The evolution of asynchronous programming in JavaScript

Introduction

By design, JavaScript is single-threaded, which means that it can only handle one operation at a time. Because there is a single execution thread for our program to run, a question then arises: How do we go about executing a long-running operation without blocking the thread of execution? Well, welcome to asynchronous programming.

Asynchronous programming in JavaScript offers a great way of handling operations (I/O) that are not immediately executed and therefore have no immediate response. Rather than waiting for long-running operations to return, blocking the execution thread in the process, they are delegated to callbacks, which are functions that are called when these operations finally return.

An execution thread in this case helps to keep track of an active running operation called a sub-routine, and when that sub routine should return control to its calling sub-routine after execution.

Nowadays, there are a bunch of applications that require one form or another of async behavior. Making network or AJAX requests offers a very good use case when it comes to explaining these concepts in JavaScript.

In this article, we will use callbacks, promises, and async/await to illustrate the concepts of async JavaScript and explain how they work.

JavaScript’s nature and why we need async programming

Earlier, we learned that JavaScript is single-threaded with a global execution context. This means that, by nature, JavaScript is synchronous with a single Call stack. Therefore, code will be executed in the order it is called, commonly known as the last-in, first-out (LIFO) method.

For example, say we were to execute two functions, A and B, where function B depends on the output of A to run. Supposing function A takes some time to return with the output needed for function B to start executing, we end up blocking the thread of operation for our program. This sort of behavior leads to a very slow application, which is detrimental to user experience.

Let’s take a look at an example of a synchronous or blocking operation in JavaScript.

const fs = require('fs')

const A = (filePath) => {
  const data = fs.readFileSync(filePath) 
  return data.toString()
}

const B  = () => {
  const result = A('./file.md')
  if (result) {
    for (i=0; i < result.length; i++) {
       console.log(i)
    }
  }
  console.log('Result is back from function A')
}

B()

// output is shown below
0
1
2
3
4
5
6
7
8
9
10
Result is back from function A

In the example above, we are waiting for the output of the function A() on line 9 before we continue executing the code logic in the B(). For now, this is fine — well, until we have to read a very large file. In that case, it would take a very long time to wait for A() to finish before we get the input needed for B() to execute. Again, this is not advisable,

Note 1: Based on the output from above, B() is invoked and pushed to the top of the call stack. After it is done executing all its code logic synchronously — which includes executing the function A() — it is popped off the stack and the thread is free again for use. Here is a link to the playground to run the example code.

Note 2: The readFileSync function is an inbuilt method in the fs module in Node.js. It reads synchronously from a file input with a specified path.

Therefore, for a synchronous call or operation, the event loop is unable to continue executing any other JavaScript code until that operation is done.

What does async solve in JavaScript?

Asynchronous programming makes it possible to have many input/output operations, all happening at the same time. For JavaScript, this is made possible via the event loop, the call stack, and async APIs like callbacks.

Let’s look at an example of an asynchronous operation to understand better:

const fs = require('fs')

const A = (filePath, callback) => {
  return fs.readFile(filePath, (error, result) => {
    if (error) {
    return callback(error, null)
    }
    return callback(null, result)
  })
}

const B  = () => {
   // a callback function attached
  A('./file.md',  (error, result) => {
    if (result) {
    for (i=0; i < result.length; i++) {
       console.log(i)
    }
  }
})
  console.log('Result is not yet back from function A')
} 

B()

// output is shown below
Result is not yet back from function A
0
1
2
3
4
5
6
7
8
9
10

Here is a link to the playground to run the code example above. As we can see, we have defined an asynchronous callback. Therefore, function A() is not executed immediately when B() is invoked.

Instead, it does so only after the Node.js readFile module has finished parsing and reading the contents of the file. Therefore, when the code is run, we see that the code on line 21 with the output Result is not yet back from function A is immediately executed.

In forthcoming sections, we will get to learn about callbacks and other async patterns that have evolved over time in JavaScript. But before that, let’s touch on the event loop.

The JavaScript event loop

From our earlier discussion, JavaScript handles non-blocking input/output operations via event-based callbacks. In this section, we are going to understand the execution order of our code via the event loop, the call stack, and the callback API, which is the earliest async API in JavaScript for both browser and Node.js.

According to MDN, callbacks and the concurrency model in JS are made possible via the event loop. The event loop takes care of executing our code, handling events like callbacks, and also scheduling other queued tasks for execution. Let’s use our previous callback example to demonstrate how the event loop works.

• To begin, before the function B() is executed, the call stack and the event loop are empty.
• When the function B() is being executed, it is then pushed onto the call stack.
• Since the first function A() on line 14 has a callback attached, it is pushed to the callback queue for processing.
• Meanwhile, the last line, console.log('Result is not yet back from function A'), is executed and leaves the call stack.
• After the function A() is done and we have a response, the execution is then moved to the event loop.
• At this point, the call stack is empty, so the JS execution context checks the event loop for any queued task.
• The chain of control now moves the function A() from the event loop to the call stack, where it is then executed with the response (result) returned.
• At this point, the result is now available, and the call stack is empty again.
• The for loop is then moved to the call stack for execution.
• On every iteration of the for loop, the console.log on line 17 is moved to the call stack for execution until done.
• Lastly, because execution is now complete, the function B() is taken off the call stack, which ends the typical flow.

The event loop and the call stack

The event loop acts as a bridge that keeps track of the call stack and the callback queue. When the call stack is empty, the JS execution environment occasionally checks to see if anything is queued for execution. If it is, the event loop takes the first task from the queue (FIFO) and moves it to the call stack, which then executes our code.

The call stack is a stack data structure that helps to keep track of currently running or executing functions in our program. For stack data structures, the last item pushed onto the stack is the first item that leaves — more like LIFO.

A final point of note here is that while callbacks are not part of JavaScript’s engine implementation, they are APIs made available for both browser and Node. These APIs do not push code execution directly onto the call stack, as that could interfere with code that’s already executing, hence the event loop.

Callbacks

Callbacks are one of the earliest approaches for handling async behavior in JavaScript. As we have seen earlier in our async example, a callback is a function passed as an argument to another function, which is then later executed with a response.

In essence, after async operations are completed, errors or responses returned are handled by callbacks or other similar async APIs like promises or async/await in JavaScript.

Note: As a convention, the first argument passed to a callback is the error, with the reason that error occurred, while the second argument is the response data or the result.

Again, creating a callback can be as simple as the example below. Here’s a link to the playground to run the code.

const callbackExample = (asyncPattern, callback) => {
  console.log(`This is an example, with a ${asyncPattern} passed an an argument`)
  callback()
}

const  testCallbackFunc = () => {
  console.log('Again, this is just a simple callback example')
}

// call our function and pass the testCallbackFunction as an argument
callbackExample('callback', testCallbackFunc)

Problems with callbacks

It should be noted that since the result of each async behavior happens on its own call stack, error handlers might not be on the call stack at the time an exception is thrown. This might lead to errors not properly propagated to the calling functions

Also, there is the issue of the dreaded “callback hell” — too many nested callback functions tangled like spaghetti. When this happens, failures don’t get reported to the right callback, as we might even forget to handle all errors in each callback. This can be especially confusing for new developers.

const fs = require('fs')

const callbackHell = () => {
  return fs.readFile(filePath, (err, res)=> {
    if(res) {
      firstCallback(args, (err, res1) => { 
        if(res1) {
          secondCallback(args, (err, res2) => {
            if(res2) {
              thirdCallback(args,  (err, res3) => {
                  // and so on...
              }
            }
          }
        }
      }
    } 
  })
}

A typical callback hell is shown in the above example. One approach to handling these issues is splitting the callback into smaller functions, like we have done in the previous example. Additionally, promises and async/await can solve some of the associated challenges.

Converting a callback to a promise

Using our earlier callback-based example, in this section, we are going to promisify it — rewrite it to use a promise instead. Let’s go:

const fs = require('fs')

const A = (filePath) => {
  const promise = new Promise((resolve, reject) => {  
  return fs.readFile(filePath, (error, result) => {
    if (error) {
    reject(error)
    }
    resolve(result)
   })
 })
  return promise
}

const B  = () => {
  A('./file.md').then((data)=>{
     if(data) {
      for (i=0; i < data.length; i++) {
        console.log(i)
     }
   }
 }).catch((error)=>{
    // handle errors
    console.log(error)
  })
  console.log('Result is not yet back from function A')
}  

B()

// output as above
Result is not yet back from function A
0
1
2
3
4
5
6
7
8
9
10

Note: As we have seen above, we have been able to convert our earlier example from a callback to a promise using the Promise() constructor. We will explore promises in depth in the next section.

Converting a callback to a promise is even easier in Node since there is improved support for promises via the inbuilt util.promisify() API. Here is a link to the playground to run the code.

Promises

A promise is an object representing the eventual completion or failure of an async call. What this means is that, just like callbacks, promises intuitively help us handle both the error and success responses of actions that are not immediately executed, albeit in a nicer, cleaner way.

Standardized in the ES2015 spec, a promise is a wrapper function around regular callback functions. To construct a promise, we make use of the Promise() constructor, as seen in our earlier example of converting or promisifying a callback to a promise.

The Promise() constructor takes two parameters: resolve and reject, which are both callbacks. We can run an async action within the callback, then resolve if it’s successful or reject if there’s a failure. Here’s how we’d declare a promise using the constructor:

const promiseExample = new Promise((resolve, reject) => {
    // run an async action and check for the success or failure
    if (success) {
      resolve('success value of async operation')
    }
    else {
      reject(throw new Error('Something happened while executing async action'))
  }
})

The function above returns a new promise, which would initially be in a pending state. The resolve and the reject act as callbacks in this case. When a promise resolves with a success value, we say it is now in a fulfilled state. On the other hand, when it returns with an error or is rejected, we say it is in a rejected state. In order to make use of the above promise:

promiseExample.then((data) => {
  console.log(data) // 'success value of async operation'
}).catch((error) => {
  console.log(error) // 'Something happened while executing async action'
}).finally(() => {
  console.log('I will always run when the promise must have settled')
})

Note: In the above example, the finally block helps to handle other stuff — for example, cleanup logic — when the promise is settled or has completed operation. It is not meant to process a promise result, but rather process any other cleanup code.

Additionally, we can manually convert a value into a promise, like below:

const value = 100

const promisifiedValue = Promise.resolve(value)

console.log(promisifiedValue)

promisifiedValue.then(val => console.log(val)).catch(err => console.log(err))

//output below
Promise { 100 }
Promise { <pending> }
100

Note: This also applies to rejecting promises by using Promise.reject(new Error('Rejected')).

Promise.all

Promise.all returns a promise that waits for all promises in the array to resolve and then resolves to an array of the values that these promises return, usually in the same order as the original array. If any promise in the array is rejected, the result of Promise.all is itself rejected. The signature is shown below:

Promise.all([promise1, promise2]).then(([res1, res2]) => console.log('Results', res1, res2))

In the above, promise1 and promise2 are functions that both return a promise. To learn more about Promise.all, take a look at the awesome documentation for promises in the MDN docs.

Promise chaining

One of the sweet spots of working with promises is chaining. We can chain a couple of then together to transform a returned value from an earlier promise or run other additional async actions one after another. Using our earlier example, let’s see how we can chain promises below:

const value = 100

const promisifiedValue = Promise.resolve(value)

promisifiedValue.then( (val) => {
  console.log(val) // 100
  return val + 100
}).then( (val) => {
  console.log(val) // 200
})
// and so on

Problems with promises

The most prolific promise anti-patterns out in the wild are:

• The deferred anti-pattern, when “deferred” objects are created for no reason, making your code verbose and hard to reason about
• The .then (success, fail) anti-pattern, using promises as glorified callbacks

More details about these topics can be found here via the Bluebird wiki.

Async/await

Over the years, JavaScript evolved from callbacks to promises, which were standardized in ES2015, to async/await, standardized in ES2017. Async functions allow us to write an asynchronous program as if it were synchronous. It’s especially important that we just covered promises in the previous section because async functions use promises under the hood.

Therefore, understanding how promises work is key to understanding async/await.

The signature of an async function is marked by the word async before the function keyword. Additionally, methods can be made async by writing async before their name. When such a function or method is called, it returns a promise. As soon as it returns, the promise is resolved; if an exception is thrown, the promise is rejected.

Every asynchronous function is actually an AsyncFunction object. For example, let’s say we have an async function that returns a promise:

const asyncFun = () => {
  return new Promise( resolve => {
    // simulate a promise by waiting for 3 seconds before resolving or returning with a value
    setTimeout(() => resolve('Promise value returned'), 3000)
  })
}

Now we can wrap the above promise with an async function and await the result of the promise inside the function. The code snippet is shown below:

// add async before the func name
async function asyncAwaitExample() {
  // await the result of the promise here
  const result = await asyncFun()
  console.log(result)  // 'Promise value returned' after 3 seconds
}

Note that in the above example, the await will pause the execution of the promise until it is resolved. More details about async/await can be found here via MDN.

What async/await solves

Async/await offers a much cleaner syntax when it comes to handling async behavior. While promises come with a lot of boilerplate, async functions build an abstraction on top of it. Therefore, async functions are just syntactic sugar over regular promises. In summary, for async functions:

1. The resulting code is much cleaner and, thus, easier to debug
2. Error handling is much simpler, as it relies on try…catch just like in any other synchronous code, and so on.

Top-level await

Top-level await, which is currently at stage 3 in the ECMAScript specification, allows developers to use the await keyword outside of an async function. Before now, this was not a supported feature of the language, both for browser and Node.

So, from our earlier example on async/await, if we had done this:

// here the returned `asyncFun()`promise is not wrapped in an async
const result = await asyncFun()

console.log(result) 
// this would throw a SyntaxError: await is only valid in async function

Before now, in order to simulate this kind of behavior, we utilized immediately invoked function expressions:

const fetch = require("node-fetch")
(async function() {
  const data = await fetch(url)
  console.log(data.json())
}())

In essence, since we are used to async/await in our code, it is now possible to use the await keyword alone, imagining that a module can act as a big async function in the background.

With this new top-level await feature, the snippet below works the way you’d expect an async/await function to work. In this case, it enables ES modules to act as global async functions.

const result = await asyncFun()

console.log(result)  // 'Promise value returned'

Note: To learn more about the use cases and the caveats of the top-level await feature, we can take a peep at the V8 doc here.

Async vs. parallelism in JavaScript

As we discussed earlier, JavaScript has a concurrency model based on the event loop and async APIs. On the other hand, web workers, supported by major browsers, make it possible to run an operation in a background thread in parallel, separate from the main execution thread of operation.

The Web Worker API

Async functions come with some limitations. As we learned earlier, we can make our code asynchronous by using callbacks, promises, or async/await. These browser and Node APIs really come in handy when we want to schedule and handle long-running operations.

But what if we have a highly computation-intensive task taking a long time to resolve — a very large for loop, for example? In this case, we might need another dedicated thread to handle these operations, freeing the main thread to do other work. This is where the Web Worker API comes into play. It introduces the possibility of parallel execution of our code.

Async functions come with limitations and solve only a small part of the issues associated with JavaScript’s single execution thread. Web workers execute JavaScript code without blocking the event loop by introducing a separate thread for our program to essentially run code in parallel.

Let’s use an example to understand how to create a web worker:

const worker = new Worker('file.js')

From the above, we have created a new worker with the constructor. We have also specified the path of the script to execute in the worker thread. Because they run in an isolated thread in the background, the code to be executed is contained in a separate JavaScript file.

To send messages to and from a dedicated worker, we can use the postMessage() API and the Worker.onmessage event handler. To terminate a worker, we can call the terminate() method. To learn more, check out this section and this section of the MDN docs.

Web worker limitations

Web workers are limited in the sense that they:

• Have no access to the browser DOM
• Have a different global scope, called the WorkerGlobalScope
• Compulsorily load files from the same origin

Conclusion

In this article, we have looked at the evolution of async programming in JavaScript, from callbacks to promises to async/await. We have also reviewed the Web Worker API.

We have seen that callbacks are simple functions passed to other functions and are only executed when an event is completed. We have also seen that callbacks and promises are equivalent, as callbacks can be wrapped to expose a promise-based interface, and vice versa.

Furthermore, we have seen that async functions run independently in the background, without interfering with the main thread of our application. Due to their nature, they can return with a response (data or error) whenever they are ready, thus not interfering with other running processes in our application.

We have also learned how web workers spin up a new thread separate from the main thread of execution of our program.

To learn more about these concepts, the MDN documentation on asynchronous JavaScript and other topics covered here is always a great place to start.

Thanks again for reading, and please do drop your questions and comments in the comment section below or reach out on Twitter.