This post will cover the following topics:
- The history and philosophy behind Node.js
- Why task runners were developed for Node.js
- Different approaches taken by popular task runners
- How bash may be used as a simpler alternative
A brief history of Node.js
The tooling ecosystem for JavaScript is incredible. You’ll be hard pressed to find any other language with as much tooling or as many different users contributing to this tooling. From its humble beginnings as a language originally designed in 10 days to its C10K-achieving foothold in the server space, you will not find another language as malleable as this one.
Node.js, the popular server-side implementation of JavaScript, was first introduced in 2009. This platform, nearly overnight, allowed frontend developers to quickly become viable as backend developers, unblocking front-end teams everywhere. Its success warranted a tool for easily distributing source code and, in 2010, this need was satisfied by npm.
Node.js is heralded as being fast, approachable, and perhaps most alluring of all, simple. It began syphoning users from other platforms. One such platform is PHP — a language created to generate dynamic websites. PHP has perhaps thousands of global functions available at any time and requires a stack of configuration files.
Node.js allowed developers to migrate to the platform and get a fresh start. Being so new it hadn’t yet developed the “batteries included” frameworks of other languages. One of the guiding principles of Node.js is to keep the core simple. You won’t find built-in tools for connecting to MySQL, generating a UUID, or calculating Levenshtein distance.
The JavaScript language was transforming as well. Some features are backwards compatible thanks to user-land “polyfills”. But, in order for a language to advance, it simply must add the occasional new syntax. Developers yearn for new syntax, yet old browsers are the reality, which led to the development of transpilers.
The simplicity of working with Node.js was eventually dwarfed in importance by the fact that code is written in JavaScript, the lingua franca of the web. Node.js gained more and more traction as a tool for transforming frontend assets from one representation, such as ES7 or SASS, to another representation, such as ES5 or CSS. There was just one catch, though. JavaScript engineers typically want to keep writing JavaScript. This led to the development of task runners: specialized Node.js tools designed to run other tools.
The rise of the task runner
There are essentially three technologies required to construct a website, each of which is consumed directly by the browser. The first is HTML, controlling the structure of a webpage. The second is CSS, controlling the appearance of a webpage. And finally, we have JavaScript, which is used for programming website interactivity.
For simpler websites, or small teams, working with these languages directly is usually a fine approach. However, with complex websites, or websites being built by teams of engineers, each with their own specializations, working directly with these basic languages can start to fall short. Consider, for example, when the branding for a corporate website changes. A hexadecimal color code used in several different style files may need to be changed. With raw CSS this operation would require orchestrated changes across a few teams. With SASS, such a change could be made in a single line. Similar concepts apply to HTML where we generate markup using templating tools like Mustache or virtual DOMs like React. They also apply to JavaScript, where an engineer may write code using the async/await ES2017 syntax which then gets transpiled into a complex ES5 switch statement with callbacks.
At this point, we may have a site which needs to have SASS compiled into CSS, ES2015 code which needs to be transpiled into ES5, and React/JSX templates which need to be converted into raw JavaScript. Other operations are also beneficial, such as minifying compiled code and compressing PNG images into their smallest representation. Each one of these tasks needs to be run in a particular order when a website is being built. Depending on the context of a particular website build — such as it being built for development/debugging purposes or production — some tasks must be altered or skipped entirely. Such complexity has inspired the creation of task runner tools.
Two popular Node.js task runners came to the rescue. The first is Grunt, with a first commit made in September 2011. This tool takes an imperative approach to configuring different tasks, building out deeply nested objects and calling a few methods. The second one is Gulp, having an initial commit in July, 2013. This tool takes a different approach, more functional in nature, piping the output of one function into the input of another function, streaming the results around.
Let’s consider a simple web application we’d like to mockup using a subset of these technologies. This application depends on multiple SASS and JS files. We’d like to convert the SASS files into CSS, concatenating the result. For sake of brevity, we’ll also simply concatenate the JS files together, and assume the module pattern, instead of using CommonJS require statements. Let’s see how such a configuration might look using these different task runners:
gruntfile.js
This approach requires the following modules be installed: grunt, grunt-contrib-sass, grunt-contrib-concat, and grunt-contrib-clean. With this approach, we can run grunt style, grunt script, or grunt build to do the work of both.
gulpfile.js
The equivalent Gulp version of the previous Gulp example is as follows. This requires we have gulp, gulp-sass, gulp-concat, and node-sass installed. With this approach, we can run gulp style, gulp script, or gulp build to do the work of both.
As you can see, the Gulp example is a little more terse than the Grunt example.
Philosophically, the two tools take different approaches to implementing runnable tasks, but ultimately they allow you to do similar things. Again, Grunt was introduced before Gulp. They’ve both have had comparable popularity throughout their lifespans:
Both projects are highly modular, allowing developers to create specialized plugins. These plugins allow an external tool, such as eslint or sass or browserify, to easily integrate into the task runner. We actually have an example of this in the code we looked at earlier: the popular SASS tool has both a grunt-contrib-sass module, and a gulp-sass module available.
These two tools may be essentially “done”. As of this writing, Grunts last publish was made eight months ago, and Gulps last publish was a year ago. What does it mean to be “done”, a word which is both literally and figuratively a four letter word in the JavaScript community? Well, in this case, it probably means the core task runner modules do everything they need to do and that any additional functionality can be added via plugin.
Webpack is a tool that is similar to Grunt and Gulp in that it can also be used to take source files, combine them in various ways, and output them into single files. However, it’s different enough that it wouldn’t be fair to compare it against Grunt and Gulp. It is primarily a tool for transforming JavaScript, based on requires and a hierarchy of dependencies. It’s definitely worth mentioning as its popularity has surpassed that of Grunt and Gulp.
The first commit to Webpack happened in March 2012, between the first commits to Grunt and Gulp. As of this article being written, it is still under very active development and its last contribution occurred a few hours ago. Whereas Grunt and Gulp aide in performing many types of generic tasks, Webpack is specifically more interested in building frontend assets.
Webpack can also be configured in a manner similar to Grunt and Gulp using a file called webpack.config.js. It is also highly modular and we can achieve similar results using plugins like sass-loader. It has its own philosophical differences from the aforementioned tools. But, it’s still similar in the sense that a Node.js based process ultimately transforms assets and is configured via a JavaScript file.
Task runner alternatives
For the most complex of build systems, it makes total sense to use a Node.js Task Runner. There’s a tipping point where the build process can get so complex that maintaining it in a language other than the one the application is written in just doesn’t make sense. However, for many projects, these Task Runners end up being overkill. They are an additional tool that we need to add to a project and keep up to date. The complexity of Task Runners is easy to overlook when they’re so readily available via npm install.
With the previous examples, we looked at we needed 32MB of modules to use Grunt and 40MB of space to use Gulp. These simple build commands —concatenate two JavaScript files and compile/concatenate two SASS files— takes 250ms with Grunt and 370ms with Gulp.
The approach used by Gulp of taking outputs from one operation and piping them into another operation should sound familiar. The same piping system is also available to us via the command line, which we can automate by use of bash scripts. Such scripting features are already available to users of macOS and Linux computers (WSL can help with Windows).
We can use the following three bash scripts to achieve what our Grunt and Gulp examples are doing:
Shell scripts
When we use this approach we’ll only need a 2.5MB sass binary (executable). The time it takes to perform the entire build operation is also lessened: on my machine the operation only takes 25ms. This means we’re using about ~1/12 the disk space running 10x as fast. The difference will likely be even higher with more complex build steps.
package.json
They can even be in-lined inside of your package.json file. Then commands can be executed via npm run style, npm run script, and npm run build.
This is, of course, a trade-off. The biggest difference is that bash is a shell scripting language with a syntax completely unlike JavaScript. It may be difficult for some engineers who work on a JavaScript project to write the appropriate scripts required to build a complex application.
Another shortcoming is that bash scripts require that some sort of executable is available for each operation we want to incorporate. Luckily for us they usually are. Browserify, a tool for resolving CommonJS requires and concatenating output, offers an executable. Babel, the go-to transpiler, also offers an executable. Sass, Less, Coffeescript, JSX: each of these tools has an executable available. If one isn’t available we can write it ourselves, however, once we reach that point we might want to consider using a task runner.
Conclusion
The command line scripting capabilities of our machines are very powerful. It’s easy to overlook them, especially when we spend so much time in a higher level language like JavaScript. As we’ve seen today they are often powerful enough to complete many of our frontend asset building tasks and can often do so faster. Consider using these tools when you start your next project, and only switch to a heavier solution like a task runner if you reach a limitation with bash scripting.
Plug: LogRocket, a DVR for web apps
LogRocket is a frontend logging tool that lets you replay problems as if they happened in your own browser. Instead of guessing why errors happen, or asking users for screenshots and log dumps, LogRocket lets you replay the session to quickly understand what went wrong. It works perfectly with any app, regardless of framework, and has plugins to log additional context from Redux, Vuex, and @ngrx/store.
In addition to logging Redux actions and state, LogRocket records console logs, JavaScript errors, stacktraces, network requests/responses with headers + bodies, browser metadata, and custom logs. It also instruments the DOM to record the HTML and CSS on the page, recreating pixel-perfect videos of even the most complex single page apps.
