node-gyp, short for generate your projects, is a tool that lets us build native add-ons for Node.js. Due to historical reasons, which we’ll outline in this article, using node-gyp can easily result in problems that are hard to resolve.
In this article, we’ve grouped all the major issues with node-gyp into three categories, including dependencies of node-gyp, add-on compilation errors, and binding contract violation. We’ll review each category one-by-one, offering solutions along the way.
If you aren’t already familiar with node-gyp or haven’t run into these issues yourself, I recommend reviewing the project’s README file first. Let’s get started!
Before we start, it’s important to note that if you have Node.js installed, you probably already have node-gyp installed but not exposed globally. If you only need to compile add-ons during the project setup, Node.js should cover it for you. However, if you are an add-on developer, you probably need to install node-gyp globally.
To use node-gyp, first, we’ll need to install a Python runtime, the
make utility, and a C or C++ compiler. Already, we run into an issue with the first requirement. node-gyp expects Python ≥v3.6, not Python v2.x. If you’re not a Python developer, you might not realize that Python v.3 isn’t backward-compatible with its previous major version.
Additionally, many modern operating systems provide default runtimes from the first boot-up, however, these runtimes may not comply with node-gyp’s requirements. Be sure to properly install the correct dependencies for your system before starting with node-gyp.
Lastly, many software developers tend to miss the CPU architecture, frequently assuming that the 64-bit version of the x86 instruction set will work. But as new CPUs constantly appear on the market, this could not be further from the truth.
Add-on compilation errors
Usually, developers write add-ons for Node.js in C with N-API bindings or C++ with NAN bindings. We’ll cover bindings in depth later on, but for now, keep in mind that there are at least two ways to bridge the add-on code to the Node.js runtime.
Usually, when you install an external dependency that uses node-gyp, node-gyp will download pre-built binaries that match your architecture and the Node.js version on your system. node-gyp will compile the add-on code only when it doesn’t find a match, which can cause problems.
Most programming languages are backward-compatible, or at least have a minor release version that is. Imagine that the author of an add-on wrote the code but used a higher version of a language than the one supported on your system. You simply would not be able to compile the add-on.
To solve compilation errors, you can either fork its codebase or upgrade or downgrade your compiler toolchain.
Binding contract violation
Previously, we mentioned two types of bindings, NAN, Native Abstraction for Node.js, and Node-API, formerly known as N-API. The third type of bindings include V8, libuv, and Node.js header files.
The first binding method, NAN, provides header files that enable add-ons to interact directly with the V8 runtime. It is similar to using plain V8 header files, just with extra precaution. Since every Node.js release might use a different V8 version, add-ons that use NAN indirectly depend on the V8 version utilized by their users.
If an add-on code expects more or different functionality from V8 than what it provides, the add-on can fail during the linking phase. Or worse, linking may succeed, only for the add-on to crash during runtime due to the presence of incorrect header files.
Sadly, there is no simple solution for binding contract violations. If you’re the code’s author, you could adjust it, otherwise, you may need to upgrade or downgrade your Node.js version. Later in the article, we’ll cover a few more generic ways to deal with similar issues.
You may have already realized that the problems reported as node-gyp issues are, in fact, issues with the toolchain used for building add-on code. Now, let’s cover how to categorize the issue to apply the correct solution. When you run into a problem, you can ask yourself the following question:
- Who is the issue reporter? Is it an add-on publisher or an add-on consumer?
- What is the architecture and the toolchain of your machine?
- What are the target Node.js versions?
- Does the add-on use NAN or Node-API?
- Which programming language does the add-on use?
Simply put, the add-on publisher has more tools at their disposal when fixing potential problems than a consumer. The architecture and the toolchain influence the whole build process. The target Node.js versions might impair the way NAN cooperates, and some low versions do not support Node-API. The programming language of the add-on influences the contact point between Node and the NAN/Node-API bindings.
It’s worth noting that one installation of an add-on that uses node-gyp for building could trigger many issues, each in different categories.
Checking the architecture and the toolchain
Checking the architecture and toolchain requires knowledge about the software and hardware of your machine. You should acquire information about your operating system and the CPU/GPU. You likely work on an x86-64-compatible instruction set, but it’s better to verify this information.
To check the version of Python 3 currently used on your terminal, run the following command:
In your terminal, run the command below to check what version of the C/C++ compiler toolchain you’re using:
To check the Node.js version on your machine, use the following code:
You can dynamically change the executed Node.js version with a Node.js version manager, like either n, available on Linux and macOS, or nvm. However, to avoid using each tool in an automated way, which could cause some of the issues we’ve previously listed, we’ll check the add-on bindings.
Checking the add-on bindings
We need to have access to the codebase of the said add-on. The most important file we need to look into is the
binding.gyp file, usually located in the root directory of the add-on. We should search for all
require statements in it;
require("nan") indicates that the add-on uses NAN, and
require("node-addon-api") indicates that the add-on uses Node-API.
We can also search the entire project for either NAN or Node-API macros,
#include <nan.h> and
#include <node_api.h>. Some projects may also use V8 header files, which use
#include <v8.h> macros.
Checking the programming language
You should see the programming language by the filename extensions. C uses
.h for header files and
.c for source files. On the other hand, C++ uses
.hxx for header files and
.cxx for source files.
Add-ons that use the
node-addon-api module are C++ projects. Otherwise, it would be pointless to get declarations of C++ wrapper classes with no intention of using them.
Generic troubleshooting strategies
At this point, if you’ve implemented the above solutions and still haven’t solved the problem you’re facing, we’ve prepared a few generic ideas to try out.
Setting up the Node.js environment and node-gyp dependencies might not be straightforward on your operating system. As an alternative, let’s try building the add-on in a Docker image instead of your local machine, allowing you to control all the dependencies, including the operating system. If you can build the Docker image but still have trouble with your local setup, you’ll know where the problem lies.
As an example, we’ve prepared the following Dockerfile snippet, which builds the SQLite3 add-on on an Alpine Linux instance:
FROM alpine:3.14.2 RUN apk update RUN apk add --no-cache \ build-base \ curl \ git \ linux-headers \ nodejs \ npm \ py3-pip \ python3 \ python3-dev \ sqlite RUN pip3 install --upgrade pip WORKDIR /opt/showcase RUN npm init -y RUN apk add sqlite RUN npm install sqlite3 --verbose --build-from-source
You can save it as
Dockerfile in the directory of your choice and run with
docker build .
We used Alpine Linux because of its lightweight nature. You may have noticed that we had to add our dependencies by hand. As an exercise, you can remove some dependencies and see how each influences the installation of SQLite3. Additionally, you might check the proposed Dockerfile for node-rdkafka, although it is somewhat dated.
You might also try changing the architecture within the Docker image, for instance, you can choose a 32-bit version of Ubuntu. We trust you can use this hint for old projects, but please note that it is rather unlikely for you to run a 64-bit virtualized operating system on a 32-bit host.
Upgrading or downgrading a Node.js version is trivial with Docker. To speed up the process, the Node.js Docker Team provides Docker images for different versions.
If you have access to the codebase of a NAN-based add-on, you might try to switch to Node-API. It requires you to have prior experience with writing add-ons. You should as well have enough resources to undertake such a task.
Conclusion and resources
Hopefully, by the end of this tutorial, you’ve been able to resolve your issue with node-gyp. We learned how to categorize issues into three common categories by asking a series of questions, then offered unique solutions tailored to each category. We then provided a more generic, universal solution to node-gyp issues that uses a Dockerfile.
In rare instances where the steps proposed above don’t work for you, you can browse the following resources and support channels. I hope you enjoyed this tutorial and have a stronger understanding of node-gyp.
200’s only Monitor failed and slow network requests in productionDeploying a Node-based web app or website is the easy part. Making sure your Node instance continues to serve resources to your app is where things get tougher. If you’re interested in ensuring requests to the backend or third party services are successful, try LogRocket. https://logrocket.com/signup/
LogRocket is like a DVR for web and mobile apps, recording literally everything that happens while a user interacts with your app. Instead of guessing why problems happen, you can aggregate and report on problematic network requests to quickly understand the root cause.LogRocket instruments your app to record baseline performance timings such as page load time, time to first byte, slow network requests, and also logs Redux, NgRx, and Vuex actions/state. Start monitoring for free.