Solving common issues with node-gyp

node-gyp, short for generate your projects, is a tool that lets us build native add-ons for Node.js. Due to historical reasons, which we’ll outline in this article, using node-gyp can easily result in problems that are hard to resolve.

In this article, we’ve grouped all the major issues with node-gyp into three categories, including dependencies of node-gyp, add-on compilation errors, and binding contract violation. We’ll review each category one-by-one, offering solutions along the way.

If you aren’t already familiar with node-gyp or haven’t run into these issues yourself, I recommend reviewing the project’s README file first. Let’s get started!

node-gyp dependencies

Before we start, it’s important to note that if you have Node.js installed, you probably already have node-gyp installed but not exposed globally. If you only need to compile add-ons during the project setup, Node.js should cover it for you. However, if you are an add-on developer, you probably need to install node-gyp globally.

To use node-gyp, first, we’ll need to install a Python runtime, the make utility, and a C or C++ compiler. Already, we run into an issue with the first requirement. node-gyp expects Python ≥v3.6, not Python v2.x. If you’re not a Python developer, you might not realize that Python v.3 isn’t backward-compatible with its previous major version.

Additionally, many modern operating systems provide default runtimes from the first boot-up, however, these runtimes may not comply with node-gyp’s requirements. Be sure to properly install the correct dependencies for your system before starting with node-gyp.

Lastly, many software developers tend to miss the CPU architecture, frequently assuming that the 64-bit version of the x86 instruction set will work. But as new CPUs constantly appear on the market, this could not be further from the truth.

Add-on compilation errors

Usually, developers write add-ons for Node.js in C with N-API bindings or C++ with NAN bindings. We’ll cover bindings in depth later on, but for now, keep in mind that there are at least two ways to bridge the add-on code to the Node.js runtime.

Usually, when you install an external dependency that uses node-gyp, node-gyp will download pre-built binaries that match your architecture and the Node.js version on your system. node-gyp will compile the add-on code only when it doesn’t find a match, which can cause problems.

Most programming languages are backward-compatible, or at least have a minor release version that is. Imagine that the author of an add-on wrote the code but used a higher version of a language than the one supported on your system. You simply would not be able to compile the add-on.

To solve compilation errors, you can either fork its codebase or upgrade or downgrade your compiler toolchain.

Binding contract violation

Previously, we mentioned two types of bindings, NAN, Native Abstraction for Node.js, and Node-API, formerly known as N-API. The third type of bindings include V8, libuv, and Node.js header files.

V8 is a JavaScript and WebAssembly engine used by Node.js. Libuv is a C library used by Node.js to get support for asynchronous I/O.

The first binding method, NAN, provides header files that enable add-ons to interact directly with the V8 runtime. It is similar to using plain V8 header files, just with extra precaution. Since every Node.js release might use a different V8 version, add-ons that use NAN indirectly depend on the V8 version utilized by their users.

If an add-on code expects more or different functionality from V8 than what it provides, the add-on can fail during the linking phase. Or worse, linking may succeed, only for the add-on to crash during runtime due to the presence of incorrect header files.

To remedy these pitfalls, Node.js now exposes Node-API, a stable API independent from the JavaScript runtime. However, this method is not perfect either. The target Node version needs to implement Node-API in the version used by the add-on. To know which version of Node-API you can use against a particular Node.js installation, you should check the version matrix.

Sadly, there is no simple solution for binding contract violations. If you’re the code’s author, you could adjust it, otherwise, you may need to upgrade or downgrade your Node.js version. Later in the article, we’ll cover a few more generic ways to deal with similar issues.

Issue categorization

You may have already realized that the problems reported as node-gyp issues are, in fact, issues with the toolchain used for building add-on code. Now, let’s cover how to categorize the issue to apply the correct solution. When you run into a problem, you can ask yourself the following question:

• Who is the issue reporter? Is it an add-on publisher or an add-on consumer?
• What is the architecture and the toolchain of your machine?
• What are the target Node.js versions?
• Does the add-on use NAN or Node-API?
• Which programming language does the add-on use?

Simply put, the add-on publisher has more tools at their disposal when fixing potential problems than a consumer. The architecture and the toolchain influence the whole build process. The target Node.js versions might impair the way NAN cooperates, and some low versions do not support Node-API. The programming language of the add-on influences the contact point between Node and the NAN/Node-API bindings.

It’s worth noting that one installation of an add-on that uses node-gyp for building could trigger many issues, each in different categories.

Checking the architecture and the toolchain

Checking the architecture and toolchain requires knowledge about the software and hardware of your machine. You should acquire information about your operating system and the CPU/GPU. You likely work on an x86-64-compatible instruction set, but it’s better to verify this information.

To check the version of Python 3 currently used on your terminal, run the following command:

python3 --version

In your terminal, run the command below to check what version of the C/C++ compiler toolchain you’re using:

gcc --version

To check the Node.js version on your machine, use the following code:

node --version

You can dynamically change the executed Node.js version with a Node.js version manager, like either n, available on Linux and macOS, or nvm. However, to avoid using each tool in an automated way, which could cause some of the issues we’ve previously listed, we’ll check the add-on bindings.

Checking the add-on bindings

We need to have access to the codebase of the said add-on. The most important file we need to look into is the binding.gyp file, usually located in the root directory of the add-on. We should search for all require statements in it; require("nan") indicates that the add-on uses NAN, and require("node-addon-api") indicates that the add-on uses Node-API.

For further clarification, you can see examples on GitHub of binding.gyp files in NAN and Node-API.

We can also search the entire project for either NAN or Node-API macros, #include <nan.h> and #include <node_api.h>. Some projects may also use V8 header files, which use #include <v8.h> macros.

Checking the programming language

You should see the programming language by the filename extensions. C uses .h for header files and .c for source files. On the other hand, C++ uses .h, .hpp, and .hxx for header files and .cpp and .cxx for source files.

Add-ons that use the node-addon-api module are C++ projects. Otherwise, it would be pointless to get declarations of C++ wrapper classes with no intention of using them.

Generic troubleshooting strategies

At this point, if you’ve implemented the above solutions and still haven’t solved the problem you’re facing, we’ve prepared a few generic ideas to try out.

Setting up the Node.js environment and node-gyp dependencies might not be straightforward on your operating system. As an alternative, let’s try building the add-on in a Docker image instead of your local machine, allowing you to control all the dependencies, including the operating system. If you can build the Docker image but still have trouble with your local setup, you’ll know where the problem lies.

As an example, we’ve prepared the following Dockerfile snippet, which builds the SQLite3 add-on on an Alpine Linux instance:

FROM alpine:3.14.2
RUN apk update
RUN apk add --no-cache \
   build-base \
   curl \
   git \
   linux-headers \
   nodejs \
   npm \
   py3-pip \
   python3 \
   python3-dev \
   sqlite
RUN pip3 install --upgrade pip

WORKDIR /opt/showcase

RUN npm init -y
RUN apk add sqlite
RUN npm install sqlite3 --verbose --build-from-source

You can save it as Dockerfile in the directory of your choice and run with docker build .

We used Alpine Linux because of its lightweight nature. You may have noticed that we had to add our dependencies by hand. As an exercise, you can remove some dependencies and see how each influences the installation of SQLite3. Additionally, you might check the proposed Dockerfile for node-rdkafka, although it is somewhat dated.

You might also try changing the architecture within the Docker image, for instance, you can choose a 32-bit version of Ubuntu. We trust you can use this hint for old projects, but please note that it is rather unlikely for you to run a 64-bit virtualized operating system on a 32-bit host.

Upgrading or downgrading a Node.js version is trivial with Docker. To speed up the process, the Node.js Docker Team provides Docker images for different versions.

If you have access to the codebase of a NAN-based add-on, you might try to switch to Node-API. It requires you to have prior experience with writing add-ons. You should as well have enough resources to undertake such a task.

Conclusion and resources

Hopefully, by the end of this tutorial, you’ve been able to resolve your issue with node-gyp. We learned how to categorize issues into three common categories by asking a series of questions, then offered unique solutions tailored to each category. We then provided a more generic, universal solution to node-gyp issues that uses a Dockerfile.

In rare instances where the steps proposed above don’t work for you, you can browse the following resources and support channels. I hope you enjoyed this tutorial and have a stronger understanding of node-gyp.

Resources:

• https://nodejs.org/api/addons.html
• https://nodejs.org/api/n-api.html
• https://github.com/nodejs/node-addon-api
• https://github.com/mapbox/node-pre-gyp

Support:

• StackOverflow
• GitHub Issues of NAN
• GitHub Issues of Node-API
• Node.js Add–on API Working Group