Using Node.js to create an HTTP proxy for IPFS content

In this article, we’ll demonstrate how to create a Node.js server to access IPFS content via the clearnet. Before we explain how to do this, let’s define the key terms and create some context for why this matters.

What is the clearnet?

The clearnet is essentially the publicly accessible internet. It refers to all web content consumed over public networks using standard protocols supported by all browsers, like HTTP, WebSocket (used for real-time communications), and WebRTC (used for most audio and video streaming platforms).

These protocols often require central servers and service providers to work, which can potentially become a vector for gatekeeping, particularly in countries with restrictions on free speech and privacy.

Beyond the clearnet

There are a number of other non-standard network protocols, although they may not be supported by all browsers. Examples include:

• Onion routing: a technique that enables anonymous communication over a network, notably implemented in the Tor project
• Gemini: a communication protocol that strives to a be an alternative to Gopher and HTTP. There are a number of Gemini clients available
• InterPlanetary File System (IPFS): a protocol and peer-to-peer network for sharing data. IPFS is supported by the popular Chromium-based Brave browser

IPFS explained

Similar to BitTorrent, IPFS is a protocol for storing and sharing data in a distributed file system. The key difference, however, is that IPFS aims to create a single global network. In IPFS, an individual file that is hosted is assigned a content ID (CID), which is used to access it.

To illustrate the difference, in HTTP, we use a URI like https://www.google.com, which is then sent to a DNS server. The server identifies the resource’s IP address and serves the resource. If that host is turned off or otherwise inaccessible, however, that resource is just… gone.

In IPFS, we use a CID like ipfs://xjd9809bnuiue900sdfnuiwerwwer, which is identified by searching through the distributed hash table (DHT) hosted by all the participants in the network. Instead of identifying one single provider of the resource, all hosts who have a copy of the resource are identified, and we download portions of the resource from all of them.

As with any internet protocol, there are pros and cons to IPFS’ design.

IPFS pros

• Once content has been added to the DHT, as long as there’s a host connected to the IPFS network with a copy of the file, that file is accessible on the network
• Over time, the accessibility of the file grows since every user who accesses the file becomes another host to retrieve that file
• No external entity can singlehandedly remove a file that has been added to the network

IPFS cons

• When you initially add content to IPFS, you’re the only host. As such, accessibility is limited at the outset
• You can’t mutate files, so if you need to fix a type, you’ll have to add a new copy of the file with a new CID and alert others users to go to the new CID
• Not currently supported by all browsers

Getting started with IPFS

Let’s install the IPFS CLI and add some content to IPFS. You can reference the IPFS documentation as we progress.

First, install IPFS. For Linux and Mac, you can install it with Homebrew using brew install ipfs. When it completes, make sure to read the output for important information; for Linux, it should give you the exact path for running the daemon.

Once installed, run ipfs init and, again, make sure to read the output for some important details — you may want to save it somewhere safe. The output should share a cat command to see the documentation; run that command to confirm everything has been set up.

Start the daemon with ipfs init. To add files to IPFS, just use the command ipfs add FILENAME and you’ll receive the CID. Easy as that!

Consuming IPFS content

To consume IPFS content, simply run ipfs cat /ipfs/CONTENTID in the command line to retrieve the content.

To consume IPFS content via browser, you’ll need to use Brave, which allows you to view content from IPFS. In the browser, use this pattern ipfs://CONTENTID to access content. For example, here’s a link to a “Hello, World!” text file I posted to IPFS: ipfs://QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u.

Delivering IPFS content over HTTP

IPFS is pretty cool, but most everyday internet users don’t want to download an alternative browser or use a CLI tool to browse resources on the web. So maybe you want to deliver your IPFS content to clearnet users by creating a proxy application to fetch the content in a mainstream browser. Let’s do that with Node.js.

N.B., this demo application must be running on a machine that is also running an IPFS server. This means you’ll want to use a virtual private server (VPS) for deployment, where you can run an IPFS server parallel to your Node.js application.

Setting up the app

Start by creating a new folder — you could name it ipfs-proxy or similar — and change directories into that folder. Create a server.js file by running touch server.js, then initiate a new Node.js project with npm init -y. Lastly, install Fastify (although any web framework will do) and ipfs-http-client by running npm install fastify ipfs-http-client.

Getting the Node server running

Add the following code to your server.js file:

// import fastify
const fastify = require("fastify");

// create the server object
const server = fastify({ logger: true });

// example route
server.get("/", async (request, reply) => {
  return { message: "The Server is Working" };
});

// start server function
const start = async () => {
  await server.listen(3000).catch((err) => {
    fastify.log.error(err);
    process.exit(1);
  });
  console.log("Listening on port 3000")
};

// turn server on
start();

Run the file node server.js, go to localhost:3000 in your browser, and make sure you see the message. Turn the server off with Control+C in the terminal.

[H3] Using the ipfs-http-client

We need to update our server.js to the code below, but first let’s walk through what we’re doing.

First, we’ll import the ipfs-http-client library as ipfs. We then create a client that connects to localhost:5001, which is the default port our local IPFS server should be running on.

We use the client’s get method to get the data CID. The function returns an Uint8Array of bytes wrapped in an AsyncGenerator (fun!).

We use a for await loop to loop over the generator, then use a text encoder to decode the array of bytes into a string. After the loop, the resulting text has several null characters in it, so we remove them using string.replace. Finally, we send back the resulting string as JSON.

// import fastify
const fastify = require("fastify");
// import ipfs-http-client
const ipfs = require("ipfs-http-client")

// connect to local ipfs node
const client = ipfs.create()

// create the server object
const server = fastify({ logger: true });

// example route
server.get("/", async (request, reply) => {

  // fetch the content from ipfs
  const result = await client.get("QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u")

  // create a string to append contents to
  let contents = ""


  // loop over incoming data
  for await(const item of result){
      // turn string buffer to string and append to contents
      contents += new TextDecoder().decode(item)
  }

  // remove null characters
  contents = contents.replace(/\0/g, "")

  // return results as a json
  return { message: contents };
});

// start server function
const start = async () => {
  await server.listen(3000).catch((err) => {
    fastify.log.error(err);
    process.exit(1);
  });
  console.log("Listening on port 3000")
};

// turn server on
start();

Notice the results aren’t the most friendly in the world. The resulting string includes the CID and a timestamp; in other words, additional info along with the contents of my “Hello, World!” file. If this were an HTML file, I’d probably have to do some additional cleanup to remove all the non-HTML, but at least we’re getting the content.

So, theoretically, we could host this app on a VPS that also has IPFS installed, and users on the clearnet/HTTP can see our content without having to download a special browser or command line tool.

Conclusion

IPFS presents some really interesting ideas on how the internet can be more decentralized and free, but it does come at the cost of some of convenience. With clever use of Node.js, however, we can start to bridge that gap for the masses.