A guide to working with OrbitDB in Node.js

Introduction

When it comes to a database for the decentralized web, OrbitDB is a name to know. OrbitDB is not a regular database, with one central repository for all entries; instead, it relies on a protocol for building P2P decentralized applications (DApps), where each connected peer has a specific database instance or copy.

Typical DApps rely on a distributed computing model where system components run on a P2P basis. A P2P network affords an instance where files can be replicated or synced amongst other peers residing on the same network.

There is a sort of direct communication in this kind of network architecture, as opposed to the client-server model of the centralized web, wherein a server acts as an intermediate layer for other systems to connect to and communicate with.

In an actual P2P model, there is a connection to and interaction with other peers in the global network, although peers usually start out alone, locally, and unconnected to the network.

Let’s now move on to reviewing some background terms around distributed databases with OrbitDB as a case study.

‘Distributed databases’ and other background terminologies

Just like distributed systems, distributed databases can be easily replicated and duplicated. Data is usually stored in multiple independent systems across different locations or regions, or even in a data center. Let’s look at OrbitDB in the context of distributed databases.

Firstly, it utilizes libp2p, which is a network protocol. It is a publish-subscribe (pub/sub) protocol useful for easily syncing database updates from multiple peers.

Secondly, for data homogeneity and uniformity, an OrbitDB instance must be replicated amongst peers. This happens in such a way that a peer only needs to subscribe to a database address, and it’ll automatically start replicating it.

Also, OrbitDB uses conflict-free replicated data types (CRDTs). This is a data structure whose network replicas can be updated concurrently and independently without the replicas coordinating. This means peers can go off the grid, and when they return, their state will be in sync with other peers/nodes, and all inconsistencies in the entire system will be resolved.

Lastly, for distributed databases like orbitDB, access to data can be specified for a set of peers that can write to the database. Note that, by default and if not specified by the creator of the database, only the creator will have write-access.

IPFS

OrbitDB utilizes the InterPlanetary File System (IPFS). IPFS is a protocol for storing and sharing data in a distributed file system and, according to its documentation, powers the distributed web.

It uses content addressing to give each resource a unique identifier. This means that when we add content to IPFS, it’s given an address, which usually contains a hash.

This creates a group of data that is stored and that can be subsequently accessed by connected peers. Therefore, for data retrieval, multiple peers can respond at the same time if they have the data, leading to improved performance for high-latency networks. The data can also be verifiable.

DApps

The driving force behind decentralized systems is to create a faster, safer, more secure web. For decentralized applications (DApps), third-party applications seeking to connect to a host system must request permission to run locally.

With OrbitDB for example, there is an access control layer where peers can define a set of public keys when a DB is created. This allows peers to have a DB instance that multiple other peers can update at once.

P2P

With ObitDB, each peer or node in the network hosts an instance of the database. libp2p pub/sub protocol allows OrbitDB to easily sync database updates from multiple peers.

This allows a user’s database to be actively backed up on multiple peer instances with no need for the user to explicitly connect to any of them. However, the connecting peers need to have a public address of the peer they want to connect with in the network.

As a result, OrbitDB allows for the creation of a network of databases that anyone can join so long as they have the right access to help keep data available, making the entire network chain more robust and sustainable.

Getting started with OrbitDB

To easily follow along with this tutorial, it is advisable to have Node.js and npm installed on our development machines. Otherwise, we are good to go. Note that we’ll talk about other necessary dependencies as we proceed.

OrbitDB encompasses a whole lot of the technologies we defined above. It is a serverless, distributed, P2P kind of database that relies on IPFS pub/sub for data storage and syncing amongst connected peers. This allows peers to either subscribe to new messages on a given topic or publish messages to a specific topic.

Peers or nodes in the network store only the data they need and some extra metadata for the next node. Therefore, users or peers can hold a portion of the overall data in the network, and therefore can serve files by their respective addresses.

Installation and setup

Because OrbitDB depends on IPFS, we need to have it installed. For details about the various ways to install IPFS, take a look at this section of the documentation. However, because we are focused on Node.js and we intend to use it programmatically, we can run npm install ipfs to install it as an npm package.

Alternatively, we can run the following command:

npm install orbit-db ipfs

Creating an Orbit instance is as easy as calling the createInstance() method. In this method signature, we are allowed to pass as arguments the IPFS instance already set up. This instance comes with optional settings just in case we need to specifically make it configurable.

To create an OrbitDB instance, we can do the following:

const IPFS = require('ipfs')
const OrbitDB = require('orbit-db')

// optional settings for the ipfs instance
const ipfsOptions = {
  EXPERIMENTAL: {
    pubsub: true
  }
}

 // Create IPFS instance with optional config
const ipfs = await IPFS.create(ipfsOptions)

 // Create OrbitDB instance
const orbitDB = await OrbitDB.createInstance(ipfs)

//create KV database
const db = await orbitdb.keyvalue('test-db')

As we can see above, we have imported both ipfs and orbit-db after installation. We then created a new IPFS instance, passing an optional ipfsOptions argument for special configurations. After that, we proceeded to create an OrbitDB instance, which allows us to interact with the various kinds of supported database structures, e.g., KeyValue.

For a full, working implementation of the code block above, take a look at the accompanying GitHub repo. 

The default options settings object passed to the newly instantiated IPFS instance can contain the following:

• EXPERIMENTAL: { pubsub: true } – this enables IPFS pub/sub, which is basically a method of communicating between nodes, as earlier discussed
• config: { Bootstrap: [], Addresses: { Swarm: [] }} – this sets to empty both our Bootstrap peers list (which are peers that are loaded on instantiation) and swarm peers list (peers that can connect and disconnect at any time)
• repo: './ipfs' – designates the path of the repo, usually in Node.js only. In the browser, this is not really necessary. Note that the default setting is a folder called .jsipfs in our machine home directory

Additionally, the signature of the createInstance method is shown below:

createInstance(ipfs, [options])

The optional options settings useful for further configuration is an object that could contain any of the following properties:

• directory – path to be used for the database files. By default, it uses './orbitdb'
• peerId – by default it uses the Base58 string of the IPFS peer ID
• identity – by default, it creates an instance of Identity
• offline – start the OrbitDB instance in offline mode. Note that databases are not replicated when the instance is started in offline mode

Note: If the OrbitDB instance was started in an offline mode and you want to start replicating databases, the OrbitDB instance needs to be recreated.

To interact with a Key-Value database, for example, we can use the newly created OrbitDB instance like this:

const db = orbitDB.keyvalue('test-db')

After we are done with this step, we can get access to the database address, which serves as an identifier and also as a pointer for other database peers or clients to replicate. To access the address, we can do the following:

console.log(db.address.toString()) // convert the database address object to a string with the toString() method.

When this is logged to the console, we get an output similar to the below:

/orbitdb/zdpuB1ccfqAVXPhf4zBBCohvvbDWV1k6S6thTujzy2CHQBPAx/test-db

The database address contains three parts, namely:

• The protocol, signified by /orbitdb
• The IPFS hash in the middle, which is usually an object containing a combination of the database info, known as the manifest and signified by an hash zdpuB1ccfqAVXPhf4zBBCohvvbDWV1k6S6thTujzy2CHQBPAx
• The database name, which is test-db in the example above

Also note that OrbitDB provides an API method, isValidAddress, to verify the validity of a DB address. Note that it returns a Boolean value. Let’s see how we can do so below:

OrbitDB.isValidAddress('/orbitdb/Qmd8TmZrWASypEp4Er9tgWP4kCNQnW4ncSnvjvyHQ3EVSU/test-db')
// true

Supported data model for creating a database

OrbitDB has different types of databases. It organizes its functionality by separating different APIs into stores, where each store satisfies a different purpose.

Each store has its own specific API methods to create, delete, retrieve and update data. To get a list of supported stores, we can do the following below –

OrbitDB.databaseTypes // Returns supported database types  as an Array of Strings
// [ 'counter', 'eventlog', 'feed', 'docstore', 'keyvalue']

Also, we can check if a provided String is a supported database type by calling the isValidType method as shown below, which returns a Boolean value.

OrbitDB.isValidType('docstore')
// true

A list of predefined stores is discussed below:

• log – an immutable, write-only DB, useful mainly for message queuing systems or transaction lists
• feed – a mutable log, where entries can be added and removed. Mainly useful for shopping carts, blog posts, comments, or a Twitter-style timeline
• doc – a document database that stores JSON documents ,which can be indexed by a specified key. Useful for building search indices
• keyvalue – a simple key-value database that supports JSON-serializable data
• counter – an increment-only integer counter useful for counting events or usually ordered data.

Note: To add a custom database type, all we have to do is call the addDatabaseType API method exposed by OrbitDB, as shown below:

const CustomStore = require('./CustomStore')
OrbitDB.addDatabaseType(CustomStore.type, CustomStore)

Also, it is important to note that every database store has the following methods available in addition to their specific methods:

• store.load() – this loads the locally persisted database state to memory
• store.close() – this closes the database
• store.drop() – this removes the database locally. However, this does not delete any data from already connected peers
• store.type – this returns the type of the database as a String

More details and information can be found here in the documentation. Now, let’s look at how to interact with the different databases from a newly created OrbitDB instance:

# for a log database type
const db = await orbitdb.eventlog()

# for a feed database type
const db = await orbitdb.feed()

# for a key keyvalue database type
const db = await orbitdb.keyvalue()

# for a docs database type
const db = await orbitdb.docs()

# for a counter database type
const counter = await orbitdb.counter()

Note: We can decide to write our own stores based on our specific use cases.

Working with OrbitDB in the wild: Demo application to play around with

Communication between database peers

In this section, we are going to look at how to identify as a connected peer and gain needed access to a database. First, let’s construct our IPFS node, as shown in the index.js file:

// optional settings for the ipfs instance
const ipfsOptions = {
    EXPERIMENTAL: {
      pubsub: true
    },
  }

const ipfs = await IPFS.create(ipfsOptions)

The create() method accepts an optional ipfsOptions, which is an object with different properties we can pass as an argument. Here, we have passed an EXPERIMENTAL property, which is an object that allows us enable or add pubsub to the ipfs instance we just created.

Next, we can go ahead and create identities for our database. Each entry in a database is signed by who created it. To do so, we can use the createIdentity method, like so:

const identity = await Identities.createIdentity(options)

Before we do so, we should make sure to import orbit-db-identity-provider. The output of the above command when we log the identity to the console is shown below:

console.log(identity.toJSON()

//output 
{
  id: '034b8a8931164238b1a8c598fcf0d73245780174bf0cb100d93cb3098ba4b19ff2',
  publicKey: '04ad4d2a7812cac1f0e6331edf22cec1a74b9694de6ad222b7cead06f79ec44a95e14b002ee7a0f6f03921fcf2ff646724175d1d31de4876c99dcc582cde835b4c',
  signatures: {
    id: '304402203a7fa472dc584f02aabb27111eab48bc50b0c2137876cd08db89842870aa5abe022069a05962ab9d3d28ff5d7587503852c210e3de65e7fe4bfa0a25ba96a5f078f3',
    publicKey: '3044022049a5885d613a7dd70cd21bad46e159645202911e2d2c16e1be7681ec6b84a272022024575ef612119fbb8e374862d8178b4c0a44f3655400626de4b6ea89e12fb488'
  },
  type: 'orbitdb'
}

Note: The object contains signatures proving possession of some external identifier and an OrbitDB public key. This is included to allow proof of ownership of an external identifier within OrbitDB.

In the above, the id property returns the ID of the external identity. The publicKey returns the signing key used to sign OrbitDB entries, while the signatures return an object containing two signatures, as shown.

To get the public key for our created OrbitDB instance, we can run the following command:

console.log(identity.publicKey)
//output
04ad4d2a7812cac1f0e6331edf22cec1a74b9694de6ad222b7cead06f79ec44a95e14b002ee7a0f6f03921fcf2ff646724175d1d31de4876c99dcc582cde835b4c

Next is to create an OrbitDB instance with the identity we created earlier. To do so, we can use the createInstance method, which accepts the ipfs instance already created and an optional settings object:

const orbitdb = await OrbitDB.createInstance(ipfs,  { identity: identity })

Next up is to actually create a database from the supported database stores with OrbitDB. However, before we do so, we can set access control options for who has write access to our database.

To do so, we can go ahead and define a set of peers that can write to our database or allow anyone to write to a database via a wild card.

Note: By default, only the creator of the database has write access.

const optionsToWrite = {
    // Give write access to the creator of the database
    accessController: {
      type: 'orbitdb', //OrbitDBAccessController
      write: [orbitdb.identity.id, '04ad4d2a7812cac1f0e6331edf22cec1a74b9694de6ad222b7cead06f79ec44a95e14b002ee7a0f6f03921fcf2ff646724175d1d31de4876c99dcc582cde835b4c'],
    }

From the above, we can see that we have granted write access to ourselves to the database using the accessController options property. Note that to allow anyone write to a database, we can do the following:

write: ['*'] //enable write access to the public

Note: To give write access to another database or peer, we can use their public key when we call identity.publicKey.

Now, to create a doc database, for example, we can do the following:

const db = await orbitdb.docs('test-db', optionsToWrite)

optionsToWrite is our access control rights to our database.

Next, let’s add an item to our database:

await db.put({ _id: 'test', name: 'test-doc-db', category: 'distributed' })

To get our database address, we can run the following command:

const address = db.address.toString()

//output
orbitdb/zdpuB1ccfqAVXPhf4zBBCohvvbDWV1k6S6thTujzy2CHQBPAx/test-db

Also, we can grant access to our database after it has been created. To do so explicitly, grant write access to the database by running the following:

await db.access.grant('write', '04ad4d2a7812cac1f0e6331edf22cec1a74b9694de6ad222b7cead06f79ec44a95e14b002ee7a0f6f03921fcf2ff646724175d1d31de4876c99dcc582cde835b4c') // grant access to database2
//where the hash is the `identity2.publicKey`

Because OrbitDB saves the state of the database automatically to disk, you can load a database locally before using it. Therefore, upon opening a database, we can choose to load locally the persisted data before using the database.

To do so, we can use the load() method:

await db2.load()

Also, we can get a value or entry from the database. To do so, we can call the appropriate functions, which are different for the various database types:

const value2 = db2.get('') // this gets all the entries in the database store

console.log(value2)
//output
[
  { _id: 'test', name: 'test-doc-db', category: 'distributed' },
  { _id: 'test2', name: 'test-doc-db2', category: 'nil' }
]

All database models in OrbitDB are implemented on top of ipfs-log, which is an immutable, operation-based CRDTs. It’s an append-only log that can be used to model a mutable, shared state in P2P applications.

Note: Every entry in the log is saved in IPFS and each points to a hash of previous entries thereby forming a graph.

Detailed snippets for working with ipfs-log are available in the sample section of the documentation.

CRUD actions

OrbitDB comes with a clean and easy-to-use API interface. It has functions/methods like get, put, add, set, and others. Details about the different exposed APIs can be found in this section of the documentation.

Let’s look at some examples of API methods exposed by the supported databases below:

keyvalue

In the keyvalue database type, the exposed API methods include, put, set, and get. The signature of put for example is put(key, value), which accepts a key or database name and the value we intend to update. More details can be found in this section of the docs.

log

In the log DB type, we have add, get, and iterator methods, which are explained in more detail here.

feed

In the feed database type, we have add, get, remove, and iterator. More details about these can be found here in the docs.

doc

In the doc database type, exposed API methods include put, get, query, and del. More detailed use cases for each can be found here in the docs.

counter

In the counter database type, the exposed API methods include the value and inc. More detailed information can be found here in the docs.

Conclusion

Why is OrbitDB a superb choice for DApps and blockchain applications? Well, because it uses a particular kind of data structure known as a conflict-free replicated data type (CRDT) for maintaining eventual consistency. This means operations can occur at different times without coordination, with the assumption that they will eventually sync up.

Each time we interact with an OrbitDB database, we are interacting with a snapshot in time. This is how distributed databases are designed. They work both online and offline. However, it does require at least one node or peer to be willing to persist the database so that data is not lost upon a disconnect.

Data are interlinked via content addresses, as opposed to the location-based addressing of the centralized web, where the application code runs on a centralized server.

It should be noted, however, that OrbitDB and the underlying IPFS layer are currently alpha-stage software. They both work in Node.js applications as well as in browsers, but Windows operating systems are not currently supported.

More information about the capabilities of OrbitDB can be found in the project repository on Github. The script used for this tutorial can also be found here on GitHub.