When it comes to a database for the decentralized web, OrbitDB is a name to know. OrbitDB is not a regular database, with one central repository for all entries; instead, it relies on a protocol for building P2P decentralized applications (DApps), where each connected peer has a specific database instance or copy.
Typical DApps rely on a distributed computing model where system components run on a P2P basis. A P2P network affords an instance where files can be replicated or synced amongst other peers residing on the same network.
There is a sort of direct communication in this kind of network architecture, as opposed to the client-server model of the centralized web, wherein a server acts as an intermediate layer for other systems to connect to and communicate with.
In an actual P2P model, there is a connection to and interaction with other peers in the global network, although peers usually start out alone, locally, and unconnected to the network.
Let’s now move on to reviewing some background terms around distributed databases with OrbitDB as a case study.
Just like distributed systems, distributed databases can be easily replicated and duplicated. Data is usually stored in multiple independent systems across different locations or regions, or even in a data center. Let’s look at OrbitDB in the context of distributed databases.
Firstly, it utilizes libp2p, which is a network protocol. It is a publish-subscribe (pub/sub) protocol useful for easily syncing database updates from multiple peers.
Secondly, for data homogeneity and uniformity, an OrbitDB instance must be replicated amongst peers. This happens in such a way that a peer only needs to subscribe to a database address, and it’ll automatically start replicating it.
Also, OrbitDB uses conflict-free replicated data types (CRDTs). This is a data structure whose network replicas can be updated concurrently and independently without the replicas coordinating. This means peers can go off the grid, and when they return, their state will be in sync with other peers/nodes, and all inconsistencies in the entire system will be resolved.
Lastly, for distributed databases like orbitDB, access to data can be specified for a set of peers that can write to the database. Note that, by default and if not specified by the creator of the database, only the creator will have write-access.
OrbitDB utilizes the InterPlanetary File System (IPFS). IPFS is a protocol for storing and sharing data in a distributed file system and, according to its documentation, powers the distributed web.
It uses content addressing to give each resource a unique identifier. This means that when we add content to IPFS, it’s given an address, which usually contains a hash.
This creates a group of data that is stored and that can be subsequently accessed by connected peers. Therefore, for data retrieval, multiple peers can respond at the same time if they have the data, leading to improved performance for high-latency networks. The data can also be verifiable.
The driving force behind decentralized systems is to create a faster, safer, more secure web. For decentralized applications (DApps), third-party applications seeking to connect to a host system must request permission to run locally.
With OrbitDB for example, there is an access control layer where peers can define a set of public keys when a DB is created. This allows peers to have a DB instance that multiple other peers can update at once.
With ObitDB, each peer or node in the network hosts an instance of the database. libp2p pub/sub protocol allows OrbitDB to easily sync database updates from multiple peers.
This allows a user’s database to be actively backed up on multiple peer instances with no need for the user to explicitly connect to any of them. However, the connecting peers need to have a public address of the peer they want to connect with in the network.
As a result, OrbitDB allows for the creation of a network of databases that anyone can join so long as they have the right access to help keep data available, making the entire network chain more robust and sustainable.
To easily follow along with this tutorial, it is advisable to have Node.js and npm installed on our development machines. Otherwise, we are good to go. Note that we’ll talk about other necessary dependencies as we proceed.
OrbitDB encompasses a whole lot of the technologies we defined above. It is a serverless, distributed, P2P kind of database that relies on IPFS pub/sub for data storage and syncing amongst connected peers. This allows peers to either subscribe to new messages on a given topic or publish messages to a specific topic.
Peers or nodes in the network store only the data they need and some extra metadata for the next node. Therefore, users or peers can hold a portion of the overall data in the network, and therefore can serve files by their respective addresses.
Because OrbitDB depends on IPFS, we need to have it installed. For details about the various ways to install IPFS, take a look at this section of the documentation. However, because we are focused on Node.js and we intend to use it programmatically, we can run npm install ipfs
to install it as an npm package.
Alternatively, we can run the following command:
npm install orbit-db ipfs
Creating an Orbit instance is as easy as calling the createInstance()
method. In this method signature, we are allowed to pass as arguments the IPFS instance already set up. This instance comes with optional settings just in case we need to specifically make it configurable.
To create an OrbitDB instance, we can do the following:
const IPFS = require('ipfs') const OrbitDB = require('orbit-db') // optional settings for the ipfs instance const ipfsOptions = { EXPERIMENTAL: { pubsub: true } } // Create IPFS instance with optional config const ipfs = await IPFS.create(ipfsOptions) // Create OrbitDB instance const orbitDB = await OrbitDB.createInstance(ipfs) //create KV database const db = await orbitdb.keyvalue('test-db')
As we can see above, we have imported both ipfs
and orbit-db
after installation. We then created a new IPFS instance, passing an optional ipfsOptions
argument for special configurations. After that, we proceeded to create an OrbitDB instance, which allows us to interact with the various kinds of supported database structures, e.g., KeyValue
.
For a full, working implementation of the code block above, take a look at the accompanying GitHub repo.
The default options settings object passed to the newly instantiated IPFS instance can contain the following:
EXPERIMENTAL: { pubsub: true }
– this enables IPFS pub/sub, which is basically a method of communicating between nodes, as earlier discussedconfig: { Bootstrap: [], Addresses: { Swarm: [] }}
– this sets to empty both our Bootstrap peers list (which are peers that are loaded on instantiation) and swarm peers list (peers that can connect and disconnect at any time)repo: './ipfs'
– designates the path of the repo, usually in Node.js only. In the browser, this is not really necessary. Note that the default setting is a folder called .jsipfs
in our machine home directoryAdditionally, the signature of the createInstance
method is shown below:
createInstance(ipfs, [options])
The optional options
settings useful for further configuration is an object that could contain any of the following properties:
directory
– path to be used for the database files. By default, it uses './orbitdb'
peerId
– by default it uses the Base58 string of the IPFS peer IDidentity
– by default, it creates an instance of Identity
offline
– start the OrbitDB instance in offline mode. Note that databases are not replicated when the instance is started in offline modeNote: If the OrbitDB instance was started in an offline mode and you want to start replicating databases, the OrbitDB instance needs to be recreated.
To interact with a Key-Value
database, for example, we can use the newly created OrbitDB instance like this:
const db = orbitDB.keyvalue('test-db')
After we are done with this step, we can get access to the database address, which serves as an identifier and also as a pointer for other database peers or clients to replicate. To access the address, we can do the following:
console.log(db.address.toString()) // convert the database address object to a string with the toString() method.
When this is logged to the console, we get an output similar to the below:
/orbitdb/zdpuB1ccfqAVXPhf4zBBCohvvbDWV1k6S6thTujzy2CHQBPAx/test-db
The database address contains three parts, namely:
/orbitdb
zdpuB1ccfqAVXPhf4zBBCohvvbDWV1k6S6thTujzy2CHQBPAx
test-db
in the example aboveAlso note that OrbitDB provides an API method, isValidAddress
, to verify the validity of a DB address. Note that it returns a Boolean value. Let’s see how we can do so below:
OrbitDB.isValidAddress('/orbitdb/Qmd8TmZrWASypEp4Er9tgWP4kCNQnW4ncSnvjvyHQ3EVSU/test-db') // true
OrbitDB has different types of databases. It organizes its functionality by separating different APIs into stores, where each store satisfies a different purpose.
Each store has its own specific API methods to create, delete, retrieve and update data. To get a list of supported stores, we can do the following below –
OrbitDB.databaseTypes // Returns supported database types as an Array of Strings // [ 'counter', 'eventlog', 'feed', 'docstore', 'keyvalue']
Also, we can check if a provided String
is a supported database type by calling the isValidType
method as shown below, which returns a Boolean value.
OrbitDB.isValidType('docstore') // true
A list of predefined stores is discussed below:
log
– an immutable, write-only DB, useful mainly for message queuing systems or transaction listsfeed
– a mutable log, where entries can be added and removed. Mainly useful for shopping carts, blog posts, comments, or a Twitter-style timelinedoc
– a document database that stores JSON documents ,which can be indexed by a specified key. Useful for building search indiceskeyvalue
– a simple key-value database that supports JSON-serializable datacounter
– an increment-only integer counter useful for counting events or usually ordered data.Note: To add a custom database type, all we have to do is call the
addDatabaseType
API method exposed by OrbitDB, as shown below:const CustomStore = require('./CustomStore') OrbitDB.addDatabaseType(CustomStore.type, CustomStore)
Also, it is important to note that every database store has the following methods available in addition to their specific methods:
store.load()
– this loads the locally persisted database state to memorystore.close()
– this closes the databasestore.drop()
– this removes the database locally. However, this does not delete any data from already connected peersstore.type
– this returns the type of the database as a String
More details and information can be found here in the documentation. Now, let’s look at how to interact with the different databases from a newly created OrbitDB instance:
# for a log database type const db = await orbitdb.eventlog() # for a feed database type const db = await orbitdb.feed() # for a key keyvalue database type const db = await orbitdb.keyvalue() # for a docs database type const db = await orbitdb.docs() # for a counter database type const counter = await orbitdb.counter()
Note: We can decide to write our own stores based on our specific use cases.
In this section, we are going to look at how to identify as a connected peer and gain needed access to a database. First, let’s construct our IPFS node, as shown in the index.js
file:
// optional settings for the ipfs instance const ipfsOptions = { EXPERIMENTAL: { pubsub: true }, } const ipfs = await IPFS.create(ipfsOptions)
The create()
method accepts an optional ipfsOptions
, which is an object with different properties we can pass as an argument. Here, we have passed an EXPERIMENTAL
property, which is an object that allows us enable or add pubsub
to the ipfs
instance we just created.
Next, we can go ahead and create identities for our database. Each entry in a database is signed by who created it. To do so, we can use the createIdentity
method, like so:
const identity = await Identities.createIdentity(options)
Before we do so, we should make sure to import orbit-db-identity-provider
. The output of the above command when we log the identity to the console is shown below:
console.log(identity.toJSON() //output { id: '034b8a8931164238b1a8c598fcf0d73245780174bf0cb100d93cb3098ba4b19ff2', publicKey: '04ad4d2a7812cac1f0e6331edf22cec1a74b9694de6ad222b7cead06f79ec44a95e14b002ee7a0f6f03921fcf2ff646724175d1d31de4876c99dcc582cde835b4c', signatures: { id: '304402203a7fa472dc584f02aabb27111eab48bc50b0c2137876cd08db89842870aa5abe022069a05962ab9d3d28ff5d7587503852c210e3de65e7fe4bfa0a25ba96a5f078f3', publicKey: '3044022049a5885d613a7dd70cd21bad46e159645202911e2d2c16e1be7681ec6b84a272022024575ef612119fbb8e374862d8178b4c0a44f3655400626de4b6ea89e12fb488' }, type: 'orbitdb' }
Note: The object contains signatures proving possession of some external identifier and an OrbitDB public key. This is included to allow proof of ownership of an external identifier within OrbitDB.
In the above, the id
property returns the ID of the external identity. The publicKey
returns the signing key used to sign OrbitDB entries, while the signatures
return an object containing two signatures, as shown.
To get the public key for our created OrbitDB instance, we can run the following command:
console.log(identity.publicKey) //output 04ad4d2a7812cac1f0e6331edf22cec1a74b9694de6ad222b7cead06f79ec44a95e14b002ee7a0f6f03921fcf2ff646724175d1d31de4876c99dcc582cde835b4c
Next is to create an OrbitDB instance with the identity we created earlier. To do so, we can use the createInstance
method, which accepts the ipfs
instance already created and an optional settings object:
const orbitdb = await OrbitDB.createInstance(ipfs, { identity: identity })
Next up is to actually create a database from the supported database stores with OrbitDB. However, before we do so, we can set access control options for who has write access to our database.
To do so, we can go ahead and define a set of peers that can write to our database or allow anyone to write to a database via a wild card.
Note: By default, only the creator of the database has write access.
const optionsToWrite = { // Give write access to the creator of the database accessController: { type: 'orbitdb', //OrbitDBAccessController write: [orbitdb.identity.id, '04ad4d2a7812cac1f0e6331edf22cec1a74b9694de6ad222b7cead06f79ec44a95e14b002ee7a0f6f03921fcf2ff646724175d1d31de4876c99dcc582cde835b4c'], }
From the above, we can see that we have granted write access to ourselves to the database using the accessController
options property. Note that to allow anyone write to a database, we can do the following:
write: ['*'] //enable write access to the public
Note: To give write access to another database or peer, we can use their public key when we call
identity.publicKey
.
Now, to create a doc
database, for example, we can do the following:
const db = await orbitdb.docs('test-db', optionsToWrite)
optionsToWrite
is our access control rights to our database.
Next, let’s add an item to our database:
await db.put({ _id: 'test', name: 'test-doc-db', category: 'distributed' })
To get our database address, we can run the following command:
const address = db.address.toString() //output orbitdb/zdpuB1ccfqAVXPhf4zBBCohvvbDWV1k6S6thTujzy2CHQBPAx/test-db
Also, we can grant access to our database after it has been created. To do so explicitly, grant write access to the database by running the following:
await db.access.grant('write', '04ad4d2a7812cac1f0e6331edf22cec1a74b9694de6ad222b7cead06f79ec44a95e14b002ee7a0f6f03921fcf2ff646724175d1d31de4876c99dcc582cde835b4c') // grant access to database2 //where the hash is the `identity2.publicKey`
Because OrbitDB saves the state of the database automatically to disk, you can load a database locally before using it. Therefore, upon opening a database, we can choose to load locally the persisted data before using the database.
To do so, we can use the load()
method:
await db2.load()
Also, we can get a value or entry from the database. To do so, we can call the appropriate functions, which are different for the various database types:
const value2 = db2.get('') // this gets all the entries in the database store console.log(value2) //output [ { _id: 'test', name: 'test-doc-db', category: 'distributed' }, { _id: 'test2', name: 'test-doc-db2', category: 'nil' } ]
All database models in OrbitDB are implemented on top of ipfs-log
, which is an immutable, operation-based CRDTs. It’s an append-only log that can be used to model a mutable, shared state in P2P applications.
Note: Every entry in the log is saved in IPFS and each points to a hash of previous entries thereby forming a graph.
Detailed snippets for working with ipfs-log
are available in the sample section of the documentation.
OrbitDB comes with a clean and easy-to-use API interface. It has functions/methods like get
, put
, add
, set
, and others. Details about the different exposed APIs can be found in this section of the documentation.
Let’s look at some examples of API methods exposed by the supported databases below:
keyvalue
In the keyvalue
database type, the exposed API methods include, put
, set
, and get
. The signature of put
for example is put(key, value)
, which accepts a key or database name and the value we intend to update. More details can be found in this section of the docs.
log
In the log
DB type, we have add
, get
, and iterator
methods, which are explained in more detail here.
feed
In the feed
database type, we have add
, get
, remove
, and iterator
. More details about these can be found here in the docs.
doc
In the doc
database type, exposed API methods include put
, get
, query
, and del
. More detailed use cases for each can be found here in the docs.
counter
In the counter
database type, the exposed API methods include the value
and inc
. More detailed information can be found here in the docs.
Why is OrbitDB a superb choice for DApps and blockchain applications? Well, because it uses a particular kind of data structure known as a conflict-free replicated data type (CRDT) for maintaining eventual consistency. This means operations can occur at different times without coordination, with the assumption that they will eventually sync up.
Each time we interact with an OrbitDB database, we are interacting with a snapshot in time. This is how distributed databases are designed. They work both online and offline. However, it does require at least one node or peer to be willing to persist the database so that data is not lost upon a disconnect.
Data are interlinked via content addresses, as opposed to the location-based addressing of the centralized web, where the application code runs on a centralized server.
It should be noted, however, that OrbitDB and the underlying IPFS layer are currently alpha-stage software. They both work in Node.js applications as well as in browsers, but Windows operating systems are not currently supported.
More information about the capabilities of OrbitDB can be found in the project repository on Github. The script used for this tutorial can also be found here on GitHub.
Deploying a Node-based web app or website is the easy part. Making sure your Node instance continues to serve resources to your app is where things get tougher. If you’re interested in ensuring requests to the backend or third-party services are successful, try LogRocket.
LogRocket is like a DVR for web and mobile apps, recording literally everything that happens while a user interacts with your app. Instead of guessing why problems happen, you can aggregate and report on problematic network requests to quickly understand the root cause.
LogRocket instruments your app to record baseline performance timings such as page load time, time to first byte, slow network requests, and also logs Redux, NgRx, and Vuex actions/state. Start monitoring for free.
Would you be interested in joining LogRocket's developer community?
Join LogRocket’s Content Advisory Board. You’ll help inform the type of content we create and get access to exclusive meetups, social accreditation, and swag.
Sign up nowconsole.time is not a function
errorExplore the two variants of the `console.time is not a function` error, their possible causes, and how to debug.
jQuery 4 proves that jQuery’s time is over for web developers. Here are some ways to avoid jQuery and decrease your web bundle size.
See how to implement a single and multilevel dropdown menu in your React project to make your nav bars more dynamic and user-friendly.
NAPI-RS is a great module-building tool for image resizing, cryptography, and more. Learn how to use it with Rust and Node.js.
6 Replies to "A guide to working with OrbitDB in Node.js"
I’m at my wits end. I know this should be simple, but there is literally ZERO documentation anywhere on the web that I can find to show WTH I’m supposed to put in my tag!
In my script tag, that is. I had brackets around the word script, and it got stripped from my comment above.
Very useful, thank you!
Never expected this from logrocket. Used await outside of async.
Hello Lwkedio,
Apologies for any misconception. That was just an explanation and therefore does not show the full code block.
In the full code which is available on GitHub, the await is wrapped around the async keyword. Please do check that as well. I have added the link to the code on GitHub at the end of the post.
Also, as an aside, you should check top level await functions, they exist nowadays. Thanks for reading and for your comments and hope I was able to clarify your questions.
Hi
It might be simpler to use dexie.js, and sync it to orbitDb.
like in here:
https://github.com/shemeshg/dexie-sync-orbitdb/