8 external services for Rust

Developers often use external services, such as databases, worker queues, and cache or data processing pipelines, to add new dimensions and functionalities to their web apps. As Rust has grown in popularity, naturally, so has its ecosystem of high-quality crates for implementing communication protocols and other external services.

In this guide, we’ll look at some of the best high-performance external services the Rust community has to offer.

1. lapin

lapin provides a complete async implementation of the Advanced Message Queuing Protocol (AMQP) used for communication with the RabbitMQ message broker. It’s the base library for creating microservices, worker queues, and interservice communication.

lapin is the base library for multiple crates with support for Tokio runtime, async-io runtime, and async-std runtime through different wrappers. It has a production-level implementation of the AMQP and support for native SSL and OpenSSL. The lapin crate also supports different types of authentication, e.g. SSL certificate and username/password authentication.

Production-ready: Yes
Async support: Yes

2. rdkafka

rdkafka provides an implementation of Kafka for Rust. It’s fully asynchronous.

The rdkafka crate is based on the librdkafka C library. It provides a safe and complete interface for the librdkafka library. It supports all versions of Kafka greater than 0.8.

rdkafa is extremely fast; it can process up to 1 million messages per second, according to the benchmark on rdkafka'sreadme. rdkafka internally uses the rdkafka-sys crate for FFI in the librdkafka library.

Features of rdkafka include:

• Ability to consume from single or multiple topics
• Automatic consumer rebalancing
• Customizable rebalance, with pre- and post-rebalance callbacks
• Synchronous and asynchronous message production
• Customizable offset commit
• Ability to create and delete topics and add and edit partitions
• Ability to alter broker and topic configurations

Production-ready: Yes
Async support: Yes

3. nats

The nats crate provides support for the NATS messaging system in Rust. It has both sync and async support and works with both Tokio and async-std out of the box. It is written in pure Rust.

nats is the official client for Rust and is maintained by the nat.io team. It has support for JWT, based, token, Nkeys, and username/password-based authentication.

The main advantage of the nats crate is its ability to generate iterators.

Features of the nats crate include:

• Basic publish/subscribe
• Request/reply (Singelton and Streams)
• Authentication
• Reconnection in case of connection loss
• TLS support
• Direct async support

4. arrow and datafusion

arrow provides data types required for Apache Arrow and datafusion provides an SQL like a query over the arrow array and data layout.

Generally speaking, the arrow crate offers the functionality to develop code that uses Arrow arrays, and datafusion offers most operations typically found in SQL, with the notable exceptions of join and window functions.

arrow implements all formats in the specification, except certain dictionaries. It also supports SIMD operations to some of its vertical operations.

datafusion supports async execution, user-defined functions, aggregates, and whole execution nodes.

Production-ready: Yes
Async support: Yes

5. tantivy

tantivy is a building block for full-text search. Tt has support for the tokenization of 17 Latin languages and many other regional languages. tantivy also supports multithreading and SIMD for fast and efficient indexing and search.

tantivy has a natural querying language (e.g., “logrocket” and “frontend”) and is extremely fast, according to the benchmark provided by the tantivy crate.

Notable features:

• Single-valued and multivalued u64, i64, and f64 fast fields (the equivalent of doc values in Lucene)
• &[u8] fast fields
• Text, i64, u64, f64, dates, and hierarchical facet fields
• LZ4 compressed document store
• Range queries
• Faceted search

Production-ready: Yes
Async support: n/a

6. elasticsearch

The elasticsearch crate is an official Elasticsearch client. It supports async using Tokio runtime, as well as both native-tls and rust-tls.

elasticsearch is in alpha version but it has a complete implementation of the Elasticsearch protocol. It supports both credential- and certificate-based authentication.

Production-ready: No
Async support: Yes

7. redis

redis is an in-memory database that can be used for creating a cache, worker queues, and creating microservices. Rust has excellent support for the Redis database.

As noted in “11 database drivers and ORMs for Rust that are ready for production,” redis provides both high- and low-level APIs. All the queries are pipelined, meaning more than one query can be sent simultaneously.

redis has a full implementation of the Redis communication protocol, but does not yet support pub/sub. The crate uses Tokio as async runtime and supports Lua scripts, auto-reconnect, and connection pool using the r2d2 crate.

8. pulsar

pulsar is an async client for Apache Pulsar. It’s written in pure Rust and doesn’t rely on the Pulsar CPP library.

The pulsar crate supports both Tokio and async-std runtime out of the box.

Notable features:

• URL-based (pulsar:// and pulsar+ssl://) connections with DNS lookup
• Multitopic consumers (based on a regex)
• TLS connection
• Configurable executor (Tokio or async-std)
• Automatic reconnection with exponential backoff
• Message batching
• Compression with LZ4, zlib, zstd or Snappy (can be deactivated with Cargo features)

Production-ready: Yes
Async support: Yes

Comparing Rust external services

Library	Production-ready	Async support	External service
lapin	Yes	Yes	Redis, AMQP protocol
rdkafka	Yes	Yes	Apache kafka
nats	Yes	Yes	NATS.io
arrow	Yes	No	Apache Arrow
datafusion	Yes	Yes	Apache Arrow (SQL query)
tantivy	Yes	Not Applicable	Full Test Search
elasticsearch	No	Yes	Elasticsearch
redis	Yes	Yes	Redis
pulsar	Yes	Yes	Apache Pulsar