How to use Redis as a database with go-redis

Redis is an in-memory data store used as a database, cache, or message broker. Go-redis/redis is a type-safe, Redis client library for Go with support for features like Pub/Sub, sentinel, and pipelining.

NOTE: We will be referring to the client library as “go-redis” to help differentiate it from Redis itself.

In this article, we will explore go-redis and use its pipeline feature to build a leaderboard API. The API will use Gin and Redis’ sorted sets under the hood. It will expose the following endpoints:

• GET /points/:username — to get a user’s score and their rank in the overall leaderboard
• POST /points — to add or update a user and their score. This endpoint will also return the new rank of the user
• GET /leaderboard — returns the current leaderboard, with users sorted in ascending order of their ranks

Prerequisites

To follow along with this post, you will need:

• A Go installation with modules support
• Redis installed on your local computer (alternatively, you can use the Docker image if you have Docker installed)
• Experience writing Go

Getting started

To get started, create a folder for the project in your preferred location and initialize your Go module:

$ mkdir rediboard && cd rediboard
$ go mod init gitlab.com/idoko/rediboard

Install the application dependencies (gin-gonic/gin and go-redis/redis) with the commands below:

$ go get github.com/gin-gonic/gin github.com/go-redis/redis

Next, create a main.go file to serve as the project’s entry point. While at it, we will also create a db folder in the project root directory to hold the code responsible for interacting with Redis:

$ touch main.go
$ mkdir db

Get familiar with go-redis

With our application scaffold in place, let us go over some go-redis’ basics. Connection to a Redis database is handled by the “client” – a thread-safe value that can be shared by multiple goroutines and typically lives throughout the lifetime of the application. The code below creates a new client:

client := redis.NewClient(&redis.Options{
   Addr:     "localhost:6379", // host:port of the redis server
   Password: "", // no password set
   DB:       0,  // use default DB
})

Go-redis provides lots of configuration options through the redis.Options parameter. Some of the options include PoolSize to set the maximum number of connections and TLSConfig for connecting to a TLS-protected Redis server.

The client then exposes commands as receiver methods. For instance, the code shows how we can set and get values from a Redis database:

ctx := context.TODO()
client.Set(ctx, "language", "Go", 0)
language := client.Get(ctx, "language")
year := client.Get(ctx, "year")

fmt.Println(language.Val()) // "Go"
fmt.Println(year.Val()) // ""

The library requires a context parameter to allow for things like context-based cancellation of a running command. Since we don’t need the benefits it provides here, we create an empty context with context.TODO(). Next, we set the language to “Go” and give it no expiration date (by passing in a value of 0). We proceed to get the values for language and year, but because we did not set a value for the year, it is nil, and year.Val() returns an empty string.

Connect to Redis with Go

To create the Redis client for our application, create a new db.go file in the db folder we created earlier and add the code snippet below to it:

package db

import (
   "context"
   "errors"
   "github.com/go-redis/redis/v8"
)

type Database struct {
   Client *redis.Client
}

var (
   ErrNil = errors.New("no matching record found in redis database")
   Ctx    = context.TODO()
)

func NewDatabase(address string) (*Database, error) {
   client := redis.NewClient(&redis.Options{
      Addr: address,
      Password: "",
      DB: 0,
   })
   if err := client.Ping(Ctx).Err(); err != nil {
      return nil, err
   }
   return &Database{
      Client: client,
   }, nil
}

The code above creates a Database struct to wrap the redis client and expose it to the rest of the app (routers, etc.). It also sets up two package-level variables – ErrNil used to tell the calling code that a Redis operation returned nil and Ctx, an empty context to use with the client. We also created a NewDatabase function that sets up the client and checks that the connection is alive using the PING command.

Open the main.go file and call the NewDatabase() function as shown in the code below:

package main

import (
   "github.com/gin-gonic/gin"
   "gitlab.com/idoko/rediboard/db"
   "log"
   "net/http"
)

var (
   ListenAddr = "localhost:8080"
   RedisAddr = "localhost:6379"
)
func main() {
   database, err := db.NewDatabase(RedisAddr)
   if err != nil {
      log.Fatalf("Failed to connect to redis: %s", err.Error())
   }

   router := initRouter(database)
   router.Run(ListenAddr)
}

The snippet above attempts to connect to the database and prints any error it encounters in the process. It also refers to an initRouter function. We will set that up in the next section.

API routes with Gin

Next, create the initRouter function for creating and registering the application routes. Add the code below in main.go beneath the existing main function:

func initRouter(database *db.Database) *gin.Engine {
   r := gin.Default()
   return r
}

For now, the function returns an instance of gin.Engine. We will add route-specific handlers later on.

Transaction pipelines in go-redis

A Redis Transaction queues operations and provide a guarantee that either all or none of the operations are executed. Another interesting Redis feature is pipelining, a network optimization that allows a Redis client to send multiple requests to the server without waiting for replies and reading all of them at once.

Go-redis wraps both transactions and pipelines in the TxPipeline method. Below is a set of sample transaction commands executed on redis-cli:

127.0.0.1:6379> MULTI
OK
127.0.0.1:6379> SET language "golang"
QUEUED
127.0.0.1:6379> SET year 2009
QUEUED
127.0.0.1:6379> EXEC
1) OK
2) OK
127.0.0.1:6379>

The commands above can be translated to the Go code below:

pipe := db.Client.TxPipeline()
pipe.Set(Ctx, "language", "golang")
pipe.Set(Ctx, "year", 2009)
results, err := pipe.Exec()

Save users to the sorted sets

Create a user.go file in the db folder and add the code below to it:

package db

import (
   "fmt"
   "github.com/go-redis/redis/v8"
)

type User struct {
   Username string `json:"username" binding:"required"`
   Points   int `json:"points" binding:"required"`
   Rank     int    `json:"rank"`
}

func (db *Database) SaveUser(user *User) error {
   member := &redis.Z{
      Score: float64(user.Points),
      Member: user.Username,
   }
   pipe := db.Client.TxPipeline()
   pipe.ZAdd(Ctx, "leaderboard", member)
   rank := pipe.ZRank(Ctx, leaderboardKey, user.Username)
   _, err := pipe.Exec(Ctx)
   if err != nil {
      return err
   }
   fmt.Println(rank.Val(), err)
   user.Rank = int(rank.Val())
   return nil
}

The code above creates a User struct to serve as a wrapper around users in the leaderboard. The struct includes how we want the fields to be represented when transformed to JSON as well as when they are translated from HTTP requests using Gin’s binding. It then leverages pipelines to add the new member to the sorted set and gets the member’s new rank. Because the user parameter is a pointer, the Rank value is updated across the board when we mutate it from the SaveUser() function.

Next, alter main.go to call the SaveUser function declared above when it gets a POST request to /points. Open main.go and add the route handler below to the initRouter function (just before the return r line):

r.POST("/points", func (c *gin.Context) {
   var userJson db.User
   if err := c.ShouldBindJSON(&userJson); err != nil {
      c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
      return
   }
   err := database.SaveUser(&userJson)
   if err != nil {
      c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
      return
   }
   c.JSON(http.StatusOK, gin.H{"user": userJson})
})

Get users’ scores and ranking

Similarly, add the code below to user.go to fetch a single user’s ranking and their score:

func (db *Database) GetUser(username string) (*User, error) {
   pipe := db.Client.TxPipeline()
   score := pipe.ZScore(Ctx, leaderboardKey, username)
   rank := pipe.ZRank(Ctx, leaderboardKey, username)
   _, err := pipe.Exec(Ctx)
   if err != nil {
      return nil, err
   }
   if score == nil {
      return nil, ErrNil
   }
   return &User{
      Username: username,
      Points: int(score.Val()),
      Rank: int(rank.Val()),
   }, nil
}

Here, we are also leveraging pipelines to get the user’s score and rank, with their username as a key.

We also signal to the caller if no matching record was found (using ErrNil) so that it is up to the caller to handle such cases separately (for instance, they could choose to display a 404 response).

Next, add the corresponding route handler in main.go as follows:

r.GET("/points/:username", func (c *gin.Context) {
   username := c.Param("username")
   user, err := database.GetUser(username)
   if err != nil {
      if err == db.ErrNil {
         c.JSON(http.StatusNotFound, gin.H{"error": "No record found for " + username})
         return
      }
      c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
      return
   }
   c.JSON(http.StatusOK, gin.H{"user": user})
})

The snippet above retrieves the username path parameter and passes it to the GetUser function declared earlier. It also checks for cases where the error returned is of type ErrNil and shows a 404 response in that case.

Fetch complete leaderboard with ZRangeWithScores

To get the complete leaderboard, Redis provides the ZRange command, used to retrieve the members of a sorted set in ascending order of their scores. ZRange also accepts an optional WITHSCORES argument that tells it to return the score of each member as well. Go-redis on the other hand, splits the command into two, providing ZRange and ZRangeWithScores separately.

Create a new file in the db folder named leaderboard.go with the following content:

package db

var leaderboardKey = "leaderboard"

type Leaderboard struct {
   Count int `json:"count"`
   Users []*User
}

func (db *Database) GetLeaderboard() (*Leaderboard, error) {
   scores := db.Client.ZRangeWithScores(Ctx, leaderboardKey, 0, -1)
   if scores == nil {
      return nil, ErrNil
   }
   count := len(scores.Val())
   users := make([]*User, count)
   for idx, member := range scores.Val() {
      users[idx] = &User{
         Username: member.Member.(string),
         Points: int(member.Score),
         Rank: idx,
      }
   }
   leaderboard := &Leaderboard{
      Count: count,
      Users: users,
   }
   return leaderboard, nil
}

The leaderboardKey represents the key used to identify the set in our Redis database. Since we are only running a single command now (ZRangeWithScores), there is no longer a need to batch the commands with transaction pipelines anymore so we store the result directly in the scores variable. The value stored in scores contains a slice of Go maps, whose length is the number of members stored in the set.

To run our application, ensure that you have Redis installed and running. Alternatively, you can pull in the Redis Docker image and run it with the command below:

$ docker run --name=rediboard -p 6379:6379 redis

You can now build and run (or directly run) the main.go file with the commands below to test out the sample project:

$ go build ./main.go
$ ./main

Here are some sample cURL commands and their responses.

Feel free to try the API out with cURL, Postman, HTTPie, or your favorite API client.

cURL command:

$ curl -H "Content-type: application/json" -d '{"username": "isa", "points": 25}' localhost:8080/points

Response:

{
  "user": {
    "username": "isa",
    "points": 25,
    "rank": 3
  }
}

cURL command:

$ curl -H "Content-type: application/json" localhost:8080/points/mchl

Response:

{
  "user": {
    "username": "jude",
    "points": 22,
    "rank": 0
  }
}

cURL command:

$ curl -H "Content-type: application/json" localhost:8080/leaderboard

Response:

{
  "leaderboard": {
    "count": 7,
    "Users": [
      {
        "username": "ene",
        "points": 22,
        "rank": 0
      },
      {
        "username": "ben",
        "points": 23,
        "rank": 2
      },
      {
        "username": "isa",
        "points": 25,
        "rank": 3
      },
      {
        "username": "jola",
        "points": 39,
        "rank": 5
      }
    ]
  }
}

Here is a screenshot of the app running in the terminal and the cURL response:

Conclusion

If you are looking to explore further, the Redis and Go-redis’ documentations are places to start. For unsupported commands, go-redis also provides generic Send() and Do() methods.

In this article, we went over how to interact with a Redis database using the go-redis library. The code for the sample project is available on GitLab.