Optimizing CI/CD pipelines in your Rust projects

Rust has been the most admired programming language for seven years in a row, according to Stack Overflow. But it is also infamous for being slow at compile time. This is by design and for good reason; the compiler enforces constraints that make Rust safe at runtime.

But the thing with slow compilation time is that, if ignored, the waiting time will compound, the cost of running the CI/CD will increase, and the feedback loop will gradually lengthen. We don’t want the Rust compiler to steal our productivity! Ultimately, we want to use less electricity and pay less for the CI/CD. If you happen to hotfix a bug in production, you’ll realize the importance of faster iteration and faster CI/CD.

In this article, we‘ll focus on different strategies we can use to optimize CI/CD pipelines in Rust projects. We’ll review them in order of their potential impact on performance. For our demo, we’ll use a simple “Hello World!” REST API.

Let’s make our Rust projects faster!

Jump ahead:

• What is CI/CD?
• Optimizing the Docker image build
  • Using a basic Dockerfile
  • Using multi-stage builds
  • Minimizing the binary size
  • Using a scratch image
  • Using an alpine image
  • Using a distroless image
  • Speeding up build time with cargo-chef
  • Comparing performance gains
• Speeding up Rust app build time
• Speeding up app testing
• Speeding up binary crate installation

What is CI/CD?

CI/CD, or continuous integration/continuous development, is a collection of pipelines that runs code through various steps and platforms, helping ensure that code quality is met, tests pass, and builds are successful in each target platform.

In most cases, the pipelines are executed on the server, even if it is possible to run them locally. Most code hosting platforms have a customizable, integrated CI/CD feature. The pipelines can be run after any event or at a particular time, and there are several operating systems to choose from.

Optimizing the Docker image

A Docker image is simply a file with a list of instructions for building a Docker container. To look at different ways to optimize the build process, we’ll use a simple “Hello World!” project. We will implement a variety of methods with the goal of reducing the Docker build time and producing a slimmer image.

In each iteration, we’ll build the image four times. Before each build step, we’ll change the code to ensure it gets recompiled. The first build is considered a warm-up, so the result will be excluded. We will look at the results of the last three builds for each iteration.

Take note that I am using Podman instead of Docker; it is an alternative to Docker (a 1:1 replacement) and has a similar API.

Using a basic Dockerfile

For our first round, we’ll start with a Rust Docker image Installing and setting up Rust manually using a plain base image will take a longer build time:

# Dockerfile.1

ARG BASE_IMAGE=rust:1.70.0-slim-bullseye

FROM $BASE_IMAGE
WORKDIR app
COPY . .
RUN cargo build --release
CMD ["./target/release/hello"]

Now, we’ll run the build using the following command. The time prefix is added to the command to measure the execution time of the podman build process. By using time, we can see how long it takes for the build to complete, providing valuable information on its performance. time will output the elapsed real time, user CPU time, and system CPU time taken by the podman build command.

time podman build -f Dockerfile.1 -t hello-1:0.1.0

Next, we must ensure that the image doesn’t include any files unrelated to the build process.

# Run the image
$ podman run -p 7000:7000/tcp hello-1:0.1.0

# Check what is inside
$ podman exec -it container_name bash
root@2d3c937d40f0:/app# ls
Cargo.lock  Cargo.toml  src  target

Before exploring other methods, the crucial first step is to fine-tune the .dockerignore file. Docker doesn’t use .gitignore. .dockerignore is the file that Docker uses to exclude files unrelated to the build, such as documentation, changelogs, and the local target directory. Next, we’ll employ multi-stage builds where the .dockerignore plays a minor role, as only the binary enters the final image. However, completely disregarding the .dockerignore can impact overall build time due to the copying process of the first build stage.

This approach produces a 1.12GB image and took 59 seconds to build:

154.76s user 15.64s system 280% cpu 1:00.84 total
162.36s user 15.93s system 299% cpu 59.557 total
173.79s user 17.33s system 300% cpu 1:03.53 total

Using multi-stage builds

For our second round, we’ll take advantage of Docker’s multi-stage builds to optimize the image size. Before multi-stage builds were available, we needed to remove and clean up the build resources manually to keep the Docker image small.

As the name suggests, this approach works with multiple stages of a build. We can selectively choose what to pick from the previous build, resulting in a smaller final image:

# Dockerfile.2

ARG BASE_IMAGE=rust:1.70.0-slim-bullseye

FROM $BASE_IMAGE as builder
WORKDIR app
COPY . .
RUN cargo build --release

FROM $BASE_IMAGE
COPY --from=builder /app/target/release/hello /
CMD ["./hello"]

This approach produces an 838MB image and took 1 minute to build:

156.80s user 15.98s system 282% cpu 1:01.10 total
174.36s user 17.60s system 305% cpu 1:02.92 total
174.25s user 17.74s system 304% cpu 1:03.03 total

There is some improvement in the image size with this approach, but no reduction in build time.

Minimizing the binary size

Rust provides an option to reduce its binary size output. This option is set as an opt-out because Rust prefers fast compilation speed and ease of debugging.

Stripping the symbol, optimizing the binary size, and enabling Link Time Optimization (LTO) are some options that can be considered:

# Cargo.toml

[profile.release]
strip = true  # Automatically strip symbols from the binary.
opt-level = "z"  # Optimize for size.

This approach produces an 832MB image and took 48 seconds to build. Somehow, taking additional steps for stripping the binary size took less time than the previous approach:

99.40s user 12.71s system 230% cpu 48.728 total
106.82s user 13.66s system 216% cpu 55.738 total
117.68s user 15.26s system 236% cpu 56.221 total

Although the Docker image size is only 6Mb smaller than achieved with the previous approach, this still represents a huge win for small applications.

In an effort to further improve build time and file size, let’s look at different image files that we can use.

Using a scratch image

First, let’s try scratch, Docker’s official, minimal image. The scratch image does not contain glibc, so we’ll need to do the static linking to the musl C library to run the “Hello World!” app in the scratch image:

# Dockerfile.3

FROM rust:1.70.0-alpine3.18 as builder
# fixes `cannot find crti.o`
RUN apk add musl-dev
WORKDIR app
COPY . .
RUN cargo build --release

FROM scratch as runtime
COPY --from=builder /app/target/release/hello /
CMD ["./hello"]

Here’s the error we’ll get if we try to build the app with glibc and run it in a scratch image:

$ podman run -p 7000:7000/tcp hello-3:0.1.0
{"msg":"exec container process (missing dynamic library?) `//./hello`: No such file or directory","level":"error"}

This approach produces an 878kB image and took 1.22 minutes to build.

178.83s user 28.14s system 244% cpu 1:24.67 total
170.81s user 27.31s system 239% cpu 1:22.73 total
167.39s user 27.25s system 225% cpu 1:26.16 total

This iteration produced a large improvement in the file size, but the build took 34 seconds longer than the standard Docker image.

Using an alpine image

Rather than using a scratch image as a builder and setting up Rust manually, it’s easier to use an alpine image. Let’s give the alpine image a try:

# Dockerfile.4

FROM rust:1.70.0-alpine3.18 as builder
RUN apk add musl-dev
WORKDIR app
COPY . .
RUN cargo build --release

FROM alpine:3.18.0
COPY --from=builder /app/target/release/hello /
CMD ["./hello"]

This approach produces an 8.5MB image and took 1.23 minutes to build:

175.92s user 27.78s system 241% cpu 1:24.36 total
170.38s user 27.52s system 231% cpu 1:25.62 tot
178.53s user 28.50s system 249% cpu 1:23.07 total

Using a distroless image

With distroless, we get a bigger image size but a faster build time. This approach produces a 25.4MB image and took 45 seconds to build:

# Dockerfile.5

FROM rust:1.70.0-slim-bullseye as builder
WORKDIR app
COPY . .
RUN cargo build --release

FROM gcr.io/distroless/cc-debian11
COPY --from=builder /app/target/release/hello /
CMD ["./hello"]

This approach produces a 25.4MB image and took 45 seconds to build:

98.97s user 12.84s system 232% cpu 48.012 total
98.96s user 12.68s system 237% cpu 47.092 total
104.45s user 13.26s system 220% cpu 53.389 total

Speeding up build time with cargo-chef

Another way to speed up the build time is to use cargo-chef. Cargo-chef speeds up the build time by leveraging the Docker layer cache.

First, let’s try applying cargo-chef to the distroless image:

# Dockerfile.6

FROM lukemathwalker/cargo-chef:latest-rust-1.70.0 as chef
WORKDIR app

FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json
COPY . .
RUN cargo build --release

FROM gcr.io/distroless/cc-debian11
COPY --from=builder /app/target/release/hello /
CMD ["./hello"]

This approach produces a 25.4MB image and took an astonishing 14 seconds to build:

16.07s user 1.80s system 121% cpu 14.651 total
16.02s user 1.83s system 120% cpu 14.802 total
16.24s user 1.77s system 121% cpu 14.851 total

By inspecting the build log, I can see that compiling the app is the only slow process. When I run the build command, I don’t see any dependencies being recompiled.

Now, let’s try applying cargo-chef with the scratch image. This approach produces an 878 kB image and took only 22 seconds to build:

# Dockerfile.7

FROM lukemathwalker/cargo-chef:latest-rust-1.70.0-alpine3.18 as chef
WORKDIR app

FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json
COPY . .
RUN cargo build --release

FROM scratch
COPY --from=builder /app/target/release/hello /
CMD ["./hello"]
25.77s user 3.55s system 132% cpu 22.190 total
25.67s user 3.71s system 131% cpu 22.329 total
25.83s user 3.64s system 131% cpu 22.346 total

Comparing performance gains

Below is a summary of the performance gains of the different approaches we’ve reviewed in this article.

Approach	Image size	Build time
Basic Dockerfile	1.12GB	59 seconds
Multi-stage builds	838MB	1 minute
Minimizing binary size	832MB	48 seconds
scratch image	878kB	1.22 minutes
alpine image	8.5MB	1.23 minutes
distroless image	25.4MB	45 seconds
cargo-chef with distroless image	25.4MB	14 seconds
cargo-chef with scratch image	878kB	22 seconds

Of the options we reviewed, the best approach to get a tiny image size and fast build time is using cargo-chef with a scratch image. If you can’t use musl due to the particular requirements of your application, opt for cargo-chef and distroless image. Then, use SlimToolkit (previously DockerSlim) to minify your final image. In our case, It can trim the distroless image size up to 33.07%.

Speeding up Rust app build time

Using a caching tool in the CI, such as rust-cache or sccache, will greatly improve your Rust app’s build time since caching restores the previous builds artifact in the next build. In GitHub Actions, rust-cache is easier to set up and more commonly used.

Another strategy for speeding up app build time is to switch to a faster linker, such as lld or mold. mold is faster, but lld is more stable and mature. You can also consider splitting your application into smaller crates to further improve build time.

Speeding up app testing

Another opportunity for CI/CD process optimization is the application testing phase. cargo-nextest claims to provide 3x times faster execution than cargo-test. In a simple “Hello World!” application, the difference is not much, but it has a significant impact on large applications as shown in its benchmark page.

Here, the cargo-nextest is slower than regular cargo:

# cargo
0.09s user 0.04s system 98% cpu 0.137 total
0.09s user 0.03s system 98% cpu 0.123 total
0.10s user 0.04s system 98% cpu 0.143 total

# cargo-nextest
0.11s user 0.05s system 105% cpu 0.150 total
0.11s user 0.05s system 105% cpu 0.155 total
0.11s user 0.05s system 105% cpu 0.157 total

Speeding up binary crate installation

Installing a binary crate in CI using cargo install <app name> is time consuming. We can speed up the installation by using cargo-binstall instead. It pulls the binary right away, and does not require any compilation.

To speed up binary crate installation in GitHub Actions, use install-action. This is my personal favorite; most of the contribution I make to speeding up the GitHub Actions workflow is attributed to simply changing cargo install to install-action.

Conclusion

In this article, we discussed different strategies for building the smallest and fastest Docker image for our Rust projects. The most effective method is using cargo-chef with a scratch image. We also reviewed other portions of the CI/CD process that can be optimized, such as testing, building the app, and binary crate installation.

I hope you enjoyed this article. If you have questions, please feel free to leave a comment.