Server-side A/B testing with Node.js

A/B testing is a common way for developers and product teams to understand how users engage differentially with their tools.

For example, if a team is launching a new feature, it would want to know if this feature does what it is meant to do — increase engagement, signups, purchases, and so on. By testing the new feature in an experiment, they’ll be able to determine the precise way the new feature affects the user experience vs a control group.

On my site, Solitaired, we A/B test on an ongoing basis. We test new features (using painted doors), new games, and new layouts. We start our A/B tests off at 10 percent and then scale our testing as we see positive engagement.

A big issue for us was setting up the A/B testing in the first place. Of course, there are tools out there that purport to make A/B testing easy — tools like Optimizely and Google Optimize.

However, the main focus of these tools is client-side — meaning the A/B testing changes happen after a page is loaded. While ease of use is one of the benefits of client-side testing, there are some major downsides to client-side testing:

• Page flickering as the A/B testing kicks in
• Limited mostly to visual changes like text, colors, etc.
• Multi-page testing is nearly impossible

That’s why most applications or serious A/B testing teams use server-side testing. Server-side testing is a little trickier to set up (but not that tricky), but has some added benefits:

• Improved speed and no on-page flickers
• Multi-page/multi-step testing
• Ability to sync with backend databases

We were happy enough with our A/B testing software that we released it open source. Here, we’ll walk through how to use our middleware for A/B testing for Node.js applications.

Install A/B testing middleware

Requirements

• Node and Express
• express-session for session management

You can start by installing the npm library, easy-abtest:

npm install easy-abtest

Then add the package to your app.js file:

const abtest = require('easy-abtest');

Further down in your file, add the middleware with the options argument (we’ll get into this below):

let options = {
  enabled: true,
  name: 'experiment-ID-here',
  buckets: [
    {variant: 0, weight: 0.40},
    {variant: 1, weight: 0.60}
  ]
}
app.use(abtest(options));

Note: if you use express.static, add the middleware code after it. Otherwise, it’ll run on every static asset call.

The options object can be described as follows:

• enabled (Boolean): this is so you can easily turn on or off your A/B testing code
• name: experiment name. This is a slug you can add, or if you are using Google Analytics or Mixpanel, you will need to add their slug into this field
• buckets: This is the good stuff. This is an array where you describe your variants. Each variant is an object with the following keys:
  • variant: 0 for control, 1 for the first cell, 2 for the second, and so on. Only the 0 bucket is truly required, but you should have an experiment cell as well
  • weight: this is the percentage of traffic this cell should take up. A value of 0.1 equals 10 percent, for example. All of your weights should add up to 100 percent

Now when a new user comes to your application, the middleware will run and assign a bucket to the user’s session. It also assigns the bucket to the local variables that can be used in your view templates.

• In your routers: req.session.test
• In your views: abTest

By being available in both your routers and your views, the bucket can be used to segment your users any way you’d like, e.g.:

• If you want to send one view template to your control users, and a different one to your experiment cell, you can call different render() functions:

if (req.session.test.bucket == 0) {
  return res.render('index');
} else if (req.session.test.bucket == 1) {
  return res.render('index-new');
}

• If you want to show different headlines to your users right in the view, you can do that too:

in homepage.pug
if abTest.bucket == 0
  h1 The best thing since sliced bread.
else if abTest.bucket == 1
  h1 The best thing since apple pie.

That’s it for the setup side. With access to the backend, view-templates, and client side, you can instrument your tests any way you want. (E.g., the team at Mojomox uses easy-abtest to figure out the order of steps to give to users on a multipage experience).

Connecting the testing system to reporting system

Although you can now run A/B tests in your app, you still need to know which tests won. That means you need to connect your experiments to some reporting backend. I’ve made a couple of suggestions below:

Connect to Google Analytics

Let’s say you want to track if one experiment results in more clicks on a button than another.

You can do this easily by adding the A/B test bucket data to the view as a JSON object, and then push up the appropriate events:

script.
    let abTest = !{JSON.stringify(abTest)};

  if abTest.bucket == 0
    button#cta Click here now
  else if abTest.bucket == 1
    button#cta Start today!

  script.
    $('#cta').on('click', function() {
      gtag('event', abTest.bucket, {
        'event_category': abTest.name,
        'event_label': 'start today'
      });
    });

If you want to use Google Optimize for your A/B testing product, you just need to follow the tutorial here and use Optimize Experiment ID as the experiment slug in the middleware options.

Create your own database

Similarly, you can track events in a database table you write yourself. Keep in mind the high volume of transactions that may occur. Regardless, you should include the following fields in your table:

1. Experiment name
2. Bucket variant
3. Value (e.g., 1 for “one button click”)
4. Date/time

Conclusion

Server-side A/B testing is clearly advantageous to the client side, but in the past it has required thinking about how to set it up. With the easy-abtest middleware, you can easily integrate A/B testing into your app.

What will you test next? A product price increase, or perhaps a new logo? With A/B testing, you can experiment continually to see what makes the best product for your users and your business.