Multi-armed bandits in UX experiments: Faster testing with smarter traffic splits

Traditional A/B testing splits incoming traffic equally (50/50%) for design versions A and B. Then, half of the incoming traffic will receive the original version, and the other half will receive the new design variant during the experimentation period. This fixed equal traffic distribution generates more statistically accurate results with unbiased data, but is slow, and negatively affects the conversion rate during experimentation since users who will receive the poorly-performing design version may not convert.

The multi-armed bandit strategy helps conduct A/B, A/B/n, or multivariate tests faster by dynamically splitting traffic based on the winning version. Let’s understand how multi-armed bandits handle A/B tests faster without affecting the conversion rate by comparing with the traditional A/B testing traffic distribution method.

What are multi-armed bandits?

In UI/UX A/B testing, the multi-armed bandits (MAB) strategy is a dynamic incoming traffic distribution method that redirects more traffic to the current winning design version. It initially starts with a traditional 50/50% split, but later dynamically increases traffic to the highest-performing version to stabilize the selected test metrics and traffic percentages to conclude the test faster:

MAB isn’t limited to A/B testing — you can use it to optimize traffic for A/B/n and multivariate tests.

The slot machine example is the simplest way to understand MAB:

Assume that there are two slot machines with unknown reward rates. How do you find the most rewarding machine without losing more money?

• First, you play equally with A and B
• If machine B performs well, you’ll use it more often
• Still uses machine A occasionally and starts playing it often if it rewards better
• Selects the machine that rewards the most

How do multi-armed bandits work in UX?

Let’s discuss each generic step that an MAB A/B test performs to optimize traffic:

Over 200k developers and product managers use LogRocket to create better digital experiences

Learn more →

1. Fair initialization: Initializes the A/B test with 50/50% traffic distribution to give each version a fair amount of traffic. In A/B/n testing, the algorithm initially distributes traffic based on the number of versions, e.g., 25% for a four-version A/B/n test
2. Monitoring: Monitors metric changes and identifies the top-performing version for A/B tests and ranks all versions for A/B/n and multivariate tests
3. Traffic distribution adjustments: Adjusts traffic distribution percentages based on monitoring results. For example, if version B has 5% conversion, but version A has only 4%, the algorithm increases traffic for version B since it’s more successful
4. Stabilizing: The algorithm’s goal is to stabilize metrics to stop the process. This usually happens when new traffic doesn’t heavily alter the current traffic distribution percentages
5. Stopping: The MAB process will stop when traffic distribution is stabilized (the dominant version may stabilize the traffic percentage at 80%-95%) and no significant changes in metrics are observed. For example, if the conversion rates for versions A and B stabilize at 6% and 5%, the process will stop

Benefits of using multi-armed bandits

Using MAB instead of traditional fixed traffic distribution has the following benefits:

• Shorter tests: MAB tests stop when a specific version dominates, so the test is usually faster than a traditional A/B test that waits till the result achieves statistical significance
• Better conversion during the experiment: The top-performing version gets more traffic, so the conversion rate will increase, even if a specific version performs poorly, unlike the fixed, equal traffic split
• Less wasted traffic: MAB optimizes the incoming traffic during the experiment by sending less traffic to versions that perform poorly, so the incoming traffic isn’t wasted
• No fixed upfront sample size requirements: No need to calculate and meet statistical significance requirements for incoming traffic before starting the test — flexible start and progression with traffic flow

Limitations and pitfalls of multi-armed bandits

MAB has benefits, but the following issues made designers rethink using it over the classic statistical method:

• Depends on the MAB algorithm: The accuracy and trust of the result depend on the implementation quality of the MAB algorithm that dynamically splits traffic. Weak algorithms can generate inaccurate results
• No exact stopping time: You don’t know exactly when a MAB test will end, since it waits till traffic distribution stabilization, unlike classic A/B tests that start with a pre-defined duration
• Statistical bias: MAB starts with a fair traffic distribution, but dynamically adjusts percentages later, so the result is biased to the initial traffic
• Statistical significance isn’t guaranteed: You can’t present MAB test results with the usual 95% statistical significance since it doesn’t use a hypothesis-based statistical foundation for decision-making

Multi-armed bandits vs. traditional A/B testing

Here is the summary of MAB vs. the traditional A/B testing comparison:

Use case examples

An MAB test works best when you need to optimize traffic while quickly evaluating design versions. Here are some examples:

• CTAs: CTAs decide the product’s conversion rate, so MAB helps test design versions without losing traffic
• Onboarding flows: Testing two onboarding flows with a fixed 50/50% traffic split can increase the product abandonment rate if one version poorly performs. Using MAB prioritizes the top-performing flow without running the experiment for a long time
• Recommendation systems: Imagine you need to do a full-stack test for two e-commerce product recommendation algorithms on a homepage. Using MAB helps you find the good algorithm, also maximizing the revenue, unlike traditional A/B testing

Tips for designers

Here are some practical tips to run effective MAB tests:

• Avoid premature conclusions: Don’t stop the test forcefully even if you see a rapid increase in the early phase, always wait til traffic and metric stabilization
• Monitor continuously: Monitor for top-performing version changes, seasonal traffic effects, and stop it properly without overrunning
• Combine with human insights: The results depend on the quality of the multi-armed banding algorithm and early traffic, so analyze results yourself without blindly trusting the algorithm’s result

Conclusion

The multi-armed bandit strategy can be preferred over the traditional traffic distribution with A/B, A/B/n, or multivariate testing if you care more about wasted traffic and don’t care much about statistically significant results

FAQs

Is the multi-armed bandit method better than the traditional A/B testing?

Depends on the scenario. MAB is better if the losing traffic is critical, and traditional A/B testing is better when you prioritize statistical significance

Should I need to calculate traffic percentages frequently and notify developers?

No, A/B testing tools that support MAB handle everything — you just need to initialize the test

LogRocket helps you understand how users experience your product without needing to watch hundreds of session replays or talk to dozens of customers.

LogRocket's Galileo AI watches sessions and understands user feedback for you, automating the most time-intensive parts of your job and giving you more time to focus on great design.

See how design choices, interactions, and issues affect your users — get a demo of LogRocket today.