A great product manager should know how to use analytical tools to validate assumptions and drive product insights. One of these tools is an odds ratio analysis.
In this article you will learn what odds ratio analysis is, how to calculate it, and strategies for testing the strength of your findings.
Odds ratio analysis is a statistical tool used to measure the strength of an association between two events. In simpler terms, odds ratio analysis tells us the likelihood of situation A occurring when situation B has occurred.
For example, odd ratio analysis can answer business questions like:
You can use odds ratio analysis to test your assumptions and experiments to explore patterns within your product.
At first glance, odds ratio analysis and correlation analysis seem to answer similar questions, however they differ in the type of outcomes each seeks to analyze. Odds ratio analysis looks at discrete, or binary outcomes, whereas correlation analysis examines continuous outcomes.
We can use odds ratio analysis to analyze discrete outcomes. These are the “yes or no” outcomes, such as:
On the other hand, correlation analysis helps us analyze continuous outcomes. These are the “how much” outcomes, such as:
In this article, we’ll focus on analyzing discrete outcomes with odds ratio analysis. If you need to measure a continuous outcome, then check out our correlation analysis article.
In order to calculate the odds ratio use the following four steps:
The first step is gathering all the data you want to analyze and labeling it. You will need:
(It doesn’t matter whether you use true/false, Y/N, or 1/0 for inputs as long as your analytical tool can analyze it)
For example, if you want to analyze how a particular feature adoption impacts 30 day retention, you’d need to label
You need both users that adopted the feature, as well as those who didn’t to have a comparison group for your analysis.
The next step is to bucket that data into four specific groups. Let’s call them quadrants:
Quadrant A — Users who completed both the action and achieved the desired outcome
Quadrant B — Users who completed the action, but didn’t achieve the desired outcome
Quadrant C — Users who didn’t complete the action, but still achieved the desired outcome
Quadrant D — Users who neither completed the action nor achieved the desired outcome
This exercise will give you four groups of users to serve as the basis for our odds ratio analysis:
For our example, let’s assume that after labeling all our data, we have:
You calculate the odds ratio by dividing the odds of the desired outcomes (e.g., retention) when users do the action (e.g., feature adoption) by the odds of the desired outcome when users don’t do the action:
To calculate the odds of the desired outcome if the user does X, we divide the number of users who experienced the action and achieved the outcome (quadrant A) by the number of users who experienced the action and didn’t achieve the outcome (quadrant B):
In this case:
Next, let’s compare the odds of the outcome even if people didn’t adopt the feature. Here, we divide people who didn’t adopt the feature, but retained anyway (quadrant C) with those who didn’t (quadrant D):
In our example:
Now that we have both these numbers, we can now calculate our odds ratio:
The odds ratio of 1.07 tells us that people who adopted feature X are seven percent more likely to retain it for 30 days than people who didn’t.
However, it’s not a definitive statement. Other variables, seasonality factors, and sometimes just pure luck influence calculations.
As a rule of thumb:
We can derisk our odds ratio analysis by calculating the negative predictive value (NPV) and the positive predictive value (PPV) as guardrail checks.
These values help us assess what proportion of our outcomes may have been false positives and negatives.
For example, if out of 100 people that reached the desired outcome, 100 of them did the action, then it’s a powerful signal. If only 20 people did the action, but 100 reached the outcome, there’s a higher chance that the outcome isn’t related to the action. NPV and PPV help us measure that.
Below is the formula for both NPV and PPV:
The closer both of these values are to one, the stronger the relationship is between the outcome and the action.
If you achieve:
Then even though the odds ratio seems promising, there’s a high chance it’s just pure luck. Consider exploring different opportunities or conducting an odds ratio analysis on a more significant sample to rule out luck.
On the other hand, the results of:
Give us a strong signal that a specific action could potentially double the chances of achieving the desired outcome.
Always stress-test your findings.
Odds ratio is a powerful tool to help you validate your experiments and explore new hypotheses.
It’s used for measuring discretionary types of outcomes, and the overall process has the following four steps:
If you are exploring new opportunities, it’s rare to get strong insights on your first try. You often need to run an odds ratio analysis multiple times with different variables to discover valuable insights.
Remember, you want your odds ratio to be above two, and your NPV and PPV as close to one as possible for an insight to be worth pursuing.
LogRocket identifies friction points in the user experience so you can make informed decisions about product and design changes that must happen to hit your goals.
With LogRocket, you can understand the scope of the issues affecting your product and prioritize the changes that need to be made. LogRocket simplifies workflows by allowing Engineering, Product, UX, and Design teams to work from the same data as you, eliminating any confusion about what needs to be done.
Get your teams on the same page — try LogRocket today.
The globalization of your product opens up opportunities for growth, however, every new market comes with its own challenges.
Hypergrowth happens when a company experiences an exceptionally rapid rate of expansion, typically more than 40 percent annual growth.
Detractors have long-term effects like negative brand perception, reduced customer loyalty, and a decrease in sales.
To proactively address liability concerns, you can create an internal product recall team for dealing with risks and ensuring quality.
Learn more →
