As a product manager, one of your most important tasks involves launching new products. Imagine you spend months conducting market research, determining your MVP, and troubleshooting bugs. Finally it comes time to get the word out about your product to your target audience. With your product marketing team, you decide to use four different promotional channels: Email, WhatsApp, TV, and social media.
After, with your launch in the rearview, your stakeholders ask you whether you experienced a significant difference in sales generated by these campaigns. They want to know if certain campaigns performed better or worse than others to determine a best practice for the future.
How would you go about doing this?
One of the best ways to tackle the request would be to use ANOVA (analysis of variance). Keep reading to learn about what ANOVA is, the different types you can run, and how to perform it.
ANOVA, which stands for analysis of variance, is a statistical method researchers use to compare multiple groups simultaneously to determine whether there are any statistically significant differences between them.
ANOVA’s history dates back to the early 1900s when it was primarily developed by the renowned statistician Ronald Fisher. The core principle of ANOVA lies in partitioning the total variability in the data into variability between and within groups. ANOVA compares these two sources of variability and tests the null hypothesis that assumes that all group means are equal.
You can use ANOVA when you need to compare more than two groups and see if there are any significant differences in their performances.
Overall, ANOVA is best suited when:
ANOVA is widely applicable where comparing outcomes across groups is essential for decision-making. Some examples include:
As a statistical method, ANOVA comes with its own set of terms that are important to understand before you attempt to implement it within your product team. These include:
ANOVA is categorized based on the number of variables participating in the experiment and the involvement of subjects in the study. The most important ones include:
As the name suggests, one-way ANOVA tests the difference in groups with one independent variable.
For example, say you want to test the effect of different diets on weight loss. You divide participants into three groups based on their diet type (e.g., diet A, diet B, and diet C) and measure weight loss in kilograms. In this case, there is only one independent variable (diet type) with multiple (three) levels, and the dependent variable is weight loss.
Two-way ANOVA analyzes the effect of each independent variable and its interaction with a dependent variable.
In this case, you might run a study on the effect of exercise type (cardio, strength, no exercise) and diet type (diet A, diet B, diet C) on weight loss. Here, there are two independent variables: exercise type and diet type, and the dependent variable is weight loss.
Repeated measures ANOVA is used when you perform the tests on the same participants under different conditions or at different times. This type helps to assess if there are significant differences in the dependent variables across the repeated measurements within the same subjects captured under different conditions or times.
For example, if you measure participants’ blood pressure before, during, and after a treatment. Since each participant’s blood pressure is recorded three times in different conditions from the same subject, this requires repeated measures ANOVA.
Repeated measures can be applied on one-way or two-way ANOVA where tests are done based on certain conditions over time.
If you have multiple dependent variables, you might want to use multivariate ANOVA (MANOVA). It allows you to assess if the mean vectors of multiple dependent variables differ across the levels of one or more independent variables.
For example, if you were examining the effect of different teaching methods (in-person, online, hybrid) on students’ performance across several subjects (math, science, and language). Here, you have one independent variable (teaching method) with three levels, and multiple dependent variables (scores in math, science, and language).
The chart below illustrates the different types of ANOVA and when to use each:
ANOVA Type | Independent Variables | Dependent Variables | Repeated Measures? | When to Use |
One-way | One | One | No | Comparing one factor with three plus groups |
Two-way | Two | One | No | Comparing main effects and their interaction effects |
One-way Repeated | One | One | Yes | Same subjects but different conditions or multiple time points. |
Two-way Repeated | Two | One | Yes | Two factors repeated measurements on the same subjects |
MANOVA | One plus | Two plus | No | Multiple outcomes (mean vectors) measured |
To help you understand how to run ANOVA, consider the following example and the steps required:
Problem — A product marketing team runs three campaigns in different regions for a newly launched product: Social media ads, email marketing, and TV commercials
The marketing team wants to know if the average sales generated by different marketing campaigns are significantly different.
Gather the sales performance data for each marketing campaign. For example, say each campaign runs in three regions and generates the following sales for each region:
Now, for ANOVA, you would:
With a p-value of 0.19 (not less than 0.05), you fail to reject the null hypothesis. This means there isn’t enough evidence to suggest that the sales performance varies significantly across the marketing campaigns.
Since you didn’t reject the null hypothesis, post-hoc tests aren’t needed in this case. However, if the null hypothesis was rejected and an alternate hypothesis was selected, post-hoc tests such as Tukey’s HSD would need to be used to compare pairwise between all group means to identify specific differences.
ANOVA is a statistical method used to compare the means of three or more groups to determine if at least one group’s mean is significantly different from the others. It helps you identify variations among group means and assess the impact of different factors on a dependent variable. Understanding ANOVA enables you to make data-driven decisions, understand relationships and variability within and between key drivers, and run post-hoc analysis.
Featured image source: IconScout
LogRocket identifies friction points in the user experience so you can make informed decisions about product and design changes that must happen to hit your goals.
With LogRocket, you can understand the scope of the issues affecting your product and prioritize the changes that need to be made. LogRocket simplifies workflows by allowing Engineering, Product, UX, and Design teams to work from the same data as you, eliminating any confusion about what needs to be done.
Get your teams on the same page — try LogRocket today.
Concept evaluation bridges the gap between what seems like an out-of-the-world idea and what users truly need.
Nick Ehle talks about minimizing dependencies by designing teams and organizations to be as empowered as possible.
Value-based pricing is about using the perceived value, also referred to as willingness-to-pay, to set the right price points for the product.
Carolina Devia Angarita about the importance of understanding who your customers are and what they’re thinking when they come to your site.