What is beta testing and how to do it

Have you ever purchased and used a product and thought to yourself, “Who designed this? Who approved it for release?” Either the product just didn’t make sense to you as a user or it was full of bugs and errors. Sound familiar?

Look I just have one question. And I’ve been on it since childhood. Who approved this product as tried and tested to be SUPER SOFT? I’m waiting for the answer paaaaa. 😔 pic.twitter.com/C0owZuB9Jz

— Francis Abban (@francisabban) September 27, 2019

When I am using this product I am so angry I want to punch the Product Manager who approved it so that I can have some measure of satisfaction from something. https://t.co/HwgkVrNORk

— Alan Cooper (@MrAlanCooper) June 10, 2020

If you haven’t felt the rage and confusion that results from using a terrible, buggy, or imprecise product, the tweets above illustrate that emotional toll quite well.

As a product manager, the last thing you want is to be punched. It hurts, both physically and in terms of your ego. That’s why you should always test your product before releasing it to the market.

In this guide, I’ll show you how to avoid punches and maybe even earn some hugs from your customers by doing beta testing.

Tests you should be doing as a product manager

There are many tests product managers can and should conduct to ensure the team is on course to deliver a stellar product. In no particular order, these activities include:

• Concept tests to validate the idea you would like to convert into a product
• A/B tests, whereby you offer two variants of your product and see which one performs better
• Multivariate tests to determine which of more than two variants of your product performs the best
• Alpha tests to determine whether your product meets business requirements by having employees use it internally
• Software tests to ensure that the software works as designed and to identify bugs and performance issues. In most teams, this is done by the QA specialist
• User observation tests to monitor how people use the product without your interference. In my experience, this is the fun part because users are interesting and full of surprises 😃
• Beta tests, which are the focus of this guide

As a product manager, you shouldn’t feel pressured to do all these tests for every single release. Pick the most relevant tests based on available resources and the data you’re trying to get from the test.

What is beta testing?

PMs need to gather information at every stage of the product development process to help measure, learn, and then design the right solutions for their users.

Beta testing is an important activity to help product managers validate the product hypothesis and gather initial feedback from real-life users. It’s very common for product teams to be so close to the product that they’re unable to see issues or identify weird behavior.

Beta testing gives PMs a fresh pair of eyes to view the product differently. Best of all, these extra eyes belong to the actual users of the product, and they’ll most likely tell you their true feelings. This might hurt a bit at first, but it’s critical to the long-term viability of the product and the business.

Why should you beta test?

Beta testing is a great way to anticipate potential issues and bugs before you launch your product to the wider market.

Beta testers are often better equipped to observe things product teams overlook during development and other testing sessions. This enables the product team to collect meaningful user feedback and input about the product they wouldn’t otherwise receive from internal stakeholders.

In addition, beta testing is a valuable exercise because it enables product teams to:

• Collect data quickly
• Monitor engagement and retention before launch
• Identify previously undetected bugs and functionality issues
• Tweak the product as necessary before launch
• Observe real-life results
• Gradually introduce product features to the core user base

How to do beta testing

Beta testing is conducted in three phases:

1. Preparation and design
2. Conducting the beta test
3. Post-test feedback and communication

Preparation and design

The preparation and design phase involves the following steps:

1. Define goals and metrics
2. Ensure alignment on goals and metrics
3. Set test dates and duration
4. Gather your beta testers
5. Set up the test environment

Define goal and metrics

You and your team should be clear on what exactly is being tested for and why. What doesn’t get measured doesn’t get managed.

Some examples of metrics and KPIs for beta testing:

• Bugs detected
• User engagement by feature
• NPS score
• Product usability

Ensure alignment on goals and metrics

A common mistake PMs make is setting goals without the input of stakeholders.

You can get away with this for QA and AB tests, but beta tests require the input of customer-facing teams that are in touch with the users constantly. Naturally, these teams are well-positioned to receive customer feedback that is valuable to the product team.

In addition, the results of a beta test might dictate changes to the way operations teams work, so it’s always good to keep them in the loop from the get-go.

Set test dates and duration

A beta test shouldn’t run for too long. Failure to properly time your beta test can result in skewed data and a decline in user interest. Not to mention, you have a product launch to plan.

At the same time, a beta test shouldn’t be too short; it needs to be long enough to allow you to collect data properly. I recommend between 14–60 days to beta test for a typical feature. The industry, goal, and type of product impacts this.

Gather your beta testers

You don’t just let anyone into your space, so why would you let anyone join your beta group? PMs need to be specific about their target audience for a beta test.

Your product most likely has more than one user type, so you need to determine whether you want results from all user types or you’d rather focus on one user type.

I recommend recruiting a diverse, representative group of users in your beta test community.

Beta testers are a different breed. Where would product teams be without them? They are usually:

• Early product adopters
• Willing to dedicate time to test your product and give you feedback
• Quite invested; the solution your product is offering should solve an actual problem for them

You can find beta users in the following places:

• On social media
• In online communities
• Within organizations and industries you’re building specifically for
• Your user base; don’t be afraid to ask!
• Your sales and customer service teams; they’re in touch with real and potential users daily

You might also want to have beta users sign legal T&Cs and NDAs, depending on the type of product you’re testing.

It is sometimes hard to find beta users. It’s a good practice to develop a pool of beta users you can reach out to over a period of time. That way, you don’t need to search for new users every time you want to beta test a product.

You should enable users to easily opt in or out of beta tests. If you have the resources, you might also consider offering an incentive. Ideally, this incentive should be something other than monetary compensation. People generally love free things, after all!

To save you from having to answer the same questions repeatedly during the beta testing period, you should prepare the following resources ahead of time for your beta users:

• Onboarding materials
• Demos
• Instructional documentation
• Blog posts
• FAQ documents
• Feedback materials
• Surveys

Set up the test environment

Beta tests happen before the product is launched which means the product shouldn’t be publicly available yet. You want to set up an environment that’s as close to production-ready and easily accessible as possible.

Conducting the beta test

In product management, there are very few one-size-fits-all approaches. Beta testing is no exception.

There are myriad ways to do beta testing, depending on the type of product, what you are testing for, and the availability of your users. But as a general framework, here’s what you should do:

1. Make an announcement — Send an announcement to the beta group you already gathered or to your potential group of beta users. This can be via email, blog post, community post, or even a phone campaign to distribute information about the beta test (e.g., duration, how to get involved, any incentives, etc.)
2. Onboard beta users — Guide users on how to use your new product or feature and help focus their attention
3. Monitor behavior — Use analytics to track actual behavior during the beta tests. Sometimes, the way users interact with your app may surprise you

Post-test feedback and communication

After you complete your beta test, you should take the following steps:

• Collect feedback — Collect as much feedback as possible from your beta testers. If possible, get both quantitative and qualitative feedback
• Thank your beta users — Show appreciation for your users who joined the beta tests and inform them of next steps. For example, if you’d like them become beta users in the future, now is the time to inform them
• Share next steps — Explain to your beta users what’s coming next on the roadmap. Remember, these are actual users of your product, so you want to keep them excited about your product, even if they give negative feedback about the feature they just tested

Featured image source: IconScout