A guide to product testing

Every course I teach on product management contains an early section about the scientific method. With the scientific method, you can only draw a factual conclusion if you can devise an experiment that produces close to identical results in the same conditions. I try to imprint this level of scientific thinking on the PMs I meet with.

Much like a scientific experiment, you need to develop products that produce identical results time and time again. But how can you ensure this? Start with product testing.

This article covers all aspects of product testing from the different types, to benefits and challenges.

What is product testing?

Product testing, also known as consumer or comparative testing, evaluates a product’s performance, safety, quality, and compliance with established standards and set goals. This leads to the creation of products that meet customer expectations and regulatory requirements before they’re released to the market. It also minimizes the risk of failure and helps uncover improvements that can elevate any product work to new heights!

Types of product testing

Depending on the stage of your product, select a type of product testing best suited for what you aim to evaluate. Some of the most common ones include:

Concept testing

You might also hear this called early product discovery. It aims to evaluate the feasibility and potential success of a product idea or concept.

If you have competing ideas and resources, you can also use it to help determine the best approach to take.

Example: Coca-Cola conducted concept testing for Sprite Zero, a low-calorie version of Sprite, by partnering with Grofers, a grocery delivery app, to gather feedback from potential customers before a full-scale launch.

Quality assurance (QA) testing

QA might have been the first thing that came to mind when you saw the title of this article.

Nonetheless, QA seeks to discover problems or bugs in the product before it goes out to the customer.

You can have either manual or automated QA, but since almost every code development already incorporates QA, I would focus more on learning about the other types of testing.

A/B testing

With A/B testing, you compare two (or more) versions of a product (usually changed and unchanged) to determine which performs better. You should use this anytime you want to test or prove the value of a proposed product change.

Example: HubSpot Academy conducted an A/B test on its homepage hero image to see if changing the value proposition would increase user engagement. It found that a clearer value proposition significantly improved user interaction.

Market testing

Market testing looks like concept testing, but instead of testing an idea, you make a completed product (or its early version) available to a small group of clients/users to gather feedback and offer a better product to the wider audience.

Example: In October 1985, Nintendo released 100,000 units of the NES to select retailers in New York City. Encouraged by the positive response in New York, Nintendo expanded the test market to Los Angeles in February 1986, followed by Chicago and San Francisco. After successful test runs, the NES was launched nationwide in September 1986.

User testing

User testing involves real users interacting with the product to identify usability issues. This can be done either on a prototype or a finished product. Doing this helps to ensure you develop a user-friendly product that meets customer needs.

Example: Bannersnack utilized user testing to improve its landing page. By analyzing heatmaps and conducting A/B tests, it identified areas for improvement and increased sign-ups and feature adoption.

Regression testing

Regression testing analyzes the existing functionality to ensure that new changes haven’t introduced any issues. Instead of testing if the new update works, this aspect checks if the whole product still works well with the addition of a new code.

Example: Running tests on a software application after adding new features to ensure everything still works correctly.

Benefits of product testing

Product testing offers numerous advantages, ensuring that products meet customer expectations and regulatory standards before they hit the market. Here are some key benefits:

QA

• Customer satisfaction — High-quality products lead to higher customer satisfaction and loyalty, or at least fewer people not liking the product. Remember, happy users rarely leave a review!
• Brand reputation — Consistently delivering quality products enhances brand credibility. You want to be famous for great quality
• Compliance — Ensure products meet industry standards and regulatory requirements, avoiding legal issues, particularly by picking up any privacy issues, intended or otherwise

Incorporating user feedback

• Improved usability — Direct feedback helps identify usability issues and areas for improvement. Simply put, some early users might need to experience sub-optimal UI so that you, the product manager, know how to make it perfect
• User-centric design — Align product features with user needs and preferences. This way the product primarily serves the users, not the business goals

Reduction of risks and costs

• Early issue detection — Identify defects and quality issues early in the development process to prevent costly redesigns and recalls. You want to catch issues and fix them long before the product hits the users
• Market validation — Testing ensures that the product meets market needs, reducing the risk of launching a product that fails. It’s great to know that the product idea will likely turn into financial success before spending millions building it
• Cost savings — By addressing potential problems early, you can save on future support and maintenance costs, not to mention lost confidence
• Risk management — Minimizes the risk of product failures, recalls, and negative user experiences, protecting the brand’s reputation. The more you test, the lower the risk, but also the later the release

Product testing process

Now, to help you get started, try implementing the following product testing process within your team:

1. Define goals and objectives

Set clear goals and objectives so your testing process remains focused and purposeful. Well-defined objectives allow testers to concentrate on critical areas, ensuring that the product meets specified quality standards.

In simpler words, you want to define what you want to achieve by the end of testing (i.e. successful release of a new feature) and how to measure the success (i.e. the new feature increases monthly active users by at least 10 percent).

2. Plan

Determine the type of testing to use and the scope. Keep in mind that the more testing you run, the later a new feature goes live. While you may think you want to simply apply all listed aspects, they all require time.

Remember to agree with the development team and write down a detailed test plan outlining the testing strategy, resources, timeline, and responsibilities

3. Secure the right people

While you may have Q/A engineers on staff, they might not be exactly what you need in the discovery phase. You might need to recruit willing users and have a dedicated department (or external company) to do that for you.

4. Test!

Execute your plan and see what another high-quality and well-tested feature gets made.

5. Data collection and analysis

Be sure to collect quantitative and qualitative data from the tests. This includes performance metrics, user feedback, and defect reports. This might be useful later when presenting results to the stakeholders and/or deciding what to do next with the new update.

6. Documentation and communication

Remember to document the findings, especially coming from discovery and A/B testing. This information might be extremely useful for you or other PMs in the company when they decide to run a similar update in their respective areas of the product.

In fact, go the extra mile and inform all potentially interested parties (say, all the product people in the company) about your results using email or group chat.

7. Next steps and decisions

Based on the test results, make informed decisions about product modifications, improvements, or readiness for market launch.

Challenges in product testing

There are two main challenges when it comes to the product testing:

Biases

Allowing biases to creep into your tests can defeat the point of testing in the first place. Keep an eye out for these three:

• Sampling bias — When the sample used for testing fails to represent the entire population. To counteract it, use random sampling. You may even want to check a few samples just to be certain
• Confirmation bias — Testers may focus on results that confirm their preconceptions, ignoring data that contradicts them. Here blind testing comes to help! The less the tester knows, the less likely their bias impacts the result
• Observer bias — The tester’s expectations influence the outcome of the test. To widen your perspective, involve a diverse group of testers to bring different perspectives and reduce individual biases

Limited resources

You need to test as much as you can within the limits of your update, which can include:

• Time constraints — Your stakeholders might hold you to a specific release date for your update. In that case, you need to prioritize which testing aspects are essential and which can be dropped
• Budget constraints — You have time for discovery, but an insufficient budget for necessary investments. If you can’t recruit users, try discovery on at least your friend circle and people in your company outside your team
• Lack of skilled personnel — You may have all your QA engineers unavailable, say during flu season. However, for QA testing you and your dev team can simply get your hands dirty and test yourselves! However, if this situation drags on, you may want to invest in automated testing tools to save time and resources

Conclusion

As you can see, testing introduces some challenges into your workflow. However, without it, products can fail because of quality concerns and poor market fit. Apply these product management testing practices and come back for the next article!

Featured image source: IconScout