Every course I teach on product management contains an early section about the scientific method. With the scientific method, you can only draw a factual conclusion if you can devise an experiment that produces close to identical results in the same conditions. I try to imprint this level of scientific thinking on the PMs I meet with.
Much like a scientific experiment, you need to develop products that produce identical results time and time again. But how can you ensure this? Start with product testing.
This article covers all aspects of product testing from the different types, to benefits and challenges.
Product testing, also known as consumer or comparative testing, evaluates a product’s performance, safety, quality, and compliance with established standards and set goals. This leads to the creation of products that meet customer expectations and regulatory requirements before they’re released to the market. It also minimizes the risk of failure and helps uncover improvements that can elevate any product work to new heights!
Depending on the stage of your product, select a type of product testing best suited for what you aim to evaluate. Some of the most common ones include:
You might also hear this called early product discovery. It aims to evaluate the feasibility and potential success of a product idea or concept.
If you have competing ideas and resources, you can also use it to help determine the best approach to take.
Example: Coca-Cola conducted concept testing for Sprite Zero, a low-calorie version of Sprite, by partnering with Grofers, a grocery delivery app, to gather feedback from potential customers before a full-scale launch.
QA might have been the first thing that came to mind when you saw the title of this article.
Nonetheless, QA seeks to discover problems or bugs in the product before it goes out to the customer.
You can have either manual or automated QA, but since almost every code development already incorporates QA, I would focus more on learning about the other types of testing.
With A/B testing, you compare two (or more) versions of a product (usually changed and unchanged) to determine which performs better. You should use this anytime you want to test or prove the value of a proposed product change.
Example: HubSpot Academy conducted an A/B test on its homepage hero image to see if changing the value proposition would increase user engagement. It found that a clearer value proposition significantly improved user interaction.
Market testing looks like concept testing, but instead of testing an idea, you make a completed product (or its early version) available to a small group of clients/users to gather feedback and offer a better product to the wider audience.
Example: In October 1985, Nintendo released 100,000 units of the NES to select retailers in New York City. Encouraged by the positive response in New York, Nintendo expanded the test market to Los Angeles in February 1986, followed by Chicago and San Francisco. After successful test runs, the NES was launched nationwide in September 1986.
User testing involves real users interacting with the product to identify usability issues. This can be done either on a prototype or a finished product. Doing this helps to ensure you develop a user-friendly product that meets customer needs.
Example: Bannersnack utilized user testing to improve its landing page. By analyzing heatmaps and conducting A/B tests, it identified areas for improvement and increased sign-ups and feature adoption.
Regression testing analyzes the existing functionality to ensure that new changes haven’t introduced any issues. Instead of testing if the new update works, this aspect checks if the whole product still works well with the addition of a new code.
Example: Running tests on a software application after adding new features to ensure everything still works correctly.
Product testing offers numerous advantages, ensuring that products meet customer expectations and regulatory standards before they hit the market. Here are some key benefits:
Now, to help you get started, try implementing the following product testing process within your team:
Set clear goals and objectives so your testing process remains focused and purposeful. Well-defined objectives allow testers to concentrate on critical areas, ensuring that the product meets specified quality standards.
In simpler words, you want to define what you want to achieve by the end of testing (i.e. successful release of a new feature) and how to measure the success (i.e. the new feature increases monthly active users by at least 10 percent).
Determine the type of testing to use and the scope. Keep in mind that the more testing you run, the later a new feature goes live. While you may think you want to simply apply all listed aspects, they all require time.
Remember to agree with the development team and write down a detailed test plan outlining the testing strategy, resources, timeline, and responsibilities
While you may have Q/A engineers on staff, they might not be exactly what you need in the discovery phase. You might need to recruit willing users and have a dedicated department (or external company) to do that for you.
Execute your plan and see what another high-quality and well-tested feature gets made.
Be sure to collect quantitative and qualitative data from the tests. This includes performance metrics, user feedback, and defect reports. This might be useful later when presenting results to the stakeholders and/or deciding what to do next with the new update.
Remember to document the findings, especially coming from discovery and A/B testing. This information might be extremely useful for you or other PMs in the company when they decide to run a similar update in their respective areas of the product.
In fact, go the extra mile and inform all potentially interested parties (say, all the product people in the company) about your results using email or group chat.
Based on the test results, make informed decisions about product modifications, improvements, or readiness for market launch.
There are two main challenges when it comes to the product testing:
BiasesAllowing biases to creep into your tests can defeat the point of testing in the first place. Keep an eye out for these three:
You need to test as much as you can within the limits of your update, which can include:
As you can see, testing introduces some challenges into your workflow. However, without it, products can fail because of quality concerns and poor market fit. Apply these product management testing practices and come back for the next article!
Featured image source: IconScout
LogRocket identifies friction points in the user experience so you can make informed decisions about product and design changes that must happen to hit your goals.
With LogRocket, you can understand the scope of the issues affecting your product and prioritize the changes that need to be made. LogRocket simplifies workflows by allowing Engineering, Product, UX, and Design teams to work from the same data as you, eliminating any confusion about what needs to be done.
Get your teams on the same page — try LogRocket today.
Colin Brooks shares how digital product management influences a traditionally in-person institution — The Whitney Museum of American Art.
We’ve all read “The Lean Startup.” We get the theory. But when it’s time to ship, somehow we mess it up anyway.
Dev Ward shares how she’s helped curate the digital experience for the WNBA’s growing fanbase, including redesigning and launching a new app.
From now on, app developers can avoid the 30 percent cut Apple takes from most transactions and opt to process payments on their own.
Learn more →
