In 2016, Volkswagen agreed to pay approximately $ 14.7 billion to settle allegations by the U.S. Environmental Protection Agency and the U.S. Department of Justice to compensate affected vehicle owners of the “Dieselgate” scandal. VW cheated in product tests by altering the software of VW vehicles when emissions tests were done. The software was designed to detect special testing conditions when the vehicle was undergoing structured emission testing.
If the software detected the test the engine power was cut down and an emissions control system was activated. The scandal came to light when researchers and independent organizations conducted real-world emission testing on the road.
This is only one example of product tests and how they help to figure out if products meet the expectations of the users and the authorities. Done right product testing can be very helpful to ensure that expectations are met, done wrong and the tests are useless and inconclusive.
In this article, you will learn more about what product testing is, different types of testing, and you can conduct your own.
Product testing is the systematic process of evaluating and assessing a product’s quality, performance, safety, and compliance with industry standards and regulations to ensure it meets or exceeds customer expectations before being introduced to the market.
e the market and product concept only once and cannot adapt to changing conditions.
Product testing is important within product management because it can help you:
Testing hypotheses and assumptions in the problem space helps you gain market insight and insights into the needs of the users. This helps to reduce the risk of developing a product that isn’t needed by the market or users.
By testing the quality of the product and gathering feedback through tests in the solution space, product testing can help to improve the product to better meet quality standards and user expectations.
Through early feedback and elimination of risks, product testing can help save development time. Only faceable features are developed that will actually be appreciated by the user.
Product tests can also test if the users are willing to pay for the product and can even figure out the right price point for a product. Testing the product reduces development risks and assures that the business model is viable.
You conduct product testing to reduce product risk and validate the assumptions and hypotheses you make. Good product managers involve the entire team in testing and create a culture of regular testing and learning. Agile teams collaborate with users and product managers to conduct continuous product discovery.
Each test is different and serves a different purpose. Some tests are easier, others are more difficult and require more effort. It is important to plan the test properly and prioritize the tests accordingly. Conducting the right test also depends on:
You can also distinguish the tests according to what you’re testing. Are you testing in the solution space or the problem space? Another way to distinguish between the different types of tests is to look at how the test is conducted. If the test is done with a small group of users and you interact directly with the users, it is categorized as a qualitative test, whereas if the test is conducted on a broader basis without direct interaction, it is a quantitative test:
| Qualitative tests | Quantitative tests | |
|---|---|---|
| Problem space | Marketing materials test Marketing video test Copy writing and analytics |
Landing page performance Explainer video responses Marketing A/B tests Conjoint analysis |
| Solution space | Wireframe interviews Mockup tests Concierge tests Testing interactive prototypes User story maps with emotion graphs Usability tests User acceptance testing |
QA tests Smoke tests Product A/B tests Feature A/B tests Fake door tests 404 pages tests Load testing Analytics Regression testing Functional tests Component tests Integration tests System tests Performance tests |
Testing a product is time-consuming. Depending on the type of test and the product, the effort required to perform a test varies greatly. However, the purpose of the test and the type of product also determine whether it is possible to automate tests.
Functional tests, for example, are used to assess whether the product’s features and functions work as intended. They are used for quality control and quality assurance. For a software product, it may be possible to automate much of the testing.
In general, manual tests cost more than automated tests. For this reason, tests in software products should be structured as shown in the test pyramid:
Unit tests are automated tests that are written in code. They’re the smallest possible tests in a software system. They test small blocks of code, called units, and check whether they work as intended.
Component tests are also automated tests written in code that test larger components often via interfaces. They ensure that entire components behave as expected.
Integration tests are used to test the integration and interaction of components and larger parts of the software system. They test how different components interact with each other and can be fully automated.
System tests evaluate the entire software system. They test end-to-end functionality from the user’s perspective. System tests are often automated UI tests.
Manual tests are performed by human testers who interact with the application’s user interface and functionality.
Automated tests are developed once and can be run regularly. This ensures that the quality of the product is maintained permanently. If the product changes, the tests must also be maintained, which increases costs, but in the long run, automated tests ensure quality and save costs because they make bugs and errors visible early in the process.
This is illustrated by the Rule of 10 and the following graphic:
The Rule of 10 states that finding bugs early in the process saves 10 times more money and effort than finding them at the next step. This means that you save a lot of costs by detecting bugs and errors right away when they occur with automated testing. It’s also why assumptions about the market and user needs need to be tested from the beginning.
Conducting a product test is a systematic process that involves planning, execution, and analysis. Here’s a step-by-step guide on how to conduct a product test:
Clearly define the objectives of the product test. What are you trying to achieve or evaluate with this test? Identify specific goals, metrics, and success criteria.
Alexander Osterwalder recommends using test cards to structure and plan the tests.
In a test card, you specify the assumption you want to test, how you will verify it, with which test method, and how you know your assumption is right or wrong.
Determine the scope of the test, including the features, functionalities, or aspects of the product to be tested.
Identify the target audience or user group for the test.
Create a detailed test plan that outlines the test scenarios, test cases, and test data required.
Set up the testing environment, including any necessary hardware, software, or tools.
If the test involves real users, recruit participants who represent your target audience. Ensure they are willing to provide honest feedback.
Clearly communicate the purpose and expectations of the test to participants.
Execute the test plan according to the defined scenarios and cases.
Encourage participants to use the product as they would in a real-world situation.
Document observations, issues, and feedback as they arise during the test.
Collect both quantitative data (e.g., performance metrics, completion times) and qualitative data (e.g., user feedback, observations).
Use surveys, questionnaires, or structured interviews to gather user opinions and impressions.
Evaluate the collected data against the defined objectives and goals.
Identify any issues, defects, or areas of improvement.
Consider both common patterns and outliers in user feedback.
Prepare a comprehensive test report summarizing the test process, findings, and recommendations.
Clearly document any defects, issues, or areas of concern.
Include actionable insights and suggestions for improvements.
Based on the test results, make informed decisions about whether to proceed with the product as is, make improvements, or conduct further testing.
Prioritize issues and improvements based on their impact and importance.
If improvements are necessary, work with the development team to implement changes and fixes based on the test findings.
Ensure that any modifications align with the project’s timelines and objectives.
If significant changes were made, conduct follow-up testing to validate that the issues have been addressed and that the product now meets its goals.
Use the feedback and insights from the test to continuously improve the product.
Consider conducting further testing as the product evolves or new features are introduced.
After any issues have been resolved and improvements made, perform a final evaluation to ensure that the product now meets the desired quality and performance standards.
Throughout the process, clear communication and collaboration between the test team, product managers, developers, and stakeholders are essential to successful product testing. Customize the process and steps as needed to meet the specific requirements and complexity of the product under test.
Spotify is known to actively use feature toggles (also known as feature flags) and A/B testing as part of their software development and product optimization strategies:
Spotify uses feature toggles to enable or disable specific features within its applications or services without the need for a code deployment. This approach allows it to have more control over feature releases and to test new functionality with select groups of users. Some key aspects of how Spotify utilizes feature toggles include:
Spotify employs A/B testing to optimize user experiences and test hypotheses about new features or changes. Here’s how A/B testing works at Spotify:
Both feature toggles and A/B testing align with Spotify’s agile and data-driven approach to product development, allowing it to release features incrementally, respond to user feedback, and optimize the user experience over time.
Product testing is a big and important topic that, when done correctly, helps product development teams bring their products to success.
Product testing is important and can save you a lot of time and money. Experienced agile product teams know that product tests are done all the time, and ideally they are designed to cover as broadly as possible the topics that need to be tested.
Product tests serve to confirm assumptions and hypotheses and to create products that are needed by the user. Therefore, tests that are in the problem space reduce the product risk. Successful product development teams also have a large number of functional tests that are automated and run regularly. This makes it possible to find bugs early and save costs.
Featured image source: IconScout
LogRocket identifies friction points in the user experience so you can make informed decisions about product and design changes that must happen to hit your goals.
With LogRocket, you can understand the scope of the issues affecting your product and prioritize the changes that need to be made. LogRocket simplifies workflows by allowing Engineering, Product, UX, and Design teams to work from the same data as you, eliminating any confusion about what needs to be done.
Get your teams on the same page — try LogRocket today.
In our pursuit of innovation, we must be aware of the crucial aspect of letting go — the decision to shut down a product feature.
Justin Kim, VP, Product at Vimeo, discusses the importance of applying ruthless prioritization across the board and curating a clear vision.
Managing a diverse range of products is an effective way to showcase strategic planning, forward-looking thinking, and a commitment to meeting the evolving demands of customers.
Firmographic data refers to traits about a company that can be used to segment it into different categories.
Learn more →
