Using concept evaluation to avoid product failures

Back in 2016, I had a startup called shoptosurprise, which was a venture into the online gifting space. I set out to build, what to me was one of the coolest features of the website — create your own gift.

The idea was simple: Users could create a hamper from a big selection of products that we had and send it to their friends and families as gifts. On paper, the idea was interesting.

As often happens with the startup mindset of “build fast, fail fast,” we decided to build the feature on the weekend and take it to live. Except, no one used it, even after a month.

Why? I talked to a few users, and they said they wouldn’t want to consider a hamper simply because they preferred pre-curated gifts that make their buying easier. Additionally, some users mentioned that they were unsure about what combinations would make for an ideal gift, which added to their hesitation. Essentially, while the concept sounded exciting to me as the co-founder and I was too excited for it, in reality, it didn’t resonate with the actual needs or preferences of the target audience.

This experience was a pivotal lesson in the importance of concept evaluation in product development. If I had conducted proper user research or tested the idea with a prototype, I could have uncovered these insights early on, saving both time and resources. Instead, we skipped the evaluation phase, leading to a feature that failed to deliver value.

The story underscores why concept evaluation is so critical in product management and how it bridges the gap between what seems like an out-of-the-world idea and what users truly need. It’s not just about building quickly; it’s about validating the right thing to build before committing to development.

This article explains what concept evaluation is and how you can leverage this concept to build better products — the ones that users actually use and pay you their hard-earned money.

What is concept evaluation?

Concept evaluation is when you take a concept and instead of immediately building it out, you try to understand the following:

• What is the concept? Define the idea clearly, including what problem it aims to solve
• Who are the users of the concept? Identify the target audience and understand their demographics, behaviors, and pain points
• Do the users need the concept? Validate whether the concept addresses a real need or problem for your target audience
• What’s the impact of the concept? Assess how the concept will affect users, the market, and your business objectives

Once this analysis is complete and you’ve gathered positive feedback through user research, surveys, or prototypes, you proceed to build the concept. This process ensures that you’re not just building fast, but building the right thing — and this took me a few product failures to understand.

One of the famous examples of concept evaluation is Amazon’s PR FAQ and its working backward approach. It’s a document that comprises of five main questions:

• Who is the customer?
• What is the customer’s problem?
• What is the most important customer benefit?
• How do you know what customers need or want?
• What does the customer experience look like?

The PR FAQ serves as a living document that undergoes multiple iterations, integrating inputs from all relevant teams. Only after the document is approved, reflecting clarity and alignment across the organization, does development begin.

This meticulous evaluation ensures that the product is designed with the customer at the center, significantly reducing the risk of failure while increasing the chances of creating impactful, user-driven solutions. Considering every company has limited resources and time, it’s super important to build products that the customers will eventually use.

Why concept evaluation matters for product success

My example at the start of the blog was good enough to understand that concept evaluation is mandatory. But in this section, I want to specifically explain what can go wrong if you don’t evaluate a concept beforehand.

Below are some of the important parameters that the concept evaluation can help with:

Identifying user needs

In today’s world, user’s needs change quickly. What might be true yesterday might not be true today. Hence as a PM, it’s important to be aware of what user needs you’re solving and whether they’re even relevant and worth solving.

Concept evaluation can help you discover this via user interviews, group interviews, market research, questionnaires, etc. By conducting user research and validating assumptions, teams can avoid building solutions for non-existent problems.

Optimizing resource allocation

As shared earlier, companies have limited resources and time, and it’s important to use them wisely. Thorough evaluation ensures resources — time, money, and talent — are invested in ideas with the highest potential for impact, reducing wasted efforts on unviable concepts.

This is even more important since the market is very competitive and incorrect resource utilization can prevent you from achieving your goals.

Reducing the risk of failure

With building new features comes the risk of failure. While no one can predict the success or failure of a concept, as a PM, you can definitely decrease the risk. How?

By rigorous validation, including early prototyping and user testing, identifying potential flaws, and giving control to the teams to address them before full-scale development. This minimizes the risk of costly post-launch failures thereby saving money, time, and resources.

Improving user trust

Better products mean better user trust. When you take the time to evaluate and validate concepts, the result is often a product that better meets user needs and expectations. Delivering thoughtful, user-centric solutions shows that you understand your audience and value their input.

This is one of the most important parts of product management. This not only builds trust with existing users but also attracts new customers, enhancing your brand reputation in the process.

Concept evaluation case study: Google Glass product failure

A few days ago I met a friend who was wearing Ray-Ban glasses. He could easily record anyone with just one click. Everyone at the party tried his glasses.

Smart glasses are becoming a part of our lives. But in 2013 Google came up with Google Glass. It looked something like this:

Unfortunately, the product didn’t do well and it was one of the biggest product failures of Google. Here are a few of the reasons:

Design

Google Glass had a futuristic design, but it was far from appealing to the average consumer. The unconventional frame made it look overly geeky and hard to relate to. People were reluctant to wear something that didn’t fit seamlessly into their daily lives or style preferences.

Lack of clear use-case

Google marketed Glass as a revolutionary product but failed to identify specific, compelling use cases. Consumers were unsure how it fits into their lives beyond being a novelty item. Without a clear value proposition, mass adoption was limited.

Unusually high price point

Priced at $1,500, Google Glass was prohibitively expensive for most consumers, especially given the lack of practical benefits. The cost further alienated potential adopters and combining this with a lack of a clear use case, it was a clear no-go.

Premature launch

Looking back, the technology was too early for mass-market adoption. The battery life was poor, the interface was clunky, and the product lacked polish. Google Glass felt more like a beta project than a finished product, leading to user dissatisfaction.

Lessons for PMs from Google Glass

While Google surely did a concept evaluation there were quite a few things that weren’t tested. Here are a few lessons PMs can learn from the failure of Google Glass:

Understanding the market

Concept evaluation helps you understand the market. Back then Google Glass was a revolutionary product without any use case. The lack of a strong use case means no real benefits for the users. Proper concept evaluation lets you understand the market early in the process and reduce risk.

Understanding the user

While Google tested the product with the initial set of users, there were a lot of bugs in the product that made the early users not like it. This can be considered a failure, but thanks to the concept evaluation, Google didn’t invest a lot and decided to kill the product. Lean on concept evaluation instead of building out a full-fledged product for mass consumption.

Balancing innovation with usability

Google Glass comes in at 100/100 for innovation but 20/100 for usability. This shows that a product that introduces cutting-edge technology must also be user-friendly. Work closely with design and engineering teams to ensure the technology seamlessly integrates into users’ lives without overwhelming them.

Timing is crucial

This was one of the most important reasons and reiterates the fact that timing is critical. Google Glass was released in 2013, at a time when wearable technology was still new. The product was a decade ahead of its time, as the necessary supporting infrastructure (e.g., seamless mobile integration, more consumer-ready wearables) was not yet fully developed. Concept evaluation can help understand the timing and its importance.

Best practices for a comprehensive concept evaluation

To help you get started with your own concept evaluation, follow these key steps:

1. Define the problem and understand the users

This is the first and foremost step. What is the problem and how does it affect the users? There are multiple tools, processes, and techniques available in the market for defining the problem and understanding the user:

• Market research
• Competition analysis
• SWOT analysis
• Customer interviews
• Questionnaires

2. Document your insights

Depending on how complex the product is, you’ll end up with multiple insights from engineering, design, higher management, and users. Structure your document and categorize the insights in a way that they are easily accessible:

• Feasibility and technical constraints — Understand the technical limitations or challenges that could impact the product’s development (e.g., compatibility with existing systems, scalability, performance issues)
• Time estimates and dependencies — Document time and resource estimates provided by engineering and design to better plan the project timeline
• Risk and mitigation — Identify any technical risks (e.g., bugs, resource constraints) and the proposed solutions or mitigations
• Innovation or limitations — Document new technology proposals or limitations of existing systems that may influence the product’s architecture or features

3. Review and prioritize concepts

More often than not, you’ll work on multiple concepts at a time and it’s important to understand which concept to prioritize and which one to let go. Consider the following points before prioritizing:

• Use frameworks like RICE or opportunity solution tree to compare concepts against your criteria
• Prioritize concepts that align with long-term business strategy, have clear market potential, and can be realistically executed with available resources
• Document your evaluation process and rationale for the chosen concept to ensure transparency and alignment among stakeholders

Final thoughts

With the rise of Generative AI tools, concept evaluation will become easier. The right prompts that outline the details you need to know will let you evaluate concepts much quicker. But whether you use Generative AI tools or not, concept evaluation will help you with potential failures and this is vital for building successful products.

Have you had an experience where you pushed a product without a necessary concept evaluation? What lessons have you learned from your failures? Write to me in the comments and let’s make product management better.

Featured image source: IconScout