Creating proof: How to balance gut feeling with empirical evidence

We’ve all been there: the meeting where you present the infamous product roadmap — usually created with the best intentions, but sometimes optimistically slapped together. Different organizations have different processes for handling them and different expectations for how they’re executed.

In that fateful product roadmap meeting, someone questions why certain features are on the roadmap and not others. Someone even asks, “Why are we building that product at all and not pursuing the other list of ideas in our company strategy?” Or, worse, the dreaded provocation, “Why does the product team get the budget for that? The [insert other team desperate for budget] team has a ton of high priorities!”

You grin and clench your teeth, urging your blood pressure to remain under control. Even if the inquirer is the most perpetually unsatisfied person in the office, they still have a right to ask it. It’s a fair question, and it deserves a fair examination.

The problem is not the question. The problem is how you determine the answer. That’s where empirical evidence comes in.

In this article, you will learn what empirical evidence is, the difference between opinions and facts, and strategies for creating proof.

What is empirical evidence?

Empirical evidence is data or information that can be physically observed or measured. This differs from theoretical evidence in that it’s factual and not based on someone’s personal opinion. You can collect empirical evidence through observational studies, testing, and analytics softwares.

As a PM, you should lean on empirical evidence when presenting decisions to your product team. This will allow you to speak from a position of authority and minimize the potential for pushback.

Facts vs. opinions

Some people debate opinions until they’re blue in the face. Opinions are nice, and occasionally useful, but they can only get you so far. Instead of allowing conversations about your roadmap to run amok, think like an attorney.

To win legal cases, attorneys don’t waltz into the courtroom and spout their opinions about what happened on the night of the crime. They find proof. They’re attentive to detail. They force the jury to consider all the evidence and they attempt to make an undeniable case for their client.

To help illustrate the difference between facts and opinions, let’s consider two fictional scenarios:

• Opinion example
• Fact example

Opinion example

Jane is a product manager in a microelectronic division of a large enterprise. She gets questioned about the features on her product roadmap for a suite of products she’s running and immediately grows defensive, saying these have been circulated among many stakeholders and everyone signed off.

In her opinion, it’s critical to build new features for A product and not Z product, because customers seem to respond better to A, based on a customer comment she overheard on a support call once.

Plus, she says this is what Bob from [insert influential department] wants. If they don’t follow this roadmap, Bob is going to be really mad, because he’s counting on A product to meet his team’s sales goals.

Someone asks Bob to weigh in, and he acquiesces, saying if Z product is a focus for everyone else, he’s fine with that, as long as he also gets most of the features Jane promised for A product.

Now everyone’s looking at Jane, wondering why she can’t do it all. Under pressure, Jane caves and over promises to everyone, but her product development team can only handle one product at a time. At capacity, it demands she set more realistic expectations with the stakeholders, so she crawls back to everyone with her tail between her legs and begs for more time, money, or scope reduction.

Fact example

Emily is a product manager at a small healthcare company focusing on digital wellness tools. She gets questioned about why they’re creating another mobile app. Her colleagues ask, “Don’t we already have all this in the portal?”

She is quick to cite detailed statistics about how frequent customer engagement creates loyalty, which produces recurring revenue, and how much less expensive it is to keep existing customers than to obtain new ones. In fact, the development costs associated with her product (she cites a number for a specific period of time) are offset by the cost savings that would otherwise be incurred to obtain new customers.

Their customers are X times more likely to use a mobile app while on the go than to sign in to a web application regularly, she says, pointing to results from a customer engagement survey she performed last year.

Furthermore, she states why their customer base has specific needs unfulfilled by other apps on the market, and how their preliminary market testing showed validation of the top features customers want in order to consider the app useful. They’re similar to the top-used features available on the company’s web application, with a few exceptions.

She whips out a pivot table showing the statistical analysis on the existing website usage behavior and reiterates she used all that research to prioritize and define requirements for the MVP.

In the opinion-based example, the conversation revolves around what the company and its employees want. The absence of statistical evidence left a glaring hole, which was quickly filled by opinions and team dynamics. The negotiation spiraled out of Jane’s control into the hands of her colleagues, who turned the expectations back on her and held her opinions ransom.

In fact-based example, by citing data and having detailed research ready at hand, Emily shifted the focus of the discussion onto what the customers want and need. Now, instead of infighting, the teams focused on which features were truly priorities for the web versus mobile application. Team dynamics will still inevitably be a factor here, but Emily remains in control of the roadmap because she keeps the discussion rooted in facts.

How to create proof for your product

So, how do you create proof for your product (whether it’s for the product overall or its features)? Believe it or not, it starts in a nebulous, non-statistical domain. Product management requires a certain degree of feeling — an instinct, perhaps — that provides direction about what a particular customer or group of customers wants or needs.

This is generally enabled by a high capacity for emotional intelligence, which means you can empathize with others. Being highly emotionally intelligent means using your inner eye. First, you must know your customer.

For example, a product manager for a pet harness would be more successful if they had a pet, or frequently engaged with others’ pets to understand typical pet behavior and learn why a particular harness may or may not work for different pet breeds, sizes, and temperaments.

Once you’ve researched and actively engaged with your customers, be sure to communicate your findings to others across your teams. One way to do this is by creating personas (one-page templates with a brief outline of a fictional customer with realistic traits).

Gathering empirical evidence

Now that you know your customer(s), you can dig a level deeper into their wants and needs by using the following steps:

1. Research addressable market statistics relevant to your product domain and target users, document them, and cite your sources. These stats help answer the question: ‘Why this product?’
2. Ask your sales or marketing teams if they have statistics to share about the sales conversion funnel, Net Promoter Score (NPS), etc.
3. Maybe you have a product in the market already. If so, do you have a monthly report showing behavioral engagement statistics?
   1. For digital products, this might include stats like: new vs. returning users, bounce rates, page views and time spent on page, CTA click rates, etc.
   2. For physical products, this might include store or site traffic, sales and return figures, customer support data (volume and type of tickets, top complaints, bugs, known issues), etc.
4. Evaluate customer feedback channels to see what people are asking for. What problems still exist that your product might help solve?

Compile all these facts and stats into a common location for easy reference across your teams and encourage them to peruse it themselves.

With that background research in your pocket, now you can plan the product. Conducting a customer journey mapping session can help you take an objective look at how the user will engage with your product, and you can make a list of all the features you think the early product will need. Segment those features into two groups:

1. What you know you need (such as a user identity management system)
2. What you think customers want (such as notification settings)

This is important to convey to stakeholders so they understand product development isn’t all bells and whistles. For example, there may be certain features or non-functional requirements on your roadmap that are legally mandated to comply with certain standards or protocols.

Sketch some early designs to test how users react to what you think they want. Run the test with customers, or a group of people who have similar characteristics.

Once the product is live, testing continues in an iterative cycle, focusing on both the existing features (you can use A/B tests to evolve and improve your existing features) as well as new features. Design Thinking is a cyclical journey.

Think you don’t have time to conduct tests? Remember, it’s not about you. Even a product manager who thinks they are right should test hypotheses to prove it before building.

Summary

In product planning, whether you are in the position of being questioned or you are the questioner, you have a right to an opinion, but you have a  responsibility to consider the facts. Luckily, it’s never been easier. Myriad tools are at your fingertips to craft hypotheses, test ideas, analyze the outcomes, segment your audiences, and prioritize your roadmap.

Balance the skills of emotional intelligence and ‘gut feelings’ with empirical evidence from surveys, user testing research, trends and statistics, etc. to prove why your product roadmap is valid. Then you can stand tall, knowing you’ve built a solid case, and ask the jury to render a decision in your favor.

Featured image source: IconScout