Why PMs need to master qualitative tests now more than ever

I’m often brought in to fix a leaky user activation funnel. The brief usually goes something like this: “Please help us improve our new user activation rate. Out of all the users who sign up, only 10 percent become active, habitual users of the product.”

You’ve probably dealt with this scenario yourself. And as a well-educated, “data informed” PM, you dig into the data, build detailed onboarding funnels to figure out where users drop off, determine your “aha” and “habit” moments, and build a nice signup → setup → aha → habit funnel.

At this point, you can see exactly how many users are dropping off at each stage and drill down to see who’s dropping off. Then, you go ahead and run some A/B tests.

You modify the setup wizard, you tweak the copy, and maybe strategically place a testimonial, hoping to nudge the activation rate up by a few percentage points. But after a while, you inevitably have to raise your hand and ask the awkward questions: Who is this for? What does value actually mean to them? Why should this product exist at all in a crowded market?

Because if you can’t answer those questions, all the data driven optimization work and A/B testing is just lipstick on a pig.

The reality is that you can’t build an innovative product by relying on quantitative data alone. Successful products merge the quantitative with the qualitative. To help you get started, this article unpacks how you can do just that and covers how things have shifted in the new AI world.

Quantitative data is a microscope

Quantitative data is a microscope. It gives you a precise, almost scientific view of what users are doing inside your product:

• It’s backward-looking, showing you what has already happened
• It’s incredibly precise but also incredibly narrow
• It’s the perfect tool for optimizing what’s already there

But you can’t discover a new galaxy by looking at a petri dish.

Qualitative data is a telescope

Think of qualitative data like a telescope. You can use it to look up and out, towards “the stars:”

• The user’s entire jobs to be done — Your product probably only solves one small step of a much larger workflow. What are the other steps?
• Goals and motivations — The why behind the what. What are they truly trying to achieve? What do they expect versus what would truly delight them?
• The alternatives — What were they using before you? What else have they considered? What solutions have they cobbled together to solve their problem?
• Other people — Additional people involved in your users process, either directly or indirectly
• Trends and circumstances — What technological shifts, economic pressures, or life events (a new baby, a big promotion) are influencing their needs, behaviors, and expectations?

All of this lives outside your product, and for innovation, it’s infinitely more interesting than what happens inside. Qualitative data is imprecise, but it lets you see the forest, not just the trees. This is the key to building something that matters.

Over 200k developers and product managers use LogRocket to create better digital experiences

Learn more →

The product market fit treadmill

There used to be a clean-cut distinction in the startup world: pre-product-market fit was about innovation and finding what clicks. Post-PMF was about growth and optimization.

This distinction is hopelessly oversimplified. I’ve always hated it, but today it’s more wrong than ever as AI has changed the rules around PMF.

For anyone who hasn’t read it yet, I highly recommend Brian Balfour’s essay on product market fit collapse. To explain the change more clearly, let’s take a look at the pre-AI era and how things have evolved.

Pre-AI era

Look at the “PMF threshold” line at the bottom. This line corresponds to user expectations. Before the AI era, this threshold rose in a relatively linear, predictable way. Imagine the blue line (“company performance”) as a startup, struggling to find PMF.

In the middle of the chart the startup meets the expectations of its target audience — something “clicks” — and it achieves PMF. Now it’s comfortably in the green zone, where it can stay as long as it keeps us with those linearly rising user expectations.

AI era

But in the AI era, the PMF threshold no longer rises. It can jump exponentially, almost overnight.

Groundbreaking tools pop up out of nowhere, making the impossible possible. Users adapt instantly, and their expectations skyrocket.

The result is what Balfour calls PMF collapse. Your product might’ve been essential yesterday, but today it’s obsolete because a new AI-native tool just solved your customer’s core problem 10 times better, faster, or cheaper. Take the following LinkedIn post:

Considering the fact that Elena is handling growth for the $100M revenue AI company (Lovable) this means a lot. I would argue that this isn’t just a problem for AI companies. It’s a problem for every company being disrupted by them (we’ve all seen Chegg, Stackoverlow, and even Adobe tumble).

We’re all on a PMF treadmill now. At this point, innovation isn’t optional — it’s a survival mechanism. And the only way to innovate is to use the telescope.

Qualitative tests: More than just interviews, easy to mess up

When I say “qualitative data,” most PMs think of customer interviews. But that’s just one tool in a much larger toolkit — and it’s often not the right tool for the riskiest assumption at hand.

You need to wield the full range of qualitative methods, each with its own strengths and weaknesses.

I bucket them into three main categories:

1. Self-reportage — What users say they do. It includes interviews, focus groups, and surveys. It’s great for understanding motivations and perceptions but can be unreliable for predicting actual behavior
2. Watching users in their environment — What users actually do in their natural habitat. Think ethnography or contextual inquiry. This is where you uncover the workarounds and pain points they don’t even think to mention
3. Watching users with your artifacts — How users interact with what you’ve built (or plan to build). This includes moderated and unmoderated usability tests, prototype tests, and landing page tests. It’s fantastic for de-risking specific solutions

Each category comes with its own strengths and weaknesses:

• Self-reportage:
  • Pros:
    • Fight your confirmation bias: Find out what’s important to the user by asking non-leading questions and letting them tell their story
    • Uncover the why, goals, motivations, values
  • Cons:
    • People don’t always do as they say (bias)
    • You can miss latent needs/hard fact
• Watch user in their environment:
  • Pros:
    • See real workarounds, constraints, context
  • Cons:
    • Set up and execution complexity (esp. B2B)
    • Interpretation (tester assumptions)
• Watch user with your artifact, in your environment (this is where vibe coding shines):
  • Pros:
    • Spot resonance and delight (latent needs? hard fact?)
    • Spot usability issues
    • Fast setup and iteration
  • Cons
    • Artificial environment (your watching over their shoulder)
    • Small “solution-space box” (you don’t fight your own idea bias)
    • People don’t always do as they say

Mastering the qualitative test toolkit, and knowing which method to use for which question, is more important now than ever.

Unfortunately, many PMs step into this with a hopelessly naive (or arrogant) view of how easy it’ll be to:

• Set up and run good qualitative tests
• Collect good qualitative data (garbage in = garbage out)
• Extract the right insights

The bad news: Qualitative tests are difficult to get right, and very easy to get wrong. If I had a penny for every time a founder/PM stormed off in the exact wrong direction, feeling fully confident because “our users say so”…. I’d be a rich woman.

Remember, modern behavioral science and UX research are almost half a century old. It pays off to invest time and resources to learn from these well-established fields.

Final thoughts

Optimization has its place. Using the microscope to fine-tune your onboarding funnel can yield important incremental gains.

But the innovations that’ll keep you on that PMF treadmill won’t be found in your funnel report. They’re out there in the imprecise outside world.

So, grab your telescopes, and figure out how to use them properly.

Featured image source: IconScout

LogRocket generates product insights that lead to meaningful action

LogRocket identifies friction points in the user experience so you can make informed decisions about product and design changes that must happen to hit your goals.

With LogRocket, you can understand the scope of the issues affecting your product and prioritize the changes that need to be made. LogRocket simplifies workflows by allowing Engineering, Product, UX, and Design teams to work from the same data as you, eliminating any confusion about what needs to be done.

Get your teams on the same page — try LogRocket today.