What is a screener for a survey? How to best design them?

It doesn’t matter how great your interviewing screens are and how amazing your research methods are if you recruit the wrong participants.

Talking to the wrong people is the worst thing that can happen during the research — it can give you false confidence and beliefs in insights that are opposite to those of our target group.

That’s exactly what screener surveys are supposed to protect us from. Assuming we do them right. More on this in this blog today.

What is a screener for a survey?

A screener survey is a set of initial questions we ask potential research participants to decide whether they are a right fit for the research.

In most cases, they help us define if we are talking to our intended user persona.

There are different methods to screen participants:

• E-mail — The most common method is to send a link to a survey via email or other communication. It’s the most efficient way to screen at a scale
• Phone/video call — For more nuanced research, or often in the case of B2B studies, a screening can happen on a call to get more details and add a personal touch
• In-person — The least efficient method to screen in person. If the research participant turns out to be out of our target group, we are wasting both our and their time; use only if truly essential (e.g., medical studies)

Why are effective screener surveys important?

Well-thought screener surveys are essential in the user research process.

Most importantly, they ensure high data quality. Screener surveys help us ensure we are talking to the right audience and help us further segment our results if we notice diversity in research outcomes.

They also help with efficiency immensely.

Discovering that you are talking to the wrong people during the research phase itself is quite expensive. You waste time and often money (e.g., for incentives).

Screeners are a filter that helps us segment a big group of people into well-segmented sub-groups for our research.

Types of screener questions

What do you ask in a screen survey?

Short answer? Anything that helps you decide whether the person is a viable candidate for research or not — put all of it in when creating your screener survey.

The exact questions differ depending on the specific nature of your research but might include:

• Demographic questions — If you’re looking for a specific gender, age range, location, or income criteria; examples are questions like: What’s your age range? What’s your gender?
• Behavioral questions — If you’re looking for people with specific habits or experiences; questions like: How often do you shop online? Have you used any ride-sharing apps in the past month?
• Knowledge questions — If you’re looking for people with specific expertise (or lack thereof); questions like: How familiar are you with cryptocurrency? Have you used our product in the past?
• Attitude questions — If you’re looking for people with a specific level of interest in a specific subject; questions like: How would you describe your interest in health and fitness? How important for you is regular exercise?
• Intent questions — If you’re looking for people with specific intentions; questions like: Are you planning to buy a house in the next year?
• Qualification questions — Any other type of question to determine if the surveyed person is the right fit for research; questions like: Are you a primary decision-maker for subscription purchases in your company? Is your job primarily focused on marketing?
• Control questions — To filter out people who randomly fill in the screener, you can consider asking control questions to double-check consistency in answers; questions like asking “Do you drink coffee?” on one screener page and then “How many cups of coffee do you drink a day”? or asking the same question twice

Best practices for designing screener surveys

Making your screener effective doesn’t have to be complicated — just follow a few best practices to get it right. With these tips, you’ll be well on your way to finding the perfect fit for your research.

Keep question straightforward

Unless you are recruiting highly specialized people and have to ask them specific details, go for simple questions.

You want to limit any confusion that could lead to inaccurate answers. Plus, if the survey becomes too confusing, many people can leave without finishing the survey.

People are most likely not incentivized at the screener stage yet, so don’t put too much mental work into them.

Limit the number of questions

Five to ten questions are usually a sweet spot for surveys.

Fewer than five, and you might not have enough information to segment participants properly.

More than ten or more people will drop out or start giving out random answers.

This is just a guideline, though. If you truly need to ask fifteen or twenty questions to recruit a very specific audience, go for it.

Favor closed-ended questions

Whenever possible, ask closed-ended questions. They are both easier to fill and easier for you to interpret.

You are not fishing for any new insights, so open-ended questions are mostly pointless.

Allow the ‘other’ option

The main disadvantage of closed-ended questions is the limited choice they give. You can’t predict all possible answers.

As a fallback, let people answer “Other” and later either analyze these answers yourself or disregard them completely.

It’s still a better option than inaccurate results.

Include control questions

The more important it is to recruit the right audience, the more control questions (e.g., asking the same question twice to check for consistency) you should include.

Still, having just one control question already reveals many spammers.

Avoid leading questions

As with any other good research survey, test your questions for bias.

Read each of your questions and ask yourself — Does this give off “correct” answer in any way? If yes, change it.

For example, “Do you prefer our product because it’s high quality?” suggests that high quality is the “correct” answer. “What is the main reason you chose our product?” is more neutral.

Test the screener

Test the screener for accuracy.

Find at least two people you know are part of your target group and two who are not, and send them the screener. Then, check if the screener results align with reality. You might need to adjust the screener to be more accurate if there’s a mismatch.

Fine-tuning your screener

You’re off to a great start with the earlier best practices, but these advanced strategies can help you add the final touches for a truly efficient screener.

Use conditional logic

Consider adding conditional logic if you need a long screener to recruit a specific segment.

For example, if after a few initial questions, you can already say the surveyed person is not a fit, finish the survey earlier.

Or, if your target segment consists of a few smaller subsegments, move people through different questions based on initial responses. This will allow you to ask particular questions without bombarding users with irrelevant questions.

Cover legal basics

First of all, be transparent about how you will use the data and how long you will store it. State it explicitly if you plan to keep screener responses and contact answerers in the more distant future.

Ask for consent at the very beginning of the screener to set up the expectations from the beginning.

Compensate complex screeners

If your screener takes a lot of time and effort to fill in and you recruit a specific, pre-qualified audience, consider compensating people for going through the screener even if they don’t qualify.

This can help greatly with recruitment efforts. However, it works mostly for invitation-only screeners. You don’t want a mass audience to start filling in the screener at random just to get the compensation.

Cater for diversity

Whenever you ask demographic questions, include a broader range of responses and/or allow an “other” option.

For example, people identify with more genders than men and women, whether you like it or not.

Closing thoughts

Let me repeat myself. It doesn’t matter how great your interviewing screens are and how amazing your research methods are if you recruit the wrong participants.

Whether using an online survey, phone calls, or even real-life screening, talk to the right people before proceeding with your research.

Ask relevant demographic, behavioral, attitude, knowledge, intent, and qualification questions to ensure the potential participants fit your desired user profile.

You’ll end up with a good screener if you:

• Keep questions simple and clear
• Limit the number of questions whenever possible
• Focus on closed-ended questions for easier analysis
• Include an “other” option to cover all bases
• Add control questions to verify participant consistency
• Avoid leading questions to maintain objectivity
• Test your screener thoroughly before launching it

For complex screeners, use conditional logic and consider compensating participants in the survey stage.

And in all cases, cover your legal and diversity basics.

Although developing a perfect screener might be time- and energy-consuming, it’s still cheaper than building a wrong product based on wrong insights.