User testing is an important part of user-centered design. It helps ensure that digital products and services are intuitive, easy to use, and meet the needs of the users.
One of the main types of usability testing is moderated testing, where a researcher guides a participant through a set of tasks while observing them and taking notes during the process. Because it needs observation, moderated testing takes a long time and is expensive, particularly for large-scale studies.
Unmoderated testing is an alternative approach that allows participants to complete tasks on their own time and in their own environment, without a moderator guiding them. This kind of testing is both an effective and efficient way to gather feedback, but it presents unique challenges and considerations.
In this article, we’ll explore the benefits and drawbacks of unmoderated user testing, as well as some best practices for designing and conducting these tests.
Table of Content
- What is unmoderated user testing?
- Advantages of unmoderated user testing
- Disadvantages of unmoderated user testing
- The step-by-step unmoderated remote usability testing process
- Tips for unmoderated usability testing
What is unmoderated user testing?
To obtain design feedback from users in a cost-effective and efficient manner, there are two types of remote user testing available: moderated and unmoderated.
- During a moderated remote test, researchers use remote screen-sharing software to meet with a participant. The researcher can instruct the participant, observe how participants interact with the design in real time, and ask relevant follow-up questions for that particular part of the test
- In contrast, for an unmoderated remote test, there is no need for a researcher to attend each test session. Instead, instructions are given to users by a software application, which also records their actions and may ask pre-determined follow-up questions
Usability tests can be conducted both in person or remotely. The important thing is to select the more functional method.
Advantages of unmoderated user testing
Time-aware research
Unmoderated tests offer designers an opportunity to understand the context in which people typically use an interface, including the physical environment. Remote research provides an opportunity to investigate how people use interfaces in their everyday lives while they carry out interesting tasks.
Consider these two scenarios and think about their differences:
You have been selected to test a website. The website is a recipe platform with baby food recipes for mothers. The manager of the test informs you about your tasks and you work on them. The test environment is quite silent and you are completely focused on the test. You perform the tasks given and it is not very difficult.
In the other scenario, you are at home. While your baby is asleep, you want to look at baby food recipes from the website in the other example. Your baby starts crying after you’ve wandered the website, chosen your recipe, and started cooking. Meanwhile, you lower the heat and run to your crying baby immediately. Once you have your baby, you return to your computer because you forgot the recipe. Now you have to try to calm your baby, look at the recipe, and monitor dinner on the stove.
Geographic diversity
Interfaces are often designed to be used globally, which means you may want to gather insights from users located in different parts of the world. Otherwise, you would have to do a test, for example, in Japan to understand if your application is intuitive enough for the users living in Tokyo.
Learn more →Minimal set-up requirements
You can conduct unmoderated usability tests almost anywhere with minimal setup requirements, as long as you have a computer and an internet connection. This is particularly advantageous for lone-wolf consultants or start-up teams working from cafes who may have difficulties finding a suitable office space for in-person testing.
Cost effective
Unmoderated usability tests generally reduce costs associated with lab testing, such as travel expenses and the need for costly testing accessories, because you can use relatively inexpensive software to conduct sessions.
Quicker set-up times
Additionally, unmoderated usability tests enable quicker setup times because you can recruit online while research sessions are ongoing. This method is generally much faster compared to a moderated study because there is no need to schedule individual meetings with each participant. Unmoderated studies are efficient methods for data collection, and they allow you to collect feedback from dozens of users simultaneously.
Disadvantages of unmoderated user testing
Unmoderated studies may not be appropriate for all kinds of research, such as studies that require more in-depth engagement or guidance. One example might be ethnographic studies because the researcher must be present to observe and interact with participants in their natural environment. Another example could be user interviews, which include conversations between researchers and users, where researchers ask open-ended questions to understand user behaviors, motivations, and pain points of the user deeply.
Unmoderated studies also lack the personal interaction and rapport building that provide deeper insights and a better understanding of the participant’s perspectives, which would be essential in the examples above.
Automated (moderated) research can also be limited in its ability to capture the nuances of human behavior and can’t provide the same level of detail and context that can be obtained through qualitative research methods. Thus, it is important to carefully consider the research questions and goals when deciding which research method to use.
For early-prototype testing, a moderator makes the test process easier by explaining and helping participants to recover from errors or limitations of the prototype. Participants tend to be less engaged in unmoderated studies, as they are not interacting with a real person and may not feel as accountable for their actions.
This lack of engagement may lead to less authentic responses and may not accurately reflect the user’s experience. In addition, unmoderated studies can be risky in data collection because your participants may not follow instructions or may provide incomplete or inaccurate feedback, thus affecting the data.
In other words, the choice between moderated and unmoderated studies (or automated research) depends on your specific research goals, the nature of the research questions, and the type of data and insights you need. You can use a combination of both methods to obtain a more comprehensive understanding of user behavior and attitudes.
The step-by-step unmoderated remote usability testing process
1. Define the goal of the study and criteria for participant recruitment
Before selecting testing software, it’s essential to determine research objectives. This will lead you to select a tool that aligns with your research goals instead of constraining the study to fit within a particular tool’s technical limitations. Defining study objectives helps in identifying critical testing software requirements.
For example; if the goal of the research is to learn how long it takes to complete a payment process for a user, you need to have a tool that is able to evaluate time on task, total execution time of the task, etc.
On the other hand, if the goal of the research is to understand why users do not complete their purchase after they add some products to their shopping lists, you need to have a tool that is able to record the screen and voice of the user.
2. Select your software
Selecting the right software for unmoderated studies is crucial because it guides participants and records results. You have plenty of software options. However, what matters the most is choosing one that suits your project’s requirements and the type of your research.
According to the type of research result you want to gather (i.e., collecting qualitative or quantitative data) and depending on how you expect the data you have gathered to be interpreted and presented to you in the tool, you have several options. Be sure to thoroughly investigate and pilot your test tools to avoid technical limitations or data integration issues.
3. Define tasks and questions
In order to create effective directions for an unmoderated study, you need a clear understanding of what you want your user to achieve. This could include tasks such as navigating through a website to find specific information, making a purchase, or providing feedback on a product or service.
Once you have identified the objective of the task, provide instructions that are detailed, realistic, and challenging enough to accurately measure the user’s abilities.
The nature of the study, whether it is qualitative or quantitative, will also play a role in the type of instructions you create. For instance, in a qualitative study, you may want to encourage users to explore and navigate through a website on their own without any guidance or hints in order to observe their natural behavior. On the other hand, in a quantitative study, you may need to provide more structured and specific instructions to ensure consistent data collection.
When conducting an unmoderated study, it’s important to plan follow-up questions that will provide valuable insights into user behavior and help improve the overall user experience. These can take the form of quantitative rating questions or open-ended prompts that encourage participants to provide detailed feedback about specific aspects of their experience.
Quantitative methods such as closed-ended survey questions can help us understand participants’ attitudes and perceptions, while qualitative methods like open-ended survey questions allow participants to provide free-form text responses about their task experience.
Examples of these questions are:
- How clear was this task for you? Please rate out of 5. [Quantitative]
- If there are parts of the testing process that you find confusing, can you briefly explain it to us? [Qualitative]
However, when phrasing follow-up questions, choose your wording carefully to avoid encouraging participants to talk about their previous experiences rather than the system they are just using. By tailoring your questions to the specific task or experience being tested, you can gain valuable insights into user behavior and improve the overall user experience.
4. Run a pilot test
Before beginning an unmoderated study, conduct a pilot test to identify any potential problematic points with your design or procedure. Pilot testing is crucial for unmoderated studies because there is no moderator present to address issues during the study.
While pilot testing can’t catch every problem that a moderator could, it can help detect issues like ambiguous task instructions, improperly presented tasks, and missing functionality in prototypes. In addition, technical limitations or incompatibilities can prevent your testing software from capturing necessary information, particularly when testing an intranet or native mobile application. Therefore, it is essential to test the recording process of your service before setting up tasks.
Another important consideration when conducting a pilot test is to ensure that you thoroughly analyze your software’s data collection to verify that your study design and procedures are well developed before launching your actual study.
While some issues can be identified by going through the study as a test participant, others may not be apparent until real participants use their own mobile phones or computers.
By conducting a thorough pilot test and addressing any issues that arise, you can increase the reliability and validity of your unmoderated study and obtain more meaningful insights into user behavior. This will enable you to make informed decisions about how to improve your user experience and optimize your digital products or services to better meet the needs of your users.
5. Recruit participants
To ensure effective user testing, it’s important to have control over your sample. You can achieve this by using screening questions or by recruiting your own participants. While some user testing tools offer participants from their own panel for unmoderated tests, this may not be ideal for websites targeting either a specific or a more general audience.
To improve the quality of your participants, consider asking behavior-based questions during screening to ensure that they are a good fit for your study. However, keep in mind that some panel participants may have experience in this type of screening and will try to bypass it, so it’s important to include extra participants and exclude those who may not be fully engaged.
If your website aims to reach an international audience, unmoderated testing services can be a great way to reach a diverse range of people. Just keep in mind that if you recruit participants who speak different languages, you’ll need to have a researcher and maybe a translator who can accurately interpret the results.
In summary, while websites with a niche or general audience may need to recruit their own participants, using a panel can be beneficial depending on your study. However, be sure to consider your specific needs when selecting a panel, as some may take longer to fulfill specific study requirements.
6. Setting up and launching the test
Screen sharing
Remote unmoderated usability tests can be challenging, especially when it comes to screen sharing. Setting up the tool beforehand can make the process easier in general. However, about 25 percent of remote sessions may still fail due to technical issues.
That’s why it is crucial to choose a screen sharing tool that is compatible with different browsers and operating systems. Consider the number of observers the tool can support, and whether chatting is possible between observers and moderators.
Taking notes
Taking notes on the computer during remote testing has advantages over handwritten notes — you don’t have to take your eyes off the screen and can transcribe faster.
Take notes on what users say and do as well as what’s on their screens. You can use a word processing document or Excel spreadsheet for basic note taking, but time-stamped notes can make analysis easier, especially if you’re planning to make highlight videos or conduct detailed analysis later.
Webcams
While webcams are not commonly used in remote usability sessions, they can be a valuable tool that allows you to observe your users’ facial expressions. Some screen sharing applications now offer inbuilt webcam functionality, but participants must have the required hardware and software.
Ensure that any webcam software does not impact your computer’s performance and that you obtain explicit consent before recording any identifying information. After those conditions are met, you should follow the facilitator guide for conducting the test:
The remote facilitator guide:
- Begin by quickly introducing yourself and the purpose of the study. Then, provide the terms and instructions in small chunks to prevent participants from becoming disengaged
- Be prepared to go off-script if unexpected user actions provide relevant insight into product usage. However, stay within the allotted time and keep users on track
- To take more efficient timestamped notes, use spreadsheets or software such as transcription services or voice-to-text programs
- Inform observers of what to expect during the session and gather their feedback and insights throughout the study
- It is likely that you will encounter challenging users. Use effective moderating skills, such as asking guiding questions, to handle quiet, shy, and bored users
- If you don’t use webcam recording, don’t worry about not seeing users’ faces; your tone conveys expression
7. Analyzing the data
Unmoderated studies can generate vast amounts of data, requiring a well-structured and analytical strategy to transform it into practical insights for your site. Although your analysis capabilities rely heavily on the software you choose, which determines how the findings are visualized and analyzed, it is important to consider specific aspects for different types of usability tests.
For qualitative studies:
- Review each session recording to analyze behavior of the participant
- User comments, vocal or written, can be misleading, so watch behavior to understand what really works and what doesn’t work
- Use audio recordings to capture participants verbalizing their actions, and use screen recordings to capture behavior
- Tag videos with timestamped notes and export/share/visualize them for analysis
- Look for unmoderated testing tools offering video analyzing features
For quantitative studies:
- Identify and exclude inaccurate values to clean the data
- Carry out statistical tests to evaluate and ensure the accuracy of your results
- Visualize the data for showing your findings to others in an easy understandable way
- Ensure your tool has the ability to export the data for further analysis using specialized statistics software
Tips for unmoderated usability testing
Test with people outside your department
Test the tool with someone you know, from outside the company or your department, and check if the instructions are understandable or not.
Revise your instructions
Write down the tasks well enough in advance to pilot test them. In a moderated session, the facilitator can get the user back on track if a task is misunderstood. Every scenario, task, and/or question needs to be adjusted well to prohibit the possibility of misunderstanding.
Have a large sample size
In remote unmoderated research, participants are prone to not participating in your research. For this reason, you should always consider this possibility and create a backup participant list.
To sum up
Unmoderated remote usability testing can be a cost-effective and efficient way for UX designers or teams to get design feedback from users. There are many advantages of unmoderated testing, such as the capability to understand the physical environment and context in which users or participants use interfaces of the digital products, together with the minimal setup requirements and faster setup times.
However, unmoderated testing isn’t an appropriate method when it comes to extensive research or testing where participants must be guided through tasks. In addition, unmoderated research may lack personal interaction and rapport, which can lead to richer insights and deeper understanding of participant’s perspectives.
To conduct an effective unmoderated test, it is essential to take into consideration what this research aims to learn, and next, to decide the questions you’ll ask according to the research method you’ll use. To enhance the reliability and validity of the study and collect significant insights into user behavior, piloting study design and procedures may be used inclusively. Unmoderated usability test method is a type of test that can be used very efficiently for designers and researchers who want to improve their products in terms of user experience and improve possible problematic areas identified as a result of the test.
Should I run a moderated test or an unmoderated test?
What you need to focus on to choose the right test method is the type of data and information you want to obtain. Moderated and unmoderated tests complement each other because both qualitative and quantitative data are needed to build a good product. Therefore, you should treat usability testing as an iterative process and test your product with different testing methods. This will allow you to achieve the healthiest result.
Header image source: IconScout
LogRocket: Analytics that give you UX insights without the need for interviews
LogRocket lets you replay users' product experiences to visualize struggle, see issues affecting adoption, and combine qualitative and quantitative data so you can create amazing digital experiences.
See how design choices, interactions, and issues affect your users — try LogRocket today.
