User testing is an important part of user-centered design. It helps ensure that digital products and services are intuitive, easy to use, and meet the needs of the users.
One of the main types of usability testing is moderated testing, where a researcher guides a participant through a set of tasks while observing them and taking notes during the process. Because it needs observation, moderated testing takes a long time and is expensive, particularly for large-scale studies.
Unmoderated testing is an alternative approach that allows participants to complete tasks on their own time and in their own environment, without a moderator guiding them. This kind of testing is both an effective and efficient way to gather feedback, but it presents unique challenges and considerations.
In this article, we’ll explore the benefits and drawbacks of unmoderated user testing, as well as some best practices for designing and conducting these tests.
Table of Content
To obtain design feedback from users in a cost-effective and efficient manner, there are two types of remote user testing available: moderated and unmoderated.
Usability tests can be conducted both in person or remotely. The important thing is to select the more functional method.
Unmoderated tests offer designers an opportunity to understand the context in which people typically use an interface, including the physical environment. Remote research provides an opportunity to investigate how people use interfaces in their everyday lives while they carry out interesting tasks.
Consider these two scenarios and think about their differences:
You have been selected to test a website. The website is a recipe platform with baby food recipes for mothers. The manager of the test informs you about your tasks and you work on them. The test environment is quite silent and you are completely focused on the test. You perform the tasks given and it is not very difficult.
In the other scenario, you are at home. While your baby is asleep, you want to look at baby food recipes from the website in the other example. Your baby starts crying after you’ve wandered the website, chosen your recipe, and started cooking. Meanwhile, you lower the heat and run to your crying baby immediately. Once you have your baby, you return to your computer because you forgot the recipe. Now you have to try to calm your baby, look at the recipe, and monitor dinner on the stove.
Interfaces are often designed to be used globally, which means you may want to gather insights from users located in different parts of the world. Otherwise, you would have to do a test, for example, in Japan to understand if your application is intuitive enough for the users living in Tokyo.
You can conduct unmoderated usability tests almost anywhere with minimal setup requirements, as long as you have a computer and an internet connection. This is particularly advantageous for lone-wolf consultants or start-up teams working from cafes who may have difficulties finding a suitable office space for in-person testing.
Unmoderated usability tests generally reduce costs associated with lab testing, such as travel expenses and the need for costly testing accessories, because you can use relatively inexpensive software to conduct sessions.
Additionally, unmoderated usability tests enable quicker setup times because you can recruit online while research sessions are ongoing. This method is generally much faster compared to a moderated study because there is no need to schedule individual meetings with each participant. Unmoderated studies are efficient methods for data collection, and they allow you to collect feedback from dozens of users simultaneously.
Unmoderated studies may not be appropriate for all kinds of research, such as studies that require more in-depth engagement or guidance. One example might be ethnographic studies because the researcher must be present to observe and interact with participants in their natural environment. Another example could be user interviews, which include conversations between researchers and users, where researchers ask open-ended questions to understand user behaviors, motivations, and pain points of the user deeply.
Unmoderated studies also lack the personal interaction and rapport building that provide deeper insights and a better understanding of the participant’s perspectives, which would be essential in the examples above.
Automated (moderated) research can also be limited in its ability to capture the nuances of human behavior and can’t provide the same level of detail and context that can be obtained through qualitative research methods. Thus, it is important to carefully consider the research questions and goals when deciding which research method to use.
For early-prototype testing, a moderator makes the test process easier by explaining and helping participants to recover from errors or limitations of the prototype. Participants tend to be less engaged in unmoderated studies, as they are not interacting with a real person and may not feel as accountable for their actions.
This lack of engagement may lead to less authentic responses and may not accurately reflect the user’s experience. In addition, unmoderated studies can be risky in data collection because your participants may not follow instructions or may provide incomplete or inaccurate feedback, thus affecting the data.
In other words, the choice between moderated and unmoderated studies (or automated research) depends on your specific research goals, the nature of the research questions, and the type of data and insights you need. You can use a combination of both methods to obtain a more comprehensive understanding of user behavior and attitudes.
Before selecting testing software, it’s essential to determine research objectives. This will lead you to select a tool that aligns with your research goals instead of constraining the study to fit within a particular tool’s technical limitations. Defining study objectives helps in identifying critical testing software requirements.
For example; if the goal of the research is to learn how long it takes to complete a payment process for a user, you need to have a tool that is able to evaluate time on task, total execution time of the task, etc.
On the other hand, if the goal of the research is to understand why users do not complete their purchase after they add some products to their shopping lists, you need to have a tool that is able to record the screen and voice of the user.
Selecting the right software for unmoderated studies is crucial because it guides participants and records results. You have plenty of software options. However, what matters the most is choosing one that suits your project’s requirements and the type of your research.
According to the type of research result you want to gather (i.e., collecting qualitative or quantitative data) and depending on how you expect the data you have gathered to be interpreted and presented to you in the tool, you have several options. Be sure to thoroughly investigate and pilot your test tools to avoid technical limitations or data integration issues.
In order to create effective directions for an unmoderated study, you need a clear understanding of what you want your user to achieve. This could include tasks such as navigating through a website to find specific information, making a purchase, or providing feedback on a product or service.
Once you have identified the objective of the task, provide instructions that are detailed, realistic, and challenging enough to accurately measure the user’s abilities.
The nature of the study, whether it is qualitative or quantitative, will also play a role in the type of instructions you create. For instance, in a qualitative study, you may want to encourage users to explore and navigate through a website on their own without any guidance or hints in order to observe their natural behavior. On the other hand, in a quantitative study, you may need to provide more structured and specific instructions to ensure consistent data collection.
When conducting an unmoderated study, it’s important to plan follow-up questions that will provide valuable insights into user behavior and help improve the overall user experience. These can take the form of quantitative rating questions or open-ended prompts that encourage participants to provide detailed feedback about specific aspects of their experience.
Quantitative methods such as closed-ended survey questions can help us understand participants’ attitudes and perceptions, while qualitative methods like open-ended survey questions allow participants to provide free-form text responses about their task experience.
Examples of these questions are:
However, when phrasing follow-up questions, choose your wording carefully to avoid encouraging participants to talk about their previous experiences rather than the system they are just using. By tailoring your questions to the specific task or experience being tested, you can gain valuable insights into user behavior and improve the overall user experience.
Before beginning an unmoderated study, conduct a pilot test to identify any potential problematic points with your design or procedure. Pilot testing is crucial for unmoderated studies because there is no moderator present to address issues during the study.
While pilot testing can’t catch every problem that a moderator could, it can help detect issues like ambiguous task instructions, improperly presented tasks, and missing functionality in prototypes. In addition, technical limitations or incompatibilities can prevent your testing software from capturing necessary information, particularly when testing an intranet or native mobile application. Therefore, it is essential to test the recording process of your service before setting up tasks.
Another important consideration when conducting a pilot test is to ensure that you thoroughly analyze your software’s data collection to verify that your study design and procedures are well developed before launching your actual study.
While some issues can be identified by going through the study as a test participant, others may not be apparent until real participants use their own mobile phones or computers.
By conducting a thorough pilot test and addressing any issues that arise, you can increase the reliability and validity of your unmoderated study and obtain more meaningful insights into user behavior. This will enable you to make informed decisions about how to improve your user experience and optimize your digital products or services to better meet the needs of your users.
To ensure effective user testing, it’s important to have control over your sample. You can achieve this by using screening questions or by recruiting your own participants. While some user testing tools offer participants from their own panel for unmoderated tests, this may not be ideal for websites targeting either a specific or a more general audience.
To improve the quality of your participants, consider asking behavior-based questions during screening to ensure that they are a good fit for your study. However, keep in mind that some panel participants may have experience in this type of screening and will try to bypass it, so it’s important to include extra participants and exclude those who may not be fully engaged.
If your website aims to reach an international audience, unmoderated testing services can be a great way to reach a diverse range of people. Just keep in mind that if you recruit participants who speak different languages, you’ll need to have a researcher and maybe a translator who can accurately interpret the results.
In summary, while websites with a niche or general audience may need to recruit their own participants, using a panel can be beneficial depending on your study. However, be sure to consider your specific needs when selecting a panel, as some may take longer to fulfill specific study requirements.
Remote unmoderated usability tests can be challenging, especially when it comes to screen sharing. Setting up the tool beforehand can make the process easier in general. However, about 25 percent of remote sessions may still fail due to technical issues.
That’s why it is crucial to choose a screen sharing tool that is compatible with different browsers and operating systems. Consider the number of observers the tool can support, and whether chatting is possible between observers and moderators.
Taking notes on the computer during remote testing has advantages over handwritten notes — you don’t have to take your eyes off the screen and can transcribe faster.
Take notes on what users say and do as well as what’s on their screens. You can use a word processing document or Excel spreadsheet for basic note taking, but time-stamped notes can make analysis easier, especially if you’re planning to make highlight videos or conduct detailed analysis later.
While webcams are not commonly used in remote usability sessions, they can be a valuable tool that allows you to observe your users’ facial expressions. Some screen sharing applications now offer inbuilt webcam functionality, but participants must have the required hardware and software.
Ensure that any webcam software does not impact your computer’s performance and that you obtain explicit consent before recording any identifying information. After those conditions are met, you should follow the facilitator guide for conducting the test:
The remote facilitator guide:
Unmoderated studies can generate vast amounts of data, requiring a well-structured and analytical strategy to transform it into practical insights for your site. Although your analysis capabilities rely heavily on the software you choose, which determines how the findings are visualized and analyzed, it is important to consider specific aspects for different types of usability tests.
For qualitative studies:
For quantitative studies:
Test the tool with someone you know, from outside the company or your department, and check if the instructions are understandable or not.
Write down the tasks well enough in advance to pilot test them. In a moderated session, the facilitator can get the user back on track if a task is misunderstood. Every scenario, task, and/or question needs to be adjusted well to prohibit the possibility of misunderstanding.
In remote unmoderated research, participants are prone to not participating in your research. For this reason, you should always consider this possibility and create a backup participant list.
Unmoderated remote usability testing can be a cost-effective and efficient way for UX designers or teams to get design feedback from users. There are many advantages of unmoderated testing, such as the capability to understand the physical environment and context in which users or participants use interfaces of the digital products, together with the minimal setup requirements and faster setup times.
However, unmoderated testing isn’t an appropriate method when it comes to extensive research or testing where participants must be guided through tasks. In addition, unmoderated research may lack personal interaction and rapport, which can lead to richer insights and deeper understanding of participant’s perspectives.
To conduct an effective unmoderated test, it is essential to take into consideration what this research aims to learn, and next, to decide the questions you’ll ask according to the research method you’ll use. To enhance the reliability and validity of the study and collect significant insights into user behavior, piloting study design and procedures may be used inclusively. Unmoderated usability test method is a type of test that can be used very efficiently for designers and researchers who want to improve their products in terms of user experience and improve possible problematic areas identified as a result of the test.
What you need to focus on to choose the right test method is the type of data and information you want to obtain. Moderated and unmoderated tests complement each other because both qualitative and quantitative data are needed to build a good product. Therefore, you should treat usability testing as an iterative process and test your product with different testing methods. This will allow you to achieve the healthiest result.
Header image source: IconScout
LogRocket lets you replay users' product experiences to visualize struggle, see issues affecting adoption, and combine qualitative and quantitative data so you can create amazing digital experiences.
See how design choices, interactions, and issues affect your users — get a demo of LogRocket today.
Thinking of switching from graphic design to UX? The transition isn’t as daunting as you might think. Here’s what you need to know about their differences, similarities, and career paths.
Users shouldn’t have to guess what to do next. Here’s how you can close the two UX gulfs — the gulf of execution and evaluation — for better designs.
Error messages don’t have to be frustrating! In this blog, I discuss writing clear, user-friendly messages that keep people on track. Tips, examples, and best practices in here.
This article aims to inspire you with 25 carefully hand-picked UX case study examples, each offering valuable lessons.
Learn more →
