Remote testing and feedback: the key to customer empathy

Repeat after me: you are not your customer. That’s why putting yourself in their shoes is arguably one of the most important skills customer experience (CX) professionals should learn (and practice) in order to develop the customer empathy needed to build products and experiences their users need and enjoy. While that may seem like a challenge, remote testing can make it less of a burden.

Through remote qualitative testing, designers, product managers, marketers, and researchers alike will uncover and understand how real people respond to products and experiences. From what they like and dislike, to where they get stuck and confused, to areas of improvement, the valuable insight gathered from these tests are sure to be eye-opening.

The benefits of remote feedback

Remote testing is an efficient way to capture real people completing real tasks. Here are some of the key benefits:

1. Test where your customers are

One major benefit of this type of research is that your test participants aren’t limited to a specific location. This allows you to get the most accurate representation of the user experience—out in the wild.

Some key benefits include:

• Research that happens in the participant’s natural environment is more realistic than lab research
• Feedback and outcomes are honest and unbiased because the setting isn’t artificial
• Feedback is representative of a wider population because the geographical location can be more diverse
• Contextual findings emerge when a product is used naturally

2. Some research isn’t suitable for a lab

Some research just can’t be done in a lab. For example, if you’re trying to answer questions about how users interact with a product in a specific context—like a fitness app at the gym—then you’ll need to make sure your research happens in that location. We call that destination-based testing. 

3. Some experiences aren’t digital

There may also be times when you need to observe something other than a digital experience on a computer or mobile device. Tasks, like navigating a theme park or installing a home thermostat, are great practical applications of remote testing.

Additionally, users often don’t complete an activity in one sitting (or through one channel), so it’s important to evaluate your omnichannel experience. That means any interaction a user has with your company while completing a single activity across multiple channels, like websites, apps, and even physical locations.

4. It’s faster and more cost-efficient

Probably one of the greatest business benefits of remote testing is that it’s faster and more cost-efficient. Because you’re not burdened with the logistics of in-lab testing, the constraints of travel and scheduling are nonexistent. Moreover, by testing with an insight platform like UserTesting, you have access to a panel of test participants (or as we like to call them, contributors) that allows you to bypass the hassle of recruiting.

Rather than trying to find and compensate a group of target users on your own, you just designate your target audience and UserTesting will recruit users from its panel that match your description. This shortens the testing period to a few hours, rather than days or weeks, allowing you to get feedback and iterate right away.

In summary, remote testing allows you to:

• Gather feedback from real people for products and experiences
• Test location-specific experiences 
• Test experiences across the customer journey, both digital and in-person
• Complete more tests in a shorter amount of time

Teams are changing the way they work—with more work being done remotely. While working remotely is easier than ever, it requires us to think differently about how we engage with and get feedback from our customers and users. 

And customer feedback is critical in this volatile environment. Truly understanding behaviors, motivations, expectations, and needs will give companies the insight they need to pivot business models to meet shifting demands. 

As companies respond to the current environment, many are transitioning even more of their customer feedback sessions to remote methods and technologies, and some are introducing new teams to remote approaches in lieu of in-person and face-to-face feedback sessions. Some in the industry are even calling for companies to shift to fully remote research as a way to define their role in “minimizing transmission of the virus.”

To help those who are making or leading the shift to more remote approaches, we’ve created some best practices that may help as you get going: 

1. Conduct real-time, face-to-face interviews remotely through a web conferencing platform

For those that are used to doing in-person feedback sessions, it’s best to start with interviews over a web conferencing platform, such as UserTesting’s Live Conversation, as they allow you to have real-time conversations over video. Interviewers can interact with customers using verbal and non-verbal queues, ask follow-up questions, probe, and see their reactions—just as they would in-person. Some tips for before, during, and after your session:

• Show up early and build in time for troubleshooting. As with an in-person meeting, aim to show up early. With the added complexity of software and tech, allow time to work out any technical challenges by adding 10-15 minutes to each session. 

• Create a backup plan. Think about poor internet connection, poor audio quality, and delayed start times when planning a Live Conversation interview. Consider providing the interviewer’s phone number to the participant ahead of the session in case they have technical difficulties and need some support.

• Be focused. Turn off all notifications, both browser-based and computer-based, so that there aren’t alerts, chimes, or other noises that might prove distracting while giving instructions or observing.

• Review your interview skills to improve over time. After you’ve done an interview, go back and watch it. Find places to improve your interview skills—whether that is by asking clearer questions, finding better opportunities to probe for more information, or conversely, learning when to stop talking so the customer can give you more details and context. 

2. Gather self-guided, recorded customer feedback

You can still get rich feedback from your customers without speaking to them 1:1 and in real-time. Self-guided customer feedback is a great way to reach your customers when it is convenient for them, and it adds efficiency to the team’s workflow and processes, as the sessions are completed while they are focused on other priorities. For example, 80% of UserTesting’s self-guided sessions fill within a few hours. If you’re newer to self-guided, recording customer feedback, here are a few tips to get started:

• Ask open-ended exploratory questions in an interview-style format. Open-ended questions encourage people to discuss and share details and perspectives. This approach provides a depth of feedback that is similar to real-time, face-to-face interviews, which allows teams to build a stronger sense of empathy with customers. For example, asking people to describe their current challenges with financial planning will garner richer feedback as compared to simply asking people if they agree or disagree with specific challenges that you have pre-defined. 

• Give participants the option to turn on their webcam. Even though you won’t be attending the interview, you can still ask people to turn on their webcam. Being able to see the person as they answer questions helps to build a level of empathy that you don’t get with a voice recording alone.

• Get feedback on a product or idea—a mobile app, brand campaign, rough sketch—anything, really! This could be anything from a landing page to a new product idea to a competitor’s experience. Keep the scope narrow and try to focus on a single objective, such as: “Do people find value in this new product idea?” or “How do our competitors describe their offerings?” Our test plan templates are a good place to start for common questions that UserTesting customers look to answer with our platform.

3. Analyze and share learnings through various channels

When gathering customer feedback remotely, teams lose the opportunity to observe in a co-located group session. While this can be a challenge, there are several ways to consume insights via remote technologies:

• Invite observers to remote interviews. Zoom, UserTesting, and other technologies allow others to join and observe in real-time. Consider hiding these remote observers so the participant doesn’t feel overwhelmed or “on the spot” if lots of people join.

• Use technology to analyze and prioritize findings. Whiteboard sessions, affinity diagramming, and group discussions are common approaches when analyzing data from an interview or a usability study. If you can’t do this in person, lean on virtual whiteboards, such as MURAL or even within a Google Doc. 

• Use categories or tags to organize findings. After you’ve done your analysis, you’ll likely see strong themes. Decide as a team how you want to categorize your findings. This could be by customer “feelings” (such as motivations, fears, needs, etc.) or, if you are evaluating a message, concept, or experience, by design elements (such as content, navigation, interaction, etc.) Use these tags as you document your findings and create highlight reels or clips of customer feedback.

• Share with teams through various channels. There are many ways to share customer feedback. From email to Slack as well as within an insight platform, such as UserTesting, there's no shortage of ways to allow others to consume insights.

As your practice of gathering customer feedback moves to more remote approaches, you'll learn best practices that are specific to your industry and team, and you’ll want to build those into your overall strategy as well.

Being customer-obsessed isn’t just a competitive advantage these days, it’s a basic necessity. Having a human-centered mindset at an organizational level is important, and one way teams can validate their efforts is through qualitative studies like moderated and unmoderated testing. While unmoderated and moderated testing can sound intimidating, we’re here to help break it down for you with everything you need to know.

One question that we often hear from customers is which is better: moderated or unmoderated testing? The answer, as you might’ve guessed, is that it depends. Like any research strategy, there are differing benefits for each approach making them more or less suitable depending on what you’re studying and your ultimate research goals.

Methods of remote research—moderated vs. unmoderated testing

Remote testing can be conducted using two different methods: moderated or unmoderated. While there are pros and cons to both methods, choosing which is right for you will depend on your specific situation and what your goals are.

Moderated vs. unmoderated tests, what’s the difference?

We’re about to dig into the pros and cons of each approach, but before we do, it’s important to understand how the two strategies differ.

Moderated testing

With moderated testing, a real person will be there to help facilitate (i.e., moderate) the test. The moderator will work directly with the test participant, guiding them through the study, and will be there to help guide and answer questions if the participant encounters any challenges while completing their tasks. What’s great about moderated studies is that they can be conducted either remotely or in-person. One of the major challenges with moderated testing, however, is that it can require a bit more planning upfront since you’ll need to coordinate schedules with participants. It usually requires a designated, quiet location, where you can interact with your participant, free of interruptions.

Unmoderated testing

Unmoderated testing is just like it sounds. It’s not monitored or guided, so there’s no one else present during the study except the participant. The participant completes any tasks and answers questions at their own pace, on their own time, at a time and location of their choosing. Pretty nice, right? Unmoderated testing tends to be faster as participants can complete their tests at any time without any disruption to your daily workflow. Now that we’ve established the differences between the two let’s dive into how and when to best apply each strategy based on your research needs.

When to use moderated testing

It helps to think of moderated user testing as an interview or a real-time conversation that you’re having with a participant or customer. Everything that you’d do to prepare for that type of interaction is what you’ll need to consider for a moderated study.

Pros

Moderated testing works best when you need a high level of interaction between you and your participant. For example, if you want to study a prototype with limited functionality, or a complicated process or concept, moderated testing provides you with the interaction you’d need to guide a participant through the study. It's also an excellent way to conduct interviews, understand the customer journey, and discover pain points. Moderated user testing is also a great way to observe body language and pick up on subtle behaviors and responses. It enables you to probe participants for more information if they seem stuck or confused. This type of test allows you to develop a rapport and have a natural conversation with your customers which helps establish trust and can lead to candid feedback that might not be possible with other qualitative research methods.

Cons

Because of the additional time and resources, moderated testing does cost more than unmoderated testing. However, you can reduce some of that cost by conducting remote moderated studies, rather than in-person. This will give you the opportunity to connect with participants all around the world and reduce the need to block out a dedicated time and space for onsite interviews. Who doesn’t love getting some hours back in their day?

When to use unmoderated testing

This type of testing is best for validating concepts and designs quickly with a diverse group of participants. Unmoderated testing works great if you have specific questions that you need to be answered, you need a large sample size, you need feedback quickly, or if you want to observe a participant interacting in their natural environment.

Pros

The beauty of remote unmoderated testing is that it can be done anytime, anywhere, and you typically have actionable feedback within a day if not sooner. Because a moderator isn’t needed, the cost is typically much lower than moderated tests, which can enable you to run more tests with a wider variety of participants.

Cons

Since unmoderated tests are completely unsupervised, it requires a fixed set of questions and tasks for participants to complete and doesn’t allow for follow up questions or clarification. If participants run into issues, have technical difficulties, or don’t understand the tasks or questions, you won’t have the ability to step in and guide them.   At the end of the day, the choice is up to you, but now you can make an informed decision. 

To keep pace with the changing attitudes, needs, and behaviors of people adapting to the new normal, you’re required to gather insights faster and more frequently than ever. To help with this, we pulled together a list of our top ten remote research strategies to help you uncover valuable insights more quickly while conducting remote unmoderated tests on the UserTesting platform.

Taken together, these strategies will help you pinpoint aha moments so that you can identify the trends and themes you want to understand by watching videos more strategically.  

1. Use structured test plans 

Instead of relying on verbal response questions and listening to entire video sessions for all participants, consider using a combination of tasks, multiple-choice questions, and rating scale questions to structure participants’ time. A little extra time spent upfront planning and structuring the test will help you pinpoint moments you care about quickly—without having to watch a lot of videos to find that needle in a haystack. How this might look:

Set participant expectations

By providing an expectation for how long you want people to spend on a task, you’ll set the expectation for how they should spend their time and control the amount of time you need to devote to watching videos.

• Task: Spend about 2 minutes reviewing the content on this page. Remember to talk about what you’re thinking. When you’re done, move on to the next task.

Use rating scale questions

Use rating scale questions to get a quick sense of how people feel, create counts, and pinpoint outlier participants. When you prompt people to speak to their rating, you can then watch an automatic highlight reel to provide more context around the ratings.

• Rating scale: On a scale of 1-7 where 1 is ‘Not at all understandable’ and 7 is ‘Very understandable,’ please rate how easy it was for you to understand the meaning of the text. Remember to describe your rating as you make a selection. 

Use multiple-choice questions

These questions enable you to create counts as well and help you understand how participants think or feel about a concept. In addition, you can create an automatic highlight reel to listen to all of the feedback, back-to-back.

In another example, you might consider adding a multiple-choice question that is structured like a quiz question that enables you to gauge whether participants understood the text—again, without having to watch all of the videos.

• Multiple choice (single select): How likely would you be to change your behavior based on what you read:

  • I would be unlikely to change my behavior because….

  • I would change my behavior because….

  • I would be very likely to change my behavior because….

  • I am not sure I would change my behavior because….

2. Use automatic highlight reels

Now that you have a highly structured test, you can use automatic highlight reels to easily watch all of your participants do the same task or answer the same question back-to-back. The advantage of using this strategy is that themes surface very quickly. 

Typically, it’ll take between 5-15 minutes to watch all participants complete a certain task, where the average time each participant took to complete the entire test was 15 minutes. This also means that you don’t need to watch all participants complete the entire test to get quick answers.

3. Understand and frame the story you want to tell

Make sure you’re very clear about your research objective and only review the data points and videos from tasks that help you answer that question. Typically, this means that you’ll want to select 1-3 objectives; the more you have, the longer the test will need to be to cover all of them.

Consider framing out your report first so that all you need to do is plug in your findings. This has the added benefit of forcing you to recheck the tasks and questions to make sure they capture the findings you want to share.

4. Capture a pulse

A limited set of study objectives doesn’t mean that you have to limit the test to only capture the information related to your current research objectives. On the contrary, consider collecting more information than you need. For example, ask a few questions about needs, behaviors, and perspectives, so you can go back and use the information later for tracking and trending purposes. 

In this way, you can create a data set over a series of studies that enables you to get a pulse on what people are thinking, feeling, and wanting over time. This can be an efficient way to gather information that you can use to inform persona research rather than conducting a larger scale study all at once.

5. Use the metrics tab 

Use the visualizations available in the platform to pinpoint specific videos and moments of interest that exemplify what is normal, show why specific participants behaved in unexpected ways, or help you to pinpoint the ‘why’ that you want to focus on.

6. Use the Excel download

When you download the Excel spreadsheet, you’ll bring together all of the text data that’s generated from a study. It’s faster to read text than watch videos. In addition, you can create charts and graphs based on the counts you’ve created from metrics or multiple-choice questions. Transcripts are also available in the download. These can be used to pull quick quotes for use in your report.

7. Use the minimum number of participants

To develop a sense of themes or trends, you get good data testing 3-5 participants per segment. If you find that you need more information, you can always add more later. By starting with as few as three participants and watching their entire videos, you’ll be able to get a sense of what a larger group might tell you. You can use the “long and deep” (watch the entirety of a smaller number of videos for studies with a smaller number of participants) or “short and broad” (watch single tasks or groups of tasks across all participants).

8. Develop and use templates

The UserTesting Professional Services team has developed a number of templates that are available both within the platform and via our help page. They cover a wide variety of objectives, from discovery to validation. Additional templates have been developed that focus on COVID-19. Using these templates as a starting point to develop your own Saved Test Plans that can save you a substantial amount of time as well as ensure that you start with a robust approach.

9. Time your test launch

Think through your availability and workday as you plan when you want to launch tests. We find that launching at the end of the work day means that you can start the following day over a cup of coffee watching videos that spark ideas or next steps that you can share with your team as they also start their days. 

Other teams that are working on quick iterations can launch tests first thing in the morning, iterate prototypes in the middle of the day, and launch the next test immediately to get through 2-3 rounds of testing in a day. 

Alternatively, schedule launch for a time that people are naturally thinking about or doing the activity that you want to capture. This might be over the weekend, during a special incentive period, or during a holiday.

10. Watch videos at an accelerated rate

The video player allows you to speed up the rate of playback up to 2.8x. We find that most people tend to be comfortable with watching videos at 1.5 to 1.8x. This enables you to discern what people are saying as well as watch and compare what they are doing.

Take a balanced approach that works for you

While structure can provide you with speed to insights across a typical interaction, another potential strategy to consider is asking a small number of questions, around 3-5, that are more exploratory in nature or a single, very short task. 

This approach enables you to get quick answers, but mirror actual behaviors more realistically. Balancing these approaches and integrating them into your workflow can help you to make decisions based on user feedback more frequently.
 

Discover how you can empathize with customers remotely with UserTesting