Keeping score: Why and how you should benchmark your UX

UX benchmarking is the process of testing the progress of a product or brand’s user experience over time. It can be measured through

• Different iterations of a prototype 
• Different versions of an app
• Before and after versions of a feature or function update
• Comparing your products to your competitors’

Why benchmark your UX?

Benchmark studies add a quantitative component to your UX research that helps support your findings with trackable data. Over time, you’ll see trends that will help inform better design and business decisions.

There are numerous benefits to benchmarking, but here are a few of the most common.

Gain buy-in with quantitative, historical data

Any user experience, by definition, is subjective to each individual. It’s not surprising, then, that in the UX community, we tend to rely on qualitative measures when evaluating user experience.

As a result, it can sometimes be challenging to get buy-in from data-driven stakeholders who prefer to reach statistical significance before making a change.

So does that mean you can’t measure user experience and present these findings to your leaders? No.

Benchmarking your UX research not only adds a quantitative layer to your UX research but also enables you to track your results with data everyone can quickly understand historically. It also empowers teams to demonstrate the impact of UX research on key business metrics like revenue and retention rates.

For example, let’s assume you know that product videos increase a customer’s likelihood to make a purchase—as well as their total order size. Benchmarking your UX over time will allow you to demonstrate how improvements to your video experience lead to increases in purchases and total purchase amounts.

Compare to competition to gain an advantage

It’s not just your products you should be interested in. While your teams will naturally strive to create the best experience possible, always improving upon a previous version, it’s hard to know how much better your product is if you’re not comparing it to your competition.

When you incorporate competitive UX benchmarking with your UX research strategy, you give yourself a clear, measurable way to keep score with your competition.

No matter how great your product is now, there’s always a chance your competition will catch up with you. By establishing a baseline of KPIs and measuring them against your competition’s, you’ll always know where you stand and where you can improve.

Understand omnichannel expectations

Your users think of their interactions with a company as one continuous experience, not a series of separate experiences. That means they expect the same look, feel, performance, and level of service across all channels. Benchmarking your UX across channels will help you better prioritize features and improvements, ensuring a consistent experience across all channels.

If your desktop site is where most of your digital efforts are focused, use your desktop UX as the baseline to benchmark your other channels against. Does the site fall short on smartphones or tablets? How does your mobile app measure up? To take it one step further, how do those experiences compare to any in-person experiences customers have with your brand?

Uncover hidden strengths and weaknesses

Like it or not, your competitors train your users on what to expect. Customer expectations around ease of use, clarity, and speed are impacted by the rising quality of customer experiences that all companies set—not just your competitors.

The good news is you can learn from your competition’s successes and failures. If you know that the competition will implement a new feature, run benchmark studies to see how that feature resonates with users before and after the change. The results will help inform future design decisions. Find out what’s working and what isn’t from your competitors and iterate to improve upon that.

With UserTesting’s Human Insight Platform, you can conduct UX benchmarking studies quickly and easily. You can set up, run, and analyze your benchmarking study within just a few days from anywhere in the world.

Competitive benchmark studies are conducted similarly to other UX studies; however, there are a few special steps to take when conducting this type of research. Here’s how to get started.

Establish your goals and areas of focus

Before you can begin your competitive research, you first need to establish what you want to study. Typically, these goals should be tied to improving KPIs like:

• Time on task
• Ease of use
• Satisfaction
• Success rate
• Number of errors

Additionally, identify your areas of focus, like key flows or tasks. Are you trying to gauge your overall user experience, or are you interested in a specific feature? Determining your goals now will help you write an effective test plan.

Write your script

Once you’ve established what you want to track, it’s time to write your test script. For
benchmark studies, test scripts should be easy to understand, non-leading, and consistent.

Remember that to achieve the most accurate results, your test script should be the same for every subsequent test after your initial baseline study and be achievable across any other sites you may be comparing against. For example, if you’re trying to study the journey of a user finding a red duffel bag that costs less than $100, make sure that all the sites you’re testing sell an item that fits that criteria.

Within the UserTesting platform, you can drag and drop common tasks and questions, write your custom tasks, or even choose from our popular test templates.

Tips

Remember that when benchmarking against your competition, the relative changes in results matter most—not just the score. 

For example, if you’re benchmarking your brand against a global leader in your industry, your scores may not be as high, and that’s okay! What does matter, however, is how your score changes over time compared to their score. Are you gaining ground? Where are the biggest opportunities for improvement? What inspiration can you take from the user experience of a best-in-class brand?

Use goal-oriented tasks: Tasks should be straightforward and with a specific goal in mind. Tasks like “Sign up for an account” or “Find a product you’d like to buy” are good examples. The task should present your test participants with a specific goal without providing them with any guidance on how to accomplish that goal.

Include evaluative questions: Each task should be paired with evaluative questions to help quantify the experience. These typically consist of a rating scale (e.g. “How difficult (1) or easy (5) was it to complete this task?”) or multiple choice (“Compared to other similar sites, was it easier, harder, about the same, or none of the above to complete this task?”).

Conduct a dry run: We recommend conducting several dry run studies with just a few users before committing to your script. This will allow you to make any necessary changes to questions that users may find confusing or that aren’t eliciting the type of feedback you were looking for.

Identify your participant demographics

Determine who you’ll want to get feedback from. In most cases, you’ll want to choose participants who match your target market. Consider basic demographics (like age, gender, and income) as well as criteria like job function, whether they are already customers, how often they shop online, or other relevant factors. With UserTesting’s large Contributor Network, you can choose demographics and set up custom screener questions to recruit your exact target audience to your study.

How many participants should I use?

For most UserTesting studies, we typically recommend using 5-7 participants for the best results. However, if your budget allows, benchmark studies can benefit from using 10-50 participants. A larger sample size helps provide more quantitative data, making it easier to identify trends.

Maintain the number of participants

While you don’t have to run your studies with the same individuals every time, you should use the same number of participants. If you establish your baseline study with 20 participants, then each subsequent study should also use 20 participants.

Use a different set of participants for each competitor

If you’re benchmarking the user experience across multiple brands, to avoid fatiguing your test participants, don’t have each user evaluate each brand in your study. Instead, use a different set of participants for each experience you want to study and then compare the results objectively.

Conduct a baseline study

To conduct effective UX benchmarking studies, you first need to understand where you’re starting from. Once you’ve decided what you’ll be tracking, conduct the studies necessary to establish your baseline. You’ll compare your user experience (and your competitors’) to this over time.

Commit to regular testing

Establishing a regular testing schedule to track your progress over time is important. Once you’ve established a baseline, you can compare all future measurements against it.

Some companies benchmark their UX against their competition quarterly, while others do it monthly. Whatever cadence you choose, be sure to stick with it consistently.

Lean on our expert research team

Our expert research team benchmarks your experience using QX score, a standardized measure of user experience. Benchmark experiences is an end-to-end delivery that provides an experience-based benchmark that communicates the value of research to the business and guides business or product investment decisions.

UX research can reveal a wealth of insights you’d never have access to otherwise. Conducting UX research regularly and comparing your progress over time gives you the metrics you need to prove the ROI of your UX research.

Benchmarking your UX gives you solid, quantitative results paired with qualitative insights. When combined, you’re armed with the knowledge you need to not only build a better experience for your customers but to measure and prove business impact.