Test Your Competitors with These 2 Methods

A lot of  competitor tests come our way here at UserTesting’s Research Team. It’s great to learn how your site measures up in relation to your competitor’s site, but there are a couple of different methods of comparing sites that are appropriate for different situations. To get the feedback you need, you’ll have to decide which method to use and how to structure your study.

But not to fear! We’re here to explain the differences between the two methods so you can determine which type of study is right for you.

For starters, if you’re unfamiliar with competitor testing, check out this article on how and why to test your competitors’ sites.

Once you’ve got the hang of how competitor tests work, then keep reading!

Why are there two different methods?

After watching users go through the steps of a comparison competitor test, I started to get frustrated. I kept asking myself, “Why do we need to show multiple sites to the same user? There HAS to be a way around this.”

Here’s the problem: If you want users to complete a full test on your site and a full test on a competitor’s site, it’s going to take twice as long. And while you can always offer an extra incentive for tests that take extra time, humans will be humans. Test participants are going to get tired and frustrated. Would you want a surgeon who’s been awake for 4 days straight, with no coffee running in their system, to perform open-heart surgery on you? While testing a website isn’t the same as conducting a triple bypass, I think you get the point.

Some of you might be saying to yourself, “But we HAVE to do it to get the results we need.”

We found out, after doing many of these tests, that you don’t always have to send users to multiple sites to get comparable results among competitors. What’s more, there are a few ways to go about getting these competitor insights without wearing users out, whether the user is viewing multiple sites (Direct Comparison) or just one site (Objective Comparison).

Direct comparison testing

This type of testing helps when you want to identify specific parts of your site that your competition does better.  It can be done within a fairly short amount of time.

How to do it

Direct comparison testing is what you probably picture when you think of competitor testing. You have a handful of sites you want to test: your site, and, let’s say, your top 3 competitors. It might seem efficient to test all of them at once, but that would be exhausting for users, and the results likely wouldn’t be very helpful.

Instead, test specific parts of each site against each other. For example, test your site’s checkout process to Competitor A’s checkout process. Then, with a different batch of testers, do the same thing with your site vs. Competitor B, and so on.

Advantages

What’s nice about this type of testing is that if you want to implement something that your competition does, but don’t know the best way to do it, you can test one competitor against another to see which one users prefer.

Another advantage is that you can ask users outright which site they would choose to buy from. It can be very helpful to hear a user say explain their choice, like in the clip below.

Objective comparison testing

But let’s say you want to gain an understanding of how your site is doing compared to your competition over time. This is where Objective Comparison Testing comes into play.

How to do it

For an objective approach, you’ll create a standard set of questions, and then you’ll run the same test for each site separately. Then you’ll do it again whenever either site is updated. (You can use a tool like ChangeDetection or Website Alert to get notifications about any updates on a site.)

Advantages

This method is great because you don’t need to show multiple sites in one test. It won’t fatigue users; in fact, they’ll never even know that the test is part of a comparison study. To them, it’ll just be a regular test of a regular site.

On the surface, you won’t get the immediate feedback of, “I preferred the first site,” like you would in a Direct Comparison Test. You’ll have to watch every test video, analyze the metrics, and draw the conclusions on your own.

But you’ll gain an initial baseline for your site and your competition, and then, over time, you’ll have a basis for several different comparisons.

• Your current site vs. your competitor’s current site. Which do users prefer and why?

• Your competitor’s newest version vs. their previous version. Did they make important improvements? What can you learn from them?

• Your newest version vs. your previous version. Did your updates make a positive difference in the user experience of your site?

Iterating is inherently apparent with Objective Comparison Testing. When using this method, you’ll need to run tests on each site whenever there is an update. Simply testing each site once is not enough.

Keep in mind, this method is used to track progress over a longer period of time, and is not meant to help fix small issues that your site is facing today.

Use competitor insights to get a leg up on the competition

To recap, it’s important to do some kind of competitor testing to make sure your site can stay, well, competitive with other companies. Here's a quick overview of the two types of competitor testing you can do:

Think about your goals to determine which method to use.

If you need help setting up your study, contact your Client Success Manager; we're always available to work with Enterprise clients. To get started and see how the UserTesting can help you, sign up for a free trial here.