Preference Testing: What to Do Before You Run A/B Tests

In the world of user research, A/B testing is one of the most popular types of tests we hear about. And while they’re great for optimizing something that’s already built, they can’t really help you early on in your product’s lifecycle. But don’t worry, there’s a test for that: preference testing. While preference testing doesn’t get as much press as A/B testing, it’s still a valuable UX tool. If you’re not familiar with preference testing, read on for a quick overview of what it is, when to use it, and what kind of results you can expect.

What is preference testing?

Preference testing is just like it sounds. You provide users with two different choices and ask them which they prefer. But the results go beyond just picking option one over option two. You’re looking for more qualitative feedback. You want to know which option users prefer and why. That feedback is then used to help make larger, big picture decisions that will guide you on choosing the best design for your users. Preference testing is geared toward understanding what users prefer and why, before the product is completed. This is in contrast to A/B testing, which gives you focused statistics based on small changes to something that’s already living in the wild that you want to improve. Designers typically use this approach to test two drastically different concepts or designs. If, for example, you’re doing a site redesign, you could show participants a wireframe of two very different styles of navigation you’re considering. Their feedback will not only tell you which design they prefer, but also how they felt about each. This gives you valuable context that can shape each iteration of the design until you get it right.

When to use it

Preference testing works great early on in your product’s life cycle. As in, right from the idea-on-a-napkin stage. It’s a great way to get proof of concept before ever committing the time and resources to develop a finished project. And by using this type of test early on, you’ll gain a better understanding of what your customers really want and why.

What you’re looking for

Despite the name, there is a bit more to preference testing than simply asking users what they prefer. And while you may start out with that style of question, a big part of what you’re looking for will come from observing their interactions and hearing the “why” behind their response. For example, if you’re testing two different wireframe designs for a website’s home page, simply collecting the quantitative data—percentage of preference to design one versus design two—isn’t going to help you much. What will be important, however, is the qualitative data. Getting feedback that option one elicits feelings of excitement and adventure, while option two makes users feel like they’re on a bland, corporate website will give you a lot of guidance on the design—before you’ve even written a line of code.   While A/B testing has it’s place, it’s not always the solution you’re looking for. Until your site or app is alive and well, preference testing can give you the insights you need to craft a great user experience—even if all you have is a sketch on a napkin.