How to test your app store listing

You’ve probably heard of search engine optimization (SEO) and its importance to your brand. App store optimization (ASO) is equally as important. ASO is a method for increasing your app’s visibility in an app store’s search results to boost traffic, downloads, and app conversion rates. In addition, it connects your app with customers.

Looking at some of the most popular app downloads globally, you’ll see household brands like Instagram, TikTok, Facebook, and WhatsApp. And while your app probably won’t dethrone Instagram (at least not yet), you want your mobile app to appear—and remain—at the top of app store search results when users type in a relevant keyword. This is what ASO helps you achieve. 

Before creating your listing, decide where you’ll list your app. Your listing needs to include both iOS and Android users, which requires tailored, separate ASO strategies. Experiment using the app store where it's easiest for you. For example, Google Play is known to offer a less time-consuming submission process. Doing a soft launch on one app store will give you room to make mistakes before your second listing. Or maybe most of your audience uses the Google Play store, and that's why you don't want to start with it. 

While it’s ideal to launch your product on both the iOS Store and Google Play, it makes sense to initially hone in on one, especially for the purposes of testing. Think about your app’s target audience, and even consider running a test to find out the demographics before making a decision. This could be as simple as setting up a test to gauge who’s most interested in your app and making note of their demographic information and device preferences. 

If it seems like your audience is split down the middle on their phone preference, and you prioritize a large market share, launching on Android first may be the better choice. If your organization primarily uses Apple products, then it’s a no-brainer to test on the iOS Store. You may also weigh in factors like app store commission fees, consumers’ app spending habits, submission process speed, competition, and language support. 

Whether you’re testing your mobile app for either the Apple App Store or Google Play Store, or both, listed below are ranking factors that you should evaluate (and examples of what you can test for). However, keep in mind that app store algorithms are consistently evolving, so these factors are subject to change. 

The iOS store prioritizes: 

• Accurate keywords 
• Compelling app name and description 
• Correct category assignment 
• Positive ratings 
• In-app events 
• In-app purchase promotion 

Google Play lists several factors that determine how they organize and rank apps, including: 

• User relevance 
• Quality of the app experience 
• Editorial value 
• Ads
• User experience 

Your app store ranking is influenced by various factors, from the device (e.g., a TV versus a smartphone), where on the app store one is looking, and user preference. By establishing an evergreen strategy and consistently keeping your app’s information up-to-date, you’ll be ahead of the curve when the next update comes around. 

Apple App Store requirements

When crafting your iOS store product page, you’ll need to create the following: 

• App name
• Icon 
• Subtitle
• App previews
• Screenshots
• Description 
• Promotional text 
• Keywords 
• In-app purchase 
• What’s new 
• Ratings and reviews 
• Categories 
• Localization 

Google Play Store requirements

And when creating a Google Play store listing, you’ll need the following: 

• App icon 
• Short description 
• Long description 
• Feature graphic 
• Screenshots
• Preview video 

One thing is certain. Test your app store listing early and often. Depending on your timeline and needs, you should opt to test again during prototype development efforts or after a redesign or product launch. And as you can see from the ranking factors above, there’s no shortage of what you can test.

Before you start testing, identify your audience. You can easily find the right audience with the help of screener questions. When planning your screener questions, opt for users who match your target audience. If you’re hoping for unbiased feedback, you might even screen out those who’re familiar with your app and competitors.

You can even think outside the box. If your app has undergone a significant change recently, you may consider targeting users who used to subscribe to your app but no longer do to learn whether or not they’d be convinced to return with the new app listing.  

1. Test app store description screenshots

An app store’s screenshots, images that offer a glimpse into the app experience, are an eye-catching element. When a user scrolls through the app store or search results, the screenshots take up most of the screen and can entice them to stop scrolling and click on the listing. In fact, an optimized screenshot has the ability to increase conversion rates by 28%, which may be attributed to the fact that typical app store users are 10x more likely to look through a listing’s screenshots over its description. 

Understanding this impact, Rovio Entertainment turned to SplitMetrics before the release of Angry Birds 2 to test app screenshots and icons, analyzing everything from vertical and horizontal orientations to color. Through A/B testing, they discovered that portrait app screenshots boosted conversion rates and found valuable new insights about their top users. The result is that the organization was able to boost conversion rates by 13%, which translates to 2.5 million extra downloads during its first post-launch week. 

Source: SplitMetrics

Apple’s App Store sets a limit of three screenshots for the search results, and 10 total screenshots within the product page. For more detail, view their requirements here. Meanwhile, on Google Play, a maximum of eight screenshots are offered, and you can find more of their guidelines here. 

You should pick carefully when deciding how to use this precious real estate—and take advantage of the amount you’re given. You can use this space to highlight any signature branding or offer a glimpse of the app experience when downloaded. For an extra layer of customization, pick between vertical or horizontal screenshots, though the vertical layout seems more common.

Keep in mind that within search results, your screenshots’ text appears smaller than how it would in the product page. A best practice would be to enlarge your font or keep any copy short and sweet. 

To implement user feedback into this process, consider a preference or A/B test, to gauge reactions from your ideal audience on multiple designs. For more thorough feedback, or an opportunity to probe your test participants, consider a Live Conversation—which may be preferred if you’d rather share your screen over a screenshot. 

Here are examples of what you can ask: 

• Which design do you prefer?
• Which design do you find the most visually appealing? Why?

2. Testing app store icons

Your app store icon is an integral part of your brand that helps distinguish your organization from others and makes it easy to find. As a design principle, remember that the simpler, the better, especially to make it compatible across different devices. 

Remember that your icon isn’t set in stone, as Apple’s App Store and the Google Play logos have evolved over the years, along with many others like Uber and Instagram. As customer preferences change, so do logos, which means testing your icon is a step that can’t be skipped in your ASO strategy. 

Source: 9to5Mac

Like with screenshots and other visuals, you may consider conducting a preference test to test several different design options. If you’re making a drastic change from an existing app icon, be sure to get user feedback on the new design concepts early in the process. This early feedback will help you effectively iterate your design because it’s based on the opinion of your customers. When you land on your final design, launch it with confidence knowing your target audience has already shared their perspectives on the design. 

Here are some sample questions you can ask when testing your app’s icon:

• What adjectives come to mind when you see this icon? 
• Does the icon match what the organization is trying to achieve? 
• What would you change about the icon? Why? 
• How does the new icon compare to the original? 

3. Testing app store reviews

We’ve all had that moment of being in an app and receiving a pop-up notification to leave a review or rating—and sometimes, this may have been more unwelcome than others. To increase your chances of receiving a review, conduct a test beforehand on the ideal moment for this message to appear (like when a user may satisfied with the app), and hear user feedback on any contributing factors. 

Additionally, you can rely on the existing reviews you already have to help guide your future tests and pinpoint areas needing improvement. You may also find it useful to evaluate your competitors’ reviews to target common complaints or pitfalls which you can strengthen within your own app. 

Understandably, consistent negative user reviews can decrease an app listing’s ranking. And with the app rating being one of the first few elements users can see in search results, which can make or break a first impression, this makes it an area worth prioritizing. 

Potential questions to ask include: 

• What motivates you to leave an app review? 
• What do you think about being asked to leave a review? 
• How much weight do reviews and ratings have when you consider downloading an app?

Whether your goal is figuring out if your app store listing's imagery or messaging align with your brand, UserTesting templates help you get the answers you need. Templates are customizable to fit your exact testing requirements. They're also great for browsing to get inspiration on what's possible with testing. 

Template examples:  

• App store feedback
• Metaverse and VR
• Visual design
• Messaging comprehension

These templates are designed by research experts to give you the unique ability to: 

• Evaluate the clarity of your app’s description and if it comes across as intended 
• Assess whether your listing resonates with your target customers and what's most or least compelling 
• Understand customer expectations

Armed with templates, you may choose to pursue moderated or unmoderated testing. Both of these test types are solid choices for evaluating your app store listing, but your decision ultimately boils down to your needs, resources, and time constraints. 

Moderated tests, known as Live Conversations, are one-on-one interviews that offer plenty of opportunity for follow-up questions and probing. Meanwhile, unmoderated tests can be completed on a contributors’ own time. 

If your focus is your visual assets, like screenshots or preview videos, consider A/B testing. These can be structured as a Live Conversation or unmoderated test. If you can’t decide between two images or videos, contributors can serve as tiebreakers and either offer validation or voice insight you may not have even thought of. 

Potential questions to ask contributors include: 

• What’s your first impression of this app store listing? 
• When it comes to device compatibility and languages offered, does this app seem accessible and inclusive?
• Is it clear what the app offers and why you should use it? 
• Is it clear what the app's organization does? 
• If you had a magic wand, what would you change about this listing?  
• Is the listing transparent about data usage and privacy policy? 
• What, if anything, is holding you back from downloading the app? 

Still not convinced? Take it from our customers, who have successfully used UserTesting’s Human Insight Platform for their challenges. Augmented reality gaming app Krikey conducted tests on their Google Play store listing, where the majority of their customers make a decision to install or not. Contributors gave valuable feedback on everything from product descriptions, imagery, and their preference for certain terms over others. 

After implementing feedback from their initial tests, Krikey found that their Google Play Store install conversion rates increased from 5% to 40% in just six months. And that’s not all; combined with tests on their mobile app itself, Krikey has since reached over 1 million users. 

For more on their story, check out the video below. 

Krikey is just one example of a customer who has successfully tested their app store listing. Healthcare app Your.MD, also known as Healthily, aims to help anyone find reliable and customized support for their health. Looking to enhance their product’s UI and UX, they found recruitment agencies too time-consuming for their needs, turning to UserTesting. While their main focus was assessing their app experience, they also found that the core feature’s description misled and confused customers. After removing the description from the US app store, US download and retention rates increased. 

Ready for the next step? 

And when you're ready to focus on your mobile app itself, reach for the mobile app evaluation template or the mobile app comparison template. 

While your app store optimization efforts are just one aspect of your mobile strategy, they can make a huge impact on the success—or failure—of your app. Using human insight as your north star to evaluate and optimize your listing will validate your data metrics, and get you on track to improving them with real, customer-centered feedback. Combine this with a mobile app that lives up to what its listing promises, and you have an app store force to be reckoned with.