How to get feedback over time with diary studies

Diary studies are a form of longitudinal research (research that takes place over a long period with the same participants). Typically, users self-report their activities at regular intervals to create a log of their activities, thoughts, and frustrations. It’s a useful approach for capturing organic feedback on activities that are repetitive, long, or unpredictable. Research questions that involve habits, prolonged processes, or perception change over time (likelihood to use, learnability, etc.) are prime candidates for diary studies. As with all longitudinal studies, the amount of time required for the study is important. If you can get a reasonable snapshot of behavior in an hour or a day, for example, diary studies may not be right for your questions—try an unmoderated study instead. While diary studies are more complex than typical usability studies, they can be done using the UserTesting platform.

When to use a diary study

Here are some examples of scenarios that might lend themselves well to diary studies:

• Long processes: Tracking how users complete a process that’s likely to take several days, such as shopping for a large/expensive item or looking for a new house or apartment.
• Initiating specific actions: Discovering what motivates users to initiate a specific action, like using a search engine or looking up recipes.
• Understanding habits: Determining how an app or website factors into a user’s regular habits, such as fitness or fertility trackers.
• Engagement: Assessing retention or learnability; for example, seeing whether your users stay engaged with a game after a certain period.

Diary studies are also great for capturing information that you might not normally think to do with a remote, unmoderated test. For example, you could recruit users from the UserTesting panel and ask them to review a physical product like a fitness tracker, or you could ask them to play a Wii game for several days. Similarly, you can use a diary study to discover the habits and motivations that users might not be able to give accurate, nuanced, or complete enough answers about during a standard, one-time test.

Diary studies vs. iterative testing

Although they may seem similar, don’t confuse diary studies with iterative research. Many UX practitioners test the same product consistently and often (as you should!), and may even use the same participants if they so choose. This type of testing provides a snapshot of a user’s experience with the product. Diary studies, on the other hand, provide a look at what prolonged exposure to a product or process is like for the user. This can also include usability issues, but these studies are typically of a broader scope—looking for problems that affect or derail the overall process, rather than small annoyances or design preferences.

Pro tip: Diary studies are good for “how” questions: How do people use this? Regular usability studies are best suited to answer “why” and “what” questions: What’s is broken on my site? Or, Why don’t users like this feature?

Complex but compelling

While diary studies are a bit more time-consuming than regular usability testing, their uses are diverse and fascinating. They provide a clear slice-of-life image of your users, their needs, motivations, and reactions. UserTesting customer and e-learning platform, Udemy, relied on a diary study to better understand how students utilize both desktop and mobile devices when using the platform. UX Researcher, Michelle Fiesta noted one advantage to the diary study methodology,

We wanted to observe our users over a longer period of time, noting behaviors as they happened to determine how/if these user groups were different, which wouldn’t have been possible with a one-off interview.

How to conduct a diary study

As with any customer research, the effectiveness of a diary study depends on proper set-up and study design. It’s important to note that diary studies can be more complex than remote, unmoderated studies. When possible, it helps to have direct contact with your participants, typically through customer interviews before and after the study is completed.

Recruitment

We recommend keeping the demographics and requirements as flexible as possible. This helps keep the recruitment schedule on track while giving your team more time to inform participants about the study length as well as what will be expected of them. Additionally, because diary studies require more time from participants, higher compensation than standard customer research will be required.

Pro tip: Diary studies are more intensive than traditional unmoderated studies. We recommend limiting the total number of participants to around five or six per demographic group, which should reveal plenty of valuable insights.

Testing

It’s important that the instructions that you send to participants are clear and explicit. Outline exactly how often you expect them to keep a diary, how long the diary should be, and whether you expect images, screenshots, survey responses, etc. Try to keep your instructions as clear and simple as possible. It can be helpful to provide a list of questions that you want participants to answer or keep in mind as they complete their diary. For example, you might want to ask participants what prompted them to start [activity] each day, what websites/apps they used to complete the activity, and what their experience was like each day. Because of how organic diary studies are, users may take your question list as a suggestion rather than a strict plan, so if there are questions that you must have answered, be sure to make that explicit as well.

Pro tip: Be sure to pilot your study with at least one participant before fully launching your study. This will help ensure that participants understand your instructions and highlight any potential pitfalls.

Pre and post study best practices

At the beginning of the diary study, you can schedule a live interview session (30 or 60-minute sessions) to help set expectations and get to know your participants. Establishing this connection can help participants stay committed and responsive throughout the study while also helping reduce participant dropouts. Additionally, it can be helpful to schedule follow-up interviews with participants once their diary study is complete. This gives both you and your participants the opportunity to expand on their overall experience and address any questions that may have arose during the study. This approach is especially helpful if you have a limited number of participants, as it will give you additional insight and detail to complement the findings from the diary studies.

Want to learn more?

If you’d like to learn how UserTesting can help you understand your customers through on-demand human insights, contact us here.