The following is a guest post by Lorraine Chapman, Director Research at Macadamian. If you want to submit your own guest post, click here for more information.
With the rise of analytics and big data, there is a ton of information out there that can give you insight into how customers use your products and feel about your brand. For example, tools like Google Analytics, Flurry, and Preemptive Mobile give you more data than you know what to do with if you want to analyze usage patterns of your web and mobile devices.
Analytics solutions like Netbase allow you to parse through millions of tweets on Twitter and get a sense for how customers perceive your brand. It’s all very exciting, but these reams of data can also be highly misleading.
Have you really spotted a trend?
If you ask enough questions, and gather enough data, meaningless patterns will emerge. Simple example: if you ask enough people what month they were born in, and a hundred other research questions, you will “discover” things like people born in November drink a disproportionate amount of Budweiser compared to everyone else, or that 86% of people with July birthdays like driving foreign cars over North American cars.
You haven’t actually discovered anything except a statistical phenomenon known as clustering. In a large set of random data, you get clusters of the same type of information. Roll a pair of dice 100,000 times and you’ll get strings of snake eyes or 12’s and other unlikely sequences. It doesn’t reveal a strange meaningful pattern. There are just a lot more clustered patterns than non-clustered possibilities out there.
To find out if the trend has meaning, you would have to run a separate follow-up experiment. Automatically assuming that a cluster of data has meaning — and making design decisions based on that — can lead you astray.
For example, a client once asked us why their website was so popular on Tuesdays, as shown by their demographic data. It didn’t make sense, so we asked them to redo the test. In the second test, the pattern disappeared.
When gathering web or mobile analytics, survey data, or any kind of quantitative measure, your biggest asset is knowing the right questions to ask. Start with a theory that you want to prove or disprove, or use analytics and surveys to steer more detailed user research. For example, you could use survey data to get an overall preference for a feature, then use more qualitative methods like user interviews to understand that preference in context and detail.
Misunderstanding statistical significance
Conclusions drawn from large amounts of data (from social media, analytics, BI and even traditional methods like customer surveys) are often highly regarded because they are “statistically significant”. However, when it comes to testing how users actually interact with your product (and hence what new features, improvements and innovations would best help them), statistical significance is far from a sufficient criterion for meaningful or proper research on its own.
On the other hand, you can’t draw conclusions from interviewing only two or three users.
So how many users do you need to speak with or observe?
It depends on the range of user groups your product is targeted to, the scope of product interactions you want to observe, how the results will be used, and how many rounds of research you conduct.
If you want to test a software application for usability and opportunities to improve the design, focus on one or two primary user groups, and five to six users each time will detect most usability issues.
Usability testing
Jakob Nielsen has shown that approximately five to six users will likely detect 80% of usability problems for a specific use of a product. Keep in mind this “formula only holds for comparable users who will be using the product in fairly similar ways.”
Laura Faulkner tested Nielsen’s theory on a web-based employee timesheet. She ran tests with a group of 65 users, then selected random groups of five participants and compared the percentage of issues each group of five detected compared to issues detected by all participants.
Nielsen’s 80% rule held, but variability meant some groups of five identified as few as 55% of the issues. By increasing the number of users to ten, groups found an average of 95% of issues.
Ultimately, the number of users you need to draw on depends on what type of information you want to get from your users. For a product benchmark study, you’ll need to run with a much greater number of users than a usability test, and you have to make sure you test across all user groups. For customer interviews, the number can vary significantly.
In summary
Many important research goals can be satisfied with a limited amount of research from a limited number of participants. Product Managers can seek advice from User Experience researchers with a background in experimental practices to help determine the extent needs of a product.
As analytics solutions get more powerful and the Big Data movement takes off, there are more and more novel ways to glean insights from customers about a product and brand. But having loads of data in and of itself doesn’t guarantee anything. Product teams need to maintain a healthy skepticism and balance these new data trends with traditional user research methods like contextual interviews and usability testing in order to truly understand how users are interacting with products, and where the opportunities for improvement and innovation lie.
Lorraine
——
Tweet this: Customer Analytics – A Double Edged Sword – http://wp.me/pXBON-3id #prodmgmt #ux