How many users is enough in a Remote UX test?

For a simple user journey five users is the optimal starting point, especially if you are new to Remote UX testing. However, other factors need to be taken into account when there are multiple audience types and the test objectives are more advanced.

Why Five Users?

In a seminal and often misquoted piece, written more than 13 years ago, Jakob Nielsen explained why five users is the optimal number of participants for a usability test – citing a diminishing ROI for the the sixth, seventh and eighth user.

Since then, the number five has stuck in the psyche of UX practitioners and unfortunately has been misunderstood by many. What Nielsen actually said was:

“The best results come from testing no more than 5 users and running as many small tests as you can afford.”

Evidence from the WhatUsersDo test platform supports this idea: testing in small chunks (of five each time) and frequently is the optimal approach for many organisations, especially those who use our self-service platform. This is because:

– after five users you tend to see the same issues repeating

– five users account for idiosyncratic behaviour

– your time is precious: our user videos average about 20 minutes duration and (including the time to use our “video tagging tool”) you’ll need half a day to analyse five videos.

If it’s your first Remote UX test then our advice is to start with five users and plan on two further rounds of testing as you iteratively fix the issues revealed in the first round. In our experience it’s more important to focus on writing effective user tasks, rather than worrying too much about the number of participants.

You should aim, as Jakob Nielsen puts it, to “collect insights to drive your design, not numbers to impress people in PowerPoint”.

When is Five Not Enough?

There are, of course, exceptions to the rule. In the following cases you should consider increasing the number of participants.

Multiple user segments: where you have distinct groups of customers a minimum of five users per group should be applied, particularly when your site services multiple territories.

Design comparison tests: where you are asking users to compare “flat image” design candidates (perhaps of a new landing page or product page) a minimum of ten users is recommended.

Tree and Card Sorting tests: these tests help you organise the content on your site and blend both quantitative and qualitative methods meaning a statistically significant number of users is required. We recommend 20 users minimum.

There’s one instance where you might think more than five users are required, but our experience shows they are not. That’s when testing semi-functional online prototypes using a tool like Axure. In our experience the “test with five and iterate, then test again” holds true for prototype testing even if that prototype is far from fully formed.

Need advice?

If you’re a WhatUsersDo Managed Service client our UX Team will help design your tests and recommend the most effective number of users, just contact your Account Manager.

If you are a self-serve client and need some help, leave a comment in this blog post and we’ll share our experience with you.

Trackbacks & Pings

Leave a Reply