Confessions of a remote testing convert!

By Suzanne Hutson – user experience, information design expert

Maybe the words ‘confessions’ and ‘convert’ in the above title smack of an almost religious fanaticism. And yes I admit, I am a bit of a remote testing ‘evangelist’.

So, if you don’t wish to risk renouncing your faith in lab-based testing and become converted to the joys of remote testing; look away now!

Note: this article is about remote, un-moderated (‘asynchronous’) testing, not the remote, moderated variety. Read this article for a comparison

Users are good at user testing!

As a UX consultant and die-hard fan of lab-based usability testing, when I first encountered remote un-moderated testing, I was sceptical.

Sure, the cost benefits of remote testing were obvious, but the word ‘un-moderated’ was a major sticking point for me. Surely ‘moderation’ is the whole point? Doesn’t a UX ‘expert’ need to be present to ask probing questions, remind participants to ‘think aloud’ and guide them through the tasks?

With a fresh perspective (gained of remote testing experience at  WhatUsersDo); I now believe that I was both overemphasising the value of the moderator and underestimating the skill of test participants.

While reviewing videos of test participants in un-moderated tests, I’m frequently amazed by users’ capacity both to articulate their problems and to provide valuable insights about the user experience, without the benefit of a UX expert peering (however inconspicuously) over their shoulder.

UX experts are human

Even the most objective UX expert brings baggage to the lab-testing suite.

If you’ve been a moderator for lab-based testing, can you hold your hand up and honestly swear that you never, ever ‘led’ users in think aloud? That you’ve never relentlessly probed a particular problem (that you ‘know’ to be a major issue) nor glossed quickly over another (that you ‘know’ to be a minor one)?

And who has not, at the end of a long day moderating user testing, been spotted by the test participant smothering a yawn, glancing at their watch, or providing other subconscious ‘let’s wrap this up’ signals?

If you are never guilty of these transgressions, I would offer that you are a rare gem and your salary should be doubled (tell your boss I said so).

Every conscious or subconscious signal we give the user has an influence on their behaviour, which will influence the test outcome and hence validity.

Of course it’s difficult to make reliable comparisons between different usability evaluation methods (see Damaged Merchandise? by Gray and Salzman), but there is a strong case for the argument that un-moderated tests provide more realistic (and therefore more useful) results.

It’s a bit like watching Big Brother

Even assuming the moderator is a robot incapable of bias or boredom, their very presence may influence the test participant. In lab-based testing, participants are often keen to please and/or impress the moderator, and this can manifest itself in attempting to do things the ‘right’ way, or provide a ‘balanced’ viewpoint.

In contrast there is a ‘Big Brother’ feel about remote un-moderated testing. Test participants appear to forget their responses are being recorded, and are prone to letting out exasperated sighs, snorts of derision and disbelieving laughter.

In my experience, without a moderator present most participants are less inhibited, less eager to please and more likely to vent their true feelings than they are in the lab (see The Hidden Benefits of Remote Research).

Choice comments from recent projects include “Seriously? You’re kidding me, right?”, “This is really, really, really bad”, and “Please don’t make me wade through all that crap again!”

The outcome of this honesty is that it is crystal clear which usability issues cause the most frustration, and this provides crucial signposts to the design changes which will lead to higher conversion rates.

Furthermore, video clips of such candid responses make an immensely powerful presentation tool to get your message across.

Not quite ethnography … but nearly

You can see where I’m going with this. It seems self-evident that remote testing in users’ home or work environment has greater validity, because it’s closer to the real experiences that users have, leading to more natural behaviour.

This is particularly the case in time-aware testing, in which test participants are recruited on the fly when about to embark on a ‘real life’ task (such as booking a holiday). The timing and relevance of the task is completely genuine, leading to more ‘ecologically sound’ results (see Rush, Halko et al); surely the closest we can get to applied ethnography in the user testing sphere

It’s more democratic

Remote un-moderated testing is much, much cheaper than lab-based testing. So, smaller companies and start-ups are more likely to be able to afford it.

Anything that gets more people doing user testing and watching users gets my vote.

Learn faster

In my lab-testing days I thought I knew quite a lot about users. But as a UX analyst with the What Users Do team I’m gaining new insights every day.

Partly these insights arise from the honesty of users in the absence of a moderator.

Partly this arises through not having to moderate. There are no distractions from purely watching users’ behaviour. I can get through 8 or more videos a day, and my learning rate is correspondingly high. It’s intense, but hugely satisfying.

The list goes on …

The benefits I’ve listed above are my particular favourites. There are others:-

  • It’s easier to recruit busy, employed people to do remote testing than lab-based testing, for obvious reasons. So it’s easier to get a representative test sample.
  • Testing a global audience is much easier and cheaper, with no time-zone issues
  • It’s quicker. Users can be recruited, tests run and the results analysed in a couple of days. This supports testing ‘little and often’ (see How many users is enough in a remote UX test?), a perfect approach for Agile and fast-moving business sectors.
  • It’s easy to collect metrics as you go. Likewise the test can include an online questionnaire or a written summary by users.

Ok so it’s not perfect

I’m not such an evangelist that I can’t recognise some of the limitations:-

  • Not every single user is great at think aloud, and without the opportunity to query their actions it can be tricky to work out what’s happening, and why. But this is rare and it’s easy to replace a video if this happens.
  • More care must be taken over task design, to ensure users don’t get stuck and abandon the test

Try it …

Don’t take my word for it, try it out for yourself. WhatUsersDo offers subscription plans, which are quick and easy to sign up for (though you’ll need to write your own tasks and analyse the videos), as well as Managed Service Plans, where you get the benefit of their in-house UX Researchers and Planners.

6 Responses to “Confessions of a remote testing convert!

  • I am also quite keen on remote un-moderated testing but recognise that ‘pre complete production release’ lab testing is often the only possible option, because the functionality exists as distinct pieces. For example a log in may exist but after that the application goes no where. Does this mean you should not test the log in isolation, I believe you should. In a lab context it is possible to fabricate the links or even just talk them through. I think producing working pieces of software and completing behavioral tests as they are produced has value. Take the log in example, the user is given the option of logging in with their Social Media account. The functionality may all look good and work in isolation but be written so that subsequent log ins rely on the fact that the user recalls which Social media account they used for their initial log in. User behavior tests may show they use the SM account that they are using most at that moment in time and highlight the possible use of a client side routine to present the credential type they used the last time they logged in. So even though a snippet of the application is written it benefits from very early testing in a mock environment.

  • I am also quite keen on remote un-moderated testing but recognise that ‘pre complete production release’ lab testing is often the only possible option, because the functionality exists as distinct pieces. For example a log in may exist but after that the application goes no where. Does this mean you should not test the log in isolation, I believe you should. In a lab context it is possible to fabricate the links or even just talk them through. I think producing working pieces of software and completing behavioral tests as they are produced has value. Take the log in example, the user is given the option of logging in with their Social Media account. The functionality may all look good and work in isolation but be written so that subsequent log ins rely on the fact that the user recalls which Social media account they used for their initial log in. User behavior tests may show they use the SM account that they are using most at that moment in time and highlight the possible use of a client side routine to present the credential type they used the last time they logged in. So even though a snippet of the application is written it benefits from very early testing in a mock environment.

  • Interesting article, and lots of food for thought!

    Based on my own experience, I would agree that having participants complete the usability session ‘undisturbed in their own environment’ can be beneficial. I do believe though that your yield will depend on the type of interface you are testing and the level of maturity of the interface. Testing a consumer app, especially one that is similar to others out there, could be very straightforward and yield interesting results.
    However, having participants wade through a novel idea or work with an immature version of the software will likely result in people wandering off-task or not providing you with the type of feedback you were looking for. As much as it is the task of the moderator to be invisible and not influence, there is value in bringing participants back on the right track if necessary.

    As always, I think there’s a time and a place for every methodology.

    Thanks for the well-written post.

  • Interesting article, and lots of food for thought!

    Based on my own experience, I would agree that having participants complete the usability session ‘undisturbed in their own environment’ can be beneficial. I do believe though that your yield will depend on the type of interface you are testing and the level of maturity of the interface. Testing a consumer app, especially one that is similar to others out there, could be very straightforward and yield interesting results.
    However, having participants wade through a novel idea or work with an immature version of the software will likely result in people wandering off-task or not providing you with the type of feedback you were looking for. As much as it is the task of the moderator to be invisible and not influence, there is value in bringing participants back on the right track if necessary.

    As always, I think there’s a time and a place for every methodology.

    Thanks for the well-written post.

  • Suzanne
    4 years ago

    I totally agree that there are times when lab testing is much more appropriate (indeed, sometimes the only option), and your point about pre-production testing is well made. I should have included this in my list of limitations (although as I was playing devil’s advocate a little, I didn’t want to list too many limitations and shoot down my whole argument in flames!).

  • Suzanne
    4 years ago

    I totally agree that there are times when lab testing is much more appropriate (indeed, sometimes the only option), and your point about pre-production testing is well made. I should have included this in my list of limitations (although as I was playing devil’s advocate a little, I didn’t want to list too many limitations and shoot down my whole argument in flames!).

Leave a Reply