3 aspects of running UX tests that I didn’t enjoy

UX testing challengesWhatUsersDo pays me to write nice things about UX testing – so… what’s possessed me to write an article with a title like this one?

The fact is nothing’s perfect – not even the process of running user experience tests. If it were, everyone reading this blog post would be a customer.

Plus, we aren’t serving people honestly if we pretend otherwise.

We’re encouraged to always walk in our customers’ shoes at WhatUsersDo. So, while I was creating our storybook site, on increasing eCommerce sales, I worked like a bootstrapping, time-poor, one-man team.

I became the archetype which most desperately needs UX testing, but has (perhaps) the strongest excuses for not giving it a try.

The fact that I was personally and professionally attached to the work presented me with challenges I hadn’t faced while creating customer success stories.

Here are the 3 things I least enjoyed while running remote UX tests – and how I got around them.

1. The crippling sense of self-doubt

I can be a cocky bastard – especially when it comes to creative endeavours. Watching users interact with my creation tamed me.

I wanted to stop watching people rip my work to shreds almost as soon as I’d started. I was only testing a prototype… but that didn’t matter.

The feeling was strong enough that I could see how a user on a free trial of our platform might have just tuned out – no money is on the line after all. But refusing to watch would be a terrible mistake. Let me explain.

First, listen to this lady poopoo the central concept of my labour of love:

Nonetheless, I watched (in full) all 5 videos from this round of testing. After a (very short) while, I became so excited about all the things I could improve (and relieved because I discovered them), that I stopped giving a damn about my ego.

What this lady was giving me was an opinion, not a real usability issue. She wasn’t confused by the words, iconography or design – she was simply expressing a preference.

I’d have missed genuine usability issues had I been too sensitive to take a few opinions on the chin.

By the end of my analysis, all I could think was, “OMG… this website would’ve been so shit had I not run UX tests. How could I have missed so many opportunities for improvement?

2. The distracting lure of compliments

There were a few users who loved our storybook site – even at prototype stage. I mean, this guy was all over it like white on rice:

I found it hard to stay focussed while watching users like this –
the rainbows and lilac-scented unicorn farts had me feeling like a genius.

I almost missed it when he struggled to find key information. I watched the video for this particular user twice because I was feeling too smug to pay attention the first time round.

I’m so glad I did because it reinforced my decision to put an index (of sorts) on the home/cover page.

3. “When the hell will I get time to watch and analyse videos? I still have my real job!”

Man, I feel you guys on this one. As the sole content writer/manager/strategist at WhatUsersDo, time is (by far) my most precious commodity.

The same is true for every single person in our company – so, who’s gonna spend 5 hours watching and analysing videos… for an experimental storybook site?

The answer? No one. Not even me.

But I came up with a hack that let me watch and analyse videos in a fifth of the time it would ordinarily take.

My hack? Pen and paper.

If I’m found dead in a ditch, any time within the next couple of days, our UX and product teams did it. This confession will probably have them seething (I’m sorry, guys).

Instead of using the tagging features on our platform to categorise and save insightful moments from videos, I scribbled short notes while watching recordings.

I wasn’t gonna share tags with anyone else in our company. I wasn’t gonna deliver big presentations with key moments of insight extracted. I was the only one who’d be using these notes. I was working like a one-man team, remember?

Using pen and paper meant I never had to pause playback. I never had to catch exactly which words were said. I never had to enter detailed or organised notes. I just used my own special form of shorthand.

Instead of spending an hour analysing each video (as we, correctly, advise), I spent only the duration of a video (about 15 mins, on average) taking notes.

Would I have gotten better, more useful insights if I’d used the correct methodology? Abso-freaking-lutely.

But if I’d waited until I could do everything perfectly – until I felt like I had time to spare – I might not have done UX testing or analysed the videos at all.

That would’ve been disastrous. Some form of testing and insight is infinitely better than no UX testing at all. Believe me – I realised that once my neck/work was on the line.

And that’s how a full round of UX testing cost me just £150 and about 80 mins.

Check out our storybook site: UX and eCommerce – Separated at Birth

UX eCommerce Optimisation

Eco (eCommerce) and Uxley (UX) – the stars of our microsite


Reunite UX and eCommerce

Timi is a London-based copywriter and full-time marketing sceptic – there are now more unvalidated opinions out there than ever.

He became a UX testing enthusiast after seeing its power while working at TUI – the world’s largest travel, leisure and tourism company. He then joined WhatUsersDo to sharpen his UX knowledge and work side-by-side with the field’s best and brightest.

Leave a Reply