Why you shouldn’t UX test by gender Luke Richards tackles the reasons why testing along gender lines is to fall at the first hurdle

plugs with bulbs instead of genitalia

Those of us with an interest in mobile and internet usage are familiar with data that deals with demographic breakdowns.

It’s natural to try and get a handle on what is driving the popularity (or not) of an online platform by looking at the gender, age, education, etc. of those using it. So it was surprising when I began learning about UX that testing along lines of gender is not favoured within the specialism.

At WhatUsersDo for example, they haven’t worked on anything that needed to compare different genders. So why is this?

Psychographic vs. Demographic

To begin understanding this, it’s worth going back to one of the earliest texts on UX, The Elements of User Experience [2002] written by Jesse James Garrett.

In it Garrett points to the important distinction between studying people across demographic and psychographic lines.

Demographic profiles can be quite general… Psychographics often correlate strongly with demographics: people in the same age group, location, and income level often have similar attitudes. But in many cases demographically identical people have very different ways of seeing and interacting with the world. (Just think of everybody you went to high school with.) That’s why uncovering the psychographics of your users can give you insights you can’t get from demographics.

The level of insight we can get from psychographics rather than demographics is far more useful and in-depth for UX testers.

An important virtue of UX begins to emerge here, that of reportage in comparison to actual behaviour. We might be able to come to a superficial conclusion about how certain genders report they feel about a platform, but until we analyse the individual behaviour of users on that platform we still have a very foggy idea of actual experience.

This arguably became a common agreement among UX specialists around a decade ago. It’s highlighted in the 2009 paper Understanding, scoping and defining user experience: a survey approach which collects the opinions of 275 researchers and practitioners who delineate UX “as something individual (instead of social) that emerges from interacting with a product, system, service or an object.”

Gender: performative and relative

On the topic of gender more specifically, what we do and the differences between us is more defined by our attitudes and behaviours than anything inherently biological. So my DNA might mean I’m a male and have all the physical attributes that accompany that classification, but my DNA doesn’t define my gender.

In her 1990 book Gender Trouble, Judith Butler delves into the theory that gender is ultimately a performative thing:

When we say that gender is performed, we usually mean that we’ve taken on a role; we’re acting in some way… For something to be performative means that it produces a series of effects. We act and walk and speak and talk that consolidate an impression of being a man or being a woman… we act as if that being of a man or that being of a woman is actually an internal reality or simply something that is true about us. Actually, it is a phenomenon that is being produced all the time and reproduced all the time.

While the biological sex of a person may often at least be a binary one or other (and certainly not relevant to most UX!), gender is in reality a spectrum dependent on the individual’s decision to conform to facets of maleness and/or femaleness – and for many people it’s always changing.

Think again of when Garrett highlights the diversity of behaviours on display among those you went to high school with. There was no doubt a range of adherences to masculinity and femininity across the sexes.

And then consider how difficult it is to depend on groupings of people when those people are defining such groupings so differently.

In UX, to analyse high school students as they purchase cinema tickets via desktops vs. smartphones and then seeking to attribute behaviour to such a broad, relative and fluid notion as gender is counter to the practice of studying actual individual behaviour.

Gender: irrelevant?

In an effort to promote screening for behaviours rather than demographics, another UX specialist Dr. David Travis refers to an example phenomena in his piece 4 forgotten principles of usability testing – he asks UX specialists to imagine a “Norman Door” (named after Don Norman, the author of The Design of Everyday Things).

A door with a handle on it that screams ‘Pull!’ but the architect decided to make people push it — take a seat nearby and watch a handful of people use it. You’ll find that you won’t need to observe many people before you see that there’s a problem with the design of the door. It won’t matter if the people you observe are men or women, young or old, tall or short — virtually everyone will experience this problem.

I noticed a website study which suffered a Norman Door type problem just recently.

In their article A usability study of the obamacare website: Evaluation and recommendations Viswaneth Venkatesh, Hartmut Hoehle and Ruba Aljafari produce a very comprehensive critique of healthcare.gov – a site which needs to be accessible to the entire US population, ranging from the digitally proficient to those who go online less frequently.

The study is informative, but while it suffers from asking users to report rather than observing actual behaviour, it also goes to great lengths to segregate out users by gender, age, as well as whether users did or didn’t vote for Obama in 2012.

Of course the reams of data it presents show slight but little difference between male and female users of the website and the article offers the insightful (though perhaps for the wrong reasons) conclusion: “By a miracle, even if the infrastructure issues were resolved, even Obama backers are likely to continue to find the site to be wanting.”

In conclusion…

UX testing by its very nature should be about observing how a range of individuals interact with a platform, rather than merely asking them how they intend to use it or how they found the experience to be.

As is the case with the healthcare.gov website, even if Obama voters are kinder when asked about the webpage, their actual observed interaction with the website is not much different to that of a staunch republican if the site is as roundly unusable as that example is purported to be.

Embarking, then, on a UX test along gender lines is to fall at the first hurdle. The focus turns to relative, hard-to-define groupings rather than actual behaviour-led actions. Users aren’t suffering from a bad user experience or enjoying a smooth customer journey because of how much they do or don’t conform to a fluid social performance.

The best insight comes from putting psychography before demography and studying what users actually do – rather than what we, society or even they expect to do themselves.

Join our UX Community on Slack

Ask and answer questions, take part in exclusive AMAs, share your own experiences... Discuss all things UX with the best and brightest in the industry!

(It’s like a constantly evolving networking event, and even though you have to provide your own refreshments, nobody will know that you’re wearing pyjamas.) ☕️ 💻 🛏

Main image by Michael Prewett

Luke Richards is a freelance writer based in Plymouth. He specialises in covering areas where technology and digital have surprising and positive effects on society, culture, and business. He can be found on Twitter @myyada.

Leave a Reply