4 forgotten principles of usability testing

Over the years, I’ve sat through dozens of usability tests run by design agencies. Clients have asked me to oversee the tests to make sure that the agency really puts their design through its paces. This is a good thing as it shows that usability testing is now becoming a mainstream activity in the design community. But many of the usability tests I’ve sat through have been so poorly designed that it’s difficult to draw any meaningful conclusions from them. No wonder that Fast Company mistakenly believe that user centred design doesn’t work.

Picture a usability test

If I ask you to picture one of these usability tests, you’ll probably conjure an image of a participant behind a one-way mirror, with video cameras and screen recording software. Although your picture would be accurate, there’s something missing: the hand of an experimental psychologist checking that other factors are in place behind the scenes. You need this guiding hand because the technology can often obscure the key goals and principles of usability testing. If I wear a white coat and dangle a stethoscope around my neck, it doesn’t make me a medic. Similarly, if I record a picture-in-picture video of someone using a web site, it doesn’t mean I’m running a usability test.

Here are 4 principles of usability testing that have been absent in many of the tests I’ve observed.

• Screen for behaviours not demographics.
• Test the red routes.
• Focus on what people do, not what they say.
• Don’t ask users to redesign the interface.

Screen for behaviours not demographics

You’ll find that you won’t need to observe many people before you see that there’s a problem with the design of the door. It won’t matter if the people you observe are men or women, young or old, tall or short — virtually everyone will experience this problem. The ones that don’t experience the problem have probably used the door before, and know what to expect.

Instead, recruit users based on their behaviour: people’s previous experience with the domain that you’re testing. If you screen participants based on their demographic characteristics you’ll end up compromising on the kinds of domain knowledge you need. You’ll end up with a demographically representative, but behaviourally biassed, sample.

what they say

Lab-based usability tests use a small number of participants: 5 is a common sample size and more than 20 is rare. These small sample sizes work because usability testing focus on cognitive, problem solving behaviours and when it comes to how the brain works, people really aren’t that different from each other.

Problems occur when usability testers shoehorn other research objectives into their usability tests. For example, questions like, ‘How much would you pay for this service?’, ‘What kind of brand values does this site elicit?’ or ‘What are your feelings about this design?’ are totally meaningless when asked in the context of a usability test. You might as well ask people who they will vote for at the next election: you’ll get an answer but the small sample size means it won’t have any predictive value. A more subtle kind of bias occurs when you ask participants to introspect on why they did a particular behaviour. user because people can’t reliably introspect into their motivations.

Don’t ask users to redesign the interface

When participants struggle with some aspect of the user interface, it’s very tempting to ask them how they would like it designed. But if design was that easy, we could all go home and leave it to participants to develop the next generation of user interfaces. So, after the participant has failed to choose the appropriate button, it’s frustrating to hear the moderator ask, ‘What would have made that button more obvious to you?’ The inevitable answer from the participant is, ‘Make it bigger,’ or even that old chestnut, ‘Make it a different colour.’

Asking people to redesign things introduces solutions into a process that’s designed to find problems. Just when you had them in ‘problem mode’, this line of questioning shifts their thinking. It tends to close down promising investigative threads and opportunities and you then risk not properly understanding the problems. Once users are shown a solution, or are asked to think of one for themselves, they get fixated on it and they struggle to think of anything else. The truth is that participants don’t know what’s possible, and asking them to generate design solutions will make them focus on the blindingly obvious or get them to focus on solutions that may work in one context but may not work in yours.

Conclusion: Avoid cargo cult usability testing

The physicist Richard Feynman once wrote about user testing where researchers adopt the paraphernalia of doing scientific activity but forget its core principles of empiricism, integrity and avoidance of bias. In the same way, people sometimes adopt the paraphernalia of usability testing, such as the one-way mirror and the video cameras, but forget the core principles of doing user research. Get those core principles right and you can run a great usability test with just a pencil and paper.

My favorite usability testing question

It’s pretty well-known that you shouldn’t directly ask a usability test participant whether they liked your product or not. As neutral as they may try to be, people tend to be more positive during a study than they would with a close friend. It’s the social pressure of not disappointing someone. So how do you gauge their overall impressions of the product? My favorite approach is to ask,

“Who do you know that would like this product? What are they like?”

Why? It draws really honest answers out of people because it’s a product question disguised as a casual question about the people they know. Here’s how it does that:

• There’s no social pressure. The premise of the question is that the product is good (for some people) — which means that your participant doesn’t have to give negative feedback. It avoids the social pressure issue entirely. Hooray!
• Strangers are surprisingly happy to talk about the people they know. Here’s a great example. When I was working at Imgur, I was running a usability testing on our Android app with a middle-aged woman. She had two kids, two jobs, and doesn’t use social media too often. When I asked who of her friends would like Imgur, she immediately went ‘Ohhhhh boy. I have this friend…he’s always posting all sorts of weird things on Facebook’. Her tone and facial expression was clear — this person is not someone she associates with — and she has no intention of being associated with this app. It turns out, people are fine with sharing honest assessments of other people they know, especially when they don’t know you. But asking them to expressing dissatisfaction for your product in front of you? No way.
• It highlights the people who would be high on the NPS scale. If you ask the question and your user says “I don’t know, I’d love to use the product myself!”, that’s a really good thing. The fact that they’re not thinking of their friends means they’re more excited to get their hands on the product after the study. These types of people will eventually share the product with their friends too.
• It gets right to the personality differences between your product and the user. Every product has a personality. It’s a function of the design of the app, the voice used in the copy, the reliability of the app, etc. You’ll be able to understand how your participant perceives the product’s personality by asking them follow up questions about their recommended person. Are they describing this person as lovely? They probably think your product is lovely. Are they describing this person as someone erratic? Well, your product’s personality might be unpredictable of user testing.

Give the question a shot. Let me know how it works, and if you have any other personal favorite questions!