Shut Up!

Mar 4, 2020

You should stay silent when conducting a usability test.

As part of my work in Somia Customer Experience (check us out at somiacx.com), I’ve been running and teaching a lot of Usability Testing. Despite being a very simple form of research, I found that a lot of professionals are still confused about how to conduct good Usability Testing.

Here’s one good pro tip: shut up!

Usability Testing is an observational type of research. That means the only thing that you should do is to observe how the participants work on the given task. You are not supposed to ask questions, probe, or disturb them when they’re working on it.

Photo of me training a room full of VPs — From my experience training people from VP levels to juniors, because of its simplicity, Usability Testing can be one of the toughest to digest as people tend to overthink and worry about not getting the most of the session by sitting silent and observing.

Why? To avoid bias.

Bias is the enemy of research and there are a lot of them that you can encounter whether it is before, during, or after the research sessions (go ahead look these research bias up on the internet as I would not discuss in length about them here—or you can leave comments below and force me to discuss it in later posts). The more bias there is in a study, the less accurate its results are.

When you’re doing observational research, you want to observe your participants in their most natural state. You want them to act like they should when there’s no one around watching them so that you know exactly what happened.

Photo of a fictional young girl named Jenny (image generated by AI) — Jenny looks sweet, but she’s something else if your teacher’s not there.

Here’s a story: You probably have that one friend in the class who is a bully, let’s call her Jenny. You and your friends are fed up and have complained about her behavior, but the teacher never really does anything as your teacher thinks Jenny is a sweet girl—and yes, that’s because she’s always acting nice when the teacher’s around.

Yep, Jenny changed her behavior when she’s aware that the teacher’s around so that she’s not getting into trouble. The same goes for your participant too.

Just from them knowing you’re there, your participant might shift their behavior. Whether it is to make you feel it’s worthwhile to invite them there, to avoid hurting your feelings, or simply to not look like a fool in front of you. In any case, they shift their behavior to a degree. You want to actively manage the degree of that shifting so that it’s not over skewed your research results.

Now, with that issue already at hand when you start asking questions, you essentially introduced even more bias in the process. When you asked them a question midway, you make them think and that might be about something that they’ve probably never thought about when doing the task. This ends up cutting off their train of thoughts, or worse, change the next course of action.

A participant seen talking to impress — When you probe, the participants can end up talking too much about their past experiences and less about experiencing the product that’s being tested. This might give you good insights on her past experience, but in a Usability Testing, you want to see them experiencing the product more than hearing stories of it.

If you think this never happened to you, think again. Because most likely, when I told you to “look these research bias up on the internet” in the fifth paragraph above, you’ve got your train of thoughts cut off—or worse, changed your course of action from continuing reading to literally looked for research bias online. Even if you don’t, I’m pretty sure there are a lot of moments when you got distracted from reading on Medium because you’ve stumbled on an interesting link in the article.

Now that you understand and can sympathize, as a moderator, what should you do then? Simple: just shut up and observe what your participants do.

What you need to remember is that you can always ask the questions when your participant’s done with their task at the end of the session.

“But what about those thinking out loud things I’ve read a lot like in Steve Krug’s books?” you asked?

A clip from Ed Sheeran's song — To be clear, by ‘thinking out loud’, Steve Krug’s not talking about Ed Sheeran’s song. And if you’re wondering about the books I’m referring to, they’re called Rocket Surgery Made Easy and Don’t Make Me Think. They’re good starting points to learn about Usability.

Asking your participant to think out loud is a different thing from probing. It’s just requesting your participants to speak up on anything that’s one their minds during the process. Sure, it creates bias too—like some people who don’t read texts might end up reading the text—but far less disruptive than making your participants answer questions on stuff that’s not in their train of thought.

Now, let me teach you some mantra for you to learn. I suggest that you try to repeat after me, okay?

Ready? Here we go:

When your participant says “I think this is not right”, you say: “…”
When you notice your participant did not take the right path, you say: “…”
When your participant did something interesting, you say: “…”

Great, you did well!

Remember, all you got to do during the session is to take notes on interesting things your participants do and ask them later at the end of the session. Silence is golden.