← Back to  Writings

Remote Usability Testing Learnings

Apr 23, 2020

Here’s how we did our fully remote Usability Testing sessions during COVID-19 and what we learned from it.

In Somia CX, we believe that direct observation is preferable to remote as you can get immersed in the context so we hardly ever do remote Usability Testing sessions. Unfortunately, the situation provides us with no choice. We know this will not be easy, but we accepted the challenge, and here’s what we learned.

If you just want to know the setup, go to the last part of this post.

The majority of communication comes from non-verbal

The tricky thing with an online setup is that you cannot see the participant directly.

So we decided early on that we wanted to see participants so that we know how they react when interacting with our prototype. As we’re doing a mobile app, we tackled this by asking the participant to join via both Laptop and Mobile phone.

Screen capture of a call where the moderator's and participant's faces can be seen as well as the screen sharing
We’ve decided early on that we wanted to see the participants while they’re using the prototype.

The laptop is used to cast the video of the participant while they’re using the prototype, and the mobile is used as a platform for the user to interact while being screen-shared at the same time. Both are connected to a single conference call platform so that the moderator and the observer can see both the video and screen-shared at the same time.

Of course, you can switch the other way around if you want to test a desktop app.

Streaming with voice, video, and screen

As the participants will stream three things at the same time: voice, video, and screen, we need to make sure that the participants have a decent internet connection.

In Indonesia, we generally have a very low upload stream rate compared to the download, based on my experience, they’re usually only around less than half of the download rate.

Looking at the requirements, most platforms (Zoom, Microsoft Teams, GoToMeeting) require 1 Mbps download and upload rate minimum for a conference call with both video and screen-share. That means a minimum 4G connection is required. Despite this, we usually still screen for private wifi connection availability to ensure a smooth session.

As for the platform, we’ve tried using several different platforms out there (Lookback.io, GoToMeeting, Microsoft Teams, Skype, Google Hangout/Meet, Cisco, and even TeamViewer for seeing mobile screens remotely) and convinced that Zoom is the one to go with. We chose it for multiple reasons, but in this specific context, it’s due to it:

Thinking out loud for the win

I also noticed that this is where the thinking-out-loud protocol shines. As we’re using an online setup, we don’t have a way to identify where the participant is tapping on the screen, so we need to ask them to inform us where they tap. So we make additional efforts to make sure the thinking-out-loud process is clear.

Not that I am not aware of the ability to turn on Show Tap area on Android, but it has its flaws. For (1) there’s none in iOS, (2) even if we go with Android users only, it’s troublesome to tell the user how to do it by themselves—not to mention there are a lot of different Android skin making it more troublesome for us to prepare the instruction or would take time to dingle dangle when guiding the user through the setting.

Screen capture of 'Show tap area' setting in Android

For those who don’t know where the Show Tap area (also known as Show Touches in other versions) setting is, you need to go and activate Developer Options first by tapping your Android build number in some subcategory of your setting (varies among Android skins) eight times, then open the developer option, look for the setting, and turn it on. Imagine explaining that to someone on a call.

Prepare in details

As we limited the session time to 60 minutes (one of our quick tips on remote research that you can read here), we wanted to make sure that the session ran as effectively and efficiently as possible.

We wanted to make sure our participants felt their time was valued and fully appreciated. We don’t want to waste our time fixing technical issues that we can prevent prior, or look unprepared for the session. We wanted the session to be less stressful for us. So we prepare more than we think we need.

I personally believe this is the one thing that ended up really helping us get through the session.

Not only that it make the sessions more enjoyable and less stressful for everyone, but it also helps us can focus on the content rather than the technicals during and after the session. It also makes running back-to-back sessions and their transitions much easier.

I know it’s called planning, but the way I see it, it’s about designing the experience of the study session. So that’s the way we did it. We planned by making the participant’s journey and identifying things that can go well or not during the session and preparing for it.

We started planning from the recruitment up to the end of the session:

Essentially, we planned in detail. We also make sure to do this in a single document so that we don’t have to scramble around looking for stuff and have everything we need within reach.

Screen capture of our pilot test session where the moderator's and participant's faces can be seen as well as the screensharing
Our pilot test is to check all the setup and identify other preps that might be missed.

Then we did a pilot test of how the session runs while taking notes on things that we might have yet to cover. We did it like it was the real deal. We acted the parts and the role, we ran the whole study from the recruitment to the session. We make sure we know when to start recording, and who will record. We evaluate what to say during instructions and information parts to make sure it’s crystal clear. We even have a person enacting the role of note-taking so that we know how to do it well. This is so that we’re familiar with the setup we made and it paid well.

The overall setup

My screen setup during remote usability testing session: TOP-LEFT is the call app with participant list control open for easy access to muting unwanted voice and private chat open for sharing links and scenarios with the participant; BOTTOM-LEFT for note-taking; BOTTOM-MIDDLE for chat for internal discussion with observer; RIGHT is the discussion-guide
This is my overall setup when running the Usability Testing where I can see and monitor everything I need. Note that I used a 27" screen so you might want to adjust when using a smaller one such as a laptop screen.