While our book is a novel, we also wanted it to be a reference book. Our first piece of instruction arrives in chapter 3. Our user experience designer is learning how to run Usability Tests from reading the blog of Angela, our coach. The blog introduces the three types of tests: Open, Closed, and Guided. Everyone and their project manager knows what a “guided” test is. It’s a bug test. If you’ve ever done Acceptance Testing, then you also know how to do Closed tests. Few people, however, ever learn about the Open test. And the Open test is the gateway to people-centered thinking.

The team is chasing down a problem where there was a new release, and now users are leaving the app in droves. They must figure out why! For clarity, we wanted to remove all excuses. There are no major bugs preventing the app from working. All of the automated tests are passing with flying colors. Even the User Acceptance Testing is reaching a dead end. 90% of organizations can’t see the real problem because of the bugs and the failing User Acceptance Tests. This is because 90% of testing is designed to identify what we don’t already know.

The biggest dangers in any project are the unknown unknowns, the things that we don’t yet know that we don’t know. The world of the unknowns is the perfect situation for an Open Test! It gets us out of our own heads and into the heads of our users, but… There are very important, specific rules that we MUST follow to run an open test, rules that prevent our own biases from taking over. We introduce them in this chapter, and return to them in chapter 9. 

If you like the excerpt below, check out the book to learn more. And if you’d like to learn more about Usability Testing for your organization, contact us.

An excerpt from "Shift: from Product to People", Chapter 3, "Every Which Way You Can":

On the day of the tests, Clyde got up early and readied the screen recording software. He also decided to include the design practice of A/B Testing. Test ‘A’ would include the previous, working release before the decline in user adoption as the control group. Test ‘B’ would use the newest release. Don was able to reach out to five potential customers to try out the new features live, and they all agreed to be recorded. Two of them had accounts already, but three of them did not.

Usability Test #1: New User, Old Version

First up for the day was one of the new users. Clyde had him test the old version of the software first. After reconfirming his consent, Clyde started the recording. The test subject had to get his personal assistant to come set up the screen sharing, but after that, he was ready to go. Clyde’s only prompt was, “This is an app that allows you to transfer large sums of money to other people. For testing purposes, you can send a transfer to yourself and that will tell the system that this is a fake transfer.”

The man expressed an immediate dislike of having to create his Breeze account via his existing bank account, “What do you mean I can’t create an account for your app in your app?” Clyde just nodded and kept his mouth shut, taking notes. The test subject called his personal assistant again, this time to bring him the bank password. The assistant walked in and handed the man a sticky note. He typed the password into the computer.

As Clyde had seen many times before, he struggled with finding the link to create his Breeze account on the partner website, but unlike what Clyde had seen before, the guy actually just got up from the chair and stepped out of view. The assistant then sat down and proceeded to create the account for him, and then navigate back to the Breeze website. Clyde made several notes and took note of the time. The entire account creation process had taken 4 minutes and 26 seconds.

From that point on it really was a breeze. The first test user quickly found the button for creating a new transfer. When prompted to enter the email address of the user to transfer the money to, he entered his own email address, followed the legal prompts (scrolling through them quickly without reading), and finished the process. Clyde checked his watch again. The entire transfer was done in 1 minute and 30 seconds.

Usability Test #2: Existing User, Old Version

The second test was an existing user. Since she had already created an account, she had no problem logging in. Just like the first tester, though, she called her assistant to bring her the list of passwords for her accounts. Once logged in, though, she clicked the button to create a new transfer, entered her own email address as the destination, read the legal prompts quickly, and finished the process in just over 1 minute.

Usability Test #3: Existing User, New Version

Next up was the other existing user. Clyde sent him a link to the new version on their test server. Upon logging in he immediately said, “Oh, this is new. I don’t need to log in through my bank anymore? Ok. Whatever.” He went through the steps to create his new account name and set his Breeze password. The total time was 1 minute and 39 seconds. After that, he sent the test transfer in 1 minute and 17 seconds.

Clyde’s hopes were starting to drop. He was learning things about their users, many of whom were not that tech-savvy, but he hadn’t seen anything that would be blocking adoption. These results all matched the UAT results.

Usability Test #4: New User, New Version

The first new user in Group B was running a little bit behind, and when she finally joined the video conference she was very obviously not in a good mood. “I’m sorry I’m late,” she said. “I almost canceled, but let’s go ahead and get this thing over with. What do you want me to do?” Clyde gave his usual speech describing the purpose of the test.

The woman loaded up the Breeze homepage and began the new user workflow. She easily entered her email address and chose a password and was taken to the security questions. “What’s this?” she exclaimed, obviously irritated. “You want to know the bank where I have my car lease? How should I know? That’s my accountant’s job. What am I supposed to do with this? None of these options look familiar.”

Clyde tried redirecting the question back at her the way the blog recommended, “Can you think out loud? What do you think the application wants you to do next?”

“Oh, geez,” exclaimed the woman, exasperated, her eyes rolling back in her head. “Ok. Let me look… Here’s an option that says ‘none of the above’. Since I don’t recognize any of these, I’ll select that. There. It says I’m wrong. Ok, so it is one of the others.” She tried two more answers before exclaiming, “You’re going to lock me out for too many failed attempts? I don’t have time for this. Tell Don that I’m sorry. I did what I could, but his application isn’t working.” And with that, she ended the call and was gone.

Clyde sat there for a minute dumbfounded. “Who doesn’t know what bank their car lease is with?” he thought. “They have to pay the lease every month… Except they don’t. They have people for that. Everybody so far has had an assistant.”

Usability Test #5: New User, New Version

The next person did handle his own finances, so when he was asked to identify the bank that held his current mortgage, he got it right on the first try. He got blocked, though, when the security questions asked him to identify a street address from four years ago. He chose “None of the above” and was given an error. He countered, “I did? Did I own a house on one of those streets? Can you give me a city name? I can probably figure it out from the city name. It’s hard for me to remember a house from four years ago. Do you realize how many houses I own? I flip houses. That’s one of my major businesses.”

Clyde said nothing and just replied with an, “Mm-hmm. Say more, please.”

After searching his computer for about a minute, the test subject gave up looking. “Why do I have to do this anyway? Why can’t you just use my phone to authenticate? It’s got face recognition, fingerprints, passwords, lock codes. I mean, for Pete’s sake, if it can’t be trusted to authenticate that I am who I say I am, what’s the point of all this crap?” It sounded like a great point, and Clyde added it to his notes.

The man switched to a browser and started searching for the address names in the multiple-choice answers. After searching all four address names he said, “Ok, this third one looks like it’s in an area that I recognize.” He selected it and clicked through successfully. “Finally!” Clyde checked his watch and notes that it had taken 5 minutes and about 20 seconds so far.

The next step for the new user was adding in a bank account. “Ok, next we add funds,” he said. “That makes sense… looks like a wizard… selecting my bank… and it’s asking for my account and routing number. Well, heck. I don’t know what those are off the top of my head. I’m going to stop screen sharing while I do this step.” The screen went back to showing their faces in full view. Clyde thought to himself that this was actually probably better to see the emotions on this guy’s face during that whole thing.

The test subject continued narrating, “...Ok, I found one of my bank statements. Entering the account number. Where do I find the routing number? I don’t have a checkbook with them… Let’s see. That should be on their website. Hold on. Let’s see. I remember that it wasn’t on the homepage. Let me click into my Profile… Nope, not there. Account details? No, that’s not the place. Let me try the help section. Oh, there it is! On the help page. What the heck is it doing there? That makes no sense. If you know that people need help finding it, why don’t you just make it easier to find?” Clyde counted off another 2 minutes and 55 seconds before the guy shared his screen again.

The tester initiated the screen share again, “Sorry,” the guy added apologetically, “that’s not your fault. My stupid bank likes to hide the link for looking up the routing number. It always takes me a while to find it again, but I’ve entered the account and routing numbers now. It looks like the next step is sending a transfer. Let’s see, I do owe Michelle thirty thousand. I can enter her email address in this box, but wait... I don’t really want to make her go through this process just to get the money. What was that thing you said about making a test transfer?”

“You can make a test transfer by putting your own email address in the destination box,” Clyde reiterated. He jotted down in his notes about the anti-viral nature of the process and how it discouraged this user from sharing the app.

“Perfect. I’ll do that. Ok, let me read this legalese.” The tester went silent for a good minute or so. “Ok,” he said. “Sorry. I used to practice law. I don’t click something I haven’t read.” He clicked the button and initiated the transfer. “What’s next?” he asked. “Is that it? Ok. That part was a breeze. The other parts, not so much, but that part was good. A decent app you’ve got here. Do we need to do anything else?”

“Is there anything else you would want to do?” asked Clyde.

“Maybe add a couple more accounts,” the guy said, “but not right now if that’s ok with you. That’s probably going to take a while. I’ll do that later.”

“Ok,” responded Clyde. “Thank you so much for your time! You’ve been really helpful, and it’s been super useful for me to see what you think about our app and how you navigate it.”

“I hope I did alright,” the man laughed. “Did I pass the test?”

“Definitely!” Clyde chuckled “I’m not sure we did, but you definitely did. Thanks again!”

“Ok,” he said. “No problem. Happy to help Tim out. If I have time I can maybe do more later in the week. Have a great day.”

“You too!” Clyde chimed in, and with that, the guy ended the video call. Clyde recorded the time. In total, they had been on the call for a little over 17 minutes. Clyde made a few more notes and then shut everything down.

It had been a brutal series of tests, but Clyde was elated. He had found it! He now had recorded evidence for Naomi and Tim to clearly see the reason behind the declining user base! He was excited and couldn’t wait to share his discoveries with Vera.