Applying User Testing to Make UX Decisions in Real Time


On a recent project, we (the project team) were having difficulty deciding on a particular method of navigation for our new app. It was for an app that would replace one currently being used by millions of customers of a large Telecom company, and there was a wide variance of opinion on which method of navigation was best. And the stakes for decision were high: the app had been through numerous redesigns and had never really lived up to user expectations, largely due to confusing navigation.

So how could we decide which method of navigation would be most understandable and engaging for users? And how could we sell our decision to skeptical stakeholders? By testing, of course.

The Contenders

After some discussion and research into highly successful apps that users love, we came up with three basic contenders. First, the iOS standard bottom bar navigation as featured in most leading iOS apps. Second, the almost universal web standard for mobile navigation, the “hamburger” menu. Third but not least, we came up with a custom made design based on a circular menu that would appear when the user tapped an icon at the bottom of the screen (that just happened to look like the client’s logo).

The Method

We built three identical prototypes, the only difference between them being the navigation paradigms described above. Users had the ability to tap through all three designs on their phones to explore the experiences. The three mobile prototypes were tested in person and remotely, using, and users were asked a short series of survey questions.

Users were selected carefully: telecom customers who had recently used their provider’s billing app (our clients’ or another), as well as front line customer service agents who had been dealing with customers using the current client app. We tested the prototypes against users’ experience with their current telecom provider’s app, as well as each of the three prototypes against each other.

By gathering qualitative data through user zoom, and qualitative feedback through user interviews with customers and front line customer service agents, we were able to learn a lot about how users would respond to each. And what’s more, we were able to come up with a pretty reliable basis on which to make our decision and sell it to our stakeholders.

The Winner


And the winner is: iOS standard navigation by a nose! All of the details are in the User Testing Results presentation. It was a very close result. It turned out that all three navigation paradigms were numerically almost identical in performance. Users could complete all of the tasks nearly all of the time with any of the three methods. Bit we also asked the qualitative NPS question “How likely are you to recommend this app to a friend or colleague?”. This is where the apps differentiated themselves. Users were more likely to recommend to iOS tab bar. Reasons why can be debated over a beer any time, but the stats don’t lie. Side note: the tab bar was a lot more economical to implement and maintain than the other two which would have required developing custom controls, which helped to nudge it in front with the project team.


Be The Interface

An experience, in the context of digital business, can be defined as the totality of visual, intellectual or other inputs that a user encounters through a digital platform or device, in relation to a product or service. Experience design, therefore, is the effort to create the optimal series of inputs and resulting user actions that achieve the desired result – an excellent user experience. In the best examples, this design effort encompasses both business and user goals in optimal balance. A good digital experience should be good business, as well.

Read More

The Case for User Testing

Who needs to user test their web site or apps? You do.

Not to presume too much, but I am willing to bet that most of the people reading this post have never actually watched a user interact with their designs or products even in a casual environment. For those of you that test and test again, my apologies. And for those of you that have tested a little – congratulations, and keep going. I bet you learned some great stuff right? But by and large, most of the UX and visual designers I talk to have not conducted or benefited from much actual user input. Read More

Why don’t architects and print designers use agile?

If you work in digital technology, in an Agile environment, you have probably heard the following statement: “Why do we need Agile? People don’t use Agile to design buildings, or cars, or airplanes?” And from the design team; “print designers don’t use Agile” or some variant of the “It’s not done this way” refrain. “Shouldn’t we follow established design paradigms (read: waterfall methods)?”

This statement is usually made by designers (as opposed to developers), and usually by those who are earlier on the “agile continuum”. And it’s a fair question, which I have heard myself ask a few times in the past – the truth is that we DON’T design cars or airplanes or large buildings this way. And traditional print design has never worked this way. But why is that? Read More

Lean UX – First Principles

As a start to my series on Lean UX, here is a presentation that I created and gave at Shaw Media to help explain Lean UX concepts and foundations to the wider Online Experience Team. The presentation is in PowerPoint format, but is rather weighty due to the graphics.

Lean UX Concepts

Make Good Worlds

You Are Not A Gadget by Jaron Lanier

Mr. Lanier’s fine book – this is a must read in my opinion

Just a quick thought here: remember that what we do is to create new spaces for people to explore, new digital worlds for people to exist in. Remember that these are imaginary spaces that we can shape in any way we choose, but that the decisions we make in this regard are not as casual as we may often think.

Consider: if you design a form that requires a person to enter the sum total of their experience and human existence into three lines (“Name, Age, Gender”), what are you saying about humanity? By asking a person to reduce themselves to three lines of bland data, are you not devaluing that person at the same time? Are you not saying that the massive complexity of a person can be (on some level) reduced to these three meagre data points. Of course, we don’t intend this, but are we inferring it? Read More

Lean UX – Why and How

“Phase 2 is the biggest lie in software development.”

That’s opening line of Josh Seiden’s 2013 book Lean UX. And if you’ve spent more than ten minutes in the web and software business, you know the truth of that statement. So often, UX designers work hard to create a fully functional, achingly cool interaction design only to have the developer, or worse, the client come along and tell you that they aren’t going to build it or pay for it in “phase 1”. The thought is that we will come back and polish things up in phase 2, but phase 2 never seems to come.

Sound familiar? It’s not anyone’s fault, really. It’s just that the software and web businesses function on tight timelines and almost non-existent budgets, so often the finer points of interface and interaction design are lost in the charge to deliver the product to market, by any means necessary. Read More