Uber tech / comms / media

30.07.2011

Things to Know about Usability Problems View More Blog Posts

By Harris

by Jeff Sauro | July 26, 2011

If you collect nothing else in a usability test it should be a list of problems encountered by users.
It seems so simple yet there is a rich history of how many users you need to test, what constitutes a problem and which method to use.

A usability problem should have a name, description and often a severity rating. Problem severities can be based on a 1-5 or 1-7 scale or assigned according to some criteria like whether the problem.

prevents task completion
causes a significant delay or frustration
has a relatively minor effect on task performance
is a suggestion.See Dumas & Redish and Rubin & Chisnell for more.

You should keep track of which user encountered which problem so you can better predict sample sizes, problem discovery rates and the number of undiscovered problems. Use a grid like the following one:

	User 1	User 2	User 3	User 4
Problem 1	X	X
Problem 2	X
Problem 3	X	X	X

If you observe a problem in a usability test with a small sample size, it is more likely that this problem affects a lot of users than just a few.

There are many factors that impact how common usability problems are such as the stage of development and type of interface. In a recent analysis of several datasets I found that business to business application usability problems were about 10 times more common than those found on public facing websites. These are averages from a sample, so the specific context of an application will play a major role in how common usability problems are.

Usability Problems are found in both Heuristic Evaluations and User Testing: There is some debate on whether User Testing or Heuristic Evaluation reveal more problems or whether HE finds more false positives. When possible, conduct both. Of course, some usability problems are so obvious you don’t need to torture any users to know they need to be fixed–and these are the types you hopefully find in Heuristic Evaluations.

Users can report usability problems: Asking users to report what problems they encountered, where they were in the interface and potential solutions can reveal around half the problems as trained usability evaluators. Users aren’t substitutes for professional usability evaluators but they can be an easy and cheap option.

Five Users will reveal most of the obvious problems not all problems: Five users will encounter around 85% of problems if the problems tend to affect 31% or more of the users. Five users reveal most of the obvious issues, but not all issues—especially not the ones that tend to affect a small percentage of users (like 10% or fewer). This issue continues to be hotly debated. The controversy is not about the math, it’s about the best way to model the percent of problems detected and the percent not found.

Problem severity and frequency are likely independent: It’s difficult to separate the number of users that are impacted by a problem from a severity rating, so this will likely generate some correlation, especially when the same evaluators who find the problem rate its severity. This means that more difficult usability problems are just as likely to be encountered as cosmetic ones by users.

Different Evaluators Tend to Find Different Problems: This is due to a number of factors including different definitions of usability problems, different methods, different task scenarios and different users.

Disagreement among usability problem definitions is not unique to usability, many professions, including Radiology and Psychiatry have similar problems. Where judgment is needed and the definition of the “problem” is not specific and measurable, disagreement occur

Leave a comment | posted in mobile