Saul Greenberg > courses > lecture topics in HCI > evaluating interfaces: qualitative methods

Evaluating Interfaces with Users: Qualitative Methods

Evaluating interfaces with users' involvement form the basis for usability studies. A good evaluation process means that designers will catch major problems (and successes!) early on, with lesser problems being ironed out as the interface is being refined.  Fortunately, there are many low cost evaluation methods for discovering usability problems. Most of these are based on observing people as they use your system to perform particular tasks.

An assignment on usability studies can be used to complement this section by providing students with hands-on practice in various usability study methods.

Overheads

Topics Covered

Required Readings

  1. Chapter 6: Usability testing. Nielsen, J. (1993) In Usability Engineering, p165-205, Academic Press. [1 page / side or 2 pages / side]
  2. Discussion of guidelines for user observation. Kathleen Gomoll & Anne Nicol (1990) User Observation: Guidelines for Apple Developers, January.  

In-Class Teaching tips

I have found that performing usability studies in class is an excellent use of time. I do several of them, each emphasizing slightly different methods of performing a usability study.

  1. Initial conceptual model formation plus think aloud of a paper prototype. I place a stylized picture of a Cannon fax machine on the overhead (included in the qualitative evaluation overheads) and ask them to explain the meaning of labels and controls. I then asked them to pretend they are sending a one page fax, and have them use think aloud to say what controls they are selecting and why.
  2. Think aloud on a physical artifact. I have a student volunteer do a think-aloud exercise as they try to display a slide on an overhead projector rigged with a burnt bulb---it usually takes the student 20 minutes to discover the problem and realize that the projector has a spare bulb that they can switch to. I also have them try to change the bulb, and most have difficulties figuring out how to open the projector to reach the bulb. The class, who are are taking notes, then critically analyze the design of the overhead projector (relating back to Norman's design principles of everyday things) and suggest improvements. As most recommendations are simple changes to the plastic overhead case, a "better" projector could probably be built for the exact same price. The class often wonders why the manufacturer (3M) had never bothered doing this simple exercise!
  3. Constructive interaction/co-discovery learning on an existing computer system. A pair of students are given a PC (the display is linked to a large screen so the class can see what is going on). One student is usually familiar with conventional GUI file systems, while another is a novice to it. The novice is the driver, while the 'expert' is the coach. Students are asked to do a few tasks on well known systems; one year we had people try to figure out the image placement system in Microsoft Word. While the scenario is somewhat rigged to bring out the interfaces worst features, it is surprising how much difficulty students have doing even the most basic things. I also have done this for the IBM RealPhone Prototype.

Additional Readings

  1. A Practical Guide to Usability Testing. Revised Edition. Dumas, J.S. and Redish, J.C. (1999)
  2. Usability Engineering, Nielsen, J. (1993) Academic Press.
  3. Jakob Nielsen's Alertbox, March 1, 2004: Risks of Quantitative Studies

Videos

  1. Ghostbusters (commercial video, available from any video store) Ghostbusters  has a very funny clip early on that shows an example of poor ethics. In it, two subjects are in a parapsychology experiment, with shock treatments being administered for a subject who guesses wrong on a card. The experimenter, however, is totally oblivious to how the subjects are performing, and always administers shocks to the "nerdy guy" subject (who happens to start getting it right), while encouraging the beautiful woman subject (who gets it wrong)! Ok, not a huge educational value here, but the students like it as a break from the normal routine!

Major sources used to prepare lecture material

  1. How to design usable systems. Gould, J. (1995) In Baecker Grudin Buxton and Greenberg, Readings in Human Computer Interaction: Towards the Year 2000 (2nd Edition), Morgan Kaufman. Originally written in 1988, this great article shows how user involvement and prototyping work together effectively
  2. Usability Engineering, by Jakob Nielsen, helped structure the idea of evaluating systems with a user's involvement.
  3. For qualitative evaluation, the readings in Baecker, Grudin, Buxton and Greenberg: How to design usable systems and Discussions and guidelines for user observation are particularly valuable.
  4. Dumas, J. and Redish, J. (1993) A Practical Guide to Usability Testing. Ablex.This book provides complete coverage of usability testing: