Evaluating Interfaces with Users: Qualitative Methods

This section introduces qualitative methods for evaluating interfaces with users' involvement. These form the basis for usability studies. Many HCI people (including myself) strongly believe that evaluation should occur continually through the design and implementation process. Students should remember that these and other evaluation methods are applied as the interface is being developed, rather than something that's "slapped on" at the end. A good evaluation process means that designers will catch major problems (and successes!) early on, with lesser problems being ironed out as the interface is being refined.

An assignment on usability studies can be used to complement this section by providing students with hands-on practice in various usability study methods.

Overheads

• Qualitative methods of evaluation: [Acrobat] [PowerPoint]

Topics Covered

• Evaluating interfaces with Users
  • Why bother?
  • Natural vs. experimental approaches
  • Reliability and validity concerns
  • Ethics
• Qualitative evaluation methods
  • Inspections
  • Direct observation: think aloud and co-discovery learning
  • Recording results
  • Query techniques: interviews and questionnaires
  • Continuous evaluation: user feedback and field studies

Required Readings

• (In handout for assignment 2) Kathleen Gomoll & Anne Nicol (1990) Discussion of guidelines for user observation. From User Observation: Guidelines for Apple Developers, January [Available as HTML].  

Optional Readings

• Nielsen, J. (1993) Chapter 6: Usability testing. In Usability Engineering, p165-205, Academic Press.
• Baecker, Grudin, Buxton and Greenberg Chapter 2: Design and Evaluation, p.73-91, gives an overview of the life cycle and methodologies of design and evaluation. This chapter is really an introduction to the rest of the course, as it touches upon themes that are covered later. The sections directly relevant to qualitative evaluation are: a) Design and evaluation considered together; b) Usability testing.
• Baecker, Grudin, Buxton and Greenberg How to Design Usable Systems, p.93-121, is another overview to the process of system design, and shows how evaluation fits within many parts of the design cycle.
• Baecker, Grudin, Buxton and Greenberg Methodology Matters: Doing Research in the Behavioral and Social Sciences, p.152-169, covers many fundamental issues in empirical research methodologies.
• Baecker, Grudin, Buxton and Greenberg Using Video in the BNR Usability Lab, p.182-185, discusses the value of co-discovery learning and video as a means to record observations within a product environment setting.
• Baecker, Grudin, Buxton and Greenberg Conducting and Analyzing a Contextual Interview, p.241-253, introduces and describes the process of contextual interviews, an interviewing approach that acquires information on users' work and tool use, and that can actively involve them in co-design of the system.
• Sharpe, Preece and Rogers Chapter 10: Introducing Evaluation, explains why evaluation is a necessary part of the design process as well including some interesting "war stories".
• Sharpe, Preece and Rogers Section 12.4 compares the benefits of different recording techniques used in usability studies.
• Sharpe, Preece and Rogers Section 13.1 - 13.3 describes structured vs. unstructured interviews and well as comparing different types of strategies for the design of questionairres.
• Sharpe, Preece and Rogers Section 14.1 - 14.4 talks about test procedures, test participants as well the role of ethics in testing.

Videos

Ghostbusters (commercial video, available from any video store) has a very funny clip early on that shows an example of poor ethics. In it, two test participants are in a parapsychology experiment, with shock treatments being administered for a subject who guesses wrong on a card. The experimenter, however, is totally oblivious how the participants are performing (or choose to totally ignore their performance), and always administers shocks to the "nerdy guy" participant (who happens to start getting it right), while encouraging the beautiful woman participant (who gets it wrong)! Ok, not a huge educational value here, but the students like it as a break from the normal routine!

• Ghostbusters (commercial video, available from any video store)

In-Class Teaching tips

I have found that performing usability studies in class is an excellent use of time. I do several of them, each emphasizing slightly different methods of performing a usability study.

1. Initial conceptual model formation plus think aloud of a paper prototype. I place a stylized picture of a Cannon fax machine on the overhead (included in the qualitative evaluation overheads) and ask them to explain the meaning of labels and controls. I then asked them to pretend they are sending a one page fax, and have them use think aloud to say what controls they are selecting and why.
2. Think aloud on a physical artifact. I have a student volunteer do a think-aloud exercise as they try to display a slide on an overhead projector. The class, who are taking notes, then critically analyze the design of the overhead projector and suggest improvements. As most recommendations are simple changes to the plastic overhead case, a "better" projector could probably be built for the exact same price. The class often wonders why the manufacturer (3M) had never bothered doing this simple exercise!
3. Constructive interaction/co-discovery learning on an existing computer system. A pair of students are given a PC (the display is linked to a large screen so the class can see what is going on). One student is usually familiar with conventional GUI file systems, while another is a novice to it. The novice is the driver, while the 'expert' is the coach. Students are asked to do a few basic tasks i.e., one year we used Windows 95 and had them copy files around, finding files, and so on. While the scenario is somewhat rigged to bring out the interfaces worst features, its surprising how much difficulty students have doing even the most basic things. I also have done this for the IBM RealPhone System.

Major sources used to prepare lecture material

• Gould's reading in Chapter 2 lays much of the foundation for evaluating interfaces
• Usability Engineering, by Jacob Nielsen, helped structure the idea of evaluating systems with a user's involvement.
• For qualitative evaluation, the readings How to design usable systems and Discussions and guidelines for user observation are particularly valuable.
• This book provides complete coverage: Dumas, J. and Redish, J. (1993) A Practical Guide to Usability Testing. Ablex. I