Saul Greenberg > courses > lecture topics in HCI > evaluating interfaces: Quantitative methods

Evaluating Interfaces with Users: Controlled Experiments

This section introduces quantitative methods for evaluating interfaces with users, concentrating on experimental design, controlled experimentation, and simple statistics.

An assignment on quantitative evaluation gives students practice in a controlled experiment.

Overheads

• Controlled experiments: [PDF] or [Powerpoint]
• Analysis of Variance [PDF] or [Powerpoint]

Topics Covered

• Quantitative evaluation methods
  • User performance data collection
  • Controlled experiments
  • The experimental method
  • Planning an experiment
  • The value of statistics
  • Example: T-test
  • Significance levels, types of error
  • Other statistical tests
• Analysis of variance
  • terminology
  • factorial design
  • case study

In-Class Teaching tips

In-class quantitative controlled experiment. Sometimes, there was no time for students to do the related assignment. When this is the case, I run a subject in class through a controlled experiment to give people an idea of what they have to do (their task was mouse-typing on two soft keyboards with different layouts, specified in the assignment 1). Students then run each other as subjects out of class (since the software is set up for them, it takes them only 15 minutes). They hand in the data, and we analyze and interpret it in a later class.

Additional Readings

1. Chapter 2: Design and Evaluation, p.73-91. The sections directly relevant to quantitive evaluation are c) Controlled experiments.
2. Methodology Matters: Doing Research in the Behavioural and Social Sciences, p.152-169, covers many fundamental issues in empirical research methodologies.

Videos

1. Touch Typing with a Stylus, by Xerox (1993, SGVR 88) shows a pen-based character input system. This is presented as a radically different alternative to the screen-based keyboards evaluated in the example assignment.

Major sources used to prepare lecture material

1. For quantitative evaluation, the reading in Baecker, Grudin, Buxton and Greenberg: Methodology matters provides good general background.
2. Other sources I used include general psychology statistics books, and chapters in HCI textbooks dealing with controlled experiments (e.g., Chapter 11.5.1 in Human Computer Interaction by Dix, Finlay, Abowd and Beale).
3. The research planning chart included in the slides came from an old CHI tutorial, but I've since lost track of who actually created it.