CPSC 481 Foundations of HCI Saul Greenberg, Instructor

Contents

Back to lab materials for teaching assistants

Notes for Teaching Assistants — Labs 4 

Assignment 2: Usability study:

• • schedule of dates and labs (see lab schedule)
  • brief discussion of assignment
    • go over handout
    • what they are supposed to do
    • what is in the write-up

Assignment 2: Usability assignment-introduction

• Stress that many usability problems can be fixed at little or no extra costs in production
• Stress that usability studies are a cheap, effective way of discovering design flaws
• For the assignment, briefly describe the system to be evaluated and its purpose (you should try it out yourself)
• Mention that each group should use at least 3-4 subjects: themselves (excluding experimenter), and preferably one or two others. The more the better.
• another way to do it is to use a ''partner group'' as a subject base-> some tasks will differ from what yours are, and you will be able to discuss the results together.

Group preparation

• each group select an experimenter
• the experimenter will prepare the example tasks ahead of time (preferably by discovering what real tasks are by interviewing system users)
• the experimenter will also prepare a short pre-test questionnaire that solicits people's prior experience with computers and the chosen system (why is this important?) and the user's expectations of their experience. Similarly, the experimenter will prepare a post-test questionnaire. Make sure that they justify their choice of questions i.e. that they think the results will give them meaningful information.

Tests

• run each subject through pre-test questionnaire
• when all is ready, the experimenter will run the tests according to the assignment spec:
• silent observer:
• • subject 1
  • observer & subject not allowed to speak to each other
  • observer takes notes of subjectís behavior, especially where breakdowns occur
• think aloud
• • subject 2
  • subject asked to talk aloud explain what they are doing/thinking elaborate on problems/solutions
  • observer takes notes
• constructive interaction/co-discovery learning
• • subjects 3 & 4 (or subjects 1 & 2 on a more complex task)
  • both work together on task, with one being the system driver
  • observer again takes notes
• administer post-test questionnaire
• interview the subject, using questions based on observations and questionnaire results.

Write-up

Go over the assignment sheet specs. Some added comments for different sections are below.

• Perspective: Pretend that you are a product evaluation team for the company that developed the system, and that you are looking for major flaws to repair for a new version. 
• Section 4: Interpretation system strengths & weaknesses
• • Identify common and important problems & strengths
  • • break into categories
    • should be more than a checklist of all the problems seen!
  • Try to generalize problems when necessary, although you can use examples to highlight them
  • The intent is to highlight what you think are the important observations
• Section 5: Suggested improvements
• • Pick at least 5 important changes you would make
  • Must be restricted to low-cost changes e.g., same hardware and interface style
• Appendix 1: Comparison of different techniques
• • Assume we know what the techniques are
  • Summarize your experience with each technique comparison, ease of use, information obtained, advantages, etc
  • How would you do this if you were going to do it again? What would you keep? Throw away?
• Appendix 2: Raw data
• • This MUST be included

Assignment 1 marking

• students will likely have questions for you
• tell students overall things you noticed when marking assignments
• • what made write-ups excellent
  • what made write-ups weak
  • where marks were typically lost
• while you should answer high-level questions about marking, defer picky questions (e.g. actual grade) until the end of the formal lab
• I strongly suggest you ask students with complaints over their grading to give in their assignment with a note on it that says exactly where they feel the grading was wrong, and exactly why they should be regraded.
• encourage groups to discuss their assignment 1 reports with you. This is the best way to give feedback!

Assignment 2 expectations/grading

• give out grading sheet
• go over main differences (organization and final section)
• length should be long enough to tell the story! But probably no shorter than 8 pages, no longer than 20.
• Stress:
  report should be oriented toward manager
  must be readable
  must show deep thought
• Refer to lab 3ís notes for details of writeup

System discussion

• have people present their sample tasks, and why they thought they were good (or bad) ones. In particular, did they explore a common thing that users did in the system, or an arcane part of the system? Did the task really matter that much?
• have people discuss their general observations and anecdotal evidence of what happened
• try to elicit some problems of the system, at both the low level (particular examples) and the high level (ie how they generalize)
• try to elicit design changes of the system that use the same technology
• If you have time, have them design a new interface to the system.