Notes for Teaching Assistants — Tutorial 4 

Assignment 2: Usability study:

• Introduce this assignment
  • Give a brief discussion of assignment
    • Go over handout
    • What they are supposed to do
    • What is in the write-up

Assignment 2: Usability assignment-introduction

• Stress that many usability problems can be fixed at little or no extra costs in production
• Stress that usability studies are a cheap, effective way of discovering design flaws
• For the assignment, briefly describe the system to be evaluated and its purpose (you should try it out yourself)
• Mention that each group should use at least 2 - 4 test participants.  The more the better.
• If all else fails and they have trouble finding another group tell them to form a  ''partner group'', two groups pair up and they each act as each other's test participants (although this is far from the most optimal solution)

Group preparation

• Each group selects an experimenter
• The experimenter will prepare the example tasks ahead of time (preferably by discovering what real tasks are by interviewing system users)
• The experimenter will also prepare a short pre-test questionnaire that solicits people's prior experience with computers and the chosen system (why is this important?) and the user's expectations of their experience. Similarly, the experimenter will prepare a post-test questionnaire. Make sure that they justify their choice of questions i.e., that they think the results will give them meaningful information.

Tests

• Give each test participant through pre-test questionnaire
• When all is ready, the experimenter will run the tests according to the assignment spec:
• Silent observer:
  • Participant 1
  • The observer & participant not allowed to speak to each other
  • The observer takes notes of participant's behaviour, especially where breakdowns occur

• Think aloud
  • Participant 2
  • The participant is asked to talk aloud explain what they are doing/thinking elaborate on problems/solutions
  • The observer takes notes

• Constructive interaction/co-discovery learning
  • With a new set of participants 3 & 4 (or reuse participants 1 & 2 on a more complex task)
  • Both participants work together on task, with one being the system driver
  • The observer again takes notes

• Administer the post-test questionnaire
• Interview the participant, using questions based on observations and questionnaire results.

Write-up

Go over the assignment sheet specs. Some added comments for different sections are below.

• Perspective: Pretend that you are a product evaluation team for the company that developed the system, and that you are looking for major flaws to repair for a new version. 
• Section 4: Interpretation system strengths & weaknesses
• Identify common and important problems & strengths
  • Break them into categories
  • They should be more than a checklist of all the problems seen but tell them to focus on the major problems by combining little nit-picky problems into a higher level symptom
  • Try to generalize problems when necessary, although you can use examples to highlight them
• The intent is to highlight what you think are the important observations

• Section 5: Suggested improvements
  • Pick at least 5 important changes you would make
  • Must be restricted to low-cost changes e.g., same hardware and interface style

• Appendix 1: Comparison of different techniques
  • Assume we know what the techniques are
  • Summarize your experience with each technique comparison, ease of use, information obtained, advantages, etc
  • How would you do this if you were going to do it again? What would you keep? Throw away?

• Appendix 2: Raw data
  • This MUST be included not only so that students can show that they have the ability to take good notes but also because it may be used to verify their results (i.e., did what they say happened really happen).
  • If there are pertinent things that they want you to look at then indicate that they should tell you exactly where to look (I don't expect that you will spend 10 hours looking through raw video on the off chance that you might see something interesting).

Assignment 2 expectations/grading

• Give out grading sheet
• The length should be long enough to tell the story! But probably no shorter than 8 pages, tell them to try and make it no longer than 20 pages.
• Stress that the report should be oriented toward a busy manager, it must be readable and that they must demonstrate superior analysis and communication skills in over to get an above-average grade 
• Refer to lab 3 notes for details of the write-up.

System discussion

• Have people present their sample tasks, and why they thought they were good (or bad) ones. In particular, did they explore a common thing that users did in the system, or an arcane part of the system? Did the task really matter that much?
• Have people discuss their general observations and anecdotal evidence of what happened
• Try to elicit some problems of the system, at both the low level (particular examples) and the high level (i.e., how to generalize low level symptoms into higher level problems)
• Try to elicit design changes of the system that use the same technology
• If you have time, have them design a new interface to the system.