Notes for Teaching Assistants Labs 4
Assignment 2: Usability study:
- Schedule of dates and labs (see lab schedule)
- Brief discussion of assignment
- Go over handout
- What they are supposed to do
- What is in the write-up
Assignment 2: Usability assignment-introduction
- Stress that many usability problems can be fixed at little or no extra costs in
production
- Stress that usability studies are a cheap, effective way of discovering design flaws
- For the assignment, briefly describe the system to be evaluated and its purpose (you
should try it out yourself)
- Mention that each group should use at least 3-4 participants the more the better.
- Another way to do it is to use a ''partner group'' as a subject base-> some tasks
will differ from what yours are, and you will be able to discuss the results together.
Group preparation
- Each group select an experimenter
- The experimenter will prepare the example tasks ahead of time (preferably by discovering
what real tasks are by interviewing system users)
- The experimenter will also prepare a short pre-test questionnaire that solicits people's
prior experience with computers and the chosen system (why is this important?) and the
user's expectations of their experience. Similarly, the experimenter will prepare a
post-test questionnaire. Make sure that they justify their choice of questions i.e., that
they think the results will give them meaningful information.
Tests
- Run each subject through pre-test questionnaire
- When all is ready, the experimenter will run the tests according to the assignment spec:
- Silent observer:
- Participant 1
- Observer & test participant not allowed to speak to each other
- Observer takes notes of the test participants' behavior, especially where breakdowns occur
- think aloud
- Participant 2
- Participant asked to talk aloud explain what they are doing/thinking elaborate on
problems/solutions
- Observer takes notes
- Constructive interaction/Co-discovery learning
- Participants 3 & 4 (or Participants 1 & 2 on a more complex task)
- Both work together on task, with one being the system driver
- Observer again takes notes
- Administer post-test questionnaire
- Interview the participant(s), using questions based on observations and questionnaire
results.
Write-up
Go over the assignment sheet specs. Some added comments for different sections are
below.
- Perspective: Pretend that you are a product evaluation team for the company that
developed the system, and that you are looking for major flaws to repair for a new
version.
- Section 4: Interpretation system strengths & weaknesses
- Identify common and important problems & strengths
- Break into categories
- Should be more than a checklist of all the problems seen!
- Try to generalize problems when necessary, although you can use examples to highlight
them
- The intent is to highlight what you think are the important observations
- Section 5: Suggested improvements
- Pick at least 5 important changes you would make
- Must be restricted to low-cost changes e.g., same hardware and interface style
- Appendix 1: Comparison of different techniques
- Assume we know what the techniques are
- Summarize your experience with each technique comparison, ease of use, information
obtained, advantages, etc
- How would you do this if you were going to do it again? What would you keep? Throw away?
- Appendix 2: Raw data
- Tell them that this MUST be included
Assignment 1 marking
- Students will likely have questions for you
- Tell students overall things you noticed when marking
assignments
- What made write-ups excellent
- What made write-ups weak
- Where marks were typically lost
- While you should answer high-level questions about
marking, defer picky questions (e.g. actual grade) until
the end of the formal lab
- I strongly suggest you ask students with complaints over
their grading to give in their assignment with a note on
it that says exactly where they feel the grading
was wrong, and exactly why they should be
re-graded.
- Encourage groups to discuss their assignment 1 reports
with you. This is the best way to give feedback!
Assignment 2 expectations/grading
System discussion
- Have people present their sample tasks, and why they
thought they were good (or bad) ones. In particular, did
they explore a common thing that users did in the system,
or an arcane part of the system? Did the task really
matter that much?
- Have people discuss their general observations and
anecdotal evidence of what happened
- Try to elicit some problems of the system, at both the
low level (particular examples) and the high level (ie
how they generalize)
- try to elicit design changes of the system that use the
same technology
- If you have time, have them design a new interface to the
system.