Guide to A2

Quickie introduction

· You are consultants preparing a usability report to a VP

· This person is busy (so make it clear but concise)

· She already knows the 3 observation methods (silent observer, think aloud, constructive interaction) so no need to explain what they are (in the Appendix you are supposed to explain what new insights you gained into the technique from completing Assignment 2).

Test procedure

1) Before test

• Read the assignment description (please)
• Try system out yourselves (familiarize yourself with it but make sure not to bias participants)
• Prepare some example tasks (make at least four of your own plus you can use as additional tasks any of my examples too).
• Decide on who will be the test participants (remember the type of person run can effect the data you get from observations), minimum 2 to 4 people (you can use group members or members but other groups but try to get as large and diverse a group as possible).
• Decide which group members will do what job. You need at least two people:

a) The person who administers the test (explains and introduces the test to the participant, answers questions – the one who runs the test and interacts with the participant).

b) The scribe – takes down notes (observations of what occurred during the session).

c) (Optional: the security person) – prevents other people from interrupting the test session – useful if you are conducting this test in a public place. If you only have a group of two then you get a friend to do the 3^rd job.

• Set up the system e.g., for classes that are evaluating a web page set up the browser ahead of time (load it up, clear the cache and history – you might also have to get the browser set up between tasks too e.g. bringing back to main homepage).

2) Test procedure

• Introduction (who you are, what test is about – watch out that you don’t bias people here!)
• Administer the pretest questionnaire (to get background experience and perhaps any expectations that they have of the system, what they expect it to do).
• Running the test (for all 3 cases) – the person gets the instructions for the tasks, must be complete enough so that the person knows exactly what they are supposed to do. (Problems that arise should not come from an unclear/incomplete task description but from the system itself). In each of the three cases the person(s) should try to complete all of the tasks. Don’t forget to try to determine the person’s conceptual model of the system at significant portions of the test (look at how it matches how things really work and note if the model changes during the test)

a) Silent observer case

• You need one person to run through this (can repeat the silent observer case with different people).
• Give the person the instruction sheet with the tasks on it
• Have the person try to complete the tasks
• The scribe watches the person and takes notes.

b) Think aloud scenario

• You need one person to run through this (can repeat the think aloud case with different people)
• Show the person how to do this by example (do it yourself) and run the person through the tasks – you might have to remind the person to continue to think aloud if they forget and stop doing it.
• Give the person the instruction sheet with the tasks on it
• Have the person try to complete the tasks
• The scribe watches the person and takes notes

c) Constructive interaction

• You need two other people to run through this (although can repeat with other pairs of people if you can get them)
• Don’t explain the procedure (the idea is that the communication that occurs between the two people is a natural dialog so don’t prompt them as the case of the silent observer just tell them that they should use the system together. They can set it up so that one can be the operator the other can give directions if they wish).
• Give the person the instruction sheet with the tasks on it
• Have the people try to complete the tasks
• The scribe watches the people and takes notes

3) After the test

• Administer the post-test questionnaire- what they though about the system (subjective satisfaction, usability of the system, how easy or hard was it to complete each task etc.)
• Interview participants after the test – what they did think of the system (which parts were strong, which parts were weak etc – adapt it to your observation as you ran the test and/or use the post-test questionnaire as a discussion tool)
• Write up: sections (see Section 3 of the main assignment (“The write up”) for more details regarding the 6 sections and 2 appendices that are required to hand in).
• Tips on the analysis of results (I’ll try talk about in lecture sometime after finishing the last set of slides on qualitative evaluations).

Final points

· Don’t forget lecture material about ethics!

• Remember that the questionnaires, interviews and observations provide raw data. Your data drives analysis observations (what is good or bad about the system) and how improve it (recommendations). So, it is crucial that your data provides you material that you need to build upon.