Guide to A2
Quickie introduction
·
You are consultants preparing a usability report to a
VP
·
This person is busy (so make it concise)
·
She already knows the 3 observation methods (silent observer,
think aloud, constructive interaction) so no need to explain what they are (in
the Appendix you are supposed to explain what new insights you gained into the
technique from assignment 2).
Test procedure
1) Before test
- Read
the assignment description (please)
- Try
system out yourselves (familiarize yourself with it but make sure not to
bias participants)
- Prepare
some example tasks (make at least four of your own plus you can use as
additional tasks any of my examples too).
- Decide
on who will be the test participants (remember the type of person run can
effect the data you get from observations), minimum 2 to 4 people (you can
use group members or members but other groups but try to get as large and
diverse a group as possible).
- Decide
which group members will do what job.
You need at least two people:
a) The person who administers the
test (explains and introduces the test to the participant, answers questions –
the one who runs the test and interacts with the participant).
b) The scribe – takes down notes,
which are observations of the session.
c) (Optional: the security person)
– prevents other people from interrupting the test session – useful if you are
conducting this test in a public place.
If you only have a group of two then you get a friend to do the 3rd
job.
- Set up
browser (load it up, clear the cache – you might also have to get the
browser set up between tasks too e.g. bringing back to main homepage for
the Future shop).
2) Test procedure
- Introduction
(who you are, what test is about – watch out that you don’t bias people
here!)
- Administer
the pretest questionnaire (to get background experience and perhaps any
expectations that they have of the system, what they expect it to do).
- Running
the test (for all 3 cases) – the person gets the instructions for the
tasks, must be complete enough so that the person knows exactly what they
are supposed to do. (Problems that arise should not come from an
unclear/incomplete task description but from the system itself). In each of the three cases the
person(s) should try to complete all of the tasks. Don’t forget to try to determine the
person’s conceptual model of the system at significant portions of the
test (look at how it matches how things really work and note if the model
changes during the test)
a) Silent observer case
- You
need one person to run through this (can repeat the silent observer case
with different people).
- Give
the person the instruction sheet with the tasks on it
- Have
the person try to complete the tasks
- The
scribe watches the person and takes notes.
b) Think aloud scenario
- You
need one person to run through this (can repeat the think aloud case with
different people)
- Show the
person how to do this by example (do it yourself) and run the person
through the tasks – you might have to remind the person to continue to
think aloud if they forget and stop doing it.
- Give
the person the instruction sheet with the tasks on it
- Have
the person try to complete the tasks
- The
scribe watches the person and takes notes
c) Constructive interaction
- You
need two other people to run through this (although can repeat with other
pairs of people if you can get them)
- Don’t
explain the procedure (the idea is that the communication that occurs
between the two people is a natural dialog so don’t prompt them as the
case of the silent observer just tell them that they should use the system
together. They can set it up so
that one can be the operator the other can give directions if they wish).
- Give
the person the instruction sheet with the tasks on it
- Have
the people try to complete the tasks
- The
scribe watches the people and takes notes
3) After the test
- Administer
the post-test questionnaire- what they though about the system (subjective
satisfaction, usability of the system, how easy or hard was it to complete
each task etc.)
- Interview
participants after the test – what they did think of the system (which
parts were strong, which parts were weak etc – adapt it to your
observation as you ran the test and/or use the post-test questionnaire as
a discussion tool)
- Write
up: sections (see page 4 of the assignment description for the 6 sections
and 2 appendices that are required).
- Tips
on the analysis of results (I’ll talk about in lecture next week after
finishing the last set of slides on qualitative evaluations).
Final points
·
Don’t forget ethics
lecture material!
- Remember
that the questionnaires, interviews and observations provide raw
data. Your data drives analysis
observations (what is good or bad about the system) and how improve it
(recommendations). So, it is
crucial that your data provides you material that you need to build upon.