Assignment: Usability Studies

Additional materials

• Tips on questionnaire design

Handouts

Some of the projects used assignments used in past terms

• Microsoft Hotmail: A free web-based mail service
• MSN IM: An instant messenger system that provides many supporting features.
• The website for Travel Cuts: A site which lets people make on-line travel reservations.
• The website for WestJet: A site which lets people make on-line travel reservations.
• The website for the University of Calgary library
• The website for Air Canada: An airline-specific flight schedule and booking system.
• The website for The Future Shop: A web that was developed to allow people browse and buy without coming into the store.
• The University of Calgary Degree Navigator: A system that supposedly helps students at the University of Calgary plan their courses.

Introduction

How can we tell how easy or hard it will be for someone to use a computer system?  Most developers simply create the system, try it out themselves until they are satisfied with it, and then dump it onto the user audience.  The result is usually an end product that many people have problems with.

One of the easiest methods for getting to "know the user" and for evaluating the human-computer interface is through usability studies. Although usability studies come in many flavors, all of them require an observer to watch a typical user try the system performing a real task.  It is surprising how many design flaws can be detected this way!

Usability studies are becoming increasingly popular in industry.  Many modern software companies now have usability labs staffed by HCI professionals whose job it is to find usability problems in products as they are being developed.  Most labs contain all the equipment permanently in place (e.g., computers), and are augmented with additional equipment such as high-end audio/video systems, screen-capture software, one-way mirrors, and so on.

However, usability studies are meant to be extremely practical, and you can do them without these special usability labs (ala the "discount usability" approach).  The simplest studies just require you to: pull up a chair next to a typical user; to watch him or her do their work (and perhaps have them explain what they are doing as they are doing it); to jot down any noteworthy events that occurred; and to listen to the user's comments.

Quick synopsis of the assignment

• You work for the Ace Consulting Company (™), a consulting firm that specializes in evaluating interfaces.

• You and your team have been contracted to do a usability study of the system described in the attached handout.

• You are to prepare a usability report for the vice-president in charge of that system's use and redevelopment.

• This person is extremely busy (so make things clear but concise).

• Your report will describe how you went about looking for design problems, what problems you saw, and what changes you recommend.

• The VP is already familiar with the 3 observation methods (silent observer, think aloud, constructive interaction) so there is no need to explain what they are.  In one of the appendices you are supposed to explain what new insights your group gained from trying these techniques out in the usability study (i.e., what you learned about the value or weaknesses of each technique from actually running test participants through each technique beyond the points that were covered in lecture).

• Depending upon how convincing you are, the VP will use your report to authorize changes in the upcoming version.

The major steps for this assignment are summarized below.

1) Things to prepare ahead of time

Administrative

• Read the assignment description (please).

• Try system out yourselves (familiarize yourself with it but make sure not to bias participants with your own preconceptions during the test).

• Prepare some example tasks (you can use the ones that I made but you must also create several of your own).

• Decide who you will employ as participants in your study (remember the type of person that you run through the study can effect the data you get from observations), minimum 2 to 4 people (you can use your own group members or members of other groups if you are really pressed but try to get as large and diverse a group as possible).

• Decide which group members will do what job.  You need at least two people:

1. The test administrator: Makes all the introductions, describes the test to the participant, answers their questions and so on.  This is the person who runs the test and interacts with the participant.

2. The scribe: Writes down observations of what occurred during the session.  Depending upon your group size you may have more than one person fulfill this role because one person may catch events that the other person missed.

3. (Optional: The 'security' person): Prevents other people from interrupting the test session which useful if you are conducting this test in a public place like the lab on the main floor of Math Sciences.   If you only have two people in your group, then you can get a friend to do the third job.

• Set up the system e.g., for classes that are evaluating a web page set up the browser ahead of time (load the browser, clear the cache and the history, navigate away from the web site to be evaluated).

Write-up the pre-test questionnaire.

Create a short pre-test questionnaire (~10 questions) that is used to determine the person's experiences and beliefs as specifically related to the task and system.  More than likely each person will have to indicate their prior experience with the system being tested or with similar systems.  They should also list any expectations or preconceptions that they may have about the system.  The pre-test questionnaire should be given to the person before he or she has used the system.  Also, be sure to administer the usability instructions to participants, as indicated in the handout. 

Example questions about users experience level with regard to the system to be tested include:

• Never used it.
• Used it once or twice over the last few years.
• Used it ~3-7 times this year.
• Use it on a daily basis.

Example questions about user's beliefs may include:

• Will need personal instruction to get started.
• Will learn it after a bit of playing around.
• Will be able to use the simple features of the system with no problems.
• Will be able to use the advanced features of the system.

Write up the post-test questionnaire.

Good post-test questions will give you information about how participants judge the system's usability, where they think they had most problems, and so on. You may want to leave space after each question for comments, where you would encourage people to say why they answered a question a certain way.  For example, here is a question that uses a rating scale:

I found the system:	Easy to use				Hard to use.
	1	2	3	4	5
Reason for your rating:______________________________________________________________________

Questionnaires should include questions that ask about people’s satisfaction with the system both in broad terms.  E.g. one, “I would rather use the WestJet website rather than book a flight through a travel agent.” E.g. two, “I would never want to use the online WestJet system”, and with more specific ratings (Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree). 

For both questionnaires (pre and post test) there should be a reason for each question that you ask – don’t ask for information that has no bearing on the test and which you won’t be using.   Finally make sure that you read my tips on designing questionnaires!

Select a core set of typical tasks. 

Usability studies requires an observer to watch someone go through their paces with 'typical' tasks. It is your job as the experimenter to prepare a set of example tasks ahead of time that the participants will try to perform. These tasks should be realistic ones that typical users would try to do with the system!  But how do you discover what those typical tasks are?

The first way is to let people use their own real tasks. To do this, you would have to solicit participants who already use the system to be evaluated, and ask them if you could watch them go about their daily business as they use the system. This is only an option if the system being studied is one that all your test participants have used before.

The second way is to ask a random sample of people who are using the system what they typically do with it, and then generalize those as tasks to give to test participants.  More than likely this is the approach that you will have to take for this assignment.

The third way is for you to use the system, and contrive a few sample tasks through intuition. Although this will not produce a set of reliable tasks, you may not have any other choice.  (By the way, jot down any problems with the system you see as you try it. You can compare these later with the problems you notice in the actual study).

To get you started, I have enclosed a few example tasks in this year's assignment description, but you must come up with your own as well.

2) Conducting the Usability Study 

See the handout on "User Observations" for a basic description of the method.  

    Quick synopsis of the test process

• Provide the introductions (who you are, what test is about – again watch out that you don’t bias people here!)
• Administer the pre-test questionnaire (to get the background experience of participants and perhaps to find out if they have any expectations about the system, what they expect it to do etc.).
• Run the test (using one of the three approaches: Silent Observer, Think Aloud or Constructive Interaction) – the test participant gets the instructions for the tasks, these instructions must be complete enough so that the person knows exactly what they are supposed to do.  Problems that arise should not come from an unclear/incomplete task description but from the system itself.  Make sure that you run a pilot study to iron out these types of problems beforehand!  In each of the three cases the person should try to complete all of the tasks. 
• Administer the post-test questionnaire and conduct any necessary debriefing.

    Details of the test process

• Throughout the exercise: Active intervention and conceptual model formation. You can gain a sense of a person's initial conceptual model of the system by having them explain each major screen as it appears, what each major interface component does, and what they think they can do with it. This conceptual model will be formed from prior experiences with similar systems and their interpretation of the visuals on the screen. You are looking for places where the model is incorrect or undeveloped.  For example, people may not understand the meaning of labels and icons, what they are supposed to do, and how they are supposed to do it.

  You are looking for places where the model is incorrect. Start doing this as soon as they get to the main window or screen for the application.

  • Have them explain the meaning behind the different components of the introductory window/screen.
  • Have them do their task.
  • Ask them if they can now explain how some of the components actually work (ones that they couldn't explain or ones that they explained incorrectly).  You should only do this only if the person couldn't explain how all the controls worked in the previous step.
  • Redo this exercise for all major screens, but try to minimize interference with the task e.g., for a form-based window you would likely do it after they fill out the needed information but just before they press a button or other control that takes them to the next screen. 

  All your test participants should begin with this step. Using this information as a baseline, you can see how a person's conceptual model develops (correctly and incorrectly) during system use merely by asking them to re-explain the display after major dialog steps (e.g., after reading documentation or after completing a transaction) or at the end of their session.  Note that this means you are actively intervening in a person's session, for you are disrupting them in the middle of their task, and their act of explaining the screen to you may result in extra learning by them.  Thus you should use this approach carefully, and only at opportune moments (such as when a major screen is encountered).

• Run participants through the three main test cases:

  Do not run every person through every case.  The idea is to take your group of participants and have approximately a quarter of them run through each of the Silent Observer and the Think Aloud cases with the remainder of people being run using Constructive Interaction.

  a) Silent observer case (~25% of the total number of participants are run through this case)

  • You need to run one person at a time through your study using this approach  (try to repeat the Silent Observer approach with more than one person).
  • Give the person the instruction sheet with the first task on it.
  • Have the person try to complete the task (remember that you are not allowed to talk to the test participant).
  • Repeat the previous two steps for all the tasks.
  • The scribe watches the person and takes notes.

  b) Think aloud scenario (~25% of the total number of participants are run through this case)

  • You need to run one person at a time through your study using this approach  (try to repeat the Think Aloud approach with more than one person)
  • Show the person how to think aloud by example (do it yourself with a system other than the one being evaluated) and run the person through some task (not the tasks that he or she is supposed to complete!) e.g., "I'm going to try to do this task ... O.K., this is probably the menu item I should select. Hmmm ... It's not doing anything, what's wrong? Oh, I see, I have to double click it..."
  • You might have to remind the person to continue to think aloud if they forget and stop doing it but make sure that you do not otherwise interfere with the test procedure.
  • Give the person the instruction sheet with the first task on it.
  • Have the person try to complete the first task.
  • Repeat the previous two steps for all the tasks.
  • The scribe watches the person and takes notes.

  c) Constructive interaction (~50% of the total number of participants are run through this case)

  • You need to run two people at a time through the study using this approach (try to run more than one pair using this approach - you can re-run the test again with the first pair of people (that you ran through the Silent Observer and Think Aloud if you can't get anyone else - ideally you should try to get as many pairs to run through the constructive interaction cases as possible).   Note: if you do have test participants run through the study multiple times make sure that you give them different tasks to complete each time!
  • Don’t explain the procedure (the idea is that the communication that occurs between the two people is a natural dialog so don’t prompt them to talk as you did with the think aloud scenario, just tell them that they should use the system together.  They can set it up so that one can be the operator with the other can give directions - if they wish - other approaches may be possible).
  • Give each person the instruction sheet with the first task on it.
  • Have the pair try to complete the task.
  • Repeat the previous two steps for all the tasks.
  • The scribe watches the people and takes notes.

  Caveat to not interfering with your test participants: If your test participants get stuck. While the experimenter should not help the person with the task, there are a few exceptions to this rule.

  • If the person has problems getting started, record the problems that they are having and give them a hint to get going. This is O.K., because if they can't get started, they will not be able to do the tasks!
  • If the person cannot complete a particular task after a reasonable amount of time, tell them to stop and start them on the next task.  Alternatively you can give them a hint if they cannot overcome some conceptual problem necessary to trying out other parts of the system. Again, be sure to record all problems!

  Remember that getting stuck is discouraging for test participants. Try to give them an early success experience, and don't forget to remind them that they can quit at any time for any reason if they wish.   (Remember the lectures dealing with ethics).

• Post-test and debriefing

• Administer the post-test questionnaire which asks them what they thought about the system (subjective satisfaction, usability of the system, how easy or hard was it to complete each task etc.)
• Interview participants after the test – what they did think of the system (which parts were strong, which parts were weak etc. – adapt the interview to your observations as you ran the test and/or use the post-test questionnaire as a discussion tool rather than slavishly sticking to a fixed script).  As before, the observer should be taking detailed notes.

At this point, you are encouraged to repeat the experiment with your friends, classmates and so on.  The more people you observe, the better!  Just make sure that they are all volunteers.  If appropriate you can also allow people to perform open-ended tasks where they set their own goals.

3) Writing up your results

Recall that your write-up should be oriented towards a senior executive in your company that will make the major decisions on the software changes and it should be structured in the following format:

Section 1. Scenario

Give a very brief reminder to the VP about what the system is, and then explain the role of your product evaluation team. Make sure you tell him/her the point of your work!

Section 2. Methodology

Explain what you did. Assume that the VP is familiar with the usability techniques that you will employ in the study (as described in this sheet) and their purpose.  However he or she will not know how you employed these techniques.  Some information to include: the number of participants, which type of participants were assigned to each case (list relevant background information rather than providing identifying information like names) the pre-test and post-test questionnaires (don't just list each question but also indicate why you are asking that particular question), task descriptions (which tasks did you use and why they were included) and which participant (be sure to anonymously identify each one but list their relevant skills and experience) participated in which test scenario.

Section 3: Observations

Summarize your observations in this section. Use selected raw and collapsed data, paraphrasing, comments, questionnaire and interview results, etc.  It is important to present the relevant information with economy!   Your TA will not be impressed with your report if he or she has to read an exact "click-by-click" narration of each participant's session.  Again:  It is your job to summarize and point out the important highlights of your study.  However your observations should be separate from your interpretations.  This section is only a summary of the results but should not include any interpretations or opinions of why you got those results.  This should be done in the next section.

Section 4: Interpretation: System strengths and weaknesses

Identify common and important problems and strengths of the system. This should be more than a checklist of all the problems seen.  Try to generalize all the minor 'symptoms' into a higher-level description of what the major problems are with this system.   You can then provide some illustrations of the higher-level problems with lists of specific examples.

Section 5: Suggested improvements

Describe the five most important changes that you would make to the design of the system, with explanations of why you think they are important.  To do this you should refer back to your observations and the discussion on design as covered in class.

Section 6: Conclusion

Summarize what you found and list your recommendations.

Appendix 1: Comparison of the different techniques

For future usability studies, you want to tell your product team what worked well and what didn't in this usability study.  Briefly summarize your experiences with each method, contrasting them for ease of use, the richness of the information obtained, their advantages, etc.  Then recommend the methods you wish your group to use in the future.  Which was most useful?  Which was least useful?  What would you keep?  What would you throw away?  Why?  Your comments in this section should describe the additional insights that you gained about each technique from your own experiences in running people through the study rather than just reiterating what was said in lecture.

Appendix 2: Raw data

All original observations/recordings, etc. should be attached here.   Although your group is including the raw data for reference you should point out where the 'choice' bits can be found in the raw data so that your TA can quickly find and view it.

Final points

• Don’t forget lecture material that deals with test ethics!
• Remember that the questionnaires, interviews and observations provide raw data.  Your data drives your analysis (what is good or bad about the system) and how improve it (recommendations).  So, it is crucial that your data provides you with sufficient material to build your case.
• Usability studies are immensely practical: you can and should use it every time you design (or wish to select!) a user interface. Good luck, and have fun!