CPSC 481: Foundations of HCI

James Tam

Assignment: Usability Studies

(Worth 12 %, due Tuesday February 28th)

Additional materials

Handouts

  • This year's system: Hotmail
  • Discussion of guidelines for user observation (This is one of the required readings that I handed out at the start of the term)
  • Grading Sheet

    • Introduction

    How can we tell how easy or hard it will be for someone to use a computer system?  Most developers simply create the system, try it out themselves until they are satisfied with it, and then dump it onto the user audience.  The result is usually an end product that many people have problems with.

    One of the easiest methods for getting to "know the user" and for evaluating the human computer interface is through usability studies. Although usability studies come in many flavors, all of them require an observer to watch a typical user try the system performing a real task.  It is surprising how many design flaws can be detected this way!

    Usability studies are becoming increasingly popular in industry.  Many modern software companies now have usability labs staffed by HCI professionals whose job it is to find usability problems in products as they are being developed.  Most labs contain all the equipment permanently in place (e.g., computers), and are augmented with additional equipment such as high-end audio/video systems, screen-capture software, one-way mirrors, and so on.

    However, usability studies are meant to be extremely practical, and you can do them without these special usability labs (ala the "discount usability" approach).  The simplest studies just require you to: pull up a chair next to a typical user; to watch him or her do their work (and perhaps have them explain what they are doing as they are doing it); to jot down any noteworthy events that occurred; and to listen to the user's comments.

    Quick synopsis of the assignment

    The major steps for this assignment are summarized below.

    1) Things to prepare ahead of time

    Administrative

    1. The test administrator: Makes all the introductions, describes the test to the participant, answers their questions and so on.  This is the person who runs the test and interacts with the participant.

    2. The scribe: Writes down observations of what occurred during the session.  Depending upon your group size you may have more than one person fulfill this role because one person may catch events that the other person missed.

    3. (Optional: The 'security' person): Prevents other people from interrupting the test session which useful if you are conducting this test in a public place like the lab on the main floor of Math Sciences.   If you only have two people in your group, then you can get a friend to do the third job.

    Write-up the pre-test questionnaire.

    Create a short pre-test questionnaire (~10 questions) that is used to determine the person's experiences and beliefs as specifically related to the task and system.  More than likely each person will have to indicate their prior experience with the system being tested or similar systems.  They should also indicate if they have any prior expectations about the system and if so what these expectations are.  The pre-test questionnaire should be given the person before he or she used the system.  Also, be sure to administer the usability instructions to participants, as indicated in the handout. 

    Example questions about users experience level with regard to the system to be tested include:

    Example questions about user's beliefs may include:

    Write up the post-test questionnaire.

    Good post-test questions will give you information about how participants judge the system's usability, where they think they had most problems, and so on. You may want to leave space after each question for comments, where you would encourage people to say why they answered a question a certain way.  For example, here is a question that uses a rating scale:

     

    I found the system: Easy to use       Hard to use.
      1 2 3 4 5
               
    Reason for your rating:____________________

    This second questionnaire should include questions that ask about people’s satisfaction with the system both in broad terms.  E.g. one, “I would rather use the WestJet website rather than book a flight through a travel agent.” E.g. two, “I would never want to use the online system”, and with more specific ratings (Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree). 

    For both questionnaires (pre and post test) there should be a reason for each question that you ask – don’t ask for information that have no bearing on the test and which you won’t be using.   Finally make sure that you read my tips on designing questionnaires!

    Select a core set of typical tasks. 

    Usability studies requires an observer to watch someone go through the paces with 'typical' tasks. It is your job as the experimenter to prepare a set of example tasks ahead of time that the participants will try to perform. These tasks should be realistic ones that typical users would try to do with the system!  But how do you discover what those typical tasks are?

    The first way is to let people use their own real tasks. To do this, you would have to solicit participants who already use the system to be evaluated, and ask them if you could watch them go about their daily business as they use the system. This is only an option if the system being studied is one that all your test participants have used before.

    The second way is to ask a random sample of people who are using the system what they typically do with it, and then generalize those as tasks to give to test participants.  More than likely this is the approach that you will have to take for this assignment.

    The third way is for you to use the system, and contrive a few sample tasks through intuition. Although this will not produce a set of reliable tasks, you may not have any other choice.  (By the way, jot down any problems with the system you see as you try it. You can compare these later with the problems you notice in the actual study).

    To get you started, I have enclosed a few example tasks in this year's assignment description, but you must come up with your own as well.

     

    2) The Usability Study 

    See the handout on "User Observations" for a basic description of the method.  

        Quick synopsis of the test process

        Details of the test process

        Post-test and debriefing

    At this point, you are encouraged to repeat the experiment with your friends, classmates and so on.  The more people you observe, the better!  Just make sure that they are all volunteers.  If appropriate you can also allow people to perform open-ended tasks where they set their own goals.

    Recall that your write-up should be oriented towards a senior executive in your company that will make the major decisions on the software changes and it should be structured in the following format:

    Section 1. Scenario

    Give a very brief reminder to the VP about what the system is, and then explain the role of your product evaluation team. Make sure you tell him/her the point of your work!

    Section 2. Methodology

    Explain what you did. Assume that the VP knows all about the usability techniques that you will employ in the study (as described in this sheet) and their purpose.  However he or she will not know how you employed these techniques.  Some information to include: the number of participants, the pre-test and post-test questionnaires (don't just list each question but also indicate why you are asking that particular question), task descriptions (which tasks did you use and why they were included) and which participant (be sure to anonymously identify each one but list their relevant skills and experience) participated in which test scenario.

    Section 3: Observations

    Summarize your observations. Use selected raw and collapsed data, paraphrasing, comments, questionnaire and interview results, etc. It is important to present as much information as possible with economy!   Your TA will not be impressed with your report if he or she has to read an exact "click-by-click" narration of each participant's session.  Again:  It is your job to summarize and point out what were the important parts.  However your observations should be separate from your interpretations.  This section is only a summary of the results but should not include any interpretations or opinions of why you got those results.  This should be done in the next section.

    Section 4: Interpretation: System strengths and weaknesses

    Identify common and important problems and strengths of the system. This should be more than a checklist of all the problems seen.  Try to generalize all the minor 'symptoms' into a higher-level description of what the major problems are with this system.   You can then provide some illustrations of the higher-level problems with lists of specific examples.

    Section 5: Suggested improvements

    Describe the five most important changes that you would make to the design of the system, with explanations of why you think they are important.  To do this you should refer back to your observations and the discussion on design as covered in class.

    Section 6: Conclusion

    Summarize what you found and list what your recommendations were.

    Appendix 1: Comparison of different techniques

    For future usability studies, you want to tell your product team what worked well and what didn't in this usability study.  Briefly summarize your experiences with each method, contrasting them for ease of use, the richness of the information obtained, their advantages, etc.  Then recommend the methods you wish your group to use in the future.  Which was most useful?  Which was least useful?  What would you keep?  What would you throw away?  Your comments in this section should describe what additional insights did your group gain about each technique from your own actual experiences in running people through the study rather than just reiterating what was said in lecture.

    Appendix 2: Raw data

    All original observations/recordings, etc. should be attached here.   Although your group is including the raw data for reference you should point out where the 'choice' bits can be found in the raw data so that your TA can quickly find and view it.

    Final points