Assignment: A Usability Study (12%)

 

How can we tell how easy or hard it will be for someone to use our system?  Most developers simply create the system, try it out themselves until they are satisfied with it, and then dump it onto the user audience. The result is usually an end product that many people have problems with.

One of the easiest methods for getting to "know the user" and for evaluating the human computer interface is through usability studies. Although usability studies come in many flavors, all of them require an observer to watch a typical user try the system performing a real task. It is surprising how many design flaws can be detected this way!

Usability studies are becoming increasingly popular in industry. M any modern software companies now have usability labs staffed by HCI professionals whose job it is to find usability problems in products as they are being developed. Most labs contain all the equipment permanently in place (e.g., computers), and are augmented with additional equipment such as high-end audio/video systems, screen-capture software, one-way mirrors, and so on.

However, usability studies are meant to be extremely practical, and you can do them without these special usability labs (ala the "discount usability" approach).  The simplest studies just require you: to pull up a chair next to a typical user; to watch them do their work (and perhaps have them explain what they are doing as they are doing it); to jot down any noteworthy events that occurred; and to listen to the user's comments.

Quick synopsis of the assignment

 

The major steps for this assignment are summarized below.

1) Things to prepare ahead of time

Administrative

a) The test administrator: Makes all the introductions and describes the test to the participant, answers their questions and so on.  This is the person who runs the test and interacts with the participant.

b) The scribe: Takes down notes which are observations of what occurred during the session.

c) (Optional: The security person): Prevents other people from interrupting the test session which useful if you are conducting this test in a public place like the lab in  the main floor of Math Sciences.   If you only have two people in your group then you can get a friend to do the third job.

Write-up the pre-test questionnaire.

Create a short pre-test questionnaire (~10 questions) that is used to determine the person's experiences and beliefs about the system.  It is extremely important that you ask relevant questions that helps you understand a participant's background and beliefs, as related to the task and system.  More than likely each person will have to indicate their prior experience with computers, the windowing system, and the system being tested.  They should also indicate if they have any prior expectations about the system and if so what these expectations are.

Example questions about users experience level with regard to the system to be tested include:

Example questions about users beliefs may include:

Write up the post-test questionnaire.

Good post-test questions will give you information about how participants judge the system's usability, where they think they had most problems, and so on. You may want to leave space after each question for comments, where you would encourage people to say why they answered a question a certain way.  For example, here is such a question that uses a rating scale:

I found the system:             easy to use                                                                   hard to use.

                                                 1                   2                    3                    4                   5   

Reason for your rating:____________________

There should be a reason for asking the questions that you ask – don’t ask for information that have no bearing on the test and which you won’t be using. 

Some of the typical questions that you might ask in the pre-test questionnaire if you were evaluating the usability of a web site may ask about the person’s computer experience, their Internet and browser experience, their experience with this particular web site and if this person actually uses this web site or the web sites of other airlines.

At the end of the test, administer the post-test questionnaire.  This second questionnaire should include questions that ask about people’s satisfaction with the system both in broad terms e.g., “I would rather use the WestJet airline rather than book a flight through a travel agent” e.g., “I would never want to use the online system”, and with more specific ratings (Strongly agree, agree, neither agree nor disagree, disagree, strongly disagree).  Also, be sure to administer the usability instructions to participants, as indicated in the handout. 

Finally make sure that you read my tips on using questionnaires!

Select a core set of typical tasks.

Usability studies requires an observer to watch someone go through the paces with 'typical' tasks. It is your job as the experimenter to prepare a set of example tasks ahead of time that the participants will try to perform. These tasks should be realistic ones that typical users would try to do with the system! But how do you discover what those typical tasks are?

The first way is to let people use their own real tasks. To do this, you would have to solicit participants who have a real need, and ask them if you could watch them do their tasks. This is only an option for you if the system being studied is a popular one.

The second way is to ask a random sample of people who are using the system what they typically do with it, and then generalize those as tasks to give to test participants.  More than likely this is the approach that you will have to take for this assignment.

The third way is for you to use the system, and contrive a few sample tasks through intuition. Although this will not produce a set of reliable tasks, you may not have any other choice.  (By the way, jot down any problems with the system you see as you try it. You can compare these later with the problems you notice in the actual study).

To get you started, I have enclosed a few example tasks in this year's assignment description, but you must come up with your own as well.

 

2) The Usability Study 

See the handout on "User Observations" for a basic description of the method.  

    Quick synopsis of the test process

    Details of the test process

    Post-test and debriefing

At this point, you are encouraged to repeat the experiment with your friends, classmates and so on. The more people you observe, the better!  Just make sure that they are all volunteers.  If appropriate you can also allow people to perform open-ended tasks where they set their own goals.

 

3) The write up.

Recall that your write-up should be oriented towards a senior executive in your company that will make the major decisions on the software changes. Your TA will describe details and format of the write up to you. Here is a general template for you to follow.

Section 1. Scenario
Give a very brief reminder to the VP about what the system is, and then explain the role of your product evaluation team. Make sure you tell her the point of your work!
Section 2. Methodology
Explain what you did. Assume that the VP knows what the particular usability methods are (as described in this sheet) and their purpose. Include the number of participants, the pre-test evaluation, task description, etc. You must provide a list of the tasks that you have developed, and why you included them. You must also provide the pre- and post- test questionnaires, and why you included each question.
Section 3: Observations
Summarize your observations. Use selected raw and collapsed data, paraphrasing, comments, questionnaire and interview results, etc. It is important to present as much information as possible with economy!   Your TA will not be impressed with your report if he or she has to read an exact "click-by-click" narration of each participant's session.  Again:  It is your job to summarize and point out what were the important parts.
Section 4: Interpretation: System strengths and weaknesses
Identify common and important problems and strengths of the system. This should be more than a checklist of all the problems seen.  Try to generalize all the minor "symptoms" into a higher-level description of what the major problems are with this system.   You can then provide some illustrations of the higher-level problems with lists of specific examples.
Section 5: Suggested improvements
Describe the five most important changes that you would make to the design of the system, with explanations of why you think they are so important.  To do this you should refer back to your observations and the discussion on design as covered in class.
Section 6: Conclusion
Summarize what you found and what your recommendations were.
Appendix 1: Comparison of different techniques
For future usability studies, you want to tell your product team what worked well and what didn't in this usability study. Briefly summarize your experiences with each method, contrasting them for ease of use, the richness of the information obtained, their advantages, etc. Then recommend the methods you wish your group to use in the future. Which was most useful? Which was least useful? What would you keep? What would you throw away?
Appendix 2: Raw data
All original observations/recordings, etc. should be attached here.   Although your group is including the raw data for reference you should point out where the "choice" bits can be found in the raw data so that your TA can quickly find and view it.

 

Final points