The University of Calgary library recently installed an electronic on-line catalog called Clavis. It is heavily used by undergraduate students from all faculties to search for books and periodicals held by the library. One of its advantages is that it can be accessed via the Web. You and your team will evaluating the Web interface to Clavis. The Vice President in charge of Clavis' use and redevelopment wants to know if undergraduate students can successfully accomplish their typical catalog search tasks. Your job is to find major flaws with Clavis, and recommend ways to fix them. If there are problems, the Vice President may authorize changes to the system in upcoming versions.
It is up to you to decided what set of typical tasks should be given to the users. The assignment sheet has a section that indicates how you can go about this. As well, the experimenter should try the system ahead of time, and come up with at least six other reasonable tasks to give to subjects, preferably more. A good task one that is likely to be used by many end-users. Tasks should also be selected to investigate different (but still heavily used) parts of the system functionality.
To get you going, I've included a few sample tasks below. Notice that they are phrased as directions that will be given to the user.
If you are reusing a browser (e.g., in your login account) clear the browser's cache before each session, as you don't want the link highlights to give the user clues of where to go. Depending on how you run subjects, you may want to have a browser (e.g., Netscape) up and ready at the University of Calgary's home page (http://www.ucalgary.ca/).
Administer the pre-test questionnaire. Questions must at least probe for people's experience with the computer they are using, the web browser they are using, library catalogues (both paper and electronic), and the current University of Calgary library catalog.You may also want to ask them if they are familiar with Boolean and/or data base search systems.
Administer the usability instructions to subjects, as indicated in the handout.
The university library has an on-line catalog accessible via the web. It is an electronic database search and retrieval system that lets you search through titles held by the library. With your preferred web browser, go to the University of Calgary home page (http://www.ucalgary.ca/). Now try and find and enter the University of Calgary's on line catalog.You will want to find the web page that will let you begin your actual search of the catalog.
Note: Don't tell the person the name of the system (Clavis). Remember, if they get stuck and are obviously having problems, jot down where the problem is and give them a hint.If they enter the wrong place (but think they are in the right place), you may want to let them proceed to the next task and see if they can find the right place.
Reason for choosing this task. Getting onto the system is a pre-requisite to using it. Although the University of Calgary's web pages and links to Clavis are technically not part of Clavis, their design is still vital to its use.
Note: the reason you are doing this is to see what initial conceptual model people have of the system, based upon their prior experiences and their interpretation of the visuals on the screen. You are looking for places where the model is incorrect. Start doing this as soon as they get to the main web screen of Clavis.
Find out if the library has the book "The User as Toolsmith". If it does, tell me if its in and exactly where I would have to go to find it (i.e., what building and where in that building).
Reason for choosing this task. This is a core task. Students often have partial information about a particular book that they need (such as its title only), and need to know if the library has it and if its in. They also have to know how to physically find the book.
You want to do more background reading for CPSC 481. Find out if the library has any recent textbooks (other than the ones we use) on the topic. Give me two textbook titles that you think would be worth getting from the stacks.
Reason for choosing this task. This is a core task. Students often have to look up information on a subject. However, catalogues may not use the same words for defining that subject, so students will have to enter their own search terms. As well, this is a different type of search request and the student will have to navigate to the correct screen.
Find out all the books written by either Saul Greenberg, or by Ian Witten. Email the description of these books to the observer (you will have to give them your email address).
Reason for choosing this task. There are two parts to this task. First, it is a Boolean 'or' search, and the student has to find out how to phrase this type of Boolean request, as well as find the screen that allows one to create this request. (Both or and and searches, as well as complex searchers combining Ors and Ands are typical of complex queries). Second, students may be performing many searches, where each search may offer some interesting things. Consequently, the student needs a way to easily capture those items of interest and save them (e.g., by email or by printing). These are both core tasks, although they are expected to be rarer than simple searches.
When complete, repeat the exercise the subject did at the beginning i.e., have them explain what each interface component does in each main screen. Has their conceptual model changed through their experiential learning? Is their model correct?