Endeca vs. Web2 Catalog Search Usability Test

Usability Test Script: Endeca catalog

March 2006

Facilitator name: ______

Participant name: ______

Undergrad Graduate Faculty Staff Other

Note to facilitators: User should be using catalog search only. If they begin to use other tools (databases, Google, website, etc) gently return them to catalog by asking them to try finding it in the catalog. (http://catalog.lib.ncsu.edu).

Preparations

The facilitator brings to the test:

___ Usability test script (this document) and a pen to write with

___ Usability Test Video Release Form for participant to sign

___ Envelope w/ monetary compensation, contact info and copy of consent form

Before the participant arrives, the facilitator should:

___ Arrange the physical space in the Usability Research Lab so that the participant will sit comfortably in front of the computer screen; a second chair for the facilitator should be placed next to the participant

___ Verify that the Usability test compensation fund book is available for signing at the end of the session

___ Close the door to the observation room

___ Verify that the lens cap is off the webcam on top of the computer screen

___ Clear the cache and history of the browser so visited links are not visible (on Firefox this is available via Tools > Options > Privacy menu)

___ Open a Firefox browser window and go to:

http://catalog.lib.ncsu.edu

Greeting the participant

When the participant arrives, you the facilitator should:

·  Bring the participant into the testing room using the hallway entrance

·  Introduce yourself (including name and position within the library)

“Thanks again for coming today and agreeing to participate in our library catalog usability test. My name is ______, and I am a/the ______here at the library. Although it might sound awkward, I’m going to be reading from this script so that all the tests we conduct will be the same.

·  Discuss compensation:

“As we agreed previously, you will receive $15 in cash for your participation in this evaluation, which will last no more than an hour. I will give this to you at the end of the session.”

·  Explain the purpose of the testing:

“You may already know, but let me explain why we’ve asked you to come here. We’re testing the design of our library catalog so we can see what it’s like for students who are actually using it. In the library catalog, you can find books, ebooks, journals, movies, music, government documents, and other resources owned by NCSU Libraries.”

·  Describe the open-ended nature of the evaluation:

“I want to make it clear right away that we’re testing the catalog, not you. You can’t do anything wrong. Your honest feedback will be very helpful as we work to improve our catalog.”

“I will ask you to complete some specific tasks, and there are no right or wrong ways to perform them. If you are unable to complete one of the tasks, it is fine to let me know. In some cases I may stop you before you finish just to make sure that we don’t run out of time.”

·  Encourage talking—thinking out loud:

“What’s really most helpful to us is if you share your thoughts with me as you are completing the tasks, so please try to think out loud as much as you can. Let me know what you are thinking as you make decisions about how to conduct your search, and why you make those decisions. For example, if you’re confused, tell me what is confusing. The more you tell us, the more data we’ll have to help us improve the catalog. Feel free to offer criticism, and don’t worry about hurting my feelings.”

·  Mention that the usability test will be recorded and have participant sign consent form.

“While you are working, I’ll be taking notes. In addition, this session will be recorded using a software product called Morae that will capture your comments and your activity. This allows our staff to review the session in the future so that I don’t have to try to write everything down. Please sign this consent form, which indicates that you know this session will be recorded, and that the recording will not be released publicly.”

·  Ask the participant whether s/he has any questions or comment

“Do you have any questions or comments before we begin?”

A. Pre-test Questions

“We’re going to start with just a few questions in order to get a sense of your familiarity with using the library catalog, which is what you can see right here on the monitor.”

[Click mouse to start recording now…]

1.  “How frequently do you use the library’s catalog to find materials? Have you used it this semester?”

Maybe offer examples: once a semester, a few times a semester, about once a week, etc.

2.  “What do you use the library catalog for most often?” – Examples: for personal use or for class use; to find specific items or to find resources about a general topic; to find books or to find other materials like DVDs

3.  “What has your previous experience with the library catalog been like? Anything particularly positive or particularly negative you’d like to comment on?”

C. Tasks

“What you are seeing is the current version of the library catalog. You will be using this catalog interface to work through the following tasks. I will read each task to you. Please let me know when you feel you have completed the task or if you feel you will be unable to complete it. You can ask me to repeat any task or spell any terms for clarification, but I cannot offer other information about how the library catalog works or how to complete the task.”

Please circle the appropriate observed level of difficulty in completing the task. Refer to the legend below for more information.

Easy / 1st or 2nd try - no problem
Med / 1st or 2nd try - observed difficulty
Hard / 2nd or 3rd try - expressed difficulty
Fail / Failed or gave up

Task difficulty legend

  1. “Your history professor has requested you to start your research project by looking up background information in a book titled Civilizations of the ancient near east”
  1. “Please find this title in the library catalog.”
  1. “Where would you go to find this book physically?”

Task difficulty: Easy Med Hard Fail

Observations:

  1. “For your literature class, you need to read the book titled Gulliver’s Travels written by Jonathan Swift. Find the call number for one copy of this book.”

Task difficulty: Easy Med Hard Fail

Observations:

  1. “You’ve been hearing a lot about the physicist Richard Feynman, and you’d like to find out whether the library has any of the books that he has written.”

Note to facilitator: Feel free to spell the last name for the participant if asked.

  1. “What is the title of one of his books?”
  1. “Is there a copy of this book you could checkout from D.H. Hill Library?”

Task difficulty: Easy Med Hard Fail

Observations:

  1. “You have the citation for a journal article about photosynthesis, light, and plant growth. You can read the actual citation for the journal article on this sheet of paper.”

Note to facilitator: Hand separate sheet of paper with citation to participant now.

Alley, H., M. Rieger, and J.M. Affolter. "Effects of Developmental Light Level on Photosynthesis and Biomass; Production in Echinacea Laevigata, a Federally Listed Endangered; Species." Natural Areas Journal 25.2 (2005): 117-122.

  1. “Using the library catalog, can you determine if the library owns this journal?”
  1. “Do library users have access to the volume that actually contains this article (either electronically or in print)?”

Task difficulty: Easy Med Hard Fail

Observations:

  1. “Please find the titles of 2 books that have been written about Bill Gates (not books written by Bill Gates).”

Task difficulty: Easy Med Hard Fail

Observations:

  1. “Your cat is acting like he doesn’t feel well, and you are worried about him. Please find 2 books that provide information specifically on cat health or caring for cats.”

Task difficulty: Easy Med Hard Fail

Observations:

  1. “You have family who are considering a solar house. Does the library have any materials about building passive solar homes?”

Task difficulty: Easy Med Hard Fail

Observations:

  1. “Can you show me how would you find the most recently published book about nuclear energy policy in the United States?”

Note to facilitator: eBooks are fine.

Task difficulty: Easy Med Hard Fail

Observations:

  1. “Imagine you teach introductory Spanish and you want to broaden your students’ horizons by exposing them to poetry in Spanish. Find at least one audio recording of a poet reading his or her work aloud in Spanish.”

Task difficulty: Easy Med Hard Fail

Observations:

  1. “You would like to browse the recent journal literature in the field of landscape architecture. Does the Design Library have any journals about landscape architecture?”

Task difficulty: Easy Med Hard Fail

Observations:

  1. “You are working on a presentation about the history of North Carolina State University, and you’d like to include some older pictures of campus. Can you find a book in the library catalog which should have historical pictures of North Carolina State University?”

Task difficulty: Easy Med Hard Fail

Observations:

  1. “I’ve asked you to find some specific resources about topics, now we’re interested in seeing how you would use the catalog to find materials when you’re working on a research paper.”
  1. “Have you had to write a research paper for a class in the last year?”
  2. [If yes]:
  3. “Can you describe your topic to me? If you don’t have a topic or you don’t remember it well, I can give you one to use.”
  4. [If participant remembers and is comfortable with topic]

“Can you use the library catalog to find several books that you think would help you write a paper on this topic?”

  1. [If no (or participant is unsure/uncomfortable about topic)] :

i.  [Point participant toward topics listed on sheet you game him/her earlier]

  1. “Can you read through these research topics and pick one you would like to use?”
  2. “Can you use the library catalog to find several books that you think would help you write a paper on this topic?”

Task difficulty: Easy Med Hard Fail

Observations:

D. Post-test Questions

“Thanks again for helping us out today; you’ve given valuable feedback about how to make our services more effective. I’d like to ask you a couple of questions about your general impressions after using the library catalog.”

1.  “What did you think of using the library catalog? What did you like? What did you not like?”

2. “The first tasks asked you to locate items where the title or author was already known. Would you rate these tasks as difficult, average, or easy to complete? Why?”

3. “The later tasks asked you to locate items about general topics. Would you rate these tasks as difficult, average, or easy to complete? Why?”

4. “Did you notice the navigation options at the top and left of the results page? What do you think they do?”

5. “If you had three things to tell the website manager to improve this catalog, what would they be?”

[Stop recording now…]

E. Wrap up

At the end of the session, the facilitator should:

·  Thank the participant for their time.

“Thanks again for coming in today. We appreciate getting your helpful feedback.”

·  Present the compensation envelope to the participant

“This envelope contains the compensation we agreed on, your copy of the video consent form, and a business card with contact information. Feel free to email [me / Emily] if you think of any questions or comments later. You can also pass it along to anyone you think might be interested in participating in a future usability test.”

·  If using NCSU Libraries funds, ask the participant to sign the usability test compensation book

“On your way out, I need you to sign this receipt. It simply indicates that you received the compensation we previously agreed on.”

·  Escort the participant out of the Usability Research Lab. May need to help them find their way out of the building.


Journal Article Citation

Alley, H., M. Rieger, and J.M. Affolter. "Effects of Developmental Light Level on Photosynthesis and Biomass; Production in Echinacea Laevigata, a Federally Listed Endangered; Species." Natural Areas Journal 25.2 (2005): 117-122.

Sample Research Topics

Forest Fires

How do forest fires effect the environment? Describe the effects of large scale wild fires in relation to other environmental factors such as air or water pollution, emission of greenhouse gases, or weather patterns.

Gene Therapy

Summarize the ethical issues related to the use of gene therapy for patient treatment. Describe some of the moral and social issues surrounding gene therapy, such as patient rights, privacy, standards of care, or eugenics.

Women as scientists or engineers

What are some barriers that have kept women from becoming scientists or engineers in the past? What is the latest thinking on how to achieve equity for girls and women in science classrooms?