Usability Testing Script
Introduction
I’m a student from Carnegie Mellon University, working on developing a system to help City managers track performances within the city. It should allow them to build graphs and charts for performance metrics for data analysis. Now, we need your help to find out possible problems with the interface, and see what seems to work for you and what doesn’t. We have a few scenarios for you and we would like to observe how you interact with the interface in each scenario.
I would like you to think aloud as you play with the prototype. Tell me what you are thinking of as you are using the interface. For example, if I were going to think about how many windows were in my parents house, I would give you all the steps thinking through the process such as, “well starting with the front of the house, left side, you have two on the first floor of the garage, one on the second floor, then one in the half bath, etc. (Use your own Think-aloud example if you want)
We are testing the system, NOT YOU! So don’t worry about making any mistakes. As you are doing the tasks, I will talk a little as possible. . If there is something you are expecting from the interface, do say it out loud. If you have any questions, I’ll try to answer them at the end. You may stop at anytime.
Background Questions
- How often do you use computers?
- Have you used software to create graphs before?
o If so, what software?
o Do you use it to do any data analysis?
Post Questions
- How do you feel about the interface?
- Is there anything you like or dislike about the interface?
- Did you learn anything new or interesting?
- Do you understand this part of the interface (Graph Data, Compare By)?
Instructions
- Fill in the user testing results sheet respective to the set of scenarios used.
- Allow them look for the metric themselves. If they take more than 30 seconds, or give up, prompt them a little.
- Take note of which requirements they complete for each scenarios
o It doesn’t matter if they do not complete the requirements.
- Write up the UARs
o The template file provided is “/User_Testing/VB_Prototype_3/TA_Template.xls”
o Use “VB3” for the interface ID of the UAR code
o Include a brief description of the background of the user and if he/she actually uses any graph drawing software before
Testing Goals
General Graphing Options
- Do they understand the Graph Data components (Values, % change, etc)?
- Do they understand the Compare By components?
- Do they have problems with the time axis control?
Metric / Hierarchy
- Can they navigate to the required metric?
- Do they use the search function?
- Do they know even to do multi-selection?
- Can they do multi-selection with the tree structure?
- Do they use the Collapse/Expand All function?
Graphing
- Are they comfortable with the graphing behavior (eg. Titles, axis labels, the time divisions)?
*Bar
*Line
*Diff div
*Diff years
Export
*% change
*% planned
data table
*change graph options
saved graphs
Your Task
Use this new software to help review the performance and efficiency of the Department of Public Works in the following scenarios. Be sure to think aloud as you go.
Scenario 1
You were told about the Mayor’s concerns about the number of grass cuttings done in each month so far this year. He needs a line graph of the relevant information. Copy and paste the graph into the latest briefing book.
Scenario 2
Compare the performance of Division 2, 4 and 5 for the monthly percent change of the number of potholes filled in 2004. Use bar graph for better visual comparison. Send the graph as an image in an email to the Mayor. Save the graph settings so that you can re-use this in future.
Scenario 3
Review the yearly performance of the area of litter pickup for March to September of 2002, 2003 and 2004. Set up a single graph to compare the values among each year easily. Create a print-out of the graph and table similar to the following, previously created by the City of Metropolis staff.
Scenario 4
Is there any significant difference between the number of obscene graffiti incidents removed and the number of gang-related graffiti incidents removed for January to March of 2005? Put a graph of the comparison of these two sets in the latest briefing book.
Scenario 5
Compare the performance of Division 1, 2 and 3 in the monthly percent change of the number of potholes filled in 2004. Since you saved settings of a similar graph in Scenario 2, you can use that graph for this task.
Scenario 6
In the previous CitiStat meeting, there were concerns over the maintenance of Play Areas.
Investigate any interesting trends or observations in the Play Area data so far this year. Put any relevant graphs and tables in the briefing book for the next meeting.
- 3 -
User Testing Results Sheet
Scenario 1Requirements
o Correct metric (# grass cuttings)
o Time (Jan ‘05 – today)
o Line graph drawn
o Copy and paste / Others
o Search function used
o Expand all used
o Collapse all used
Notes
Scenario 2
Requirements
o Correct metric (# potholes filled)
o Time (2004)
o Compare by divisions 2, 4 and 5
o Bar graph drawn
o % monthly change selected
o Export graph as JPG image
o Save graph settings / Others
o Search function used
o Expand all used
o Collapse all used
Notes
Scenario 3
Requirements
o Correct metric (area of litter pick up)
o Time (Mar - Sep)
o Line graph drawn
o Compare by 2002, 2003 and 2004
o Print / Others
o Search function used
o Expand all used
o Collapse all used
Notes
Scenario 4
Requirements
o Two correct metrics (# of obscene & gang- related graffiti incidents)
o Time (Jan – Mar of 2005)
o Line graph drawn
o Bar graph drawn
o Export / Others
o Search function used
o Expand all used
o Collapse all used
Notes
Scenario 5
Requirements
o Correct metric (# potholes filled)
o Time (2004)
o Retrieve saved graph settings
o Bar graph drawn
o Compare by divisions 1, 2 and 3
o % monthly change selected
o Export graph as JPG image / Others
o Search function used
o Expand all used
o Collapse all used
Notes
Scenario 6
Requirements
o Correct category (Play Area Maintenance)
o Time (Jan ‘05 – today)
o Line graph drawn
o Bar graph drawn
o Export to Word document / Others
o Search function used
o Expand all used
o Collapse all used
Notes
User Testing Results Sheet
How often do you use computers?
o Once a week
o 2 - 4 times a week
o Everyday
o Others ______
Have you use any software to create graphs?
o No
o Yes
If yes, what software is used?
o Excel o SPSS
o Minitab o R / S+
o Others ______
If yes, do you use it for any data analysis?
Notes
Post Questions
How do you feel about the interface?
Is there anything you like or dislike about the interface?
Did you learn anything new or interesting?
Do you understand this part of the interface?
- Graph Data
- Compare By