GSR A – 3/5/03
User Test Script
Hi User Name, my name is facilitator name, this is notetaker name, who will be taking notes and computer name, who will play the part of the computer.
We really appreciate you agreeing to help us test the interface for the CBE data mining tool. Since we’re in the first stage of designing the interface, we created a low-fi prototype because it makes it easier to make changes based on your feedback.
I will begin the test by explaining a scenario in which you’d be using this tool, and ask you to complete several tasks with the prototype. As you go through the task, computer name will act as the computer and change things on screen based on the your interaction. You can use your finger as a mouse pointer, or say what you would like to do – whichever you’re more comfortable with. To scroll, move the paper up and down, like this… To highlight something, just press on it. When something’s disabled and you click on it, we’ll inform you by making this noise…
Please remember, we’re testing the interface, not you, so if anything seems confusing, unclear, or didn’t do what you expected, please let us know. This feedback is valuable to us, and we hope to use it to make the tool easy to use.
As you go through each task, please try and “think aloud”, to let us know how you’re thinking about the task and the interface. This may seem a bit awkward at first, so I may prompt you a bit if you seem confused or surprised by anything, but it’ll probably feel more natural as we go on.
Do you have any questions before we go on?
Okay, great. Now before we begin, I have a few questions for you:
How familiar are you with the Occupant Satisfaction Survey and the data it collects?
Very
Somewhat
A Little
Not At All
How familiar are you with computers?
Very
Somewhat
A Little
Not At All
Great, thanks.
Now, I’m going to explain a scenario, followed by several tasks for you to try and complete with this interface. As I explain each task, please face me until I tell you to begin.
Scenario
You are working on the Individual Thermostat Research Project, and are interested in the effect of thermostat control on the perceived thermal comfort of building occupants. You just wrapped up the survey and want to use the data mining tool to look at the results and see if there are any interesting patterns.
Legend
(Notes to myself)
Question: User’s questions
User: User’s comments
Do you have any questions?
- User: “There seems to be 2 ways to look at the survey data. One is to browse and discover patterns and questions. The other is to have a strategy and look at specific questions. What is the goal of this tool?” (We hope to do both)
Great.
Task #1
This is the first time you’re using the system. You’ve just completed a survey on individual thermostats and would like to look at the data for the first time. Please login, select the survey you’re interested in, and find the thermal comfort satisfaction graph.
Do you have any questions?
Great. Now begin.
- Logs in
- Goes to the “Dataset” pull-down menu
- Clicks on thermostat survey
- Sees summary page
- User is confused by the ‘thermostat survey 147’ shown on the screen. He tries to figure out what it means and realizes it is the survey within the dataset
- (When there is only one survey in the dataset, we should make clear that it is still a dataset they are looking at, just one with a single survey in it)
- User looks at ‘Query’ and clicks on ‘category’ drop-down menu and selects ‘thermal comfort’
- User: “I expected to see something change on the screen but it didn’t”
- User sees the send drop down menu in ‘Query’ and reads all the questions. He selects the thermal comfort question
- User sees another drop-down menu for the answers but still no result graph (User is still looking at the summary window and hasn’t selected any category yet)
- User seems confused and doesn’t know what do to. He feels that he should look elsewhere to find the thermal comfort satisfaction graph.
- User clicks on the “single graph” button and sees an expanded view of all the buttons
- User: “I will probably click on all the buttons to see what they do”
- User clicks ‘thermal comfort’ in the category window (This is the correct way)
- He sees single question graphs displayed on the screen and realizes this is what he wants.
- User finds the thermal comfort satisfaction graph
- complete
Comments:
Task #2
Now you’d like to see the data underneath one of the graphs.
Do you have any questions?
Great. Now begin.
1.User: “I am looking for a button that says “data for graphs”
2.User browses through all the labels and buttons and does not find the “data” button
3.User scrolls down the main window to see if there are more buttons below. He doesn’t find anything.
4. User checks the ‘select’ box next to a graph
5.User sees that the ‘stats’ window updates and he studies the data
6.User: “I think I’m done”
7.complete
Comments:
- User: “The ‘select’ checkbox next to each graph is confusing. I’m not sure what it does. The data eventually showed up in the ‘stats’ window but I did not expect that. Maybe it should be a button instead of a checkbox with a label on it.”
Task #3
Now you are interested in looking to see if there is a relationship between having control over thermostats and thermal comfort satisfaction levels.
Do you have any questions?
Great. Now begin.
1.User: “I want to do a cross tab on perceived control.”
2.Looks at single question graphs and finds the question “What do you control….”
3.User checks the ‘select’ box
4.The data in the ‘stats’ window from Task #2 disappears
5.User: “No stats? That is strange.” (The data disappears because the system can not show data for 2 selected graphs. We should display a message that says ‘can not show data for multiple graphs’ instead of leaving it blank)
6.User looks at the ‘Query’ and is confused by the answer checkboxes (same question Researcher E had)
7.User: “The check boxes for answers look like selection for cross tabs” (The query is still showing the thermal comfort question from Task #1)
8.User selects top 3 satisfaction answers. (binned question)
9.User sees new graph display in the main window
10.User: “Out of all the people who are satisfied, these where what they controlled”
11. User does not understand why 2nd query box appears next to the 1st one. His instinct tells him to stay with the 1st query box.
12.User modifies the 1st query. He chooses “thermal comfort” category, “What do you control…” question and “thermostat” answer
13.User sees a graph with a single bar displayed in the window.
14.User: “I see a relationship”
15. complete
Comments:
- User found it confusing when the data in the ‘stats’ window disappeared after he checks the ‘select’ box next to the graph. He thinks the ‘select’ button is inconsistent and unclear
Task #4
Now you decide you want to export the interesting graphs and their related data for further analysis in a statistical analysis tool.
Do you have any questions?
Great. Now begin.
1.Presses “export” in Dataset window
2.User realizes that he is about to export the entire survey 174 and doesn’t want to do that, he cancels
3.User checks the ‘select’ button next to the graph (nothing happens again because more than one graph is selected. We need a ‘clear all selected’ somewhere because it is hard to keep track of all selected graphs especially since some of them are not visible on the screen)
4.User looks at the buttons
5.User clicks on “save query question”
6. A query object appears in the ‘library’
7.User doesn’t really know what the icon for the query object means (upside down funnel)
8.Clicks ‘export’ in ‘library’ and hears ‘disabled’
9.User doesn’t know what to do now
10. He notices that there is another button labeled ‘save in library’ and clicks on it
11. Sees 2 new objects in the library with icons that look like graphs (these are from the 2 selected graphs from his previous tasks)
12.Question: “What do the select boxes next to the library objects do?”
13. Question: “Why didn’t I see the stats before? Is it because I had more than one graph selected?” (this is the correct answer)
14.User checks ‘graph2’ in library and clicks on the ‘export’ button.
15.Export pop-up window appears and the user enters the file name. He notices a check box that says ‘export data with graph’ and checks it. He realizes that he is exporting both the graph and the data into the same folder.
16.Clicks “OK”
17. Complete
Comments:
- (User was able to complete task but had to make many mistakes on the way. He found his way through exploration)
- (User did not really get to the ‘library’ functions until much later. Currently our ‘library’ is our work palette and is an important part of the tool. However, people are not really using it the way we expected.)
Task #5
Now you’d like to compare this survey with other buildings in California.
Do you have any questions?
Great. Now begin.
1.User: “I haven’t seen anything that deals with multiple surveys. Maybe the dataset window will lead me to it”
2.Clicks “manage dataset” and sees “choose dataset” pop-up window
3.User: “Looks like this is what I want”
4.User wonders if he should create a new dataset that includes all the buildings in California
5.Clicks “new” and sees “new dataset” pop-up box
6.Types ‘california’ into the name field
7.Looks at ‘by Survey ID” and is confused
8.Looks at ‘by building category’
9.Finds the drop-down menu labeled ‘region’
10.Finds and selects ‘california’
11. Clicks “OK”
12.“new dataset” window closes and now he is back at the “choose dataset” window
13.User: “The ‘enable’ and ‘disable’ button doesn’t really mean anything to me. They look like buttons on an ftp interface.”
14.Clicks “OK” and pop-up window closes
15. User is back at our main interface and sees a page with 2 summaries
16.User is confused by the ‘thermostat 147’ title. “Is this only the thermostat survey”? (Again we need to clearly show that this is a dataset with one survey)
17.Question: “Does the application know not to put the current survey in the benchmark dataset? If survey 147 is from a building in California and I am comparing it with other surveys in California, then 147 should not be included in benchmark dataset. I don’t want to get false relevance because I am comparing the same survey.”
16. Complete
Comments:
General Comments
- ‘select’ checkbox next to graph is confusing. What are the actions that apply to the selected
- The fact that the library is linked to the graphs in the main window is confusing. It is also not consistent how we display the linked information. When user selects the graphs, he needs to press the ‘save in library’ button before the graphs get saved. However, he only needs to select the objects in the graph (without pressing another button) to display them in the main window.
- Why can’t the user export ‘query object’ from the library? (Task #4 step 8) Is it because there is no actual data associated with it?
- Is seems confusing to have so many ways to get to the same thing
- For cross tabs (User agrees that this is a more advanced function, although it was the first thing he said he wanted to do.)
- Expects to see pop-up window with a list of settings
- Wants to be able to fix a specific question
- Wants to be able to ‘block out’ certain data (e.g. all the data from the 2nd floor responses)
- It is important to show comparison of two graphs side by side (maybe a split screen?)
- “Are the multiple query boxes a combination of one query? ‘and/or/not’”
- User suggests applying the query at the dataset level (!!! Same as Researcher E) A query (or filter) is powerful when applied for investigation rather than to create a single graph for presentation.
- User feels that the ‘stat’ window is a good thing to have but seems a bit small to display all the data
- User wants to be able to see the comments on individual questions but they should be optional when exporting. Exporting comments create large files that they don’t need.