Visualizing Someplace to Eat: a Comparative Experiment of Three Interfaces for Searching

IS 271 Final Experiment Report

12/15/2000

Produced by

Sacha Pearson, Kim Garrett, and Jennifer English

Visualizing Someplace to Eat: A Comparative Experiment of Three Interfaces for Searching a Restaurant Database

IS 271:
Quantitative Research Methods for
Information Management
Fall, 2000
Instructor: Rashmi Sinha

Kim Garrett

Sacha Pearson

Jennifer English

Abstract

Motivation for the Study

Previous Research

Preparation for the Experiment

User Needs Assessment

Applying our findings to the Experiment

Experimental Method

Introduction and Hypothesis

Dependent Variables

Confounding Variables

Equipment

Tasks

Materials

Participants

Pre Experiment Procedures

Training Procedures

Pilot Study Results

Changes to the Interfaces

Changes to the Test

Changes to the Instruments

Qualitative Results

Post Interface and Made-up Task Questionnaires

General Feedback ~ Post Test Questionnaire

Quantitative Results

Testers

Results

Design Recommendations

Attribute Explorer

Dynamic Query

Proposed Implementation

Conclusions

Changes to Experiment for Future Iterations

Acknowledgements

References

Appendix

Appendix A: Problems with Visual Attribute Explorer

Appendix B: Instructions for Using the Attribute Explorer (see attached)

Appendix C: Tasks (see attached)

Appendix D: Test Key (see attached)

Appendix E: Sample Metrics Sheet (see attached)

Appendix F: Script and setup instructions (see attached)

Appendix G: Sample Pre Test Questionnaire (see attached)

Appendix H: Sample Dynamic Query Tasks and Questions (see attached)

Appendix I: Sample Attribute Explorer Tasks and Questions (see attached)

Appendix J: Sample Forms Tasks and Questions (see attached)

Appendix K: Sample Made Up Tasks and Questions (see attached)

Appendix L: Sample Post Test Questionnaire (see attached)

Appendix M: Old Instruments (see attached)

Abstract

Traditional, form-based queries do not give the user any information about what the database can offer them and how their query constraints limit the data. When user goals are relatively fluid, it is helpful to have information about how loosening constraints will affect the returned set. For instance, if spending $5 per night more on a hotel room means twice as many hotels to choose from, the user would like to have that information. It could be that using techniques which provide immediate feedback will allow the user to see how their constraints limit the data and will help them to make more informed choices in their query specifications.

In this experiment, we compared an interface that uses visualization to two forms-based interfaces, one flat and one that dynamically shows query results. Tasks had to do with choosing a set of restaurants. We wanted to find out how the interfaces compared in terms of task completion time, quality of query results and user confidence. In addition, we collected user satisfaction information.

We also wanted to determine whether textual training had any impact on success with the visualization interface.

Motivation for the Study

This experiment was conducted as part of a larger graduation project at the School of Information Management and Systems.

TraveLite is a web-based, customized travel guide publisher. It allows travelers to sort through a database of travel content and choose only what they decide they need or want. Through a series of tasks, users create a customized guide, which they can later download to a PDA or other portable format. In creating guides based on their interests and needs, travelers will have the opportunity to purchase their guide, rather than a static, bland product designed for a generalized perception of what a generic traveler in a region may need. One of the major design hurdles for the project, however, is how to support user queries over the vast amount of travel information that is available.

One of the primary tasks we will need to support in the prototype of TraveLite is building a guide online using a web-based interface. In order to build a guide, users will need to use some type of tool to sort through the large amounts of nominal and ordinal data available, and filter out the elements of interest specific to their needs. The search/filter task can be daunting given a large database of content, and the possibility for failed queries (0 hits) is high as more constraints are added to queries.

Traditional, form-based queries do not give the user any information about what the database can offer them and how their query constraints limit the data. When user goals are relatively fluid, it is helpful to have information about how loosening constraints will affect the returned set. For instance, if spending $5 per night more on a hotel room means twice as many hotels to choose from, the user would like to have this information. It could be that using techniques that provide immediate feedback will allow the user to see how their constraints limit the data and will help them to make more informed decisions regarding what to collect for their customized guides.

We expect that a visualization tool such as the Visual Attribute Explorer will enable users to interact with the database in an intuitive manner that facilitates exploring, searching and selecting sets of data based on attributes relevant to the individual's travel needs.

In order to determine the appropriateness of using the Visual Attribute Explorer, we conducted an experiment to evaluate the comparative usability of the Visual Attribute Explorer. We compared the tool to a traditional, form-based query interface, as well as a dynamic query interface. The task was choosing a set of restaurants to include in a guide.

In the final prototype, the Visual Attribute Explorer would have to be redesigned to function in a web interface. Before investing time in such a redesign, we wanted to determine whether such an interface would help or hinder users in their tasks.

In addition, since our eventual audience will be Internet users, we sought to make the experiment as close to the conditions our eventual users would face as possible. For example, the training we provided to some of the users was text based rather than provided by a live teacher.

Previous Research

As the amount of information we deal with on a daily basis increases, we need easier ways to manage and filter that information. Visualization tools are a way to represent data that takes advantage of our visual cognitive skills. Humans can recognize and understand shapes and colors much faster than we can process text. Furthermore, dynamic query and visualization tools allow the user to manipulate datasets through a graphical interface. These systems all incorporate:

• Rapid incremental and reversible operations with immediate visual feedback on each action
• Smooth graphical feedback of results
• Continual visual representation of the dataset
• Physical based interaction with the data, usually using sliders or buttons, to allow the user to form and develop queries
• Further details on demand
• A layered approach to learning that allows both naïve and experienced users to use the tool
• Eliminate the zero hits returned problem. If zero hits occur, user simply sets the results back to the previous stage [1]

A main benefit of these systems is that they enable the user to reduce the found set to a manageable size (based on desired attributes/constraints) and then allow for deeper exploration. Furthermore, the sense of actual control over the data and query process bolsters user confidence in the results of the search. [2]

Much research has been performed on visualization tools using these dynamic query principles. We will summarize a few of the most closely related below. First we will look at two systems that allow exploration of travel related information. FareBrowser is an interactive tool for finding and comparing airfares with a visual element displaying results of the query. The Restaurant Finder is a dynamic query system that allows users to explore the metadata and preview the refined data set.

In the FareBrowser study, the authors developed a tool for searching for airfares using a visual display of the results. The FareBrowser, like TraveLite, was developed with the goal of allowing users to access the system via the Internet and therefore needed to be simple enough for the average user to learn and use but also handle complex queries based on many specific constraints. The user can change constraints, including destination and flight type, to locate those flights that best match their needs. The system also enables the user to manipulate the time scale using sliders. The study compared FareBrowser with Travelocity's FareFinder, a text based search tool. The results of the experiment showed that non-technical users felt that the FareBrowser was too complex and relied too heavily on the users ability to interpret the graphs, and was thus too complicated to learn in comparison to the task. Those with a more technical background appreciated the ability to see all details in one screen and to have direct control over the data. Experienced users were also able to complete the more complicated tasks in less time with FareBrowser. [3]

In this second example, Catherine Plaisant, et al. were interested in using a dynamic query and query preview to explore very large datasets over slow network systems. The system allows the user to explore the dataset over the metadata, combining browsing and querying, until the user arrives at a more usable set of data that they can then request and explore in depth. The authors developed the Restaurant Finder, a search interface for EOSDIS (NASA's Earth Observing Data Information System), and a film database for exploring the possibilities behind this type of system. They performed a user experiment on the film database and found that users on the query preview system were twice as fast in searching for items than those using a form fill-in interface. The query preview system also performed higher on user satisfaction. [4]

Octavio Juarez evaluated the performance of visualization tools based on the task they are intended to support. The experiment was a between subjects test using two tools: one using an interactive table system and one using an interactive visualization. Both groups looked at a set of environmental data and the experiment captured two measures: time to perform the task and the quality of the results. The researcher discovered that most users only needed to look at the data on a summary level. Unless they actually needed the more complex data feedback, the visualization tool was too complex and unfamiliar to the users for exploring the data at this level. However, users that did work with the visualization tool were more efficient in completing the tasks. [5]

The HomeFinder employs dynamic query and direct manipulation with immediate feedback to enable users to rapidly explore real estate listings. The system uses sliders to change the range of the dataset based on several criteria. To submit a query, the user changes a range on the sliders and the system automatically updates the found set. The tool facilitates multiple and reversible queries, there are no errors because the user can return to a previous found set. [6] Christopher Williamson and Ben Shneiderman performed a study comparing the Dynamic HomeFinder with two other interfaces, a natural language query system and traditional paper listings. The experiment measured time to find the correct answers on a series of increasingly complex tasks and the user's subjective satisfaction in using the interface. The results of the experiment showed that the dynamic query interface provided the best search results overall and also scored highest on satisfaction. [7]

The Alphaslider is the interface element employed by all of these visualization tools to quickly and graphically move the user through a data set. In this study, the authors looked at evaluating different designs of sliders for selecting text from a list using RSVP (rapid, serial, visual presentation of text) in dynamic query systems. The experiment compared four designs in a controlled environment where they compared the interfaces by looking at time to locate an item and subject satisfaction. Each interface employed a different method of allowing the user to indicate desired granularity of a search. [8]

Comparative evaluations of interfaces using visualization seem to fall into two general categories: either the interface is compared to old technology and the interface using the visualization interface is clearly superior (as in the case of the Homefinder experiment) or they were compared to technology better suited to that task, in which case the visualization is preferred by expert users but though too complicated by novice users. In comparing three interfaces, one which does not provide pre or post query information about data density across attributes, one which provides post but not pre information, and one which provides pre and post query information, we can elicit which parts of the visualization paradigm are useful and which are overwhelming to users.

Preparation for the Experiment

User Needs Assessment

In order to determine user needs for the task of building a customized travel guide online, we held a focus group. We invited eight experienced travelers who use a guide when traveling. Six were graduate students from the UC Berkeley Computer Science Department and School of Information Management and Systems, two were professional travel writers.

All of the participants travel at least once a year to locales that are unfamiliar. All the participants use the Internet and other technical tools prior to and during travel to research, purchase and communicate. This group roughly approximates the target customer of TraveLite - web savvy, frequent travelers who are accustomed to researching and purchasing travel related information and services from the Web.

While the focus group covered a wide range of topics, we focus here on the information gathered specifically for the experiment. The most important aspect of the product to determine up front was whether users wanted to be able to filter information before they traveled. As we expected, they liked the idea of a web interface that allowed them to eliminate content they knew they would not use (i.e. hotels in a price range they could not afford, restaurants in cities they would not visit, etc.). Given that users would like to be able to choose a group of restaurants to include in their guides, we explored further to determine what sort of metadata are important to them in choosing restaurants.

At the end of the focus group session, we asked users to rate the features they had brainstormed in order of importance to them. This analysis helped us to design the tasks for the experiment which is the focus of our analysis. The results were as follows:

Dimensions:

• Price (Categories)
• Cuisine
• Location (Neighborhood)
• Food Rating
• Smoking/non
• Credit cards accepted

We also researched restaurant listings already in existence to see what attributes they allowed users to search over. From these we gleaned several more attributes for our test interfaces, including:

• Open Sunday
• Open Monday
• Meals served (breakfast, lunch, dinner, etc.)
• Noise level
• Appropriate Attire
• Service Rating
• Open after 10

Some attributes which are specific to the TraveLite product were included as well:

• Content provider (because TraveLite will aggregate content from many providers)
• When updated (to inform users about the currency of the information).

Applying our findings to the Experiment

Part of the goal in using a visualization tool to present data such as our is that users can quickly process and understand the meaning of their search in the context of the overall data.

After considering the results from this focus group session and how to apply them to choosing attributes for the experiment, we realized that there is a distinction between the metadata you will want to search over and the information you want to know about a restaurant. This will also differ depending on the destination and type of trip for which the user is planning. Therefore we realized that, in the eventual implementation, we will need to allow the user to choose which metadata to search over. This is particularly important when considering how to apportion screen space. In order to be readable, these visualizations need to be of a certain size and therefore need to be limited in the amount employed in the system.

Experimental Method

Introduction and Hypothesis

Is visualization useful for the task of selecting content to include in a customized travel guide, specifically for searching over a database of restaurant information?