Workshop on Evaluation: Possible Metrics to Consider for Evaluation
Workshop on Evaluation: Prioritized Metrics
Jean Scholtz, Pacific Northwest National Laboratory
Georges Grinstein, University ofMassachusetts Lowell
Catherine Plaisant, University ofMaryland
During the 2007 InfoVis workshop on Metrics for Evaluation of Visual Analytics the participants discussed a large number of metrics. We decided to generate a list of high priority metrics. This was done by looking at categories of metrics and voting to determine high priority metrics in any given category. The following are the categories used and the metrics considered in these categories. A short description of the category is given to set the context for the metrics. In some cases the category has been broken down into subcategories.
Analysis
Metrics in this category measure how helpful a tool was to the analyst during the analytic process. All of these metrics will be based on an analyst using the tool and could possibly be compared to a similar task without the tool. Acceptable/non-acceptable ratings by the analyst could also be collected.
-Was the analyst able to explore a comprehensive set of hypotheses?
-How many hypotheses was the analyst able to generate?
-How many hypotheses was the analyst able to eliminate?
-How many red herring hypotheses did the analyst follow?
-How many red herring hypotheses was the analyst able to eliminate?
-Was the analyst able to track multiple hypotheses?
Collaboration
Metrics in this category measure the collaboration capabilities of the tool.
-Does the tool provide multiple view points to facilitate collaboration between different analysts with different expertise?
Visualization
As we are interested in visual analytics, it is also important that we have metrics for the visualization components of the tools. There are three subcategories that are a high priority for evaluations: data types, usability, and utility.
Data types
-What kinds and amounts of data can be considered by the tool?
Visualization - usability
In particular we are interested in interactive visualizations and the usability of these as measured by the common effectiveness, efficiency, and user satisfaction. However, the tasks used for measuring usability should be established within the context of the user’s work/goals.
-Effectiveness, efficiency, user satisfaction BUT in context of user’s work/goals
Visualization – utility
Utility measures the usefulness of the visualizations for the different analytic tasks that users are asked to perform.
-Flexibility/ adaptability of visualizations
Work Practices
Under work practices we are interested in metrics that are relevant to the productivity of the analyst. Many analysts read some sort of textual documents or messages and decide which of those are relevant for their analysis. For these analysts, a possible productivity metric would be:
-Document decisions per hour
Quality
It is essential that tools facilitate analysts in producing high quality work. The end goal of most analysis is a report. Analytic reports can be rated for quality; however quality can be further broken down in accuracy, objectivity, usability (for the customer or decision maker), and relevance of the report to the question asked by the customer.
-Report quality
- Accuracy
- Objectivity
- Usability
- Relevance
Insertion
Analysts need a number of tools and have established work practices. Tools that are easy to insert into the current work flow will be more readily used by analysts.
-Ease of integration of tools into current work practices
Overall Usability of the tool
-Number of steps needed to accomplish basic tasks
Training
-Learnability
- Time/ amount of training neededto learn basic functionality
- Time/ amount of training needed to learn advanced functionality