Helen He. CPSC681 – Topic: Results Synthesis. 11/01/07.

RESULTS SYNTHESIS

Overview of the method

Results synthesis is a participatory process, involving all participants such as developers, end users, and evaluators to interpret and analyze the raw data gathered from heuristic evaluation by the individual evaluators, into a single, unified report of concise problem descriptions, their severity and possible solutions. Results synthesis aims to overcome the limitations of heuristic evaluation by trying to discover the root causes of the problems, rather than considering only the symptoms. The final output is a single, unified report that describes the problems in a clear, concise and meaningful way allowing participants to successfully communicate to developers what the problem is, its severity and possible solutions.

How it works

Results synthesis is a collaborative interpretation process, ideally involving everyone such as evaluators, developers, end users, documentation specialists, etc. Together, the participants’ use as input the raw problem descriptions gathered from heuristic evaluation by individual evaluators. These information fragments are of limited use, as they may contain duplicates, have unclear phrasing and meaning, are too low-level or interface specific and may be indecipherable to anyone but the original evaluator. Thus, these fragments cannot effectively be used to communicate problems in the interface to developers. A solution to this is results synthesis, where the raw data fragments are interpreted and discussed, and insights into the root causes of the problem are found and courses of actions determined.

Results synthesis works best in a spatial medium that supports the process of emergence, such as a whiteboard or a table. Emergence is the idea that: “Ideas do not arise well informed. At first there are expressions of fragments of thoughts. Once there is some rough material to work with, interpretations gradually begin to emerge as they are discussed” [5]. Using this spatial medium over a period of time, extended consideration can be given to the raw data input and information is discussed, manipulated and interpreted to come up with insights into the root causes of the problems and possible remedies.

The stages in results synthesis include the following: familiarizing with the entire collection of data, an initial grouping of the data according to heuristic (duplicates may be found), an initial interpretation of the data, emergence of a new understanding by grouping and re-grouping and moving problems to new labels or new groups, and finally, a consensus reached on the best possible conceptualization and grouping of the data. The workspace allows participants to use spatial reasoning, informal groupings and free-hand annotations such as sketching, circling, drawing arrows, etc to show relationships between problems.

The final output of results synthesis is a single, unified problem report where each problem consists of a clear and concise description of what the problem is in terms of users and their tasks, description of severity, whom it affects and recommended courses of action as well as alternate solutions. This single formatted report containing all the problems is given to the developers in order to communicate the results with the goal of improving the usability of the product.

Developed by

Donald Allan Cox and Saul Greenberg.

When it is used

Results synthesis is performed after the heuristic evaluation of an interface is completed. Results synthesis takes as input the raw data from the heuristic inspection, and produces as output a single, unified picture of the interface’s problems and possible solutions. The next step after results synthesis is to communicate these results in a final report to designers and developers.

Heuristic evaluation  raw data  results synthesis  communicate results to developers

Benefits and problems of this method

Benefits:

  1. Results synthesis is a natural and intuitive process.
  2. Participatory results synthesis ideally involves everyone (developers, evaluators, documentation specialists, end users, etc) in the process. This gives the benefit of:
  3. Educating those involved about usability issues and concerns
  4. Participants get a deeper understanding about the interface and its problems, as well as a better overview of how the system should work
  5. Encourages valuable team interaction, and a sharing of knowledge and perspectives
  6. Deeper understanding of target users and their tasks
  7. Allows usability to be seen as a helpful and positive contribution to development efforts
  8. Results synthesis aims to discover the root causes of problems, overcoming the limitations of heuristic evaluation that usually provide low-level symptoms of problems, rather than a focus on deeper, high-level issues.

Problems:

  1. The quality of solutions generated as the output of results synthesis solely depends on the expertise and experience of the participants. As there is no formula for how to find the “correct” or “best” solutions, the quality of the final problem and solution report is entirely dependent on the discussions formed and the evaluators’ expertise and insight.
  2. The input to results synthesis as gathered from heuristic evaluation directly affects the output. If the raw data gathered during heuristic evaluation is insufficient or inadequate, there is less material to analyze and will be harder to come up with insightful solutions for the output.
  3. Good solutions “emerge” after a period of time. Therefore, results synthesis requires a period of time to come up with insightful solutions.

Worked example

Please see the attached “Worked Example of Results Synthesis”.

ANNOTATED BIBLIOGRAPHY

[1] Beyer, H., Holtzblatt, K. (1998). Contextual Design: Defining Customer-Centered Systems. San Francisco: Morgan Kaufmann, p. 154-162 (Affinity Diagrams). A similar method to results synthesis is Affinity diagrams. Affinity diagrams use individual notes captured during interpretation sessions as input and build in a bottom-up fashion a hierarchy of common issues and themes that are present in the input data. Unlike results synthesis, affinity diagrams are not created by starting off with a predefined set of categories or groups; rather, one note is put up and then individual notes go around it if there is a natural affinity towards it. As the diagram is grouped and regrouped over time to find the best arrangement, the use of structure, post-its and labels reveal new insights about the data. The creation of affinity diagrams should be a group activity and encourages discussion and team consensus.

[2] Cox, D. and Greenberg, S. (1998). Dealing with Heuristic Evaluation Data. In Proceedings of the UPA '98 Usability Professionals' Association Conference, Poster presentation. This short paper introduces results synthesis; a method to convert the raw information fragments as gathered from heuristic evaluation into a unified set of meaningful problem descriptions to give to developers with the goal of improving the usability of the product. Results synthesis is used after the heuristic inspection stage, and before we communicate the results to developers. Results synthesis is most successful in a spatial medium that supports emergence, allowing group members to have an extended consideration of the data and through discussion with team members, group and re-group the data, achieving a final arrangement that provides us with a new and insightful understanding of the root causes of the problems in the raw data. The output of results synthesis is a final problem report to give to developers, containing concise problem descriptions, severities and possible courses of action.

[3] Cox, D. (1998). Supporting Results Synthesis in Heuristic Evaluation. M.Sc. thesis, Department of Computer Science, University of Calgary, Calgary, Alberta, Canada. November. Ch 1, 2, 3. In Chapter 1: “Introduction”, Cox provides an overview of the goals of his thesis, which are defining and describing results synthesis, identifying requirements for supporting results synthesis, and implementing and evaluating a prototype to support results synthesis. In Chapter 2: “Heuristic Evaluation”, Cox provides a description of heuristic evaluation, what it is, and how it works, as well as how results synthesis makes use of data gathered from heuristic evaluation to produce a final meaningful output. In Chapter 3: “Illustrating Results Synthesis Activity”, Cox provides a very detailed description of the results synthesis process in a paper-based environment using a comprehensive scenario. The scenario illustrates the various steps in results synthesis, such as preparation, familiarizing, emergence, interpretation and finalizing. An observational study of a group performing results synthesis in a paper-based environment was conducted, providing important conclusions regarding the importance and benefits of results synthesis.

[4] Cox, D. and Greenberg, S. (2000). Supporting Collaborative Interpretation in Distributed Groupware. Proceedings of the ACM Conference on Computer Supported Cooperative Work, 289-298, ACM Press. This paper presents a groupware system to support collaborative interpretation in small, distributed groups. Collaborative interpretation occurs when a group of people interprets a set of information fragments and through the process of emergence, transforms these into a unified set of concise, insightful, and meaningful descriptions. In this paper, design principles for distributed groupware supporting collaborative interpretation was developed and evaluated for participants performing results synthesis using raw data fragments gathered from heuristic evaluation.

[5] Moran, T., Chiu, P. and van Melle, W. (1997). Pen based interaction techniques for organizing material on an electronic whiteboard. Proc. ACM UIST’97, 45-54. The Tivoli on LiveBoard system is pen-based whiteboard system to support informal information interaction techniques to organize and rearrange material, and manage the space in a lightweight manner. The system supports pen-based interaction and allows for freehand/freeform creation and manipulation of materials on the whiteboard. General design principles such as social perceptibility, openness, emergence and agility were supported, as well as goals for using space efficiently and how to group materials. Interaction techniques involve basic selection and editing of materials, structured borders, freeform enclosures and collapsible annotations. Evaluation of this system confirmed that these interaction techniques must be learned, but once learned, can successively support a fast-paced meeting situation.

[6] Nielsen, J. (1993). Usability Engineering. Academic Press, Ch. 5: Usability Heuristics. In Chapter 5: “Usability Heuristics”, Nielsen provides a detailed description of his ten usability heuristics (principles), supporting each principle with relevant examples and illustrations. In the last section of this chapter, an overview of how to conduct a heuristic evaluation is presented, including an explanation on how evaluator expertise affects the overall success of heuristic evaluation.

[7] Nielsen, J., and Mack, R.L. (Eds.) (1994). Usability Inspection Methods, John Wiley & Sons, New York, NY, p 25-62 (Ch 2: Heuristic Evaluation). In Chapter 2: “Heuristic Evaluation”, Nielsen first introduces a revised set of usability heuristics. Then, a detailed step-by-step description of the heuristic evaluation procedure is provided using a case study of an evaluation of a system, explaining details such as what the best number of evaluators is, how to prepare and run the evaluation session, as well as how to judge severity and reliability of problems found, and characteristics of the types of usability problems found in heuristic evaluation.