Analysis-Constructed Shared Mental Model Methodology: Using Concept Maps as Data for the Measurement of Shared Understanding in Teams
International Workshop and Mini-conference onExtending Cognitive Load Theory and Instructional Design to the Development of Expert Performance
August 29 and 30, 2005
Open University of the Netherlands
Tristan E. Johnson
Learning Systems Institute
Florida State University
Tallahassee, FL
Johnson, T.E. (2005). Analysis-Constructed Shared Mental Model Methodology: Using Concept Maps as Data for the Measurement of Shared Understanding in Teams
1
Introduction
Everywhere we look groups or teams of people discuss, manage, and solve difficult problems and issues. Teams and teamwork are a critical and functional part of our society, especially when dealing with complex or ill-structured situations and decision-making tasks not easily addressed by a single individual (Cooke, Salas, Kiekel, & Bell, in press; Stout, Cannon-Bowers, Salas, & Milanovich, 1999). Taking a definition of teams from Salas, Dickinson, Converse, and Tannenbaum (1992), teams are “a distinguishable set of two or more people who interact, dynamically, interdependently, and adaptively toward a common and valued goal/objective/mission, who have each been assigned specific roles or functions to perform, and who have a limited life-span of membership” (p. 4).
Teams or team members utilized for repetitions of the same or similar tasks may differ. Team task performance processes for the same team may vary over time. Team products and outcomes may not be concrete or accurately measurable. Additionally, teams often have members who work on different portions of the overall task with the knowledge necessary for the completion of the task being distributed among various team members. In other teams, team members work with more knowledge about the team task and/or the task process being shared by all members of the team. Shared understanding, or team cognition, has been linked to team performance (Stout, Cannon-Bowers, & Salas, 1996).
Recent studies have shown that shared understanding among team members is important. They have also provided indicators that shared understanding in teams develops over time. Most of these empirical studies of team shared mental models have been conducted in experimental or artificial environments and with teams that were put together specifically for the purpose of the study. Examples of such teams include flight cockpit crews using computerized flight simulation scenarios (Bowers, Jentsch, Salas, & Braun, 1998; Mathieu, Heffner, Goodwin, Salas, & Cannon-Bowers, 2000) and firefighting teams responsible for dispatching appropriate equipment to fight fires as they appeared on a computerized map (Rasker, Post, & Schraagen, 2000). There is little empirical research that has investigated shared mental models utilizing teams from real-life situations (McIntyre & Salas, 1995). Thus, there is a need to study shared understanding in teams working in their natural environment.
In order to study shared mental models in teams working in an applied setting, we need strategies and methods appropriately suitable for use in this context. In a recent study, the authors utilized concept mapping as a data elicitation strategy with slower-paced, non-emergency decision-making teams in order to investigate the development or change in teams’ shared knowledge over the course of team task performance. This paper presents the analysis methodology developed to measure shared mental models of teams working in an applied setting.
Analysis Constructed Shared Mental Model (AC-SMM) Methodology
Each member of a team holds a mental model of the team task. Mental models are represented through concept maps. In this study, team members were asked to represent their understanding or mental model of the task process. Teams in this study were working on the task of revising personnel qualifications standards (PQS) books for the military. For the creation of their individually constructed mental model (ICMM), team members were provided a list of 22 concepts derived from a task analysis. Team members were told they did not have to use all of the concepts on the list. They were also told they could add concepts of their own if the additional concepts were needed to complete their concept maps. ICMMs were created by each team member before the team started working on the task, during the team task process, and immediately following completion of the task.
The Analysis Constructed Shared Mental Model (AC-SMM) methodology was designed as a qualitative analysis technique for utilizing data from team members’ ICMMs and, though analysis of the ICMM components, constructing a representation of the team’s shared mental model, the AC-SMM. Resulting AC-SMMs were compared to determine the change in team mental model as a result of the team task performance process.
The AC-SMM methodology includes three phases. First, data from each ICMM were coded and analyzed using factors of concepts, sequence, links, clusters, and important concepts. Second, a shared analysis was conducted to determine which items within each factor were shared by team members. The third phase was the construction of a team shared mental model based on the shared analysis, and supported by observation notes and an initial analysis of the team task. This paper provides a detailed description of the AC-SMM methodology. The following hypothetical ICMM (Figure 1) is used for purposes of demonstration throughout the paper.
Figure 1 Hypothetical Example: ICMM for Team X, Pre-Task Map, Team Member 1
Phase I: ICMM Analysis
The first phase in utilizing the AC-SMM methodology is to analyze the individually constructed concept maps. In order to compare and measure a degree of sharedness in ICMMs, common factors were used in analyzing individually constructed concept maps such as the number of concepts, links, and node-link-node combinations (Doyle, Radzicki, & Trees, 1998; Jonassen, Reeves, Hong, Harvey, & Peters, 1997; Novak & Gowin, 1984). Additionally, it was necessary to use criteria that were appropriate for the domain (Jonassen, et al.). Because team members of this study focused their concept maps on the process of performing their team task and not the content within the book they were revising, it was better to use causal measures (directional links, sequence of concepts, and clusters) than hierarchical measures and cross-links as suggested by Novak and Gowin (1984).
The ICMM analysis phase involved several steps. The first step was to identify the factors of concepts, sequence, links, clusters, and important concepts as used in the ICMMs. Each of these factors is defined below.
Factor 1: Concepts
Concepts in this study included terms derived from the task analysis and those added by the team members. Concepts are displayed as nodes in ICMM concept maps. Concepts may be related to people involved in the task of revising the book, steps or components of the revision process or others as indicated by how the label is used within the context of the map. Concepts included both those resulting from the task analysis conducted prior to data collection and those added by individual team members. The same coding scheme was used for all ICMMs from each data collection period (pre-, mid-, and post-task).
Each concept used by a team member was coded alphabetically and compiled into a table as shown in the example provided in Figure 2. In compiling the concept list, each coded concept was listed only once even if a team member used a concept multiple times. Note that in the table descriptions of the concepts are used along with the code for the concept. The “Y” for each item under Team Member 1 indicates that this team member included this particular concept in his/her concept map.
ConceptsTeam X, Pre-Task Map / Team Members (N=4)
1 / 2 / 3 / 4
A – Facilitator / Y
B – Analysis – Determine Level of Change / Y
C – Model Manager / Y
Figure 2 Example of Concepts from Hypothetical ICMM Coded as A, B, and C.
Factor 2: Sequence
Concepts that were sequentially ordered within the ICMM by numbers, letters, or other explicit indicators (i.e. list) were coded (Figure 3). Concepts not explicitly ordered were not coded for the sequence factor. Sequenced concepts from the hypothetical case were listed in alphabetical order.
SequenceTeam X, Pre-Task Map / Team Members (N=4)
1 / 2 / 3 / 4
J=1st / Y
K=2nd / Y
L=3rd / Y
Figure 3 Example of Sequence from Hypothetical ICMM Coded as J=1st, K=2nd, and L=3rd.
Factor 3: Links
Where concepts and connectors are the building blocks of concept maps, links provide the primary foundation of mental model representations. ICMMs are made of links that consist of nodes (concepts) and connectors (lines, unidirectional and bidirectional arrows). The analysis factor of links refers to node-connector-node combinations. A link indicates a relationship between two connected concepts. Two types of links are considered in the AC-SMM analysis methodology: explicit and implicit.
Explicit Links
Two concepts can be explicitly linked in several ways. There are two general types of explicit links: simple and complex (including bracket, branch, and open-end). In each case, the link connector may be a line, a single-headed arrow, or a double-headed arrow.
Explicit Simple Links
Explicit simple links are merely a connector (with or without arrowheads) between two concepts (Figure 4). Concepts connected by a single line are related, but the connector does not provide an indication of order, sequence, or cause/effect. These links, such as the explicitly linked relationship between concepts AF and AE shown in Figure 4, are considered non-directional links. All non-directional links are coded by listing the concepts in alphabetical order, [AE,AF]. This alphabetical coding represents both the relationship between concept AF and concept AE and the reciprocal relationship between concept AE and concept AF.
Concepts linked with arrows were coded as directional links. Single-headed arrows indicate a unidirectional relationship between the connected concepts. For example, in Figure 4, the relationship flows from concept A to concept B, but not from concept B to concept A, and is coded as [A>B].
Double-headed arrows indicate a bidirectional relationship between concepts. Bidirectional links indicate that the relationship between the connected concepts flows in either direction (Figure 4 [I>J]). With bidirectional links, the reciprocal relationship is included in the notation, as in the case of [I>J], the notation also includes the relationship of [J>I]. Like concepts, sequence, non-directional and unidirectional links, bidirectional links are only listed once and are listed in alphabetical order. However, if unidirectional and bidirectional arrows are used simultaneously between the same two concepts, all instances represented by both arrows [I>J], and [I>J] or [J>I] are coded appropriately.
Directional LinksTeam X, Pre-Task Map / Team Members (N=4)
1 / 2 / 3 / 4
[A>B] / Y
[I>J] / Y
Non-Directional Links
Team X, Pre-Task Map / Team Members (N=4)
1 / 2 / 3 / 4
[AE,AF] / Y
Figure 4 Example of Explicit Simple Links from Hypothetical ICMM Coded as Directional and Non-directional links.
Explicit Complex Links
Explicit complex links are minimally two concepts related through two or more intersecting/shared connectors that do not intersect with another concept before establishing the link. For example, in the Figure 5, concepts J and T qualify as a link. However, concepts J and V do not qualify because their connection is interrupted by concept T.
Non-Directional LinksTeam X, Pre-Task Map / Team Members (N=4)
1 / 2 / 3 / 4
[J,K] / Y
[J,L] / Y
[J,O] / Y
[J,P] / Y
[J,T] / Y
[K,L] / Y
[K,O] / Y
[K,P] / Y
[K,T] / Y
[L,O] / Y
[L,P] / Y
[L,T] / Y
[O,P] / Y
[O,T] / Y
[T,G] / Y
Figure 5 Example of Complex Link from Hypothetical ICMM
These explicit complex links also may include cases where multiple connectors intersect/share in a bracket, branch, or open-end fashion.
Explicit Complex Links Containing Bracket Connectors
Brackets indicate groupings of concepts. A bracket link is minimally a complex link containing one concept that is related to two or more concepts within a bracket. Again, because links are node-connector-node combinations, the linked concepts are individually coded and noted alphabetically. In this case, a bracket connector links to all concepts contained within the bracket. Concepts contained within the bracket are coded as individual links (Figure 6).
Bracket links involving unidirectional connectors consist of all combinations of concepts to or from the point of origin and the set of concepts contained within the bracket. Bracket links may also include bidirectional connectors (Figure 6). For the purpose of recording complex bracket links that include bidirectional connectors, bracket links consist of the single concept as the point of origin to and from the concepts contained within the bracket.
Directional LinksTeam X, Pre-Task Map / Team Members (N=4)
1 / 2 / 3 / 4
[AA>AB] / Y
[AA>AC] / Y
[AA>AD] / Y
Non-Directional Links
Team X, Pre-Task Map / Team Members (N=4)
1 / 2 / 3 / 4
[AB,AC] / Y
[AC,AD] / Y
Figure 6 Example of Explicit Bracket Link from Hypothetical ICMM
Explicit Complex Links Containing Branch Connectors
Explicit complex branch links are minimally a complex link containing one concept that is related to two or more concepts where multiple connectors intersect in a hub or cluster-like fashion (Figure 7). As complex links, branch links may include unidirectional or bidirectional connectors. The relationship between the originating single concept and each concept connected through the branching connector is recorded individually. Additionally, each concept related through the branch connector is recorded as a separate link
Directional LinksTeam X, Pre-Task Map / Team Members (N=4)
1 / 2 / 3 / 4
[N>Q] / Y
[N>R] / Y
[N>S] / Y
Non-Directional Links
Team X, Pre-Task Map / Team Members (N=4)
1 / 2 / 3 / 4
[Q,R] / Y
[Q,S] / Y
[R,S] / Y
Figure 7 Example of Explicit Branch Link from Hypothetical ICMM
Explicit Complex Links Containing Open-end Connectors
Open-end links contain connectors that end short of another concept or cluster of concepts. Explicit open-end links are complex in that they not only contain an explicit connector (non-directional, unidirectional, or bidirectional), but also require the analyst to make an implicit and logical decision about the relationship of the concepts.
In open-end links, the relationship is between the originating concept and the nearest concept or set of concepts at the end of the connector such as [U,W] and [U,X] found in Figure 8. Open-ended links are coded according to the type of connector employed.
Non-Directional LinksTeam X, Pre-Task Map / Team Members (N=4)
1 / 2 / 3 / 4
[U,W] / Y
[U,X] / Y
Figure 8 Example of Complex Link Containing Open-end Connector from Hypothetical ICMM
Implicit Links
As seen in complex open-end links, the AC-SMM methodology analysis factor of links also includes implicit relationships between concept nodes within the ICMM. Implicit relationships primarily focus on concepts that are related spatially. However, implicit links are determined not only by their spatial relationships, but also structural and logical relationships.
Spatial determination of implicit links depends on the orientation of the concepts within the context of the concept map. Implicitly related concepts are generally placed more closely together as compared to concepts surrounding them and other concepts within the map.
In terms of structural support for determining implicit links, implicitly related concepts have either a limited explicit structural support (Figure 9, [W,X], [W,Y]) or no explicit structural support (Figure 9, [G, H]). Concepts are often placed adjacent (node-node) to another without the node-connector-node relationship. In the case of open-end links, the incomplete structural support is interpreted as complete so long as there is a logical relationship between the concepts.
The logic component for determining implicit links refers to the conceptual relationship among the concepts in the identified implicit link. The conceptual relationship does not have to be complete, but it does need to have logical merit.
Non-Directional LinksTeam X, Pre-Task Map / Team Members (N=4)
1 / 2 / 3 / 4
[G,H] / Y
[U,W] / Y
[U,X] / Y
[W,X] / Y
[W,Y] / Y
Figure 9 Example of Implicit Links from Hypothetical ICMM
Like explicit links, simple and complex implicit links are recorded as node-connector-node combinations and grouped according to directionality. Links connected by simple lines are recorded separately from links involving unidirectional or bidirectional arrows. Specific notation denotes the type of links as directional or non-directional.
Factor 4: Clusters
Next, concept maps were analyzed with the specific objective of looking for components of the ICMM that contained more complex knowledge than what is represented by explicit and/or implicit simple and/or complex node-connector-node combinations.
Clusters are minimally two simple and/or intersecting/shared connectors bridging three or more concepts. Clusters can include simple or complex link types. Like implicit links, when identifying clusters, three key components must be considered: spatial, structural, and logic information.
Spatial information refers to the location of concepts within the map. Interpreting spatial component information requires the consideration of visual groupings of the concepts as they are presented in the ICMM.
Structural information component refers to any type of explicit/implicit relationship between concepts. Valid cluster structures are determined based on cluster concepts all being adjacent to each other or all cluster concepts being adjacent to a single central concept (in a hub like fashion).
The logic information component refers to the conceptual relationship among the concepts in the identified cluster. Again, the conceptual relationship does not have to be complete, but it does need to have logical merit. In identifying clusters, at least two of the three information components need to be present in order to include the cluster in subsequent analysis, and one of the information components must be the logical relationship.