Network maps . . .1
The influence of relational and proposition-specific processingon structural knowledge and traditional learning outcomes
by
Roy B. Clariana and Maria T. Poindexter
Presented at the annual meeting of the American Educational Research Association, April 12-16, 2004San Diego, CA.
Jonassen, Beissner, and Yacci (1993) describe a number of approaches for assessing structural knowledge. Of these, concept maps and semantic maps are most commonly used in the classroom.Though there are still a number of important unanswered questions about the role of concept maps in measuring knowledge, there is substantial evidence supporting the reliability and validity of concept maps for assessment (McClure, Sonak, & Suen, 1999; Ruiz-Primo, Schultz, Li, & Shavelson, 2000; Ruiz-Primo & Shavelson, 1996; Ruiz-Primo, Shavelson, Li, & Schultz, 2001; Wallace & Mintzes, 1990). One fundamental question is,which aspects or components of knowledge are captured by concept map scores?
The purpose of this investigation is to examine the relationship ofstructural knowledge as measured by network maps to three traditional learning outcomes, specifically, identification, terminology, and comprehension multiple-choice posttests (Dwyer, 1972).Instructional treatments were provided that encourage either proposition-specific processing or relational processing, and the effects of these lesson treatments on traditional and map scores areprovided in order to describe more clearly what aspects of structural knowledge can be captured by network maps.
Methodology
This investigation is a follow-up of a recent dissertation using the same participant population, materials, instruments, and procedures with the exception of the addition of a network mapping activity after the final multiple-choice posttest. The Methodology section is described here in brief, since detailed information on participants, instructional materials, and posttestsis available from the original investigation (Poindexter, 2003).
Participants
The participants in this investigation were undergraduate students from a north-eastern college campus (n = 23). Undergraduate students were recruited by informational e-mail from all majors at the campus including education, biology, communications, engineering, and business. Those interested in volunteering used a web-based sign-up sheet to select the time and dates that worked best for them. Volunteers were compensated with $2.00 certificates that could be redeemed at the campus restaurant.
Materials and Procedure
The1900-word print-based expository text(and posttests) used in the study were developed by Dwyer (1972) and covered the structure and function of the human cardiovascular system. Nineteen simple black-and-white line drawings were added to complement each page of the written text. Each visual had labels to describe only the structure or function of the heart that was described by the text on that page.
Three instructional treatment booklets were used. Two of these treatments were devised in order to direct reading behavior, one influenced the reader to focussimultaneouslyon multiple propositions (relational) in the text and the other influenced the reader to focus mainly on individual propositions.The third treatment was areading only control to serve as a baseline for comparison. All three treatments used the same text portion(with complementary line drawings) divided into five sections, butin the relational condition, participants were required to “unscramble” sentences (following Einstein, McDaniel, Bowers, & Stevens, 1984) in one paragraph in each of the five sections or about 20% of the total text content; while in the proposition-specific condition (following Hamilton, 1985), participants answered three or four adjunct constructed response questions (taken nearly verbatim from the text)provided at the end of each of the five sections, for a total of 17 questions covering about 20% of the total text content (no feedback was provided).
Posttest Measures
Three separate 20 item multiple-choice tests developed by Dwyer (1972) were used to measure various levels of achievement including identification, terminology, and comprehension. The identification test (ID) was designed to measure declarative knowledge, specifically the learner’s ability to identify and label the parts and positions of the human heart using visual cues. The terminology test (TERM) was designed to measure declarative knowledge of facts, terms, and definitions taken nearly verbatim from the text, but different from the lesson questions of the constructed response treatment. The comprehension test (COMP) was designed to measure a more thorough understanding of the processes of the human heart, with a specific focus on the functions of different parts of the heart (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956). Cronbach alpha reliability of the three posttests used here areID =0.88,TERM =0.84, and COMP = 0.86 (total test =0.94).
Network Map Dependent Measures
Concept maps and othersimilar visual representations of knowledge contain at least two qualitatively different but related kinds of information, actual explicit links between terms and also implicit associations among terms represented by the spatial distances between terms. Concept map scoring approaches typically focus on hierarchical relationships and on the actual links between terms (links), but the spatial distances among terms are usually not considered (associations). This investigation uses the term “network map” rather than concept map in order to show our focus on the meaningful associationsamong the terms and the posttest directions for creating a network map intentionally emphasizedproximity of terms and placedno emphasis on drawing links between terms.
Two kinds of data were derived from each network map, an association array (distances) and a proposition array (links). For example, the network map that was shown to the participants as an example is represented as both an association array and a proposition array in Figure 1.
Figure 1. The network map example used in the directions represented as a proposition link array and as an association distance array.
Specifically, S-Mapper software (Clariana, 2002) was used to establish association arrays that contained all pair-wise distances between the 26 terms. The number of pair-wise comparisons required for 26 terms is equal to (262 – 26)/2 = 325. Next, an Excel spreadsheet job aid was used to establish proposition arrays that captured the links between terms (1s indicate a link while 0s indicate no link). Each proposition array also contained 325 elements.
The resulting proposition arrays were transformed into two distinct scores. First, the number of “correct” propositions was determined by adding together the links (propositions) that agreed with the expert’s array(called Link-Ex). In set terminology, Link-Ex is the intersection of the participant’s and the expert’s proposition arrays. In other words, Link-Ex is a whole number that ranges from 0 to 36 in this case and is analogous to raw score in traditional tests. The second score derived from proposition data is closeness (called Link-C), a set-theoretic measure of node configural similarity (Goldsmith & Davenport, 1990).Link-C is calculated by dividing the intersection of the participant’s and the expert’s proposition arrays (i.e., Link-Ex) by the union of the two arrays, thus correcting for over-linking. Link-C is a real number that ranges from 0 (no similarity) to 1 (perfect similarity) and is analogous to traditional test scores determined by dividing the number correct by the total number of tries.
Next, the resulting association arrays were first converted into network representations of structural knowledge (converts pair-wise distances into 1s and 0s) called PFNets (Goldsmith & Davenport, 1990) witha software tool called Knowledge Network and Orientation Tool (KNOT) using the standard recommended parameters of r (infinity) and q (n - 1). Then, using the same approach described for the proposition data above, the PFNets were transformed into two distinct scores for each participant, association agreement with an expert (Dist-Ex) and closeness (Dist-C).
Results
The posttest data were analyzed by MANOVA with the lesson treatment main factor (relational, proposition-specific, and control) and seven dependent variables including ID, TERM, COMP, Dist-Ex, Dist-C, Link-Ex, and Link-C. Only COMP reached significance, F(2,20) = 5.25, MSe = 17.836, p = 0.015, none of the other dependent variables reached significance. Follow-up Scheffé tests revealed that the proposition-specific group’s COMP mean was significantly greater than the control group’s COMP mean (see Table 1).
Table 1. Posttest means and standard deviations (in parentheses).
Treatments / PosttestsID / TERM / COMP / Link-Ex / Link-C / Dist-Ex / Dist-C
control / 15.1 / 12.3 / 7.3 / 14.1 / 0.37 / 9.0 / 0.23
(4.4) / (4.6) / (5.4) / (4.6) / (0.14) / (3.6) / (0.11)
proposition- / 16.3 / 14.6 / 13.8 / 16.5 / 0.45 / 11.5 / 0.30
specific / (5.6) / (5.7) / (3.7) / (8.3) / (0.24) / (3.4) / (0.11)
relational / 17.0 / 12.7 / 12.4 / 13.9 / 0.36 / 10.7 / 0.29
(2.6) / (3.5) / (3.0) / (9.4) / (0.27) / (4.6) / (0.16)
Network Map Criterion-Related Validity
All four of the network map scores were significantly correlated (p 0.05) with the three traditional multiple-choice posttest scores, and in some cases network map scores were more strongly related to multiple-choice posttest scores than were the multiple-choice posttest scores to each other (see Table 2). Contrary to expectations, the four network map scores were least related to the ID multiple-choice posttest (though ID was most related to the TERM multiple-choice posttest). Although the ID multiple-choice posttest was a visual labeling task that on the surface seems to best align with network mapping, nevertheless, ID scores were least related to the network map scores.
Table 2. Intercorrelations of the seven dependent variables.
ID / TERM / COMP / Link-Ex / Link-C / Dist-ExID / --
TERM / 0.71 / --
COMP / 0.50 / 0.74 / --
Link-Ex / 0.56 / 0.77 / 0.53 / --
Link-C / 0.55 / 0.75 / 0.53 / 0.99 / --
Dist-Ex / 0.45 / 0.69 / 0.71 / 0.73 / 0.78 / --
Dist-C / 0.44 / 0.67 / 0.68 / 0.73 / 0.78 / 0.99
All are significant at the p < 0.05 level.
Perhaps most interesting is that Link-Ex, a proposition-specificnetwork map score was more related to TERM, a proposition-specific multiple-choice posttest score, than to COMP, a relational multiple-choice posttest score(r = 0.77 compared to r = 0.53); whileDist-Ex, a relationalnetwork map score, was slightly more related to COMP, a relational multiple-choice posttest score, than to TERM (r = 0.71 compared to r = 0.69).
Further, though Link-Ex accounts for 58.9% of the TERM variance (i.e., r = 0.77), multiple regression combiningDist-ExandLink-Ex accounts for 62.4% of TERM variance. Thus network map association data (Dist-Ex) provides slightly different proposition-specific information than network map proposition data only, with the two types of information accounting for more TERM variance than either alone.
Similarly, Dist-Ex accounts for 50.9% of the COMP variance (i.e., r = 0.71), but multiple regression combiningDist-Ex and Link-Ex accounts for only 51.0% of COMP variance. Thus network map proposition data (Link-Ex)provides no additional informationto account for comprehension. These finding are consistent with our expectations that network map links and distances represent different, though related (here, r = 0.73), aspects of structural knowledge, specifically proposition-specific knowledge and relational knowledge.
Discussion
This investigation examined the effects of relational versus proposition-specific reading orientations on both network map and traditional multiple-choice posttests that measured proposition-specific outcomes (Link-Ex, Link-C, ID, and TERM) and relational outcomes (Dist-Ex, Dist-C, and COMP). Only COMP posttest scores were sensitive to the treatment effects, although counter-intuitively, the proposition specific treatment (adjunct constructed response lesson questions) had the greatest effect on COMP, a relational multiple-choice posttest form. This indicates that proposition-specific processing of the expository text strongly influences comprehension, and thatnetwork map scores were NOT more sensitive in this case. Note that, since the map activity was given last, it is likely that the intervening multiple-choice posttests altered or flattened the effects of the lesson treatments on network map scores.
Further, the criterion-related validity of these network map scores was supported. Specifically, the proposition-specific network map and multiple-choice measures Link-Ex and TERM were strongly related, with an r = 0.77, and the relational measures Dist-Ex and COMP were strongly related with an r = 0.71. Note that these correlation values are equivalent to the correlation between the TERM and COMP multiple-choice measures, r = 0.74.
From a practical viewpoint, thenetwork map scoring approaches used in this investigation can be fully automated within a mapping software program. For example, a student could group and link 26 selected terms on the computer screen, and when done, the software could instantly compare the association and proposition data in that student’s map to an expert map or other reference map (i.e., the teacher’s map or a best student map) previously stored in the system in order to calculate the student’s Dist-Ex and Link-Ex scores. If this software system were implemented on a larger scale (such as online), then norm-referenced scorescould also be provided for use by students, teachers, and parents as well as school districts and even state departments of education.
Further research should consider questions like: what is the optimal number of terms for a network map, should links be labeled, how do mapping directions influence the importance of association and proposition data, what other traditional assessment approaches (such as essays) correlate with association and proposition data, would having students weight links (such as with dashed lines, single width, and double width lines) improve the predictive quality of the proposition data, and many others.
References
Bloom, B.S. (Ed.), Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: Handbook I: Cognitive domain. New York: David McKay.
Clariana, R.B. (2002). S-Mapper, version 1.0. Available online:
Dwyer, F. M. (1972). A guide for improving visualized instruction. State College, PA: Learning Services.
Einstein, G. O., McDaniel, M. A., Bowers, C. A. & Stevens, D. T. (1984). Memory for prose: The influence of relational and proposition-specific processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10, 133-143.
Goldsmith, T.E., & Davenport, D.M. (1990). Assessing structural similarity of graphs. In Schvaneveldt (ed.), Pathfinder associative networks: studies in knowledge organization, 75-87. Norwood, NJ: Ablex Publishing Corporation.
Hamilton, R. J. (1985). A framework for the evaluation of the effectiveness of adjunct questions and objectives. Review of Educational Research, 55, 47-85.
Jonassen, D. H., Beissner, K., & Yacci, M. (1993). Structural knowledge: techniques for representing, conveying, and acquiring structural knowledge. Hillsdale, NJ: Lawrence Erlbaum Associates.
McClure, J.R., Sonak, B., & Suen, H. K. (1999). Concept map assessment of classroom learning: reliability, validity, and logistical practicality. Journal of Research in Science Teaching, 36 (4), 475–492.
Poindexter, M.T. (2003). The effect of cognitive processing strategies and level of prior knowledge on student achievement of different educational objectives. Unpublished Doctoral Dissertation at the PennsylvaniaStateUniversity.
Ruiz-Primo, M. A., Schultz, S. E., Li, M., & Shavelson, R. J. (2000). Comparison of the reliability and validity of scores from two concept mapping techniques. Journal of Research in Science Teaching, 38 (2), 260-278.
Ruiz-Primo, M. A., & Shavelson, R.J. (1996). Problems and issues in the use of concept maps in science assessment. Journal of Research in Science Teaching, 33, 569-600.
Ruiz-Primo, M. A., Shavelson, R.J., Li, M., & Schultz, S.E. (2001). On the validity of cognitive interpretations of scores from alternative concept-mapping techniques. Educational Assessment, 7 (2), 99-141.
Wallace, J.D., & Mintzes, J.J. (1990). The concept map as a research tool: exploring conceptual change in biology. Journal of Research in Science Teaching, 27, 1033–1052.