Cara KuglerCECS 5610: Article Critique
Reference: Moore, Joi L. and Marra, Rose M. (2005). A Comparative Analysis of Online Discussion Protocols. Journal of Research on Technology in Education, 38 (2), 191-212.
Principle:This study comparestwo different online discussion board protocols, a structuredformat (constructive argumentation) vs. a less structured format, and measures the effect on knowledge construction.
Type of Design:According to the Campbell and Stanley classification, the type of design of this study is The Posttest-Only Control Group Design. Graduate-level students in an online class were randomly divided into two groups. The independent variable in the study was the type of discussion protocol used and the dependent variable was the degree of knowledge construction, which was measured using the IAM model (Interaction Analysis Model) Students in both sections utilized identical course materials and retrieved all course content from a single course Web site, but participated in separate discussion boards, utilizing the two different protocols. Students in both sections were given general posting guidelines, which would be considered the ‘basic protocol’. Section one students would follow the ‘basic protocol’ while students in section two were asked to also label their postings as one of the following components of arguments: thesis, evidence, assumption, and synthesis. The discussion posts were rated over a 9 week period using the IAM model, which is an instrument designed to detect evidence of knowledge construction
Factors Jeopardizing Internal Validity: The nature of the ‘posttest-only control group’ design generally controls the internal factors quite well, but the factors of selection and instrumentation pose some threat to the internal validity of this particular study. Although selection was randomized between the two sections, there were more participants in section one (21) than section two (16), and there was unequal distribution of the academic level of students in these sections. Although the majority of students were master’s students, section one had three doctorate students and section two had one doctorate, and one undergraduate. These differences might have affected the data.Instrumentation might also jeopardize the internal validity because human observers were used to provide observations and they can be prone to fatigue and the process of learning. The IAM protocol was used as the primary instrument, and can be somewhat subjective because although the phases are clearly defined, it still depends on the rater’s interpretation of both the student postings and the defined phases used in the IAM instrument.
Factors Jeopardizing External Validity: This study was performed on a limited sample size; a small group of graduate level students, in only one content area. The results observedin this study were based on an advanced level of students, and the results might not be true for students of lower academic levels, such as undergraduates. Also, the argumentative protocol was tested in only one online course and therefore applied in only one content area. Therefore, the findings in this study cannot be generalized to include all online classes of various student levels and content areas.
Adequacy of Statistical Procedures Used: Data analysis methods for the study were based on Chi’s eight-step method for “quantifying qualitative analysis of verbal data”. The IAM model was an appropriate choice as it gave researchers a way to systematically produce quantitative results from an underlying qualitative coding of the discussion board transcripts. Inter-rater reliability checks were used and reliability figures were calculated for each section using Krippendorff’s and Cohen’s kappa and indicated an acceptable to moderately strong agreement rate by the raters. Chi square analysis were performed on the phase totals for each section, and showed that the frequency differences between the two sections were significant (p<.05 )with students in section one having both more coded postings and postings at higher phase levels. The measures of central tendency used in this experiment were the mean and standard deviation.
Summary of Logic:The argumentation protocol that was designed to facilitate knowledge construction in section two did not show evidence of having that effect. This reasoning was inductive based on the findings in the study that show there were overall fewer postings in section two, a lower number of postings in phases III and above (indicative of knowledge construction), and that the overall level of interactions in section two was also lower. The authors propose several explanations for these observed differences. They propose that there were fewer responses in section two because students may have found the ‘constructive argumentation’ protocol too difficult to implement because it wasn’t well suited for the content. Section two also had fewer students and that could have further impacted both the quantity and quality of posts. The authors conclude that further research is needed to investigate the effect of participation protocols with different discussion board activities. Based on this study, one cannot conclude that argumentation does not promote knowledge construction, as it may indeed promote it in a different context.
Design Improvement: To strengthen the internal validity, the students in both sections should be of the same academic level. Those doctorate studentsin the study might alone have attributed to the higher numbers in section one. Researchers might choose to use a different measurement instrument than the IAM, possibly one they create themselves. The reliability scores indicated that the raters were likely having trouble interpreting the IAM phases consistently. It is possible that that the reliability figures for inter-rater would improve if the researches did not have to interpret another researchers coding scheme (like in the IAM model). Finally, the researches mention that students were asked to agree or disagree with controversial statements. By simply changing these discussion prompts, it might encourage students to achieve higher levels of construction of knowledge.
Extension of the Study:I would like to see a similar study done on undergraduates, or even high school students. It is possible that a more structured protocol, like the argumentative one, might show a positive effect on knowledge construction in younger students who are lacking the more advanced skills typically found in graduate level students. If different discussion protocols have different results on various levels of students, this type of information would be very helpful to those designing online content.