Reaction to a Research Report on Learning

Wolfskill, L. A.

ALEC 601

Reaction to a Research Report on Learning

Boyd, B.L., & Murphrey, T. (2002). Evaluation of a computer-based, asynchronous activity on student learning of leadership concepts. Journal of Agricultural Education, 43(1), 36-45.

Boyd and Murphrey begin their paper with a very brief review of the literature that describes the thought on teaching leadership skills. The then quickly transition to instructional methods that use technology, leaning heavily throughout the paper on Alessi and Trollip’s book (1991). They then focus even further in their literature review to the use of simulation, especially asynchronous, and the ability of these to support teaching of the higher order thinking skills (Bloom, 1956). Generally, the authors provide an appropriate knowledge base for the introduction. They use the literature to lay the theoretical groundwork for the current quasi-experimental study.

Boyd and Murphrey clearly identify their research purpose. They use a section entitled “Purpose and Objectives” to succinctly state this purpose, that of determining whether or not the use of “asynchronously delivered simulation activity to teach leadership styles and ethics theory would impact learning” (p. 38). They continue with three enumerated objectives that the research project will complete. Two of them begin with the word “Compare,” while the other begins with “Determine,” indicating that they are testing hypotheses in this study, and expect to make statistically determined conclusions.

The authors do not explicitly describe a population for the study, but it is assumed from the conclusions that they intend for consumers of this research to apply it to other undergraduate classes teaching leadership styles and ethics theory. They did not indicate the sampling frame or accessible population, or specifically how they selected their sample. It is assumed that it was a sample of convenience, probably related to a course taught by one of the researchers.

Key results (whether or not the intervention actually occurred) were based on self-reported data by the subjects. There is no indication in the article that the researchers verified or sampled to check these data. Due to the design of the data collection, nonrespondents were apparently not an issue, so they did not need to control for that.

Data were collected through course examinations given to the control and experimental groups, one semester apart. The specific questions used from the exam were checked for content validity by “a panel of two faculty members experienced in curriculum design and familiar with the taxonomy.” The authors did not indicate whether any changes were made based on comments from the panel. Cronbach’s alpha reliability coefficient was calculated for the Objective One measurement section, and was admittedly low, at 0.62. The authors explain that away with some reason like, “Yes, but we’re sure the questions are valid and reliable because we’ve been using them a long time!” (p. 40). That is not a convincing argument, in my opinion.

The authors used the commonly accepted measure of a 0.05 type I error level for statistical significance. In their first objective, that of determining whether overall learning was affected by the treatment, they conclude that there is a statistically significant difference in learning based on number of questions answered correctly by each group. They report the t-statistic for this test, as well as each of the others.

For the second objective, that of determining the impact based on the level of cognitive learning as developed in Bloom’s taxonomy (cognitive domain), the results were mixed. Of the four levels identified, three (knowledge, comprehension, and analysis) were shown to have statistically significant differences in the measure of learning, while at the application level, no significant difference was noted. It strikes me as odd that the learning difference (or not) was seemingly unrelated to the level in the hierarchy. The two lowest levels, and the highest level, were significant, while the penultimate level was not. The authors made no direct comment on this, except to note that that the nonsignificant level was based on only three question in the assessment, while the others had five, six, and eight. It would have been nice if they had reported p-values so that we could determine at what level the results would have been significant.

The third objective was to compare results of the assessment to the learning styles of the participants. Prior to the assessment, students completed the VARK Learning Styles Inventory to determine their preferred learning style. Analysis of Variance was used to determine whether assessment score varied depending on learning style preference. In each case, there were no significant differences in learning with respect to learning styles. I would have thought that this finding would have generated some comments in the conclusions, but the only assessment was that “the simulation facilitated learning regardless of student learning preference.” The researchers did come back to the topic in the Implications section, noting that the instructional design principles used in the intervention covered all the bases, and so would have had something to aid any style of learning.

In the Implications section, the authors start by showing how the results of one part of the experiment fit into the current literature, aligning with the results of a previous study.

As noted above, it would have been more interesting if the authors had reported the actual p-values from the statistical tests, rather than the t-statistics, which are not directly readable. I was also a little surprised at the statistical results of the Objective 2 data. While I do not have access to the data, and have no reason to doubt the statistical accuracy, it just seems odd to me that, for example, the two compared sample means for Comprehension would be one third of a standard deviation apart, and yet at a 95% confidence level be determined to have significantly different means, especially when the number of questions are so low. I admit that I have a long way to go in being able to look at statistics and make guesses as to significance, but this is a reaction paper, and that’s my reaction and I’m sticking with it!

Overall, I thought the experiment was fairly well designed, and to me has plenty of face validity. I think that further research should be conducted into the relationship between levels of learning objectives in the cognitive domain, using an assessment that has more questions for each level, and a higher Cronbach’s Alpha score.

It was hard for me to relate this paper specifically to the learning theories that we have been learning in class. Perhaps it fits into David Ausubel’s schema, in that through the use of the CD with structured information, the learner can more readily organize the new information into the existing structure, and/or adapt that structure to better classify the data. (I should cite here, but don’t have a good document to reference)

REFERENCES

Alessi, S. & Trollip, S. (1991). Computer-based instruction: Methods and development. Englewood Cliffs, NJ: Prentice Hall.

Bloom, B., Engelhart, M., Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of educational objectives book 1: Cognitive domain. New York: David McKay Company.