Notes from the editors:
- Green highlights as well as comments need the attention of the author; yellow highlights require attention from the Handbook editors; Be sure that your file – when you open it - also shows the comments!
- The number of words exceeds the chapter limit, therefore the chapter needs to be slimmed down wherever possible; see the suggestions from the Handbook editors.
- This is a nice chapter; the proposed changes particularly intend the decrease the length of the chapter. Please review these comments/changes and revise your chapter accordingly.
- We did not yet check the references (complete/APA); we might come back to you when necessary.
Section 6: Chapter 3:The logic and logic model of technology evaluation
Yong Zhao
Michigan State University
Bo Yan
Blue Valley School District
Jing Lei
Syracuse University
Correspondence:
Yong Zhao, Ph.D.
University Distinguished Professor
Educational Psychology and Educational Technology Director, Center for Teaching and Technology Director, US-China Center for Research on Educational Excellence College of Education Michigan State University 115D Erickson Hall East Lansing, MI 48824
Phone: 517-353-4325
Fax: 517-432-4797
Email:
Abstract: After decades of investment in information technology, the promised revolution in education has yet to come. With the growing accountability pressure, evaluation of educational technology programs is laden with the hope of demonstrating technology does make a positive impact on education, especially student achievement. Troubled by a flawed question, however, this approach to technology evaluation is shown to be simplistic and inadequate for understanding the complex and dynamic relationship between technology and education. This chapter further introduces recent development in evaluation theory and method as well as latest perspectives on technology in education, and discusses their implications for technology evaluation. Based on the discussion, a logic model is proposed to serve as prototype for evaluating technology programs with adaptations and adjustments that are subject to local context.
Key words: Effectiveness, Program evaluation
1. Introduction
Evaluation is an unavoidable task for schools as they embark on the journey of adopting technology programs. The various stakeholders all want to have answers to their questions concerning the program. The funding agency, be it the community, a government agency, a private business, or a charitable organization, wants to know if its investment is worthwhile. The school governing board and parents wants to know if the program will help or hinder the education students receive. The teachers and other staff want to know how the program might affect their work. And of course the students, although often overlooked in many evaluation efforts, are entitled to know if and how the program might lead to more effective and efficient learning.
But evaluation is not an easy task. The multiplicity of stakeholders, the complexity of the context where the program is implemented, and the ill-definition of outcomes complicate the matter tremendously. Different stakeholders have different views of “the value or worth” of the same program, which is usually affected by the various factors of the school context and the program itself. Adding to that complexity is the fact that technology is often asked to simultaneously produce many outcomes, some of which are not well defined or lack precise measures. Consequently evaluation has to serve many purposes (U. S. Department of Education, 1998):
· Provide information to funding agencies so they can determine whether to continue the funding or invest in similar programs in the future.
· Provide information to school leaders so they can decide whether to continue the implementation and engage in similar ones in the future.
· Provide information to teachers and school staff so they can decide whether or how they might to support and participate in the program.
· Provide information to program staff so they can take actions to improve the program.
· Provide information for future evaluation efforts.
· Provide information to the general public.
Depending on its purpose, evaluation can be formative or summative. Formative evaluation is intended to provide ongoing information about whether things are proceeding as planned and whether expected progress is being made and if any changes need to be made while the purpose of summative evaluation is to provide information about the program’s overall merit or worth.
A recent evaluation study of technology in education exemplifies the various issues surrounding evaluation. In April 2007, the US Department of Education released a report entitled “Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort”(Dynarski, Agodini, Heaviside, Novak et al., 2007). This document reports findings of an evaluation study that intended to assess the effects of 15 computer software products designed to teach first and fourth grade reading and sixth grade math. It found that “test scores in treatment classrooms that were randomly assigned to use products did not differ from test scores in control classrooms by statistically significant margins.”
As soon as it was publicly released, the report caught the attention of the media. A day after the release, for example, the Washington Post published a report about the study with the title eye-catching title “Software's Benefits On Tests In Doubt: Study Says Tools Don't Raise Scores” (Paley, 2007). The influential education magazine Education Week published an extensive story on this study and reactions a week later under the title “Major Study on Software Stirs Debate: On whole, school products found to yield no net gains.” (Trotter, 2007)
The software industry and supporters of educational technology quickly responded to the study as well, challenging the study from every possible angle. For example, the Consortium for School Networking (CoSN), the International Society for Technology in Education (ISTE), and the State Educational Technology Directors Association (SETDA) issued a joint statement two days after the release, cautioning readers of the report "to scrutinize the findings carefully, as even the U.S. Department of Education [ED] states that the study 'was not designed to assess the effectiveness of educational technology across its entire spectrum of uses.'" The Software & Information Industry Association (SIIA) also responded with a statement about the study: "As this study recognizes, proper implementation of education software is essential for success. Unfortunately, it appears the study itself may not have adequately accounted for this key factor, leading to results that do not accurately represent the role and impact of technology in education."(Nagel, 2007).
The treatments this report received reflect the very issues of evaluation of educational technology. First, there is much demand for serious evaluation of technology in schools. Uncertainty about the effectiveness of technology still exists despite (perhaps because of) years of heavy investment and widespread euphoric claims about the power of technology for transforming education. The public, policy makers, and education leaders are in desperate need for sound evidence to guide their investment decisions. Second, evaluation of the impact of educational technology is not easy, to say the obvious. As critics of the study pointed out, there are many factors affecting the effectiveness of any given technology. It is very difficult if not impossible to isolate the effects of technology from those of the uses and the people who use them. Moreover, technology is a catch-all word that can include all sorts of hardware and software. Hence it is unwise but easy to do to over-generalize the results of one or two applications to all technologies. Furthermore, the impact of technology may not necessarily be on test scores. Thus selecting the desirable outcome measure becomes another issue when evaluating the implementation and impact of technology.
2. A critical appraisal of the Evaluation Literature
In the short history of studying technology in education, there has never been a lack of evaluation studies that are indented to gauge the impact of technology on education. No matter what form technology took (from Skinner’s teaching machine to radio and television) and in what way it was employed (face-to-face or distance learning), few technologies and technology applications have not been evaluated. When it comes to modern information technologies that are represented by personal computers, the Internet, and a variety of mobile electronic devices, there is no exception. Globally, the movement of infusing information technologies in schools has lasted more than a decade. This continuous investment is accompanied by all sorts of evaluation efforts that have increasingly become an inseparable part of many educational technology programs.
From 1995 to 1998, the European Commission funded the Targeted Socio-economic Research Programme whose focus was specifically on studying the results of those projects that were involved in ICT-supported learning innovations (Barajas, 2003). Case studies were conducted in OECD countries to examine the impact of information technology on teaching and school improvement (OECD, 2000). The Second Information Technology in Education Study Module 2 (SITES M2), a project of the International Association for the Evaluation of Educational Achievement (IEA) that involved research teams from 28 countries in Europe, North America, Asia, Africa, and South America, took an in-depth look into how curriculum is influenced by technology (Kozma, 2003). In the United States, with the push from the federal government for scientific research, a growing number of studies have been conducted or are under way, which involves randomly assigning research participants into a control group in which people do business as usual and a treatment group in which technology is integrated into curriculum (Dynarski, Agodini, Heaviside, Carey et al., 2007). Although there is a great variety among nations with respect to educational system, technological infrastructure, and resources for technology integration, the evaluation efforts are strikingly similar in terms of what questions they intend to answer and how evaluations are approached and conducted. Across the globe, people seem to care about similar issues and evaluations are engaged with addressing same concerns.
These evaluation efforts, plus countless smaller scale studies conducted by researchers in universities and research institutions, constitute a significant part of the rich repertoire of evaluation on the relationship between technology, learning, and teaching. A review of this literature suggests that while previous evaluation efforts have certainly contributed to our understanding of the role and impact of technology in education, there are a number of fundamental problems that must be addressed if we wish to make evaluation more useful practically and more insightful theoretically.
2.1 Flawed Question
In spite of the fact that a wide variety of questions are explored by evaluators, the evaluation efforts of technology and education is largely driven by the so-called “works” question, which often takes the form of “does technology work” or “is technology effective?” Though phrased differently, in essence, there are all concerned about one thing: do students using technologies perform better than those who do not on a certain outcome measure. Studies trying to tackle this question often employ experimental or quasi-experimental design and later become the data source of meta-analysis (Kulik, 2003; Murphy, Penuel, Means, Korbak, & Whaley, 2001; Pearson, Ferdig, Blomeyer, & Moran, 2005). The “works” studies flooded the educational technology literature, especially in a special field of educational technology: distance learning, where the research is dominated by studies trying to answer whether a significant difference exists between distance learning and face-to-face learning (Russell, 1999). The “works” research directly addresses policy makers’ concern about the efficacy of investing money in technology. In addition, it is often perceived by many to be methodologically easy to conduct. However, a careful examination reveals that it is not only theoretically impoverished but also methodologically problematic.
Epistemologically, the question of “does technology work” assumes that technology either produces expected outcomes on some pre-conceived outcome measures or not. This notion is troublesome in two aspects. First, it assumes that the effect of technology is largely invariant or at least consistent regardless of population, specific technologies, and the context in which it is used. However, synthesizing research (Kulik, 2003; Pearson et al., 2005; Waxman, Lin, & Michko, 2003; Zhao, 2005) has repeatedly observed large variation of outcomes existing within the technology treatment groups, which sometimes could be even larger than the between-group variation. These results have led researchers to believe that a dichotomized answer is not attainable and the effect of technology depends on how it is used and for what purposes. Second, in contrast to the varied effects and diverse goals of technology integration, outcomes in the “works” research have been mostly defined and exercised as standardized test scores. On the one hand, this narrowed view deemphasized other expected outcomes that deserve attention. On the other, it prevented us from examining unexpected outcomes that could be equally or even more important.
Methodologically, the question of “does technology work” can not be successfully answered because it in fact encompasses two distinct questions whose answers confound each other. Decades of research suggests that technology could lead to improved learning outcomes if used appropriately (Educational Testing Service, 1998; Fouts, 2000; Wagner et al., 2005). At the same time, there is evidence showing that appropriate technology uses are associated with conditions for technology uses (Dexter, Anderson, & Ronnkvist, 2002; Mann, 1999; Noeth & Volkov, 2004; Penuel, 2006; Zhao, Pugh, Sheldon, & Byers, 2002). Taking these two important findings together, it is not hard to notice that a significant difference in the “works” research is the result of two occurrences. First, certain technology uses are supported in a school environment. Second, those certain technology uses are effective in producing desired outcomes. Thus, “does technology work” is really asking: 1) are certain technology uses effective, which has been extensively studied in the past; 2) can those effective technology uses survive in a certain school environment, which has recently drawn a growing amount of attention. Therefore, a significant effect means not only a certain use is effective but also the use is achievable in a particular school environment. Similarly, a non-significant result could be that the use is realized but not effective or the use is effective but not achievable.
2.2 Simplistic approach to a complex phenomenon
The predominance of the flawed research question has several unhealthy ramifications to the evaluation of educational technology. One of them is that researchers are often confined to take a simplistic approach to studying educational technology. After years of exploration, it has been widely recognized that technology integration is a continuing process, which usually involves setting goals, laying out an action plan, implementing the plan, and making adjustments during implementation. The effect of a technology program depends on what happens in the integration process, which has been identified to be non-linear (Molas-Gallart & Davies, 2006) and dynamic (Mandinach, 2005), which could be complex and messy (Zhao et al., 2002).