Notes from the editors:

-  Green highlights as well as comments need the attention of the author; yellow highlights require attention from the Handbook editors; Be sure that your file – when you open it - also shows the comments!

-  We propose keywords for the chapter; please add/change when you think it is necessary

-  We did not yet check the references (complete/APA); we might come back to you when necessary. Also mention all authors in text (up to 6 authors), when first referenced.

-  This chapter exceeds the number of words available for a chapter; we provide suggestions to reduce the number of words;

-  The chapter is very much on the design and procedures which are needed for conductiong WISCEAS, but not so much on what we learned from WISCEAS about ICT in education;

-  I expected when reading the abstract a different chapter than the chapter I read; e.g 1.. information about which indicators appeared useful and relaible for an ICT-related WISCEAS and which indicators need further development; 2. how did different ICT policies of countries affect the CompEd and SITES studies; 3. what ICT-related WISCEAs would be desirable as a next step in the near future

-  We allow the use of ICT or IT, but ask the author to use one of these terms in a consistent manner throughout the chapter. In this chapter the use of ICT seems more appropriate

-  Please review these comments/changes and revise your chapter accordingly.

Section 10: Chapter 7 – Methods for large scale international studies on ICT in education

Authors – Willem Pelgrum (Danish University of Education) and Tjeerd Plomp ( University of Twente, Netherlands)

Abstract

This chapter reviews the different large scale international comparative studies which have been conducted over the last decade such as the COMPED, SITES and OECD Pisa researching the impact of ICT in education to consider the range of techniques and their usefulness and reliability. These are analysed in relation to various national and international policies and programmes in order to determine what indicators are likely to be reliable and useful on a large international scale. The authors also consider the different priorities of different participating countries and how these may affect the data collected through such international studies. The chapter concludes with recommendations for valid, reliable and useful (from a policy perspective) indicators for measuring the impact of ICT in education on a large international scale.

Keywords
International comparative assessment; Worldwide International Comparative Assessments (WISCEASs; SITES, CompED, PISA
Introduction

International comparative assessment is a research method applied for describing and analyzing educational processes and outcomes. They are used to ‘describe the status quo’ in educational systems from an international comparative perspective. This type of assessment started in the 1960s and has been mainly focused on core subjects, such as mathematics, science and reading (literacy). Over time, assessments were also conducted that were in particular focusing on the use of IT in education, the first one being the Computers in Education study (CompEd) conducted in the late 1980s and early 1990s under the auspices of the International Association for the Evaluation of Educational Achievement (IEA). (Pelgrum & Plomp, 1993).

Nowadays different types of international comparative assessments exist, such as:

1.  Projects by international organizations, e.g.such as:

a.  Projects funded by the European Commission (Eurydice, 2004)

b.  Exchange of experiences via the World Summits on the Information Society by UNESCO (see: www.unesco.org)

c.  Secondary analyses of assessments conducted by the Organization for Economical Cooperation and Development (OECD, 2006)

d.  WORLDBANK (Hepp et al, 2004)

2.  Case studies of selected schools in a number of different countries (e.g. SITES-Module 2, a study looking at innovative pedagogical practices utilizing IT; Kozma et al, 2003)

3.  International assessments using national representative samples of schools, teachers, and/or students, and focusing on collecting and producing comparative indicators regarding educational processes and outcomes.

This chapter will in particular address the last category of large scale international comparative assessments[1]. Although these assessments also exist on a regional basis[2], this chapter will primarily focus on large scale international assessments that are conducted on a worldwide scale by the International Association for the Evaluation of Educational Achievement (IEA) and the Organization for Economical Cooperation and Development (OECD). We will call these types of assessments Worldwide International Statistical Comparative Educational Assessments (WISCEAs) in order to distinguish them from qualitative assessments and from regional statistical comparative assessments.

In this chapter, the following questions will be addressed:

1.  Which WISCEAs addressed issues related to ICT?

2.  What were the major concepts and indicators that were addressed in these WISCEAs?

3.  How are national and international policies taken into account in designing WISCEAs?

4.  How are WISCEAS designed?

5.  Which recommendations for measuring ICT-impact via WISCEAs can be given?

These questions are covered in the following sections, starting with a short description of the history of WISCEAsthis type of assessment in terms of purposes, methods, audiences, and organizationss in particular with regard to ICT related matters. This will be followed by zooming in on the research questions underlying WISCEAs, conceptual issues, measurement (in terms of what and whom), and populations and samples in so far as assessments are concerned that address ICT related issues. Finally challenges will be discussed with an eye on (methodological) developments that are likely to occur in the future.

Historical sketch of WISCEAs

While scientific interest was the main purpose of the first WISCEAs initiated in the 1960s by the IEA, nowadays policy interests are the main driver for this type of assessment as is witnessed by the fact that the OECD has become a main player in this field by conducting the so-called Program for International Student Assessment-PISA (e.g. OECD, 2004). Although these organizations also run other types of comparative assessments (policy analyses, qualitative studies) they are most widely known (as far as press coverage and awareness of the general public is concerned) for their ‘league tables’ showing rank ordered country scores on achievement tests.

Since the first feasibility study of IEA on mathematics was conducted in the 1960s, more than 15 WISCEAs were reported by the IEA up until 2005. The OECD PISA assessment has been conducted and reported twice. Most WISCEAs were focused on basic school subjects such as mathematics, science and reading. The first WISCEA that was dedicated to ICT using a quantitative method was the so called ‘Computers in Education Study’ (Pelgrum & Plomp, 1993) that was conducted in two phases between 1986 and 1993. The first phase (with data collection in 1989) targeted schools and teachers (users as well as non-users) at the primary and secondary level, while the second phase also included samples of students. More details about conceptual issues and indicators will be provided later in this chapter. The second wave of IT-WISCEAs consisted of the so-called Second Information Technology in Education Studies (SITES), which started with a quantitative part targeting school samples at the primary, lower secondary and upper secondary level (1998-1999; see Pelgrum & Anderson, 1999, 2002). This was followed by qualitative case studies (174 in 27 countries) of IT-supported pedagogical innovations (Kozma, 2003). The next phase was again a quantitative assessment at school and teacher level in lower secondary education, called SITES2006, that is explained in more detail later in this chapter (see Figure 2). Since TIMSS-1995 the assessments of IEA and OECD (that were focused on traditional school subjects, such as mathematics, science and reading) included a few indicators of IT-use by students and teachers. Although these indicators were quite general (despite the fact that PISA-2003 contained a special IT-questionnaire for students), those studies allowed researchers to start with examining the relationship between IT-use and achievement of students in core subjects.

Questions underlying WISCEAs

WISCEAs may have several functions. Howie and Plomp (2005) distinguish the following functions: description (mirror), benchmarking, monitoring, enlightenment, understanding and cross-national research. Some of these functions (such as benchmarking, monitoring, understanding, cross national research) can be explicitly addressed by the research design, while other functions are more or less collateral (mirror, enlightenment). In general a basic purpose of WISCEAs is to contribute to educational improvement. Target audiences of the WISCEAs are therefore mainly macro-level actors: policy makers, inspectorate, researchers, and special interest groups in educational practice.

The design and methodology of WISCEAs are determined by policy and research questions that are posed at their start. In generic terms one may distinguish descriptive and analytical questions. Descriptive questions in general are phrased like: “What is the current status of our country with regard to a particular set of educational characteristics as compared with other countries?”, whereby ‘educational characteristic’ is specified in terms of, for instance: achievement of students in mathematics, differences in achievement between boys and girls, level of education of mathematics teachers, etc.. More Sspecifically ICT related questions might be: “is the access to IT in schools measuring up with other countries?, to what extent (comparatively) are our teachers skilled for using IT in teaching and learning?”, etc.

Analytical questions are usually phrased in terms of potential causal factors, for instance, in generic terms: “What factors cause that our country is scoring lower on a particular educational characteristic than other countries?” (Can you replace this with an ICT related question)

Although assessment is the primary function of WISCEAs, several secondary functions can be distinguished as is illustrated in Figure 1, which shows that policy issues and derived research questions constitute (ideally) the basis for designing an assessment. The results of the assessments are used for making inferences about strengths and weaknesses of the education systems of the participating countries (evaluation and reflection). Once weaknesses are spotted analysis activities need to be undertaken in order to find the potential causes (diagnosis). This part of a study is guided by analytical research questions. The results of such analyses of the available data (in countries sometimes supplemented with an additional data collection) may be used for undertaking interventions aimed at improvement of the education system.

Figure 1. Policy cycle showing several functions of WISCEAs.

Figure 1 illustrates that posing analytical questions a priori (that is when designing a WISCEA) can be problematic when the descriptive results are not yet available. Therefore analytical questions in WISCEAS are often based (a priori) on hypothetical outcomes inferred from reviews of research literature, or they are generated a posteriori and based on the observations resulting from the descriptions.

Policy and research questions form the start of any WISCEA. Concepts that are referred to in these questions (in the examples above: mathematics achievement, gender, teacher characteristics, access to ICT) are organized in a conceptual framework which on an abstract level forms the link to indicators, andwhich constitute the basis for developing instruments.

In the next sections each of these aspects are discussed briefly.

Conceptual frameworks

Conceptual frameworks mainly have a descriptive and structuring function: they more or less describe the ‘landscape’ contours of the area that will be assessed and at the same time provide a structure for deriving relevant constructs. For instance an important policy issue is equity (concept), which may be defined as gender and minorities (constructs) for which indicators can be developed (such as responses to questions like ‘what is your gender’ or ‘in which country were you born’). In the context of this type of study, an indicator is often called ‘variable’ or ‘statistic’.

An important distinction is between independent and dependent indicators, which both can be used for descriptive purposes, whilst the independent variables play an important role in addressing the analyses questions shown in Figure 2 below.

Figure 2 – Links between independent and dependent indicators

The construction of a conceptual framework for a WISCEA is a complicated process that is usually not a linear one, but rather based on an initial proposal developed by an international coordinating centre which is further elaborated via intensive interaction with the representatives of participating countries who together take decisions about (political) relevance and feasibility. Quite often, when initial concepts are further operationalized, it appears that not always suitable and cost-effective indicators can be found for important concepts (e.g. life long learning skills of students). Hence, initial conceptual frameworks often need to be revised during the elaboration of a WISCEA-design and after the pilot-testing of instruments in the participating countries. Furthermore, time and budget constraints prevent all the favourite issues of the participating countries from being addressed. This problem is often solved by allowing countries to add so-called national options to the international assessment measures.

An important distinction between the conceptual frameworks of IEA and OECD is the curriculum-perspective: IEA-frameworks strongly focus on the common denominator in curricula of participating education systems, while the OECD framework is “literacy” oriented, looking at the yield of education: “…PISA assesses the ability to complete tasks relating to real life, depending on a broad understanding of key concepts, rather than limiting the assessment to the possession of subject-specific knowledge” (OECD, 2004). An example of a conceptual framework of an ICT-relateddedicated WISCEA is shown in Figure 3. This framework is taken from SITES2006, an assessment for which the data were collected in 2006 in 22 education systems. The starting point for this assessment was the policy issue of pedagogical innovation and the role IT is playing in these innovations. This issue has grown from the more than 20 years of experience in a substantial number of countries regarding the introduction and use of IT in education.

The SITES studies focussed on several concepts that, according to earlier research, influence the implementation of pedagogical innovations. These were operationalized in SITES2006 in terms of pedagogical paradigms. A; the most important ones a contrasting pair was distinguished:, viz traditional pedagogy (teacher-directed, whole class teaching) and emerging pedagogical approaches (more responsibility and autonomy of students for their learning, working in small groups, lifelong learning skills). As illustrated in Figure 3, it was hypothesized that the extent to which these two approaches exist would be influenced by several (often interacting) ‘conditions’ at the school and teacher level. IT was conceived as one of the important conditions.