School Self-Evaluation in Primary Education in the Netherlands

SCHOOL SELF-EVALUATION IN PRIMARY EDUCATION IN THE NETHERLANDS

HENK A. MOELANDS

MARIUS J. OUBORG

THE NATIONAL INSTITUTE FOR EDUCATIONAL MEASUREMENT

ARNHEM, THE NETHERLANDS

Henk A. Moelands

+3126 3 521 562 (voice)

+3126 3 521 356 (fax)

e-mail:

Marius J. Ouborg

+31263 521 422 (voice)

+3126 3 521 356 (fax)

e-mail:

CITO

P.O. Box 1034

6801 MG Arnhem, The Netherlands

Paper presented at the European Conference on Educational Research (ECER98)

Ljubljana, Slovenia

September 20, 1998
School self-evaluation in primary education in the Netherlands

Abstract

The Dutch educational decentralization policy is aimed at giving schools more freedom and making them more responsible for their outcomes. This has resulted in a growing interest for quality control procedures at the school level. School self-evaluation has become an important issue. The purpose of this paper is to present a procedure for school self-evaluation in primary education. It will be shown how a system based on the outcomes of education can contribute to school self-evaluation. In the system the outcomes are considered as the starting point for determining educational quality. Moreover, the system allows outcomes to be adjusted for the quality of educational input and educational processes.

Cito has developed a student monitoring system that allows to monitor the progress of individual student over time. By applying item response models the system shows how much a student improves with respect to a previous measurement moment. In the presentation it will be demonstrated how the aggregated data from a student monitoring system, in combination with electronically recorded data from the school administration, can be used in a school self-evaluation procedure to indicate whether or not the progress of a group is up to expectation.

The system for self-evaluation being developed will have the following features:

• The ability to monitor student achievement within a school over time. The increase in student achievement is considered as an indicator of the quality of the educational process within the school.
• Student achievement will be related to input- and process data, in order to control the influence of these factors.
• The collection of input and process data should not result in an additional burden for schools.
• Users of the system have to be able to make judgements that are scientifically meaningful, without complicating these by the use of difficult statistical procedures.
• The set of instruments should be easy to handle and should lead to results that are easy to interpret.

In this paper attention wil be paid to the three first points mentioned above. The problems encountered and the way they are solved will be discussed.

School self-evaluation in primary education in the Netherlands

[噉䴂O쇎1]Dutch educational decentralization policy, developed to give schools more freedom and make them more responsible for their results, has resulted in a growing interest in quality control procedures at the school level. Schools are being held more accountable for the quality of the education they provide. Therefore they have to active engage in quality control. They must choose methods and instruments, set targets that are to be achieved, and determine which adjustments have to be made. To help schools improve the quality of their education, CITO, the National Institute for Educational Measurements, in close cooperation with OCTO, The Centre of Applied Research in Education of the University of Twente, and SLO, the National Institute for Curriculum Development, is developing a system consisting of a set of instruments, procedures, and logistics for school self-evaluation (the ZEBO project). This system emphasizes the outcomes of education which are seen as the core criteria for educational quality

The system for self-evaluation being developed will have the following characteristics:

-The ability to monitor student achievement within a school over time. The increase in student achievement is considered an indicator of the quality of the educational process within the school.

-Student achievements will be related to input and process data, in order to control the influence of these factors.

-The collection of input and process data should not result in an additional burden for schools.

-Users of the system have to be able to make judgments that are scientifically meaningful, without complicating these by the use of difficult statistical procedures.

-The set of instruments should be easy to handle and should lead to results that are easy to interpret.

The purpose of this paper is to present a procedure for school self-evaluation in primary education. A definition of school self-evaluation in the context of this project, the function of the self-evaluation system and a number of prerequisites for output oriented school self-evaluation will be presented. It will be shown how a student monitoring system operates at the micro level and how this system can also be used for self-evaluation at the school level. Moreover, the paper will deal with the problems concerning data collection on a large scale. Finally, the paper will concentrate briefly on the problems related to multi-level modelling.

School self-evaluation

The term school self-evaluation[쇎쇤䴂2] means that it is the school itself that evaluates instruction. door schoolzelfevaluatie worden de sterke en zwakke punten van de school in kaart gebracht en wordt vastgesteld of het gegeven onderwijs ertoe leidt dat het beoogde resultaat wordt bereikt. Through school self-evaluation, the strengths and weaknesses of schools are summarised, and it is determined whether or not the instruction in question leads to the achievement of the intended results. On the basis of the evaluation, a school can decide to set priorities and take decisions about improvements. These improvements can relate to various (educational) factors. Factors that are important for the quality of education can be categorised according to input, process, and output (CIPO model). In the procedures that are to be developed, special emphasis will be placed on student achievement, since this is considered to be the most important indicator of the quality of education. However, the circumstances under which instruction takes place have to be taken into account in the evaluation of student achievement. The set of instruments will therefore offer the possibility of relating the test results achieved to input and process factors, such as the social background of the students, the quality of the teaching staff, material means, and the time available.

Thus, self-evaluation includes:

-collecting information about student achievement

-collecting information about teaching activities

-collecting information about the student backgrounds

-drawing conclusions about (1) the extent to which one is satisfied with the results within the context of input and process factors, and (2) about possible measures to improve quality.

Student monitoring system

The system will emphasize student achievement and one of the features relates to monitoring student achievement within the school over time. For that reason, the system makes use of of the Cito Student Monitoring System tests. Repeated measurements of student achievement over time are important in this system, which consists of a concrete set of tests for certain subjects. consistent sequence of tests for certain subjects. By applying item response models, scales which allow the school to monitor the progress of individual students, have been developed for different subjects. Results for different tests administered at different times are transformed into the same scale. This scale shows how much a student improved with respect to a previous moment of measurement. When problems (such as improvement that lags behind expectations) are detected, further analysis can be done to find possible causes and to provide help. The system supports the teacher in this task. Characteristic of the student monitoring system is the fact that it is not based on a single assessment; progress in a subject matter component is monitored periodically, throughout the years, on the same fixed scales. This means that a balanced picture of progress may be given in the course of the years.

The tests from the CITO-Student Monitoring System are administered twice a year. Afterwards, raw scores are transformed in scale scores, which are forming the basis for determining progress over time. The scale scores are represented in a graph (see Figure 1), from which the students' improvement (Tom and Anne in this example) can be read. The horizontal axis represents the different moments of test administration of measurement, while the vertical axis represents the corresponding scale scores.

Figure 1: Student report

The rising line in Figure 1 indicates that Tom improved or, in other words, has learned steadily more things. Whether or not this improvement is satisfactory for this student has to be decided by the teacher/school. To help teachers in this decision, the graph also represents the results of a national reference group. The scores that were obtained by this group at different moments are divided into 5 levels, A to E. Level A corresponds to the 25% of students with the highest scores; level B refers to 25% of the reference group that is situated just above the national mean score; level C refers to those 25% of the reference group that are located just under the national mean score; level D refers to the 15% of the reference group that obtained scores far below the national mean; and finally, level E represents the 10% lowest scoring students.

Figure 1 shows that although Tom improved, he obtained a score that is some distance below the national mean. Nevertheless, his development runs parallel with that of his level group. Given his performance, Tom is a student that one has to monitor. Anne shows a different development. Initially, she belonged to the same level group as Tom (see moment M3). However, Figure 1 shows that her development is stagnating. At moment E4, Anne belongs to the group of lowest scoring students. This means that the school has to take action to determine the reasons for this stagnation and provide aid. Monitoring students in this way is not a question of a single moment. Signals are received showing that a student's development is not going well, or at least does not seem to be.

Group level monitoring

After administering the tests and recording the results improvements of individual students, a teacher must be able to obtain information about the achievement of a group of students by aggregating these data. By doing this over several years, a group report is created that provides an overview of the mean scale scores that have been obtained during successive years by different groups. Figure 2 shows a group report which represents the results of several cohorts of students. A cohort is defined as a group of students that start at the same moment at level 3 and afterwards share the same instructional program. The horizontal axis represents the moments of test administration; measurement the vertical axis represents the corresponding mean scale scores for the cohorts.

Figure 2: group report

Since we still do not have national reference data on the school level, we make use of the scores of the school in previous years. In this way it is possible to get information about whether or not the results obtained by a cohort correspond to the results obtained in previous years. If a school is aware that the composition of its students is different from national patterns - and therefore can expect to have structurally better or worse results compared with a national referece group - a school may be decided to apply its own achievement standards. Figure 2 shows the scores of the students of the eight cohorts from grade 3 through grade 8 and reflects the usual scores for this school. In Figure 2, it can be seen that cohort '87 shows a drop in grade 8. By relating the results at this moment to some student characteristics it is possible to get more specific information about the kinds of improvements that should be made. See, for example Figure 3, where the background of the students is taken into account which leads to a classification of the students into three groups: 1.0-, 1.25- and 1.7-students. Figure 3 shows that is not the 1.0-students who are responsible for the drop, but the 1.25- and the 1.7-students.

Figure 3: group report, separated

Available information at schools

If this set of instruments is to be successful, then using them should not be an extra burden for schools. It has therefore been decided to use information which already exists in schools (in an electronic database). In the Netherlands, most schools use the same commercial school administration database. With the help of this program much of the input and process data can be filed. In 1994, CITO launched a computer program with which schools are able to file the test results of their students. By combining the data from the CITO program with the data from the school administration database, schools quickly obtain information for the evaluation of their programs without any additional effort.

The problems we have encountered and the solutions we have developed will be discussed in the next section.

Data collection

The most important purpose of the system is integrating data from school administration databases and test administration databases to support and evaluate school policy using the CIPO model.

Most schools in the Netherlands make use of the same commercial school administration database. The problem is how to make use of these data without burding the schools. Extracting the desired information is very labor-intensive, mainly because these administation databases were not developed for these purposes. Two possible solutions are: (1) adapting the databases and (2) adding a new system to the already existing school administration databases. The first solution is very radical and therefore possibly not so interesting. But, what most important is that adapting the databases is not so functional for the school administration databases as developed. The second solution, known as the Data Warehouse concept (DWH) in the literature, offers more perspective. First, the problems involved in using the school administration databases will be discussed and then the basic assumption of a DWH will be elaborated on.

The school administration databases used by schools are known as operational systems in the information technology, developed for administrative transactions. The purpose of the ZEBO project is to integrate data from school administration databases and test administration databases to supply schools with information about the quality of their education. For this purpose, we need a instrument for policymaking.

In our test case the most important problem turned out to be the difference in characteristic features of both systems. The operational systems do not acquire the demands for policymaking desired by the ZEBO project. There are a number of important differences between the two systems:

Operational systems:

• Are directed at processing a great number of administrative transactions
• Contain a lot of detailed information
• Contain data that constantly changes (there is a possibility of mutation)
• Are not aimed at integration of data from different operational systems and they do not collect historical data by mutation of data
• Need to have a highly performance
• Have a static structure with a variable content

Policymaking systems:

• Are directed towards a limited number of complex questions, aimed at providing data
• Are directed towards content (rather then transaction)
• Contain consistent, static (not changeable) data
• Contain historical data, in which aggregation of data is crucial
• Do have a dynamic structure

The most distinctive difference between the two systems is that, in the case of an operational system, a change in data generally leads to a mutation of the data. In a policymaking system, a change in data leads to a new - mutated - record. Thus, in a policymaking system, both the original and the mutated record are available.

Comparing the examined operational system with the features from both before- mentioned operational and policymaking systems, it appears that the operational systems:

• Offer too much possibilities (freedom) for mutation of data
• Do not record historical data for background informations of pupils and test results
• Do not offer sufficient possibilities to validate the data necessary for the ZEBO-project
• Do not offer sufficient possibilities to integrate the data from different sources. The data are not adequately geared to each other.

Given the above-mentioned observations, it can be concluded that the application of the Data Warehouse concept can be a solution problems mentioned above. Applied to school-selfevaluation, it leads to the idea reflected in figure 4

Figure 4 shows that a data warehouse concept consists of a locale and a central part, also known as the 'distributed data warehouse' (Immon, 1992). Locale means that every single school must have a facility in which the data from school administrative and test registration databases can be brought together in the data structure desired for school self-evaluation. In figure 4, this is indicated with the term 'Local data warehouse'. The validation and integration of this data, as well as the aggregation, must be executed within every school. These analyses concern only the school itself. To do some analysis beyond the school - for example, to provide schools with reference data - the locally stored data be collected in the central data warehouse (see Figure 4). By applying techniques known in the literature as 'data-mining' and 'knowledge discovery', the data can be further integrated and extra analysis can take place. The schools will be offered the possibility to extract the reference data they desire from the DWH.

Locale Data Warehouse (school)