Quality Assurance and Evaluation (QAE) - Support Or Nuisance

Work in progress – please do not quote without permission of the authorsFab-Q M5-1sw

‘Quality assurance and evaluation (QAE) - support or nuisance?Swedish teachers’ views’

Håkan Karlsson ()

Christina Segerholm ()

Department of Education, MidSweden University

Paper presented to the European Conference on Educational Research (ECER) Network 23 Symposium: “Fabricating Quality in European Education (FABQ): Teacher Professionalism, Quality Assurance Regimes and Performance: a comparative study”, held in Göteborg, 10th-12th September 2008.

Abstract

In Sweden, a web of intertwined evaluative activities has been developed in the last 15 years. Most of them are mandatory and directed at all levels in the school system, be it pupils, schools, municipalities or the entire nation. Teachers are one central group in these processes. What are their views and experiences of QAE? How do these activities influence teachers’ work? Answers to these questions are essential in order to assess QAE impact on educational practice. Following the agreed methodology of the international project ‘Fabricating Quality in European Education’ the results of a nationally distributed survey directed to teachers in public mandatory schools are presented in this paper. The sample consisted of teachers in 10 municipalities. Descriptive statistical analysis was used to give an overview of teachers’ views, reported QAE activities, and the influences of QAE. Multivariate analysis was used to convey response patterns. Since both external and internal validiy are weak, the results are only tentative. Results show that ……………………the survey are discussed in relation to the web of evaluative activities and QAE policies in Sweden and in relation to trans-national policies (ie EU and OECD). QAE influence on teacher professionalism, autonomy and control is brought to the fore as well. The seductive power of governing through QAE is examined.

Introduction

During the last 15 years, a web of evaluative activities has been developed in Sweden (a comprehensive evaluation system as defined by Leuuw & Furubo 2008). The aim of these activities is said to promote quality, but are naturally also part of the governing instruments used in present national governance of Swedish education. Moreover, the rhetoric in Swedish policy documents provide many examples of how this quality assurance/assessment and evaluation (QAE) policy and practice is a mix of ideas of school development/improvement, accountability, student and parent influence, control and steering – all in the name of good or better quality (Segerholm 2007).

Most of these activities are mandatory and directed at all levels in the school system. At national level, administered by the Swedish National Agency for Education (SNAE), there are:

• national inspections carried out every third year, including site visits in all municipalities and all schools within each municipality, both public and independent
• national tests in school years 5 and 9 (and school year 3 soon) in the school subjects English, mathematics, Swedish and Swedish as a second language
• yearly follow-ups on statistical measures of for example student/teacher ratio, grades, test results, resources for special education, and
• involvement in international studies like PISA, PIRLS, TIMMS administered by researchers.

Municipalities are required to carry out:

• yearly municipal audits of public education and deliver to the municipal school administration and local (municipal) parliament
• yearly quality accounts of public education and deliver them to them SNAE.

Schools (principals and teachers) have to undertake:

• yearly quality accounts of each school and send it to the municipal administration
• continuous evaluations of school subjects, projects, instruction etc.
• student assessments (various kinds like teacher developed tests, port folios), and develop
• individual study plans for students (based on student performance related to national objectives in different school subjects).

Taken together, these activities may be characterised as a rationale based on collecting data on preconditions, process/work in school, and on attainment/goal fulfilment at all levels. Furthermore, self-evaluations are common in local quality accounts in schools and municipalities, as well as in recommendations to schools of recommended evaluation approaches. Making comparisons possible between schools, municipalities and nations from collected data (e.g. on particular indicators[1]), is thought to be the basis for improvement. Finally, the design of these evaluative activities sustains the goal-result governing doctrine incrementally induced at the end of the 1900s by its increased emphasis on the relation between goals/objectives and results/attainment (Segerholm 2007). Only recently has national inspection deviated from this rationale, when a new (neo-liberal coalition) government took office after the elections in 2006. Student performance and subject knowledge is now to be further stressed in the inspections (Skolverket n.d.).

Sweden is not alone in its efforts to emphasise and promote quality in education. The international project this study is part of (Fabricating Qualtiy in European education), shows that education (as learning) is a central part of making Europe a successful competitor on the global market. It is also clear that transnational policies on QAE crafted by the European Union and the OECD appear to be similar across nations, at least in name. They also play a significant role in what may be labelled global governance in the field of education. However, the success of global policies and governance in education is yet to be assessed. There is no simple hierarchical chain of compliance from transnational organisations’ efforts, through national and local authorities to schools. Public education is still a policy domain much governed by the nation states, and by local authorities in varying degrees in different national settings, so global policies are not easily implemented. Although the nation state (as government) still has and exerts power, it is not as sovereign in governing processes as some decades ago (Pierre & Peters 2000). Public-private co-operation in education deliverance, parental choice, and other new public management methods as well as different local, national and international networks influencing education policy, have changed the context for the nation state when it comes to governing (Ball 2006a, Pierre & Peters 2000, Sassen 2007). Rosenau describes global governing as a mobius web. “A mobius web is top-down, bottom-up and side by side governance all at once (Rosenau, 2002:81-83 as cited in Rosenau 2005:145). He continues to point out that efficient global governance is not easily achieved since there are many levels, actors, and networks, etc. whose compliance and concerted efforts are needed in effectuating policy. But, this also means that education policy and QAE influences may not only come from the state, but also move between levels, nations, regions and networks both horizontally and vertically.

Teachers are one central group when carrying out QAE activities. They are presumably also affected both by the procedures and the results of QAE activities. Earlier research has for instance showed distortions of teacher instruction in contexts where test results play a significant role in detention, teacher salary and accountability (school board, school district and state) (see e.g. Black 2001, Linn 2000). Evaluative processes also impact education practice and teachers’ work in that the evaluation criteria direct principals’ and teachers’ attention and actions (Ball 2006b, 2001, Segerholm & Åström 2007). Depending of what is considered good education, the impact of QAE may be valued positively or negatively (Ball 2006b).

Our interest in the study reported here, was directed to teachers views and experiences of QAE, and to inquire how these evaluative activities influence educational practice, in this case teachers’ work. The following questions were central to the study:

• What are teachers’ views and experiences of QAE?
• How do these activities influence teachers’ work?

This paper reports the answers to those questions as well as discusses the them in relation to the web of evaluative activities and QAE policies in Sweden and in relation to transnational policies, (i.e. EU and OECD). QAE influence on teacher professionalism, autonomy and control is brought to the fore as well.

Procedure

In order to answer the above questions, and to make comparisons between the nations in the international project Fabricating Quality in European Education (FabQ), a survey study was conducted. The international team developed a set of core questions used in the Swedish survey (a process described in a paper by Grek, Pedersen, Segerholm & Simola 2008). The aim was to carry out nationally representative studies, in our case representing the views and experiences of Swedish teachers working in compulsory school.

Pilot studies

Once the comparative questions were agreed on, and we, the Swedish team, had translated and accommodated them to the Swedish context, a pilot study to try the questions was carried out. A small group of teachers answered the questions (paper and pencil). Afterwards an informal interview took place concentrating on the concept QAE, and on the relevance and clarity of the questions. A couple of questions and response alternatives were omitted before transforming the questions into electronic form.

A second pilot test was then undertaken directed to all teachers in one small municipality. The aim was primarily to test the functionality of the electronic format distributed by e-mail. The response rate was extremely low due to a spring brake, but we nevertheless concluded that the electronic survey worked as planned.

Sample

The sample was a random XXXXXX (XXXXXX) on municipality level, based on nine categories that are used for most of the national statistics in Sweden (see table 1 below).

Table 1: Categories of municipalities from Statistics Sweden

One municipality from each category was randomly chosen together with one municipality that was randomly chosen from all Swedish municipalities, total 10 municipalities. All teachers in these municipalities were sent a questionnaire. This resulted in a sample size of 4588 teachers. That is about 5% of the total amount of teachers working with pupils between 6 and 16 years, total N 90540.

Distribution

The teachers were sent an e-mail in March 2008 with a request to participate in our web-based survey. Unfortunately spring semesters are intervened by a number of week-long holidays and brakes in Sweden, a fact that may have impacted on the very low response rate.

The electronic questionnaire was designed in such a way that we could control who answered without having to know their e-mail addresses – a design that fully guaranteed respondents’ anonymity. Reminders were sent to those who did not respond at two occasions.

Responses and dropouts

The response rate was 12,3 %, 564 persons. The response rate in the different municipalities varied between 8,1% and 18,8%. We can not see any particular pattern in response rates depending on type of municipality. There was also a systematic dropout on the school level, nine independent schools refused to hand out their e-mail addresses. The responses still need to be analyzed for internal dropouts. A more thorough non-response analysis of external dropouts has yet to be carried out.

Respondents

The respondents work with the following age-groups of pupils

Pre school class n = 54

Early years (school year 1-5) n = 275

Later years (school year 6-9) n = 319

141 men and 423 women responded to the survey and that corresponds fully with the total gender distribution among Swedish teachers. The age distribution still needs to be analyzed further.

The respondents’ age varies according to the following: Mean 45,3 years Std. D 10,3; Median 45,0 Range 48, Min 20, Max 68

The number of years the respondents have worked as teachers are: Mean 16,9 years Std. D 11,4; Median 13, Range 42, Min 1 Max 43. The majority 360 work full time, 102 part-time, and most (525) hold permanent positions (102 temporary).

One observation is that 20% (111 respondents) of the respondents have more than one education.

Validity

Since the response rate was so low, it is unfortunately not possible to generalise to the entire teacher population in Sweden. We can only report the responses for this group of teachers from 10 municipalities. Because of the teachers’ generally positive attitude towards QAE, we also believe that the respondents may be positively biased towards QAE. However, even so there are answers that point to negative sentiments or doubts that should be taken seriously precisely because of the general positive view of QAE.

We also assess the internal validity (construct validity XXXXXX) to be low. Even though the survey displayed a definition of QAE, we think that the concept meant different things for different respondents, perhaps depending of what QAE activities they recently had been involved in. Results from this survey must therefore be treated with caution and be viewed as tentative.

Analysis

Descriptive statistical analysis was used to give an overview of teachers’ views, reported QAE activities, and the influences of QAE. Multivariate analysis was used to convey response patterns. Här behövs specifikation

We have refrained from reporting results that stem from an analysis based on cross-tabulations by age/gender/area/class. This is not because we lack interest or ambition, but because the cross tabulations we did carry out presented no significant differences related to these background variables.

Teachers’ views on QAE

The following sections present the results of the survey study. A description of teachers’ views of what high quality in education is provided followed by an account of their experiences of QAE practices. Teachers’ responses on issues concerning QAE influence concludes the presentation of results.

Views of high quality in education

According to table 2, one observation is that individual development seems to be an important ingredient in teachers’ perception of high quality in education. However they do not find individual development in terms of critical thinking or independent thought that vital. Basic skills like literacy, communication and numerical skills are also central to education quality. Interestingly enough the response alternative “Responsible citizens that participate in political, economic, social and cultural life” ended up last(almost),even thoughthe democratic value base has been the core of the Swedish national curricula at least since 1994.

Table 2. Ranking of teachers’ views of high quality in education

Quality is measured and assessed in many ways, but common in all of them is the necessity to decide what is worth measuring. Performance indicators target particular areas in order to measure quality, but not others. We asked which types of indicators teachers find important. Table 3 shows the ranking of teachers’ responses to the presented alternatives.

Table 3: Ranking of performance indicators teachers find important

Core skills seem to be very important as performance indicators according to these teachers. The respondents think that it is important that the pupils reach the goals when it comes to reading, writing and math. Truancy/absence, health, and attainment in national tests appear to be more important to measure than social injustices.

Together these answers point at views on education quality and how to measure it, that are less concerned with social inequalities and engagement in a democratic society than with sustaining an individualistic development of each individual child.

Views on QAE

The respondents’ general view of QAE is mostly positive. Almost 80% of the respondents agreed/agreed strongly with the statement “In general I feel thatquality evaluation is beneficial for teaching”. Furthermore, the teachers’ responses indicate that evaluations can be used to address some real problems - a statement that 45% partly agreed and 36% agree fully with. There is nonetheless a negative side to evaluation and quality assurance/assessment on the whole according to a third of the respondents. They think it is a waste of time. But there is a contradictory tendency in the views of respondents working in schools with a mix of middle class and working class students to not think of evaluations as a waste of time. Most of our respondents estimated the amount of time spent on QAE to be 1-2 hours per week. When asked to assess the number of hours ideally spent on these activities a majority answered between 2-3 hours. So it seems as if these teachers are quite satisfied with the quantity of time they use to evaluate and work with quality issues.

QAE experiences and practice

When it comes to QAE practice, only a third of the Swedish respondents feel that they have sufficient competence in the field of evaluation and quality assurance. There also seems to be a lack of support from the municipal centre in these matters since almost half of the teachers said they did not get such help.

Despite this lack of competence, the vast majority of the teachers (83%) said that they willingly share their evaluations with colleagues, and around 40% also use evaluation results in their communication with parents.

Do these teachers find any particular evaluation instruments useful in measuring the kind of quality they find most relevant? The Swedish survey did not include specific questions about quality and evaluation instruments as such. We did however try to indirectly check this relation by correlations between variables on teachers’ views on quality and their opinions about various QAE activities. There were no significant correlations. Frequencies on variables describing such activities tell us that 82% of the respondents think that evaluation and other evaluative activities carried out by the schools themselves can be used to address real problems (e.g. bullying). 46% also think that internal self-evaluation (by the schools themselves) is better than external evaluation. This does not mean that external evaluation is of less importance to quality in education in the view of these teachers. On the contrary, half of them think that external evaluation increases quality, even though half of them also find it to be a control instrument. This is a bit interesting and we have not yet had the time to look closer into this relation. Do these teachers not bother about external control but find it helpful, or is it not the same respondents who answered like this?