Taking the Evidence

Draft. This version not for citation.

‘Taking the Evidence. Comparing a National Policy Review with more
Conventional Policy Research.

The Review of Initial Teacher Training
Capacity in Wales'.

John Furlong, Hazel Hagger, University of Oxford

and Cerys Butcher, NIACE Dysgu Cymru

Paper presented at the British Educational Research Association Annual Conference, University of Warwick, 6-9 September 2006

'Good government is thinking government. And a good department is a thinking department. Rational thought is impossible without good evidence and to get this we need a revolution in the relations between government and the research community’ (Blunkett, 2000, para 62)

‘Policy making in a modern, complex, plural society like Britain is unwieldy and complex. It is often unscientific and irrational, whatever the claims of policy makers to the contrary’ (Ball 1990: 3)

1 Introduction - Evidence Based Policy Making in England and Wales

The mantra of ‘evidence based policy making’ has been one of the hall marks of British politics since the election of New Labour in 1997. It emerged‘somewhere on the journey from Opposition to Government’ (Soulesbury 2001) and found its first full expression in the 1999 White Paper ‘Modernising Government’.

This growth of interest in evidence based policy making has in turn had a profound impacton the position of the social sciences in Britain. There has been an emergence of a ‘new utilitarianism’ (Young et al 2002) best expressed by the then Secretary of State for Education, David Blunkett, in his speech to the ESRC in 2000. ‘We need to be able to rely on social science and social scientists to tell us what works and why and what types of policy initiatives are likely to be most effective’ (Blunkett 2000). Others have referred to a ‘new social contract’ for social scientists; increased funding and opportunities for policy relevant research but matched by a growing emphasis on accountability, relevance and value for money (Demeritt 2000).

At the level of rhetoric at least, the evidence based policy movement is about challenging old strategies for policy formation, based as they were either on ideology or elites. This aspiration is well captured by Soulesbury (2001) of the ESRC UK Centre for Evidence based Policy and Practice, when he writes

‘…..there is something new in the air which gives both a fresh urgency and a new twist to the issues around evidence-based policy. To my mind the key factor is the shift in the nature of politics; the retreat from ideology, the dissolution of class-based party politics, the empowerment of consumers’ (9).

The rhetorical appeal of social science in providing part of the evidential base forpolicy is therefore that it indeed can be seen as scientific. It purports to stand outside the political process, giving policy advice based on rational evidence rather than ideology or sectional interest.

Significantly, the growing interest in the evidence based movement has not beenconfined to England. Wales, with its newly achieved democratic devolution has also embraced the role of evidence in the development of policy. In an early policy speech in December 2002, the First Minister, Rhodri Morgan, referred to a ‘new pluralism’ in policy making in Wales, with the Welsh Assembly seeking a broader engagement with civil society in Wales. As Daugherty (2003) suggests, this has been exemplified by the establishment within the civil service of a cross-departmental policy unit and by overtures to higher education in Wales to make research expertise available to government in Wales.

This commitment to the role of evidence in policy development has been particularly evident in the field of Education where there has been an emergence of an increasingly different set of educational policies from England.As G. Rees (2004) notes, those policies are based on

‘deep-seated social democratic virtues, albeit in changed and unfamiliar circumstances: equality of opportunity through universal provision; the necessity for the state’s role in ensuring this; the rights and obligations of citizenship; partnership between the central state, local education authorities and professional groups; and so on’ (p31)

As a result, to a greater extent than at any time previously, commentators are pointingto the emergence of a truly national education system in Wales(Egan and James 2003, Phillips 2003,G. Rees 2002; G. Rees 2004). And in the development of that distinctive agenda, the use of evidence, it is claimed, has been central. As Jane Davidson, Minister for Education and Lifelong Learningstated in the Introduction to her most recent policy document ‘Learning Country 2’ (NAfW 2006a).

‘Our policies will also continue to be evidence based. Over the last five years we have drawn both on practitioner expertise and high quality educational research. We commission independent evaluations of all our major policy initiatives. Leading research bodies have complimented us on this approach. (We) will continue to operate in this way.’

However, in one regard, the position of Wales is profoundly different from that of England and that is in relation to educational research capacity. AsFurlong and White (2001) demonstrated early in the life of the new Assembly, research capacity in the field of education is extremely weak. Unlike England, or indeed Scotland, the Welsh Assembly Government’s aspiration for research based evidence isnot matched by the capacity of higher education or indeed any other bodies within Wales to provide that evidence through conventional social science.

It is perhaps for this reason that rather than commissioning significant amounts of formal research, over the last five years, the Welsh Assembly Government has adopted a different strategy for assembling relevant evidence for its policy development. That strategy has been the commissioning of ‘Independent Reviews’. So far there have beena number of independent reviews in education in Wales: ‘Devolution of the Student Support System and Tuition Fee Regime in Wales’ (2005) chaired by Professor Teresa Rees, ‘Learning Pathways Through Statutory Assessment: Key Stages2 and 3’ (2004) chaired by Richard Daugherty and the recent ‘Independent Review into Part Time Higher Education Study in Wales’ (2006) chaired by Dr Heather Graham. Each Review hasthus been commissioned to focus on a key area of policy within the Welsh educational agenda. The focus of this paper is on one further independent Review: The Review of Initial Teacher Training Capacity in Wales, which was led by the current authors.

But if such Reviews are a key means by which research and other evidence is assembled on behalf of the Welsh Assembly Government, it is important to understand how they work. Each, it would seem has had a common structure. They have been chaired by an independent academicand have worked with a Support Group representing both lay and expert opinion from different relevant communities. Each has also demonstrated a strong commitment to the broadest possible consultation from across Wales. T. Rees (2002) reports having consulted over 1500 individuals and groups in the course of her first Review on student funding, while we took evidence from over 150 individuals and 50 organisations. Where the Reviews would seem to vary is the extent to which they have themselves commissioned original research(our Review concentrated more on assembling a range of existing evidence and opinion from within and outside Wales) and on the degree of support provided by seconded civil servants (our Review had relatively little such support because of staffing shortages in the relevant unit).

The processes involved are therefore rather different from conventional policy research. For one thing, what counts as ‘evidence’ is more broadly conceived. For example, our evidence included, among other things, highly technical reports from the Higher Education Funding Council for Wales (HEFCW), the General Teaching Council for Wales (GTCW)and Estyn (the inspectorate), and hand written letters from individual members of the public with a personal interest in teacher education. There is also an explicit expectation that Reviews will themselves move on from evidence to policy recommendations; in this regard, going further than conventional social scientific research.

But despite these differences, we can see that in many ways, the aim of such Reviews, particularly from the policy makers’ point of view, is to do the same thing as commissioned research. Their appealis that they appear to be at one at the same time both democratic, consulting a wide cross section of opinion, and ‘independent’; in this sense they may be seen as sidestepping politics, moving policy making beyond ideology or sectional interest. This is how our study was presented in public. For example when first discussing our Final Report in the Education and Life Long Learning Committee, the Minister said

We are (today) discussing an independent report, on which we are now taking evidence so that the Assembly Government can give a response. We could not possibly have given a response prior to the publication of an independent report. (NAfW 2006b)

But is this actually how these Reviews work; are they really ‘independent’; are they really rational and scientific, standing outside politics? So far we have had two retrospective accounts of these Welsh Reviews –on the first Rees Review (T. Rees, 2002 and Stroud 2002) and one on the Assessment Review (Daugherty 2004). Both of these retrospective accounts present themselves as success stories and as being entirely rationally based. As T. Rees, (2002) states

‘In many senses, the investigation resembled a policy-oriented research project: we identified the questions, we collected evidence, we analysed it, we discussed possible interpretations and their implications with key actors, we came to some conclusions and we made our recommendations’

The only difference from conventional research that T. Rees notes is the great enthusiasm with which people gave evidence!

Both Rees and Daugherty therefore continue the ‘story’ of rational evidence based policy making. And while we could indeed tell a similar story, in this paper we want to do something different; we want to highlight some of the complex political processes that went on in and around our Review; processes which suggest that it was anything but outside the political arena. In this sense we would like our retrospective account to contribute to the growing literature on how research and policy processes actually interrelate; we want to contribute to a greater understanding of what Shulock (1999) calls the ‘paradox of policy analysis’, a paradox arising from‘a mismatch between notions of how the policy process should work and its actual messy, uncertain, unstable and essentially political realities’. (p218)

Our experience is that the Review process, just like any other strategy for assembling evidence, is deeply involved in that paradox . Facing up to those contradictions and tensions in the process is, we believe, an important step in assessing the real value of incorporating the Review process within a modern democratic state.

2 The Review Process

There are many different stories that could be told about the way in which our work was ‘political’; the Review found itself inserted into many different political discourses (Foucault 1984; Olssen et al 2004). Some we experienced at a personal level (for example a chance meeting in Tesco’s with a senior civil servant who wanted to discuss our Interim Report; informal discussions with key informants in the pub and in meetings about other things). Politics at this level is perhaps the product of Wales being a small country, with many cross cuttingpersonal and professional networks. But we were also aware that the Review was a very small part of many larger political discourses – the relationship between the Minister and the Higher Education sector in Wales; the relationship between the Higher Education Funding Council and the National Assembly; the relationship between Wales and key bodies in England such as the Department for Education and Skills (DfES) and the Training and Development Agency for Schools (TDA). Of the many stories that could be told, we want to concentrate here on just three that illustrate different ‘moments’ in the Review process. They are the setting up of the Review; the way in which we attempted to engage the Higher Education sector in Wales in the Review process; and finallyourrecommendations and the responses to them.

2.1 Setting up the Review.

There are two key issues that we wish to highlight in the setting up of the Review. The first concerns the Terms of Reference and the second concerns the membership of the Support Panel. Both of these were highly significant in shaping the way we worked and perhaps in the final outcomes of our Review.

The idea that there should be a Review of initial teacher education had been discussed for a number of years amongst the teacher education community in Wales; certainly since the establishment of democratic devolution in 1999. Estyn, in successive reports, had expressed concern that although most higher education based provision was ‘good’, only a very small proportion was ‘excellent’; they had also expressed concern about the variability in school based provision. (Estyn 2003; 2005). Some schools and LEAs that had begun experimenting with employment based training though the Graduate Teacher Programme were keen to expand that system from its current limited number of 50 places per year. And the Welsh Language school system had for many years expressed concern about the numbers of teachers willing and able to teach through the medium of Welsh, with particular concern being expressed about Welsh second language teaching in primary schools.

Perhaps the largest number of concerns however, were expressed by those in Higher Education. In the early 1990’s, initial teacher education inWales, along with that in England, had been swept up into the radical reforms of the day. A system of teacher training (based on a model of government imposed ‘standards’ or competencies, to be acquired in large part through practical school experiences, combined with a strong inspection regime, with the threat of course closure for courses that failed to achieve the appropriate standard) had been imposed from London with no consultation. While there were and are some important differences from the English model in the management of the system ( for example, the absence of a TDA or its equivalent; a slightly ‘lighter touch’ inspection regime;more regular contact between the key players in the system – the inspectorate, the funding council, Welsh Assembly Government officers and representatives of HEIs) the underlying model of teacher training itself was and is the same system that was imposed by the Westminster Conservative government in 1992.With the emergence of a different agenda for manyaspects of educational policy in Wales following democratic devolution, there was an appetite and a hope amongst university based teacher educators for a fundamental review of the system. From their point of view there were a number of concerns that might have merited review: the apparent under funding of the system which in some institutionsappeared to be resulting in the progressive casualisation of staffing; the difficulties of establishing effective school partnerships in some parts of Wales; and perhaps most fundamentally of all, the continuing lack of recognition of a distinctive role foruniversitiesin teacher education rather than simply inteacher training.

However, in the end, it was none of these concerns that actually precipitated the establishment of the Review; instead it was the issue of the numbers of teachers being trained, a particular concern for the Welsh Assembly Government itself.

For several years prior to the establishment of the Review there had been growing concern about the apparent difficulty that some newly qualified teachers in Wales, especially primary teachers and especially those who had trained in the south of the country, were having in finding a post. Throughout the early 2000’s there were regular letters and comments in the Welsh press about the difficulties many newly qualified teachers were facing in finding permanent posts; there were reports of newly qualified teachers not being able to complete their statutory induction which at the time had to be undertaken in a full time post within one year of qualifying. In response to these concerns, the NUT was running a campaign to encourage Wales to establish an entitlement for all newly qualified teachers to a one year placement in order to complete their induction, as had been introduced in Scotland. The initial response of the Welsh Assembly Government was to change the regulations, allowing newly qualified teachers 5 years rather than one in which to complete their induction. HEIs were also given advance warning in 2004 that there would be a 5% cut in primary numbers from September 2006. Despite these moves, concerns about the oversupply of primary school teachers remained. And when combined with demographic forecasts for the future reduction in the school population,it was becoming apparent that in the future, the position might get worse rather than better. From the point off view of the Welsh Assembly Government, this was a nettle that had to be grasped.

There were therefore many competing expectations and aspirations about what questions a Review of Initial Teacher Training Provision in Wales might address. Initially, discussion withthe Minister and her Civil Servants suggested that the Review should confine itself entirely to the issue of numbers of teachers needed; conversations implied that the Review should be primarily statistical in nature with clear recommendations about percentages and location of cuts needed over the next 5 to 10 years. Behind this question was the issue of whether or not it was technically possible or indeed sensible, for Wales only to focus on producing its own teachers. Our response was that addressing the issue of numbers was clearly very important but that simple percentage cuts were unlikely to be successful in that they could very easily destabilise the system; the complexities of staffing both primary and secondary courses had to be taken into account as well as the implications of any cuts in numbers for other forms of provision, such as Continuing Professional Development(CPD) and research. We were also keen to look in more detail at the substance of current provision and suggested that we address the issue of ‘quality’. This proposal was warmly accepted with two provisos. Firstly, that we should not look at the issue of ‘partnerships’ as HEFCW had recently commissioned its own Review on this matter.Secondly, that we should not address the issue of ‘content’. Following a period of consultation, officers were only now completing their latest revisions of the revised ‘standards’ for ITT and there was clearly no appetite (or indeed capacity) to entertain further discussion. The opportunities for asking many of the more fundamental questions proposed by those in higher education were therefore effectively curtailed. However, further discussion suggested that we might also examine the relationship between initial training and induction; there was also agreement that we should address the issue of the strategic management of the system.