The future for educational research post-RAE 2001
Paper presented at the Annual Conference of the British Educational Research Association, University of Exeter, England, 12-14 September 2002
Stephen Gorard
Cardiff University School of Social Sciences
[Please note that this paper was created at very short notice as a favour to BERA Council to replace the usual Lawrence Stenhouse Memorial Lecture. It should therefore be treated as a draft for a face-to-face presentation intended to provoke a debate. It refers several times to a ‘brief’ published before the session in the BERA Annual Conference booklet. Readers are advised to look at this brief first.]
Introducing the problem
Some comments at meetings such as UCET Research Committee, at the All-Wales Education Conference I attended recently, the summary of this session in the BERA Programme, and an RAE article by Michael Bassey in RI (2002), all portray a similar problem facing educational research. In short, our genius is, as yet, unrecognised. Educational research is intrinsically harder than in cognate disciplines and, despite our radical and innovative approaches to overcoming these difficulties, educational research is judged more harshly than these other disciplines. 'The RAE panel in Education articulated a particularly demanding set of criteria by which to judge international excellence, more so than Panels in most other subjects'. The unfairness is exemplified by the fact that 'it seems extraordinary that somewhere which includes researchers with the undoubted international reputations of [list of names] should end up with the description [virtually all national, and some international]’ (Bassey 2002, p.1). The RAE results for education in 2002 were hardly better than 1996 or even 1992, and considerably worse than in most other subjects. This means little prestige and less money for UK educational research, and its virtual elimination in some institutions. Yet, at the same time, we are faced with greater demands for evidence-based policy-making and practice. This session has been set up to help find a solution. How can we persuade others (funders, policy-makers, institutions) that the results of the RAE are misleading, and that educational research, as it is, is excellent and worth preserving?
Refining the problem
I feel sure that you all recognise the problem, the current demands, their ramifications and purported solutions, so I recapitulate these only briefly here. UK educational research is under a barrage of attack from funders and policy-makers, from practitioner bodies, local government, and some academics. The permanent secretary at the Department of Education believes that a few minutes thought brings greater insight than most educational research, and asked:
Why is it that educational research has such a bad name among the reasonably informed general public? [Partly] we are so short of good active educational researchers (Wrigley 1976, p.2).
More succinctly, ‘so much rubbish and trash is published’ (RI 1993, p.19). A recent conference on improving training for educational researchers agreed:
on the need for harder definitions of research that would enable better and tighter evaluations to be made… if educational researchers were to avoid the criticism of merely producing self-validating gloss (RI 1975a, p.72).
And was
unanimously of the opinion that there was a need for more information, and a wider circulation of such information as was available, concerning taught courses in educational research (RI 1975a, p.72).
According to a former permanent secretary at the Department of Education, research is
not providing the clarity, vigour and generality policy-making requires but rather, in carrying out in public its internecine debates over methodology, such research puts its whole cause at risk (Broadfoot 1985, p.11).
It is no surprise therefore that research does not usually influence education policy. Because of the call for more relevance of research, there is a danger of political over-control of the research process. So a BERA conference symposium has been set up to discuss
how do we establish an appropriate balance between central co-ordination and independent initiative? (RI 1975a, p.77).
We need to concern ourselves more with research impact and the standards of initial research training (RI 1977). BERA has started a debate over the need for greater dissemination, and whether/how we can make research findings more comprehensible to the public (RI 1977). There is
daily increasing pressure for such research to be tailored to the needs of policy-makers and away from the more general questions of social science per se (RI 1984).
We therefore need to help policy-makers see the conditions necessary for producing good research, and see that a rise in quality might also help utility.
Above all, the growth of HE means that more teaching is required, and the RAE results for education may mean that research ceases in some HEIs. The
policy of the present government is to transfer much training to schools… hence in the next few years there may be a substantial reduction in the amount of educational research being carried out (Bassey and Powney 1992, p.11).
Even from the funding councils ‘the financial support for educational research has been slight in the past’ (Wragg 1982). There is anger that despite the huge public expenditure on education itself, there is a reduction in the funding for research. BERA Council deplores the small amount of Research Council funding for educational research (RI 1980a). Past presidents of BERA have written to the Secretary of State of the Department of Education deploring the likely loss of basic educational research in favour of work only on practical problems.
we wish to express our concern at the present pattern of funding for educational research… There is a tendency, too, to ignore the findings of previous research and to seek to reinvent the wheel… questions can be effectively tackled by a careful synthesis of existing research findings (RI 1988, p.2).
There are too few full-time researchers, and those that exist complain about the lack of career structure and the difficulties of short-term contracts (RI 1980b). There are calls from some on the BERA Council for ‘the concentration of good researchers into certain/fewer institutions… [which] may well prove to be a more efficient structure for the support of high quality research’ (RI 1993, p.15). In their response to the new ESRC frameworks and priorities for research in education, BERA Council admit that this concentration may be unwelcome personally and painful institutionally. A new study of research careers and funding is concerned with the creation and maintenance of a body of social science researchers – known as building ‘research capacity’ (RI 1981, p.21). This study conducted a large-scale survey of educational researchers, and found that many had no formal research training (RI 1981). One third of researchers had no higher degree (one quarter had yet to publish anything). One third of trained researchers did no research, and few made a career of research because of the lack of development and promotion prospects.
There are calls for greater ‘interdisciplinary’ (Wrigley 1976, p.2), debates about the relevance to the UK of a report from the US Department of Education on ‘What works?: research about teaching and learning’ (RI 1987). There are training sessions on the potential of video for the analysis of classroom interaction. School effectiveness and school improvement is the favoured way forward. Greater use of multi-level modelling will provide many of the answers where previous weaker analyses have failed (RI 1986). To help impact NFER has set up a national Register of Educational Research (sponsored by DES/SSRC, RI 1976). There are calls for greater synthesis of existing research, using techniques such as meta-analysis – an ‘innovative approach to the integration of research findings [which] could… provide a much firmer basis for decision-making’ (RI 1982).
Reconsidering the problem
I hope that most of you will have realised that, although all of these things could have been said today, this is all history. BERA history, coming from the pages of Research Intelligence since 1975. The fears about loss of funding, and the closure of departments comes from 1992, three RAEs ago. The debate over what works and the promise of school effectiveness was in 1987. The failure to provide answers for policy and practice because of internecine debates over methodology comes from 1985. The survey of researchers, and the call for capacity-building comes from 1981. Concerns about slight funding are from 1980, and 1982, and 1988. The call for inter-disciplinarity comes from 1976, the same year as the setting up of the ‘first’ NFER register of research findings. The comment about self-validating gloss comes from the very first issue of RI on the formation of BERA 27 years ago. There was no ‘golden age’ of educational research. It would be hard even to sustain an argument that there is not more funding and more research today than there ever has been. What does not appear to have changed is BERA’s take on the high quality of our work, and the unfairness with which educational researchers are treated. Re-reading 27 years of RI over one weekend, I was struck by how several of the same people have been involved with BERA and with high-profile UK educational research over much of that period. They have been writing and re-writing about many of the same substantive and methodological issues, and arguing that others ‘out there’ are not listening. Now this does not make them wrong, but we have to ask whether we want this to continue. Has BERA been listening to the messages from outside? Is there a danger that some of us will still be sitting here in another 27 years bemoaning our lowly funding, and how much harder it is for researchers in education than it is in crime, health services or housing? And arguing interminably with each other about the intrinsic superiority of particular methods over all others for answering all research questions. And hiding our fear of the unknown and the unfamiliar in research behind the vocabulary of epistemology. And will outsiders still be saying of us – but they haven’t answered even the most basic questions of learning? Will we still be trying to run before we can walk?
Before looking for a solution to the problems outlined in the introduction, it is worth considering another perspective. We could assume that the RAE panels all tried to make fair and moderated judgements within the rules they were provided with. They will, of course, have made some errors, as would be consistent with our own experience of any system of assessment – even those we are involved in with our own teaching. But we have no evidence of systematic bias, and no reason to assume that the education panel was any more demanding than any other. Nor that this situation was any worse this time than in 1996 or 1992. In 1992 Michael Bassey reported that ‘the process of peer assessment carried out by the Education panel was done with great thoroughness, fairness and professionalism’ (RI 1993, p.18). Why should we think that things are any different today? There has been a growth in RAE results overall in 1996 from 170 5* UoAs (10.8% staff) and 248 5 UoAs (20% staff) to 326 5* UoAs (18.7% staff) and 755 5 UoAs (36% staff) in 2001 (The United Kingdom Parliament 2002). There has not been anything like the equivalent growth for UoA 68 (meaning, of course, that the improvement for other UoAs is even better than above, on average). There may have been misclassified submissions (in either direction), but surely the simplest explanation overall is that the results for education suggest that we need to improve. To say ‘it’s a fair cop’. But then, presumably, we would have wanted to improve anyway. So the key issue is not how to persuade others that we are, in fact, better than we appear. Rather it is how actually to be better than we are.
But BERA has not traditionally supported those urging the improvement of its own research field. Rather, it has generally sought to evade any evidence of the need for improvement by mis-representing figures, and relying on the lack of critical awareness among its members concerning the use of numbers. For example, in 1996 education was officially ranked 58th out 68 subjects on the basis of the number of researchers in high-grade departments. However, RI (1998) points out that there were 564 educational researchers in schools/departments graded 5 or 5*. This was more than every other subject except five, so according to BERA education is really the 6th best subject from 68. Of course the latter point is nonsense, but similar approaches were tried in 1992 and again in an RI editorial this year. For example, in 1996 there were only 34 researchers in Celtic Studies in schools or departments graded 5 or 5*. In BERA logic, this means that Celtic Studies is far worse than education with 564 such researchers (RI 1998, p.17). But in the last three RAEs education has been consistently one of the largest submissions thereby gaining one of the largest amounts of funding, while Celtic Studies has been one of the smallest. Therefore, education also had many more researchers in poorly-graded departments than Celtic Studies. Presumably, because 6th is more flattering than 58th at least some BERA members agree with the BERA line on this (some colleagues in my own institution still do not understand what is wrong with it). Perhaps this lack of logic and lack of concern for the simple nature of evidence is symptomatic of the kind of problems that we as a field are accused of having.
Further, an overemphasis on the RAE results for UoA 68 leads to sidelining what is perhaps the majority of current educational research. We are in danger, therefore, of neglecting research relevant to education in other disciplines. A recent OECD/NERF analysis suggested that funding for educational research and development in England, while not lavish, was 67% greater as a proportion of expenditure on education than in six comparable countries. It also suggested that 40% of that funding did not go through HEFCE. Lots of work on learning, cognitive development, memory, organisational behaviour and so on will have been submitted as psychology or physiology. Lots of work on differential life chances by class, age, sex or ethic group, and on labour market up-skilling was submitted to sociology or social policy. There is work using GIS, field trials in health education (probably the single largest and best-funded area of educational research in the UK today), economic and political analysis. There are researchers who may conduct only one project relevant to education before moving to another topic. There is continuing education. There is the work of TTA, NFER, LSRN, NIACE and others – not submitted to any RAE panel. There are research councils, charities, government departments, quasi-political bodies, local authorities and institutions both conducting and commissioning education. There are numerous private companies conducting this commissioned research. In my department there are home office projects on crime reduction via schooling, and National Assembly projects on identity formation in Welsh-language schools, and on smoking cessation trials in schools. Projects such as these are not counted as educational research, and their existence suggests that the 40% of R&D funding not from HEFCE is probably an underestimate. I doubt if there has ever been more educational research, as widely envisaged, conducted in the UK. Many of these bodies are expanding, not contracting, their work. Perhaps the problem is that this work is not currently of interest to BERA (or rather ESRA, the English Schools Research Association, as I termed it in Leeds last year) – a point to which I will return later.
Finally, and perhaps most importantly, there is work conducted outside of the UK. The brief for today bemoans the lack of research evidence to help make teaching a research-based profession. In doing so it appears to assume that the DfES and others can only obtain decent evidence from UK research. Rather, using evidence from elsewhere gives us a free-ride, and allows us to focus whatever funding there is into issues of specifically UK concern. Of course, as the past presidents of BERA pointed out in 1993, this may cause personal and institutional disquiet. But that is very different from arguing that research itself will be the worse, or that there will be less research or fewer researchers.
From a wider perspective, therefore, we have no reason to assume that there is less UK funding for educational research than in some other golden age. Even on a narrow vision, such as that of the brief for today, it is incorrect to relate the RAE outcomes to the overall level of funding. And there is, therefore, no excuse to try and demonise our colleagues who comprised the RAE panel. The level of funding for UoA 68 depended on two factors – the funding factor, and the number of researchers submitted. It does not depend on the resultant grades, for these merely determined the relative allocation within education. Those who say ‘the education panel has shot us in the foot’ have misread the financial situation.
The RAE is not in itself a funding mechanism… HEFCE calculates the QR research grant to each university by employing a weighting formula… In addition each subject is assigned one of three weight costs… a final weighting is made for the volume of research being conducted. The principal factor is the number of research active academic staff funded from general departmental funds… The allocation for each UoA is calculated according to the number of active researchers submitted in that subject. This means that a 5* department in a UoA with large numbers of other 5* departments will get less per active researcher than… in a less successful UoA… Universities are not obliged to spend their Funding Council block grant on any particular departments. In theory at least, a university could close down a 5* department and spend the money elsewhere, for example in attempting to bring a low-rated department up the scale in the new RAE (The United Kingdom Parliament 2002).