University of Durham

Joint Funding Bodies Review of Research Assessment: Invitation to Contribute

Recognising that the purpose of this invitation is to generate ideas and insights as the first stage of consulting stakeholders on the future of research assessment, the following response is submitted on behalf of the University of Durham and has been prepared following individual consultation with a number of our key University members.

We are very pleased to have been given this opportunity to contribute to the debate on research assessment in higher education, which is of fundamental importance to the University’s research community. Should you have any queries about this response please contact:

Wendy Harle

Research & Economic Development Support Service

Telephone: 0191 374 7680

.

Summary

Consideration has been given to the four approaches to research assessment and to the cross -cutting themes and key points raised relating to each are summarised below.

Overall, we would seek an expansion of the options to be considered under the “expert review” model for further exploration and debate. An element of external review is essential in any research assessment model. External appraisal and measurement contributes to three essential characteristics of a research assessment method: transparency, equity and robustness. However, expert review/external assessment does not now and should not in the future dictate research policy to institutions. The critical mass of the research community and the quality of their research and vision for the future must shape research policy.

APPROACHES TO RESEARCH ASSESSMENT

Expert Review

Overall, it is felt that expert review has the broad support of the academic community, and can be sensitive to the circumstances of individual departments. Experts, nominated as they are at present, should have the trust of the subject communities but it is recognised that there is an inevitable subjective element to this method of assessment.

I. Assessment should be primarily retrospective. An ideal would also include some elements of prospective assessment. Prospective assessment would necessitate consideration of future plans and so must contain an element of self-assessment in relation to vision and planning. This poses great difficulty for reviewers to determine and assess what is realistic.

II. Research income, research student numbers, numbers of research degrees awarded (or completion rates) etc should be considered by assessors.

III. Assessment should be made at subject level with autonomy given to institutions to determine whether departments or other forms of organisation/groupings are submitted for assessment.

IV. Given the above, it is unavoidable that assessment should be based on subject groupings. Real consideration needs to be given to cross disciplinary themes.

Algorithm

Elements of the algorithm approach to research assessment may, in certain circumstances, be considered for areas of science research but are not appropriate for arts and humanities research. Given this and the comments below, it is felt that this method would not provide reliable results and would affect the behaviour of Universities in attempts to improve their performance. Strong reservations are expressed as to the validity of this option as a method for research assessment.

I. It is not acceptable to assess research entirely on the basis of metrics.

II. The metrics available that could be used are citation indices, which would be inadequate.

III. For the above reasons it is felt that the available metrics could not be combined to provide an accurate picture of research strengths.

IV. If funding were tied to the available metrics, Universities attempting to influence the figures could lead to certain discipline and research areas being seriously damaged.

Self-assessment

Overall there are extremely strong reservations as to the validity of this method of research assessment. Self-assessment would lack credibility and would have to be backed up by some form of expert review. In answering the questions raised, only existing quantifiable data could be subject to retrospective self–assessment and institutions would have to be bound by the use of universal criteria in the self –assessment process. The only way to credibly validate self-assessment would be by means of expert review.

Historical ratings

The historical ratings method of research assessment provides no measure of current performance and is not an acceptable method. It is felt that this method would have am extremely detrimental effect on the morale of institutions. Institutions rated low in the initial baseline would have little incentive to improve whilst those that were high rated would not have sufficient incentive to maintain a good performance.

CROSS-CUTTING THEMES

What could/should an assessment of the research base be used for?

Assessments of research should be used by funding councils to inform funding and by research councils to determine the distribution of research studentships. However, funding councils should be much more explicit about how the money should be used and agreement should be reached with the institutions on this. Alongside this transparent agreement, mechanisms should be established by funding councils to ensure that the money distributed as a result of any form of research assessment is spent on the intended and agreed purpose.

How often should research be assessed?

This is a key consideration and much more consultation on the frequency and method of assessment is required. Rolling programmes of assessment could be worth consideration but only after the advantages and disadvantages of being involved in a rolling programme of assessment are clearly identified, documented and fully considered by institutions.

What is excellence in research?

Excellence in research makes a fundamental difference to thinking in the subject area and can bring about new approaches in the subject. However, one problem with the previous research assessment exercise has been that it implies just three divisions into which research can be categorised - international excellence, national excellence, and sub-national. These are then mapped onto the seven grades by complex descriptors involving words such as ‘a majority’, ‘some’ and ‘up to half’, which are capable of diverse interpretation.

Should research assessment determine the proportion of available funding directed to each subject?

It must be recognised that the infrastructure to support certain subjects can be widely different, in particular between the arts and the science subjects. This question requires much more detailed consideration, taking into account the nature and requirements of individual subject areas, before subject pots should be considered.

Should each institution be assessed in the same way? / Should each subject or group of cognate subjects be assessed in the same way?

Institutions should be assessed in the same way and present submissions in accordance with the assessment framework. More onus to be placed on those doing the assessment to be aware of differences between institutions and take these in to account as part of the assessment process.

Equality of treatment for all groups of staff

It is felt that the past assessment exercises have not consciously discriminated against any groups of staff, except, quire rightly, those who produce poor quality research. Consideration should however be given to the following circumstances:

  • Institutions should be allowed to submit members of staff who are new to the profession with fewer publications than experienced staff, providing those new members of staff are within their first three years.
  • Key technical staff, especially in the science subjects, who are at a senior level are currently not recognised in the assessment exercise and are an important component of research and contributor to research output.

1