Appendix 1

Research AssessmentCriteria[1]

Review panels are asked to evaluate research performance, relative to the appropriate international disciplinary norms, under the following criteria:

  • Published output
  • Other research outputs
  • Peer esteem
  • Research income
  • Postgraduate research education

Panels are also asked to provide an overall assessment of the Theme derived from an integrated overall assessment of the research activity of the Theme as a whole.

  1. Quality Levels

Panels will recognize the diverse range of disciplines represented by the Themes assignedtothem.Setoutbelowarebroadparametersfortheassessmentofthe qualityof research foreachoftheresearch assessment criteriawithinwhichindividualpanelsmay exerciseadegreeofvariation. The levelsrefertoqualitystandards ofscholarship thatare the disciplinary norm within the international academiccommunity.

Level5 / Qualitythat isofworld-leadingstandardby international disciplinarynorms.
Theresearch workoractivitywill beoutstanding,displayinga veryhigh level oforiginality,significance tothe discipline and rigour;it will be innovative andpotentiallyagenda-setting in researchand/orpolicy fields.
Level4 / Qualitythat isofexcellentstandard by international disciplinarynorms intermsof originality,significance and rigour comparable with suchworkinternationally.
Theresearch workoractivityhashad or islikely tohave asignificant impact onresearchand/orpolicyagendas.
Level3 / Qualitythat demonstratessignificancetothe discipline and rigourtoa
goodstandardby international disciplinarynorms.
Theresearch workhashad or islikelytohave asignificant impacton researchand/orpolicyagendas.
Level2 / Qualitythat demonstratessignificanceto the discipline and rigourtoan
adequatestandardby international disciplinarynorms.
Theresearch workoractivityhasonlyhad oris likelyto have a marginal impact upon existing paradigmsandagendas within the discipline.
Level1 / Qualitythatis poor and fallsbelowthe adequatestandardofrecognizedworkwithin the discipline.
Theresearch workoractivityhashadno impact nor isit likelyto have animpactupon existingparadigmsand agendaswithin the discipline.

‘World-leading’ quality denotes an absolute standard of quality in each Theme. ‘World-leading’, ‘excellent’, ‘good’, ‘adequate’ and ‘poor’ in this context refer to quality standards. They do not refer to the nature or geographical scope of particular subjects, nor to the locus of research nor its place of dissemination.

For the purposes of determining the overall Theme rating and because of the differences that exist between the research assessment criteria, appropriate standards will be employed for each one. Individual outputs will be evaluated againstoriginality, significance and rigour while total output will be evaluated against extent, diversityand quality. Peer esteem will likewise be evaluated against extent, diversityand quality international disciplinary norms.Research income will be evaluated relative to the funding levels for the specific Theme and cognate disciplines available to researchers in Ireland.

Each panel is asked to rate each Theme under four major criteria as follows.

  1. PublishedOutput

The panel is asked to indicate the percentage of the selected published outputs produced by the Theme that would be categorized according to the scale of quality levels provided in section 1 above in terms of originality, significance and rigour by current international disciplinary norms. Panels will be required torateeachoftheselectedpublicationsforeachmember of the theme–it is suggested that eachpublicationberatedbytworeviewers.

I.e.:

5:Selectedpublicationisoutstandingintermsoforiginality,significanceandrigour bycurrent international disciplinarynorms

4:Selectedpublicationisexcellentintermsoforiginality,significanceandrigourbycurrent international disciplinarynorms

3:Selectedpublicationisgoodintermsoforiginality,significanceandrigourby currentinternational disciplinarynorms

2:Selectedpublicationisadequateintermsoforiginality,significanceandrigourby currentinternational disciplinarynorms

1:Selectedpublicationispoorintermsoforiginality,significanceandrigourby currentinternational disciplinarynorms

Quality Level
5 / 4 / 3 / 2 / 1
% of published output of Theme
  1. Other ResearchOutput

The panel is asked to indicate the percentage of the selected research outputs produced by the Theme that would be categorized according to the scale of quality levels provided in section 1 above in terms of originality, significance and rigour by current international disciplinary norms. Panels will be required torateeachoftheselectedoutputs foreachmember of the Theme–it is suggested that eachoutputberatedbytworeviewers.

I.e.:

5:Selectedoutputisoutstandingintermsoforiginality,significanceandrigour bycurrent international disciplinarynorms

4:Selectedoutputisexcellentintermsoforiginality,significanceandrigour bycurrent international disciplinarynorms

3:Selectedoutputisgoodintermsoforiginality,significanceandrigourby currentinternational disciplinarynorms

2:Selectedoutputisadequateintermsoforiginality,significanceandrigour by currentinternational disciplinarynorms

1:Selectedoutputispoorintermsoforiginality,significanceandrigour by currentinternational disciplinarynorms

Quality Level
5 / 4 / 3 / 2 / 1
% of research outputs of Theme

Each panelwillbefurther requested toallocateeach Theme member’stotalresearch, including publication,outputinthelast ten years to one of fivelevels.

I.e.:

5:Totaloutputisoutstandingintermsofextent,diversityandqualitybycurrent international disciplinarynorms

4:Totaloutputisexcellentintermsofextent,diversityandqualitybycurrent international disciplinarynorms

3:Totaloutputisgoodintermsofextent,diversityandqualityby currentinternational disciplinarynorms

2:Totaloutputisadequateintermsofextent,diversityandqualityby currentinternational disciplinarynorms

1:Totaloutputispoorintermsofextent,diversityandqualityby currentinternational disciplinarynorms

QualityLevel
5 / 4 / 3 / 2 / 1
%ofresearchersinTheme
  1. Peer Esteem

In assessing the quality of esteem indicators and defining a profile, each panel will make a judgement about the level of impact and recognition of the members of the Theme on research, scholarship, policy and practice; and the contributions made to the sustainability of scholarly academic activity, and of the appropriate research and policy communities. This assessment will be based on the Thematic Champion’s statement in the submission describing the esteem within which the Theme as a whole is held and on a listing for all members of the Theme of significant indicators of esteem presented in their profiles. These criteria will lead to the definition of an agreed profile based on the quality level descriptors below. The panel is asked to indicate the percentage of staff whose overall esteem based on the submitted profile is judged to be at each of the five quality levels:

Evidenceofpeeresteem,acrossthecareerasa whole,includesresearchoutput (including significant prior research activity,e.g., seminal orwidely-citedpublications), Fellowships,Honours,International Prizes, InvitedPlenaryPresentationsatsignificantdisciplinary conferences, service on appointmentpanelsatotherinstitutions,external examining, refereeing/editingofjournalsetc. The ratinggiventoanindividualshouldreflecttheleveloftheindividual’sachievementsacrosshis orher researchcareer asa whole.

Two reviewerswill assesseach individual researcherusingoneof thefollowingratings:

5:Acrosstheircareertodate,individualisoutstandingintermsofpeeresteemby currentinternational disciplinarynorms.

4:Acrosstheircareertodate,individualisexcellentintermsofpeeresteemby currentinternational disciplinarynorms.

3:Acrosstheircareertodate,individualisgoodintermsofpeeresteembycurrent international disciplinarynorms.

2:Acrosstheircareertodate,individualisadequateintermsofpeeresteembycurrent international disciplinarynorms.

1:Acrosstheircareertodate,individualispoorintermsofpeeresteembycurrent international disciplinarynorms.

Thepanelwilldeterminethequalityprofileforeachindividualresearcher.

Quality Level
5 / 4 / 3 / 2 / 1
% of members of the theme
  1. Research Income

Each member of the Panel isaskedtogive asinglequality level forthecollective research- relatedincome of the Themebasedontheir professional judgementoftheresearch area, taking intoaccount the research landscaperelevanttoresearchers in Ireland asdescribedin the briefing documentsprovided, based on the quality level descriptors.

I.e.:

5: Total researchincomeisoutstandinggiven the researchfundinglevelsfor this and cognatedisciplinesavailable to researchersin Ireland.

4: Total researchincomeisexcellenttheresearchfunding levelsfor thisand cognate disciplinesavailable toresearchers inIreland.

3: Total researchincomeisgoodgiventheresearchfunding levelsforthisand cognate disciplinesavailable toresearchers inIreland.

2: Total researchincomeisadequategiven theresearchfundinglevelsfor thisand cognate disciplinesavailable toresearchers inIreland.

1: Total researchincomeispoorgiven theresearchfunding levelsforthisand cognate disciplinesavailable toresearchers inIreland.

Themodal (mostfrequentlyoccurring) ratingacross reviewerswill be taken astheresearch- relatedactivityscore. The higher ratingwill be preferred where thedistribution ofratingsis multimodal.

  1. Postgraduate ResearchEducation

Panel members are asked to each give a single quality level for the collective activities related to postgraduate education. This rating should reflect the professional judgement of the reviewers concerning the quality level descriptors provided, taking into account the number of students graduate or studying for research degrees, culture of support (i.e., arrangements for supervision), research training environment, and opportunities available for research students and subsequent career profile.

Each member of the Panel is asked to give a single quality level for the collective postgraduate training related activities of the theme based on their professional judgement referenced to the following quality level descriptors, which should summarise the overall culture and standard of postgraduate research education present.

I.e.:

5: Postgraduate research education is outstanding by current internationaldisciplinary norms

4: Postgraduate research education is excellent by current international disciplinarynorms

3: Postgraduate research education is good by current international disciplinarynorms

2: Postgraduate research education is adequate by current international disciplinary norms

1: Postgraduate research education is poor by current international disciplinary norms

The modal (most frequently occurring) rating across reviewers will be taken as the research- related activity score. The higher rating will be preferred where the distribution of ratings is multimodal.

  1. Overall Assessment of the Theme

The Panel is asked to give a single quality score for all the collective research activities of the Theme based on the professional judgement of the peer reviewers.

  • A. International reference point for research in the discipline. Clearly a world leader. The majority of research activity as assessed under the various criteria is of an excellent standard of scholarship.
  • B.Internationally competitive in a majority of areas of its research. A significant player internationally in the field.Some of the research activity as assessed under the various criteria is of an excellent standard of scholarship and virtually all other research is of a good standard of scholarship
  • C. Internationally competitive in only a minority of its research. Major part of research of national importance.The majority of research activity as assessed under the various criteria is of a good standard of scholarship
  • D. None or very little research of international significance. Some research of national importance.The majority of research activity as assessed under the various criteria is of a fair standard of scholarship.
  • E. Almost no research of national importance.Some of the research activity is of a fair standard of scholarship.

[1]Derived from UCC Research Quality Review2008-2014Guidelines, April 2014.