Improving the Peer-Review Process for ARC Grant Applications: Reliability, Validity, Bias

"Improving the Peer-review Process for ARC Grant Applications: Reliability, Validity, Bias, and Generalisability"

Professor Herb Marsh, University of Oxford (University of Sydney Visiting International Fellow)

MondaySeptember 28th

Public Lecture 1-2pm

Lecture Theatre 101, Law Bld

Peer review is a gatekeeper, the final arbiter of what is valued in academia, but it has been criticised in relation to traditional research criteria of reliability, validity, generalisability, and potential biases. Despite a considerable literature, there is surprisingly little sound peer-review research examining these criteria or strategies for improving the process. This presentation summarizes a research program based on data from the Australian Research Council (10,023 reviews by 6,233 external assessors of 2,331 proposals from all disciplines). It is comprehensive due to its size, but also because it includes peer reviews from all science, social science, and humanity disciplines, includes peer reviews from assessors from all over the world, includes assessors chosen by the applicants themselves as well as panel-nominated assessors. Using multilevel models, we critically evaluated peer reviews of grant applications and potential biases associated with applicants, assessors, and their interaction (e.g., age, gender, university, academic rank, research team composition, nationality, experience). Peer reviews lacked reliability, but the only major systematic bias found involved the inflated, unreliable, and invalid ratings of assessors nominated by the applicants themselves. We propose a new approach, the reader system, which was evaluated with psychology and education ARC grant proposals and found to be substantially more reliable and strategically advantageous than traditional peer reviews of grant applications.

Herb Marsh is Professor in Educational Studies at OxfordUniversity. Prior to his appointment at Oxford he was Research Professor of Educational Psychology at the University of Western Sydney where he served as Dean of Graduate Research Studies (1996-2000) and Pro-Vice-Chancellor (1995-96, 1997). His substantive interests span self-concept and self-esteem, achievement motivation, evaluation of teaching, peer review, and student achievement. His methodological interests are multi-level modelling, longitudinal modelling, meta-analysis, and construct validity. He is widely published with 350 articles in more than 70 different journals, 60 chapters, 14 monographs, and 350 conference papers; and co-edits the International Advances in Self Research monograph series. Professor Marsh's research has consistently attracted external funding, including success on 24 proposals to the Australian Research Council during the last 25 years as well as more recent United Kingdom grants from the Economic and Social Research Council, Higher Education Funding Council for England, and Higher Education Authority. In 2008 Professor Marsh was awarded the ESRC Professorial Fellowship which provides professorial salary, support staff and infrastructure for an extended research program, a highly competitive fellowship awarded to only 3-5 social science researchers across all of the UK.