CHEM 519 Guidelines for Peer Review
Overview: Each paper will be scored and reviewed by a primary and a secondary reviewer. During the meeting of the review panel, the papers will be discussed and comments will be summarized to highlight the strengths and weaknesses.
Assessment: Dr. Campbell will mark your reviews (/3) and also give a participation mark (/2) for your involvement in the grant panel discussion. These marks will be added to the score assigned to your proposal (/5) and Dr. Campbell’s mark on your proposal (/10) to give a total score out of 20.
1. Before the Grant Panel Discussion
Review your two papers, writing up a one-paragraph summary of each proposal, including merits and perceived problems, and assigning a score from 0-5 (to one decimal place). Your review of the primary grant should be more thorough than your review of the secondary grant.
Use the Reviewer score sheet for your primary and secondary reviews.
Your review should be a paragraph (~250-300 words) that highlights the strengths and weaknesses of the proposal in light of the criteria listed below. This will be the summary that you will verbally present to the panel to support the initial score that you have submitted. Keep in mind that this will eventually be given to the applicant and thus should be constructive criticism. The criteria are:
a. Significance. Does this study address an important problem? If the aims are achieved, how will scientific knowledge or clinical practice be advanced? What will be the effect of these studies on the concepts, methods, technologies, treatments, services, or preventions that drive this field?
b. Approach. Are the conceptual framework, design, methods, and analyses adequately developed, well integrated, well reasoned, and appropriate to the aims of the project? Does the PI acknowledge potential problem areas and consider alternative tactics?
c. Innovation. Is the project original and innovative? For example: Does it challenge existing paradigms or clinical practice or address an innovative hypothesis or critical barrier to progress in the field?
d. Quality of the written document. Note that the guidelines I provided for writing the document was not written in stone and there is some flexibility. The proposal should not be evaluated only on the basis of how closely it follows those suggestions. It should be judged on its own merit. That is, the author may have a different style that is equally effective. However, things like granting agency-imposed page limits and formatting requirements do matter and these rules must be followed. Is the document well written? Is there proper grammatical usage and no spelling mistakes? Have figures been used appropriately? Has the author made the best possible use of the space provided to them? Sometimes a panel might think that more details should have been provided, but this is only a valid criticism if there is space to add these extra details. That is, if the document is so well written that none of the existing text could be cut, and the authors have used all of the space provided, it’s not reasonable to have expected them to put even more into the document.
You must also read the grants on which you are neither the primary or secondary reviewer. Your goal is to have an approximate ranking of all the grants in your section and to be able to intelligently comment on each one and defend your ranking.
2. At the Grant Panel Discussion
“Welcome to Fight Club. The first rule of Fight Club is: you do not talk about Fight Club. The second rule of Fight Club is: you DO NOT talk about Fight Club!”
Tyler Durden: Fight Club (1999)
These two rules also apply to Grant Panel Discussions. They must be kept confidential!
Each proposal will be dealt with in turn. For each proposal, the scores of the Primary and Secondary reviewers will be revealed. The Primary reviewer will then present a summary of the proposal, which will then be followed by the Secondary reviewer’s comments. You are welcome to just read what you have written and you do not have to submit hard copies to the rest of the panel. All panel participants are then invited to comment on and discuss the proposal, with the Primary reviewer acting as moderator of the discussion. Any additional comments that the panel as a whole feels a need for the author to have will be added to the primary/secondary reviews as part of the “Summary comments”. The primary reviewer, as moderator, will keep track of these comments. If a member of the panel is the author of the grant or if they have a ‘conflict of interest’ due to a personal connection with the author of the grant, they will leave the room during the discussion of their proposal.
Following the discussion, the Primary or Secondary reviewers must adjust their initial score and settle on a single consensus score. All panelists will then individually decide on a score for the proposal +/- 1 from the consensus score. Please resist the urge to score everything average! This is not helpful!
Use the Panel Score Sheet available on the website.
These scores will not be revealed until the end of the panel. At this point a final score will be calculated for each proposal by taking the average of the scores provided by all members of the review panel.
3. After the Grant Panel Discussion
It is the responsibility of the primary reviewer to submit a final review report that contains the primary review, the secondary review (you will be responsible for getting a copy from the secondary reviewer), the summary comments, and the final overall score for the grant. Make sure it is completely anonymous! This document will be returned to the author.
Use the Review Summary sheet available on the website.
A summary statement is not an exhaustive critique. Instead, it recaps the highlights of the review discussion as far as it progressed, providing general feedback. This may or may not be the same information that is found in the reviewer reports. The overall goal is to help the author understand why they got the score that they did and to help them improve their proposal in the future.This statement should consist of a few sentences which provides the author of the grant with a bit of insight about how the panel felt about their proposal overall. It should point out the major positive points (i.e., The panel felt that the goals of the work were important and the proposed approach was imaginative) and the primary concerns (i.e., The panel felt that aim 2 should have been better supported with additional details of the experiments that will be undertaken). The summary should not contain any additional comments on relatively minor issues.
Score: working range 0-5 (5 best; 0 worst; 3.5-3.9 typical average in practice)
Rating Range / Benchmark4.5 to 4.9
Outstanding / Extraordinary optimization of: challenge to the candidate, scientific importance and feasibility of completion during the funding period.
An ideal project that is faultlessly outlined
4.0 to 4.4
Excellent / Excellent optimization of: challenge, importance and feasibility.
A highly suitable project that was superbly outlined.
3.5 to 3.9
Very good / Strong optimization of: challenge, importance, and feasibility.
A very suitable project that was very clearly outlined.
3.0 to 3.4
Good / Good optimization of challenge, scientific importance and feasibility.
A suitable project that was well outlined.
2.0 to 2.9
Average / Mediocre
1.0 to 1.9 / Below average
0 to 0.9 / Not acceptable
Adapted from documents provided by Andrea Gorrell, UNBC
CHEM 519 Reviewer Score Sheet
Title:
Author PIN:
Reviewer PIN:
1° or 2° or reader (circle one)
Significance /5
Approach /5
Innovation /5
Document quality /5
Overall /5 (not an average)
Paragraph review for 1° and 2° reviews (or notes for others):
CHEM 519 Review Summary Sheet
Title:
Author PIN:
Final score as decided by the review panel: /5
1° reviewer comments:
2° reviewer comments:
Summary Statement
CHEM 519 Panel Score Sheet
Your PIN: ______