RFA AI-12-044 Centers of Excellence for Translational Research (CETR) U19
Review Strategy
Introduction.
There are 107 Letters of Intent that have been submitted in response to this announcement. These LOIs represent ~300 different institutions and >1200 individuals for the conflict of interest list. Many have listed 4-6 projects and 2-4 core components.
I anticipate that the average number of projects will be between 4.5-5, and the average number of cores will be 2-2.5. Given these numbers, I anticipate 480-540 projects and 210-270 cores to review, as well as the overall applications (~107). With each project to receive 3 critiques and each core to receive 2 critiques, I anticipate soliciting 1860-2160 critiques. As each reviewer must receive a manageable workload with adequate time for review, I anticipate the number of assignments per reviewer to be 9, with an average of 7.5-8. This yields an anticipated number of reviewers at 230-290.
If we were to do this the traditional way of streamlining 60% of the applications and discussing each component of the remaining 40% (~40 applications), at 2h/application, this would take >80 hours of meeting time.
Given the magnitude of the conflict of interest list, the large number of reviewers required, and the projected time it would take to conduct review meetings, a different approach is in order.
Proposal.
Overview:
This will be a two-stage review process. The first stage will consist of highly specialized reviewers who will write critiques for each component.The second stage will consist of a Special Emphasis Panel of generalists, who by rank and experience have a broad view of significance and feasibility of therapeutic, vaccine, immunotherapeutic, or diagnostic development.These panel members will write critiques of the overall U19 applications, streamline up to 65%, and discuss and score the remainder.
Detailed approach:
Stage 1: Specialized reviewers will be recruited to critique the U19 components. There will be the usual number of reviewers for each project and core component. These first-stage reviewers will write detailed critiques and assign preliminary scores, including criteria scores where appropriate, to individual components. They will upload these to IAR. There will be no Special Emphasis Panels in this stage.
Stage 1 has the following advantages: 1. Critiques will be generated by experts in the required fields. 2. Several reviewers may be recruited from one institution. 3. Conflicts of interest concerns will be limited to only those applications that are reviewed by each individual. 4. Each component receives the same number of critiques as the traditional review procedure, and applicants will be informed of the range of preliminary scores for each component.
Stage 2: The second stage will consist of25-40 generalists who have a broad view of significance, synergy, and feasibility of the types of products proposed. These reviewers will be assigned to review theoverall U19 applications with the benefit of the Stage 1 critiques. Two reviewers will be assigned to each U19 application. They will be asked to write overall critiques with preliminary scores and upload these to IAR. A single Special Emphasis Panel will be convened. Streamlining of ~65% will take place using either the Stage 1 or Stage 2 preliminary scores for initial screening. The remainder, ~35 applications, will be discussed, each in a single, overall discussion. Panel members will assign final, overall scores to the discussed U19 applications.
Stage 2 has the following advantages: 1. The discussions will be informed by the critiques submitted by the Stage 1 reviewers. 2. The same panel will discuss (or decide not to discuss) each U19 application, providing for a high level of consistency in review. 3. The discussions will focus on the overall U19s rather than on the detailed approach of individual components. 4. Each U19 will be reviewed in its entirety by two reviewers, who will provide critiques and final scores. 5. The SEP will last a reasonable 2-3 days. 6. The applicants, whether their applications were discussed or not, will have the benefit of these overall critiques.
These stages may overlap in time and reviewers. Thus, a Stage 2 reviewer may receive review materials consisting of the application and instructions, and after the Stage 1 critiques are completed, receive these as well. It is conceivable that a particular individual may serve as both a Stage 1 and Stage 2 reviewer, though if stages overlap in time, this would be very difficult.
I appreciate your consideration in this challenging endeavor.
Lynn Rust
March 5, 2013