Division of Independent Review (DIR)

Chairperson Orientation Guide

Objective Review Committee Meeting

TABLE of CONTENTS

1Qualifications…………………..…………………………………………...... ……….3

2Pre-Review Responsibilities.……….……...... 4

3Review Responsibilities.……….…………...………..…………….……....…………4

4Post-Review Responsibilities………………………………………………...... 6

5Conflicts of Interest and Confidentiality………..………..……………….…………6

6Roles and Responsibilities of HRSA and Contractor Staff…….………..….………...7

7 Frequently Asked Questions……..…...... ……………………………………...... 8

ATTACHMENT A – Tips for Successful Meetings…………………………………..11

Chairperson Orientation Guide for Committee Review Meetings

The Chairperson (Chair) has a critical role in ensuring the success of the Objective Review Committee (ORC). The Chair is an extension of HRSA’s Division of Independent Review (DIR). The Chair presides over the ORC meeting, ensures that each application receives a fair, equitable, and objective review, and manages the review meeting schedule to ensure timely processing of all applications. The Chair certifies the review results and also provides an assessment of Reviewer performance to the DIR Review Administrator (RA) at the end of the review.

The purpose of this Guide is to orient the Chair in his/her ORC role and to provide additional information necessary to fulfill his/her responsibilities.

Qualifications

The Chair must be fully eligible and qualified to review applications in the program area, as well as experienced with objective reviews. In addition to technical knowledge, the Chair must also have excellent writing and facilitation skills.

Any of the following disqualifies an individual from serving as a Chair:

1)A conflict of interest with any application in the competition, including applications that are not on the Chair’s panel. Conflict of interest is a critical matter to HRSA and is described in more detail later in this Guide;

2)Status as a Federal employee;

3)Inability or unwillingness to perform the responsibilities of chairing the ORC.

If the prospective Chair finds that there is basis for disqualifying him or herself from being a Chair, he or she must alert the RA as soon as possible.

As a facilitator, the Chair must be able to:

  • Lead the review meeting;
  • Effectively guide the panel through the order of review, and monitor the amount of time the group spends on each application;
  • Make no decisions for the review panel, but suggest ways that will help the participants to move forward;
  • Create a safe environment in which participants feel comfortable contributing ideas, and not allow more assertive members of the group to overtake the discussion;
  • Paraphrase contributions to ensure understanding and agreement by the panel;
  • Encourage reticent group members to come forward with contributions; and
  • Guide participants efficiently through the development of the final summary statement.

Pre-Review Responsibilities

Prior to the review, the Chair must confirm that there is no conflict of interest with any application under review. The Chair will receive a copy of everything that the Reviewers receive. While the Chair will not review applications, he or she must be familiar with the applications and content of the program guidance and other documents that the Reviewers read and use for their reviews.

The Chair must also participate in the Chairperson orientation call. During this mandatory call, the RA will discuss Chair responsibilities. In addition, there is a pre-review orientation call that is mandatory for Reviewers and it is strongly recommended that the Chair also listen in, or at a later date, listen to the recorded version of the call.

Review Responsibilities

The primary responsibility of the Chair is to preside over the ORC panel to assure that each application is accorded a fair and equitable review and manage the schedule to assure timely review of all applications. During the meeting, the Chair ensures that the contributions, views, and questions of all Reviewers are heard, discussed, and given due consideration. The Chair does not participate as a Reviewer and therefore does not contribute subject matter to the discussion or assist in the determination of an application’s merit. The Chair does not score individual applications. At the end of the process, the Chair certifies the results of each application being reviewed.

During the review, the Chair is responsible for the following:

  • Directing the order of review. The RA will provide an initial ordering of the applications to be reviewed, however, the Chair has discretion to make changes due to a variety of factors. Examples of factors that could change the application order include reviewers called upon at the last minute to replace reviewers unable to attend, or unforeseen conflicts of interest requiring recusal.
  • Leading the panel in the review process. The Chair will announce the review of an application and officially move the panel through the review steps. The Chair should refer to and carefully utilize Steps in Conducting the Committee Review that follow. The Chair must be cognizant of the amount of time each step is taking (recommended timeframes correspond to the steps), and schedule panel breaks accordingly. After consulting with the RA, the Chair may modify the panel’s start time on subsequent days, depending on the number of applications that have yet to be reviewed.
  • Clarifying and facilitating the progress of the review. If there is any uncertainty regarding the review process, the Chair is the “first line” of resolution, consulting with the RA, as needed. The Chair will also facilitate the process by encouraging all Reviewers to participate, keeping the discussion focused, and ensuring that the review is conducted in a timely and equitable manner.
  • Screening for conflicts and reminding participants of confidentiality and no discussions outside the call.
  • Monitoring the introduction of outside information. An application should be evaluated on its response to the published review criteria, and Reviewers are to only use information that is included in the application. The introduction of outside information could bias Reviewer discussion and the review results. In the event the Chair hears outside information being introduced, the Chair must instruct Reviewers to disregard that information, and remind them that outside information cannot be considered in the review.
  • Mediating questions and comments from the panel to Program and Grants Management staff and vice versa. Programmatic and/or budget related questions posed by Reviewers should be addressed to the appropriate HRSA staff through the Chair. Likewise, Program or Grants Management staff must be recognized by the Chair to ask questions or clarify information for the panel.
  • Preparing the review summary statement. The Chair asks the First Reviewer to lead the panel in the summary statement editing process. The SSO edits the summary statement based upon input from the Reviewers. The Chair actively listens to the discussion and facilitates preparation of the summary statement. The Chair requires that the panel resolve contradictions and repetitions between written comments and the need for clarification. The Chair should suggest rephrasing sentences when needed to better match the discussion. The Chair must ensure that the strengths and weaknesses listed support each criterion score. The Chair should encourage reviewers to keep use of “none noted” to a minimum because it provides no feedback to applicants. The Chair should also ensure that comments are captured under appropriate criteria. For example, a comment may be more appropriate under another review criterion or more useful as a recommendation to HRSA.
  • Helping to ensure the accuracy of the summary statement. If a Reviewer indicates that a required item is missing, the Chair should ask other assigned Reviewers for confirmation. In addition, the Chair can ask Program staff to ensure the accuracy of the Reviewer statement. The Chair does not serve as a Reviewer and offer an opinion.
  • Certifying the review process. The Chair completes a Chair Certification Form for each application and follows directions for submission. This certification records the panel information, conditions, recommendations, budget recommendations and completion of the summary statement. For programs that have funding factors, the chairperson certification also records decisions related to funding preferences, funding priorities, and special considerations. In programs where there is a vote to recommend or not recommend for approval, the certification also records those votes.
  • The Chair guides the Reviewers in composing/crafting language for recommendations and/or conditions. Reviewers sometimes have advice for the program office to provide to the applicant organizations. The Chair assists the reviewers to clarify these concerns through the provision of recommendations to the applicants through the program office. Reviewers sometimes have advice for the Grants Management Office. The Chair assists the reviewers to voice these concerns through the provision of conditions to the applicants through the Grants Management Office.
  • Breaking a tie. On occasion, where there is voting to recommend or not to recommend an application for approval, the Chair will need to vote in case of a tie. Often, further discussion among the reviewers will break the tie. Before resorting to the tie-breaking procedure, the Chair will encourage the Reviewers to further discuss the recommendation to approve or not approve, then take another vote.

The Chair is vitally important to the review process. The outcomes of the Objective Review Committee enable HRSA to fund the best applications and allow the applicants, both successful and unsuccessful, to receive clear and constructive feedback on how their applications were evaluated.

Post-Review

After the review meeting is completed, the Chair leads the debriefing discussion using the outline provided by DIR. The debriefing discussion should cover the review process, the program guidance, the budget information, and any other points the reviewers would like to discuss. The Chair also completes an evaluation for each reviewer to provide feedback on his or her performance, as well as a feedback form on the entire process. These forms are then faxed to the contractor.

Once the review process has concluded, Reviewers and Chair must return all “hard” materials, such as applications, worksheets, and CDs, to the RA or contractor for destruction. All electronic materials containing confidential information, such as computer files created by the Reviewer, e-mails, etc., must be deleted.

Conflicts of Interest and Confidentiality

The Chair has a crucial role in ensuring that the objective review is free from conflicts of interest. Prior to the review meeting, the Chair must certify that he or she has no conflicts of interest with any application under review. A conflict of interest (COI) is defined in the box to the right:

Prior to reviewing any application, the Chair must ask whether anyone on the panel has a conflict of interest. If a Reviewer has a conflict of interest, the Federal RA must be notified before the review of that application can start. If the Chair finds that, during the application review, he or she has a conflict of interest, the review of that application must stop and the Federal RA must be notified. In both instances, the Federal RA will decide how to proceed with the review.

The Chair must also preserve the confidentiality of the review. All copies of applications and review materials must be safeguarded before and during the review process, and destroyed after the review. Neither the Chair nor the Reviewers may discuss the applications, comments, recommendations, review scores, names of applicants, and/or their participation with anyone not involved in the review process. There should be no discussions of the review before, during, and after the meeting. The Chair should adhere to the confidentiality requirements, and remind Reviewers to delete all electronic review materials once the review concludes.

Roles and Responsibilities of HRSA and Contractor Staff

There will be other staff present at the review besides the Chair and the Reviewers. The following provides a brief description of their roles and responsibilities.

The RA provides oversight over the entire review process, and has the final responsibility regarding the conduct of the review. The Chair will be working closely with the RA before, during, and sometimes after the review.

Program staff is responsible for providing clarification on reviewers’ programmatic questions, bringing to the Chair’s attention any inconsistent or erroneous statements regarding the review of individual applications, and reviewing the use of terminology included in the summary statements to ensure consistency of Reviewer evaluations with the program guidance.

Grants Management staff serves as a resource to review participants by providing grants policy advice and guidance on business management matters, specifically regarding budget issues.

Summary Statement Operators (SSOs) compile Reviewer summaries, project the consolidated summary statement, edit the summary statement language as directed, and record the debriefing. The SSO assigned to the panel will take direction from the Chair to make these edits. It is important that the Chair work closely with the assigned SSO during the editing process.

Frequently Asked Questions

1)How long do I spend on an application?

This depends on the number of applications to be reviewed, the complexity of the program, the complexity of the application, and, on the amount of time available. Generally an application for a typical program should take a little over an hour to review, however your Federal RA will be able to give you more assistance on this question.

2)Do I have to provide a written evaluation form on each Reviewer?

Yes. Your input is valuable in advising HRSA as to the perceived performance of these Reviewers as well as in defining areas that HRSA needs to emphasize for Reviewer training.

3)Last time, the review was conducted differently. Why did it change?

The review process changes because DIR is constantly evaluating its performance and making refinements and improvements to ensure a fair, ethical, and objective review. If you have any suggestions on the review process, please bring them to the attention of the Federal RA, or describe them in your process feedback form at the end of the review. Do not implement any changes without the prior approval of the Federal RA.

4)I think that this particular review would be better if we changed the procedures. When should I bring this up?

Please bring this up during the pre-review conference call that the Federal RA has with the prospective Chairs. The Federal RA will consider your suggestion, and communicate his or her decision to all the Chairs. Changes to the procedures cannot be made during the review unless directed by the Federal RA. This ensures that each panel is following the same procedure for every application, thereby ensuring a consistent, as well as fair, ethical, and objective review.

During the debriefing this type of feedback is also solicited and would be considered for future reviews.

5)What is the relationship of the Chairperson and the SSO?

The SSO assists with the summary statement development. Please direct the SSO in writing strengths or weaknesses, re-wording sentences, deleting points, etc.

6)Do I have to do any pre-review work?

The Chair must submit a COI and Confidentiality Form. The Chair must also read and understand the program guidance and the steps in the objective review process. In addition, it is helpful if the Chair is familiar with the Reviewers’ materials and tools, as well as the application short sets. A short set consists of the application face page, abstract/summary, budget pages, and budget justification. The Chair must also participate in the mandatory chairperson pre-review conference call, and it is strongly suggested that the Chair attend or listen to the recording of the Reviewer pre-review conference call.

7)If the application addresses or meets various elements of a review criterion, is that a strength?

Meeting requirements (e.g., the guidance requires the applicant to provide an XYZ plan, and the XYZ plan is in the application) does not constitute a strength, but not meeting them is a weakness. Most criteria are stated as “the extent to which the applicant does X and Y.” HRSA is looking for qualitative judgments rather than statements that elements are included in the application. See “The Reviewer Instruction Guide –Guidelines for Writing Strengths and Weaknesses.”

8)What should I do when comparisons of applications are made?

If Reviewers begin to make comparisons between applications, please advise them that it is not permissible. Each application is to be reviewed on its own merit based solely on the review criteria, and not compared to other applications.

9)The applicant organization referred to websites in the application. What should we do?

Reviewers should not visit websites referenced in the application in order to gain “a better understanding” of the application. The application should be reviewed as a stand-alone document. Information on the application or the applicant organization contained in websites is considered outside information and will not be utilized.

10)What if a reviewer is late for a review?

If discussion of the review criteria has begun (use your judgment – is the ORC 30 seconds into it or ten minutes into it?), the late reviewer should be advised that he/she cannot participate in that discussion or vote on that application. They should be given the option to stay on the line (not participating in the discussion or scoring) so they are available immediately for the next review. They must complete a signed score sheet marked “Absent”.

11)What if a reviewer must leave the call DURING a review?

Similarly, if a reviewer simply MUST leave the call DURING a review (which we discourage – it really should be an emergency), he/she must advise the chair. The Chair then has the option to halt the review for everyone. If the Chair continues the review and, when the excused person returns, more than one review criterion has been discussed, that person can no longer participate in the review and score. This is an RA decision and an RA responsibility to advise the Chair.