Page 1
Office Work Instruction
HQOWI 7040-Y012B
Effective Date: November 13, 2000
Responsible Office:YS/Research Division
YO/Applications, Commercialization, and Education Division
YF/Program Planning and Development Division
Subject: Conduct Peer Review
OFFICE WORK INSTRUCTION
CONDUCT PEER REVIEW
(Conforming to ISO 9001 Quality System Requirements)
Original Signed By:
Ghassem R. Asrar
Associate Administrator
Office of Earth Science
CHECK THE MASTER LIST at
TO VERIFY THAT THIS IS THE CORRECT VERSION BEFORE USE
Earth Science Enterprise Office Work InstructionConduct Peer Review / HQOWI 7040-Y012 / Revision: B
Date: November 13, 2000 / Page 1of 12
DOCUMENT HISTORY LOG
Status(Baseline/
Revision/
Canceled) / Document
Revision / Effective
Date / Description
Baseline / 1/26/99
Revision / A / 5/10/99 / Section 2.0, paragraph 2.1: Clarified relationship to other OWIs.
Section 4.0: Deleted references not called out in the procedure. Added reference on unsolicited proposals.
Section 5.0: Added outputs to Activities 1 and 2. Rearranged layout to fit on a single page, without changing the flow.
Section 6.0, Activities 1 and 2: Added descriptions of Letters Of Intent (LOI) and proposal logs. Clarified use of LOIs, and handling of unsolicited proposals.
Section 6.0, Activity 3: Expanded discussions of proposal screening and non-responsive proposal disposition. Clarified references and relationship to other OWIs.
Section 6.0, Activity 7: Added references for unsolicited proposals.
Section 6.0, Activity 10: Described contents of documentation. Clarified relationship to other OWIs.
Section 7.0: Added NPG 1441.1 reference. Added text to describe how to retrieve quality records from support contractor. Changed mapping to NPG from Schedule 5, Item 1, “Procurement Files,” paragraph D (retain for two years) to Schedule 7, Item 9, “R&D Peer Review and Evaluations.”
Revision / B / 11/13/00 / Section 5.0, step 3, and Section 6.0, step 3, revised to include Code IY notification of unacceptable Foreign Letters of Endorsement. Section 6.0, paragraph 4 revised to reflect new name for Code YO. Footnotes in section 7.0 revised to reflect new Peer Review Contractor.
PREFACE
The NASA Office Work Instruction (OWI) for Conduct Peer Review documents the tasks and activities in conformance with the International Organization for Standardization’s (ISO) 9001 requirements for quality systems. The OWI supplements the NASA Strategic Plan, the NASA Strategic Management Handbook, and other higher level NASA directives, which form the basis for how NASA conducts business.
This OWI is not intended to duplicate or contradict any other NASA policy, procedures or guidelines, which currently exist. As such, the OWI will reference prevailing documents where a topic is addressed and existing coverage is deemed adequate. Additional information provided within is intended to supplement existing documentation regarding Headquarters (HQ) implementation of strategic and program/project management, as well as HQ conformance with the ISO 9001 Quality Management System (QMS) requirements.
TABLE OF CONTENTS
PARAGRAPHPAGE
1.0 Purpose
2.0 SCOPE and Applicability
3.0 DEFINITIONS
4.0 references
5.0 Flowchart
6.0 PROCedure
7.0 QUALITY RECORDS
1.0 Purpose
The NASA Earth Science Enterprise (ESE) uses a peer review process, sometimes also called merit review or scientific review, to evaluate both solicited and unsolicited proposals. This OWI documents the ESE procedure for conducting peer reviews.
2.0 SCOPE and Applicability
2.1 Scope. The ESE peer review procedure includes receiving and logging proposals, identifying reviewers, determining the type of review (mail and/or panel), conducting mail and panel reviews, and documenting the results.
The ESE conducts peer reviews to evaluate proposals. As such, ESE executes this process in support of higher-level processes as documented by HQOWI 7120-Y003, Formulate and Approve Flight Missions, and HQOWI 8310-Y005, Solicit and Select Science, Applications, Education, and Technology Investigations.
2.2 Applicability. This work instruction for Conduct Peer Review applies to the NASA Office of Earth Science (OES, Code Y) offices and divisions. The Associate Administrator for Earth Science is responsible for maintaining this document. The controlled version of this OWI is available on the World Wide Web (WWW) via the HQ ISO 9000 Document Library at Any printed version of this OWI is uncontrolled (reference: HCP 1400.1, Document and Data Control). Proposed revisions will be accomplished by following HQOWI 1410-Y015, Approve Quality Documents.
3.0 DEFINITIONS
Appendix B of the Earth Science Enterprise Management Handbook provides ESE-specific terms and definitions.
4.0 references
The following documents contain provisions that, through reference in this OWI or in policy or procedure documents, constitute the basis for the documented procedure:
FAR 6.102(d)(2)Use of competitive procedures
FAR 35.016Broad agency announcement
NFS 1835.016-70NASA Research Announcements
NFS 1837.204 Guidelines for determining availability of personnel
NFS 1852.235-72(J)Instructions for Responding to NASA Research Announcements
NPG 5800.1 Sec A Part 1260.11(a)Grant & Cooperative Agreement Handbook
NFS 1872.705-3Glossary of Terms & Abbreviations Associated with Investigations
FAR 15.6Unsolicited Proposals
NFS 1815.6Unsolicited Proposals
for the Preparation and Submission of Unsolicited Proposals
5.0Flowchart
The flowchart depicts the procedure described in Section 6. The outputs in boldface type represent the quality records in Section 7.
6.0 PROCedure
Peer review is used extensively in NASA science, applications, education, technology (SAET) and appropriate flight mission acquisitions. It is essential for a high quality, relevant program. The use of external peer review enhances the quality of NASA’s investigations and activities because it brings the best and most critical national and international experts to the evaluation process. External peer review ensures that fresh viewpoints, alternative perspectives and state-of-the-art understanding are included in the evaluation process. Each review includes a written record of the evaluation and evaluation records are maintained. The evaluation results are used to make a judgment about the merits of each proposal and ultimately are used as the basis to make a selection for an award.
The Federal Acquisition Regulations (FAR) 35.016 “Broad agency announcements” dictates that peer review will be the method used to evaluate and select research for funding. This applies to the acquisition of all research investigations in which the applicable type of solicitation is used, for example, a NASA Research Announcement (NRA), an Announcement of Opportunity (AO) or a Cooperative Agreement Notice (CAN). The scientific, data and information systems, and engineering communities are asked to participate in the peer review of proposals submitted for consideration.
Peer review processes are described in each NASA solicitation. Although the mechanical details of the review process may vary from program to program, the basic use of discipline experts to evaluate and document their findings is universal across NASA. The sponsoring program office for each solicitation is responsible to implement the peer review and maintain the integrity of the evaluations.
Solicitation initiators from the Research Division (Code YS), the Program Planning and Development Division (YF), and the Applications Division (Code YO) conduct this process. The solicitation initiator is usually a science, applications, or education program manager, or a Program Coordinator but could be a division director, deputy division director or someone such as a discipline scientist who is not a program manager. The Research Opportunity Administrator from the Business Division (Code YB) provides administrative support for the solicitation process.
The flowchart of Section 5 depicts typical mail and panel peer review processes. Variations on these processes or somewhat different peer review processes may be employed from time to time, as appropriate. The following table describes the flowchart of Section 5.
Actionee / ActionSupport Contractor
Solicitation Initiator / 1 / Receive, Log, and Assess Letters of Intent to Propose. At the discretion of the solicitation initiator, the solicitation may request that proposers submit a Letter of Intent (LOI)[1] within a given time frame[2] after solicitation release and in advance of proposal submission. The support contractor receives and logs the LOIs, noting the related solicitation identifier, NASA-assigned proposal number, Principal Investigator (PI) name, PI institution, proposal title, and the date received.
The solicitation initiator uses the LOIs to scope the peer review effort by assessing the number of proposals to be reviewed, and the science and technical discipline skills likely required by the peer review team.
Support Contractor / 2 / Receive and Log Proposals. Prospective researchers (proposers) generate responses to a solicitation, and submit these solicited proposals to NASA for evaluation. Proposals also may be submitted in the absence of a solicitation wholly on the initiative of the prospective researcher. If such unsolicited proposals are for science investigations and are responsive to the NASA ESE program, they will be evaluated. For flight missions and technology development, however, proposals are accepted and evaluated only in response to a formal solicitation.
The ESE support contractor receives and logs both solicited and unsolicited proposals. The proposal log identifies the related solicitation, NASA proposal number, Principal Investigator (PI) name, PI institution, proposal title, and the date received.
Solicitation Initiator / 3 / Screen Proposals. The solicitation initiator screens the proposals for compliance with the solicitation requirements, and rejects any non-responsive proposals. Generally, the solicitation initiator looks for completeness of technical and cost information (that is, sufficient for evaluation), and responsiveness to the research areas solicited. The solicitation initiator also may further identify the skills and expertise required by the peer review team.
Additionally, the solicitation initiator screens unsolicited proposals in accordance with the NASA FAR Supplement (NFS) Part 1815.606-70, Relationship of unsolicited proposals to NRAs: "An unsolicited proposal for a new effort or a renewal, identified by an evaluating office as being within the scope of an open NRA, shall be evaluated as a response to that NRA (see 1835.016-70), provided that the evaluating office can either: (a) State that the proposal is not at a competitive disadvantage, or (b) Give the offeror an opportunity to amend the unsolicited proposal to ensure compliance with the applicable NRA proposal preparation instructions. If these conditions cannot be met, the proposal must be evaluated separately."
The solicitation initiator receives notification from the Code IY Action Officer for the NRA of any unacceptable letters of endorsement from foreign institutions. The solicitation initiator will take action, often after consultation with Code IY, to either reject these foreign proposals or to seek an acceptable letter of endorsement from the proposer’s sponsor.
If a proposal is rejected, the solicitation initiator informs the offeror of the reasons for rejection in writing and of the proposed disposition of the proposal. These letters may be posted at this point in the process, or in conjunction with the final Accept/Reject letters as described in HQOWI 7120-Y003, Formulate and Approve Flight Missions, and in HQOWI 8310-Y005, Solicit and Select Science, Applications, Education, and Technology Investigations.
Solicitation Initiator
Cognizant Reviewing Division Director / 4 / Determine Peer Review Type. The solicitation initiator, with the concurrence of the appropriate division director(s) -- for example, the Research Division Director when reviewing proposed science investigations -- determines the characteristics of the peer review process to be used. The characteristics include how the review is to be conducted and the source of peer reviewers. For example, reviews may be conducted by mail, by panel, or by some combination or variation of these methods, including single or multi-step proposal review processes. Also, ESE may make full use of external peer reviewers, or conduct an internal peer review -- that is, involving only civil servants. The solicitation initiator bases these decisions on the urgency of the schedule for a selection, the complexity of the proposals to be evaluated, the number of responses, what type of process was advertised in the solicitation and other considerations unique to the solicitation.
ESE evaluates proposals using a mail peer review, a panel peer review, or both, and involves external reviewers from all types of institutions. If both mail and panel reviews are to be used, the panel review almost always follows and receives input from the mail review.
Situations in which internal peer reviewers alone may be appropriate include proposals that involve minor funding for non-research activities such as workshops or symposia, or research situations requiring a quick response to an unexpected opportunity such as a volcanic eruption or foreign aircraft mission.
Solicitation Initiator / 5 / Identify Mail Peer Reviewers. Mail peer review allows for selecting reviewers with the very specialized expertise required for delving deeply into the technical and scientific merits of the solicitation topic and technical approach.
The NASA ESE draws its peer reviewers from the entire scientific, technical, educational, information systems, and engineering communities, including experts from both public and private academic institutions, industry, government (such as NASA Centers, other government laboratories, and Federally Funded Research and Development Centers), and foreign countries. NASA and its support contractors maintain a list of discipline experts within the System for Codes E, Y, F, U, and S (also known as SYS-EYFUS system), and this list is drawn upon to find the appropriate reviewers. Additional names are added as new people enter the field.
The solicitation initiator identifies potential reviewers based on their experience, knowledge of the SAET technical areas, information systems, and engineering or management areas. In addition recommendations may be obtained from other program managers, acknowledged experts, NASA management, university professors and corporate executives. Criteria considered reflect the reviewers’ scientific, technical, and professional expertise, credentials, and abilities. For example, criteria may include quality of scholarly research, relevant publications, amount and relevance of research, and knowledge of and experience within the discipline. Availability and willingness to accomplish the review are also factors.
The ability of a potential peer reviewer to conduct an impartial review is a critical factor. The solicitation initiator screens potential reviewers for any perceived or real conflicts of interest. "Conflict of interest" means that because of other interests, activities, or relationships, a person is unable or potentially unable to render impartial assistance or advice to the Government, or the person's objectivity in performing the review is or might be otherwise impaired. Other interests, activities, or relationships include financial interests, institutional affiliations, professional biases and associations, as well as familiar relationships. Conflicts could further occur as a result of imbalance between Government and non-Government appointees, a member evaluating a proposal from their own parent organization, or panel members representing a singular school of thought in discipline areas involving competitive theories and approaches without appropriate balance from those representing competitive schools of thought.
The solicitation initiator sets a due date for the mail reviews and directs the support contractor to distribute the proposals and evaluation criteria to the mail reviewers. Prior to distributing the proposals, the solicitation initiator may elect to contact the reviewers and request their participation.
Support Contractor / 6 / Distribute Proposals and Mail Review Evaluation Criteria. The support contractor distributes the proposals and evaluation criteria through the mail to the peer reviewers. Mail reviewers are sent the proposals for which their reviews are needed, with a letter requesting their review and asking for immediate return of the proposal(s) if the reviewer cannot comply. Reviewers perform the review on a voluntary basis without payment for services.
Mail Peer Reviewers / 7 / Evaluate Proposals (Mail Peer Review). Mail reviewers individually evaluate each proposal according to its own merits and using the evaluation criteria provided by NASA and listed in the solicitation or in the NASA guidelines for unsolicited proposals (FAR Subpart 15.6, Unsolicited Proposals; NFS Part 1815.6, Unsolicited Proposals; and"Guidance for the Preparation and Submission of Unsolicited Proposals," available on the Internet at
The mail peer reviewers return their reviews and evaluations by mail (sometimes FAX and e-mail) to the support contractor.
Solicitation Initiator / 8 / Identify and Invite Panel Peer Reviewers. A panel peer review brings together scientific and technical experts that cover an appropriate breadth of professional knowledge and expertise and offer balanced perspectives on the topics to be evaluated. It allows for a thorough discussion of the merits of each proposal and the opportunity to reconcile differing evaluations on the part of individual panelists. If a mail review has preceded the panel review, the results of the mail review are made available, and the panel will be asked to reconcile differences among the mail reviews as well as to put the work proposed into a larger scientific and programmatic context.
The solicitation initiator identifies potential panel peer reviewers in the same way he or she identifies them for mail reviews. Refer to Activity 5, Identify Mail Peer Reviewers, for a discussion of how peer reviewers are identified, screened, and chosen.
The solicitation initiator solicits participation in a formally convened panel, or in some cases asks the support contractor to call the selected reviewer on his/her behalf. This invitation is commonly extended by phone and followed by formal letter, but may be extended through a formal letter only. Reviewers perform the review without payment for services. Travel costs for the panel meeting are reimbursed.
The solicitation initiator, with the support contractor, arranges the panel meeting by planning the schedule, budget, and logistics.
The solicitation initiator often assigns a lead reviewer for each proposal from amongst the panelists. The lead reviewer summarizes the proposed research, the mail review results (if a mail review was conducted), and his or her own evaluation as a means of initiating the panel discussion of a proposal.
Support Contractor / 9 / Distribute Panel Review Materials. The support contractor distributes the proposals, the evaluation criteria, and any mail review results to the panel members.
Panel Peer Reviewers
Solicitation Initiator
Support Contractor / 10 / Evaluate Proposals (Panel Peer Review). The peer review panel meets as a group and discusses the scope, strengths and weaknesses of the various proposals. The proposals are rated in accordance with the evaluation criteria and the panel gives an overall rating based on a vote or consensus. The peer panel may also be asked to recommend aircraft campaign payloads or assemblages of proposals that best meet focused program objectives.
The solicitation initiator, often with help from the panelists and/or a recording secretary provided by the ESE support contractor, documents the panel review evaluation and results. The documentation may contain, for example, a summary of the analyses performed, scores and ratings, statements of fatal flaws, and summaries of proposal strengths and weaknesses. At a minimum, it must summarize the reviewers' findings and recommendations in sufficient detail for making a selection decision.
The peer review results feed subsequent selection recommendation activities as documented by HQOWI 7120-Y003, Formulate and Approve Flight Missions, and HQOWI 8310-Y005, Solicit and Select Science, Applications, Education, and Technology Investigations.
7.0 QUALITY RECORDS