Independent Verification & Validation Program / IV&V Catalog of Methods / [IVV ##-#-#]
Revision: Basic
Effective Date:
DRAFT

DOWNLOADED AND/OR HARD COPY UNCONTROLLED


Independent Verification & Validation Program / IV&V Catalog of Methods / [IVV ##-#-#]
Revision: Basic
Effective Date:
DRAFT

Verify that this is the correct version before use.

{The Approval Signatures table contains the name of the Approving Official, role of the Approving Official within the IMS, and date the Approving Official signed the DCR to approve the addition or modification of the document.}

APPROVAL SIGNATURES / DATE
<Name>

{The Revision History table contains the revision (represented by letters of the alphabet after the initial release, which is represented as the “Basic” release), a description of the document revision, the Author who implementedthe change, and the effective date of the document.}

REVISION HISTORY
Revision / Description of Change / Author / Effective Date
Basic / Initial Release / <Name> / <Date>
REFERENCE DOCUMENTS
Document / Title
IVV QM / NASA IV&V Quality Manual
IVV 16 / Control of Records
NPR 1441.1 / NASA Records Retention Schedules
IVV 9-4 Rev AA / Project Management
<Doc ##> / IV&V Office Concept of Operations
<Doc ##> / TQ&E Method Evaluation Guidance
IVV 9-1 Rev M / IV&V Technical Framework

If any process in this document conflicts with any document in NODIS, this document shall be superseded by the NODIS document. Any reference document external to NODIS shall be monitored by the Process Owner for current versioning.

1.0Purpose

The purpose of this system level procedure (SLP) is to establish a consistent method for adding, modifying, and otherwise managing Methods for performing software IV&V.

The Catalog forms the basis for an engineered approach to achieving the desired level of software assurance on Mission Projects for which IV&V is required and authorized. Methods in the Catalog support one or more Goals of IV&V as defined in System Level Procedure (SLP) 09-1, "Independent Verification and Validation Technical Framework."The Catalog defines the set of Methods approved for implementation on IV&V Projects, and embodies lessons learned and best practices in applying those Methods productively.

IV&V Project Managers select Analysis Methods from the Catalog to fulfill the IV&V Analysis objectives for the IV&V Project, in compliance with requirements for analysis derived from the Portfolio Based Risk Assessment (PBRA) and/or Risk Based Assessment (RBA) of the Project. The collection of selected Analysis Methods applied to Mission Project artifacts constitutes the Technical Approach to fulfilling IV&V analysis requirements, as documented in the IV&V Project Execution Plan (IPEP).When selecting analysis Methods of sufficient maturity, Project Managers avoid the necessity of generating and reporting on a risk to track the potential impact of using immature unproven analysis Methods on their Projects.

Analysis methods are used as an aid for estimating levels of effort in preparing the IPEP's resource requirements section, including the IV&V Project schedule.

The Methods listed in the Catalog are only approved for use in achieving the IV&V Analysis Goal(s) as directed in this Catalog. Other uses of the Methods must be approved in advance by TQ&E.

Each Method may have optional tailoring guidelines to permit the Method to be adapted to the particular Mission Project's characteristics. The scope of such tailoring is defined by the tailoring guidelines; any additional modifications to the Method must be approved in advance by TQ&E.

The authority to perform catalog and method maintenance, including approval of methods, method use guidelines, and method tailoring guidelines is the responsibility of the TQ&E.The decision to apply a particular method on a particular IV&V Project to meet a particular technical goal is captured in a Technical Scope and Rigor document for that IV&V Project, which is an appendix to the IPEP, and that IPEP is reviewed by TQ&E and IV&V Office Management, and approved by the IV&V Office Lead.

IV&V Office Management uses the Analysis Methods Catalog to demonstrate how the Office fulfills IV&V Program goals for IV&V analysis. It also uses the Catalog to help assess the quality and rigor of planned IV&V analyses per Project, and to assure proper coverage of IV&V Analysis Goals for the subject Project.

IV&V Office TQ&E uses the Catalog as its primary reference in evaluating Technical Approaches proposed by Project Managers.It also uses the Catalog as the primary organizational tool for implementing process and quality improvement initiatives, and tracking the organization's process maturity levels.

Analysts use the Catalog's descriptions to gain a better understanding of the objectives of each Method, and to identify appropriate work instructions and guidelines for implementation of the Method.

R&D uses the Catalog to assess proposed research and development efforts against existing Methods to ensure best allocation of R&D resources.

2.0Scope

This SLPapplies to IV&V Methods utilized by NASA IV&V and its contractors to provide software assurance to Mission Projects selected to receive IV&V services from the NASA IV&V Facility. The Catalog covers IV&V Methods that meet technical goals 2.0 through 7.0 (and associated sub-goals of 2.0 through 7.0) from the IV&V Technical Framework.

3.0Definitions and Acronyms

Official NASA IV&V roles and terms are defined in the Quality Manual. Specialized definitions identified in this SLP are defined below.

3.1IV&V Catalog of Methods (“Method Catalog” or simply “Catalog”)

A set of documented methods that can be applied to perform IV&V. The Catalog is currently maintained in an Excel spreadsheet here:

Livelink/ecmles (online)/Enterprise Workspace/IV&V OFFICE/TQ&E/Knowledge Sharing/IV&V Catalog of Methods

3.2IV&V Method (or simply “Method”)

An IV&V Method is a set of activities that when executed under the specified conditions and assumptions achieves a specified technical goal or goals.

3.3IV&V Method Lifecycle

When a method is initially provided to TQ&E for review, it is in the Draft state. Once TQ&E approves the Method and includes it in the catalog, it is in the Documented state. When the Method is no longer to be used, it is removed from the Catalog and is in the Retired state.

TQ&E will review all of the Methods in the Catalog at least annually to ensure they are still applicable and viable.

3.4IV&V Method Maturity Continuum

Documented (TQ&E approved) methods are in a sub-state using a TRL-like progression (relative to IV&V use) called the IV&V Method Maturity Continuum:

  • Conceptual: the Method exists as an idea with no more than theoretical basis; academic. All new Methods begin life at the "Conceptual" stage.
  • Studied: some effort has gone into elaborating or exploring the viability of the conceptual Method; focus of study could be on parts of the Method, a simulation of the Method, etc.
  • Demonstrated: the Method has been executed once in a specific set of circumstances under controlled conditions, perhaps as a pilot project.
  • Operational: the Method has been shown to produce results.
  • Deprecated: the Method has been superseded by others or is otherwise no longer viable, and should be avoided.

3.5IV&V Method Template

The table below defines the data that is required and desired to define a Method in the Catalog.

Title / Simple, unique title
Maturity / See above
Analysis Target / Artifacts or parts of artifacts that are targeted for analysis
Inputs / Prior analysis outputs, checklists, etc. needed to execute the method
Prerequisites / Any previous internal or external activity needed before this method can be applied
Outputs / Products delivered as a result of execution of the method (e.g. TIMs, summary reports, external risks)
Required Tools / Tools that are required to execute the Method - generic (capability-based) and specific (e.g. ORBIT)
Assumptions / Unconfirmed statement(s) that must be true for the Method to be applicable and/or successfully applied
WBS Coverage / Mapping to WBS element(s) partially or fully addressed by employing the method
Technical Goals(s) / Statement of the objective(s) to be attained through completing the IV&V analysis specified in a Method
Required Specific Knowledge or Skills / Any special domain or tool knowledge that is required to execute the Method
Analysis Steps / Description of the method to be executed in sufficient detail to provide understanding by an analyst who will be performing the process. There is no specific format that is required. Formats could include:
1. Use case
2. Flow chart
3. Step-by-step description
4. Natural language
5. Pointer (URL or other reference) to such a description elsewhere that is generally accessible to all IV&V analysts and staff, such as on the ERC or KMS
Empirical Evidence / Evidence produced by or during execution of this Method to support IV&V mission assurance conclusions and outputs
Analysis Risks / Risks inherent in employing the analysis method
Other / Anything else of relevance
See also / Names/references/pointers to other related methods
Source
BOLD Items are required

3.6Submitter

The Submitter submits a new, modified, or tailored Method for TQ&E consideration. The Submitter may be any engineer, analyst, IV&V Project manager, or any other person in the IV&V Program with an interest in extending or improving the Methods in the Catalog.

3.7Acronyms

IMSNASA IV&V Management System

IVVOIV&V Office

NPRNASA Procedural Requirement

QMQuality Manual

SLPSystem Level Procedure

TQ&ETechnical Quality and Excellence group

4.0Process Flow Diagram

The following diagrams depict processes described in this document, and the responsibilities and actions that shall be performed by process participants or their designees. Any information supplemental to the depicted process will appear after the diagram.

4.1Adding a New Method

New Methods can be added to the Catalog as needed. New Methods can be suggested by experience, relevant technology or advances from other related fields, result from on-going research and development, by new technologies and tools, or it could simply be the documentation of a tried-and-true Method that has not, as yet, been documented and included in the Catalog.

Adding a New Method

  1. A Submitter documents a new method using the Method Template.
  2. The Submitter provides the draft Method to TQ&E.
  3. TQ&E organizes and leads a review of the Method to include inputs from R&D and SWAT and provide feedback to the Submitter per the Review Guidance Document. TQ&E will coordinate R&D and SWAT review and feedback collection.
  4. NOTE: At any point after the initial review, TQ&E can opt to reject the Method.
  5. If needed, the Submitter updates the Method per the review comments and resubmits the Method for re-review.
  6. TQ&E approves the Method – the Method is in the Documented state.
  7. TQ&E adds the Method to the Catalog.

4.2Tailoring an Existing Method

An existing method may need to be tailored under certain circumstances to make it applicable to a project, to enhance its applicability to a project, or remove or change step(s) to make it more appropriate. Tailoring can be defined for a specific project only, or proposed for general purpose use when certain circumstances are met.

Tailoring an Existing Method

  1. A Submitter documents a tailoring of an existing Method using the Method Template. The tailoring shall include the circumstances under which the tailoring is applicable as well as the change to the method.
  2. The Submitter provides the tailored Method to TQ&E.
  3. TQ&E organizes and leads a review of the Method to include inputs from R&D and SWAT and provide feedback to the Submitter per the Review Guidance Document. TQ&E will coordinate R&D and SWAT review and feedback collection.
  4. NOTE: At any point after the initial review, TQ&E can opt to reject the tailored Method.
  5. If needed, the Submitter updates the Method per the review comments and resubmits the Method for re-review.
  6. TQ&E approves the Method
  7. If the tailoring is of use to multiple projects, then the Tailoring is identified in the Method in the Catalog.
  8. Otherwise, the tailoring is limited to a single IV&V Project.

4.3Modifying an Existing Method

An existing method may need to be modified to incorporate an improvement or remove a defect.

Modifying an Existing Method

  1. A Submitter documents a modification of an existing Method using the Method Template.
  2. The Submitter provides the modified Method to TQ&E.
  3. TQ&E organizes and leads a review of the Method to include inputs from R&D and SWAT and provide feedback to the Submitter per the Review Guidance Document. TQ&E will coordinate R&D and SWAT review and feedback collection.
  4. NOTE: At any point after the initial review, TQ&E can opt to reject the modified Method.
  5. If needed, the Submitter updates the Method per the review comments and resubmits the Method for re-review.
  6. TQ&E approves the modified Method.
  7. TQ&E notifies all active projects that are using the Method that was modified of the change. The IV&V Project may choose to execute the Method as originally planned prior to the change or as changed.

4.4Method and Catalog Business Rules

Methods in the Catalog are approved by the IV&V Office Technical Quality & Excellence(TQ&E) function of the NASA IV&V Program, which retains sole configuration control authority.

If a project needs a method to achieve an IV&V goal and no method exists, then a new method should be proposed through the new method process.

All new Methods must go through the Method Maturity Continuum. Methods that have been shown to meet one or more goals from 9-1 and are simply being documented in this format can be grandfathered in as operational methods.

Existing methods that are defined in this format for the first time may enter the Maturity Continuum at whatever level criteria is met, up to the Operational level.

All modifications to operational Methods must go through the Method Maturity Continuum.

TQ&E defines and applies standards to determine if a Method may transition to a higher maturity.

Only operational Methods should be employed by an IV&V project, if at all possible.

Operational Methods may be tailored with TQ&E approval.

If an IV&V project employs a Method that is not operational, then a risk must be opened and managed until the Method is completely executed or abandoned.

Each method is to be reviewed annually for viability. If a method is no longer used because other methods are demonstrated to be more effective or efficient, it is deprecated. If a method is no longer viable under any circumstances, then it is retired.

5.0Metrics

Any metrics associated with this SLP are established and tracked within the NASA IV&V Metrics Program.

6.0Records

There are no records associated with this SLP.