IDnr / 125_BP_E
Date adjusted / 18122006
Description of the content / This document contains the procedure for Planning and performing Quality Assurance reviews for projects and programs ofam and
handling QA violations
Type of document / Procedure
ASL Process / Quality Management
Size of the application management team:
Small (number fte < 10),
Medium (number fte 10 – 20)
and/or Large (number fte> 20). / Small Medium √ Large
Remarks
Quality Assurance Procedure
procedure
Place / Amsterdam
Date / 181206
Author / ASL
Status / Concept 0.1
Version Control
Version / Date / Author / Description0.1 / Author / Initial document
Distributionlist
Version / Date / To0.1
Quality Assurance Procedure
Overview
This document contains the procedure for:
Planning and performing Quality Assurance reviews for projects and programs ofam.
Handling QA violations (a QA violation is the non-execution of a mandatory QA review).
1. Quality Assurance in Application Management
1.1 Purpose
Ensure and deliver advice on that what is being delivered is provided on time, within budget, meets the documented requirements and with high customer satisfaction.
Provide the Project Manager and AMS management with:
Advice on the proposed solution, contract, or work effort with respect to its management, quality and financial exposures (via Solution Design reviews and the QA1/2/3 reports)
A health assessment and related advice on the project as it is being delivered (via Solution Delivery reviews and the QA4/5/6 reports)
Target audience:
Opportunity Business Manager (OBM), Opportunity Owner (OO), Proposal Teamleader (PpTL), Project Team Leader (PjTL), Quality Assurer (QA), Senior Business Area Manager (Sr. BAM), Project Manager (PM)
1.2 Scope
This procedure applies to:
All projects and all programs.
This procedure describes only one area of Software Quality Assurance activities, the independent review of project management and the related deliverables or work products.
1.3 Special Conditions
This procedure applies only to QA reviews on projects and programs performed byam.
1.4 Entry Criteria
An opportunity has been received and discussions are starting which will affect customer expectations towards the solution or offering.
1.5 Exit Criteria
The project ended either successfully (project completion criteria are met) or unsuccessfully (proposed solution has been rejected by the customer).
The end of the agreed program period is reached (normally end of year).
2. Overview of Quality Assurance
2.1 Description
The activities related to QA reviews are divided in two parts:
Activities related to the planning and execution of QA reviews
For a description of the different QA reviews see chapters 2.3 and 5.
Activities related to the detection and reporting of QA violations.
For a definition of a QA violation see chapter 2.4
2.2 Activity and Logic Flow
QA reviews
QA violations
2.3 Activity List for QA reviews
QA reviews are referred to by e.g. a company´s Worldwide Quality Assurance designations of QA1 through QA6. Different types of reviews (technical, project management, business) are conducted at various stages of a project, as described below.
During Solution Design:
QA1 - Solution Assurance Review
QA2 - Business Assurance Review
QA3 - Proposal Assurance Review
During Solution Delivery:
QA4 - Project Management Review (Initial)
QA5 - Project Management Review
QA6 - Deliverable Readiness Review
Role(who will do) / Process-Activity / EvidencePpTL/PjTL / Step 1 Plan for Review
When (at what frequency)
At start of Solution Design for the QA1, QA2 and QA3 review
During Solution Delivery for the QA4, QA5 and QA6 review
How (will it be done)
Refer to appendix C for the frequency of QA reviews and to the activity lists for the timing of the reviews.
Refer to 'AM SQA plan template for Quality Planning activities.
Contact the QA team to plan a specific QA review,
Update Business Management reporting with the planned QA review dates / Where (will it get reported)
PROJECT TEAM DATABASE
BMRS
What (format will be used)
SQA plan
PpTL/PjTL / Step 2 Prepare for Review
When (at what frequency)
For a QA3 review: 5 days before the planned review.
For the other reviews: 2 days before the planned review.
How (will it be done)
Provides requested project materials to the Quality Assurer.
Ensures that corrective actions and deficiencies from previous reviews that should have been completed or corrected have been completed or corrected.
Prepares a short presentation on the project background and current status, if necessary.
Works with the Quality Assurer to identify specific review participants and roles they will play. / Where (will it get reported)
PROJECT TEAM DATABASE
What (format will be used)
For QA1-3:
Technical and Proposal Risk Assessment
Project Management Plan (including auxiliary plans)
(Peer) Review results.
For QA4-6:
Delivery Risk Assessment
Detailed PMP
Actuals
(Peer) Review results.
QA / Step 3 Perform QA Review
When (at what frequency)
At date planned (see step 1)
How (will it be done)
Reviews materials.
Holds in-person interviews and review meetings with the Project Team as needed to resolve questions or issues. Leads interviews and review meeting(s). Obtains additional materials from the PpTL/PjTL as needed.
Creates the QA Review Report, the findings, recommended actions with their completion dates and the lessons learned.
Completes the QA Risk Assessment
(Optional) Reviews preliminary version of the QA Review Report with the PpTL/PjTL and team and finalizes the report.
Provides final version of the QA Review Report and Risk Assessment to the OO and the PpTL/PjTL.
In case of troubled projects (health rating C or D) or high/exceptional risk projects, reporting will be extended to the Sr. BAM, and for large projects also to the QA (refer. appendix B). / Where (will it get reported)
PROJECT TEAM DATABASE (via PjTL)
QA Tracking database (for projects between 1000 and 5000 hours and for programs)
QA Workbench GAMSD (for projects >= 5000 hours)
What (format will be used)
QA Review Report containing:
Findings
Recommended actions
Completion dates for actions
Lessons Learned
Rating (for delivery projects > 1000 hours)
QA Risk Assessment.
PpTL/PjTL and OO / Step 4 Respond to Review Findings
When (at what frequency)
At or before completion dates for recommended actions.
How (will it be done)
Reviews report from the Quality Assurer.
Performs the recommended actions before the requested completion dates.
Informs QA of completion of actions.
Document non-concurrence with the recommendations and review this with QA and management.
Documents the rating and actual review date in BMRS / Where (will it get reported)
PROJECT TEAM DATABASE
BMRS
What (format will be used)
Updated QA Review Report
Other documents as appropriate
2.4 Activity List for QA violations
A QA violation is the non-execution of a mandatory QA review.
For the mandatory QA reviews see appendix A.
Role(who will do) / Process-Activity / EvidenceQA / Step 1 Detect and report QA violation
When (at what frequency)
Detect: during scheduling of QA reviews by QA team or during a QA review.
Report: within 2 working days after detection of violation
How (will it be done)
Detect a violation
A portfolio of projects is maintained by QA. This portfolio is derived from several sources (PpTL/PjTL, OO, Business Operations). If a QA1-3 review is performed then the SQA plan is known. Based on the project portfolio and/or the SQA plan a violation can be detected.
Report a violation
For projects > 1000 hours, the violation will be reported to the OO with a cc to:
The PpTL/PjTL
The Teamleader of the Independent QA team foram
The Manager of QA Netherlands
The OBM
For projects < 1000 hours, the violation will be reported as an issue in the QA3-6 report. / Where (will it get reported)
PROJECT TEAM DATABASE: Violation Report (via PjTL)
QA team database: Violation Status Overview (projects > 1000 hours)
What (format will be used)
QA report issue (projects <= 1000 hours)
Violation report (projects > 1000 hours)
Violation Status Overview (projects > 1000 hours)
PpTL/PjTL / Step 2 Respond to QA violation
When (at what frequency)
Within five working days after receiving the violation report.
How (will it be done)
Reviews report from the Quality Assurer and determines root cause of violation.
Determines the possible corrective actions that will alleviate the root cause determined for the violation.
Defines action plan.
Reviews action plan with Quality Assurer and management. / Where (will it get reported)
PROJECT TEAM DATABASE: Action Plan
QA team database: Violation Status Overview (projects > 1000 hours)
What (format will be used)
Action plan
Violation Status Overview (projects > 1000 hours)
QA / Step 3 Close QA violation
When (at what frequency)
At completion of action plan.
How (will it be done)
Track the action plan to completion.
Close the QA violation. / Where (will it get reported)
PROJECT TEAM DATABASE: Action plan (via PjTL)
QA team database: Violation Status Overview
(projects > 1000 hours)
What (format will be used)
Updated Action plan
Violation Status Overview (projects > 1000 hours)
3. Measurements
Number of QA reviews held within a business unit.
Health rating of the projects within a business unit.
Risk rating of the projects within a business unit.
Number of QA violations within a business unit.
The basic data for these measurements is stored in the QA Tracking database, the QA Workbench and the QA team database.
4. Verification
All review results and violations are reported in the monthly meeting between the teamleader of Independent QA and the Sr. Business Area Manager
An overview of review results and violations is reported by the teamleader of Independent QA in the monthly meeting of the Sr BAM with his OO's.
An overview of review results and violations is reported in the periodic meeting between the AM Manager, the QA manager and the teamleader of Independent QA
An overview of review results and violations on projects > 5000 hours is reported monthly to the European QA manager (excluded program reviews).
A more detailed description of the findings and action plan on these projects is reported in the following cases:
The project has a health rating of C or D (troubled projects)
The project has a risk rating of 7 or higher (High or Exceptional)
5. Tailoring
No tailoring is defined for the Activity List for QA Violations.
The Activity List for QA Reviews is completed with the additional descriptions in Appendix A, B and C (defining clip levels, related review authorization and review validation details, report distribution and review frequencies). The following tailoring is possible on this set of rules:
For Program reviews, full tailoring is allowed.
Project reviews below the clip level have to be done by the project team, and can be conducted using a QA Checklists for Small Projects and following a Peer Review procedure. In addition, the same tailoring as with projects above the clip level is allowed.
Project reviews above the clip level are mandatory, but some tailoring is allowed. For QA1, QA2 and QA6, the review topics need to be assured by the PpTL/PjTL, but reporting may be combined with another appropriate review (e.g. QA3 for QA1/QA2, or QA5 for QA6) or in another format. For QA4 and QA5 or QA5 and QA6, it can be decided during QA3 whether these reviews may be combined.
6. Quality Records
Quality Record / Where Stored / Retention PeriodQuality Assurance Review Reports / PROJECT TEAM DATABASE
QA Tracking db for projects between 1000 and 5000 hours and programs
QA Workbench GAMSD for projects >= 5000 hours / One year after project completion
Risk Assessments / PROJECT TEAM DATABASE
QA Tracking db for projects between 1000 and 5000 hours and programs
QA Workbench GAMSD for projects >= 5000 hours / One year after project completion
SQA Plan / PROJECT TEAM DATABASE / One year after project completion
Other Quality Documents / PROJECT TEAM DATABASE / One year after project completion
Violation report / PROJECT TEAM DATABASE
Mailbox of taskid 'AM QA' / One year after project completion
Violation status report / QA team database / One year after project completion
7. References
7.1 Standards & Procedures
Worldwide Quality Assurance Policies and Procedures
Quality Management System
Peer Review procedure
7.2 Definitions / Acronyms
QA: Quality Assurance
BMRS: Business Management Reporting System (central database for storing business management information used to get overview on the business and support decision making)
Independent QA: Quality Assurance, independent from the Line of Business
Aligned QA: an independent QA'er aligned to a business unit in AM.
7.3 Other References
Capability Maturity Model for Software, Software Engineering Institute, CarnegieMellonUniversity
8. Tools, templates and forms used in the Process
8.1 Forms and Templates
AM SQA plan template
AM QA Checklists for Small Projects
Solution Design, QA123 review report
Solution Delivery, QA456 review report
8.2 Tools
QA Workbench : on server XXXX (only for independent QA)
QA Tracking database: on server XXXX (only for independent QA)
AM QA mailbox: on server XXXX (only for independent QA)
QA team database: on server XXXX (only for independent QA)
Appendix A. Clip Levels and authorized reviewers
The Worldwide Quality Assurance (WWQA) process mandates quality assurance reviews, the requirements for which are documented in the following table.
For projects below the clip level the validation activities as mentioned below in column 'Review Validates' have to be performed by the PjTL/PpTL and have to be recorded in a Project Control Book, but do not need to be reported formally.
For program reviews, no clip level is defined.
Authorized reviewers by Project Clip LevelType of Review / Review Validates / Above 1000 hours / On and Below 1000 hours
QA1. Solution Assurance Review / Proposed solution:
fulfills customer's requirements
is technically viable
complete and reasonable estimates and schedules
technical risks identified, assessed and controlled. / PM / Informal
(peer review)
QA2. Business Assurance Review / Verification of:
solution satisfies both the companies´ and customer business objectives
complete cost case, contingencies identified
complete and reasonable estimates and schedules / PM / Informal
(peer review)
QA3. Proposal Assurance Review / Cost, planning, and technical baselines previously reviewed have been updated to reflect changes, and can now be regarded as final. Documentation clearly defining the nature, scope, special conditions, delivery schedule, responsibilities and dependencies of all parties (including payment streams) is in place to establish clear project baselines. Overall agreement that the offer can be responsibly made and represents good business for the offering organization reached in QA2 is verified, if applicable. / Independent QA
(also review of QA1/QA2) / Informal
(peer review)
QA4. Project Management (PMR) Review - Initial / The project planning baselines, and all other project management documentation sufficient to commence work on the project, are in place and represent an adequate project management foundation. Draft schedules for performance of quality assurance reviews have been established. / Independent QA / Informal
(peer review)
QA5. Project Management (PMR) Review / Project performance is evaluated to ensure that activities are all on track, key dates will be met, quality deliverables are being provided, processes, standards and procedures are being adhered to, financial and resource commitments are being met, risks are being mitigated, and issues are carried forward for Service Line Management focus as appropriate. QA5 reviews are repetitive based on the review classification (health) assigned to the project. / Independent QA / Informal
(peer review)
QA6. Deliverable Readiness Review / Adequate and appropriate pre-delivery testing has been performed by the delivery team and validated by the project SMEs. This review is scheduled before major delivery milestones and
before overall project delivery and completion. / Independent QA / Informal
(peer review)
Note: Scope change may result in a proposal or project previously below a particular clip level passing into a new clip level. If this occurs the PpTL/PjTL informs the assigned Quality Assurer. The review should be held in accordance with the new clip level.
Appendix B. Project Health Rating for QA4, QA5 and QA6
Health Rating / Description / Report Distribution / Next ReviewA / Project is under control. Minor problems may exist, but the PpTL/PjTL has an effective plan in place to solve the problems. No major existing or potential problems have been identified. / PpTL/PjTL
OO
Other review participants / No later than six (6) months past the current project review date.
B / Project is currently under control. However, existing or potential problems have been identified which will require positive management attention in order to keep the project under control. / PpTL/PjTL
OO
Other review participants / No later than five (5) months past the current project review date.
C / Significant problems exist. Probability exists for exceeding estimates or budgets, customer dissatisfaction, and/or limited financial exposure. Aggressive management action is required to bring the project under control. / PpTL/PjTL
OO
Manager of OO
Other review participants
For projects of > 5000 hours:
EUROPEAN GAMSD QA Manager
QA Manager / No later than four (4) months past the current project review date.
D / Major problems exist with definite, serious financial exposure and/or customer dissatisfaction. / PpTL/PjTL
OO
Manager of OO
Other review participants
For projects of > 5000 hours:
EUROPEAN GAMSD QA Manager
QA Manager / No later than three (3) months past the current project review date.
Note 1: QA1, QA2 and QA3 reviews do not report a Health Rating. The advice can be 'Proceed', 'Do not Proceed', or 'Complete actions, then proceed'.
Note 2: C & D Projects are considered 'troubled'.
Note 3: Each QA review also produces a Risk Assessment. High Risk projects are subject to the same management attention as troubled projects.
Appendix C. Frequency of reviews
QA1-4: Once during the lifecycle of a project
QA5: See Appendix B, column 'Next Review', but at least once during the lifecycle of a project and no later then 30 days before end of project
QA6: Before sending a major deliverable to a customer and at the end of each project
The timing of each type of QA review is prescribed in the Activity Lists of AM Management System
End of DocumentQuality Assurance Procedure
181206
1/15