IAF/ILAC-A3:2006IAF/ILAC Mras: Key Performance Indicators

IAF/ILAC-A3:2006IAF/ILAC MRAs: Key Performance Indicators:

IAF/ILAC-A3:2006 IAF/ILAC MRAs: Key Performance Indicators:

PREAMBLE

The international community of accreditation Regional Groups, recognized Accreditation Bodies and their stakeholders cooperate through the International Laboratory Accreditation Cooperation (ILAC) and the International Accreditation Forum, Inc. (IAF). A principal objective of ILAC and IAF is to put in place world-wide, multi-lateral Mutual Recognition Arrangements (Arrangements). Both ILAC and IAF aim to demonstrate the equivalence of the operation of their Member Accreditation Bodies through these Arrangements. As a consequence, the equivalent competence of conformity assessment bodies accredited by these accreditation bodies is demonstrated. The market can then be more confident in accepting certificates and reports issued by the accredited conformity assessment bodies.

ILAC and IAF are linking the existing multi-lateral, mutual recognition Arrangements of the regional accreditation cooperations (Regional Groups) and are encouraging the development of new Regional Groups to complete world-wide coverage. For the purposes of their Arrangements, both ILAC and IAF delegate authority to their “recognized” Regional Cooperation Body Members (Regional Groups) for the evaluation, surveillance and re-evaluation of full Member Accreditation Bodies within their defined territory and associated decision making relating to the membership of the ILAC and IAF Arrangements in that territory. Formal “Recognition” of a Regional Group with respect to the ILAC and IAF Arrangements is based on an external Evaluation of the Regional Group’s competence in mutual recognition Arrangement management, practice and procedures by a team composed of evaluators from other ILAC and IAF Member Regional Groups and Accreditation Bodies.

Evaluation relating to the development and maintenance of the ILAC and IAF Arrangements operate at two levels:

the Evaluation of the competence of single Accreditation Bodies to accredit;
the Evaluation of a Regional Group’s competence in managing the operations of regional mutual recognition Arrangements.

The general requirements to be used by ILAC and IAF and their recognized Regional Groups, when evaluating the competence of a single accreditation body for the purpose of qualifying to sign the applicable mutual recognition Arrangement(s) are set out in IAF/ILAC A2.

The requirements to be used by ILAC and IAF when evaluating the competence of a Regional Group in managing, maintaining, and extending a regional mutual recognition Arrangement for the purposes of ILAC and IAF “Recognition” are set out in IAF/ILAC A1.

PURPOSE

This document provides a tool in the evaluation process: (1) to allow an evaluated accreditation body (AB) to present information about how it addresses thirteen topics important to its performance; (2) to focus the evaluation agenda on important topics; and (3) to provide a framework to present this information in an evaluation report. The effective date for application of this document is the date of publication.

AUTHORSHIP

This publication was prepared by a joint ILAC/IAF working group on Harmonization of Peer Evaluation Processes and endorsed for publication by the respective General Assemblies of ILAC and IAF in 2004.

This latest version was endorsed by the respective General Assemblies of ILAC and IAF in 2005.

IAF/ILAC-A3:2006 IAF/ILAC MRAs: Key Performance Indicators:

PREAMBLE......

PURPOSE......

AUTHORSHIP......

TABLE OF CONTENTS......

Introduction: The value of Key Performance Indicators (KPIs) in the peer evaluation process...

The use of KPIs by the AB under evaluation......

The use of KPIs by the peer evaluation team......

KPI 1: Access to Expertise......

KPI 2: Accreditation criteria, scope of the AB and extension of the scope......

KPI 3: AB staff, assessors and experts......

KPI 4: Assessor support system......

KPI 5: The assessment and the assessment team......

KPI 6: Impartiality of Assessors, Committees and Decision-Making Bodies......

KPI 7: Monitoring Performance of Assessors and Experts......

KPI 8: Dealing with non-conformities and corrective actions of the accredited bodies, including decision making on accreditation

KPI 9: Internal audits and management reviews......

KPI 10: Proficiency testing......

KPI 11: Calibration, traceability, and reference materials......

KPI 12: Program of surveillance activities......

KPI 13: Value-adding services......

Introduction

The KPIs in the peer evaluation process

The use of KPIs for evaluation

Abbreviations

KPI 1: Access to Expertise

KPI 2: Scope of the AB, extension of scope and accreditation criteria

KPI 3: Competence of AB staff, assessors, experts and committees

KPI 4: The assessment and the assessment team

KPI 5: Impartiality

KPI 6: Dealing with non-conformities, corrective actions of the CAB’s and decision making by the AB’s

KPI 7: Internal audits and management reviews

KPI 8: Proficiency testing

KPI 9: Calibration, traceability, and reference materials

KPI 10: Surveillance and Re-assessment

KPI 11: Supporting activities

IAF/ILAC-A3:2006 IAF/ILAC MRAs: Key Performance Indicators:

Introduction: The value of Key Performance Indicators (KPIs) in the peer evaluation process

Peer evaluation aims at establishing whether accreditation bodies are reliable and competent to deliver services according to harmonised standards. The result is a multilateral mutual recognition arrangement (Arrangement) among accreditation bodies. The Arrangement confirms reliability and competence to the market. Effectively, each accreditation body (AB) that is part of the Arrangement can confirm to its market that the result of the work of accreditation bodies signatory to the Arrangement is reliable. This necessitates a high degree of confidence between the accreditation bodies in the Arrangement. The Key Performance Indicators (KPIs) described in this document are designed to help justify that confidence. They do this by providing focus points for the evaluation process.

In formal terms, nonconformities can only be raised against the criteria contained in the relevant standard(s) and the terms of the Arrangement. The KPIs are not supplementary requirements. The KPIs do not constitute an additional basis for nonconformity. Therefore, evaluation teams cannot issue nonconformities against any part of a KPI.

The purpose of the KPIs is to provide a framework for a balanced picture of the performance of the AB. They are meant to provide a detailed description of the core processes of accreditation. They also provide a description of the situation in selected fields known to provide crucial evidence for the reliability and credibility of accreditation bodies. In this manner, KPIs complement the image given by nonconformity reports with a narrative description of the core processes.

Naturally, reports need to include the formal nonconformities found, but must also provide a picture of the ability of the AB to perform. Such reporting should provide decision-making bodies of the various Arrangements with a complete picture of the capabilities and performance of the signatories and applicants and enable sound decisions.

The considerations that call for KPIs apply to accreditation as well as to evaluation, but their development in the field of peer evaluation underlines the extent to which this process aims to confirm performance rather than conformity. Peer evaluation leads to entry into a partnership of increasingly interdependent accreditation bodies. The KPIs enable benchmarking among accreditation bodies, opening up possibilities of, not only measuring and confirming performance capabilities, but also of improving them. The focus is on outcome rather than mere conformity.

Not all the KPIs included in this document are applicable to all situations. They sometimes extend beyond the confines of the accreditation standard, and it will be a question of judgement for each peer evaluation team to decide how much to focus on a particular KPI in a specific evaluation. For example, proficiency testing may be appropriate in more situations that have been traditionally identified, but not in others. The KPIs are not to be used as a checklist but to stimulate thought in the design of the evaluation, to give hints to crucial points and to encourage the reporting of a full picture of the AB for the benefit of its potential partners. Covering them is not an end in itself.

The KPIs normally focus on aspects that the peer team is expected to highlight about the performance of the AB, but they also extend to identify aspects that the peer team will expect the AB to examine in its customers. This reflects the fact that any evaluation at the accreditation standard level (e.g. ISO/IEC Guide 58) only serves to ensure that the work of accredited bodies (operating at the level of, for example, ISO/IEC 17025 or TR 17020) are effective. In time, the KPIs may usefully expand from their present focus to provide balanced guidance on the application of ISO/IEC 17011:2004.

The use of KPIs by the AB under evaluation

An AB that is going to be evaluated (or re-evaluated) will be asked to provide the peer evaluation team with two sets of documents apart from its quality system documentation. KPIs comprise one of these sets of documents.

Prior to the evaluation, the AB will be asked to make a self-assessment and in a "written narrative description" address all relevant points in the KPIs. This will result in a complete set of KPIs filled in by the AB to be evaluated. This will be submitted to the peer evaluation team as part of the evaluation documents. The peer evaluation will thus be able to have a picture of how the AB performs from the point of view of the AB itself. This description will make it easier to understand the core processes of the AB.

The use of KPIs by the peer evaluation team

The evaluation team will use the KPIs as part of its preparation for the evaluation. During the evaluation visit, the evaluation team will find out whether the description given by the AB in the KPIs is compliant with the reality that it encounters.

It is the obligation of the evaluation team to delete, add to, re-write the KPIs describing the processes of the AB so that at the end of the evaluation the KPIs reflect exactly what the evaluation team has seen and experienced.

This tool, if submitted in a comprehensive manner by the AB prior to an evaluation, could save considerable time during an evaluation. The peer evaluators would only have to verify on-site the facts in the KPI report submitted by the AB. A broader picture of the AB activities would also assist in planning the evaluation more effectively.

Introduction

The KPIs in the peer evaluation process

Peer evaluation aims at establishing confidence whether accreditation bodies are reliable and competent to deliver services according to harmonised standards. As a result of a positive evaluation outcome, the accreditation body can join a multilateral agreement (MLA)[1] between accreditation bodies, confirming reliability and competence to the market.

The purpose of the KPIs is to provide a balanced and detailed picture of the performance of the accreditation body at core processes and focal points of its operation. The KPIs are not to be used as a checklist but to stimulate thought in the design of the evaluation, to give hints to crucial points and to encourage the reporting of a full picture of the AB. Thus, only facts and the results of the implementation of policies should be reported, not the policies themselves as they are evaluated at another place.

Findings can only be raised against the standards or agreements between the MLA signatories but they can be explained in the KPIs in appropriate detail. Such reporting, together with the other documents, provides the evaluation team and the committee of the MLA with a true picture of the capabilities and performance of the signatories and applicants and enables sound decisions.

Since the KPIs deliver a true picture of the AB, they can be used

-as a tool for the peer evaluation team

-for benchmarking between the MLA members

-as a tool for the AB for improvement.

Not all the KPIs included in this document are applicable to all situations. They sometimes extend beyond the confines of the accreditation standard, and it will be a question of judgement for each peer evaluation team to decide how much to focus on a particular KPI in a specific evaluation. For example, proficiency testing may be appropriate in more situations that have been traditionally identified, but not in others. On the other hand, the peer evaluation team must verify that the facts stated in the KPIs are valid for all fields of activity of the AB. If the procedures or outcomes in the fields are different, the situation should be clearly described.

The use of KPIs for evaluation

Prior to the evaluation, the AB will be asked to make a self-assessment, addressing all points in the KPIs. It will be submitted to the peer evaluation team as part of the evaluation documents. The peer evaluation will thus be able to have a picture of how the AB performs from its own point of view. This description will make it easier to understand the core processes of the accreditation body.

The evaluation team will use the KPIs as part of its preparation. During the evaluation visit, the team will find out whether the description given by the accreditation body in the KPIs is compliant with the reality that it encounters. It is the obligation of the evaluation team to delete, add to, re-write the KPIs describing the processes of the accreditation body so that at the end of the evaluation the KPIs reflect exactly what the evaluation team has seen and experienced.

In other words, the KPIs will start off as a report submitted by the accreditation body under evaluation and end up as a report written and confirmed by the evaluation team.

This tool, if submitted in a comprehensive manner by the accreditation body prior to an evaluation, could save considerable time during an evaluation. The peer evaluators would only have to verify on-site the facts in the KPI report submitted by the accreditation body. A broader picture of the accreditation body activities would also assist in planning the evaluation more effectively and will help the MLA members to understand better the processes in the AB.

Abbreviations

ABAccreditation Body

CAAConformity Assessment Activities

CABConformity Assessment Body

PTProficiency Testing

RMReference Material

KPI 1: Access to Expertise

References:

ISO/IEC 17011:2004: 4.2.6, 4.2.7, 4.2.8, 4.6.2, 4.6.3

Accreditation is essentially an assessment and confirmation of the competence of conformity assessment bodies (CABs) to perform specific tasks.

Attention should be paid to selection procedures, training needs and exchange of experience. These arrangements, their suitability, coverage, frequency of use, depth, performance, balance and fairness have to be examined.

An AB needs to have access to the right kind of expertise to:

1-1.Establish suitable specific accreditation criteria, applying the relevant standards as appropriate for special fields;

1-2.Assess CABs on-site;

1-3.Assess ongoing performance of its customers;

1-4.Cover its scope of offered accreditation services;

1-5.Make appropriate accreditation decisions.

KPI 2: Accreditation criteria, scope of the AB and extension of the scope

References:

ISO/IEC 17011:2004: 4.6.1, 4.6.2, 4.6.3, 7.1.1, 7.1.2a, 7.1.2b, 7.1.2j, 8.2.4

The credibility of an accreditation body (AB) and the value of its services to community in large, depend on its ability to assess the technical competence of its clients. AB should declare and document their accreditation criteria. AB that claims to work on an unlimited technical scope will be expected by their clients to act in fields or to deal with applications where it has no previous experience. AB may be involved in the assessment bodies performing different conformity assessment activities. For example inspection bodies that perform testing or use subcontracted testing or product certification bodies performing inspection and testing.

AB should have a clear policy concerning the selection of conformity assessment activities (CAAs) they accredit. A decision not to accredit a certain CAA should be taken based on such a policy and should not discriminate against parties on the market.

Thus, the AB should have and implement effective policies and procedures for establishing accreditation criteria, extending its scope of activities into new fields (in different CAAs as well as within the same type of CAA) and reacting to demands from interested parties. (see also KPI 1).

The AB should define and implement effective policies and procedures for assessing the CABs that make use of combinations of different fields of accreditation.

The AB procedures should address at least the following issues when considering an extension of scope:

2-1.Analysis of its present competence;

2-2.Analysis of the suitability of extensions

2-3.Access to resources, etc. in a new area;

2-4.Evaluation of the need for application documents;

2-5.Initial selection and training of assessors for extensions;

2-6.Training AB staff in the new area;

2-7.Arrangement issues and requirements of regulatory bodies.

2-8Co-operation with AB's in different fields of accreditation for assessment of CABs applying for more than one area of accreditation.

2-9Transparent and internationally harmonised decisions on what may be accredited and what should not be accredited.

Notes:

As possible activities of co-operation, the team should consider:

1.Joint assessments with experienced ABs;

2.Building up of long-term know-how in the AB;

3.Receiving feedback from initial accreditation activities as a tool for tuning the activities in the new field to an internationally acceptable level;

4.Accessing and employing know-how from other national or international sources.

KPI 3: AB staff, assessors and experts

References:

ISO/IEC 17011:2004: 4.3.5, 4.6.2, 6.1.1 – 6.1.3, 6.2 – 6.4, 7.5.2

The AB staff responsible for assessments has a central role in the operation of the system.

The evaluation team should evaluate whether the AB has applied rules and procedures covering the competence of the staff, assessors and experts including the following.

At an individual level:

3-1.Job descriptions,

3-2.Curriculum vitae

3-3.Fields of responsibility,

3-4.Roles and authority in the accreditation processes,

3-5.Technical qualifications and practical experience,