/ EUROPEAN COMMISSION
HEALTH AND CONSUMERS DIRECTORATE-GENERAL
Food and Veterinary Office
Quality, planning and development

Grange,

D(2012)

Note to the file

Subject: First meeting of the "Effectiveness" sub-group of the Working Group on National Audit Systems, held at Grange, 15 May 2012

Participants: Ann-Christine Salomonsson, Susanne Sylven (SE), Tony Devlin, Eibhlin O' Leary (IE), Chris Hagen, Rob de Heus (NL), Dorit Stehr (DE), Thomas Luethi (CH), Carla Campagnoli (IT), Juha Juntilla, Jenny Egan, Maura Waters (part), Aidan O'Connor (part), Klaas Steijn (part) Andries Olie (part), (FVO).

Apologies for absence: Jean-Baptiste Rodes

The session began with a presentation by Aidan O'Connor (FVO, Unit F1) of a working paper on the application of requirements for verification in the context of Article 8 of Regulation 882/2004. While this paper was not strictly within the scope of the sub-group, it was considered potentially relevant as CA verification systems could be audited to determine their effectiveness. MS participants provided feedback and were invited to provide further written feedback following consultation with colleagues in their own MS (a revised version of the document was circulated to MS participants by email on 16 May).

Several MS participants suggested that in the recast of Regulation 822/2004, it would be helpful if certain aspects, such as verification, supervision, objectives and the role of the 'PDCA' cycle, could be clarified.

Carla Campagnoli described the system used in Italy for checking effectiveness, with three stages: at the planning stage (a priori), in real time, on-the-spot (proficiency test) and afterwards (a posteriori), determining achievement of indicators/planned outputs.

This was followed by the presentation of an example from the Swedish system, presented by Susanne Sylven:

The sub-group then sought to agree a more precise scope for their considerations, as the current brief, 'effectiveness', was too general to allow meaningful discussion. In this context a number of concepts were raised. These include:

  • The definition of effectiveness. It was agreed that for consistency we would use the same definition used in the risk-based planning guidance document;
  • The need to consider the initial risk assessment/prioritisation of controls;
  • The need to consider follow-up, corrective action and sanctions;
  • The need to be able to determine if there is continual improvement over time;
  • The problem of interpretation of results that can arise of a high level of detection of non-compliance where the risk assessment/prioritisation of controls has been good (the need for information on the 'baseline' level of compliance);
  • The importance of suitable objectives/planned outcomes if effectiveness is to be determined;
  • The relevance of the PDCA cycle;
  • The scope of this group does not include developing guidance on the 'compliance' aspects of audits, but should focus on determining whether 'arrangements are implemented effectively and are suitable to achieve objectives' (article 2(6), regulation 882/2004, from the definition of 'audit'). Thus the question to be addresses in 'effectiveness' auditing is not "are they doing things right?" but rather "are they doing the right things?";
  • The audit of effectiveness should not seek to duplicate the verification to be done by CAs (article 8), however the arrangements put in place by CAs to verify the OCS could be audited to determine their effectiveness;
  • The focus should be on the system, not just on verifying the correctness on individual official controls, however, on-the-spot verification of audit findings is necessary;
  • The legal basis available may be a limiting factor for some auditors;
  • The importance of consideration of activities which the support the OCS (and are required by Regulation 882/2004) but which are not actually official controls was highlighted. Examples include training, co-ordination/communication between Competent Authorities and responses to alerts;
  • Whatever guidance this group produces should be practical;
  • To determine effectiveness there is a need to focus not on outputs such as the number of controls done (this would fall within the scope of auditing compliance with planned arrangements) but on the outcomes, such as the improvement in FBO compliance, achieved.

The group agreed the following 'working text' that provided a common position for further discussion but can be further refined as necessary when the guidance document is drafted:

Title: How audits assess the effectiveness of official control systems

Scope: Include considerations of official control systems OCS):

  • at local and national level
  • Inputs and outputs of OCS
  • Processes of OCS
  • Outcomes

But does not include:

  • Monitoring
  • Surveillance
  • FBO controls
  • Impacts

Objectives:

  • to produce (practical) guidance for Member States (MS) to audit the effectiveness of the OCS
  • to assist the MS to achieve the objectives of Article 4(6) of Regulation 882/2004
  • to share best practices.

As discussion continued on the basis of the agreed title, scope and objectives, clear differences of views regarding the best approach to auditing effectiveness emerged. Some favoured an approach based on analysis of the OCS, with subsequent verification on the spot, as necessary. Others favoured an approach beginning with examination of outcomes on the spot, with a root cause analysis to identify systemic failures, where problems were encountered. It appeared that the approach preferred may depend on the context in which the auditing body operated:

  • Where the audit body is considering effectiveness in the context of an audit that also examines compliance, and/or where the audit body regularly audits the auditee, the components of the OCS may already be well know to the auditor and a separate analysis of these components to determine 'effectiveness' may not be the most efficient approach;
  • Where the audit body is auditing purely to determine effectiveness or where the auditing body does not regularly audit the auditee and hence is not already familiar with the components of the OCS, an analysis of the components of the OCS may be the most efficient starting point.
  • The best approach to auditing effectiveness may be influenced by the context in which the auditing body works. Thus the guidance should provide a 'toolbox' where auditing bodies can find things to help them in their particular context;

Therefore the group concluded that any guidance should provide a 'toolbox' where auditing bodies can find things to help them in their particular context. It was proposed that examples of a number of different approaches be included in an annex to the guidance. MS participants were invited to provide such examples, based on their preference/experience.

One example of the use of outcomes to evaluate effectiveness suggested was the use of data on the shift of establishments from the high risk category to a lower risk category (based on lower levels of non-compliance with food safety requirements) over time.

It was also suggested that it would be useful if the group could identify general principles to be applied when auditing effectiveness, as the group on risk-based planning did when producing their guidance document.

The group agreed to continue its work with an e-mail exchange and discussion of the abovementioned examples.

The annex to this note includes the legal provisions that were particularly relevant to this discussion.

Jennifer EGAN


Annex

Relevant Legal Provisions:

Regulation (EC) No 882/2004 of the European Parliament and of the Council on official controls to verify compliance with feed and food law, animal health and animal welfare rules

Article 2(6): definition of ‘audit’

“Audit” means a systematic and independent examination to determine whether activities and related results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives

Article 4(6): operational criteria for competent authorities

Competent authorities shall carry out internal audits or may have external audits carried out, and shall take appropriate measures in the light of their results, to ensure that they are achieving the objectives of this Regulation.

Commission Decision 2006/677/EC setting out the guidelines laying down criteria for the conduct of audits under Regulation (EC) No 882/2004 of the European Parliament and of the Council on official controls to verify compliance with feed and food law, animal health and animal welfare rules

Purpose and Scope – Annex, point 1

….. The purpose of audit systems is to verify whether official controls relating to feed and food law and animal health and animal welfare rules are effectively implemented and are suitable to achieve the objectives of the relevant legislation, including compliance with national control plans….

6. Implementation of the Audit Process

6.1. Guiding Principles: (a) Compliance with planned arrangements; (b) Effective implementation; (c) Suitability to achieve objectives

To comply with the requirements of Article 4(6) of Regulation (EC) No 882/2004, the audit system should cover the following three points set out in Article 2(6):

(a) Verification of compliance with planned arrangements in order to provide assurances that official controls are carried out as intended and that any instructions or guidelines given to staff carrying out the controls are followed. This may largely be addressed by document review, but will also require on-site verification. The audit team will require good generic audit knowledge and skills to address this audit objective.

(b) Verification of the effective implementation of planned arrangements. In order to assess effectiveness, that is the extent to which planned results are achieved, on-site operational implementation must be included. This should include an assessment of the quality and consistency of the controls and should involve on-site audit activities.

The audit team will require the relevant technical expertise in order to address this audit objective.

(c) The audit system should also seek to assess whether the planned arrangements are suitable to achieve the objectives of Regulation (EC) No 882/2004, and in particular the single integrated multi-annual national control plan. This should include assessing the suitability of official controls, with regard, for example, to their frequency and the methods applied, having regard to the structure of the production chain(s) and to production practices and volume. The audit team should have substantial knowledge and understanding of system auditing, together with relevant technical input to address this audit objective.

In order to determine whether the planned arrangements are suitable to achieve the objectives set out in (c) above, the following should be considered:

Audit criteria should include strategic objectives stemming from Regulations (EC) No 178/2002 and (EC) No 882/2004 (including the single integrated multi-annual national control plan) and national legislation.

The primary focus of audits should be the control arrangements relating to the critical points for control in the production chain(s). The emphasis should be on assessing whether planned arrangements are capable of delivering sufficient guarantees on (a) the safety of the end-product(s) and (b) compliance with other feed and food law requirements and with animal health and welfare rules. In order to achieve this, audit(s) should where possible extend beyond and across administrative boundaries.

5