Measuring and Assessing the Performance of Regulators

Issue 52September 2014

Measuring and Assessing the Performance of Regulators

Mark Pearson and Simon Haslock*

Recently, the question of how best to measure and assess the performance of regulators has been gaining significant traction in Australia, and also internationally. A range of reports released in Australia over the last 12 months focus on best-practice principles for conducting performance assessments of regulators, such as the Productivity Commission’s (2014) Regulatory Audit Framework and the Australian National Audit Office (2014) revised guide to administering regulation. These reports feed into the broader objectives of the Australian Government, which is committed to regulatory reform, with the stated aims of boosting productivity, increasing competitiveness, reducing unnecessary regulation and lifting regulatory performance. Internationally, the Organisation for Economic Co-operation and Development (OECD) (2014b) has made a contribution to the field through the release in July 2014 of its own best-practice principles for regulatory policy and governance. These principles have been the subject of significant consultation with national regulators over the past two years, including the Australian Competition and Consumer Commission (ACCC). This article provides some insight on the key elements of the OECD principles, while also noting the ongoing work that is being conducted by the OECD’s Network of Economic Regulators to devise a fitforpurpose performance assessment framework for economic regulators.

Background

In 2012, the OECD’s Council on Regulatory Policy and Governance made a number of recommendations focused on developing a systematic governance framework that could deliver ongoing improvements to the quality of regulation in member countries. One of these recommendations (OECD 2014a) was that countries develop ‘a consistent policy covering the role and functions of regulatory agencies in order to provide greater confidence that regulatory decisions are made on an objective, impartial and consistent basis, without conflict of interest, bias or improper influence’. This recommendation has acted as the impetus for the OECD’s work on best-practice principles for regulatory policy and governance in the intervening period.

The Council’s recommendations were the result of assessments made by the OECD’s Regulatory Policy Committee, established in 2009 to provide intellectual and practical support to countries in their regulatory reform efforts. OECD policy analysts had identified a gap in support mechanisms for countries facing reform pressures, and established the Regulatory Policy Committee in response. The Regulatory Policy Committee aims to provide a platform to help countries adapt regulatory policies, tools and institutions and through that, support member countries to undertake effective regulatory reform. It has a broader remit than economic regulation per se, which led to the more recent decision to establish the Network of Economic Regulators.

The establishment of the Network of Economic Regulators was first considered at an April 2012 meeting of the Regulatory Policy Committee, where two major themes emerged. Firstly, appropriate governance models for regulators, including the institutional setting; and, secondly, the criteria that relate to a world-class regulator. The Regulatory Policy Committee decided to sponsor the Network of Economic Regulators on the basis that effective economic regulation is a fundamental pillar of economic reform and development.

The Network of Economic Regulators aims to be a forum in which regulators can develop best practices, identify key operational principles, provide advice on challenges and how to overcome them, and develop case studies for the benefit of member countries and their reform processes.

The Network of Economic Regulators considers the ability to move away from a pure sectoral approach (that is, communications, energy, post, ports) allows a much broader perspective to be brought to bear on regulatory issues, thereby improving the performance of individual regulators and the policy processes underpinning them.

The OECD’s Best Practice Principles for Regulatory Policy

The OECD Best Practice Principles for Regulatory Policy seek to construct an overarching framework to support initiatives to drive further performance improvements across regulatory systems in relation to national regulatory bodies or agencies. The OECD considers that efficient and effective regulators, with good regulatory management and governance practices, are needed to administer and enforce regulations.

Regulatory activity has become increasingly important in the modern state in both policy formation (regulatory design) and in policy execution (regulatory delivery) because regulators have special expertise in drawing on the relevant evidence from the natural and social sciences, including economics, finance and behavioural theory (OECD 2014b).

While the OECD acknowledges that there are different institutional models for regulators, it considers improving the governance arrangements of regulators can benefit the community by enhancing the effectiveness of regulators, and, ultimately, the achievement of important public policy goals.

The OECD (2014b, p. 19) has identified two broad aspects of governance relevant to regulators, which are:

external governance (looking out from the regulator) – the roles, relationships and distribution of powers and responsibilities between the legislature, the minister, the ministry, the judiciary, the regulator’s governing body and regulated entities; and
internal governance (looking into the regulator) – the regulator’s organisational structure, standards of behaviour and roles and responsibilities, compliance and accountability measures, oversight of business processes, financial reporting and performance management.

The OECD Best Practice Principles for Regulatory Policy mainly focus on external governance arrangements and their effect on the performance of regulators. However, some important elements of internal governance are addressed, including performance evaluation for regulators. Internal governance is the main focus of this article.

The OECD has identified seven principles for good governance, being:

role clarity;
preventing undue influence and maintaining trust;
decision making and governing body structure for independent regulators;
accountability and transparency;
engagement;
funding; and
performance evaluation.

Below is a summary of some of the key guidance provided by the OECD in relation to measuring and assessing the performance of regulators. Where relevant, the ACCC’s approach to these issues has been drawn upon as a case study in the Australian context.

Accountability and transparency

The OECD suggests that a good mechanism for ministers and regulators to achieve clear expectations is for ministers to issue a statement of expectations to each of their regulators. These statements should outline relevant government policies, including the government’s current objectives relevant to the regulator, and any expectations on how the regulator should conduct its operations (OECD 2014b, pp. 81-82). The regulator should then formally respond by outlining how it proposes to meet the expectations of government in its corporate plan or a statement of intent. This document should include key performance indicators (KPIs) agreed with the relevant minister.

This process is already in place in Australia. Earlier this year, the Australian Government issued its Statement of Expectations to the ACCC and other regulators (Australian Treasury 2014a). The ACCC responded in turn with a Statement of Intent (ACCC 2014a). The ACCC has published both documents on its website for review by the public.

In addition to publishing objectives, the OECD recommends that regulators produce and publish clear operational policies covering compliance and also enforcement and decision reviews. The regulator should also disclose what rules, data and informational inputs will be used to make decisions (OECD 2014b). The ACCC already implements these measures, updating the ACCC Compliance and Enforcement Policy on a regular basis to ensure its priorities and strategies remain relevant (ACCC 2014a). The ACCC also provides public versions of all draft and final regulatory determinations on its website.

Performance evaluation

The OECD notes that it is important that regulators are aware of the impacts of their regulatory actions and decisions. This will help drive improvements and enhance systems and processes internally. It also helps to build confidence in the regulatory system. The OECD considers this is best achieved by the identification and implementation of performance measures (OECD 2014b, p. 107).

The OECD recommends that the regulator should report against a comprehensive set of meaningful performance indicators, set with reference to the goals it is expected to achieve. These indicators should incorporate quantifiable aspects of the regulator’s activities that provide metrics to assess its performance and the costs that it imposes (OECD 2014b, p. 107). Regulators should consider which operational indicators can be used to demonstrate the systems, processes and procedures that are applied within the organisation to complete tasks. In addition, regulators should consider which outcome indicators can be linked to their actions to demonstrate the overall strategic results of regulatory intervention (for example, investment in infrastructure) (OECD 2014b, p. 106).

The OECD suggests that regulators should conduct internal performance evaluations as part of good internal governance practices. These should be complemented by external evaluations. The OECD notes that, while regulators have a number of audiences for their performance evaluation, including government, regulated entities and citizens, the main purpose of the evaluation should be towards achieving self-improvement and accountability (OECD 2014b, p. 108).

Traditionally, the ACCC’s primary reporting mechanism on its performance has been its Annual Report, which has been produced in conjunction with the Australian Energy Regulator.[1] This report has responded to the framework in the Treasury portfolio budget statements, which outlines a number of outcomes the ACCC is expected to achieve during a financial year (Australian Treasury 2014b). The Annual Report provides a detailed account of deliverables against each of these outcomes.

The ACCC, along with all other Commonwealth entities, is currently working with the Australian Government in relation to the development of a Commonwealth Performance Framework. The Public Governance, Performance and Accountability Act 2013 (PGPA Act), which came into effect on 1 July 2014 and replaced the Financial Management and Accountability Act 1997, established the Performance Framework as one of its four core objectives. The Performance Framework is being developed to promote improvements in the quality, reliability, and availability of descriptive and instructive information about the non-financial performance of Commonwealth entities.

The Public Governance, Performance and Accountability Act 2013 establishes a number of requirements that are related to the development of the Performance Framework (Parliament of the Commonwealth of Australia, 2013). These include:

a new requirement that all Commonwealth entities prepare a Corporate Plan;
a new requirement that all Commonwealth entities prepare an Annual Performance Statement; and
a restatement of the current requirement for all Commonwealth entities to prepare Annual Reports.

Performance statements will be part of an integrated Annual Report that brings together information about an entity’s strategy, governance and financial and non-financial performance. A copy of the statement will need to be included in an entity’s Annual Report when it is tabled in Parliament.

Development of a Performance Assessment Framework for Economic Regulators

Building on the OECD Best Practice Principles for Regulatory Policy, the OECD’s Network of Economic Regulators is committed to developing a Performance Assessment Framework for Economic Regulators. It is widely accepted that there are many challenges in performance measurement and assessment and developing appropriate KPIs for economic regulators. For example, gaming by businesses can impact significantly on the ability of regulators to achieve desired outcomes in regulatory processes. However, these challenges must be overcome as governments, parliaments and other stakeholders are increasingly demanding assurances around effectiveness, efficiency and impact of economic regulators.

Regulators play key roles in various industry sectors, thereby influencing growth, development and investment. Having an effective measurement framework is likely to lead to better outcomes for society, by providing for robust benchmarking of regulatory performance to ensure regulators are meeting the objectives they were established for and achieving value for money.

Discussions amongst regulators at the OECD have been aimed at identifying ways of measuring their performance and impact on the sectors they regulate, and economic welfare more broadly. These discussions have included case studies on the development of KPIs, regulatory audits (especially of economic regulators undertaken by the UK’s National Audit Office)[2] and papers prepared by the Secretariats of the Regulatory Policy Committee and the Network of Economic Regulators.

The Performance Assessment Framework for Economic Regulators that is being developed under the auspices of the Network of Economic Regulators aims to operationalise in an economic regulatory context the principles of the Regulatory Policy Committee’s Framework for Regulatory Policy Evaluation (OECD 2014a). This framework was developed by the OECD Secretariat, working with member countries, to address the lack of guidance on exactly how to undertake assessments of regulatory policy within individual jurisdictions. There was also a recognised need to develop appropriate methodological tools to assist jurisdictions and those undertaking the assessments. The aim of the framework is to provide countries with a methodology that assists the capture of information sufficient to allow them to make decisions about where to invest scarce resources.

While there are particular issues relevant to specific sectors, there is a growing understanding of the broad similarities and issues that all economic regulators face in measuring and assessing performance. The Performance Assessment Framework for Economic Regulators recognises and will build on the efforts of a number of economic regulators to develop assessment processes for their own performance. It aims to help fill any gaps in particular regulators’ work programs and to provide solutions across sectors through a common framework for regulatory learning.

The Network of Economic Regulators is currently testing the Performance Assessment Framework for Economic Regulators, including utilising a questionnaire to assess Columbia’s telecommunications regulator, the Comision de Regulacion de Comunicaciones. A set of recommendations will flow from this assessment and include:

information on approaches and methodologies for setting measureable objectives and targets;
methodologies for developing specific indicators;
measurement techniques; and
the institutional processes and arrangements for using performance measurement.

The Australian Context

As noted earlier, a number of reports have been released in Australia focusing on the performance of regulators. Two in particular provide frameworks for measuring and assessing this performance. The Productivity Commission’s Regulatory Audit Framework (2014) offers guidance for auditing the performance of regulators in regard to the compliance costs they impose on business and other regulated entities. The Australian National Audit Office’s Better Practice Guide – Administering Regulation (2014) has a broader focus, providing a framework to assist regulators in assessing the quality of their administrative practices and identifying improvements that can be made.

The Productivity Commission (2014, p. 5) asserts that the need for a process to audit regulator performance reflects ongoing concerns that the way some regulators interact and engage with businesses and other regulated entities is responsible for much of the unnecessary cost imposed by regulation. The Productivity Commission (2014, p. 5) considers there is currently ‘no systematic process by which the costs that regulators impose on business are assessed ex post.’

The Productivity Commission’s (2014, pp. 9-10) audit framework sets out a number of steps that it considers should be taken by a regulator to measure and assess its performance, with particular regard to the compliance costs they impose. These steps are:

Establishment of an agreed set of indicators of good performance appropriate to each regulator. This should be documented in an audit plan. It should form part of the regulator’s stated intent for administering their regulations in a way that imposes the least cost on business.
Collection of information and data on the chosen indicators. The audit plan should set out what data should be collected for annual reporting, and the form in which they should be collected and collated.
Conduct of an external audit. A written assessment of the regulator’s performanceagainst the indicators should be published in a central location.

The Productivity Commission notes (2014, p. 15) that a key step in developing an audit plan is identifying the particular metrics or measures that will reflect achievement of the chosen indicators (step one, above). The Productivity Commission suggests that the metrics chosen should ideally reflect outcomes rather than processes or outputs.

The Australian National Audit Office considers that well-documented and carefully-structured management systems and procedures provide a regulator with the tools to define regulatory outcomes and administrative priorities, and measure and report on performance (Australian National Audit Office 2014, p. 27). The Australian National Audit Office suggests that performance information systems should be designed to inform internal and external stakeholders about the performance of the agencies’ activities including (p. 27):

whether the regulation is achieving the Australian Government’s stated policy objectives;
the costs associated with administering the regulation; and
the cost of compliance for regulated entities.

The Australian National Audit Office outlines a set of key considerations in measuring, reporting and reviewing regulatory performance. These considerations are similar in nature to the steps proposed by the Productivity Commission. Initially, the Australian National Audit Office suggests that a regulator should define relevant effectiveness and efficiency indicators to support reporting for internal management and external accountability purposes. The regulator should then undertake periodic reviews to consider the effectiveness of the regulation being administered, and the efficiency and effectiveness of the agency’s regulatory administration. Throughout this process the regulator should draw on stakeholder views to understand their expectations about the effectiveness of the regulatory regime, whether an appropriate balance is being achieved in relation to risk, the underlying regulatory burden, and the efficiency and effectiveness of the regulatory regime (Australian National Audit Office 2014, p. 28).