Module 8 – Review and Evaluation

  • Better Regulation: The Search and the Struggle – Robert Baldwin (2010)Available on Docushare
  • Toward a Culture of Persistent Regulatory Experimentation and Evaluation – MichaelGreenstone (2009)
  • The Evaluation of Regulatory Agencies – Jon Stern (2010)
  • The Making of Good Supervision: Learning to Say “No” – IMF Staff Position Note(2010)
  • The business of sustainable salmon – Gabrielle Walker (2017)

Supplementary/Reference:

  • Successful Leaders Focus on Strong Execution – Harvey Schachter (2017)

Available on Docushare

Module 8 –Executive Summary of Readings

Better Regulation: The Search and the Struggle – Robert Baldwin (2010)

Most government administrations often seek to deliver “better regulation” through initiatives designed to improve the delivery of high quality regulation. Such efforts tend to encounter three central challenges: benchmarks, strategies, and measurement.

Benchmarks: Concerns for allocative efficiency have driven regulatory reform in recent decades, however, there is difficulty in identifying the social objectives that good regulation should seek to provide. Some prefer to focus on Rawlsian social justice, or Sunstein’s prescriptive regulations based on autonomy and welfare. Others focus on evaluating regulation based on consensus reached between relevant entities. Others still have suggested understanding evaluation in terms of the degree to which regulation is implemented in a cost-effective matter, the fairness of decisions and policies, accountability of relevant actors, and transparency and due process. Regardless of the preferred method, the challenge of weighing the various considerations remains – regulators must justify a given balancing of benchmarks. Efforts have been made by a number of governments to provide guidelines on weighing benchmarks; reports published by the UK Better Regulation Task Force (2003), Australia’s Regulatory Performance Indicators (1999) and Office of Regulatory Review (2003), the USA Program Assessment Rating Tool (2002), the World Bank (2004), the EU (2002), and the OECD (2005). These reports are consistent in underlining the importance of low-cost, least intrusive, evidence-based, and transparent regulations. In addition, efforts to address some benchmarks have spillover effect into others. Finally, the same set of guidelines can lead to diverse approaches given differences of philosophy, policy, and implementation.

Strategies: OECD countries have differed in their pursuit of better regulatory systems. Some have focused on prospective policies and instruments, while others have invested in improving the existing stock of regulations. Some have aimed to reduce regulatory and administrative burdens, while others have focused on strengthening compliance and enforcement. The OECD’s Regulatory Impact Assessments (RIAs) are at the centre of the regulatory reform conversation – they recommend a number of criteria for states to pursue in order to achieve regulatory excellence. Many governments have pursued many of these goals simultaneously, which has at times produced confusion and inconsistency. Three contradictions stand out: the clash between an expert, evidence based regulation and the desire for less intrusive regulation; the intent to target enforcement activities while reducing the information/reporting burden on businesses; and the inability of regulation to actually influence the policy and legislative processes, despite rhetoric to this regard.

Measurement: ‘Better regulation’ must involve the measurement of the quality of regulation as well as the performance of regulatory improvement tools, institutions, andpolicies. Policymakers need to know whether compliance and performance is improving, and whether this is due to reforms to the regulatory system, or independent activities of regulated entities. However, this measurement is extremely challenging because it depends on benchmarks and policy objectives, and the level of quality sought will vary according to audience, market position, and sector. For example, the Dutch and Belgian regulatory frameworks assess regulatory excellence according to administrative burdens and regulatory complexity, whereas the American and Canadian systems focus on regulatory outcomes and benefits to citizens.

The idea of ‘smart regulation’ is to find optimal mixes of control methods that can be applied by thestate as well as other actors including trade associations, stakeholder groups, and corporations. Itrequires a reflexive, dynamic approach in which regulatory strategies are constantly revised accordingto changes in circumstances and preferences – these ongoing “check and adjust” activities are notamenable to evaluations in a one-time process. Further, those proposing regulatory reforms might beinclined to develop straightforward regulatory designs that can be fed through the machinery of the

RIA process; such bureaucratic incentives may offset the incentives to adopt a complex, multi-stakeholder,problem-centered approach. The worry is that if the RIA process is at the core of ‘betterregulation’ efforts, this might not lead to ‘smart’ mixes of regulatory instruments.

Measuring regulation requires deciding what is to be the focus: the quality of regulatory design, itsimplementation and enforcement, or whether desired outcomes were achieved. Further, it must bedetermined whose regulatory performance is being assessed. Modern regulation is carried out withinnetworks of controls that involve various regulators, controls, and policies; decentralized systems inparticular are susceptible to regulatory outputs resulting from accumulations of regulatory systems thatmay vary according to the degree of coordination among stakeholders.

Questions provoked:

  1. What is your organization’s dominant motivation for undertaking evaluation: to assess the quality ofregulatory design, assess quality of implementation and enforcement, or determine whether outcomesare being achieved?
  2. Are the assessments of design, implementation and enforcement secondary objectives important inunderstanding, first and foremost, why outcomes are being achieved or not?

Key takeaways for CPRL:

  • Policy tensions in a given regulatory context must be mapped out in order for regulators to progress towards a better system.
  • The application of benchmarks and measurements are contentious, and trade-

Toward a Culture of Persistent Regulatory Experimentation and Evaluation – Michael Greenstone (2009)

The current system for making regulatory choices is broken. It is largely based on faith, rather than evidence. Many regulations are only evaluated before they are implemented. The result is that our regulatory system frequently takes shots in the dark and we often fail to measure if intended targets have been hit. Greenstone calls for a move toward a culture of persistent regulatory experimentation and evaluation. His goal is to rigorously evaluate every regulation in order to expand upon the ones that work and weed out the ones that fail to improve our well-being (or worse, harm it). At the heart of such reform is the recognition that we cannot fully know a regulation’s benefits and costs until it has been tested. The rewards for such testing are better regulations.

Proponents of a new regulation inevitably argue that its benefits are substantial, while opponents inevitably argue that the costs are too high. The difficulty is that the evidence needed to assess such claims is almost always unavailable, and therefore many regulatory decisions are based on rhetoric.

Ideally it would be feasible to observe simultaneously the same subjects in two different states: one in which the regulation is applied, and one in which it is not. Of course, it is impossible to observe both states simultaneously. This impossibility is labeled the fundamental problem of causal inference. This problem is relevant for cost-benefit analysis in that many regulations are implemented for an entire population, which makes it impossible to develop a valid counterfactual case for what would have happened in the absence of a regulation’s implementation. In the absence of a counterfactual, it is impossible to know the policy’s causal impacts. Another problem occurs when a regulation is applied to some people or places and not to others, while these two differ in important ways. This is called selection bias and it occurs when there is a control group, but the regulated or treatment group differs from the control group. These issues can be addressed by conducting a randomized experiment, whereby it is valid to assume that a comparison of outcomes among treatment and control groups yields an estimate of the causal effect of the regulation.

The current regulatory problem is not a lack of cost-benefit analysis. Rather, the problem is the poor quality of evidence and the assumptions underlying many applications. Most regulations are subject to a cost-benefit analysis only in advance of their implementation. This is the point when the least is known and any analysis must rest on many unverifiable and potentially controversial assumptions. The key is to develop a culture of regulatory experimentation and evaluation that involves rigorously testing all regulations and then expanding upon the regulations that work and dropping the ones that do not. The US Food and Drug Administration’s drug approval process is a good example of this. The key to regulatory reform is to instill a culture of experimentation and evaluation. This can be established in four steps: Experiment (structure regulations so that evaluations are feasible, fund evaluations, collect data, and publicly release the data); Create a Regulatory Review Board; incorporate Automatic Sunset and Expansion Provisions; and Develop and Apply a Code of Ethics.

Question provoked:

What are the greatest barriers in your organization that slows adoption of a culture of experimentation and evaluation?

Key takeaways for CPRL:

  • Pursuing cost-benefit analyses only in advance of a regulation’s implementation is effectively the “government tying at least one hand behind its back”.
  • We are currently in an era of scrutiny and reconsideration of the role of government. This opportunity should be harnessed and the four key reforms should be implemented in the regulatory system.

The Evaluation of Regulatory Agencies – Jon Stern (2010)

The term ‘impact assessment’ has often been used to cover both ex ante (before the fact) assessments and ex post (after the fact) evaluations. Evaluating regulatory agencies is an especially difficult task, given the technical and economic considerations that must be taken into account.This chapter focuses on the economic regulation of infrastructure industries such as electricity, telecoms and water, and on the evaluation of outcomes. Although consumer protection is the core function of regulators, there are major differences between those regulating different industries and sectors. Infrastructure regulation has at its heart the issue of regulating monopoly networks that have substantial economies of density, scale, and scope.

Political and budgetary pressures are initially always for ex ante assessments. However, after a while, pressure for ex post evaluations grows. This is in order to see how well the initial assessments performed as guidance and also to learn lessons for the future. Major pressures for evaluations often arise when there is a perceived need for a strategic review of a line of policy. The World Bank has always taken ex post evaluation very seriously and has an Independent Evaluation Group that conducts regular evaluations of programs and strategies. A major issue for the World Bank is always how far approaches should or should not vary between international regions and countries.

The World Bank Handbook for Evaluating Infrastructure Regulatory Systems proposed three levels of evaluation: ‘Short, Basic’; ‘Mid-Level’ (the main focus of the Handbook); and ‘In-Depth’. The first is intended as a diagnostic check on the basic characteristics of the sector via a structured questionnaire. This has good coverage on regulatory governance issues and industry structure, but relatively little on outcomes – primarily a few open- ended questions on successes and setbacks. The mid-level evaluation is intended to probe rather harder on regulatory governance issues (legislative and in practice) but is the first level at which evaluation of outcomes can seriously be achieved. The in-depth evaluation is essentially the mid-level evaluation but with much more probing and a wider remit.

A regulatory system can be effective only if it satisfies three basic principles:

  • Credibility: Investors must have confidence that the regulatory system will honour its commitments.
  • Legitimacy: Consumers must trust that the regulatory system protects them from monopoly power.
  • Transparency: Investors and consumers must know the ‘rules of the game’.

Governments often have multiple objectives and they may conflict with one another. In those circumstances, a major function of the regulator is to identify potential conflicts and to discuss with governments how they might be resolved. The most common problems are where governments have policy targets, which require substantial industry investment, but the government is unwilling to allow prices to be raised to cover the expenditure. Whether and how far regulators further those objectives is an important issue in evaluating the performance of regulatory agencies. A fundamental evaluation issue is how to isolate the impact of regulatory decisions from those of other major factors. This means that the evaluator must be aware of and take account of all relevant factors besides regulatory decisions. This is because these other factors often have a larger impact on industry outcomes than regulatory decisions.

Questions provoked:

  1. Do regulators infrequently evaluate the effectiveness of their regulations because they think the evaluations will be politically controversial?
  1. How best can Canadian regulators overcome this risk?

Key takeaways for CPRL:

  • The growth of regulatory evaluations is from the political and general population’s demands for greater government accountability and the need to monitor performance.
  • The regulator’s task is facilitated the more clearly and fully governments state their policy objectives

The Making of Good Supervision: Learning to Say “No” – IMF Staff Position Note (2010)

The quality of financial sector supervision has emerged as a key issue from the financial crisis.While most countries operated broadly under the same regulatory standards, differences emerged in supervisory approaches. While progress is being made in putting regulation in place, work remains to be done in many countries to strengthen supervision. Key elements of good supervision are that it is intrusive, skeptical, proactive, comprehensive, adaptive, and conclusive.

The ability to supervise requires appropriate resources, authority, organization and constructive working relationships with other agencies. Supervisors must be willing and empowered to take timely and effective action, to intrude on decision-making, to question common wisdom, and to take unpopular decisions. Developing a will to act requires that supervisors have a clear and unambiguous mandate, operational independence coupled with accountability, skilled staff, and a relationship with industry that avoids regulatory capture. These essential elements must be given as much attention as regulatory reforms.

Supervision is about implementation, monitoring, and enforcement of regulations, but also the task of figuring out whether an institution’s risk management controls are adequate, and whether the institution’s culture and its appetite for risk significantly increase the likelihood of problems.

Supervision failed to recognize or address growing risks in the financial sector before the 2008 global financial crisis. Although the specific reasons differed according to jurisdiction, a general examination highlights the following lapses in supervision:

  • Not intruding sufficiently into the affairs of regulated institutions.
  • Not being proactive in dealing with emerging risks and adapting to the changing environment.
  • Not being comprehensive in scope.
  • Not taking matters to their conclusion.

The analysis of financial sector supervisory and regulatory standards shows that while most countries have the necessary legislation, regulations, and supervisory guidance appropriate to their national systems, a significant proportion of these does not do as well when it comes to the nuts and bolts of supervision across the different sectors.

Among the deficiencies in risk supervision are the lack of supervisory awareness and training; inadequate and dated tools and methodologies to evaluate risk management approaches; and the absence of authority to require regulated entities to take protective measures against risks. In the case of enforcement powers, while most countries have a range of legal powers to take action, generally there is a lack of clarity as to the means by which the sanction meets the severity of the infringement—resulting in the powers not being applied consistently, regulatory forbearance, and supervisory actions not being seen as credible.

Only if supervision is strengthened can regulators hope to effectively deliver on the challenging regulatory reform agenda. For this to happen, society must stand with supervisors as they play their role as naysayers even when everything seems to operate smoothly. The policy and institutional environment must support both the supervisory will and ability to act.

Question provoked:

Does your organization have a strategic focus on evaluation training for your staff?

Key takeaways for CPRL:

  • Key elements of good supervision are that it is intrusive, skeptical, proactive, comprehensive, adaptive, and conclusive.
  • The “ability” to supervise must be complemented by the “will” to act.

The Business of Sustainable Salmon – Gabrielle Walker (2017)

This piece by Gabrielle Walker narrates the story of the Global Salmon Initiative (GSI), a collaboration of companies from around the world that was created in 2012, whose members produce more than half of all farmed salmon. Their stated aim is for the farms of all their members to achieve a “gold standard” certification for sustainability—one that’s so stringent that the World Wildlife Fund (WWF) endorses it. The idea is to improve the sustainability of farmed salmon across the entire sector, and hence to earn the social license to grow. They believe this will allow them to play a greater role in satisfying the world’s increasing appetite for protein—to the benefit of all players.

Avrim Lazar, the current head of the GSI secretariat, was previously the Canadian policy lead for the Kyoto climate agreement, and as CEO of the Forest Products Association of Canada successfully brokered a deal between its members and a host of environmental groups including Greenpeace. According to Lazar, the real challenge is to get change at both speed and scale. The companies involved have to do three important things: set ambitious goals that can be measured; be radically transparent about their practices; and share solutions very widely among themselves so that their collective progress accelerates.