Policy influence of indicators – POINT
Pia Frederiksen1, Zuzana Barankova2, Tom Bauler3, Simon Bell4, Louis Cassar5, Elizabeth Conrad5, Ken Eason6, Henrik Gudmundsson7, Claus Hedegaard Sørensen7, Zita Izakovicova2, Petrus Kautto8, Markku Lehtonen9, Jari Lyytimäki8, Stephen Morse10, Janne Rinne8, Lea Sébastién11
1National Environmental Research Institute, AarhusUniversity,
2Institute of Landscape Ecology, SlovakAcademy of Sciences,
3Université Libre de Bruxelles,
4Communications and Systems Department, MCT Faculty, Open University
5Institute of Earth Systems, University of Malta
6Bayswater Institute
7Department of Transport, TechnicalUniversity of Denmark
8Finnish Environment Institute
9Science and Technology Policy Research, University of Sussex
10Centre for Environmental Strategy, University of Surrey
11Université de Toulouse II-Le Mirail
Abstract
Indicators and indicators sets are increasingly being supplied for communicating evidence to policy processes and to the broader public. The paper presents an EU-FP7 project, POINT (Policy Influence of Indicators) which has aimed to elucidate the roles indicators may play in policy processes and contexts, and some of the factors that may
Keywords: Indicators, influence, sustainable development policy
1.Introduction
The demand for and supply of indicators for environmental and sustainability policies have increased during the last decades. Main drivers behind this trendinclude a wish from international institutions tocompare the environmental or other performance across countries and sectors, a need to satisfytransparency and accountability requirements in policy performance evaluations, and a call for general information and communication with the public on sustainable development and the state of the environment.
But are such indicators actually used in policy processes and do they have any influence on policy outcomes? These were the key questions posed in the EU FP7 project POINT – Policy Influence of Indicators.
The POINT project began in 2008 with the aim of exploring the use and influence of indicators broadly with the area of sustainable development policies as its main focus. A number of case studies were conducted, covering indicators in sector integration,indicators for sustainable development and also composite sustainability indicators, such as the Ecological Footprint (EF).
2. Background
Indicators are developed and applied for a variety of purposes. They may be aimed for monitoring and evaluation of progress towards specific policy objectives. Another major area of use is the production of national stateoftheenvironment reports, which employ a range of monitoring indicators and other indicators. At the supra-national level, indicators are used for comparison of environmental performance across countries, such as in the European Environment Agency’s(EEA) State Of the Environment Report (SOER), or for thematic assessments, such as those produced by the EEA or by OECD. The main purpose in these effortsis to communicate complex information in a simplified way to policy makers, to stakeholders, and essentially to the public at regional, national and local levels.
One basic idea behind the development of indicators for monitoring and performance evaluation of policies is making operational the concept of evidence based policymaking – indicators being viewed as knowledge-agents serving simplification and communication of evidence in a form suited for policy- and decision-makers. Such an approach is rooted in the rationalistic conception of the instrumental role of knowledge for decision-making, implying that making available robust evidence for policy-makers will aid producing the most efficient policy outcome (Hertin et al. 2009). A linear process from production of indicators, through use to influence on policy processes and outcomes is an implicit assumptionunderpinning this conception.
While the use of evidence in policy-making has a long track-record, adherence to the principle of evidence-based policy-making as a specific approach took foothold in the second half of the last century.It was specifically voiced by the UK government in 1999 in a white paper on ‘Modernising Government’ (Holmes & Clark, 2008), and recentlypromotedin the ‘Scientific Evidence for Policy-Making’ report from DG Research (EC, 2008). Research into the processes behind the use of evidence in policy-making has an equally long track-record (e.g. Weiss 1999, Sandersson 2004, Owens 2005).
Turnpenny et al. (2009) distinguish between different types of researchon policy appraisal. They conclude that when actors disagree on the aims and the performance in policy implementation, this may lead to a search for improvement of the design of the tools used in appraisals or of the appraisal frameworks themselves, such as how different dimensions of appraisal (e.g. economic vs environmental) is represented, or how stakeholder audiences are selected. This type of research aims at improving the instrumental use of evidence, by improving its quality. Other types of research (ibid) focus on other than instrumental roles of appraisal, studying appraisals in the context of knowledge utilisation and examine the in motivations to appraise such as legitimization, or political control.
Weiss (1999) argues that other than instrumental forms of influence can be identified – “more subtle, needing more time, and functioning as a percolation of new ideas and perspectives into the policy making arenas”.Owens (2005) highlights the strategic perspective to the use-influence chain, pointing to the use of knowledge as ammunition for political argument. She argues, however that due to the need for political actors to debate and convince, evidence may play a role, which is not merely strategic, but where the substance of the argument is integral to decision-making. She further considers that this perspective is one of policy learning, which she finds is fundamentally different from the instrumentalist perspective, as it acknowledges that evidence may stem from different scientific disciplines, and that different forms of knowledge (including contextual and lay knowledge) may be relevant. Sanderson (2004) extends this perspective to also including the political culture as another factor, which preconditions the influence of evidence. Criticising what he sees as an increasingly ‘managerial accountability’ culture, he points to the institutional framework as a critical factor for promoting a dialogue between researchers and policy-makers. Such as dialogue should be structured to foster mutual understanding of the contexts within which each of the actors work, based on responsiveness, mutual trust andactive forms of having a dialogue on research and evaluation findings.
The policy learning perspective is also emphasised by Swartling et al. (2007), in a discussion of theory and methods for studying environmental policy integration. Differentiating between first and second order learning, they argue that first order learning may also be viewed as technical or instrumental learning, where frames and policy problems do not change but where changes are taking place in instruments or levels of measures. In contrast, second order learning involves changes in goals and system descriptions, and thus represents outcomes of conceptual changes.
Turnhout et. al. (2007) takes this argument further to the discussion of the use of evidence in the form of indicators. They argue that ecological indicators have been produced since the 1980s as a response to an increasingly effect-oriented political culture. Discussing the ecological indicators as boundary objects between science and policy they maintain that indicators belong to the applied research category in which knowledge is negotiated (as effect of bounded rationality) and need other criteria for acceptance than ‘science for science’. They use the label of ‘serviceable truths’ (Jasanoff 1990, in Turnhout et al. 2007), and discuss how the degree of structure to the policy problem in question also affects the way knowledge may be used – from direct use of data in a well structured policy problem, to the use as ideas, concepts or arguments in unstructured, badly structured or moderately structured policy problems respectively.
Based on these insights the aim of the POINT project has been to explore the use and influence of indicators in all aspects of policy, with a thematic focus on the role of indicators in fostering and supporting change in areas of policy making towards sustainable development. Specific objectives have been to identify the ways in which indicators influence policy, including non-intended ways and situations of ‘non-use’, and to identify factors that condition the way in which indicators influence policies and processes.
3. Method
The first step in the project was to develop an analytical framework, proposing concepts and analytical categories that could guide the individual studies in POINT.This work was based in the original research proposal, where explanations of indicator influence in policy processes were depicted as belonging to three domains: first the indicator factors, representing well-known criteria for indicators such as validity, robustness and relevance; second the user factors, concerning the characteristics of the persons or groups involved in policy processes, e.g. factors that condition their propensity to demand or use indicators; and third, the policy factors, representing the type of policy and the institutional framework within which indicators are being designed and applied.
During this phase, the framework was reviewed based on literature and turned into categories used as guidelines for the further work (Gudmundsson et al, 2009). An important differentiation made in the project was to follow Henry and Mark (2003) in their separation of use and influence(of evaluations), and the distinction between influence at different levels on interaction. ‘Use’ signifies in this context the collection and processing of information, and application in e.g. assessments or evaluations.Influence at the individual level implies either a change, such as change in attitude, behaviour or even worldviews or the reassuring of existing views or positions. At the inter-personal level it may involve persuasion or justification, and at the collective level outcomes such as policy learning and policy change may take place.
Moreover, it was suggested that a distinction could be made between instrumental, conceptual and political roles played by indicators, and that their potential use and influence could be studiedin different policy phases such as policy formulation phases as well as later phases of implementation and evaluation. In addition, studying influence through different phases of indicator production was suggested, as emerging indicators may define a “window of opportunity” for ideas, frameworks and dialogue (Gudmundsson et al. 2009)
During the empirical phase, explorative studies were undertaken in seven countries and in ten different indicator settings, of which the majority represented national policy settings (agri-environmental and aquatic environmental indicators in Slovakia and Denmark, transport indicators in Sweden and energy sector indicators in the UK, as well assustainable development indicators in Finland, Slovakia and Malta);two represented the use of indicator sets produced at the EU level (ASSESS indicators for transport policy evaluation and EU sustainable development indicators), while one focussed on composite indicators for sustainability, without studying an explicit policy setting.
The studies rest on document analysis, questionnaires and semi-structured interviews, and for the aggregated indicators analysis of stakeholder group interactions in conferences and workshops, as well as media analysis at EU level and in UK. About130 semi-structured interviews were carried out across the project. Related to these studies, 7 stakeholder workshops were carried out with a newly developed participatory methodology called ‘Triple Task’. The primary aim of the workshops was to provide a triangulation for the findings in the individual studies and they also allowed space for other issues to emerge.
During the synthesis phase triangulation between the different studies isattempted, drawing on the common framework. This phase is ongoing.
4. Some findings
The POINT project is now in a phase where possible issues for synthesis across the studies are being explored. In the following, we present a non-exhaustive sample of some of the issues that have been discussed.
The results from POINT have revealed that while use of indicators is sometimes, but not always identifiable, influence is not so frequently found. This findingmay, of course, be due to a lack of influence in the policy situations that were studied, but also that influence may take placein more subtle ways that our studies have not been able to identify using the selected approaches and methods.
Results across the POINT project clearly illustrate how indicators work in a variety of contexts, for a variety of purposes and only sometimes responding to a specific demand. Some indicators are selected and developed following a policy preparation process where later evaluations are foreseen and a monitoring system set up. Other indicators are introduced or published by academic researchers or think tanks and may enter the policy scene through various pathways in which presentations and discussions of new ideas, approaches and frameworks of thought may be first steps towardsa more conceptual influence.
The three categories of factors that were initially selected as categories of ‘explanatory factors’ all proved relevant and could be used to help understand various outcomes in the studies, and most studies identified factors within all three categories that played some role. The indicator factor ‘poor timeliness’ (updated indicator sets are not present when policy decisions are made) is for example mentioned in several studies as part of the reason why some indicators were not very useful or influential. A shared economic belief system is a specific user factor reported to be conducive to indicator influence in the transport study, while the same factor was more of a barrier to indicator use and influence in the energy study. A policy factor like the presence of a strongly institutionalized regime of policy objectives is found to be conducive for use, but not necessarily influence, in several cases. In fact, many different factors contributed to the use or non-use of indicators in these studies, and often it was more a cluster of factors than individual aspects which was decisive for their use or non-use.
Mandatory reporting related to ‘management-by-objectives’ systems illustrates one set of interacting factors. EU policy, for instance, is increasingly met by demands for transparency and accountability, and European Commission initiatives are scrutinised through an impact assessment procedure that includes suggesting procedures for later monitoring and evaluation. Consequently, EU directives frequently include requirements to monitor and evaluate policy performance. Examples of this are the Rural Development Strategy (2007-2013) and the Water Framework Directive to which member states are later expected to report using pre-determined indicators. While the use and possible influence of these indicators at the EU level has not been analysed in POINT, it is evident that a lack of adaptation to national and regional scales andsometimes lack of timeliness for national policy processes seems to make supra-national indicator systems less relevant for national policies. In new member states with less structured policies this may result in a lack of financial and institutional back-up for the indicator production.
The influence on policy processes may however depend on the political agenda, for instance when political opposition sees the issue as an important arena for policy-making, and raise the issue of non-compliance e.g. in Parliament. Such political, strategic use of indicators can also serve to legitimize or reinforce existing positions, as found across several studies. The “ranking” of issues on the policy agenda may be influenced by the extent to which policy performance is linked to financial penalties or rewards or carry EU subsidies for which accountability is important.
It seems that in the case of indicators produced for cross-scale issues such as EU mandated indicators produced at national scale, it may bepolicies that influence indicators rather than vice versa.
On the contrary, when indicators were produced at the EU level for EU purposes, both use and some influence was observed. The ASSESS study was directly commissioned to produce indicators to inform a politically mandated, scheduled policy review effort of the European Commission’s transport policy White Paper ‘Time to Decide’. These indicators were used in the Mid Term Report ‘Keep Europe Moving’, and seem to have had someinfluence on formulation of policy agenda and goals, at least at the European level. Likewise, the EU Sustainable Development Indicators produced by Eurostat have an established role in the evaluation of the EU Sustainable Development Strategy. The study finds that the indicators have in fact affected the framework for sustainable development as well as the revision of the strategy.
The use of indicators in mandatory evaluations at national level may sometimes be expected to result in influence on national policies such as revisions of policy targets or policy measures. However, this was only occasionally observed in our case studies. Some case studies showed how reporting on lacking goal fulfilment can also become a routine that triggers little interest, especially if different indicators show mutually conflicting results.
Evolving policy structuring (Turnhout et al., 2007)encompassing consensus on policy goals as well as agreement on the appropriateness of measures selected, seem to set a favourable context for use of indicators. Linking indicators to clear policy targets increased the likelihood of their utilisation, according to informants. This is in line with other findings that indicator frameworks may enhance indicator use, as they express a common conceptual understanding of the substance and processes at stake.
Institutional conditions may be important in terms of the kind of influence this use maylead to. If, for example, policies are backed up by monitoring, reporting and dissemination systems, paradigms and procedures, this may enhance, although not guarantee, the influence of the indicators in policy processes. Such a situation can be difficult to obtain if policy targets, measures and frameworks are fluctuating. In contrast, a relatively stable policy framework may be favourable to the direct influence of indicators. This may provide the required time for gaps in knowledge to be filled, models behind indicator production to be developed and validated, and policy to be structured along agreed frameworks.
On the contrary, policies that are inherently more unstructured, with less clear goals and without major consensus on cause-effect relationships, form a less conducive context for indicator use. This may sometimes be the case for policies of sustainable development, in which indictors are often developed with the double purpose of reviewing the policy and at the same time providing information to the broader public of the progress towards sustainability. It was found that sustainable development indicators were often very close to statistical data, and that technical indicators factors such as lack of accurate long term data sometimes impeded the development of proxy-indicators,which might not be accepted for policy appraisal, but could have served communication of trends well.