Evidence Based Policy for Local Development: Some Considerations for Good Practice

Peter Lloyd (Peter Lloyd Associates) and Michael Blakemore (Ecorys Ltd)[1]

Introduction

Setting the context: the drive for evidence-based policy

Let us start with a deceptively simple opening question: “is it important to have evidence-based policy?” The answer surely is an unequivocal “yes”. Having no evidence on which to base any intervention is clearly unthinkable and some evidence is better than none. But it depends on what kinds of evidence and what threshold conditions of rigourare set. What we do in this Chapter is to unpack the idea of the evidence base and the issues arising from it in the local employment and economic development (LEED) policy domain. Using six case study areas that took part in the OECD-LEED programmeacross the EU, we explore the sort of evidence used, the subject matter, who collects what and for whom and what quality standards are applied. A particular theme of the chapter, which also came out strongly from the case studies, is that acting locally has its own special challenges when it comes to evidence gathering. There are particular issues surrounding what ranks as sound information for policymakers to be able to operate with a degree of confidence. Setting the framework for the discussion that follows, we want to look at the evidence base in the round – what sorts of things need to be known for engaging in effective local intervention and what sorts of information and analysis are most appropriate.

Evidence: problems, choices, actions and outcomes

An obvious point of entry (that can sometimes be forgotten against a modern obsession with gathering evidence to measure policy outcomes) is that acting locally involves at the very least being properly equipped to decide which problems should be addressedlocally in the first place. What for example are the most pressing development issues to be confronted for the local citizenry; which of them are the most critical and which should be tackled first? Answering such questions requires a properly worked out information and issue-selection strategy to support the decision-making process. That in turn demands a sound information base, built on a set of appropriate analytical methods and importantly some sense of causes and effects. Since intervention means some form of remedial action, there are questions not just about the specification of the problems and their causes but also a “justification in evidence”that what is proposed will actually work and whether or not it does so. We are dealing here with the credibility of the evidence-base.

Historically the evidence used to assess local development has been ‘official’; that is largely drawn from official statistics. The data carries with it the authority and credibility of being provided from the ‘centre’ – usually the national Statistics Office or Observatory. One undoubted advantage of this is that it opens the door to the creation of a common set of indicators comparable across geographical spaces. But a significant disadvantage, as we shall go on to show, is that central models of evidence are poorly tuned to the contingency of local conditions. As a direct contrast, local-level evidence can be well-attuned to local circumstances but with only the most limited opportunities for commonality and for making comparisonsacross geographical spaces. Here we confront a fundamental conundrum – how to ensure both cross-comparability and local “granularity” in the evidence-base for local development.

It is clear also that a discussion of evidence-based policy takes us directly into questions of theory – for instance what causal processes does the evidence suggest are behind the identified problems;and what actions are hypothesised as acting on them to produce some defined outcome? There is an obvious link here between politics and methodology. No matter how rigorous the ex-ante analysis;the interpretation of the primary problems, what causes them and how they should be fixed is not a given but a choice among alternative positions. There are therefore critical questions to be explored for any evidence base about who does the interpretation, makes the choices and in whose interestsall this is done(Haughton, Peck et al. 1995). Of course once the chosen intervention is running we canmove on to a less complicated terrain. In progress and ex-post; policy can be monitored and evaluated to see if it was efficient and effective and met the objectives set for it. But even here - especially when we open the box of impact assessment - we are drawn back into what may be potentially contested politicaland methodological questions about the causes and effects of the intervention; how well it worked; what were the expected outcomes and the unexpected ones; and what were the implications for the underlying theory? Impact inevitably takes us back to perspectives on how the local economy works and how the intervention is connected to it. So in the end, grand exhortations to adopt an evidence-based approach to local economic development inevitably open a Pandora’s Box of complications. The issue for us here is that “acting locally” makes things more and not less challenging. The next section outlines some of those challenges as they relate to LEED, before the chapter moves onto highlight findings from our EU case studies around what sorts of indicators and evidence-gathering methodologies are appropriate for supporting LEED policy. Finally, the chapter looks at some emerging underlying principles of developing an information strategy for local economic and employment development.

Local Employment and Economic Development (LEED): An Overview of the Data Challenges

A complex integrated policy terrain

In terms of evidence-based policy, one thing we can confirm from empirical observation is that practitioners in the LEED field are very busy gathering information about what they do and how well it works. For over two decades data has been widely assembled across all the classic elements of the policy cycle – problem definition, policy formulation, policy intervention; evaluation and the assessment of value-added and impact. Since the local policy domain has begun to have a wider currency and the whole technological paradigm is shifting,now is a good time to look back at where we stand in relation to evidence-based policyaround LEED - since it may have profound implications for the overall direction of travel.

From the LEED perspective we focus attention on three distinct (but naturally co-related) areas of policy activity. All have different forms of data, information and evidence associated with them and all three come together to create the reality of economic development and employment policy as it operates in local places. These are:

a)Helping people into jobs and improving skills: supply-side actions where the subjects for the policy are the people currently both working and unemployed or in some form of education or training (the latter two being potentially available for work) – but all of whom can potentially supply more of labour/productivity than they currently do;

b)Economic development and the creation of more and better jobs: demand-side actions where the subjects for the policy are businesses or public employers whose activities produce (that is open up more of) the “job slots” that wage earners can supply their labour to fill;

c)Helping disadvantaged groups (those experiencing difficulty in gaining paid work) to access the labour market: social and economic inclusion where the subjects for the policy are people and groups at some distance from the labour market and where a diverse portfolio of measures is deemed to be needed to address their disadvantage. This means working from both the demand-side and the supply side – both by creating new jobs sources that are appropriate (sheltered workshops, for example) and helping to remove the barriers that impede access to jobs generally.

Issues of scale and the local level of resolution

There is clearly no one-size-fits-all local geographical framework that would neatly package up all three policy domains across a single unit of space. Going up-scale geographically to the regional or national levelwould, of course, give more chance of “policy closure” (i.e. captured in the same spatial unit) while going down-scale to the neighbourhood or rural village would clearly give less[2]. Across the three defined LEED policy components above, some are also clearly more susceptible to small area local intervention (inclusion and disadvantage perhaps) while others require actions at a wider spatial scale (say education and training and economic development). For some, the causal processes that drive them are more manipulable locally, for others the local is more dominantly the end-point of causal chains driven from elsewhere.

In policy terms interventions deemed "local" are probably best defined by what appears to work bestin contextfor the task in hand and this is what happens in practice. For example, the scale efficient "most local" level of action for, say, a major national labour activation or skills development policy may well be sub-regional, or even regional with local delivery points on the ground. By contrast, a policy for getting to the most disadvantaged groups into work will probably need to be local in a very real sense – “going where people are” to capture the variety of social and cultural conditions that may be holding them back(Haughton 1999). From an evidence-based perspective, therefore, spatial scale or level of resolution is demonstrably a critical variable and the data that underpin the key tasks of problem definition, policy formulation and policy interventionare highly sensitive to issues of scale and methodology.

Hard and soft spaces

The fact is that the local plays differently depending on the purpose of the policy action. It also matters who and what authorities, agencies or groups are the promoters of that action(Peck 2011; Prince 2012). Where the promoters’span of jurisdiction is fixed in place, say as municipalities or sub-regional councils, for example, there is a constant issue of how well the levers that they have to hand and the spaces they occupy map onto the causal processes they seek to influence to achieve their objectives. What we see in practice tends to be a far from perfect fit and for local policy tools to be confronted with a span of control that can only partially have purchase on the causal forces involved. In response, the “hard spaces” of government –fixed administrative boundaries linked to democratic electoral processes tend to be restrictive. Over time they have tended to give way to a variety of“soft” spaces that can be invented and re-engineered more flexibly for policy purposes(Allmendinger and Haughton 2011). These softer spaces are generally overlapping and tend to operate against loosely-defined fuzzy boundaries that work better with the fluid processes in hand(Haughton, Allmendinger et al. 2010). Pure pragmatism suggests that the spatial frame for acting locally needs to be contextually defined and fluid. This is critically important from our perspective hereon evidence-based policy since the “hard” spaces tend to be those against which standard data series are collected while flexibility in production and the labour market increasingly favours “soft” spaces. An immediate problem for local information strategies is a root issue of spatial mismatch unless data can be sourced under some system of flexible geographical referencing.

Standard practice; statistical series, indicators and well-tried methods

But in observed practice questions like these – while probably understood - tend to be rarely addressed. To carry out the day-to-day function it is more often a question of “what information is available and how can we bend it to what we need?” The approach across the board is to draw on data from the standard series (usually provided by national Statistical Offices) to capture a local (“hard” spaces) perspective across the key domains of LEED policy. Much of the economic, employment and inclusion information that local policymakers work with tends to be easily recognisable and handled by well-tried methods. Information from higher-order datasets is used to generate local estimates based on proportional shares (national to local/ regional to local) by the application of equations of varying sophistication. Once downloaded the information is processed for analysis – through the Excel suite or perhaps some form of sectoral model - to turn it into evidence for use in policy documentation and strategy development.

One of the core elements is usually a set of indicators that characterises the state of the local economy presented in terms of comparison with national or regional conditions for the set of indicator variables. Cross-comparable indicator sets also enable comparisons to be made with other cities, regions or localities. Time series indicators are normally a key part of the information output – looking at local trends and inviting comparisons. There is, however, a special issue with this standard portfolio of orthodox practice when it comes to applying it at local level. Questions of error - probably well understood by the data handlers at source - tend to be lost in the process of converting indicator and trend information into policy papers and strategy-making. While Standard Errors are normallycarefully controlledfor the national or regional source data sets; as the level of spatial disaggregation falls thequestion of error and its distribution tends to capture much less attention. This is far from trivial since error goes up sharply as spatial scale comes down with disaggregated top-down data[3].

Sectoral lenses and flexible production systems

In the creation of cross-comparable indicator sets involving local areas a number of other key things also tend to get glossed over. Most of the standard indicator sets tend also to depend heavily on data aggregated by sector, and where the data are classified according to a particular industrial classification (itself the result of methodological compromises) at an establishment level (the internal heterogeneity of activity within the establishment is subsumed to one or a limited number of classifications). Unfortunately, however, this too abstracts from what we know about modern production systems – not least that the idea of a sector as a collection of establishments producing homogeneousoutputs is hardly tenable(Storper and Scott 1986). The functional disaggregation around complex value chains that is normal for modern production systems makes reading the local by sector potentially open to serious misinterpretation. When the same sectorial category can contain very wide variety (the norm); what goes on locally depends on which bits of the functional-occupational, spatially-distributed elements are actually in place on the ground. Flexible production systems and their flexible labour markets are not captured effectively by the sector-based indicators that tend to be so readily accepted as “evidence” in local development. Once again, it is operating locally that exaggerates the issue – regional and national data series will have some degree of under-representation of this kind but scale and scope will render it less important.

Complex data landscapes and a morass of information challenges

In the end, then, local policymakers (local authorities and other responsible agencies) normally have to deal not only with the contingent complexity of economic and social life in their areas, they also find themselves having to cope with highly complex data landscapes where authoritative information from central sources may not map onto the spaces they need and where data error is hard to control for. So,accepting the need for more evidence-based and rigorous approaches to local policy-making runs them headlong, as we said earlier, into a morass of information challenges. Many of these challenges are in practice very difficult to surmount - but this is not a good reason for not making ourselves aware of them up-front. At the very least we should all be more sceptical about information and evidence when it comes to assessing local action and estimating its broader economic or social effects. The simplest approach is to provide the user with some form of “health warning” that flags up the issue of error and alerts policymakers to what they can and cannot accurately claim from the data series they employ. Naturally,this runs counter to the tendency to put the best light we can on a policy approach that all the anecdotal evidence seems to suggest works well and is effective. But it represents a challenge to do better with issues of error and bias even if it means stepping outside the bounds of received practice.

The position of monitoring and evaluation

One place where local data gathering activity has grown substantially and where things are much easier is where the locally promoted actions have already been selected and resourced as being the ones that will make a difference and actions have been put into operation. This is the task of gathering evidence of effectiveness and efficiency locally through programme or project evaluation(Stern 1987). This time there are no complex relational issues of spaces or causes to consider – that is until questions of impact assessment come into play. Problem definition, policy formulation and policy intervention have all been sorted out in advance. Once launched, the project or programme is a self-defined entity and the evaluation process to justify spend and measure its local added value can then proceed. Data is collected internally to the project with due rigour and usually according to standardised evaluation methods. It is only when the issues of complex spaces and multi-causalities reappear againin estimating the overall economic or social impact of such a project or programme that we re-engage once more with the complexities of multi-causalities, proportional effects and appropriate spaces.