THE EVALUATION OF
SOCIO-ECONOMIC DEVELOPMENT
The GUIDE
December 2003
Tavistock Institute
in association with:
GHK
IRS
3
Draft Materials (Second Revision) 30 September 2003
CONTENTS
INTRODUCTION 1
PART 1 THE CONTRIBUTION OF EVALUATION TO SOCIO-ECONOMIC DEVELOPMENT 5
1.1 THE BENEFITS OF EVALUATION 5
1.2 INTRODUCING EVALUATION: HISTORY AND PURPOSE 10
1.3 METHODS AND THEIR ROOTS 17
1.4 EVALUATION TO STRENGTHEN SOCIO-ECONOMIC DEVELOPMENT 24
1.5 GOLDEN RULES 35
PART 2 DESIGNING AND IMPLEMENTING EVALUATION FOR SOCIO-ECONOMIC DEVELOPMENT 38
2.1 DESIGNING AND PLANNING YOUR EVALUATION 38
2.2 IMPLEMENTING AND MANAGING EVALUATIONS 56
2.4 GOLDEN RULES 81
PART 3 DEVELOPING CAPACITY FOR SOCIO-ECONOMIC EVALUATIONS 84
3.1 DEVELOPING INSTITUTIONAL CAPACITY 84
3.4 GOLDEN RULES 100
PART 4 CHOOSING METHODS, TECHNIQUES AND INDICATORS AND USING EVIDENCE IN EVALUATION 103
4.1 FACTORS INFLUENCING THE CHOICE OF METHOD, TECHNIQUES, DATA AND EVIDENCE 103
4.2 METHODS AND TECHNIQUES FOR EVALUATING DIFFERENT TYPES OF SOCIO-ECONOMIC INTERVENTIONS 109
4.3 METHODS AND TECHNIQUES FOR DIFFERENT EVALUATION PURPOSES 115
4.4 METHODS AND TECHNIQUES APPLICABLE AT DIFFERENT PROGRAMMES/POLICY STAGES 117
4.5 METHODS AND TECHNIQUES APPLICABLE TO DIFFERENT STAGES IN THE EVALUATION PROCESS 120
4.6 ACQUIRING AND USING DATA IN EVALUATION 123
4.7 CREATING INDICATORS AND INDICATOR SYSTEMS 127
4.8 USING INDICATORS TO IMPROVE MANAGEMENT 140
4.9 GOLDEN RULES 146
Annexes:
Annex A The Main Stages of Evaluation
Annex B Changes in Structural Fund regulations
The GUIDE December 2003
Evaluating Socio Economic Development, The GUIDE: Introduction
INTRODUCTION
This GUIDE, is intended for those involved in the evaluation of socio-economic development in Europe. It is a successor to the MEANS collection (methods for evaluating structural policies). Whilst the GUIDE has a specific focus on evaluation within European Structural Funds, it is not confined to the evaluation of these interventions. Socio economic development is, after all, strongly featured in many national and regional programmes that are not funded by the EU. In most countries in Europe, as elsewhere in the world, improving regional and local economies that have fallen behind, re-integrating marginalized groups, and adjusting to the challenges of global competition and technical change are priorities. Given the scarcity of resources and the often innovative strategies that socio-economic development requires, the demand for evaluation has expanded alongside the development of policies and complex interventions themselves.
Evaluating socio-economic development; policies, programmes, themes and projects
The GUIDE is concerned with the evaluation of socio economic development in Europe which gives it a particular focus on European Structural Funds. The funds are organised in programmes and evaluation takes place at ex ante, mid term and ex post stages. The programmes are one means of achieving wider policy goals and programme evaluation contributes to policy evaluation. The programmes comprise many interventions and projects. Evaluation at the level of the measure/intervention/project forms a part of programme evaluation. Many different programmes and their elements contribute to thematic objectives and thematic evaluation builds upon project and programme evaluation. The principles stressed in this GUIDE generally apply in socio economic programme, policy, project and thematic evaluation. Thus the GUIDE will be of use to those who have to plan, commission and manage thematic, policy and project evaluations as well as programme evaluation.
Who is this GUIDE for?
The readers of this GUIDE will come from many of the different communities active in the evaluation of socio-economic development programmes. These will include:
§ Policy makers who have an interest in what evaluation can do for them including what are the strengths and limitations of evaluation and the resources and capacities they will need,
§ Public sector managers and civil servants who may commission evaluations and would like an overview of what it available including the choices of approach and methods that they should be drawing on,
§ Programme managers who will wish to incorporate evaluation results into the way they manage and plan their programmes,
§ Programme partners who are increasingly involved as stakeholders in evaluations, consulted about evaluation agendas and expected to use evaluation findings,
§ Evaluators, many of whom will have detailed knowledge of specific areas of evaluation but will benefit from an overview of a wider range of methods and approaches to support collaborative work with other members of an evaluation team.
Although the GUIDE itself is intended for general users and readers, rather than specialists, we have also taken account of more specialist needs by preparing a number of sourcebooks to back up the content of the GUIDE. This sourcebook material is available via the Internet at: http://www.evalsed.info.
Why another evaluation GUIDE?
These days we are not short of evaluation guides, textbooks and source material! As the profession and practice of evaluation has grown, a considerable library of evaluation books has been published. Whilst this literature mainly originated from North America, the expansion of evaluation in Europe - often in response to Structural Fund requirements - has spurred many new publications in Europe. The European Commission has published detailed Methodological Guidance - on indicators, ex-ante evaluation, macro-economic evaluation etc - that is specific and closely aligned with the Structural Fund Regulations. There is also a Financial Regulation that requires ex ante and ex post evaluation of all EU funded programmes that has to be adhered to[1]. Public authorities at member state level also publish guidance for those who evaluate national and European socio-economic development programmes and policies.
The obligations to evaluate and the Guidance published by those who share responsibility for the socio economic development programmes are bound to change. Evaluation needs to be closely aligned to the circumstances in which the socio economic development is taking place and the key policy choices that need to be informed. We need to be clear that this GUIDE is not a substitute for other sources and indeed it draws on and cross-refers where relevant to such sources. This GUIDE is intended to speak to a wider audience - and to present evaluation approaches and practice in these kinds of programme and policy areas 'in the round'. Very often other sources are very specialised, addressing narrow areas of evaluation at an expert level. This GUIDE intends to fill a gap in the market to broaden understandings of sound methods and good practice in an accessible form.
Updating MEANS
Of course the main source of such generic guidance up to now has been the MEANS collection - a valuable and comprehensive set of handbooks published by the European Commission in 1999. The MEANS collection has become a standard text for European evaluators and has enjoyed a justifiable reputation for its scope, ambition and coverage. Indeed many aspects of that collection have stood the test of time and have been incorporated into this new GUIDE. However, times have also moved on since 1999. In particular:
§ There have been major changes in the world of evaluation practice, with the emergence of new evaluation tools, a heightened role for theory, new participatory and qualitative approaches (especially relevant to socio economic development) and an emphasis on the institutionalisation of evaluation).
§ European policy has moved on, especially following the Lisbon Agenda. The role of human and social capital, the importance of information society and the knowledge economy as a means of achieving greater competitiveness and priorities of sustainable development and equal opportunities have all been brought to the fore.
§ The accession of ten new member states in 2004 also poses challenges for evaluation. Structural Funds are being introduced into public administrations with a relatively short experience of evaluation and consequently without a well developed evaluation culture. The authors of this GUIDE have had in mind throughout its preparation, the capacity building needs of many new member states. In practical terms we are concerned to maximize what can be achieved pragmatically with available resources, skills, institutional arrangements and data sources.
Together these changes are substantial and this GUIDE has been written to take account of their consequences. It has also been planned and written to anticipate future change. There is no reason to believe that the future pace of change of evaluation and of socio economic policy will slow down. For this reason and to enable the ready updating of material the GUIDE, Sourcebook material and Glossary are accessible and searchable via the Internet. In future it is intended that the material will be further developed so that users will be able to structure and configure their own 'personal' GUIDE that matches their needs, the evaluation role that they have and the particular programmes and policies that they are evaluating or managing.
Content and structure
The new GUIDE is published as single volume. It is supported by a series of Sourcebooks that provide more specialised and in depth material and which can be accessed and downloaded via the internet.
The GUIDE itself is in four parts.
Part 1 provides an introduction to evaluation and its benefits. This begins with a general overview of what evaluation can do to improve policies and programmes and ultimately to strengthen socio-economic development. This is followed by an introduction to some basic ideas in evaluation: its history; some of the different traditions which evaluators draw on; and, the different purposes of evaluation. Finally, the specifics of socio-economic development as an object of evaluation are discussed. This includes unpicking the specific characteristics of this socio economic development policy and its implications for evaluation as well as the main theories and ideas on which policies and programmes are built and which evaluators need to take into account.
Part 2 takes readers through practical issues in designing and implementing evaluations. It begins by considering design issues including how to plan an evaluation, defining evaluation questions and choosing methods, as well as launching and commissioning evaluation work. It then goes on to consider the management issues once an evaluation has been designed including the choice of evaluators and the role of Steering Committees, managing communications to ensure influence and managing quality assurance in evaluation.
Part 3 discusses how to develop evaluation capacity and strategies for capacity development are discussed. The argument is structures around an ‘idealised’ model that suggests four stages in capacity development. This part includes discussion of internal capacity within administrations, as well as external capacity within professional networks and partnerships.
Part 4 introduces the methods and techniques of evaluation, in terms of their strengths and weaknesses – and appropriateness. Methods and techniques are discussed within a number of frameworks: of different types of socio-economic programmes, different programme stages, different stages in the evaluation process, and different evaluation purposes. Finally, types of data (quantitative and qualitative), indicator systems and data sources are introduced.
Each section of the GUIDE ends with some ‘golden rules’ highlighting both good practice and rules of thumb that can be recommended to those who manage, commission, undertake and use evaluations. However, in general this GUIDE avoids being too prescriptive. This is partly because there is often no single right way in evaluation and different approaches each have their strengths and weaknesses in different settings. Pragmatically also, the ideal preconditions for evaluation often do not exist – whether because of lack of data, problems of timing or availability of skills. Doing the best we can whilst still trying to improve evaluation capacity in the future is a theme that runs through this GUIDE.
To support the GUIDE a series of Sourcebooks has also been prepared, which will be of particular interest to specialists and practitioners.
Sourcebook 1 is entitled ‘Evaluation approaches for particular themes and policy areas’. It considers important priorities such as sustainable development, the information society, social inclusion and equal opportunities and the range of policy areas within which interventions to promote socio economic development take place. The types of evaluation methods, data and indicators that are appropriate to these themes and policy areas are elaborated and examples of their application provided.
Sourcebook 2 is entitled ‘Evaluation methods and techniques’. This includes the elaboration of a wide range of tools and techniques both quantitative and qualitative that are useful at different stages of an evaluation.
Sourcebook 3 is entitled ‘Resource material on evaluation capacity building’. This includes case histories of the development of evaluation capacity in the EU, Italy, Netherlands and Ireland; and, references to other regional, national and international experience – including the accession countries. It illustrates the advice provided in the GUIDE and is intended to stimulate the development of evaluation capacity.
Finally, there is a Glossary that contains definitions of the terms used in the GUIDE and Sourcebooks.
4
The GUIDE December 2003
Evaluating Socio Economic Development, The GUIDE: Part One
PART 1 THE CONTRIBUTION OF EVALUATION TO SOCIO-ECONOMIC DEVELOPMENT
This first part of the GUIDE begins with a reminder of why evaluation is so important in socio-economic development programmes. It introduces some basic evaluation ideas, their origins and purposes. It then highlights some of the characteristics of socio economic policies and interventions and the implications of these for how they can be evaluated.