Fifth European Conference on Evaluation of the Structural Funds Budapest, 26/27 June 2003
“Challenges for Evaluation in an Enlarged Europe”
Fifth European Conference on Evaluation of the Structural Funds
Budapest, 26/27 June 2003
“Challenges for Evaluation in an Enlarged Europe”
Building capacity for evaluation: Lessons from Italy
Paola Casavola and Laura Tagle [(*)]
ABSTRACT
In recent years, Italy has dramatically increased its evaluation capacity through conscious efforts. The strategy followed has already produced significant results which are even more relevant considering that the starting point was low. In the paper, we present the strategy implemented and the results obtained, discuss what we have learned from the experience so far - from both achievements and disappointments - and draw lessons that might be useful for accession countries.
Italy has a civil law system and a continental administrative structure, both not conducive to an evaluation culture. In the field of evaluation of development programs, the only source of evaluation requirements and guidance have long been EU regulations and European Commission’s guidelines, while national authorities mostly disregarded the issue. Such neglect resulted in a restricted market and fostered compliance-oriented practices on the part of both evaluators and administrations.
The evaluation unit of the Ministry for the Economy (UVAL) has in the last few years been in charge of building capacity for the evaluation of regional development interventions. In its strategy, it utilizes Structural Funds’ evaluation requirements to introduce evaluation in regional and central administrations’ practice. The strategy aims at improving both a) the incentive structure and b) the capacity to ask for, utilize, and conduct evaluations.
a) In order to alter incentives for evaluations, UVAL provides assistance for administrative authorities to formulate their own evaluation questions and ask for appropriate feedback from evaluators. Among the causes of poor evaluation capacity, lack of incentives from potential users is rampant. In absence of a clear perception of utility, managing authorities resist evaluation or, at best, formally comply with regulations requests. Social and institutional partners fail to put pressures on managing authorities to obtain reliable evaluation results. Hence, UVAL focuses its strategy on utility: evaluations must be useful for managing authorities, social partners, and other stakeholders. The chosen tactic is based on creating an institutional environment for evaluation that encourages the expression of evaluation needs of regional and central governments alongside those of the European Commission and involves social and institutional partners in utilizing evaluation results.
b) For a long time administrations have viewed evaluation as too difficult a task to tackle. This perception has disfavored evaluations. Italy’s strategy builds on the hypothesis that reinforcing technical skills internal to administrations improves comprehension and acceptance of evaluation role. Hence, UVAL is also striving to enhance administrations’ capacity to conduct evaluations. It is leading a network of evaluation units at regional and central level, promoting the launch of internal evaluations, and coordinating self-assessment exercises.
This multi-pillar strategy has now been implemented for four years. It has benefited from the lessons learnt from positive results (such as the increase in resources devoted to evaluations or evidences of utilization of preliminary evaluation results by partners) as well as from the persistence of obstacles and constraints, such as a limited market.
BUILDING CAPACITY FOR EVALUATION: LESSONS FROM ITALY
Summary
1. Introduction 2
2. The construction of the strategy 3
3. The starting point 3
4. The initial strategy 4
Performance reserve and mid-term evaluations 5
Carving a space for evaluation questions to emerge 5
Meta-evaluation activities 6
Opening the market 7
Construction of evaluation capacity within the administration: setting up a network of evaluation units 8
Construction of evaluation capacity within the administration: involve the MA’s on the issue of Indicators 9
Enhancing Supply 9
5. Achievements and disappointments 9
Achievements from the performance reserve 10
The incentives provided by the performance reserve proved effective. MA’s dedicated time and resources to launch the bids along with the required timeline. By early 2002 almost all operational programs included in the OB.1 CSF had hired their evaluators. 10
Difficulties in emergence of evaluation questions and effects of national guidelines 10
The taking up of a more interacting model in evaluation processes 11
Role of evaluation units 11
Results and reflections from the”Indicators” project 12
Few progresses for the market 12
6. Features of a revised evaluation capacity building strategy 14
7. Conclusions 15
References 16
1. Introduction
European programs, such as structural funds or Phare, offer a tremendous opportunity for evaluation capacity development in both member states and candidate countries, because they impose evaluation requirements. In a number of cases, such requirements have been the road through which evaluation has entered the administrative practices of both member states and candidate countries.[1] This sequence contains both opportunities and risks. It provides the opportunity to overcome resistances and to support internal tendencies to reform. At the same time, risks are strong that evaluation is perceived as mere compliance and that it fails to respond to the needs of national actors.
The enlargement is an opportunity to re-think this sequence, and more generally, to propose a more proactive strategy for evaluation capacity building. The Italian experience with evaluation capacity building has a number of lessons to offer. Italy started building its capacity to evaluate development programs from a particularly low point. European requirements played a substantial role in shaping the way evaluations are conducted. There have been many phases in evaluation capacity building, and there are a number of models within Italy: evaluation has long been istitutionalized in social and health programs, there are evaluations within the university system, and regularization policies also are evaluated. This variety of approaches and of institutional actors allows for adaptation of the institutional framework to each task.
This paper presents the evaluation capacity building strategy pursued by the Department for Development and Social Cohesion Policies (DPS). This experience is particularly interesting for accession countries because here the influence of the European Union has been the strongest, where the risks of a normative approach to evaluation have shown most clearly, and where neither the public or the administration initially perceived the value of evaluation. The strategy has been developed gradually, overtime, and building on the knowledge we accumulated by working and by taking into account achievements and disappointments. There is still much to do.
What our experience shows is that inputs from the EU (in terms of requirements, guidelines, or methods) cannot be taken as a blueprint. There is no single model, and prescriptive approaches damage, rather than sustain, the creation of evaluation capacity. It is necessary to carefully build a set of instruments that satisfy the information needs of planners, implementers, policy makers, social partners, and the public. In order to do this, it is necessary to partner with the European Commission (EC), with civil society (e.g., evaluation associations), and with counterparts in other countries or in the international environment.
After a brief story of how the strategy has been developed, the paper contains an assessment of the initial situation. The fourth paragraph portrays the main features of the strategy. Results and disappointments are accounted for in the fifth paragraph, while the sixth one describes how the strategy is changing in response. A brief paragraph of conclusions closes the paper.
2. The construction of the strategy
The strategy for capacity building now pursued by the Department for Development and Social Cohesion Policies (DPS), and in particular by its Evaluation Unit does not arise from a conscious, systematic decision to alter the way evaluation is conducted in Italy. Although we can now describe it as a coherent strategy, it developed over time, starting from internal debates and decisions taken in 1998 and 1999, at the time DPS was developing the strategy for the Community Support Framework for Objective 1 (CSF Ob1.).
This way of developing the strategy has actually helped finding innovative solutions and dealing more effectively with other actors, such as the European Commission and Managing Authorities (MA’s). Had they been confronted from the start with an entirely new approach, they would most likely have resisted it.
Another advantage is that this way of proceeding—a step at a time, altering one instrument at the time—allowed for learning and for testing. Although we now possess a much clearer view of where we want to be in ten years’ time and how we intend to get there, we continue to adopt a cautious pace and to be open to changes in our own strategy and instruments.
The first, important decisions were to give importance to evaluation tasks. Such decisions were in line with the general philosophy of the CSF Ob. 1: contrary to the prevailing attitude of the Italian administration, now requirements, decisions, and deadlines were to be taken seriously. Formal compliance and substantial indifference would not be tolerated anymore.
Evaluation was also deemed a condition for the program to be credible, even though at the time evaluation results from the previous programming period were still not available. What was clear from the past period—and it was strongly signalled by the European Commission (EC)—was that in previous programming periods evaluation requirements had been substantially avoided: managing authorities (MA’s) had delayed or even omitted to hire evaluators. When they had hired them, there had been very few interactions, and most evaluations appeared to be of little of no use.
Despite this unsuccessful history, we chose to rely on evaluation requirements mandated by the EU to strengthen evaluation capacity. This choice appeared capable of legitimising evaluation and of providing a tradition and a body of regulations, guidelines, and methodological indications on which to build national and regional capacity. The paper will explore how this strategy evolved over time as a consequence of experience.
3. The starting point
In mid-1998, when DPS started being operational, the situation of structural fund evaluation in Italy was not favourable. Evaluation requirements were ignored or complied with in a purely formal way. Evaluators and evaluation commissioners (usually MA’s) had few—if any—interactions during the evaluation process. The EC considered available evaluations to be of low quality. Neither the EC nor any other partner, such as MA’s, the member state, or social partners, used evaluation results in negotiations or in decision-making.
The evaluation framework emanated from the EC and there was no reinterpretation by the member state or MA’s. Evaluators faced no evaluation question—just the request to comply with EC requirements. Evaluation commissioners, in other words, paid evaluators to formally answer evaluation questions they did not share, need, or believe in and that came from just one of the relevant stakeholders: the EC. Paradoxically, such questions produced evaluations that, by and large, disappointed the EC.
The evaluation framework has been defined “unfriendly” to MA’s. The evaluation question the EC expresses caters to its needs—and rightly so, since the EC is accountable for the use of EU money to the European Parliament. The way such needs have stratified in evaluation of structural funds, however, respond to administrative accountability—the receiving administration is (or should be) accountable to the funding administration. As a consequence, evaluations are required to cover all aspects of each program. This might be acceptable, were not for the fact that some programs are huge and span a number of sectors. Many measures – the program operational building blocks - might be evaluated as a program in themselves.
Evaluators faced such requests by avoiding to focus on parts of the program and by performing superficial data collection, synthesised in “indicator sets” developed by (other) consultants and rarely used by the MA’s or any other actor. By doing this, they missed most non anticipated effects, failed to provide explanations based on causal links between the program and its effects, and fell short of building good theories. Little wonder nobody used evaluations.
The market for evaluation of structural fund interventions was (and still is) restricted. Few operators got most contracts. Contracts were so large that only large firms could participate in bids. Requirements to be admitted to bid in call for tenders also unwillingly favoured large firms—financial capacity was measured in previous turn-overs of three times the amount of the contract. The way technical capability was to be demonstrated further raised barriers to entry: bidders were required to have previously worked on structural fund evaluation.
Not only was (and is) the market restricted. It was also segmented: other evaluators operated in the social programmes evaluation market. These are generally smaller firms, operating on a local scale. It appears difficult to attract such operators in the structural fund market. They claimed (and still do) that rules governing structural funds evaluation are not easy to understand.
Albeit restricted and segmented, the market for evaluation of structural fund interventions was (and still is) not specialised. Firms found (and still find) impossible to survive “only on evaluation” and operated (and still operate) at the same time both on the evaluation and on the technical assistance market. This lack of specialisation disastrously affects the independence of evaluators, who tend to share the MA’s assumptions and to offer consulting services instead of information and judgements.Being this the state of the art, it is somewhat surprising that, for Objective 1 programs in Italy, the EC prompted to hire evaluators from the market in order to support the required independence.
Another element should be added to this initial context. For a long time, Universities have not provided formal training for evaluation. A large cohort of trained experts was, thus, lacking. Most researchers involved in actual evaluations were acquiring their skills on the job. In the best of cases, they used in evaluation tasks what they had learned in applied research in social sciences. In the worst cases, they tried to replicate general guidance offered by EC documents, but missing the necessary experience to adapt those advices to the specific cases they were working on.
4. The initial strategy
Initially, the Italian strategy for strengthening evaluation was to rely on structural fund requirements, to enforce them, to comply with all of them, and to add a few elements in the attempt to change some of the unsatisfactory features outlined above. DPS started by separating the performance reserve monitoring from mid-term evaluations. By developing national guidelines and by providing support to MA’s engaged in ex ante and then mid-term evaluations, DPS created a space for MA’s to express their own evaluation demand—to request information and judgement on issues that were of particular relevance to them. It also undertook activities to create evaluation capacity within administrations, by setting up evaluation units and by launching an initiative of meta-evaluation. Other actions aimed at improving the market situation and the availability of evaluation skills.