Forming evaluation capacity and culture in the Czech Republic: experience with the first set of ex ante evaluations of programming documents
(with special focus on evaluation of NDP)
Jiří Blažek, Charles University, Prague,
Jan Vozáb, Berman Group, Prague,
Introduction
This paper aims at deriving lessons from the first set of ex ante evaluations of programming documents in the Czech Republic (with special regard to ex ante evaluation of Czech National Development Plan) and outlines the needed steps in forming evaluation capacity in the country. Though the paper focuses mainly at practical and organisational aspects of evaluation it also establishes some broader questions in terms of the programming documents as an expression of public policies and development needs.
1. Context for support policies and programmes in the Czech Republic
One of the major problems faced by both authors of programming documents and teams responsible for their ex ante evaluation was the question how to fit the future programmes of EU cohesion policy within the framework of existing national support programmes. In the sphere of support policies and programmes the Czech Republic suffers from several principal shortcomings. These shortcomings stem from an inadequacy of the public development policies in the Czech Republic. In general it can be stated, that Czech Republic does not have any comprehensive, structured and objective oriented development policy or set of policies and corresponding development programmes. Instead – as argued later – there are many fragmented and partial programmes focused on short-term difficulties. These programmes are mostly supportive instruments with no explicit development goals/objectives. Hence they are reactive forms of redistributions open to various partial needs of certain interest groups (e.g. SMEs, territorial units, larger companies, local governments). The Czech national policies are not based on pro-active or even strategic programmes focused on medium/long-term deliberate changes, neither in their individual cases nor in their whole complex. Consequently the managing and implementing structures for national programmes are created on these assumptions.
Current status of the public support/development policy is closely related to the way how the package of Czech support programmes has been developing since the beginning of the transition.
1.1. Design of policies in the sphere of economic and social cohesion
In the first period of transition, approx. until 1996, the role of regional policy was intentionally marginalized. The reasons for this „low profile“ approach of the Czech government were historical and geographical (small inter-regional disparities inherited from the former Communist regime), economic (until 1996 unusually low rate of unemployment of only about 3 to 4 percent), and political (proclaimed one - sided liberalism, unwillingness to intervene and make any “exceptions” in market rules). The official regional policy was very modest and consisted only in the offer of modest support to small and medium size firms (in the form of soft loans) in assisted regions selected in principle on the basis of the unemployment rate (e.g. more than 5 percent in 1996) and did not require any strategic/programming document.
Nevertheless, the Czech government gradually developed a whole array of different policies with (intended or unintended) significant regional impacts. But these policies do not comprise a comprehensive approach to economic, social and regional development. Almost every ministry developed some programme with important regional impacts. There are numerous examples: the Ministry of the Environment distributes the resources from the State Environment Fund to projects aiming at improvement of environment in most polluted regions; the Ministry of Transport supports public transport in rural areas; the Ministry of Agriculture supports farmers in less favourable conditions or in environmentally protected areas; the Ministry of Labour allocates funds on active employment policy to district job centres according to unemployment rate; the Ministry of Trade and Industry supports by regionally differentiated incentives the foreign investors through its agency Czech Invest. Altogether, in the Czech Republic, there are currently dozens or even hundreds of support programmes, each with different priorities, incentives, financial resources, project selection criteria and different time-schedule of assistance. In addition, the whole „system” of support programmes is due to frequent changes rather unstable which limits its effectiveness and efficiency.
The main weakness of this approach lies in its institutional fragmentation and lack of horizontal co-ordination at the governmental level. This problem had been until 2000 even multiplied by missing regional self-government. Excessive fragmentation of support programs significantly increases the administrative costs of implementation of these programmes, especially of projects elaboration, evaluation, selection and contracting, but also adds up the costs of financial and physical controls. Huge fragmentation of support programmes thus limits also the amount of money allocated to support measures and thus further decreases efficiency of these programmes.
Even when the context for development policies has changed significantly (both in political and economic sense), overcoming of these accumulated problems proved to be quite difficult. Consequently, preparation for the EU policy of economic and social cohesion (ESC) is still hampered by the lack of several key elements: missing stable and legally codified division of competence among different level of public administration (esp. due to recent abolishment of the level of districts and introduction of new layer of “smaller districts”), unfinished reform of administration on a central level, not completed system of financial management and control, missing system of evaluation. Another problem is the lack of qualified people able to prepare and implement projects according to demanding EU regulations as proved to be the case in several pre-accession programmes. Therefore, a key question remains how in this context design a proper strategy in the form of programming document.
1.2. Conceptual problems in drafting of programming documents
Considerable problems concern the poor horizontal and vertical co-ordination of different bodies, subjects, programmes and policies. However, there are also conceptual problems related directly to design of programming documents. These problems are ranging from the very elementary problems like whether to focus the strategy on narrowing the gap between the Czech Republic and the EU or on fighting with growing internal regional and social disparities, through missing strategy in some sectors, the lack of multi-annual budgeting of public budgets, unclear criteria for selection of partners, to lack of experience with evaluation and monitoring (esp. selection of indicators, quantification of targets and missing consideration of alternative strategies).
Traditional weakness of Czech programming documents is insufficient outline of underlining strategy based on a logical chain of intervention. Other problems are safeguarding an internal consistency of all programming documents and an elimination of mutual overlaps. Nearly chronic problem of majority of programming documents is disproportionate stress put on analytical part to the detriment of strategic part, namely, alternative development strategies were considered only exceptionally and development strategies are outlined only very loosely, rather in the form of list of priorities covering nearly every field of possible intervention than explanation of a real strategy.
To this “wide” all encompassing approach contributed fears about insufficient absorption capacity at the beginning of drafting of these documents in the late 1990´s when it was expected that the Czech Republic would receive form Structural Funds and Cohesion Fund 4% of its GDP. Therefore, motivation for this broad approach was not to exclude any potential project by a narrow definition of priorities and measures. As the financial framework has changed (decreased) significantly (to only about 1% of Czech GDP), this tactic proved to be counterproductive, but than it was too late to change the strategy and to make significant cuts in list of priorities.
The key problem, and likely the most important one, underlying the whole programming effort of creation of the NDP and operational programmes (OPs) is missing experience and tradition of using supportive tools/incentives for development purposes. Moreover, many of the tools and measures introduced by EU cohesion policy are either new ones or are proposed in a new context. Given that the NDP and OPs form the very new kind of programmes in the Czech Republic, and given the tools/measures are sometimes proposed with only limited previous experience, the resulting complex set of strategies, priorities, measures and implementing structures is to a certain extent artificial. Moreover, it is only partially linked to the existing Czech national policies. Though many of the elements of new structural funds programmes are aimed at using existing support mechanisms, it is very difficult to assess/estimate their mutual determination, individual results or even common impacts at various levels of implementation.
2. Procedures and organisation of the ex ante evaluation.
Ex ante evaluation was organised as a new exercise in the Czech Republic. There has been no previous experience in evaluating national programmes and policies. Only EU pre-accession instruments (namely various components of Phare programme) have been evaluated so far. Yet, this evaluation was made mostly at project level or focused on annual „programmes“ which scope was very limited. Therefore, the capacity to carry out the ex ante evaluation was rather limited among the Czech institutions (consultants, companies, universities). On the other had, since the programming effort and preparation of documents for Structural Funds has started about five years ago, there has been a number of organisations and individual consultants experienced in planning and programming procedures according to the methodologies and regulations of the EU.
2.1. Selection of ex ante evaluators.
The ex ante evaluation had to cope with two different and to some extent contradictory requirements. First, the need for a previous experience (both with programming as well as evaluation techniques and methodologies) so that the evaluation could be done properly in terms of the EU standards. Second, need to command with a knowledge of the Czech environment, which was necessary to evaluate the proposed activities and programming documents on the background of national context. Resulting decision of future managing authorities, which were responsible for contracting ex ante evaluators, was to hire local consultants in most of the cases (except OP for Tourism, which was at the very end integrated into Joint Regional Operational Programme).
Such a decision was complicated with the issue of conflict of interests. Most of the evaluators had been involved in some of the previous programming efforts. Therefore, the attention was paid by the managing authorities to select such an evaluator which had not participated in preparation of a programming document which was subject to evaluation and which had not have previous relationships to respective authorities. Based on the experience of authors and given the results of the ex ante evaluations, the conclusion may be rather positive. The independence of ex ante evaluations has been ensured to the extent that they have not been bound by any previous obligations to the management authorities. It was also ensured that the evaluators did not participate in the previous preparation of the programming documents they were to evaluate.
International expertise in ex ante evaluations.
Some of the ex ante evaluation teams were also strengthened by involving international experts. They were invited in order to facilitate transfer of know-how with ex ante evaluations to the Czech Republic. Such an expertise has been used in several major areas.
First, foreign experts responded to ad hoc questions of other evaluators, but brought in also the skills and knowledge of non-written but commonly known practices. At least in the case of NDP the stress on well developed strategy supported by solid arguments and carefully linked to other parts of the document was partially result of the international expertise. They also assisted the national evaluation teams by pointing out to the facts (or missing points) which the national evaluators might find less important.
Second, the foreign experts provided methodological guidance in cases when national evaluation teams were in doubts. The general environment in the EU and in accession countries is different which resulted in different approach to ex ante evaluation, but certain common points and methodological approaches had to be followed. It was also helpful to confront methods and techniques used nationally with techniques (and relevant results) used abroad. At least the view of ex ante evaluators was broadened, which may have contributed to a better standard of ex ante evaluation. On the other hand, the teams, which did not use the service of international experts, produced the evaluation of comparable quality. Therefore, the role of international experts should not be overestimated.
Third, but exceptionally, international experts worked on some specific parts of ex ante evaluation by themselves and were directly involved in drafting parts of ex ante evaluation. Rather considerable obstacle was in this case a limited knowledge of national environment. Therefore, this approach was used particularly in case of the NDP because the Plan at national level may be less specific and more formal. As such it is more suitable to be dealt with by foreigners. A specific case of NDP was that the expert involved in the team had a long-term knowledge and experience from the country.
To sum up, international experts were involved only to the limited extent in the ex ante evaluations. Where they were involved they proved to be helpful members of the team, particularly when being supported by national consultants who provided national framework. The international experts focused mostly on methodological issues, on explanation and use of common practices, on explanation of regulatory background and formal requirements.
2.2. Process of ex ante evaluations.
Ex ante evaluations were organised, in line with the EU working paper requirements, as parallel exercise along with drafting programming documents. It was expected, from the very beginning, that the evaluation will be carried out in two phases. First, it will be an ongoing evaluation, which would deal with individual chapters, and second, the final evaluation which would provide for the evaluation of the whole programming document in its complexity and various internal and external relationships. Though the attitude of several ex ante evaluation teams was a bit different, generally all teams followed this model. The main goal of this „two-level“ evaluation was to improve the programming document on an ongoing basis and finally to provide the comprehensive and overall evaluation to be used in the next stages in programming process.
First phase, ongoing evaluation, provided for informal remarks and then formal recommendations on individual chapters of programming documents as they were continuously drafted. It was assumed this approach would allow both to incorporate a standpoint of evaluators as well as to proceed with preparation of programming documents at the same time. However, a serious obstacle appeared when using this approach. Chapters of programming documents were not always drafted continuously and in a logical order. Rather they were often prepared almost in a random order which resulted in weak relationships and low internal coherence among them. This situation made it difficult for the evaluators to issue fully developed recommendations dealing with all the necessary questions as well as for the authors of programming documents to incorporate all the recommendations of evaluators.