16th IFOAM Organic World Congress, Modena, Italy, June 16-20, 2008
Archived at
Development of criteria und procedures for the evaluation of the European Action Plan of Organic Food and Farming
Schmid, O.[1] , Lampkin, N.[2], Jeffreys, I. 2, Dabbert, S.[3], Eichert, C. 3, Michelsen, J.[4]
Key words: action plan, agricultural policy, organic agriculture,programme evaluation
Abstract
Within the EU funded project ORGAP, a toolbox for the evaluation of the European as well as nationalaction plans for organic food and farming has been developed (). This toolbox was based on a comparative analysis of national action plans in eight countries (CH, UK, DE, IT, DK, SI, CZ, NL, ES), a meta-evaluation of existing evaluations of national organic action plans, workshops with national stakeholders, discussions with evaluation experts and a European Advisory Committee. Furthermore conflicts and synergies between national and the European Action Plan were identified.
Introduction
The European Commission released in June 2004 the European Action Plan for Organic Food and Farming (EUOAP). In May 2005 the 3-year,EU funded research project ORGAP (“Evaluation of the European Action Plan for Organic Food and Farming”) started. 10 partners from 9 countries (CH, UK, DE, IT, DK, SI, CZ, NL, ES) and the umbrella organisation of the organic agricultural movements in Europe (IFOAM EU Regional group) participatedin the project,ensuring a broad stakeholder consultation process and dissemination all over Europe.
Materials and methods
The overall objective of this project was to give scientific support to the implementation and evaluation of the EUOAP. This wasachieved by the identification of a set of suitable indicators and concepts as a basis for the development of an integrated evaluation tool to assess the long-term and short-term effects of the implementation of the EUOAP. The project consisted of the following workpackages:development of a toolbox to evaluate the implementation of the EUOAP;testing of the toolbox on a selected number of ongoing national action plans (desk research, interviews with experts); identification of conflict areas between national and EUOAP targets (national workshops); early assessment of potential risks and problems associated with the implementation of the EUOAP (through expert consultation); policy analysis for the implementation processes and procedures; and recommendations for different actors developed with involvement of essential stakeholders.
Results
Comparison of national organic action plans
A comparative documentation about national action plans for organic agriculture describes in terms of a desk-top study the current status quo of eight national and regional action plans for organic food and farming.The case study action plans vary with regard to theirdevelopmentprocess, targets and objectives, and emphasis of measures on certain areas. The action plan of Andalusia (ES), CZ, DK and SL address a broad portfolio of areas and measures. In contrast to this, the Dutch, Italian and English action plan give high priority to measures targeted at market development and consumer information. The Dutch action plan has a strong market driven approach. The German Federal Organic Farming Scheme has a clear focus on measures related to public information.These differences are due to quite different political/socio-economic framework conditions for organic farming in these countries at the time when these plans were established (Stolz, Stolze, Schmid, 2006).
Meta evaluation of evaluations of national organic action plans
For the development of an evaluation toolbox one important step was to get an insight into already conducted evaluation studies in the field of organic action plans in Europe via meta-evaluations from DE, DK, NL and partly from England (UK). The resulting report contributed to a methodological learning process, helped to optimize the ORGAPET toolbox and provided information on the content level about the success and failure of organic action plans in general (Dabbert and Eichert, 2007).
ORGAPET development
The development of the Organic Action Plan Evaluation Toolbox (ORGAPET) is the central part of the ORGAP project. During the project, ORGAPET has been developed in an iterative process with several versions regularly updated and further enhanced. ORGAPET has been developed as an electronic toolbox for use on-line ( or as a CD-ROM, with key documents incorporated in the toolbox andhyperlinks between the different elements designed to make navigation easy. The structure ofORGAPET consists of four main sections.Section A consists of background/contextual documents on organic action plans, organic farming policy, stakeholder involvement and evaluation principles and procedures. Section B deals with evaluation methods relating to action plan development and the implementation process, including conflicts and synergies, coherence, implementation failure risk and stakeholder engagement. Section C covers evaluation methods relating to action plan outputs, effects on the organic sector and impacts on public policy goals. Section D focuseson approaches to synthesising overall conclusions including interpretation issues relating to cause and effect relationships, interactions between elements and likely developments in the absence of the action plans or specific action points.Each section is sub-divided into a number of specific topics, with an overview document functioning as guide to key issues and possible solutions, and a series of annexes providing illustrative examples, specific methodological details or useful data sources. A manual provides an accessible guide to action plan development, implementation and evaluation at national and regional level as well as EU level.
ORGAPET testing and assessment by stakeholders and evaluation experts
Comments on the ORGAPET toolbox were collected via a comprehensive testing process in all ORGAP participant countries, in order to get feedback under the different needs and circumstances in the countries involved. The results helpedto improve ORGAPET and providedan opportunity to involve stakeholders further in the development of the evaluation methodology. Overall, the conclusion from the extensive testing process of an intermediate version of ORGAPET was that stakeholders and experts view ORGAPET in principle as a useful tool.Suggestions for structural and general changes from the experts were taken into account for the revision of ORGAPET (Dabbert and Eichert, 2007).
Focus group discussions on the national implementation of the EUOAP
As part of the analysis of the implementation of the EUOAP, focus group discussions with important stakeholders were held between November 2006 and February 2007 in 8 EU member states: the Czech Republic, Germany, Denmark, England, Italy, the Netherlands, Slovenia and Spain. The general objective of the focus group series was to make some judgments on how the recent EUOAP is expected to be implemented in the member states. The intention was to identify how national stakeholders perceived the EUOAP and its interplay with national policies and which strategies they would suggest in coping with predictable implementation problems.One topic common to all discussions was the proposal for a revised regulation on organic production, which will be implemented by all member states by 2009. It represents an attempt to realize many of the recommendations from the EUOAP. The second discussion topic in six focus groups was the recommendations on a more transparent market development included in the EUOAP.In Italy and England the focus group members choseto discuss the issue of funding organic food and farming policy through rural development plans i.e. as part of the common agricultural policy (CAP).
The main conclusion from the analysis was that the level of implementation success of the EUOAP in any member state is a matter of national balances between positive and negative aspects of the three main properties associated with all stakeholders involved in implementation: stakeholder willingness, capability and comprehension (Vedung 1997). Regarding capability, participation in the focus groups indicates that capabilities are available on both the politico-administrative level and on the level of the groups of organic food and farming. The EUOAP aims at expanding organic food and farming within a predominantly non-organic food market, and hence implementation demands capabilities of actors with only few organic activities. But nearly no stakeholder of this type appeared in the focus groups and it indicates that some of the capabilities necessary for realising the market orientated aspects of the EUOAP are not at hand. Regarding willingness, the focus groups’ views on the EUOAP varied from positive to negative. Only in the focus groups of CZ and SI did participants find the EUOAP important and had positive expectations to it; in the Danish group,expectations to the EUOAP were positive but the EUOAP was considered insignificant; in the discussions held in DE, EN and IT,expectations were neutral and the EUOAP was considered insufficient; in the Spanish focus group the EUOAP was considered insufficient and expectations negative.Regarding comprehension, each focus group mentioned many types of implementation problems and coping strategies. Only two problems appeared in most focus groups. Both related to the EU regulation. Seven focus groups discussed the scope of the new EU regulation, some preferring a narrow and others a broad scope. Six groups discussed a suggested threshold of GMO content in organic produce and all agreed that a threshold should be very low if it was to be allowed at all. All other issues were specific to the national context, suggesting that implementation problems are specific to each EU member state. On a more general level, the analysis revealed a deep scepticism about the market orientated basis of the EUOAP in all focus groups, which in itself may cause implementation problems since it counters one of the main ideas of the EUOAP.(Michelsen and Tyrol Beck, 2007),
Discussion
A major issue in the project was stakeholder involvement both in the elaboration of organic action plans as well as in their evaluation. A lack of sufficient stakeholder involvement in the elaboration of the new Council regulation (EC) 834/2007 was one of the main criticisms, which came from the national focus group discussions. Therefore the ORGAP project team elaborated proposals and gives exampleshow to improve stakeholder involvement. (Dabbert and Eichert, 2007).
Another mainissue in the project was to develop appropriate indicators for the evaluation of organic action plans, taking into account the problem of data availability and limited resources for data collection.
A third issue was how to measure the effectiveness and the direct effects of the policy separate from the general performance of the organic sector. What is the impact of exogenous events and how can these be addressed in an evaluation? As conclusion, it is important to focus on the performance of the measures against indicators. This does provide an overall picture on the impact of the OAP on the organic farming sector or the wider bio-physical, social and economic environment.
Conclusions
The evaluation toolbox (ORGAPET) has been developed as a help and guidance for planning and evaluation of action plans both on EU level or national level. However it should not be used as a “cook book” but more as a source of tools, techniques and background information, to be adapted to the specific context of a particularaction plan.
Acknowledgments
The authors acknowledge the support of the European Commission for funding this research work. The views expressed are those of the authors and do not necessarily reflect the views of the European Commission, nor do they in any way anticipate the Commission's future policy in this area.
References
Dabbert S., Eichert C. (2007): Public synthesis report on the scope of national action plans, their evaluation procedures and the operability and appropriateness of the developed evaluation concept at national level, as well as the impact of conflict/synergies and policy proposals for implementing the EU Action Plan in member states. Project Deliverable D8. University of Hohenheim. Project website:
Michelsen J., Tyroll Beck A.-M. (2007): Implementing the European Organic Action Plan in EU member states. Stakeholders’ perceptions of implementation problems and coping strategies. Project Deliverable D7. University of Southern Denmark. Project website:
Stolz H., Stolze M., Schmid O. (2006): Documentation about national Action Plans for Organic Food and Farming. Project report. Research Institute of Organic Agriculture (FiBL). Project website:
Vedung, E (1997): Public Policy And Program Evaluation, London: Transaction Publishers.
[1]Research Institute of Organic Agriculture FiBL, Ackerstrasse, 5070 Frick, Switzerland, E-Mail:, Internet:
[2]Institute of Rural Sciences, AberystwythUniversity, Llanbadarn Campus, AberystwythSY233AL, United Kingdom, E-Mail:, Internet:
[3]Institute of Farm Economics (410A), University of Hohenheim, D-70593 Stuttgart, Germany, Email: , Internet:
[4] Department of Political Science and Public Management, University of Southern Denmark, Niels Bohrs Vej 9-10, 6700 Esbjerg, Denmark,E-Mail: , Internet: