Ex-post evaluation of the sustainability of the Belgian governmental development cooperation.
Executive summary
Evaluation of sustainability
The Special Evaluation Office of the Belgian development cooperation commissioned an ex-post evaluation of the sustainability of results of the Belgian governmental development cooperation. The evaluation focused on two partner countries – Benin and Bolivia – and on two sectors – health and agriculture – and covered the period 2000-2015. A consortium of two consulting companies – hera and ADE – implemented this evaluation.
At the start of the evaluations two Theories of Change - one for the Belgian cooperation with Benin and another for Bolivia - were elaborated. These Theories of Change formed the basis for the evaluation matrix with five main evaluation questions that guided the data collection and analysis process. The evaluation team made use of various methods to assess the sustainability of development results, including document review, interviews and focus groups. Learning is the main objective of this evaluation, in addition to accountability.
Sustainability in the context of development cooperation is concerned with measuring whether the benefits of an intervention are likely to continue after donor funding has been withdrawn. Seven components of sustainability have been distinguished: 1) political; 2) institutional; 3) economic and financial at the level of beneficiary institutions; 4) economic and financial at the level of the targeted population; 5) socio-cultural; 6) environmental; and 7) technological sustainability. Given the fact that the evaluation team had to rely to a large extent on secondary sources of variable quality, not all components of sustainability could be assessed in equal detail. In particular, the assessment of socio-cultural sustainability proved to be problematic given the relatively short duration of the field work and the absence of good secondary sources.
Given the focus on drawing relevant lessons, the evaluation team paid due attention to the analysis of factors that explain why results are sustainable or lack sustainability. External explanatory factors are related to the evolving context in partner countries and in the specific sector. Internal explanatory factors are related to the Belgian governmental development cooperation. A traffic light analysis has been made to assess and illustrate whether explanatory factors contributed in a positive, a neutral or mixed, or a negative way to the sustainability of results. The scope of the evaluation was limited to two countries and to two sectorsand therefore, the evaluation results cannot be generalized. Nevertheless, the evaluation results have been situated in the evolving context of the Belgian governmental development cooperation, including the management contracts between the Belgian state and the Belgian Technical Cooperation (BTC), the implementing Belgian Development Agency.This allowed to formulate some general lessons on how to better address sustainability throughout the entire policy cycle.
Sustainability is one of five main evaluation criteria in development cooperation, as specified by the Organisation for Economic Cooperation and Development (OECD), but is given relatively limited attention because ex-post evaluations after finalization of the interventions do not frequently occur. Furthermore, mid-term and final evaluations of projects do not provide insight into the sustainability of results, but can only assess potential sustainability. Nevertheless, useful methodological lessons can be derived from recent ex-post evaluations of impact. An important common lesson from this evaluation of sustainability and other impact evaluations is that good quantitative and qualitative baseline data are required in combination with longitudinal data that allow for assessing the impact and sustainability of development interventions. The evaluation team presents specific methodological lessons on how to improve the evaluation of sustainability in a separate methodological learning note.
Differences in country and sector context Benin and Bolivia
Both countries have a population size of approximately 10 million people. Benin, however, has a high population growth rate of 3% per year, while the growth rate for Bolivia is 1.6% (2013 figures). Both countries also have relatively stable democratic regimes and have experienced gradual economic growth, although Bolivia grew since 2006 at a faster pace than Benin. Therefore, Bolivia has reached the low middle-income status (GNP per capita US$ 2620 in 2013), while Benin is still a low-income country(GNP per capita US$ 790 in 2013). The growth in Bolivia led to a reduction in moderate poverty, although 45% of the population is still affected by poverty and income inequality remains high. The percentage of the population living in poverty in Benin is estimated at 36% and does not show changes despite the moderate economic growth.
In general, Bolivia is characterized as a country making gradual progress and pursuing clear socio-economic policies. International development support to Bolivia has been reduced considerably and is now approximately 1% of GDP. Many bilateral donors have exited or are preparing their exit from bilateral governmental cooperation. Benin on the other hand has also developed new policies, but faces important implementation challenges and setbacks. Benin still remains rather aid-dependent and approximately 50% of government expenditure is funded by donors. In Benin, poverty is still widespread and the economy remains undiversified and vulnerable to external shocks. Poverty and food insecurity is more widespread in rural areas than in urban areas. The general differences in the country context are also reflected in the two sectors. Health indicators in Bolivia are considerably better than in Benin and show gradual improvement, while improvement in Benin is slower. Regarding the agricultural sector, the contribution to the national economy varies considerably: in Benin it is 35-37% of GDP, showing a slight increase and in Bolivia it is 13-15% of GDP, showing a decrease between 2005 and 2013. In Benin agriculture is considered as one of the motors of economic growth.
The Belgian governmental development cooperation in Benin and Bolivia
The Belgian development cooperation with partner countries is guided by Indicative Cooperation Programmes (ICPs), which are formulated jointly, in principle for periods of four years. While the main focus of the ICPs is on governmental or direct cooperation, which is implemented by BTC, the ICPs also cover the indirect cooperation with partner countries through Non-Governmental Organizations (NGOs) and universities. The bilateral policy dialogue between Belgium and its partner countries is linked to the ICPs.
An overview of the duration and funding committed in the ICPs for the two countries is presented in table 1. This overview shows that for Benin the ICPs were mainly implemented according to planning with only one extension (for the third ICP). For Bolivia the implementation was more problematic with several long extensions. The amount of funding for Benin was considerably higher than for Bolivia. The Belgian government has announced that the governmental cooperation with Bolivia will be ended on 30 June 2019.
Table 1. Indicative Cooperation Programmes (ICPs) Benin and Bolivia, 1998-2017
Years / 98 / 99 / 00 / 01 / 02 / 03 / 04 / 05 / 06 / 07 / 08 / 09 / 10 / 11 / 12 / 13 / 14 / 15 / 16 / 17Benin
million € / ICP 1998-2003 / ICP 2004-2007 / ICP 2008-2011 / ext / ICP 2013-2017
25 / 30 / 52 / 55
Bolivia
million € / 96-98 / ICP 2000-2003 / Ext / ICP 2008-2011 / ext / ICP 2014-2016 / ext
30 / 40 / 15
The cooperation with the two partner countries has been remarkably stable over the last years and in both countries the two sectors – health and agriculture – have been supported by Belgium for more than 20 years. In the past, Belgium supported more sectors, but over time, in line with international agreements on aid effectiveness, the Belgian support to partner countries has been concentrated in two sectors. Crosscutting themes such as gender and environment are an integral part of the ICPs.
The ICPs for Benin are characterized by a clear focus on two regions: Atacora-Donga in the North and Mono-Couffo in the South. In Bolivia the regional focus changed over time: initially there were quite some interventions in the lowlands around Santa Cruz, but the ICP 2000-2003 focused on the poorer highland regions, in particular the province Chayanta. In later ICPs for Bolivia, the regional focus was less explicit although the agricultural interventions were more or less concentrated in the area of the departments of Cochabamba and Chuquisaca, plus the province Chayanta belonging to the department of Potosi.
In both sectors in both countries a series of projects were implemented by the Belgian development cooperation, with attempts to adopt more programmatic approaches. However, in none of the countries was a real sector approachadopted, based on agreements between the national government and donors on the implementation of sector policies and a regular policy dialogue. All interventions however were closely aligned with national policies. The dominant aid modality was project support. Only in Bolivia has another aid modality been applied which consisted of Belgian contributions to basket funding for national malnutrition and agricultural programmes. In Benin, due attention is paid to the institutional anchorage of the various projects trying to link interventions at the local level with interventions at the national level.
Outputs and outcomes
In line with the Theories of Change developed by the evaluation team, ongoing outputs to which Belgium contributed through its various interventions have been grouped and are summarized below.
Type of outputs / Benin / BoliviaHealth
Improvements of the supply side / Provision of infrastructure and equipment for health facilities;
Strengthened capacity of health staff at various levels (including national level);
Improved health planning and management systems at various levels;
Improved national policies such as the national policy for blood transfusion. / Provision of infrastructure and equipment for health facilities;
Some strengthened capacity of health staff at local level;
Improved planning and implementation of health strategies at the level of municipalities.
Improvements of the demand side / Strengthened health user platforms;
Income-generating activities at local level for community health workers, women groups, and other members of the health user platforms;
Set-up of a results-based financing system for health. / Strengthened capacities of local authorities, community leaders and community groups related to health and nutrition services.
Agriculture
Strengthening of the capacity of state and private actors / Some provision of infrastructure and equipment to state and private actors; Strengthened capacities of state actors at various levels and of some producer groups. / Provision of infrastructure and equipment in particular to organizations of producers: increase in productive capacity;
Strengthened organizations of producers;
Strengthened capacities of agricultural teams/departments in municipalities and in the Department of Cochabamba;
Formulation of national policy for irrigation basins.
Improvements of agricultural production techniques or other improvements in the agricultural value chain / The focus on intermediaries led to improved service delivery to producers regarding commercialization of crops and some improved distribution of inputs;
It is plausible that Belgium contributed to the observed increases of agricultural production in Atacora-Donga. / Improved irrigation basins and adequate irrigation management of these basins;
Introduction and adoption of improved agricultural techniques;
Introduction and adoption of improved livestock practices for animal health;
Improved commercialization;
Diversification and increase of agricultural production.
There are important differences and similarities between the approach of the Belgian cooperation in the two countries and the outputs realised. In Benin, the main focus has been on strengthening the state actors at various levels in order to provide better services to the population, while at the same time aiming to strengthen the involvement of the population. This can be related to the double institutional anchorage at local and national level that is the core of the Belgian development cooperation with Benin. In Bolivia, the focus has been primarily on the strengthening of the delivery of services at local level and the strengthening of community and producer groups. The outputs in particular at the level of the target group in Bolivia are more concrete and tangible than in Benin. In both countries, the performance of projects was variable with some rather successful projects, but also projects that failed to a large extent to realise the intended outputs. A health project in the Chayanta province of Bolivia focused primarily on the construction of health facilities and provision of equipment (provision of hardware), while the adequate staffing of these facilities was given relatively less attention. Some agricultural projects in Benin and Bolivia tried to introduce agricultural innovations that failed to be adopted, although in Bolivia there was successful introduction of new fruits such as peaches. Four of the ten selected completed interventions in the two countries were considered to be effective as they realised most of the planned outputs and outcomes.
Given the difference in context and approach, it is no surprise that more tangible outcomes have been realised in Bolivia than in Benin. Nevertheless in Benin the health performance indicators - such as the attendance of health facilities and the number of assisted birth deliveries - in the two health districts that received long-term Belgian support are significantly better than the average in the region, although not all the national targets are realised. For Bolivia, there is a clear improvement in the nutrition situation of children, in particular below the age of two years and there is also an improvement of knowledge on nutrition among the target population.
The agricultural interventions supported by Belgium – together with other donors – contributed directly to increases in production of specific crops, fruits and vegetablesin Bolivia and better marketing. In Benin, however, no direct linkages can be established between the Belgian support and increases in agricultural production, although it can be assumed that Belgium through its bilateral support contributed to some extent to the positive dynamism. The extent of the Belgian contribution to agricultural production increases cannot be assessed given the limited role of the Belgian cooperation and the varying economic improvements in both countries.
The ex-post assessment of the outputs and outcomes in Bolivia is more positive than was expected on the basis of the analysis of secondary sources – including final evaluations – and interviews in Brussels prior to the field visit.
Political, institutional and economic-financial sustainability
Political and institutional sustainability received most attention in policy design, project formulations and evaluations. It is clear that through the ICPs all interventions are closely aligned to the national sector policies. This can be considered as a pre-condition for the political sustainability of results. However, the main issue affecting sustainability is whether sector policies have been effectively implemented. This requires good leadership and the willingness to change. In general, the political and institutional conditions are better in Bolivia than in Benin, and therefore more favourable to realise sustainable development results.
In Bolivia, the decentralization policies that were implemented since 1994 provide resources to municipalities to deliver services to the citizens. The Belgian support, in particular for agriculture, focused on the strengthening of capacities at the level of municipalities and communities, while making the necessary linkages to the departmental and national level where appropriate. Some of the achieved outputs and outcomes in the two sectors are sustainable from a political, institutional and economic-financial point of view. Nevertheless, there are still significant weaknesses in the systems such as frequent rotation of health staff, insufficient knowledge and tensions between the central level and the municipalities. If some projects had been better formulated and had aimed for a better balance between the ‘hardware’ and ‘software’, especially in health, more sustainable results could have been achieved. In reality, municipalities and other actors proved to be capable of continuing the activities once the Belgian development cooperation stopped. This does not mean that there is no room and need for further improvement, but first steps have been made.
In Benin, the assessment of the sustainability of results is challenging, because the Belgian development cooperation interventions are still being continued in the same (thematic and geographic) areas. Therefore, the hypothetical situation – what would happen if support was withdrawn – has to be assessed. Despite institutional strengthening at various levels, the assessment of the expected institutional sustainability of results is not very positive. First of all, there is insufficient funding available for delivery of services. With 10% of government funding allocated to the health sector, Benin fails to meet the target set by the Abuja Declaration to allocate at least 15% of their annual budget to improve the health sector. The delays in implementation of the necessary structural reforms agreed upon in various policies hinder sustainability and this is related to a lack of leadership. In general, the institutional environment in Benin – with few exceptions – remains rather weak and fragile, despite all efforts of strengthening it. The improvements in the blood transfusion system, where an appropriate national system has been set up, are a positive exception. This system has good perspectives to remain functioning at adequate levels, at least if funding is guaranteed. However, overall aid dependence remains very high and the main political, institutional and economic-financial conditions for sustainable results are not yet achieved in Benin.
Socio-cultural and socio-economic sustainability at the level of the targeted population
The sustainability of results at the level of the population received relatively limited attention, although in both countries and in both sectors efforts were made to involve the population in the activities indicated above. The sustainability of strengthened community and producer organizations requires an appropriate institutional framework and continued availability of funding. As the Bolivian municipalities have (some) resources and are used to working with community groups, the perspectives for sustainable results are relatively positive. This is to a lesser extent the case in Benin, where it cannot be expected that the government will start funding community groups out of its own budget in the near future. On the other hand, the projects did not assess the capacity of the population to auto-finance their own organizations (although this is not expected to be very high). This means that the sustainability of community and producer organizations would be at stake if donor funding would be withdrawn.