Evaluation of Development Activities at the Macro Level – Challenges and Experiences in Cambodia

Paper prepared by THENG Pagnathun, Director General, Ministry of Planning, Royal Government of Cambodia.

Measuring Results is Paramount

If you do not measure results, you cannot tell success from failure

If you cannot see success, you cannot reward it

If you cannot reward success, you are probably rewarding failure

If you cannot see success, you cannot learn from it

If you cannot recognize failure, you cannot correct it

If you can demonstrate results, you can win public support

  1. Statement of the Problem

Evaluation should be a systematic and impartial assessment of aproject, programme, sector or development process at theaggregate level;that it should quantifyaccomplishmentsby examining the results-chain, contextual factors and causality to assessoutcomes; and it should aim at determining the relevance, effectiveness, efficiency and sustainability of interventions.Additionally, evaluation is also an important source of evidenceof achievements in addition to it being a contributor to knowledge-building and organizational learning. Evaluation is not independent of monitoring, a process of tracking or measuring the performance more frequently, and auditing, which addresses financial accountability. Evaluation is necessarily evidence-based. This is the Cambodian government’s position.

This paper notabout evolvingyet another definition of evaluation. Instead, it examines various nuances that Cambodia(or any small developing country) faces in conducting evaluation exercises [or Monitoring and Evaluation (M&E) in general]for periodic reporting to the senior management in the government and the development partners.More specifically, the concerns presented are:

1. Whatshould be the structure of a Results Framework applicablefor macro level evaluation, this level of reporting being fundamental innational planning?

2. How to generate the necessary data required for populating the Results Framework forms?

Theseconcerns forma part of a debatewithin the Ministry of Planning (MOP)while it prepares an M&E Framework for the next National 5-year Development Plan 2014-2018. It is believed that other least-developed countries (LDC) as well, face similar challenges and there is need to address them in a practical and meaningful manner.

  1. Country Context

Cambodia, classified it as anLDC,had a per capita income of about US$931 in 2011. It went through war and turmoil between the early-1970s and mid-late 1990swhenits institutions of governance, infrastructure, human capital, social fabric and more, were severely damaged.Only in the new millennium, substantial activities relating to socioeconomic development were taken up.

Cambodia pursues a development strategy through planned development in a market framework. Through the last 17 years the Cambodian economy on average has grown ata rate 7-8% annually and if the downturn of 2009 is deleted from the trend, the rate is yet higher. In the recent 4-5 years, the global economic turndown and the commodity/petroleum-led inflation have resulted in a global economic turmoil leaving out few countries unaffected. The Cambodian economy, however, has shown the capacity to bounce back after facing a severe setback,though the turmoil has left its scars: many development programs have had to be rescheduled or staggered. International development assistance plays an important role in funding and providing technical assistance.In 2011, international assistance was estimated at about 8.5% of the GDP. More than 60% of the total developmental expenditure in the public sector (typically, agricultural extension, irrigation, education, health and infrastructure) is funded by grants and soft loans from development partners. In this regard, a strong M&E Framework and Evaluation assume increased importance.

  1. The Results Framework for M&E

3.1 The Conventional Wisdom

The government has adopted a Results Framework (RF) to account for and evaluate all developmental activities for providing vital directions at different stages in project/programme management, allowing senior government staff to answer four key questions:

  1. Is the plan being effective?
  2. How does one know whether one is on the right path?
  3. If not, where is the deviation?
  4. How does one use this information continuously for regular corrective action?

It believes that expenditures made need trackingfor ensuring that theyget translated into outcomes. Take the case of the Education Sector:

  1. It is not enough to allocation of $XXXX for a school building has been adequately spent (Inputs)
  2. The school should have actually been constructed as per the specifications and on time, andhas its hardware, trained and qualified personnel, teaching curricula, etc. (Activity/Output)
  3. Additionally, the school attracts children from the catchment-area (Outcome)
  4. Finally, children attend schools, pass and become literate/ educated, and the society moves towards becoming more productive, more jobs are created, poverty reduces, etc. (Impact)

This framework has been generally endorsed by the development partners as well. A more generalized form of a RF could be seen in Figure 1.

Figure 1: Basic components of a results chain

3.2 Some Practical Considerationsin UsingRF at Macro level

3.2.1 Measuring Outcomes and Impacts

In practice, there are issues in identifying and measuring variables at differentstages in the RF.In national planning, impacts (and in most cases, outcomes) are macro level phenomena, while interventions could be policy or launching projects. Three examples illustrate the macro-micro disjoint.

Case1:Poverty

Most development projectsmention the outcome to be ‘poverty reduction’, irrespective of whether the projects are of infrastructure, potable water,health, sanitation or education.

  1. Poverty, a macro phenomenon, is reduced by a number of factors ranging from individual attributes to performance of the the economy. It is difficult to linkpoverty reduction with one intervention; a full identification of the influencing factors is required. The RF thus requiresbeing made significantly more complicated.
  2. Next, the relationship between poverty alleviation and its determinants could change in time. A typical case is of safe water which when provided improves health, in turn which is expected to improve educational attainments and incomes. However, after everyone has been provided with clean water, other factors influence poverty reduction and not safe waterper se.
  3. Finally, outcomes (poverty, inequality) being macro level variables, it is not easy to link these with a RF drawn-up for individual projects or sectoral programs: the typical macro-micro disjoint.

Case2:Agriculture

In the Agricultural Sector, take the cases of crop yield rate.Government inputs like agricultural extension work and irrigation facilities certainly help. But yield rates increase also becausefarmerstake up to scientific agriculture, seeing profits. When a new scientific method of doing agriculture emerges, farmers learn and then draw upon services that an emerging group of private providers offer as government services do not reach everywhere. They sink wells for irrigation, quite independent of government-created large dams. Actually, it has been witnessed in Cambodia that yield rates began to increase much earlier to the completion of dams in many areas in the western parts of the country. The causality is thus warped.

Case3: Education

In SchoolEducation,the government provides buildings, teachers and other wherewithal,which have a definitive impact on the enrolment rate.However,in Cambodia it is seen that children are sent to schools not only because the government has built schools but also because parents want their off-spring to be educated. Ifgovernment schools are not available the private sector provides services. Recent reports on school enrolments suggest that in Cambodia, even poor households prefer expensive private schools because children at least get some quality education there, often scarce in government schools.

To empirically validate this point, a regression equation was estimated to determine whether supply or demand side variables are more important in explaining the proportion of children aged (6-17 years) in schools,using village-level data for all villages in Cambodia for which data were available. The estimates,given in Table 1, amply demonstrate that supply-side variables, namely distance of government schools (primary, junior secondary or senior secondary) from the villages, are statistically not significant in explaining school attendance. Instead, the demand side variables are statistically significant (measured by level of affluence for which the proxy variables are: possession of assets like motorbikes and cycles, having (not) to live in thatch houses, accessing potable drinking water, having sanitary latrines, etc.

Table 1:Regression Results Explaining School-attendance in a Cross-section of 11,882 Villages in the Country, 2011
Dependent Variable: percent children 6-17 years in school / Coefficient / t-statistic / Significance
(Constant) / 73.073 / 99.406 / 0.000
Distance primary school (Km) / .002 / .302 / .763
Distance Junior secondary school (Km) / -.002 / -1.127 / .260
Distance Senior secondary school (Km) / -.002 / -.997 / .319
Wet season paddy yield / 1.290 / 5.850 / .000
Distance of village to province town / -.035 / -6.528 / .000
[(Total motorcycles)/(Total families)]X100 / .011 / 1.798 / .072
[(Total cycles)/(Total families)]X100 / .055 / 15.664 / .000
percent families living in thatch houses to total houses / -.138 / -15.785 / .000
[(Number of toilets)/(Total families)]X100 / .085 / 14.778 / .000
[( Numberof families accessing clean water)/(Total families)]X100 / .051 / 14.257 / .000
Percent families farming less than 1 ha land / -.003 / -2.683 / .007
R2 = 0.376; F = 177.40; n= 11,882

There is evidence thus, that a conventional RF could be flawed if it is applied in its generic form in macro level evaluations. Thereis need to bring-about significant sophistication in the models;also, one-type of model will not fit all situations.

3.2.2 Specification Problems

A second type of problem arising when outputs stand for outcomes. E.g. the output variable, ‘increased visitsto health centers’is often taken to depict outcomes. This is because outcomes of health interventionare slow and captured in large surveys carried out once in 5-10years. The only tangible indicator for an (annual)reporting is the number of visits (or other process variables like malaria/tuberculosis cases treated). In a conventional RF, the following hazards emerge:

  1. Health outputs taken for outcomes could result in ‘moral hazard’ (over-use – like counting the (free) visits of patients to Health Centers).
  2. On the converse, there could be under-utilization of outputs owing to reasons like their location, access by users, etc. (schools constructed, but children do not go).

Once again, the Results Framework requires to be sufficiently flexible to accommodate such nuances.

3.2.3 It Follows Therefore…

Thesecasesdo not deny the importance of a Conventional Results Framework but make a case for a more realistic Framework, forit becoming a tool for evaluation rather than be a ‘one size fit all’ wand.

Alternatives have been suggested like constructing approaches based on Theory of Change. Putting these into practice, though, is not easy: themodelsare extremely complex, requiring a great deal of data and resources which are simply not there. Additionally, there are issues in human capacities: normal government officials do not possess skills to construct/interpretcomplex models.

3.3 Alternatives in RF for Evaluation in Cambodia

Recognizing the issues discussed above, the governmenthas adopted a RF which matches the performance with a stipulated target at the beginning of a 5-Year Plan. These are mainly outcome/impact indicators, though a few output indicators also appear in the list. At the sectoral level though, the government encourages use of the conventional RFs with whatever modifications the sector authorities/ministries prefer to make.At the programme and project levels it is mandatory for the concernedmanagers deploy anear-conventional RF.

The Ministry of Planning in consultation with other line ministries and agencies has identified some 64 core M&E indicators and some 125 auxiliary M&E indicators to assess the annual progress made in the economy (growth, inflation, trade and balance of payments, debt, government budget), the Millennium Development Goals, and other key sectors (external financial assistance, employment, transport, infrastructure). While the core indicators are mainly multi-sectoral and cross-cutting (e.g. poverty, growth or child health), the auxiliary indicators are mainly sectoral. The list will expand further depending upon the need and the requests from various line departments.Consultations are continuing with different stakeholders. The indicator list includes all the identified MDG Indicators.

The core indicators are divided into five categories:

  1. Aggregate outcome indicators – GDP, Poverty, Inequality, Inflation;
  2. Aggregate output indicators which stand for outcomes – Balance of Payments, Import/Export, Structure of GDP and Workforce;
  3. Sectoral outcome indicators – Infant Mortality Rate, Maternal Mortality Rate, School Completion Rates;
  4. Sectoral output indicators which stand for outcomes – Crop Yield Rates, Area Under Crops, Roads Made, Attended Births, Enrolment Rates;
  5. Proxy indicators – E.g. for Governance, Inclusive Growth

Note: Classification of the auxiliary indicators is more complex, them being mainly sector-specific and serving the needs of stakeholders in the concerned sectors.The indicators are a mixture of outcome, output and process. Even NGOs draw up their own list of indicators. Additionally, the extent of standardization has been more limited in them than inthe core monitoring indicators. This aspect needs strengthening.

3.4 Reporting

Earlier, the MOP brought out a macro level evaluation report of development activities only in 3-5 years. In 2010 and 2011 it brought out twosuccessive (annual) MDG Progress Reports. Starting from 2012, the reporting was for both the National 5-Year Plan and MDGs, and isannual. In the 2014-2018 cycle as well, it should stay that way.

At present, only the aggregate country-level indicators are being reported upon. From 2013 onwards, some province-level disaggregated data too would be presented (esp. on indicators developed from Administrative Statistics), because regional disparity is a recognized concern.Effort is also being made to bring-in some non-official project data into the Official Statistics Framework.

Finally, in the last few years the government, under its sub-national governance programme, has begun to collect administrative data at the village and commune levels. While as of now their validity is yet to be established, in the times to come itis hoped to mainstream and strengthen this database.

  1. Data Related Issues

An evaluation exercise requires the right data and ofhigh quality. In most LDCs this is the ‘Achilles Heel’.Sample or census surveys are conducted with external funds and expertise, and by agencies which determineboth,the data generation process and definitions of variables. To an extent, the government officials have begun to participatein the exercises but this is yet limited. As a result, continuity of surveys and comparability of definitions across surveys (and also the same surveys over time) are not guaranteed. Cambodia has now established a Standing National Working Group on M&E, which is in the process of standardizing definitions of variables and indicators. The government believes that the recommendations of this committee will bring some significant changes in the data systems; consequently, the M&E.

Next, it is now recognized that effort should be made to strengthen Administrative Statistics in ministries and departments since their coverage a broader, they could be more regular, and be less expensive. The United Nations also recommends strengthening Administrative Statistics. In Cambodia almost all ministries collect administrative data but their quality is not above question. Among the reasons:

1. While the central offices are somewhat better staffed, the provincial and district staff are not. At the district level, there is only one official belonging to the Planning Department and s/he has neither asupport staff nor the resources to scientifically collect,collate and present data. S/he relies on village and commune chiefs (who are elected representatives and as such, have no formal training in statistics, data management or for that matter in any field) to collect data by means of a Village Book or a Commune Book.

2. Some ministries collect data based on rather small and not necessarily scientific surveys.

3. In a few ministries, administrative data are not collected every year for want of adequate resources; instead, linear projections from past data are made.

4. Data management facilities in many ministries is weak on many counts: knowledge of basic statistics, availability of computers, and devices to store data. The position progressively deteriorates from center to province to district. Some communes/villagesyet have no electricity – no devices work there.

5. Commune- and village chiefs and other local representatives are to an extent being trained, but one problem is their weak initial exposure, another is that they have other chores to do as well (they are not dedicated staff), and finally, they could change every 5 years, as elections are held 5-yearly.

It is of concern to note that the development partners make a great deal of effort to draw up forms for measuring success but leave aspects relating to populating these forms with quality data to national governments. This is particularly true for macro/sectoral data.

  1. Approach

The key question is, how to strengthen the statistics for M&E?MOP, in its effortto make M&E more useful in planning, is trying to train government staff in line ministries on generating and interpreting data, managingAdministrative Data CollectionSystems (selectively) and on conducting Small-Sample Studies/Case Studies. A standard Training Module has been developed for this purpose, and until so far several batches of personnel in 7 ministries have been trained. Once the staff at the center are trained, it is believed that they would then pass on this knowledge to staff at the provincial and district levels.

Central to making the M&E improvement is the M&E Working Group. Riding on the success of defining a new poverty line in 2012 (by the government. For the 1st time – earlier, the World Bank did the exercise), MOP is following the approach of engaging officialsfrom line ministries in task forces. At the central level, it has been possible to change certain definitions, alter/improve the data collection process at least in surveys conducted by the MOP, and open doorsin other ministries for engagingthem in discussionson their data collection processes (e.g. Agriculture, Rural Water Supply andGender).These processes have been extremely compartmentalized until so far—each ministry is still extremely compartmentalized even now—but a beginning has been made.