HOW ISSUES OF TIMING OF MID-TERM EVALUATION AND ACCOUNTABILITY HAVE INFLUENCED THE IMPLEMENTATION OF MfDR IN NILE BASIN INITIATIVE.
by
Pamella Lakidi Achan
M&E Assistant, Nile Basin Initiative Secretariat, Entebbe, Uganda.
17th October, 2008
The objective of this paper is to highlight some of the challenges in the implementation of MfDR in the intergovernmental organization of the Nile Basin Initiative. Pre-mature mid-term review implementation has resulted in a focus on implementation rather than the output-outcome link which is the principle of MfDR. Accountability for the outcome level of results calls for partnership between the project implementers and those having oversight on it.
Background
The Nile Basin Initiative (NBI) is a partnership initiated in 1999, and led by the riparian states of the Nile River, that seeks to develop the river in a cooperative manner, share substantial socioeconomic benefits, and promote regional peace and security (World Bank, 2003:1). NBI’s Vision which also it’s long-term goal is; “To achieve sustainable socio-economic development through equitable utilization of, and benefit from, the common Nile Basin Water resources”. NBI is converting it’s shared vision into action, through implementation of two complementary programs: the Shared Vision Program (SVP) of basin wide projects and Subsidiary Action Programs, investment programs at a sub-basin level involving all potentially affected countries (World Bank, 2003:1). The SVP program aims at promoting the shared vision of the Nile riparians through “creation of an enabling environment for investments and action on the ground, within an agreed basin wide framework (World Bank, 2003).”
1.0 SVP projects MTR Timing and MfDR Implementation
The Mid-Term Reviews (MTRs) of the SVP projects was conducted in 2007 as shown in Table 1. For the 3 SVP projects, EWUAP, RPT and SDBS, whose MTRs were conducted immediately after the actual project effectiveness, the results were generally unsatisfactory. The review team recommended that these projects should not continue beyond 3 years, even though, they initially were to be implemented in two phases, each of 3 years. This premature closure of the 3 projects has had two major negative and unintended effects. The first is that staff in these projects feel that they have been unfairly evaluated. They sight technical issues such as late or prolonged durations taken in staff recruitments through UNOPS and other administrative bottlenecks that hindered an early or prompt start in project implementation as having been overlooked. According to them, due to these constraints, by the time of the MTR, the projects had hardly “delivered” anything.
Second negative effect of the closure of these projects soon after their dates of effectiveness is that staff are entirely concentrating on “finishing”. In all the 3 projects the delivery of project outputs (implementation) is the major focus with little to no attention being given to the output-outcome linkage (results) on the likely effects of these outputs. According to World Bank, 2004, this is the traditional approach to M&E which is not in line with MfDR principles. In an RBM context, carrying out or completing a program/project activity does not in itself constitute a developmental result (CIDA, 1999).
OECD/DAC defines an Outcome as “the likely or achieved short-term and medium-term effects of an intervention’s outputs” (OECD/DAC, 2002). In my opinion, focusing on the likely effects of the outputs would entail the project team making an assessment of the quality of the outputs they are producing and ensuring that the target group accesses or is facilitated to utilize the project outputs. However, in the circumstance that the project teams find themselves in, there is no time for the project team to hold stakeholder meetings and other kinds of dialogue which can be avenues for promoting the utilization of project products and services by the target group, thereby leveraging the realization of the output-outcome link.
Table1: SVP Projects and their MTR
SVP Project Name / Project Duration / MTR Date / MTR Rating / StatusApplied Training Project (ATP) / 25 September, 2004 – June, 2010) / July 16-27, 2007
(approx 3 years after project effectiveness) / Satisfactory
Confidence Building and Stakeholder Involvement Project (CBSI) / I: March 2004-December 2008
II: Jan 2009-June 30, 2010 / July 16-27, 2007
(approx 3yrs after project effectiveness) / SI/CB: satisfactory
PI: marginally unsatisfactory[1]
Efficient Water Use for Agricultural Production Project (EWUAP) / Three years: Started in 2007 and is expected to
finish by 2008 / July 16-27, 2007
(zero yrs after project effectiveness) / Marginally unsatisfactory / Ending December 2008
Nile Transboundary Environmental Action Project (NTEAP) / 5 years: started October 2003(RPM & FPO
hired)/May 2004(Project Launch) / July 16-27, 2007
(approx 3-4yrs after project effectiveness) / Satisfactory
Regional Power Trade Project (RPT) / Four Years : Phase 1 is for 2.5 years ending in June 2008 / July 16-27, 2007
(approx 1.5 years after project effectiveness) / Marginally satisfactory / Ending December 2008
Socioeconomic Development and Benefit Sharing Project (SDBS) / Three years (2005 – 2008) / July 16-27, 2007
(actually 1.5 yrs after project effectiveness)[2] / Marginally satisfactory / Ending December 2008
Water Resources Planning and Management Project (WRPM) / From 2004 to 201[3] / July 16-27, 2007
(Approx 3 years after project effectiveness) / Satisfactory
Source: SVP MTR report 2007 and Project Annual Reports, 2007
One would argue that leveraging utilization of project outputs is not project manager’s work. That would be the case if the government counterparts (usually Project Steering Committee, Task Force and other Working Group members) had explicitly taken up the role of leveraging for the utilization of the project outputs. Unfortunately, this is not the case as these groups focus mostly on the technical issues related to the projects they are involved with. Therefore, we are left with a scenario of assuming that the delivery of outputs by the projects will automatically result into “enabling environment”.
There is an “SVP mainstreaming strategy” through which SVP outputs will be sustained in the institution. It is important to note that the mainstreaming strategy is addressing sustainability and not change as a result of the outputs produced by these projects. The focus of this paper analysis is the developmental change that is expected from the utilization of the project outputs. This cannot be attained at the corporate level, but at the project level where dialogue and negotiation takes place. In conclusion, for the 3 SVP projects that close in December 2008, the result chain is incomplete. It is inputs →activities →outputs and then stops at outputs. The assumptions and risks that influence the output-outcome link are not being taken care of. Is it possible that this aid going to be effective?
2.0 Accountability Roles and Responsibility for Results and MfDR Implementation
Currently, the reporting at NBI is such that individual SVP projects compile their Result based progress report and submit it to the Nile Secretariat M&E . The Nile Secretariat M&E unit compile the SVP project level reports and the SAP program reports into a corporate level report. Typically, the reports submitted to Nile Secretariat are activity based. A few reports do focus on results, and when they do so it is at the outputs level. There is not much information on the outcomes of the project interventions.
One of the reasons, for this lack of reporting on outcomes is that there is currently no simple methodology and tool in place for capturing outcome level data, on an annual basis. Another reason is that even for the current outcome level results, for most of the projects, the statements are not yet concluded on. Project teams have met with various implementation obstacles that have made them realize that the initial “outcomes indicators” or “outcomes” they envisaged will take longer than they thought to materialize and also involve stakeholders outside of their control. An example is taken from NTEAP. Table 2 below illustrates this point.
The dilemma that the above project has found itself in is the challenge of how to deal with capacity development in defining the logic of a project (Lavergne, 2002). According to Lavergne, “improved capacity is a relatively intangible type of outcome, compared to more tangible results such as reduced child mortality, improved access to education”. SVP is typically a capacity development program and all it’s projects are facing a challenge in clearly defining the outcomes and who is held accountable for them. In this context, the shift by the project team to a lower “achievable” outcome result leaves the next level of outcome result without anyone being held accountable for it. This is so mostly because, internally, at the Nile Secretariat there is no specific staff that is held accountable for such results. Externally, the counterparts (who participate in the projects in the form of PSC (project steering committee), TF (Task Force) and WGs (Working groups) are typically focused on providing technical oversight but are not held accountable for the outcome results.
Concluding Remarks & Recommendations
The 3 projects that are ending in 2008 would have been more effective, if they had been given ample time for implementation of the development interventions initially envisioned, thus completing their development logic framework up to the outcome level of results.
There is a need to expand the current M&E system at the NBI to include government counterparts. For instance we can adopt the first-order, second-order and third-order approach to identification of the outcome results, proposed by Lavergne. If adopted, the project teams would be accountable for the first-order outcomes, the government counterparts (PSC, TF, WG and Nile Secretariat designated staff or unit) for the second-order outcomes and then higher government stakeholders (TAC, COM) for the third-order outcomes and impacts. Each category would commit to do their part and also follow through. I believe that this is in line with the Paris Declaration on global partnership in development as well as enhances participation and local ownership which are the principles of MfDR.
References
1. IBRD/World Bank, 2004, p16. Ten steps to a results based M&E system. A handbook for development practitioners.
2. World Bank, 2003. PAD for the shared vision program of the Nile Basin Initiative
3. CIDA, 1999. “Guide to Project Performance Reporting For Canadian Partners and Executing Agencies”, May 1999. Last updated 2006.
4. OECD/DAC, 2002. OECD/DAC Glossary of Key Terms in Evaluation and Results-Based Management 2002.
5. Nile Basin Initiative, 2007. SVP MTR report 2007
6. Nile Basin Initiative, 2007. SVP Project Annual Reports, 2007
7. Lavergne, Real (2002). Results Based management and accountability for enhanced Aid effectiveness. A reference paper for CIDA officers engaged in capacity development and Program-Based approaches such as SWAps, CIDA Policy Branch, July. Pp. 16.
Table 2: NTEAP’s Revision of Outcome level indicators to match what is achievable, July 2008
RESULTS / INDICATOR(before July 2007) / REVISED INDICATORS OR DECISION MADE ON INDICATORS (July 2007) / COMMENTS OR REASONS FOR THE REVISION /
OUTCOME
Institutions Strengthened to facilitate regional cooperation / At least 2 Protocols/agreements on environmental issues formulated and submitted for approval / · Number of tentative/provisional instruments that have been developed and agreed among NB countries / These indicators was revised because
· It was observed that the establishment of a protocol was not easy; reaching an agreement and signing it was seen to be even more difficult.
· Besides, it was considered that a protocol is very difficult to handle at project level; rather it should be handled at the NBI Secretariat level
· Furthermore, it was noted that to-date, a protocol cannot be presented for approval before the NBI Corporate Framework is agreed and endorsed by all countries
· In the meantime, however, it was considered that the agreements that have been realized at technical level and with high level meeting involving Ministers of Environment show that there is good will to cooperate although this level of cooperation among NB countries can only be seen as an expression of interest because it is not legally binding.
Policy recommendations approved by at least one country
OUTCOME
Environmental education improved and public awareness enhanced / Two best practices on environmental awareness documented and streamlined in related institutions in at least 5 countries / · Number of tertiary institutions equipped to deliver environmental education (trained teachers, training materials provided, etc)
· Number of countries that have integrated environmental education in their secondary school curriculum
· Number of Civil Society organizations equipped to deliver environmental education (trained teachers, training materials provided, etc) to the general public
· Percentage of target groups knowing at least three environmental degradation/pollution factors / The indicator was changed because:
· the baseline and target values for the original indicator neither measure documentation nor streamlining as they should be according to the indicator
· Role of NTEAP is not to sustain the practices, but rather to identify and make them known to governments; thereafter, it is the governments that should have interest, commitment and obligation to disseminate the best practices for example by providing appropriate policy environment or making it part of its policy in management of environment
· It was also noted that the baseline and targeted values of the indicators did not match with what the indicator was to measure.
Source: Draft interim report on Data Collection tool consultancy, July 2008 (unpublished
[1] SI= Stakeholder Involvement; CB: Confidence Building; PI: Public Information
[2] Some of the time was lost in the recruitment of the one Lead Specialist for the project.
[3] The current (2nd) phase of the Water Policy component is expected to come to the end by 31 December 2008