Report Preparation Guideline

Preparation Guideline for

Outcome Evaluation Report

March 2015

Prepared by Cap-Net
/

Introduction of Outcome Evaluation

Cap- Net is a result- based management organization which needs constant feedback, learning and improving.Existing and future plans have to be regularly improved and developed based on the lessons learned from monitoring and evaluation. To ensure Cap- Net’s goal and objectives can be achieved, evaluation plays a vital role in measuring Cap-Net’s contributions to the change of water resources management.

Purpose of This Guideline

This guideline provides guidance for Cap-Net’s affiliated network toprepare their annual outcome evaluation report of capacity development activities. For Cap-Net’s monitoring and evaluation framework in detail, please refer to Cap-Net’s Monitoring, Evaluation, and Learning Plan (MELP).
This guideline highlights the definitions of terminologies of outcome evaluation, overview of the outcome evaluation process, tools for outcome evaluation, guideline for result categorization, and overview of the contents of outcome evaluation report. A reporting template has been prepared and it serves as a reference for network’s outcome evaluation report. Networks are allowed to modify the content of the report (e.g.: change the order of the section, add more details), but networks are encouraged at least submit the information that stated in the template.

Report Preparation Guideline

Overview of Cap-Net’s Monitoring, Evaluation and Reporting Framework

Report Preparation Guideline

Definition of Terminology

Capacity Development Activity-CDA

Water Resource Management-WRM

Terminology / Definition
Immediate Outcome / Immediate outcome is the most immediate things we are trying to change. Hence, for Cap-Net’s CDAs, the most immediate change should be participants’ knowledge or skills on WRM. For example, immediate outcome of IWRM Training Course shouldbe improved knowledge of the participants on IWRM subject.
Mid-term outcome / Mid-term outcome is the change happens on the person who participated in Cap-Net’s CDA, for instance, mid-term outcomeis when a person use knowledge gained through Cap-Net’s CDAs in his or her day-to-day work and cause to changes. Evaluating mid-term outcome is to prove that the knowledge taught from Cap-Net’s CDA is useful and applicable, and also to prove the competency of a person in WRM has already been improved after he or she attended the Cap-Net’s CDA.
Long-term outcome / Long-term outcome is the change happens within the institution that the participant is working for OR changes happen because of shared knowledge or interventions by the participant using the knowledge gained through the training. Hence, expected long-term outcome of Cap-Net’s CDAs aims to improvethe efficiency and effectiveness of an institution in solving water issues through the participants of the CDAs.
Example:Mr.Xruns a company that provides water loss reduction service to a city, he improve the operational efficiency of his team in solving pipe leakage issue by introducing new water loss policy. And such policy is developed base on the knowledge that he gained from a water loss management training which was organized by Cap-Net.
Impact / Mid-term outcome refers to the change of a person, long-term outcome refers to change of a water related institution, and thusimpact should refer to the change of water sector.Impact of Cap-Net’s CDAs refer to the improvement of WRM practices, improvement of water and sanitation service quality, and improved application ofWRMinstruments in addressing climate change impacts.By using the example in long-term outcome, impact of the CDAs should be the improved water loss management in the city. Please be note that the impact is usually long-term and it could be indirectly caused by Cap-Net’s CDA as big changes always involve multiple factors and actors. Thus, the networks are required to focus on evaluating the contribution not the attribution.

Evaluating Outcomes and Impacts of

Knowledge and Capacity Development Activities by Networks

Overview of the Process

Note:

Result of immediate outcome evaluation is not covered in outcome evaluation report, but it is included in the course/ activity report,which should be submitted after every CDA.Generally, Tool 2 serves as a basic tool to evaluate the immediate outcome of CDA. As the content of every course is different, networks are recommended to develop an additional tool to evaluate the immediate outcome of the CDA.
Only one outcome evaluation report should be submitted, and it should report the mid-term outcomes, long-term outcomes, and impacts of network’s CDAs in the reporting period, which are usually the CDAs that networks organized in the previous year. For example, if you are preparing outcome evaluation report in 2015, you should report the mid-term outcomes, long-term outcomes, and impacts of the CDAs that your network organized in 2014.Tool 3 will be use in evaluating the outcomes of CDA, and knowledge development and networking activities.

6. Overview of the Content of Outcome Evaluation Report

Title and cover page / -Logo and name of network, title of the report, name of the person who prepared the report,and report submission date
Report’s content page / -The content of the report should be presented in this section, together with the page number
Introduction / -Introduction of network
-Aim and objectives of outcome evaluation report
Tools and Methodologies of Outcome Evaluation / -Explain how did you evaluate outcome in details
-Describe the tools, instruments, and indicators that you used for evaluating outcomes and impacts of your capacity building activity
-Attach the questionnaire that you used for outcome evaluation as per the Tool 3 (you can choose to attach it in appendix section)
Results and Findings / -State which course have been evaluated
-Present and discuss respondents’ profile (e.g.: the response rate, respondents’ gender, respondents’ institution etc.)
-Present and discuss the results of outcome and impact evaluation
-Please provide necessary figures in this section
Outcomes of Case studies/ Training Materials/ meetings / -This sub section (as a part of results and findings) reports about any follow-up of the case studies produced in the previous year/s, and important meetings conducted in the previous year
-Tool 3 Form B, C, D of the Cap-Net MELP provides the tables for recording follow up, but network are advised to come up with innovative methods and reporting in this section.
Lesson Learned and Recommendation / -Present the lesson learned from the outcomes of capacity building activity, as well as the lesson learned from monitoring and evaluation activities.
-Provide recommendations with specific findings from previous sections
Conclusion / -Concluding remark
Appendix 1 / -Attach the questionnaire that you used for outcome and impact evaluation
Appendix 2 / -Provide the tables that you used for categorizing the answers
Appendix 3 / -Quotes from the respondents (optional)
Reference / -Please include the references that you used for preparing report

7. Revised Outcome Evaluation Questions (Tool 3 – Form A)

When should this be applied?
All participants should be followed up between 6 – 12 months after course completion to assess the outcomes and impacts of the training.
How are the results to be used?
Compile necessary information on the activities of the network against indicators and targets. Primary is used to prepare the evaluation report. Used to review and revise work plans and priorities. Used for promoting network and increased visibility. Improve management, relevance and efficiency of future activities.

The answer of question 1 and question 2 have to be reported by using the tables in a excel sheet template which can be downloaded from Cap-Net’s website. The answer of question 3 need to be categorized by the network and/or reported in short stories format.

------

Dear……………

You participated in the course “…………………..” held from “……………..” at “……………”. Could you please take a few minutes to answer these 3 questions? This will help us to improve our work.

Question 1

a)Have you used the knowledge from the course to improve your working performance?

Yes [ ] No [ ]

b)If YES, please explain how you have used the knowledge. If NO, please mention the limitations.

Aim of first question
To investigate whether the respondent has already apply the knowledge gained through Cap-Net’s CDAs in his or her day-to-day work and bring about changes.

Question 2

a)Have you shared or spread the knowledge gained from the course within or beyond of your organization?

Yes [ ] No [ ]

b)If YES, please explain how you have shared/spread and to whom you have shared/spread the knowledge. If NO, please mention the limitations.

Aim of second question
To investigate the way that the participant shared the knowledge as well as to investigate how far Cap-Net’s information has reached.

Question 3

Have you identified positive or any specific change in water resource management that contributed by the knowledge that you gained from the course? What change was that?

Aim of third question
To obtain more information about how water resources management has already been changed due to the knowledge taught by Cap-Net’s training programmes. This question is crucial to provide more information to identify the potential long-term outcome or impact of Cap-Net’s training. The answer of this question will be used to develop the short stories. Specifically, this question provides more information to network to decide which participant they should follow up to obtain more information for developing the short story.

------

Note to networks: Networks are allowed to modify the questions or add more questions. But networks are encouraged at least keeping the keywords in the questions.

8. Result categorization Guide

Below tables serves as a reference to networks for categorizing results. Besides, by referring below tables, network should think about the expected answers for the question 1 and question 2 of the Tool 3 - Form A. More tables are available in the excel sheet (result categorization guide);networks are required to fill in the tables in the excel sheet and paste the tables in the report, then Cap-Net can refer to them in case need of more information.Networks are allowed to add more categories or section in the table according to the answers they received.

Table 1: How the participants have used the knowledge (Answers for Question 1)

Category of answer / Sub category / Number of respondent / Percentage
Raising awareness on water management / -Improve knowledge of water management
-Improve sensitivity/ perception towards water management
-Improve working motivation/ attitude
-Other (please specify)
Enhancing skills / -New technical skills have been developed
-Improve skills in solving water issues
-Other (please specify)
Educational or academic purpose / -Teaching or lecturing
-Conducting capacity building activity
-Conducting water campaign
-Conducting research
-Developing or improving educational or course material
-Other (please specify)
Formulating water policy/ regulation/ strategy / -Propose or draft water policy / regulation
-Improve or initiate water management strategy
-Engage stakeholders in water management
-Other (please specify)
Supporting water related project / -Design or plan water related project
-Manage water related project
-Implement water related project
-Other (please specify)
I haven’t used the knowledge from the course
Total / 100 %

Table 2: How the participants shared/ spread the knowledge (Answers for Question 2)

Category / Respondents’ Answer / Number of Respondents / Percentage
Formal conversation / -Formal meeting
-Debate
-Group discussion
-Other (please specify)
Informal conversation / -Daily/ casual conversation
-Informal group/ personal discussion
-Other (please specify)
Capacity building activity / -Training of Trainer programme
-Course training or capacity building activity
-Engaged as facilitator of capacity building activity
-Other (please specify)
Academic or educational activity / -Academic conference
-Lecturing or teaching
-Research activity
-Water campaign
-Other (please specify)
Document sharing / -Distribution or share course material/ book/ other document from the course
-Sharing of course report
-Other (please specify)
Collaboration or partnership of project/ programme / -Knowledge spreading was occurred through partnership or collaboration with other institute or organization of a water related project/ programme
-Other (please specify)
Stakeholder involvement in water management / -Knowledge spreading was occurred through community water management programme or stakeholder involvement in water management
-Other (please specify)
Media / -Mass media (TV, radio, newspaper, etc)
-Social media (Internet, Facebook, twitter, email, blog, etc.)
-Other (please specify)
I haven’t shared the knowledge from the course
Total / 100 %

9. Guideline for Developing Short Stories of the Impact of Capacity Development Activities

The purpose of impact and long-term outcome evaluation is to provide evidence to prove that CDA is a crucial element in enhancing sustainable water management. Short story is used as a tool to present the long-term outcome and impact of Cap-Net’s CDAs.

Generally, the answers from tool 3 provide basic information to identify which short stories should be reported, especially answer of question three. Network is suggested to obtain more information through an interview by choosing respondent according to the answers from tool 3, particularly answer of question number three.

Networks should report the short stories by including below elements:

Who has taken action and what course did he/she attend?
What has he/ she done to move the country towards WRM and service provision?
How has the mode of WRM and service delivery changed/ improved?

Example

Short story from Burundi:

The Climate Change Adaptation: Strategies and Tools training workshop was conducted by the Nile Basin Capacity Building Network with the objective of strengthening the capacities of the Nile basin water professionals in dealing with climate change issues. Dr. NyandwiVenant is an assistant lecturer at the University of Burundi and a member of Friends of the Earth (Les Amis de la Terre Burundi – ATEBU), Bujumbura, Burundi. He says that ‘After the course I’ve gained new ideas and I use them in some work like consultancies and meetings as much as possible.’ He has discussed some climate issues with members of Nile Discourse Forum (NDF) especially on the need to include climate change issues in any project. The NDF is involved in building some national policies as a team. Delegates from the NDF were active where Burundians were elaborating a water policy and poverty reduction strategy framework.

Dr. Venant has trained 18 journalists in order to build their capacity on climate change. This training was held on 12th September, 2011 in Bujumbura Press House. His colleague Mr Apollinaire Niyirora conducted a session on environmental reporting techniques for journalists. After the training course, the facilitators noticed some impact of the training course. They were invited by some journalists to answer questions related to the environment after the workshop and nowadays. Media and papers in Burundi talk more about environmental issues now.

(Source: Gunawardana et al., 2013)

10. References

Cap-Net Monitoring, Evaluation and Learning Plan (MELP) (2009)
Cap-Net MELP Guidance Sheet (2014)
Cap-Net Phase III Joint Donor Review Report (2014)
Cap-Net Annual Report 2013
Cap-Net Phase III Strategic Plan
Cap-Net Phase IV Strategic Plan
Cap-Net Network Attachment MELP Reports (2011- 2013)
Gunawardana, I., Leendertse, K., Handoko, W. (2013). Monitoring outcomes and impacts of capacity development in the water sector: a Cap-Net UNDP experience. Water Policy 15.
UNDP Handbook on Monitoringand Evaluating for Results (2002)
UNDP Handbook on Planning, Monitoring and Evaluating for Development Results (2009)