Final version – Handicap International’s policy on the evaluation of projects
/Handicap International’s policy on the evaluation of projects
Version validated by the PBD[1] – 04/06/2004 – P. Villeval – 9p.
This positioning document applies to all new projects once validated by the Programme Board of Directors .
Having gradually developed its evaluation practice and acquired a certain culture in this area, Handicap International now wishes to formalise its evaluation policy further.
The motivations for this and the system introduced are described briefly in this note, which seeks to reply to 6 questions:
- Why evaluate?
- Who decides to evaluate?
- When to evaluate?
- Who evaluates?
- What to evaluate?
- How to evaluate?
It also offers a bibliography on project evaluation.
This note is intended for people working for the association, its partners and for consultants wishing to reply to its calls for tender.
Why evaluate?
Evaluation, as perceived by Handicap International, responds to 4 objectives:
- To analyse an operation. An evaluation is a time for examining a project underway, re-orientating it, if necessary, and reviewing the intervention methods, indicators and/or the means implemented. It also looks at any changes in the project’s environment. The results of the evaluation help the project operators in making any decisions regarding possible changes to the project.
- To give account. An evaluation must contribute towards accounting to the association’s decision-making bodies, partners, funding bodies and possibly donors for the actions that have been carried out as a result of their support, the results obtained or the management of the resources allocated.
- To learn from experience. An evaluation should allow us to step back from the operation and analyse exactly what this operation can teach us about our identity as an Organisation of International Solidarity or about our professional practices. This analysis should lead to improvements in our practices, and even to re-examining our political, strategic or institutional positioning. It is part of the capitalisation process. The evaluation should allow us to analyse the extent to which the lessons learnt from previous experiences have been taken into consideration.
- To contribute towards social change. An evaluation is time for confronting the opinions of the different participants in an operation. This encounter and this exchange are in themselves factors of social change, which can also lead to developing new types of relations.
Who decides to evaluate?
The decision to carry out an evaluation may come from several sources :
-Handicap International: at the initiative of a member of the programme team (field programme director and/or project manager) or a representative of head office (Director of Programmes, Methods and Techniques department; Desk Officer, technical adviser);
-The funding body(ies) financing the project;
-The project partner(s).
It is essential to clearly identify who made the initial request and the reason(s) for this request.
The recommendations will be all the more appropriate and acceptable if the project actors are behind the evaluation and participate in it.
The decision to carry out an evaluation is made officially by the desk officer, in consultation with the field programme director and the technical adviser responsible for accompanying the project concerned.
When to evaluate?
An evaluation can take place during the implementation of an operation, at the end of an operation or at a later date (ex-post evaluation).
The system for evaluating the project is defined jointly by the project manager, the technical adviser and the partners during the development stage of the project.
All projects of 3 years’ duration or more must be evaluated.
In addition to the evaluations provided for during the planning stage, any participant in the operation may of course take the initiative of suggesting an evaluation at any time.
Who evaluates?
The evaluation may be run by a single person or by a team, and it may be internal (a project manager from another project, an adviser who was not involved in the planning stage, a desk officer or field programme director from another programme…), external (with no contractual connections with the association before the evaluation) or mixed.
If a project of 3 years’ duration or more is only to be evaluated once, this evaluation will be carried out by an external consultant, unless there is an explicit reason otherwise. In this case, the reason should be stated.
To avoid being judge and judged, an internal evaluator must not have been involved in the development or implementation of the operation.
An external evaluator may be chosen by mutual agreement from among the evaluators that the association has already used or as part of a call for tender, either broad or restricted. Calls for tender may thus be diffused within the association network, or more widely on its internet site and via its professional networks.
What to evaluate?
An evaluation may focus on one or other of the following:
-a programme, a project or a specific section of a project,
-a partnership,
-a dynamic, a process,
-the effects, impact of an operation,
-an approach, a concept,
-a sector of activity
-a structure.
The objective of the evaluation should be defined with the main partners concerned.
How to evaluate?
To guarantee both the quality and interest of an evaluation, different criteria need to be respected. The criteria below are of particularly note :
Impartiality & independence of the evaluators
Credibility & transparency of the evaluators
Utility of the evaluation, and the way it is run, in decision-making
Participation of the parties involved in the project so as to strengthen their capacity in the field of evaluation, but mainly to make it easier for them to appropriate the lessons learnt from this exercise
The table below summarises the main stages in an evaluation, and the imperatives to be respected. A programme correspondent may be called in at different stages, if the project to be evaluated is funded via a section.
Stages in an evaluation / Imperative to be respectedDecision to plan an evaluation / The evaluation should be provided for in the planning stage, and figure in the logical framework.
If one of the participants suggests carrying out an evaluation that was not scheduled at the outset, the decision must be validated by the Desk Officer
Drawing up of the terms of reference / The terms of reference are drawn up jointly by the projects actors and partners, the technical adviser and the Desk Officer, on the basis of the F3E form.
The partners should also participate in this stage. Their level of participation will depend essentially on the objective of the evaluation and the interest they see in it.
The association has fixed a ceiling for the daily allowance of external consultants. At the time of writing, the ceiling is fixed at 427 € / day excluding per diem.
Selection of the evaluator(s) / They are selected by the party requesting the evaluation by mutual agreement or via a call for tender, either wide or restricted, on the basis of an analysis of their proposed methodology and budget and their competence (CV).
A restricted call for tender – if this option is chosen – aimed at international consultants must at least include those consultants with a suitable profile on the database updated by the Documentary and Methodological Resources Unit.
Definition of the method / The evaluation method is defined by the evaluator(s), on the basis of the terms of reference.
The methodology note is discussed with the project actors until a compromise has been found.
Preparation of the mission / To prepare their mission, the evaluator(s) receive a briefing and are provided with all the documents recommended in the terms of reference.
On the basis of the methodology note and aspects discussed in the briefing, the project manager suggests a programme for the mission.
The project manager prepares the mission by mobilising all the necessary logistical resources, providing all the documents related to the project, making appointments with the partners and resource people concerned…
Completion of the mission in the field / The mission is carried out in accordance with the pre-established methodology note and the decisions taken during the briefing.
Writing of the report / An evaluation report must always include a summary of less than ten pages in French and English.
This summary will be translated into the national language or the vernacular.
Presentation of the report / The evaluator(s) presents the report to the programme team in the field and to the partners immediately after the evaluation.
The project manager writes a note of less than 2 pages giving his/her appreciation of the evaluation. F3E’s form is given in Annex 1 as guidance.
A meeting including at least the desk officer, technical manager for the field concerned and the technical adviser accompanying the project is then organised at head office. If the evaluator is unable to participate, the presentation should be made by the technical adviser. The director of Programmes, Methods and Techniques department and a representative of the Documentary and Methodological Resources Unit are invited to this meeting.
The final report is accepted officially by the desk officer, after consultation with the other actors concerned.
Diffusion of the report / The project manager, in consultation with the field programme director and the technical adviser, identifies the persons who should receive a copy of the report and the form it should take.
Do not forget the partners .
A paper and electronic copy of the final report must systematically be sent for archiving to head office’s documentation centre by the party that requested the evaluation.
Follow up of the recommendations / The follow up of recommendations is carried out jointly by the party that requested the evaluation, the field programme director and the technical adviser.
Valorisation of the report / The technical adviser writes a report of less than 3 pages, in French and in English, on the interest of the report for HI. This note is forwarded to the director of Programmes, Methods and Techniques department, the desk officer, the manager of the Technical Coordination Unit concerned and the manager of the Documentary and Methodological Resources unit.
The project methodology adviser reads over all of the summary notes and evaluation reports each year to identify all the strong and weak points highlighted by the evaluations and the lessons to be drawn from them. This analysis is then written into a summary document, which is discussed with the Desks and Technical Coordination units, before being validated by the director of Programmes, Methods and Techniques department. It is then presented to the Executive Director’s office and the Administration Council, and sent out to all field programme directors for information.
Impact of the evaluation / One year after the final report of the evaluation has been submitted, assuming that the association is still in the field, an analysis of the application of the report’s recommendations is carried out.
This analysis is written up into a document of less than two pages by the project manager and forwarded to the programme director, the desk officer and the technical adviser. F3E’s form is given in Annex 2 as guidance.
Bibliography on evaluation
Barbedette L. et al., September 1995, Charte de l’évaluation dans le cadre d’un travail de développement/ Charter for evaluations made during development work/ Estatuto de la Evaluacion en el marco del trabajo de desarrollo , Fondation de France, 12p.
COTA, 2002, Organiser l’évaluation d’une action de développement dans le sud [Organising the evaluation of an operation in the south ], 82p.
Feuerstein Marie-Thérèse, 1986, Partners in evaluation – Evaluating development and community programmes with participants, Macmaillan publishers, TALC, 194p.
Heeren Nicolas, 2002, Evaluation, a management tool, Handicap International, 10 p.
Heeren Nicolas & Villeval Philippe, 2002, Une pratique et une culture de l’évaluation qui permettent une conduite effective du changement [A practice and culture of evaluation for the effective management of change]
, Contribution to the seminar on 16th and 17th December 2002 entitled « Les pratiques d’évaluation des acteurs français du développement et du l’humanitaire [Evaluation practices of French development and humanitarian stakeholders ]
», MAE/HI, Lyon, France, 3p.
Graugnard Gilbert & Heeren Nicolas, 1999, Prise en compte de l’impact et construction d’indicateurs d’impact [Consideration of the impact and construction of impact indicators]
, F3E/ CIEDEL, 50p. – Document available on F3E’s internet site.
Guéneau Marie-Christine & Beaudoux Etienne, 1996, L’évaluation, un outil au service de l’action [Evaluation, a tool to assist the activity ]
F3E/ IRAM, 71p. – Document available on F3E’s internet site.
Save the Children, 2003, Toolkits – A practical guide to planning, monitoring, evaluation and impact assessment, second edition, 341 p.
Schmitz Jean-Louis et Willot Paul, octobre 1997, Méthodes et outils d'évaluation participatives - Manuel d'orientation [Methods and tools for participative evaluation, a handbook], Fondation de France, 58p.
Tricot Emmanuelle, Octobre 2001, Guide d’évaluation de la viabilité d’un service de rééducation et d’appareillage [Guide for evaluating the viability of a physical rehabilitation and orthopaedics centre], Handicap International – currently in production
______
Annex 1 –F3E guidance form « just after the evaluation»
«Although the contents of this text is free, here are some questions that should be dealt with :
- What is your opinion on how the mission went?
- What do you think of the form and the contents of the report you have received ?
- Do you feel that the report satisfies the terms of reference for the study?
- Which points do you find satisfactory?
- What do you feel is lacking from this report?
- What was the immediate impact of this report for the project under consideration, for your association?»
Annex 2 –F3E guidance form «One year on…»
«At the outset ...
1.Who expressed the need for this evaluation to be carried out : head office, field, funding body, partners, others ?
2.What were the reasons behind it?
3.Did you encounter any difficulties with this evaluation request
in writing the terms of reference?
in choosing the evaluators?
in planning the work schedule ?
in estimating the budget?
in other areas?
4.On what aspects has the F3E been of assistance to you and what do you think of this assistance at this stage in the process?
During…
5.Who monitored the progress of the evaluation? What function does this person occupy within the NGO ?
6.In the course of the evaluation, was this person called upon by the evaluator, the field team, your partners ?
At the end…
7.What is your opinion of the report that you have received :
in terms of form?
in terms of content?
8.How were the conclusions of the evaluation presented to you?
Final analysis…
9.What have been the concrete effects of this evaluation :
at head office?
in the field?
in your relations with your partners?
in defining your future projects?
10.Has this evaluation resulted in new toolsbeing produced : for monitoring, capitalisation, decision-making?
11.Has this evaluation resulted in the emergence of any new ideas or orientations for your project?
12.How have you incorporated the evaluator’s recommendations ?
13.For what reasons have you decided not to follow certain recommendations?
And now …
14.How have you valorised this evaluation (wide diffusion, meetings for debate...) ?
15.Have you learnt anything from this experience with regard to:
your conception of evaluations ?
your way of approaching and running a development operation?
16. Today, and a few words, how do you define an evaluation?
Page 1 / 9
[1]This version takes account of comments made by: Nicolas Heeren, Eric Delorme, Sonia Lokku (SARC), Barthélemy Batieno (CWARCO), Françoise Ledru.