EX-ANTE EVALUATION
OF THE PROJECT “PARTNERSHIP FOR INCLUSION”
Implemented by Charitas CR and “Women and Child–Protection and Support” NGO
Evaluation Report
Duration of the Project: October 2015 – June 2017
Duration of the Evaluation: October – December, 2015
Evaluator: Radka Bzonkova, Ph.D.
Content
Executive Summary
Purpose of Evaluation
Specific Tasks to Be Answered
Methodology Employed
Original project description
Findings and Recommendations
Expected outcomes – Theory of Change
Annex 1 Methodology
Annex 2 Schedule of meetings and people interviewed
Executive Summary
This evaluation was held between October 25, 2015 and December 14, 2015 and included desk research, field mission and interviews with relevant stakeholders in the Czech Republic (3 persons) and in Moldova (36 persons).
Based on the Theory of Change approach of this evaluation the project has capacity to achieve at least three behavioral changes in the society:
1. Approach of the educators/ directors of preschools towards children with special educational needs can be changed.
2. Children with special educational needs can break their social isolation.
3. The attitude of the society towards children with special needs can be changed.
Most significant recommendations are as follow
-The budget must be increasedto keep the current number of trainings or the number of trainings must be decreased according to the estimated costs
-A steering committee of the main stakeholders should be established and should meet every half of the year
-An overall indicator could be added to the project (number of children with special needs in preschools at the beginning of the project and at the very end of the project)
Modification of programmatic activities for 2015:
-selection criteria for the gradual recruitment of trainees should be developed
-the role of social workers should be emphasized in the project
-the trainings must include official information for the mayors about their responsibility to allot a part of the budget for the inclusive education
-the trainings should provide educators with practical methods how to manage a child with special needs and the rest of the class
-the trainings must offer a practical knowledge to the educators about how to communicate with parents and how to establish a mechanism for sharing information between them
Follow-up activities and changes for the year 2016, 2017
-practical guidelines for a coordination among all relevant stakeholders reagarding the inclusive education should be developedon the local level, Czech experience for this cooperation guidelines should be used
-final beneficiaries (families) must be included in this projects and in a follow-up project
-a mobile team of specialists visiting regions could be established before the end of this project to strengthen the sustainability of the outcomes
Purpose of Evaluation
The objective of this ex-ante evaluation is to provide CCR with sufficient information on effectiveness, feasibility, sustainability and impact of proposed project activities. The evaluation must provide a set of recommendations for improvement, withdrawal or adding any component of the project including objectives, outputs, activities, stakeholders, experts, partners or beneficiaries. This recommendation has to reflect needs of the target groups. Collected data must be disaggregated by gender, type of residence (urban or rural), age of respondents, and occupation. Furthermore, the evaluation has to reflect that implementation of the project started in October 2015 and will provide adequate information on which to proceed to concluding additional agreement with donor agency.
Specific Tasks to Be Answered
Objectives and proposed implementation strategy:
-the relevance and clarity of the proposed project activities (including the measures proposed) and indicators, especially to the needs of target groups;
-internal coherence of the project (consistency of objectives and activities);
-whether the quantified target values for indicators are realistic (within the financial allocation);
-the relation of the project with other relevant interventions (properly set up mechanisms to avoid overlaps);
-sustainability of the project (including the financial and economic sustainability) and potential of its multiplication;
-the adequacy of planned measures to promote equal opportunities between men and women and to prevent any discrimination;
-recommendations for an improvement of the project design at the activity level (re-formulation of the activities, proposals for additional activities) and to prove if the target values of the indicators are realistic.
Stakeholders and target groups:
-the relevance of chosen key stakeholders and target groups;
-the degree of local ownership and participation;
-recommendations for an improvement of local ownership and participation.
Management / coordination:
-the adequacy of human resources and administrative capacity of the partner for management of the project;
-recommendations for capacity building activities of the partner
Methodology Employed
For this type of evaluation the evaluator used Result-Based Evaluation based on mapping of a proposed project casual chain using the evaluation concept of the Theory of Change, which gives us realistic idea on how the intervention can deliver results. The final product (evaluation report) is descriptive, based on a module used for an ex-ante evaluation and suggested in ToR by Charita CR.
The methodological work was divided into five phases:
1. Desk review including a targeted review of relevant program documentsfrom Charita CR, ToR and policy papers
2. Development of work plane, meetings schedule and evaluation questions
3. Qualitative and quantitative research through interviewing key informants during the field mission in Moldova
4. Revision of funds allocated in budget and creating changes acceptable for both Charitas CR and the implementation organization
5. Analysis of findings.
Detailed data collection andevaluation toolsare included in Annex 1. Schedule of meetings and interviews are included in Annex 2. The country visit took place in November 18-21, 2015. The evaluator interviewed 36 stakeholders in Chisinau and Crahuleni regions to assess their ownership and readiness for the project. The evaluator has interviewed 3 stakeholders from the Czech Republic.
Original project description
The project aims to contribute to the decrease of discrimination approach towards children with special educational needs in Moldova. The project will ensure access to inclusive education to children with special educational needs. Within the project, the training needs of teachers, administrative staff and local public authorities (LPA) representatives will be re-assessed, atraining module will be created and approved by the Ministry of Education and the Republican Psycho-Pedagogical Assistance Centre. At least 1,500 stakeholders (teachers, administrative staff, LPA representatives) will be trained in a set of skills required for proper addressing the requirements of children with special educational needs and facilitate their inclusive education. Czech experts will be engaged in elaboration of the training module to share their experience of transformation of inclusive early education sector in the Czech Republic. Through regional meetings and a national conference, the project aims at contributing to increased inter-sectoral cooperation in the sector of inclusive early education in Moldova. The project will also be dealing with dissemination of the message of importance of inclusive early education through massmedia.
Findings and Recommendations
Objectives and proposed implementation strategy:
-the relevance and clarity of the proposed project activities (including the measures proposed) and indicators, especially to the needs of target groups
The relevance and clarity of all the planned activities is high, but the amount of money allocated for the activities might not be sufficient. One of the stakeholders at the initial meeting stated that the total budget allocated for sixty trainings doesn’t correspond to the real costs of four-days-trainings.
The needs of the target groups were identified carefully by the partner organization. Based on my interviews and focus group interviews, all four target groups have deep need in training on inclusive education, because their knowledge of the inclusive education standards is either poor or with big gaps: local government representatives (mayors, social workers, nurses), members of Pedagogical Assistance Service (PAS) at district level, members of the Department of Education from the Ministry of Education on the district level and educators from the preschools.
REC:The indicators are clear, quantity-oriented. To evaluate the overall goal reached by the implementer could ascertain number of pre-school children with special educational needs in pre-school institutions which were selected for the trainings at the beginning of the project (2015) and then compare this number to the number of pre-school children with special educational needs in these pre-school institutions at the end of the project or at the end of a follow-up project (2017 or later). The difference in thenumbers of children with special needs involved in the pre-school education would definitely prove the need and measurable achievements of this project.
-internal coherence of the project (consistency of objectives and activities)
The inner logic of the project seems to be understandable to all implementing partners as well as to the target groups. The objective of the project “to ensure access to inclusive education to children of pre-school age with special educational needs” seems to be achievable and all the activities contribute to its achievement.
REC: The cooperation between the pre-schools and the social worker proved to be crucial for the inclusion of children with special ed. needs and strengthening of this cooperation should be emphasized in the project more clearly.
This finding (special importance of the collaboration among families – social workers – preschool institution) was identified during my observation and interviews with beneficiaries of this project – with the families with children with special educational needs. In one case the pre-school refused to accept a boy with special needs, but the director of the school changed her mind after an intervention of a social worker one year later.
The gradual selection of participants for the trainings is not properly specified in the project.
REC: Based on a suggestion from a stakeholder from the Ministry of Education, it would make sense to include into the training first those preschools, where some children with special needs are present already or those preschools which expressed their interest already.
The project doesn’t include cooperation with the final beneficiaries – with the families. Based on the interview with stakeholders in Chisinau and with families in the region – it is obvious that the educators in preschools are not used to communicate about the way they treat the children in preschool institution. I have also heard complains about a huge fluctuation of teachers in the educational institutions. It is necessary to improve the knowledge and overall approach to families in order to get the information circulating and to support the sustainability of the project.
REC: If the project keeps the logical selection of the trainees as described above, the follow-up of the project could be an awareness campaign focused on parents. Anyway, in a framework of this project,as one of thetraining topicsthat could be also included in could be some basic cooperation strategies for the educators and parents. The parents should know what they can expect from the preschools and the educators should communicate the problems and their approach towards the children to the parents.
In the field, the relationship and cooperation among PAS (on the local level) and all the institutions around was absolutely unclear. The representative of PAS could not describe any systematic way of getting information about children with special needs. They could neither provide me with any systematic information about their assistance to the cases.
REC: The Czech expert coming to Moldova for the ToT training could focus on the Czech experience of sharing knowledge about this gradual insitutionalization process. The Czech expert could show how to gradually share information coming from the family doctors to families, PAS, social workers, etc. Some practical guidelines could be developed, they should contain concrete steps of action, concrete addresses, contacts and responsibilities.
-whether the quantified target values for indicators are realistic (within the financial allocation)
Based on calculation made by three different local stakeholders, the budget is not adequate to the indicator of 60 realized trainings (based on the following expectations: duration of 1 training = 4 days, trainings in each region, 25 participants in each trained group, 1 trainer for 1 group) Therefor there can be achieved only smaller number of trainings/ trained participants and some activities will have to be modified. The other option is to increase the budget in a corresponding way to the costs of 60 trainings.
-the relation of the project with other relevant interventions (properly set up mechanisms to avoid overlaps)
The inclusive education is a field coordinated by the Ministry of Education. MoE plays an active role in all the projects – approves the curricula, monitors activities, motivates the participants of the trainings to invest some own money into the training (travel costs) through providing them with an accreditation. The MoE also coordinates projects focused on the inclusive education and serves as a meeting point for all the stakeholders. For this reason the inclusive education is considered to be one of the well-organized spheres of the donor involvement.
REC: There is a need to keep all the stakeholders informed about the project implementation. To improve the information exchange and share the approaches among the stakeholders it would be worthy to arrange a steering committee meeting of all relevant stakeholders every half a year. Stakeholders will be able to share the updates and the common understanding of the outputs and outcomes can be reached.
-sustainability of the project (including the financial and economic sustainability) and potential of its multiplication
The project can be sustainable under a condition that at least one more multiplication will be implemented. At least one half of all pre-school institutions should be involved in this project. Based on the filed interviews,a mobile team of specialists (speech therapist, psychologist, etc.) operating through the region was very appreciated by the interviewed educators, directors of the preschools and families.
REC: As a follow-up of the project, a mobile team of specialists operating in the regions can be established to visit regularly regions and preschools where staff was trained. This team of expert would give pieces of advice on concrete cases and situations. This mobile team would ensure monitoring, tailored approaches to specific cases and it would definitely increase the sustainability of the project.
-the adequacy of planned measures to promote equal opportunities between men and women and to prevent any discrimination
No discriminative approach toward target groups or beneficiaries was observed during the field mission and no discriminatory practice was found in the planned activities
-recommendations for an improvement of the project design at the activity level (re-formulation of the activities, proposals for additional activities) and to prove if the target values of the indicators are realistic
Based of the consultations in the country, following calculation for the minimum cost of one training was made: 25 participants, 4 days, under the condition that one group will be taught only by one trainer:
1.260 CZK / costs for 1 participant
31.500CZK / costsof 1 training
! Be aware there are isproject calculation for the following budget lines: accommodation for participants, accommodation for trainers, local transportation costs for participants, and rent of a conference room.
______
REC:Possibility 1:
In the budget approved by the Czech Agency was a budget for trainings in 2016-2017: 1.157.312 CZK. If we divide this sum by 1.260CZK (cost of the training for one participant) we get total of 918 participants to be trained.
To get at least 1.000 participants trained I suggest following changes in the budget:
year 2016: 750 participants trained = 30 trainings = 945.000 CZK
year 2017: 250 participants trained = 10 trainings = 315.000 CZK
in total year 2016+2017 = 1.000 participants trained = 40 trainings = 1.260.000CZK
______
budget for trainings approved in 2016-17 = 1.157.312
cancellation of the activity A.3.1 /line 5.13 in the budget in year 2017/ = + 84.776 (this activity can be postponed to the follow-up period of the project, after all the planned stakeholders are trained or can be incorporated in the handbooks of the training participants)
decrease of the activity A.3.1./line 5.09 in the budget in year 2016/ = + 8000
decrease of the activity A.3.1./line 5.09 in the budget in year 2017/ = + 9.912
in total: costs needed for trainings for 1.000 participants = 1.260.000
______
the composition of participants of the trainings:
35 representatives of local MoE
35 representatives of Psycho-pedagogical units
280 representatives of local government (mayor + social worker/ social assistant+ optionally a nurse/
650 representatives of educators from kindergartens (director + teacher, or a methodologist - max of 2 persons from each kindergarten)
Under these parameters it is possible to achieve following indicators: more than 300 kindergartens covered (325), coverage of all the regionswill be as planned.
REC:Possibility 2:
Other considered solution would be the budget increase in a following way:
In the budget approved by the Czech Agency was a budget for trainings in 2016-2017: 1.157.312 CZK. To get 60 trainings and 1.5000 participants trained the following amount will be needed:
60 trainings x 31.500CZK / cost for 1 training/ = 1.890.000CZK
Stakeholders and target groups
-the relevance of chosen key stakeholders and target groups
There are two levels of stakeholders: on the national level, the Ministry of Education is deeply involved in the project and the idea of inclusive education has full support of the Ministry. The partner organization has enough experience with advocacy and lobbying on the national level. The Centre of Pedagogical Assistance Service within the Ministry of Education has been established through cooperation between the Ministry and the partner organization and can play also significant role for a dissemination of all the directions and orders from the ministry to the local level.
On the local level, representatives of the local governments (mayors, social workers), preschools, local departments of the MoE and Pedagogical Assistance Service are involved. All these stakeholders should participate in the trainings, but the mayors often don’t use budget allocated for the inclusive education for this purpose and, in some cases, they are even not aware that there is this state budget available. One part of the training should cover this topic and offer solution for the distribution ofthe budget in a proper way, in accordance with the law. The Ministry of Education should emphasize this issue the mayors and should play more decisive role in influencing the mayors to use the budget for inclusive education in a proper way.