Evaluation of Concern Sierra Leone S Education Programme (2006-2010)

1

Evaluation of Concern Sierra Leone’s Education Programme (2006-2010)

Final Evaluative Report

December 10, 2010

Bib’s Notes: still some highlighted bits in this report

Submitted to: Mr Manoj Kumar, Country Director, Concern Sierra Leone

Submitted by: Dr Leslie Casely-Hayford (Evaluation Team Leader) and Zakaria Sulemana (Evaluator)[1]

Table of Contents

Executive Summary

1.0Introduction

1.1The National Context

1.2Sierra Leone’s Commitment to Attaining UPE and Gender Equity

1.3The Evaluation Approach and Methodology

1.4Concern’s Education Programme Objectives and Scale in Sierra Leone

1.5. Main Education Programmatic Shifts

2.0The Outputs of Concern SL’s Education Programme

2.1Findings at the output levels of Concern’s Education Programme (2006-2010)

3.0Findings at the Outcome level

3.1Outcome 1: Access and Retention

3.2Outcome 2: Quality of Education

3.3Outcome 3: Child-friendly, Hygiene and Health

3.4Outcome 4: Management and Accountability of Schools

4.0Other Key Evaluative and Performance Criteria

4.1Targeting within Concern’s Education Programme

4.2Relevance and Appropriateness

4.3Efficiency and Effectiveness

4.4Impact of Concern’s Education Work at National and District levels

4.5Replication

4.6Sustainability

4.7Monitoring and Evaluation

4.8Advocacy

4.9Partnerships

4.10Learning and Innovations in the Concern SL Education Programme

5.0Conclusions and Key Recommendations

5.1Overall Recommendations

5.2Final Conclusions

Annex 1: Persons Interviewed during Field work

Annex 2: Final Itinerary for Evaluation Team

Annex 3: School Construction and Rehabilitation by Concern SL from 2003 to 2010

Annex 3: Costing Data for Concern Schools

Annex 3b: Cost data for Concern Schools

Annex 4: Tonkolili District Education Office Enrolment 2009/10 of Primary Teachers in the District

Annex5:

Sababu textbook distribution for Tonkolili district

Annex 6: Numbers of SMC Trainings from 2004 to 2010

Annex 7: SMCs Income Generating Activities (IGA) Project Profile

Annex 8: Breakdown of Income Generating Activities by school in 2009

Annex 10: Detailed Breakdown of Income Generating Activities by School in 2010

Annex 11: Enrolment and Performance Data Across Five Selected Concern SL Schools

Annex 12: Tonkolili District Primary and JHS School Statistics

Annex 13: Concern Primary School Summary Data Set (from 2004/05 to 2009/10

Annex 14: Number of Schools in the Tonkolili District (Primary to SSS)

Annex 15: Enrolment Growth Rates across Concern Schools Tonkolili district

List of Acronyms

CCTTChild Centred Teaching Techniques

CLOSATConcern Local Organisation Solution Assessment Tool

DESDistrict Education Statistics

DEODistrict Education Office

DHSDemographic Health Survey

EFAEducation For All

EMISEducation Monitoring Information System

FAWEFoundation for Women Educationalists

FIMFood, Income and Markets

FSUFamily Support Unit

GHIGlobal Hunger Index

HDIHuman Development Index

HTHead Teacher

IGAIncome Generating Activities

ISEDImproving School Education

JHSJunior High School

JSSJunior Secondary School

KAPKnowledge Attitudes and Practices

MDGMillennium Development Goals

MESTMinistry of Education

MICSMulti Indicator Cluster Study

MOEMinistry of Education

NPSENational Primary School Examination

ODFOpen Defecation Free Zone

OVCOrphans and Vulnerable Children

RBMResults Based Management

SASDASights and Skills Development Association

SBGRVSchool Based Gender Related Violence

SLSierra Leone

SMCSchool Management Committee

SSSSenior Secondary School

TDCTonkolili District Council

TLMTeaching/Learning Materials

UPEUniversal Primary Education

UUUntrained and unqualified

VAMVulnerability Assessment Mapping

WFPWorld Food Programme

Executive Summary

Concern Sierra Leone’s Education Programme has been working in the Education sector in Sierra Leone since 2002 providing a variety of education support services to Government and communities in some of the most deprived areas of the country. This programme evaluation was designed to assist Concern Sierra Leone assess its performance, relevance, efficiency, effectiveness and sustainability within Sierra Leone’s education sector. By looking at the issues of mainstreaming, replicability, partnership, advocacy and learning innovations, it was also designed to assist Concern SL in its forward planning to scale up geographically and strategically in Sierra Leone.

Concern Sierra Leone is striving to provide a model of community driven school improvement across 99 communities in the Tonkollili district. The model involves supporting activities to attain universal primary education among all children in the localities using:

• Community led infrastructure approaches for child friendly schools,toilet and water facilities in the community;

• improving quality education through access to more core text books and training of untrained teachers using distance education;
• creating a child friendly learning environment which are equitable and clean;
• using a child to child approach through school clubs which are addressing child rights, gender and health issues;
• strengthening child and community school governance structures which include a large level of systematic training at school base, at cluster and chiefdom levels;
• empowering communities through the usage of income generating activities; and
• establishing procedures and mechanisms to monitor and make SMC’s more accountable for child welfare at community level.

Outputs of the Programme:

Concern Sierra Leone (SL) has made a significant investment in the education sector over the last four years. On average, approximately 700,000 to 800,000 Euros were invested per year since the beginning of the programme. There has also been a significant expansion in the number of primary schools across the three chiefdoms over the last five years. For instance, during the baseline (2004), there were approximately 240 schools in the district (2004/05); currently there are over 570 primary schools in the district, of which 99 are supported by Concern.

Over the last five years the 66 schools across the three chiefdoms where Concern SL began work have achieved the following outputs related to targets between 2006 to 2010[2]:

• 15 community-led school construction projects and about 20 contractor-led school construction projects;
• 54 gender-friendly toilets constructed;
• Over 20 water points, wells and bore holes constructed - introduction of cost-effective methods of community-led well construction;
• By training 688 previously untrained and unqualified (UU) teachers through the distance education programme, Concern has decreased the number of UU teachers in the district by at least 25-30%. A large proportion of them have been accepted on the government payroll;
• Two teacher resource centres have been established (one of which is fully active), staffed and functional, providing services in Teaching/Learning Materials (TLM) development and training for in-service teachers;
• School clubs have been established in the majority of the 66 schools, with some ripple effect on neighbouring communities;
• Over 8700 SMC members directly trained by Concern SL; 860 SMC workshops held (from 2004 to 2010) with 100 participants on average; and
• Over 30 District Training of Trainers sessions held on SMC management issues, including resource mobilization, comprehensive assessment and usage of Teaching/LearningMaterials (TLMs).

Programme Outcomes and Impact

Data from a sample of the schools visited reveal modest increases in enrolment at P1 level over the five to six year period of the programme with evidence that pupil retention rates were increasing from P1 to P3. After P3, retention rates, particularly among girls, dropped significantly in all the Concern and non-Concern schools visited, with the exception of the urban town schools.

Untrained and unqualified (UU) teachers are beginning to manage their classrooms more effectively, with less corporal punishment and physical /verbal abuse, in the classroom, but work is still needed to ensure that this transformation is sustained and observed among all teachers in Concern schools.

Building gender-friendly toilets along with hygiene and health education in schools and the child-to-child methodology has helped children to learn the basics of proper hygiene and to start demanding higher standards in their own homes, the school environment, and within the community. The child-to-child methodology – creation of school clubs, and the appointment of child committees which focus on different aspects of sanitation and hygiene promotion at the school level, has been adopted in all the schools visited. The provision of water and hand washing facilities close to toilets in some of the schools is improving personal hygiene practices and behaviours among children. An outcome of these interventions has been children empowered to demand that their school environment as well as school sanitation facilities (e.g. compost fences and gender- friendly toilets) are well maintained and kept clean. The effort made by Concern SLto ensure access to potable water in most of its schools is another achievement in this area.

The strongest outcome identified in the evaluation resulted from Concern SL’s support for strengthening SMCs in the district, using several strategies including SMC training, incentive grants and ongoing school-based planning with the SMC and Head teachers. It was also the area in which key national stakeholders recognised Concern’s contribution nationwide.[3] Interviews with the SMCs and head teachers in 7 Concern rural schools suggested that the SMCs are taking the lead in developing long-term visions for their schools and solving immediate problems. There was also evidence of a growing capacity for financial management and oversight of the schools among SMCs, even in the absence of Concern staff. Factors leading to this outcome, include Concern staff’s responsiveness to levels of capacity and initiative in the SMCs, using the Incentive Grant Approach (IGA) approach. However, much more work is needed to ensure that all SMCs in the district are trained and that training costs are determined by government and donors so that investments can be sustained by agencies such as Concern.

Other Evaluative Indicators

Concern’s latest targeting efforts include an emphasis on chiefdoms where existing programmes can achieve greater integration, and more rigorous targeting methods have been used, including a full assessment of the district’s education needs; new areas of operation and selection of chiefdoms within the district have been identified through the need to develop stronger synergy with the Food, Income and Market (FIM) and health programmes in SL. There was also a scoping exercise to identify the communities with the most need within the two new chiefdoms (Barina and Konike). Vulnerability Assessment Mapping (VAM) has also been used to target some of the work of Concern in these areas.

There are signs that significant change is developing in Concern’s focal chiefdoms in relation to community empowerment; communities visited are taking greater control of their school improvement efforts. Community-led infrastructure, incentive grants tailored to the capacity needs of the community, school development planning efforts, and the stronger oversight and management capacities of the SMC through numerous training efforts, have all helped to activate and build leadership within Concern focal communities.

Concern’s Education Programme in Sierra Leone is likely to contribute to the MDGs by providing, at least, some aspects of replicable models where school improvement has taken place, and from which enough evidence for whole school development can be gathered for scale-up by communities, government and other INGOS. The change processes relate to the increased ownership of SMCs over school development, appropriate infrastructural development in relation to school buildings and school toilets, and the child empowerment approaches being used through school-based clubs, and hygiene and sanitation efforts.

Aspects of the programme that show a strong potential for sustainability and replication include: SMC training (based on the acceptance of the SMC training manual); the coupling of SMC training with IGA for greater community empowerment and self-reliance, community-led infrastructure and the ODF certification concept, and teacher resource centres (based on capacity-building in the District Education Office which will manage them). Aspects of the Concern SL work in education, which appear less likely to be sustained after the withdrawal or exit of Concern SL, include the training of UU teachers and school construction. The exit of Concern from “ready” communities may be achieved if Concern makes even more effort to assist these communities draw up strategic plans with their IGAs and subvention funds.

Overall, Concern SL’s Education Programme established a participatory monitoring and evaluation process intended to assess change, generate learning, and promote accountability. The programme has employed and continues to develop tools for better monitoring and tracking of project inputs and outputs, as well as bring about change at the school/community level in the district.

There was also evidence that Concern SL staff are using some methods of assessing outcomes at the district levels through comparing Concern-supported with non-Concern schools over time. Unfortunately outcomes data will be difficult to track if the Education Monitoring Information System (EMIS) and school census data are not available over the next year, limiting the ability of agencies, such as Concern, to track change on access, quality and management, comparing Concern-supported with non-Concern schools, using external data sets. Performance results based on the NPSE and the JHS examinations were available from the district offices but not systematically collected by Concern’s education staff.

Concern is increasingly looking to use local partners in the education programme and has assessed several potential NGO’s in areas of their programme operations. The programme has established only one partnership relationship so far with ’Pikin-to-Pikin’ to implement school health hygiene and sanitation activities through school clubs. The education programme plans to establish one more partnership with the Sight and Skills Development Association (SASDA) – a local organisation focused on adult literacy. Though the education programme is committed to working through local partner organisations, many local organisations fail to meet the assessment standards and the long process is a time consuming task for partners to achieving this level of recognition.

Although some advocacy work particularly related to School Based Gender Related Violence and the need for a national review within the education sector has been ongoing with Concern’s work at national level with its partners such as Ibis, much more work could be done to identify key areas of advocacy priority at the grassroots and bring these to the attention of Government at the national level. Stronger focus is needed by Concern to use the SMC forums as an instrument for this advocacy work at district and national levels. For instance the barriers to girls education including the lack of examination centres and the costs for girls from remote rural areas or ensuring that the fee free tuition for girls at JHS should be key advocacy areas of focus for Concern at national level. Concern schools will likely benefit if more facilitatation of newly registered JHS’s is given higher priority by District Education Office officials in the rural communities in Concern’s focal district. Much more work is needed to ensure that SMC’s are provided with training on a district wide level and that government cost for this training to ensure that investments are sustained.

Concern SL’s Education Programme is well positioned both at the national and local level to advance its advocacy agenda in areas of improving quality education through training UU teachers, ensuring that schools have a minimum set of standards making them child-friendly, and ensuring that the national level take up resourcing for SMC training on a regular basis. Concern, along with its partners, is also well positioned to take up national advocacy issues on promoting girls’ education through its work on gender-friendly school infrastructure, and School Related Gender based Violence (SRGBV).

Overall Programme Strengths and Weaknesses

The strengths of the programme have been Concern SL’s emphasis on empowering communities to plan and make changes happen within the school environment and in relation to educating their children. Concern’s focus on training SMCs, teachers and head teachers has had a ripple effectby mobilising broader groups within the community to assist in school improvement, such as engaging the women and youth in communities, along with ensuring a higher level of transparency through the emphasis on literacy classes for SMCs. The weaknesses in this connection have been that the programme has sometimes ignored important opportunities to spread the impact of its work to neighbouring communities by more effectively engaging with “feeder” communities to the Concern focused schools. Reports from the district suggest that this is something that should be considered in the coming programmeespecially when Concern is helping communities in a specific focal area of the chiefdom and not supporting other schools which are located on route to Concern focal schools.

Strengths of the programme also include the training and empowerment of unqualified and untrained (UU) teachers in the district through distance education. The distance education programme has been a success in exposing and introducing new methods and concepts of teaching and learning in the classroom to the teachers resulting, in most cases, in higher confidence levels among teachers and less verbal and physical abuse of pupils in the classroom. A process has begun to ensure that all teachers in the district programme improve their lesson planning and make use of textbooks and other TLMs within the classroom.