CARE TANZANIA

MID TERM EVALUATION OF KAHAMA EDUCATION ENHANCEMENT PROJECT (KEEP)

DRAFT REPORT SUBMITTED BY

DR. NANCY DROST – CARE CANADA

DR. JOYCE NDALICHAKO – UNIVERSITY OF DAR ES SALAAM

FEBRUARY 2004
TABLE OF CONTENTS

Executive Summary

Introduction

Evaluation Methodology

Findings

Impact Result: Increased Access

Outcome Result #1: Sufficient Infrastructure

Outcome Result #2: Enhanced Teaching/Learning Process

Outcome Result #3: Increased Community Involvement

Outcome Result #4: Secondary School

Management

Conclusions

Recommendations

Annexes

  1. Terms of Reference for the Mid-Term Evaluation
  2. MTE Schedule
  3. Education Framework
  4. Update on Progress from Baseline
  5. School Statistics
  6. Stakeholders Meeting
  7. Evaluation Tools
  1. Interview Guide for District Education Officer
  2. Interview Guide for Ward Development Committee
  3. Interview Guide for Project Advisory Committee (PAC)
  4. Interview Guide for KMCL Staff – Social Responsibility Manager
  5. Interview Guide for Management (PM, Field Officers, Controller, Office Manager)
  6. Interview Guide For School Committee Members
  7. Interview Guide For Village Government Members
  8. Interview Guide for Head Teachers
  9. Guide for Teachers Small Group Discussion
  10. Classroom Observation Tool
  11. Interview Guide for Pre-School Teacher
  12. Interview Guide for COBET Coordinators and Facilitators, Learners
  13. Guide for Pupils’ Group Discussion
  14. Physical Environment and Infrastructure, KEEP log book
  15. Grounds and Buildings
  16. School Statistical Data

Executive Summary

After only two years of implementation, the overall progress of KEEP is impressive. In a short period of time, Bugarama ward has gone from a situation in which there were insufficient facilities and resources for education, to a new state in which most primary pupils have desks and textbooks. In total 34 classrooms were built and 300 desks and 12,961 books provided. 40 toilets (holes) constructed through the project provide better access, especially for girls. Furthermore, the impetus for learning caused by the existence of new facilities and equipment has almost immediately resulted in an improvement in pupil enrolment, attendance and performance. Teacher training opportunities and teacher morale has also positively affected performance. Overall examination results and the exceptional performance of individual students, has given the community a new sense of value for education.

KEEP has not only confined itself to primary education, but has taken significant risks to begin two new programmes in the ward to make provision of education for pre-primary pupils for six-year-old children and also for out-of-school youth. A total of 249 learner (112 girls and 137 boys) are enrolled in COBET while 308 children (168 girls and 140 boys) are enrolled in pre-schools. Although the Ministry of Education encourages all local governments to begin pre-primary and complementary basic education (COBET), many find the challenges too daunting to start. Thus, the eight COBET centres in Bugarama are the only ones in the entire region of Shinyanga.

One outstanding indication of the community’s new interest and commitment to education is the demand for a secondary school. Parents want their children to continue to go to school in order to have a hopeful future.

Of significance to CARE, the mid-term evaluation proceeded with full cooperation from all stakeholders, none of whom had any criticisms about KEEP and its operations. It is too early to see what lessons can be learned from the first stage of KEEP. However, there are some conclusions that can be drawn at this stage, which will assist in planning for stage two of the project’s life.

The project is clearly moving from issues of a quantitative nature (increasing numbers of facilities, materials and supplies) to greater concerns of a qualitative nature (enhancing the teaching/learning process for teachers and pupils at every level). This stage requires CARE to recruit greater expertise to meet demands of quality education and innovation from all stakeholders.

Within a short period of time, the community and school committees have gone from hopelessness and indifference to a new stage in which they desire to be empowered as actors and decision-makers within education processes. Their roles in pre-primary, primary and secondary school decisions affect the futures of their children. Good governance and active citizenry are required to ensure that resources are well directed and that all children, especially girls, have the opportunity to benefit from education. CARE’s role in managing this community process will indeed decide the success of the next phase of KEEP.

Major recommendations centre around KEEP’s ability to re-focus its priorities for the second phase and to streamline its processes for capacity building and good governance within the education system and the community. KEEP has met expectations in its first phase, but the challenges of the second phase are, in many ways, greater. The re-planning tools and recommendations stress the need for more expertise and concentration on processes that lead to evidence that proves qualitative change in the management and implementation of education.

Introduction

In August, 2001, CARE International in Tanzania signed an agreement with Kahama Mining Company Limited (KMCL) to function as its education expert and to facilitate and implement the Kahama Educational Enhancement Project (KEEP). KMCL is an affiliate of Barrick, a Canadian mining company, which operates in Bugarama ward, the catchment area of the project. Because of the project’s relationship to Barrick, CARE Canada has a role in giving management support to the project.

The overall goal of Kahama Education Enhancement Project (KEEP) is “to enable communities affecting or affected by activities of KMCL to have access to high quality relevant basic education for children and adults and increasing access to secondary education for children completing primary seven in the targeted area.” The project is implemented in collaboration with the education offices at the district and ward level, and also with school committees and other stakeholders at the community level. The project works mostly with seven primary schools, but also is responsible for other education-related activities.

The project aims to achieve four results by the end of the six-year contract:

  1. Improved school infrastructure
  2. Enhanced teaching/learning process in schools
  3. Increased community involvement in education
  4. Effective secondary school

Under each of these expected results are a number of shorter-term results or outputs that contribute to their achievement. At the inception of the project, a baseline study was accomplished by Dr. Joyce Ndalichako. This baseline study provided the basis for which to study change.

Thus the purpose of the mid-term evaluation of KEEP was to assess:

  • Whether the processes followed in implementing the project were effective and had the potential of achieving the expected overall goal, intermediate goals and expected outputs
  • Areas of weaknesses, strengths and proposed strategies to rectify and improve implementation
  • The way forward based on the implementation status of the current phase

Detailed terms of reference for the assignment are provided in Annex 1.

The mid-term evaluation of KEEP took place on-site from February 4-18. Team leaders were Dr. Nancy Drost (CARE Canada) and Dr. Joyce Ndalichako (University of Dar Es Salaam). The team of seven, made up of education and community development officials and a data analyst, were divided into three teams. The teams were engaged in evaluating progress toward outcome results related to 1) infrastructural improvement, 2) enhanced teaching/learning process and 3) increased community involvement.

To guide the investigation, sixteen different instruments (see Annex 7) were devised by the team, including interview and group discussion guides, observation guides and checklists. All stakeholders of the project were interviewed, including officials at district and ward levels. In addition to the seven schools included in the project, pre-primary facilities and COBET centres were visited.

The findings of the team are given in this report, which is organized according to four sections corresponding to the four outcomes and their output results. Additionally, there is a section on the management of the project, which includes sections of re-visioning, re-planning and staffing and capacity. Conclusions point to some of team’s impressions and forward-looking suppositions about the project. Recommendations are re-iterated at the end of the report for easy reference, even though they are also given in their relevant sections. A number of annexes provide a catalogue of all the frameworks and tools used to guide the evaluation. There are also crucial documents related to school statistics and quantitative changes evident in comparison to the baseline study. The team leaders felt that the report itself should contain useful tools, and also be a good tool in its organization and content, so that it becomes a model for future KEEP reports.

Methodology

Mid Term Evaluation Team

A team of seven individuals took part in the mid-term evaluation. The team was made up of the two team leaders (Dr. Nancy Drost and Dr. Joyce Ndalichako), representative from the Ministry of Education and Culture (Ms. Grace Rwiza), two representatives from district education office (Ms. Consolata Nyanda and Mr. Elisha Mahobhe), community development officer (Ms. Anna Mwakalinga) and a data analyst (Mr. Tobias Swai). Mr. Alex Soko, Project Manager (PM) of Tusome Vitabu Project also participated in the Team up to the instrument development stage. Team members were divided into three teams that were engaged in evaluating progress toward outcome results related to 1) infrastructural improvement, 2) enhanced teaching/learning process and 3) increased community involvement.

Basic Approach

The mid-term evaluation adopted a qualitative approach in order to gain insights of the approaches and processes used in accomplishing KEEP activities. Interviews, Focus group discussion and observations of school environment and classroom lessons formed the main techniques used for data collection. A questionnaire covering major basic statistics for schools was also distributed to all schools. The study covered all key stakeholders in the project from district to school levels. More specifically, the respondents included governments leaders in the villages surrounding seven schools participating in the project, school committee members, head teachers, ward development committee members, Project Advisory Committee (PAC), KEEP staff, KMCL, District officials, primary school teachers, pupils, COBET coordinators, facilitators and learners, and pre-primary school teachers.

Instrument Development

Each team developed instruments needed for data that they were supposed to collect. Framework provided in Annex 3 and KEEP logframe served as basis for instrument development. The draft instruments were reviewed and improved by all team members to ensure that the instruments were capable of capturing relevant information based on project results. The team devised a total of sixteen different instruments. These included:Interview Guide for District Education Officer, Interview Guide for Ward Development Committee, Interview Guide for Project Advisory Committee (PAC), Interview Guide for KMCL Staff – Social Responsibility Manager, Interview Guide for Management (PM, Field Officers, Controller, Project Administrator), Interview Guide For School Committee Members, Interview Guide For Village Government Members, Interview Guide for Head Teachers, Guide for Teachers’ Small Group Discussion, Classroom Observation Tool, Interview Guide for Pre-School Teacher, Interview Guide for COBET Coordinators and Facilitators, Learners, Guide for Pupils’ Group Discussion, Physical Environment and Infrastructure, KEEP log book, Grounds and Buildings, and School Statistical Questionnaire. The Draft instruments were presented to the KEEP PM, CARE Education Sector Coordinator for comments and relevant revisions were made.

Data Collection Process

The initial administration of the instruments served also as pilot testing of the instruments. Relevant changes were made in accordance to the observations made. All team members participated in the interview of the district education officer (DEO) and paid a courtesy call was paid to the District Executive Director. The infrastucture improvement team (Team 1) was responsible for inspection of school buildings and grounds based on the checklist developed for that purpose. They also contacted the head teacher or the teacher responsible for infrastructure to obtain information related to the number of buildings constructed and facilities provided by KEEP. The team on enhanced teaching and learning process (Team 2) conducted classroom observations, held group discussion with teachers especially those who attended INSET, interviewed head teachers and pre-primary school teachers. Team (3) was responsible for interviewing all the school committee members, village government leaders, KEEP and KMCL staff, COBET coordinators, facilitators and learners and PAC members. The schedule used for data collection is attached as Annex 2.

Data Analysis

Each team was responsible for the analysis of data collected on daily basis. Team members of the small teams were required to identify main issues and major recommendations from data collected. In the evenings, all team members met where each team made a presentation of what they found and their recommendations to KEEP. Through discussion and reflections of data collected, the team members revised and agreed on recommendations. The recommendations were revised in iterative manner following each day of data collection. After four days of data collection, data saturation was reached whereby most of the observations were similar to those found on initial days. Therefore, little changes and few additional recommendations were made on the last two days of data collection. Finally, each team was asked to draw main conclusions based on the findings for each project result. The conclusions drawn formed the main content of the presentation of the preliminary findings to the stakeholders in Kahama which was done on 14th February, 2004 (see Annex 5). Presentation to CARE Tanzania’s top management was done on 24th February 2004.

Constraints

Unavailability of reliable statistical data was the major constraint faced in the mid term evaluation. Another constraint was the mismatch between the schedule provided to team members and the schedule circulated to the respondents. In some cases there were double bookings and we had requested the field officers to assist to ensure that all the respondents meet the evaluation team as expected. However, in Igwamanoni it was not possible to meet the school committee and village government members due to confusion in time schedules.

The evaluation team was composed without careful consideration of knowledge and skills each member could contribute to the exercise. Some members in the team had no any experience in doing research or undertaking evaluation of a project. Thus the team leaders spent a lot of time in providing guidance to the inexperienced team members, some of whom had difficulties in coping with the requirements of the exercise.

IMPACT RESULT (OVERALL GOAL): Increased Access to Education

The final goal of KEEP project is to enable communities affecting or affected by activities of KMCL to have access to high quality relevant basic education for children and adults and increasing access to secondary education for children completing primary seven in the targeted areas[1]. It was expected that the all children enrolled to primary school will be able to complete standard seven after six years of project implementation. However, the mid term evaluation team was not able to establish the current percentage of pupils who complete standard seven. This was mainly due to the fact that data were not well kept in schools hence it was not possible to trace a cohort over the period of seven years of primary education.

The enrolment to standard I was regarded as an indicator of access to basic education although a proper indicator to use is the proportion of school age children who are enrolled in school. Unfortunately, the school age population data were not available in the ward. Thus, Table 1 provides number of children enrolled to standard I from 2002 to 2004.

Table 1: Standard I Enrolment in Bugarama Ward for 2002-2004

Year / Std I enrolment / % Increase
Girls / Boys / Total
2002 / 632 / 481 / 1113 / -
2003 / 696 / 690 / 1386 / 24.5
2004 / 688 / 631 / 1319 / -4.8

Table 1 shows an increment in enrolment by 24% for the year 2003 in comparison to year 2002. The 2004 enrolment shows a decrease by 4.8% of the number of children enrolled in Standard I. As already indicated, the school age population could not be established. Hence the decrease in enrolment does not reveal much about the general proportion of children enrolled. The Bugarama ward consists of mobile residents and fluctuations in enrolment and attendance is quite common.

As far as adult education is concerned, the ward has no adult education classes. Although the project intended to establish and revive the adult education classes, the needs assessment study for adult education indicated that adults considered youth to be the priority and suggested that they be provided with education before the project consider the adults. As a result of the needs assessment study, the Complementary Basic Education Project (COBET) was initiated. COBET provide education opportunity to two cohort groups, cohort I comprising of children age 8-13 and Cohort II comprising children of age 14-18. The COBET program has been well received by the stakeholders and beneficiaries. A total of 249 (36% - 112 girls and 137 boys) learners