TENMET
UWEZO TANZANIA
REPORT ON THE FINDINGS OF PILOT ASSESMENT ON LEARNING OUTCOMES CARRIED OUT IN SIMANJIRO AND TEMEKE DISTRICTS
DRAFT
(Confidential)
Not to be circulated
Do not quote
July 2009
Acknowledgement
We would like to thank the following for supporting the pilot. First, our thanks go to the Uwezo Tanzania Advisory Board members. They helped at every stage of the pilot – identifying the villages, participated in training and assisted in the assessment. We would like to thank Mr. Seka, Ms. Yahya and Ms. Mlekwa of Tanzania Institute of Education, for assisting in developing the assessment tools.
We would like to thank the district authorities, both in Simanjiro and Temeke for ensuring that the assessment was done without any hinderance. Both the districts provided vehicles to assist in taking the volunteers to the respective villages/mtaa. We would like to thank all the volunteers who made time for this exercise and assisted us in collecting data from the field.
We would like to thank Hakielimu and TEN/MET for assisting in numerous ways to ensure that the pilot was successfully carried out. We would particularly like to thank ASER for all the support they have provided for the past year and in particular by allowing three of their young, knowledgeable, and energetic people; Ms. Sakshi Kapoor, Ms. Bhargavi Sundramurti and Mr. Mohit Anand; to come and assist us during the training. Their help was invaluable.
We would like to thank Hivos for providing guidance at different stages of the development of pilot exercise. Lastly, but not the least, we would like to thank William and Flora Hewlett Foundation for funding the project.
Introduction:
Uwezo East Africa project, based on the ASER model in India, started in August 2008. The purpose of Uwezo is to carryout assessment of learning outcomes among children in all the three countries of East Africa. The national assessments will be done later in the year, in September or October 2009. In preparation for the national assessments, Tanzania and Kenya carried out pilots to test out the assessment tools and the processes that will be used during the national assessments.
The pilot was carried out to provide answers to the following questions:
· Do we have people of sufficient calibre to carry out the exercise? The pilot will enable the coordinators to judge the quality of regional coordinators and the volunteers. How should we appeal to the right kind of people and how do we select them? What criteria should be used in selecting the regional, district coordinators, and volunteers?
· How much support can be expected from various CSOs in the field?
· What level and kind of cooperation can we expect from the community and from the government officials at various levels?
· Are the tests of required level of difficulty? Are the data forms simple enough for volunteers to fill?
· How do we handle variations that exist between districts, villages, and households?
· How much does it cost to do an assessment in a district?
· Was the training provided adequate? Can regional coordinators be depended on to provide the training to volunteers?
How was the pilot carried out?
Preparation for the Pilot:
· Development of the Assessment Tools: Selected people from the Tanzania Institute of Education developed the assessment tools. The children were tested in their literacy ability in Kiswahili and English, and in their numeracy ability. These tests are attached to the report.
· Recruiting volunteers: Volunteers were recruited through TEN/MET. As the number of volunteers required was small, it was decided not to recruit by advertising in newspapers. The response was encouraging. The information spread through the word of mouth. Young people, from secondary schools, from colleges and Universities applied to assist after hearing of Uwezo from their friends. The number of volunteers who applied to assist was much higher than the need. Selection of volunteers was done based on their educational background and work experience. It was not possible to interview the applicants, given the short period of time available.
· Preparation of the training manual: The training manual prepared was based on the one that ASER used in India. Some changes were made to reflect different conditions in Tanzania.
· Training of volunteers: The training of volunteers was conducted in two centres, Arusha and Dar es Salaam. In both these places, a team from India assisted the Uwezo Director. In each centre the training was done over a period of two days. The training focused on the following:
o Understanding the aim of the assessment
o Understanding the tools
o Getting familiar with the assessment procedures
· Assessment: Twenty villages were identified in Simanjiro District for data collection. In Temeke, 20 mtaa were selected. In each village or mtaa, 20 households were selected, using the every fifth household approach. In each household, all the children between the ages of 5 and 16 were tested. The testing was done by a team of two volunteers for each village. Data was also collected from one school in the respective village or mtaa.
Why Simanjiro and Temeke Districts?
Temeke and Simanjiro districts were deliberately selected to see how the assessment can be carried out in two widely different settings. Simanjiro is a large district with small population which is scattered. Majority of the people residing in Simanjiro are Maasai and Waarusha. Maasai are cattle herders while Waarusha cultivate crops. Population that is scattered over a large land area pose a special problem for provision of education services. Coupled with the fact that children are an essential part of the cattle keeping economy, makes this area particularly challenging for education provision. Simanjiro proved a challenge for the researchers as some villages covered very large areas. In one village, for example, the distance between one end to the other was 8 kilometres.
Temeke was selected to see if the assessment can be done in an urban setting. In India, ASER covers only rural areas given the difficulty of assessing children in an urban setting. In Tanzania, we wanted to find out if the local government structure of mtaa in urban area can be utilised to select a sample.
The initial decision was to have a smaller pilot in each country, done perhaps in one village and one school. However, during the national coordinators visit to India, the ASER methods and sampling expert advised Uwezo to conduct a full district pilot to learn from practical experience. The coordinators agreed to change the pilot design and do the pilot in two districts. As districts in the three East African countries vary considerably in size, population, distribution of households, socio-cultural factors, it was agreed that piloting will be done in two districts – one a “normal” district and another where population is scattered (as in pastoralist areas). Findings of the pilot phase will be key inputs in the development of the full funding proposal.
Processes used during the pilot:
Availability and Calibre of Volunteers: In order to decide where the volunteers would come from and how we should pitch our appeal to them, it was decided to take people from different walks of life as volunteers. In Tanzania the ages of selected volunteers ranged between 65 and 20. They were from church organizations, students from the Open University, student teachers from teachers colleges, secondary school graduates, members of CSOs and CBOs, and one even from the Immigration department.
CSO capacity: In order to test out CSO capacity to support Uwezo in terms of logistical and personnel support, HakiElimu, and SNV were invited to participate in the pilot exercise.
Government and Community Cooperation: Cooperation with the government is essential for such a large scale of research. For the pilot, regional and district academic officers were invited to participate in the exercise. The MOEVT supported the exercise through personnel and assisting in the transport. MOEVT also mobilised communities in the two districts for the assessment.
Training: The training was carried out in two centres Arusha and Dar es Salaam. The training was done over a two day period followed by field assessment. The training manual was also piloted to see if the participants were able to learn from and use the manuals in the field. Three ASER people from India assisted in the training.
Cost: In order to calculate the actual cost of doing assessment in a district, the costs that were budgeted for the national assessment were used. A per diem of 50,000 was paid to the regional coordinator and 30,000 to the volunteers. Even with doing a smaller sample of 20 villages (compared to 30 during the national assessment) the total cost of pilot assessment was high. It cost at least $10,000 for training and assessment per district.
Implications for the National Assessment:
Volunteers:
Overall the quality of volunteers was average. The problem arouse mainly because the training was conducted in English and the data forms were in English. Even university students felt uncomfortable using English.
· All the data collection tools will have to be translated into Kiswahili;
· One page laminated instructions will have to be prepared for different stages of data collection to ensure that these are adhered to.
· Participants will have to go through several sessions of supervised practice before they are sent to the field.
CSO Capacity:
Both HakiElimu and SNV were supportive of the initiative and helped in the pilot exercise. HakiElimu provided vehicles to support the field work both in Temeke and Simanjiro. As Uwezo had not formally been housed at TEN/MET at the time, procurement of stationery for the pilot was done using the HakiElimu system. SNV and Care International requested their staff to participate in the pilot. SNV was particularly generous in covering the costs for the training hall and meals in Dar es Salaam.
· The nature of cooperation between Uwezo and CSOs will have to be clearly spelled out.
· The ownership of the process will lie with TEN/MET – Uwezo.
· MoU will be developed to stipulate the roles of CSOs during the national assessment.
Community Government Support:
Cooperation with the Government is essential for the success of Uwezo activities. During the pilot there was high level of government support for the activity. The government ensured that people were informed of the exercise, and that proved useful during the assessment. The Ministry of Education and Vocational Training (MpEVT) was particularly helpful in the exercise. Apart from other support, the Ministry provided vehicles for the pilot in Simanjiro.
· Community and government support will be important for successful national assessment.
· Transparency of all the processes will be ensured.
· Communication strategy will have to be developed to ensure communities in areas where the assessment is done are informed of the exercise.
Training:
· The training will be conducted in Kiswahili
· The training manual will be translated into Kiswahili.
Assessment Tools:
Overall, the assessment tools were appropriate to assess the literacy and numeracy of children in Tanzania. However, the following problems were observed:
· Although there are differences in the way alphabets are pronounced in English and Kiswahili, children used the English pronunciation when reading letters in Kiswahili.
· The story level was high for most children.
· Children did not feel comfortable doing sums horizontally. Many changed the additions, subtractions, and multiplication so that they could do sums vertically.
The tools will be revised to reflect these observations.
Cost implications:
The cost of carrying out the assessment in a district, based on the experiences in Simanjiro and Temeke, was around $10,000. This is far too high and ways have to be found to cut down the cost. The following is proposed:
· Reduce the number of regional coordinators per region from 2 to 1; similarly number of district coordinators in a district be reduced to 1.
· In order to save on transport cost, volunteers as far as possible be identified within the district.
Learning Environment
It is schools where majority learn to read and write, and, gain skills and knowledge that shape their adult working life. Poor schools and poor teachers breed poor quality of education and good well-resourced schools deliver good quality education. Conditions of schools can reflect on the learning outcomes among the children in the population. The aim of collecting data from schools was to see what kind of environment existed for children to learn. Data were collected from 20 schools in Simanjiro and 18 in Temeke. In Temeke some selected streets (mtaa) shared schools between them. Table 1 shows enrolments and the Teacher Pupil Ratio in schools visited during the fieldwork.
Pupil Teacher Ratio (PTR)
Table 1: Pupil Teacher Ratio in Sampled Schools, 2009SIMANJIRO / TEMEKE
SCHOOL / ENROLMENT / TEACHERS / PTR / SCHOOL / ENROLMENT / TEACHERS / PTR
Endiamtu / 182 / 4 / 46 / Juhudi / 2858 / 44 / 65
Engonongoi / 400 / 3 / 135 / Annex / 1603 / 33 / 49
Kampuni / 509 / 6 / 85 / Keko Magurumbasi / 1263 / 43 / 29
Kiruani / 468 / 11 / 43 / Kekomwanga / 1754 / 42 / 42
Landanai / 514 / 7 / 73 / Kibada / 634 / 14 / 45
Loorng’oswani / 245 / 2 / 123 / Kibasila / 736 / 32 / 23
Losinyai / 330 / 7 / 47 / Kiburugwa / 3158 / 52 / 61
Loswaki / 489 / 9 / 54 / Kilamba / 3244 / 40 / 81
Magadini / 678 / 8 / 85 / Kingugi / 3521 / 60 / 59
MIrerani / 1207 / 25 / 48 / Kizinga / 3162 / 63 / 50
Msitu wa Tembo / 530 / 19 / 53 / Maji Matitu / 3600 / 46 / 78
Naberera / 570 / 11 / 52 / Mbagala / 3161 / 62 / 51
Namalulu / 708 / 8 / 89 / Mchikichini / 3285 / 41 / 80
Oiborkishu / 738 / 6 / 123 / Mgulani / 2200 / 43 / 51
Okutu / 245 / 4 / 61 / Mizimbini / 341 / 10 / 34
Oichoronyi / 289 / 4 / 72 / Nzasa / 3487 / 52 / 67
Oljoro No. 5 / 402 / 6 / 67 / Rangi Tatu / 1536 / 34 / 45
Songambele / 1509 / 15 / 101 / Wailesi / 1448 / 47 / 31
Tanzanite* / 77 / 5 / 15
Terrat / 476 / 11 / 43
Total / 10,566 / 162 / 65 / Total / 40,991 / 758 / 54
Source: Field data * Need to re-check data
Data clearly reflects differences between rural and urban schools. Given the scattered nature of population in Simanjiro, the schools are small. The average number of pupils per school in Simanjiro was 556 compared to 2277 in Temeke. The average PTR was higher in Simanjiro than in Temeke, but in both the districts it was higher than the PTR of 45 proposed by the government.