Process Evaluation of Stop Malaria Project’s LLIN Distribution to Pregnant Women throughAntenatal Care Services
Final Report
August, 2013
Executive Summary
This evaluation was aimed at reviewing the process of Stop Malaria Project’s LLIN distribution through ANC to identify best practices, achievements, lessons learned and areas for improvement. The evaluation applied both qualitative and quantitative approaches. Data collection was conducted at national level and in six (6) districts out of the 34 in which Stop Malaria project is distributing LLINs through ANC. Information was collected on all points within the LLIN distribution chain, including coordination, supervision, record keeping and reporting, LLIN distribution to beneficiaries, behavioural change communication; and beneficiary assessment among others. The districts selected for the evaluation were: Hoima, Kayunga, Kumi, Mityana, Sembaule and Serere districts- covering a total of 38 health facilities.
The evaluation revealed that out of the 25 facilities which had completed data on stock-outs 22 facilities (or 88%) recorded LLINs availability rates of 80% and above. Lower level facilities tended to record better rates of LLIN stocks, largely due to the relatively small number of pregnant women visiting these facilities. The frequency of LLIN delivery from the centre to districts ranged between an average of 77days to 130 days (or 2.6 to 4.3 months).The recorded frequency of LLIN delivery is comparable to that given in the guidelines (i.e. quarterly delivery). LLIN deliveries between the district and health facilities ranged between averages of 27 days to 137 days (or 1 to 4.6 months) days from the district to health facilities. The proportion of pregnant women receiving LLLIN at ANC1 varied considerably, ranging from 30% to 90%, and performance was found to better at Hospitals and HC 4 level.
The key strengths of the LLIN distribution included: (i) the transport logistical support by SMP to deliver LLINs from the central stores to the district headquarters and from the district to the HFs, without which most districts would not sustain the program. Contracting local transporters within the districts, who were very familiar with the roads and location of the facilities, made the LLIN delivery faster and less costly. (ii) A proper safety and security system for LLINs in terms of documenting LLIN delivery; and keeping stores under lock and key and a guard at night. (iii)Positive beneficiary response reflected in terms of higher ANC attendance and ANC-1 attendance in first trimester; and improved knowledge on dangers of malaria in pregnancy and benefits of sleeping under an ITN. (iv)Using the existing MoH structures to deliver nets and avoid duplication of effort for supervision and improve ownership of the program. (v) The SMP LLIN program has strengthened capacity of districts and health facilities, through the trainings, mentorship, joint support supervision; and developing tools and guidelines, to manage future LLIN distribution programs.
The evaluation revealed weaknesses in terms of: (i) failure to maintain a continuous flow of LLINs to manage LLIN stock-outs, which was by far the most, reported challenge in the LLIN distribution process.Stock-outs resulted from weak coordination across the distribution chain and non-availability of LLINs at the centre to the districtsand/or districts. Coordination between districts and health facilities in terms of monitoring LLIN stocks and reporting impeding stock-outs to district MFP was in some cases not adequate.
(ii) Poor records management, reflected in the high inconsistencies between the ANC registers and LLIN distribution record book at health facility level and delivery notes at the district level. The main reasons given were: limited orientation of ANC staff not trained at inception of program and heavy workload due to shortage of staff at most health facilities – leading to inconsistent recording of LLIN data. (iii) Forecasting and LLIN quantification system based on expected pregnancies did not adequately reflect need.There was lack of quality data from the facilities on ANC1 attendance which better reflect the need for LLINs.
One of the emerging issues from this evaluation, which is important for future health interventions, is the apparent weaknesses within the health system. There was evidence of attitude problems at different levels by viewing SMP ANC LLIN and other such donor-supported programs as an added responsibility for which an extra allowance should be paid. This attitude problem could explain the slow response by staff at some of the health facilities to act proactively and report impending stock-outs rather than waiting for the district/national level supervision team to monitor stocks. Another factor which needs improvement in future relates to limited emphasis on records management and reporting on LLIN distribution early on in the program. The introduction of specially designed LLIN distribution record books and adapting the HIMS ANC registers to capture data on LLIN distribution at facility level was done months after the start of the program. Prior to this, facilities did not have a proper way of recording and reporting data to districts and the national level. Records management had improved after the ANC registers were adapted to collect SMP ANC LLIN specific data. Recording and reporting of data is a key component in program implementation that should be given proper attention to make it possible to monitor and evaluate programsadequately.
To address the records keeping and data management challenges, there is need for updating existing HIMS forms and standardize the recording of LLIN distribution data on the ANC register at all levels. . The LLIN distribution record books were not being used consistently by majority of the health facilities visited, and the excessive workload given as the main reason for not doing so, is unlikely to be overcome soon. It is unlikely that sensitization alone could compel/motivate ANC staff to maintain both the ANC register and LLIN record books consistently. Use of existing innovative ways such as M-Track and m-Health, which have been piloted by UNICEF with maternal health services, should be considered to allow ANC staff to transfer monthly reportsas well as reporting impending stock-outs to districts through short-text messaging (SMS) services more promptly and inexpensively.
The training and sensitisation should target all staff at the facility level involved in the delivery of ANC services rather than focusing at only the mid-wife and facility in-charge as was commonly reported to have been the case.This is because the cascading training approach was not doneeffectively. Orientation of new staff should be emphasized during routine support supervision, especially on record keeping and reporting- which areas are weak currently.
Districts without data managers (Bio-statisticians) should be supported to maintain a functional data management system so that information on LLIN distribution (and other health interventions) can be captured properly and evaluated.
The current transport logistical support should be maintained given its central role in the LLIN supply chain. In addition, there should be regular flow of LLINs to districts and health facilities to counter the recurring stock-outs experienced over the evaluation period. Maintaining a constant supply of LLINs was cited as the main challenge at both national and lower levels of the supply chain and is perhaps the single most important determinant of the success of the program.
Improvements in coordination are required in terms of prompt transfer of information from health facilities to districts and from the district to the centre. The zonal SMP coordinators should work more closely with district FMPs to monitor stock-outs and share reports on distribution of LLINs at all levels.
System-wide weaknesses in the health sector in terms of poor ownership of programs, staff attitude and coordination should be addressed from the national level to lower levels in order to improve performance of health programs.
Table of Contents
Executive Summary
list OF TABLES
LIST OF ACRONYMS
1.0Introduction
1.1LLIN distribution to Women in Pregnancy
1.2The Stop Malaria Project
1.3Experience of LLIN Distribution from other countries
1.4Description of the ANC LLIN distribution mechanism by SMP
2.0 Aim and Objectives
3.0Methodology
3.1Overview of methods and sampling
3.2 Description of the data
3.2.1Quantitative data
3.2.2 Qualitative data
4.0Results
4.1 Quantitative Analysis
4.1.1 LLIN stock-outs at Health facilities
4.1.2Consistency in LLIN records management in the supply chain
4.1.3ANC attendance and LLINs distribution
4.2Qualitative Analysis
4.2.1Roles of district and health facility staff in ANC LLIN distribution.
4.2.2Planning and forecasting demand for LLINs
4.2.3Training and sensitisation
4.2.4Supervision of LLIN distribution
4.2.5Record keeping and Reporting
4.2.6 Transportation of LLINs
4.2.7Storage and security of LLINs
4.2.8ANC LLIN distribution and its impact on health service provision
4.2.9Beneficiary assessment of the LLIN distribution program
4.2.10 Behavioral change communication
4.2.11 Monitoring and Evaluation
5.0Summary: Strengths and Weakness in the LLIN Distribution Processes
5.1Strengths of the LLIN distribution process
5.2Weaknesses in the LLIN Distribution Process
5.3Other Emerging Issues
6.0Recommendations
7.0 Documents Reviewed
8.0Annexes
Annex A1: A synthesis of the benefits, drawbacks, threats and lessons learned of LLIN distribution
Annex A2: Districts and health facilities sampled for data collection
Annex A3: Frequency of LLIN delivery from SMP Central stores to Districts
Annex A4: Frequency of LLIN delivery from District to Health Facility
LIST OF TABLES
Table 1: Analysis of the quantitative data and data sources
Table 2: LLIN stock-out analysis for selected HFs (July, 11 – June 2012)
Table 3: LLIN distribution records at district and facility level (June2011 – June2012)
Table 4: ANC attendance and LLIN distribution in selected health facilities in Kumi, Kayunga and Mityana districts (July 2011 - June 2012)
Table 5: Table 4: ANC attendance and LLIN distribution in selected health facilities in Sembabule, Serere and Hoima districts (July 2011 - June 2012)
LIST OF ACRONYMS
ANCAntenatal care
DHODistrict health officer
HCHealth centre
HFHealth facility
HMISHealth management information system
ITNInsecticide Treated nets
LLINLong-lasting insecticide treated nets
MFPMalaria focal person
NGONon-governmental Organisation
NMCPNational Malaria Control program
PMIU.S President’s Malaria Initiative
SMPStop Malaria Project
UDHSUganda Demographic and Health Survey
USAIDUnited States Agency for International Development
WHOWorld Health Organisation
1.0Introduction
1.1LLIN distribution to Women in Pregnancy
Malaria remains the leading cause of morbidity and mortality in Uganda, contributing more than any other illness to the high burden of disease in the country, and undermining investment in social and economic development (NPA, 2010, MoH, 2012).
Malaria infection during pregnancy is a major public health problem, with substantial risks for themother, her foetus and the neonate (WHO, 2012). Studies have shown that approximately 25 million pregnant women in Sub-Saharan Africa are at risk of Plasmodium falciparum infection every year, and one in four women have evidence of placental infection at the time of delivery. Malaria infections during pregnancy in Africa rarely result in fever and therefore remain undetected and untreated. Results from trials on malaria prevention during pregnancy suggest that successful prevention of malaria infections reduces the risk of severe maternal anaemia by 38%, low birth weight by 43% and perinatal mortality by 27% among paucigravidae (Desai, et al 2007). Low birth weight associated with malaria in pregnancy is estimated to result in 100,000 infant deaths in Africa each year. Pregnant women infected with malaria usually have more severe symptoms and outcomes, with higher rates of miscarriage, intrauterine demise, premature delivery, low-birth-weight neonates, and neonatal death. They are also at a higher risk for severe anaemia and maternal death. Malaria can be prevented with appropriate drugs, bed nets treated with insecticide, and effective educational outreach programs (Schantz- Dunn & Nour, 2009). The World Health Organization (WHO) recommends a package of interventions for controlling malaria during pregnancy in areas with stable transmission of Plasmodium falciparum, which includes the use of insecticide treated nets (ITNs) ( WHO, 2009).Long-lasting insecticidal treated nets (LLIN)[1] and treated nets generally is a cost-effective intervention for malaria prevention during pregnancy. Distribution of long lasting Insecticide TreatedNets (LLIN) to reach universal coverage is considered as a key intervention for the prevention of malaria. Mass distributions are the best method to rapidly scale-up the coverage while continuous distribution system is essential to maintain the results achieved (Teklehamanot, et al 2007).
Stop Malaria Project (SMP) with funding from the U.S President’s Malaria Initiative (PMI), in partnership with Malaria Consortium is distributing LLINs to pregnancy women during antenatal care (ANC) in selected districts across the country. SMP’s target is to reach 85% coverage of children under five years of age and pregnant women with LLINs. The2011 Demographic Health Survey estimates show that while 59% of pregnant women reported sleeping under any net the night prior to the survey, only 46% had used a LLIN (MoH, 2012). This implies a large number of pregnant women do not have access to LLINs and interventions such as SMP LLIN ANC distribution aim to increase ownership to the Roll Back Malaria (RBM) target of 80%.
1.2The Stop Malaria Project
The Stop Malaria Project (SMP) is funded by the U.S President’s Malaria Initiative (PMI) and implemented by Johns Hopkins University Bloomberg School of Public Health Centre for Communication Programs (JHU-CCP) in partnership with Malaria Consortium (MC), the Infectious Diseases Institute (IDI), and Communication for Development Foundation Uganda (CDFU). SMP is designed to assist the Government of Uganda in reaching the PMI and Roll Back Malaria (RBM) goal of reducing malaria-related mortality by 70% by 2015 (MOP FY 2011), and subsequently contribute to the attainment of the Millennium Development Goals (MDGs). SMP’starget is to reach 85% coverage of children under five years of age and pregnant women with proven preventive and therapeutic malaria interventions, over a period of five years, including: Artemisinin-based Combination Therapy (ACTs), for treatment of uncomplicated malaria, Intermittent Preventive Treatment of malaria in pregnancy (IPTp), and Long-lasting Insecticide Treated Nets (LLINs).The project activities are implemented in close collaboration with the Ministry of Health’s National Malaria Control Program (NMCP) and district local governments. The project currently covers 34 districts in three regions: (i) Central region covering 21 districts, (ii) Hoima region covering 5 districts and (iii) Teso region covering 8 districts.
SMP has been receivingLLINs from USAID/PMI since March 2011 for distribution to pregnant women through antenatal care services (ANC). Actual distribution of LLINs to pregnant women started in May 2011, reaching full scale in June 2011 in 1,025 health facilities across the 34 project supported districts. The NMCP plans to distribute LLINs to pregnant women through the ANCunder the Global Fund (GF) round 10.SMP will carry on distributing LLINs to pregnant women through ANC with continued USAID/PMI support through the end of the project (Sept 2013). SMP would like to conduct a process evaluation of the ANC LLIN distribution to inform on-going SMP and future NMCP ANC LLIN distribution. It is expected that this evaluation will identify challenges, achievements and lessons learned during the first year of ANC LLIN distribution in Uganda, and to define best practices in the country, in terms of LLIN distribution through this channel. This evaluation is aiming to review the process of distribution; it is not intended to provide population-based outcome results.
1.3Experience of LLIN Distribution from other countries
In the past ten years, millions of LLIN have been distributed to households in Sub-Saharan Africa through catch-up (mass campaigns) and keep-up strategies (continuous distributions) Ref PMI projects in Malawi, Zambia, Madagascar, Rwanda, Senegal, etc). The RBM partners capitalized on the various experiences in different countries and identified key elements[2] for the success of LLIN distribution via ANC. The key pro-conditions for success have been documented. A well-functioning network of health facilities providing ANC services is essential and a continuous flow of LLIN must be ensured by a solid supply chain management. ANC attendance must be high in order to reach a substantial proportion of pregnant women and they must be aware of LLIN availability and benefits, especially during pregnancy.
The experiencesfrom Kenya and Malawi show that training of health workers, supervision, and demand forecasting are important aspects in the successful implementation of ANC LLIN distribution programs. In Kenya, training and supervision is conducted jointly between district health management team and PSI members of staff, dividing supervision of health facilities from supply chain follow up. This provides opportunity to emphasize on areas specific to supply chain management of commodities including LLINs as forecasting, supply reorder at each levels and record keeping. Supervision is intense and the training approach is flexible, to allow adapting to staff turnover and specific needs. Also, training and supervision should emphasize on the generation of good quality data to generate evidence on achievements and lessons learned.
Successful forecasting was attributed to needs estimation based on population figures as opposed to ANC attendance in both countries. Flexibility appeared to be a key element for success; in Malawi, additional temporary space is available at national level, enabling to quickly adapt storage capacity if needed. Also, a maximum stock quantity was set at national level, enabling each health facility to top up their net supply if judged appropriated. In Kenya, large buffer stocks of 2 to 3 months of regional needs ensure continuous flow of nets. Both countries have a dedicated fleet of vehicles delivering LLIN throughout the country all year long. In Malawi, responsiveness of supply chain is thought to be crucial; the model combines pull and push mechanisms for the resupply of LLIN. Also, to facilitate net handling at HF level, PSI in Malawi deliver nets in bags of 10 that make stock count easier to handle and to monitor damaged goods. Finally, effective internal control measures during warehousing, transportation and at health facility level ensure accountability of the system.