Process Evaluation of Stop Malaria Project’s LLIN Distribution to Pregnant Women through Antenatal Care Services
Protocol
Table of Contents
1 Introduction 3
1.1 Context 3
1.2 Description of the ANC LLIN distribution mechanism in Uganda 6
2 Aim and objectives 6
3 Methodology 6
3.1 Quantitative data 7
3.2 Qualitative data 10
4 Expected outcome of the audit 15
5 Resources 16
6 Annexes 16
6.1 Interviewer guidelines 16
6.2 Observation checklist 16
1 Introduction
1.1 Context
Distribution of long lasting Insecticide Treated Nets (LLIN) to reach universal coverage is considered as a key intervention for the prevention of malaria. Mass distributions are the best method to rapidly scale up the coverage while continuous distribution system is essential to maintain the results achieved. The Stop Malaria Project (SMP) is funded by the U.S President’s Malaria Initiative (PMI) and implemented by Johns Hopkins University Bloomberg School of Public Health Centre for Communication Programs (JHU-CCP) in partnership with Malaria Consortium (MC), the Infectious Diseases Institute (IDI), and Communication for Development Foundation Uganda (CDFU). SMP is designed to assist the Government of Uganda in reaching the PMI and Roll Back Malaria (RBM) goal of reducing malaria-related mortality by 70% by 2015 (MOP FY 2011), and subsequently contribute to the attainment of the Millennium Development Goals (MDGs). SMP’s target is to reach 85% coverage of children under five years of age and pregnant women with proven preventive and therapeutic malaria interventions, over a period of five years, including: Artemisinin-based Combination Therapy (ACTs), for treatment of uncomplicated malaria, Intermittent Preventive Treatment of malaria in pregnancy (IPTp), and Long-lasting Insecticide Treated Nets (LLINs).The project activities are implemented in close collaboration with the Ministry of Health’s National Malaria Control Program (NMCP) and district local governments. The project currently covers 34 districts in three regions: (i) Central region covering 21 districts, (ii) Hoima region covering 5 districts and (iii) Teso region covering 8 districts. The last Demographic Health Survey (2011) estimated the proportion of households owning any ITN at 60% and the proportion of pregnant women having used an ITN the previous night at 47%.
SMP has been receiving LLINs from USAID/PMI since March 2011 for distribution to pregnant women through antenatal care services (ANC). Actual distribution of LLINs to pregnant women started in May 2011, reaching full scale in June 2011 in 1,025 health facilities across the 34 project supported districts. The NMCP plans to distribute LLINs to pregnant women through the ANC under the Global Fund (GF) round 10.SMP will carry on distributing LLINs to pregnant women through ANC with continued USAID/PMI support through the end of the project (Sept 2013). SMP would like to conduct a process evaluation of the ANC LLIN distribution to inform on-going SMP and future NMCP ANC LLIN distribution. It is expected that this evaluation will identify challenges, achievements and lessons learned during the first year of ANC LLIN distribution in Uganda, and to define best practices in the country, in terms of LLIN distribution through this channel. This evaluation is aiming to review the process of distribution; it is not intended to provide population-based outcome results.
In the past ten years, millions of LLIN have been distributed to households in Sub-Saharan Africa through catch-up (mass campaigns) and keep-up strategies (continuous distributions). The Roll Back Malaria partners capitalized the various experiences in different countries and identified key elements[1] for the success of LLIN distribution via ANC. A well-functioning network of health facilities providing ANC services is essential and a continuous flow of LLIN must be ensured by a solid supply chain management. The ANC attendance must be high in order to reach a substantial proportion of pregnant women and they must be aware of LLIN availability and benefits, especially during pregnancy. A health facility survey done by SMP in December 2011, showed a good level (87%) of ANC services in the health facilities surveyed.[2]However, the same survey showed that only 49% of all surveyed facilities reported no LLIN stock out of drugs in the previous three months, indicating a supply chain management challenge in ensuring continuous flow of LLIN.
1.2 Experience of LLIN Distribution from other countries:
Experience from Kenya and Malawi is synthesised in the table below showing the benefits, drawbacks, threats and lessons learned of ANC LLIN distribution[3]:
Benefits/opportunities / Drawbacks/challengesØ Increased access to LLIN for a biologically vulnerable group
Ø Providing free LLIN to pregnant women generally promotes the ANC attendance
Ø ANC attendance generally high (over 80% for at least one visit) in many countries
Ø The use of existing structures, sometimes supported by other partners provides an opportunity for resources sharing and increase efficiency
Ø Staff members are already trained in delivering health services and managing commodities / Ø Storage space must be adequate to supply frequency
Ø Staff turnover is a challenge specific to continuous distribution mechanisms, as opposed to mass campaigns
Ø Stock disruption can lead to clients frustration and impact on ANC attendance
Ø The support to the system must be continuous
Ø Motivation of health staff already often overworked must be considered
Ø Record keeping through HMIS is often a challenge while essential to respond to donor requirements
Ø NMCP as technical division of MoH often have no line authority within health system while RH and MCH divisions are often in charge of the ANC. This could have consequence on the quality of supervision.
Ø Free LLIN distribution is likely to increase the health facility workload without increasing the resources of the centre or the salary of the staff members.
Ø Uneven funding to sustain continuity of the distribution mechanism
Ø Slow disbursement of funding
Ø LLIN stock outs
Ø Incorrect or incomplete reporting of results
Ø Decreased motivation of health staff
Ø Lack of ownership by district and sub district staff who view the LLIN distribution as NMCP activity rather than as a part of a comprehensive package of activities
Ø Insufficient motivation from ANC implementing divisions to add tasks to their activities, competition with NMCP
Ø Lack of clarity in budgeting to cover additional costs of malaria services via existing channels
Lessons learned
Ø Early planning is essential to identify gaps in resources and funding and to involve each operational level in early stages
Ø Coordination committee and technical subcommittee must be clear on their roles and responsibilities that should be outlined in Terms of References. Effective communication channels among partners must be defined
Ø Training and supervision at all levels is key to success; adult learning techniques as well as pre and post-test are good ways to ensure quality
Ø Technical assistance from other countries is a valuable way to share experience between countries
Ø Identifying a focal point within NMCP has proven to be useful
Ø Accountability must be ensured via record keeping to sustain funding
In addition to these general benefits, drawbacks and lessons learned, case studies from continuous distribution system in Kenya and Malawi[4] provide valuable insights on key elements of success.
Training and supervision
In Kenya, training and supervision is conducted jointly between district health management team and PSI members of staff, dividing supervision of health facilities from supply chain follow up. This provides opportunity to emphasize on areas specific to supply chain management of commodities including LLINs as forecasting, supply reorder at each levels and record keeping. Supervision is intense and the training approach is flexible, to allow adapting to staff turnover and specific needs. Also, training and supervision should emphasize on the generation of good quality data to generate evidence on achievements and lessons learned.
Supply chain management
Successful forecasting was attributed to needs estimation based on population figures as opposed to ANC attendance in both countries. Flexibility appeared to be a key element for success; in Malawi, additional temporary space is available at national level, enabling to quickly adapt storage capacity if needed. Also, a maximum stock quantity was set at national level, enabling each health facility to top up their net supply if judged appropriated. In Kenya, large buffer stocks of 2 to 3 months of regional needs ensure continuous flow of nets. Both countries have a dedicated fleet of vehicles delivering LLIN throughout the country all year long. In Malawi, responsiveness of supply chain is thought to be crucial; the model combines pull and push mechanisms for the resupply of LLIN. Also, to facilitate net handling at HF level, PSI in Malawi deliver nets in bags of 10 that make stock count easier to handle and to monitor damaged goods. Finally, effective internal control measures during warehousing, transportation and at health facility level ensure accountability of the system.
1.3 Description of the ANC LLIN distribution mechanism by SMP
The LLIN distribution mechanism aims to provide a continuously available source of free LLIN to help maintain high LLIN coverage levels within households in Uganda, to ensure all pregnant women are able to access an LLIN to protect themselves and their unborn child from malaria. The distribution mechanism in the 34 districts is led by SMP/Malaria Consortium following NMCP ANC LLIN distribution guidelines. Malaria Consortium is directly responsible for LLIN distribution in 26 districts, and works with Uganda Health Marketing Group (UHMG) in the remaining 8 districts. The NMCP ANC-LLIN distribution guidelines stipulate that LLIN are delivered to the health sub-district and then to the health facilities. However, under SMP, currently LLIN are procured centrally (by PMI) and delivered to SMP stores in Kampala; SMP then delivers to district stores. Districts deliver consignments of LLINs to health facilities, quantified based on previous ANC attendance figures, to each health facility offering ANC services. Consignments are delivered once every three months. Standard MOH HMIS forms are used to record LLIN stock management information. One LLIN should be given to every ANC client on her first visit (or later if she has not yet received one); her ANC card is supposed to be marked accordingly. All ANC clients are supposed to be counselled on malaria, the benefits of LLIN and how to use them. Nets given to ANC clients should be recorded in the ANC register and reported along with routine HMIS data. Support supervision of the exercise is integrated into routine support supervision activities and tools, with a specific section added to guide this. The success of this distribution mechanism thus depends on ANC attendance, good awareness of the danger of malaria and prevention methods by health workers and pregnant women as well as a solid supply chain ensuring that LLIN are continuously and freely available.
2 Aim and objectives
The evaluation aims to critically review the process of SMP’s LLIN distribution through ANC to identify best practices, achievements, lessons learned and areas of strengthening/improvement. The objectives are to:
· Assess the extent to which and how the current distribution process adheres to the expected guidelines and how this affects project results in regard to LLIN distribution
· Explore what appears to be working well and any issues/ challenges in relation to the different elements of the system(coordination, supply chain management, training, supervision, LLIN distribution to beneficiaries, data collection, communication and monitoring and evaluation)
· Identify and provide recommendations on the additional technical and logistical support needed to improve the routine distribution of LLIN through ANC
3 Methodology
3.1 Overview of study methods and sampling
The assessment will consist of a retrospective and cross-sectional process evaluation. It will cover the following levels: national, district, health sub-district and health facility. Quantitative and qualitative methods will be used. Preliminary work will include reviewing the literature (potentially a landscape review) to identify best practices, successes and lessons learned from documented experiences of LLIN continuous distribution through ANC in selected countries in Africa. The review will also include a review of quantification methods used in other ANC LLIN distributions. Existing guidelines and tools used in Uganda will be reviewed as well. Currently, the quantitative (monitoring) data from the project for all the 34 districts is only available up to district level. In order to assess the supply chain from the centre to facility level, we will collect data on LLIN distribution from a sample of districts and health facilities selected for the study. This assessment will give insights on consistency in supply and record keeping processes for the LLIN distribution. The field work will also consist of collecting qualitative data through key informant interviews, focus group discussions (FGD) and direct observations.
Six districts out of the 34 will be purposively selected to include 3 better performing and 3 weak performing based on: SMP ranking of district performance on a range of indicators (such as percentage of facilities reporting accurate data; where ANC health education talks are given, etc); and overall assessment of the SMP staff based on monitoring and support supervision reports. The other consideration for selecting study districts is regional representation for the three regions (central, eastern and western) where SMP in collaboration with UHMG is implementing the ANC-LLIN distribution. Based on these criteria, the following districts will be targeted for data collection: Kayunga, Mityana, Sembabule, Kumi, Serere and Hoima.
In each district, a sample of 3 ANC providing health facilities per HSD will be selected. The health facilities will include: the general hospital/HC-4, one HC-3 and one HC-2 providing ANC. In each district, we will consider both Public and PNFP facilities at each level of health facility (i.e. hospital/HC4, HC3s and HC2s). Table 1 below summarises the sample selection.
Table 1: Districts sampled for data collection
Region (No. of implementing districts) / District selected / SMP ranking / No. Of HSD / No. of HFs / Category of HFs per HSDCentral (21) / Mityana / Strong / 3 / 9
Kayunga / Strong / 2 / 6
Sembabule / Weak / 2 / 6
Eastern (8) / Kumi / Strong / 1 / 5 / 1 Hosp-Public, 1 Hosp (1-public, 1-PNFP; 2 HC-3,
Serere / Weak / 2 / 6
Western (5) / Hoima / Weak / 2 / 6
3.2 Description of the data
3.2.1 Quantitative data
Quantitative data will be collected from the sampled districts and selected health facilities. The data collected will relate to key process and output indicators of the project activity as already agreed with PMI. Table 2 summaries the quantitative assessment to be done and the respective data sources.