Fourth Meeting of the UNAIDS Monitoring and Evaluation Reference Group (MERG) : Lausanne

Fourth Meeting of the UNAIDS Monitoring and Evaluation Reference Group (MERG) : Lausanne

FOURTH MEETING OF THE UNAIDS MONITORING AND EVALUATION REFERENCE GROUP (MERG)

Lausanne 15-16 November 2001

DRAFT SUMMARY REPORT

TABLE OF CONTENTS

  1. Introduction1
  1. Objectives of the Meeting1
  1. Overview of the Main Issues in HIV/AIDS Monitoring and Evaluation2
  2. Frameworks for Monitoring and Evaluation2
  1. Progress on the Five-year Evaluation of UNAIDS6
  1. Terms of Reference of the MERG7
  1. Conclusions and Recommendations7
  1. Other Issues9

Annexes

Annex IList of Participants

Annex IIProvisional Agenda

Annex IIIRevised Draft Terms of Reference of the MERG

1. Introduction

The fourth meeting of the UNAIDS Monitoring and Evaluation Reference Group (MERG) was held at the Hôtel de la Paix, Lausanne, from 15-16 November 2001. The meeting was convened with the purposes of reviewing the he recent monitoring and evaluation iinitiatives, in particular,the indicators for the United Nations General Assembly Special Session (UNGASS) on HIV/AIDSDeclaration of Commitment, the UNAIDS Monitoring and Evaluation Pplan,; and the Terms of Reference (TOR) for the reconstitution of the MERG in the sSpring of 2002, . and providing advice on emerging evaluation needs. The pParticipants (Annex I) at the Lausanne meeting included representatives of the UNAIDS Cosponsors, Centers for Disease Control and Prevention (CDC), United States Agency for International Development (USAID), Gesellschaft für Technische Zusammenarbieit (GTZ),Department for International Development (DfID), Family Health International (FHI), Futures Group International, Swedish National Audit Office and Measure Evaluation Project. .

The participants of the meeting were welcomed by Dr Jim Sherry, Director, Programme Development and Coordination Group, (PDC), UNAIDS Secretariat;,

Dr Michel Caraël, Evaluation Unit, PDC, UNAIDS Secretariat; and the MERG Chair, Professor Fred Paccaud, Director, Institut Universitaire de Médecine Sociale et Préventive (IUMSP), Lausanne.

The provisional agenda was reviewed and agreed upon (Annex II).

In opening the meeting, The members introduced themselves briefly. Dr Jim Sherry reminded emphasised that the the members of the MERG that the efforts of the MERG would be were importantin the follow-up to the Declaration of Commitment of the UNGASS. He also stressed that it is very importantthe importance of to havehaving a more harmonized approach to monitoring and evaluation (M&E) by identifying and working with stakeholders and exploiting their comparative advantages in specific areas of M&Eand by developing a unified M&EPplan to be implemented at the country level.

2. Objectives of the Meeting

The objectives of the meeting were:

  1. To technically review the recent M&Eiinitiatives including theUnited Nations General Assembly Special Session on HIV/AIDS (UNGASS) UNGASS Declaration of Commitment, , the International Partnership againstAIDS in Africa (IPAA), the World Bank Multi-Country HIV/AIDS Programme (MAP) for Africa, the UNAIDS Country Response Information System (CRIS), etc. and to provide advice on the processesfor the i implementation and follow-up at global and country levels.
  2. To review the outputs and activities of the UNAIDS M&E Plan.
  3. To provide advice on emerging evaluation needs, including research, and to assist in mobilizing technical resources for planned activities. and to
  4. To review the revised Terms of Reference (TOR)forof the MERG.

3. Overview of the Main Issues in HIV/AIDS Monitoring and Evaluation

This presentation on the main issues in HIV/AIDS M&E elaborated on the three core areas of monitoring and evaluationGlobal initiatives to improve Monitoring and Evaluation of HIV/AIDS programmes at the national level exist. The presentation broke Monitoring and Evaluation into three core areas - : Data, Structure and Capacity:.

Data:

In terms of data, there were three points that were discussed: firstly the lack of and/or irregularity of input data collection, financial, and otherwise; secondly, the lack of compilation and systemization of those data that are available leading to underutilization of data for planning purposes; and thirdly, the need to merge epidemiological data which is available in most countries and behavioural data which is becoming available in many countries into second generation surveillance to ensure that resources and expertise are used efficiently. There is a need for data to be compiled in one central system. Data is presently underutilized The prevalence of inadequate links of input data (financial, resources, national), are not routinely collected and process data is often not captured in national Monitoring and Evaluation Systems. It was noted that national databases in most countries have not yet been established.

Structure:

The existence of national M&E plans, and especially of a budget for M&E in the plan, the responsibility for implementation of M&E activities and the role of the international players formed the basis of discussion vis a vis the structure for M&E. It was noted that most countries either already have nNational sM&Etrategic plans for HIV/AIDS or are in the process of developing them. The inclusion of a budget for M&E in the planning process is crucial, as resources for conducting M&E are not part of many national programmes. It was also pointed out that, even though countries have national M&E plans or are in the process of developing them,there was confusion as to who should be responsible for M&E activities: National AIDS Councils (NAC) or the Ministries of Health.are in place in most countries. National Monitoring and Evaluation plans are being developed by new National AIDS Councils. Finally, it was stressed that The main concerns are with the M&E capacity at national level and the difficulty in identifying, who should take the lead in M&E – the new National AIDS Councils or Ministries of Health.

The inclusion of a budget for monitoring and evaluation in the planning process is necessary, as resources for conducting Monitoring and Evaluation are not part of many programmes. The issue is how the international environment can assist countries to mobilize these resources at country level. it is absolutely imperative that the international players and the international environment assist countries in mobilizing resources at the country level for monitoring and evaluation.

Capacity:

It was noted that while most countries have the capacity for Some capacity for monitoring and evaluationM&E does eat the central levelxist in countries, , districts and regions are far less equipped to carry out M&E activities. More resources should go into training district level staff and building up capacity at the district level. Opportunities do exist for training national professionals, but there is need for coordination of Monitoring and Evaluation efforts deployed (??????) by various partners at country level. The focus should be placed on training and capacity-building in monitoring and evaluation at the district level.

Finally, the session ended with highlighting the point that with the With the onset of the UNGASS Ddeclaration of Commitment, additional emphasis has been , stronger emphasis is placed on M&EMonitoring and Evaluation across international efforts which will ultimately - result ing in an increased focus on strengthening M&E capacity at the country level. The implementation of UNGASS necessitates a variety of data sources and integrated information systems. There is a need for the development of new indicators on partnerships, policies and resources.

A major challenge regarding Monitoring and Evaluation at the national level is the need for clarification and guidance of the institutional roles in monitoring and evaluation (more specifically with NAC and NACP).

4. Frameworks for Monitoring and Evaluation

During this session, the various frameworks for HIV/AIDS monitoring and evaluation were presented, including M&E activities of CDC, WHO, UNICEF, and USAID.Integration (??) (perhaps the term is not integration- it should be harmonization or synchronization or something like that) various of Monitoring and Evaluation Frameworks

Frameworks:

It would be good have one introductory sentence here of what this long section is talking about)

a)UNGASS Declaration of Commitment

A framework for evaluation

One of the goals of UNGASS is to ‘Identify problems and obstacles to achieving progress and disseminate the results of reviews/evaluation.’ The UNGASS Declaration of Commitment provides an exceptional opportunity to emphasize the need formobilize further efforts to improvedM&Eevaluation of the country responses. The goal of the Declaration will need to be achieved through actions both at the national, regional and global levels - in M&E terms this will lead to a strengthening and expansion of national and sub national programs.

To date, data on inputs and outputs are particularly weak.

A draft plan that includes a framework and indicators has been developed tto monitor the progress towards the goals set in the UNGASS Declaration of Commitment was presented to the participants. There was general agreement that the framework contained too many indicators and that further work was required to review and revise the indicators. The UNAIDS Secretariat is in the process of reviewing the framework to come up with operational indicators which will be acceptable to all Governments. To the extent possible, the process of developing the indicators will take into account the recent international efforts to strengthen and harmonize monitoring and evaluation of the HIV/AIDS epidemic by UNAIDS, bilateral agencies and other international technical organizations. . The draft is based on recent international efforts to strengthen and harmonize monitoring and evaluation of the AIDS epidemic and its consequences. UNAIDS, WHO, World Bank, European Commission, USAID and other bilateral donors, and other international technical organizations are collaborating in a global effort to strengthen monitoring and evaluation of national programs. The draft plan is based on recent efforts to strengthen monitoring and evaluation of national programs by UNAIDS, major bilateral agencies and other international technical organizations. (is that so?)

A framework (Annex III) for Monitoring and Evaluation of the goals in the UNGASS declaration of commitments has been developed. The goal of all efforts will need to be achieved through action at national, regional and global levels In Monitoring and Evaluation terms this will lead to a strengthening and expansion of national and sub national programs.

A set of indicators has been established derived from existing M&E efforts of AIDS programs. Indicators at national and international levels to measure the progress against UNGASS goals has been developed, also indicators at input, output, outcome and impact levels. 57 indicators at global and regional level were (have been) extracted. The data collection will mainly be regular reviews of progress in terms of international partnerships, policies and resources for HIV/IAIDS. 24 indicators at national level could be used based on review of national documents, laws and budgets. These reviews can be combined and incorporated into such surveys as the AIDS Programme Effort Index (API) – as this instrument could easily be expanded to include all process indicators for monitoring of the Declaration goals. At the national level, sample surveys – national household surveys, and facility surveys are required. 19 indicators already exist. A review of the UNGASS indicators with the development of a monitoring framework plan shows that UNGASS indicators belong to all these levels.

b)International Partnership againstAIDS in Africa(IPAA)

The draft Monitoring and EvaluatioM&En framework of the IPAA initiative with a list of indicators for measuring the progress of the four key objectives of the IPAA was field–tested in August 2001. The framework consists of four overall objectives, thirteen expected outputs linked to the four objectives and a cluster of quantitative and qualitative indicators to measure the progress and performance for each of the outputs. The format includes also a scoring system for rating performance for each output and source of information for each indicator. The results from the field tests revealed that much

Some of the main findings of the field test were as follows:

mMore information is available than systematized, collated and used for planning purposes. While impact level data is collected and available, outcome data is not regularly collected. Similarly, process data is scarce and usually not captured in monitoring systems even though it is critical in assessing the relevance, legitimacy and effectiveness of programmes. In general, very little input data (financial, supplies) is available. Furthermore, data collection at the district level is particularly weak and there is no regular flow of information from districts to the central level.

A large number of studies, surveys and reports have been prepared by donors, consultants, government ministries, and international research programmes in the area of HIV/AIDS over the last few years, but results have not been collated, systematized and used for monitoring and planning purposes.

District level data are weak

As part of decentralisation of the health sector, more resources and responsibility for implementation are delegated to districts. There is to date, no regular flow of information from districts to national level providing a comprehensive overview of what is happening in the area of HIV/AIDS.

2.Information and data about inputs are scarce - Financial inputs, Expenditure and Supplies.

3.Outcome data is not regularly collected

Studies have been carried out to measure knowledge, attitudes and behaviour, but not systematically and not on a national basis. Efforts are underway for establishing second-generation behavioural surveillance systems.

4.Impact data is collected and available

In the area of HIV/AIDS, data collection is strongest on impact: HIV prevalence. Antenatal surveillance systems produce impact data in countries – although the quality and reliability of data varies.

5.Process information is available indirectly

Process data information is scare and usually not captured in monitoring systems; but is critical in assessing relevance, legitimacy and effectiveness. The “AIDS Programme Efforts Index” (API) is a useful tool and source of information for most of the process indicators.

The results of the field tests-testinghave beenwere presented to a Ttechnical Wworking Ggroup for review and were revised accordinglythe IPAA M&E framework has been revised accordingly. A final version was has been prepared with practical guidelines on how to use the framework (Guidelines for using the monitoring framework -Indicators for measuring the national response to HIV/AIDS). This framework should be adapted and integrated into the UNGASS framework as part of the efforts to harmonizemonitoring of national responses to HIV/AIDS.

c)Country Response Information System (CRIS)

The idea of the CRIS is that it will compile in one database a variety of information on the situation, impact and response to the HIV/AIDS epidemic in countries.The CRIS is intended to compile in one database, a broad range of information, organized in modules on the situation, impact and response to HIV/AIDS in countries. The systemdatabasewill specifically beis in the process of being developed by the UNAIDS Secretariat, with inputs from the Country Programme Advisors (CPAs), United Nations Theme Groups on HIV/AIDS and the countries and will be field tested in twenty countries in the initial phase of its implementation in 2002.. Thus the work plan for CRIS includes the finalization of the tool, training of CPAs and other country operators, consolidation of negotiations with countries and strengthening the presence of Cosponsors in the reference networks to be established.

The sources of information for the indicators will be taken from the National AIDS Programmes, Epidemiological Fact Sheets, the IPAA monitoring and evaluation framework , the API, annual Theme group assessments, World Bank Development Indicators, synergy profiles (??) and other working documents on monitoring and evaluation TIt is anticipated that the implementation of the CRIS at country level will require innational commitment and country ownership, identification of funding sources and ensuring information technology capabilities. The UNAIDS Secretariat has proposed that 20 countries be field tested for the initial phase of implementation, with a reference network to be established and an operational calendar and management needs to be identified. The CRIS work plan for the initial phase will include finalization of the tools, conducting of training of CPAs and other country operators, the consolidation of negotiations with countries and efforts to strengthen the presence of Cosponsors in the reference network. The foreseeable risks with the CRIS are low analytical power (of whom) or what – be specific), no cooperation by international partners and under management of the system by whom (???). One of the most critical constraints of the CRIS is going to be the lack of data available in countries and the quality/authenticity of these data.

d)AIDS Programme Effort Index (API)

The AIDS Programme Effort Index (API) has been developed to measure the amount of programme efforts, both national efforts and international contributions, in the response to the HIV/AIDS epidemic. The index is designed to provide a profile that describes the national effort and the international contribution to that effort . The API was applied into 40 countries in 2000. The purpose of the API is to measure the amount of effort put into national HIV/AIDS programs by domestic institutions and by international organizations. The summary reports as well as country data profiles specific to each participating country washas been disseminated to UNAIDS representatives at country level inviting them to review the results and to comment.for review and comments. Based on the initial results of the API in those 40 countries, theA new human rights component of the APIhas beenwas revised and a new one has been developed and will be field tested by the end of 2001. In addition, in the wake of the UNGASS Declaration of Commitment, the participants agreed that the API should be re-revised to incorporate, integrate and complement the new UNGASS indicators. It is envisaged that there will be some overlap with the API and the CRIS – the CRIS will focus on quantitative measures while the API tries to capture qualitative measures and programme efforts. Efforts will be made to harmonize/integrate the API into the CRIS and the IPAA framework.

e)Multi-Country HIV/AIDS Programme (MAP) for Africa

Top View