DEVELOPING A COUNTRY OFFICE MEASUREMENT AND LEARNING SYSTEM FOR CARE BANGLADESH

Report of the 8th – 11th September 2008 Workshop


INTRODUCTION

Overview

This report documents the process and outputs of a four-day workshop (September 8th to 11th, 2008) held by CARE Bangladesh (CARE B)in Dhaka to discuss ongoing work for the development of a CO level Measurement and Learning system. The workshop was facilitated by the CARE Bangladesh Programme Quality Unit with support from members of the CARE USA PIKL team and was attended by some 40 members of CARE B staff. Workshop participants included the Team leaders/Project coordinators and M&E staff from 13 existing CARE B projects, the directors of the CARE B units for Economic Development, Governance and Food security, Communications, Relations and Media, Gender, Social Development, the Country Director and three Assistant Country Directors (see Annex 1 for list of participants).

Objectives

The workshop involved a series of presentations followed by group exercises by workshop participants and plenary reflection and discussion sessions. This interactive approach was designed to meet the workshop objectives of:

•  Building awareness of the organisational change processes that are ongoing in CARE at the Country Office and international levels and which relate to the work for development of a CO measurement and learning system;

•  Deepening understanding of the Theories of Change (ToC) and identifying breakthrough and incremental changes for the CARE Bangladesh Impact Statements focussing on extremely poor people in rural areas, and economically, politically and socially marginalized women;

•  Promoting dialogue to develop indicators for the ToCs, against which the progress and impact of programming by the CO will (ultimately) be measured, and around which learning will be focussed;

•  Mapping the breakthroughs and indicators identified against the Unifying framework and the Millennium Development Goals to promote deeper understanding of how the ToCs will enable CARE B to deliver its mission of achieving poverty reduction and social justice.

•  Planning the next steps for the development of the CO measurement and learning system to test the ToCs and measure the progress, process and impact of CARE B programming based on the active participation of programme staff at all levels of the CO.

Report Content & Structure

This report documents the workshop process and the key learning points of that process (see Sections II and III) as part of the CARE Bangladesh experience as a learning lab. As such it is intended as a resource for external readers interested in understanding how CARE Bangladesh has begun to work on developing a CO measurement and learning system. The report also presents the outputs of the workshop discussions together with a brief commentary on their implications for future work on the development of the CO M&L system (Annexes 3 to 5). This documentation of workshop content is intended to provide a basis for engaging the internal audience/ readership (i.e. CARE B management and programme staff) in the next steps of the process of reflection and dialogue that will be necessary to take forward the development of the CO M&L system.


II WORKSHOP PROCESS

Step 1: Building awareness of the organisational change processes in CARE and CARE Bangladesh

Process: The first day of the workshop was devoted to a series of presentations[1] focussing on:

Achieving Lasting Impacts through Program Quality. This presentation by Maliha Khan was designed to promote understanding of the rationale for the shift to a programmatic approach in CARE and the implications of that shift for Country Offices, and included discussion of the definition and characteristics of a program.

The CARE Bangladesh context. This presentation by Eliza Islam looked at where the development of a CO M&L system fits with the work being done by CARE B to develop and operationalise Impact Statements as a framework for long-term programming commitments focussed around four marginalised and vulnerable population groups[2].

Developing and Operationalising Theories of Change: This presentation by Michael Drinkwater set out the argument for developing theories of change to provide the rationale for program design aimed at achieving a long term impact goal.

The first day also included a session during which an external speaker, Tahera Yasmin, spoke about the experience of Oxfam in shifting towards a program approach and discussed the relevance of the program shift at CARE Bangladesh to the women’s movement in Bangladesh.

Recommendations for follow-up work
·  Questions raised by workshop participants about the multiple challenges of moving towards a programme approach (including concerns about feasibility, complexity, funding arrangements) require ongoing discussion, and are an important part of the learning journey for the CO.
·  The IS and constituent theories of change are evolving and will continue to do so. The workshop outputs represent an important stage in this process as a starting point for wider dialogue and further work around the identification of breakthroughs, incremental changes and indicators to measure progress towards the impact vision.
·  The question of how to align existing projects and interventions with the IS programming statement will be a focus of work across the CO in the coming months. The issue of ensuring that project staff have dedicated time to engage meaningfully with this process needs to be addressed.
Step 2: Deepening understanding of the Theories of Change (ToC)

Process: The second day of the workshop involved a series of exercises designed to generate discussion around the theories of change for the impact statements on extremely poor people in rural areas and economically, politically and socially marginalized women.

Workshop participants were asked to work in four groups to discuss the “paths” that the theories of change for these impact statements might take in terms of

i) the key breakthroughs that would need to happen to achieve the impact vision, and

ii) the important incremental changes that would need to occur for the breakthroughs to take place.

Each group was asked to work on two domains of change from each IS and to capture their ideas on cards organised in the form of a spider diagram showing the linkages between the incremental changes and breakthroughs. The focus was on identifying causal pathways for achieving theory of change.

The groups were also asked to discuss the causal relationships between the various domains of ToC, to think about the completeness of the ToC, and to make suggestions for any major additions or changes to the ToC. The outputs of the group work, which are summarised in Annex 3, were then presented and discussed in plenary.

Recommendations for follow-up work:
·  In general terms the outputs of this exercise need to be processed and further refined by the CARE Bangladesh learning groups for each impact statement to reach a clear articulation of a set of key breakthroughs for each domain of change (based on the definitions of the concepts of these concepts put forward in Annex 2). The revised breakthroughs should then be circulated more widely for comment by core groups of staff from different projects and levels within the organisation.
·  The outputs for both the women’s empowerment and rural extreme poor ToC are strongly focussed around requirements for institutional reform and policy change. As part of the follow-up work on the breakthroughs it would be useful to explore the extent to which this focus reflects experience from the field.
·  Discussions at the workshop also raised a specific question regarding the role of the organisation, namely whether CARE Bangladesh will need to take on an “activist” role to facilitate the pathways of change suggested in the impact statements, and if so, whether the organisation is ready to do so. To move forward usefully with this discussion it will first be necessary for the organisation to consider what is meant by the term “activist” and whether, given the frequent misunderstandings of that term, it is an appropriate word to describe the kind of organisation CARE Bangladesh aims to become.
Step 3: Promoting dialogue around potential indicators for the ToCs

Process: For the second group exercise workshop participants were asked to review and, where necessary, refine the breakthroughs and incremental changes they had identified, and to suggest possible indicators for their measurement. The groups were asked to assess their suggested indicators in relation to the following questions:

·  Are the indicators useful? Do they help us measure change over time?

·  Are the indicators helpful in understanding impact? Will they help us to understand whether we are contributing to the achievement of impact goal over time?

·  Are these indicators feasible? Will it be possible for information to be generated on the indicators in reasonably cost-effective ways?

·  Are these indicators aggregable? Are there a group of indicators which could be common across all four CO programs (relationship to MDIs)

The exercise outputs were discussed in plenary and are summarised with the revised breakthroughs and incremental changes in Annex 3.

Recommendations for follow-up work:

·  The indicators proposed at the workshop will need to be reviewed and refined following establishment of an organisational consensus on the key breakthroughs and incremental changes to be measured for the ToCs. The review and refinement of the various proposed indicators relating to governance issues was highlighted as an important strategic focus for this follow-up work.
·  Reflection by the PQ team on the outputs of this exercise also identified the need to define indicators at the level of the impact visions as well as at the level of the ToCs, for example, to reflect changes such as reduced hunger of extreme poor populations or improved maternal health of the most marginalised women. It was recognised that it can be difficult to differentiate indicators at these different levels, as for example in the case of the proposed indicators for the IS on women’s empowerment.
·  Overall the further development of indicators will need to build on the work of the CARE Bangladesh learning groups and will aim to ensure the coherence and consistency of indicators identified across the four impact statements.
·  The further development of indicators for the ToCs and impact visions should also take into consideration and build on the indicators in use by existing projects. It will therefore be necessary to map out where and how the indicators in use by existing projects speak to the domains of the ToCs and the impact visions, and to explore the experience of projects in using those indicators. The PQ team will need to work closely with projects on this part of the process.
·  The indicators for the IS on women’s empowerment should also be cross-checked against the indicators and findings of the 2004 SII on women’s empowerment.
Step 4: Mapping the breakthroughs and indicators identified against the Unifying framework

Process: The third group exercise involved a process of mapping the breakthroughs, incremental changes and indicators identified from the earlier group work exercises onto the three outcome categories (Human Conditions, Social Positions & the Enabling Environment) of the CARE Unifying Framework. The workshop participants were divided into three groups (one group working on each outcome category) and instructed to cluster the breakthroughs, incremental changes and indicators from the “universe” of output cards which had been generated from the earlier exercises and posted on the wall of the workshop venue in terms of their relevance to each outcome category. Where one breakthrough/ incremental change or indicator was relevant to more than one outcome category duplicate cards were prepared. The purpose of the mapping exercise was to explore how the suggestions for programming by CARE Bangladesh to move forward against the ToCs would contribute to addressing the underlying causes of poverty identified in the Unifying Framework (see outputs in Annex 4).

Suggested follow-up:
·  As outlined above it would be useful to map the indicators currently in use by existing projects in relation to the outcome categories of the Unifying Framework, as the basis for understanding where CARE Bangladesh has experience of measuring change and where there are gaps that will need to be addressed in the development of the CO M&L system.
Step 5: Mapping the breakthroughs and indicators identified against the Millennium Development Goals

Process: The last exercise of the workshop was done in a smaller group consists of team leader Shouhardo, Director Food security and Governance, Project coordinators, M&E specialists of projects and members of Program Quality Team involved in a review of the MDGs, MDG Targets and MDG Indicators to identify those most relevant for CARE Bangladesh. The relevance of the MDIs for CARE Bangladesh was assessed in terms of whether the indicator would be useful for

i) Measuring the contribution of project and programme interventions towards delivery of the impact visions for the four target populations;

ii) Testing the theories of change set out in the impact statements for those target populations; and

iii) Assessing the contribution of work by CARE Bangladesh on non-project initiatives such as advocacy and on cross cutting issues like governance or gender, knowledge generation and management.

The exercise was designed to meet the UBORA requirement for COs to identify a set of relevant MDIs to which the CO will be able to contribute significantly through its programs, projects and non project initiatives, and which could therefore provide a set of internally consistent and externally comparable indicators for the measurement of outcomes and impacts at the CO and global levels.

The MDIs identified as being relevant were then mapped against the breakthroughs and indicators that had been identified in the preceding group exercises, to check for which MDIs we already have indicators and which are the gaps where we need to develop new indicators or modify some existing indicators. The output of this work is presented in Annex 5.

Suggested follow-up:
The group also felt that we need to do some wider consultation with external stakeholders to decide on whether or not or how CARE Bangladesh's work can contribute to some of those MDIs for example MDIs related to MDG 7 & 8.
Step 6: Planning the next steps for the development of the CO measurement and learning system

Process: The final session of the workshop involved discussion by the smaller group of the next steps to be taken for the development of the CO M&L system, and the agreement of a provisional work-plan with a suggested timeline and focal persons for each step as follows: