Unit 4 Participant Involvement and Data Collection

Challenges in Evaluating Health and Development Programmes

Introduction

Welcome to the sixth, and final, unit of Monitoring and Evaluation in Health and Development Programmes.

What might some of the challenges be in evaluating how, and whether, the programme is meeting those principles? Principles are often broad statements of desires or ideas about how to operate, and to what end. To this extent, assessing them can present a number of challenges. You’ve briefly encountered some of these challenges when thinking about the indicators for your programme, for example, how qualities like “policy” environment”, “community participation”, “quality of care”, etc could be measured.

This unit the focuses on some of the challenges related to evaluating programmes in terms of their underpinning principles, such as those commonly accepted and practised in the field of the programme.

The discussion is slanted towards challenges particularly relating to Primary Health Care programmes. However, whatever the particularity of the field in which you practise, the discussion that follows will provide useful pointers to challenges you may well face. If you are not in the PHC field, as you read, you could easily, substitute the guiding principles of your field where PHC is used as the reference point.

The unit is strongly based upon the ideas and reading selections in the module by Susan Pettifer (2001) from FlindersUniversity in South Australia. Their module is called Evaluation in Primary Health Care. We gratefully acknowledge their support in this unit.

There is only one studysession in this unit.

Study Session 1: Challenges in Primary Health Care Evaluation.

In the Session, we explore some of the challenges raised by undertaking evaluation in the field of Primary Health Care, one of which is to involve stakeholder or community participation in the evaluation process.

Learning outcomes of Unit 6

• Discuss how evaluation could reflect the principles of Primary Health Care.
• Discuss selected challenges in PHC evaluation.

Unit 6 – Session 1

Challenges in Primary Health Care Evaluation

Introduction

“Primary health care refers both to an approach to health care and a range of health services. Primary health care combines efforts to:

• Restore health
• Promote health and
• Prevent illness and disability (SAHC, 1993)

[It]… may include the individual treatment or counselling services, group based support and health education, community development programs aimed at strengthening community capacity, and prevention programs targeting both risk factors and socio-environmental or structural issues. Some of these activities, particularly those aimed at reducing risk and strengthening community capacity are also referred to as health promotion.” (Pettifer, 2001: 31)

Thus far, we have explored monitoring and evaluation in general. In this session, you are invited to reflect on some of the opportunities and challenges you may face in evaluating in the context of Primary Health Care (or any given context).

We start by considering how the principles of Primary Health Care can be incorporated not only into the programme, but also into the programme evaluation; in doing so, we raise some of the potential challenges that evaluators may experience in this process. To refresh your memory about the principles of Primary Health Care, take a look at some of the readings included in the Health, Development and Primary Health Care II module.

Contents

1Learning outcomes of this session

2Readings

3Appropriate evaluation methods

4 Some challenges in Primary Health Care evaluation

5Incorporating PHC principles into the evaluation process

6Session summary

7References and further reading

Timing

This session contains two readings and three tasks. It should take you about two and a half hours to complete.

1LEARNING OUTCOMES OF THIS SESSION

• Discuss how evaluation could reflect the principles of Primary Health Care.
• Discuss selected challenges in PHC evaluation.

2READINGS

The readings for this session are listed below.

3APPROPRIATE EVALUATION METHODS

Thus far, we have approached and discussed evaluation as a generic process. In this session, we debate which approaches or methods are best suited to the field, e.g. of Primary Health Care. We explore the relationship between Primary Health Care principles and the most appropriate approach to evaluation. The range of activities included under the banner of Primary Health Care leads one to conclude that a wide range of evaluation approaches and methods will be needed to respond to such diverse contexts of practice.

We have already established that evaluation is not a value-neutral process, and that findings can be determined by the values of the evaluation design. But to what extent should the design be guided by the values and principles of Primary Health Care?

A summary by Wass (in Pettifer, 2001: 32) states that the PHC field is grounded in the following principles:

• “Accessibility
• Affordability
• Equity
• Co-operative intersectoral action
• Community participation
• The use of multiple strategies to address complex health issues
• Adopting a stronger population focus
• Adopting a stronger preventive and health promotion focus.” (Pettifer, 2001: 32)

Pettifer asks two key questions in this regard:

• “How can a program’s ability to translate these principles into action be judged or evaluated?
• To what extent should primary health care evaluation also adopt and reflect these principles?” (Pettifer, 2001: 32)

In the course of this session, we would like you to consider some of the challenges involved in evaluating a Primary Health Care project and to question to what extent the approach you are using fits the principles of the field, i.e. the second question.

4 SOME CHALLENGES IN PRIMARY HEALTH CARE EVALUATION

Planning a programme evaluation is a responsive process; you respond not only to programme issues such as improving child nutrition, but also to the principles of Primary Health Care which drive health provision in our context. A quick review of the Declaration of Alma Ata will remind you of these principles.

TASK 1 – DOES THE AIM OF YOUR EVALUATION FIT WITHIN PHC?

Think about the aim of your programme evaluation. Does it aim to clarify something that falls within the realm of PHC? Put another way, does it address one of the principles of PHC (or other relevant guiding principle structure)?

FEEDBACK

The aim of a PHC programme evaluation is likely to follow PHC principles. For example, you may be evaluating the achievement of programmes in: providing access to growth monitoring services, providing equitable numbers of skilled nursing staff in an informal settlement or raising youth awareness of HIV/AIDS. These aims fall comfortably within the principles of Primary Health Care. However, it is more challenging to ensure that the evaluation process is also appropriate in terms of PHC principles because it is to some extent more time-consuming and challenging to ensure stakeholder participation.

We will focus on three of the challenges which arise when evaluating Primary Health Care programmes: the first is evaluating community participation in a programme, another is evaluating community empowerment through an intervention and the third relates to the developmental nature of programmes.

It is primarily the developmental nature and community orientation of PHC which makes it difficult to evaluate.

4.1 Evaluating Community Participation

If the focus of your evaluation is participation, you need to be very clear about exactly what aspect of participation you are evaluating. Pettifer warns that clarifying what we mean by participation at the outset of an evaluation of this issue is critical because “participation can operate at different levels and be motivated by contradictory intentions”. (Pettifer, 2001: 44)

TASK 2 – WHAT IS MEANT BY PARTICIPATION?

Imagine that you would like to evaluate the participation of care-givers in a child nutrition, growth and weight monitoring project in an informal settlement in your area.

Define exactly what you mean by participation in this context. You could use these questions adapted from Pettifer (2001: 44) to broaden your thinking:

• Is it the health care organisation or its strategy that is being evaluated?
• What is done to support participation?
• Is the capacity of the organisation to create a supportive environment for participation being evaluated?
• Are the participants being evaluated and what have they or have they not achieved?
• Is the focus on the outcomes of participation or the processes of participation?
• Whose views shape the judgements about the worth or value of participation in the program?
• Who is, and who is not, participating?
• Whose interests are served through participation?

(Pettifer, 2001: 44)

FEEDBACK

Apart from being very specific about what is meant by participation in the context of the evaluation, judging participation must also be undertaken in the context of the community being evaluated. This suggests that lower levels of participation may be considered relatively “high” if the norm for a particular social group, e.g. young working class women, is particularly low. Participation is also always connected to “[r]ace, class, age, education level, gender, sexuality, disability and so on … [These attributes] are often closely linked to opportunities and types of participation. This complexity calls for an unpacking of questions such as what does participation mean and to whom, who is participating and whose interests are served.” (Pettifer, 2001: 43)

Rifkin (1986), who reviewed about 200 case studies from developing country programmes, concluded that there is no universally applicable definition of community participation, and that considering the political context is critical when evaluating participation. In the case of the informal settlement in Task 2, what political factors should be taken into account when considering community participation?

Pettifer draws attention to Arnstein’s ladder of participation which is still a useful model for understanding participation 30 years after it was developed. It may be helpful in distinguishing different levels of participation for evaluating the example in Task 2. Arnstein equates participation with levels of power which range from citizen control at the one extreme, to manipulation at the other. Three levels of citizen power are distinguished, each in turn broken into three levels .

Imagine that you are trying to evaluate the participation of mothers or caregivers from an informal settlement: you could grade their participation in terms of Arnstein’s ladder of participation.

However, unless the growth and weight monitoring project proactively promoted participation in the sense of partnership, or even citizen control of the programme, it is likely that participation will mean no more than “informing”, which is at the lower end of the “tokenism” scale. Can you see that Arnstein’s ladder forces one to be quite critical of the extent of participation one tends to expect within a programme? By unpacking the concept of participation, one is forced to address whether there is in fact any real possibility of participation. This is obviously influenced by whether participation would enhance the programme’s aims in the first place.

Oakley (1991), in the following reading, takes up this issue in more detail. If participation is the focus of your evaluation, you may want to consider the quantitative indicators suggested on page 248 of the reading. The discussion moves beyond the design phase to consider issues of interpretation.

READING

Oakley, P. (1991). Evaluating Participation. Projects with People: The Practice of Participation in Rural Development. Geneva: International Labour Office: 239-268.

4.2 Evaluating Developmental Programmes

At some later stage, you may be faced with evaluating a developmental programme, or one which can only be completed over time. Inherent in the concept of health development is that of community development which aims to collectively address issues identified by communities themselves. Any such initiative, for example addressing socio-economic inequities, is a long-term process. Any evaluation of its success must take account of the extended timescale of the process and whose views shape the judgement of development or progress.

An example of differing judgements of development can be seen in the training context, where illiterate adults attending literacy classes regarded their progress as substantial because some of their own aspirations - accessing a social grant, comparing prices in shops and attending classes had been fulfilled. In the eyes of those who funded the classes, however, the classes were a failure because after a year, none of the participants were able to sit for an exam.

In addition, any community development initiative should be driven by the vision of community empowerment, which is again a long-term process and difficult to measure. This is discussed further in section 4.3.

Community development programmes can also present problems because they are complex and involve a number of agencies or sectors. Such programmes present moving targets wherein “political and organisational contexts, the stakeholders and the programme goals or aims may change over time. The challenge for evaluation is how to measure performance and make judgements when the goal posts keep moving.” (Pettifer, 2001: 48-49)

Four important points about evaluation in relation to community development are made by Baum, Cooke and Murray working for the South Australian Community Health Research Unit (1998). They are:

• “Evaluation needs to be an integral part of the community development process.
• The evaluation needs to be of a scale appropriate to the time and resources available.
• Program outcomes are not easily determined in advance, so evaluation plans and methods need to be continually developed and revised.
• The difficulty of attributing impact and outcomes to community development strategies means that evaluations often focus on process.”

(Pettifer’s adaptation of Baum, Cooke and Murray, 1998: 46)

In the process of evaluating developmental programmes, the complexities of contextual change may also play a role – national and local political changes may take place, donor and stakeholder interests may also change. This process becomes in effect a process of evaluating moving targets.

4.3 Measuring Empowerment

The third Primary Health Care challenge selected for discussion is that of measuring empowerment.

Laverack and Labonte comment on the evaluation of community empowerment as follows:

“Community empowerment can be a long and slow process, and is one that, almost by definition, never fully ends. Particular outcomes in the community empowerment process may not occur until many years after the time frame of the programme has been completed. Thus, evaluation of community empowerment within the limited time frame of a programme context more appropriately assesses changes in the process rather than any particular outcome. In effect, the process becomes the outcome.” (2000: 260)

Through a case study of Hispanic immigrant men in Canada, they emphasise that community members themselves are the only ones who can determine empowerment goals and develop indicators of their own empowerment. “More importantly [they note], empowerment objectives are likely to change for programme participants as their own experiences of capacity and power increase over time.” (Laverack & Labonte, 2000: 261) You will find this article under further readings.

5INCORPORATING PHC PRINCIPLES INTO THE EVALUATION PROCESS

Designing one’s evaluation plan to be responsive to the principles of Primary Health Care is another of the challenges that we face as evaluators.

In the following reading, Kennedy (1995) discusses an evaluation design developed by the Drumchapel Community Health Project. Their goal was to evaluate the extent to which the Health for All principles were incorporated into their project. However, in the process of evaluation, this group piloted a research methodology which reflected the Health for All principles. Read the paper, considering whether, in a similar way, you could incorporate Primary Health Care principles into your evaluation process, given the time and resources.

READING

Kennedy, A. (1995). Measuring Health For All – A Feasibility Study in a Glasgow Community. Research and Change in Urban Community Health. In B. N. Sringett, J. Hotchkiss. & A. Scott-Samuel. Aldershot: Ashgate Publishing: 199-217.

TASK 3 – REFLECT ON THE INCORPORATION OF PHC PRINCIPLES INTO YOUR EVALUATION PROCESS

Having read the article by Kennedy (1995),

a)Summarise how the programme incorporated the Health for All principles into the evaluation process.

b)Present a short argument on whether you believe your evaluation process could incorporate the Primary Health Care principles in the same way, and how you would do it.

FEEDBACK

The Drumchapel Community Health Project incorporated the principles of participation, collaboration and communityempowerment into the evaluation process by using the following strategies, amongst others:

• Encouraging the participation of a range of members of the project in the evaluation process;
• Using the members’ criteria for perceptions that would be counted as an achievement;
• Using the evaluation as a training opportunity for staff and volunteers;
• Using varied means of collecting data, including a range of traditional research methods as well as theatre, video production, painting, etc;
• Being flexible in the research design and allowing key research questions to emerge through the process.

The article by Kennedy (1995) demonstrates how the Health for All approach influenced the way in which the evaluation was planned and carried out. In the conclusion of the article, Kennedy raises a concern about the divide between community development practice and research, noting the suspicion that often exists amongst practitioners, as well as the way in which the research community undervalues the knowledge of practitioners. She suggests that: “Both the research world and the world of community health … practice need to move towards more of a shared position where there are trade offs on both sides.” (Kennedy, 1995: 215)