Local and National Outcomes Of

LET’S PROVE IT

PROVIDING EVIDENCE

ON THE

LOCAL AND NATIONAL OUTCOMES OF

COMMUNITY LEARNING AND DEVELOPMENT ACTIVITIES

GUIDANCE AND FRAMEWORK

August2010

CONTENTS

Introduction 1

Terms Used 1

Purpose 3

Guide to the Framework 4

National Outcomes 5

National Indicators 5

Single Outcome Agreements 6

Developing and adapting the framework 6

Framework for Evidence on the Outcomes of CLD Activities 8

Wealthier and Fairer 8

Smarter10

Healthier12

Safer13

Greener14

Strong, resilient and supportive communities15

Implementation and Evidence 16

Sources of evidence16

Use of assessment tools18

Making the Case18

Use of logic models19

Attributing outcomes to CLD19

Future Directions20

Further Reading and Resources21

List of National Outcomes24

List of National Indicators25

INTRODUCTION

All providers of services to the public face the common challenge of showing how their activities help to achieve desirable social and economic outcomes, particularly at a time of intense pressure on resources. They need to be able to say more about the difference that they make in the lives of individuals, groups and communities.

Organisations that deliver Community Learning and Development (CLD), in both local government and the voluntary sector, are under special pressure to demonstrate their contribution to shared outcomes. Though CLD provision has a clear statutory basis, there are no mandatory requirements indicating a required level of provision.

Policies on what outcomes we should pursue are enshrined at national level in the National Performance Framework and at local level in the Single Outcome Agreements agreed by Community Planning Partnerships.

HMIE Inspectors see evidence of CLD making a range of impacts across all the Strategic Objectives in the National Performance Framework. They believe that it has a much wider, cross-cutting impact than many other services do. But this impact is often not fully understood by practitioners, or communicated to and understood by others.

When decisions are made, it is often what can be measured that matters. Practitioners need to capture hardevidenceof the outcomes of their work and show how this relates logically to helping to achieve shared outcomes.

CLD practitioners are very familiar with monitoring, evaluation and inspection of their activities. But HMIE inspection reports find that almost 80% of CLD partnerships are less than ‘good’ in ‘improving performance’ and ‘improving services’ (even thougharound 80% are ‘good’ or better in their impact on young people and adults).

This guidance and framework has been prepared by Community Learning and Development Managers Scotland, in consultation with YouthLink Scotland, Learning Link Scotland, Learning and Teaching Scotland, HMIE, and other partners.

It is intended as a contribution to helping CLD practitioners to provide evidence that will help them to improve the outcomes from their services and to gain wider recognition and understanding for these.

TERMS USED

Intermediate and End Outcomes Within a Logic Model,End Outcomes are the outcomes that an intervention ultimately seeks to achieve, and Intermediate Outcomes are necessary steps that allow the intervention to achieve the End Outcomes, as explained by the model. Intermediate Outcomes may also be seen as ‘process outcomes’[1].In the ‘Delivering Change’ report on the outcomes from CLD, a clear distinction is drawn betweenmeasurement of the competences, skills and confidence of individual community leaders and theirorganisations and measurement of the impact that such individuals and organisations have in tacklingspecific community problems or achieving community aspirations. The former are defined asIntermediate Outcomes and the latter as End Outcomes.

Local IndicatorsA measure of performance against local outcomes set by community planning partners, in addition to any National Indicators that they may use. COSLA and SOLACE have agreed a set of local indicators, which isreferred to as a Menu of Local Indicators. A Community Planning Partnership is able to includeindicators from this menu in their Single Outcome Agreement where relevant..

Logic Model A logic model sets out how an intervention is understood or intended to produce particular results. It analyses work into steps that show the logical relationships between inputs, activities, outputs, and outcomes.

National Indicators TheNational Outcomes are backed by 49 National Indicators and Targets (see list, p25)defined by the Scottish Government, providing a measure of performance against National Outcomes.

National Outcomes The Scottish Government’s National Performance Framework specifies a set of 15National Outcomes (see list, p24)which will be sought by the Government, and which also form the basis for local Single Outcome Agreements, along with additional local outcomes.

Outcomes The changes or consequences that happen as a result of the activities of an organisation or partnership of organisations. Outcomes are normally what an organisation is trying to achieve, though some outcomes are unintended.

Single Outcome Agreements As part of the concordat agreed between the Scottish Government and local authorities, each Community Planning Partnership must reach a Single Outcome Agreement (SOA) with the Scottish Government, covering all local government services in the area and some of the responsibilities of other Community Planning partners. These agreements are based upon National Outcomes and Indicators, plus a range of Local Outcomes and Indicators to take account of local priorities.

Strategic Objectives The Scottish Government has aligned its work around five Strategic Objectives (see p.5) that underpin its Purpose and the National Performance Framework.

PURPOSE

Our purpose in this document is to provide:

a framework to support people in planning, improving and providing evidence of the outcomes from their CLD work
guidance on how to interpret and use the framework.

We concentrate on the specific aim of relating our evidence more closely to the National Performance Framework. Since local Single Outcome Agreements referextensively to that Framework, this guidance is alsovery relevant to them.

We are deliberately issuing this document quickly, without field testing or more extensive consultation, because we believe that CLD services are facing immediate challenges. We hope that they will find in it both particular items and general ideas that they can use and adapt.

If the demand is there, we hope that more work could be done in future to help people to improve their ability to gather, interpret and communicate evidence of outcomes, and to make the relationship to existing performance management systems clear.

A guide to the range of outcomes that we can expect CLD to bring about or contribute to has been available since 2007. The ‘Delivering Change’ report (see list of Further Reading and Resources)gave examples of areas where, we know from research and experience, CLD can achieve or contribute to end outcomes. But it took the view that in most cases CLD providers would find it more helpful to concentrate on the intermediate outcomes of their work.

Demonstrating intermediate outcomes is vital to explaining why what we do works. But we believe that there is now an urgent need to go beyond them and look directly at the contribution of CLD to shared end outcomes. The source of much relevant evidence is close to hand: the people and communities that we work with and the changes that they experience through their engagement with us. Thisguidance suggests that we can and should do more to record the measurable end outcomes that they achieve.

It also suggests that we can do more to show how our work at the ‘intermediate’ stages of engagement, capacity building, preparation for learning etc, is related to achieving those desired end outcomes.

So what does this framework offer? It is intended to have uses at three different levels:

as an aid to self-evaluation
for evidencing contributions to Single Outcome Agreements
to demonstrate the overall contribution of CLD to National Outcomes.

It is intended to provide:

a simple, basic framework, offering a common language, not a comprehensive solution
tools that can be‘owned’ by CLD practitioners, developed collectively, and used across all strands of CLD, including both local authority and voluntary sectors
not a template that can or should be used in all circumstances, but something to help people with their thinking; they can use parts of it where they make sense, and as local circumstances require.

It does not suggest that we should measure only end outcomes instead of intermediate ones, or that CLD workers can always prove in full their contribution to end outcomes. But we can do more to:

Show the logical steps connecting what we do to the end outcomes valued locally and nationally
Show the direct relevance of our work to those outcomes, by tracking what happens to participants and collecting information about change in communities.

GUIDE TO THE FRAMEWORK

The framework suggests indicators, or types of indicator, which could be used to record the performance of CLD services at four levels:

Engagement
Self-perception of impact
Measurable Outcomes
Sustained impact.

It is based on the principle that evidence of the relationship of CLD work to outcomes can and should be available at all of these levels. Naturally, it might be thought most persuasive if we could simply relate everything we do directly to end outcomes which are measurable changes in individuals and communities, particularly where these can be shown to be sustained over a longer period. But it is also important to show thatour work in engaging people and communities and helping them to prepare for change is often donein ways where the relationship to the end outcomes is clearly identifiable. We also need to do more to capture what people have to tell us about the changes that they themselves experience.

National Outcomes

The framework suggests indicators, at each of these levels, that could help relate CLD activities to the achievement of outcomes relating to each of the Scottish Government’s Strategic Objectives, namely to make Scotland:

Wealthier and Fairer
Smarter
Healthier
Safer and Stronger
Greener.

Indicators have also been included that relate more specifically to one particular National Outcome that does not relate exclusively to any of these Objectives, and is of particular importance for CLD work, namely:

NO11: We have strong, resilient and supportive communities where people take responsibility for their own actions and how they affect others.

This is about building social capital, and the personal skills and resilience that go with it. It is an important aim in itself, as well as being an intermediate stage along the way to achieving many other aims.

All of the indicators that we propose show CLD activity working towards or achieving a contribution to at least one National Outcome. These links are shown in the framework (users may be able to think of others). Some National Outcomes, such as NO6: ‘We live longer, healthier lives’, relate closely to the whole of one Strategic Objective, in this case ‘A Healthier Scotland’. Others relate only to part of oneObjective, or to more than one.

National Indicators

The National Outcomes are backed by a list of 45 recommended National Indicators and Targets (and a menu of possible Local Indicators for use in Single Outcome Agreements). The list at the end of this paper shows 26 National Indicators towards which different types of CLD activity may be able to demonstrate a contribution - more than half of the total, though some contributions are more indirect than others. In the framework, we highlight those indicators where we see a specific relationship to any National Indicator.

This is not to say that the most important indicators are those highlighted. The National Performance Framework is not a neat structure in which everything fits into everything else. For example there is no National Indicator that relates to employment or employability (except for school leavers and business start-ups). Yet the central purpose expressed in the National Performance Framework is increasing sustainable economic growth.

The measurement of National Indicators is mostly based on overall statistics showing what is happening to the whole of the relevant section of the population, the economy etc. CLD outcome indicators will mostly be based on the experience of the users of services or perhaps of particular communities. The extent to which they contribute to shifts in population totals, such as crime or ill-health rates, may be hard to quantify. But aligning CLD outcome indicators closely with national and local indicators helps to demonstrate the contribution of CLD.

Single Outcome Agreements

Sometimes Single Outcome Agreements include indicators which allow CLD services to demonstrate directly that they are responsible for the achievement of a defined proportion of the overall target. Such explicit links, according to research currently in progress,are most likely to be found in the areas of positive destinations for learners, adult literacy and numeracy, volunteering and community activity and engagement.

But often the contribution of CLD is acknowledged, if at all, ‘below the waterline’: in the strategies that explain how outcomes will be pursued, rather than in the Outcome Agreements themselves. In those cases, evidence of the outcomes of CLD activity should show that it is working in the right field, on a large and significant scale. It should provide persuasive pointers to show that CLD is making a positive contribution.

In many cases these indicators also need to be supported by a clear understanding of the logic that links the building of personal and community capacity to other outcomes.

Developing and adapting the framework

Different uses for the framework willrequire different approaches to all ofthese issues.Users can develop the framework in several ways. For some purposes we may need to present information which concentrates on the numbers of participants who are from disadvantaged communities and groups, or in the case of ‘smarter’ outcomes, of learners who had no or few qualifications at the outset. This would allow more direct links to be made to NO7: ‘We have tackled the significant inequalities in Scottish society’.

Youth work services could adapt many of the indicators by specifically counting participation of or impact on young people, enabling the contribution of their services to be demonstrated, and more links to NO4 (Curriculum for Excellence) to be shown.

There are two aspects to the framework. It focuses on both:

personal development
building community capacity.

Much, if not all, CLD practice cuts across these two areas, and we encourage people to identify outcomes from both. In a community capacity building context it may be desirable to include additional indicators (as well as those that show the creation of ‘strong, resilient and supportive communities’ and those based on individual learning) showingchanges that communities have achievedthrough action related to specific Strategic Objectives.

FRAMEWORK FOR EVIDENCE ON THE OUTCOMES OF CLD ACTIVITIES

Items that relate most directly to National Indicators (NIs) (see list below) are highlighted

(NO = ‘National Outcome’) (‘N’ = ‘Number’)

More links to NO4 "Our young people are successful learners, confident individuals, effective contributors and responsible citizens" (Curriculum for Excellence) and NO8 "We have improved the life chances for children, young people and families at risk can be made by specifically counting participation of or impact on young people" (or families for NO8)
More links to NO7 " We have tackled the significant inequalities in Scottish society" can be made by specifically counting participation of or impact on disadvantaged people or communities

WEALTHIER AND FAIRER

Links to National Outcomes / Engagement / Self-perception of Impact / Measurable Outcomes / Sustained Impact
All relate to NO1: We live in a Scotland that is the most attractive place for doing business in Europe
NO2. We realise our full economic potential with more and better employment opportunities for our people / N of participants in activities that create jobs in the social economy / N of jobs created in the social economy
NO2.
AND
NO3. We are better educated, more skilled and more successful, renowned for our research and innovation / N of participants in activities that improve employability, including self-employment
(Related to NIs if measuring ‘N of young people’ and/or ‘self-employment) / N of participants who report that their employability has improved
(Related to NI if measuring ‘N of young people’) / N of participants who after the activity ends are in sustained
a)voluntary work
b)part-time work
c)full time work
d)self employment
(Related to NIs if measuring ‘N of young people’ and/or ‘self-employment) / N of participants who six months after an activity ends are in sustained
a)voluntary work
b)part-time work
c)full time work
d)self employment
(Related to NI if measuring ‘N of young people’)

WEALTHIER AND FAIRER (cont)

Links to National Outcomes
All relate to NO1 / Engagement / Self-perception of Impact / Measurable Outcomes / Sustained Impact
NO7. We have tackled the significant inequalities in Scottish society / N of participants from disadvantaged communities & social groups
N of participants in activities that improve the take up of benefits
N of members of credit unions and young people in savings clubs
N of participants in activities that improve money management / N of participants who report that their money management has improved / Value of credit union savings and loans / Cumulative annual gain to individuals or families from improved benefit take up
NO7
AND
NO13. We take pride in a strong, fair and inclusive national identity / N of participants in activities that challenge discrimination and celebrate people’s identities and differences. / N of people who report that their understanding of activities that challenge discrimination, identity and difference has improved

SMARTER

Links to National Outcomes / Engagement / Self-perception of Impact / Measurable Outcomes / Sustained Impact
NO3. We are better educated, more skilled and more successful, renowned for our research and innovation / N of participants in activities that improve literacy and numeracy
N of participants in activities that lead to accredited awards or qualifications / N of participants who report that their reading, writing and number work has improved
N of participants who report that their readiness for further learning or involvement has improved / N of participants who achieve their learning goals fully or partly
N of participants who have completed achievement awards or qualifications
N of participants who progress to positive destinations at the end of an activity
(Related to NI if measuring ‘N of young people’) / N of participants who have sustained a positive destination six months after an activity
NO11. We have strong, resilient and supportive communities where people take responsibility for their own actions and how they affect others / N of participants in activities that prepare them for further learning, volunteering or community involvement / N of people who report that they have benefited from volunteering and community activity
NO3.
AND
NO11. / N of participants in activities that improve their confidence as learners / N of participants who report that their confidence as learners has improved / N of participants whose confidence as learners has improved based on reliable and valid measures
NO4. Our young people are successful learners, confident individuals, effective contributors and responsible citizens / N of participants in activities that address school attendance and behaviour / Improvements in school attendance, reductions in exclusions and improvements in behaviour attributable to work with young people

SMARTER (cont.)