Further information on soft outcomes for House of Lords ESF Inquiry
During the presentation of evidence WEFO officials agreed to provide their Lordships with some further information on how WEFO measures‘soft outcomes.’
WEFO’s methodology for monitoring and evaluating projects has been built up as part of its experience of managing the ESF Programmes in Wales.
Evidence from 2000 – 2006 Programmes
Evaluations from the 2000 – 2006 Programmes found that ESF projects were assisting highly disadvantaged individuals at some distance from the labour market. For these individuals, the achievement of hard outcomes such as qualifications and employment was a long-term process requiring multiple and often intense interventions. As a result these individuals progressed through a number of intermediate milestones as part of their journey towards employment. Some of these milestones were achievements of basic life-skills such as increased confidence or motivation. It was important that ESF projects assisting these individuals could measure and demonstrate the progress made by individuals as a result of the support provided.
In 2003, WEFO and the Department for Work and Pensions commissioned jointly a consultancy organisation called GHK to produce the first practical guide to measuring soft outcomes and capturing the distance travelledby beneficiaries from inactivity towards employment. The research underpinning the production of this guidance identified that there is no single, ‘off-the-shelf’ approach to suitall projects because of the wide variety of ESF supported activities, theresources/skill-sets available to them, and the different client groups with whomthey work.
To determine the extent to which soft outcomes measurement systems were used across the 2000 – 2006 Programmes and to investigate sponsors’ experiences of measuring soft outcomes, WEFO revisited this issue through fieldwork with ESF project sponsors as part of the Mid Term Evaluation Updates (MTEUs) for Objective 1 and Objective 3 undertaken in 2005. The MTEUs found that overall, project sponsors attached great importanceto the achievement of soft outcomes and several collected soft outcomes data at beneficiary level. However, many project sponsors were not using the information in any systematic way: the information enabled them to demonstrate progress of individual beneficiaries but did not allow project sponsors to aggregate data to demonstrate the project’s achievement as a whole. The MTEU also found that the systems developed by projects varied considerably in theirsophistication and the level of input required by the project sponsor.
To explore project sponsors’ experience of measuring soft outcomes further and to understand the barriers to systematic measurement, WEFO undertook qualitative research witha sample of Objective 3 project sponsors in 2006. The research found that sponsors useda variety of soft outcomes indicators and tools but weresceptical of the possibility of aggregating soft outcomes data.
The key messages from the research on the 2000 – 2006 Programmes were that soft outcomes are important but that project sponsors all had their own systems commensurate to the client groups with which they were helping. And it was apparent they perceived it as very difficult to aggregate these systems into a coherent aggregate individual indicator.
Approach in 2007 – 2013 Programmes
Based on the 2000-2006 research WEFO has designed its monitoring system for the 2007-2013 Programmes using a two-pronged approach.
Firstly, the needs of participants at some distance from the labour market and what this requires in terms of support and delivery by projects is better understood. WEFO has developed a progression indicator, ‘Participants gaining other positive outcomes’, to enable projects to report intermediary outcomes, such as completing courses, entering voluntary work, or attending a job interview. This indicator represents soft outcomes but with a hard edge: it allows progression towards hard outcomes to be captured but, because the progression is demonstrated by tangible actions set out in the definition[1] of this indicator, it can be measured consistently and therefore aggregated in a way that concepts such as ‘increased confidence’ cannot because they mean different things to different people.
Secondly, as many projects have their own soft outcomes monitoring systems, WEFO encourages projects to provide details from these systems in their project evaluations. WEFO has increased its emphasis on project evaluation in the 2007 – 2013 Programmes and has set up a team to support project sponsors to meet their evaluation requirements. This gives sponsors an opportunity to report their soft outcome achievements as well as the harder outcomes to WEFO.
This approach to soft outcomes complements the wider ESF monitoring system, which involves the collection of information at participant level. Collecting this level of information gives WEFO a better understanding of the journey undertaken by participants as they progress from unemployment / inactivity to employment. The enhanced ESF monitoring system and project evaluation arrangements gives WEFO a richer set of information on which judgements of the effectiveness of the Programmes can be made.
[1]The number of participants gaining intermediary outcomes as a result of participation in an ESF-funded project. Intermediary outcomes, such as completing courses, entering voluntary work, or attending a job interview, are those which are achieved as part of the journey to achieving final outcomes, such as entering paid employment or gaining qualifications.