1
Prof. Dr. Erwin Seyfried
The Technical Group
Contents
Preface
1.Introduction: the usefulness of indicators
2.Clarification of objectives
2.1The European policy priorities
2.2Key dimensions for measuring employability, matching, access
2.3Main categories of objective for the European policy priorities
3.A coherent set of indicators
3.1Context indicators
3.2Input indicators
3.3Output indicators
3.4Outcome indicators
3.5Examples from Member States
4.Indicators, evaluation and feedback of conclusions into the policy cycle
5.Proposals for indicators
5.1Indicators addressing the policy level from a European perspective
5.2Indicators addressing the European policy priorities
5.2.1Indicators addressing employability
5.2.2Indicators addressing matching
5.2.3Indicators addressing access
5.3Indicators to be used in European surveys
6.General requirements for the use of indicators
6.1Hard and soft indicators
6.2Data collection efforts
6.3Limits in the use of indicators
7.Proposals for further action
6.1Proposals concerning the implementation of indicators
6.2.Proposals to support European cooperation
6.2.1A peer review process
6.2.2Comparative European bench-learning clubs among VET providers
8.Points for further discussion
Annex 1List of policy objectives for initial VET and continuing VET
Annex 2Possible indicators according to policy priority objectives
Annex 3Table with indicators which have been reviewed
Preface
In its ‘Proposal for Action’[1] to support the quality of vocational training the European Commission outlined the agreement on common quality indicators in the field of initial and continuing vocational training as one of the main areas for cooperation between the Commission and the Member States. Subsequently the European FORUM on quality in VET agreed to make the topic of indicators a priority in its working programme and to form a subgroup amongst its members to work more intensively on this subject with the support of a member of the Technical Group .[2]
In the last two decades Europe has seen a growing awareness in most Member States that quality in VET is an important but complex policy issue. In most Member States quality management approaches have been developed to meet specific national policy objectives and concrete challenges. This developments resulted from the demand to increase the efficiency and effectiveness of VET as a key instrument to achieve economic competitiveness. Nowadays, in principle, many tools that enable quality in VET to be measured are available in Europe. However, these tools are mainly addressing the level of VET institutions or training providers. The approach of the FORUM could not be to repeat or to double the lists of indicators used by these quality tools but it had to be to look for an original approach on the level of VET systems and to support a European strategy for the improvement of quality. Against this background the subgroup on indicators chose a European perspective as its starting point, where common objectives on European level, increased cooperation between Member States and comparability are important aspects. From this perspective the work of the subgroup focused on two points:
- the identification of indicators to support the policy objectives which are seen as common priorities on European level, (see section 5.1);
- the development of proposals for indicator-based European cooperation supporting the improvement of quality in VET (see section 5.1).
To achieve these objectives the work of the subgroup was based on a complex methodological approach which consisted of several elements.
To make the best use of already existing experiences and thus following a bottom-up approach, the members of the FORUM had been asked to provide examples of the use of indicators for quality improvement in their Member States and these experiences were then reviewed from in a European point of view. In addition to the identification of quality indicators used by the Member States, a second approach of the subgroup consisted in reviewing the indicators which are in use on European and international levels. The OECD, Unesco, Eurostat, Eurydice, Cedefop and other institutions regularly publish relevant data serving different indicators in the VET sector. Thirdly, the work of developing common indicators for quality in VET needed also to be connected with the work already being done at the European level on education, in field of lifelong learning or in the framework of employment policies.[3]
So far, over 200 indicators, together with the examples of the Member States, have been reviewed and analysed in terms of their suitability for a European approach to improve quality in VET.[4] Whenever possible the indicators described in this report have been drawn from these existing data sources.
This approach, which took into consideration the empirical situation on the Member State and international levels, was complemented by an additional theoretical approach. As the use of isolated indicators risks provoking negative side-effects, it was necessary to develop a conceptual framework for a system of indicators linking different quality indicators to one another. An additional focus of the work consisted in the translation of the European policy priorities into measurable objectives.[5]
Based on the results of the empirical review of existing indicators and taking into consideration the theoretical framework, a final step in the process was to address the issue of European cooperation. This step led to proposals for indicators for a quality policy at the European and Member State levels.
Erwin Seyfried
1.Introduction: the usefulness of indicators
In recent years there has been growing awareness of the importance of indicators for the improvement of VET systems. There is a general consensus that indicators for the observation and/or measurement of quality are key instruments for guiding and improving the quality of education and vocational training and are necessary for the good governance of training systems and structures. Though not all Member States have the same experience with the use of indicators in VET, it is generally agreed that indicators are a necessary part of every mechanism (including self-assessment) designed to ensure constant progression towards quality improvement.
Generally speaking, an indicator is a characteristic or an attribute that can be measured to assess a certain action. This may relate to the measurement of an objective to be met, a resource mobilised, an effect obtained, a gauge of quality or a context variable. The role of indicators is to describe:
-the current status or the base-line from where we are starting with our efforts to increase the quality of VET;
-to operationalise and quantify (as much as possible) the quality objectives which have been set;
-to provide continuous information on the extent to which those objectives have been met;
-to provide an idea of the factors which might have contributed to the attainment of certain results.
A further operational element is that indicators should produce information to help the relevant actors in VET not only to assess the extent to which their pre-defined objectives have been met, but also to help communicate the results, negotiate the effects, and adopt the consequent decisions. Although in the long run indicators go for valid information and measurement one should be aware that their function and their merits in practice are sometimes different as in a first step they help to gain insights into the most relevant dimension for quality. By their reflexive function they support the mutual understanding of relevant criteria for quality and thus they enable mutual learning.
This aspect is also particularly relevant for European cooperation, where indicators are of central importance for enriching the widely used instruments like the exchange of good practices and the adoption of bench-marking processes. The Barcelona European Council in March 2002 reaffirmed the importance of developing closer cooperation between Member States in the field of VET and developing a more systematic strategy for the active exchange of good practices. The use of indicators will help to arrive at commonly shared understandings of good practices, it can help to structure the exchange of experiences and it will also help to identify strengths and weaknesses of VET systems at European, national, regional and sectoral levels.
In order to support the cooperation on quality in VET on European level it also might be helpful to have some specific criteria and indicators which should help to compare the different achievements for quality between Member States but also help to reflect the extent, the level or even the “quality” of cooperation in VET on European level. This paper therefore will also make some proposals for quality indicators addressing the policy level of a European strategy for quality in VET.
2.Clarification of objectives
An indicator is not a value in itself. The definition and selection of indicators presupposes the clarification of the objectives to be attained in order to improve quality, i.e. the selection of indicators is not a technical task; the central focus must be on the quality objectives of VET.
Taking this insight as a principal guideline, the indicators for quality in VET proposed in this report refer primarily to the policy priorities identified by the European Commission, the Member States and the social partners.
2.1The European policy priorities
The European FORUM on quality in VET asked for the development of indicators to measure the quality of vocational training in relation to the following three political priorities set by the Member States, the European Commission and the social partners:
(1)better employability of the labour force;
(2)better match between training supply and demand;
(3)better access to vocational training, in particular for vulnerable groups on the labour market.
Taking these policy priorities as a point of departure, the definition and choice of appropriate indicators of quality in VET should be guided by the means of documenting, assessing, and supporting the achievement of these goals.
So far, quality has been linked primarily with how education and training organisations provide their services, in terms mainly of the results of their teaching or training activities. Thus quality has been approached mainly from the point of view of the processes within VET organisations.[6] The FORUM adopted a wider concept of quality, with the challenge of developing quality indicators at the level of systems, thus including and going beyond the level of VET institutions and training providers.
2.2Key dimensions for measuring employability, matching, and access
The three political priorities provide the FORUM with clear guidance on where to look for quality indicators, although those priorities will have to be translated into specific objectives. However, this is not an easy task, as each of the three policy priorities is in itself a complex and ambitious area of action and policy.
Employability
A complex concept cannot be measured directly, and in order to define it in terms of empirical traits, there is room for manoeuvre. The concept of employability also has no commonly agreed definition. It includes some paradigm shift in the definition of the relationship between education/training and employment. Employability refers to individuals, and their responsibility to help bring about the conditions necessary for their participation in employment; it also refers to the overall system of employment relations, including business practices, which to some extent provide incentives and raise expectations. A working definition by Cedefop to cover the level of the individual includes in particular the following two components: a willingness to be mobile and a willingness to develop occupational competences.[7] However, understanding employability as a pillar of the employment strategy refers more explicitly to creating and fostering sufficient framework conditions to bring about employability at an aggregated level.
The main factors determining employability are competences and attitudes. Those factors have, paradoxically, often been poorly measured so far. Moreover, their actual impact has rarely been demonstrated. Other - more indirect - factors are the acquisition of qualifications and several aspects of employment conditions and behaviour.
Matching supply and demand
Matching between supply and demand means not only ensuring that people’s knowledge, competences and skills correspond to the growing demands of jobs, occupations and working methods; it also means matching the needs of individuals and (potential) trainees with those of organisations, the labour market and the civil society in Europe.
In recent years VET systems have come under pressure to match supply more closely to demand. However, assessing the quality of that matching process has remained rather implicit so far. In fact, explicit measurement of the quality of the matching has proved to be rather difficult.Two basic approaches to solving the matching problem are prevalent in certain VET systems: first, the provision of broader supply profiles in order to absorb the dynamic of change by increasing the flexibility of the actors involved, especially individuals and enterprises (the flexibility approach); and secondly, the provision of more specific supply profiles that are more closely tied to demand (the specification approach).
The main factors that contribute to improving and assessing the quality of matching are proper information procedures, including forecast mechanisms, and a responsive provision of new education and training offers. There must be a clear definition and monitoring of demand and of its relation to supply in terms of comparable measurement units. Indicators for measuring the quality of matching should include information about structural variables, i.e. occupations and job vacancies, broken down by trade or sector, and education/training level, field of qualifications or competences (in most systems more than one of these dimensions will be relevant).
Access
Equal opportunities in access to the VET system is an essential part of the overall European objective of building an inclusive society in which VET provisions are based on the needs and the demands of individuals. The provision of access as a policy priority in terms of quality is a conflict-laden issue, as one main dimension inherent in education is selection. Therefore accessibility has to be seen to some extent in relation to selectivity against the background of differing values and interests. Education systems and policies serve to varying degrees to establish the relationship between accessibility and selectivity. A basic distinction can be made between general issues of accessibility and more specific problems of vulnerable groups.
In general terms access is mainly concerned with structural opportunities or obstacles to participation in VET, with participation encompassing motivational issues as well as financial, institutional and cultural necessities. In this respect access depends on the linkages and pathways between the different parts of the VET systems, and the provision of mechanisms for accreditation and certification of previously acquired skills and competencies.
The main factors contributing to better access are equal opportunities at the general level, and outreach activities and targeted offers for vulnerable groups. Indicators for measuring the quality of access should in any case provide information about group-specific distributions in VET participation, in general at least about sex, age (with special attention to young people and people older than 55) and other social characteristics. With regard to access for vulnerable groups, a meaningful and comparable classification should be developed, including variables such as disability, exclusion from employment, in long-term unemployment, ethnic minority and migrant statuses and levels of disadvantage in relation to family status, regional traits, etc.
2.3Main categories of objective for the European policy priorities
Based on the above considerations, and taking into account the theoretical and policy discussions, as well as the experience in the Member States, a conceptual classification of objectives can be developed. Table 1 provides a proposal for a set of main categories of objective, which have been selected to operationalise the three basic policy priorities.
Table 1: Main categories of objective for the European policy priorities
Employability / Matching / AccessMain categories of objective to be attained or improved as criteria for quality / - Competences and attitudes achieved (levels: basic, ICT, social, personal, technical)
- Completion of education and training pathways and acquisition of qualifications (avoidance of drop-out)
- Transition to employment, employment participation (quantity measures, duration)
- Quality of employment (stability, income, desired working time) and employment in new and flourishing sectors / - Information (knowledge about training demand acquired and transmitted)
- Responsiveness (knowledge about training demand recognised and reacted to)
- Adaptation (production of training supply related to demand)
- Innovation (training supply related to new demands) / - Basic competences for everyone (distribution among groups)
- Broadening of access to education and training pathways (to everyone who can benefit)
- Effective opportunities for vulnerable groups (targeted provision, outreach activities, acceptance, completion and utilisation)
- Permeability of systems: accreditation and certification of acquired skills and competences
Employability includes four basic categories of objective: competences and attitudes, completion of training programmes and pathways, transition to and participation in employment (quantitative), and issues relating to quality of employment. Employability cannot be measured directly. Therefore indirect measures must be elaborated depending on which kinds of aspects of the concept they refer to. To some extent the definition of objectives can be assessed in terms of how ‘near’ the measurement is to a certain concept of employability. The main measures applied so far concern employment. However, employment does not really measure employability. People who are employed are necessarily to some extent employable; however, the opposite need not be true. As the concept is strongly contextual, VET is not the only contributor to employability. Therefore overall measures cannot be directly attributed to the quality of VET. The definition of a set of different objectives which are related to each other (see para. 3) can to some extent bridge that attribution problem, as the objectives express what is expected overall from VET to improve employability.
Matching includes the following four categories of objective: information about demand including foresight, responsiveness of supply, adaptation and innovation. The first category concerns the production and dissemination of information and knowledge in a system. How are mismatches detected (informally, formally)? What time frame is involved (short-, medium- or long-term; reactive, proactive)? What are the main aspects of mismatches that are deemed relevant for improving matching? How is information about mismatches communicated among the actors involved?The second category of objective concerns the strategies and actions taken to react to mismatches (flexibility and broadening of delivery, specification and updating of VET). Which actors (individuals, enterprises, education sector, policy and public sector, research) have which implicit or explicit responsibilities in the prevailing matching practices? Which kind of information should be disseminated amongst the actors and how?