Quality Assurance
For
Vocational Education & Training
In
Romania
Final Report
External Monitoring Procedure
2007/2008
Content
1. Introduction
2. Summary
3. European Context (Background)
4. National Context (Background)
5. Scope
6. Methodology
6.1 Visit Process
6.2 Visit Feedback
6.3 Continuous Professional Development
6.4 Analysis
7. Difficulties
8. Findings
8.1 Strengths
8.2 Weaknesses
8.3 Examples of Good Practice
8.4 Data from the Checklists
9. Conclusions
10. Recommendations
1. Introduction
The national implementation of quality assurance (QA) in technical vocational education & training (TVET) in Romania is facilitated through a European-funded PHARE[1] project. As part of the TVET QA Mechanisms TVET schools receive external monitoring (EM) visits to support them in their self-assessment process as well as validating their self-assessment report and improvement plan.
This report details the findings of the analysis of external monitoring visit reports, draws some conclusions, and makes recommendations for the further improvement of EM visits in particular and TVET QA Mechanisms in general.
2. Summary
The EM visits were carried out by the school inspectorates between 20 March and 6 May 2008. The calendar for EM visits was developed in agreement with TVET schools and in compliance with the period of time set by the MERY[2]. Although there are 1316 TVET schools operating in Romania, only 150 of them were assisted by the PHARE project, of which 50 schools are in rural areas. These 50 rural TVET schools were visited by PHARE project local experts.
External monitors[3] produce a written report after their visit, which is sent to the inspectorate. The PHARE project received 941 EM visit reports for analysis; this represents a return rate of 71.5%. In addition, some inspectorates forwarded their synthetic reports (4) or inspection reports (6), which were also analysed.
Although TVET schools are implementing the TVET QA Mechanisms of self-assessment and improvement planning, there is still a lack of statistical data being used for measuring and comparing performance. Hence, it is difficult to validate the strengths and improvements that TVET schools and external monitors have identified.
However; the results demonstrate clearly that the formal appointing and resourcing of external monitors at the inspectorate level who provide permanent assistance to TVET schools should be instigated at the system level. County inspectorates should be supported in developing and maintaining reliable databases, which include statistical information as well as qualitative one, so that synthetic reports could become valuable and valid tools for evaluation on the system level. Synthetic reports should become a recognised tool for analysing EM and inspection reports on county inspectorate level, thus facilitating the gathering of information for the annual quality report on the system level.
3. European Context (Background)
Total Quality Management (TQM) is probably the best known quality assurance system. There are a number of widely used TQM models and some, like the ISO9000 family of standards[4] and the European Foundation for Quality Management (EFQM) Excellence Model[5], are used by VET organizations in some countries. However, neither ISO9000 nor EFQM addresses the core activity of “learning”.
Therefore, the Technical Working Group Quality in VET designed the Common Quality Assurance Framework for VET in Europe[6] (CQAF), which was ratified in May 2004 by the European Commission (Council Conclusion on Quality Assurance in VET of 18th May; 9599/04LIMITE/EDUC117/SOC252), and is the umbrella framework under which each nation state will develop its own National Quality Assurance Framework for VET.
The CQAF, which was updated in September 2005[7], is a framework model that is to serve the EU member countries as a reference for the development and reform, respectively, of quality systems in VET at national level. The CQAF describes basic principles, criteria, and instruments that are to be taken into consideration in the implementation of national quality assurance systems in VET.
In July 2007 the European Parliament and the Council of the European Union published their recommendation for the establishment of a European Quality Assurance Reference Framework for VET (EQARF-VET)[8], which was revised in 2008. The objective of this recommendation is to establish the EQARF-VET as a reference tool to help Member States to monitor continuous quality improvement of their VET systems, and to examine how the existing quality assurance models work.
The EQARF-VET includes three strands:
· the CQAF, which is based on the quality cycle and designed to facilitate a systemic approach to VET target setting, implementation, evaluation, and review at all levels, through specific criteria
· models for VET provider self-assessment, internal evaluation, and external evaluation by third parties, to be identified by Members States as appropriate
· Key Performance Indicators (KPIs) as measuring tools to facilitate the evaluation and improvement of VET systems and provision by Member States and relevant stakeholders, through a reference set of quality indicators
4. National Context (Background)
Romania has been adopting European developments in QA for TVET schools since 2003. First the CQAF was adapted to the national context and a proposal for QA Mechanisms in TVET was developed together with two instruments[9], which are based on the European Guide for Self-Assessment of VET Providers[10]. Between 2004 and 2006 Romania further developed, piloted, implemented, and revised quality assurance instruments for the TVET system. Since 2006 the TVET QA Mechanisms have been implemented nationally in all TVET schools.
The QA Mechanisms for TVET in Romania include all three aspects of quality: assurance, control, and improvement; and they have continuously been aligned with best European practice (EQARF-VET and Peer Review Manual[11]) and will ensure the quality of TVET provision. However, the TVET QA Mechanisms also maintain Romanian values; for example, TVET will continue to be a school-based provision incorporating generic skills and general learning along with vocational skills and knowledge.
One of the quality assurance instruments is the self-assessment process undertaken annually by VET providers. The self-assessment process is supported and monitored through external monitoring activities. During the academic year 2003/2004, with the help of European-funded PHARE projects for TVET development, the use of the 2 instruments and associated manuals was piloted in 22 TVET schools. Another 100 TVET schools joined the pilot phase during the academic year 2004/2005. In 2005/2006 an additional 50 TVET schools especially selected from rural areas were added to the 122 existing pilot TVET schools to undertake further piloting of TVET QA Mechanisms.
In 2006 the National Framework for QA in Education was validated at national level through the ratification of Law 87/2006 on Quality Assurance in Education to approve the Government Emergency Ordinance 75/2005, which set up the legal institutional framework for developing and implementing quality assurance mechanisms at both system and provider levels.
On the basis of Law 87/2006 two national quality agencies, ARACIP and ARACIS[12], were established and charged with developing quality assurance frameworks for Pre-University and Higher Education respectively. According to Law 87/2006 TVET schools and their programmes are approved against the ARACIP Authorization and Accreditation Standards, and graded against a 5-point scale. To be judged “satisfactory” a TVET school has to meet all 43 indicators of the Accreditation Standard. Higher grades of “good”, “very good”, and “excellent” are measured against the ARACIP Reference Standards.
During 2007 CNDIPT[13] adapted the QA Framework in Education (Law 87/2006) to the specific requirements of TVET; and, since the academic year 2006/2007 the TVET QA Mechanisms have been implemented in all 1316 TVET schools by ministerial order. According to Ordinance no 4889/09.08.2006 issued by MERY all TVET schools have to draft an annual self-assessment report and to submit it to the general school inspector in charge with quality assurance. TVET schools measure their findings from the TVET QA Mechanisms self-assessment process against the indicators and 5-point grading scale when completing the ARACIP self-assessment report format.
5. Scope
For the quality assurance aspect the Terms of Reference for the 2007/2008 PHARE project stipulated external monitoring activities for only the 150 assisted TVET schools. However, as the TVET QA Mechanisms have been implemented on a national level since 2006/2007, the PHARE project has always analysed all 1316 EM visit reports, as well as the 1316 self-assessment (SA) reports. For the academic year 2007/2008 again the EM and SA reports of all TVET schools were scrutinized and evaluated.
EM visits to all 1316 TVET schools were organised and carried out by the school inspectorates, and the PHARE project kept tracking sheets of visits to the 150 assisted schools. In addition, and to gain further insight into the EM process, the 50 rural TVET schools of the 150 PHARE assisted schools received EM visits, which were undertaken by 8 trained local quality experts.
External monitoring is a system level quality assurance mechanism; and as part of quality control and improvement, the system level self-assessment process has to collect feedback from stakeholders regarding the impact of its QA mechanisms and instruments. Therefore, external monitoring visits have the following objectives:
· evaluation of the teaching, practical training, and learning process, with particular stress on the utilization of learner-centred methods
· reporting on the implementation of the self-assessment process
· validating the self-assessment report and improvement plan
· obtaining feedback from a variety of stakeholders
· examining documentation of the self-assessment process
· gathering information on the nature and efficiency of TVET school partnerships and/or networks
· reporting on strengths, improvements, and examples of good practice at system and TVET school level
· reporting on weaknesses and difficulties at system and TVET school level
· providing advice and guidance, and supporting TVET schools in the implementation of self-assessment and quality improvement
The external monitoring process may be carried out twice per year. The first visit, which is performed halfway through the annual self-assessment cycle, aims to support TVET schools in their self-assessment, whereas the second visit validates their self-assessment report. External monitoring visits are at the discretion of the inspector; and an inspector might make one or more visits per year. The main visit should take place during the period between February and May for giving advice and guidance; a second visit in September/October for validating the self-assessment report.
The analysis of external monitoring during the academic year 2007/2008 took into account the reports from the EM visits carried out between 20 March and 6 May 2008.
6. Methodology
A number of different processes are part of external monitoring activities and have contributed to the analysis of EM visit reports.
6.1 Visit Process
After having been allocated to relevant TVET schools (either by the inspectorates or by the PHARE project) external monitors have to contact their TVET school in order to establish a suitable visit date. External monitors also have to scrutinize the school’s self-assessment report and other information they deem necessary for the preparation of their visit. From this scrutiny external monitors devise a visit plan; this stipulates what documentation is to be examined, which lessons are to be observed, which stakeholders are to be interviewed, and which sites are to be visited during the EM visit.
During the visit external monitors should follow the external monitoring procedure and adhere to the code of conduct at all times. At the end of the visit the external monitor will give oral feedback on the findings and note any examples of good practice found during the visit. For a 1-day visit no more than 2 to 4 hours should be spent in writing the final EM visit report.
External monitoring visits are not inspections; they usually only last 1 day in order to provide support and guidance to TVET schools during their self-assessment cycle. The visit focus may be organisational, thematic, or concentrate on one or two specific quality principles. Therefore, each individual visit follows its own patterns in terms of content and subject matter.
6.2 Visit Feedback
During the academic year 2007/2008 a new system level quality assurance procedure was introduced in the form of visit feedback questionnaires. These questionnaires were piloted with the 150 PHARE assisted TVET schools and their visiting external monitors. The purpose of the questionnaire research was to evaluate the compliance with and usefulness of external monitoring procedures and outcomes.
The 31 questionnaire items covered activities before, during, after, and between EM visits, as well as the performance of external monitors and TVET school quality coordinators, and some general issues. It was important to gain feedback from both players; therefore, in order to get the perspectives from both the school and external monitor two slightly different questionnaires were developed.
113 feedback questionnaires were returned by TVET schools; this represents a return rate of 75.3%. 177 questionnaires were received from external monitors. The results of the questionnaire analysis can be found in the Evaluation Report of EM Visit Feedback[14].
EM visits also have the purpose of collecting feedback from other stakeholders through interviews. External monitors should talk to learners, teachers, trainers, and quality coordinators during their visit; but, they might also interview parents, employers, auxiliary staff, management team members, local council representatives, or representatives from TVET school partnerships or networks. For the interviews external monitors were provided with an interview schedule for different types of interviewees so that approximately the same questions were asked of all stakeholders, so that their feedback could be compared and analysed.
6.3 Continuous Professional Development
The 1316 TVET schools have been implementing self-assessment against the 7 quality principles of the TVET QA Mechanisms (3rd edition, 2007) since 2003. Although quality coordinators have been previously trained, it is essential that their competences are continuously being enhanced. Furthermore, the self-assessment process needs to be standardized across providers and good practice needs to be shared, especially regarding the writing of the self-assessment report and making judgments about evidence. The same issues apply to the continuous development of external monitors.
Therefore, directors and quality coordinators of the 150 PHARE assisted TVET schools, as well as external monitors attended training events on the subject of quality assurance, held in Sinaia during March 2008. An additional training event was organized for technical inspectors of all 43 regional inspectorates.