EVALUATION REPORT:
LIFE SKILLS PROJECT IMPLEMENTATION
IN
THE ARMENIAN EDUCATION SYSTEM
A PROJECT SPONSORED BY
UNICEF
THE MINISTRY OF EDUCATION AND SCIENCE
Coordinated and Written by:
Carolyne Ashton, Evaluation Consultant
In Collaboration with:
Karen Melkonyan
Inspection Department, MOES
Lala Sargsyan
Coordinator for Curriculum and Methodology Implementation
Center for Education Reforms
Julietta Gulamiryan
Professor
State Pedagogical University
Marine Soukhudyan
UNICEF Education Project Officer
Yerevan, Armenia
March 2001
45
TABLE OF CONTENTS
Executive Summary
Section I: Methodology
Section II: The Need for a Life Skills Curriculum in the Armenian Education System
A. The Status of Educational Reform in the Republic of Armenia
B. The Life Skills Curriculum: A Description
C. The UNICEF Role
Section III: Planning and Implementation of the Life Skills Project
A. Organization and Management of the Project
B. Project Administration
C. Two-Phased Implementation of the Project
Section IV: Evaluation Team Report on Site Visits and Interviews
A. Principal, Teacher, and Student Interviews
B. Classroom Observation
C. Teacher Training from the Perspective of the Core Team and Other Trainers
D. Institutional Support Within the Ministry of Education and Science
E. Current Status of the Core Team
F. Life Skills Media Coverage
G. The UNICEF Perspective
H. Conclusion
Section V: Summary of Findings and Recommendations
Appendices: A through D
45
Executive Summary
This report represents an evaluation of implementation of the Life Skills Project being conducted in the Armenian education system as component of an overall effort in education reform. Dialogue about the project began in 1997. A variety of meetings were held with various stakeholders interested in education reform and in 1998 UNICEF and the Armenian Ministry of Education and Science (MOES) agreed to implement the Life Skills curriculum. The project was piloted in the first and fifth grades in 16 schools in 1999-2000. In 2000-2001 the project was expanded to 100 schools and to the second and sixth grades. UNICEF provided funding and some logistical support and the MOES provided administrative and logistical project support and workspace for the curriculum development team.
Section I of the report describes the methodology used. The evaluation reported here included a review of archival documents on the Life Skills project and a series of unstructured interviews with principal stakeholders including UNICEF staff, ministers and staff of the MOES, principals, teachers and students in Life Skills schools. The methodology for the research was primarily qualitative analysis as there is presently no evaluation in place to assess student and teacher short-term outcomes and long-term impacts. An external evaluator conducted the evaluation with support from a national team of education experts.
The past and current state of the Armenian education system and a description of the Life Skills curriculum are discussed in Section II. Armenia has set reform efforts in place to accomplish several goals as a result of the transition from the Soviet system that ended in the early 90s. Three major educational aims of the reform are:
1. to contribute to the democratization of Armenian citizens of all ages,
2. to provide an education that is competitive in the current international system, and
3. to bring about the humanization of education, that is, promoting a child-centered learning environment and providing children with skills for everyday life as citizens of their country and the world.
Implementation of the Life Skills curriculum, modeled on a Canadian program, was selected as a major contributor to the reform effort because it focuses on developing a set of skills that research has shown contribute to the types of changes sought in Armenia. Those skills include:
· Decision making
· Problem solving
· Creative thinking
· Critical thinking
· Effective communication
· Interpersonal relationship skills
· Self-awareness
· Empathy
· Coping with emotions
· Coping with stress
A Canadian team trained a Core Team of curriculum developers and teacher trainers to write a curriculum tailored for Armenian students and train teachers in implementation.
Section III reviews organization and management of the project and identifies facilitators, barriers, and challenges to the implementation of the project. Administrative and resource issues such as the role of accounting procedures, lack of access to electronic communication, the development of the Core Team’s capabilities, and relationships among the partners are included in this section.
Section IV provides a narrative overview of the findings from the interviews conducted for the report. These findings and a set of recommendations for the future of the project are summarized in Section V. Overall, the findings were very positive. Students were enthusiastic about the Life Skills curriculum and wanted it expanded to more of their classes. Teachers and Principals reported positive changes in the students participating in the curriculum. Anecdotal evidence that indicated students were practicing the lessons learned in the Life Skills classes. For example, children who were disruptive before coming into the classes were now calmer and more interactive. Two fourth grade classes that were troublesome were combined into a fifth grade class that studied Life Skills and became more collaborative and settled down to enjoy learning. There were many such examples offered. Interviews with the teachers, principals, Ministry officials and the Core Team pointed to severe resource deficits that may hamper expansion of the project. The Core Team’s work in developing the curriculum and training teachers and teacher trainers was highly regarded. Though there was no formal evaluation of student and teacher outcomes, the interview results were strong enough to suggest that a more rigorous evaluation would probably show the same positive results but might have provided clearer data on identified problem areas to be addressed in order to strengthen the project.
Recommendations were made in several areas of program implementation and expansion. The following is a synopsis of those recommendations.
Structural and Financial:
Ø While the relationship between UNICEF and the MOES is a strong one, the evaluation indicated a need for stronger coordination between UNICEF and the MOES on administrative and financial processes.
Ø It was recommended that the MOES begin a concerted effort to bring more partners into the project.
Ø The Center for Educational Reform should remain the base for the project and the Core Team should receive appropriate levels of support to conclude development of the project through the seventh grade.
Ongoing Development of Curriculum, Training, and Project Evaluation:
Ø Continue development of materials through the seventh grade.
Ø Continue teacher training and expand the program gradually to other schools.
Ø Develop a multi-year plan for integration and expansion of the Life Skills curriculum.
Ø Identify resources to support the Core Team in continued curriculum development.
Ø Develop Life Skills curriculum for the special education population.
Capacity Building and Sustainability:
Ø Begin an immediate effort to identify expanded resources for funding and other types of support.
Ø Identify sources of material resources such as computers and copy machines for the Life Skills schools and the Core Team.
Ø Begin to implement teacher training in pedagogical universities as a part of pre-service.
Evaluation and Assessment:
Ø UNICEF and the MOES should work with others who are implementing the Life Skills project to develop an evaluation design and the necessary instruments to assess the short-term outcomes and long-term impacts of the project. Evaluation should be conducted on a regular basis with quarterly and annual reports.
CONCLUSION
It was the determination of this evaluation that the Life Skills project is well worth continuing if the standards of development and implementation can be maintained, if a plan for funding and integration can be achieved, and if an evaluation procedure is put in place to provide an annual assessment of short-term behavioral changes and long-term impacts resulting from the project.
The ultimate responsibility for the education of Armenian children must fall to the Armenian people. The recommendations provided for this report may seem overwhelming if taken all together, but if the parties are willing to develop a multi-year plan which simultaneously builds the implementation of the project while building the resource base from a wide variety of sources, this project is sustainable.
SECTION I: METHODOLOGY
This report was commissioned by UNICEF and prepared for UNICEF and its partner in the Life Skills project the Ministry of Education and Sciences (MOES) of the Republic of Armenia (RoA). The report was compiled by an external evaluator who led an evaluation team consisting of national experts in Armenian education: Mr. Karen Melkonyan, Deputy Minister, Inspection Department, Ministry of Education and Science, Ms. Lala Sargsyan, representing the Center for Educational Reform (CER), and Professor Julietta Gulamiryan of the State Pedagogical University. Ms. Marine Soukhudyan, the UNICEF Education Project Officer and a translator assisted the team. During the time period 05 March through 16 March, 2001, the external evaluator met with various parties in the Ministry of Education and Science and the evaluator and the national team visited several of the schools that are implementing the Life Skills project as well as the Syunik Marz Education Department of the Ministry of Education and Science in Kapan and a regional office of the Center for Education Reform in Goris.
Unstructured interviews were conducted with
v Edvard Ghazaryan, Minister of Education,
v Aida Topuzyan, Deputy Minister for Education,
v Robert Stepanyan, coordinator of the Life Skills project for the Ministry from 1998 to 2000,
v Victor Martirosyan, Director of the Center for Education Reform,
v The Core Team of curriculum developers and trainers of teachers for the Life Skills project,
v Karen Melkonyan, Inspection Department, Ministry of Education and Science,
v Lala Sargsyan, Coordinator for Curriculum and Methodology Implementation, CER,
v Julietta Gulamiryan, Professor, State Pedagogical University,
v Grisha Manoucharyan, Syunik Marz Education Department of the Ministry of Education and Science
v Artashes Torozyan, Director, Syunik Regional Office of CER
v Three Life Skills teacher trainers (non-core team members), one in Kapan and two at the CER in Yerevan,
v An accountant for the of the Center for Education Reforms ,
v Gloria Fernandez, UNICEF Assistant Representative
v Marine Soukhudyan, UNICEF Education Project Officer
Visits were conducted at schools in Yerevan, Lori Marz and Syunik Marz. The schools visited included:
Ø Lori Marz: Gagar School, Stepanavan Schools # One, Two, and Four, and Agarak School
Ø Yerevan: School #198 and School #150
Ø Syunik Marz: Goris Schools #Two and Three, and interview with the principal of the Boarding School for Children with Visual Impairment, Boarding School #27, and Boarding School #5 for the Mentally Handicapped.
Several UNICEF and Ministry of Education and Science documents served as background information for the report. These are listed in Appendix A.
In most schools, the principal or deputy principal and one or two Life Skills teachers were interviewed, as well as several non-Life Skills teachers. Life Skills classes were observed in four of the schools and students had an opportunity to share their Life Skills experience with the evaluation team.
A final debriefing meeting was held with the Minister of Education on 15 March 2001. This meeting included the Minister, Deputy Minister Topuzyan, Mr. Robert Stepanyan, Mr. Victor Martirosyan, the three national expert team members, Ms. Soukhudyan of UNICEF, and the evaluator.
The Terms of Reference for this evaluation called for a review of the implementation of Life Skills within the structure of the Armenian Ministry of Education and Science and analysis to include:
ü Analyse the pedagogical contributions of the project in the context of the national curriculum reform plans of the Ministry of Education and Science.
ü Analyse the pedagogical contributions of the project in the areas of (a) teaching methodologies and (b) introduction and development of new curriculum areas into core subjects of Armenian basic education.
ü Analyse the impact of the Life Skills education on the classroom environment as compared to standard classroom organization and environment in Armenian schools.
ü Analyse the acceptability of the Life Skills approach on the part of teachers, students, parents, and school administrators.
The program is too new to address the long-term impact sought by the Life Skills curriculum on the participating students and teachers. Short-term results are occurring, but an accurate measure of these results was hampered by the lack of a student and teacher evaluation process. The two-week time frame allotted for onsite data collection severely limited the number of teachers and students who could be interviewed for this report and there was no time to develop, administer and analyze a survey that would have provided some baseline quantitative data. Therefore there was no opportunity to conduct a controlled comparison between the Life Skills classes and traditional subjects and methods, but the evidence gathered for this report does suggest a qualitative difference.
This report is based largely on qualitative and anecdotal evidence about the structure and implementation of the first year and a half of this curriculum. It must be said, however, that the findings identified on short-term outcomes for students and teachers are very positive and there was no evidence to suggest further analysis would change the current findings. On the contrary, the strength of the qualitative data gathered would suggest that findings would be similar across all students and teachers, though we might have been able to identify more clearly the barriers and supports to implementation. One of the major points made in the Recommendations section is the need for an ongoing assessment process for this project.
One document, the Interim Evaluation Package, provided a small amount of quantitative data on the first year of the project. The Interim Evaluation Package was an evaluation done by the Core Team with the 16 pilot schools at the end of the first year of implementation. This data reflected the overall positive qualitative findings of the present report. (See a summary of this report, Appendix B.) Again, the qualitative and anecdotal information on classroom implementation presented here is overwhelmingly positive. Analysis of administrative and structural implementation indicates a mixture of positive and potentially more troublesome findings.
As the majority of the data was qualitative, a narrative analysis method was used to analyze data and determine findings and recommendations. This analysis looked for common themes among the responses given by the interviewees. Progress was also based on comparing current reports on the status of the project with a review of background materials listed in Appendix A and a determination of whether commitments noted in these documents were met.