Program Quality Assurance at
Algonquin College of Applied Arts and Technology
Program Quality Review
Ontario College Credentials
2016-2017
A Guide for Team Leaders
PREFACE
This Program Quality Review (PQR) Team Leader guide is written to aid Program Quality Reviews for Ontario College Certificate, Diploma, Advanced Diploma and Graduate Certificate programs offered at Algonquin College. It serves as the master Team Leader Guide for Program Quality Reviews. Adaptations are provided separately for Apprenticeship programs, Algonquin College Certificates and Bachelor’s degrees going through quality review.
REVISION HISTORY
The PQR Team Leader guide is updated at least annually, Revisions since 2010 are noted below.
Team Leader Guide / Major revisions / Notes2016-2017 / · Update language to note CQAAP standards and terminology, and change from MTCU to MAESD (PQR quality criteria and elements remain essentially unchanged)
2015-2016 / · Data review section and templates updated
· Minor rewordings to Quality Criteria
· Process flowchart updated
2014-2015 Issue 02 (for Winter/Summer 2015) / · Quality criteria re-ordered and modified to align with OCQAS Accreditation Quality Standards specific to Program Quality Review. Minor enhancements made to the Quality Criteria and elements.
2014-2015 Issue 01 (for Fall 2014) / · Note regarding upcoming change to Accreditation process, and guidelines for Fall Team Leaders
· Updates to data access through BI data reporting portal
· Introduction of Curriculum Services
2013-2014 / · Updates to quality criteria elements to include assessment of AODA, and recognition of the deployment of the College-wide Sustainability and Internationalization VLO
2012-2013 / · Notice that documents are retained and distributed electronically
· Addition of process for assigning an external team leader
2011-2012 / · Update to use six quality criteria, based on changes to PQAPA
· Introduction of Data Reporting Portal for data access
· Introduction of new format for Implementation Plan
· Formal inclusion of Sign-Off sheet
· Additional information on data (KPI)
2010-2011 / · Re-formatting of guide
· Introduction of Guide specific to review of Bachelor’s degrees
Contents
PREFACE 2
REVISION HISTORY 3
PROGRAM QUALITY ASSURANCE – AN OVERVIEW 8
PROGRAM QUALITY ASSURANCE AT ALGONQUIN COLLEGE 10
Annual Curriculum Review 10
Program Mix Review 10
Program Quality Review 10
Cyclical Review of Online Elective General Education courses 10
PROGRAM QUALITY REVIEW 11
PQR Goals 11
PQR Process 11
Schedule 11
Roles 12
Program Quality Assurance Administrator 12
Program Quality Review Team Leader 12
Dean 13
Chair (also Academic Manager) 13
Orientation 14
Release Time 14
Fall 2016 or Winter 2017: 14
Timeline 15
Need Assistance? 15
THE FINAL REPORT – AN OVERVIEW 16
The Main Document 16
Executive Summary 16
Introduction 16
Evaluation of the Program against the Criteria 16
Conclusions 16
Recommendations and Implementation Plan 16
The Appendices 17
Appendix A 17
Appendix B 17
Appendix C 17
Appendix D 17
Appendix E 17
DOCUMENT REVIEW 18
Program Monograph 18
Program Promotional Material 19
Program Council Meeting Minutes 19
Program Advisory Committee Meeting Minutes 19
Further Study Pathways 20
Reports from External Accreditation or Licensing Bodies 20
Program (Vocational) Learning Outcomes 20
Program of Study 20
Course Outlines and Course Section Information 20
DATA REVIEW 22
Key Performance Indicators (KPIs): Background Information 22
Student Satisfaction Survey 22
Graduate Outcomes/Graduate Satisfaction Survey 22
Employer Satisfaction Survey 23
Program Performance Tab 23
Key Performance Indicators 23
Registration and Other Data 24
Program Costing Tab 26
CURRICULUM MAPPING 27
Program Standards 27
Vocational Standard 27
Essential Employability Skills 27
General Education Requirement 28
Purposes of Curriculum Mapping 30
Performing Curriculum Mapping 30
Course Outline Review 32
Vocational Learning Outcomes – Reviewing the Map 32
Essential Employability Skills – Reviewing the Map 33
General Education Requirement – Reviewing the Map 33
Curriculum Services 33
EVALUATION AGAINST THE QUALITY CRITERIA 34
Evaluation Criteria Listing 34
Criterion 1 34
Criterion 2 34
Criterion 4 35
Criterion 5 36
Criterion 6 – 36
Responding to the Evaluation Criteria 37
Evaluation Criteria Listing – with help text 38
CONDUCTING FOCUS GROUPS 58
Memos from the Chair and Team Leader 58
Faculty Education, Experience, and Professional Development Survey 58
Program Self-Audit 59
Course Outlines 59
Setting the Focus Group Agendas 59
Tips for Planning and Facilitating Focus Groups 59
Preparing and Planning for the Meeting 59
Facilitating the Meeting 59
Faculty Focus Group 61
Facilitating the focus group 61
Preparing the focus group report 62
External Stakeholder Focus Group 63
Planning the External Stakeholder Focus Group 63
Time Frame for the Meeting 63
Leadership of the Meeting 63
Ground Rules for the Meeting 63
Writing the external stakeholder focus group report 64
Possible Items for Discussion 64
Student Focus Group 67
Planning the Student Focus Group 67
Time Frame for the Meeting 67
Leadership of the Meeting 67
Ground Rules for the Meeting 67
Writing the Student Focus Group Report 68
Possible Items for Discussion 68
COMPLETING THE FINAL REPORT – AN OVERVIEW 73
ANNUAL FOLLOW UP 74
GLOSSARY AND LIST OF ACRONYMS 75
APPENDIX A – THE FINAL REPORT 76
APPENDIX B – THE CREDENTIALS FRAMEWORK 83
APPENDIX C – TEMPLATES 84
Introductory Email from the Chair 85
Introductory Email from the Team Leader 86
Faculty Education, Experience, and Professional Development Survey 87
Program Quality Review - Program Self-Audit 91
Course Outline Check-Up Review Tool for PQR 93
Course Outline Review Summary Sheet 94
Curriculum Mapping - Terms and Concepts 95
Analysis and Review of VLO Maps for PQR 96
Analysis and Review of EES Maps for PQR 99
Analysis and Review of General Education Maps and Courses for PQR 101
Faculty Focus Group – Sample Agenda 103
Sample Invitation to External Stakeholders 104
External Stakeholder Focus Group - Sample Agenda 105
Student Focus Group – Sample Invitation 106
Student Focus Group - Sample Agenda 107
Thank You Note to Focus Group Participants - Sample 108
PQR Submission Sign Off Form 109
PROGRAM QUALITY ASSURANCE – AN OVERVIEW
One of the prerequisites for granting greater autonomy to the Colleges of Applied Arts and Technology in Ontario to govern their own programs was the implementation of a process to assure program quality. The Ontario Colleges of Applied Arts and Technology Act, 2002 outlined two elements of this expectation:
· quality assurance at the individual college level, and
· a self-regulating process at the system level
as noted within the Minister’s Binding Policy Directive, Framework for Programs of Instruction. The Program Quality Review process forms a major part of adherence to providing quality assurance at the College level, and is therefore essential to College operations.
At the system level, that is, at the level which encompasses all Colleges of Applied Arts and Technology (CAATs) in Ontario, a separate quality assurance process takes place, the College Quality Assurance Audit Process, CQAAP (formerly the Program Quality Assurance Process Audit, PQAPA). This audit is conducted through the Ontario College Quality Assurance Service, OCQAS[1]. OCQAS was established to provide effective and efficient mechanisms that ensure specific program quality and consistency standards are met by the CAATs in Ontario.
The following six quality standards describe the expectations of the OCQAS regarding quality assurance; the PQR evaluation criteria relate to these standards, and also encompass College specific requirements.
Quality Standard 1 (Program Quality Management System)
Effective quality assurance mechanisms ensure the quality of a program management system and demonstrate continuous improvement.
Quality Standard 2 (Program Development)
Effective quality assurance mechanisms ensure the quality of the programs of study being developed and demonstrate continuous improvement.
Quality Standard 3 (Conformity with Government Requirements)
Effective quality assurance mechanisms ensure the conformity of programs of study with relevant government requirements and demonstrate continuous improvement.
Quality Standard 4 (Program Delivery and Student Assessment)
Effective quality assurance mechanisms ensure the quality of program delivery and student assessment, and demonstrate continuous improvement.
Quality Standard 5 (Existence, Monitoring and Communication of Academic Policies and Practices)
Effective quality assurance mechanisms ensure the communication and monitoring of established academic policies and practices related to academic issues that support program implementation and delivery, and student achievement of vocational learning outcomes, and demonstrate continuous improvement.
Quality Standard 6 (Availability and Allocation of College-wide Resources)
Effective quality assurance mechanisms ensure the existence, availability and allocation of resources (human, physical, financial) and technological infrastructure to support student achievement of program vocational learning outcomes, and demonstrate continuous improvement.
For more information regarding quality assurance at the provincial level, please visit the OCQAS Quality Assurance website.
PROGRAM QUALITY ASSURANCE AT ALGONQUIN COLLEGE
Program Quality Assurance at Algonquin College consists of three processes:
· Annual Curriculum Review
· Annual Program Mix Review
· Cyclical Program Quality Review
Additionally, there is a cyclical review of online elective General Education courses.
Annual Curriculum Review
Each year program faculty members revise and update curriculum to maintain its currency and to ensure that graduates will have the skills necessary to be successful in the workplace.
Feedback from faculty, students, and advisory committee members is taken into consideration before updates are made to keep the program current. Revisions to the curriculum and the program narrative information are made in GeneSIS, the student information system, and are subsequently reviewed in the Office of the Senior Vice President Academic. Once the changes are approved, a new program version is created. The information in GeneSIS becomes the foundation for course loading, fees loading, scheduling, registration and College publications.
Program Mix Review
Staff in the Academic Operations and Planning department compile data that reflect the Quality Measures for a program. Admissions and registration data, employment rates, capstone questions from Key Performance Indicators (KPI) and program costing data are reviewed. Programs not meeting college benchmarks are noted and strategies are developed by the Chair and Dean to improve the quality of the program and/or the financial contribution.
Program Quality Review
Program Quality Review (PQR) ensures that each program goes through a comprehensive review every five years. The PQR process builds on annual review activities by including analysis of three to five years of program data, and soliciting direct feedback from stakeholders and students into the program review. Each PQR results in an implementation plan containing resourced actions to maintain and improve the program over the next five years.
This manual guides Team Leaders when conducting a Program Quality Review.
Cyclical Review of Online Elective General Education courses
Online General Education courses form an integral part of many programs offered at the College. Each online General Education elective course in the College pool is reviewed according to a published cycle, first by the General Education committee for overall adherence of the course to General Education objectives, and then in detail by individual faculty and student reviewers.
PROGRAM QUALITY REVIEW
PQR Goals
· To assess the program against established quality criteria in order to determine whether the program meets or exceeds College and Ontario College Quality Assurance Service (OCQAS) standards (for Ontario College Credentials) or Postsecondary Education Quality Assessment Board (PEQAB) standards (for Bachelors’ Degrees).
· To establish and implement a quality improvement plan[2] for the program, in response to the findings of the review.
PQR Process
Program Quality Review follows a systematic process, and is outlined in the Program Quality Assurance Framework graphic. After the Team Leader has been identified to conduct the review, information pertinent to the program is collected and analyzed, allowing an initial evaluation of the program against established quality criteria. The program information reviewed includes published and internal documentation, program data, current curriculum maps, and input collected from key players in the program – faculty, students, and external stakeholders.
Once the review is complete, the evaluation against the criteria is finalized, and all findings and recommendations for improvement are captured in a report. An implementation plan for the recommendations is drawn up, with an update on the actions provided to the PQAA at least annually until completion.
The PQR Team Leader conducts the PQR, with assistance from the Chair and the PQAA.
Note that this guide is written to provide assistance with each of the main components of PQR:
· Document Review
· Data Review
· Curriculum Mapping
· Evaluation Against the Quality Criteria
· Conducting Focus Groups
· Completing the Final PQR Report
Schedule
Programs are reviewed according to a defined schedule, normally every five years. The schedule is revised annually, in consultation with the Academic Departments, and aligned with other Quality Assurance requirements, for example, accreditations or consent renewals for bachelor’s degrees.
Roles
Program Quality Assurance Administrator
Program Quality Reviews are coordinated by the Program Quality Assurance Administrator (PQAA), who reports to the Dean, Academic Development, who in turn reports to the Senior Vice President Academic (SVPA).
The responsibilities of the Program Quality Assurance Administrator include
· ensuring the College is compliant with Ministry requirements regarding Program Quality Assurance.
· consulting with major academic constituencies College-wide regarding Program Quality Review.
· preparing documents related to Program Quality Review.
· planning and organizing orientation/training workshops for Team Leaders.
· supporting the Program Quality Review process in programs.
· meeting regularly with PQR Team Leaders during the review process to monitor progress and provide guidance.
· facilitating evaluation of the Program Quality Review process.
· ensuring that the final report is signed off fully by the PQAA, Chair and Dean
· arranging archiving of the final report; forward e-copies to the TL, Chair, Dean, and providing access to all reports to the SVPA.
Program Quality Review Team Leader
The PQR Team Leader is normally a faculty member who is assigned to conduct Program Quality Review for the program in which s/he teaches.
The responsibilities of the Team Leader include
· attending meetings scheduled by the Program Quality Assurance Administrator.