Program Quality Assurance at
Algonquin College of Applied Arts and Technology
Program Quality Review
Bachelor’s Degrees Supplement
2014-2015
A Guide for Team Leaders
PREFACE
All programs at Algonquin College are reviewed on a cyclical schedule following the Program Quality Review (PQR) process. A master Team Leader Guide directs the comprehensive review, and is located at http://www.algonquincollege.com/academic-development/program-quality-review-ontario-college-credentials/. Program review includes data and document review, conducting focus groups with key program stakeholders, and an assessment of the program against established quality criteria.
For Bachelor’s degrees, the final PQR report, also referred to as the ‘program self-study’, forms part of the documentation pertaining to Postsecondary Education Quality Assessment Board (PEQAB) degree consent renewal. The self-study is provided to members of an external Program Evaluation Committee, who independently carry out a quality assessment of the Bachelor’s Degree program. PEQAB requires that “The self-study should be conducted in accordance with Benchmark 1 a) to k) of the Program Evaluation Standard”. The PQR process and quality criteria elements encompass the PEQAB Program Evaluation Standard; however for clarity and ease of review an insert directly mapping to the PEQAB benchmarks is included in the final PQR report for Bachelor’s degrees.
Documentation and guidelines pertaining to any degree consent renewal, including the external Quality Assessment, are available from the Program Quality Assurance Administrator.
REVISION HISTORY
The master PQR Team Leader guide is updated at least annually. Revisions specific to Bachelor’s degrees are noted below.
Team Leader Guide / Major revisions / Notes2014-2015 Issue 02 (for Winter/Summer 2015) / · Explicit insert of PEQAB Program Evaluation Standard
· Updated PQR Quality Criteria number, to align with OCQAS Accreditation Quality Standards.
2011-2012 / · Update of dedicated PQR Team Leader Guide for Bachelor’s degrees
2010-2011 / · Introduction of Guide specific to review of Bachelor’s degrees
Contents
PREFACE 2
REVISION HISTORY 3
STRUCTURE OF THIS SUPPLEMENTAL GUIDE 9
PROGRAM QUALITY ASSURANCE – AN OVERVIEW 11
Program Quality Assurance Process Audit (PQAPA); Accreditation Process 11
Program Quality Review Quality Criteria 12
PROGRAM QUALITY ASSURANCE AT ALGONQUIN COLLEGE 14
Annual Curriculum Review 14
Program Mix Review 14
Program Quality Review 14
Cyclical Review of Online Elective General Education courses 14
PROGRAM QUALITY REVIEW 15
PQR Goals 15
PQR Process 15
Schedule 15
Roles 17
Program Quality Assurance Administrator 17
Program Quality Review Team Leader 17
Dean 18
Chair 18
Orientation 19
Release Time 19
Fall 2014 or Winter 2015: 19
Timeline 20
Need Assistance? 20
THE FINAL REPORT – AN OVERVIEW 21
The Main Document 21
Executive Summary 21
Introduction 21
Findings (Evaluation of the Program against the Benchmarks/Quality Criteria) 21
Conclusions 21
Recommendations and Implementation Plan 21
The Appendices 22
Appendix A 22
Appendix B 22
Appendix C 22
Appendix D 22
Appendix E 22
DOCUMENT REVIEW 23
Program Monograph 23
Program Promotional Material 24
Program Council Meeting Minutes 24
Program Advisory Committee Meeting Minutes 24
Further Study Pathways 25
Reports from External Accreditation or Licensing Bodies 25
Program Learning Outcomes 25
Program of Study 25
Course Outlines and Course Section Information 25
DATA REVIEW 27
Registration Data (Tab: Program Performance) 27
Registration Milestones by Program (PQR1000) 27
Application / Registration Ratio (pending, to be deployed in PQR1000): 27
Intake / Projected Enrolment (PQR1000) 27
Program Withdrawals (PQR1020) 27
OCAS Program Choice (PQR1040) 28
Student Success Data (Tab: Program Performance) 28
Course Failure Rate (PQR1030) 28
Course Assessment Roll Up (not currently published) 29
Student, Graduate and Employer Satisfaction Data (Tab: Program Performance) 29
KPI Student Satisfaction – Capstone Questions (KPI0001) 29
KPI Student Satisfaction – All Questions (KPI0004) 30
KPI Graduate & Employer Satisfaction – Capstone Questions (KPI0002) 30
KPI Graduate & Employer Satisfaction – Detailed Responses (pending) 31
Net Contribution Data (Tab: Program Costing) 31
Costing Year Detail by Program (ACA1000) 31
Additional 31
Default rate on Ontario Student Assistance Program (or other student loan) 31
CURRICULUM MAPPING 32
Degree Level Standard 32
Program Learning Outcomes 32
Purposes of Curriculum Mapping 34
Performing Curriculum Mapping 34
Curriculum Maps 34
Course Outline Review 37
Reviewing the Curriculum Maps 37
Note on PEQAB Degree Consent Renewal 38
Curriculum Services 38
PROGRAM EVALUATION AND FINDINGS 39
PEQAB Program Evaluation Standard 39
Program Evaluation – Program Quality Review Criteria 47
Evaluation Criteria Listing 47
Responding to the Evaluation Criteria 50
Evaluation Criteria Listing – with help text 51
CONDUCTING FOCUS GROUPS 71
Memos from the Chair and Team Leader 71
Faculty Education, Experience, and Professional Development Survey 71
Program Self-Audit 72
Course Outlines 72
Setting the Focus Group Agendas 72
Tips for Planning and Facilitating Focus Groups 72
Preparing and Planning for the Meeting 72
Facilitating the Meeting 72
Faculty Focus Group 74
Facilitating the focus group 74
Preparing the focus group report 75
External Stakeholders Focus Group 76
Planning the External Stakeholders Focus Group 76
Time Frame for the Meeting 76
Leadership of the Meeting 76
Ground Rules for the Meeting 76
Writing the external stakeholders focus group report 77
Possible Items for Discussion 77
Student Focus Group 80
Planning the Student Focus Group 80
Time Frame for the Meeting 80
Leadership of the Meeting 80
Ground Rules for the Meeting 80
Writing the Student Focus Group Report 81
Possible Items for Discussion 81
COMPLETING THE FINAL REPORT – AN OVERVIEW 86
ANNUAL FOLLOW UP 87
GLOSSARY AND LIST OF ACRONYMS 88
APPENDIX A – THE FINAL REPORT 89
APPENDIX B – THE ONTARIO QUALIFICATIONS FRAMEWORK 96
APPENDIX C – TEMPLATES 97
Introductory Memo from the Chair 98
Introductory Memo from the Team Leader 99
Faculty Education, Experience, and Professional Development Survey 100
Summary of Faculty Members’ Credentials and Experience 101
Program Quality Review - Program Self-Audit 102
Course Outline Review Tool (Bachelor’s Degrees) 110
Mapping: Degree Level Learning Outcomes to Degree Level Standards 113
Mapping: Courses to Program Learning Outcomes 114
Course Outline Review Summary Sheet 115
Curriculum Mapping - Terms and Concepts 116
Analysis and Review of Program Learning Outcome Maps for PQR 117
Faculty Focus Group – Sample Agenda 119
Sample Invitation to External Stakeholders 120
External Stakeholder Focus Group 121
Student Focus Group – Sample Invite 122
Student Focus Group - Sample Agenda 123
Student Focus Groups – Sample Purpose and Objectives 124
Purpose of the Student Focus Group 124
Thank You Note to Focus Group Participants - Sample 125
PQR Submission Sign Off Form 126
APPENDIX D – Program Data 127
Key Performance Indicators (KPIs) 127
Student Satisfaction Survey 127
Graduate Outcomes/Graduate Satisfaction Survey 127
Employer Satisfaction Survey 127
Algonquin College Performance Measures 128
Application/Registration Ratio: 128
Intake/Projected Enrolment: 128
KPI Graduation Satisfaction: 128
KPI Total Employment: 128
KPI Related Employment Rank: 128
KPI Employer Satisfaction: 128
Algonquin College Quality Measures 129
KPI Student Satisfaction 129
Instructional Quality 129
Quality Average: 129
School Average: 129
Benchmark: 129
APPENDIX E – COURSE ASSESSMENT SURVEY QUESTIONS 130
Paper-based Survey – 2009 and previous 130
Electronic Survey – 2009-2011 131
Electronic Survey – 2011-2012 133
Electronic Survey – 2012-13 134
Electronic Survey – 2013-14 135
STRUCTURE OF THIS SUPPLEMENTAL GUIDE
The supplemental guide for Bachelor’s degrees is compared against Issue 2.0 of the Master Program Quality Review Team Leader Guide, 2014-2015.
Team Leader Guide Section / Comparison with Master Team Leader GuideProgram Quality Assurance – an Overview / As noted in the preface, for Bachelor’s degrees, an explicit evaluation against each element of the PEQAB Program Evaluation standard is made, with reference to specific PQR quality criteria elements.
Program Quality Assurance at Algonquin College / The cyclical review of online elective General Education courses is out of scope for Bachelor’s degrees. However, a commensurate item is the Breadth component of the Bachelor’s degree.
Whether the review of the breadth file is required for the PQR/self-study will be determined as each program reaches the review stage.
Program Quality Review / The process outlined in figure 1 in the master Team Leader Guide is augmented to reflect the additional review by the Program Evaluation Committee.
The Final Report – an Overview / The final report template is augmented with a section dedicated to the Evaluation of the program against the PEQAB Program Evaluation Standard Benchmarks. This section references forward to related PQR Quality Criteria Elements.
As noted in the master guide, the nomenclature surrounding program learning outcomes, and the curriculum mapping section is adjusted to reflect review of a Bachelor’s degree.
Document Review / Program Learning Outcomes section updated to reflect degree specific language.
Data Review / OSAP default rate data (or equivalent) must also be reviewed. These data are obtained from the Financial Office and supplied to the Team Leader by the PQAA.
Curriculum Mapping / An updated section is provided reflecting degree-specific nomenclature and program structure.
Program Evaluation and Findings / An additional section detailing expectations of the PEQAB Program Evaluation Standard, and mapping it to the Quality Criteria elements is provided. This section is also included in the PQR Template.
Minor updates are made to the PQR Criteria and elements thereof to reflect degree–specific language as required.
Conducting Focus Groups / Language modified to reflect degree-specific terms (i.e., program learning outcomes)
Completing the Final Report – an Overview / No updates required
Annual Follow Up / No updates required
Glossary and List of Acronyms / Additional of PEQAB
Appendix A – The Final Report / Inclusion of reference to Program Evaluation Standard
Appendix B – The Ontario Qualifications Framework / No updates required
Appendix C - Templates / Slight modifications as required to reflect degree specific language
Appendix D – Program Data / No updates required
Appendix E – Course Assessment Surveys / No updates required
PROGRAM QUALITY ASSURANCE – AN OVERVIEW
Program Quality Assurance Process Audit (PQAPA); Accreditation Process
One of the prerequisites for granting greater autonomy to the Colleges of Applied Arts and Technology in Ontario to govern their own programs was the implementation of a process to assure program quality. The Ontario Colleges of Applied Arts and Technology Act, 2002 outlined two elements of this expectation:
· quality assurance at the individual college level, and
· a self-regulating process at the system level.
The Program Quality Review process forms a major part of adherence to providing quality assurance at the College level, and is therefore essential to College operations.
At the system level, that is, at the level which encompasses all Colleges of Applied Arts and Technology in Ontario, a separate quality assurance process takes place, the Program Quality Assurance Process Audit (PQAPA). This is run through the Ontario College Quality Assurance Service (OCQAS[1]). OCQAS was established to provide effective and efficient mechanisms that ensure specific program quality and consistency standards are met by the Colleges of Applied Arts and Technology (CAATs) in Ontario.
Building on the PQAPA, Ontario’s Colleges are moving to an accreditation process effective September 2015. In preparation for this transition, six Accreditation Quality Standards have been developed from current PQAPA quality criteria, and approved in March 2015 by the Management Board of OCQAS. The essence of the PQAPA criteria has been retained, although the ordering/number has changed.
Quality Standard 1 (Program Quality Management System)
Quality assurance processes ensure program development, program review, and on-going program quality assessment involve faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement.
Quality Standard 2 (Existence and Communication of Policies and Practices)
Quality assurance processes ensure the existence and communication of effective policies and practices related to academic issues that support program development and student achievement of program learning outcomes. This would include, for example, admission, credit for prior learning, promotion, graduation, and other related academic policies.
Quality Standard 3 (Program Design)
Quality assurance processes ensure that: program-level learning outcomes are established for all program of instruction; they are consistent with the programs’ intended purposes; and, they are appropriate for the credentials offered upon successful completion of the programs.
Quality Standard 4 (Program Delivery and Student Assessment)
Quality assurance processes ensure that methods of program delivery, including the design, development and execution of teaching and learning activities and student evaluation strategies, are consistent with the program learning outcomes.
Quality Standard 5 (Conformity with Government Requirements)
Quality assurance processes ensure that all programs of instruction conform to current government policy[2] related to the design and delivery of programs of instruction, are consistent with accepted college system nomenclature / titling principles, and maintain relevance.
Quality Standard 6 (Availability and Allocation of College-wide Resources)
Quality assurance processes ensure the existence and availability of human resources, physical resources, financial resources, student support resources, and technological infrastructure to support student achievement of program learning outcomes wherever and however they are delivered.
For current information on the transition to the Accreditation model, please visit the OCQAS web page at http://www.ocqas.org/en/?p=5014.
Program Quality Review Quality Criteria
Based on the above Accreditation Quality Standards, PQR Quality Criteria have been updated and are noted below. These PQR Quality Criteria are effective Winter 2015.
PQR Quality Criterion 1 Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.
PQR Quality Criterion 2 Admission, credit for prior learning, promotion, graduation, and other related academic policies support student achievement of program learning outcomes.
PQR Quality Criterion 3 Program-level learning outcomes are established for all programs of instruction, are consistent with the programs’ intended purposes, and are appropriate for the credential offered upon successful completion of the program.