April 8, 2010: Drafted by Perry Deess, Fadi Deek, and Norbert Elliot

May 11, 2010: Presented to President Altenkirch, Provost Gatley; Committee for Department and Program Assessment (Steve Tricamo, Chair as Associate Provost, Joel Bloom, Sunil Saigal, Urs Gauchat, Fadi Deek, Narain Gehani, Bob English, Rich Sweeney, Jack Gentul); and Sub-Committee on Assessment (Perry Deess, Co-chair as IRP Director, John Bechtold [Member as Master Teacher], James Geller, Norbert Elliot [Member as MSCHE Chair], John Cays, Shanti Gopalakrishnan, Bruce Bukiet, Ala Saadeghvaziri, Barry Cohen)

Updated: September, 2012: Perry Deess, Judy Redling

Updated Guidelines reflect Committee structure changes, edits to Table of Contents (including Part III-A-iv, details on external assessment plan), and additional requirements for the Appendix: syllabi for core courses (with states learning outcomes) and NJIT Quality in E-learning rubric for all online courses.

NJIT Program Review Process: Guidelines

Background

The enhanced program review process is responsive to increased demands for accountability. A synthesis of program review guidelines from accreditation agencies such as the Accreditation Board for Engineering and Technology, the Association to Advance Collegiate Schools of Business, the National Architectural Accrediting Board, and the Middle States Commission on Higher Education—as well as university strategic planning initiatives—the program review process allows both transparency and collaboration: In reviewing all NJIT program, the process allows NJIT curriculum directors to learn from each other and to create innovative strategies for achieving enhanced program outcomes.

Introduction

Beyond a mere audit, the Program Review Process has four innovative objectives: 1) To provide a forum for the assessment and improvement of all degree and university programs; 2) To demonstrate continuous improvement in the delivery of educational curricula; 3) To promote a culture of assessment by building a cohesive assessment strategy; 4) To create a central, web-based repository for assessment design and supporting documents. These four objectives will yield a university-wide forum for the following: 1) collaboration on establishing program vision, strategies, tactics, and metrics; 2) investigation of new ways of understanding student achievement; 3) collaboration on common ways of reporting student outcomes; and 4) creation of a web-based system of promoting program assessment models.

While all degree-granting programs will participate in program review to advance the key objectives, programs accredited through the Accreditation Board for Engineering and Technology, the Association to Advance Collegiate Schools of Business, and the National Architectural Accrediting Boardwill not be required to complete a separate program review report. Program directors, working with a review team within their academic units,will be asked to submit their existing accreditation reports with key sections highlighted that address the NJIT program review guidelines. Program directors will also submit a brief cover memo, of no more than 2 pages, identifying the key sections and describing changes made in response to the accreditation self-study. The reason for requesting review of accredited programs is to allow a sense of university-wide collaboration, to study new ways of coming to terms with the complexities of student achievement, to encourage collaboration on strategies of reporting student outcomes, and to create a new model by which we may promote our innovative assessment models.

Vision

The NJIT Program Review Process will answer the following questions:

  • How are institutional and program level goals integrated and articulated?
  • How have planned assessment processes been implemented?
  • How do assessment results provide convincing evidence of student learning?
  • How do program administrators use assessment in decision making?
  • Whatchanges have been made based on assessment?
  • How have program assessment results been communicated to shareholders?
  • What is the cohesive framework of program assessment at NJIT?

Governance and the Program Review Process

The NJIT Program Review Process is governed by a standing university committee, The Committee on Assessment of Student Learning, reports to the Deans’ Council (the Provost and deans from Newark College of Engineering, the College of Architecture and Design, the School of Management, the College ofComputing Sciences, the College of Science and Liberal Arts, and the Albert Dorman Honors College. Committee members include the Associate Provost for Academic Affairs, The Director of Institutional Research and Planning, Associate Provost for Graduate Studies, Associate Deans, identified faculty, and identified administrators (UCRC Chair, TLT Chair, Graduate Council Chair).

The process begins with the selection of the university’s degree programs to move through review on a five-year rotating basis. That is, the process calls for approximately22 programs to be reviewed each year. Once these programs are identified by the Committee on Assessment of Student Learning within the five year cycle, selected programs will work collaboratively with the assessment team in Institutional Research and Planningto add available measures to the existing review strategy.

Program directors will then develop an analysis focusing on student learning outcomes. Directors will include a detailed description of how data have been used to make improvements in the past and how they will use the self-study process to drive future improvements.Once the report is ready, the Committee on Assessment of Student Learning will undertake the review on behalf of the Dean’s Council.

After the program review is completed, the Committee makes one of three levels of recommendations: an exemplary recommendation; a qualified program recommendation; or review for monitoring. If a program receives an exemplary recommendation, a concerted effort will be made to strengthen other degree programs by advocating process similar to those practiced by the exemplar program. If a program receives a qualified program recommendation, a concerted effort will be made to identify the standards of achievement of the qualified program so that all programs may achieve similar standards; if a program needs further monitoring, the committee will document the case, work with the program director to improve program effectiveness, and report that process to the Committee for Assessment of Student Learning.The Committee may then refer the program to the provost for further action.

Timeline

The timeline consists of five phases that begin with the agenda established by the Committee on Assessment of Student Learning.

1.Initiate Process

Operating within the five year cycle, the annual process begins with a review of the yearly roster of the programs to be reviewed in the five year process. Concurrently, the NJIT Office of Institutional Research and Planning plans to provide technical assistance to directors of the programs to be reviewed. The fall process also includes the charge to the Committee.

  1. Orientation of the Program Directors

Members of the Committee orient the identified program director to the review process. The orientation consists of a review of guidelines, a review of procedures, and an overview of resources available to assist the director.

3. Assemble Data

The Office of Institutional Research and Planning provides the program director with the necessary data to needed to complete or enhance the program review document.

4. Submit the Report

With the support of Institutional Research and Planning, the program director submits the report to the Committee.

5. Committee Review

The Committee reviews the report and makes recommendation with a 1 page summary of program strengths and weaknesses to the Dean’s Council.

6. The Deans’ Council Makes Final Recommendations

The Deans’ Council makes final recommendations and issues a brief report citing program strengths and weaknesses to the provost.

The timeline is described in the figure below:

Program Review Process: Timeline of Annual Activities


Charge to Program Directors

In preparing the self-study, the program and the academic unit’s program review team director should keep twogoals of this process in mind:

  1. To ensure that NJIT is offering quality academic programs (e.g. has appropriate educational objectives, and is accomplishing those objectives as evidenced by student learning outcomes assessment)
  2. To demonstrate continuous improvement in the delivery of the educational curriculum.

In preparation for the review, the program director should review the following:

•Program oversight and management

•Program objectives

•Core course objectives

•Relationship between curriculum and objectives

•Resources—human, physical, financial—required to support the curriculum

•Evaluation student learning outcomes according to a previously established assessment plan

•Measurement of student performance in courses

•Measurement of student performance across courses

•General measurement of student learning outcomes

•Measurement of student attitudes

•Obstacles to meeting objectives

•Continuous curricular improvement

•Signature features, such as undergraduate research experience

•Submission of the Institute Information Literacy Report

•If eLearning is involved, review the Institute eLearning Quality Assurance Report

Charge to Institutional Research and Planning

The Office of Institutional Research and Planning will provide each director with the following information:

  • Graduation rate
  • Retention rate
  • Time to degree
  • Five year enrollment
  • Most recent diversity measures
  • Average GPA by student level

Collaboration on the NJIT Program Reporting Process

Program directors, members of the academic unit review team, and representatives from Institutional Research and Planning will work together to assemble the following evidence related to program review.

  • Describe relationship of program outcomes and educational objectives;
  • Describe the processes used to assure that graduates have achieved the program outcomes: identify direct and indirect measures of the learning outcomes.
  • Provide qualitative and quantitative data to demonstrate that graduates satisfy planned program outcomes;
  • Describe how the assessment results are implemented to improve the program.

The NJIT Program Report Document

The NJIT program report provides an overview of the program in general and it’s the assessment of student learning outcomes in particular. It directly addresses the twin objectives of assessment and continuous improvement by demonstrating how the program meets its own educational objectives and how it has made changes to improve the program. The program review report should employ Middle States terminology, unless the program has external accreditation. For additional assistance see the Middle States guide to assessing student learning:

For a more comprehensive discussion see:

Programs participating in an external accreditation process such as ABET, NAAB, or AACSB will be asked to submit existing reports with a detailed cover letter and relevant sections marked.

An additional program review report will be scheduled for the assessment of student learning outcomes in the GUR. This report will follow a similar model, but with a different structure because it involves the assessment of learning across the curriculum.

Definitions:

Institutional Learning Goals:

As students seek disciplinary mastery, they will achieve skills and knowledge in:

  • Research-based Inquiry: Students employ investigative methods
  • Ethical Conduct: Students understand professional and civic responsibility
  • Economic Opportunity: Students understand economic reasoning and demonstrate that they are able to allocate resources effectively and logically under operating constraints
  • Collaboration: Students work effectively in teams to engage multidisciplinary perspectives
  • Engagement: Students are active and committed learners

Program Learning Goals:

Statements that describe what students are expected to know and be able to do by the time of graduation. These relate to skills, knowledge, and behaviors that students acquire as they progress through the program.

Course Learning Goals:

Formally defined expectations for what students should know and be able to do after they complete a course. These relate to skills, knowledge, and behaviors that students acquire as they progress through the course.

Learning Outcomes:

The knowledge, skills, and abilities that students have attained as a result of their involvement inprogram based educational experiences.

Assessment of student learning:

Assessing student achievement employs evidence from student courses, projects, activities, or performances to determine the extent to which student learning achieves the stated program goals

Sample Table of Contents

I. General program information (2 page)

  1. Program background information:
  2. Specify program title(s): all degrees awarded for the program under review, including options, as specified in transcripts;
  3. Indicate program mode: (e.g., face-to-face or distance learning; percentage of students enrolled part-time).
  4. Program details: discuss metrics and implications for program, including how data have been used to make improvements in the past and how they will use the self-study process to drive future improvements.
  1. Graduation rate
  2. Retention rate
  3. Time to degree
  4. Five year enrollment
  5. Most recent diversity measures
  6. Average GPA by student level

II Program mission and goals (1 page)

  1. Describe program mission and program learning goals
  2. Provide a matrix showing how the institutional learning goals are mapped to the Program Learning Goals (See Table 1)

Table 1. Matrix of Institutional Learning Goals and Program Goals

Institutional Learning Goal 1: Researched-based Inquiry / Institutional Learning Goal 2: Ethical Conduct / Institutional Learning Goal 3: Economic Opportunity / Institutional Learning Goal 4: Collaboration / Institutional Learning Goal 5: Engagement
Program Learning Goal:1
Program Learning Goal: 2
Program Learning Goal: ETC.

III Learning outcomes (4 pages)

  1. Discuss how the studentachievement of program learning goalsis assessed
  2. Describe the program assessment plan:

How are student learning outcomes assessed over the course of the program? Indicate measures of student learning and criterion-based standards (rubrics) against which learning is evaluated.

  1. If the assessment plan includes assessment at the course level, how are course learning outcomesrelated to the achievement of program learning goals.(see Table 2). Syllabi with learning outcomes for all core courses should be provided in the appendix.
  2. If the assessment employs portfolio analysis or the application of rubrics to specific student work describe the process, including how standards are defined.
  3. Provide details on external assessment plan for comparative quality assurance: discipline-specific exam, external review of final project, portfolio, or exam.
  4. For programs where there are no well defined core courses, assessment should be based on a senior project, thesis, qualifying exam, publications or dissertation.
  5. For undergraduate programs, identify core courses with an information literacy component. Describe the information literacy assessment process and results.
  6. Undergraduate programs should address the contribution of the GUR to learning outcomes in the program. (For GUR program review guidelines see Appendix 1)
  7. If students can enroll in a online courses, include in the appendix acompleted NJIT Quality in eLearning Rubric for each course:

Table 2. Matrix of Program Learning Goals to Core Course Goals

Program Learning Goal:1 / Program Learning Goal:3 / Program Learning Goal: ETC.
Core Course 1
Goals
Core Course 2
Goals
Core Course 3 Goals, etc.

IV Using student learning assessment (1 page)

  1. Discuss the results of the assessment and to what extent these results provide evidence that program learning goals have been achieved.
  2. Describe what specific program strengths and weaknesses are revealed by the assessment of student learning outcomes.
  3. Show how the assessment results are being used to implement changes. (What are those changes?)

V. Assessment in the next cycle (1 page)

A. Discuss the sustainability of assessment and how it is integrated into the program.

B. Describe planned improvements in the assessment process.

VI Conclusion (1 page)

  1. Summary
  2. Future improvements

Appendices—Supporting Documents (possible examples shown below)

A)Syllabi with learning outcomes for all core courses

B)Program assessment supporting data including rubrics

C)Summary of test results

D)Summary of ePortfolio evaluation

E)NJIT Quality in E-learning rubrics and evaluation report

F)Presentations and publications associated with student learning assessment

Implementation

The guidelines described in this document are effective for program review reports written after Spring 2012and will remain in effect until superceded by a new set of guidelines.

APPENDIX I

ASSESSMENT OF THE GENERAL UNIVERSITY REQUIREMENTS: A fundamental guiding principle in the development of the General University Requirements (GUR) is the formulation of a foundational curriculum encompassing the necessary preconditions for success in undergraduate disciplines, a curriculum that establishes a devotion to lifetime intellectual discovery and personal development. Learning Outcomes relevant to this guiding principle are articulated in the NJIT Core Competencies. Assessment of the Core Competencies is achieved through the use of locally-developed methods and nationally-based tests.

The NJIT Program Review Process for GUR will answer the following questions:

  • What learning outcomes are specific to a thematic area within the GUR?
  • Which Core Competencies are addressed? How is a given Core Competency integrated and articulated in the GUR thematic area?
  • How have planned assessment processes been implemented?
  • How do assessment results provide convincing evidence of student learning?
  • How do program administrators use assessment results in decision making?
  • How have actions been taken as a result of assessment?
  • How have assessment results been communicated to a variety of shareholders?

Review Process

The GUR program review report will be prepared by the Program Review Committee as a whole, based on separate contributions from each of the disciplines offering GUR courses. The review will follow the overall structure discussed in the Program Review Guide, but with separate sections for each discipline and will cover all standard GUR courses in the GUR disciplines. The process of aligning and assessing goals from the institutional goal to the program goal, to the discipline goal, to the course level will be followed with an additional focus on the extent to which core competency benchmarks have been achieved.