Executive Summary on Lit Search 2 Pages

Executive Summary of the Literature Search

Dashboard Systems for Program Evaluation

Background

In the fall of 2002, five statewide community college organizations came together to research, design, and deploy a new system for program evaluation. The purpose of the team is to create an alternative evaluation system for PROE (Program Review of Occupational Education). The Michigan Department of Education funded the project to complete five tasks:

Conduct a literature review of existing dashboards and variables used to monitor instructional activities.
Collect and analyze input from faculty and administrators to identify the critical key data elements that will be used for the pilot dashboard.
Pilot the dashboard at three community colleges (Mid-Michigan, Schoolcraft, and Grand Rapids)
Design a process that will help to use the dashboard in a quality system model.
Coordinate communications about and training on the dashboard.

This report is the findings of the literature review.

Introduction

The dashboard system is a powerful tool that can provide a consolidated view of an organization’s performance, making it easy to:

Take advantage of a balanced scorecard approach to management
Measure and understand an organization’s key performance indicators and performance metrics

As in a car, the dashboard uses indicator lights to provide the “driver” information about performance. In a program dashboard, defined variables regarding programs, services, and finances are displayed with colors to portray the following:

High (green)

Medium (yellow), and

Low (red) performance levels

For example, if enrollment trends decline over a three-year period lower than the pre-set minimum of the accepted range, a red bar will appear on the dashboard. Clicking on the variable name will open the spreadsheet (or database) that contains the specific data for further review and explanation.

The Issues

After some initial research, community colleges in Michigan are proposing a model that would use a dashboard system to track and evaluate career programs. This will help our organizations with the following issues:

Tardy and inaccurate access to key performance indicators.
Inability to spot negative trends early.
Human errors and eliminating duplicate data entry.
Generating detailed reports to show emerging trends.
Identifying operational efficiencies.
Identifying and proactively applying preventive measures.

The Solution

A dashboard will help Michigan Community Colleges to:

Set performance goals at each level.
Track performance indicators.
Establish measures and criteria for monitoring progress.
Identify, track, trend and correct problems.
Design and print performance indicator reports.
Understand our organization's financial health.
Meet regulatory requirements.

Dashboard Users

Senior Leaders

Board of Trustees

Management personnel

Other institutional leaders and decision makers

Provides a snapshot of key performance indicators
Keeps users current on issues related to organizational performance
Increases the timeliness and effectiveness of management decisions

Dashboard Effectiveness

In order for Dashboards to be effective;

Organizations must accurately identify and model linkages between overall performance objectives to the specific metrics often already tracked routinely in the organization.

Regular monitoring of your dashboard “gauges” in order to steer your organization by looking forward instead of reverse

The process requires an annual review of performance indicators and an in-depth program analysis by an external evaluation team every 3-5 years

Dashboard Literature Search Methodology

The literature search was conducted primarily on the Internet.

Dashboard Analysis

The research found supports that dashboards are widely used in the corporate world. Industry examples include the healthcare industry, small business, government, and automotive.

The evaluation tool is making its way into the educational arena. For example Dashboard Lite ( is a free on-line tool that allows you to include your school’s technology situation in a national dashboard evaluation. The dashboard tool is also being used at schools such as Northwest Missouri State University, Fox Valley Technical College, Minnesota Association of Charter Schools, and is being implemented in the Michigan Education Information System (MEIS).

“What better method could Michigan – the automotive capital of the world – use to present its annual school report to its community than a digital dashboard?” - Governor John Engler, 2001 State of the State Address entitled Building the Next Michigan: to make Michigan's public schools the best in the world.Hot Links

The literature search is supported by data from some of the following web sites:

This web site provides a PowerPoint presentation from Dr. Stein of the Missouri Coordinating Board for Higher Education on Quality and Measurement. Slide 14 begins discussion on the uses of dashboard as an evaluation tool. Slide 21-23 defines the dashboard concept. Slide 26-29 provides graphics of a dashboard. Slide 30-on shows performance indicators that the dashboard can measure.

Fox Valley Technical College provides “Facts about the College” on their home page using dashboard key performance indicators.

Dashboard Lite is a web site that offers schools a free and easy way to quickly evaluate their current technology situation across 7 technology-related categories. The data can be compared against schools across the country.

Michigan State University (MSU) provides information on the Steps of Evaluation, Methods of Gathering Evaluation Data, Data Collection techniques, observation, case study, and key informants.

Woodward Equity is a company that specializes in improving the quality of your business through the use of dashboard development.

Press release on Chenango Memorial Hospital implementing a dashboard evaluation system

This web site shows the use of a dashboard to access the state of the U.S. economy

Companies that can help you create dashboards for performance measurement

Franklin Memorial Hospital’s web site provides examples of the Dashboard Indicators of Quality

Literature Search Analysis of Variables Used In Dashboards

Educational Examples

Northwest Missouri

Graduates by Degrees, Majors

Academic Profile Results

Major Field Test Results

Freshman Success

Student Opinionnaires of Teaching Modules

General Education Local Module

Major Field Senior Capstone Module

Alumni Satisfaction module

Placement Data Module

Financial Data Module

Comparative Data for Targets

Student Success (Freshmen Success, Academic Profile, Teacher Education, Major Field, Graduation Rates)

Satisfaction (Student, Faculty, Staff)

Enrollment (Recruitment)

Financials (Appropriations, Capital, E&G Operations, Auxiliary Operations, Fund Balance, Endowment)

Strategic Initiative Achievement (Financial Flexibility, Faculty/Staff Development, Integrating Technology, Teacher Education, Assessment, Communications, Diversity)

Academic Workload

Fox Valley Technical College

Students:

Total Students Served

Program Students Served

Basic Education Students Served

Apprentice Students Served

New Application Conversion Rate

Market Share of New High School Graduates Attending a Technical College

Racial/Ethnic Minority Students

Financial:

Contract Revenue Generated

Operating and Debt Service (Combined Millage Rate)

Instructional Cost per Full-time Equivalent Student

Grant/Project Revenue Generated

Fund Balance/Operating Expenditures

Student Outcomes:

Total Graduates

Employed Graduates Working in Field of Study

Graduates Average Yearly Pay

Degree Students Retained (Fall to Spring)

Students Successfully Completing Courses

Top Five Programs

Degree Students Who Took Basic Skills

Students Transferring Credits into Wisconsin Colleges and Universities

Lansing Community College

PROE Dashboard (All Targets are Set)

Three-Year Enrollment (Unduplicated Headcount, Credit Hours, Contact Hours)

Three-Year Graduates and Placement

Perkins Core Performance Indicators

Summary of Perception of Administrators and Faculty

Summary of Perception of Students

Summary of Perception of Advisory Committee

Samford

Satisfaction with Major Field

Standardized Achievement Tests

Professional Licensing Exam

Graduation Rate

Return from Fall to Fall

Graduation within Four Years

Applicants to Enrollment

Regional Enrollment

Student Opinion Survey

Student Athletic Scholarships

Operating Funds

Endowment Yield

Instruction, Academic Support and Library Expenses

Alumni Giving Rate

Benchmarking Measures (U.S. News & World Report)

Academic Reputation

Mid-point SAT

Persistence Rate

Graduation Rate

% Alumni Giving

% E&G to Instruction

Average Faculty Salary

Satisfaction Index

Corporate Benchmarks

Market Share

Sales

Operating Profit

Inventory

Customer Satisfaction

Cycle Time: Order to Delivery

Employee Retention

New Product Development

Another Example of Corporate Dashboard

Client Count

Complaint Resolution

Brand Recognition

Warranty Claims

Inventory Turnover

Average Years of Service

Gross Margin

Share Price

Profit Per Share

Training