Staff Recommendations for Revisions to Scored Indicators

Attachment 2

P&A Committee

March 20, 2001

Consideration of proposal for implementing revisions of indicators measured and scored for annual performance assessment for performance funding year 2001-02 (Year 6)

On February 1, 2001, the Commission approved recommendations of the Planning and Assessment Committee effective July 1, 2001, to limit the number of indicators scored annually to allow a more concentrated focus on sector and institutional missions and to assess those indicators no longer scored for purposes of identifying emerging issues or problems. In order to implement the recommendations approved, the Commission recognized the need for consideration of revisions to some measures and possibly to the overall scoring scheme and charged staff to work with institutions in developing its recommendations. To do so, staff drafted preliminary recommendations and met with institutional representatives on February 1 to begin discussing revisions needed and to solicit feedback from institutions regarding staff’s initial suggestions. Following that meeting, staff met with institutional representatives by sector at least once and received feedback from institutional representatives as measures and revisions were discussed. On March 1, staff met again with institutional representatives to discuss revised staff suggestions for revising measures. Presented here are staff recommendations resulting from its work with institutional representatives to address measurement issues for scored indicators. Staff appreciates all the good work that those involved have contributed in assisting staff in producing these recommendations. Work to address unresolved issues as indicated below as well as to address other issues such as monitoring of non-scored indicators is underway and recommendations will be made at a later point regarding these issues.

Staff Recommendation: Staff recommends that the revised measures as indicated below and in accordance with recommendations approved by the Commission on February 1 be approved for use beginning with the 2001-02 performance year with the understanding that measurement details and in some cases standards for revised indicators may require further consideration as outlined in the explanation below.

Staff Recommendations for Revisions to Scored Indicators

for Implementation in 2001-02 (Year 6 of Performance Funding)

The following summarizes staff recommendations by indicator for needed measurement revisions includes staff recommendations for resolving identified but as of yet unresolved measurement issues. A supplement to this summary outlining each measure in keeping with the current workbook format is attached (see file PA032001.Attt2a.yr6proposal wkbkform.) This supplemental information should be considered as an initial draft. Staff is at present working to ensure its accuracy and to provide additional measurement clarification where needed.

1BMISSION FOCUS, CURRICULA OFFERED TO ACHIEVE MISSION

Applicable to All Institutions

1CMISSION FOCUS, APPROVAL OF A MISSION STATEMENT

Applicable to All Institutions

Staff Explanation: There are no proposed changes to this indicator from its current definition. For the current definition and standards, see Performance Funding Workbook, September 2000, pages 73 - 75. Note that for Year 7, staff will review this indicator again in the fall to determine any needed modifications.

1D/EMISSION FOCUS, COMBINED (1D) ADOPTION OF A STRATEGIC PLAN TO SUPPORT THE MISSION STATEMENT and (1E) ATTAINMENT OF GOALS OF THE STRATEGIC PLAN

Applicable to all Institutions, with institutional differences

Staff Explanation: In order to reflect CHE’s approval of assessing through a single indicator performance on one institutional goal reflective either of the institution’s strategic plan or CHE’s strategic plan for higher education in South Carolina, staff proposes that the current definitions of 1D and 1E be combined and revised as appropriate to result in a single scored indicator. Necessary changes to 1D and 1E as they appear in the September 2000 Workbook on pages 77-82 include: limiting the selection to one goal and scoring of the indicator each year beginning in Year 6. Institutions have recently had two goals approved as part of 1D requirements for Year 5 and for which performance in meeting those goals is to be scored beginning in Performance Year 6 as part of current 1E definitions. Staff recommends that institutions be allowed to select one of the two goals for continuation and scoring in Performance Year 6. The resulting measure is indicated below. Technical measurement details for the recommended combining of 1D and 1E may be reviewed in the attached supplement to this document.

Proposed Measure (1D/E for all institutions):Each institution is to be assessed on its performance in attaining a measurable goal over a three-year period. Institutions are to identify, subject to the approval of CHE, the measure to be used in determining performance in attaining the selected goal and the appropriate quantitative standards for each of the three-years for which performance will be scored. Goals and their measures and targets are to be approved such that there will be no delay between ending one goal and beginning another for performance scoring purposes. The identified goal and the selected measure and standards to be used in determining achievement of the goal will meet at a minimum the following requirements:

Be in keeping with an institution’s own institutional strategic plan or the strategic plan for higher education in South Carolina as approved by the Commission on Higher Education and the Council of Public College and University Presidents;
Support the institution’s mission and not be in conflict with the sector mission;
Be maintained for three years;
Include annual as well as third year goals;
Be quantifiable;
Not duplicate an existing performance funding measure;
Not include capital projects; and
Be subject to approval by the Commission on Higher Education.

2AQUALITY OF FACULTY, ACADEMIC AND OTHER CREDENTIALS OF PROFESSORS AND INSTRUCTORS

Applicable to all Institutions, but defined differently for Technical Colleges vs. all other institutions

Staff Explanation for 2A as applicable to Technical Colleges: Technical colleges will continue to be scored on indicator 2A1 for purposes of the scored 2A indicator. Staff recommends no change from current definitions and standards for 2A1 that are found on pages 83-88 of the September 2000 Workbook. Staff suggests from this point forward that this indicator be referred to as 2A for Technical Colleges.

Staff Explanation for 2A as applicable to the Research, Teaching and Regional Campuses sectors: Staff recommends revising the two current subparts, 2A2a and 2A2b, into one part referenced as 2A for Research, Teaching, and Regional Campuses. The measure recommended considers all full-time faculty regardless of courses taught. Faculty included in calculating average salary data for indicator 2D will serve as the basis for faculty included here in determining the percentage of those with terminal degrees in the primary teaching area. Revised data based on the proposed measure indicated below are unavailable because institutional reporting has been limited to date to those faculty identified as teaching an undergraduate credit course in the fall. Staff recommends that all other measurement details related to determining whether a not a faculty member has the terminal degree in the primary area be retained (see pages 83-88.) For standards for Year 6, staff recommends that they be those currently defined for subpart 2A2b with the understanding that the standards will be revisited as institutional data becomes available or as applicable data from national and/or regional sources is identified. For Year 6, if there a concerns related to the proposed standards, staff will receive feedback and make a revised recommendation for consideration such that the standards would be finalized not later than the July meeting of the Commission. The proposed measure is as follows:

Proposed Measure (2A for Research, Teaching, and Regional Campuses Sectors): The quality of the faculty as represented by the academic and other credentials of professors and instructors is to be measured as the percent of all full-time faculty who have terminal degrees as defined by SACS in their primary teaching area.

2DQUALITY OF FACULTY, COMPENSATION OF FACULTY

Applicable to all Institutions, but defined differently for Technical Colleges and Regional Campuses as opposed to Research and Teaching Institutions

Staff Explanation: In implementing the recommendations approved by the Commission, staff proposes that this indicator remain the same as in Year 5 (definitions and methodology for standards) for the regional campuses and technical colleges. Staff recommends that for all four year institutions, the measure remain the same as defined in Year 5 (definitions and methodology for standards), but with the rank of instructor being excluded from consideration for scoring purposes. Staff finds that this rank should be excluded for purposes here due to definitional differences related to the treatment of this rank across institutions and due to the relatively low numbers of faculty at this rank for some institutions. These differences lead to volatility in performance that may not be reflective of an institution’s performance and as such the deletion of this rank for scoring purposes in no way reflects a position that instructors should not receive competitive compensation by institutions. Data for instructors will continue to be reported to the Commission and as a result, the Commission can continue monitoring institutions’ compensation of instructors to identify any emerging problems.

Proposed Measures:

2D for Research Institutions and Four-year Colleges and Universities: The measure is the average faculty salary by rank for the ranks of assistant professor, associate professor, and professor.

* 2D for Regional Campuses of the University of South Carolina: The measure is the average of faculty salaries. Faculty with ranks of instructor, assistant professor, associate professor and professor will be included in determining the average.

* 2D for Technical Colleges: The measure is the average of faculty salaries. (Technical Colleges do not utilize ranking of faculty.)

* No changes from current measure as effective in Year 5.

3DCLASSROOM QUALITY, ACCREDITATION OF DEGREE-GRANTING PROGRAMS

Applicable to All Institutions

Staff Explanation: There are no proposed changes to this indicator from its current definition. For the current definition and standards, see Performance Funding Workbook, September 2000, pages 121 - 122. Note that the current methodology for the “counting of programs” will continue in Year 6 and that for Year 7, staff will review this indicator to determine whether any revision to this methodology is to be made.

3ECLASSROOM QUALITY, INSTITUTIONAL EMPHASIS ON QUALITY TEACHER EDUCATION AND REFORM

Applicable to Teaching Sector Institutions

Staff Explanation: There are no proposed changes to this indicator from its current definition. For the current definition and standards, see Performance Funding Workbook, September 2000, pages 121 - 122. Staff intends to make recommendations in the future, possibly to become effective in Year 6 to align this measure, particularly subparts 3E2a and 3E2b, with Title 2 reporting requirements when known. Such a proposed revision may require a different reporting cycle for parts 3E2a and 3E2b and corresponding data for 7D in the upcoming year.

3CLASSROOM QUALITY, INDICATOR TO BE DEFINED ASSESSING CLASSROOM/ INSTRUCTIONAL QUALITY OF PROGRAMS

Applicable to Regional Campuses

Staff Explanation: At present, measurement details are being worked-out. Staff is working with regional campus representatives to identify a measure that could be used to provide a measure to assess classroom/instructional quality. For this measure, the Commission approved the development of a measure for regional campuses. Staff’s expectations continue to be that a measure be developed by the fall so that data collection could occur and a determination made as to whether or not the measure could become effective the following year. Any measure developed would therefore not be applicable to the Year 6 scoring process. Preliminary discussions with regional campus representatives suggest that an expansion of questions asked on the current course evaluation instrument could provide the basis for this measure, with consideration given to uniformity of administration across campuses.

4A/BCOOPERATION AND COLLABORATION, COMBINED, (4A) SHARING AND USE OF TECHNOLOGY, PROGRAMS, EQUIPMENT, SUPPLIES, AND SOURCE MATTER EXPERTS WITHIN THE INSTITUTION, WITH OTHER INSTITUTIONS, AND WITH THE BUSINESS COMMUNITY and (4B) COOPERATION AND COLLABORATION WITH PRIVATE INDUSTRY

Applicable to All Institutions, with differences across sectors

Staff Explanation: Staff is in the process of working with sectors to define this indicator as applicable for all institutions within a sector. Staff recommends that for the upcoming year, the measure be scored as a compliance indicator for all sectors as measures are developed and data collected. Because of the nature of this indicator and timeframes involved, staff will continue to work with sector representatives to clarify measurement expectations. Staff recommends that the sectors which have not yet identified a sector focus area provide a progress report on measurement development to the Planning and Assessment Committee on May 22. Staff recommends that that all measurement issues related to identifying the focus area and determining the measure be resolved not later than October 2001. It is recommended that data collection and identification of standards for the defined sector measures should occur by Spring of 2002 so that the measure can be implemented and scored beginning in Year 7 (2002-03). Below, staff indicates the recommended language for the general measure, followed by details of sector activity regarding measurement development.

Proposed Measure (4A/B general measure as applicable to all institutions): Indicator 4A/B is defined tailored to each sector. 4A/B is intended to measure sector focused efforts of institutional cooperative and collaborative work with business, private industry and/or the community. Each sector, subject to approval of the Commission, will develop a common measure that will be the focus of the sector for a timeframe to be determined in excess of 1 year. Standards will be adopted for use in scoring individual institutional performance annually after the first year of implementation.

Sector Definitions of 4A/B to begin with Year 6 (2002-03):

The Research Sector has identified a specific focus area and also goals and targets to be assessed for five years. The area of focus, outlined below, is to enhance collaborative research efforts within the sector. This sector will continue to work on definitional issues.

Proposed Measure (4A/B, Research Sector): To enhance collaborative research within the Research Sector including the development and use of an integrated faculty and grants database system.

The Teaching Sectorhas identified focusing on business, community, and public school representation on academic program advisory boards as the area for which the sector would like to craft a measure. The teaching university sector will pursue a measure aimed at assessing institutional involvement in the community or with area business and industry by focusing on the representation of business and community and public school representatives on academic program advisory boards. The measure being discussed would identify current involvement, increasing involvement where needed, and optimum levels of representation. The sector is working to define a measure and standards focusing the institution’s activities related to outreach efforts to gain involvement by such groups on campuses.

The Regional Campuses are in the process of identifying a focus area in order to begin discussion of potential measures and continue to work with staff in identifying a sector focus that will lead to measurement development. One suggestion the sector has determined it may wish to pursue is a measure that assesses community outreach activity by the faculty and staff of the campus. The sector has suggested that the focus could borrow from a recently adopted faculty senate document outlining service activities which include, but are not limited to: service to the community, the local campus, the regional campuses/greater University and the profession. Staff will continue to work with the campuses and expect to have a progress report from the sector available for the Committee in May.

The Technical Colleges are in the process of identifying a focus area in order to begin discussion of potential measure. The technical colleges also continue to work with staff in identifying a sector focus that will lead to measurement development. One suggestion the sector is pursuing is the identification of an area of focus related to the mission of the sector and relevant state-level strategic plans in order to develop a “quantifiable” best practices-type measure. Staff will continue to work with the colleges and expect to have a progress report from the sector available for the Committee in May.

5AADMINISTRATIVE EFFICIENCY, PERCENTAGE OF ADMINISTRATIVE COSTS AS COMPARED TO ACADEMIC COSTS

Applicable to All Institutions

6A/BENTRANCE REQUIREMENTS, COMBINED (6A) SAT AND ACT SCORES OF STUDENT BODY and (6B) HIGH SCHOOL STANDING, GRADE POINT AVERAGES, AND ACTIVITIES OF THE STUDENT BODY

Applicable to All Research, Teaching and Regional Campuses Sector Institutions, defined differently for MUSC vs. other institutions

Staff Explanation for 6A/B for Research, except MUSC, Teaching and Regional Campuses Sector Institutions: In implementing the recommendations approved by the Commission, staff recommends that indicators 6A and 6B be combined such that the measure consider the percentage of first-time entering freshmen who have an SAT of 1000 or ACT of 21 or a high school GPA of 3.0 or high school class standing in the top 30% of those reporting any of the aforementioned credentials. Staff and institutional representatives have reviewed recalculated historical data encompassing this proposed definition. Staff is in the process of reviewing other relevant data in order to make recommendations related to the standard for this revised indicator. Staff expects to resolve the issue of the standard by the Committee’s May meeting.

Proposed Measure (6A/B Research, except MUSC, Teaching, and Regional Campuses Sectors): Percent of first-time entering freshmen who take the SAT or ACT test or who have reported a high school grade point average (GPA) or who have reported a high school class standing who meet or exceed the Commission-approved target score on such tests. NOTE: Target scores are defined as 1000 on the SAT or 21 on the ACT: both are based on approximate national averages for test takers. For high school GPA, the target is 3.0 or higher on a 4.0 scale and for high school class rank, the target is within the top 30% of their senior year class.