Discussion, Consideration and Possible Action Regarding Fiscal Years 2017-2021 Strategic Plan Budget Structure Elements, Including Strategy and Performance Measure Changes and Assessment of Advisory Committees.

Overview of Performance Measure Recommendations

Introduction – The Governor’s Office of Budget and Policy (GOBP) and Legislative Budget Board (LBB) issued the “Instructions for Preparing and Submitting Agency Strategic Plans for Fiscal Years 2017 to 2021” on 4/6/16. In the instructions, state agencies were directed to prepare budget structure (including performance measure) change requests and assigned one of five due dates. TWC’s due date for requested changes is 5/13/16. Staff are planning to present recommendations for Commission approval on 5/3/16 with 5/10/16 serving as a second opportunity for Commission approval should Commission-direction on 5/3/16 require staff to perform additional work prior to final approval.

The Division of Operational Insight has coordinated with agency and DARS staff on a set of change recommendations. The recommendations fall primarily into 3categories:

1)Replacing the old TWIC Common Measures to with the new WIOA-based TWIC Common Measures;

2)Consolidating Vocational Rehabilitation (VR) measures (primarily combining Blind and VR General into integrated measures); and

3)Removing several measures that were either less meaningful or not aligned with the proposed measure structure.

------

Common Measure Changes

TWCoriginally implemented a set of TWIC Common Measures based on DOL’s Common Measures in SFY06. WIOA now replaces those old Common Measures with new Common Measures which DOI staff helped TWIC adapt for their formal measures.

Staff recommend that the new Common Measures form the foundation of TWC’s SFY18-19 measures for Vocational Rehabilitation (VR), Adult Education & Literacy (AEL), and Career & Training (C&T)[1] services. Staff will be able to apply the new measure definitions to historic participant data to enable comparisons of performance over time. In addition, using new measure names and new measure IDs in the LBB reporting system will reduce the likelihood of someone mistakenly combining old and new data. VR, AEL, and C&T will each have the following measures:

  • Participants Served
/
  • Average Cost per Participant Served – VR

  • % Employed/Enrolled 2nd Qtr Post-Exit
/
  • % Employed/Enrolled 2nd–4th Qtrs Post-Exit

  • Credential Achievement

The following matrix provides a summary of the key changes between the old and new Common Measures:

Old Measure / New Measure / How the New measures Differ from the Old
Customers Served / Participants Served / 1) Not all Customers are Participants
2) Excludes Self-Serve/Information-Only Job Seekers (consistent with expected WIOA regs)
Average Cost per Customer Served / Average Cost per Participant Served / 1) This is not a TWIC measure but will be aligned with the TWIC measure
2) The proposed Numerator includes all salary, grant, and client service expenditures in the Strategy to increase transparency and comparability across strategies
Entered Employment
(By Q1 Post Exit) / % Employed/Enrolled 2nd QtrPost-Exit / 1) Considers both Employment or Enrollment in Education/Training as successful outcomes
2) Includes all Participants, not just those unemployed at participation
Employment Retention
(Employed Qtrs 1-3 post exit) / % Employed/Enrolled 2nd-4thQtrs Post-Exit / Considers both Employment or Enrollment in Education/Training as successful outcomes
Educational Achievement
(By Q3 Post-Exit) / Credential Rate
(by Q4 Post-Exit) / 1) Doesn’t Exclude those who leave early to enter employment but failed to achieve credential
2) Only counts Diploma/GED if accompanied with employment or enrollment in post-sec education.

Additional Details on how the measure definitions and related concepts change (or remain the same):

1)PARTICIPANTS SERVED / AVERERAGE COST PER PARTICIPANTS SERVED

  1. Career & Training Service Participants – DOL has proposed that those receiving only Self-Service and Information (SSIO) are NOT Participants and that continued utilization of self-service or information services does not extend participation. TWC strenuously objected to the proposal and backed those objections up with national data but it seems unlikely that DOL will budge on this point. If the final regulations largely match the proposal, then Participants Served and Avg Cost measures will exclude SSIOs job seekers. Over the last 3 calendar years, 33-35% of job seekers were SSIO with the vast majority being in Wagner-Peyser only.
  2. VR Participants – VR consumers only become participants if they have a signed, approved IPE and receive services in accordance with it. Roughly 27% of DRS customers and 33% of DBS customers served each year don’t become participants (at least not in that year).
  3. AEL Participants – TWC previously modified the AEL Customer Served definition to only count AEL Participants as defined by the U.S. Department of Education (ED). Therefore, the change should not functionally change what has been historically reported though the performance period will shift to the SFY to align with other output measure reporting.

Although the final WIOA definition will likely not be released by the deadline for submission of measure/definition change requests in the Strategic Plan, we can address them outside the definitions by clarifying what is and is not considered a “qualifying service” based on federal guidance.

DOI will provide updated history for the Participants Served and Average Cost measures to ensure that people do not attempt to compare the old Customer Served data with the new Participant Served data.

2)OUTCOME MEASURES OVERVIEW – TWC is only accountable for outcomes on Participants. The New Common Outcome Measures are all EXIT-based (as they were before). Generally, upon becoming a Participant (called the Date of Participation), an individual receives qualifying services from one or more programs and then eventually exits. As proposed in WIOA Regulations, exit occurs when

  1. The Participant goes 90 days without receiving STAFF-ASSISTED services and this lapse in service was unplanned (an example of a planned gap that would prevent exit is if the Participant were to begin a training course but the training was not scheduled to begin for more than 90 days); OR
  2. The VR Case File is closed in accordance with federal standards.

The period of time between the Date of Participation and the Date of Exit is called the Period of Participation (POP). Under the old Common Measures, TWC used “Common POPs” for C&T programs. This mean that although Participants/outcomes could be tracked by program that funded services, we didn’t have separate POPs and separate instances of accountability if several programs funded services for an overlapping period.

Staff are recommending that this integrated reporting model be extended to AEL and VR Participants as well, in line with the vision of an integrated workforce system. If the final regulations do not permit common POPs across DOL and ED programs, we will need to revise our programming slightly to provide for common POPs for all C&T programs as we have today and siloed, program-specific POPs for AEL and VR.

3)ACCOUNTABILITY CHANGES – As noted, TWC is generally accountable for outcome results on all Participants but there were some exclusions for people who at or in the measurement period following exit were dead, hospitalized or otherwise institutionalized. This was also the case under the old Common Measures for C&T programs but not for AEL and VR. However, there are some other accountability changes of note:

  1. Historically AEL only counted those who were “in the labor force” in the employment outcome measures. It appears that this will no longer be the case. If not, it will likely result in lower performance than if they were still excluded but this it strengthens accountability and provides a fuller picture of the employment outcomes for Participants.
  2. There used to be an exclusion for those for whom the state did not get a valid SSN. There are indications that this will not be the case under the new WIOA-based measures. If not, this could reduce reportable performance (especially in AEL given substantial portion of AEL Participants are missing SSNs in TEAMS.
  3. Historically, TWC had a “noncompliance” exclusion that was usedto exclude Choices & SNAP E&T Participants who failed to meet program requirements and exited as a result of a sanction. This exclusion is not part of the New Common Measures because:
  4. Recent Federal legislation required FNS is apply WIOA measures for SNAP E&T; and
  5. Many TANF reauthorization drafts over the past two years have included WIOA measures.

We will still have the “compliant” version of the results so we could say something like “60% were employed post exit but 80% of those who actually participated in services as required were employed post-exit.” That would let us show the impact that these services can really have.

Consolidation of VR Measures – In SFY17, 18 VR measures transfer from DARS to TWC:

  • 5 for Blind Consumers5 for General VR Consumers
  • 3 that included both Blind and General VR Consumers
  • 5 BusinessEnterprises of Texas (BET)

Blind and General VR – Given the statutory requirement to integrate VR Services by 10/1/17, staff are recommending using a single set of measures that include all Participants receiving VR, regardless of disability type. DOI will develop performance reports that break out results by category of disability in order to allow management to monitor and provide stakeholders information on services to and outcome of those with different types of disability.

Other Miscellaneous VR Measures – Staff propose to delete 5 other VR measures as outlined in the detailed side-by-side document. Generally these measures were redundant to other measures or did not provide meaningful information.

BET – Staff are not proposing any changes to BET measures for SFY18-19 but will evaluate the program and whether the current measures are effective and provide the information needed for program oversight and improvement.

The following table shows the changes in the number and types of measures that TWC would have should these recommendations be adopted:

2017 / 2018 – 19
VR Key measures: / 9 / 7
VR Non-Key measures: / 12 / 3
VR measures - Total / 21 / 10
Non-VR Key measures: / 26 / 30
Non-VR Non-Key measures: / 21 / 13
Non-VR measures - Total / 47 / 43
Total Key measures: / 35 / 37
Total Non-Key measures: / 33 / 16
Total measures: / 68 / 53

Attachments:

1)Side-by-Side Changes – A detailed list of recommended measure changes side-by-side with notes regarding each proposed change. Most changes relate to either the implementation of the new TWIC WIOA-based Common Measures with the VR measures also consolidating outcomes for all VR participants (i.e. not broken out into Blind and VR General).

2)Measure Definitions – Recommended measure changes are reflected in the detailed measure definitions which are also attached. The LBB requests that these changes be made in underlining and strikethrough. However except for a few technical corrections, nearly all staff recommendations involve deleting existing measures and replacing them with new measures rather than attempting to amend old measures that in most cases are fundamentally different than their replacements. As noted, this also has the benefit of making it less likely that someone might accidentally mix data from the old and new measures.

Note that neither the side-by-side measure change document nor the detailed measure definitions reflect changes to goals, objectives, or strategies that may be adopted by the Commission. Should any such changes be approved by the Commission, staff would request to be allowed to simply update the measure documents to match the structural changes as “technical updates.”

- PAGE 1

[1] “Career & Training” is the recommended umbrella phrase to refer to TWC historic workforce programs outside of VR &AEL.