NYSILC SPIL Evaluation Report, FY 2015-2016
NYSILC 2016 SPIL Evaluation Report
Consultant’s Report on the Third Year of the 2014-2016
State Plan for Independent Living (SPIL)
Submitted by Alan Krieger
Krieger Solutions, LLC
May 5, 2017
- 19 -
NYSILC SPIL Evaluation Report, FY 2015-2016
Contents
Background 1
2016 SPIL Evaluation Findings 2
Summative Analysis – How well were the objectives and targets met? 2
Overview: 2
Summary of the evaluation of each of the objectives and consultant observations. 3
Objective 1: NYSILC Operation - 3
Objective 2: Support a Statewide Systems Advocacy Network (SSAN) 6
Objective 4: NYS CB (formerly CBVH) will provide IL services on a fee-for-service basis to eligible individuals who are legally blind. 8
Objective 5: Capacity Building 9
Objective 7: Coaching and consulting for Centers 11
Objective 9: Develop a Database 12
Overall Summary 13
Formative Analysis 13
Recommendations 15
SPIL COMMITTEE REPORT 16
NYSILC 2016 SPIL Objectives/Evaluation Summaries 16
Objective # 1 – NYSILC Operations 16
Objective # 2 – SSAN 22
Objective # 4 – NYSCB IL FFS 29
Objective # 5 – CBILCO 31
Objective # 7 – CIL Coaching 33
Objective # 9 – Database 35
Background
This evaluation covers the activities from October 1, 2015 through September 30, 2016 related to objectives in the statewide plan for independent living. This is the final year of the plan and some objectives were already complete or otherwise not active this year. The evaluation consultant had limited discussions with most of the committees about their evaluation process. NYSILC staff, as always, did a great job pulling together the data and working with the committees to finalize their assessments. The evaluation consultant participated in committee meetings to review the data collected and to continue to focus on the shift from activity or inputs to outputs or impacts. The evaluation consultant did not review any of the original data, only what the Council staff pulled together.
As noted in previous reports, the Council has come a long way with writing a more specific and measurable plan. The objectives and targets are more outcome or impact oriented; vague terms and standards have been defined more clearly to make the evaluation more objective. The Council is currently in the process of developing a monitoring protocol so the Council can take an active role in measuring success in accomplishing the objectives and to get an early awareness if issues arise so they can be addressed during the program year rather than after the fact.
This year’s evaluation consulting activities included helping finalize the objectives and targets for the new SPIL, and continuing to look at “value added” components for each objective that focused more on outcomes and results. Earlier plans consisted mostly of activity or process oriented objectives. They discussed what the Council or its subcontractors would be doing. The new SPIL has a few more outcome oriented objectives, but in the push to get it finished, a number of activity oriented objectives remained in place. So, we continued to add some “value added” elements to bring more of an outcome orientation to the evaluation.
There were several targets that were not clearly defined in the SPIL (e.g. what is a “system change”) and committees have taken steps to better define some of these so they can be more accurately evaluated.
At the June 2015 Council meeting additional training was provided for Council members on the evaluation process in advance of the development of the new SPIL. The training focused on developing outcome based goals and objectives to help continue the transition from process/activity objectives to outcome objectives. In addition, as new individuals continue to be appointed to NYSILC, it may be useful to plan additional training for them. New members continue to be provided a link to the webinar recorded earlier. The outcome evaluation training materials developed and presented the first year the evaluation consultant worked with the Council are also included in the new member orientation manual.
Some objectives from the previous SPIL were purposely delayed to provide enough time to develop the RFP’s. The final one or two years of these objectives were extended into this SPIL with the use of unspent Part B funds.
Similar to the earlier years, each committee developed an evaluation process that generally relied on the NYSILC staff gathering the relevant data from records for projects being conducted by NYSILC, or from the DSU for projects that were handled through RFP’s. Some additional work was done to evaluate some of the “value added” components. Committees are paying more attention to the actual impact of the projects, not just the amount of effort that was expended.
Each of the committees conducted their evaluation between November 2016 and February 2017. The phone calls in which the SPIL evaluator participated usually have very few committee members participating. This low level of involvement is a concern. Only one committee had a SPIL evaluation session where they were fully engaged in discussion. This would be the goal for every committee. Written summaries of the committee meetings were transmitted to the SPIL evaluator who then drafted an evaluation report based on those summaries and committee discussions. The full committee summaries are appended to this report.
2016 SPIL Evaluation Findings
Summative Analysis – How well were the objectives and targets met?
Overview:
Most of the objectives in the SPIL initially look to be very straight forward to evaluate. While the objectives themselves are not usually written in measurable and specific terms, they each have measurable indicators that are measurable. These indicators generally relate to whether or not certain activities took place, how many people participated in the activity, what the impact was of the activities, and/or how often or to what degree the activity was accomplished. On a deeper review, some questions emerge. Some of the objectives were technically met, i.e. the number of activities/people served were achieved. However, the impact or outcome of these activities in some cases appears to be minimal and not what the objective’s intention was.
The reverse is also true, some of the targets that the committee felt were “not met” due to data problems or technicalities were partially met when the data was examined more closely and the intention or desired outcome was considered. In some cases, the objectives or indicators were written too specifically. Therefore, the evaluation must show that the objective was technically not met, although the intention appears to have been met. Similarly, by writing some of the objectives too broadly, the objectives were technically met although the impact appears to be minimal and not what was hoped for.
This report is generally based on the technical rating – did the objective and indicators, as written literally get met. In addition, the consultant added comments where a different rating would actually be more accurate.
There were 10 objectives in the 2014-2016 SPIL. Four of the objectives (#6 Voter Registration, #7 CIL Coaching and Consulting, #8 Capacity Building and #10 Deaf-Blind SSP) were carried over from the prior SPIL for only one or two years, so were inactive this year. There was no IL Conference this year, so Objective 3 was also inactive. The 5 remaining objectives had 30 specific measurable performance targets set for this year, but only 28 were active. The Council has adopted a scale or range for rating each target and objective. The scale includes: exceeded, fully met (100%), substantially met (at least 60%), partially met, not met/no progress. This reflects the understanding that even when an objective or indicator is not fully met, if there was substantial progress, that has substantial impact and is worth noting.
Overall, based on strictly numbers, the Council fully met and exceeded the measures for one objective (#2). Three of the five active objectives were substantially met (objectives 1, 4, and 5) and one was partially met (#9). The fully completed rate of 20% is lower than any previous year. Four of the five (67%) were at least substantially met which is slightly lower than previous years, but overall, the SPIL was substantially met for the past year.
For the five active objectives, there was a total of 30 performance targets, but only 28 were active this year. Thirteen exceeded the target (46%) and five met the target (18%), so 18 out of 28 active targets were accomplished, a rate of 64%, about the same as last year, but falling short of the ambitious goal of 90%.
As noted in the introduction, these numbers do not tell the whole story. As explained in the report which follows, the numbers often show the extent of the activities that were conducted, but don’t always reflect the intention of the objective or the target in terms of impact or outcomes.
Summary of the evaluation of each of the objectives and consultant observations.
(For more detail see the SPIL committee’s full report, which follows the consultant’s report)
Objective 1: NYSILC Operation - In order to effectively coordinate, monitor, and evaluate the SPIL, NYSILC will strengthen its operations and capability.
This objective has nine targets, three were exceeded, three were met, one was substantially met and two were partially met. Overall, the objective was substantially met.
As noted in prior year reports, this objective was revamped in this SPIL to show more of an impact or outcome. In the earlier SPIL, one target was that meetings were held or the 704 report was completed. In this SPIL, some qualitative elements were added to these targets – Council meetings had to have a quorum so business could be transacted. The 704 report was completed accurately and submitted on time to RSA (which has since transitioned to the Administration for Community Living (ACL).
In actuality, the six targets that were exceeded or met have for the most part been met for many years and this does not show a “strengthening” but shows a maintenance of good or excellent performance. To measure “strengthening” the Council would need to identify areas of weakness, measure the baseline and show improvement after one or more years.
Two of the targets relate to Council administration (704 report and audit). These have been fully met for many years. One target related to Council operation – the number of meetings with a quorum – has also been met for many years. It was an issue several years ago, and did show a strengthening, but since it’s been met for several consecutive years, it no longer shows a strengthening, but a maintenance.
Two other targets related to Council operation (number of people actively serving on committees and number of issues addressed) could be a basis for showing strengthening.
· Number of people actively serving on committees – the Council reports a very high number (45), but each time I have participated in committee meetings, the numbers have been extremely low. The definition of “active” participation may need to be clarified. I have difficulty accepting the Council’s report on this target based on my experience, but without more complete data, it should be counted as the committee identified it, as being exceeded. However, perhaps it is a matter of understanding how this number was counted toward the outcome. Does it reflect the members who just attend the committee meetings, or should it be adjusted down to just those individuals who actively participate during the meetings or to those who attend on a regular basis? The Council will be now be using Outlook invitations as a way to ensure good attendance at each committee meeting. They may wish to additionally develop a definition of what constitutes an “active” committee member.
· The “number of issues addressed” was a vague term that the committee has worked on to clarify and now refers to the fact that the Council took some affirmative action in relation to the issue and in many cases, the Council has recorded the impact of those actions. This third target shows real improvement in Council operations and impact.
Two other targets relate to the overall accomplishment of SPIL objectives and targets. One was substantially met (#of targets achieved) and the other was only partially met (# of objectives achieved). The Committee discussing this felt that since this was out of the Council’s direct control, it should not be counted against the Council if the numbers fell short. This has been dropped from the upcoming SPIL. They also felt that the group of outcomes should be specific to the operations and activities relevant to the council. I disagree. The Council sets the targets and while it does not select the contractors, the Council is allocating funds for these projects and is ultimately responsible for the administration of these funds. I think the Council should be accountable for this, perhaps in partnership with the DSU. I have indicated repeatedly over the years that the targets the Council has set (often in the 80-100% range) is unrealistically high, and the Council has never met these targets. Ambitious outcome measures are rarely met at such a high level. If the targets were set at 50% fully achieved and 20% substantially achieved, this would be a strong result and in many years, the Council could have met this target. I would encourage the Council to reinstate this as a measure of success in future years.