SILC Needs Assessment Survey: a Discussion on Current Approaches and Practices in Needs

SILC Needs Assessment Survey: A Discussion on Current Approaches and Practices in Needs Assessment in SILCs

Presented by Valerie Barnum-Yarger, Deb Cook, Susan Fager, Jeff Sheen, Larry Wanger and Brad Williams on November 21, 2013
> TIM FUCHS: Good afternoon. I'm Tim Fuchs
with the national coin sill on independent living
here in Washington D.C. I want to welcome you all
to SILC-NET's newest webinar SILC needs assessment
survey: A discussion on current approaches and
practices and needs assessment at Centers for
Independent Living. Excuse me. Statewide
independent living councils. What a way to start
a call. Today's webinar is being presented by the
the SILC-NET, a program of the IL NET training and
technical assistance project for CILs and SILCs
around the IL NET is operated through a
partnership among ILRU, NCIL and APRIL with
support provided by RSA at the U.S. Department of
Education.
So we are recording today's call so we can awr
Clive it on ILRU's website and we're going to have
a different format today. For those of you that
posh dissipate in these calls from time to time
you know it's typically kind of a lecture-based
program with Q&A but we really do want today's
call to be an open discussion. We're going to
start off with a presentation. But after we
listen to Jeff Sheen's presentation we're going to
open the call up, have some panel questions from
our speakers today, and then take your comments
and questions to lead the discussion. So the
second half of the call is really intended to be
very interactive. We're going to open the line up
and hear from all of you, too.
So I'll give you some instructions about that when
we get there. Until that time your lines are
muted. When we open up the line I will ask you to
individually mute your lines so we don't have a
lot of background noise. I'll walk you through
that when we get there, about halfway through the
call.
Before we start I want to ask you to please fill
out the evaluation form for today's call. Despite
the fact it's a little less formal than some of
our other calls we still need to know what you
thought of it and if it was helpful for you in
your role with the SILC. And it only takes a few
moments to complete. It's very easy. We really
would appreciate your thoughts. That will be
included on the final slide of the webinar if
you're on the webinar today. If you're listening
on the phone you can access that link in the
confirmation email that was sent to you.
We do have a short PowerPoint today. If you are
on the webinar it's going to display automatically
but for those of you that have just called, which
is fine, you'll want to open up that PowerPoint
that was sent to you in the confirmation email.
If you don't have that in front of you, you can
email me. I'm at . Thanks.
That's as much housekeeping as I had to handle
today. I just want to introduce some of our
speakers before we begin. First and foremost I
want to thank Jeff Sheen for all of his work
organizing this call. Jeff is with us from Utah
State University's center for people with
disabilities, and Utah state is also an integral
part of the IL NET project, and Jeff has done a
lot of work putting together this SILC survey,
analyzing the results from many of you that
responded. So thank you for that. And helped us
put together our presenters for today that I'll
walk through now. And I want to thank all of
them, too, for responding to the survey. Jeff was
instrumental in helping us identify states that
really had some promising practices, and I'm
pleased to say that all five of the states that we
reached out to agreed to participate today. We're
excited -- were excited to participate today.
It's wonderful. Really, I think, it is a
testament to the peer support that happens in IL
on a macrolevel as well as on a service level at
centers. So thanks in advance to all of you.
With us from the Michigan SILC we have Valerie
Barnum-Yarger. From the Washington SILC Deb Cook.
From the DSU in Colorado, Susan Fager. From the
Arizona SILC we have Larry Wanger. And from the
New York SILC we have Brad Williams.
We had a call the other day to prepare and just a
wonderful discussion, and I know we'll repeat that
today. Thanks to all of you for being with us.
In the meantime I'm going to turn it over to Jeff
to get us started with the presentation. Jeff?
> JEFF SHEEN: Thank you, Tim. Good afternoon,
everyone. It's nice to be here with you this
afternoon. I'm going to take just a few minutes
and go over some of the -- give you an overview of
the survey that we conducted and some of the key
findings that informed the recommendations that we
have come up with based on conducting this survey.
So to give you a sense, the overall purpose of
this survey was just to get a better understanding
of how the statewide independent living councils
are currently going about conducting their needs
assessment activities. In particular in relation
to their state plan development. So that the
folks at the IL NET project could get a better
sense of how we might shape training and technical
assistance efforts in the future to help the SILCs
do a job that they're comfortable with as far as
conducting needs assessments, can really help them
move forward in productive and comprehensive ways.
The basics of the survey, we developed a
23-question telephone survey that many of you
participated in. That was developed by staff here
at the center for persons with disabilities as
well as staff at ILRU, and we conducted these
surveys between July and August. The thing I'm
most excited about, we had 54 out of the 56 SILCs
that compleaptd this survey. So we almost had
100% response rate. That is fantastic. So we're
really pleased that so many of your colleagues and
yourselves participated when we called and helped
us complete this important survey.
As far as who we talked to when we reached the
Statewide Independent Living Councils, 39 of the
respondents were executive directors, six were the
designated states units liaisons and eight of the
surveys were completed by current SILC chairs or
vice chairs.
Just a little bit of demographic information, the
median number of years with the SILC was six. So
half of the people that we talked to had been at
the Statewide Independent Living Council for more
than six years and half had been there just under
six years. We had three that had been with the
SILC for less than a year and so you can imagine
this was a new experience for them. And then we
had three who have been with their Statewide
Independent Living Council essentially since the
inception, over 20 years ago. So kind of a broad
cross section of folks that we talked to.
I want to go into a few of the key findings from
this survey, and the ones that we selected to
present today are really because they tie into
some of the primary recommendations that we have
moving forward for how we can address these
issues.
So the first two bullet points on this shrined
this slide is dealing with key findings around
methods, methodology of conducting needs
assessments. Over half of the folks we talked to
use some form of survey to collect data, a mail
based survey, a web-based survey or telephone
survey. But only 5% of these folks knew actually
how many surveys were distributed. That's one of
the issues we have with web surveys. We send out
a link, and we ask people in our network to send
that link out to their network and then to their
network, and so it's really difficult to know how
many surveys actually get distributed or how many
people actually get the link to a -- in particular
to a web-based survey. That has some implications
for methodology as far as sample size that I'll
talk about in a minute.
The majority of the respondents didn't on the
other side have a good sense of how many responses
they ultimately received from not just their
surveys but other data collection activities which
often include focus groups or public hearings.
Part of that is because we caught people on the
phone and we were kind of asking how many
responses, and they might not have had the exact
number in front of them at the time. But most of
the folks we talked to had essentially an educated
guess but weren't certain really how many
responses their data collection efforts had led
to.
That third bullet point, we only had three
respondents we talked to out of the 54 that had
actually established a predetermined response rate
for data collection. So they had looked at the
demographics of their state, and they had decided
that in order for their data collection to be
complete they needed to reach a certain number of
people, and that varied across respondents. But
most folks were simply using a cut-off date. The
surveys were open. Focus groups were held until a
certain point in the year, and then at that point,
whatever data had been collected was the data that
was going to be used. And so that has some
implications again, like I said earlier. Sample
size is -- having a Good Sam pull size when
you're -- a good sample size is important giving
confidence that you have a representative
understanding, a good representation of the
different opinions and thoughts and things that
are going on among the population that you're
trying to survey. So if we're not clear on how
many surveys we're not sending out, we're not
clear on how many we're getting back and we're not
establishing kind of ahead of time a number of
folks that we need to get to to get a
representative sample, it kind of makes our
methodology a little less rigorous, which can have
some implications for essentially how many
confidence we can place in the we're getting the
best information that we can. And so that's kind
of an interesting finding. And I'll come back to
a recommendation around that in just a minute.
As far as satisfaction, we asked respondents how
satisfied they were with the needs assessment
process and results of their most current needs
assessment, and if you look at this graph, we had
a small number that was not very satisfied with
the process. We had almost a third that were
somewhat satisfied. We had slightly higher,
another third, that was satisfied, 31%. And we
had 30% of our respondents that were very
satisfied. So the nice thing is two-thirds
essentially of our respondents were either
satisfied or very satisfied with the needs
assessment process and results. The group that is
the somewhat satisfied, you know, that's
essentially folks that have a recognition that
they would like to do things differently. They
maybe would like to tighten some things up or do a
better job of data collection, or something about
what they did this last round wasn't up to their
satisfaction. Those that were satisfied are more
in the group -- they felt like they did an
adequate job but realized there could still be
some things they could improve.
The next slide talks about some miscellaneous
findings but some things that I think are
important as far as how they relate to our
recommendations. We did have 33 respondents,
which is 61% of our sample, that indicated they
use center-level consumer satisfaction data as
part of the needs assessment. Just over half of
those, which is 18 respondents, indicated that the
center consumer satisfaction data is actually
standardized. And I'm going to come back to that.
So there's a number of centers, over half of the
SILCs, excuse me, are using consumer satisfaction
data, which we think is a good thing. But the
data they're getting is maybe not standardized
across the centers in their state. So that's an
issue that we'll try to address in the future.
The next bullet point, and this was really an
important one that we'll come back to as far as
promising practices and we'll talk about this in
our discussion... 20 respondents reported they
used the results of their needs assessment for
purposes beyond the state plan development. They
have invest add lot of time and resources in
completing the needs assessment and they have
proactively kind of found ways that they can use
that information for multiple purposes beyond
creating a nice state plan.
Finally, on this slide 14 respondents indicated
that members of the SILC had not received training
on how to conduct an effective needs assessment
and there was an additional 13 that were unsure.
If anybody on the council had actually received
training specific to needs assessment. So we had
just over half of our respondents that either did
not have any training or weren't sure if anybody
on the council had received training. So that
obviously gives us some information that we can
use to develop some additional targeted training
around this topic.
Got a handful of slides left and then I'll go
ahead and take questions regarding any of this
information.
When we looked a little bit closer at the data
from the survey, there was some interesting
relationships that we found, or correlations. We
found that the longer somebody had been with the
council, the greater chance that they had a higher
level of satisfaction with the needs assessment
process. There was a higher likelihood that they
had used CIL data as part of the needs assessment.
And they were more likely to have used the
information they collected for purposes beyond
developing the state plan. So that kind of tells
us the longer somebody has been around, they've
kind of figured out some things that increase
their satisfaction. They figured out that the CIL
data is good to pull in and they found some
additional ways to use all of the information that
they're gathering for purposes beyond developing
the state plan. So we felt that was a fairly
important relationship.
The next relationship we found was between folks
that expressed a higher level of satisfaction.
These folks were typically more likely to use
data -- collect data from sources beyond the
typical sources, such as VR data or CIL data.
They were a little bit more comprehensive in who
they worked with to get information about the
needs for people with disabilities in their state.
And these folks that had high levels of
satisfaction were also more likely to use -- again
to use their data for purposes beyond the state
plan. They essentially found a way to get more
bang for the buck. If you're going to already
conduct a needs assessment and put the time and
effort into it, the more you can leverage that
information, the more efficient that is for you
and your council.
So from these kind of key findings and results, a
couple of key recommendations. The first one is
based on the methods slide where we found out a
lot of folks didn't have a really solid sample
size in mind. They weren't sure how many surveys
had been distributed or returned. To work with
the councils to develop and provide training on
the fundamentals of really effective needs
assessment in order to improve the rigor of the
methods that are being used and the idea behind
that is to improve the quality of data that
ultimately is collected, and the reason we spend
this kind of time trying to collect good data is
we need good quality data and information to make
good strategic plans and decisions going forward.
So that's an issue that we think that we can
address in the future. The second piece is really
going back to the CIL consumer satisfaction data.
Where we don't have standardized data across the
centers in a state, and it's certainly not across
the country, we'd like to explore the pros and
cons of working towards that, that at least the
data is standardized at the state level. Whether
it makes sense to standardize that across the
country is a discussion to be had, but it does
make sense that at least at the state level there
would be some pros, there would be some benefit to
standardizing how the centers are collecting
consumer satisfaction data. That may be my bias
coming from a relatively small state that has six
centers. That may be a different discussion in
states that have many more.
The third recommendation is to encourage the
statewide independent living councils to really
broaden the sources of data they're pulling in,
and these are existing data sources. So figuring
out who in the community is gathering data that
could really help us get a more complete picture