Introduction

This is a brief introduction to the latest pieces that we are sending to you. The main focus of the material is on the development of a strong performance measurement/monitoring and continuous quality improvement system. We emphasize this because we have found it to be an important component of effective service systems. It is also our experience that within states that have had class action lawsuits, the inclusion of a strong performance measurement and monitoring system has been one of the most effective components for bringing about change, and for guiding that change.

In the material that we are sending, we have emphasized that a performance measurement system should be based on the overall outcomes that a system is seeking to achieve, and on the theory of what will lead to the achievement of those goals (we call this a “theory of change” approach). Basically we see two main outcomes, and have included figures to illustrate each: an increase in access to care for children in need of mental health services; and an increase in effectiveness of care for children who receive mental health services. Within the goal on improved effectiveness, the primary emphasis of our focus is not on symptom reduction but on measures of community functioning. This is consistent with the Rosie D. ruling, and the emphasis within Medicaid on improving overall functioning. It is also consistent with the recommendations of the President’s New Freedom Commission on Mental Health, which emphasized a recovery and resilience model designed to enhance community functioning rather than a more narrow model targeted at the goal of symptom reduction (See Figure 1). We encourage you to think of the development of a performance measurement system as a staged process. The system has to begin by developing capacity for identifying effective practices and outcomes, and then to benchmark effective practices, next to create and evaluate supports for reaching those benchmarks, and finally to create mechanisms for re-evaluating benchmarks. Over time, the system should become more efficient at delivering effective services; thus benchmarks should evolve over time to reflect system maturation. From this perspective, performance monitoring is a way to teach persons about system expectations for performance, monitor system progress, and to evolve the system to continuously pursue better outcomes.

Figure 1

We are therefore providing the following:

Two figures that provide an overview visualization of levels of monitoring;

A summary of general principles of outcome assessment;

A summary of measures of child outcomes and youth and family satisfaction with recommendations;

A more detailed description of what is being currently done in different domains that we consider to be state of the art.

Overall, we make the following recommendations:

  1. Whatever is done should include transparency and public presentations of findings, regular and frequent review by key decision-makers, feedback loops to those who have provided the data, and culturally and linguistically competent measures and data collection procedures;
  2. The major focus should be level of success in achieving the desired outcomes: increased access to care and access in a timely manner; increased effectiveness of care resulting in better functioning in the community, and child and family satisfaction with care;
  3. The child measures that seem to be most closely aligned with the goals of increasing community functioning, and have both the best psychometric data and field experience are the CAFAS and the CANS-MH, and the MHSIP measures of child and family satisfaction (YSS and YSSF);
  4. It is also essential to collect data on a regular basis on the process of providing services, with a focus on those processes that are believed to be most essential for achieving the desired outcomes;
  5. Overall, it is recommended that information be collected, at a minimum, on:

The screening process with regard to its timeliness, its adherence to standards, the information it provides on characteristics of children and families, and its role in providing access to care for those in need;

The treatment planning process, using a measure that involves direct observation such as the Wraparound Observation Form;

The overall treatment process, using an in-depth approach such as Service Testing and System of Care Practice Review, a measure that looks at the whole process such as the Wraparound Fidelity Index, and through gathering data, much as Hawaii does, on the implementation of the recommendations of the treatment planning process;

The actual services provided and the cost of the services;

Periodic semi-structured interviews with key stakeholders from the mental health and other related systems, and also with parent leaders, to secure their feedback on how well the system is functioning and what might be done to improve it;

Child outcomes, as measured by the CANS-MH or the CAFAS, with overall data also being disaggregated to determine the degree of change for children with different constellations of problems, and different backgrounds;

Child and family satisfaction using measures such as the YSS and YSSF and also incorporating within the Service Testing or SOCPR some less structured questions on the child and family experience with treatment;

System outcomes, such as number and percentage of children being served in out-of-home care, and percentage of children in need who are receiving home and community-based services

Figure 2

Performance monitoring is an essential element of system reform and continuous quality improvement. Planful identification of key data points and feedback loops that inform practice and decision making can be powerful instrument for system change. Such data can be used to highlight meaningful progress and areas for improvement at the agency and system levels, as well as supervision tools and a means of monitoring the progress of individual children and families. This seems to be particularly effective when the expectation is that the results of the performance monitoring will be presented publicly. Public dissemination of system or agency ‘report cards’ have been used as system reform tools in Hawaii, where regional directors are required to present the results for their region each quarter to a meeting of key stakeholders including state-level administrators, legislators and staff, and family members, and quarterly reports are posted on the agency website for the use of consumers, key decision makers and the general public alike. Likewise, the state of Indiana posts consumer satisfaction data for all of its community mental health centers for review and use in health care decisions. The transparency and power of such methods of accountability extends beyond the behavioral health sector – for example, in his administration as New York Mayor, Giuliani required city agencies to post a set of relevant indicators on a weekly basis (Giuliani, 2002).

Recognizing the both the need for information on which to base decisions, as well the need for parsimony, it is suggested that performance monitoring systems be premised on the principles of relevance, timeliness and appropriateness. Relevant systems answer the questions, ‘What worked?’, ‘What didn’t work?’, ‘What could be better?’. Successful outcomes should be defined at the appropriate level, but should originate from the point of view that children and families are experiencing improvements in their well being and are satisfied with the services and supports they receive.

While Massachusetts currently collects some system level indicators, such as readmissions to acute settings (Commonwealth of Massachusetts, 2004), and child and family level indicators, such as change scores on the Child and Adolescent Functional Assessment Scales (Hodges, 1994), it is recommended that the current system monitoring efforts be expanded to include indicators that reflect the processes and outcomes at the child and family level, the agency or provider level and at the system level. Determining the key activities at each of these three levels, monitoring these activities, establishing rapid and ongoing feedback loops for each activity and making this information easily available to all stakeholders is essential to system reform efforts. In this section, key activities at each level are identified and suggestions for indicators are proposed.

While service systems are inherently dynamic, discussion herein will be discussed in terms of three levels; child and family level, the agency and / or program level, and, the service system level. Within each of these levels, there are multiple points at which data should be collected and inform processes and decisions internal to these domains. Ideally, information also flows between the levels.

Child and family level – just as a key principle of service and support delivery is that services are family driven and child focused, development or reform of performance monitoring systems should begin with the examination of key points in service and support delivery.

Screening and comprehensive assessment– Most children and families first encounter the system at the point of screening, and so it is critical that data be collected on how well the screening function is being carried out, as well as on child and family characteristics. These data inform activity at the service system level, the agency and within the child and family level (See Table 1 below). Those children, youth and families that ‘screen in’ must participate in a comprehensive assessment process that is ecological, encompassing all life domain areas, and inclusive of child and family strengths to be used in service and support planning.

In Michigan, for example (Wotring, personal communication, 2006), the Child and Adolescent Functional Assessment Scales (CAFAS) is used as one element of eligibility criteria for services and is also used in the determination of the most appropriate level of care, e.g., clinic based or intensive home based services (Hodges & Wotring, 2000). The CAFAS is then collected quarterly while the child is enrolled in services and at discharge. These data are used in service planning and monitoring at the child and family level. They are also used in supervision of direct care staff, at the agency level. At the agency level, data are also used to assess whether children and families are being served at the appropriate level of care. If, for example, children served in outpatient services were not improving over time, the plans of these children would be revisited, to determine a more appropriate type or intensity of services. At the system level, these data are also aggregated. One example of how these data are used was in the selection of evidence based practices that match the characteristics of the children and families being served.

Table 1

Activity / Data element(s) / Level Informed/ Level of aggregation / Examples of data use
Screening & comprehensive assessment / Child and family characteristics & functioning / System level /
  • Planning service mix that addresses needs and characteristics of the population
  • Eligibility determination

Child and family characteristics & functioning / Agency & direct care staff level /
  • Ensuring staff characteristics, staff development, and agency policy are appropriate for children and families served (cultural competence);
  • Eligibility determination

Child and family characteristics & functioning / Child & Family level /
  • Needs and strengths are identified for further assessment and use in service planning.

Treatment, service and support planning, and service and support implementation – The activity areas of treatment / support planning and service and support implementation are difficult to disentangle once an initial plan has been developed because they are joined in an iterative cycle of continuous planning, service delivery, monitoring and feedback. For this reason, the two areas of activity are discussed in a single section.

On completion of the comprehensive assessment and outcomes monitoring tool, e.g., CAFAS or CANS, the needs and strengths of the child and family identified in the assessment guide the service and support planning process, as well as inform other levels within the service system. (See Table2) At this level, emphasis of data should focus on the quality and inclusiveness of the planning process. Quality should include measures of whether identified services address the needs and strengths of children and families across life domain areas, are inclusive of professional and informal supports that touch the family, and are consistent with principles of family-driven and culturally competent care. Of equal importance is the collection of the intensity and duration of identified services and interventions to meet the needs and build on strengths.

While most states collect information on the type, intensity and duration of services, if only for billing purposes, few states systematically capture information on the process of service planning. Two notable examples of states that do collect and use these data are Hawaii and Arizona. [Please refer to treatment planning information sent under separate cover.]

Hawaii uses multiple data sources to monitor service planning and delivery. One approach to better understanding service delivery is in Hawaii’s requirement that providers submit monthly reports that 1) document the service type, intensity and duration, 2) include what it calls ‘core practice elements’, i.e., the nature of the intervention, e.g., cognitive behavior approach, 3) documents progress toward treatment or service goals, and 4) provides the child and family specific outcomes of services. In tandem with functioning and level of care data gleaned through quarterly completion of the CAFAS and CALOCUS, care managers[1] (a.k.a. case managers / care coordinators) use service data in planning and service delivery for individual child and families. Likewise, their supervisors have access to these data for individual supervision and staff development purposes. These data are also aggregated and reviewed at the Branch (regional) and state levels to identify effective services for expansion and to provide corrective feedback to provider agencies.

Hawaii also uses a qualitative approach to understanding service planning and practice, called service testing (Foster & Groves, need cite). This process uses case level data, i.e., the services and processes that support individual children and families, to provide agency and system level feedback on system performance. The service testing approach used by Hawaii uses teams of state employees and Parent Partners (caregivers of children with EBD) that conduct interviews with caregivers, youth, providers and state employees. These data are then used to provide feedback at the agency level to identify strengths and weaknesses in their work with children and families, and identify agency processes and policies that promote or inhibit improvement in child and family functioning. These data are also aggregated for use by state level committees to assess system performance and provide feedback to the executive level staff. The use of methods like service testing and system of care practice review (SOCPR) is time intensive and relatively costly but seems to be well worth it—it appears to be one of the most powerful mechanisms for genuinely determining how well the system is functioning, and how well young people are doing, and for identifying areas in need of improvement.

Specific measures that address the service planning and / or delivery process used elsewhere, although not on a statewide level, include the Wraparound Fidelity Index (WFI, Bruns, Burchard, Suter, & Force, 2005), the Wraparound Observation Form (WOF, Epstein et al, Epstein M. H., (1998) and the System of Care Practice Review (SOCPR, Hernandez, Worthington & Davis, 2005). Measures like the WFI are completed by direct care staff, caregivers and youth, and are brief and simple to complete, and would be relatively easy to incorporate into practice to provide rapid feedback to those making service and support decisions, the multidisciplinary child and family team. These data can then be used to guide subsequent planning and activities, in supervision of direct care staff and to identify system barriers to seamless service provision.

Measures such as the WOF and the SOCPR provide detailed information on staff behaviors and system factors that facilitate or impede provision of appropriate services in the context of the child and family team process and service delivery. Administration of theses measures for a sample of youth and families, provides rich information that can guide advocacy for change at the system level and in supervision. Further, both measures are in the public domain and can be incorporated into and enhance existing quality assurance processes or be used to frame current monitoring. In our work with local communities in Florida, practitioners and families are very receptive to feedback generated by these two measures, because strengths are identified, the areas for improvement identified are specific in nature, feedback is provided in the context of quality assurance, and enough detail is provided to generate plans for coaching and supervision. The combination of quantitative and qualitative data collection within a single measure, means data can be aggregated for use at the policy level, for example, to advocate when service gaps are identified or when cross system funding or training is called for. These more intensive data collection procedures, such as service testing, SOCPR, and WOF, can be used for a sample of children and families—they need not be used for all children.