April 2016 /
Child Outcomes Summary (COS)
Quality Self-Assessment Process
Overview
The Child Outcomes Summary (COS) Quality Self-Assessment Process is intended to help early childhood school district leaders participate in the Department of Public Instruction (DPI) Exceptional Children (EC) Division’s local education agency (LEA) self-assessment that is part of its State Systemic Improvement Plan (SSIP), a requirement of the Individuals with Disabilities Education Act (IDEA). The LEA self-assessment replaces the previous LEA Continuous Improvement Program Plan (CIPP). Regional EC Division consultants provide professional development to LEA EC Directors/Leaders during regional meetings each year on the overall LEA self-assessment process.
The sections of the LEA self-assessment process that directly align with the COS process Quality Self-Assessment include: Core Element 4: Problem-Solving for Improvement and Core Element 6: Communication and Collaboration. Upon analyzing the child outcome data, an LEA may use the data to identify the need for research-based initiatives to ensure students’ mastery of the North Carolina Foundations for Early Learning and Development, preschool standards that align with the NC Standard Course of Study, which addresses Core Element 5: Research-based Instruction and Practices.
The purpose of the COS Quality Self-Assessment is to ensure that the LEA’s child outcomes data are of high quality through a system for monitoring the data and taking appropriate follow-up action as needed. Professional development resources are available on understanding the Child Outcome System, children’s developmental trajectories, and explaining child outcomes to families (http://modules.nceln.fpg.unc.edu/outcomes/module-intro).
The following documents provide guidance for engaging in the COS Quality Self-Assessment Process:
· COS Program Quality Self-Assessment
· COS Data Quality Self-Assessment
· COS File Review for Supporting Evidence
· Charting a Child’s Progress Against Age-Expected Development
For assistance with these materials, please contact your regional Early Learning Network regional consultant (http://nceln.fpg.unc.edu/staff).
Child Outcome Summary (COS) Quality Self-Assessment Process: Program Quality Self-Assessment
Program Administrator(s) Completing Self-Assessment:
Date of Self-Assessment:
Question / Level of ImplementationNot Yet / In Process / Achieved / Don’t Know
1. COS initial ratings for all children are monitored to ensure data are in place two times during a reporting period
2. COS exit ratings for all children are monitored to ensure data are in place prior to submission at the end of reporting period
3. Process in place to ensure that data is reported for all children who exited the program who were in the program for at least 6 months of service (less than 5% of missing data)
4. File review for quality of supporting evidence for rating is conducted on at least 10% (sampling) of enrolled children, or a minimum of 10 files (see, File Review for Supporting Evidence tool)
5. Data analysis for overall program is conducted at least annually (see, Data Quality Self-Assessment tool)
6. Stakeholder process including families is conducted regularly to consider program implications of child outcome data (e.g., administrators, teachers, related service providers, and family members)
7. Leadership team that oversees the COS program data analysis develops follow-up long-term and short-term program improvement plans
8. Information on “Understanding the Child Outcome System” is regularly provided to families
http://modules.nceln.fpg.unc.edu/node/26
9. Professional development on the COS system is planned and delivered at least annually to all staff (teachers and related service providers)
http://modules.nceln.fpg.unc.edu/outcomes/module-intro
Child Outcome Summary (COS) Quality Self-Assessment Process: Data Quality Self-Assessment
Program Administrator(s) Completing Self-Assessment:
Date of Self-Assessment:
Question / Yes / No / Don’t Know1. COS data patterns across progress categories for each outcome are consistent with expected distribution
2. Comparison of entry and exit ratings by rater/team shows appropriate variety of ratings ( e.g. not all children given a rating of 6/7 for all three outcomes)
3. COS data analysis compares patterns by different populations for this LEA to identify trends (race, ethnicity, gender, disability categories)
4. Distribution of almost all ratings at entry are centered on a lower number than exit
5. In most cases, COS ratings increase by no more than 3 points
6. Groups of children with more severe disabilities will have distributions with a larger percent in categories a – c
7. Groups of children with less severe disabilities will have distributions with a larger percent in categories d and e
Recommendations for follow up:
Overall program data analysis suggests that most entry ratings are consistently too high or too low: Develop follow-up action plan (e.g. provide staff with professional development on formative assessment and age referencing of data and use of the COS decision tree)
Overall program data analysis suggests an unexpected gain/loss in performance (e.g. as evidenced by a larger than 3 point increase over the time in program): Drill down to identify root cause and/or data quality issues; develop follow-up action plan
Analysis of data by individual rater/team shows patterns that suggest need for improvement (e.g. entry ratings of three and exit ratings of five for all children in a classroom or caseload): Develop follow-up action plan
Child Outcome Summary (COS) Quality Self-Assessment Process: Child Outcome Summary File Review for Supporting Evidence
Child:
Raters:
Reviewer:
Question / Outcome I / Outcome II / Outcome IIIYes / No / Yes / No / Yes / No
Is formative or summative assessment data provided that supports the rating in this outcome?
Is the data age-referenced and compared to the child’s chronological age for this outcome?
Does the “summary of relevant results” relate specifically to this outcome?
Does the summary of relevant results include examples of the child’s every-day functioning in this outcome?
Was there evidence of family input that supported the rating for this outcome?
Was there evidence that the rating was determined with more than one person’s input for this outcome?
Can this reviewer estimate, within one point in either direction, the rating based on the information provided for this outcome?
Total
Follow-up Recommendations:
Commend raters for job well done
Have exemplar raters assist others
Review results with rater and provide information about needed areas of improvement
Require raters to attend COS training
Charting a Child’s Progress Against Age-Expected Development
Instructions
Select a child in your classroom and plot his developmental trajectory for each of the three outcomes (starting on Page 2). To do this, you will need to find his entry Child Outcomes Summary Form ratings in his files. Then, you will estimate his level of functioning at the current point in time, based on information you have collected from assessment tools, observation, and talking with family members and other caregivers. We will call this estimation an ‘interim rating.’ Plot the child’s entry rating on the left side of the rainbow graphic. Plot his interim rating on the right side of the rainbow graphic. Draw a line between the two points. This is his developmental trajectory.
What does the trajectory tell you? Is he making progress in this outcome area? Is he staying within the same trajectory? Has he changed developmental trajectories? Is he catching up with same-age peers?
Here is an example. In the graphic below, the child’s entry rating was a ‘3,’ meaning that he was not yet showing age-expected skills. His interim rating was a ‘5,’ meaning he now shows some age-expected functioning in this outcome area. The line drawn between the points shows a steep upward slope. This tells us that, between the time the child entered the program and this point in time, he acquired new skills and accelerated his rate of growth. He made progress toward catching up with same-aged peers but is still functioning below age expectations. He changed growth trajectories and “narrowed the gap” between his level of functioning and that of a child who is developing according to age expectations.
Child Name: / Teacher:Date of Birth:
Child Outcome 1- Positive Social Skills
Observational Data
Interview Data
Assessment Data
Child Outcome 2- Acquisition and Use of Knowledge and Skills
Observational Data
Interview Data
Assessment Data
Child Outcome 3- Taking Appropriate Action to Meet Needs
Observational Data
Interview Data
Assessment Data
Definitions for Outcome Ratings:
Overall Age-Appropriate / Completely means: / 7 / • Child shows functioning expected for his or her age in all or almost all everyday situations that are part of the child’s life. Functioning is considered appropriate for his or her age. • No one has any concerns about the child’s functioning in this outcome area.6 / • Child’s functioning generally is considered appropriate for his or her age but there are some significant concerns about the child’s functioning in this outcome area. These concerns are substantial enough to suggest monitoring or possible additional support. • Although age-appropriate, the child’s functioning may border on not keeping pace with age expectations.
Overall Not Age-Appropriate / Somewhat means: / 5 / • Child shows functioning expected for his or her age some of the time and/or in some settings and situations. Child’s functioning is a mix of age-appropriate and not age-appropriate behaviors and skills. • Child’s functioning might be described as like that of a slightly younger child*.
4 / • Child shows occasional age-appropriate functioning across settings and situations. More functioning is not age-appropriate than age-appropriate.
Nearly means: / 3 / • Child does not yet show functioning expected of a child of his or her age in any situation. • Child uses immediate foundational skills, most or all of the time, across settings and situations. Immediate foundational skills are the skills upon which to build age-appropriate functioning. • Functioning might be described as like that of a younger child*.
2 / • Child occasionally uses immediate foundational skills across settings and situations. More functioning reflects skills that are not immediate foundational than are immediate foundational.
Not yet means: / 1 / • Child does not yet show functioning expected of a child his or her age in any situation. • Child’s functioning does not yet include immediate foundational skills upon which to build age-appropriate functioning. • Child functioning reflects skills that developmentally come before immediate foundational skills. • Child’s functioning might be described as like that of a much younger child*.
For Use with the Child Outcome System Child Progress Monitoring Tool
*The characterization of functioning like a younger child only will apply to csome children receiving special services, such as children with developmental delays.
NC Early Learning Network is a joint project of the NC Department Of Public Instruction, Office Of Early Learning
and UNC Frank Porter Graham Child Development Institute