Introduction to Pattern Checking Workshop
July 30, 2010
Pattern Checking
Included in this worksheet are a number of exercises to work through with your own data. Some example data has been provided for group discussion during the workshop. However, we recommend reviewing these questions with data from your own state or local areas.
Pattern 1 – Distribution of Scores and Progress Categories
a. Children will differ from one another in their entry scores in reasonable ways (e.g., fewer scores at the high and low ends of the distribution, more scores in the middle).
b. Children will differ from one another in their exit scores in reasonable ways.
c. Children will differ from one another in their OSEP progress categories in reasonable ways.
d. Relatively few children will have entry scores on all 3 outcomes that meet age expectations
Analysis
- Look at the distribution of rating/scores at entry and exit and the data reported to OSEP.
- Look at the percentage of children who scored as age appropriate (or not) on all three outcomes at entry and at exit.
Question: Is the distribution sensible? What do you expect to see?
Can you analyze your distribution of entry scores/ratings? Of exit scores/ratings?
Look at the distribution of your own scores/ratings and OSEP progress categories. Is the distribution reasonable? Does anything concern you about the distribution?
How does the percentage of children in progress category a look? Category e?
Here is sample data from two different states. In each case, the OSEP progress categories for outcome 1 are shown.
Arethe distributions reasonable? Does anything concernyou about the distribution?
How does the percentage of children in progress category a look? Category e?
What else would you want to know? Here is additional information about ratings from the two states:
What initial conclusions could you draw from looking at the patterns in this data?
What, if any, actions might be appropriate now that you have seen these patterns?
Pattern #8 – Relationship to “non-predicted” characteristics
Scores at entry (and exit) should not be related to certain characteristics (e.g., gender, race/ethnicity).
Analyses:
1. Frequency distributions for each group
2. Means and standard deviations for group
Question: What do we expect to see?
Make a list of non-predicted characteristicsthatcapture in your database and and can use to look at relationships to entry, exit, and OSEP categories.
1
Early Childhood Outcomes Center
Measuring Child and Family Outcomes Conference – July 30, 2010
1.
2
3.
4.
5.
6.
7.
8.
9.
10.
1
Early Childhood Outcomes Center
Measuring Child and Family Outcomes Conference – July 30, 2010
Which of these non-predicted characteristicsshould be related to entry, exit, and OSEP categories?
If you find a relationship, does it necessarily mean there is a problem with the data?
Example: Percentage of children by progress category for males and females on outcome 1.
A / B / C / D / EMales (n=312) / 1.6% / 28.4% / 29% / 23% / 18%
Females
(n=299) / 1.0% / 14% / 26.5% / 31% / 27.5%
What do you notice about this data?
What else would you want to know?
What initial conclusions might you draw?
What, if any, actions might be appropriate as a result of this data?
Pattern #3 – Comparison of Entry and Exit Scores
Functioning at entry within an outcome area will be related to functioning at exit (or – children who have higher functioning at entry in an outcome area will be the ones who are high functioning at exit in that outcome area).
Analyses:
1. Crosstabs between entry and exit scores for each outcome
2. Correlation coefficients between entry and exit scores for each outcome
Question: What do we expect to see?
Note: This analysis examines the entry score compared to the exit scores. It requires entry and exit scores/ratings. If you do not have access to these at the state level, local programs can perform the analysis at the local level.
Have you looked at the relationship between entry and exit scores?
You can also look at these relationships for children who entered in different years (’07,’ 08, ‘09) and at different ages (24 month, 30 months, etc)?
What difference would year make?
What difference would age make?
Example: State data comparing entry and exit ratings on outcome 1
Exit Outcome 1 / Entry Outcome 11 / 2 / 3 / 4 / 5 / 6 / 7
1 / 7 / 1 / 1 / 0 / 0 / 0 / 2
2 / 20 / 16 / 4 / 1 / 2 / 0 / 0
3 / 33 / 36 / 26 / 5 / 8 / 1 / 0
4 / 16 / 28 / 39 / 1 / 4 / 12 / 4
5 / 22 / 62 / 72 / 40 / 24 / 9 / 8
6 / 34 / 63 / 106 / 64 / 93 / 26 / 11
7 / 20 / 39 / 58 / 44 / 44 / 31 / 14
Note: Scores/Ratings could be groupings of raw scores (e.g., 45-60, 61-75, 76-90, 91-110, 111-125, 126-140), COSF ratings (1-7), or distance in standard deviations from the mean at entry and exit
What does the diagonal mean?
How much change would you expect to see in most cases between entry and exit?
Which child is an outlier (i.e, unusual pattern)?
What initial conclusions might you draw?
What, if any, actions might be appropriate as a result of this data?
The activity below may be helpful for becoming familiar with the meaning behind the boxes in crosstab data. It can be helpful to use this activity with other stakeholders who you share data with if interpretation is challenging.
Crosstabulation (Crosstabs) Activity
Child / Entry Score/RatingOutcome 1 / Exit
Score/Rating
Outcome 1
Z / 4 / 3
Y / 2 / 2
X / 4 / 5
W / 1 / 5
V / 3 / 4
U / 3 / 3
T / 4 / 4
S / 2 / 2
R / 5 / 5
Fill in a letter for each child in the appropriate box.
Entry – Outcome 1Exit Outcome 1 / 1 / 2 / 3 / 4 / 5
1
2
3
4
5
What does the diagonal mean?
Which child is an outlier (i.e, unusual pattern)?
1
Early Childhood Outcomes Center
Measuring Child and Family Outcomes Conference – July 30, 2010