Using Information from National Databases to
Evaluate Systemic Educational Reform
Janet H. Kane
()
Wisconsin Center for Education Research
University of Wisconsin, Madison
Presented at the annual meeting of the
American Evaluation Association
Arlington, VA, November, 2002
This research was supported by a grant from the National Science Foundation to the Study of the Impact of Statewide Systemic Initiatives (REC-9874171) at the Wisconsin Center for Education Research.
Using Information from National Databases to
Evaluate Systemic Educational Reform
Janet H. Kane
Introduction
In 1991, the National Science Foundation (NSF) launched the Statewide Systemic Initiatives (SSI) program—an ambitious effort to raise student achievement in mathematics and science throughout an SSI state by providing seed money of $10 million a year and ongoing consultation for five years to promote statewide systemic educational reform. The SSI resources were to serve as catalysts to spur the state’s reform efforts and to leverage additional resources.
Twenty-one states and Puerto Rico completed the full five years of SSI funding, and eight of the 22 SSI jurisdictions received an additional five years funding. Funding for Cohort I began in 1991, with Cohorts II and III starting in 1992 and 1993. Table 1 lists the fully funded states in each cohort. Asterisks identify states receiving Phase II funding.
Table 1
States in the Statewide Systemic Initiative Program for the Full Five Years
Cohort I (1991) / Cohort II (1992) / Cohort III (1993)Connecticut*
Delaware
Louisiana*
New Jersey*
Ohio
South Carolina*
South Dakota / California
Georgia
Kentucky
Massachusetts*
Maine
Michigan
New Mexico
Puerto Rico*
Texas*
Vermont* / Arkansas
Colorado
Montana
Nebraska
New York
In this study, the goal was not to say which SSI states had “better” or “more effective” systemic reforms, but rather to use available information to develop and refine hypotheses about the features of systemic reform that lead to statewide gains in
mathematics achievement. Using a variety of information sources, this study addresses three questions about the Statewide Systemic Initiatives:
- What features differentiate SSI states with steady increases in mathematics achievement at grades 4 and/or 8 from the SSI states with little or no increases? Are these features shared by SSI states with some increases in mathematics achievement?
- Can statewide achievement gains be attributed to a state’s SSI?
- When is statewide achievement gain an adequate measure of systemic reform efforts?
Sample
The State Assessment Program of the National Assessment of Educational Progress (State NAEP) provides common measures of student achievement for each participating state. State NAEP was first administered in 1990 at grade 8 and was expanded to include grade 4 in 1992. The sample for this report includes the 14 SSI states, i.e., two-thirds, that consistently participated in State NAEP in 1992, 1996, and 2000. Although the SSI program addressed both mathematics and science education, only mathematics achievement is included in this report.
The state longitudinal sample provides 14 case studies of systemic reform. Proposed explanations of reform effectiveness based on one or two states can be evaluated in light of the remaining states.
For each state, grade 4 and grade 8 state means on the mathematics composite score were plotted across 1992, 1996, and 2000. The states were categorized into three groups, based on mean achievement gains (see Table 2). A gain of 3.5 points was used as the cut point because the national gain averaged between 3 and 4 points for each four-year interval. The groups were:
Steady Increase: The mean mathematics composite increased by more than 3.5 points from 1992 to 1996 and from 1996 to 2000 at grade 4 and/or grade 8.
Some Increase: The mean mathematics composite increased by more than 3.5 points in one of the two four-year intervals at both grade levels.
Little/No Change: The mean mathematics score increased by less than 3.5 points from 1992 to 2000 at grade 4 and/or grade 8.
Table 2
State Groups Based on Mean Mathematics Achievement Gain
Gain fromGain fromTotal
1992 to 19961996 to 2000Gain
Grade 4Grade 8Grade 4Grade 8Grade 4Grade 8
Steady Increase
Texas* 10.79 5.61 3.96 4.65 14.75 10.26
New York 4.18 3.81 3.93 6.03 8.11 9.84
Michigan 6.38 9.52 4.63 1.58 11.01 11.10
Kentucky 4.94 4.35 1.00 4.97 5.94 9.32
Massachusetts* 2.37 4.79 5.99 5.55 8.36 10.34
Louisiana* 4.88 2.40 8.94 6.60 13.82 9.00
Some Increase
South Carolina* 0.69 0.01 7.23 5.57 7.92 5.58
Georgia -0.13 3.11 4.10 3.86 3.97 6.97
Connecticut* 5.23 5.85 2.21 2.41 7.44 8.16
Arkansas 5.64 5.34 1.21 0.71 6.85 5.05
Little/No Increase
Maine 0.57 6.62 -1.64 -0.42 -1.07 5.00
California 0.73 1.88 4.44 -0.60 5.17 1.28
Nebraska 2.21 5.12 -1.59 -2.15 0.62 2.97
New Mexico 0.54 2.36 0.03 -2.13 0.57 0.24
*Phase II states
Information Sources
Several different sources of information were used to compare and contrast the three groups of SSI states.
State NAEP–Achievement data. The State NAEP mathematics achievement test was the same as the Main NAEP. The NAEP Mathematics Framework includes five content strands: 1) Number Sense, Properties, and Operations, 2) Measurement; 3) Geometry and Spatial Sense; 4) Data Analysis, Statistics, and Probability; and 5) Algebra and Functions. Calculator use is permitted on approximately one-third of the test questions. Results reported here are based on data from public school students under conditions that did not offer accommodations to special-needs students.
State NAEP–Reform indicators. NAEP also includes a teacher questionnaire, with items about teachers’ preparation and instructional practices. The State NAEP teacher questionnaire included a wide variety of questions prior to 2000, but the number of questions was kept to a minimum in 2000.
Items from the teacher questionnaire were used to create six indicators of reform related practices (Webb, Kane, Kaufman, & Yang, 2001, pp. 107-237). The indicators are listed below.
I(RC), Relative Emphasis on Reasoning and Communication—how much reasoning and communication were addressed, relative to facts and procedures.
I(MD), Mathematical Discourse—a scale of students’ opportunities to discuss, present, and write about mathematical ideas.
I(C), Calculator Use—a scale of the extent to which students used calculators in the classroom and on tests.
I(S), NCTM Standards—a single item that asked about teachers’ knowledge of the NCTM Standards.
I(PD), Last Year’s Professional Development—a single item that asked how much time teachers spent in professional development in mathematics or mathematics education during the last year.
I(RT), Reform-related Topics Studied—a count of the number of reform-related topics teachers have studied out of the seven topics listed in the NAEP questionnaire.
The first three indicators describe teachers’ classroom practices, while the others ask about what teachers know, how much time they spent in staff development, or which topics they have studied.
Data from all states participating in the State NAEP in a given year were used to standardize the indicators to a scale with a mean of 0 and a standard deviation of 1. For each state, comparisons between 1992 and 1996 were used to identify relative changes in the indicators. Comparisons with 2000 were not possible because the items were not administered to teachers.
State reports. Project staff compiled reports on many of the SSI states, based on interviews with SSI leaders and documents about state reform efforts. The reports focus on reform efforts from 1990 through 1996 and include background on SSI, non-SSI reforms, the target population, saturation, form and systemicness, and the nature of mathematics. State reports were completed on 12 of the 14 states in the longitudinal sample, all but South Carolina and New Mexico.
Analyses of state SSIs. Clune has proposed a model of systemic reform that has been used to analyze individual SSI states (Clune, 1998). To date, 16 SSI states have been examined, including 11 of the 14 in the longitudinal sample (Osthoff, 2002). The three missing states are South Carolina, Nebraska, and New Mexico.
Evaluations of the SSI program. The National Science Foundation commissioned evaluations of the SSI program as a whole, as well as specific program components. Information from prior reports was used to supplement information from other sources.
Annual surveys of state student assessment programs. The Council of Chief State School Officers (CCSSO) provides extensive data on statewide student assessment programs, based on surveys mailed to states each Fall for the prior year’s program. In the present report, information from CCSSO’s 1995–96 and 1999–2000 surveys was used.
Findings
This section presents selected characteristics of the SSI state data from the sources listed above. The data sources provided a wide variety of information about each state. The information in the tables illustrates differences between the three groups of SSI states in the longitudinal sample.
Table 3. Table 3 presents information from interviews with SSI leaders in each state. The leaders were asked to rate the relative effort directed to each of four components of systemic reform: policy, curriculum, instruction, and accountability. Respondents rated each component on a scale from 1 to 5, with 1 representing Low Effort and 5, High Effort. Respondents were not trained to the same standards, so ratings across sites are not comparable, but ratings within sites indicate where resources were directed. The 1990 ratings indicate the extent of reform prior to the SSI, and the 1996 ratings indicate reform efforts in the third, fourth, or fifth year of the program, depending on the state’s cohort group.
The table includes ratings for two components of systemic reform, policy and instruction. The first two columns indicate the 1990 ratings, and the next two the 1996 ratings. The third pair shows the difference between the 1996 rating and the 1990 rating, with a negative sign indicating a decrease in the rating from 1990 to 1996. The last column shows the difference between the two 1996 ratings. A positive sign indicates the policy rating is higher than the instruction rating, and a negative sign indicates the instruction rating is higher than the policy rating.
Looking at the 1990 ratings, three of the 14 states in the longitudinal sample had relatively high ratings prior to the start of the SSI. All three had steady increases from 1992 to 2000 in students’ mathematics achievement.
Table 4. Table 4 presents mean ratings of selected components of Clune’s model of systemic reform. Clune’s model evaluates both the breadth and depth of each component. The values in the table present means of the breadth and depth ratings both pre-SSI and in 1997. A detailed description of the analysis system is presented in Clune (1998). The numbers were obtained through personal communication (Osthoff, 2002).
1
Table 3
Ratings by State SSI Leaders of the Relative Effort Directed to Selected Components of Systemic Reform—Scale of 1 to 5, with Anchor Points of 1 (Low Effort) and 5 (High Effort)
19901996ChangeDifference
PolicyInstructionPolicyInstructionPolicyInstructionin 1996 Ratings
Steady Increase
Texas 5.0 3.0 4.5 4.0 -0.5 1.0 0.5
New York NA NA NA 4.0 NA NA NA
Michigan 4.0 4.0 4.0 4.0 0.0 0.0 0.0
Kentucky 4.0 4.0 5.0 5.0 1.0 1.0 0.0
Massachusetts 1.0 1.0 4.0 4.0 3.0 3.0 0.0
Louisiana 1.0 1.0 3.5 3.0 2.5 2.0 0.5
Some Increase
South Carolina ------Ratings not available ------
Georgia 1.0 1.0 4.0 4.0 3.0 3.0 0.0
Connecticut 2.5 1.5 3.5 3.0 1.0 1.5 0.0
Arkansas 2.0 1.0 5.0 5.0 3.0 4.0 0.0
Little/No Change
Maine 1.0 2.0 4.0 5.0 3.0 3.0 -1.0
California 2.5 1.0 1.5 2.5 -1.0 1.5 -1.5
Nebraska NANA 3.0 5.0 NANA-2.0
New Mexico------Ratings not available ------
Note: Values are comparable within site but not across sites, since raters were not trained to the same standard.
NA – Not available
Table 4
Mean Ratings of Selected Components from a Model of Systemic Reform
aInterviews were conducted in Spring, 1997. Cohort I had completed five years of Phase I funding, Cohort II was completing the fifth year and Cohort III was
in the fourth year.
Mean ratings for state policy and infrastructure are reported both pre-SSI and near the end of the Phase I funding. In addition, ratings for instructional reform are presented for 1997. Change during the SSI was computed by subtracting the pre-SSI values from the corresponding 1997 values. The last column in the table reports the difference between the 1997 ratings for policy and instructional reform.
Ratings generally went up during the pre-SSI to 1997 period for all states, with the one exception of California.
Tables 5. Tables 5a and 5b list selected characteristics of state mathematics assessment programs in 1996 and 2000, coinciding with the years the State NAEP in mathematics was administered. The information was compiled from the annual surveys of CCSSO (Council of Chief State School Officers, 2001; Roeber, Bond, & Braskamp, 1996).
The first column specifies whether the assessment is criterion-referenced or norm-referenced. In this context, criterion-referenced means that student performance can be mapped back to specifications in the state standards, while norm-referenced is a set of items assembled to identify differences among students. One state, Maine, had an innovative state assessment comprised primarily of open-ended items, with the tests containing different sets of items that were selected via matrix sampling to provide for broad content coverage.
Table 6. Table 6 provides a brief summary of the statewide assessment programs in mathematics for the 14 states in the longitudinal sample, to complement the information in Table 5. Major changes to the assessment between 1995 and 2000 are noted.
Table 5a
Characteristics of State Assessment Programs in Mathematics and Accountability Policies, 1996
TypeTimeNegativeHigh School
OfGradesofConsequencesGraduation
TestTestedTestingfor SchoolsTest
Steady Increase
TexasCRT3-8;10-12Oct, SpringWrn,PWL,TO,DisYes
New YorkCRT3,6SpringWrn,PWL,TOYes
MichiganCRT4,8,11Sept, Oct, MarWrn,PWL,TO,DisNo
KentuckyCRT5,8,11SpringWrn,PWL,TO,DisNo
MassachusettsCRT4,8SpringNoneNo
LouisianaCRT/NRT3,5,7/4,6Spring/SpringNoneYes
Some Increase
South CarolinaCRT/NRT3,6,8,10/4,5,7,9,11Spring/SpringTOYes
GeorgiaMS/NRT3,5,8,11/3,5,8,11Spring/SpringNoneYes
ConnecticutCRT4,6,8,10FallNoneNo
ArkansasNRT/CRT5,8,11/4Fall/SpringNoneNo
Little/No Change
MaineMS4,8,11SpringNoneNo
California------Statewide assessment program limited to end-of-course high school exams------
Nebraska- ------No statewide assessment program ------
New MexicoNRT/CRT3,5,8/10SpringNoneYes
Type of Test:Possible Negative Consequences for Schools:
CRT – Criterion-referenced testWrn – Give warnings to schools
NRT – Norm-referenced testPWL – Put on probation or watch list
MS – Matrix-sampled testTO – Take over schools
Dis – Dissolve schools
Table 5b
Selected Characteristics of State Assessment Programs in Mathematics and Accountability Policies, 2000
TypeTimeNegativeHigh School
OfGradesofConsequencesGraduation
TestTestedTestingfor SchoolsTest
Steady Increase
TexasCRT3-8;10-12VariousWrn,PWL,TO,DisYes
New YorkCRT4,8SpringWrn,PWLYes
MichiganCRT4,7SpringWrn,PWLNo
KentuckyCRT/NRT5,8,11/3,6,9Spring/SpringNoneNo
MassachusettsCRT4,8,10SpringNoneNo
LouisianaCRT/NRT4,8/3,5,6,7,9Spring/SpringNoneYes
Some Increase
South CarolinaCRT/NRT3-8/5,8,11Spring/SpringWrn,PWL,TOaYes
GeorgiaCRT/NRT4,6,8/3,5,8Spring/SpringWrn,PWLaYes
ConnecticutCRT4,6,8,10FallWrn,PWLNo
ArkansasNRT/CRT5,7,10/4,8Fall/SpringWrn,PWLbNo
Little/No Change
MaineCRT(MS)4,8,11Fall,SpringNoneNo
CaliforniaNRT2-11SpringNoneNo
Nebraska------No statewide assessment program ------
New MexicoNRT3-9SpringPWLYes
aConsequences based on the results of the criterion-referenced tests.
bConsequences based on the results of the norm-referenced tests.
Type of TestPossible Negative Consequences for Schools:
CRT – Criterion-referenced testWrn – Give warnings to schools
NRT – Norm-referenced testPWL – Put on probation or watch list
MS – Matrix-sampled testTO – Take over schools
Dis – Dissolve schools
Table 6
Brief Descriptions of Selected State Assessment Programs at the End of the 1990s
Steady Increase States:
New York’s state testing program is the oldest in the nation, first administered in 1865. Tests are based on the learning standards, and results provide a level of accountability for state schools. In 1998-99, the Board of Regents adopted new standards and approved the development of new tests based on the standards.
The Texas Assessment of Academic Skills (TAAS), a criterion-referenced program that assesses mathematics in grades 3 through 8, began in 1990. New standards, Texas Essential Knowledge and Skills (TEKS), were adopted in 1997 and changes were made to TAAS so it would be aligned with TEKS
by 1999-2000. Legislation in 1999 mandated a new testing program for 2002-2003.
In Michigan, the next generation of MEAP tests was under development, based on the new curriculum content standards approved in 1995.
In Kentucky, the Kentucky Instructional Results Information System (KIRIS) was replaced with the Commonwealth Accountability Testing System (CATS), required by legislation passed in 1998.
The Massachusetts Assessment Program was first administered in 1986 and then in 1990, 1992, 1994, and 1996. A new state assessment system was authorized by legislation in 1993, and tests based on the new curriculum frameworks were first implemented in 1997–98.
In 1996, the Louisiana Assessment Program had criterion-referenced tests in grades 3, 5, and 7 and norm-referenced tests at grades 4 and 6. New content standards were adopted, and criterion-referenced tests based on the mathematics standards were implemented at grades 4 and 8 in 1998–99, along with norm-referenced tests at grades 3, 5, 6, 7, and 9.
Some Increase States:
Connecticut had criterion-referenced tests, with implementation of the third generation of tests scheduled for 2000-2001. In Connecticut, tests are administered in the Fall.
Arkansas had a norm-referenced test in 1996 and expanded to include criterion-referenced tests along with the norm-referenced tests in 2000.
Georgia administered both criterion- and norm-referenced tests to the same grades in 1996, but moved to norm-referenced tests in 2000.
South Carolina had criterion-referenced tests in grades 3, 8 10 and 11 and norm referenced tests at the other grades in 1996. By 2000, criterion-referenced testing was expanded to grades 3 through 8 and 10 to 12, and norm-referenced testing of a sample of students was continued at grades 5, 8, and 11.
Little/No Change States
California’s criterion-referenced testing program, the California Learning Assessment Program (CLAS), was discontinued in 1994–95 as a result of the governor’s veto. Local districts were encouraged to select their own standardized tests. Legislation in 1995 and 1996 required development of new standards in the major subject areas and a statewide pupil assessment program. By 2000, California was using the Stanford Achievement Test, Ninth Edition, in grades 2 to 11, supplemented with standards-based test items.
New Mexico administered the Iowa Test of Basic Skills to students in grades 3, 5, and 8 in 1996. In 2000, the state used a different norm-referenced standardized test, the California Test of Basic Skills 5/TerraNova, supplemented with items linked to state standards, in grades 3 through 9.
Nebraska had no statewide testing program. Local districts were required to select a test for their reporting requirements.