Section4.Appendices

Appendix A.The Development ofthe Massachusetts EWISModel

(updated February2016)

Introduction

TheMassachusettsEarlyWarningIndicator System(EWIS)is based onseveralyears ofresearch examiningstatedata andidentifyingthosedatathatcontributetoindicators thatcanmostaccurately predictthelikelihoodthata studentwilleither achieve ormisskeyacademicmilestones. Byhelpingto earlyidentifystudentswhoareatrisk,theEWIS allows educatorstointerveneandsupportstruggling studentstohelpthemget back ontrack.

Background

TheMassachusettsDepartmentofElementaryandSecondaryEducation(ESE)createdEWIS inresponse todistrictinterestintheEarly WarningIndicator IndexthatprecededEWIS;the indexidentifiedrising grade9studentswhowereatriskofnotgraduatingfromhighschool.Perdistrictrequest,EWISwas createdtoincludetheidentificationofstudentsin grades1–12 whoareatriskofmissingkeyacademic outcomesormilestones.

Academic Milestones and Age Groups

ESE collaborated withAmericanInstitutes for Research (AIR)todeveloptheEWIS riskmodels for each gradelevel andgradecluster(alsocalled agegroups;e.g., earlyelementaryschool, lateelementary school,middle school, andhighschool).Althoughtherearesome commonindicators across agegroups andgradelevels,the risk models dovarybygradelevel.A teamfromESEworkedcloselywith AIRto determinetherecommended modelsforeachgradelevel,andanEWIS advisory groupfromESEand otherstateagenciesreviewedtheresearch findingsanddiscussed thekeydecisions.

For each gradelevel,academicmilestoneshavebeen identified thataredevelopmentallyappropriate, based onavailablestatedata,importantto thesuccess ofMassachusetts students, and meaningful and actionableforadult educatorswhoworkwith thestudentsineach gradegrouping. Exhibit A-1 overviewstheagegroups andtheacademicoutcomesormilestones.

ExhibitA-1.MassachusettsEWIS AgeGroups,GradeLevels,andAcademicMilestones

AgeGroups / Grade
Levels / AcademicMilestones
(ExpectedStudentOutcomesfor EachAgeGroup)
Early elementary / 1–3 / Readingbytheend of grade3:ProficientorAboveProficientonthe grade3ELAStateAssessment
Late elementary / 4–6 / Middle school ready:ProficientorAdvancedonthegrade6ELA and mathematicsStateAssessment
Middle grades / 7–9 / Highschoolready:Passinggrades onallgrade9courses
Highschool / 10–12 / Highschoolgraduation:Graduatingfromhighschool infouryears

EWISRisk Levels

TherearethreeEWISrisklevels: low,moderate,andhigh.Theserisklevelsrelatetoastudent’spredicted likelihoodforreachinghisorheragegroup’s definedacademicmilestone.Arisklevelquicklyprovides informationtoeducatorsaboutthelikelihoodthatastudentwillorwillnotmeetanacademicmilestone.In otherwords,therisklevelindicateswhetherastudentiscurrentlyontrackorofftracktoreachthe upcomingacademicmilestone.Forexample,astudentwhoisidentifiedasbeingatlowriskis predictedas likelytomeettheacademicmilestone.

A risklevelisassignedto everystudentwho was enrolled inaMassachusetts publicschool andfor whom therearestateleveldata in theprioryear.Therisk levelsare determinedusingdata fromthe previousschoolyear. Therisk levelsare determinedonan individual student basis,noton astudent’s relativelikelihoodofachievinganacademicmilestonewhen compared with otherstudents. Asaresult, therearenosetamountsof studentsforeach risklevel. Forexample, itis possibletohaveall studentsin the low-riskcategory. Exhibit A-2overviewstherisklevels.

ExhibitA-2.MassachusettsEWIS StudentRisk Levels

LowRisk / Likelytoreach theupcomingacademicmilestone.Approximately 90percentofstudentswho are atlow riskwillmeetthis academicmilestonewithineachage group.
Moderate Risk / Moderatelyatriskfor not reachingtheupcomingacademicmilestone. Approximately60percentof students atmoderateriskmeetthisacademic milestonewithineachagegroup.
HighRisk / At risk for notreachingtheupcomingacademicmilestone.Approximately
25percentof studentsathighriskmeetthisacademicmilestonewithineachage group.

TodeveloptheEWIS risklevels,researchers used a rigorousstatisticalmethod19tocreateavalidEWIS model for each gradelevelfromelementarythroughhighschool.The model allowsusers toidentify studentswhoareatriskofmissing keyacademicmilestones withinthe K–12 educational trajectory.

Earlywarningsystems cannotpredictwith 100percentaccuracy whetherstudentswillachieve academicoutcomes.

ESE intentionallydesignedEWIS torelysolelyondataavailable fromexistingstatewidecollections. The data included comefromseveraldata sources:theStudent Information ManagementSystem (SIMS)and Student CourseSchedule(SCS)data collections, aswellasassessment dataresultsfromState AssessmentsandtheAssessingComprehensionandCommunicationin English State-to-StateforEnglish LanguageLearners(ACCESS for ELLs).Dataincluded intheEWIS risk models arefromthepriorschool year,toprovidearisklevelatthe beginningof thenextschoolyear.EWIS isvalidatedandupdated annuallyasmorerecentyears ofdata becomesavailable and toaccountforchangesandshiftsindata sources.Exhibit A-4showsthe indicators usedforEWIS bygradelevels.

ExhibitA-4.Overview of theFinalEWISModel,by GradeLevel

GradeLevel
AgeGroup / EarlyElementary / LateElementary / Middle School / High School
AcademicMilestone / ProficientorAdvanced
Proficienton Grade3
ELA StateAssessment / ProficientorAdvanced on Grade6 ELA and Mathematics State Assessment / PassAll Grade9
Courses / GraduateFromHigh
School in
4 Years
IndicatorsIncluded / 1 / 2 / 3 / 4 / 5 / 6 / 7 / 8 / 9 / 10 / 11 / 12
Attendance rate / x / x / x / x / x / x / x / x / x / x / x / x
Schoolmove (ina singleyear) / x / x / x / x / x / x / x / x / x / x / x / x
In-school and out-of-school suspensions / x / x / x / x / x / x / x / x / x / x / x / x
ACCESSforELLs level / x / x / x / x / x / x / x / x / x / x / x / x
ELA StateAssessment / x / x / x / x / x / x
MathematicsStateAssessment / x / x / x / x / x / x
Retained / x / x / x / x / x / x / x / x / x / x / x
Special education levelof need / x / x / x / x / x / x / x / x / x / x / x / x
Gender / x / x / x / x / x / x / x / x / x / x / x / x
Urban residence / x / x / x / x / x / x / x / x / x
Overage forgrade / x / x / x / x / x / x / x / x / x / x / x / x
SchoolwideTitleI / x / x / x / x / x / x / x / x / x / x / x / x
TargetedTitleI / x / x / x / x / x / x
Mathematics courseperformance / x / x / x / x / x / x
ELA course performance / x / x / x / x / x / x
Science courseperformance / x / x / x / x / x / x
Social studies course performance / x / x / x / x / x / x
Noncore courseperformance / x / x / x / x / x / x

ACCESSfor ELLLevelswas introducedin 2012-13;prior tothatyear MEPAwasused.

Low-income,as definedbyeligiblefor freeandreducedpricelunch, was includedin previousiterationsof EWIS. However, in2015,severalcommunitiesin MassachusettsparticipatedintheUSDACommunity Eligibility Programandwithoutfree andreducedpricedlunch datafromsome of thestateslargestdistricts, ESE developeda new metric,“Economically disadvantaged”which is basedonstudentsparticipationin otherstate-administeredprograms.This measureis not availablefor previouscohortsof studentsandissubstantiallydifferentfromlow-incomesowecouldnolonger include “low-income”in model development. Once sufficientyears of “economically disadvantaged” are available,theywillbe testedfor inclusion.

Overagefor early elementary,lateelementaryandmiddleschool isdefinedas one year older thantheexpectedagefor the gradelevel.For thehighschool, studentstwoor moreyears older thanexpectedgradelevelareconsideredoverage.

Retentionvariablewasnotusedas anindicator in highschool agegrouping,becausethe variablewasdirectlyrelatedto the outcomebenchmarkinhighschools,i.e.,on-timegraduation. Retainedstudentsin thesegradesare automatically assignedas high riskbecausethey areunlikelytograduateon-time.

Limitations and Usesfor EWIS

EWIS wascreatedtoprovideeducatorswith a tooltohelp predictwhichMassachusettsstudentsmay

bemostin needof additional interventionsorsupportsbecauseof thelikelihoodthattheyarecurrently off trackfor reachingan upcomingacademicmilestone. It isnotanaccountabilitymetric. Foreach grade level,theprobabilityofastudent notmeetinganacademicoutcomehas been calculatedbasedon existing, historicaldata. Thepredictivenatureof thesedata andindicatorsisnotabsolute, andtheEWIS risk level shouldneverbethesolepieceof data used for studentplanning. For example,major life experiences(amorestableor unstableenvironment for achild, parental [or guardian] marriageor divorce,birthand/ordeathof afamilymember,etc.)mayhaveanimpactthatoffsetsthe EWIS risklevel assigned to a student. Also,school impacts,suchastargeted student interventions, canmoveastudent frombeingoff track(highrisk)tomeetinganacademic milestone. Research is clear that avarietyof factors contributeto astudent’slikelihoodof achievingkeyacademicoutcomes.

TheEWIS model is intendedasa startingpointforidentifyingandsupportingstudentswhomaybeat risk fornotreachingan upcomingacademicmilestone. BecausetheEWIS dataarecalculated usingdata collectedbyESE,districts andschools alsomay considerusinglocal datatoidentify,diagnose,and supportstudents. Formoredetailed information on EWIS development, refertoESE’sEWISTechnical DescriptionsofRiskModelDevelopmentreports (