Smarter Balanced

Student Interaction Study FAQ–2014

2014 Smarter Balanced Student Interaction Study

Purpose

The student Interaction (SI) Study is an in-depth, qualitative study to evaluate how students interact to Smarter Balanced assessment items or tasks. The study utilizes cognition laboratories (cog labs)that prompt student interactions with assessment questions by asking students to respond to items/tasks and then engage in “think aloud” interviews.

Participating students will represent English language learners (ELL) and students with disabilities (SWD) as well as students who have performed in the lowest and highest performance levels on state summative assessments.

Student and Site Participation

The study will include approximately 700 students from across Smarter Balanced Governing States. These students will be enrolled in grades 3, 4, 6, 8, and 11.The study will target student subgroups representing the following:

  • English language learners (ELL),
  • Students with disabilities (SWD),
  • low performing* students, and
  • high performing*students.

*performance of students will be based on previous state assessment results in Math and Reading.

Representation of gender, ethnicity, race, and socioeconomic status within each of the targeted student subgroups will also be considered.

The students will be selected from districts and schools within the Governing States. CTB/McGraw-Hill is expecting 35 districts or sites to participate in the study with approximately 20 students per site.

Data Collection Requirements

The student characteristics of socioeconomic status (SES), school type, race or ethnicity, gender, ELL status, performance on state assessments, home language, and special education status will be collected at the school and classroom level. Classroom level data will be collected prior to the labs to identify the sample. Student level demographic data will be collected at the time of administration to evaluate the sample and analyze results. Table 1 below defines the data elementsrequested for the study organized by level of information (student or school).

Table 1: Data Collection

Data Collection Level / Data Type / Definition
Student Data / Home language group / The student’sprimary language spoken in their home.
Disability / The primary disability designated on a student’s Individualized Education Plan (IEP)
Performance level on state summative assessment in Math and Reading / Overall Performance Level on the previous year’s state summative assessment
Accommodations/designated supports for summative assessment / Accommodations or designated supports for the summative assessment as provided in the student’s IEP.
Number of computers at home / The number of computers in students home that he/she would have access
Gender / Designation of Male of Female
Race/ethnicity / Race/ethnicity based on the federal definition used for school accountability reporting
Socioeconomic status / Socioeconomic status based on the student’s participation in the free and reduced lunch program
School Data / Community type / Type of community Urban, Suburban, Rural
Socioeconomic status / Percent of students participating in the free/reduced lunch program.
Number of Computers per classroom / Average number of computers provided in the schools classrooms that are accessible to students.
Performance on state assessments in Math and Reading / Percent of students in each of the performance levels on the state assessments in Math and Reading disaggregated by represented student populations

Study Approval

The SI design and Plan was reviewed and approved by the Smarter Balanced Technical Advisory Committee (TAC). Members of the TAC are highly regarded national experts who have been widely published in the fields of assessment design, assessment accommodations, uses of tests, mathematics, and English language literacy.

In addition, a research functional group represented by Governing State members, Smarter Balanced Staff, and CTB/McGraw-Hill, American Institute of Research (AIR), and Data Recognition Corporation (DRC)staff will review and oversee the study throughout the development and implementation.

Benefits of the Study

Your district’s/school’s participation will contribute to the Smarter Balanced commitment of creating fair summative assessments that give access to all students, and developing assessments that produce scores that can be interpreted validly. The assessment program’s success at achieving these goals must be evaluated through multiple methods. The information collected in this study evaluates whether students’ cognitive processes align to item development targets. Smarter Balanced will use the evaluation findings to inform future development of assessments that will provide valid, reliable and useful results.

In addition, a small incentive will be provided to each participating district/site.

Risks of the Study

There are no known risks to this study. Study administrators will be experienced and trained to minimize any discomfort to student participants. Full disclosure regarding study details will be provided to students, parents, and school staff. The records of this study will be kept private. Only summary information and information that does not make it possible to identify students or student responses will be included in any published articles or publications. The study will summarize findings across the sample and will not associate any responses/quotes with a specific student, school, district or state.

Study Schedule

CTB/McGraw-Hill will develop localized plans for each district with key dates that consider the school and district needs and coordinate administration dates with other testing and vacation demands. Table 1 below provides a high level schedule for activities associated with the study.

Table 1: High level Schedule

Activity / Timeline
Participant Recruiting / February- April
Administrator Trainings / March through April
Administer Cognitive Labs / April - May
Research Analysis / April - May
Deliver Cognitive Lab Technical Report / June

Expected Time Commitment for Sites

The total time required for administering cog labs will include test delivery setup, study administration and training, and student data collection activities. Initial setup, training and administration-related tasks are estimated at three hours.

In addition, each student protocol administration is estimated between 30 to 45 minutes including a short break. The total time required for administration of student protocols will depend on the total number of students and the total number of study administrators. CTB/McGraw-Hill will work with each district to determine acceptable time commitments based on the districts capacity and other educational demands.

All student protocols for a specific district are expected to be administered within two days. However, this is dependent on the availability of the districts or sites computers and space for administration of protocols.

District or Site Responsibilities

As a participating district, you will:

  • Supply location for administering student protocols.
  • Supply computers for students to view test items/tasks. This includes minimal setup activities for computers to administer secure items. Study administrators will conduct the setup.
  • Recommend students for recruitment.
  • Provide student characteristic data for participating students (i.e. socioeconomic status (SES), school type, race or ethnicity, gender, ELL status, home language, special education status,primary disability, and previous performance results on state assessments in Math and Reading ).
  • Support study administrators during protocol administration (e.g., answer questions, support student administrator for some students with special needs, encourage students).

Technology Requirements

During the study students will be administered protocols which include questions from the Smarter Balanced Field Test. These questions will be delivered from a secure browser. Districts will need to download the secure browser application on all computers used to administer student protocols. In addition, one computer will need to be dedicated to administration tasks for the delivery of protocols.

The protocol delivery system includes the same technical requirements as the Smarter Balanced Field Test and will support the following systems:

  • Desktops/Laptops:
  • Windows XP (with Service Pack 3), Vista 7, 8.0, 8.1: Windows Secure Browser 6.3
  • Mac 10.4 (all) and 10.5 (PowerPC): Mac Secure Browser 5.6
  • Mac 10.5-10.9 (Intel): Mac Secure Browser 6.3
  • Linux Fedora Core 6+, Ubuntu 9-12: Linux Secure Browser 6.3
  • Tablets:
  • Apple iPad running iOS 6.0-7.0: Mobile Secure Browser from App Store
  • Supported iPad versions: 2, 3, Retina display
  • Android tablets running OS 4.0.4-4.2: Mobile Secure Browser from Google Play Store
  • Supported Android tablets: Google Nexus 10, Motorola Xoom, Motorola Xyboard, Samsung Galaxy Note (10.1), Samsung Galaxy Tab 2 (10.1)
  • Google Chromebooks:

AIR has worked with Google to develop a secure kiosk application for use with Chromebooks. The application can be added to both managed and non-managed Chromebook devices.

Study Administrators and Training

Trained study administrators provided by CTB/McGraw-Hill, American Institute of Research (AIR), and Data Recognition Corporation (DRC) will administer the study protocols. Teachers/School personnel who will be supporting the study are requested to participate in a 30 minute, self-paced, online training covering the support activities. The training topics will include how to identify student participants, how to register students, and how to support study administrators.

All study administrators will pass background checks and complete fingerprinting. Qualifications will be reviewed to ensure, when applicable, that they have experience working with special populations.

In addition, study administrators will participate in training related to secure data handling, protocol administration, and data coding.

1