Office of Research, Evaluation and Professional Learning

Online Test Scoring and Item Analysis Protocols

Protocol also included for tests not in Online Test Scoring

Objectives:

  1. Determine area(s) of focus for improved student learning
  2. Identify students to target for improved outcomes

Protocol Activities:

  1. Online Test Scoring (DSC)

Online Test Scoring Assessments for 2015: ELA Grades 2-12 (includes item analysis)

Social Studies Grades 6-11 - Averages only at this point in time

Mathematics: Middle School Algebra 1 end of year (averages and item analysis)

  • Provides reports at the teacher level, school level, and for
    administrator monitoring of district benchmark (curriculum) assessments
  • Summary of student performance:
    Average by class, Average by standard, Item analysis
  1. Item Analysis guides for Common Formative or school-based benchmark assessments
  • Use excel form for analysis or manually
  • Bambrick-Santoyo guiding questions

Additional Resources:

Test in hand

  • ELA assessments are available on the DSC module “Curriculum Tracker”. The assessment can be found in the units.
  • Social Studies materials are found at

Excel-Test Item Analysis for school-based common formative or benchmark assessments

Appendices

(Red Clay) Assessment Literacy: More Information on Assessment

(Bambrick-Santoyo) Increasing Rigor Throughout the Lesson: Data-Driven Classroom Best Practices

Reference(s): Driven by Data: A Practical Guide to Improve Instruction (2010)Paul Bambrick-Santoyo University of Virginia, School Turnaround program at UVA

Protocol 1: Online Test Scoring Analysis

Step 1. PREPARE: Accessing the Data

Log into DSC and select Online Test Scoring.

Teacher Reports

  • Item Detail & Summary Reports
  • Item Detail Report-shows class list of students with their overall score and answer selection for each item, also provides multiple choice summary of the questions most answered correctly and least answered correctly
  • Class Standards Summary Report-provides the average correct and number of questions by standard for students in the class selected
  • Class Standards Detail Report-shows class list and score (number correct and percentage) by standard

Note: each student’s name is linked to an individual standards summary report

  • Student Search- enter student name or ID for a summary of a student’s scores across all tests in the system

Administrators can run school level and monitoring reports. School Level reports and Item Validation Monitoring reports are useful for collaborative teams to review.

School Reports

  • Standards Summary Report
  • Select Summary Type “All Standards Summary by School” for a bar graph with percent correct by standard
  • Select Summary Type “Standards Summary by Course” for a bar graph with percent correct on each standard reported by course/teacher
  • Average Scores By Class- shows percent correct on overall test by course/teacher, district and school average, link below graph to each class’ item detail and standards summary
  • Average Disaggregated Summary- provides average score by sub-groups

Data Monitoring

  • School Assessments Overview(Admin)- provides school and district averages by assessment, as well as link to reports specific to each assessments
  • Item Validation Report- shows the percentage of students responding to each answer choice by question
  • Item Validation Report by Standard-shows the percentage of students responding to each answer choice by question, grouped by standard
  • Good/Bad List-(Admin)- reports completion rates: percentage of online tests entered versus expected, and scanned tests returned versus expected

Protocol 1: Online Test Scoring Analysis (continued)

Step 2: INQUIRE: Analyzing the Data

  • Come with “test in hand” (use a hard copy of the test or view the test online)
  • As you examine the reports in Online Test Scoring, respond to the following questions:
  1. What 3 data points stand out to you?
  2. What did the students do well?
  1. To what do you attribute that success?
  1. How do you address areas of deficit?
  1. What’s one struggle that surprised you?
  1. What’s one struggle you expected?
  1. Who did well (which individuals)?

Step 3. ACT: Responding to the Data

Identify targeted students and create action plans to address individual needs.

Product(s):Select one or more of the following products provided, or use one similar.

Assessment Analysis Sheet and Instructional Plan

Cumulative and Targeted Review of Standards

Six-Week Instructional Plan

Digging into Data

Student Reflection Template

Revisit (Cycle of Inquiry):

Monitor student progress.

Plan when to reassess student for mastery of standards addressed.

Determine next steps in the instructional plan for struggling students.

Protocol 2: Item Analysis of Tests not in Online Test Scoring

Step 1. PREPARE: Accessing the Data

Enter data into the excel spreadsheet or into a table (see page 10)

Step 2: INQUIRE: Analyzing the Data

Come with “test in hand”

Respond to the following questions (source: Bambrick-Santoyo):

Larger Picture Questions

  • How well did the class do as a whole?
  • What are the strengths and weaknesses in different standards?
  • How did the class do on old versus new standards taught?
  • How were the results in the different question types (multiple choice vs. open-ended, reading vs. writing)?
  • Who are the strong and weak students?

“Dig in” Questions

  • Bombed questions – did students all choose the same wrong answer? Why or why not?
  • Break down each standard – did students do similarly on each question within the standard? Why?
  • Sort data by students’ scores – are there questions that separate proficient and non-proficient students?
  • Look horizontally by student – are there any anomalies occurring with certain students?

Step 3. ACT: Responding to the Data

Identify targeted students and create action plans to address individual needs.

Product(s): Select one or more of the following products provided, or use one similar

Assessment Analysis Sheet and Instructional Plan

Cumulative and Targeted Review of Standards

Six-Week Instructional Plan

Digging into Data

Student Reflection Template

Revisit (Cycle of Inquiry):

Monitor student progress.

Plan when to reassess student for mastery of standards addressed.

Determine next steps in the instructional plan for struggling students.

Page1 | 15

Office of Research, Evaluation and Professional Learning

Assessment Analysis Sheet and Instructional Plan for Teacher: ______

Subject: ______Assessment: ______Grade/Class: ______Date______

Standards Analysis / Analysis of Why Students Did Not Learn Standard / Instructional Plan—What Techniques Will You Use To Address These Standards
What standards warrant more time for whole-class instruction, re-teaching and review
Whole class instruction:
•
•
•
•
•
•
•
Small Group Instruction: / Instructional Plan How or When Will You Structure Small Group Instruction
What standards warrant more time for small-group instruction and review?
•
•
•

Source: Driven by Data: A Practical Guide to Improve Instruction (2010)Paul Bambrick-Santoyo
Cumulative and Targeted Review of Standards

Write the standards you will address with each of the following

Spiral in Homework / Spiral in Do Now / Do Mini-Lesson / Do Now with Mini- Lesson / Spiral in Quizzes or Tests
Students Of Major Concern / What They Need Most Help With / Instructional Plan—When or How Will They Get Tutored, Supported, Addressed
•
•
•
•
•
•

Source: Driven by Data: A Practical Guide to Improve Instruction (2010)Paul Bambrick-Santoyo

Six-Week Instructional Plan for
______

WEEK 1—Date ______ / WEEK 2—Date ______ / WEEK 3—Date ______
Standards for Review / Standards for Review / Standards for Review
New Standards / New Standards / New Standards
WEEK 4—Date ______ / WEEK 5—Date ______ / WEEK 6—Date ______
Standards for Review / Standards for Review / Standards for Review
New Standards / New Standards / New Standards
Source: Driven by Data: A Practical Guide to Improve Instruction (2010)Paul Bambrick-Santoyo
Materials to bring with you include:
Your completed student assessments
Class summary Report from DSC (Online Test Scoring)
Your plan book/lesson plans
Overview chart for the NEXT unit
The common assessment for NEXT unit / DIGGING INTO DATA

Let’s look into the data and come up with a plan! / Guiding Questions:
  1. What do we expect our students to learn?
  2. How will we know they have learned it?
  3. How will we respond when didn’t learn?
  4. How will we respond if they already know it?

Step 1
Provides your team key information / 5 Minutes / Overall Conclusions / Guiding Questions:
  • How well did the class do as a whole?
  • When looking at the standards, which one(s) was a strength or area of need?
  • What were the results in different question types? (Multiple choice vs. open-ended, etc.)

Step 2
Preparation for making your plan of action / 5 Minutes / Dig in Deeper / Look vertically. Are there any outliers? For instance:
  • Were there any questions where most students missed the right answer/chose a different answer? Why?
  • In math, were there (what are the) misconceptions?
  • Were there questions that half the class chose one answer and the other half chose a different one? Why?
  • Who on your team got a different result? What strategy/technique did they use? Share.
  • Which questions did your students do really well on? Share with your team!

Step 3
15 minutes
Whole Class Instruction Adjustments
(most students missed this standard)
Make a PLAN based on the DATA! / Based on the data, what critical standards justify more time for whole class re-teaching or review?
When will you fit this in during the coming unit? (What can be taken out or where does it logically fit in?)
What different strategies or techniques will you utilize to address these standards? (Ask your team for help.)
Step 4
20 minutes
Small Group Instruction Plan
(Only a few struggled with this standard)
Make a PLAN based on the DATA! / Based on the data, what critical standards do you need to reteach in a small group?
Standard(s) / Name of Students / Strategy/Technique / When will you fit it in?

Source: S. Rifenburg and BSD

Page1 | 15

Office of Research, Evaluation and Professional Learning

Student Reflection Template(based on: Bambrick-Santoyo, Williamsburg Collegiate School, Uncommon Schools)

Standards/Skill / Did you get the question right or wrong? / Why did you get the question right or wrong?
Question / Right / Wrong / Careless Errors / Did Not know how to get the answer
1
2
3
4
5
6
7
8
9
10

Type of Error (count)

Careless Errors / Did not know how to get the answer

Hints:

If you have more careless errors than don’t knows:

  • In classwork: Slow down, stop and check your work with a teacher or partner, explain your thinking, look to see if you keep making the same mistakes
  • On assessments: Slow down, double check our work

If you have more don’t knows than careless errors:

  • In classwork: Ask questions about homework or classwork if you are not sure, use every opportunity to check in with teachers and classmates to see if you are doing work correctly
  • On assessments: Do the questions/problems you are sure about first, take your time on the others using everything you know, ask questions as soon after the assessment as possible

Page1 | 15

Office of Research, Evaluation and Professional Learning

Assessment Results Analysis Template (based on Bambrick-Santoyo)

Questions # / 1 / 2 / 3 / 4 / 5 / 6 / 7 / 8 / 9 / 10 / 11 / 12 / 13 / 14 / 15 / 16 / 17 / 18 / 19 / 20 / 21 / 22 / 23 / 24 / 25
Write standards addressed by each question in the space below
Teacher:
Course: / MULT. CHOICE: Total correct / MULT. CHOICE: % CORRECT / OPEN-ENDED: % POINTS POSSIBLE / COMBINED PROFICIENCY SCORE:
Correct Answer
Student Name
TOTAL CORRECT PER QUESTION
PERCENTAGE CORRECT BY QUESTION:
PERCENTAGE CORRECT BY STANDARD:
Standard: / % / Standard: / % / Standard: / %
Standard: / % / Standard: / % / Standard: / % / Team Performance: / Mult-Choice / Open-Ended / Combined
Standard: / % / Standard: / % / Standard: / % / % / % / %

Page1 | 15

Office of Research, Evaluation and Professional Learning

Appendix: Assessment Literacy

More information on assessment

Assessment is a valuable part of Red Clay’s educational program. This important component allows teachers to plan, document and evaluate student learning in an ongoing cycle. This teaching process is crucial to promote student achievement. Tests and other assessments provide evidence of student learning and help teachers identify next steps in their instruction.

Appendix: Increasing Rigor Throughout the Lesson: Data-Driven Classroom Best Practices (Paul Bambrick-Santoyo)

1. Objectives: Rewrite and tighten with assessments in mind:

  • Connect objective to how the students will be assessed.
  • Write “know/do” objectives: Students will know ______by doing ______.
  • Look at test questions beforehand to be sure the skills assessed on the test were worked into the daily lesson.
  • Write an assessment of the skills immediately after the objective, at the top of the lesson plan.
  • First write assessment questions that align to objective; then break the objective into smaller chunks that will ensure mastery of all the skills needed to answer each question correctly.
  • Use verbs from Bloom’s taxonomy to ensure that the objective is rigorous.

2. Do Now (five- to ten-minute individual exercise to start class)

  • Use Do Now as a re-teach tool: Write questions that students struggled to master on the last interim assessment.
  • Use mixed-format questions for a skill: multiple-choice, short answer, open-ended, and so on.
  • Organize questions sequentially according to difficulty.
  • Spiral objectives, skills, and questions from everything previously learned to keep student learning sharp.
  • Develop Do Now tracking sheet for teachers and students that shows student performance on the skills in each Do Now.
  • Make Do Nows that look like test questions and make sure they are reviewed in class.
  • Observe students’ answers during Do Now and note kids with wrong answers to follow up with them during oral review.
  • Add multiple-choice questions to Do Now to allow real-time assessment.
  • Add why and how questions (for example, Why did you choose this answer? How do you know your answer is correct?) for different levels of learners and to push thinking.
  • Revisit yesterday’s objectives in the Do Now.
  • Collect and grade four straight Do Nows, and for the fifth day let students correct their first four Do Nows for extra points toward their Do Now grades.

3. Questioning to check for understanding and increase engagement:

  • Develop whole class responses to student answers (for example, snap if you agree, stomp if you don’t) to engage 100 percent participation.
  • Use cold call: Avoid just calling on students with hands raised.
  • Move from ping-pong to volleyball: Instead of responding to every student answer yourself, get other students to respond to each other: “Do you agree with Sam?” “Why is that answer correct (or incorrect)?” “What would you add?”
  • Script questions in advance of the lesson to make sure they scaffold appropriately and address rigor at varied levels.
  • Have an observer record teacher questions: highlight where students are succeeding and where they can grow.

Appendix: Increasing Rigor Throughout the Lesson: Data-Driven Classroom Best Practices (Paul Bambrick-Santoyo, continued)

3a. Student error (techniques for helping students encounter the right answer):

  • Have a student who struggled initially repeat the correct answer eventually produced by the class.
  • Use whiteboards to have every student write a down response to question: whole class shows answers simultaneously so teacher can immediately check to see how many students answered correctly.
  • Write questions in plan to specific students who are struggling with a standard; jot down their responses in the plans during class.
  • Note in your book or lesson plan what questions students answer incorrectly; call on them again when you revisit that sort of question later in the week.
  • Choose “No opt out”: do not let students off the hook when struggling with an answer.

3b. Think ratio (techniques to reduce teacher talk and push student thinking):

  • Require students to support answers with evidence from the text.
  • Feign ignorance (for example, write wrong answer that student gives on the board, let students find the error rather than correcting it yourself; pretend you don’t even know that the answer is wrong).
  • Ask students: “put it in your own words” about a classroom definition, concept, and so on.
  • Reword question to force students to think on their feet about the same skill.
  • Use Wait Time to give more students the chance to think through the answer.
  • Model “Right is right”: press to get the 100 percent correct answer.
  • Check for student use of specific strategies and not just correct answers.
  • Ask “what if” question: “What if” I took away this information from the problem, how would you approach it?

4. Differentiated instruction (teaching students at different levels):

  • Create leveled questions for assessments.
  • Include a bonus section of challenging questions.
  • Prepare different Do Nows, worksheets, and so on for students at different levels.
  • Use data (tracking sheets, interim assessment results, exit tickets) to determine the degree of scaffolding and extra support each student needs.
  • Group students according to the skills they need to develop.
  • Communicate and collaborate with skills room and special education teachers to develop appropriate scaffolding for special needs students.
  • Implement station work.
  • Create individual “work contracts” so students have a clear path of what they are working on.
  • Use Do Now, exit tickets, and interim assessment data to drive small group re-teach sessions.
  • Create assignments with menu options by level (easy, medium, hard)—students can choose or teacher can assign.
  • Have observers sit by lower-achieving students during an observation to provide extra support.

Appendix: Increasing Rigor Throughout the Lesson: Data-Driven Classroom Best Practices (Paul Bambrick-Santoyo, continued)