STATE BOARD OF EDUCATION – TOPIC SUMMARY

Topic: Math and Science Updated Scoring Guides

Date: May 19, 2011

Staff/Office: Tony Alpert, Leslie Phillips, Jim Leigh, Winnie Miller

Action Requested: Informational Only Adoption Later X Adoption Adoption/Consent Agenda

ISSUE BEFORE THE BOARD: The math and science scoring guides have been aligned with the content standards adopted within the last few years, and have followed similar processes in the design, statewide review, revision and piloting of these tools. These scoring guides are ready for adoption by the State Board of Education.

·  The new scoring guide for mathematics would replace the current scoring guide as the official scoring guide for scoring work samples for the Essential Skills requirement in “Apply mathematics in a variety of settings.”

·  New scoring guides in Science are proposed as the official state scoring guides, supporting local educators’ efforts to fully measure Science Processes content standards not completely assessed as testable content for the OAKS science assessment.

·  For both mathematics and science, the new scoring guides would become the default scoring guides for scoring work samples for the Local Performance Option.

Information requested by the State Board on March 10, 2011

·  Can you please submit an outline of an Analytic Report such as interrater reliabilty?

A full pilot study of student work provides evidence for meeting essential skills requirements. In Appendix 1, the following steps are outlined and completed for the Mathematics Problem Solving Scoring Guide:

o  Purpose

o  Scoring Guide Development

o  Task Development

o  Selection of Pilot Sites

o  Scoring of Sample Student Work

o  Inter-rater agreement

o  Analysis of Scored Student Work

o  Relationship with OAKS Scores for 7th Grade

·  What policy is affected by the adoption of new scoring guides for math and science? Are there implications that the Board needs to be aware of in making a decision?

o  In the case of mathematics, there are policy implications due to the tie-in to essential skills (as an optional way to demonstrate the Essential Skills). Oregon Council of Teachers of Mathematics and ODE advise that the new scoring guide better reflects the 2007/2009 Oregon Mathematics Content Standards and the Common Core State Standards.

o  The State Board adopted new science content standards in February, 2009 which necessitates revision of the scoring guides. Stakeholders advised that both scientific inquiry and engineering design processes should be represented in separate scoring guides.

·  What career technical education is being provided?

The Offices of EII and OAIS are working collaboratively with a group of CTE and math teachers to develop examples of math problem solving tasks that can be used in CTE classes and scored using the Official Math Scoring Guide. Currently, students are piloting the sample tasks and exemplar student work will be posted on the ODE website along with other math problem solving examples. The first student exemplars are located in the high school section at http://www.ode.state.or.us/search/page/?id=281.

·  School District Accountability for Work Sample quality and opportunity:

The Office of Assessment and Information Services will work with the Office of Educational Improvement and Innovation (EII) to provide evidence as part of Division 22 monitoring as defined in the OAR Division 22 handbook, pages 32 and 33.

BACKGROUND: In June 2008, the Oregon State Board of Education adopted the Assessment of Essential Skills Oregon Administrative Rule (OAR 581-22-0615). This rule requires that school districts and public charter schools provide opportunity for:

·  students in the class of 2013-14 to demonstrate proficiency in the Essential Skill of mathematics for the diploma. One option for this requirement is demonstrating proficiency through two work samples scored using the Official Mathematics Problem Solving Scoring Guide.

·  students in Grades 3 through 8 and once in high school to demonstrate their progress in skill areas including Mathematical Problem-Solving, and Scientific Inquiry. The local performance assessment shall consist of one work sample per grade scored using the official state scoring guides or comparable measures adopted by the district.

Mathematics Background:

The State Board of Education adopted the current Mathematics Problem Solving Official Scoring Guide for use beginning in 2000-2001. Since that time the current scoring guide has been used by teachers scoring mathematics problem solving work samples. In 2007, the State Board adopted the new Mathematics Content Standards for grades K-8 and in 2009, adopted the new Content Standards for High School. The new content standards necessitate revision to the scoring guide because they state, “It is essential that these standards be addressed in contexts that promote problem solving, reasoning, communication, making connections, and designing and analyzing representations.” The proposed 2011-2012 scoring guide captures the intent of Oregon’s new content standards as well as Mathematical Practices of the Common Core State Standards, which will be assessed beginning in 2013-2014.

Upon adoption of the new mathematics content standards in December 2007 and June 2009, the Mathematics Content and Assessment Panel and the Oregon Council of Teachers of Mathematics as well as other groups within the mathematics education community reviewed and provided feedback on the Draft Mathematics Problem Solving Official Scoring Guide for grades 3-8 and high school. The draft scoring guide was developed through collaboration between the Office of Assessment and Information Services, the Office of Educational Improvement and Innovation, and the Oregon Council of Teachers of Mathematics (OCTM) Professional Development Cadre. Every effort has been made to capture the intent of:

·  Mathematical Reasoning, as presented by the National Council of Teachers of Mathematics document, Principles and Standards for School Mathematics, 2000.

·  Problem solving as described in the overarching statement for every grade of the new Oregon Core and Content Standards, “It is essential that these standards be addressed in contexts that promote problem solving, reasoning, communication, making connections, and designing and analyzing representations.”

·  Mathematical Practices, as described in the Common Core State Standards.

Partnering with ODE, the OCTM Professional Development Cadre has conducted numerous teacher training sessions since fall 2010 in the use of the proposed scoring guide and has begun collecting student work and developing anchor papers with commentaries. ODE and OCTM expect to have twenty such anchor papers by spring 2011 to post on the web site.

The current Official Mathematics Problem Solving Scoring Guide will be used through 2010-11, while the Proposed Mathematics Problem Solving Official Scoring Guide is being piloted and finalized.

Mathematics Scoring Guide Development Process:

The revised scoring guide was drafted by a small committee of experienced work sample scoring guide trainers partnering with ODE. Oregon teachers statewide were interviewed to determine which aspects of the scoring guide they felt should be changed. The drafts were deliberately written to align to the newly adopted mathematics standards, the topic of mathematical reasoning (as recommended in a 2006 evaluation of Oregon’s assessment system), and the Mathematical Practices section of the Common Core State Standards for Mathematics. The committee also worked with Derek Brown, Mark Freed, and Jim Leigh to revise and send the draft to the OCTM Professional Development cadre, the Oregon Math Specialists (Teachers on Special Assignment), and the Mathematics Content and Assessment Panel to further refine the document. The process included field testing the scoring guide by scoring new student work in targeted classrooms by teachers trained in the use of this newly revised scoring guide, modifying the guide based on input from these field testing sessions, sending the revised guide out to the mathematics community, including the Oregon Mathematics Education Council and Teachers of Teachers of Mathematics, for review both in person and through an email survey, further revising based on this input. This year, the proposed scoring guide has been presented, along with a training pilot, at the Oregon Math Leaders conference, the southern Oregon fall mathematics conference, numerous training sessions conducted by OCTM PD Cadre in spring 2010 through spring 2011, including full-day sessions at the North West ESD and Multnomah ESD. The draft has been edited in small increments throughout the process as teachers come to consensus to make the scoring more objective.

A Scoring Guide Study was conducted and analysis is underway. In February and March, 2011, fourteen teachers had their students work through a work sample task, approved by ODE as being at an appropriate level of difficulty and aligned the 2007/2009 standards for their grade. The students will take the OAKS Online mathematics test within a month of developing their work sample. The student work samples were scored by a team of expert raters in late March. A comparison will be made between the students’ OAKS scores and their work sample scores to determine whether the work samples scored using the Revised Scoring Guide and OAKS are of comparable rigor. This study will provide student work that may be used as anchor papers for future teacher training. Preliminary results are in Appendix 1.

Science Background:

The State Board of Education adopted the 2001 science content standards and phase-in schedule for scientific inquiry work samples to begin in 2003 with full implementation by 2005-2006. The State Board adopted new science content standards in February, 2009 which necessitates revision of the scoring guides. Stakeholders advised that both scientific inquiry and engineering design processes should be represented in separate scoring guides.

Upon adoption of the new science standards in February 2009, the Science Content and Assessment Panel, and Science Focus groups reviewed and provided feedback on draft scientific inquiry scoring guides for grades 3-8 and high school. In addition, engineering design scoring guides were developed at their recommendation. It was determined that engineering design skills and processes required specific language and score level descriptions to fully align to the 2009 standards. The draft scoring guides were reviewed by stakeholders from throughout Oregon. Moving Science Forward professional development conducted by the Office of Educational Improvement and Innovation staff provided multiple teacher groups opportunities to respond to the newly developed scoring guides.

The current Official Scientific Inquiry Scoring Guides will be used through 2010-11, while the proposed official 2011-12 Scientific Inquiry and Engineering Design scoring guides are being evaluated within districts and by higher education pre-service classes.

Science Scoring Guide Development Process:

The science scoring guides were drafted by teams of Oregon content experts and educators including current classroom teachers, science content panel leaders, University and College level educators, and working professionals. The drafts were deliberately written to align to the newly adopted science standards for each grade group as interpreted by the teams of focus panel members. The process included building the scientific inquiry guide for high school, the building of the engineering guide for high school, then, for middle school, then elementary. Each level of development included vertical articulation as well as structural review to ensure parallel development between scientific inquiry and engineering design. Once first drafts were created, on April 23-24 2010, Portland State University partnered with ODE to convene experience in-service teachers to provide initial feedback regarding the developmentally appropriate proficiency levels of the new scoring guides and preliminary inter-rater reliability. Once this feedback was incorporated into subsequent drafts, other review sessions were completed by the 2009-11 Science Content and Assessment Panel, Oregon Science Teacher Association (OSTA), Oregon Science Specialists, and multiple statewide reviews completed at the Moving Science Forward workshops held in Oregon May-August 2010. In addition, two online surveys were posted on the ODE website in May and September, to collect targeted feedback.

It will be determined if a grade 3 engineering design guide is appropriate and the Office of Assessment and Information Services will continue collecting, evaluating and scoring student work samples to post as exemplars of each score level.

Summary:

Results from the teacher training sessions indicate that the proposed math problem solving, scientific inquiry and engineering design scoring guides exhibit a high degree of inter-rater reliability (within one point) on the scoring of student work. The ODE web site has been updated to offer sample work sample tasks for all grades aligned to the new standards. This web site will be further updated with additional tasks, teacher commentaries, and training materials for school and districts to conduct their own training. Supporting documents are also in development: student language version, scoring forms for feedback to students, and work sample task planning templates.

Once adopted by the State Board of Education these new scoring guides will become official, and will then be translated into Spanish. Student version scoring guides will be finalized. The Office of Assessment and Information Services will continue collecting and evaluating student work for anchor papers, in partnership with the professional organizations. Scoring guide trainings hosted by ODE and ESDs will be expanded to provide information regarding the use of the new materials and for partnerships to develop high quality work sample tasks.

STAFF RECOMMENDATION:

·  No change to the local performance requirements in the Assessment of Essential Skills OAR.

·  Adopt the scoring guides at the May meeting; action must be taken no later than May, as teachers and school administrators need official state scoring guides to plan and prepare for the 2011-2012 school year.

Attachments:

1.  2010-2011 Mathematics Problem Solving Official Scoring Guide

2.  Proposed Official Mathematics Scoring Guide for 2011-2012

3.  2002-2011 Scientific Inquiry Official Scoring Guides

4.  Proposed Official Scientific Inquiry Scoring Guides and Official Engineering Design Scoring Guides (Side-by-Side)

5.  Appendix 1: Overview of Proposed Mathematics Problem Solving Official Scoring Guide and Scoring Guide Study

Edits to proposed official scoring guides following the March 2011 State Board Meeting highlighted in yellow

1

2010 - 2011 Mathematics Problem Solving Official Scoring Guide* 2010 – 2011

Conceptual Understanding
Interpreting the concepts of the task and translating them into mathematics
WHAT?
/

Processes & Strategies

Choosing strategies that can work, and then carrying out the strategies chosen

HOW?

/ Verification
In addition to solving the task, identifiable evidence of a second look at the concepts/ strategies/ calculations to defend a solution

DEFEND!

/ CommunicationUsing pictures, symbols, and/or vocabulary to convey the path toward the identified solution

THE CONNECTING PATH!

6 / The translation of the task is enhanced through connections and/or extensions to other mathematical ideas / Elegant, complex and/or enhanced mathematical processes / strategies used to solve the task are completed / The review is related to the task, and enhanced, possibly by using a different perspective as the defense / The connecting path is enhanced (e.g., graphics, examples) allowing the reader to move easily and make connections from one thought to another
5 / The translation of the task into mathematical concepts is thoroughly developed / Pictures, models, diagrams, and/or symbols used to solve the task are thoroughly developed / The review is a thoroughly developed look at the concepts/ strategies/ calculations in relation to the task / The path connecting concepts, strategies, and/or verification toward the identified solution is thoroughly developed
4 / The translation of the task into adequate mathematical concepts using relevant information is completed / Pictures, models, diagrams, and/or symbols used to solve the task are complete / The review is completed (concepts/ strategies/calculations), and supports a solution / The path connecting concepts, strategies and/or verification toward the identified solution is complete
3 / The translation of the major concepts of the task is partially completed and/or partially displayed / Pictures, models, diagrams, and/or symbols used to solve the task may be only partially useful and/or partially recorded / The review is partially completed, partially recorded, and/or partially effective / The path connecting concepts, strategies and/or verification toward the solution is partially complete, and/or partially displayed with significant gaps that have to be inferred
2 / The translation of the task is underdeveloped or sketchy / Pictures, models, diagrams, and/or symbols used to solve the task are underdeveloped or sketchy / The review is underdeveloped or sketchy (e.g., focusing only on its reasonableness) / The path connecting concepts, strategies and/or verification toward a solution is underdeveloped or sketchy
1 / The translation of the task uses inappropriate concepts or is minimal or not evident / Pictures, models, diagrams, and/or symbols used to solve the task are ineffective, minimal, not evident, or may conflict with their solution / The review is ineffective, minimal, inappropriate and/or not evident / The path connecting concepts, strategies and/or verification toward a solution is ineffective, minimal or not evident

Accuracy: