Teaching and Learningphysical Science in Urban Secondary Schools

Teaching and LearningPhysical Science in Urban Secondary Schools:

Assessing the Assessments

Session Organizer and Chair: Joan Whipp, Associate Professor of Education, Marquette University

Discussant: David Hammer, Associate Professor of Physics and Education, University of Maryland

Panel of Presenters:

Michele Korb: Clinical Assistant Professor of Physics and Education, Marquette University

Mike Politano: Assistant Professor of Physics, Marquette University

Mel Sabella: Associate Professor of Physics, Chicago State University

Kristi Gettelman, Physical Science Teacher, Wisconsin Conservatory of Lifelong Learning, Milwaukee WI

Kelly Kushner, Physical Science Teacher, Shorewood Intermediate School, Shorewood WI

Megan Ferger, Physical Science Teacher, Roosevelt Middle School, Milwaukee WI

Overview

In the current testing climate brought on by No Child Left Behind, much has been written about the negative effectives of high-stakes testing on teaching and learning (Linn, Baker & Betebenner, 2002) and the great need and importance of using ongoing and multiple assessment measures to evaluate instructional programs, student learning and teacher development (Shepard, 2000). This interactive symposium explores the complexity of doing so in a teacher development project aimed at improving the teaching practices of middle and high school physical science teachers and the learning of their students in science. Drawing from multiple assessment measures, we describe the conflicting findings from our study on what happened to both teachers and their students in this project and the limitations and strengths of the various assessments instruments we used for evaluation. At the same time, we hope to demonstrate the rich and complex picture of “what happened” that the ongoing and multiple assessments provided.

Connection to Literature and Theoretical Frameworks

There have been numerous calls for reform in science teaching during the last decade (National Commission on Science and Mathematics Teaching in the 21st Century, 2001; National Research Council, 1996, 1999; National Center for Educational Statistics, 1999). These calls for reform emphasize the need for what is called inquiry-based, learner-centered or constructivist teaching practices which have been outlined by many (Driscoll, 2005; Herrington & Oliver, 2000; Jonassen & Land, 2000; Land & Hannafin, 2000; McCombs & Whisler, 1997; Windschitl, 2002). Such practices include use of: 1) complex, real-world situations where students actively construct meaning, linking new information with what they already know; 2) problem-based activities; 3) varied ways for students to represent knowledge; 4) opportunities for knowledge-building from a variety of perspectives; 5) collaborative learning activities; 6) activities that develop students’ metacognitive and self-regulation processes; 7) emphasis on critical thinking rather than just getting “right answers”; 8) support for student reflection on their learning processes; and 9) ongoing assessments that provide continual feedback on the development of student thinking.

Physics education researchers have been leaders in answering the call for science teaching reform through a combination of research on student learning and use of that research to construct instructional materials that promote conceptual change and learner-centered teaching. Some examples of physics research on student learning include the identification and address of student difficulties (Shaffer & McDermott, 2005), the development of a theoretical framework to understand student knowledge (Redish, 2004), and the understanding of student epistemologies (Hammer, 1994). Such research has been used to guide the development of a number of nationally known reform-based curriculum materials, including Physics by Inquiry (McDermott and the Physics Education Group, 1996), Modeling Instruction in High School Physics (Hestenes, 1996), and Constructing Physics Understanding in a Computer-Supported Learning Environment (Goldberg, 1997). Development of these materials is based on a cycle of research into student learning, development of curricula, implementation and then assessment of student learning. Up until now, however, the focus of this research and development has been largely in high school and college physics classes. Very little of this work has been attempted with middle or early high school teachers who teach physical science and who often lack strong content background in physics (Ingersoll, 1999). Furthermore, even less of this work has been attempted with middle or early high school teachers who work in urban settings.

Context and Project Description

For two years (2004-2006), eighteen middle and high school physical science teachers from ten urban Milwaukee middle and high schools participated in a U.S. Department of Education Title II (Improving Teacher Quality) project, Modeling in Physical Science for Urban Teachers Grades 6-10 that included two-week summer modeling institutes, four half-day follow-up modeling workshops, an online community, and classroom mentoring with a Marquette science educator. The curriculum for the summer institutes and follow-up activities was adapted from the Physical ScienceModeling Workshop, a curriculum for middle school teachers that was developed at Arizona State University and based on the more well-known Modeling Instruction in High School Physics. The project aimed at increasing teachers’ content knowledge of physical science, improving their instructional strategies (discourse management, assessment, content organization, inquiry methods, cooperative learning methods, and use of classroom technologies) and increasing collaboration among physical science teachers in Milwaukee urban schools. During the summer institutes and Saturday workshops, taught by a high school physics teacher trained in modeling at Arizona State, teachers had opportunities to work in teams on specific applications of the modeling curriculum to their particular grade level and school context as well as share materials, methods, and reflections on progress in implementing modeling in their classroom. In addition, throughout the life of the project, networking and support among project participants were encouraged through ongoing dialogue in an electronic community and on-site mentoring by a university science educator who visited each of the participating teacher’s classrooms at least once each semester for clarification, reinforcement, and context-specific adaptations of the professional development activities.

In order to evaluate the effectiveness of this project, we used a variety of methods to assess a number of different outcomes: pre and post test scores of teachers and their students, surveys, interviews, online discussions, classroom observations, and action research studies. As we looked at results from the assessment of content knowledge in both teachers and their students, using tools like the Physical Science Concept Inventory (PSCI), we saw small changes from pre- to post-instruction. Despite this, most of the teachers involved in the program were able to cite major changes in both their teaching and in what was happening to the students in their classes. These changes were cited by the teachers in interviews and in responses to self-assessment surveys given to the teachers. The discrepancy between these two very different types of assessment measures underlines the importance of utilizing multiple means of assessment in the evaluation of any professional development project.

Summary of Individual Presentations on Modes of Inquiry and Findings

#1 Using Pre and Post Test Scores to Assess Teacher and Student (Mike Politano, Assistant Professor of Physics, Marquette University)

Changes in both teacher and student understanding of physical science and related math skills were evaluated with pre and post administrations of The Physical Science Concepts Inventory (PSCI). The PSCI is a 25 item multiple-choice diagnostic that assesses understanding of topics in physical science. It consists of released questions from Trends in International Mathematics and Science Study(TIMSS), the National Assessment of Educational Progress (NAEP), the Classroom Test of Scientific Reasoning (Lawson, 1999), and other research-based instruments.

Different teachers used the modeling method to different extents and some did not cover all topics covered in the PSCI. Consequently, the expectation was that there would be a gain in scores as a result of participating in the program but that gain would vary from class to class. In addition, we might have expected that because this was an early implementation of these new materials, teacher and student gains on the PSCI would be modest.

The gain was measured using the equation developed by Hake (1998). The Hake gain is calculated using . The PSCI was given to the participating t4acher in the program as well as to the students in the middle school classes. To evaluate the effectiveness of the modeling materials for the middle school students, the PSCI was given pre/post two times during the grant period. In the second year of the study, the only teachers who administered the PSCI were those who implemented the modeling method with their students at some significant level.

Probably because the baseline scores of the teachers were quite high (73%), their gains on the test were not significant. The students taking the PSCI also had relatively small gains (4% to about 40% in the final implementation). Looking at the overall group, the scores went from about 30% correct to 36% in the final implementation. Interpreting data from multiple-choice tests is often extremely challenging and care must be exercised in interpreting the results. It is often unclear whether improved performance on the instrument is an indicator of improved conceptual understanding or whether a lack of improvement on the instrument indicates a lack of improvement in conceptual understanding.

The small gains from pre- to post-test, for the majority of classes, were surprising to those of us involved in the project because they conflicted with our classroom observations of the teachers and their students as well as our interviews with the teachers over the course of the project. These conflicting findings suggested the limitations of simply using a pre and post diagnostic test like the PSCI as a measure of what happened. in this project. First, teachers reported that many of their urban students who eventually became enthusiastic about learning through modeling have serious reading problems and tune out when faced with any standardized objective test that requires reading skills. Secondly, teachers in this project struggled with initial implementation of modeling due to student resistance although were successful in implementing modeling during the project’s second year. This slow start suggested that by the end of the project, teachers were actually in the early stages of implementation, which could explain their students’ modest student gains on the diagnostic. Finally, unlike the well-tested Force Concept Inventory (Hake, 1998; Hestenes, Wells & Swackhamer, 1992), the PSCI is a new instrument with less data supporting its reliability.

#2Using Self-Report Instruments to Assess Improvements in Teaching and Learning. (Mel Sabella, Associate Professor of Physics, Chicago State University)

To get further perspective on how the participating teachers were changing their teaching practices and how they were thinking and feeling about those changes, we used a variety of surveys, interviews, and reflection questions throughout the project. For example, several surveys were given to the participating teachers that provided them with the opportunity to self assess their implementation of the new materials.

One of these instruments was an attitude survey that drew from the Maryland Physics Expecations Survey (Redish, Steinberg & Saul, 1998), the Epistemological Beliefs Assessment for Physical Science (White, Elby, Frederiksen & Schwarz, 1999), and the Epistemological Questionnaire (Schommer, 1990).These instruments were used to better understand what teachers valued regarding the program . By doing this as a pre- and post-test, the instrument provided evidence on how teachers and students changed as a result of participating in the physical science modeling project. We found that there was little change toward expert-like views after approximately one year of implementing the modeling method. This is somewhat to be expected since these attitudes are very difficult to change. Other researchers who have studied the evolution of student attitudes during the university introductory physics class have observed that student attitudes tended to diverge from the attitudes and expectations of experts (Redish, Saul & Sternberg,1998).

Despite these modest changes in the performance on the PSCI and Attitudes Surveys, interviews and online reflections indicated that teachers were seeing important changes in their classrooms. These included students being more active in class, students gaining confidence in their abilities, etc. Teachers believed that the use of the modeling method benefited their students by helping to improve their understanding of the underlying concepts and the connection between mathematics and physical science. Teachers also remarked that the modeling approach helped them create an interactive learning environment in which they were in a position to conduct ongoing assessment of their students’ knowledge and conceptual development.

Although, by themselves, self-report measures can often be limited, taken together, they can often elaborate and enrich other data sources. In this project, the surveys, interviews and online reflections also indicated that many of the participating teachers were significantly adapting the modeling curriculum to fit their particular teaching contexts and students. Many of the middle school teachers, for example, reported how they were applying modeling not only in their physical science teaching but also in their teaching of chemistry, biology, and social science. In addition, many teachers reported significant adaptations of the modeling curriculum to fit the needs of their particular students, for example those with low reading skills. The fact that these teachers felt comfortable making these adaptations of their own initiative emerged as a major strength in this program.

Note: The tables in the Appendix of this report provide detail on teacher performance on the various assessment instruments used with the participants in this project and discussed by the first two presenters.

#3Using an Observation Tool to Assess Teaching Reform: The RTOP. (Michele Korb, Assistant Clinical Professor of Education and Physics, Marquette University)

Based on current national science and mathematics standards for learner-centered, inquiry-based teaching, The Reformed Teaching Observation Protocol (RTOP), developed at Arizona State University, is a standardized observation assessment instrument that attempts to detect the degree to which science classroom instruction exhibits “reformed” or “constructivist” teaching methods. The instrument specifies a set of 25 scored, observable classroom behaviors or items related to lesson design and implementation, propositional and procedural content knowledge, communicative interactions and student/teacher relationships. The instrument has been widely studied and used and demonstrates high inter-rater reliability. High scores on the RTOP correlate well with increased student conceptual understanding of science in a number of studies (Lawson, Benford, Bloom, Carlson, Falconer, Hestenes, Judson, Piburn, Sawada, Turley & Wyckoff, 2002). Throughout this project, I visited each of the participating teachers in their classrooms twice a year and completed the RTOP as part of my observation. Data from the RTOP observations indicated that over time there was a considerable increase in a number of behaviors that characterize reform-based instruction in these teachers, including use of instructional strategies and activities that respected students’ prior knowledge and preconceptions; lessons designed to engage students as members of a learning community; demonstration of a solid grasp of the subject matter content; encouragement of abstract thinking, student use of models, drawings, graphs, concrete materials, and manipulatives to represent phenomena; teacher questioning that triggered divergent modes of thinking; high proportion of student talk between and among students; climate of respect for what others had to say; encouragement of the active participation of students; teacher patience with students; and teacher acting as a resource person, supporting and enhancing student investigations. The implementation of the R-TOP seemed to be an important component to our modeling project and the professional development of the teachers. It provided teachers with a common language to talk about their teaching and to reflect on their changing teaching practices. However, the RTOP may not be as useful as a program evaluation tool since the teachers were only observed once a semester. It was difficult to use it as a basis for generalizing about the quality and quantity of teacher changes in this project.

#4 Using Action Research to Assess the Improvement of Teaching and Learning (Megan Ferger, Kristi Gettelman, and Kelly Kushner, middle school physical science teachers)

We have been conducting action research on our implementation of modeling into our teaching of physical science concepts and principles of measurement. In our research, we have been addressing these questions: 1) How did our students’ understanding of physical science concepts and measurement principles change? 2) How, if at all, did our students’ attitudes toward science change? 3) In what ways, if any, has student engagement in the scientific process increased – i.e. making hypotheses, collecting and analyzing data, and forming conclusions to understand and explain physical science concepts and principles of measurement? Using both quantitative and qualitative methods of inquiry for our research, we have drawn from multiple data sources: pre and post testing of our students on the physical science and measurement concepts, our own classroom observation logs, student science logs, student attitude surveys, and student interviews.

Our data analysis indicates that: 1) Our students showed the greatest gains on standardized tests when they were taught with a blended approach of traditional and modeling methods; 2) Our students showed the greatest gains in their understanding of principles of matter (based on pre/post testing and our observation data); 3) Our students showed the least gains in their understanding and ability to apply advanced measurement principles (based on pre/post testing and observation data); 4) Student resistance to modeling was a barrier that needed to be addressed initially; our students were too used to being given right answers rather than thinking for themselves. Over time, however, student attitudes toward science became more positive. 5) Since we have implemented the modeling curriculum, student engagement in scientific processes has increased.