Section 2: Evaluation
The evaluation of the teacher quality component of EMPOWER will examine both the quality and impacts of the interventions.
Plan A
The evaluation team will observe (1) a sample of the summer and academic-yearsessionsto determine if the EMP materials are being implemented with fidelity; and (2) a sample of the content-focused sessions, with a particular focus on the extent to which the mathematicians are able to relate to teachers’ content needs. All teachers participating in these professional development activities will be asked to complete a survey about the quality of these activities at the completion ofeach summer session, and a sample of teachers will be interviewed mid-way through the school year to gather their perspectives on the professional development and to gauge impact.
To monitor the quality of the coaching component, teacher leaders and the teachers with whom they are working will be asked to complete a reflective log each week, describing the focus of the coaching interactions, the aspects that were most valuable, and any impact on practice that resulted. Evaluators will analyze these logs to assess the quality of the coaching support being provided to teachers.
To supplement the data collected by the evaluation team, individual leadership team members will attend the MSP professional development sessions on a rotating basis to monitor their quality. The entire leadership team will conduct quarterly sessions with the professional development providers, using both evaluation data and staff documentation to reflect on the quality of the professional development.
To assess the overall impact of the professional development experiences,a survey of classroom practices will be administered to all targeted teachers in the ten districts each year. These data will be analyzed for differences by treatment (i.e., content professional development first, pedagogy professional development first, coaching, professional development delayed until year three, did not express interest in professional development). In addition, studentmathematics scores in each district will be tracked over the life of the project, beginning in the year prior to funding, to assess the extent to which EMPOWERhas had an impact on student achievement.
Plan B:
Project evaluators will observe at least one session of each of the summer professional development programs each year, and interview a sample of teachers mid-way through each school year, including some teachers working with EMPOWER coaches.
A major focus of the evaluation will be on assessing the impact of EMPOWER on both teachers and students. An instrument designed to measure teacher content knowledge and pedagogical content knowledge will be administered to four groups of teachers at the beginning of the first summer’s professional development, toward the end of the subsequent academic year, and at the end of the second academic year: (1) teachers who begin with the content-based professional development; (2) teachers who begin with the pedagogy-based professional development; (3) teachers who receive coaching support; and (4) comparison group teachers (thosein the “interested” group that are assignedto participate in the professional development after this study is completed.) These data will be analyzed for differences in gains in content and pedagogical content knowledge each year. Results will be reported separately for each of several mathematics content area scales, allowing the project leadership team to identify areas where the professional development may need to be strengthened prior to deployment with the second cohort.
All middle and high school mathematics teachers in the partner districts will be surveyed annually regarding their instructional practices. These data will be analyzed (along with data on teacher participation in the professional development) for impacts on teaching practice, allowing a determination of the effects of each of the two professional development “treatments”individually and in combination, as well as the effects of the coaching component.
Finally, data from state and district mathematics assessments will be compiled for the students of the four teacher groups designated above. These data will be analyzed for the extent of impact of the program on student achievement, controlling for prior knowledge of mathematics.
Horizon Research, Inc.1October 2004