Year Three Evaluation Report: SCUSD Small Learning Communities Grant
Prepared by:
Theresa Westover and Joanne Bookmyer
September 10, 2010
Executive Summary: Year 3 SCUSD SLC Evaluation Report
In July 2008, a U.S. Department of Education Cohort 8 Smaller Learning Communities (SLC) grant was awarded to Sacramento City Unified School District (SCUSD). The initial SLC grant was awarded for a three-year period but has been granted a one, possibly two-year extension, meaning that the funding cycle will end in 2012 or 2013.
The UC Davis School of Education’s Center for Community School Programs (CCSP), in the role of external evaluator, is charged with capturing formative feedback that might contribute to a better understanding of the program’s impact on the participating school sites and the students they serve, as well as contribute to program improvements. In years 4 and 5, the Center for Education and Evaluation Services (CEES), another division within the School of Education’s CRESS Center will serve as external evaluator. CEES has been responsible for data analysis the past three years and performed the statistical analyses presented in this report. The district itself is accountable to the funder for providing summative outcomes data.
This, the Third Annual Report summarizes external evaluation activities for the 2010/11. The evaluatorsadministered and analyzed a teacher survey that explored the extent to which the project is meeting its goal of:
- Providing professional development for teachers in a way that increases their content knowledge while developing their capacity to implement high quality instructional strategies.
A significant portion of the grant funding is allocated for professional development activities provided by five of the Region 8 California Subject Matter Projects (CSPM): Area 3 Writing Project, Sacramento Area Science Project, UC Davis Math Project, California Reading and Literature Project, and The History Project. During Year Three each of the Projects continued to provide professional development to SCUSD teachers in each of the core subject areas, defined as Mathematics, Science, English, and Social Sciences (The History Project and the California Reading and Literature Project worked collaboratively with Social Sciences teachers). While encouraged to participate (a stipend and/or credit was provided) teachers’ participation was optional and varied across school sites and across subject areas. During the summer (2011) each of the CSMPs offered a Summer Institute. Again, participation in the Institutes was encouraged but optional and levels of participation varied across subject areas and school sites.
Other grant supported activities and strategies are linked to Academic Support (e.g., common planning time, training and implementation of academic conferencing, teacher quarterly benchmark reviews, and data utilization to improve ELA, math and science outcomes) and to Advisement/Counseling (e.g., a College Career Initiative, counselor training, development and initiation of a school-wide transcript evaluation service). The external evaluation did not evaluate any of the other grant components mentioned above.
All results included in this report should be considered as descriptive and suggestive rather than definitive. Due to the uneven distribution of survey respondents’ subjects taught across schools, it is difficult to attribute any patterns in survey responses to either school location or subject taught. Similarly, the wide range in the number of respondents per school precludes drawing any firm conclusions about how survey responses may represent true variation among schools since the representativeness of the sample from each school is unknown.
Key Findings:
A total of 328 teachers were invited to participate in the survey and 153 teachers responded for a response rate of 47%.
The majority of teachers (82%) appear to have enjoyed their PD experiences and about two-thirds (66%) felt that the PD had contributed to improved student achievement at their school. Differences among schools and subject areas are not statistically significant and are difficult to interpret from a practical perspective, given the uneven distribution of respondents across schools and subject areas.
85% of responding teachers “strongly agree/agree” that the PD has increased their ability to provide students with regular opportunities to think and reason about key concepts and content in the subject area(s) taught.
85% “strongly agree/agree” that the PD has increased their ability to engage students in learning activities that allow them to deepen their understanding of key concepts and subject matter content.
80% “strongly agree/agree” that the PD has increased their ability to support students with a variety of strategies and tools that scaffold student learning.
71% “strongly agree/agree” that the PD has increased their ability to encourage students to learn from each other.
76% “strongly agree/agree that the PD has increased their ability to provide students with multiple opportunities to demonstrate their understanding of concepts, as well as to practice their skills/competencies.
Respondents were asked to respond to three open-ended questions regarding their (a) suggestions for improving the professional development, (b) the barriers they felt were most significant to student learning at their school, and (c) what they felt would be the most effective strategy to overcome those barriers.
Teachers’ suggestions about professional development were coded into nine categories: (1) subject matter content (e.g. “Keep focus on reading and writing”), (2) pedagogy/instructional strategy (e.g. “More help with differentiated instruction”), (3) changes to the PD approach or focus (e.g. “More site-based PD”), (4) assessment (e.g. “school-wide writing assessments to monitor growth”), (5) technology (e.g. “Demonstrate effective use of technology with follow-up”), (6) student engagement/motivation (e.g.” Ways to motivate students”), (7) special education (e.g. “More PD related to teaching student classified as emotionally disturbed”), (8) collaboration (e.g. “More time for inter-disciplinary work with other teachers”), and (9) general negative comments (e.g. “Not that you would listen to”).
Teachers’ responses to the “what is the biggest barrier” question were similarly categorized into nine areas: (1) student behavior, attendance, and accountability, (2) student skills, knowledge, and language, (3) students’ families and community, (4) student engagement and motivation, (5) school or district administration, (6) school or district policies such as class size and/or emphasis on standardized test scores, (7) instructional practices and/or lack of teacher skill or motivation, (8) lack of resources or technology, and (9) collaboration.
It is notable that student engagement and motivation appear to be considered the leading barrier to student learning (25% of all barriers identified). It is also interesting that nearly half of the barriers (47%) center on student characteristics largely outside the control of the teachers (e.g. behavior, attendance, skills, language, families, and/or poverty).
Teachers’ open-ended responses to the “most effective strategy for improving student learning” question were summarized into 10 categories: (1) better/more parental accountability or involvement, (2) improve student discipline/accountability, (3) improve student engagement/relevance, (4) student supports such as tutoring or individualized attention, (5) improve teacher accountability, behavior, or school climate, (6) improve instructional practice, (7) improve relationship/professional learning community, (8) common assessments/pacing guides/focus, (9) change course offerings/scheduling/class size, and (10) improve school or district administration.
Teacher respondents do appear to believe that instructional practices can overcome the barriers to learning – a third of all strategies mentioned focused on improved pedagogy. An additional 14% of the strategies mentioned involved improving the relevance of instructional materials and/ or strategies for improving student motivation. Another 24% of the strategies mentioned also addressed changes under school and teacher control – improving professional learning communities and/or small learning communities and relationships and modifying class structures, schedules, or size
In summary, it appears that the teachers who responded to the survey, the vast majority of whom had attended the subject matter professional development provided by CRESS, found value in their participation. They reported that the professional development did provide them with tools to engage and support their students. While many reported significant barriers to learning in their schools, most also felt that these barriers could be overcome by focused efforts of teachers and administrators.
SCUSD Cohort 8 Smaller Learning Community Grant: Spring 2011 Teacher Survey
Introduction and distribution of respondents
In the spring of 2011 teachers in five Sacramento City Unified School District (SCUSD) High Schools were invited to complete a web-based survey regarding their participation in professional development over the previous year. A total of 328 teachers were invited to participate in the survey and 153 teachers responded for a response rate of 47%. The majority of responding teachers were from Luther Burbank (30% of all respondents - 50% of the Burbank teachers invited) and J.F. Kennedy (29% of all respondents - 69% of the JFK teachers invited) High Schools (Figure 1). The response rates for both Hiram Johnson and Rosemont were 40%, and McClatchy’s response rate was the lowest at 26%. The uneven distribution of response rates across schools should be kept in mind when examining the following analyses. Most (90%) respondents had participated in professional development (PD) activities during the previous year. Only the 135 who participated in PD are included in most of the analyses in this report.
Figure 1: Number of Respondents by School
Most respondents (84%) teach multiple grades (9-12). A third of the respondents indicated they teach all four grade levels at their school. This varied among the schools with more teachers at McClatchy reporting teaching all four grade levels (58%) and none indicating they only teach in one grade level and more at Kennedy (23%) indicating they teach only one grade (Figure 2). It also varied among the subject areas with more English teachers (35%) reporting teaching only one grade level than was true overall (17%) – although the majority of English teachers (52%) report teaching in two grade levels. More math teachers (74%) reported teaching all four grade levels than was true overall (32%). Most science teachers also report teaching in all four grade levels (42%), while the majority (50%) of history/social science teachers teach in two grade levels (Figure 3).
Figure 2: Number of grade levels taught by teacher by school
Figure 3: Number of grades taught by teacher by subject
Overall, 32% of the respondents teach English, 25% Mathematics, 20% Science, 23% History/Social Studies and 23% “Other.” This was a “check all that apply” question so teachers might teach in more than one subject area. Since most of those in the “other” category also taught in one of the four primary subject areas, only a few teachers (5) remained strictly in the “other” category – these teachers are excluded from crosstabulations by subject area. Of these five, three are special education teachers. One teacher indicated teaching both English and History/Social studies – that teacher’s responses are included in both subject areas in crosstabulations because it is unknown which subject matter professional development he or she attended. Subjects taught by school location are displayed in Figure 4. In terms of response rates, 42% of the English teachers invited responded, 40% each of the math and science teachers responded, and 38% of the invited history/social studies teachers responded.
Due to the uneven distribution of subjects taught across schools, it is difficult to attribute any patterns in survey responses to either school location or subject taught. Similarly, the wide range in the number of respondents per school and the varying response rates per school precludes drawing any firm conclusions about how survey responses may represent true variation among schools since the representativeness of the sample from each school is unknown.
Figure 4: Subject taught by school
Professional development source, overall satisfaction, and perceived impact on student achievement
Among the 135 teachers who reported attending a content-specific professional development (PD) during the 2010-11 school year (including summer 2010), 61% reported attending one of the California Subject Matter Projects’(CSMPs) training associated with the SCUSD Cohort SLC grant, 31% reported participating in both aCSMP training and some other PD offering ,and 7% reported only attending “other” PD. The eight respondents who did not participate in aCSMP training were fairly evenly distributed among the subject areas (science has the least at 4% - 1 teacher -and history/social science the most with 10%- 3 teachers). Similarly, there is minor variation among the schools ranging from 12% (N=5) of the Burbank teachers attending “other” PD only to none of the teachers at Rosemont attending any non-CSMP PD.
When asked “Has the professional development you participated in during the 2010-2011 academic year helped you/your school meet its goal of improving students’ academic outcomes?” Most (66%) answered “yes,” 21% answered “don’t know/not sure,” and 4% (N=5) answered “no.” All of those who responded “no” and who also answered the previous question (N=4) had participated in a CSMP training; about 10% of those who were unsure of the impact of their PD on student achievement had participated in a non-CSMP training (Figure 5).
Figure 5: “Has the PD that you participated in this year helped improve student achievement?”by number of respondents participating in each type of professional development.
For purposes of statistical analyses, the categories of “no” and “don’t know/unsure” were combined into a dichotomous measure when differences among schools and subjects were examined. While there was a range of responses among schools and subject areas, the differences were not statistically significant. Among the schools, Burbank teachers had the highest proportion of “yes” responses (88%) and McClatchy has the lowest (50%). Among the subject areas, science teachers had the highest proportion of “yes” responses (80%), followed closely by English (79.5%), with the lowest proportion (62%) of “yes” responses from history/social science teachers (Figures 6 & 7).
The next survey question asked if teachers “were generally satisfied with the overall quality of the professional development teachers had participated in during 2010-11.” The majority (82%) answered “yes” with 7% answering “no” and 11% making some qualifying comments that we categorized as “somewhat.” Again, as Figures 6 & 7 illustrate, there was some variation among the schools, ranging from 72% of Kennedy teachers responding “yes” to 100% of those at Rosemont, and among subject areas (ranging from 76% of history/social science teachers responding with an unqualified “yes” to 91% of both English and math teachers). Due to the small number of “no” or “other” responses, no statistical analyses of these differences could be conducted. In terms of differences among the types of PD teachers received, as Figure 8 illustrates, there were some differences but, again, they were not large enough to accommodate statistical analysis.
Figure 6: Variation among schools regarding impact of PD on student achievement and teachers’ satisfaction with PD
Figure 7: Variation among teachers’ subject areas in PD satisfaction and perceived impact of PD on student achievement
In summary, the majority of teachers appear to have enjoyed their PD experiences and about two-thirds felt that the PD had contributed to improved student achievement at their school. Responses to these two items were significantly correlated (p=.0001) but the strength of relationship is moderate (r=.34). Differences among schools and subject areas are not statistically significant and are difficult to interpret from a practical perspective, given the uneven distribution of respondents across schools and subject areas.
Figure 8: “Were you generally satisfied with the quality of the PD you participated in?" by the number of respondents participating in each type of PD
Improved abilities attributed to PD participation
Teachers were asked five questions under the general prompt of: “The professional development I participated in during this academic year has increased my ability to:”
a. Provide my students with regular opportunities to think and reason about key concepts and content in the subject area(s) I teach, 85% “strongly agree/agree.”
b. Engage my students in learning activities that allow them to deepen their understanding of key concepts and subject matter content, 85% “strongly agree/agree.”
c. Support my students with a variety of strategies and tools that scaffold student learning (such as revisiting guiding concepts, use of different problem-solving strategies, use of models), 80% “strongly agree/agree.”
d. Encourage my students to learn from each other (through regular participation in debates, discussion, and problem-solving experiences in pairs, small groups, and as a whole class), 71% “strongly agree/agree.”
e. Provide my students with multiple opportunities to demonstrate their understanding of concepts, as well as to practice their skills/competencies,” 76% “strongly agree/agree.
Clearly, most teachers agreed with each of these statements. The item with the largest proportion (40%) of “strongly agree” responses was “Supporting my students with a variety of strategies and tools….” (item c above). The item with the highest proportion (29%) of “disagree/strongly disagree” responses was “Encourage my students to learn from each other…. “(item d above).
There are differences among the schools in the strength of agreement on each item with about two-thirds of the 36 teachers from J.F. Kennedy tending to show the lowest levels of agreement and the 12 teachers from C.K. McClatchy tending to show the highest levels of agreement with each statement (Figure 9). While interesting, it would be inappropriate to read too much meaning into these differences due to the differences among the schools in the numbers of respondents and the subject areas they teach.