Teaching American History (TAH) Year 2 Report

Teaching American History (TAH) Year 2 Report

Prepared by (in alphabetical order):

Michele Hamilton,Shani Keller,

Amy Smith, and Theresa Westover

August 20, 2010

Table Of Contents

List of Tables

List of Figures

Introduction

Teacher Content Knowledge

Multiple Choice Assessment Results

DBQ Assessment Results

Teacher Survey Results

Teachers’ Goals, Gains, and Classroom Practice

Classroom Practices

Lesson Study: 8th Grade American History Teachers

STUDENT SURVEY RESULTS

Students Attitudes and Beliefs about History

Students’ Report of Learning About American History

Student Achievement Analyses

Influence of teacher program participation on student ELA achievement

Student History CST Achievement Pre-TAH vs. Post-TAH

Student Achievement by TAH Participation components and Duration

Nested Model Analysis of Student History CST Scores

Summary And Conclusions

List of Tables

Table 1. TAH Teacher Descriptive Statistics By Cohort…………………………………………………..6

Table 2. Results of the Multiple Choice Content Knowledge Tests……………………………………….9

Table 3. Results of the DBQ Assessments………………………………………………………………..11

Table 4. Lesson Study Activities……………………………………………………………………….....17

Table 5. 2009 California Standards Test in History Social Science, Scale Score Ranges………………..26

Table 6. Comparison of Mean CST Scores and Performance Levels, TAH Participants’ Students vs.

Non-Participants’ Students………………………………………….…………………………………….27

Table 7. Significant Variables from Linear Regression Analysis………………………………………...30

Table 8. Regression Model Predicting 2009 CST History Scale Score…………………………………..30

Table 9. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Far Below Basic……………………………………………………………...33

Table 10. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Below Basic………………………………………………………………….33

Table 11. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Basic………………………………………………………………………....34

Table 12. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Proficient…………………………………………………………………….34

Table 13. Average CST History Scale Score for TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Advanced…………………………………………………………………….35

Table 14. Average CST ELA Scale Scores for TAH Participants’ Students Vs. Non-Participants’

Studentsby Prior Year ELA Level (T-test results)……………..………………………………………………………..37

Table 15. Regression Model Predicting CST ELA Scale Scores………………………………………...38

Table 16. Change in Mean CST History Performance Level From 2007-08 to 2008-09………………..38

Table 17. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Far Below Basic……………………………………………………………………………………………………...41

Table 18. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Below Basic……………………………………………………………………………………………………...41

Table 19. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Basic…………………………………………………………………………………………………...…42

Table 20. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Proficient…………………………………………………………………………………………………43

Table 21. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Advanced………………………………………………………………………………………………….44

List of Figures

Figure 1. Participating Teachers' Years of Experience Teaching American History as of the 2009-2010 School Year…………………………………………………………………………………………………7

Figure 2. Number of Sections of American History Taught in the 2009-10 School Year…………………8

Figure 3. Teachers’ Reported Goals (Pre-Survey) and Gains (Post-Survey)……………………….……13

Figure 4. Teachers’ Reported Classroom Practice at Entry and in Spring 2010………….………………15

Figure 5. 8th Grade Teachers' Report of The Impacts of Lesson Study on Teacher Collaboration………19

Figure 6. 8th Grade Teachers' Report of the Impacts of Lesson Study on Their Knowledge and Use of Instructional Strategies………………………………………………….…………………………………21

Figure 7. 11th And 8th Grade Students’ Attitudes and Beliefs About History……………...……………23

Figure 8. 11th And 8th Grade Student’s Report of Learning About American History……………..……25

Figure 9. 2008 CST ELA Scale Score vs. 2009 CST History Scale Score…………………………….…28

Figure 10. ELA vs. History Scale Score for Participants vs. Non-Participants………………………...…29

Figure 11. Mean CST History Scale Score By Prior Year CST ELA Level, All TAH Participants’ Students vs. Non-Participants’ Students………………………………………………………………..…32

Figure 12. Mean CST History Scale Score By Prior Year CST ELA Level, TAH Participants’ Students Vs. Non-Participants’ Schools at Schools With at Least 40% FRLP-Eligible……………………………36

Figure 13. Specific TAH Participation vs. Mean CST History Scores By Prior Year CST ELA Level………………………………………………………………………………………………………40

Introduction

Implementation of the Solano County Teaching American History (TAH) Grant began in the summer of 2008 and will continue through the 2010-11 school year. The grant serves 8th and 11th grade teachers of American History in six different districts within Solano County, California. This report presents the findings from the evaluation of first and second years of implementation, including findings related to teachers’ goals, the program’s impacts on teachers’ content knowledgeand classroom practices, students’ attitudes about and interest in history, and their reported knowledge gains over the past year in participating teachers’ classes, and the relationship between teachers’ participation in the TAH program and their students’academic achievement.

Over the course of first two years of implementation, teachers had the opportunity to participate in several components of the TAH program. During the first year of the grant, 8th grade teachers were invited to participate in an initial 3-day workshop held in the summer of 2008, referred to as a “Gear-Up” sessionand, beginning in Fall 2008 and continuing throughout the 2008-09 school year, a series of professional development workshops. Not all teachers participated in both summer and school year activities but all are included in the 2008-09 Cohort for analytical purposes. The 2009-10, second program yearactivities began with a ten day Summer Institute in June 2009 which included 8th grade teachers who had previously participated in the program, as well as 8th and 11th grade teachers who joined as new participants (i.e., 2009-10 cohort includes teachers who began in summer or fall of 2009). This was followed by an additional “Gear-Up” session in August, 2009, which was primarily attended by 11th grade teachers, most of whom (73%) had attended the Summer Institute. Finally, throughout the 2009-10 school year, professional development workshops were offered to 8th and 11th grade teachers and 8th grade teachers participated in Lesson Study.Overall, 81 teachers have participated in the TAH program over the two school years. The characteristics of these teachers are provided in Table 1.

Table 1. TAH Teacher Descriptive Statistics By Cohort

2008-09 Cohort / 2009-10 Cohort / Overall
N / 39 / 42 / 81
Grade Taught[1] (%)
Eighth / 89.5* / 21.4* / 53.8
Eleventh / 10.5* / 78.6* / 46.3
Hours ofParticipation
Mean (SD) / 51.4 / 44.7 / 47.4
Minimum / 16.5 / 9.0 / 9.0
Maximum / 121.5 / 109.5 / 121.5
Years Teaching American History (%)
None / 3.2 / 5.0 / 4.2
1 - 2 years / 0.0* / 30.0 / 16.9
3 - 5 years / 41.9 / 22.5 / 31.0
6 - 10 years / 32.3 / 15.0 / 22.5
More than 10 years / 22.6 / 27.5 / 25.4
Total American History Courses Taught (%)
None / 16.7 / 20.0 / 19.2
1-2 / 0.0* / 47.5 / 36.5
3-5 / 50.0 / 25.0 / 30.8
6 – 10 / 33.3 / 7.5 / 13.5

* Indicates the difference between cohorts is statistically significant (p <.05). Statistical significance levels (p) can be thought of as the likelihood that the pattern of responses is due to chance; thus smaller values of p indicate a higher level of confidence that the results are due to real differences rather than random variation.

Of the 81 teachers who have participated in the TAH program,39 teachers joined the program in the 2008-09 cohort and 42 teachers joined in the 2009-10 school year. The teachers’ total hours of participation range from 9 to 121.5 hours of TAH training, with an average of approximately 47 hours of participation per teacher. There was no statistically significant difference in the number of hours of participation between the 2008-09 and 2009-10 cohorts. In general, participating teachers were fairly experienced, with almost half of the teachers, both overall and in each cohort, reporting having taught 6 or more years. However, the 2009-10 cohort had a larger proportion of teachers with only 1 to 2 years of experience (30% vs. 0%). Differences in the teachers’ experience teaching American history are displayed in Figure 1.

Figure 1. Participating Teachers' Years of Experience Teaching American History as of the 2009-2010 School Year

In general, teachers in the 2009-10 cohort reported teaching fewer sections of American history than those who entered the program in the 2008-09 cohort. The majority of the 2008-09 teachers (predominantly 8th grade teachers) reported teaching either 3 to 5 (50%) or 6 to 10 (33%) sections; whereas the majority of the 2009-10 cohort (the majority of which are 11th grade teachers) reported teaching either no sections of American history (20%) or 1 to 2 sections (48%). Although the TAH program is designed for teachers who teach American history, a small number of teachers in each cohort reported that they were not teaching any sections of American history in the 2009-10 school year. It is possible that these teachers’ teach different courses in different years, and although they teach American history on occasion and thus participated in the program, they were not teaching it in the 2009-10 school year. Figure 2 shows the number of sections of American history taught in the 2009-10 school year for each cohort of teachers.

Figure 2. Number of Sections of American History Taught in the 2009-10 School Year

Despite the differences in the teachers’ experience teaching American history and the number of sections taught across the two cohorts, there were no statistically significant differences between 8th and 11th grade teachers in terms of their teaching experience or number of sections taught. Furthermore, there were no statistically significant differencesbetween either 8th and 11th grade teachers or between the two cohorts of teachers in the total number of hours of program participation.

Teacher Content Knowledge

Teachers’ content knowledge was evaluated in two ways: multiple choice assessments and Document Based Question (DBQ) assessments, both of which were administered at the beginning and end of each session (i.e., “Gear-Up” sessions, 2008-09 and 2009-10 school years, and 2009 and 2010 summer institutes). Within each session the same assessments were used for pre- and post-testing, but different pre-post multiple choice assessments were administered at each session (e.g. the multiple choice test for a summer institute pre-post was different, covering different material, than the test administered for the fall-spring school year).Each assessment was designed to assess teachers’ content knowledge related to the specific history topics covered in the individual sessions and to the California state standards. In the “Gear-Up” and Summer Institutes, teachers’ were assessed on the first and last days of the workshop. Participants in the 2008-09 and 2009-10 school years where assessed in the fall and again in the spring. Eighth and 11th grade teachers’ multiple choice tests covered different content.

Multiple Choice Assessment Results

As discussed in the previous evaluation report (Solano County Teaching American History Grant Report: Year One) on the first year of implementation of the TAH program, paired-samples t-tests indicated that teachers in both the 2008 “Gear-Up” session, the 2008-09 school year, and the 2009 Summer Institute scored statistically significantly higher on the multiple choice post-test than on the pre-test, indicating that teachers tended to improve their content knowledge over the course of the program. Similarly, paired-samples t-tests indicate that 8th grade teachers who participated in the TAH program over the course of the 2009-10 school year made statistically significant gains between pre- and post-testing. Over the course of the 2009-10 school year, teachers improved their scores by an average of 8 percentage points (Table 2). Please see the Year One report for details on multiple choice test preparation and administration procedures.

Despite the improvements made by teachers in the majority of the TAH sessions, 11th grade teachers who participated during the 2009-10 school year did not make statistically significant improvements between pre- and post-testing. This lack of improvement, as well as differences in the magnitude of teachers’ gains in the different TAH sessions, may be due in part to differences in the content covered and the assessments administered at each of the sessions. Because both the content and the assessments were different in each of the sessions, it is possible that the degree of difficulty varied across assessments, making comparisons across sessions difficult. However, overall these findings are promising, indicating that over the course of all TAH sessions with the exception of the 11thgrade 2009-10 session, teachers are showing improvements in their content knowledge. Results from the 2010 Summer Institute content knowledge assessments will be reported in the Year 3 report. Table 2 shows the results of the multiple choice content knowledge tests in each of the sessions.

Table 2. Results of The Multiple Choice Content Knowledge Tests

Participating Teachers (n) / Pre-Test Average Percent Correct / Post-Test Average Percent Correct / Average Percent Gain
3-day “Gear-Up” 2008 / 21 / 84 / 92 / 8*
2008-09 8th Grade / 14 / 45 / 78 / 33*
Summer Institute 2009 / 18 / 60 / 77 / 17*
2009-10 8th Grade / 13 / 65 / 74 / 8*
2009-10 11th Grade / 29 / 64 / 64 / 0

* Indicates the gain is statistically significant (p <.05). Statistical significance levels (p) can be thought of as the likelihood that the pattern of responses is due to chance; thus smaller values of p indicate a higher level of confidence that the results are due to real differences rather than random variation.

In addition to examining teachers’ gains on the multiple choice assessments, the relationships between teachers’ performance on the assessments and the number of hours they attended the TAH program, the cohort in which they entered the study, the number of sections of American history taught, and their years of experience teaching American history were examined. After controlling for the other variables, number of hours of participation, number of sections taught, and years of experience were not significantly associated with the teachers’ performance on the multiple choice content knowledge test. However, the cohort that the teachers entered the studywas significantly related to the gains they made between the pre- and post-tests in the 2009-10 school year. Teachers who participated in the 2008-09 cohort tended to make larger gains, by approximately 2.44 points (12.2 percentage points), than teachers who entered the study in the 2009-10 cohort. This effect is statistically significant even after controlling for the number of hours that the teachers participated in the TAH program and, therefore, is not attributed to teachers in the 2008-09 cohort having the opportunity to participate in more hours of training. It is possible that this finding may indicate that the effects of the TAH training may become evident after the teacher has had time to utilize what they have learned and apply the material in their classroom.

DBQ Assessment Results

The second way in which teachers’ content knowledge was evaluated was through the use of DBQ assessments. DBQs are commonly used as a student assessment approach in advanced placement (AP) history classes. According to one source:

The document based question (DBQ) is designed to enable students to work like historians, analyzing and synthesizing evidence from a variety of sources and media. Students will be evaluated on their ability to interpret such factors as purpose, source, bias, date, and place of origin, tone, etc. In order to receive a satisfactory score, students must establish and prove their thesis through accurate and sophisticated utilization of the available documents. The DBQ is designed to test the skills a historian uses in interpreting historical material.

(

For the purposes of this evaluation, participating teachers were asked to complete DBQs in response to prompts that focus on how the teacher would use materials provided to develop a lesson for his/her students. The prompts used for both 8th and 11th grade teachers for the 2009-10 school year program is listed below.[2]

Administration Prompt:

In any written format, please respond to the documents by:

a. Demonstrating the theme, topic, and/or content of the following documents.

b. Outlining a lesson plan that discusses HOW you would use these documents in

your classroom to explain the content.

c. Describing specific activities or strategies to help your students comprehend and

think critically about the documents.

d. Using at least two of the documents in your answer.

As can be seen in Table 3, the only TAH session in which teachers did not make statistically significant gains on the DBQ assessments between pre- and post-testing was the 2008 “Gear-Up” session. As discussed in the previous report, this may be because only 3 days separated the pre- and post-testing. In addition, this initial session was seen as a “pilot” exercise for the DBQ assessment: after the “Gear-Up” session the DBQ assessment prompts were refined and a more sensitive and rigorous rubric for grading was developed. Please see the Year One report for full description of the DBQ assessment development and scoring.

Over the course of the 2009-10 school year, both 8th and 11th grade teachers made statistically significant gains on the DBQ assessment (paired samples t-test). All 8th grade teachers improved their scores, with an average gain of 6.4 points and improvements ranging from 1 to 15 points. Similarly, 11th grade teachers had average gain of 2.7 points, ranging from a 9 point improvement to a decrease by 4 points. Among the 11th grade teachers, 43% showed improvements from pre- to post-testing. Teachers’ DBQ scores were not significantly correlated with years of experience teaching American history, sections taught, or hours of TAH participation.

Table 3. Results of The DBQ Assessments

Teachers (n) / Average Pre-Score / Average Post-Score / Average Point Gain
“Gear-Up” 2008 / 22 / 10.2 / 10.5 / 0.5
2008-09 8th Grade / 15 / 12.9 / 18.9 / 5.7*
Summer Institute 2009 / 18 / 12.3 / 16.7 / 4.2*
2009-10 8th Grade / 16 / 11.3 / 17.7 / 6.4*
2009-10 11th Grade / 30 / 12.1 / 14.8 / 2.7*

Teacher Survey Results

Upon entry into the TAH program and in the Spring of each year, teachers were asked to complete a survey reporting on their goals, what they learned/wanted to learn in the program, and their classroom practices. Teachers were asked what they were most interested in learning about (pre-surveys) and where they felt they had gained in expertise/skills (post-survey). In addition, teachers were asked about the frequency in which they engaged in classroom practices that are considered to be effective in order to examine whether the TAH program is associated with changes in teachers’ practice. Finally, on the post-survey administered to 8th grade teachers in the spring of 2010, teachers were also asked about how participation in lesson study impacted their knowledge of instructional practices and views on collaborating with other teachers. To date, 71 teachers have completed the pre-survey and 53 teachers completed the post-survey in the Spring of 2010.