Shelburne Farms

Middle School Geology Program Evaluation

Final Report

January 2002

Peter Bullock

Amy Powers

Shelburne Farms

Report of results and recommendations for a program evaluation conducted in Fall 2000.

Shelburne Farms

Middle School Geology Program Evaluation

January 2002

Introduction

Shelburne Farms’ is dedicated to cultivating a conservation ethic by teaching and demonstrating the stewardship of natural and agricultural resources. In fulfilling this mission the School Programs are designed to increase students’ and teachers’ awareness and appreciation of Vermont’s environment. They are also designed to address Vermont’s Framework of Standards and Learning Opportunities and to provide a connection to classroom work. Programs are conducted for pre-school through middle school students as well as for pre-service and experienced educators.

Since 1978 the field trip program has offered teachers the opportunity to extend their students’ classroom studies with hands-on day visits to the farm. The programs include age appropriate activities that are designed to develop skills and knowledge necessary for an individual to make informed decisions; to develop effective communication skills; and to provide experiences that lead an individual to develop a sense of one's place and comfort with multi-disciplinary, real-world experiences. The structure of this program has varied throughout the years. Currently, most teachers bring their class to Shelburne Farms for a one-day, onsite field trip experience.

The main purposes of this study are three-fold: 1.) to determine if the Middle School Geology field trip effectively meets the learning objectives (see below); 2.) to evaluate the benefits of conducting professional development for teachers linked to the geology field trip theme; and 3) to evaluate the benefit of a school visit (known as a pre-visit) prior to the field trip. (See appendix A for pre-visit outline.)

The results of this study are intended to help the education staff design future programs based on a solid understanding of what educational content, methodologies and practices best accomplish Shelburne Farms’ mission.

Objectives for the Geology Field Trip:

  • Students will name and identify the three types of rocks: igneous, sedimentary, and metamorphic.
  • Students will demonstrate the rock cycle.
  • Students will discover, identify, and classify rocks found on our beaches.
  • Students will explore the geologic history of our region.
  • Students will be introduced to the basic geologic processes that formed our region.
  • Students will simulate some of the weathering effects on rocks.
  • Students will search for and investigate fossils as evidence to historic geologic events.

(See Appendix B for field trip outline, including Vermont’s Framework of Standards addressed.)

Evaluation Questions

Several instruments were designed to assist us answer the following questions:

  1. Does implementation of a one-day middle school geology field trip meet the established learning objectives?
  2. Are learning objectives better met when the classroom teacher has participated in a prior professional development experience pertaining to the field trip subject?
  3. Are learning objectives better met when a pre-visit to their classroom is provided before the field trip?
  4. Are learning objectives best met (significant gains) when all three educational methods (professional development, pre-visit and field trip) are utilized?

Methods

Participants:

  • Seven schools (Representing 11 classrooms, 222 students, grades 3-6)
  • ShelburneCommunitySchool (3rd-4th grade, 44 students)
  • ChamberlinMiddle School (5th grade, 48 students)
  • St. Joseph’s School (5th grade, 22 students)
  • SwantonMiddle School (5th – 6th grade, 46 students)
  • SouthBurlingtonCentralSchool (4th grade, 38 students)
  • HinesburgMiddle School (4th grade, 15 students)
  • BellwetherMiddle School (4th – 5th grade, 8 students)
  • Shelburne Farms staff and volunteers
  • Seven full time year round staff
  • Five full time education apprentices

Program Evaluation Sequence:

  • Professional Development: Staff from Shelburne Farms conducted one-day teacher workshop specific to geology 3-5 months prior to the field trip.
  • Pre-Test: Staff from Shelburne Farms visited each class approximately one-week prior to field trip and administered a two-page test, surveying their knowledge of geology.
  • Pre-Visit: Staff from Shelburne Farms led hands on activities suitable for classroom on rock types after pre-test. (See Appendix A)
  • Field Trip Interviews: Students were orally surveyed in groups of 8-12 immediately before and after field trip.
  • Daily Comment Sheet: Written survey for staff to report conditions and experiences after teaching each field trip.
  • Post Test: One month after the field trip, staff from Shelburne Farms visited each class again, administering an identical two-page survey as in the pre-test.
  • Post Post Trip Interview: Five students per class were interviewed one month after the field trip.
  • Teacher Interview: Teacher was interviewed individually one month after the field trip.
  • Final Survey for Shelburne Farms Leaders: Written survey for staff leaders to share their reflections, experiences, and recommendations after teaching geology field trip.
  • Survey for Shelburne Farms Staff: Written survey for staff to add their insights, thoughts, observations, and recommendations after reviewing the data.

(See Appendix C for a more detailed evaluation sequence and copies of the instruments used.)

Results

Attached are seven summary tables. See Appendix D.

They include:

  • Program Evaluation Planning Form
  • Pre and Post Test Comparisons by Class

Pre and Post Test Comparisons by Grade

Pre and Post Test Comparisons for All Classes, All Grades

Pre and Post Test Comparisons for Field Trips, Pre-Visits, and Professional Development

  • Data Sheet for Staff Daily Comments
  • Pre and Post Test Comparisons for Implicit Objectives
  • Data Sheet for Pre and Post Student Interviews
  • Data Sheet for Post Post Student Interviews
  • Data Sheet for Teacher Interviews

Limitations of the Study

Several limitations in methodology and in the study sample make it difficult to answer the evaluation questions. The test survey, for one, is not worded consistently in the manner the pre-visit and the field trip were presented to students. This inconsistency may have provided for student responses out of context with what was discussed on the field trip, though the survey was expected to reflect that discussion. It is also apparent that most of the objectives of the field trip are not clear learning expectations. In regard to objective six, one staff member noted, “from observing during field trip probably 95% of kids did simulate weathering,” and another added, “kids are doing it, experiencing it.” Since the objectives cannot be formally assessed, this evaluation has focused on the implicit learning students are expected to gain by participating in activities on the field trip. It is highly recommended that in the future objectives be stated in terms of those implicit learning expectations, whereas the activities of the field trip outline are designed in meeting them.

This study also lacks a varied group of participants, i.e. classes of the same grade level that experienced all three educational methods, the field trip, the pre-visit, or professional development. On the other hand, such diverse experiences within the same grade level are not representative of classes that attend Shelburne Farms, nor possible with the small study sample. This does not allow for clear trends between the three educational methods under study and grade level to emerge.However, not surprisingly, there is a correlation between class performance and grade level (See pre and post test comparisons in Appendix D.) This has clouded the success of classes that fall into the significant gains category since they are comprised of significantly older students and do not have a younger counterpart to compare their achievements.

Fifth grade, though, is a blatant exception to the rule. Most fifth grade classes, though on average the highest achievers for pre test scores, and not indicative of trends in grade, displayed below average knowledge gains on every single question. In this case, class performance is a major issue. As has been suggested above, a trend is extant by grade level, and for fifth grade to be the exception to that trend cannot be explained by characteristics of fifth graders, but of the classes that represented them. Indeed, it was not one fifth grade class that brought down pre-visit or professional development group performance, but three of the five fifth grade classes performed well below average, below all third and fourth grade classes. Their low performance seriously taints the data generated on the influence of pre-visit and professional development experiences.

Similarly, the study group is also limited since no teacher participated in the professional development whose class did not benefit from a pre-visit as well. This has made recommendations to continue one contingent upon the other.

Other limitations are due to the nature of qualitative data, which describes rather than measures understanding or knowledge gathered by certain experiences. However, it best captures the varying quality of field trip conditions such as weather, timing, group size, and behavioral distractions, all which contribute to different experiences and learning. Of final note, this study is uniquely limited in having been planned and implemented by one group of people, yet analyzed by another.

Discussion

Meeting the Expectations of the Field Trip

Seven schools representing a total of eleven classes took part in the Middle School Geology field trip program. Teacher interviews all portray satisfaction with the content of the field trip and in meeting their expectations. One teacher suggested a more in-depth focus on the history of geologic processes in Vermont, but otherwise disappointment was focused on weather conditions at the waterfront. Though one school’s visit was brought indoors due to rain, it should be noted that staff and teachers at times identified different factors to blame on the days when the wind or the temperature presented a challenge. One teacher attributed morning inattentiveness with “too much sitting for too long” at a cold temperature as a flaw in program structure. Yet writing independently about the same group and weather conditions, one staff leader noted: “windy and cold, kids not dressed appropriately,” and another, “it was a beautiful day, environmentally the conditions were ideal.” This should highlight the importance of communicating to teachers the necessity of bringing children prepared for all types of weather, and also the usefulness of alternative, sheltered areas in which to learn. Some groups that had to miss out on outdoor activities suffered in ways that may have affected their learning. Staff lamented, “we were inside which I feel definitely affects the quality of the day,” and “…There were more distractions.” Staff should be prepared to teach effectively in an alternative indoors setting, even if it is less ideal.

Overall the field trip was labeled a success; “I didn’t need to teach how rocks are formed because they got it on the Field Trip,” a fourth grade teacher observed. Another shared that “after the field trip, the students began to bring in rocks and ask what kind they are. I’ve seen some classification and sorting, especially in the few days right after the field trip.” Another gave notice “when writing legends back in class [students] used vocabulary from the field trip such as metamorphic, pressure, shale, and limestone.”

Conclusions from staff were also positive. All viewed that the implicit learning expectations specified in objective one were met. Yet staff felt that the other objectives were met through participating in the field trip. This input from the staff would suggest that learning expectations and objectives be synonymous and clearly stated, and that staff might benefit from a discussion about the relationship of the objectives and activities before teaching the program.

Pre interviews were sprinkled with lots of comments on rocks, such as granite (one classroom visited a quarry before the field trip: “they cut granite with a diamond saw”) and minerals, precious stones (“people like diamonds for jewelry”), or incomplete knowledge of geologic processes (“rocks dissolve into water”). After the field trip, children report knowing “it takes one million years for granite to get hot enough; one million years to cool down.” They also report “how rocks are changed by weather…” that “fast moving water makes rocks smaller,” and recall learning “how Pangea broke up” and “how Adirondacks were formed by plates pushed together.” Clearly, their understanding of geology have moved beyond a familiarity of geologic terms to a functional understanding of geologic processes.

Meeting Specific Field Trip Objectives

Objective 1) Students will name and identify the three types of rocks: igneous, sedimentary, and metamorphic

Classes that had not benefited from a pre-visit nor had a teacher who had participated in a professional development workshop, displayed the least amount of knowledge concerning rock types, their names, or how to identify them. When asked to name one new thing they had learned on the field trip, most students shared that they had gained an understanding of rock types. One explained how “[it] takes heat and pressure to turn into metamorphic rock,” and another added “marble is a metamorphic rock.” Regarding sedimentary rocks, a lot had learned that pressure can combine small particles together. Though their comments about the new things they had learned and their answers on the post test surveys did not delve deeper into how small particles may be piled on top of each other (such as sediment loads in rivers and their settling in ponds, lakes, and oceans), classes that only benefited from the field trip made significant leaps in their understanding in this area. For instance they new nothing at all (0%) about sedimentary rocks before the field trip, and next to nothing about metamorphic rocks, and in these subjects displayed their greatest increases in comprehension. More knew about igneous rock before the field trip, yet made strides in this area as well.

One fourth grade student reported in the Post Post Student Interviews: “I used to just like rocks because they were interesting and now I look at them and ask if they are igneous, metamorphic or sedimentary and I have all these questions about them…” Data for all classes indicates that objective one regarding the three rock types was the most successful objective met, regardless of pre-visit or professional development impact. During the teacher interviews one teacher shared that her classroom was “writing Porqua stories, and when I mentioned a rock type off hand, a student corrected me on my rock terminology. I had said the wrong type and he was thinking back to the field trip.”

Objective 2) Students will demonstrate the rock cycle

Objective two is similar to objective one, and correlates to an activity on the field trip called Rock Cycle Roll, where students roll a die that is labeled with three different rock formation conditions and others that describe inactivity (much like geologic time). Their job is to travel from one classification of rock to another according to the changes that take place to them described by the die. On the data sheet for Pre and Post Interviews this activity is a favorite to students, second only to weathering experiments. One teacher reports: “I could tell they grasped the rock cycle through playing the dice game because when we reviewed for our test they referred to the rock cycle changes.” Another teacher adds, “the Rock Cycle Roll game got them to catch on to the idea that rocks change form.” One student explained how “metamorphic can turn into igneous and igneous to metamorphic.”

Objective 3) Students will discover, identify, and classify rocks found on our beaches

Evidence of student’s ability to identify and classify rocks, and their enthusiasm for discovering them abound in the interviews with both students and teachers. One teacher commented that “we had collected rocks before the field trip, and they expressed interest in going back to the collections to identify the rock types. After the field trip they looked more closely at their collections.” In the Post Post Student Interview, one child confesses that “I hadn’t noticed rocks before… and after the field trip rocks really stood out when we went hiking. I found five kinds: limestone, coal, shale with calcite, quartz and two I didn’t know.” Another recalls “At Lessor’s Quarry, before they told us the rock types I tried in my mind to identify them. Sometimes you can see layers and know its sedimentary.” Two students also made poignant attempts to share what they had learned with their family. One “told [my parents] about each of the rock types and it was new to them.” Another “brought home five different kinds of rocks and set them all over the supper table…” It was also clear in post interviews right after the field trip that many children were beginning to think of rocks by the three major types, and even to begin to classify them by age. In one group that had spent time discussing the lake and the slate and calcite rocks along the beach, one student recalled, “some of these rocks are 450 million years old,” while another student remarked that others were formed 100 million years ago.