Shelburne Farms
Middle School Forestry
Wildlife, Woodlots and Workskills
Program Evaluation Final Report
December 2001
Report of results and recommendations for a program evaluation conducted in Fall 2000.
Amy Powers
Shelburne Farms
Introduction
The mission of Shelburne Farms is to cultivate a conservation ethic in children and young adults by teaching and demonstrating the stewardship of natural and agricultural resources. Program goals at the middle school level are to increase awareness and appreciation of natural and agricultural resources and the working landscape and, ultimately, to inspire children to be active stewards of our environment.
The Shelburne Farms education team is always striving to improve its educational programs through assessment, reflection and evaluation. The main purpose of this study was two-fold:
1.) to determine if the Middle School Forestry Field Trip, known as Wildlife, Woodlots and Workskills (WWW), effectively meets the four learning objectives that are outlined for students; and 2.) to evaluate the impact of a pre-visit to students’ classrooms prior to their field trip on the attainment of those objectives.
Objectives for the WWW field trip:
· Students will learn and practice techniques used by wildlife biologists and foresters when assessing a forest as a resource.
· Students will learn how trees are classified and simple tree identification.
· Students will use a clinometer to determine the height of a tree.
· Students will use a Biltmore Stick to determine the board feet of lumber in a tree.
(See Appendix A for field trip outline and pre-visit outline.)
One purpose of this comprehensive evaluation is to enable staff to make sound decisions on how to best structure field trip or other programs based on evidence rather than supposition. A second purpose is to offer Shelburne Farms members and supporters documentation of the effectiveness of Shelburne Farms’ programs and reflectiveness of its practice.
Many research studies have provided insight into the effectiveness of Environmental Education. For a review of the literature that served as a foundation for the development of this project please refer to the program evaluation report titled Shelburne Farms Forestry Program Evaluation Final Report August 2000. This evaluation was conducted in fall 1999 and the report, written by Amy Powers, is housed in the education department at Shelburne Farms. (See Appendix B for a Bibliography.)
Methods
With input from Shelburne Farms’ Program Director, School Programs Director, Field Trip Coordinator and the teaching staff, the evaluation was designed and implemented in the fall of 2000. Participants in the evaluation hailed from four schools in Vermont and spanned five grade levels. Included in the study are 190 students, eight teachers and eight Shelburne Farms field trip staff.
Participants in WWW Evaluation
§ 190 students, grades 3-7
Williston Central School (3rd-4th grade, 40 students)
Rutland Middle School (7th grade, 100 students)
C.P. Smith Middle School (5th grade, 60 students)
Georgia Middle School (5th-6th grade, 40 students)
Instruments
Data collection for the evaluation consisted of a six-part sequence. (Please see Appendix C for samples of each of the data collection instruments used in this study.)
· A Shelburne Farms staff member visited each classroom approximately one week before the field trip to administer a 15-minute pre-test survey to students. For classes that had scheduled a pre-visit lesson, the pre-test was administered 15 minutes before the pre-visit activities. 190 pre-tests were collected. 26 were discarded to create a matching set with available post-tests.
· Informal interviews of the students were integrated into the field trip, occurring in small groups. Each Shelburne Farms leader of roughly ten children asked a prescribed set of questions immediately before and immediately after the field trip. Answers were recorded by a parent chaperone, staff observer or classroom teacher. All students were included.
· One month after the field trip, a 15-minute post-test, identical to the pre-test, was mailed to each teacher with instructions for administering it to the class. Tests were then mailed back to Shelburne Farms. 162 post-tests were collected.
· One month after the field trip teachers were sent a written survey to assess their observations of field trip learnings during and subsequent to the trip and to request their feedback on the program in general. 5 of 8 surveys were returned.
· Shelburne Farms staff were asked daily to complete a survey about pertinent conditions of the day’s field trip, and salient observations of the day’s events. 100% of surveys were returned.
· Shelburne Farms staff also completed a final survey reflecting on the WWW field trip as a whole. 4 of 8 surveys were collected.
Challenges
The primary challenge in conducting this evaluation was the small sample size available. Four schools indicated interest in this field trip and thus the study is only able to compare the learnings of a relatively small number of classes. One of the four schools who participated did not request or receive a pre-visit while three schools did receive a pre-visit.
The second challenge was the wide disparity of grade-levels represented in those four schools. It is difficult to compare with precision the effects of activities and the acquisition of concepts between students with as great as a five-year age gap. Particularly in childhood, five years has an immense impact on a student’s reading ability, cold tolerance, prior exposure to concepts, social skills while doing teamwork, etc. Nonetheless, it was decided that having some information to guide staff practice is better than having no information, and proceeded with the study. Additionally, the study represents an authentic glimpse of the types of variables –age differences, varying group size, different teaching staff—Shelburne Farms face in implementing any of its programs, and in that sense is useful in its lack of a precise sample.
The objective these challenges impacted most was the intention to determine the effect of a pre-visit on the students’ ability to meet program objectives. The fact that the youngest students (3-4) were the only ones not to receive a pre-visit made it hard to accurately compare those who did and did not receive the visit. This effect will be more explicitly discussed below.
Discussion
The following discussion is gathered from classroom teacher surveys, Shelburne Farms staff surveys and student interviews were combined to provide insight into the following eight categories.
Field trip objectives
Looking at the four field trip objectives, students were clearer on the concept of what a forester does than what a wildlife biologist does, though knowledge gains were made in both areas. 25% more students could correctly identify the roles of a forester after the field trip. In a post-trip interview one student commented, “My favorite part of the day was using a clinometer…I like doing a forester’s job.” (5th grader—CP Smith). Another comment a month after the field trip makes clear that students are considering the activities of the field trip as opportunities for career development “What is the life of a forester like?” This question represents a category of questions that arose in both the small group interviews and on the pre and posttests. Staff should consider carefully how they present the life of a forester as a career, and what message is sent to students.
One of the skills important to a forester is completing plot samples. Overall pre and post tests showed that students made a 43% gain in their ability to describe “a method for counting trees in an area.” Before the field trip many students described a person going through the forest counting each individual tree whereas after the field experience a large portion of students could describe the plot sampling technique wherein a forester measures off an area and uses multiplication and estimation to determine the number of trees in the larger area. It is interesting to note that despite the fact that students grasped this concept well, the activity itself had limited appeal. One staff noted low energy in students during plot sampling activity and post-trip interview question data regarding most enjoyable part of the trip show that only 5 of 182 student responses included a mention of plot sampling.
Staff observed that, of the four objectives, learning about classifying tree types was least clearly met, attributed primarily to a lack of time to spend doing tree identification in depth. Two teachers also commented that they would have appreciated more practice in tree identification for students. See recommendations below.
It is clear from the pre and post data, and from student and staff comments, that use of the Biltmore sticks and clinometers was readily grasped. By the end of the field trip some students could give verbal directions for using the Biltmore stick. Nonetheless, one teacher commented that more practice with the Biltmore sticks would have been beneficial because a few were still confused about the purpose of the stick and confuse it with the thickness of the tree’s bark. The teacher suggested simple posters to clarify the use of this tool. See pre-visit section below for further exploration of the field trip’s success in contributing to completion of the third and fourth objective.
In evaluating any program it is important to be open to the successes that are not explicitly sought, but should be acknowledged nonetheless. Though it was not specified as one of the objectives, an important part of forestry that is taught by the Shelburne Farms staff is the concept of tree life cycle. One teacher, asked to relay her observations of student learning during field trip, reported that students were able to order the life of a tree easily and also discuss the stages. (Student comment: “I learned that a tree starts as a seed and goes to a seedling to a shoot to a sapling to a full grown to a grandpa tree to a dead log. Another student adds, “Height determines that name, either a seedling, sapling, pole, etc.” And a third student, in another class completes the picture, “I know how to tell if a tree is a sapling (the size of an a-ok sign) or a pole tree (two hands in a circle).”) Finally, one post-trip student comment on this subject shows an understanding of the ecological concept of competition, “Most saplings don’t make it because they are competing for water, sun and space.”
Comments on Effects of the Pre-visit
The pre-visit was noted across the board as an important element of the field trip. One staff member commented, “I was impressed with how independent they were in using both instruments. They needed little review after the pre-visit and could read the directions. Amazing!” Another staff remarked about the class who did not receive a pre-visit, “It’s a challenge to give directions for using the Biltmore stick and Clinometer” and explain the concept (what the purpose of these tools is.) This remark would suggest that the pre-visit enables staff and students to go into more depth on the trip itself since preliminary details about equipment use have been covered. Similarly, the non-pre-visit group’s responses about what they know about forestry after the field trip tended to be more focused on their new knowledge of the Biltmore stick and clinometer while the pre—visit groups’ post-comments delved into other subject areas: tree identification, the role of wildlife biologists, facts about trees and wildlife.
Further, seventh grade students who had learned to use the instruments before the field trip were beginning to make connections between the two systems (e.g. comparing the number of 16 foot logs to the height of the tree.) Teachers also expressed satisfaction that students were so capable of doing the measuring activities without a lot of help. One teacher commented, “I enjoyed seeing my students apply what they had learned from the pre-visit.” This comment emphasizes the possibility for increasing teacher ability to do student assessment which on a field trip, a goal of Shelburne Farms’ field trip program in general.
We see evidence of the pre-visit effect in student responses to the question “What do you know about forestry?” which differed between students who had received a pre-visit and those who had not. Comments by students from C.P. Smith school, which did have a pre-visit, included: “You need to stand 66 feet away from the tree to use a Biltmore stick.” “I know that 16-foot logs are important to foresters.” “I know how to measure tree height.” “ “When you cut down trees they have to be at least ten inches wide.” In contrast, answers from students from Williston Central School who did not have a pre-visit were more random in their themes: “I know that maple syrup comes from trees.” “General Sherman is the largest tree in America.” “Coffee comes from trees” “A redwood’s bark is four feet thick.” One can infer that, regardless of actual skill level, students come to the field trip more focused and thus better prepared to study the topic at hand.
We had a pre-visit where students learned to use Biltmore sticks and clinometers and we practiced again between pre-visit and field trip. (Georgia)
Overall, this field trip is well received by teachers, enjoyed by students, and growth was seen in all knowledge/content areas. The biggest remaining concern is whether students’ attitudes about forests were changed at all. Referring to pre and post tests, in all classes but one, students wrote that they preferred to learn forestry in a forest less after attending the field trip and no class increased their preference for learning in the forest following the field trip. The average of all classes was a 3% decrease in preferring the forest to a classroom; the range was from no change in preference (seventh grade) to a decrease of 32% of students in the third grade class. This evidence points to a lack of enthusiasm for the outdoor learning experience in this case. It appears to be more directly correlated with grade level than with weather conditions. Georgia Middle School, for instance, were fifth graders who were at the farm on a very cold day, and their preference decreased by only 6%. Similarly, students were asked the open-ended question, “Why are forests important?” A tally of their responses revealed that there was a 5% increase in the understanding that forests are important for building and wood extraction but in all other categories that students mentioned (oxygen, human uses such as food, animal homes, shade, recreation, etc.) references decreased in the posttest.