Course Evaluation Report
Advanced Facility
Management Practices
Timothy Harvey
Facility Management Officer
WASO – PFMD
National Park Service
Elizabeth Dodson
Training Manager
WASO – PFMD
National Park Service
October 2009
Completed in accordance with Sub Agreement 122 of the National Park Service-
IndianaUniversity Cooperative Agreement H2420060015
Stephen A. Wolter
Executive Director
Christy McCormick Zachary Carnagey
Project Team Project Team
Eppley Institute for Parks & Public Lands
IndianaUniversityResearchPark
501 N. Morton Street, Suite 101
Bloomington, IN47404
812.855.3
Acknowledgements
The following individuals contributed to the development and instruction of the Advanced Facility Management Practices course:
Melissa AxtmanNational Park Service
Wayne ChallonerNational Park Service
Yvette ChavezNational Park Service
Jamie CupplesNational Park Service
Jenny DemsNational Park Service
Teresa DickinsonNational Park Service
Betsy DodsonNational Park Service
Aaron DoweNational Park Service
Bill EllisNational Park Service
Glen FickbohmNational Park Service
Tim HarveyNational Park Service
Steve HomanNational Park Service
Ken HornbackNational Park Service
Don MannelNational Park Service
Dayna McClureNational Park Service
Jeri MihalicNational Park Service
Bill OsterhausNational Park Service
Sandy PuseyNational Park Service
Matthew BerryEppley Institute
Zachary CarnageyEppley Institute
Stacey DoyleEppley Institute
Allene LowreyEppley Institute
Christy McCormickEppley Institute
Steve WolterEppley Institute
Dana AndersonIndiana University
Virginia SalazarNational Park Service
Richard UllmanNational Park Service
This publication may not be duplicated without the permission of the Eppley Institute for Parks and Public Lands, acting on behalf of Indiana University. The National Park Service and federal agencies may duplicate it for training and administrative purposes, provided that appropriate written acknowledgement is given. No other state or local agency, university, contractor, or individual shall duplicate this publication without the permission of Indiana University.
Copyright 2009, the Trustees of Indiana University
on behalf of the Eppley Institute for Parks and Public Lands
TABLE OF CONTENTS
INTRODUCTION
Both the Department of the Interior (DOI) and the National Park Service (NPS) have targeted the facility management profession as “mission-critical” due to the importance of maintaining the built environment and the high numbers of facility managers eligible to retire in the near future. In anticipation of the growing need for competent facility management, the NPS has developed a comprehensive set of facility manager competencies which form a roadmap for the development of the facility management workforce. Additionally, the Facility Manager Leaders program (FMLP), of which the Advanced Facility Management Practices (AFMP) is one of the core courses, aims to develop a new cadre of competent facility managers for the future.
The AFMP course builds on the knowledge gained by successful completion of the Principles of Asset Management course and courses completed duringthe first independent study period titled Distributed Learning Session One. At AFMP, students had the opportunity to use their knowledge, judgment, and experience to anticipate, prepare, and plan for critical issues facing NPS facility managers.
This course was designed to provide facility management personnel with an understanding of the importance of park strategic plans and mission goals in the development of an Annual Work Plan (AWP). The course demonstrates how mandates, regulations, and policies guide the application of the Facility Management Software System (FMSS) in the management of park assets.
Course Description and Rationale
AFMP was a two-week instructor-led course using a teaching team that provided lecture, discussion, and small group activities, concluding with the creation of an AWP. All coursework and instruction was couched in the simulated environment of a fictitious park, Sea Otter Island National Park (SEOT).A student completing this class met facility manager competencies in the NPS by being able to do the following:
- Plan operations, maintenance, repair, and alterations programs using a mission-focused Asset Management Plan (AMP), adjusting plans through evaluation to fit within budgeted funds.
- Utilize the AMP to link facility management and develop the most efficient organization, program accountability, and defendable management decisions for assets.
- Utilize data from the FMSS to develop a credible work program that manages budget allocations associated with setting priorities, meets required mandates, and responds to management decisions in the framework of the Stewardship of Federal Facilities.
- Identify and execute the required elements of the FMP Standard Business Practices (Directors Order No. 80), including related components of park unit business plans and industry standards.
- Lead colleagues, subordinates, and managing supervisors to understand the visitor, personal, park unit, and Service-wide benefits of proactive asset management and use of the FMSS.
- Coordinate the creation of an Annual Work Plan (AWP) to demonstrate and advocate the benefits of using the FMSS for park unit management through the use of data analysis and data driven decision making.
Course Assessments
Assessments for the course were both formal and informal, and individual and group. Formal assessments included the following:
- PMIS Project Submittal
- Oral Presentation with PowerPoint
- Project Plan: Annual Work Plan
- Functional Analysis
- Critical Systems Priority List
- Oral Annual Work Plan
- Written Annual Work Plan
- Team Assessment
- Class Participation
Except for the Oral Presentation with PowerPoint, Team Assessment, and Class Participation, all formal assessments were group assessments. This group assessment approach was designed to reflect and foster the development of the facility manager competencies for Leading People (VI.B) and Building coalitions and communication (VI.E).
Individual assessments, besides those mentioned above, were primarily informal and reflected principles of adult learner theory, especially the concept that adult learners take more responsibility in their learning. These assessments, therefore, gave students the chance to assess their own knowledge with the expectation that, should they find some deficit, students would seek additional instruction from their cohort or instructors.
An example of these informal assessments was the homework from the Excel Workshop. At the completion of this session, the instructor gave the students homework which required the use of skills developed during the session. The following morning, the instructor discussed the answers and provided students the opportunity to ask questions if their answers differed. For two of the homework problems, some students did have different answers, and the class discussion that followed included remediating instruction from not only the instructor, but also from fellow classmates. Though informal and not graded, assessments such as these allowed instructors to gage student learning and modify instruction as needed, without reducing classroom instruction time or subjecting students to the added stress of formal assessments.
Additionally, mastery of the content presented in the sessions during the AFMP was required in order to produce a comprehensive AWP.
PROCESS OF EVALUATION
The information contained in this report comes from directly from data collected in the form of individual student and team member evaluations of the Advanced Facility Management Program course, held in Santa Fe, NM from October 19th to October 29th, 2009.
Methodology
Course feedback was provided in several different ways. The participants were all asked to complete an evaluation of each of the instructors; if the instructor taught more than one lesson, they were asked to complete an evaluation for each lesson. Additionally, they each completed an evaluation of the entire course. Participants also provided feedback on the Mentor Workshop and its instructors. A summary of these evaluations, along with discussions and observations based on the feedback, is included later in this report.
Evaluations of the Instructors
Students were asked to provide quantitative ratings and qualitative comments for each of the instructors throughout the two-week course. The evaluations were done at the end of each day and were specific to the instructors who taught on that day, which means that if the instructor taught more than one session they were evaluated more than once.
Participant Course Evaluations
At the conclusion of the course, the students were asked to complete a participant course evaluation. The evaluation tool, developed by the Eppley Institute, included quantitative questions about both the course and student learning, as well as space for qualitative comments about different aspects of the course.
Mentor Workshop Evaluations
On the second Wednesday of the course mentors participated in a workshop which highlighted effective mentoring strategies and techniques and their role throughout the FMLP. To measure the effectiveness of this workshop and the mentor-student meetings, the mentors were asked to complete a mentor workshop evaluation. This evaluation used both quantitative and qualitative questions to assess the effectiveness of these two sessions and how well they met the needs of the mentors.
The Audience
This course was designed to meet the needs of those charged with execution of the facility management function within the NPS. Course participants came from parks throughout the country and represented a variety of park types and sizes, with a range of experience levels. The audience consisted of those accepted into the year-long Facility Manager Leaders Program (FMLP). The intended audience for this course included the following:
- New facility managers and facility managers who are new to the NPS
- Those in the NPS who want to build experience in order to apply for a facility management position or those seeking promotion to a facility management position
- Individuals in other disciplines within the NPS who wish to pursue a career in facility management
- Long-term chiefs of maintenance or facility management specialists who could benefit from facility management updates
Course Coordinators
The course coordinators were those Eppley Institute and NPS staff who worked together to develop the course, plan the course, and ultimately mount the course. They were responsible for ensuring that the sessions were timely, that the instructors had their materials, and that the entire course ran smoothly. They helped facilitate the various aspects of the course, the instruction, and the activities.
The course coordinators were highly successful in facilitating the transitions from one course session to the next. They provided and introduction for each of the instructors, including their experience in facility management both within and outside of the NPS. They also highlighted key points from the different sessions when necessary and kept the sessions moving from one to the next. By serving as the “glue” that kept the entire course together, the course coordinators provided the students with continuity and a common voice through which the key points for each session were highlighted.
Instructors
The guest instructors were selected based on identified course and student needs, including subject matter expertise, training and education skills, and the instructors’ availability and willingness to participate in the course. The instructors were highly credible and represented all levels of the NPS, including the National, Regional, and Park levels. The many perspectives of the NPS, provided through the guest speakers’ variety of experience levels, allowed the students to gain a holistic view of facility management. Participants were able to develop a basic understanding of other divisions and areas in the NPS and their applicability to Facility Management. This enhanced understanding supports the development of leadership skills.
Course Agenda and Corresponding Guest Speakers
The following tables provide the course agenda and the guest speakers for each session.
Week I
Monday, October19th, 2009
TimeSession TitleGuest Speakers
8:00 am-9:30 amCourse Introduction and Welcome to the ParkDodson, Wolter
Chief of Maintenance, BANDLiza Ermeling Santa Fe Office Representative Aaron Mahr
9:45 am-11:45 amStudent PresentationsStudents
11:45 am-1:00 pmLunch
1:00 pm-2:30 pmStudent Presentations (continued)Students
2:45 pm-3:45 pmDLS #2 PlanningChristy McCormick
4:00 pm-5:00 pmTime ManagementJeri Mihalic
5:00 pm-5:15 pmOne-Minute Paper, Instructor EvaluationChristy McCormick
Tuesday, October 20st, 2009
TimeSession TitleGuest Speakers
8:00 am-8:30 amThe Good of the Day & AnnouncementsCourse Coordinator
8:45 am-11:45 amDiscussion of Scientific Method and Zach Carnagey
Data Analysis ToolsAaron Dowe
11:45 am-1:00 pmLunch
1:00 pm-2:15 pmWork Types and their ImportanceJeri Mihalic
2:30 pm-4:30 pmCultural and Natural ResourcesVirginia Salazar
4:30 pm-5:00 pmOne-Minute Paper, Instructor EvaluationCourse Coordinators
Wednesday, October 212t , 2009
TimeSession TitleGuest Speakers
8:00 am-8:30 amThe Good of the Day & AnnouncementsCourse Coordinators
8:30 am- 11:45 amWriting for ImpactDana Anderson
11:45 am-1:00 pmLunch
1:00 pm-4:45 pmLeadership Strategies and SkillsSteve Wolter
6:00 pm-8:00 pmWelcome to SEOT! (party)Outgoing Chief
Steve Wolter
Thursday, October 22nd, 2009
TimeSession TitleGuest Speakers
8:00 am-8:30 amAnnouncements and What is a Simulation?Christy McCormick
8:30 am-9:00 amSquad Meeting-Visitation TrendsSEOT Supt
9:00 am-9:15 amLecturette-Productive Hours in your WorkforceVito Spinale
9:15 am-9:30 amLecturette-Seasonality and Special EventsSEOT AO
9:45 am-11:45 amProject PlanningChristy McCormick
11:45 am-1:00 pmLunch
1:00 pm- 3:00 pmFLREADebbie Brown
3:15 pm-3:45 pmLecturette-Unplanned WorkVito Spinale
3:45pm-4:45 pmTCFOModelZach Carnagey
Betsy Dodson
4:45 pm-5:00 pmOne-Minute Paper, Instructor EvaluationCourse Coordinators
Friday, October 23rd, 2009
TimeSession TitleGuest Speakers
8:00 am-8:30 amThe Good of the Day & AnnouncementsCourse Coordinators
8:30 am-9:30 amSquad Meeting PAMP and PAMP ExecutionBetsy Dodson
9:30 am-1:00 pmGroup Work on SEOT SimulationStudents
1:00 pm-5:00 pmField ExperienceStudents
Week II
Monday, October 26th, 2009
TimeSession TitleGuest Speakers
8:00 am-8:30 am The Good of the Day & AnnouncementsCourse Coordinators
8:30 am -10:00 amHealth and WellnessHeidi Wingart
10:30 am-11:30 amFund SourcesJamie Cupples
11:30 am-12:30 pmLunch
12:30 pm -2:30 pmPMISJamie Cupples
2:45 pm-4:45 pmInformal Visitor Contact: Supporting your EmployeesRichard Ullman
4:45 pm-5:00 pmOne-Minute Paper, Instructor EvaluationCourse Coordinators
Tuesday, October 27th, 2009
TimeSession TitleGuest Speakers
8:00 am-8:15 am The Good of the Day & AnnouncementsCourse Coordinators
8:15 am-8:45 amSquad Meeting: Fleet, Fuel and Other Fixed CostsMelissa Axtman
9:00 am-10:30 amAFMP Final ExamStudents
10:30 am -2:30 pmGroup Work on SEOT
2:30-4:30 pmMentor Student MeetingsMentors, Students
4:30-5:00 pmOne-Minute Paper, Instructor EvaluationCourse Coordinators
Wednesday, October 28th, 2009
TimeSession TitleGuest Speakers
8:00 am-8:30 amThe Good of the Day & AnnouncementsCourse Coordinators
8:30 am-9:00 amSquad Meeting: Presenting your Annual Work PlanSteve Wolter
9:00 am- 10:00 amAsset Management UpdateTim Harvey
10:00 am-11:00 amAsset Management on the GroundDon Mannel
10:00 am-12:00 pmGroup Work on SEOT SimulationStudents
10:00 am- 12:00 pmMentor Workshop (Part One)Tim Harvey
Betsy Dodson
Steve Wolter
12:00 pm-2:00 pmStudent Lunch with Mentors
2:00 pm-4:00 pmGroup Work on SEOT SimulationStudents
2:00 pm-4:00 pmMentor Workshop Betsy Dodson
Steve Wolter
4:00 pm-4:30 pmOne-Minute Paper, Instructor EvaluationCourse Coordinators
6:30 pm-8:30 pmSEOT Presentation Dress RehearsalStudents and Mentors
Thursday, October 29th, 2009
TimeSession TitleGuest Speakers
7:30 am- 8:30 amIndividual and Group Photos
9:30 am-11:30 amGroup SEOT PresentationsEvaluation Panel
11:30 am-1:00 pmLunch
1:00 pm-3:00 pmGroup SEOT Presentations
3:00 pm-4:00 pmCourse Conclusion and Evaluation
5:30 pm-8:30 pmGroup Dinner
Friday, October 30st, 2009
Travel Day
Evaluations of the Instructors
The instructors were evaluated by each of the student participants. The evaluation tool included three quantitative statements as well as a qualitative section that asked for general comments on the overall effectiveness of the instructor and any suggestion for improvements.
Quantitative Instructor Feedback
The course evaluations asked three quantitative questions related to the teaching effectiveness of each session. Therefore, instructors who taught more than one session were evaluated as many times as they had a session. Participants were instructed to indicate the degree to which they agreed with each statement using a five-point Likert scale (1=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree). The evaluations included the following questions:
Question #1: The instructor knew the subject matter well.
Question #2: The instructor made the subject matter interesting.
Question #3: The instructor encouraged student involvement.
Overall the instructors were ranked highly in all three areas.Students seem to have held the instructors’ subject matter knowledge in high regard, as the 4.85 overall mean for all the instructors was the highest of the three. Statements 2 and 3 were not far off from this mark, scoring a 4.71 and 4.73 respectively. All the scores for the three statements were fairly consistent, and with none of them dropping below a 4 it can be said that the students thought favorably of the instructors. This positive interpretation is further supported when looking at the qualitative data, which is presented next. Overall, these scores were similar to the instructor scores received last year.
Figure 1: Mean Evaluation Scores for Each Instructor for All Three Statements
Qualitative Instructor Feedback
Students were also given the opportunity to comment on the instructor’s effectiveness for each session. This section was open in structure; the students had no set questions to answer and were encouraged to simply make general comments.
The majority of the commentscited the instructor’s organization, subject knowledge, energy, and use of exercises. These positive comments support the high means scores for all instructors which were seen when reviewing the quantitative feedback. Of the negative qualitative feedback that was received, inadequate voice projection and talking directly to the computer screen were most frequently mentioned.
A sampling of comments, separated by instructor, is presented below.