Van Houten/Hilton/Rantz Protocol

1

Van Houten/Hilton/Rantz Protocol

Western Michigan University

Human Subjects Institutional Review Board (HSIRB)

Van Houten/Hilton/Rantz Research Protocol

Abstract

The airplane checklist is a job aid that is used to assist pilots in completing important functions before, during and after flight. This study will compare the accuracy of completing checklists when using an analog (paper) checklist vs. a simulated audible checklist using an alternating treatments design and whether both types of checklists performance is improved by receiving graphic feedback using a multiple baseline across participants design. The participants will be three college students recruited from junior and senior level aviation courses at Western Michigan University’s (WMU) College of Aviation (COA.) The experimental task will consist of flying six different, randomly selected flight patterns. The participants will be using the WMU Department of Psychology Personal Computer-Aviation Training Device (PC-ATD.) During the simulated flights participants will be informed to complete either analog checklists or activate a simulated audible checklist. Two secondary dependent variables will be (a) the percentage of total errors for each flight segments during each experimental phase per participant and (b) the percentage of items omitted. The experimental design will be a multi-element design with a reversal. During the baseline phase, after each simulated flight, the participant will receive post-flight technical feedback on how well they flew the pattern. During the first intervention phase the participants will fly one of six of the designated pattern and the participants will be asked to press an after-market button on the instrument panel of the PC-ATD when the pilot wants to initiate a checklist during a flight segment. The accuracy and completeness of the checklists will be graded by a trained observer. The second intervention will only be applied if participants fail to achieve 60% or higher correct of the total checklist items. Van Houten and Rantz (2009) concluded that the provision of graphic feedback on incorrect items could bring correct checklist items to above 90% consistently. This second intervention will be used to assure that the pilots leave the experiment with safer behaviors than when they arrived - if the audible checklist does not function as the experimenter predicts. If needed the second intervention phase will consist of the participants being asked to fly one of the six designated patterns and to do the simulated audible checklist as they would in the previous phase of the experiment. As the participants are flying an observer will grade the accuracy and completeness of the checklist and graphic feedback will be created to show the participant immediately after their simulated flight. The participant will be notified of specific correct, incorrect, and omitted items along with the graphic feedback and praise will be given to any participant with an increase in checklist performance. The technical feedback of the flight pattern will also be given following the checklist graphic. The reversal will be included at the end of the intervention phase(s) and the participants will be asked to fly the PC-ATD simulator and to do the checklists as they would in a regular flight. In this return to baseline condition there will be no audible checklists and also no checklist feedback or praise. The post-flight technical feedback will still follow each flight. Participants will attend 5 to 10 two hours sessions, which will include an introductory flight. These sessions will occur over a two to three week period.

PURPOSE

The purpose of this study is to assess whether there is a significant difference in checklist accuracy within pilots, who use either a simulated audible checklist vs. a commonly used analog checklist. If the participants engage in better checklist use of when using the audible checklist as it would suggest that pilots should use audible checklists during flight. If applicable, if the pilots show a significant increase in checklist accuracy following the graphic feedback/praise (introduction which follows the audible checklist intervention) it would suggest that both should be used to increase the pilot checklist use.

BACKGROUND

Airplane checklists are used during different segments of flight to sequence specific, critical tasks and aircraft adjustments that correspond to specific environmental demands (Degani & Wiener, 1990). These checklists include items for such segments of a flight as before take off, climb, cruise, descent, before landing, and after landing. Each checklist has specific tasks to the segment, as well as points that work as continued checks on the airplane’s configuration in flight. The complexity of these checklists cannot be overstated. For example, on some checklists, the “before engine start” sub-section has 76 items for the first flight of the day, and 37 items for subsequent flight segments (Degani & Wiener). Though many checklists designed by different aircraft companies have similar items, very few are identical. Even different model planes made by the same manufacturer may have different checklists that pertain to different options for those planes. Even though the differences exist, Checklists have become the main strategy to standardize pilot performance and increase flight deck safety (Van Houten & Rantz, 2009.) Thus, it is not surprising that many aviation experts have addressed their importance and design, as well as the practices and policies that surround their use (Adamski & Stahl, 1997; Degani, 1992, 2002; Degani & Wiener 1990; Federal Aviation Administration [FAA], 1995, 2000; Gross 1995; Turner, 2001;Van Houten & Rantz, 2009). Even so, the incorrect use of flight checklists is still often cited as the probable cause or a contributing factor to a large number of accidents (Degani, 1992, 2002; Degani & Wiener; Diez, Boehm-Davis, & Holt, 2003; Turner). ). Similarly, many investigations by the National Transportation Safety Board (NTSB) have revealed that the aircraft were not properly configured for flight, which usually results from improper checklist use (NTSB, 1969, 1975, 1982, 1988a, 1988b, 1989, 1990, 1997).

Studies by Lautmann and Gallimore (1987) and Helmreich, Wilhelm, Klinect, and Merritt (2001) provide more direct evidence of improper checklist use by flight crews(Van Houten and Rantz, 2009). In a study funded by Boeing, Lautmann and Gallimore ran a survey of twelve airlines and compiled the data, which showed that errors involved with using the checklist contributed to a substantial occurrence of accidents and incidents. Helmreich et al. conducted a series of studies sponsored by the National Aeronautics and Space Administration to identify the particular errors flight crews commit. Crews were observed while flying. Errors were recorded using the Line Oriented Safety Audit (LOSA) developed by Helmreich and his colleagues (Helmreich, in press; Helmreich, Klinect, Wilhelm, & Jones, 1999; Helmreich et al., 2001). Between 1997 and 1998, LOSAs were conducted at three airlines with 184 flight crews on 314 flight segments (Helmreich et al., 2001)(Van Houten and Rantz, 2009).From this study the possible errors were broken down into five categories. Checklist errors fell into the category of “Rule-Compliance” errors. The category of Rule-compliance errors had the highest frequency of errors at 54% of all errors recorded. Checklist errors accounted for the highest number of errors in that category.

Excerpt From Van Houten and Rantz, 2009. “Despite widespread recognition that checklist errors occurred relatively frequently and were major contributing factors to many accidents, the design of checklists “escaped the scrutiny of the human factors profession” until the 1990s (Degani & Wiener, 1993, p. 28). In 1988, Wiener, when testifying at a NTSB investigative hearing for a fatal crash, stated that he knew of no research on how a checklist should be designed, a fact that he and Degani confirmed by an extensive literature search of U.S. and European databases (Degani & Wiener, 1990). Degani and Wiener (1990, 1993) began their work on checklist design shortly thereafter. They observed flight crews while flying, interviewed flight crews from seven major U.S. airlines, and analyzed how the design of checklists contributed to aircraft accidents and incidents that were reported in three aviation databases. Their analytic guidelines became the industry standard in 1995 (Patterson, Render, & Ebright, 2002).”

Excerpt From Van Houten and Rantz, 2009. “Although Degani and Wiener (1990, 1993) did not pursue the behavioral factors that influence checklist use, they recognized their importance, indicating that behavioral and psychological issues were core problems that led some pilots to misuse the checklist or not use it at all. Regardless, an extensive search of the aviation checklist literature did not reveal any studies that have examined whether behavioral interventions could increase the appropriate use of flight checklists.”

Digani and Weiner (1990) mention the creation and description of a computerized device which gives audio prompts of checklist items but, they list no studies that have been done until that point which compared the use of the audio checklist with any other type of checklist. A review of the literature reveal no comparison studies or behavioral studies of the efficacy of the audio checklist in aviation. Despite the lack of studies there have been several audio checklist devices patented for the use in aviation applications. One in 1990 by Harshaw, Burkey, Doell, and Keith and another similar device was patented by O’Rourke in 2001.

Palmer and Degani (1991) indicate that computerized devices that sense completion for the pilot have increased checklist errors. In this study pilots would tend to trust the technology each time, a) detected all the mis-configured items when using the paper checklist, b) detected some of the manual-sensed items displayed by the electronic checklist and c) did not detect any of the mis-configured item when using the automatic-sense electronic checklist.

(Excerpt from Van Houten Rantz, 2009) “In organizational settings, checklists have been part of package interventions used in a variety of settings (i.e., manufacturing, hotels, banks, offices, retail establishments, and restaurants) to improve a diverse array of performances, including cleaning and housekeeping tasks (Altus, Welsh, & Miller, 1991; Anderson, Crowell, Hantula, & Siroky, 1988; Anderson, Crowell, Sponsel, Clarke, & Brence, 1982), office tasks (Bacon, Fulton, & Malott, 1982), banquet set-up times (LaFleur & Hyten, 1995), machine set-up time, (Wittkopp, Rowan, & Poling, 1990), metal yield (Moses, Stahelski, & Knapp, 2000), end-of-shift closing tasks (Austin, Weatherly, & Gravina, 2005), staff-client contact time (Porterfield, Evans, & Blunden, 1985), and customer service (Crowell, Anderson, Abel, & Sergio, 1988). Interestingly, none of the studies monitored whether or not employees actually used the checklists. This may be because the checklists were used to inform employees what they were supposed to be doing and thus were considered to be a necessary part of the intervention, but were not viewed as the important motivating variables. Rather, the other independent variables that were implemented along with the checklists (i.e., feedback, goals, and rewards) were viewed as the important motivating variables. It is also the case that in all of these studies, the checklists were used as an independent variable, not a dependent variable. Similar to our search of the aviation literature, we were not able to find any study that examined how to increase the use of a checklist in this literature base.”

Despite the lack of research in the aviation field there have been studies done in other fields which indicate that the use of audio prompting checklists can increase performance in a variety of situations. Davies, Stock, and Wehmeyer (2002) increased the performance and completion of vocational activities by individuals labeled with mental retardation by the use of a palm top computer which would give an audible checklist where items needed to be checked off after they were completed. In this study the use of the palm computer reduces the number of errors per activity from 2.25 to 0.75.

The current study will examine whether pilots will increase the accuracy of analog checklists or digital audio prompted checklists and then if applicable, whether graphic feedback will increase accuracy beyond the level obtained with the sole use of the audio prompted checklist. The presentation of the graphic feedback to improve checklist accuracy in Van Houten and Rantz (2009) has established the strong effect of the feedback in a simulated flight environment. If needed this step will further be used to assure the accurate performance of checklists in the subject pilots prior to their release from the study.

Subject Recruitment

Bryan Hilton will explain his role as a graduate student researcher and his previous role in the WMU College of Aviation. Bryan Hilton will recruit subjects from junior and senior level aviation classes as the Western Michigan University College of Aviation. The recruitment script is provided in Appendix A. Bryan Hilton will ask any student who is interested in more information about the study to print their name, phone number, and email address on a piece of paper and give it to him. Each student will be informed to write their information on separate pieces of paper to ensure confidentiality of the information provided. All documents will inform the students that participation is voluntary and that withdrawal will not effect them in any way. He will also write him name, phone number, and e-mail address on the blackboard/white board of the class room that he will be presenting in and inform the students to contact him if they prefer to obtain additional information concerning the study. Bryan Hilton will contact the individuals who have given contact information in two to three days to arrange a meeting in which to discuss the details of the study and the consent document.

In addition the recruiting students in aviation courses at WMU, he will also hang recruitment flyers at WMU’s Aviation Education Center in Battle Creek, Michigan and the Aviation Department in Korhman Hall on WMU’s main campus. The recruitment flyer is provided in Appendix B. If individuals contact the experimenter as a result of the flyer, the information from the flyer will be repeated and a meeting will be arranged to go over the details of the study and the consent form.

Informed Consent Process

Bryan Hilton will meet individually with potential participants at the College of Aviation in Battle Creek, Michigan or at the Aviation Advising Office in Korhman Hall on WMU’s main campus. The consent form will be presented and explained to each potential participant individually and privately and then the experimenter will ask the student to read through the document and ask any questions that they might have regarding the form. The experimenter will answer any questions of concern. If the student agrees to participate in the study, Bryan Hilton will ask the student to sign the form and a copy will be made for the student and one will be retained for the researcher. (The consent document can be found in Appendix C.) If the student declines to participate, Bryan Hilton will thank them for their time.

After potential participants sign the consent document, Bryan Hilton will describe the flight pattern participants will be asked to fly, noting the instrument landing requirement. He will then ask them to complete the eligibility questionnaire(Appendix K). Finally, Bryan Hilton will determine whether the potential participant can attend at least two two-hour sessions per week.

If potential participants do not meet one or more of the above criteria, Bryan Hilton will explain which requirement was not met and why. They will be thanked for their time and dismissed.

The first five potential participants who meet the above criteria will be scheduled for their first experimental session. There will be one exclusionary criterion. If a potential participant completes 36 or more of the 40 checklist items correctly during any one of the four flights during his or her first session, the participant will be excluded from further participation. This is necessary because the purpose of the study is to determine whether audio prompted checklists and performance feedback can increase the extent to which pilots use a checklist accurately. If a participant already performs at very high levels, improvement is not possible.

Before Bryan Hilton schedules the first session with participants, he will remind them (as stated in the informed consent form) that there is a possibility that they will be eliminated from the study after the first session based on an assessment of their flight performance. At the end of the first session, Bryan Hilton will tell participants whether or not they are eligible to continue. Participants who are eliminated from the study will be told that their performance was so high that it could not be improved much. Participants who are eligible to continue will be told that their flight performance qualifies them for continuation in the study. Bryan Hilton will then schedule subsequent sessions for them.

If a person is eliminated after his or her first session, another individual who expressed interest in the study will be contacted and scheduled for an introductory meeting. Once five participants have been selected based on their performance during the first session, the remaining individuals who were interested in learning more about the study will be contacted, thanked for their interest, and told that participants have been selected for the study.