The Use of Ethnographic Observation in Evaluating Training Methods in Five Navies

1

The Use of Ethnographic Observation in Evaluating Training Methods in Five Navies

Steven Wilkinson

Centre for Organisational Research, Anglia Polytechnic University

Paper presented at the British Educational Research Association Annual Conference, Cardiff University, September 7-10 2000.

Introduction

Naval training communities readily accept that a systems approach to training best meets their training needs. Improvement in training outcomes, however, can be achieved by reviewing the application of this approach. Where previous efforts in quality improvement have focused on the Training System, it is argued that since the personnel tasked with implementing naval training are naval instructors, the focus of research should be on the work of the naval instructor and on the question: ‘What is 'effectiveness' in Naval Training?’

While the concept of a 'systems approach' to training may provide a solution for the organisation of training, it is in the interpretation and implementation of the training system that effectiveness becomes an issue.

This paper discusses an observation based ethnographic approach to researching naval instructor effectiveness. It is fashioned from a doctoral thesis undertaken during 1994 – 1998 titled Naval Instruction; A Comparative Research Study. (Wilkinson S. 1998).

The Model

I was, by definition, a practitioner researcher when I conducted a study into Naval Instructor Effectiveness. I was curious about the interaction between the systems approach to training and the Instructors who were tasked with delivering training using this model. The issue of effective learning seemed to be bound within a training system model, a military culture and a range of espoused theories held by individuals and nurtured by their experiences. It was thought that an understanding of the constraints on effectiveness may help naval training organisations achieve more from their training. In the words of Schön, I was interested '. . . . in the kind of knowing in which competent practitioners engage.' (1991)

There were significant influences acting upon the selection of a research model. The vast majority of people within a navy are predominantly technological in their thinking. As people in the navy were the audience for this report, their preference for quantitative data was not to be ignored. On the other hand, as a naval trainer myself, I was aware of the limitations of a quantitative approach in this context. My views aligned with Sanger, who suggests

The vast majority of teachers know that classrooms are both unpredictable and idiosyncratic. . . . . They know that as participant observers they realise the complexity better than any outsider.' (1996)

Whilst not proposing to cross the qualitative-quantitative divide (Mason, 1996), it was necessary to devise an approach that spoke to all audiences. Noting that the two approaches did not necessarily harmonise, a research approach was devised that employed the use of a survey and then progressed into an interview and observation phase.

A sample of 30 naval instructors from each of five navies was surveyed. These were the Royal Australian Navy, the Royal New Zealand Navy, the United States Navy, the Canadian Forces - Navy and the Royal Navy. The survey data provided a comparison of perspectives across six related issues of concern. These six issues were:

1.Systems Approach - How does naval instructor training contribute to an instructor’s effectiveness?

2.Individual Virtues - Which virtues are exhibited by effective instructors?

3.Instructional Methodologies - Are the instructional methodologies taught and applied within the navies appropriate and/or effective?

4.Adult Learning - What evidence is there of adult learning approaches being applied by naval instructors?

5.Climate of Change - What are the influences effecting change within navies that impact on instructor effectiveness?

6.Military Culture - What are the cultural influences on instructor effectiveness?

Additional to the survey, this research examined the training which instructors received in training technology. This was done by observation. I attended each of the five navy's Instructional Technique courses. Two clear models of Instructor Training were evident. These can simply be polarised as Passive and Interactive. The survey data indicated that instructors who were trained in the Interactive instructional model were more satisfied with their training. Observation of instructors in their classrooms, however, revealed that they did not necessarily employ this model in practice. There was in fact a significant difference between their espoused theory and their theory in action (Argyris & Schön, 1974)

The sample of 30 naval instructors from each of the navies were interviewed and then observed while instructing. Each instructor was observed for at least one day. These data were recorded in the form of case examples (or vignettes), observations and quotations. This provided the opportunity to discuss issues concerned with instructor effectiveness at a different level to that of the survey. With this data, the research had taken on a constructivist or interpretivist dimension (Schwandt, 1998) where the researcher believes that to understand this world of meaning one must interpret it. Observation also provided a glimpse of the complex world of the naval instructor. Significantly, this data provided examples of effective and non-effective instruction.

The Case for the Use of Observation

Observation provides a powerful tool in contemporary qualitative research. It can be written to project the background of an observation to the foreground and raise the significance of an incident to the attention of the reader (Sanger 1996, Delamont 1992).

Observation was once a staple of the qualitative research diet. Going back to what has been called the traditional period in qualitative research in the twentieth century (the early 1900’s), ethnography was thought of as an adventure into a different culture to gather accounts which then stood as monuments to, and records of, that culture. The accounts provided a 'slice of life' approach (Denzin and Lincoln, 1998) to ethnography which confront the reader with evidence of a highly personalised nature. A word picture of the events occurring in a given space in time provides fuel for discussion and a starting point for analysis. The modernist phase in qualitative research (post war to 1970’s) gave voice to societies' underclasses. Contemporary qualitative research (late 1980’s to today) does not seek to produce global generalisations or grand narratives. Recent thinking emphasises small-scale theories fitted to specific problems and specific situations. (Lincoln, 1993)

Although the use of observation in qualitative research has evolved, it has not at any stage become redundant. There should be no question about why it is used, however, plenty can be asked about why it is not used more extensively. The tension created by any debate as to whether or not this methodology is science distracts from the more significant question 'what is truth ?'.

For an organisation such as a navy to be prepared to participate in such a bold and uncharacteristically qualitative approach was indeed unique. Observation is part of the naval instructor appraisal system, however, in most cases the observation is constrained by a formalised checklist and conducted over a short space of time. By contrast, this project undertook extensive observation of instructors (over 1000 hours) and was not constrained by the context of appraisal.

Validation of this Study

A significant difference between observing and simply recounting incidents, is that the observer takes steps to ensure validity. To validate this research the following measures were employed:

a.Global generalisations were mostly avoided as it was deemed difficult to draw such generalisations from relatively small, isolated and specific samples via the methodologies employed.

b.Observations and discussion of data were shared with academic staff, training officers, supervisors and other significant interested parties prior to publication with a view to enhancing objectivity by identifying occurrences of unconscious bias, conscious prejudice, incompetence or gullibility.

c.Every attempt was made during the data gathering process to reduce distortions resulting from being present at the research sites, involvement with the subjects, biases and from the manner in which data gathering techniques were employed.
Characteristics of Effective Instructors

Background reading provided a list of characteristics of effective instructors. This was developed to generate a focal point for the observation. The list was mostly borrowed from Rae (1988) which was in turn derived from a list by Burgoyne and Stuart (1976). Further input was drawn from Fawns and Nance (1993), Dunkin and Precians (1993), Dortcamp (1996) and from the Australian National Project for the Quality of Teaching and Learning ‘Competencies for Beginning Teachers’ (1996). This list of knowledge, skills and virtues was modified in response to what is known about naval instructor training and supplemented by personal experience. This list is included as an Appendix.

Case Examples

Case examples were generated during the observation of 150 instructors, their trainees and many training managers from across the five navies. Each instructor was observed for at least one day. In some cases instructors were observed for longer.

There is no attempt here to identify from where exactly these case examples originated. They represent views, trends, situations or cultural idiosyncrasies drawn from a range of instructional settings and personal exchanges. Although there were subtle differences among the locations visited during this research, the evidence reported should be viewed as being representative of all five navies.

The following set of case examples are presented here to illustrate the application of this research approach. Each example is accompanied by a discussion related to the issues raised in each.

1 - The Right Personality

While on course a trainee took the opportunity to help other trainees by mentoring and assisting with revision. This did not go unnoticed by course staff. On the completion of the course the trainee was advised that he would be instructing the next one. The prerequisite for his selection was the simple fact that he had completed the course himself. He also believed that the staff of the school at the time felt he had the right personality. When asked if he would undergo any instructor training his response was,

Yeah, I will do a basic instructor course but I don’t expect it will do much for me. It's just a tick in the box. You do so many courses in the navy. They all sort of add together, you know. . . . build on each other.

When asked if he thought the position would help his career he explained,

This job gives you rope. If you do well you will climb it; if you do badly, you will hang yourself and your career will be dead in the water.

The selection process for instructors varied among naval settings depending on availability of personnel. Ideally, an instructor would be someone who is experienced, a volunteer and motivated towards the job. This ideal combination is difficult to achieve. Personnel often did not know why they were selected as instructors. Certainly, this was the case with the instructor in the above case study. Personality seems as good a selection criterion as any other for those making the decisions. The United States Navy (USN) had a formal process for the selection of instructors, although most USN instructors stated that they were posted to their billets and not selected. Most of the senior sailors interviewed in this research accepted an instructional posting as inevitable. There were isolated situations in all five navies where instructors were nominated within a clearly defined selection process and competition for those positions was strong. In these instances there was always an advantage in being selected as an instructor. Advantages ranged from a perceived or real advantage on the promotion roster to the opportunity to convert or upgrade qualifications. Having said this, competitive selection was rare. Most often instructors were posted into instructional positions. Apart from undertaking instructor training, instructors often received a period of on the job training as the following case study demonstrates.

2 - Follow Me!

The instructor stood behind a large wooden lectern at the front of the classroom. Trainees were seated in rows sitting at large desks in pairs. The instructor quizzed the trainees on information previously learned. The lesson was a test review. Trainees were searching through their books and notes for the answers as the instructor asked each question.

At the back of the room, a new instructor sat alone at a large desk. He had before him an instructor guide and a set of reference books. He, too, was searching through the material for the answers to the questions. He was also taking notes on his copy of the instructor's guide. These notes were recording in detail the way the quiz was being conducted, the time taken to cover teaching points and even some phrases used by the instructor.

An attempt to standardise lesson content led to an attempt to standardise instructional processes. The above case study demonstrates how standardisation was covertly achieved. In many situations an instructor would sit through an entire course with the intention of learning the course content while also absorbing the instructional process.

The military model of instructor-centred delivery was established at the training development stage of the training system as the following example demonstrates.

3 - Experience or Delivery

During a Training Development Course the question was asked,

How do we get knowledge across?

The trainees on this course all had considerable instructional experience. After a short pause one of them offered the cautious response,

Through experience?

The instructor smiled knowingly and said,

Yes, but, how do we get it across initially?

The trainees began to look at each other as if they perhaps misunderstood the question. After a short while the instructor, answering his own question, said,

Through Instructor-centred delivery!!

The trainees accepted this view even though it did not necessarily agree with their own. This could have been because it was presented to them by an authority. It could also have been the case that they interpreted the answer in the context of the role of the Training Developer. The case example illustrates how the implementation of the training system determines the approach taken by instructors.

Once in an instructional position, it was rare that instructors received any more training in their specialisation than to the level to which they were required to train others. Ideally, they were required to train at one level below their own qualification. Realistically, this was not always possible. Large numbers of instructors were observed who were training at the level of their own qualification and quite often instructors were training in areas outside their own qualifications and experience.

It was observed that technical training in particular was documentation dependent. The fundamental knowledge or theory of electrical, mechanical and aviation engineering was often written into course documentation, texts and references with a high degree of technical complexity. Technical language was studded with jargon. Navies remain dependent on large numbers of technicians and technical training comprised the majority of all naval courses. It was often observed that technical instructors were inclined to read to their classes directly from texts.

4 - Reading Aloud

A technical instructor entered the classroom. The three trainees were sitting at desks in a single row, one behind the other along the right hand wall facing towards the front of the classroom. The instructor began the lesson with a brief introduction on what they would be covering that day. He then said,

Open your books to page twenty-seven.

Glancing across the room the instructor was satisfied that all three had turned to the page and were ready. A couple of the trainees shuffled in their seats and adjusted their posture. One leant on his elbows, another pushed his chair back on to the rear two legs and balanced the book on the front edge of the desk. The instructor began to read from the text. His voice was even, tone flat and volume audible without emphasis or exaggeration. He continued reading for an hour, pausing only to look up and reassured himself that the class was following him in their own texts. He continued. After the second hour had passed the instructor stopped reading and said,

Time for some fresh air and a something to drink if you need it.

Slowly the trainees stood up pausing to stretch their muscles and adjust their focus. They left the room. The trainees returned some twenty minutes later and moved back into their seats. One stretched, another swapped his chair with a spare one that was a little more comfortable. The instructor silently waited until the trainees were all seated and looking at their references. He then continued reading where he had left off prior to the break. The reading continued without a break for the next hour and forty minutes.

During this time there were two occasions when the instructor felt like having a cigarette. He reached into his pocket and pulled out his tobacco pouch and rolled himself a cigarette with one hand while holding the book in the other and continuing to read aloud. He would then walk over to the open door, take his cigarette lighter out of his pocket and pause long enough to inhale and light his cigarette. From that point on, his reading was interrupted by the occasional pause while he drew back on his cigarette and blew the smoke out of the open door.