Structuring Reflection as a Tool in Qualitative Evaluation
Carmel M Diezmann and James J Watters
QueenslandUniversity of Technology
AUSTRALIA
Reflective practice is an approach widely adopted by professionals in evaluating their practices. It has been used at all levels of education in particular to judge the quality of teaching. Reflective practice involves teachers’examination of the efficacy of their practices.Various broad processes have been advocated by researchers including reflection-in-action, reflection-on-action and reflection-about action which focus attention on the present, past and future of activities respectively. However, concern has been expressed that practitioners have difficulty stepping out of their pre-existing mind sets to challenge pre-conceived beliefs and to seek evidence that affirms or refutes the assumptions underpinning their practice.In this paper, we undertake “market research” on reflection as a cognitive tool. We identify four features of reflectionasatool: components of reflection, scope of reflection, types of reflection, and reflection through writing. We “test drive” and demonstrate the efficacy ofthis model by applying it to a Graduate Certificate (Higher Education) course. The features of the tool variously contributed to a better understanding of the course. A focus on features of reflection provides a necessary structure and specificity to guide reflective practice in higher education.
Introduction
It is widely accepted that the context of teaching in higher education has significantly changed in the past decade. Calls from industry and the professions have highlighted concerns about the quality of graduates and the extent to which universities are producing the graduates with the necessary skills to meet the needs of a post-industrial information age. Despite well engrained perceptions that teaching in higher education is a burdensome annoyance and distraction from the main game namely research (e.g.,Ramsden, Margetson, Martin, & Clark, 1995), many institutions are confronting the challenge of improving the quality of teaching through targeted professional development strategies. One such initiative is the accreditation of university lecturers through Graduate Certificate courses.The ongoing challenge for those implementing such strategies is to ascertain their effectiveness in a context where data sources are diverse and often difficult to gather.
Reflective practice is one common approach to ascertaining the effectiveness of teaching. In the school sector and higher education, reflective practice has acquired high standing as a process of monitoring effectiveness based on the assumption that practitioners should be thoughtful and analytic about their teaching (e.g.,Amundsen, Saroyan, & Frankman, 1996; Bolton, 2000; Boud & Walker, 1998; Brookfield, 1995; Cranton, 1996, 1998; Hatton & Smith, 1995; McAlpine, Weston, Beauchamp, Wiseman, & Beauchamp, 1999; McAlpine & Weston, 2000; Trigwell, Martin, Benjamin, & Prosser, 2000).Here, we take up the challenge of exploring reflection as a cognitive tool for enhancing teaching effectiveness in higher education. Paralleling practices in evaluating and reporting on the qualities of a physical tool, we undertake “market research” on reflection as a cognitive tool, we “test drive” reflection as a tool to enhance teaching practice in higher education, and finally, we present our conclusions on reflection as a tool for qualitative evaluation.
Market Research: Perspectives and Practices
Perspectives on Reflection
Reflection, as a tool for enhancing understanding, dates back to at least Confucius:
By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest. Confucius (551 BC - 479 BC) [emphasis added]
In the modern era,John Dewey was one of the first writers to focus our attention on reflection and the process of reflective thinking. According to Dewey (1933), “reflection is something that is believed in, not on its own account, but through something else which stands as evidence” (p. 8). He went on to argue “the ability to seize on what is evidential or significant and to let go the rest is the mark of an expert, the connoisseur, the judge, in any matter” (p. 104). That is, being able to let go of preconceived ideas, beliefs or opinions, suspend judgment and accept evidence that challenges an individual’s practice is central to reflection. For Dewey, reflective thinking was about solving problems through controlled thinking. However, he appears to have provided little guidance for channelling such thought into action.
Much later, Schön (1983, 1987) expanded the concept of reflection by identifying two types of reflection, namely,reflection-in-action and reflection-on-action. Reflection in-action was perceived as a spontaneous process in which the practitioner deals with the here and now using experience and intuition. For example, when problems arise during a teaching episode, ateacher spontaneously analyses the situation and responds accordingly. As Dewey had acknowledged earlier, accumulated prior experience, craft knowledge or teacher expertise enables practitioners to reflect spontaneously on events in this way.Reflection on-action is retrospective thinking in which teachers usestructured reflection to analyse their practice. The notion of structured reflection implies the application of some strategies that enable the practitioner to step outside their pre-conceived beliefs and to seek evidence that affirms or refutes theassumptions underpinning their practice.
Schön’s model (1983, 1987) has two limitations. Firstly, there is a lack of forward thinking implicit in Dewey’s writings. Reflection in-action and reflection on-action should go beyond revisiting the past or being metacognitively aware of one’s actions to be proactive and to inform future action (Killion & Todnem, 1991). Reflection-for-action links the reflective thinking process to future action. This theme was taken up by Carr and Kemmis (1986), who, from a critical theorist perspective, saw reflection in terms of action research:
Action research is simply a form of self-reflective enquiry undertaken by participants in social situations in order to improve the rationality and justice of their own practices, their understanding of these practices, and the situations in which the practices are carried out. (p. 162)
Secondly, in Schön’s model (1983, 1987),the discursive or dialogical dimension of thinking and the social context of practice are ignored (Osterman & Kottkamp, 2004; Zeichner & Liston, 1996). The premise that reflection is an individualistic process is at odds with contemporary theory about learning (Lave & Wenger, 1991; Vygotsky,1978) which proposes that learning is social process. Thus,these writers argue that teachers need to reflect by engagement in discourse with colleagues and students in a community so that their ideas and interpretations of actions are subject to discussion, argument and consideration of alternative viewpoints. Reflecting in a community requires practitioners to become critical of the assumptions underlying their intentions, values, beliefs, and feelings (Mezirow, 1997). Mezirow had argued that reflection was a process to change beliefs through transformative learning. He described a person’s experiences and the assumptions which are used to interpret experience as “frames of reference”, which:
Selectively shape and delimit expectations, perceptions, cognition, and feelings. They set our “line of action.” Once set, we automatically move from one specific activity (mental or behavioral) to another. We have a strong tendency to reject ideas that fail to fit our preconceptions, labeling those ideas as unworthy of consideration—aberrations, nonsense, irrelevant, weird, or mistaken. (p. 5)
The nature of these discourses depends on the community and each community creates a specific perspective for interpreting their world (Ovens, 2002).Nevertheless, such conversations within a community generate assertions and counter assertions which challenge practitioners’ accepted views and foster “overcoming the inertia that inclines one to accept suggestions at face value” (Dewey, 1991, p. 13).
These limitations to reflection can be addressed with attention to reflection to guide future action and the role of reflection within the social process of learning. However, turning our attention specifically to education further challenges emerge.
Reflective Practice in Education
Reflective practice in teacher education and other professional training courses has acquired high standing as a process of monitoring effectiveness based on the assumption that practitioners should be thoughtful and analytic about their teaching (e.g., Boud & Walker, 1998; Hatton & Smith, 1995). However, despite these compelling arguments for the benefits of reflective practice, concerns have been expressed about exactly what reflective practice means and how to implement it effectively (Boud, 2006; Clark, 2001; Day, 2000; Kreber, 2004, 2005). To move forward we shift our focus from the purpose of reflection, namely for enhancing our understanding of teaching, to the features of reflection as a tool.
Features of Reflection as a Tool
There are four features of reflection that can variously be used to provide insight into teaching, which is a multi-faceted and complex activity.
The first feature is the components of reflection. Drawing on the work of Mezirow (1991), Kreber and Cranton (2000)proposed that reflection focuses on three components: content, process and premise. Content reflection helps us to describe our beliefs about what we know and should be teaching. It involves examining the actions we take, for example by exploring course design, intended learning outcomes and the instructional materials used to support the course. In essence, we confront our assumptions about the course and what bodies of knowledge are appropriate. By engaging in process reflection, we seek to validate what we do. The focus on the process component is the teaching and learning assumptions underpinning our practice. In essence, how do we engage students as active learners? To obtain evidence of our practice, we collect data from students on how they engaged in the course, and the professional benefits and outcomes for them. Finally, engaging in premise reflection requires us to examine why we teach the way we do by reconceptualising the issues, justifying the approaches taken or suggesting alternatives.
The second feature of is the scope of reflection. Watson and Wilcox’s (2000) framework for reflection developed for the process of reading can be adapted to help practitioners reflect on their experiences. Watson and Wilcox propose that in reading stories the subtleties and complexities of the text can be understood if a reader applies three processes or modes to their reading. Firstly, a “quick reading” provides an initial impression of the range of issues in the text. Secondly, by “zooming in”, we are able to pursue a fine grained analysis of a particular aspect of the issue. The purpose is to ask questions of the text. Finally, a synthesis and evaluation of the text is obtained by a “zooming out” to enable a reconciliation of conjectures and a restructuring of our sense of issues and the relative significance of these issues. Thus, zooming out extends reflection from the local level to the social political context in which a practice occurs(e.g., Carr & Kemmis, 1985).
The third feature is the types of reflection. In a study of effective school principals, Day (2000) identified five types of reflection as contributing to effective leadership: the holistic, pedagogical, interpersonal, strategic and intrapersonal.Within each of these kinds of reflection the principals were able to reflect in, on, about and for action. This categorisation provided a useful structure to describe the work of these people and to foster further reflection by colleagues and researchers examining the study. Additionally, it highlights the multiplicity of reflections that are relevant in any situation at the individual and community levels.
The final feature is reflection through writing. Writing is advocated frequently as a means to stimulate reflection for example through narratives, journals and portfolios. This writing process can range from superficial engagement with ideas and experiences to deep engagement. Campbell-Evans and Maloney (1998) argue that individuals should be encouraged to reflect deeply on ideas by working through writing cycles that progressively involve reporting, reviewing and refocusing, analysing, andreconceptualising, which is the highest level of engagement (See Table 1). Campbell-Evans and Maloney’s levels provide some insight into the advantages of narrative writing. Narrativesallow practitioners to express and explore their own experiences through story telling in ways which help them to understand their practices and develop them further (Bolton, 2000). These levels of writing also indicate why a lack of challenge in some writing tasks, for example in portfolios,can fail to engage individuals in coordinated and critical thinking as envisaged by Dewey and Schön.
Table 1Levels of Writing
Type / DefinitionReporting / Describing; giving a factual recount of critical events, workshops, lectures
Reviewing and refocusing / Considering; making simple suggestions for alternatives; making plans for action; explaining; as in cause and effect; low-level questioning; reworking intentions and outcomes, making plans.
Analysing / Interpreting events, problems or activities; making sense of an activity, situation or event; figuring it out and presenting a reasoned argument or answer; diagnosing a problem; comparing and evaluating; questioning answers and proposing follow-up actions.
Reconceptualising / Reworking views and ideas; drawing upon others’ ideas and elaborating; stating a philosophy or vision of teaching a subject based on evidence, theory or valid experience; developing an image of teaching and teachers; insights into how students learn; stating a purpose for teaching a particular subject.
Based on Campbell-Evans and Maloney (1998).
These four features of reflection should neither be considered asexhaustive nor discrete. However, they should provide some guidance into how to undertake systematic reflection about teaching.
Test Drive: Enhancing Higher Education through Reflection
A “test drive” is integral to the assessment of a physical tool and provides firsthand experience of the tool. As stated earlier, according to Confucius, experience together with reflection and imitation are the three methods of obtaining wisdom. Hence, our next step is to test drive reflection as a tool for enhancing teaching in higher education.
The Context and Data Sources
The course under review is a Graduate Certificate course in Higher Education, which has operated for some ten years at the institution where this study was undertaken. Its main purpose is to provide structured professional development for university academics whose participation is sponsored by the university. Participants in the course are generally early career academics, or sessional staff. They are drawn from a range of faculties including traditional sciences, health, social sciences, business, education and humanities. These students are characterised by their high commitment and enthusiasm but in many cases are confronting multiple pressures of young families, coping with, or seeking new jobs, and institutional performance demands.Thus, they are often loaded with heavy teaching schedules and subject to considerable pressure to perform in research.
The data sources for the evaluation of the course consist of documents, survey responses, written reflections and artefacts. Documents consisted broadly of the literature and specifically of Course documentation. Survey responses were obtained from university stakeholders and course participants. University stakeholders were invited to contribute information related to relevance of the course, its value and impact on teaching and learning in various faculties, and how the Course might be strengthened. In addition, the co-author who was a course participant provided artefacts in the form of a set of assignment materials and related writing.
Testing the Features of Reflection as a Tool for Enhancing Teaching Practice
Reflection as a tool for enhancing understanding about teaching practice in higher education required three critical tests. The first test related to the utility of reflection as a tool for enhancing understanding of teaching because the primary purpose of reflection is insight. The second test checked for trustworthiness. Criticisms of reflection include the uncritical acceptance of learners’ experiences (Boud, 2006; Boud & Walker, 1998) highlight the importance of establishing trustworthiness of a qualitative investigation. The third test focussed on the role of writing in reflection. In higher education, there is a heavy emphasis on various forms of reflective writing.
Test 1. What insight does reflection provide about the quality of the course?
An examination of the course in relation to the components of reflection (i.e., content, process, premise) and the scope of reflection (i.e., quick reading, zooming in, zooming out) provided confidence that the course was generally of high qualitybut also indicated avenues for improvement (See Watters & Diezmann, 2005 for details). Adopting the approach of reflecting on content, process and premise provided an opportunity to describe and analyse the alignment of the course with the expected professional development objectives of the university. In reflecting on content, a holistic perspective was gained by examining the structure of the course and the articulated aims, goals and key concepts being taught through the screening of recommended course readings. In addition, a review of relevant literature on course evaluation provided standards against which we could compare the course content.
In zooming in, we were able to explore what the participants said about the course, its value and how the content equipped them to perform various functions in teaching. This zooming in phase drew heavily on data collected purposefully through open-ended surveys and incidentally through assignment work and unsolicited commentary. In analysing these data, we were cautious to gain extensive understandings of what participants were saying in unison as well as idiosyncratic feedback. Hence, corroborating evidence to support claims was sought and dissenting voices noted. In zooming out general conjectures were postulated and their applicability tested.
In reflecting on process, we noted several features emerged from a quick read of structures and modes of delivery. The process dimension explored modes of delivery, purposes and intended outcomes. Data informing a more focussed analysis include survey information on the operation of the course and reflections of afocus participant who was one of the authors (CMD). This process revealed limitations with the course delivery and issues around the use of technology to support learning. In zooming out, we identified the extent to which this course achieved its goals by the faithfulness of the implementation. Although alignment with institutional goals appears to be acknowledged, issues emerged with the implementation, which were addressed in subsequent implementations of the course.
Finally, in reflecting on premises we examined the assumption that professional development through a formalised Graduate Certificate taught by academic staff is necessary to ensure quality of educational experiences for tertiary students. Scanning the literature the evidence in support of this assumption was equivocal. Zooming in on our assumptions led us to believe the model of basic teacher education is effective. The course is providing a comprehensive and acceptable avenue for staff to engage in gaining qualifications in teaching in higher education. The course addresses praxis, that is, it integrates theory and practice in ways that accommodate individual disciplinary styles. Emerging from the qualitative data is an endorsement of the effectiveness and value of a course of this nature for the professionalisation of teaching. Zooming out provided a broad perspective, which endorsed the view that with guidance or support structures academic staff in non-education faculties will engage in the discourse of education.