Examination and learning Anton Havnes
Examination and learning An activity-theoretical analysis of the relationship between assessment and learning
Paper presented at the Learning Communities and Assessment Cultures Conference organised by the EARLI Special Interest Group on Assessment and Evaluation, University of Northumbria, 28-30 August 2002
Anton Havnes
Centre for Staff and Learning Development
Oslo University College
Introduction
Improving student learning implies improving the assessment system. Teachers might assume that it is their teaching that directs the students’ learning, but students orient themselves as learners in relation to what will be assessed. In spite of the underlying idea that assessment should serve to test what students have learned during a course of studying, it tends to define what is worth learning. This view, which basically says that assessment drives learning, and that assessment “overrules” teaching, has been advocated by many researchers in the field of learning and assessment over a longer period of time (e.g. Elton and Laurillard, 1979; Crooks, 1988: Biggs, 1996; Shepard, 2000). This can be seen as a paradox and the metaphor often used is that of the tail wagging the dog.
The fact that students attune their learning practice according to the demands of examinations might well be trivial. It is a way of coping with the educational system. The significance of this “backwash effect” of assessment is widely acknowledged. But the impact of assessment on how teachers teach might be even more significant than its impact on how students learn, particularly if we take into regard the impact of teaching on student learning (Trigwell and Prosser, 1999?). In this paper the core issue is the impact of assessment on learning. But I will try to widen the scope of the discourse from an assessment-learning relationship to an assessment-education-learning relationship. In stead of setting up assessment and learning as the only variables, I will also look for how assessment pushes other aspects of the educational process; particularly teaching, production of textbooks and learning material, and, more generally, the design of the learning environment. Consequently, the unit of analysis expands from the level of individual learning to the level of educational programmes and the institutional structures under which students learn, teachers teach and educational administrators work. At this level of analysis assessment is an integral part of the process of education. We are not dealing with a one-directional impact from assessment to learning and teaching, but rather an interrelatedness between assessment and the various aspects of education. This is a more complex phenomenon than the question of how individual students orient themselves as learners and requires a theoretical basis for analysis that can account for systemic complexity and dynamics. In this paper I also hope to illustrate how socio-cultural (or cultural historical) activity theory can serve as a tool for coming to terms with the interrelatedness between assessment and learning.
I shall focus on these questions by discussing the relationship between assessment and the educational process in a specific educational context in Norwegian higher education. The analysis is based on an empirical study and with students as the main informants. My starting point was an interest in understanding student learning from the perspective of students. The empirical material is from Exam Philosophicum (ExPhil) at the University of Oslo (Havnes, 1996 and 1997)[1]. The ExPhil course was a one-semester programme and was meant to serve as an introduction to scientific reasoning and studying at the university. My main argument is that the assessment structure contributed to establish a learning context that worked contrary to the declared content of the study. These observations will be accounted for and discussed on the background of activity theory.
The dominant patterns of educational practices differ from one country to another, from one institution to another, and even from one educational programme to another. In this paper I address one specific programme an one Norwegian university and do not claim that this analysis accounts for higher education in general or in Norway. One the other hand, it is relevant to talk about “the Norwegian tradition” and even of some general characteristics of higher education. It is important to approach the analysis at the level of the singular, but look for general patterns of a wider socio-cultural tradition. OECD has criticised Norwegian higher education for investing a disproportionate amount of resources in an assessment system that is not productive with respect to learning. Universities are characterised as “research institutions conducting examinations”. The strong emphasis that has been on summative assessment will come through in this analysis. Since the time this study was conducted assessment has become an issue in the public debate in Norway and there are now strong incentives from national authorities to emphasise learning, feedback and formative assessment. But the rationale underlying “the old tradition” is still strong and the danger of changes implemented being “old wine on new bottles” is absolutely there.
I will not discuss the impact of traditions extensively here, just point at three aspects. (1) There are mechanisms inherent in educational programmes that sustain a specific examination system and corresponding assessment practices (formative as well as summative) over long periods of time. These mechanisms can be seen as artefacts that represent condensed versions of the experience of generations of scholars in a particular field. Examples of such can be national, profession-specific or institutional regulations, practical arrangements for examinations, feedback and supervision, and other arrangements that sustain specific institutionally situated practices. (2) These artefacts represent stumbling blocks for innovation and thus secure continuation and stability. (3) The application of such artefacts restricts our thinking by implicitly defining something as obvious and mandatory, excluding alternatives that could be taken into considerations (Lauvaas et al, 2001). In the analysis I will account for how a wide set of artefacts closely linked to the students’ learning practice (e.g. textbooks, information material, exercises, previous exam questions, organisational structures and feedback structures) formed the educational practice of the Exphil course. Changing educational practice is not only changing how we think about teaching, learning and assessment, but also – and even more – about changing the artefacts that we rely upon in our daily practices as teachers and students.
Method
The ExPhil study is an ethnographic case study of 7 students, who were followed throughout a complete term. In addition to interviewing them several times during the semester – and after the exam – I also joined in on lectures and participated in seminars and colloquium meetings. My role was that of an active observer attempting to take part as authentically as possible in the activity of the students, reading the literature and working on the same study tasks as the students. I systematically made field notes about what took place in concrete teaching and learning situations. I also collected written material from and about the course of study. The project is thus based on data from various sources: written material describing the course of study; the literature; the sets of exercises that the students worked on, and examination tasks. My own experience as a participant observer is yet another set of data. The major part of the data is the students’ stories about their learning as they were told to me in interviews and conversation.
Data from interviews and observations have been coded to identify patterns in the students’ descriptions of their learning, the content of teaching and learning, the learning situation, and the students’ participation in the educational programme. The actions and utterances of individuals have been analysed with respect to meaning expressed about the educational programme. The analysis has focused on patterns of participation and the interrelatedness between these patterns and contextual conditions for learning. Thus it is not the participation of individual students that has been the focus of interest, but modes of participation as social practice in a particular context. Individual actions have been analysed as a forms of individualisation and particularisation of collective practice.
The epistemological basis for putting the collective level in the foreground is associated with situated learning and socio-cultural activity theory (Vygotsky, 1978; Engeström, 1987; Lave and Wenger, 1991; Cole, 1996; Engeström and Miettinen, 1999; Säljö, 2000). Engeström and Miettinen, for instance, argue that collective and historical aspects of the cultural context are integral to the actions conducted by individual agents. To understand learning (and teaching) practices we have to go beyond the individuals and look at the cultural systems and mechanisms that sustain some action patterns rather than others.
"Individuals act in collective practices, communities, and institutions. Such collective practices are not reducible to sums of individual action; they require theoretical conceptualization in their own right. When individual action is the privileged unit of analysis, collective practice can only be added on as a more or less external envelope" (ibid., p. 11).
Educational programmes are part of cultural systems and what goes on there is not reducible to a sum of individual actions. Cultural systems have “...cyclic rhythms and long historical half-lives" (Engeström et al, 1995). As we shall see, collective practices can be heterogeneous and contradictory. The tensions in the context of learning are not external to or separate from the learning. The study of learning from the perspective of activity theory implies searching for tensions in the learning context and mechanisms by which these tensions make their way into the process of learning as a social phenomenon.
This approach bears similarities with the hidden curriculum approach (Jackson 1968). A distinction is made between the intended, the taught, and the learned curriculum (Cuban, 1992). The so-called hidden curriculum is identified by going beyond the intended curriculum and taking the content of teaching and learning as the primary basis for understanding the educational process. The hidden curriculum is identified as something different from the intended curriculum. From the perspective of activity theory the complexity that emerges from the wide scope of practices (and corresponding curricula) involved is problematised. The focus is not so much on different categories of educational practices and phenomena, but rather on different levels of functioning that data and observations can be analysed on. The smallest possible unit of analysis is the activity system, not specific categories of practices within it.
Activity theory
Unit of analysis
In the socio-cultural or cultural-historical tradition, learning is viewed as social practice situated in a specific historical and socio-cultural context. Rather than emphasising individual differences, the focus is on mechanisms that make people act in similar ways within a given context. The emphasis is thus on action patterns typical of specific social contexts, that is, ways of acting that tend to be consistent across individuals and over time. The core location of the particular practice is in an organisation, a context or a social structure, rather than in individuals. It is common to raise one’s hand before speaking in a class setting or at a meeting, but rarely in a pub or another informal situation – not because the people are different, but the context is. We act contextually meaningfully (or not). The unit of analysis, then, is the person-in-the-situation, not the person as a separate entity.
Becoming a student involves adapting to, and to a greater or lesser degree accepting, certain institutional demands. Such demands are rooted in historical practice and institutional constraints and affordances, regardless of the individual person’s presence or absence. They are not to be seen as the student’s assumptions about the course of study but as affordances of the ambient environment (Gibson, 1979)[2]. In order to understand how students tackle their learning situation, I intend to examine the institutional influences inherent in the two educational programmes. It implies investigating student learning by turning the attention in the same direction as the students themselves have their primary focus; towards the surrounding environment, what it affords and demands.
Learning is also an individual action and as such is influenced by the person’s background, experience and habits. In social situations a state of tension might arise between the person’s individual skills, interests and wishes, on the one hand, and institutional demands, on the other. In this field of tension a dynamic in social practice is created. The situation is both given and at the same time created – or interpreted – by the agent. Also, there are opportunities to challenge the system. We are more or less able to help determine the social context in which we are involved. The extent to which we help create our own situation is both socially bound and dependent upon how we approach it. This line of reflection puts the notion of participation in the foreground, which means that the scope of our intention simultaneously goes in two directions; toward the context and toward the participants. Neither can be understood independently. This is a fundamental ideological basis in activity theory.
Levels of human practice
In activity theory (Vygotsky 1978, Engeström 1987) one differentiates between operation, action and activity. The simplest level of understanding of human behaviour is that of operations; routine activities which we carry out without any conscious intention. At some stage they might have been conscious, but not any more. An action, on the other hand, is individual, intentional and consciously performed in a manner unlike an automatic operation. An activity, however, is more abstract and more difficult to come to grips with. It reaches beyond the individual and is part of a wider, social context. It cannot be accounted for by referring to what individuals do or think. Relevant examples can be found in public services and functions.
Kindergarten activity, for example, is more than the actions of those working there. What I do is part of an activity, but not on the strength of my doing it. At the individual level it is an action. The cashier checking out purchases in a supermarket carries out a number of individual actions – and probably a series of habitual operations – but collectively the personnel are engaged in the activity of trade of everyday commodities. In order to understand the nature of this sales activity, we must look at the whole co-ordinated picture of a complex set of actions, not merely the actions or tasks carried out by the individual.