Reid, Lesley,
Department of Primary Education, University of Strathclyde
‘Now I know what to do!’ Teachers talking about writing assessment.
Paper Presented at SERA annual conference, Perth, Scotland, November 2003
Key words: formative assessment, summative assessment, writing assessment, moderation, criterion referencing, construct referencing
Abstract
Primary school teachers and secondary school teachers of English in Scotland have developed skills in using criterion referencing to assess the writing of pupils in the 5-14 age group, using the national testing criteria framework. This paper looks at how these skills can be further developed through assessment moderating sessions, meetings where teachers work cooperatively to refine and confirm assessment judgements. Evidence was gathered in the form of interviews of a staff tutor and 6 teachers from 3 schools (2 primaries and 1 secondary) who developed assessment rubrics using a critical skills approach, for a Local Moderation project in the Assessment is for Learning Programme. Further data was gathered in the form of field notes from project team meetings, and a cluster writing moderation meeting involving 10 schools. Teachers’ views were sought on how implementing the formative assessment principle of sharing assessment criteria with children, and the experience of moderating the writing produced affected teachers’ views of assessment. A model is proposed for the moderating process which suggests that it is beneficial for teachers to collaborate to move beyond criterion referencing to develop personal construct referencing frameworks. It is suggested that adopting such a model would help teachers resolve a perceived tension between summative and formative assessment demands in relation to the writing curriculum.
Project Aim
The aim of the research undertaken for this project was to explore the ways in which working reflectively and collaboratively to produce annotated exemplars of children’s writing affected teachers’ perceptions of using a criterion reference framework for assessment.
Research questions
- How does the experience of implementing formative assessment principles influence teacher’s views of assessment?
- What are the benefits of using the moderation process to work collaboratively cross school and cross sector in order to share assessment understandings about children’s writing?
Research Methods
Case Study/ Action Research
The researcher’s role was a dual one of offering mentoring support to practitioners engaged in action research studies for the Local Moderation Project and investigating teachers’ experience of participation in the project. (Bassey, 2003)
Interviews were conducted at the start of the projects and towards the end as teams were writing case study reports. Meeting notes and recordings were collected during the progress of the action research project (from October 2003-October 2004)
Background Context
Assessment Development in Scotland
The Assessment is for Learning Programme aims to
- encourage teachers to use formative assessment to support pupil learning
- develop procedures to quality assure teachers summative assessment judgements
It is achieving these aims through a project based structure, with individual project teams pursuing action research case study investigations
Project 5 of the Assessment is for Learning Programme is entitled ‘Local Moderation’ and has ‘sharing the standard’ purpose, in line with the programme aim of assuring the quality of teachers’ summative assessment judgements. Project 5 teams are producing local portfolios of exemplification of pupil attainment in a variety of curricular areas.
Outline of the Research
The project remit was to produce a portfolio of exemplars of pupil writing which would illustrate agreed standards of attainment at specified 5-14 levels. There was a desire expressed by the project team to have a teaching and learning focus as well as a curricular focus for their action research. This was achieved through discussions with school managers and the HEI representative, and was guided by the Kings College research (Black et al. 2002, Clarke, 2002) and the stated aims of the Assessment for Learning Programme (
In order to fulfil the requirement of generating pupil exemplar materials and to help teachers engage with formative assessment principles, it was agreed that the team would also use the generation of exemplar material as an opportunity to progress their understanding of formative assessment principles, before they engaged with the moderating activity.
The teaching and learning focus for the project team was the ‘sharing success criteria’ with pupils. The action research team team consisted of four primary school teachers and two secondary school English teachers. The moderating meeting involved staff from all schools in the cluster and was focused on exemplars of pupil work from P7 and S1.
The teachers implemented formative assessment principles by developing assessment rubrics with pupils, based on adopting a critical skills approach. (Weatherley, 2003) A critical skills assessment rubric is a grid which shows criteria and a progression of skills devised by pupils and teachers working together using child friendly language in order to develop next steps. The assessment rubric is used during writing to provide support in the form of next steps to allow pupils the opportunity to improve the quality of their work, as they work.
In order to address the research questions specified, transcribed interview data and meeting notes were analysed in relation to a coding framework
This analysis allowed the first research question relating to implementing formative assessment principles to be addressed.
From this analysis conclusions were drawn about how teachers progressed their understanding of the role of assessment in the writing curriculum, firstly by developing a more detailed criterion referenced framework. (Brown, S. & Simpson, M.1988) They then applied the skills developed to specific writing tasks. This led to increased subject knowledge (Wyatt-Smith, 1999) and to increased craft knowledge about the social context of teaching and learning.
A shift in responsibility for learning and assessment occurred in the classroom as pupils assumed more responsibility for their own learning (Torrance, H. & Prior, J. 1998) and participated in interpreting the curriculum in relation to specific tasks set by teachers.
Further codes were developed from the interviews to allow the second research questions to be addressed. From this analysis, it is proposed that the development of teachers’ subject knowledge base and their increased craft knowledge base were extended through the moderation process to become a construct referencing system (Marshall, 2001) This development led to the proposed a model for the moderation process.
Results
Implementing Formative Assessment Principles
Teachers Changing Views of Assessment
Teachers used assessment rubrics as a technique to allow them to implement the formative assessment principle of sharing success criteria with pupils. Overt use of criteria for success became an integral part of all stages of the writing lesson.
Using assessment rubrics seemed to help teachers focus their teaching aims and perceive clearer links between those and assessment criteria. Both these aspects of the teaching and learning cycle were shared with pupils.
“This is helping them to improve as they go. It really does give you are very clear format for dialogue about what they are doing.”
The value of the process of constructing rubrics jointly with children lay in the translation of these aims into ‘child speak’. The negotiation of success criteria and the subsequent mechanisms developed for feedback therefore led to improved quality of dialogue between and among teachers and pupils.
“It has sharpened the focus of my practice rather than me doing anything very different; it has made the focus clearer for children.”
Increased opportunities for peer and self assessment were a natural outcome of the use of negotiated rubrics. Teachers felt that they were more in touch with learning while it was happening in the classroom. They seemed to feel more in touch with pupils as writers rather than simply assessing the product of pupils’ writing. Pupils’ metacognitive skills also improved.
“It is obvious that children are becoming more articulate in discussing writing”
There was evidence that the rubrics enabled pupils to focus on specific aspects of writing while they were writing, thus acknowledging both the diagnostic and formative potential of assessment.
“I’ve noticed that there is a lot of checking going there with children.
This is as they are writing; this is not with a finished piece of work
They are really thinking about the different criteria while they are writing their story.”
The linear structure of the rubric seemed to encourage children to identify their own next steps rather than put a ceiling on achievement. The relationship between learning intentions planned by the teacher and the success criteria by which their achievement would be judged, was clearer for children
They are much more conscious of what they are trying to achieve.
Pupils found the negotiated rubric structure more helpful than previous forms of criterion referenced assessment, possibly because they had a sense of ownership of the ‘grid’.
They are definitely less threatened by it; they see it as a support.
They don’t get so frustrated. They can cope with their own weaknesses a bit better.
Teachers’ diagnostic assessment of writing was enhanced. Teachers commented on specific improvements in aspects of writing, rather than in improved overall attainment: these improvements related to both compositional skills (choice and use of language, selection and organisation of ideas) and the technical skills of transcription (punctuation, syntax). The specific improvements related directly to the focus of the negotiated rubrics and were apparent across the ability range.
“I am seeing far better description.”
“They have real characters with thoughts and feelings.”
There was a sense from teachers that story writing was more coherent as a result of considered thought on the part of pupil writers.
“They have stopped just rambling on.”
“They are showing much more coherent thought, even the poorest ones.”
The progression and continuity between primary and secondary was viewed as more robust for children and the assessment seen as contributing to an understanding of coherence and progression.
“If we were going back to the beginning of primary 7 I think we would have a clearer path, we might have done some of these things earlier so that’s now we could put the whole thing together.”
“They will go on to High School- there will be that progression for them, it will be built on there.”
Moderation
The criterion referencedframework provided by the national testing criteria for the assessment of writing was used as a structured starting point for the teachers’ analysis of pupil exemplars.
During early moderation discussions, teachers sought to achieve more detailed interpretations of the criterion statements, in order to achieve increased specificity of judgement. These discussions led to the production, first of all of ‘generic’ rubrics for the assessment of Imaginative Writing. Collaborating with secondary staff on the interpretations of the performance indicators seemed to have benefits for Primary teachers in that they felt they gained increased subject knowledge from the secondary staff.
“We found the secondary English teacher had a clearer idea of technical skills. She went about things in a more specific way.”
“It was helpful to go through the process of seeing the way she did it”
For secondary teachers, the discussions were also useful, but in a different way. Firstly, they felt that they helped them to articulate and share the subject knowledge they already had. Secondary teacher also perceived another benefit, that of being exposed during moderation discussions to the knowledge primary teachers have of teaching ‘craft’.
“It was useful to hear what was done by primary teachers and pupils, just to see what they were really doing, because going into a primary school is not something I really do.”
A more detailed understanding of the assessment criteria also meant that teachers felt the quality of their feedback to children had improved and was taking place in the context of a shared understanding, between teacher and pupil.
“I can honestly say I am much more certain about how to go about teaching writing.”
It seemed to be the primary school teacher’s ability to ‘fill in the classroom context’ of the exemplar of pupil work that was most helpful for the secondary teacher, particularly in relation to children who were working at the lower end of the ability range.
Teachers work at moderation meetings progressed from discussions of generic rubrics to discussions about task specific rubrics.
The teachers who had been involved in developing formative assessment principles in the generation of the exemplar materials found these discussions helped them to locate assessment more readily as an integral part of the teaching, learning cycle, rather than as a bolt-on addition to it.
“It’s made me realise this is what I’ve actually got to teach.”
It became clear that the development of task specific rubrics had great potential as a formative assessment tool, and that this represented a progression in teachers understanding of assessment from a summative, criterion referenced model.
“I think there was a need for the generic rubric. I think we needed to work through that before we started on the specific one that the specific one is important because that’s really where your learning and teaching is coming from.”
There was an awareness that individual, task specific rubrics could become a resource which was accumulated over time to build up a holistic view of ‘good writing’, which might help resolve tensions between formative and summative assessment.
Conclusions
Implementing Formative Assessment Principles:
Using Rubrics in the Classroom
The use of assessment rubrics with children seemed to enable teachers to develop their understanding of criterion referencing assessment. Initially the generic rubrics developed led to increased teacher confidence in the criterion referenced statements of the writing assessment framework.
However, jointly constructing task specific success criteria with pupils (through task- specific rubrics) enabled them to move beyond ever more detailed criterion referenced interpretations of the curriculum. The task- specific rubrics allowed assessment to become a more effective part of the teaching and learning process.
Negotiating success criteria with children, and using them throughout the teaching and learning process meant that teachers’ subject knowledge became part of the shared discourse of the classroom. This ‘sharing’ led to improved pupil engagement with leaning and improved attainment. The process of negotiating success criteria with pupils in the form of rubrics, therefore allowed teachers to share their increased subject knowledge base with their pupils. The improved quality of the focussed dialogues that arose between teachers and pupils, and between pupils, meant that children assumed more responsibility for their own learning. Assessment became a dynamic force that was part of the social context of the classroom rather than something that was done for pupils by teachers. Teachers welcomed the power shift in the classroom that resulted from the increase in pupil responsibility. Teachers positioned themselves at the interface of the learners and the curriculum, as learning was occurring. This resulted in a redefining of previous conceptions of assessment. Assessment became a dynamic part of the teaching and learning cycle, rather than a text based activity conducted in the absence of pupils. The dynamic quality of the process was activated by the social quality of the context, the prioritising of spoken discourse that arose from the negotiations between teachers and pupils.
Moderation
Familiarity with a criterion referenced framework linked to a specific curricular area therefore seems to be a facilitating factor for the development of local moderation practices.
The benefits of the moderation process varied according to the school sector background of the staff: primary teachers increased their subject knowledge, secondary teachers enhanced their craft knowledge.
The predetermined focus of the team on the formative assessment principle of sharing assessment criteria with pupils, and the pupil exemplars generated by this focus, changed the nature of the moderating discussions.
Those teachers who were part of the practitioner research team and had engaged in development work in formative assessment found the moderating discussions about the use of task specific rubrics more helpful than those about generic rubrics, and saw increased relevance for their teaching. They moved in moderating sessions from a desire for increased accuracy of definition of criteria, to discussions which explored the relationships between the teachers’ shared subject knowledge base, their knowledge of the curriculum and their craft knowledge of the teaching and learning process.
This exploration can be understood as teachers developing a construct referencing system, (such as that described William 1994, 1996, 1998 and further expanded by Marshall, 2000) rather than simply extending an already robust criterion referencing system, in an ever more detailed way. William sees construct referencing as a system which relies upon a construct or intuitive understanding of what it means to be competent in a particular domain. He views it as a more accurate description of what teachers actually do when they make assessment judgement, and as different from either norm referencing or criterion referencing. Marshall explores William’s notion of construct referencing by looking for a match between secondary English teachers’ subject philosophies and there assessment decisions. The project teachers’ experience of using critical skills assessment rubrics with pupils and of moderating the resulting written texts and rubrics together, throws further light on the how teachers actually develop construct referencing frameworks and how these differ between school sectors.
The development of construct referencing through teachers moderation discussions, allows assessment to be more fully integrated into the teaching and learning process because it draws upon the social context in which the assessment evidence was produced. This social context, explained by the moderating teacher in the context of moderating meeting, illuminates the moderation discussion, rather than obscures or prejudices it. The value of the discussion focussed on specific exemplars is precisely that the teachers can relate the work to known classroom contexts,