Assessment Careers

Pilot Report

Psychology

Author(s): Maria Kambouri

October 2013

Institute of Education, London

JISC Assessment and Feedback Strand A Project

Assessment Careers

Summary

The aims for this pilot study in a Psychology Master’s module were to improve feedback and student perception of iterative feedback as well as tutor perception of the value of ‘developmental’ feedback so that conviction of its value is shared by all tutors. A student feedback response form was therefore essential to guide the type and quality of feedback given. There were 4 staff and about 35 students involved in this pilot. A focus group with tutors and interviews with 4 students provided evidence on top of that collected through the sheets. Only 15 completed the cover sheets and very few students included real depth reflections. Though this is an improvement to the previous year, in terms of requests for formative feedback, it was clear that students did not take advantage of this formative feedback opportunity. There is some evidence that weaker students may find this type of iterative feedback difficult. What this pilot study confirmed was the desire of students for clear, direct to their text type of advice.Time is key overall, both in terms of when to despatch feedback and how many iterations are needed and continuity was also key, such as the need to have a place on MOODLE where all student work is filed including assessments and feedback.

  1. What are the headline achievements of your pilot?

•Half of the module’s students engaged with value in the formative process so far, as they possibly feel they have little time for submitting an outline- those who did found it useful. There was an increase of 20% over the last year.

•In general the most experienced learners seem to have sharp self reflection skills and ask thoughtful questions for guidance through the student feedback response form.

•There is value in thinking about the role of feedback for students and the relation between formative and summative work.

•Discussions across modules between staff have begun to make more links between learning points within and across modules.

  1. What were the key drivers for undertaking the pilot?

The module leader had noted that in an earlier module, students wrote a reflective essay but was surprised to find little reflection on this module and so there was concern about learning from module to module.

Also, there was a need at this time, to review current practice and contribute to the ongoing debate about assessment processes within the department.

  1. What was the educational/organisational context in which you undertook your pilot?

Background of course

The description of the module on which the intervention was applied and the link to other core module is set out here: The programmes chosen were the MA/MSc Psychology of Education; MA Education (Psychology). Students on these two programmes are taught as one group for core modules. (The difference in award depends on a student’s prior qualifications. The first-mentioned is accredited by the British Psychological Society (BPS); the 2nd is not.)

Focus module: Cognitive Development and Learning (CDL).

This is one of two modules between which students must choose. Around half the students choose CDL.

a)A Core module, which all students must take, would feed forward to CDL. This is Core Topics in the Psychology of Education(CTPE).

Part of its assessment is an essay-style critique of a research article, preceded by two substantial in-class sessions of critique practice using other articles. These take place at the start of the academic year, and are intended to have an early impact on students’ critical reading and writing at an early stage in the programme. They give intensive practice that we hope gives considerable formative benefit.

b)Thorough formative feedback is then given on the submitted critique. The focus on academic reading and critique fit the early stages of this module well as feed-forward material. Hence we would expect comments on academic writing and critique to be in parallel for this intervention.

c)Feed forward from CDL was to the dissertation, where the literature review and framing of research questions should benefit from the students’ cumulative experience in interrogating research. This transfer was not researched.

As CTPE and especially CDL will have substantial changes of staff by 2012-13, and also changes of content. This seemed to be a good moment for piloting changes to assessment. This pilot project was planned with the new colleague who led the CDL module. The pilot leader and writer of this research report was not teaching on the module this year and was therefore in good position to evaluate the intervention.

  1. What was the technology context?

There was a change of VLEs during the intervention. While in the previous year students submitted on Blackboard, this year they submitted on MOODLE. Studentssubmitted assessment via MOODLE (which used TURNITIN automatically). The feedback was given either directly (formative) or via MOODLE (final assessment).

There was no substantive difference in the way of using the technology other than the checking through TURNITIN.

  1. How did you approach the pilot?

Students were told about the project, asked to fill in pre-intervention sheets about assessment in general in the beginning of the module (in January). A reminder about using formative and summative feedback sheets was sent by email twice.

Description of the intervention:

The introduction of the new student feedback response form as a more structured way of students requesting feedback was the intervention. The introduction of a new form was preceded by discussion with staff a year prior to the intervention. The project was introduced the at the first lecture of the module, in January, to signal that a change in the process and purpose of formative feedback to students is intended. Students were asked:

a)to answer (anonymously) the following question on a pro-forma with details about the project and giving consent to participate in the project.

Think about the last piece of written feedback that you received on an academic course (not necessarily this course). Was your work formative (a draft you could revise) or summative (final assessed work that contributed a grade)?

b)To attach two new cover sheets to their work submitted, one for formative and another one for summative feedback (see Appendix 1).

  1. What benefits has your pilot delivered and who are the beneficiaries?

The beneficiaries were the students, 4 staff including the module leader and the pilot leader. The 4 tutors shared their reflections on the pilot as a focus group. 4 students responded to requests to participate in the focus group for evaluation purposes.

  1. Are students enabled to monitor their progress through formal opportunities to reflect on feedback?

Students were all offered the opportunity to be part of this awareness raising intervention which asked them to formally consider the feedback they were receiving in the past, compare it to now and ask for specific points of feedback. It was evident that some could benefit more than others. This was due mainly to the diversity of backgrounds both culturally, as most come from various education systems around the world and in terms of knowledge of the subject, as they can have a first degree other than in psychology.

Students always had the opportunity to submit a two-page outline of their essay for feedback. About half took this opportunity this year (15/35) as opposed to only a third last year. There was therefore an increase of 20% over last year which can be attributed to this pilot.

  1. Are students enabled to monitor their progress through formal opportunities to engage in dialogue with staff or peers?

Students need more re-assurance and therefore seek formative feedback for several reasons. This project allowed the space for more informal and formal opportunities to engage in dialogue with peers (through group interviews) and staff. We were able to hear more from those students who may have communication difficulties and need extra support: whether it is about language or about academic skills or to do with cultural differences, or indeed because they are crossing disciplines and so paradigms are different. Here are some examples of what students said organised in themes:

Language barrier: English as a second language

“... actually to be honest all…um, assessment titles are quite similar to me, and it’s quite difficult how I should write, how I organize the essay, and so I think it depends on the teacher, or lecturer, and that’s why I wanted to talk about the essay during the writing, I was about to, I think, for me, I really wanted to talk about my essay twice or three times before the meeting, otherwise I was really worried is it not OK”

“Written feedback is really helpful for me because I always don’t record our conversations, so that’s why it’s easy to forget, and so if the lecturer send me, you know, little feedback, and very helpful..”

Academic writing: critical thinking

As distinct from language which mostly applies tointernational students this theme is more about criticality and argumentation which is the core of academic writing structures; despite the trial essay they are given in the start of the course some students find this skill is not tuned enough:

“But then I was just thinking it might be helpful if we can do like a double loop sort of feedback, like you know, you got the first feedback, you kind of tried to implement it, make it work, and then saving for another time before we write the whole essay…”

Different tutor approaches(writing styles);

Some students believe that going from one tutor to the next they find marked differences of feedback styles. This was not encountered often and as we only researched one team it was limited to that group of people. Diversity is an issue that is brought in both for students and staff. However tutor’s respond with another issue that of a good outline:

Giving in a good outline/2 page draft:

Tutor:

…our students complain, they compare feedback on plans and they complain that they don’t get a similar amount of feedback. And so we’ve told them quite categorically that the amount of feedback we are able to give depends on the quality of the plan that is submitted. It’s very difficult to give more than two lines of feedback when you’ve only got two lines in a plan and so on. And so one of the things that they suggested was can we have guidance on what a good plan is?

Cultural and educational level differences;

…because I don’t have a psychology or educational background, my prior degree is actually in business, so I wasn’t taught how to actually critically analyse the validity or, you know, how trustworthy a piece of writing is, because at first I was taught that all journal articles are good!”

Timely feedback

Some students (particular those with language difficulties) expect feedback to be immediate and find that when deadlines cross over with other assignments it is more difficult for feed forward to work (and so they don’t learn much from the feedback):

…Because I had to focus on another essay, so that’s why I forgot everything about what I write,...

Specific feedback

Some students find it difficult to implement feedback that is of a general nature and there is a need for specific feedback on certain points (with concrete examples):

…You did well on this, you did well on that, but the third step you should...

…But I don’t know if I finally succeeded to be more critical, because the draft was very good, but I don’t know if the essay was as good as the draft, this was my problem. I didn’t know whether I implemented the feedback…

Continuity (via MOODLE)

Some students voiced clearly the need to have a place on MOODLE where all their work goes including assessments and feedback. This was also suggested by a member of the teaching team.

Yes, I think it’s easier to track progress really, because I took some modules in TESL classes, some modules in psychology side, so there is some discontinuity, yeah…

Peer Support

There was more peer support and discussions introduced because of this intervention.

“It is good to talk to others about it and compare notes. If we had done that more in the beginning we would have gained time…but then we don’t know each other yet!”

  1. How does the intervention change the waythat staff write feedback?

We also compared feedback given on assignments in previous years of the module with feedback given after the intervention by the same teaching team using a standardised feedback analysis tool. The feedback analysis tool was redesigned to distinguish the different purposes of feedback drawing on simple feedback categories developed by Orsmond & Merry (2011) and with an additional category of ipsative feedback.

The feedback categories were:

P1Praise for good work

P2 Recognising progress or ipsative feedback

CCritical feedback. This was subdivided to distinguish error correction C1, and critique of content C2 and critique of structure and argument C3.

AGiving advice. This was also subdivided to distinguish specific content advice for the current assignment

A1, general advice for the current assignment A2 and advice for future assignments A3.

Q Questions addressed to learners to request clarification of their ideas.

O Unclassified statements. Neutral comments, for example that describe the piece of work but do not make any judgement, were unclassified.

A feedback profile for an individual assessor or for a module or for a programme could be compiled from looking at the balance between the categories.

Psychology Feedback on final work

2013 / 2012
No. of comments (n=29) / Rank / No. of comments (n=20) / Rank
P1 / 121 / 1 / 39 / 1
P2 / 1 / 0
C1 / 52 / 3 / 8 / 4
C2 / 22 / 5 / 11 / 3
C3 / 59 / 2 / 7 / 5
A1 / 47 / 4 / 7 / 5
A2 / 10 / 7 / 22 / 2
A3 / 2 / 0
Q / 18 / 6 / 0
O / 10 / 7 / 1 / 6
Average no. of comments per script / 11.8. / Average no. of comments per script / 9.5

From the quantitative analysis above (feedback tool) there was no apparent significant change in the balance of feedback provided for this course from one year to the next. And in many ways this is true.However, this was not entirely a fair picture since:

  1. some formative feedback happens orally and this is not recorded and
  2. not all tutors adopted a change of tactic or format yet (they were simply made aware of it at this stage).

There is however clear evidence from the teaching team focus group discussion that we a more structural approach (based on examinations criteria) will be adopted which will prompt most common feedback points.

  1. What is the impact of the intervention on staff workloads?

Not a significant change in staff workloads was encountered or claimed.

  1. How has the intervention encouraged staff and students to review the value of feedback and feed forward?

It has invited students to think about not only the kind of feedback they receive but also what they do with it, which was not addressed before in an explicit way. Despite the prompting, it is not surprising to find that certain students (those with greater need for support) are not capable of incorporating general comments and they are asking for more specific input to help them re-draft their work. The degree of transfer of these aspects from one module to the next (feed forward) is therefore less evident than expected. Students were given more time to reflect and were asked to evaluate the feedback they got through interviews. Feed forward was still hard to achieve as it was not clearly signposted and only few students took advantage of this.

The intervention has also prompted discussion among the team and comparison of feedback styles. It has allowed for those few students who grasped the opportunity, to express their request for specific feedback and receive a longer explanation on certain aspects of the work submitted.

  1. What are the technological, organisational and social issues that need to be addressed for scaling up the pilot studies? i.e. what would convince other stakeholders that the change is needed?

This team was not using MOODLE enough to benefit from various options (only for submission and TURNITIN). However, through the discussions it appeared that it was crucial to find an electronic way to record data in a format accessible to all tutors involved with one student so there is some continuity in the way we feed-back and forward (eg. Through digital ‘student booklets’ proposed by one tutor).Students also valued this option as they felt it would be easier to access their portfolio of work.

In general there has to be a change of culture to accommodate more dialogue about the work particularly from students who are not sure about what is expected. This ought to be ongoing rather than introduced as it is at key dates such as course induction or in key documents such as handbooks. Examples of work could be shared on MOODLE for those who have little experience of standard required.