Development of peer collaboration models for writing formative examination questions to aid assessment and feedback.
APPLICATION TO LEARNING AND TEACHING DEVELOPMENT FUND - APRIL 2012
Joanne Burke1, (), Donald Cameron2,Vivian Binnie2, Lesley Nicolson3, Jennifer Hammond3, Carol Ditchfield1, Margaret-Ann Flynn1, Max Field1,John Hamer4, Helen Purchase4 and Amanda Sykes5 (Medicine1, Dentistry2, Veterinary3, Computing Science4, Student Learning Service5)
Summary:
This applicationbuilds on a previous LTDF projectto support and develop a computer based platform to enable students to write, review and practice two different types of examination questions.
The National Student Survey has highlighted the need toimprove formative assessment and feedback in support of student learning. In addition to teacher generated feedback, one way to enhance student feedback is to engage students in this process – to generate feedback for themselves and their peers.
PeerWise is a web-based software programme that allows students to write their own multiple choice questions, and answer and critique those set by others. Students are required to provide an explanation withtheir answer, and hence they extend their knowledge base when writing a questionand gain immediate feedback when their peersanswer thesequestions. This project aims to extend the use of PeerWise to year 1-3 students in Medicine, Veterinary Medicine and Dentistry. In order to monitor the effect on student learning, the project will interview students and look at the correlationbetween usage and exam performance.
The second aspect of this project isto develop software to accept questions called Situational Judgement Tests (SJTs). These are specialised types of questions designed to assess professional attitudes in all three disciplines, covering key graduate attributes developed during undergraduate training. SJTs are now also used as part of selection tests for medical students when entering their first postgraduate training posts. This project aims to work with the PeerWise developer to pilot the use ofnewSJTWisesoftware. Students will use SJTWise to write, review and practice SJT questions and to obtain feedback using similar principles to PeerWise. Student views will be obtained and usage data will be correlated with clinical exam performance to analyse potential enhancement to learning.
1. Aims and outcomesduring the next two years will be to:
a) extenduse of PeerWise to engage students in years 1-3 in all three disciplines in MCQ writing for formative assessment. After the next two years there will be data from 1,500 medical, 540 dental and 840 veterinary students.
b) assessthe effectiveness of PeerWise, by comparing written examination marks for students who write questions and those who simply answer the MCQs. Furtherqualitative datawill be obtained from focus groups from the different year groups.
c) build on the experience of students generating SJT questions, and integrate SJT writing sessions into medical, veterinary and dental courses to build question banksin each discipline for future use and to explore potential areas of overlap between disciplines.
d) develop the software prototype for SJT questions to produce a more sophisticated programme (to be named SJTWise) that will provide 500 medical 180 dental and 280 veterinary students with opportunities to write, answer and evaluate SJT questions and use this process as a novel learning opportunity.
e) assessthe effectiveness of SJTWiseby comparing marks for those who write questions and those who just answer questions in their OSCE examinations. This will involve reviewing questions which assess professionalism issuesas compared to questions assessing clinical skills. Qualitative evaluation will be obtained using focus groups and questionnaires.
2. Background and previous work from LTDF bid 2011:
a) Need for the Project. This project addresses two University challenges. Firstly, National Students Surveys highlighted limited formative assessment and a need for improved feedback in all Colleges including Medical, Veterinary and Dental schools. Secondly, UK regulatory bodies (General Medical and Dental Councils and to the Royal College of Veterinary Surgeons) insist that Universities teach and assess professionalism and other graduate attributes during student training. This bid will build on and utilise students’ experience to address both challenges, to develop new technology to help students with feedback and formative assessment, and continue to support staff by sharing existing experience and expertise within the schools as these developments take place.
- Both formative assessment and effective feedback are known to be powerful influences on student learning(1), andengaging students in the processis a key component of effective feedback (2,3). By developing their own questions, students can enhance their personal learning during course programmes, and, by using additional IT support that provides formative assessment with feedback from their peers, students in all three schools can build knowledge, critical thinking and personal confidence.
- The L&T strategy indicates a need to address Glasgow graduate attributes, including professionalism to develop portfolios of generic skills that, over and above basic knowledge, can help students generate successful job applications. Assessing graduate attributes is a complex area and often requires staff to assess a student portfolio including presentations, essays, written reports and simulation exercises (4). These can be difficult to assess objectivelyand time consuming to complete. New IT-based techniques can address the issue of consistency in assessing graduate attributes, but students have limited experience in use of these technologies, and this is an area that should be improved.
b) Multiple Choice Questions (MCQs) are integral to Dental, Medical and Veterinary assessment. During our present LTDF project, students in each school wrote MCQ’s and shared these with their peers using the PeerWise on line repository (5). This software (developed by Paul Denny and John Hamer) has been used in several Universities to allow students to submit questions with explanations and evidence based answers on line (6). Questionsare peer-rated, subject to student-led discussion in a forum, and provide students completing the question with immediate feedback about their performance in comparison to their peer group. In so doing,PeerWise allows students to build up a valuable question bank for self-directed formative assessment. To our knowledge this is the first time that PeerWise has been used in these disciplines and evaluated to this extent.
PeerWise:With funds from LTDFin 2011, PeerWise was introduced to year-1 students in all three subjects with different models for each discipline. Participating studentswere given information about PeerWise and remindedby staffto make use of PeerWise. Usageup to February 2012 is summarised in Table 1 and is as follows:-
1) Year 1 and 2 Veterinary students were expected to write eight and answer twenty questions (5), which contributed to 5% of annual summative assessment. In year 1, compliance was 99%. PeerWise use peaked prior to each summative assessment, and in feedback from students about all course elements (lectures etc.) students scored PeerWise second highest. Several questions submitted to PeerWise have been used in summative assessment.
2) Year 1 dental students were required to write twoand answer tenquestions as part of a course assignment,but received no reward for participating (7). Compliance was 99%. Usage peaked before examinations, and informal feedback indicated no concerns about using PeerWise in this way. Year 2 dental students heard about PeerWise use in Year 1 and expressed an interest to use PeerWise as well.
3) Year 1 Medical studentsengaged with PeerWise on a voluntary basis. About80%used PeerWise, predominantly by answering questions just prior to examinations.Focus group data showed that students perceived that practice in using MCQs was helpful, that knowledge was required to write questions,and praised the variety of questions in a single resource. Students were concerned about lack of quality assurance, that some question authors did not include full explanations for the answers, and that each question could only be attempted only once.
Table 1. Details of PeerWise usage by students in each School
School / total numberof students in year 1 / Number of students
Writing/authoring questions / Number of students
answering questions / Total number of questions
answered by students in
year 1
Veterinary Medicine (147) / 146 / 147 / 19,766
Dentistry (95) / 95 / 94 / 13,342
Medicine (273) / 45 / 217 / 16,632
c) Situational Judgement Tests: Learning about professionalism is integral to the University graduate attributes, and assessing professionalism is required bytheGeneral Medical and Dental Councils and the Royal College of Veterinary Surgeons (4, 8). SJTs can be set which use clinical scenarios to assess these graduate attributes and SJTs are already used in postgraduate selection for General Practitioners. SJTs now make up 50% of marks in final year assessments before medical students apply for their foundation year (FY) posts after graduation. Veterinary and Dental curriculum developers are reported to be considering their use. SJTs consist of questions and answers where students either rankfive appropriate actions in response to a clinical scenario, or select three most appropriate actions from eight possible correct answers (8). By assessing graduate attributes using clinical scenarios, SJTs assess students’ reflective learning skills at the highest level of Bloom’s Taxonomy, and arereadily adaptable for marking by computer.In the recent pilot study using SJTs before their applicationsfor FY posts in 2012, 50% of University of Glasgow final year medical students fell into the bottom third of the national average,a response to be improved upon in our student group. If a tool akin to PeerWise was available, Glasgow students could gain valuable personal experience in writing SJTsandusing these for practice purposes, while at the same time developing a bank of practice questions for future use.
As part of the 2011 LTDF bid, a project was set up for two Year-4medical students, aiming to assess whether students could write effective SJTs. With guidance, theywrote twenty SJTs based on the ranking five-option format(9),which were subject to revision following review by Faculty staff. Over 160 (out of 253) Year-4 students volunteered to sit a formative assessment based on these questions. Marking the questions by hand took the students 28 hours,with the distribution of marksbeingsimilar to those published for UK medical students(9).The 163 students attended predominantly for practice purposes, with only 14% perceiving themselves as prepared for SJT tests prior toformative assessment — a figure that rose to >60% after the test. More than half the students reported seeing similar scenarios during their training, indicatingpotential experience that can be tapped for writing SJTs.The majoritysaid they would purchase a book on SJTs,indicating a perceived need for practice questions. Over 40% expressed an interest in writing SJTs, indicating that a substantial number of students could be involved in building question banks for the future.
3.Student Engagement:We believethat students are an underused resource within the University, and have shown that students canuse Peer Assisted Learning to play a more active role in their own learning and assessment (10,11). Student use of MCQs in PeerWise and in writing SJTs has demonstrated the need for, and the value of, these novel educational tools.
PeerWise:In summary, preliminary studieshave seenover 370 Year-1 students engage with PeerWise for over six months. Usage shows that the majority have used thesystem extensively,and that students perceive several benefits to use of PeerWise, Studentshave also identified ways to improve PeerWise for their benefit as outlined above.
Situational Judgement Tests: Inthe national pilot study of final-year medical students, Glasgow students performed poorly in SJT tests in comparison to the national average, and there is a desire for additional practice in SJT use. Evaluation of our student project show that 4th year students have had experience of the clinical scenarios used in SJTs and can write these questions (possibly with some Faculty support) to the benefit of their peers. More importantly, students have the desire to write SJTs to help develop a question bank
Student Inclusion in this application:We propose to continue to involve students in writing, answering and evaluation of MCQs using PeerWise. In addition, students from Medicine, Dentistry and Veterinary schools will write SJTs for their peer group, provide an evidence base for their answers and, once SJTWise is further developed as an IT resource, will make use of this system as a question bank. In 2013 and 2014, we propose to offer two summer student scholarships to assist with evaluation of PeerWise and SJTWise,and to establish areas for improvement in the future.
4. Methods
This two year project consists of four phasesoutlined in Table 2:
a) Student Participationin and development of PeerWise.We will extend provisionof PeerWise to the next two consecutive Year-1 undergraduate groupsso that the students in the first three years in each school will have access to PeerWise. After the next two years,1,500 medical, 360dentaland 840veterinary students will be given the opportunity to generate, answer and evaluate MCQs for formative assessment. Early experience has identified areas in PeerWise that students perceive need development which will be put in place; e.g. allowing students access to one question more than once to help revision. Use of PeerWise has been higher than expected, but by continuing to adopt the different models in all three schools we will establish which is the most appropriate. PeerWise could be compulsory with marks awarded for participation (veterinary students), compulsory as part of an assignment with no marks for participation (dental students),or completely voluntary (medical students).Data from three year groups willprovide large student numbers to enable comparison of PeerWise use and student examination results.
b) Situational Judgement Tests and development of SJTWise. PeerWise cannot be applied to SJTs, butby the end of the 2011 LTDF project an SJTWise prototype will be available, allowing students to submit SJT questions on line in the ‘rank five’ option format, and summer students will have evaluated this prototype. In this application we will focus on:
- supporting the creation of SJTWise for both formats of SJTs,allowing individuals and student groups to submit questionsand to write a justification of their answers
- generating a forum for every SJTsubmitted to allow studentsto focus on engaging in an on line debate that resolves into a collective decision about the final version of each SJT
- providing a facility within SJTWise to enable Faculty to review relevant statistical data about administrative issues (e.g. details of student use, SJT writing and answering questions, time of use etc)
- providing a facility in SJTWise for review of SJTs by Faculty staff for quality assurance.
This will enable students to write, evaluate and modify their questions (in both SJT formats) in light of student centred discussion and for Faculty to administer the use of SJTWise as with PeerWise
c)SJTWise in curriculum development. By the end of the previous LTDF project (Table 2),the SJTWise prototype for ‘rankfive’ option questions will be in place, so that all students will have access to this SJT format.from September 2012.As with PeerWise,each schoolis likely to adopt a different approach for course integration during the project, but because writing SJTs is recognised as a complex process, students have requested a level of Faculty quality assurance which will be an essential part of development of SJTWise in each school.
- MedicalSchool. Discussions with the Faculty team coordinating the Medical school 4thyear course recognise a need to incorporate SJTs into the curriculum from 2012/13 onwards. Medical students will be timetabled a half day workshop to brainstorm SJT scenarios in small groups, debate answer choices, write explanations and submit SJTs through SJTWise. Questions will be subject to review by the student peer group usingSJTWise in the first instance,andstudent groups will submit ‘final’ revised versions of their SJT to Faculty for quality assurance. Some questions generated can be included in formative assessment planned during their 5th year as students approach the summative examination and feedback given through SJTWise.
- VeterinarySchool. Tutorials on“becoming a professional” are already incorporatedearly in the curriculum. It is proposed that from 2012/13 each student will generatea scenario withanswers for subsequent discussion in small groups during these tutorials. After agreement on a final version by the group, each SJT will be submitted through SJTWise to the rest of the year. Each studentwill have ownership of one question, and will review comments from their peers. Selected members of Faculty staff will be approached to provide quality assurance.
- DentalSchool.Situational judgements are currently underutilized for teaching purposes in the Dental school. There will be pilot development of questions by students working in small group settings in the 5th year in 2012. These will be extended in 2013 for all 5th year students providing question banks for teaching purposes but also for potential formative assessment in the future after submission to Faculty staff for quality assurance purposes.
d) Staff support through discussions. In the project’s first year,Faculty from all three disciplines have met regularly and have established strong working relationships. Staff will continue to meet to share progress, experience and discuss further project implementation and outcomes. Specific details about student recruitment, training and developing project evaluation in each of the Schools will be addressed to ensure consistency in approach and to ensure the maximum benefit for College students.
5. Application to Glasgow Graduate Attributes
In addition to building thestudent knowledge base, as well as competence and ability to deal with uncertainty in vocational subjects, writing questions/ formulating evidence based answers in MCQ and SJT formatswill also:-
a) enhance problem solving skills and benefit written communication skills;
b) encourageengagement with peers in eachdiscipline and the student community as a whole;
c) develop higher level critical thinking and discipline relevant professional skills;
d) provide an environment to encourage team working and review of ethical/professional issues.
6. Potential Applicable/ Transferability