Assessment Compact Evaluation Strategy

Proposal for an evaluation of theOxford Brookes University Assessment Compact (v1.0)

Greg Benfield, Chris Rust, Margaret Price

3 Dec 2010

Note: This documentincludes revisions to previous drafts suggested by AESC and incorporates methods that have been approved by UREC (full approval 21 Nov 2010 Approval no 100503).

1. Background and purpose of the evaluation

In the current academic year (2009/10) Schools have been implementing the Assessment Compact. Chiefly this stage of implementation has involved redesign of assessment practices to bring them into line with Compact requirements for academic year 20010-11, when all students in all courses should be working to the compact.

This evaluation aims to a) gather diagnostic, baseline information about the extent of knowledge among students and staff about the Assessment Compact, b) gain insights about staff and students’ perceptions of the impact of the Compact on their learning, c) if possible, capture the impact of new assessment practices introduced in line with the Compact on the student experience, and d) gain insights into how the Compact could make a greater contribution to the student experience.

A report describing the activities of Schools to prepare for the delivery of Assessment Compact-aligned courses was tabled at the June 2010 AESC (Benfield, Rust & Price 2010). This report is used as a source to identify examples in Schools of new and exemplary assessment and/or feedback practices from which to draw samples of students and staff for investigation of their experience of the Compact.

This evaluation will produce

an early first report - "Early messages" - in June 2011 that could influence changes for academic year 11/12
an interim report on the first year of evaluation in October 2011
and the final report in Feb 2013 (to allow all programme reviews to be considered).

The intended audience is Oxford Brookes University students, academic and support staff, School Deans and Assistant Deans, and SMT.

2. Research questions and data collection

We propose four main research questions, with subsets of more detailed questions. The relationship between the research questions and data collection is shown in Table 1.

2a.

What is the current level of awareness among students and staff of the Assessment Compact?

One of the most basic things we need to know about is the extent of awareness among students and staff of the University about the Compact and its clauses. A general level of ignorance about the Compact could be taken to indicate a failure to adequately implement it, while a high level of awareness might be taken as one measure of engagement by the Brookes community with issues around assessment.

Secondly, we need to establish a baseline map of Compact-related practices across the University: a mapping by schools, departments and demographics such as gender, age, course level and international/home status. This would allow us to track the development of assessment practices in subsequent years, assuming that depth of implementation will likely take at least three years.

Table 1: Evaluation Matrix for Assessment Compact Evaluation

Evaluation questions / Online survey of sample of students and staff / Student diaries / Module evaluation questions / Annual programme review report forms / Focus groups with course reps
Student forums
What is the current level of awareness among students and staff of the Assessment Compact? / Do you know about/have you read in your course documentation of the Assessment Compact?
Can you associate practices within your course with items within the Compact (e.g. marking exercises, peer and self-assessment, etc)?
How do students describe their engagement with assessment and feedback over the course of an academic year? / What proportion of students/staff actually experience ‘Compact-rich’ assessment and feedback practices (i.e. specific types of assessment, engagement with assessment etc)? / How do students perceive the relation between assessment and feedback practices in their course and their learning? How satisfied are they with these practices? What would they like to change about them? / How satisfied are students with assessment and feedback practices in their courses? Which ones have the greatest perceived impact on learning? / How satisfied are staff with assessment and feedback practices in their courses? Which ones have the greatest perceived impact on learning? / How assessment literate are our course reps? What are their opinions about the value of assessment and feedback in their courses?
How do students and staff relate specific assessment and feedback practices to impact on their learning? / Stories around specific assessment and feedback activities and how they are perceived to relate to performance. / Establish links betweenspecific assessment and feedback activities and how they are perceived to relate to performance. / Staff assessments of impact of new assessment practices on student cohort performance. Identification of barriers to implementation. / Which assessment and feedback practices do our course reps rate most highly as benefiting learning?
How can the Assessment Compact make a greater contribution to the student experience? / What proportions of staff and students are actively engaged in assessment practices? / How can assessment systems and processes be improved? / How can barriers to implementation be removed? How can greater impact on improving student learning be achieved? / How can assessment processes be re-shaped to make greater improvement on student learning?

2b How do students describe their engagement with assessment and feedback over the course of an academic year?

As well as a general awareness of the Compact, we would hope to find evidence that students can identify themselves as being actively involved in the assessment process in various specific ways. For example, we would hope that students could identify assessment and feedback activities in their course experience such as participation in marking exercises, opportunities for self and peer assessment, and possibly active engagement in the development of assessment-related curricula. We would also expect to find that students can give examples in their experience of opportunities to use feedback in subsequent assessments.

2c How do students and staff relate specific assessment and feedback practices to impact on their learning?

As well as evidence of Assessment Compact-related practices we need to know how students and staff perceive the impact of these practices on student learning.

2d How can the Assessment Compact make a greater contribution to the student experience?

Finally, we need to learn whether and how the Assessment Compact in particular and assessment practices at Brookes more generally can improve the student experience at Brookes.

Data collection methods

We propose five main data collection methods.

i) Baseline survey: In the first semester of each project year, we will survey students and staff about their awareness of the Compact and certain specific types of assessment and feedback practices. The survey will be conducted using Survey Monkey and will be administered via block email. (UREC has approved a one time only bulk email survey invitation and worked with us to develop appropriate wording and protocols for delivering the survey.)

ii) Audio diaries: in the second semester of each project year, 12 students will be recruited by open invitation to audio-record assessment experiences on a fortnightly basis, using specific prompts, i.e. ‘questions’, to address during the recording. Each recording, about 8 in total, is expected to take about 15 minutes. The first recording is to take place in week 0; the last recording after the final exam. The purpose is to obtain an in-depth and longitudinal student perspective on assessment literacy.

iii) Module evaluations (16 in total): We will ask the eight 2010 academic schools’AESCs to nominate two modules that have adopted new assessment practices in line with Compact implementation and incorporate additional items into their standard module evaluation forms. This will both assist staff of these modules to gather evaluation data about their course enhancements and allow us to gather evidence of the perceptions by students of the impact of these innovations on their learning.

iv) Data mining annual programme review documents: for staff perspectives on the Assessment Compact from relevant to the modules identified in the June 2010 implementation activity review as having made significant changes to their assessment practices. To gain insights in to staff perspectives on the Assessment Compact, annual programme review documents will be examined. Significant changes to their assessment practices will be identified. The Academic Registrar has asked for an item on the Assessment Compact to be included in the Annual Programme Review template and we will analyse the responses to this item at the end of the annual review round each year for two years.

v) Focus groups: In partnership with the student union, for the first project year we will invite course reps to compose 2 focus groups of approximately 8 UG students from two different modules, selected in discussion with the Student Union. A third group will consist of the reps themselves. For the second project year, we will hold two focus groups of 8 PG students, also from two different modules. The focus group topic is on the experience of assessments and a set of statements will be used to facilitate and direct the discussion. The purpose is to collect specific student assessment vernacular to further frame diary prompts and to look for evidence of assessment literacy.

Research team

Margaret Price is Professor in Learning and Assessment and leads the development of learning and teaching in the Business School through the development of strategy, sharing and enhancement of good practice and innovation in learning, teaching and assessment methods. As Director of the Centre for Excellence in Teaching and Learning, ASKe (Assessment Standards Knowledge Exchange), she is working with a team of colleagues to build a learning community centred on assessment, to encourage innovation and foster evidence-based assessment practice within the HE sector. She has undertaken research and published work on peer support for learning, criterion referenced assessment, social constructivist approaches to sharing knowledge of assessment standards with students and the effectiveness of sharing knowledge of standards within marking teams.

Dr. Greg Benfield, Oxford Brookes University, PhD, MEd, PGDipPSE, Dip Ed, BSc. Has conducted a range of synthesis and evaluation projects within HE including a literature review of learners' experiences of blended learning (HEA funded), and the JISC-funded Learner Experiences of e-Learning programme. He has conducted a variety of qualitative research projects over the last 10 years involving individual interviews and focus groups with learners.

Professor Chris Rust is Head of the Oxford Centre for Staff and Learning Development, and Deputy Director of the Human Resource Directorate at Oxford Brookes University, and for the last five years was also Deputy Director for two Centres for Excellence in Teaching and Learning - ASKe (Assessment Standards Knowledge Exchange) and the Reinvention Centre for undergraduate research (led by Warwick University). He has been Course Leader for the University’s initial training course for new teaching staff (for six years), and with (currently) thirteen colleagues has helped to provide both staff and educational development support to the University’s academic Schools and support Directorates for over twenty years.

He has researched and published on a range of issues including: the experiences of new teachers in HE, the positive effects of supplemental instruction, the effectiveness of workshops as a method of staff development, ways of diversifying assessment, and improving student performance through engagement in the assessment process. Recently he has tended to focus increasingly on assessment. He is a Fellow of both SEDA (Staff and Educational Development Association) and the RSA, and a Senior Fellow of the UK Higher Education Academy, for whom he was also an accreditor.

Birgit den Outer is the research assistant for this project and responsible for the day-to-day running. She holds Master's degrees in Cultural Anthropology from the University of Amsterdam and Mentoring and Coaching Practice from Oxford Brookes University and as a researcher for ASKe, has been involved in numerous research projects on assessment standards and learning. These have involved both staff and students, and a variety of research methods.

3. Participants

Table 2: participants completing surveys

Survey A (Baseline activity survey): / Survey B (Module evaluations):
Population / All Brookes students (and staff?) / 2 Module leaders per school who have made significant changes to their assessment practices.
Recruitment / Block email via field chairs / Nomination by School AESCs,
Incentives/rewards / Prize draw / Staff have access to standardised, comparative evaluation of their assessment-related course enhancements

Table 3: participants completing interviews

Student diaries / Focus groups
Population / Whole student body / Sample of course reps, Student Forums
Recruitment / MOD, Newsletters / Via Student Union, SSCs, MOD, Newsletters
Incentives/rewards / None / None

Table 4: Document analysis

Annual programme review documents
Population / All programme leaders
Recruitment / School/Faculty AESCs
Incentives/rewards / OSCLD/ASKe to evaluate assessment related items and provide report

4. Workplan

Table 5: Schedule of project activities.

Date
By… / Activity / Who
11 June 2010 / Proposal to AESC for comment and refinement / GB
30 Sep 2010 / Revised proposal to AESC / GB
30 Sep 2010 / Development of Survey instrument / BO
14 Oct 2010 / UREC submission / BO
12 Nov 2010 / Finalisation of Survey Monkey instrument / BO/GB
19 Nov 2010 / Finalisation of evaluation proposal and draft evaluation tools / Team
Dec 2010 / Identification of target modules for module evaluation / Team
20 Dec 2010 / Recruitment of staff leading modules with Compact enhancements willing to incorporate module evaluation items / GB/CR
Sem 2 2011 / Development of student diary protocols / BO/MP
Sem 2 2011 / Recruitment of student diarists / BO
31 Jan 2011 / Development of standardised module evaluation items / team
Feb 2011 / Staff focus group to plan module evaluations / Team
14 Feb 2011 / Survey Monkey baseline survey analysis / BO
28 Feb 2011 / First Survey Headlines report to ADSEs and ProVC (SE) / Team
June 2011 / Analysis of module evaluation data / Team
June 2011 / Analysis of annual programme reviews / Team
June 2011 / ‘Early messages report’ / Team
(First draft GB)
July 2011 / Analysis of student diaries / Team
October 2011 / Interim report / Team
February 2013 / Final report / Team

5. Expected outputs

By the end of the project, we will have produced:

an early first report - "Early messages" - in June 2011 that could influence changes for academic year 11/12
an interim report on the first year of evaluation in October 2011 (that may include recommendations for changes to Compact implementation approaches and/or enhancements to the Compact)
and the final report in Feb 2013 (to allow all programme reviews to be considered) including recommendations for on-going implementation approaches and any changes to the Compact.

6. Resource requirements

Research assistant @ circa 5 days per month over life of project

£3000 to cover: recruitment incentives, interview transcription, RA time.

References

Benfield, G., Rust, C., and Price, M. (2010). ‘Implementing the Assessment Compact:
an Activity Report for the 30 June 2010 AESC’, Internal report dated 11 June 2010.
Appendix 1: Survey Email invitation letters

Student version:

This email will only be sent to you once.

You are being invited to take part in a university-wide survey designed to improve the Student Experience.Itis being sent to all staff and students in the university and has been authorised by the Pro Vice Chancellor Student Experience.

To take part and opt-in to the study, please click on the link below now:

If however, youdo notwish to take part, then pleasedo notreply to this email. Under the Data Protection Act you will have opted-out of the survey andyour grades (where this applies)will be unaffected, whether you choose to take part or not. You willnotbe contacted again.

Staff version

This message is from Professor John Raftery:

This email will only be sent to you once.

To take part and opt-in to the study, please click on the link below now:

Appendix 2: Survey instrument