EFFECT Briefing Paper
Project Information
Project Title (and acronym) / Evaluating feedback for e-learning: centralised tutors (EFFECT)
Start Date / 1 September 2011 / End Date / 30 June 2012
Lead Institution / University of Dundee
Project Manager & contact details / Aileen McGuigan
+44(0)1382 381496
Project website / http://blog.dundee.ac.uk/effect/
Design Studio home page / http://www.jisc.ac.uk/whatwedo/programmes/elearning/assessmentandfeedback/effect.aspx
Programme Name / Assessment and Feedback Programme
Programme Manager / Lisa Gray
Summary
Evaluating feedback for e-learning: centralised tutors or EFFECT was set up to evaluate TQFE-Tutor, an innovative communications and tutoring system set up in the academic year 2010/11 and in its second full year of use in the current academic session (2011/12). The system comprises a blog and email and Twitter accounts, all badged ‘TQFE-Tutor’. All programme communications between academic tutors, administrators, participants and associate staff take place via the TQFE-Tutor utilities. The academics’ individual institutional email accounts are not utilised at all for programme business. TQFE-Tutor is utilised on the Teaching Qualification Further Education (TQFE) programme, a professional development course for in-service lecturers in the college sector. In addition to evaluating TQFE-Tutor, the EFFECT project was intended to consider the move to paperless assignment submission which was introduced on the TQFE programme at the same time.
The EFFECT project found – as anticipated – a range of advantages associated with the TQFE-Tutor system, to do with both qualitative and quantitative aspects of the research. Programme participants were found to be better supported by the new system than previously and considerable savings in time, for instance, were made possible as a result of the system.
For online distance learning programmes with a large volume of participants, a centralised, paperless system such as TQFE-Tutor, offers various advantages to different groups of stakeholders: the programme participants, the academic staff, university administrators and associate staff. The benefits experienced in the context of the TQFE programme may be replicated elsewhere, particularly in situations where the teaching team may be trained to undertake work across the programme, i.e. in a ‘generalist’ role, rather than having specific responsibility for any particular part/module.
Because they take on this centralised tutoring role, spanning across modules, the TQFE team has great resilience; for instance, if a colleague is absent, their work can easily be covered, with no need for any input from the absentee. Previously, any staff member being unexpectedly absent due to illness caused considerable disruption for programme participants – who were often left without tutor support for unacceptably long periods – and for the remaining staff, who, having no access to correspondence between the absent colleague and their tutees were ‘in the dark’ regarding the progress of that group of participants. For this and a host of other reasons, TQFE-Tutor is a highly appealing proposition.
Many colleagues are initially resistant to the idea of paperless assignment submission and turnaround, preferring to read, assess and provide feedback on ‘hard’ copies of assignments. Once a paperless system is in place, however, it offers real boons. For example, from a sustainability point of view, the lack of paper is obviously attractive; there are no extra costs – for the participant or the institution – for printing or postage. The system works much more efficiently and speedily – an assignment is received by the institution almost instantaneously instead of relying upon postal services. From a teaching and learning perspective, programme participants experience lots of advantages, such as being easily able to read typed comments (with assistive technology if necessary) – there are no legibility issues as there can be with handwritten feedback.
The TQFE programme and the TQFE-Tutor system within it, adhere to REAP (Nicol and Draper, 2009) principles, as discussed in the EFFECT project final evaluation report.
Headline findings
The TQFE-Tutor centralised system offers a wide range of advantages for the different stakeholders involved: programme participants and associate staff in their employing institutions, academic tutors and programme administrators within the University of Dundee. Evidence gathered in the EFFECT project substantiated the advantages listed below.
For programme participants and associate staff:
· Speedy turnaround of email enquiries
· Consistency of responses from tutor team
· Opportunities for participants to take part in peer assessment and collaborative activities via the TQFE-Tutor blog
· A forum in which to network and/or raise issues with peers and academic tutors
· Improved support for associate staff – to help them to mentor programme participants within their institutions
· Useful introduction to innovative use of Web 2.0 social media tools, which they can in turn incorporate into their own teaching (in keeping with Scottish Government and European Lifelong Learning objectives)
For programme administrators:
· Speedy turnaround of email enquiries
· Ease of access to tutor email correspondence - all stored in one place
For academic tutors:
· Motivating for a team that has diminished in numbers to continue to be able to promote independent, self-directed learners, through tailored responses to enquiries which are subsequently made available to all via TQFE-Tutor blog – all learners potentially benefit from advice sought by just one
· Quality assurance of the team’s online teaching – all tutors have access to all the teaching via TQFE-Tutor, which leads to tutoring that maintains a high standard and does not vary in quality or content
· The centralised system is a major boon to new team members, for whom access to other members’ tutoring is a real boon (see the EFFECT site for video of a new staff member discussing the TQFE-Tutor system and how it has helped her). It is not only useful to new team members, however: all team members learn from each other and thereby improve their practice
· TQFE-Tutor affords each of the team a very useful overview of all contact with all programme participants
· The new system negates the need for one tutor to deal alone with a participant who requires a disproportionate amount of support - responsibility for this is shared
· The utilities offer a swift overview of the ‘state of play’ during staff absences as well as an ideal resource for the induction, mentoring and training of new tutors on the programme
· TQFE-Tutor helps to avoid the necessity of repeat effort: once one tutor has dealt with a question or appeal for advice, all tutors have access to that information (as of course do all the participants).
In regard to the switch to a paperless assignment submission process, huge savings in time are made for everyone concerned – participants, mailroom workers within the University, administrators, markers. There are also major space savings for the institution, since there is now no need for filing cabinets full of hard copy assignments.
The organisational context and key drivers for change
Formerly the programme utilised a ‘personal tutor’ system, with each academic tutor allocated approximately 44 participants to support through the programme. This was not an efficient use of tutor time, often involving a lot of repetitive effort. In the case of staff absence there were considerable problems for participants, suddenly finding themselves without the support they needed and for staff attempting to cover for the absence – without access to any of the ‘back story’ which TQFE-Tutor now makes available to the full staff team.
Key drivers for change were:
· Economic - a significantly reduced team (halved from approximately eight full-time equivalent (FTE) members five years ago to less than four currently)
· Institutional strategy: the University of Dundee’s Strategic Framework to 2012 (University of Dundee, 2007) positions the institution as follows: ‘Dundee is a leader in e-learning and we are determined to keep this lead by developing and investing in this area’
· A highly motivated team, keen to continue to offer a high quality professional development programme for 200+ TQFE participants per annum and to maintain the programme’s position at the forefront of online learning.
The technology context
Wordpress was chosen as the platform for the blog as the Programme Manager had run a successful pilot blog on that platform the previous year (2009/10), having learned that it had been used very effectively within the School of Medicine at the University of Dundee. Wordpress offers a range of functionality far beyond that provided within the Blackboard My Dundee platform; for instance:
· ‘sticky postings’, allowing a blog post to be fixed at the top of a page – very useful if there’s a message that the team need everyone to see
· categories and tags (the Blackboard blog at the time only used dates for archiving which meant it was very difficult to find old postings)
· ease of use and attractiveness: it was relatively easy for non-technical staff (the academics working on the programme) to develop the Wordpress blog in line with their pedagogical concerns, making it highly accessible and good to look at. On the other hand, the My Dundee platform required a great deal of technical nous to make even slight changes to its appearance and its mechanical style of organisation of materials into folders, sub-folders and sub-sub-folders often required an unacceptable amount of clicking for programme participants.
The University set up a Wordpress installation in 2010 and the TQFE-Tutor blog was one of the first to publish on that installation.
The assessment and feedback context
Since a plagiarism-checking tool was already utilised on the TQFE programme – Blackboard’s SafeAssign – it made sense to make further use of that utility for the team’s paperless aspiration, using SafeAssign as the official submission system for administrative purposes. This means that instead of dealing with enormous amounts of post (two hard copies of each of two summative assignments for each of 200 programme participants), the programme administrator now just has to run a report once a week for inputting on the management information system (SITS) used by University of Dundee.
Markers receive an e-copy of the assignment from the participant by email attachment to TQFE-Tutor. The table below compares the TQFE-Tutor submission system with that of another similar online professional development programme: the savings are clear.
Comparison of TQFE submission systemand that of Programme B / Timings are approximate
Programme B / TQFE
a / Programme participant word processes assignment / Same / Same
b / Programme participant prints two copies of assignment / Variable / N/A
c / Participant mails assignments to Programme Administrator (visit to Post Office likely – for Recorded Delivery (recommended) and to arrange appropriate postage) / Variable / N/A
d / Participant emails assignment to TQFE-Tutor / N/A / 1 minute
e / Participant submits assignment via SafeAssign / Same / Same
f / Assignment is received by University mailroom, sorted and delivered to School of Education, Social Work and Community Education (ESWCE) / Variable / N/A
g / ESWCE School receptionist sorts mail and delivers to relevant programme staff / Variable / N/A
h / Programme Administrator (PA) collects all mail from ESWCE mailroom and returns to desk / 2 minutes / N/A
i / PA opens mail and date stamps / 1 minute per assignment / N/A
j / PA runs SafeAssign report from Blackboard / N/A / 5-10 minutes weekly
k / PA records submission on Management Information System: SITS – SALS page / 2 minutes per assignment / 2 minutes per assignment
l / PA emails marker to inform them of receipt of assignment(s) / 1 minute per assignment / N/A
m / PA delivers assignments to marker’s pigeonhole in ESWCE mailroom / 30 seconds per assignment / N/A
n / PA or TQFE-Tutor duty tutor acknowledges receipt of assignment to participant and cc’s marker / N/A / 1 minute per assignment
o / Marker collects assignment(s) from mailroom and returns to desk (often in different building) / 5 minutes / N/A
p / Marker checks plagiarism report on SafeAssign / Same / Same
q / Reads and assesses in hard copy / Variable / N/A
r / Reads and assesses in e-copy, using ‘comment’ utility / N/A / Variable
s / Word processes assessment mark sheet template drawing on hand-written comments or Word ‘comments’ from e-copy / Variable / Variable
t / Marker emails mark sheet to PA / 1 minute / N/A
u / Marker emails mark sheet to participant with cc to PA / N/A / 1 minute
v / Marker returns hard copy to PA / 5 minutes / N/A
w / PA records result on SITS / Same / Same
x / PA emails mark sheet to participant / 2 minutes / N/A
y / PA prints mark sheet / 1 minute each / N/A
z / PA files hard copy assignment and mark sheet / 2 minutes per assignment / N/A
Benefits and beneficiaries
Staff, programme participants, associate staff and administrators have all benefited from the TQFE-Tutor innovation. Focus group discussions (excerpts of which are available on the EFFECT website), evidence gathered in end of programme evaluation questionnaires, interviews, interrogation of final results of the programme as recorded on SITS are all discussed in the EFFECT final evaluation report and all point towards the efficacy of the new system.
Benefits include:
· Enhanced self-direction in programme participants
· Greater opportunities for peer assessment and collaboration/networking in a distance learning context
· Improved final results (i.e. more programme participants successfully completing the programme within one academic year)
· Improved informal staff development opportunities
· Workload savings for participants and all staff groups (administrative, academic, associate)
· Speedier response time for communication with TQFE-Tutor
· Better quality feedback and more consistent across a geographically diverse team
· Improved support for participants and associate staff
There is a high level of interest amongst programme participants in using the TQFE-Tutor model in their own contexts. Similarly within the University of Dundee various other programmes have expressed an interest in following the TQFE-Tutor’s example; in one other programme there is already a ‘virtual tutor’ operating.
The programme’s successful pilot of paperless assignment submission has been reported to the Learning and Teaching Committee within the School of Education, Social Work and Community Education and other programmes are beginning to consider this option, although there is still a distinct reluctance around this. However, the sustainability and cost saving potential of paperlessness makes it a highly attractive proposition to the institution and it seems clear that in this development the programme team is ‘pushing against an open door’: soon other programmes will follow.