The Higher Education Academy Annual Conference July 2006 – Session papers

Improving student placement learning experience through an innovative audit process

Keywords; audit, placement, student

Abstract

One of the most important and costly quality monitoring mechanisms in nursing and midwifery is educational audit of clinical placements. It is vital that the system should be transparent, effective and have beneficial outcomes for students in practice. This initiative aimed to reduce the costs of audit, improve the rigour of the process and, most importantly, demonstrate clear benefits that lead to improvements in the student learning experience. Working in partnership with eight Trusts and Primary Care Trusts a new process was devised and implemented which quickly led to an enhanced clinical learning experience. It enabled trusts to identify where to focus resources and efforts for pre-registration education.

Introduction

The need for a rigorous and effective educational audit process is vital to ensure the continuous improvement of the quality of the clinical learning experience for all healthcare students on placement. With numbers of student nurses increasing nationally, placement capacity being stretched and further placements being sought it is time to make sure quality of placements does not deteriorate due to the pressures.

“Educational audit has proved to be an effective way of reviewing current activities and learning opportunities available to healthcare students…it highlights good practice, identifies where there is a need for change as well as providing information for future planning.”

(Burke and Smith, 2000 p475)

The whole process of annual educational audit is, however, extremely costly in terms of time and manpower. With both these statements in mind, a local review of the system of educational audit revealed the following strengths and weaknesses.

Strengths

  1. Annual audit of clinical placements represents best practice (ENB, 1997.,

Ghazi, 1994). This aims to ensure that good experiences are consistently available for all

students.

  1. The audit tool used is based on standards (ENB 1997) and uses performance criteria statements to measure how the standards are being met. Such an approach has been well documented as reliable and valid (Fritz, 1997., Callaghan & McLafferty 1997). It also reflects the precepts within the QAA Code of Practice – Placement Learning (2001)
  2. The performance criteria used reflect those revealed, by the above researchers, to be the most essential to audit in healthcare learning environments.
  3. The process involves the educator, clinical staff and students raising audit profile as a significant and important activity (Ghazi, 1994).
  4. Staff are familiar with the audit tool and the ENB Standards.
  5. It is a basis for review and continuous improvement.

Weaknesses

  1. The huge cost of academic and clinician time was not efficient or effective.
  2. Students and clinicians perceived few improvements after each audit.
  3. Managers did not have an overview of audit results across a Trust or Primary Care Trust (PCT) due to so many areas being audited on different occasions throughout the whole year.
  4. No lead person was responsible for the overview of all audits. This was needed to close the quality loop i.e. make explicit links to internal and external quality monitoring mechanisms.
  5. Unfair distribution of audit workload between individual link tutors and audit teams.
  6. Consistently missed annual deadline dates due to staff unavailability to conduct the audit.
  7. Audits not fully completed eg missing signatures, assessor attendance at updates not recorded and/or staff qualifications not recorded.
  8. Link tutor bias.
  9. Inconsistent application of the audit tool, by auditors, affected the rigour of the audit.
  10. Staff not trained to carry out audits.
  11. Too many staff involved.

The Previous Educational Audit Cycle

This system required one academic (linked to the speciality being audited) and one clinician (from the area being audited) to undertake the annual educational audit for each area/ward/department. This usually took half a day with a further half day for the academic to write up the report. With over 400 clinical areas in the placement circuit this was resulting in a minimum annual cost of 400 working days to the Higher Education Institution (HEI) and approximately 200 working days to the Trusts and PCTs (a total of 600 working days across all organizations). In addition to this, there were travelling costs for academics. For large Trusts where there is midwifery, medical, surgical and other specialist services this often meant many different clinical link tutors going out to the same Trust or PCT on 30-40 different occasions carrying out 30-40 audits with their own personal approach.

Much time, effort and resources were being used by Nursing & Midwifery for the annual audits. It was clear that the costs and outcomes were neither efficient nor effective for the HEI or its partner Trusts/PCTs. It was, therefore, time for a radical change that would reduce costs but increase the robustness of the process and provide effective outcomes that benefited students,trusts and the HEI.

The New Educational Audit Cycle

The main strengths of the old process were the audit tool standards and the aim to achieve best practice of annual audits. These were retained and the new process aimed to eliminate the eleven weaknesses of the previous system.

Firstly, an Audit Team Co-ordinator/ Verifier was identified to lead the initiative forward and maintain the whole process. The next important stage was to redefine the “placement “ to mean one whole Trust or PCT with allocated areas within it. This meant that one Trust would be one “placement “ but could have up to 40 allocation areas within it but one audit would now cover an entire Trust or PCT. Thereby reducing the audits from over 400 to 8.

The next stage was to reduce variability in the process, increase consistency in application of the audit tool and remove bias. This was achieved through selecting a smaller team of staff to act as auditors, providing specific training and ensuring that link tutors did not audit their own link areas. The new audit team consisted of 12 academic staff from a mixture of adult, mental health, primary care and women & children's’ specialties. These were either selected for their robust approach to audit or for their stated commitment to improving the clinical education experience for students.

A small team, drawn from the 12 auditors, would conduct each audit. This would consist of at least 3 academic staff and the co-ordinator/verifier, along with a minimum of 2 clinical staff from that particular Trust/PCT. The whole Trust or PCT would be audited within 5 consecutive days. This would mean that the whole year of audits could be achieved in 40 days instead of over 400. The cost in working days to the HEI would therefore be reduced from over 400 to 160 and the cost to each Trust/PCT would be a maximum of 10-15 working days.

Discussion/workshops were held for auditors to devise how to ensure parity and rigour of the process and also how to implement the new process successfully. The Clinical Placement Facilitators (CPF) from each Trust/PCT would be responsible for requesting and collating the required hard evidence from all their allocation areas. The audit co-ordinator would act as verifier ensuring each auditor applied the audit tool consistently and conducted the audit in the same manner with the same degree of rigour. All auditors agreed a high quality placement learning experience for students would require relevant education, appropriate supervision and rigorous assessment for the student.

The partner Trusts/PCTs were all approached for their views on how to ensure this system would work and whether it was feasible. All were extremely supportive, with some reservations, but felt it would be a more beneficial approach.

The audit co-ordinator planned a schedule of audit weeks and auditor teams. The audit cycle was set as 1st Feb to 30th June each year. This could then help Trusts and PCTs meet their deadline for annual Placement Agreement Reports. Academic staff were informed of the new process via staff development days, team meetings, briefings, training sessions and email.

Each auditor from the HEI would participate in a maximum of 3 audits per year to provide equity of workload.

A new student placement evaluation form was devised to mirror the audit standards. This facilitated their collation and use in the audit process. Further guidance notes for auditors were developed on how to triangulate the evidence for the performance criteria. This assisted auditors in applying the audit tool consistently.

The audit co–ordinator would be responsible for the feedback of all audit recommendations to each Trust/PCT and monitoring action progress through quarterly contract review meetings and annual quality monitoring mechanisms i.e. the Workforce Development Annual Review, the Annual Placement Agreement Reports and internal HEI annual monitoring systems.

How the new audit worked

The Trust or PCT audits each required the full 5 days (smaller Trusts with few allocation areas were covered in less time). Day 1 was used to examine and scrutinize all documentary evidence that was provided along with the audits from last year and the all the appropriate student placement evaluations. Auditors reported that this gave them a background knowledge of how each of the areas functioned. Clinicians also commented that they were pleased the auditors “seemed to know the area before they arrived.” There was again discussion between auditors of how to ensure parity and maintain rigour. The auditors worked in pairs looking at the evidence relevant to the clinical allocation areas they would visit over the coming 3 days.

Where possible the CPF had arranged visit times to allocation areas (for days 2, 3, and 4 of the audit week). Most auditors and clinicians in placement stated they preferred having a specified time of day. Each visit required verification and triangulation of the evidence already seen to ensure that what was documented was actually carried out in practice. This also included talking to students, mentors, managers and clients. All auditors felt they particularly enjoyed the visits as they had a clarity of focus and knew precisely what they were looking for in each area. They also were pleased with the way they were made welcome in the majority of areas and the overwhelming perception that most students and mentors were enjoying the placement experiences.

Day 5 was spent formulating the final report for the Educational Lead of the appropriate Trust/PCT. This allowed time for all auditors to discuss and debate the clinical education issues raised from the audit along with the appropriate CPF. Staff felt this was beneficial as solutions could be suggested and the report was agreed by all involved. The Educational Leads were then responsible for ensuring all their allocation areas received a copy of this report and an action plan was set.

Evaluation so far

The whole process is continuing to prove highly successful for several reasons. It provides a concentrated focus on placement quality across a whole Trust/PCT which can then be progress monitored over the coming year . The auditors (clinicians and academics) describe the whole system as being much more robust and allowing time for productive discussion around common themes or issues.

Each report clearly identifies good practice and makes recommendations for the Trust/PCT to action. All trusts have responded with appropriate action plans for improvements.

Less time is required for each Trust/PCT to carry out the whole audit of a placement. This has a cost benefit for both the HEI and each partner organisation. Less time is also required for individual allocation visits. This is aided by the analysis of the hard evidence on day 1 of each audit week.

Less manpower is used across the whole process with subsequent reduction in travelling costs.

Each report per Trust/PCT is compiled with the clinicians involved in the audit. This creates greater ownership and commitment to actioning the recommendations. Many of the clinicians involved comment that they “learn a great deal from the audit” and can also “take innovative ideas back to their own areas.”

The auditors have a more consistent approach due to the smaller team, the presence of the verifier and agreed interpretation of the audit tool.

All allocation areas provide the same forms of evidence to meet the standards. This gives the auditors a clear organisational and allocation overview for each audit standard before they make any visits.

Good Practice Points Identified

The new audit system revealed the following good practice points across all placements;

  • All minimum standards are being consistently achieved each year.
  • Student support and supervision is generally very good across all placements and individual areas requiring improvements can be easily identified and supported appropriately.
  • Issues raised by students and staff are dealt with promptly by the appropriate CPF and/or passed onto the relevant person for further action.
  • The majority of students interviewed each year feel they have good support from their mentors and other staff.
  • The majority of students report enjoying their allocations.
  • The majority of mentors interviewed have a robust approach to assessment of clinical practice and utilise the assessment strategy and procedures well.

Common trends identified across all trusts

The new system also allowed identification of common themes/issues that all placements needed to address.

The student induction packs varied in quality and content between allocation areas. This was addressed through an agreed template being made available and CPFs providing advice on improvements.

Relevant learning resources (including staff) were often unidentified or under utilised. This meant that students often could not recognise the wealth of learning resources available to them in each area. The induction packs now include identification of these valuable resources for students.

The first year of the new audit revealed a total of 1,514 qualified nurses and midwives were on the Mentor Register but only 916 were updated and eligible to assess students at the point of the audits. Student access to mentors was therefore restricted at that point. All trusts recognised this was a priority to address and last year 1,371 mentors were in date. This made a significant impact during the clinically assessed terms.

Intra and inter mentor reliability was not routinely addressed within trusts. Mentors reported uncertainty as to how they measure or monitor this. The HEI and trusts worked together to rectify this through several mechanisms. Audits now reveal mentors are beginning to address it using appraisal systems, clinical supervision, setting up their own mentor support systems, utilising the skills of staff from the HEI and acknowledging the analysis of completed clinical assessment documents.

Mentors expressed a lack of confidence in dealing with the occasional student problems of non-attendance, lack of professional interest, poor timekeeping or unprofessional behaviour. Once again the HEI worked in collaboration with the partner trusts to implement changes that would empower the mentors to deal with such issues. These have been well received to date.

Time for mentors to assess students is not explicit and protected as part of good role modelling/practice. Some allocation areas have addressed this through innovative ways such as manpower reconfiguration to allow dedicated time for the mentor role, however, it continues to be a major issue for many mentors and most trusts.

Areas tend to perform better as a high quality learning area when an identified clinician has education of staff/students as part of their role remit. This has worked particularly well in some theatre areas and accident & emergency departments.

Most staff continue to prefer mentor annual updates being delivered on site by their own CPF.

Conclusion

The successful implementation of this initiative has provided a clear focus for improvements to the student learning experience, mentor and student support. It has also made the whole process more effective and efficient. The auditors described the previous system as a “chore” but the new system as a “pleasurable and useful” experience. It allowed the audit team to focus for a whole week on the education, supervision and assessment of students in practice in each placement. The benefits of this compared to one link tutor auditing one allocation area in one day were immense. It could also easily be used as a multi-professional educational audit. A formal evaluation of the whole educational audit process (Ashdown-Lambert & Jinks 2004) highlighted aspects for improvement within the process but also commended the initiative for it’s successful approach.