Evaluating Services
A course for trainers working in healthcare and other public sector organisations who want to deliver courses about conducting an evaluation
FOR TRAINERS
Authors: Louise Lester, Fawn Harrad, Paul Sinfield
Version: 1
Date: March 2014
Who is this training package for?
This package has been put together to provide training and development staff or other training leads with enough information to run a face to face session on evaluating services. It includes presentations, activities and templates that you can adapt to suit your needs.
Who is this course for?
Individuals and teams who want to know more about how to conduct an evaluation. Increasingly healthcare and other public sector services are assessed to decide if they should be continued, modified or withdrawn. These important decisions should be based on accurate and reliable evidence such as a properly conducted evaluation.
How will this course help the participants?
It is important to understand the principles of evaluation before starting to evaluate a service. Evaluation of a service is about collecting relevant evidence of the way a service is running, and this information is then used to help make a decision about its future.
This course introduces participants to some tools that will make it easier to conduct the evaluation and report their findings.
What will they learn?
This course will develop participants’ knowledge and skills to evaluate services in their area. Working with their peers they will gain a clear understanding of the principles of evaluation and the processes involved. This will enable them to develop a tailored evaluation plan.
On completion of this course, participants will know how to:
- Define the purpose, evaluation questions and the aimsof their evaluation project
- Identify the relevant stakeholders and evaluation team members and define their input to the project
- Select methods suitable for their evaluation project
- Identify and agree timescales, resources and areas for dissemination.
You can use the course outline template to be sent to participants and used to advertise the course.
What do I need to do to run this course?
The course has been designed for you to run through at your own pace, stopping to discuss issues as they arise or those that you think are particularly important or where you wish to illustrate with an example. We suggest that a total of around four hours is allocated, ideally in two sessions or a half day, if the tasks are to be started. We recommend groups no larger than 15 participants
About half this time is required if you just want participants to understand the principles and they are not yet ready to undertake an evaluation. You can lead participants through the documents, listen to the presentations and discuss any issues that arise. At a later date participants can return to the course and complete the tasks when they are ready to start their evaluation project.
We strongly recommend that you familiarise yourself with the course before leading a group through the material.
Teaching, Learning, Assessment
There will be no formal assessment for this course. However, there are five practical activities participants will need complete. Certificates can then be awarded (either hard copies or electronic) for participants to record in personal development plans and training records. A template is included.
Course Content
Section / Content- Introduction to the Course
- Introduction to the course and the evaluation toolkit.
- Scoping your evaluation
- Define the purpose of your evaluation
- Benefits of evaluation, and the different types of evaluation
- Who should conduct the evaluation and scoping for current/previous work
- Planning your evaluation
- Identifying stakeholders and the evaluation team
- Define evaluation questions and your aims and objectives
- Timelines and resources
- Ethics and Governance
- Conducting your evaluation
- Outcome measures
- Methods
- Data collection and analysis
- Dissemination and Implementation
- Disseminating your findings
- Implementing your findings
There is also a programme template which you may want to use to plan the session and give to participants at the start of the course so they are aware of the timings and structure involved. Activities can be completed individually, as pairs, or as a whole group using flipcharts etc.
- Introduction to the course
The evaluation toolkit
This will be referred to throughout the course; you may want to print out a copy to refer to as you go along and encourage participants to do the same or suggest they familiarise themselves with it before attending the course.
Participants’ evaluation projects
It is useful for participants to have their own evaluation project in mind as they work through the activities; it might be a good idea to let them know this before they turn up to the session so they can think about their specific project or bring any relevant information along. If they don’t have something specific to work on there are a few examples which they can use in the example evaluation projects sheet, these should be circulated at the session.
At the start of the session it is a good idea to get everyone to introduce themselves and briefly explain the evaluation they are currently working on or will be in the future and their experience of evaluations.
2.Scoping your evaluation
The first section of the course covers getting started. You can use the following presentation, either with the commentary provided, or turn the sound off and talk through the slides yourself.
After watching this presentation work through activity onewith your participants – scoping your evaluation
Activity 1: Scoping for your evaluation project
Has a similar evaluation been conducted elsewhere?
It is important that we learn from others and do not duplicate work. Participants need to be aware they should start by looking at evaluations already conducted by similar services/organisations and see whether they are willing to share methodology, tools and outcomes.
Section 1.2 of the Evaluation Guide (page 7)
Use the hand out for activity one to get participants to use the table to plan what they will do to scope previous evaluations or work.
3. Planning your evaluation
The second section of the course covers planning the evaluation, you can use the following presentation, either with the commentary provided, or turn the sound off and talk through the slides yourself.
Then complete activity twowith your participants.
Activity 2: evaluation aim
Participants will then need to identify and clarify their evaluation question and the aim of their evaluation. It is really important to do this as the beginning.A well considered and succinct evaluation question is very important for the success of the evaluation. The evaluation question will provide a useful framework for planning the evaluation.
Section 2.6 of the Evaluation Guide (pages 14-16)
How to formulate questions
The evaluation questions are essentially the questions that the evaluation project will answer. Whendeveloping the evaluation question you need to consider;
• Intended outcomes
• Who you are reporting to
• The time and resources available
Use the hand out for activity two for participants to clarify the aims of their evaluation.
Activity 3: Planning an evaluation project - stakeholders and project team
Identification of the project team
The activity can be used to identify the persons required on the project team and their current skills and experience, which will help participants to recognise any gaps that may need to be addressed. They need to consider who the stakeholders are and identify their interests or perspectives. They will also need to decide how and when to engage them in the process.
Section 2.4 of the Evaluation Guide (pages 10-12)
Use the hand out for activity three for participants to use the table to identify potential stakeholders and their project team.
4. Conducting your evaluation
The third section of the course covers conducting the evaluation. You can use the following presentation, either with the commentary provided, or turn the sound off and talk through the slides yourself.
Thenask participants to complete activity four.
Activity 4:Data collection and analysis
Use the hand out for activity four to help participants plan what evaluation tools and methods they will use to achieve their aims and objectives and who will be responsible for the analysis tasks and how these will be analysed.
Section 3.1 and 3.2 of the Evaluation Guide (pages 19-21)
5. Dissemination and implementation
The fourth section of the course covers dissemination and implementation of the evaluation findings, you can use the following presentation, either with the commentary provided, or turn the sound off and talk through the slides yourself.
Then ask participants to complete activity five.
Activity 5: Disseminating your evaluation findings
You can use the hand out for activity five for participants to plan the dissemination of their evaluation results. This will ensure that all of the stakeholders receive the key messages of the evaluation and agree on methods for wider dissemination if appropriate. Dissemination methods may include: Event/Workshop, Committee, Exec Summary, Report, Newsletter, Presentation, Article, Leaflet, Press Release, websites, social media.
At the end...
Evaluation
You can use the evaluation form template to ask participants to give feedback on the training session so you can tailor it if you decided to run future sessions. You can also use the feedback to support your own personal development.
Certificates
For those who completed the session you can use the certificate template to award them.
Further resources
The resources below can be sent to participants after completing the course for further reading and to help them. The Research and Development office on your organisation may also be able to help with evaluation support – it might be worth contacting them before running this course.
CLAHRC LNR training
Please visit the CLAHRC LNR training resources section forpresentations and useful resources previously run training sessions on evaluating healthcare services and other courses.
Commissioning evaluation?
If you or your trainees are interested in learning more about how to commission an evaluation from external evaluation providers and organisations please look at the commissioning evaluation toolkit for further guidance and templates
Attachments:
Course outline
Course programme template
Example evaluation projects
Activity one
Activity two
Activity three
Activity four
Activity five
Evaluation form template
Certificate template
Evaluating Services training package for trainersMarch 2014
Evaluating Services
Course outline
This course will develop your knowledge and skills to evaluate services. You will gain a clear understanding of evaluation and the processes involved to enable you to develop a tailored evaluation plan.
Time Scale
The session will be delivered in a (insert) hour session.
Teaching, Learning, Assessment
There will be no formal assessment for thiscourse. However, you will be required to watch the presentations and complete five associated activities.
Course Content
Section / ContentIntroduction to the Course /
- Introduction to the course and the evaluation toolkit.
Scoping your evaluation /
- Audit, evaluation or research?
- Define the purpose of your evaluation
- Benefits of evaluation, and the different types of evaluation
- Who should conduct the evaluation and scoping for current/previous work
Planning your evaluation /
- Identifying stakeholders and the evaluation team
- Define evaluation questions and your aims and objectives
- Timelines and resources
- Ethics and Governance
Conducting your evaluation /
- Outcome measures
- Methods
- Data collection and analysis
Dissemination and Implementation /
- Disseminating your findings
- Implementing your findings
Skills Gained
On completion of this course, you will know how to:
- Define the purpose, evaluation questions and the aims and objectives of your evaluation project.
- Identify the relevant stakeholders, evaluation team members and define their input to the project
- Select methods suitable for your evaluation project
- Identify and agree timescales, resources and strategies for dissemination
- Design a tailored evaluation plan
Accreditation
You will receive a certificate for your portfolio on completion of the course.
Evaluating services
[insert date, time, venue]
This course will develop your knowledge and skills to evaluate services. You will gain a clear understanding of evaluation and the processes involved to enable you to develop a tailored evaluation plan.
[insert name of trainer]
Programme:
START TIMEINTRODUCTIONS
TIMESCOPING YOUR EVALUATION
Activity 1: Scoping for your Evaluation Project
TIMEPLANNING YOUR EVALUATION
Activity 2: Planning an Evaluation Project - stakeholders and project team
Activity 3 – evaluation aim
TIMECONDUCTING YOUR EVALUATION
Activity 4 - Data Collection and Analysis
TIMEDISSEMINATION AND IMPLEMENTATION
Activity 5: Disseminating your evaluation findings
Examples ofEvaluation Projects
EXAMPLE 1:Pharmacy Support to Community Mental Health Teams
Background: The Care Quality Commission uses national surveys to assess the experiences of service users when receiving care and treatment from mental health care providers. The Community Mental Health Survey (2010) carried out nationally requests patients to rate specific areas of care they have received by their local NHS organisation. In one Trust, questions regarding the involvement of patients in their medication choices and information given about side effects, frequently resulted in low patient ratings, in particular the explanation of side effects resulted in a 5/10 for 2010. Therefore, the Trust aims to implement a Pharmacist into the Community Mental Health Teams to support patients with their medication choices, decisions, and information regarding side effects.
What needs to be evaluated? What are the potential benefits and impact this change in delivery of service will have on patient satisfaction, consultant/pharmacist satisfaction, medication adherence and cost effectiveness.
EXAMPLE 2:Principal Investigator Awareness Training
Background: In an attempt to try and engage more clinicians in large scale research projects, one NHS organisation has decided to put together a short awareness training session for clinicians about the role of principal investigator. It hopes that by disseminating this information via face to face sessions and information being available on the internet that they will be able to increase the amount of new clinicians undertaking the role and also improve the understanding of the role.
What needs to be evaluated? Does the introduction of principal investigator awareness training improve understanding of the role and increase the number of new clinicians agreeing to take part in large scale studies.
EXAMPLE 3:Type 2 Diabetes Education Programme
Background: Type 2 diabetes mellitus (T2DM) is a chronic health problem affecting over 2 million people in England alone. It shortens life, causes considerable morbidity and consumes 10% of NHS resources. There is a need, therefore, to focus efforts on preventing this devastating disease. Evidence has consistently shown that T2DM may be directly caused by physical inactivity. Unfortunately, previous diabetes prevention programmes that have focussed on patient lifestyle have not been able to maintain significant increases in physical activity. One Clinical Commissioning Group has introduced a new structured education programme for type 2 diabetes aimed at promoting walking activity in individuals identified as having an increased risk of developing T2DM
What needs to be evaluated? The Clinical Commissioning Group (CCG) would like to know how effective the structured education programme for type 2 diabetes is at improving patient outcomes.
EXAMPLE 4:Enhanced GP Role in Cardiology Care
Background: An opinion poll by one NHS organisation found that patients wanted to have care provided closer to home with more services provided by their local GP. Cardiology outpatient activity is known to be high in this area compared to national and local benchmarks. This project set out to implement an enhanced GP cardiology role in the area providing patients with a GP service with appropriate specialist support, and reducing referrals to secondary care. The GPs at selected practices undertook five sessions ofcardiology training delivered by two GPs with special interests and a local Consultant Cardiologist. Practice approval was given for participation in the project which offered eight sessions of funded time and consultant support (on line and by telephone) to manage internal GP referrals from GP practice colleagues and to provide a gateway for all non-urgent cardiology referrals, some of which would be averted through direct access to ambulatory cardiology investigations
What needs to be evaluated? TheNHS organisation would like to evaluate the impact of this project on: clinical quality, patients’ experience sand number of referrals to secondary care.
Activity 1: Scoping for your Evaluation Project
Has a similar evaluation been conducted elsewhere?
It is important that we learn from others and do not duplicate work. Start by looking at evaluations already conducted by similar services/organisations and see whether they are willing to share methodology, tools and outcomes. You may want to search the literature to find other national examples of best practice.
Section 1.2 of the Evaluation Guide (pages 7)
Organisation/Area / Details / Lead / Contacted/Date / Outcomee.g. NHS organisation / Contact Dr Smith to see if educational programme has been implemented and evaluated. / Joe Blogs / Yes (01/01/14) / Programme used but not yet evaluated. Would like to be included in the dissemination plan.
Activity 2:Evaluation aim
A well considered and succinct evaluation question is very important for the success of your evaluation.Your evaluation question will provide a useful framework for planning your evaluation.