External Review of West MidlandsQuality Review Service – Year 1
External Review of West Midlands
Quality Review Service – Year 1
August 2010
Page 1 of 47
March 2010 © 1992-2010 Finnamore Ltd
External Review of West Midlands Quality Review Service – Year 1
Contents
1Acknowledgements
2Executive Summary
2.1Background
2.2Evaluation Purpose
2.3Evaluation Framework
2.4Evaluation Methods
2.5Document and file review
2.6Key informant interviews
2.7Focus Group Interviews
2.8Literature Review
2.9Findings – Formative evaluation
2.10Formative evaluation
2.11Interview Set-up
2.12Interview Questions
2.13Interview Responses
2.14Role with WMQRS
2.15Past Work with WMQRS Team
2.16Length of Involvement with WMQRS
2.17Setting Up WMQRS
2.18Current Understanding of WMQRS
2.19Work of the Board or Steering Groups
2.20Resources for WMQRS
2.21Standards set by WMQRS
2.22Make-up of Standard Setting Groups
2.23The Peer Review Visit and Process
2.24Overall Assessment of WMQRS
2.25Potential Change achieved by WMQRS in 3 Years
2.26Should WMQRS itself be Changed?
2.27WMQRS Recommended to Other SHAs
2.28What kind of Standards?
2.29Who should implement the findings?
2.30Who is Setting the Standards?
2.31What kind of Process is WMQRS?
2.32Findings - Document Analysis
2.33Thematic findings
2.34Quality of reports
2.35Discussion
2.36Conclusions
2.37Recommendations
2.38Diffusion of Information
2.39WMQRS System Development
2.40Performance measurement and Evaluation
2.41Education and training
3Background
4Context
4.1High Quality Care for All
4.2The NHS Operating Framework
4.3World Class Commissioning
4.4Monitor
4.5What is Quality Improvement?
4.6Quality Improvement in the West Midlands
4.7Collaboration across the West Midlands
5Evaluation Purpose
5.1Evaluation Framework
5.2Evaluation Methods
5.3Conceptual Frameworks
5.4Donabedian Approach
5.5Document and file review
5.6Key informant interviews
5.7Focus Group Interviews
5.8Analysis of Interview Data
5.9Evaluation strengths and weaknesses
5.10Literature Review
6Findings – Formative evaluation
6.1Formative evaluation
6.2Interview Set-up
6.3Interview Questions
6.4INTERVIEW RESPONSES
6.5Role with WMQRS
6.6Past Work with WMQRS Team
6.7Length of Involvement with WMQRS
6.8Setting Up WMQRS
6.9Current Understanding of WMQRS
6.10Work of the Board or Steering Groups
6.11Resources for WMQRS
6.12Standards set by WMQRS
6.13Make-up of Standard Setting Groups
6.14The Peer Review Visit and Process
6.15Overall Assessment of WMQRS
6.16Potential Change achieved by WMQRS in 3 Years
6.17Should WMQRS itself be Changed?
6.18WMQRS Recommended to Other SHAs
6.19Other Feedback from Respondents
6.20WMQRS Profile
6.21What kind of Standards?
6.22Who should implement the findings?
6.23Who is Setting the Standards?
6.24What kind of Process is WMQRS?
6.25Other Comments
6.26RENAL UNITS
7Findings - Document Analysis
7.1Thematic findings
7.2Quality of reports
8Findings – Cost effectiveness
8.1Analysis of relevance and performance
9Discussion, conclusions and recommendations
9.1Discussion
9.2Conclusions
10Recommendations
10.1Reporting requirements
10.2Resourcing
10.3Leadership Recommendations
10.4WMQRS continued involvement
10.5WMQRS, systems, practices and policies
10.6Diffusion of Information
10.7Public awareness
10.8WMQRS System Development
10.9Performance measurement and Evaluation
10.10Education and training
11Summative Evaluation Framework
12Appendix 1: Literature Search Results
12.1Australian Council on Healthcare Standards
12.2United States: Systems Quality Improvement
12.3Baylor Health Care System, Dallas, Texas
12.4European Perspective of Quality Improvement
12.5Internal Responses to external comparison
12.6The Royal Adelaide Hospital in Australia
12.7Institute for Health Improvement - Triple Aim
12.8National Programmes and Initiatives
12.9Value for Money
12.10Effectiveness of Quality Improvement – learning from evaluations
12.11Quality Improvement Maturity Level
12.12Implementing Quality Improvement Systems
12.13Evaluation of quality improvement programmes
12.14Quality Improvement Report Structures
13References
Page 1 of 77
April 2010 © 1992-2010 Finnamore Ltd DRAFT
Section13
References
External Review of West Midlands Quality Review Service – Year 1
1Acknowledgements
This project was commissioned by the West Midlands Quality Review Service as part of their ongoing commitment to evaluation and continuous quality improvement. It was conducted by Finnamore Limited.
The authors would like to thank all those who participated in interviews, focus groups as well as those who provided feedback at peer review meetings.
2Executive Summary
2.1Background
The West Midlands Quality Review Service (WMQRS) is a collaborative venture developed by NHS organisations in the West Midlands region which supports the continuing improvement of health services. The service supports organisations in ensuring they have a robust framework for quality improvement across their portfolios of services. West Midlands Quality Review Service is part of the West Midlands SHA Framework for Quality and Safety.
2.2Evaluation Purpose
A commitment to evaluation was part of the Business Case for West Midlands Quality Review Service.
2.3Evaluation Framework
The evaluation framework will make the greatest possible use of WMQRS staff and other NHS resources, such as the Leadership Development Programme Alumni, in order to minimise the amount of external input required for the summative evaluation.
2.4Evaluation Methods
The evaluation design used involves a mixed methods design. It is based on a literature search and document analysis followed by a series of semi structured interviews and focus groups.
2.5Document and file review
An extensive review of the documents and files was conducted for this evaluation. This included administrative documents, business / work plans, communications and media documents, background and historical documents and plans, minutes and other governance documents. In addition, the documents produced by the WMQRS, including reviews, environmental scans, survey tools and resources, publications and reports, and recommendations. The documents reviewed were produced between April 2006 and February 2010.
2.6Key informant interviews
Interviewees were invited from all those who had been participants in a review or had themselves been reviewers. Contact communications were sent to more than 600 WMQRS participants across the West Midlands. Interviews were conducted during reviews, by telephone and in person over a 3 months period. A topic guide based on the document analysis was used by the both interviewers, responses were transcribed and collated. Participants also included WMQRS board members and team members.
2.7Focus Group Interviews
Focus group participants were selected based on recent reviews that had taken place. The End Stage Renal Failure Group were utilised for a focus group as they had most recently been reviewed and had been observed from the start of the evaluation process. The focus group discussions were transcribed and responses collated.
2.7.1Evaluation Strengths
The evaluation allowed access to a large group of participants who had direct experience of the West Midlands Quality Review Service. The qualitative evaluation design allowed interviewers to include themes as they emerged from interviews.
2.7.2Evaluation Limitations
The evaluation encountered a poor response rate from past participants, which meant the results are limited by small numbers of participants. The views of the participants can be used for qualitative analysis; they cannot be used for any meaningful quantitative analysis. Future summative evaluation will need to incorporate design features to address poor response rates.
2.8Literature Review
2.8.1Criteria for selecting programmes for the review
Types of publication
The included publications were all those in which quality improvement systems were described. Published papers included evaluations and opinions on quality improvement systems.
Types of setting
Papers were included from both national and international settings in relation to healthcare, describing programmes that operated both at a regional and national level.
2.9Findings – Formative evaluation
The early nature of evaluation was important as it allowed the Board and management team to assess the progress of the service.
2.10Formative evaluation
During December 2009 and January 2010, we carried out 22 interviews with WMQRS contacts and also visited 3 renal units to discuss their reviews in the WMQRS process. The 22 interviews were individual interviews with those who had previously participated in reviews, WMQRS team members and WMQRS Board members. The focus group discussions took place with the teams of the three renal units who had recently been reviewed. During the review process, the interviewers also observed reviews and in addition to interviewing participants also had preliminary discussions with those being reviewed as well as those reviewing.
2.11Interview Set-up
We attempted to obtain responses from several groups of contacts of WMQRS:
•Board Members;
•PCT Leads;
•Provider Trust Leads;
•CIC Steering Group members;
•Urgent Care Steering Group members.
We also interviewed key staff at WMQRS and two senior staff at the SHA.
2.12Interview Questions
The interviews were semi-structured, to allow respondents to raise issues beyond specific, closed questions. Areas explored were:
•The establishment of WMQRS as a local initiative;
•Staffing and timetable for planned activities;
•The process and people involved in standard-setting;
•The conduct and outcome of local reviews of services;
•An overall assessment of WMQRS.
We note overall that opinion was very supportive of WMQRS.
2.13Interview Responses
We were able to obtain agreement to interview from 22 contacts of WMQRS after two rounds of emailing to the list of contacts provided to us. The largest groups of respondents were managers and hospital doctors (n = 22).
2.14Role with WMQRS
The most responsive group were, unsurprisingly, Board members. Clinical and other members of individual steering groups were less responsive than PCTs and Trusts (n = 22).
2.15Past Work with WMQRS Team
More than half the respondents had not previously worked with the WMQRS Team in any other role (n=22).
2.16Length of Involvement with WMQRS
Most respondents have been involved less than or up to one year, depending on their role in a particular group (Board, Steering Group). A small number of respondents had also had past contacts through earlier versions of the peer review process.
2.17Setting Up WMQRS
Only 11 respondents felt able to comment on the way in which WMQRS was established. Of these, 7 felt that it had been done well and 4 that it had been done reasonably well. Three commented further, one indicating that the approach had been relatively consultative and two that it felt like a “top-down” process by the SHA.
2.18Current Understanding of WMQRS
Most respondents who felt able to comment on the current awareness and understanding of WMQRS.
2.19Work of the Board or Steering Groups
Of those who felt able to respond, half indicated that the Board or Steering Group that they were involved in worked very well and half that there was some scope for improvement (n = 12).
2.20Resources for WMQRS
Only 10 respondents felt able to comment on whether WMQRS has the staff and time to do the job it has set out to do. Of this group, 4 felt that it might not have and one that it definitely did not have enough staff.
2.21Standards set by WMQRS
Most respondents who felt able to comment felt that the standards set by WMQRS were rigorous. While some respondents felt there was some scope for improvement or qualified their endorsement, there was no strongly critical comment (n=14). However, 4 respondents felt that the Steering Committees did not necessarily have the right membership, particularly a sufficiently broad representation of medical staff across primary and secondary care.
2.22Make-up of Standard Setting Groups
Respondents were asked if they thought that there are sufficient clinicians on the groups setting standards (n=17).Almost half of respondents felt that there were sufficient clinicians involved but 2 felt that if anything there are too many clinicians involved. Only 13 respondents answered the parallel question on whether there are enough patients involved in groups setting standards for WMQRS.
2.23The Peer Review Visit and Process
WMQRS is still at an early stage of its development and so it is not surprising that more than half our group of respondents had not observed a review on the day, though some had been involved in responding to findings from reviews. The majority of the responses received gave the most positive opinion on the aspects of the review discussed with respondents. Overall, it is clear that the review process is being relatively well received by providers.
2.24Overall Assessment of WMQRS
Respondents were asked to say whether their overall assessment of WMQRS was that it was working well and whether it was seen positively by their own organisation and, if the respondent was not in a provider organisation, by providers in general. Again there was a similar pattern across these responses (n=38).For these questions, more than any others, an appreciable group of respondents felt that it was too soon to make an assessment of WMQRS.
2.25Potential Change achieved by WMQRS in 3 Years
This question proved difficult for respondents as they found it difficult to separate out aspirations for WMQRS from genuine expectations about what it would achieve. For those respondents that actually answered this question it was answered generally and more briefly. The focus of the review team on specific services was seen as the vehicle for achieving change in those services.
2.26Should WMQRS itself be Changed?
Respondents were asked if there were aspects of WMQRS that they would change if they could (n=15).Almost half of those responding to this question suggested borderline changes.
2.27WMQRS Recommended to Other SHAs
Respondents were asked if they would recommend an agency like WMQRS to other SHA colleagues (n=18).This figure sums up well the broadly supportive response to WMQRS from our respondents. Almost all would recommend WMQRS though in some cases with some qualifications.
2.28What kind of Standards?
A small number of respondents raised issues about the level of standards or the height of the bar which units must “jump” to reach the standards. Key comments included use of terms such as “gold standard” or “gold plated standards” when the conventional local hospital might require a “silver” standard” of everyday practice.
2.29Who should implement the findings?
Several respondents questioned whether it was for individual trusts to implement findings, particularly where these affected groups such as doctors on rotation. There was also some concern that when a network is studied, as for urgent care, responsibility for implementing actions will need to be clearly understood by the different organisations involved.
2.30Who is Setting the Standards?
There was some concern with the breadth of the membership of steering groups, linked to the limited perceived contribution of primary care professionals to some groups. While it was acknowledged in several comments that it can be hard to get GP input to groups of this kind, due to the need to make payments for their time and their own need to arrange cover, nonetheless there is clearly a place for input from primary care.
2.31What kind of Process is WMQRS?
A small number of respondents raised questions about the approach of WMQRS and the balance between a peer-led process and an “external” inspection. A key issue for WMQRS is therefore the extent to which it wishes to continue to operate a policy of publication. The few respondents who commented on this felt that this placed WMQRS much closer to a formal external inspection body, even though its standards have been developed through a peer process and its inspections are carried out by peers.
2.32Findings - Document Analysis
2.33Thematic findings
The document analysis divided the documents into structural, procedural and outcome related documents. Those that were structural related to the architecture and development of WMQRS itself, procedural documents were concerned with review process and visitations and outcome documents were those that were generated from the work of WMQRS with collaborators. The documents included in the analysis were all those held by WMQRS including those from legacy organisations.
2.34Quality of reports
All reports provided a brief description of the context and the unit being reviewed and made some reference to those involved with the review.
The reports are clear in establishing what the review was trying to accomplish. Based on feedback during the interview stage, an area for development would be to make some reference to how standards were developed, perhaps indicating how many are nationally indicated and how many were developed locally by the network.
The improvement messages are clear in all the reports, what is not always evident are the implications for patients and risk. By prioritising the outcomes reports, the recipient providers and commissioners may better understand the importance of a particular identified need.
2.34.1Process
The review process both for gathering information and the methods used to assess problems are very well documented. The written instructions provided by the WMQRS team provide clarity to all participants and are extremely beneficial when combined with actual training delivered by the team.
2.34.2Implications
Quality improvement reports are increasingly referring to what healthcare providers have learnt from review processes and outcomes. An area for development could be to ask either the lead clinician or the entire reviewed team, how the outcome report has helped or changed their understanding of the problem?
2.34.3Change Strategy
The review process does not currently carry out follow up surveys as a routine measure. The review service does however routinely evaluate change three months after the final visit. This follow up visit does include whether changes have been made. WMQRS does undertake some targeted follow up for some programmes.
A follow up survey as part of an evaluation may help identify what changes were made and within what time period following the visit.
2.34.4Effects of Change
In order to help demonstrate the benefits to patients, reviewed services should be asked to identify how improvements affected patients? For example, if a waiting list initiative or discharge policy changed as a result of a visit, what the impact was on patient waiting times or any other aspect or patient experience.
2.34.5Next steps
As part of the change cycle, reviewed services could also be asked what they learnt from the change and how they will take the outcomes forward. Part of the commitment and participation in the scheme, should include some commitment to actually making change happen. Whilst this is inherent in the programme, this is not always obvious to the entire reviewed team.
2.34.6Summary cost effectiveness
Since its establishment, WMQRS has undergone some transformations in terms of status, operational structure and relationship to the SHA, in particular where the organisation is hosted. Throughout these stages, it managed to achieve its outlined objectives. Its activities were targeted to address the standards that were not being met. Specifically, it has proven itself successful in achieving goals related to knowledge transfer, health care provider service improvement, policy and procedure improvement at the organisation and network level, adoption of best practice and co-ordination of activities.
2.35Discussion