Item 5
Paper No: CM/01/13/04
Developing a strategic framework to guide
the Care Quality Commission’s
programme of evaluation
REPORT
Kieran Walshe and Denham Phipps
January 2013
Item 5
Paper No: CM/01/13/04
Executive summary / 11 / Introduction
Background and context
Analysis of the regulatory model
Learning from research
Comparisons with other regulators
Using existing data
Structure of the report / 2
2 / The regulatory model: an analysis
Introduction
Regulatory mission and purpose
Registration
Standard setting
Information gathering and risk assessment
Inspection and reporting
Enforcement
Information provision
Reviewing the regulatory model
/ 7
3 / Differentiation in regulatory design
Introduction
Learning from research
Comparisons with other regulators
Using existing data
Conclusions / 23
4 / Regulatory standard setting
Introduction
Learning from research
Comparisons with other regulators
Using existing data
Conclusions / 33
5 / Risk based regulatory approaches
Introduction
Learning from research
Comparisons with other regulators
Using existing data
Conclusions / 43
6 / The competencies of the regulatory workforce
Introduction
Learning from research
Comparisons with other regulators
Using existing data
Conclusions / 55
7 / Conclusions / 66
References / 68
Item 5
Paper No: CM/01/13/04
Executive summary
This report sets out to show how evidence and research can be used by CQC to evaluate how well its current regulatory arrangements in health and social care work, and plan future changes in those arrangements to improve their effectiveness and efficiency. We are very thankful for the help, advice and support we received from many CQC staff in its preparation. Of course, we remain responsible for its content and conclusions, and for any errors or omissions.
We develop a “logic model” of the current regulatory arrangements which tries to map out how they are intended to work, and what assumptions are made or consequences flow from those arrangements. We conclude that as currently configured, CQC is a “safety-net” regulator, focused on dealing with poor performance but with limited capacity or capability to drive or support wider performance improvement. We note that even in those terms the current regulatory model has some inherent problems, but that the direction of future strategy signals a shift towards a more ambitious interpretation of CQC’s remit and purpose, which has important implications for the regulatory design/arrangements.
We then discuss four areas – differentiation in regulatory design, standard setting, risk-based regulation, and the regulatory or inspection workforce – and look at what evidence is available from several sources about how current regulatory arrangements work. We conclude:
· CQC’s generic regulatory model is unusual (compared with other regulators) and hard to make work. There are good reasons to consider greater differentiation between sectors (such as NHS/healthcare and adult social care) and within large sectors like adult social care. This might mean the development of more specific and tailored standards and guidance, and greater specialisation among the inspection workforce.
· CQC’s current standards are largely construed as an essential or minimal level of performance. CQC could within its existing legislative and regulatory framework create a more differentiated, demanding and service specific set of standards, and it could consider making more use of standards developed in or with the sectors it regulates.
· CQC had used a risk-based model of regulation, in which it adjusted its use of regulatory interventions like inspection with providers based on an assessment of risk and performance, but has recently returned to a universal schedule of annual inspection in most sectors. We find that even modestly proportionate or risk-based regulation requires a strong and stable database of performance data which has clear predictive validity, and a graduated range of regulatory interventions short of full inspection.
· CQC’s use of a generic inspection workforce is, as far as we can see, not emulated by other regulators and is problematic for several reasons. Regulatory staff need content expertise in the area they regulate, methods expertise in the regulatory system, and interpersonal or behavioural expertise in dealing with people and organisations in sometimes difficult and contested circumstances. Specialisation has many advantages, and investment in the development of the inspection workforce is worthwhile.
We conclude that CQC could make more use of evaluations than it has in the past both when introducing innovations in its regulatory arrangements so that they are properly tested and in the routine working of its regulatory arrangements so that it has ongoing evidence of their effectiveness and impact.
Chapter 1
Introduction
Background and context
This report is the result of a short research project commissioned by the Care Quality Commission (CQC) from the University of Manchester. Its aim was firstly, to help CQC to bring evidence to bear on examining and exploring the effectiveness and efficiency of its current regulatory arrangements; and secondly to help CQC to develop its own internal capacity to undertaken and use research and evaluation so that, going forward, it can make better use of opportunities to test, trial and assess regulatory changes and innovations, improve its efficiency and effectiveness, and gain a more robust understanding of its impact on the quality of health and social care in England.
Analysis of the regulatory model
Our first step was to develop and test a “logic model” which mapped out the underlying “programme theory” for each major component of the regulatory arrangements. In this approach we drew both on established methods for programme theory explication (see for example Bickman 1987; Rogers 2008) and recent developments in realist or theory driven evaluation (see Pawson and Tilley 1997; Pawson and Manzano-Santaella 2012; Marchal et al 2012). The purpose here was to make explicit the assumptions or presumed mechanisms by which these regulatory components bring about change in regulated organisations. In our experience, there may be multiple and sometimes contradictory theories in use, and part of the value of mapping out the programme theory is to express these alternatives in forms which then make them testable, and which bring to the surface areas of contestation or inconsistency.
We developed the logic model around the main statutory functions of CQC – registration, compliance, enforcement, and information provision. For each function, we used documents and interviews with CQC staff to explore the mechanisms at work, and to try to understand the regulatory model and the choices and consequences it represents. We were acutely conscious that this was a time of change within CQC, and the organisation had in September 2012 published a strategic review on which it was consulting stakeholders. That strategic review signalled important, even fundamental changes to the regulatory model, and so in our work while we focused on mapping the current regulatory model we tried to take explicit account of the likely future direction as well.
As part of this work, we sought to establish what were the areas where research and evaluation might make an important difference to CQC’s decision making – where there was significant uncertainty about how (or how well) the regulatory model worked, and where change seemed likely. On that basis we chose four topics for further research:
· Generic versus differentiated regulatory standards and processes – whether regulators use the same methods, standards and processes for organisations in different sectors or of different types or whether and how they differentiate and use different standards or processes for different organisations.
· How regulatory standards are set and measured – issues like whether standards are minimal, median or maximal, how they are measured, whether compliance/achievement is measured dichotomously or on a scale, how compliance is defined and how a threshold for acceptable performance is set, and whether different standards are used/applied for different organisations.
· Risk based or proportionate regulation – to what extent regulators try to make the regulatory process responsive to organisational performance, and so focus more attention on organisations which perform less well or represent greater risk, and how this is done
· The regulatory or inspection workforce – what competencies are required, what kind of people are used to undertake regulation and inspection, how are they recruited and trained/developed, how is their performance appraised.
In each of these four areas, we set about drawing together what was already known from the research literature; comparing practice at CQC with four other regulators; and exploring what was known or might be found from using existing data sources within CQC.
Learning from research
We looked for existing research evidence in the four topic areas using a wide range of bibliographic databases. The general strategy for the literature review was to identify relevant literature in public administration, healthcare, management and safety science, and where appropriate, governmental policy documents. The researchers searched the MEDLINE (1946-2012), EMBASE (1980-2012), ABI Inform (1971-2012), ASSIA (1987-2012), British Nursing Index (1993-2012), Social Services Abstracts (1979-2012), and HMIC (1979-2012) databases. For the regulatory workforce topic, PsycInfo (1806-2012) was also consulted. The keywords used for each topic were as follows:
· Differentiation of models: regulat* model*
· Regulatory standard setting: regulat* AND standard*; ("nursing home" or "care home") and regulation
· Risk-based regulation: risk based regulat*
· Regulatory workforce: regulat* AND inspect* AND (train* OR compet*)
In addition to the search of academic databases, public domain repositories (for example, government and NHS websites, and relevant professional organisations) were consulted. Also, the authors’ previous work on related topics, and the reference lists of retrieved articles, were consulted for relevant material.
Comparisons with other regulators
We identified four other regulatory agencies with whom to compare CQC in the four topic areas. We chose two healthcare regulators – the Joint Commission for the Accreditation of Healthcare Organisations in the United States and the Inspectie voor de Gezondheidszorg (the Healthcare Inspectorate) in the Netherlands. We chose two UK-based non-healthcare regulators – the Office for Standards in Education, Children’s Services and Skills (OFSTED) which regulates children’s services, particularly schools, and the Homes and Communities Agency which regulates social housing providers. Our aim was to provide a range of examples of regulatory policy practice, in both the health and social care setting and elsewhere. Some important characteristics of the comparator regulators – their overall remit/purpose, their size and scale, and who and what they are responsible for regulating – are set out in table 1.1. We gathered data about the comparator regulators through a review of published and unpublished documents, and interviews with a member of staff from each regulator. The interpretation of the comparator regulators’ arrangements contained in this report is, of course, ours rather than an official statement of the agencies concerned.
The table illustrates that the four regulators vary in some important ways, though they have much in common. Their statements of regulatory remit/purpose – taken from their own documents such as annual reports – generally all focus on performance improvement though some are more ambitious than others. They range in scale from about 120 staff to 1,400 or more, and in annual turnover from £48 million (€55 million) to £167 million pa. Most though not all are public/state organisations (JCAHO is a private not-for-profit foundation). All regulate thousands of organisations, and all but HCA regulate across multiple sectors or service areas.
Table 1.1. Comparing CQC with four other regulators
Name / Care Quality Commission / Joint Commission for the Accreditation of Healthcare Organizations / Inspectie voor de Gezondheidszorg (Dutch Healthcare Inspectorate) / Office for Standards in Education, Children’s Services and Skills / Homes and Communities AgencyAbbreviation / CQC / JCAHO / IGZ / OFSTED / HCA
Sector / Health and social care / Healthcare / Healthcare / Childrens services / Social housing
Country / England / USA / Netherlands / England / England
Organisational form / Non Departmental Public Body / Private not for profit foundation / Part of Ministry for Health, Welfare and Sport, but partially independent / Non ministerial government department / Non Departmental Public Body
Formal regulatory remit/ purpose / to “protect and promote” the health, safety and welfare of service users and the “general purpose of encouraging the improvement of health and social care services” / “To continuously improve health care for the public, in collaboration with other stakeholders, by evaluating health care organizations and inspiring them to excel in providing safe and effective care of the highest quality and value.” / “Promotes public health through effective enforcement of the quality of health services, prevention measures and medical products.” / “inspects and regulates to achieve excellence in the care of children and young people, and in education and skills for learners of all ages, thereby raising standards and improving lives” / “focus of our activity is on governance, financial viability and value for money as the basis for robust economic regulation; maintaining lender confidence and protecting taxpayers”
Annual turnover / £149 million / $165 million+ / €55 million / £167 million / £55 million (whole of HCA – regulation function not separated)
No of staff / 1,885 / 800 / 500 / 1,400 plus contracted staff / 120 in regulation function
Type of organisations regulated / Health and social care providers / Hospitals, long term care, behavioural healthcare, labs, homecare, ambulatory care / Curative healthcare, long term care, public health, pharmaceuticals and medical devices, health professionals / Schools (maintained and independent), further education, adult learning, early years/ childcare, childrens homes, childrens social care, adoption/ fostering / Registered social landlords – incl LAs, housing associations and for-profit providers
No of organisations regulated / 22k in health, social care / 20k / 3k organisations, 800k professionals / 23k schools and about 100k other providers / 1,500 though 400 large RSLs are 90% of provision
Using existing data
We were keen to see whether existing data held by CQC, particularly that produced for or as a byproduct of the regulatory process, could be used to understand current practice in CQC in the four topic areas. To that end, we met with CQC staff responsible for intelligence and data analysis, to get a better understanding of data availability and they undertook some analyses which we draw on in this report by way of examples, where possible. Our aim here has been to establish the principle that existing or routine data can be used to serve the purposes of evaluation, either retrospectively or prospectively.