Amua Evaluation Tools

RMT Quality of Care tools: Mystery Client Survey

Monitoring Quality of Care through

Mystery Client Surveys

Guidelines

Guidelines for implementing the Mystery client Surveys

for Marie Stopes International Partners

Marie Stopes International

1 Conway Street – Fitzroy Square

London W1T 6LP - United Kingdom

Contents

1.Introduction

2.What is a mystery client survey?

3.When to use mystery client tools?

4.Limitations of mystery client surveys

4.1Recruitment can be difficult

4.2Dependent on mystery client recall

4.3Need to ensure reliability

4.4Information produced can be limited

5.How to conduct mystery client surveys?

5.1Planning for Mystery Client Survey

5.2Developing Mystery Client Tools

5.3Training for Mystery Client Surveys

5.4Data Collection

5.5Data Analysis

5.6Using the evidence from Mystery client surveys

Important things to remember when using the mystery client tool

Appendix 1: Sample Facility Consent Form

Appendix 2: Sample Mystery Client Consent Form

Appendix 3: Scenario Examples

Appendix 5: Example Report Template

1.Introduction

The quality of services provided ultimately affects the outcomes of Marie Stopes International (MSI) programmes. Activities and services that are of poor quality and not delivered according to minimum standards will have suboptimal results even if they do have high coverage. Monitoring the quality of MSI services - static centres, outreach teams, social franchise providers – is an integral part of our service delivery model to ensure that we are delivering the best possible services and that we doing no harm, and that we are on track to improve our programme outcomes – ultimately, increase uptake of long term family planning and safe abortion services. As soon as services are set up and functioning, the monitoring of their quality should become a routine activity, whereby the findings of the research are acted upon in order to ensure that quality standards and benchmarks are maintained.

There are a number of different tools that can be used to monitor and assess the quality and impact of services provided. MSI’s Medical Development Team (MDT) employs a process called Quality Technical Assistance (QTA), which is a comprehensive set of activities performed annually to identify and correct gaps in clinical aspects of service delivery. Country programmes should also plan for other activities to routinely monitor quality of care that are used for different purposes or to assess different aspects of service provision. These may include exit interviews that evaluate the satisfaction and perception of quality from clients’ perspective; the use of management information system (MIS) data to track service delivery performance; MSI Starscan which is a facility audit; or mystery client surveys (MSC) which is a qualitative facility based survey.

This guide explains what MSC are and the benefits of using this type of survey, and describes the steps to planning and implementing MSC for your country programme. It also presents a checklist to document the basic elements of quality for the service provision of a) long acting and permanent methods of contraception and b) provision of termination of pregnancy (MSP and MSMP). This checklist (to be completed by the mystery client after their visit) reflects a minimum standard that all service delivery channels (static centres, social franchise providers, outreach teams) should strive to achieve.

2.What is a mystery client survey?

A mystery client is when a trained person(usually a community member) poses as a client who visits a health facility without the service provider knowing that they are carrying out research and then report (by completing a survey or through an interview) on their experience. For example, an adolescent might be sent to a healthcentre looking for contraceptive services, and then be interviewed to find out about thequality of the visit.

Because the researcher is ‘under cover’ it is more likely that they will be treated as the provider usually treats most of their clients. In many types of research a phenomenon has been observed that when people are being observed or researched the quality of the activities they are doing improves. A mystery client survey minimises the problems of bias that are encountered in direct observation (where clinic staff tend to perform better then normal as they know they are being observed). Because quality is a key component of MSI’s service delivery, mystery clients surveys provide a useful means for program staff to get a picture of how providers perform when they are not being regularly supervised or knowingly observed.

3.When to use mystery client tools?

The use of mystery client surveys is primarily for monitoring a service provider’s (centre, outreach team, social franchise provider) performance and improvement, rather than as an evaluation tool. It can also be used as an evaluation tool to measure the impact of a programme or intervention on the quality of services offered. For MCS to be used as one off evaluation tool it is necessary to have baseline information or an agreed upon benchmark, so results (and improvements) can be measured against. Information from the mystery client is fed back to programme managers, and in turn the provider, so they can improve upon their service provision.

It is recommended that clinics are involved in the planning of the mystery client visits, in addition to being informed of the potential visits (though they will not be informed which clients are the mystery clients), to increase the use of the findings (and thus improve service provision where necessary).

Because mystery clients are used to assess and improve the quality of services, they serve the interests of both clients and the programme by highlighting the ways that the facility can better serve its target audience. Mystery clients should not undergo any type of medical exam or procedure during their visits.

4.Limitations of mystery client surveys

Despite the many advantages of mystery client surveys as described above, there are few limitations that need to be considered prior setting up a mystery client survey:

4.1Recruitment can be difficult

Mystery clients are ideally community members (speak same dialect, wear familiar dress etc) that should not be recognised by the centre staff or service provider, and finding community members that will not be recognised in a small community could be difficult. This is of particular challenge for franchisees who are often located in small rural settings where they “know” all of their clients. It maybe possible to recruit an appropriate “researcher” from within the same district / locality, whose scenario and character accommodates for this problem (e.g a visiting relative, a migrant worker etc).

4.2Dependent on mystery client recall

Mystery clients will complete a short questionnaire to report on their visit following their facility visit, and thus rely on their ability to recall facts and information they saw or experienced. (i.e recall bias). Therefore it is important that the tool is quite short and focuses on a select few key issues.

4.3Need to ensure reliability

Reliability refers to the extent to which an experience or any measuring procedure yields the same results on a repeated trial. This means that the interview / survey results would be the same (as much as possible) if two mystery clients went to the same clinic and experienced the same interaction. The mystery client tool is fairly subjective.

For example, the researcher scores on a scale from 1 to 5 different aspects of the service provision. Even if two researchers were observing the same phenomenon they might give a different score. If one of the aspects is whether staff were courteous, one researcher might mark a person as 3 and another mark them as 2 even though they were greeted in the same way. Therefore to ensure reliability it is not only important to have the same researchers doing the mystery client survey, but also that mystery clients are given instruction on how to rate the facility with clear criteria for each of the different ranks (numbers) given to the elements they are assessing. This all needs to be part of a rigorous training for the mystery client researchers, whereby the researchers can discuss and agree on these criteria.

4.4Information produced can be limited

As mystery clients do not undergo any medical procedure, they cannot collect information on provision of these services. The types of areas of quality assessment suitable for a mystery client survey include an assessment of the services and equipment available, the consultation process, provision of correct information, courtesy of staff, and quality of the surroundings. Note that information on provision of procedures, can be collected through the method of exit interviews with “real” clients.

5.How to conduct mystery client surveys?

The process for using mystery clients follows the same general process as is followed for other research: planning, developing instruments, training mystery clients, data collection, data analysis, use and dissemination of findings. More detailed steps are given below:

For more support and advice regarding the planning of you Mystery Client Survey contact you Regional research manager.

5.1Planning for Mystery Client Survey

Good planning prior implementation of the exit interviews is essential to ensure there is appropriate budget and resources to produce timely, quality data that is useful for programmes as well as the partnership.

It is important to consider at the start of planning about how the data provided by mystery client surveys will be used? For instance, if the purpose of the survey is to monitor individual service provider’s performance whereby the results are reported back to improve their quality of services, then the survey should include all service providers in the programme. If a programme wants to use mystery client surveys as an evaluation tool to measure the impact of a programme on the quality of contraceptive services offered to a target population, then a sufficient sample of the same type of service providers is sufficient. However, to show an impact such as an increase in “the number of franchisees correctly counsel clients about contraceptive methods, side effects and medical eligibility”, then you will need to have a baseline to measure progress against.

The following steps are to be used as a guide to help in planning MCS for your programme:

Budget: MCS can be implemented internally or contracted out, and should not incur too much cost. Depending on the country and size of programme, and number of facilities to be monitored at one time the MCS should not exceed more US$1,000-3,000. As MCS use a small number of “researchers”[1]the budget will be mainly covering their cost, travel and per diem, as well as money to pay for the consultation services. Other expenses will include data entry, data analysis, report writing and dissemination.

Sampling of facilities:MCS are also a qualitative method and therefore do not require a robust sample size calculation. However, to ensure results are representative of similar facilities (static centres, outreach teams, social franchise providers) it is important that a sample is stratified to reflect the different types of providers and services provided. Categorise the providers according to type, such as for social franchising: clinic owner versus pharmacist / drug sellers; or client load; or type of servicesprovided (FP versus MSP /MSMP); or location (e.g. different regions, or districts, towns).

If MCS are a part of routine monitoring of services to provide information about one facility that can then be used to assess its performance and identify areas for improvement , ideally all facilities should receive at least one mystery client visit during a specified time period (e.g annually). This could be staggered with one quarter of providers surveyed in the first quarter, the next quarter surveyed in the second quarter and so on. This will of course depend on the available budget. If it is not possible to survey all facilities, such as for social franchising programmes that have a substantial number of providers within the network it may not be possible to conduct mystery client surveys for all providers within a short time period. Below is a guide to sampling different facilities for MCS during one round of the survey which should then be stratified by type of provider as suggested above.

Table showing how to select total number of facilities to sample for mystery client surveys
Type of facility / Number of facility in country / Total number of facilities to sample
Static Centre / < 10 / ALL (if >1 in country)
10 - 39 / Minimum of 3
40 - 99 / Minimum of 4
≥ 100 / Minimum of 5
Outreach / < 10 / ALL (if >1 in country)
10 - 39 / Minimum of 3
40 - 99 / Minimum of 4
≥ 100 / Minimum of 5
Franchisee / < 10 / ALL
10 - 39 / Minimum of 20
40 - 99 / Minimum of 30
≥ 100 / Minimum of 40

Nb. These are minimum numbers and the greater number of facilities included in the sample the more representative this will be of the programmes service delivery.

Once you have identified the sample of facilities you will then need to obtain written consent from the facility’s manager (see appendix for “sample centre consent form”)

Recruiting mystery clients: To determine the number of mystery clients you will require depends on the number of scenarios that you want to depict. For instance you may want to assess the quality of services for long term family planning or for safe abortion counselling and require a young woman beginning child bearing years, as well as a woman aged 30-40 years wanting to limit or stop child bearing etc. The number of mystery clients will also depend on a) the geographical coverage of the survey (cultural differences and local dialects may also need to be considered) and b) the total sample of facilities to be assessed. If the MCS had a national coverage then it may be wise to select mystery clients per region and / or district and so on and each will visit x number of facilities. It is advisable for a single mystery client to no do more than 10 visits.

Once you have decided your number of scenarios and number o mystery clients, you will need to identify potential “researchers”. Although the cornerstone for this approach of research is to ensure that the mystery clients are not recognised by the people they are assessing, ideally mystery clients should be people from the community where the facility is located, so they are more likely to not stand out in terms of dress code, dialect etc. When identified then ensure you obtain consent from each mystery client (see appendix for a “sample mystery client consent form”).

Develop a field work schedule of visits to selected facilities

Whether ethical review is required for this type of research will vary from country to country. All programmes are advised to contact the appropriate Ethics Committee at the Ministry of Health to obtain approval to conduct this research. It is important to check whether this is required as MSI may in the future wish to publish the results of the MCS to share the findings with a wider audience than just MSI at country level.