Appendix 2: Delphi Panel Methods

The methods described below provide detailed information related to the Delphi Panel project conducted in June-September 2011 by United BioSource Corporation of Chevy Chase, Maryland (biosource.com). The goal of this project was to gather expert opinion related to effectiveness of PCV13 and PPSV23 in adults. Effectiveness estimated generated as part of the project were used as data inputs to the cost-effectiveness model described in the accompanying manuscript.

Introduction: Rationale for Delphi Panel

Although vaccine effectiveness is a key economic model parameter, there are neither any known data that compare the two vaccines in an adult population, nor are there any placebo-controlled trials evaluating the efficacy of PCV13 in adults against either invasive or non-invasive disease of PCV13. There are several clinical trials and observational studies that have evaluated the efficacy and effectiveness of PPSV23 in preventing both invasive and non-invasive disease, as well as meta-analyses of these data. The data are consistent regarding efficacy for invasive disease; however, there are inconsistent meta-analysis results and interpretations regarding PPSV23 efficacy for preventing non-invasive disease. Despite these data gaps, decisions regarding regulatory approval for PCV13 as well as adult vaccine schedule recommendations regarding the use of pneumococcal vaccines will be forthcoming this year. Table S1 describes known data available in the literature related to pneumococcal vaccine effectiveness as well as where expert opinion will be sought.

Table S1: Pneumococcal Vaccines: Data Available in Literature for Estimating Pneumococcal Vaccine Effectiveness for Use in Economic Models

Efficacy/Effectiveness data / Existing Data / Need Expert Opinion
IPD / NBPP
PPSV23 vs. Placebo
Adult / Yes / Yes / No / No
Pediatric / No / No / N/A / N/A
PCV7 vs. Placebo
Adult / No / No / N/A / N/A
Pediatric / Yes / Yes / N/A / N/A
PCV13 vs. Placebo
Adult / No / No / Yes / Yes
Pediatric / No / No / No / No
PCV13 vs. PPSV23
Adult / No / No / Yes / Yes
Pediatric / No / No / No / No
Immunogenicity data / Existing data
Conjugate (PCV7, PCV13) vs. Placebo / Yes
Polysaccharide (PPSV23) vs. Placebo / Yes
Conjugate (PCV7/PCV13) vs. Polysaccharide (PPSV23) / Yes

Methods: Delphi Panel

Given a paucity of direct comparisons of the effectiveness of PPSV23 and PCV13 published in the literature, meta-analytical techniques to generate effectiveness estimates are not feasible. Therefore, a Delphi panel approach was implemented to best leverage existing information.

The primary objective of the Delphi study was to investigate the estimated effectiveness of the PPSV23 and PCV13 vaccines across different age and risk strata. Secondary objectives were related to understanding expert opinion regarding published estimates of pneumococcal disease incidence in adults and the potential for pediatric immunization to impact adult pneumococcal disease incidence due to herd immunity. Incidence of disease and the potential indirect effects of pediatric vaccination on adult incidence are key economic inputs for the economic model.

The Delphi technique was used to obtain and synthesize the opinions of experts with available evidence and build consensus among panel members by means of a structured process that uses a series of questionnaires and controlled opinion feedback. All aspects of the Delphi panel approach were executed according to the approved Delphi panel protocol.

Answers to four research questions were used to address the primary and secondary research objectives. The primary objective of the Delphi panel, to obtain effectiveness estimates, was addressed by the research questions 1 and 2 (RQ1 and RQ2), while secondary research objectives, related to incidence of pneumococcal disease among adults and indirect effects of pediatric immunization, were addressed by research questions 3 and 4 (RQ3 and RQ4).

Study Design

The Delphi panel implementation was carried out in six steps, described below.

  1. Identification of panel participants
  1. Enrollment of panel participants
  2. Design of questionnaire (described below)
  3. Iteration 1 of Delphi panel
  1. Administration of questionnaire
  2. Collection of data
  3. Analysis of data, and generation of brief summary report
  1. Iteration 2 of Delphi panel
  1. Dissemination of brief summary report from Iteration 1, and review using an online meeting forum. Online meeting was undertaken in a 1.5-hour period.
  1. Administration of questionnaire
  2. Collection of data
  3. Analysis of data and generation of brief final summary report
  1. Iteration 3 of Delphi panel
  1. Administration of questionnaire
  1. Collection of data
  2. Analysis of data and generation of brief final summary report

Identification of Panel Participants

United BioSource Corporation (UBC) identified US-based experts through publication records, conference attendance and presentations, as well as National Institute of Health grant funding. MEDLINE (as accessed online through PubMed) was used to identify those individuals who have peer-reviewed publication records relevant to pneumococcal disease and its outcomes particularly relevant to adults.

Individual experts were also identified by UBC on the basis of their publication records in the proceedings of annual meetings of relevant infectious disease associations, with the rationale that such proceedings represent the work of individuals most recently active in the field, and most likely to have a relevant body of knowledge.

All experts signed a confidentiality disclosure agreement prior to participation.

Members of the US Government and the Advisory Committee on Immunization Practices (ACIP) members were excluded to eliminate any potential conflicts of interest.

Enrollment of Panel Participants

Upon identifying potential participants, UBC sent a recruitment letter to the individuals, requesting their participation in the Delphi panel. The letter described the Delphi technique, the reimbursement offered in compensation for time contributed, a description of next steps to either decline or accept the invitation, and finally a request to the potential panel member to nominate other experts they would consider eligible for participation in such a panel. Forty potential participants were invited to participate and were identified based on their publication record, recent participation in conferences of the National Foundation for Infectious Diseases (NFID) or Infectious Diseases Society of America (IDSA), and through a search of the National Institutes of Health grant recipients in this field. An additional five participants were nominated by other expert panel members. Members of the panel were remunerated for their time.

UBC exclusively was involved in the selection of Delphi panel members. Merck was not involved in the selection of Delphi panel members. Delphi panel members were informed prior to their engagement that a pharmaceutical company was sponsoring the research; however, the specification of Merck as the study sponsor was not revealed. Furthermore, the implementation of the Delphi panel was managed exclusively by UBC. No Merck employees were involved in any capacity.

An expert panel size of 10 experts was the pre-defined target for enrollment, which is consistent with published suggestions regarding a sufficient number of participants from a particular reference group (Delbecq,1975). To achieve that end, 11 experts were engaged in the Delphi process, with the anticipation of attrition over the course of the study. One expert terminated their involvement in the process following the receipt of the first questionnaire.

Questionnaire Design

The questionnaire was based on the initial questions posted by the project team and by gaps made evident from the evidence-based review of the literature performed by UBC. The questionnaire was comprised of a series of questions related to the research objectives described. Panel members were also asked to provide a rationale for their answers in order to provide UBC with a deeper understanding of the data sources that are most critical in informing expert opinion and to provide some insight into how perceptions may change as new data become available.

In order to obtain answers specific to key age and risk strata, the following groups were used: adults aged 19–49 years with high risk conditions, 50–64 years not at high risk, 50–64 years at high risk, 65 years and older of all risk levels. The age and risk strata were defined based on the current pneumococcal vaccine recommendations (CDC 1997).

For each research question, Delphi panel members were presented with a series of evidence tables summarizing key scientific data related to the question. Respondents were then asked to complete the questionnaire using their knowledge and experience and a review of the relevant materials in the provided in the evidence tables where necessary.

For RQ1 and RQ2, UBC presented evidence to the experts that included information on immunogenicity related to PCV7 or PCV13 and PPSV23 vaccines as well as effectiveness and efficacy data for PCV in a pediatric population.

RQ1. What is the estimated effectiveness of PCV13 and PPSV23 in the prevention of NBPP, according to specific age and risk strata?

The following questions aimed to address research question 1.

Question 1.1 Based on available scientific evidence, would PCV13 be expected to be effective in the prevention of NBPP caused by vaccine serotypes in each of the age and risk groups? (Yes/No) Why? If yes: Please provide an estimate and range for expected effectiveness in % prevention. Please provide a rationale for your answer.

Question 2.1 Based on available scientific evidence, do you believe that PPSV23 is effective in the prevention of NBPP caused by vaccine serotypes in each of the age and risk groups? (Yes/No) Why? If yes: Please provide an estimate and range for effectiveness in % prevention.

Please provide a rationale for your answer.

RQ2. What is the estimated effectiveness of PCV13 and PPSV23 in the prevention of IPD, according to specific age and risk strata?

The following questions aimed to address research question 2.

Question 3.1. Based on available scientific evidence, would PCV13 be expected to be effective in the prevention of IPD caused by vaccine serotypes in each of the age and risk groups. (Yes/No) Why? If yes: Please provide an estimate and range for effectiveness in % prevention.

Please provide a rationale for your answer.

Question 3.2. Do you concur with published scientific evidence documenting significant PPSV23 effectiveness against IPD in adults? If no, please provide an estimate and range of expected effectiveness of PPSV23 against IPD in adults in % prevention.

Question 3.3 Based on current scientific evidence, will PCV13 have greater/equal/lesser effectiveness for IPD against serotypes contained in both vaccines as compared with PPSV23? Please select “lesser effectiveness,” “equal effectiveness,” or “greater effectiveness” for each of the age and risk groups.

Please provide a rationale for your answer.

Additional Questions: Gaps identified by the literature review and synthesis

In addition to addressing the four primary research questions, some additional questions were presented in the questionnaire that aimed to fill gaps highlighted by the literature review and synthesis. The following questions were included in the questionnaire.

Question 4.1. If you were a clinician or policy maker and you had PCV13 and PPSV23 available to you, what would be the optimal immunization schedule for greatest public benefit considering both NBPP and IPD? Please provide the number of doses and dosing interval for optimal effectiveness for each of the age and risk groups.

Question 4.2 Based on current scientific evidence is there a correlation between immunogenicity and effectiveness?

Merck was provided the opportunity to review preliminary drafts of the questionnaire to ensure that appropriate content was included and the essential data for inclusion in the economic model would be obtained through the Delphi panel. UBC made all final determinations regarding the wording of questions to ensure that they were asked in an objective manner.

Data Analysis

Consensus Definition

The goal of the Delphi panel was to reach consensus. A unanimous decision, however, was not necessary to reach consensus. For the numeric responses, consensus was considered to be reached when 70% of responses were within one standard deviation of the mean response. For each appropriate question, a determination of consensus among the responders was determined; it should be noted that there may still remain dissension among the panelists even if consensus was attained and alternatively that there may be levels of agreement among the panelists that are not reflected in the consensus value.

For quantitative answers, such as those required for RQ1 and RQ2 regarding effectiveness of vaccines, point estimates and measures of variation, median and range, were generated. Mean and standard deviations were also calculated to provide measures of consensus (defined as 70% of responses falling within 1 SD of the mean response). The point estimates are to be used as base case assumptions in an economic model that will estimate incremental cost-effectiveness ratios for the vaccination of adults with PCV13 compared with PPSV23, and ranges will be used in sensitivity analysis. In addition, qualitative analysis of text responses addressing each of the RQ captured commonly expressed themes and observations.

The qualitative data obtained from the open-ended questions in the questionnaire were reviewed by UBC scientists to identify key themes and important concepts raised by the participants. UBC generated compilations of participant quotes organized by theme or concept. Consistently documented themes and concepts were identified, as well as themes and concepts described with less frequency.

References

CDC. Prevention of pneumococcal disease: recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Recomm Rep. Apr 4 1997;46(RR-8):1-24.

Delbecq A, Van de Ven A, Gustafson D. Group techniques for program planning. Glenview, IL: Scott Foresman and Co.; 1975.

- 1 -