Appendix 1. Interview Guide
PCORI Interview Protocol
Developers/Researchers
Interview objectives
1. To explore the perceptions and experiences of people who have previously developed decision aids and have involved potential users (patients, clinicians, others) in diverse ways.
2. To develop ideas of how to improve development processes.
Procedure
q Connect and start the recorder
q Introduce yourself: “Hello, this is [name] from Laval University in Quebec City, Canada. I’m calling you as we had agreed. Is this still a good time for us to talk?”
q Remind person of goal of study: “Thank you for your willingness to talk to us today. As we said in our email/letter to you, the goal of this interview is to find out more about your experience in the project you led/participated in with [name of contact] in [year] about [topic, e.g., development of name of decision aid]. There are no right or wrong answers to any of these questions. We hope that you will be candid and honest with your responses, we would really like to learn from you and your experience. Does this sound OK so far to you?”
q Obtain participants’ informed consent. Key points:
q “As you know, this is a research study. We’ve already emailed you a copy of the full consent form, the questions we will be asking you, and what we understand about your development process from your papers. This conversation will take about an hour. If there are any questions that you don’t want to answer, you don’t have to answer them. If you need to leave for any reason, just let me know and I will stop. If we stop early, you can decide whether it’s ok if we keep the data we’ve already collected, or if you’d rather we discard it. Everything you say is confidential. We won’t be identifying particular projects in our reports. Is this OK with you?”
q “We are going to be audio recording this phone call, then transcribing it verbatim, taking care to remove any identifying information at that stage. After transcription, we will destroy the recording. Is it OK with you if we record this call?”
q “We will keep a separate file of your name and contact information for 7 years. We will keep this in a locked file cabinet in our offices at the university. Is that OK with you?”
q “Just a reminder that we would like to compensate you for your time. At the end of the interview, we will take your mailing address to send you a check. If you have to stop the interview early, we can take your address then.”
q Do you have any questions or comments for me before we get started?
q Conduct the interview
q Collect mailing address, record in a separate file to send compensation.
START OF THE INTERVIEW
1. To begin with, please tell me about yourself/your team and the impetus for [name of project/decision aid]. Why did you create this decision aid?
2. Tell me a little about the approach you used in the development of [name of decision aid]:
QUESTIONS (make sure to ask each item not covered spontaneously)
- Underlying model or framework that guided the process
- Mix of people, who was involved
- Process used
- Financial or material resources to support the process
- About how long did the development process take?
3. We’d like to talk a little about potential users involved in the process. When I say ‘users’ I mean people like patients who might someday use the decision aid to help them make a decision, or clinicians like physicians or nurse practitioners who might someday offer or recommend the decision aid to their patients. Can you tell me a little about the selection and participation of users in the development process?
QUESTIONS (make sure to ask each item not covered spontaneously)
- Who was involved
- How were they identified or selected?
- Were patients involved? If so, how?
- DEPENDING ON DECISION also ask about caregivers, surrogates, families.
- Were clinicians who are not decision aid researchers involved? If so, how?
- Was this the first time you involved users in this way?
[If needed, draw on extracted data to help inform this question]
4. Now if we think specifically about the users who were involved, we’d like to talk a little more specifically about their involvement.
QUESTIONS (Make sure each one is addressed)
- First of all, do you feel like you had a group of users that provided all the needed perspectives? Do you feel like there were any particular perspectives that were missing from the process? What made it easy or hard to get the particular perspectives that you needed?
- What do you see as the main barriers to user participation in the development process? (Prompts: recruitment, funding, institutional review boards, etc. Feel free to bring up things previous interviewees have said.)
- What do you see as the main facilitators to user participation in the development process? (Prompts: patient advisory boards, funders that require it, patient support groups, etc. Feel free to bring up things previous interviewees have said.)
- Was users’ feedback incorporated into your decision aid? If yes, how did you incorporate it? If no, what made it hard to incorporate?
5. [if identified as vulnerable populations] Your study was identified as one of the studies that included users who may be from socially or economically disadvantaged populations. In our quantitative analyses we found the following differences between studies that did and did not involve people from such populations: [describe differences]. I’m wondering if you can comment on those differences? To what extent do these findings reflect or fail to reflect your own experiences in this project?
6. [if users involved] Was there any sort of preparation or training for you or for the rest of the team to help everyone work and communicate with users?
PROMPTS
- If yes, please explain
- If no, do you wish there had been?
7. Thinking back to the way the project unfolded, what made it hard or easy to have patients and other potential users involved?
PROMPTS
- What, if anything, made it hard to havethem involved in the way they were?
- What, if anything, made it easy to havethem involved in the way they were?
8. Do you feel that user participation had an impact:
QUESTIONS (Make sure each item is covered. Ask if not covered spontaneously.)
- On the process?
- On the decision aid that was produced [name of decision aid if available]?
- On how much the decision aid did or did not help the target population?
- On their experience as a patient/caregiver/surrogate/person who might someday make this decision?
- On the attitudes of the other people involved?
- Directly or just by being the room?
- On your experience as a developer/researcher?
9. Today, what are your feelings about involving users in the development of tools like [name of decision aid]?
QUESTIONS (make sure to ask each item not covered spontaneously)
- Would you do it again the same way?
§ PROMPTS:
· What would you keep the same?
· What would you do differently?
10. Lessons learned “if you were talking to a group of colleagues doing this kind of work, what would you advise and why?”
QUESTIONS (make sure to ask each item not covered spontaneously)
- Identification of users
- Recruitment of users
- Numbers and mix of people
- Preparation or training
- Involvement (type, level and timing)
- The particular experience or perspective that users bring
- Incentives or other supports to participation
11. Do you have any other thoughts that you would like to share?
Thank you so very much for making the time to talk to me.
Appendix 2. Included Projects
Article(s) / Did the project specifically involve members of vulnerable populations? / Reason(s) people included or involved may be vulnerable / Clinical context of patient decision aidAllen et al., 2009[28] / Yes / Race and Ethnicity / Prostate Cancer
Anderson et al., 2011[29] / No (but they were included within a diverse population) / Older Age / Ovarian Cancer
Arterburn et al., 2011[30] / Yes / Other: Obesity / Obesity
Au et al., 2011[31], Lam et al., 2013[32] / No (but they were included within a diverse population) / Lower Literacy / Breast Cancer
Auvinen et al., 2004[33] / No members of vulnerable population(s) reported / Prostate Cancer
Bailey Jr. et al., 2012[34], Bailey Jr. et al., 2013[35] / No (but they were included within a diverse population) / Race and Ethnicity; Lower Education or Income / Fragile X Syndrome
Barnieh et al., 2011[36] / No members of vulnerable population(s) reported / End-Stage Renal Disease
Barry et al., 1995[37], Barry et al., 1997[38], Piercy et al., 1999[39], Rovner et al., 2004[40], Holmes-Rovner et al., 2006[41], Wills et al., 2006[42], Pylar et al., 2007[43] / No (but they were included within a diverse population) / Older Age; Lower Education or Income / Benign Prostatic Hyperplasia
Bass et al., 2013[44] / Yes / Race and Ethnicity; Lower Education or Income; Lower Literacy / Colorectal Cancer
Bastian et al., 2002[45], McBride et al., 2002[46] / No members of vulnerable population(s) reported / Reproductive Health; Breast Cancer
Becker et al., 2009[47] / Yes / Other: Mobility Impairments / Reproductive Health
Bekker et al., 1999[48], Bekker et al., 2004[49] / No members of vulnerable population(s) reported / Reproductive Health; Down Syndrome
Berman et al., 2011[50] / No (but they were included within a diverse population) / Older Age / Abdominal Aortic Aneurysm
Berry et al., 2010[51], Berry et al., 2013[52] / No (but they were included within a diverse population) / Race and Ethnicity / Prostate Cancer
Breslin et al., 2008[53], Mullan et al., 2009[54] / No (but they were included within a diverse population) / Older Age / Diabetes
Brink et al., 2000[55], Fagerlin et al., 2004[56] / No members of vulnerable population(s) reported / Prostate Cancer
Brohan et al., 2014a[57], Brohan et al., 2014b[58] / Yes / Mental Health Condition(s) / Mental Health
Chewning et al., 1999[59] / Yes / Younger Age; Race and Ethnicity; Lower Education or Income; Lower Literacy / Reproductive Health
Chiew et al., 2008[60] / No (but they were included within a diverse population) / Older Age; Race and Ethnicity; Lower Education or Income / Breast Cancer
Consoli et al., 1995[61] / No members of vulnerable population(s) reported / Cardiovascular Health (Heart Disease, Stroke, etc…)
Costanza et al., 2011[62] / No (but they were included within a diverse population) / Race and Ethnicity; Lower Literacy / Prostate Cancer
Cox et al., 2012[63] / No (but they were included within a diverse population) / Race and Ethnicity / Prolonged Mechanical Ventilation
Dales et al., 1999[64], Wilson et al., 2005[65] / No (but they were included within a diverse population) / Older Age; Lower Education or Income / Advance Care Planning; Chronic Obsructive Pulmonary Disorder (COPD)
Davis et al., 1998[66] / No (but they were included within a diverse population) / Race and Ethnicity; Lower Education or Income / Vaccine Preventable Disease
Dillard et al., 2010[67] / No members of vulnerable population(s) reported / Colorectal Cancer
Dodin et al., 2001[68] / No (but they were included within a diverse population) / Older Age; Lower Education or Income / Reproductive Health
Dolan & Frisina, 2002[69] / No members of vulnerable population(s) reported / Colorectal Cancer
Dolan et al., 2013[70] / No (but they were included within a diverse population) / Race and Ethnicity / Knee Osteoarthritis
Dowding et al., 2004[71], Thomson et al., 2006[72] / No (but they were included within a diverse population) / Older Age; Other: Computer Literacy / Benign Prostatic Hyperplasia; Cardiovascular Health (Heart Disease, Stroke, etc…)
Drake et al., 2010[73] / Yes / Race and Ethnicity / Prostate Cancer
Durand, 2009[74], Durand et al., 2010[75], Durand et al., 2012[76] / Yes / Race and Ethnicity / Reproductive Health
Elit et al., 1996[77] / No members of vulnerable population(s) reported / Ovarian Cancer
Emmett et al., 2007[78], Montgomery et al., 2007[79], Frost et al., 2009[80], Hollinghurst et al., 2010[81] / Yes / Reproductive Health
Evans et al., 2007[82], Evans et al., 2010[83], Joseph-Williams et al., 2010[84] / No (but they were included within a diverse population) / Older Age / Prostate Cancer
Fagerlin et al., 2010[85], Fagerlin et al., 2011[86], Banegas et al., 2013[87], Korfage et al., 2013[88] / No members of vulnerable population(s) reported / Breast Cancer
Farnworth et al., 2008[89] / No members of vulnerable population(s) reported / Reproductive Health
Feldman-Stewart et al., 2000[90], Feldman-Stewart et al., 2001a[91], Feldman-Stewart et al., 2001b[92], Feldman-Stewart et al., 2002[93], Feldman-Stewart et al., 2004[94], Feldman-Stewart et al., 2006a[95], Feldman-Stewart et al., 2006b[96], Feldman-Stewart et al., 2012[97] / No (but they were included within a diverse population) / Older Age; Lower Education or Income / Prostate Cancer
Fetters et al., 2004[98], Ruffin et al., 2007[99], Taylor et al., 2010[101] / No members of vulnerable population(s) reported / Colorectal Cancer