Evaluation of the 2015 Rheumatic Fever Awareness Campaign

EVALUATION OF THE 2015 RHEUMATIC FEVER AWARENESS CAMPAIGN

Data Collection Tools

Contents

Participant Information sheet

District Health Board Interview checklist

GSL Promotus Interview checklist

Healthline Interview checklist

Ministry of Health Interview checklist

Focus group checklist

CATI questionnaire

EVALUATION OF THE 2015 RHEUMATIC FEVER AWARENESS CAMPAIGN

Participant Information sheet

This paper provides information about the evaluation of the 2015 Rheumatic Fever Awareness Campaign. The evaluation is taking place from June to December 2015. Please read this information before deciding whether to take part in an interview, focus group, survey or workshop being undertaken as part of the evaluation.

What is this evaluation about? / The Ministry of Health has appointed Allen + Clarke to undertake an evaluation of the campaign to determine:

How well has the campaign been implemented/delivered?
Has the campaign achieved its intended outcomes, and to what extent?
What can be learned from the 2015 campaign?

What is the purpose of the evaluation? / The evaluation will determine if the rheumatic fever campaign has effectively raised awareness among Māori and Pacific parents and caregivers (aged 21-40 years) of youth aged 4-19 years, about:

the link between sore throats and rheumatic fever
the importance of getting sore throats in at-risk children checked by a health professional
the importance of completing the full antibiotic course for children who have Group A Streptococcal infection.

The evaluation will also inform the development of the 2016 rheumatic fever awareness campaign.
Why have you asked me to participate? / The evaluation team wants campaign views and experiences of a range of stakeholders. This includes national level stakeholders, such as health agencies and government organisations, and local stakeholderssuch as DHBs and community members.
What’s involved? / The evaluation will use a range of methods to engage with people, including key informant interviews, a telephone survey, focus groups and workshop discussions.
The key informant interviews will take around 60 minutes and focus groups 90 minutes. A phone survey will be launched in August 2015, will take no longer than 20 minutes to complete and target parents and whānau in each DHB.
How will the information be analysed? / The evaluation team will collate all information, and analyse the information specifically to answer each evaluation question. Facilitated workshops will take place in November 2015, lasting 2-3 hours long and involve a mix of participants in helping to analyse and interpret the emerging findings.
Do I have to take part? / No you don’t have to take part; participation in the evaluation is voluntary.
Will information I provide be confidential? / Allen + Clarke will seek to ensure your contribution is confidential. Any comments you make will not be attributed to you in our reports.
Audio recordings, notes from interviews and survey responses will be kept securely at the Allen + Clarke office or in secure storage for up to 2 years, before being destroyed. The information will only be used for the purpose of the rheumatic fever awareness campaign evaluation.
Can I change my mind and withdraw from the evaluation? / Yes. You can stop participating in the evaluation at any time and you can also withdraw the information you have given up until the time the analysis begins. You do not need to give a reason to withdraw and there will be no disadvantage to you of any kind.
Who is doing the evaluation? / The evaluation is being led by Allen + Clarke, a company which specialises in research and evaluation. For more information about Allen + Clarke, go to allenandclarke.co.nz
What if I have any questions? / If you have any questions about this project, please contact:
Patricia Vermillion Peirce, Project Manager, Allen + Clarke

Mobile: 027 449 2781

Fiona Bailey, Ministry of Health

Ph. 04 816 3676

EVALUATION OF THE 2015 RHEUMATIC FEVER AWARENESS CAMPAIGN

District Health Board Interview checklist

This list provides topics and questions prompts for evaluators to cover during the interview.

Preliminaries

Introductions.
Explain purpose of evaluation.
Explain why we want them to participate.
Provide with information sheet if have not already received it and obtain consent.

Background information

Describe your job/position/role
How long have you been doing this job?
How are you involved (or impacted by) the 2015 Rheumatic Fever Awareness Campaign?

Evaluation-criteria and question prompts

Questions prompts are provided as a starter for the discussion. If judgements /suggestions are made, the evaluator should probe to evidence to support this judgement/suggestion and test counterfactuals.

Criteria/ expected achievements / Semi-structured question prompts (as appropriate)
**Show some images of campaign**

In your own words, can you describe the current rheumatic fever awareness campaign?
Can you compare current and past campaigns

-Probe to delivery mechanism, content, language, interesting for target audience, reach)

What did you expect to achieve with regards to reach? (series of outcomes, intention in ripple effects)?

Alignment: The campaign incorporates partnership and collaboration with key relevant agencies or networks to ensure that all messages and activities are aligned /

Is the campaign (delivery mechanism/content/language/interesting) relevant to the target audiences (M&P audiences of young children)? How do you know it is relevant? (search for indicators/data sources)
In general, how does this align with other rheumatic fever activities?

-Probe to – does it work along other RF activities, or is it more of a stand-alone?

Do you or others at the DHBs use the campaign material? (if so, how?)
Does this campaign support your RFP efforts in your area? (if so, how?)

Value:The campaign resources are allocated efficiently and effectively according to outputs (what was delivered) and outcomes (what was achieved). /

How effectively could you integrate the campaign into your local work?
What things particularly helped this happen (e.g., being kept informed in campaign development, knowing what would be available and when, media templates in lead up to launch etc)?
Does this campaign support you and your role (or other DHB staff)?
Does the campaign allow you/other DHB staff to focus your rheumatic fever prevention budgets on local efforts?
How effectively could you integrate the campaign into your local work?
What things particularly helped this happen (e,g, being kept informed in campaign development, knowing what would be available and when, media templates in lead up to launch etc)?

Unintended consequences and learning /

Does the campaign impact on you, other staff or services?
Was there anything unexpected that has happened to you/your organisation resulting from the campaign (e.g. this can include from your involvement, implementing the campaign, or the campaign outputs)
What do you believe supported the current campaign?
If you could do anything differently when designing future campaigns, what would it be (and why)?
What are some things you would want to know from the target audience to improve future campaigns?

Concluding topics

Are there any other points or issues related to the 2015 Rheumatic Fever Awareness Campaign that we have not yet discussed?

EVALUATION OF THE 2015 RHEUMATIC FEVER AWARENESS CAMPAIGN

GSL Promotus Interview checklist

This list provides topics and questions prompts for evaluators to cover during the interview.

Preliminaries

Introductions.
Explain purpose of evaluation.
Explain why we want them to participate.
Provide with information sheet if have not already received it and obtain verbal consent.

Background information

Describe your job/position/role
How long have you been doing this job?
How are you involved (or impacted by) the 2015 Rheumatic Fever Awareness Campaign?

Evaluation-criteria and indicators

Criteria/ expected achievements / Semi-structured question prompts (as appropriate)
Relevance: The campaign uses appropriate language and messaging for the cultural context /

In your own words, can you describe the current rheumatic fever awareness campaign?
Can you compare current and past campaigns?

-Probe: to delivery mechanism, content, language, interesting for target audience, reach)
-(If any difference, explore reasons)

What was learned from past campaigns?

-E.g. Process, Modes of choice, Language
-What processes were instrumental in ensuring that key campaign audiences were able to access campaign messages (modes) and able to easily understand the messages; not put-off by the language/medical language; able to relate to the communications (content, imagery, cultural context, see themselves)?

How were the decisions made (e.g. to use the specific delivery mechanisms)?
What processes proved useful to making these decisions?

Alignment: The campaign incorporates partnership and collaboration with key relevant agencies or networks to ensure that all messages and activities, and the timing of the delivery, are aligned /

In general, how does the current campaign align with other rheumatic fever activities?
What processes contributed to the alignment (if applicable)?
In your own words, can you describe the current rheumatic fever awareness campaign?
How is this different/the same as other rheumatic fever campaigns?
What was considered in terms of the current campaign and other RFPP activities?

Reach: The campaign reaches those at high risk of rheumatic fever, their families and communities, and the people who care for them /

What was considered and implemented in terms of reaching the target audience?
What informed these considerations (inputs)?
What did you expect to achieve with regards to reach? (series of outcomes, intention in ripple effects)?

Value: The campaign resources are allocated efficiently and effectively according to outputs (what was delivered) and outcomes (what was achieved). /

Can you describe the process steps used to get to the campaign outputs (interviewer to map process steps)?
Did the overall process support {reach/relevance/alignment/understanding/behaviour/value for money}?
Are there any efficiencies that can be gained in this process?
Did you put in what you expected to put into the campaign (time, resource)?
Were the agreed outputs produced, on time?
Were the agreed outputs produced according to the agreed price?
What aspects do you think are particularly high-value/low value?
What were any enablers and barriers to the quality and timely completion of the outputs?

Unintended consequences and learning /

Was there anything unexpected that has happened to you/your organisation resulting from the campaign (e.g. this can include from your involvement, implementing the campaign, or the campaign outputs)?
Was there anything unexpected that has happened for the key target audience as a result of the campaign?
What are any learnings from the 2015 campaign that might enhance the 2016 campaign? Probe:

-Support for implementation
-What would you do differently e.g. design
-What are some of the things you would want to know from the target audience to improve future campaign

Concluding topics

Are there any other points or issues related to the 2015 Rheumatic Fever Awareness Campaign that we have not yet discussed?

EVALUATION OF THE 2015 RHEUMATIC FEVER AWARENESS CAMPAIGN

Healthline Interview checklist

This list provides topics and questions prompts for evaluators to cover during the interview.

Preliminaries

Introductions.
Explain purpose of evaluation.
Explain why we want them to participate.
Provide with information sheet if have not already received it and obtain consent.

Background information

Describe your job/position/role
How long have you been doing this job?
How are you involved (or impacted by) the 2015 Rheumatic Fever Awareness Campaign?

Evaluation-criteria and indicators

Criteria/ expected achievements / Semi-structured question prompts (as appropriate)

In your own words, can you describe the current rheumatic fever awareness campaign?
Can you compare current and past campaigns

-Probe to delivery mechanism, content, language, interesting for target audience, reach)
Relevance: The campaign uses appropriate language and messaging for the cultural context /

Is the campaign (delivery mechanism/content/language/interesting) relevant to the target audiences (M&P audiences of young children)? How do you know it is relevant? (search for indicators/data sources)

Alignment: The campaign incorporates partnership and collaboration with key relevant agencies or networks to ensure that all messages and activities, and the timing of the delivery, are aligned /

In general, how does this align with other rheumatic fever activities?

-Probe to – does it work along other RF activities, or is it more of a stand-alone?

Do you or others at Healthline use the campaign material? (if so, how?)
Does this campaign support your RFP efforts in your area? (if so, how?)

Value: The campaign resources are allocated efficiently and effectively according to outputs (what was delivered) and outcomes (what was achieved). /

Does this campaign support you and your role (or other DHB staff)?
Does the campaign allow you/other Healthline staff to focus your rheumatic fever prevention budgets on local efforts.
How effectively could you integrate the campaign into your local work
What things particularly helped this happen (e,g, being kept informed in campaign development, knowing what would be available and when, media templates in lead up to launch etc)
What aspects do you think are particularly high-value/low value?

Unintended consequences and learning /

Does the campaign impact on you, other staff or services?
Was there anything unexpected that has happened to you/your organisation resulting from the campaign (e.g. this can include from your involvement, implementing the campaign, or the campaign outputs)
What do you believe supported the current campaign?
If you could do anything differently when designing future campaigns, what would it be (and why)?
What are some things you would want to know from the target audience to improve future campaigns?

Concluding topics

Are there any other points or issues related to the 2015 Rheumatic Fever Awareness Campaign that we have not yet discussed?

EVALUATION OF THE 2015 RHEUMATIC FEVER AWARENESS CAMPAIGN

Ministry of Health Interview checklist

This list provides topics and questions prompts for evaluators to cover during the interview.

Preliminaries

Introductions.
Explain purpose of evaluation.
Explain why we want them to participate.
Provide with information sheet if have not already received it and obtain consent.

Background information

Describe your job/position/role
How long have you been doing this job?
How are you involved (or impacted by) the 2015 Rheumatic Fever Awareness Campaign?

Evaluation-criteria and indicators

Criteria/ expected achievements / Semi-structured question prompts (as appropriate)

In your own words, can you describe the current rheumatic fever awareness campaign?
Can you compare current and past campaigns

-Probe to delivery mechanism, content, language, interesting for target audience, reach)

What was learned from past campaigns? (if anything, was it used in this campaign)?
How were the decisions made? (explore criteria)
How do you know this is the most effective? (explore research-base, monitoring activities)

Relevance: The campaign uses appropriate language and messaging for the cultural context /

What was considered and implemented in terms of relevance with the target audience?
What informed these considerations (inputs)?
What did you expect to achieve with regards to relevance?

Reach: The campaign reaches those at high risk of rheumatic fever, their families and communities, and the people who care for them & is remembered /

What was considered and implemented in terms of reaching the target audience?
What informed these considerations (inputs)?
What did you expect to achieve with regards to reach? (series of outcomes, intention to result in ripple effects)?

Alignment: The campaign incorporates partnership and collaboration with key relevant agencies or networks to ensure that all messages and activities, and the timing of the delivery, are aligned /

In general, how does this align with other rheumatic fever activities?
What was considered in the development of the current campaign and other RFPP activities?

Value: The campaign resources are allocated efficiently and effectively according to outputs (what was delivered) and outcomes (what was achieved). /

Can you describe the process steps used to get to the campaign outputs (interviewer to map process steps)?
Did the overall process support {reach/relevance/alignment/ understanding/behaviour/value for money}?
Are there any efficiencies that can be gained in this process?
Did you put in what you expected to put into the campaign (time, resource)?
Were the agreed outputs produced, on time?
Were the agreed outputs produced according to the agreed price?
What aspects do you think are particularly high-value/low value?

Unintended consequences and learning /

Was there anything unexpected that has happened to you/your organisation resulting from the campaign (e.g. this can include from your involvement, implementing the campaign, or the campaign outputs)
What do you believe supported the implementation of current campaign?
If you could do anything differently when designing future campaigns, what would it be (and why)?
What are some things you would want to know from the target audience to improve future campaigns?

Concluding topics

Are there any other points or issues related to the 2015 Rheumatic Fever Awareness Campaign that we have not yet discussed?

EVALUATION OF THE 2015 RHEUMATIC FEVER AWARENESS CAMPAIGN