Evidence Sub Group &
Economic Evidence Sub Group
Health Improvement Review
Assessment of the Cooking Bus (CB)
The following report includes the assessments of the evidence base produced by the Evidence subgroup (section 1- Evidence of Effectiveness) and the Economic Evidence sub-group (section 2 – Evidence of Cost effectiveness).
Section 1: Evidence of Effectiveness
1. Introduction
A core component of the Health Improvement Review (HIR) has been the assessment of the evidence-base for initiatives included in the HIR. This report describes the methodology for, and findings of, the assessment of the Welsh Cooking Bus programme.
2. Methodology
Assessment of the initiative employed a dual approach:
· Assessment of the potential effectiveness of the initiative by review of research on the effectiveness of similar initiatives or of component interventions (some initiatives involve more than one intervention).
· Assessment of the actual effectiveness or impact of the initiative by review of any available evaluation reports for the initiative in Wales.
2.1 Review of potential effectiveness
The methodology adopted for this review followed systematic review principles of transparency, a priori setting of the research question, search strategy, inclusion/exclusion criteria, critical appraisal and standardised data extraction.
2.1.1 A ‘question’ was developed for each initiative following the PICO format[1]. For CB, the question was: What is the evidence that cooking sessions in the community (including schools) has any positive impacts?
2.1.2 Due to the time constraints of the Health Improvement Review, a ‘best available evidence’ approach was taken for the reviews of research on potential effectiveness of initiatives. Key words and search terms were derived from the question and a pragmatic search strategy designed using specified health databases and search-engines. For initiatives where recent high quality secondary analyses of the primary literature were found, searches were narrower and terminated at an earlier stage. Searches for questions that yielded little high quality data initially were broadened by date or by search terms in an attempt to capture related work. The time-constraints did not enable hand-searching or contacts with experts in the field (external to Public Health Wales) to search for missed or unpublished data, however, an iterative process of related article searches were run on key papers for some questions to try and capture information that the initial search strategy had not identified. All reasonable efforts were made to locate the most relevant and highest quality evidence in the short-time frame allocated.
The search terms used and databases searched were:
TRIP Database
Search Strategy / No of hits(title: cook* and (school* or community)) / 2
cook* and (school* or community) / 3,750 (664 secondary)
(title: cook*) / 70 (use this search)
cook* and (title: school* or community) / 122 (use this search)
“food preparation” and (school* or community) / 118 (no additional refs found)
NHS Evidence
Search Strategy / No of hits(Cookery or cooking or cook) / 5427
( Cookery or cooking or cook) and (school* or community) / 4106
( Cookery or cooking or cook) and (school* or community) FILTER Public Health / 512 (first 300 scanned on relevance)
( Cookery or cooking or cook) and (school* or community) FILTER Evidence summaries +grey literature+ guidelines+ systematic reviews / 1416 (first 200 scanned on relevance)
PubMed
Search / No of hitsTitle: cooking or cookery or cook or “food preparation” / 2507
[Title: cooking or cookery or cook or “food preparation”] and [Title/abstract: school* or community] / 81
Cook evaluation school / First 100 scanned
Cook evaluation community / First 100 scanned
Cooking evaluation / First 100 scanned
2.1.3 Retrieved articles were recorded in the ‘Evidence Mapping Table’ for the initiative (Table 1) and were screened for inclusion by two reviewers independently (disagreements resolved by discussion), on the basis of direct relevance to the initiative or component interventions and type of article, thus single studies were not included if higher level evidence was available:
Primary group of sources: NICE guidance, Single systematic reviews from Cochrane, Campbell Libraries, the EPPI-Centre
Secondary group (include if no primary group evidence items available): RCT or evaluation of robust design looking at appropriate outcomes
Other study designs to be included if no primary or secondary sources are available, the
quality of these to be judged separately/recorded on a case-by-case basis.
2.1.4 Information was extracted from each included article into a standardised template – the Evidence Mapping Table, for each initiative.
2.1.5 Each included article was assessed in terms of ‘reliability’, strength and direction, using the following ‘Evidence Grading Scheme’:
++ Directly relevant evidence that the intervention evaluated is beneficial/ useful/ effective - the evidence comes from a reliable source[2] and is guidance based on RCTs, SRs or robust evaluations of appropriate outcomes or is a well conducted systematic review.
+ Directly relevant evidence that the intervention evaluated is beneficial/ useful/effective -the evidence comes from a reliable source and is a robust/ large RCT or robust evaluation of appropriate outcomes.
+/- Conflicting evidence (from reliable sources) about the usefulness/efficacy of the intervention being evaluated.
0 Directly relevant evidence on effectiveness of an intervention the same as, or similar to, the initiative, of acceptable reliability, is lacking.
-- Directly relevant evidence that the intervention being evaluated is not
beneficial/useful or is ineffective - the evidence comes from a reliable source and is guidance based on RCTs, SRs or robust evaluations of appropriate outcomes or is a well conducted systematic review.
- Directly relevant evidence that the intervention evaluated is not beneficial/useful or is ineffective -the evidence comes from a reliable source and is a robust/large RCT or robust evaluation of appropriate outcomes.
Ql Well conducted studies using robust qualitative research methods which cast light on how/why intervention might be effective/ineffective or have important implications for interpretation of findings or other included studies.
The ‘evidence grades’ for each included article were recorded in the Evidence Mapping
Table for the initiative (Table 1).
2.1.6 A subjective judgment of the overall balance of evidence grades given to included articles was then made by one reviewer, to give a ‘Summary Evidence Grade’ for the Initiative:
++ There is consistent, strong relevant evidence from reliable sources that the intervention/approach employed in the initiative has the potential to be effective.
+ There is some relevant evidence from reliable sources that the intervention/approach employed in the initiative has the potential to be effective.
+/- Relevant evidence (from reliable sources) about the likely effectiveness of the intervention/approach employed in the initiative is conflicting.
0 Directly relevant evidence (from reliable sources) about the likely effectiveness of the intervention/approach employed in the initiative is lacking.
- - There is consistent, strong relevant evidence from reliable sources that the intervention/approach employed in the initiative has the potential to be ineffective.
- There is some relevant evidence from reliable sources that the intervention/approach employed in the initiative has the potential to be ineffective.
Checks were made for consistency of application of these ‘Summary Evidence Grades’
through comparison and discussion amongst the reviewer team.
2.2) Assessment of initiative
An ‘Initiative Assessment Log’ was then completed (Table 2). Information from any evaluation or other reports about the initiative in Wales was considered for relevance at this stage and pertinent information summarised into the log along with the Summary Evidence Grade and other information. A final ‘Initiative Grade’ was then applied by one reviewer using set criteria (see Annex). This therefore takes into account both the evidence of potential effectiveness and evidence of actual effectiveness in Wales, where available. This Initiative Grade’ will feed directly into the Programme Budgeting and Marginal Analysis which forms the decision-making framework for the Health Improvement Review. Initiative grades were checked for consistency by comparison and discussion amongst the review team.
Initiative Evidence Assessment Report CB FinalPage 25
Public Health Wales Observatory / Health Improvement ReviewEvidence Sub Group &
Economic Evidence Sub Group
3. Findings of the assessment
Table 1: Evidence Mapping Table for Cooking Bus
Study Details / Study design / Outcome measures / Main findings / Results / Evidence Grading / Include?Reason for exclusion
1. Silveira JA et al., 2011
School-based nutrition education
Target group:
Children & adolescents / Systematic review / Measures of body mass index (BMI), skin folds, circumferences and percentage of body fat or lean mass, or dietary outcomes / No
Included studies used range of interventions but not cooking sessions
2. Van Cauwenberghe E et al, 2010
School-based nutrition interventions
Target Group:
6-18 years / Systematic review / Dietary behaviour and/or anthropometric measures / No
Included studies used educational and environmental interventions but not cooking sessions
3. Brown T & Summerbell C 2009
School-based diet and physical activity interventions
Target group:
Children and young people aged 5-18 / Systematic review / Change in BMI or other weight outcome / No
Included studies used multiple & varied interventions - cooking sessions not specifically mentioned
4. Rees R et al.,
Interventions to introduce adults to cooking
Target group – adults of working age and older people / Systematic Review / Knowledge of, attitudes to, healthy eating and dietary change / Lack of reliable sources of evidence. For the one included study - no evidence that the cooking club had an effect on other aspects of diet or on knowledge, attitudes or physical health though the study size may have made it difficult to detect such changes. / Included 13 studies, only five of these used a comparison group design to evaluate outcomes and only one of the five was considered ‘reliable’ when selection, attrition and reporting bias were considered. This study involved peer-led cooking clubs for people aged 65 or older in sheltered housing in socially deprived areas. The outcomes showed that 1-year post course participants had increased the percentage of energy obtained from carbohydrate (2.4 % more than non-participants). However participants were also more likely to describe their diet as healthier than it actually was compared to people that had not attended the clubs after this time interval. / 0 / Yes
5. Harris LF et al., 2011
E-learning
Target group:
Children and adults / Systematic review / Dietary change / No
Intervention not relevant
6. Contento I et al., 1995 / Systematic review / Dietary change, nutrition related behaviours, knowledge & attitude / No-
Included studies used multiple & varied interventions - cooking sessions not specifically mentioned
7. NICE Pathway / Dietary Pathway / No
Not relevant type of information
8. Ciliska D et al., 1999
Community interventions to increase fruit and vegetable intake
Target group:
Ages 4+ / Systematic review / Intake of F&V, knowledge of, and attitude to, healthy eating / No
Included studies used multiple & varied interventions - cooking sessions not specifically mentioned
9. Caroline Walker Trust 2009
Report on nutrition policy in UK / Nutrition Policy Briefing Paper / No
Not relevant type of information
10. Ayliffe B and Glanville NT, 2010
Guidelines for weight management
Target group:
Teenagers / Guidelines / Weight management / No
Not relevant type of information
11. Macdiarmid JI et al, 2011
Nutrition interventions
Target group:
School-aged children / Evaluation framework / Nutrition related outcomes / No
Not relevant type of information
12. Ubido Jet al., undated
Prevention initiatives – diet and healthy eating
Target group:
All / Cost-effectiveness review / Diet related outcomes / No
Included studies used multiple & varied interventions - cooking sessions not specifically mentioned
13. NICE, 2008
Improving nutrition
Target group:
Mothers and children in low income households / NICE guidance / Breastfeeding and nutrition / None directly relevant to cooking sessions / Evidence from one (+) needs assessment study showed that cook and eat sessions and weight management classes that were made freely available on a gypsy traveller site were valued by women residents for their non-threatening environment and as a forum for discussion of health issues – as well as a way to reduce social isolation.
There were suggestions from one (+) focus group study that learning to cook traditional food in healthy ways may be beneficial to South Asian groups / No
Evidence statements do not include effectiveness of cooking sessions
14. NICE ,2011
Population & community-level interventions for prevention & treatment of Type 2 diabetes
Target group:
High risk groups and general population / NICE guidance / No
Evidence statements do not include effectiveness of cooking sessions
15. Fletcher A and Rake C, 1998
Nutrition education, counselling and policies
Target group:
65+ years / Systematic review / Dietary behaviour, knowledge, attitudes & beliefs, physiological measures / No
Only 1 included study included cooking demonstrations and this was of unreliable methodology
16. Howerton MW et al, 2007
School-based nutrition initiatives
Target group:
School children / Meta-analysis / Fruit and vegetable consumption (Net FV difference and net FV relative change (%).) / These data imply that school-based interventions, can increase fruit and vegetable consumption by modest amounts and thereby increase nutrient density of the diet. These studies used different measures of food intake, which may have biased the results owing to differences in precision and accuracy and also, interventions varied.
Only one study included ‘cooking sessions’: 24 weekly taught lessons including food preparation and eating, and for the family, 12 bimonthly, bilingual, low-literacy newsletters; nutrition classes; 2 “Family Fun Nights” . This resulted in modest increases in fruit and vegetable consumption. However, the study group was 90.5% Hispanic (US) and so has limited comparability to CB. It is not clear how long after the study the effect was maintained. / Data from 7 studies were included.. 8156 children were matched from pretest to posttest. Participants were primarily elementary school-aged (75.5%) and white (66%), and 50.4% were males. Approximately 59% of participants received
a treatment condition, and 41% received a control. Each study implemented at least 2 strategies to deliver the interventions, with each study including a classroom-based component. In addition, 6 of the studies included a family component, 4 studies included a food-service component, 2 studies included a community component, 1 study included a media campaign and 1, food preparation and eating. To calculate the combined effect for the 7 studies, a weighted average approach was used. At the individual level, the net difference in FV consumption was 0.45 (95% CI 0.33-0.59) servings; the net relative change was 19% (95% CI 0.15-0.23) servings. Sex, race, and duration of intervention had no statistically significant impact on net relative change. However, baseline grade, dose, intervention type, and the interaction between dose and grade significantly affected percent FV intake. Only 1 study included cooking sessions and for this study (n=600) the net difference was 0.35 (0.25 to 0.45, p<0.001) and net relative difference was 0.38 (0.27 to 0.49 p<0.001). / 0 / Yes