DfE Community Learning | Mental Health Research Aug. 2016 – Jul. 2017

CLMHR phase 2: Final Implementation Specification for Providers

Contents

1. Leadership and project management / 7. Workforce development
2. Project close and dissemination of findings / 8. Outsourcing/local joint working
3. The research design / 9. Evidence of your professional praxis
4. Research volunteer[i] numbers / 10. Comms and publicity
5. Research volunteer data and deadlines / 11. Meetings and training activities
6. Data quality

1.

/

Leadership and project management

From 1stAug 2016 to 31July 2017
1.1 / Ensure you have the required project team in place. As a minimum, this includes a named and accountable:
  • Project lead
  • Project manager
  • Data co-ordinator
  • Local senior strategic supporter/advocate for the project findings
Full‘live’ contact details for thesemust be provided to DfE and kept-up-to-datealwaysfor the duration of the research.
1.2 / Ensure:
  • Effective financial and project performance management, including the completion of agreed project quality assurance activities
  • Ethical research principles apply always
  • A high level of fidelity between the planning and delivery of your project and the year-2 research design and agreed descriptors for the non-equivalent group (A or B or C) to which we randomly assigned you.
  • High quality design, delivery, reporting, monitoring and quality assurance ofeach element of the specified phase 2 research intervention for your allocated group
  • You contribute to collective working and shared responsibility within your non-equivalent group to collectively secure the required sample

1.3 / Ensure delivery of the required number of non-formal, non-accredited, adult education courses to the required number of learners, consistent with the descriptors for your allocated non-equivalent group. Each course (regardless of which group you are allocated to) is to comprise 15taught hours of learning over a period of no more than 6 taught weeks and no less than 3 taught weeks
1.4 / Ensure accurate and complete reporting of all the following:
  • Research volunteer data
  • ILR data by the required reporting dates set by the SFA for the required number of new research volunteers/enrolled learners
  • Project performance and evidence of your developing professional praxis on the MHFE website.
  • Project personnel changes
  • Progress, performance against this specification and the descriptors for your assigned non-equivalent group
  • Risk management and opportunities
  • Quality assurance

1.5 / Ensuretimely internal and external communications, e.g. project marketing materials, informing and supporting your staff, performance, updates, risk register, personnel changes, etc.
1.6 / Ensure public marketing information is up-to-date and meetsthe required readability good practice[ii], the NHS(E) information[iii] and accessibility[iv] standards and in addition to local distribution is made available nationally at all times on the offers Padlet.
1.7 / Attend and contribute to mandatory:
  • Whole project weekly online FridayLunchMeetsat:|Telephone: +44 (0) 20 3695 0088 | Meeting ID:326 029 8006
  • Regular onlinequality assurance meetings for your intervention group (A, B or C)
  • Termly whole project face to face meetings on:
  • 17 March 2017 in London (Password: CLMHP-Spring17)
  • 14 June 2017 in Birmingham (Password: SummerTerm17)
  • Ipsos MORI virtual workshops on ‘understanding your data’
  • Any other virtual or face-to-face training as identified by you or DfE/the external evaluators as required to ensure that you meet the required standards in the performance of your project
  • Contribute additionally, as required by and agreed with DFE,to at least4lessons learned activities: 1 online, 1 local, 1 regional and 1 national between October 2016 and March 2018

2.

/

Project close and dissemination of findings

2.1 / Develop and publisha local plan for the internal and external dissemination and implementation of findings and lessons learned from your project and the overall CLMH research project. You must publish your plan on your MHFE project page, on or before Friday 12May 2017.
2.2 / Prepare between April and July 2017 to contribute to the national dissemination and implementation of findings and lessons learned from the study between August 2017 and March 2018.
This will include as a mandatory minimum:
  • Act as an ambassador for the findings and, if appropriate, presentation to the sector of your ongoing professional and organisational evidence of the effectiveness of community learning for people with mental health problems and how you respond to this evidence
  • Develop full and detailed electronic lessons learned content on the MHFEwebsiteabout your project - providing sufficient detail for others to be able to critique and specially to enable them to replicate what you have done in their own context/s
  • Contribute, as required by DFE/the managing agent/SFA/external evaluators, to at least 3 findings review and/or dissemination activities between August and March 2018.

3.

/

The research design (August 2016- July 2017)

3.1 / The phase 2 study is quasi-experimental and will require your project to be a team player.
The research will test the following null hypothesis:
Non-formal adult education courses of 15 hours’ duration over a period of six weeks where:
(a) Participants have mild to moderate mental health problems and courses focus on managing their symptoms;
(b) Participants have mild to moderate mental health problems and courses focus on traditional community learning topics;
(c) Participants are a mix of learners with and without mental health problems and courses focus on traditional community learning topics
Makes no statistically reliable difference to mental health outcomes, where statistically reliable is defined as >6 on PHQ9 and >4 on GAD7.
Analysis will be based on comparison of participants’ guidance session (baseline)and start of first class scores (as a double pre-test) and their most recent final score (repeat scores are collected every 2 weeks while on course and at every refresh session throughout the intervention) on the same clinical depression and anxiety self-assessment scales used to evaluate impact in the Department of Health, NHS(E) IAPT programme, plus qualitative data collection.
Specifically, the research will measure:
a)Quantitative data - Recovery and distance travelled by participants on each of the three mental health and wellbeing scales used (PHQ9, GAD7, SWEMWBS). Quantitative data will also include demographic and social mobility data as well as educational and socio-economic outcomes. a separate ‘follow-on’ Study.
b)Qualitative data – Through “A day in the Life Mental Health in Adult Education” mass observation study and through projects outputs and local evidence on the MHFE website and dissemination activities in relation to the research method, learner involvement/co-production, collective working, quality assurance of each offer, workforce development, lessons learned through professional praxis.
Each project has been randomly assigned to one of 3 non-equivalent groups, A or B or C
  • Group A will offer: Non-formal adult education courses of 15 hours’ duration over a period of six weeks where participants have mild to moderate mental health problems and courses focus on managing their symptoms;
  • Group B will offer: Non-formal adult education courses of 15 hours’ duration over a period of six weeks where participants have mild to moderate mental health problems and courses focus on traditional community learning topics;
  • Group C will offer: Non-formal adult education courses of 15 hours’ duration over a period of six weeks where participants are a mix of learners with and without mental health problems and courses focus on traditional community learning topics
Each group:
  • has the same number of provider projects (19)?
  • will deliver an identical number of courses of the same narrowly defined duration and intensity (15 hours over a period of no less than 3 taught weeks and no more than 6 taught weeks)
  • engage a similar number of learners (circa 4,000)
  • have the same overall unit cost per learner (£450)
All providers within an assigned group must:
  • deliver the same community learning offer, which must include guidance + course of 15 hours over no less than 3 and no more than 6 taught weeks + refresh sessions
  • agree and adhere to the descriptors for their respective group’s offer
  • work together to monitor and manage quality, risk, ethical, safeguarding and Prevent issues, ensure that your group contributes the overall required number of learners and data for their group as part of the overall project
  • provide a weekly update on research volunteer numbers (actuals v. planned) using the relevant Google doc for your group
  • work together to quality assure your group’s offer against the agreed published descriptors for the group, including completing randomly assigned peer observations against your group’s descriptors (as per the relevant Google doc and agreed observation checklist for your group)
  • for details of the peer observation checklist for your group see the vital information noticeboard Padlet
  • for details of the descriptors see the Padlet for your non-equivalent group.

3.2 / The fidelity of the IAG and learning opportunities offered, data collection and reporting within and between the groups nationally iscritical to the robustness of the phase 2 research design, i.e. we must control variables - except the one’s being studied and those variables that learners bring to their learning as individuals.
For a list of which providers are assigned to which group see:
Regardless of the group you are in, every project’s offer to learners must include all the following:
a)Public and cross-sector professional research recruitment and marketing activities and materials. Public marketing information can be in any media/format but must meet the required readability good practice[v],and the NHS(E) information[vi] and accessibility[vii] standards.
b)Up to 3 hours of 1:1 pre-course guidance and assessment and on-course support and guidance to every learner, using the Discovering Potential model and tools. This must:
  • help research volunteers decide if they want to be involved in the research,or
  • if the research is not the ‘right’ thing for an individual, signpost them, as appropriate, to other learning or vocational opportunities/and/or mental health services and support
  • be when you answer people’s questions and gain each research volunteer’s informed consent to the whole of the research project (peoplewho are unable to provide informed consent to the research are not be eligible to participate in this study)
  • be when you collect all baseline data from learners on an individual basis (no filling in of local or ILR forms by learners and absolutely no group sessions filling in forms together – this is NOT the required approach)
  • help learners to select their course, discuss progress, and plan for progression to employment, vocational/academic training and/or fulfilling family and community life.
c)Short, part-time non-formal adult education courseswhich must:
  • be delivered by community learning practitioners (i.e. must already be part of your Ofsted inspected provision and must be delivered by general community learning providers and teachers)
  • be delivered by the minimum number of sub-contractors and a controlled number of teachers/volunteers in order to ensure consistency in the offer within your site and across research sites)
  • use community learning teaching, learning and assessment that must be of a quality that Ofsted[viii] would recognise as at least ‘good’
  • include 15 hours of learning delivered flexibly over a period of no less than 3 taught weeks and no more than 6 taught weeks from the first to last day of a learner’s course
  • be specifically designed in terms of curriculum and teaching, learning and assessment methods to be evidence-based[ix] and inclusive of and support eligible learners experiencing mild to moderate mental health problems to share their experiences and develop the tools, strategies and resilience to manage, and aid their recovery
  • include the required data collection and online reporting timed to meet the requirements of the research
d)Non-formal ‘top-up’ mental health adult education workshops, at least 1 every 4 weeks for people who have progressed from courses who need subsequent support at times of stress/transition, in which people revisit, consolidate existing and learn new strategies for managing difficulties resulting from their mild to moderate mental health problems
e)Online data collection using the SmartSurvey provided at the required points during each person’s learning journey
f)Continuous review, quality assurance and reporting of online data as required by DfE
g)Monthly virtual review of your data quality assurance with your nominated external evaluator. Where there is concern about security, or consents, or completeness or the general quality of your data the external evaluator may (subject to approval by DfE) require you to meet with them face-to-face to resolve concerns.

4.

/

Research volunteer numbers (August 2016 – July 2017)

4.1 / The basis on which the government has agreed to continue and fund phase 2 of this research project is that we must deliver a more robust, quasi-experimental study in which we collect full match data with named consents for a minimum of 14,000 research volunteers. This is to produce results that will provide a sufficiently high confidence level for the findings to:
  • be used as evidence to inform publicly funded policy
  • be used as evidence for commissioning community learning through the Adult Education Budget
  • For us to be able to assure potential future learners that it can help them manage their mild to moderate mental health problems and will not do them harm.
We need to keep as many variables as possible fixed - except for those that the research is seeking to test out (i.e. the effect of different courses on people’s mental health as measured by PHQ9, GAD7 and SWEMWBS).
Each project has been randomly assigned to one of 3 quasi-experimental groups and has been informed individually of the number of research volunteers you need to secure full match data for.
The unit price/research volunteer[x] was arrived at by dividing the total amount of money nationally by the 14,000 research volunteers. This gave us the unit price per research volunteerinvolved in the research of £450. We then divided the amount of money allocation to you by that unit cost to work out the number of research volunteers you need to contribute. For example, the number of research volunteersyou need to secure complete match data for will be:
  • 178 if the SFA allocated you £80,000
  • 286 if the SFA allocated you £128,640
This formula is ‘generous’ in terms of funding for community learning. This is because it is not about getting the most learners/courses you can possibly deliver or about charging learners. Your priority in phase 2 is to make sure that you meet all the requirements of thisas research and that it is consistent across projects and non-equivalent groups for the whole duration. This includes securing sufficient research volunteers to can produce statistically significant and reliable findings. Without the research volunteers,we need to help us test the null hypothesis we cannot do this research.
4.2 / All research volunteer learners must be entered correctly on the ILRby the required deadline/sand submitted to the SFA monthly.
4.3 / For a research volunteer to be ‘counted’ against the target number of people we have asked you to engage:
  • they must be eligible to be involved in this research (i.e. they must meet the agreed eligibility criteria specific in the CLMH SmartSurvey for this research)
  • they must give their informed consent and sign the consent form
  • they must complete the required online SmartSurvey (or prior to you having access to your live SmartSurvey the 3 year-1 MHSAs + ILR match data) i.e.
  • Thinking: Initial Guidance Meeting (1st pre-test self-assessment and equalities)
  • Getting: Course week 1 (2nd pre-test self-assessment)
  • Getting: Course week 3 (3rd self-assessment, social mobility, and if last week, progression)
  • Getting-Keeping: Course week 5/6 (4th post-test self-assessment and progression)
  • And, if possible at least one Keeping-Thinking refresh session survey (serial post-test self-assessment and progression)

4.4 / If a research volunteer misses a session when a self-assessment is due:
If someone misses: / The rule is:
Guidance session / Can't join a course unless completed
Week 1 / OK to complete at the start of week 2
Weeks 1 and 2 / Can't do the course and need to join another/later one
Week 3 / OK to complete at the start of week 4
Weeks 3 and 4 / Skip the week-3 survey (means we don't collect social mobility data)
Week 6 / OK to complete at a refresh session
4.5 / People who access only your IAG offer (and then are either not eligible or choose not to proceed) and people who only access refresher sessions in this extension are not included in your target sample from August 2016 to July 2017.

5.

/

Research volunteer data[xi]and deadlines (August 2016 to July 2017)

5.1 / You MUST make accurate monthly ILR returns to the SFA in accordance with the 2016/17 ILR rules and timetable at:
R01 - 06 .09.16
R02 - 06.10.16
R03 - 04.11.16
R04 - 06.12.16
R05 - 06.01.17
R06 - 06.02.17
R07 - 06.03.17
R08 - 06.04.17
R09 - 05.05.17
R10 - 06.06.17
R11 - 06.07.17
R12 - 04.08.17
R13 - 14.09.17
R14 -19.10. 17
5.2 / Methods for submitting ILR data are unchanged. Ian (Knowles) has confirmed the following:
For Community Learning (CL) providers the HUB will show the next contractual ILR return. However, all CL providers, mainstream and/or part of the mental health pilot can submit CL data monthly. Please note that an ILR submission will overwrite your last return, as per normal. The SFA cannot change the ILR reporting system as it would affect all CL providers, not just those who are involved in the mental health research, so, keep returning monthly data. When it is submitted as per the ILR return requirements, the SFA can see the data.
There are 3 ways to make your monthly returns:
a)Make a full monthly ILR return, (i.e. organisation submits a clean, complete file containing all learners, including those not participating in the CLMH research), or
b)Return a file that contains only the CLMH learners (i.e. a small file containing only this sub-set of learners), or
c)Return your complete ILR file but only update the data for the CLMH learners (i.e. don’t update the data for any other learners or correct any errors triggered by other learners).
5.3 / The ILR Code for every community learning mental health research course is 340. Nothing else unless your data system also forces your MiS staff to enter PCDL (see screenshot below). You MUST NOT record any other community learning aims on the ILR for learners involved in this research.

If you are in any doubt at any time contact: YourSFA Central Delivery Service advisor or the Data Queries inbox
5.4 / In response to difficulties with year 1 data collection, DfE commissioned Good Things Foundation (formerly Tinder Foundation) to purchase and distribute to all project providers, additional IT equipment and Mi-Fi devices to support providers to collect phase 2 research volunteers’ data electronically via SmartSurvey. Good Things Foundation confirmed that all equipment has been received by providers.
This equipment must be managed and stored securely and may only be used to support the specific activities associated with this research project until the end of the research project’s data collection phase (31 July 2017), after which time DfE will formerly confirm that the equipment is available for general use by the provider.
For the duration of the research project you are required to manage these devices as the property of your organisation. You must ensure that:
•The devices are password protected
•Are available for use (in conjunction with MI-Fi) to collect online data via SmartSurvey
•Are only used to collect and temporarily store offline research volunteer data if neither Wi-Fi nor Mi-Fi works in a location
•One or more devices are available to the project lead/manager to enable them to contribute to virtual (Zoom) meetings and to be able to access project activities
•Are registered with Cisco Meraki (or similar) so that in the event of loss or theft they can be wiped remotely
•Are used to help embed digital skills in CLMH courses to help build the digital confidence and competence of research volunteers
•Are set up, controlled and managed in such a way as to be consistent with your organisation’s safeguarding policy to protect those who use them and to prevent misuse.
5.5 / To use SmartSurvey as part of your MHFE Research Project which is a contractual obligation, you must read, agree, sign and return a copy of the Terms and Conditions, these include the following requirements on you:
  • The SmartSurvey account will only be used to collect participating research volunteers’ data for your CLMH project and for internal organisation surveys to staff you directly fund
  • The Live Learner Data Collection Survey will be used solely for collecting data from participating research volunteers’
  • To abide by the data protection requirements set out in the Data Protection Act.You should be aware that the data you are collecting could be subject to a Freedom of Information (FOI) request
  • Your named SmartSurvey administrator’s email account and telephone number must not link to generic contact details

5.6 / Prior to DfE making your online SmartSurvey live:
•All Users delivering or supporting the project must have completed the appropriate training (Administration and Learner Data Collection) through either attending the national training or locally organised in-house training, as appropriate to their role.
•Your project mustprovide evidence of data collection and analysis on the practise survey supplied to you specifically for this purpose.
•Administrators (and QA staff) must have completed Administrator training and know how to:
  • Obtain appropriate reports for their MIS staff, teachers etc. without divulging any data that could compromise the Research Volunteer’s data protection.
  • Amend or add additional information to a specific record. In this instance, a full log must be kept detailing both the changes made and the reason.
  • Provide a Research Volunteer with the data requested following a FOI request.
  • Remove a Research Volunteer’s data following a specific request from them to do so. In these circumstances a full record must be kept of the request and subsequent action. As all Research Volunteers give their consent to collect data before it is taken, removal of their data is at your own discretion.
  • All guidance, teaching, learning support and QA staff, directly involved in the project must have practised each stage (guidance, weeks 1, 3 and 6) of the Learner Data Collection Survey in advance of the collecting any Research Volunteer’s data via the live survey.
  • All guidance, teaching, learning support and QA staff directly involved in the project must have practised using the Discovering Potential Wheel prior to its use alongside the Learner Data Collection Survey.

6.