Terms of Reference – Save the Children’s Quality Learning Environment Monitoring Framework
End of Strategy Review
Background
Measuring and promoting evidence-based approaches to improve programme quality is an on-going priority in various sectors, especially where measuring change has traditionally been challenging. This includes measuring the extent to which educational learning environments are “of quality,” often an end to itself given the global agenda on quality and inclusive education (reiterated at the World Education Forum). Previous work in sectors such as School Health and Nutrition and Child Protection and the recent 2015 Education for All Global Monitoring Report provide numerous examples of the direct and indirect contribution of learning environments to improved learning outcomes.
Save the Children’s Quality Learning Environment (QLE) monitoring framework can be used to inform a broad and multi-dimensional assessment of the learning environment (i.e. pre-schools, schools and other learning contexts) and its ability to meet standards in promoting psychosocial and physical well-being of learners and staff, active teaching-learning methods and engagement with the community. Data collection tools derived from the framework are employed through a mixed methods approach, including interviews and focus group discussions with students, teachers, parents and community members, as well as classroom and school observations of teaching and learning activities and the presence (or absence) of teaching and learning materials and physical infrastructure.
The QLE monitoring framework is a core component of Save the Children’s 2016-18 education strategy, serving as an indicator to measure the quality of Save the Children’s education programs at the sub-national, national, regional and global levels. The QLE monitoring framework has been used in over 35 country programs, both for Basic Education and ECCD from 2012 to the present. The QLE frameworks are primarily used by country offices for education program design and planning and program implementation, and increasingly as a guide for advocacy and policy initiatives. QLE data are reported to donors, to the SCI CEO and Board; equally important is sharing this data back with (and holding Save the Children accountable to) children, schools, communities and governments Save the Children supports. It is aligned with Save the Children’s MEAL approach, wherein monitoring and evaluation are strongly linked with learning and accountability throughout the project cycle. As such this framework and resultant data are increasingly relied upon as critical program management tools within countries, and as a check on program quality at regional and global levels.
The QLE review will consist of two phases. The outputs of this consultancy represent phase 1. Findings and recommendations will form the basis of discussions amongst Save the Children’s Education Global Initiative (EdGI) on the way forward for the QLE framework in our next strategy period, taking into account Save the Children’s investments to date in QLE. Phase 2 will consist of implementing the agreed upon recommendations.
Purpose of the QLE Review
To review and assess the utilization, design/methodology, and application of the Basic Education and ECCD QLE monitoring frameworks by country programmes between 2012-2015, to inform subsequent revision and updating of QLE framework, methodology and associated tools.
Approaches/Criteria for Assessment of the QLE Framework
Specific objectives and the activities that will address these objectives can be categorized as follows:
Objective of QLE Review / ActivityHistory and Utilization: How has the QLE framework been used in the sampled countries over the last four years? / Map, review and assess SC country office application and use of QLE framework against criteria/approaches listed below.
Must include history of the QLE framework, including the evolution of the tools and adaptations that have taken place to changing contexts and priorities.
Design: Given the current design and methodology of the QLE framework, what is the framework most suited for? E.g. program planning and monitoring. / Review and assess the QLE framework, methodology and associated tools against a set of criteria/approaches (see below) and determine strengths and weaknesses of the framework and associated materials when considered against different measures of learning environments and means of measurement.
Include focus on if/how programming and policy practices are adapted based on QLE data analysis and recommendations to improve use of data throughout program cycle.
Application: What does Save the Children want to be the primary use(s) of the QLE in the future (e.g. establish links to learning outcomes)? Given the possible additional applications of the QLE framework, what changes are needed? / Provide recommendations informing revision and updating of QLE framework, methodology and associated tools.
The consultant is expected to assess the QLE framework against several criteria:
1. SPICED methodology: subjective, participatory, interpreted and communicable (learning), cross-checked and compared, empowering (accountability) and diverse and disaggregated. Includes non-psychometric reliability.
2. Efficiency and Effectiveness of QLE framework: have the processes been adopted for data collection, entry and analysis been cost and time effective? What alternatives should be considered? How effective have these approaches been in achieving program outcomes (e.g. changes to the school environment, creating a participatory process for defining and assessing quality which promotes local ownership and a voice for multiple stakeholders)?
3. Psychometric Reliability: Establish availability of data to assess psychometric properties of QLE (and appropriateness given applications of QLE framework)
a. Internal consistency/reliability of items within and across sub-domains
b. Inter-rater reliability of the observation based criteria in the QLE
c. Adequacy of the tools developed by country programs to collect data on the QLE items
4. Validity:
a. Content validity: MECE (mutually exclusive and collectively exhaustive) assessment of QLE framework. Comparison to other measures of the quality of learning environments, including indices or composite measures.
b. Criterion validity: Relationship of QLE framework to related learning outcomes
5. Sustainability: opportunities for cooperation and alignment of QLE framework with Ministries of Education (MoE) at national and sub-national levels; extent to which communities and schools can “own” the QLE (to enhance accountability, sustainability); understanding and utilization of QLE data by stakeholders such as schools and MoE.
6. Relevance of QLE Framework: does the QLE framework suit the priorities of Save the Children’s education interventions, in different contexts? Note: QLE in humanitarian responses is out of scope for this review.
Methodology
Specific questions to guide examination of the QLE framework against the above set of criteria/approaches will be developed with the consultant and included as part of the inception report. Generally, the methodology will include:
· Review of relevant internal and external documents (to be provided by Save the Children).
· Skype/email consultation with diverse sample of country programs:
a) Sampling: Purposive stratified sampling based on a list of countries that have been grouped according to their use, success with or challenges with the QLE framework and level of investment by SC.
b) Number of case study countries and methodology for collecting data to be agreed upon with consultant.
· Skype/email consultation with SC Member education and M&E advisors (as well as School Health & Nutrition teams, Child Protection Global Initiative), consultants, fellows and external experts (as appropriate).
· Interview list to be developed by the EGI M&E Work Group and reference group for this review.
Management of QLE Review
The overall QLE review process will be managed by the EGI M&E Advisor and an EGI Steering Group member, with the support of the EGI M&E Working Group. Active involvement by all EGI Work Groups is expected.
Time Frame (30-40 working days). Duty station: Flexible (desk work).
Task / DeadlineRecruit consultant / 15 September, 2015
Inception Report (including proposed methods, prelim literature review) / 01 October, 2015
1st Draft Report and presentation of results to date / 25 October, 2015
Final Draft Report / 05 November, 2015
Presentation final draft report via teleconference / 09 November
Required Skills and Experience
· Advanced university degree (Master’s degree or equivalent) in the social sciences (e.g. international education or development), or related technical field.
· Extensive experience in application of quantitative and qualitative data collection and analysis methods, particularly its application in monitoring and evaluation of international education/development interventions.
· Knowledge of and use of participatory M&E methods. Previous experience constructing and analyzing indices, composite measures.
· Proven ability to consolidate diverse views into feasible, pragmatic recommendations.
· Excellent English-language written communications skills. Proficiency in French or Spanish is desirable.
· Strong organizational skills and ability to work to tight deadlines as part of a team.
· Demonstrable computer literacy and knowledge of analysis and database software (e.g. Excel, Access, SPSS).
2