Contents
Introduction
Terms & Definitions
How to Craft an Outcome Statement
Characteristics of a Quality Results Framework
Distinct, logically-connected results
Clear, specific and measureable results
Ambitious, yet attainable results
Checklist of Review Questions
Finding a Common Language to Discuss Results Frameworks
Introduction
A central part of the foundation’s business model, Outcome Investing, involves designing and managing investments that drive toward measureable outcomes. The approach has three defining characteristics: outcomes at the outset, logical design, and managing toward results.
- Outcomes at the outset–Program Officers and grantees align early with our grantees on the few, most important results that define what success looks like for the investment
- Logical design - Define a set of logically connected results that are on the pathway toward achieving the investment outcomes
- Managing toward results – Program Officers and grantees identify the most relevant indicators to monitor progress over the course of the investment, and then managing toward those results
This guidance notewill help you practice Outcome Investing, with a focus on developing quality outcomes andresults frameworks. It includes practical suggestions for crafting results, illustrative examples, and a checklist of reviewquestions that will help you provide actionable feedback to your grantee.
TermsDefinitions
Table 1 outlines the Outcome Investingterms and definitions relevant to the Results Framework, with an associated example for each.The investment results – primary outcomes, intermediate outcomes, and outputs – are describedin the blue box.
Table 1. Terms, Definitions & Examples
Term / Definition / ExampleStrategic Goals / The 3-5 year results of an initiative, sub-initiative or portfolio toward which the investment contributes most directly (identified in the strategy scorecard) / The sustainable elimination of Visceral Leishmaniasis (VL) from Bihar over five years
Primary Outcomes / The overall change(s) in technologies, systems, populations or behaviors the investment seeks to achieve within the context of the investment timeframe / ReducedVL incidence in four districts of Bihar to less than 1 in 10,000 by 2016
Intermediate Outcomes / The changes in technologies, systems, populations or behaviors that need to be achieved in order to realize the primary outcome(s) / Increased coverage (to 95%) of VL -affected villages in four districts of Bihar with Indoor Residual Spraying that meets quality standards
Outputs / The goods, services, events or deliverables produced during an investment / Block-level micro-plans developed for all villages in four districts of Bihar where VL patients have been identified over the past five years
Activities / The actions taken or work performed to produce goods, services, events or deliverables / Collect data on where VL patients have been identified over the past five years
How to Craft an OutcomeStatement
An investment outcome is the highest level result or changewe expect to see from a single investment. The foundation does not measure impacts (e.g. decreased poverty or mortality) as part of measuring investment results, as these are long-term, sustainable changes that are more relevant to a set of investments, an initiative or a strategy.
At its most basic level, an outcome (primary or intermediate) describes the direction and type of change among a specific group of people or system. The following rubric can be useful when crafting or reviewing investment outcomes.
Example investment outcomes:
The components of the following outcomes are color-coded according to the rubric above.
- Increased use of self-paced math programs among 25school districtsin California by 2018
- Improved early care-seeking practices for pneumonia and diarrhea among caretakers from two regions in Nigeria by year 3 of the project
- Adoption of national sanitation and hygiene policies and strategies (including non-sewer basic sanitation)infourcountries classified as fragileby 2015
While we strive to include additional specificity in our outcomes during the planning stage, it is not always possible until after the investment has begun. This is particularly true for outcome targets as it is necessary to have baseline or other data available prior to the investment to determine “how much change” you would reasonably expect to see over the course of the project. Such dataare often not availableat the time of planning, so additional detail related to targets would have to be added after the investment has been approved.
Characteristics of a Quality Results Framework
Once Program Officers and grantees align on the outcomes that indicate success for the investment, we work together to build out a set of results that if achieved, will produce theprimary outcome(s). TheResults Frameworkis the tool that shows these results and the foundation’s strategic goals toward which they contribute.
It is important to note thatthe Results Framework is just a set of boxes. Developing thecontent that goes into those boxes takes time, conversation, and collaboration between Program Officers and prospective grantees, and sometimes support fromother foundation stakeholders.
A “quality” Results Framework is one that lends itself to both alignment between the foundation and partner, and good, useful measurement. This requires three things: (1) distinct, logically connected results, (2) clear, specific and measureable results, and (3) ambitious, yet attainable results. Each characteristic is described in greater detail below.
Distinct, logically-connected results
Outputs and outcomes should be distinct from activities. Activities describe what you as an implementer will do (e.g. develop resources, hold trainings, conduct outreach), while outputs, for example, tell us what is produced from those activities (e.g. new resources, trained staff, people reached). Activities can be included in the Proposal Narrative, as needed, but we do not measure them or manage the execution of them.
Outputs and outcomes should be distinct from one another. Outputs describe what the investment will directly produce (e.g. reports, tools, products, skilled staff), while outcomes describe the changes (e.g. in behavior, coverage, policies) we hope will be achievedby the investment. As the implementing organization, you will likely have more control over achieving outputs than outcomes.
When an investment has two levels of outcomes, the intermediate outcome should representchanges that are criticalsteps on the pathway to achieving the primary outcome(s).
Outcomes should be distinct from strategic goals. Strategic goals are higher-level initiative, sub-initiative, or portfolio level resultsthat investments are contributing to (and that signal an investment is strategically aligned). Outcomes, however, are changes considered achievable giventhe specific investment context and timeframe. In most cases, it takes multiple investments to achieve a strategic goal.
Investment results should be logically connected, with one level of result leading to the next. A logical set of investment results should clearly show the causal, or if/then, relationship between a related set of results. For example, if staff are trained on maintaining and managing vaccine cold chains (output), then it would be reasonable to expect an outcome of reduced vaccine wastage to be achieved. You should include a discussion of assumptions in the proposal narrative to describe the important events, conditions, or decisions outside the control of the project that are necessary for the results logic to hold.
Logically connected resultsshould not haveany gaps, where the reader would need to inferadditional details to understand how the outputs will lead to the outcomes. Imagine if the logical flow shown in Figure 1 only included the output and primary outcome. A reader would likely identify a gap between the output of ‘village level demonstrations held’andthe primary outcome of ‘use ofthe storage bags by farmers’.There are several steps (or changes) that need to happen between these two results. Adding the intermediate outcome of ‘increased sales of bags by merchants’ helps to tell the story of howvillage level demonstrations of the new storage bags would lead to farmers usingthem.
Figure 1. Logical Flow of Results
Some straightforward investments may end in outputs (e.g. a new rapid drug susceptibility test for TB). When this is the case, it is important that the outputs are logically linked to other investment results that together contribute to the achievement of a strategic goal(e.g.WHO endorsement of two new TB molecular diagnostics by 2016). This information can be captured in the strategic fit section of theShort Form Results Framework and Tracker, which is the tool that you will receive if you and your Program Officer agree that the investment willend in outputs.
Clear, specific and measureable results
Investment results should be clear, specific, and measureable.Results language should be easy to understand by any reader, not just a few insiders that know the project well. The Results Framework does not ask about measurability, as this information is included in the Results Tracker. However, The more precise the results language, the less time the Program Officer will spend asking clarifying questions to reach alignment with you, and the easierit will be to identify indicators for those results in the Results Tracker.
A few tips:
- Avoid ambiguous, vague terms, and do not assume familiarity with the content.
- Make sure every result reflects a single change, not multiple changes together in one statement.
Ambitious, yet attainable results
Investment results should be ambitious, yet attainable. The foundation supports ambitious ideas and innovative projects, but at the same time expects the results from our investments to be feasible given yourorganization’s capacity, the timeframe of the work, and the financial resources available.
Case Study:
The Malaria team made a 21 month, $30 million investment for their grantee to lay the groundwork for demonstrating the feasibility of malaria elimination in four medium-to-high burden sub-Saharan African countries. The granteeoriginally submitted a very ambitious primary outcome of eliminating local malaria transmission in the four countries. After much discussion, the foundation and grantee realized that the pathway of change for the investment was not logical to lead to such high-level results. They were able to scale-back the original primary outcometo a more reasonable result around an increased use of evidence to inform policy andpractice.
Note that all of the “improved examples” are just that, improved over the poor example, but still not perfect. Designing quality investment results is messy work and there is no one “right” way to approach it. Remember that it is not about getting to perfect, but rather getting to the point where both you and the Program Officer are aligned on the set of investment results and feel jointly responsible for their achievement. Foundation measurement and evaluation staff are available to help you and the Program Officer get to this point..
Checklist of Review Questions
The following questions are ones you might ask yourself as you review a draft of the Results Framework. They help identify problems or common pitfalls when drafting a set of logical results. If any statements are true, then use the associated discussion prompt to help guide your internal conversation or a conversation with your Program Officer about how best to revise the Results Framework.
Distinct, logically connected results
Checklist of ReviewQuestions / If so, then…Do any of the outcomes or outputs describe how you will do the work or the specific tactics that you will use? / The investment results are likely worded as activities. Ask yourself about the end toward which the activities will be performed, and consider revisingthe results.
Are you having trouble distinguishing between what the investment will produce (i.e. outputs) and what the investment will change (i.e. outcomes)? / The investment results are likely not distinct enough. Trymapping out the ‘if, then’ logic of the results, which should help clarify what needs to be produced (i.e. outputs) in order to lead to change (i.e. outcomes).
Do any of the outcomesidentify a change that you think multiple investments are working towards, and that won’t likely be achieved by the end of a single investment? / These outcomes are likely worded as strategic goals or identify the broader vision or aspiration for the investment. Ask your Program Officer to discuss the strategic goals with you and other related investments to clarify where your investment fits in andwhat your investment can feasibly achieve within its context and timeframe. You and the Program Officer may decide to scale-back the results to reflect what you’re really trying to get out of this single investment.
Are you having trouble believing that the outputs will really lead to the achievement of outcomes? / There are likely gaps in logic. Try mapping out the ‘if, then’ logic of the results, and talk to your Program Officer about whereadditional outputs or outcomesmay be needed to fully tell the story.
Clear, specific and measurable results
Checklist of ReviewQuestions / If so, then…Do any of the results include vague or ambiguous terms that peer reviewers or your Program Officer might question what they mean? / The results maynot be clear and/or specific enough. In order to determine how how they might be framed more concretely, you might ask yourself:What does X look like? How will I know when X has occurred?This can help you reword the results to include more clear and/or specific language.
Do any of the results contain multiple changes strung together in one statement? / The results may not be clear enough. Consider breaking up the content of those results so each contains a single, measurable change. Also consider the relationship between the two changes and determine where best in the results framework to place each result.
Do any of the results lack detail aboutwhen the result will be achieved, where the result will be seen, and/or how much change is likely to be achieved within the investment timeframe? / The results could be more specific. Review the rubric on crafting an outcome statement found on page 3 of this document, and add specificity about timeframe, geography, and targets when it is feasible and relevant to do so.
Are you having trouble identifying howyou will know when the results have been achieved (or whether the investment is on track towards achieving them)? / The results may not be specific enough to be measurable, or the wrong indicatorsare used in the Results Tracker.Tryeitherrevising the results language to get increased clarity on what the specific changesare, or identify more appropriate indicators on the Results Tracker.
Ambitious, yet attainable results
Checklist of ReviewQuestions / If so, then…Does the Program Officer expect you to achieve more than what you have documented within the investment timeframe and resources available? / The results may not be ambitious enough. Think about “stretch” results for the investment, and be sure to right-size expectations about what will happen if those results are not met.
Do any of the results seem out of reach given the investment’s work streams, timeframe, and planned resources? / The results may be unattainable for the investment. Talk to the Program Officer about scaling back results or scaling upfunding and/or resources to match the type of results you are expected to achieve by the end of the investment.
Finding a Common Language to Discuss Results Frameworks
The Outcome Investing practice involves conversations between Progam Officers and partners to define and align on the content in the Results Framework. However,the foundation and our grantees often use different terminology and/or have varying definitions of the components in a Results Framework (e.g. outputs, intermediate and primary outcomes).The nuanced differences and similarities highlighted below is intended to help clarify the foundation’s language and expectations around the Results Framework.
Other Organizations’ Results Frameworks / BMGF Results FrameworkScope of Results Framework / Other organizations’ Results Frameworks articulate everything from resources/inputs to ultimate results and impact / The foundation intentionally focuses on outputs and outcomes (primary and intermediate), to allow the PO to focus on and manage to investment results rather than investment activities
Definition of Outcomes / Outcomes across organizations broadly mean the same thing: the change in behavior or systems that the project/investment seeks to achieve / In addition to the basic definition of outcomes, the foundation articulates both “primary” outcomes (i.e. overall changes) and “intermediate” outcomes (i.e. short- to- medium changes) in the Results Framework
Linking Outcomes to Higher-level Results / Many organizations require articulation of the logical link between outcomes and a higher level change or impact; e.g. USAID’s Results Framework links to a Country Development Cooperation Strategy Goal / In the foundation’s Results Framework, the higher level change an investment is contributing to is represented by strategic goals at an initiative, sub-initiative, or portfolio level, and is described in the Strategic Fit section; there is no expectation to measure impact or the strategic goal at the investment level
Articulating Assumptions / Most Results Frameworks require articulating assumptions (the general conditions under which the Results Framework will hold true) / The foundation’s Proposal Narrative and Results Framework do not include a section referred to as “assumptions”; instead, the Proposal Narrative includes a few optional questions on “critical relationships” and “external factors” that address areas commonly included as assumptions
Table 2 provides a more detailed comparison of our results language to some of our partner organizations, for reference.