Latin America and the Caribbean Region

Latin America and the Caribbean Region

Country Department II

Human Resources Operations Division

Economic Analysis in Education Projects:

Guidelines

12 October 1995

Introduction

The Bank’s decision to start lending for the education sector in 1963 illustrated the recognition of the economic importance of education as an investment in human capital. Total education lending over the last thirty years has represented close to US$20 billion through 500 projects in over 100 countries, and the Bank has become the largest single source of external finance for education in the developing world. Yet, while Bank projects in other sectors have always been evaluated against strict economic criteria, education projects have not been subject to the same degree of scrutiny. They have been exempt, in particular, from the requirement to calculate rates of returns to justify the proposed investment.

The long standing practice of not including any type of cost-benefit or cost-effectiveness analysis in education projects finds its origin in the perceived difficulty to measure with precision the outcomes of educational investments, due to the lack of data and to methodological limitations. More recently, however, the increasing availability of data sets on learning achievement, expenditures and incomes, and refinements in the methodological approaches used to estimate the returns to education have made it more feasible to apply cost-benefit analysis to the evaluation of education projects.

More economic rigor in the analysis of education projects is also warranted by the findings of a recent OED study on the quality of economic analysis in Bank projects (OED, 1995). The study concludes that the quality of the economic analysis is directly linked to the probability of success of a project. It also notes that a large proportion of education projects have failed to assess the relative costs and benefits of alternative methods to achieve a given set of goals and to pay adequate attention to the recurrent cost implications of the investments under consideration.

To avoid that important decisions regarding educational investments continue to be made without proper consideration of key economic dimensions, it is crucial to rely on existing analytical tools for the evaluation of the relative merits of alternative interventions and for the assessment of the financial sustainability of educational investments. The purpose of this note is to offer a set of guidelines in this respect, to be used for the preparation of education projects in the Department. After discussing the key questions that need to be asked for each project, this note outlines the most appropriate tools of economic analysis available to answer these questions. A few best practice cases are presented as technical annexes. This note is intended to complement the Handbook on the Economic Analysis of Investment Operations that is currently being finalized by OPR.

Dimensions of Economic Analysis

The purpose of economic analysis is to guide investment decisions and to ensure the best use of available resources. The following sequence of key questions need to be asked each time an intervention is envisaged in the education sector:

• in which educational sub-sector should one invest?
• what kind of investment should be considered in a given sub-sector?
• are the planned investments sustainable?
• what are their equity implications?

Choice of Sub-Sector. The first decision concerns the educational level where it makes most sense to invest. This important decision is usually made prior to project identification, when the overall country assistance strategy is defined and specific priorities are set for the education sector within that context[1]. Since the present note focuses on the economic analysis requirements for project preparation, it addresses mainly the last three questions raised above, i.e. the type of investment envisaged, the sustainability prospects, and the equity dimension.

Choice of Intervention. There are different ways of achieving educational goals. For example, measures to increase access at the primary level include school construction and/or rehabilitation, teacher recruitment and/or deployment, promotion of private education, introduction of multiple shifts and/or multigrade classes, and single-sex schools (Lockheed et al, 1991). Similarly, to improve the quality of basic education, one may rely on teacher training activities, curriculum reform, the distribution of textbooks and other relevant educational materials, and the strengthening of evaluation mechanisms. Therefore, selecting the most appropriate input or set of inputs to reach the chosen objectives needs to be based on rigorous economic criteria which allow to compare the costs and benefits of each measure or combination of measures. Too many Bank projects propose interventions representing a chosen package of inputs to be financed without any indication that other possibilities have been reviewed and rejected on the basis of measurable economic criteria.

Choosing among various alternatives implies the ability to measure the costs and benefits associated with each alternative. The cost side is the easier part insofar as the various inputs financed by the project have a direct monetary cost (and the indirect cost reflecting the opportunity cost linked to foregone incomes). At the very least, each project should include a table comparing various alternative approaches in terms of unit cost of each intervention and number of beneficiaries.

Notwithstanding the difficulty involved in calculating educational benefits which do not have a monetary value, each project requires a clear definition of expected outputs and benefits in terms of increased access, reduced inequity, increased efficiency, and improved quality. The task manager should specify explicit criteria for success in the form of concrete and measurable targets (enrollment rates, efficiency rates, levels of learning achievement) of what would be achieved as a result of the investment. The discipline imposed on the formulation of projects of having, first, to set explicit criteria of success and, second, to substantiate why the investment would be expected to meet the criteria, should lead to better projects. Projects that have midterm reviews should indicate how these criteria are to be used. When the projects include pilot experiments, consideration should be given to testing different alternatives and provision should be made for learning from the experience.

Sustainability. Another key element of the economic analysis is to assess whether the investment planned is sustainable during and beyond the project life. Supplementary financial resources are usually necessary to meet the additional recurrent costs linked to the investment supported by the project. For example, new schools imply additional staffing and maintenance expenditures. Furthermore, some of the investments may need to be continued to ensure sustainability beyond the project life. For instance, the establishment of a textbook production and distribution system requires the capacity to replenish damaged and outdated books on a regular basis. The purpose of the sustainability analysis is to measure the financial implications of the project in order to identify potential financing gaps and avoid financing shortfalls.

Asking the sustainability question does not mean only looking at the financing capacity of the public sector. Education investments may also generate private economic benefits that could create a “willingness to pay” among the potential beneficiaries. For instance, a new training program with good job market prospects could operate on a cost-recovery basis. This would need to be taken into account when assessing the financial sustainability of the proposed program.

Equity. Another important dimension of the economic analysis is to check the consistency of the proposed investment with the Bank’s poverty reduction strategy in the country in question. Education being a significant determinant of the distribution of income and prestige in any society, examining the equity impact of education projects cannot be dispensed with. There are also efficiency considerations as many studies indicate that investments in basic education have higher social benefits when they are targeted to the poorer segments of society than when they are uniformly offered to all families (Schultz, 1994).

Looking at these distribution issues means measuring variations in rates of access and success and in levels of educational attainment among different population groups. Dimensions of equity (and inequity) can affect the distribution of population by gender, by ethnic or language characteristics, by geographical location, and by socio-economic origin. Differences in the distribution of educational benefits are reflected, in turn, in differences in the distribution of public subsidies for education among various population groups.

In summary, the purpose of the economic analysis of education projects is to assess (i) whether the investment is justified, i.e. whether the expected benefits (physicial indicators of educational achievement, private or social returns) are greater than the costs, (ii) whether the investment is financially sustainable in the long run, and (iii) whether the distribution of costs and benefits is favorable to the poorest segments in society, or at least generally equitable.

Tools of Economic Analysis

What makes the economic analysis of education projects different from projects in other sectors is the challenge of measuring the value added of an educational investment. It is often difficult to identify and calculate monetary benefits as direct outcomes of the project. There are, however, several methods to assess benefits and relate them to the costs of the investment. This section focuses on two of these methods, cost-effectiveness analysis and cost-benefit analysis. It is argued that cost-effectiveness is more suitable to evaluate the economic impact of educational investments at the lower levels of schooling (preschool, primary education, junior secondary) while cost-benefit analysis can be used more adequately to compare the relative merits of investments in different types of education (e.g., vocational versus general) that have easily measurable labor market outcomes. The second last part of the section reviews ways of analyzing the sustainability and equity dimensions of the projects under consideration.

Cost-Effectiveness Analysis

Cost-effectiveness analysis refers to the evaluation of alternatives according to both their costs and their effects in terms of educational outcomes. This approach does not calculate the outcomes in monetary terms, but in terms of educational effectiveness, usually on the basis of non-monetary educational goals and targets. Typically, educational evaluation and decision making must focus on the choice of an educational intervention among various alternatives for meeting a particular objective, such as increasing test scores or reducing dropouts. In these cases, the results of alternative interventions can be assessed according to their effects in improving areas learned by students or on the number of potential dropouts that remain in school.

A cost-effectiveness analysis can be undertaken when costs are combined with measures of effectiveness and all alternatives can be evaluated according to their costs and their contribution to meeting the same effectiveness criterion. A cost-effectiveness analysis proceeds by comparing the different outputs expected to be achieved with various combinations of inputs. The expected outputs are estimated on the basis of prior experience with the impact of similar programs, preferably in the same country. The purpose of the comparison of expected benefits and corresponding costs is to: (i) identify the lowest cost of attaining a desired level of output or effectiveness, or (ii) choose the greatest level of output or effectiveness that can be achieved for a given cost. If the costs of two projects are the same, the cost-effectiveness analysis can identify which one achieves the most output, measured either quantitatively or qualitatively. If, however, effectiveness is the same in terms of equal standards of achievement, the purpose of the analysis is to identify the least costly alternative.

The case for using cost-effectiveness analysis is that it integrates the results of activities with their costs in such a way that one can: (i) select those activities that provide the best educational results for any given cost or (ii) provide any given level of educational results for the least cost. When assessing alternatives, it is important to emphasize that both the cost and effectiveness aspects are important and must be integrated.

The cost-effectiveness approach has a number of strengths. Most importantly it merely requires combining cost data with the effectiveness data that are ordinarily available from an educational evaluation to create a cost-effectiveness comparison (see Table 1 for a sample of effectiveness measures), and it avoids the difficult monetary valuation of the educational outcome. It builds on rather standard approaches to evaluation, simply adding a cost dimension to an overall evaluation design. Further, it lends itself well to an evaluation of alternatives that are being considered for accomplishing a particular educational goal. In fact, it can often be done by taking the normal evaluation design and integrating it with a cost component. The measures of educational effectiveness can be those that a decision maker would normally consider. This means that if competent evaluators are available to contribute to the study, the addition of the cost dimension is all that is required. Finally, cost-effectiveness evaluations generally require less time and resources than cost-benefit studies, because assessments of effectiveness can be done during the operation of the programs under evaluation.

Table 1. Examples of Effectiveness Measures

Source: Levin (1990).

In contrast with the cost-benefit approach, cost-effectiveness analysis frees the evaluator from the requirement to measure benefits in monetary terms, a somewhat constraining requirement. Cost-effectiveness analysis enables a more direct measure of impact in terms of the effectiveness of the intervention on a specific outcome. This outcome can be addressed according to its own attributes, rather than having to convert it into monetary units. Thus, student learning can be assessed in terms of test scores and dropout prevention programs can be evaluated according to the number of potential dropouts who complete a program; etc. The cost-effectiveness approach, then, gives a straightforward evaluation of alternative programs or inputs that have the same objectives.

To use the cost-effectiveness approach, it is first necessary to determine the program objectives and to define an appropriate measure of effectiveness. Then the alternatives that will be evaluated must be specified. Given these requirements, it is possible to design an evaluation of the alternatives on the particular criterion of effectiveness that has been established and to obtain cost information for each alternative. Finally, the cost and effectiveness data can be combined into cost-effectiveness ratios that show the amount of effectiveness that can be obtained for an estimated cost. Since this ratios can be compared among alternatives, it is possible to provide information to decision-makers on which alternatives seem to be most economical in terms of cost with respect to the measure of effectiveness under consideration.

Table 2 presents a hypothetical set of cost-effectiveness results for remedial science programs in a Latin American country. Four alternative methods of improving science performance are evaluated according to their cost and effects. Assuming that students have been randomly assigned to each program and that the preferences of the evaluator are linearly related to effectiveness, the effectiveness of each program is determined by test scores measuring learning objectives for the science curriculum. The cost-effectiveness ratios show the cost per student for a one-point improvement, on average, in the test scores. The most effective approach when costs are ignored (small groups) is ranked third when costs are taken into account. The most cost-effective method (additional hours of class per week) is only the third most effective. This example illustrates that the most effective approach is not always the most cost-effective. Without an analysis of costs it would be impossible to know this[2].

Table 2. Hypothetical Cost-effectiveness Results for Remedial Science Programs

*Computer-assisted instruction

Another way of using cost-effectiveness analysis focuses on the least-cost method to produce an additional unit of output. For each input, the dollar amount required to purchase enough of that input to raise achievement scores by one point is calculated. Three steps are required in this analysis. First, the regression coefficients from an education production function are used as estimates of the marginal achievement gains attributable to different inputs[3]. Second, the total economic costs per student of each individual inputs are calculated. Third, alternative inputs are ranked in terms of their contribution to improving scores relative to their costs. The ones with the largest achievement gains relative to costs are the most cost-effective.

A simple example can be used to illustrate this method, based on the results of an evaluation study of a Bank-supported educational intervention --the Northeast Rural Primary Project (Harbison and Hanushek, 1992). Table 3 presents the cost-effectiveness analysis for Portuguese achievement for second graders in 1983. The first column shows the cost per student of each input. The second column shows the achievement gain associated with each input. The third column shows the cost-effectiveness ratios, that is the quotient of the achievement gain per student attributable to the provision of the input divided by the annual cost per student of providing the designated input--achievement gain per dollar spent. Therefore, the higher the ratio, the greater the policy priority associated with that input. In that particular example, the distribution of textbooks and writing materials led to the largest achievement gains. The provision of school furniture and training do not appear to have been cost-effective policies.