Estimates in Project Evaluation Time for a New Approach

Estimates in Project Evaluation – Time for a New Approach ?

1Introduction

1.1.1This paper provides a review of the literature on project cost overruns and time delays in major infrastructure projects.

1.1.2The phenomenon of ‘optimism bias’ in which major infrastructure projects consistently and significantly run over budget in terms of time and money has been extensively examined. Much of the literature has focused on the explaining the overruns in terns of individual cost assumptions such as inflation, raw material costs or changes in project scope.

1.1.3More recent work undertaken by Professor Flyvberg and colleagues has focused on projects as ‘political’ events subject to a wide range of external factors such as personal incentives which can drive over runs

1.1.4This paper draws extensively on work undertaken by RGL, Frontier Economics and Aecom consulting in their work for the European Commission on evaluating ERDF projects.[1]

1.2Time delays and cost overruns are a significant problem in infrastructure projects

1.2.1Cost overruns and delays occur because of inaccurate estimates leading to higher actual costs, and longer construction periods, than anticipated. Numerous studies have identified a systematic tendency for large infrastructure projects to have significant overruns and delays. This tendency of projects to overrun has been termed project optimism bias.

1.2.2For example, Flyvbjerg et al (2003) found that 9 out of 10 projects, from a sample of 258, across 20 countries and 5 continents, were subject to cost overruns.[2]In a recent study for the Commission, cost overruns in 51 projects were assessed which showed cost overruns averaging 21%, as illustrated in Table 1 below.

Table 1: Cost overrun summary by country and sector – average percentage differences between estimated and actual costs

(%)/ (number of projects) / Rail / Road / Urban Transport / Water / Energy / Weighted average by sector
Germany / -4.3% (6) / -10.0% (3) / -6.2%
Spain / 12.8% (6) / 30.7% (1) / 17.4% (2) / 15.8%
France / 32.9% (1) / 32.9%
Great Britain / 110.7% (1) / 110.7%
Greece / 74.3% (2) / 19.7% (8) / 20.1% (2) / 0.0% (1) / 26.6%
Ireland / 2.1% (5) / 74.1% (1) / 14.1%
Italy / 62.4% (5) / -5.0% (2) / -0.9% (1) / 37.6%
Poland / 19.7% (2) / 80.9% (2) / 50.3%
Portugal / 9.0% (1) / 3.3% (4) / 4.4%
Weighted average by Member State / 26.9% / 9.4% / 45.4% / 11.3% / 20.7% / 21.2%

Source: RGL, Frontier Economics, Aecom[3]

1.2.3Following Flyvbjerg (2005), explanations for time delays and cost overruns can usefully be grouped together as technical, psychological and political-economic explanations.[4]

1.3Technical explanations

1.3.1Technical explanations for delays and overruns are extensively examined in previous evaluations. They include imperfect forecasting, inadequate data and honest mistakes due, for example, to a lack of experience with infrastructure cost forecasting or with forecasting costs for certain types of infrastructure. They might also include inherent problems in predicting the future.

1.3.2Technical explanations for cost overruns/delays in transport infrastructure are also the subject of a growing academic literature inspired by Flyvbjerg et al (2004) and continuing with Flyvbjerg (2008), the latter looking specifically at urban transport projects.[5]

1.3.3DG REGIO’s Guide provides a set of key determinants of delays and cost overruns. These are re-presented from the Guide in Figure 1 below.

Figure 1: Key determinants of infrastructure project delays and cost overruns

Source: European Commission DG XVI (1998), ‘Understanding and Monitoring the Cost-Determining Factors of Infrastructure Projects’, A User’s Guide, Brussels

1.3.4The Commission identified a lack of effective project management as the main cause of cost overruns. Other important factors were design changes, and input price inflation due to delays. Less important factors were land acquisition problems, unexpected ground conditions and difficulties with contractors. The Guide’s summary of the relative importance of each of the factors for different categories of project cost is re-presented from the Guide in Figure 2 below. A large circle indicates a major effect and a small circle indicates a minor effect.

Figure 2: Common causes of cost overruns and project time delays

Source: European Commission DG XVI (1998), ‘Understanding and Monitoring the Cost-Determining Factors of Infrastructure Projects’, A User’s Guide, Brussels

1.3.5ECORYS Transport’s (2005) study of 60 projects co-financed by the Cohesion Fund, in the period 1993-2000, observed similar factors to the DG REGIO Guide in explaining cost overruns.[6] Their report notes that the predominant causes of cost overruns, namely design changes, inflation impacts due to time delays and site contractor issues, all point to inadequate project preparation. The report also considers local community involvement in projects to be important.

1.3.6The ‘Ex Post Evaluation of Objective 1 1994-1999’ by ECOTEC (2003) compares water infrastructure projects with similar large-scale transport projects. They use data from France, Germany, Greece, Ireland, Italy, Portugal, Spain and the UK. They found that water projects were more susceptible to delay and cost overrun – 70% of water projects go over budget, compared to 60% for transport projects. Furthermore, water projects were found to be subject to delays of in excess of 12 months, greater than the delays for the transport projects.

1.3.7The Nichols Report (2007) on 13 of the UK Highways Agency’s largest road projects found that inflation, inaccurate cost estimation and inadequate project definition were the most important causes of cost escalation.[7] While the results are not entirely representative, however, because only the 13 largest schemes were examined, they do provide a good indication of the variety of aspects that need to be accounted for to get more accurate cost estimates.

1.3.8The UK National Audit Office (NAO, 2007) reported on a more extensive sample of road projects under the Targeted Programme for Improvement (TPI) and Local Transport Plan schemes.[8] The study supports the findings of the Commission, namely that poor project management, design changes and inflation are the most significant causes of cost overruns.

1.3.9Flyvbjerg et al (2004) conducted a statistical analysis of cost overruns and found that they could be associated with the length of the project implementation phase. They also found that the influence was not significantly different for rail, fixed-link (bridges and tunnels) or road projects. The authors also found project size to be significant for bridges and tunnels, but not for road or rail projects. Weak accountability measures for delays and overruns were also a significant influence and probably more significant than the effect of ownership incentives.

1.3.10Bordat et al (2004) made the distinction between excusable delays (due to force majeure) and non-excusable delays (attributable to contracting agencies and contractors, and therefore preventable).[9] Contracting agency errors included planning and design deficiencies such as incorrect estimates of work quantities. Contractor errors included unnecessary work, work that did not follow the design plans, and work that did not meet contract specifications. Excusable delays due to unforeseen circumstances included site conditions that differed from those described in contract documents. Majid and McCaffer (1998), upon whose work Bordat et al (2004) is built, identified 12 main causes of project time delays. They pinpoint materials, equipment, and labour-related delays as the major cause of contractors’ performance delays.

1.3.11In the recent evaluation of ERDF projects, design and legislation changes were identified as the most common source of overruns, as shown in Figure 3 below.

Figure 3 ERDF Evaluation

Source: RGL, Frontier Economics, Aecom[10]

1.4Psychological explanations

1.4.1Psychological explanations derive from the ‘planning fallacy’, the systematic tendency to be overly optimistic about the outcomes of planned actions. In infrastructure project planning, the planning fallacy leads to ‘optimism bias’, that is, the systematic tendency to underestimate costs, completion times and risks and to overestimate the benefits of planned actions.

1.4.2This category of explanation was inspired by the work of Kahneman (1979, 2003) and Kahneman and Lovallo (1993) inspired this type of explanation for overruns and delays.[11] They relate the planning fallacy to biases at the cognitive level, i.e. ‘errors’ in the processing of information that result from ‘delusional optimism’. By examining the technical and political-economic explanations, one can, according to the authors, better understand how to resolve or, rather, correct for optimism bias.

1.4.3Kahneman et al’s work on decision-making under uncertainty provided the foundation for the concept of Reference Class Forecasting, which is based on the presumption that past projects tend to be more similar to planned projects than normally assumed and are, therefore, a means of increasing the accuracy of forecasting. Rather than focusing only on the specific constituents of the planned projects (the ‘inside view’), there should be equal if not more focus on the outcomes of similar projects that have already been completed (an ‘outside view’).

1.4.4Flyvbjerg and Cowi (2004) developed an applied method of Reference Class Forecasting for the UK Treasury.[12] Their report advocates the use of explicit, empirically based optimism bias uplifts to produce more realistic forecasts of individual project capital expenditures. The study generated benchmark uplifts by developing probability (or frequency) distributions of cost overruns for the various classes of project under consideration.

1.4.5Planners and promoters can choose the uplift that corresponds with the level of risk of cost overrun that they are willing to accept. So, the authors state, they should adopt the 50thpercentile uplift if they were willing to accept a high risk of cost overrun and the 80th percentile if such a level of risk was unacceptable. (Note that the percentiles referred to here can be thought of as 1 minus the level of risk that the project planner is willing to accept. Therefore, the 80th percentile corresponds with an acceptable risk of cost overrun of 20 per cent. The lower the level of risk of cost overrun that is acceptable, the higher the optimism bias uplift needs to be.)

1.4.6The authors found that, for road projects, optimism uplifts range from 15% at the 50th percentile to 32% at the 80th percentile. For rail projects (including urban transport), the uplifts range from 40% at the 50th percentile to 57% at the 80th percentile. Fixed link uplifts range from 23% at the 50th percentile to 55% at the 80th percentile.

1.4.7The report also asserts that few project planners have a direct interest in avoiding cost overruns. As a result, they recommended that the UK Department for Transport apply optimism bias uplifts that are supported by:[13]

• establishing realistic budgeting as an ideal, and de-legitimising over-optimistic budgeting as a routine;
• the introduction of fiscal incentives to avoid cost overruns e.g. through requiring local co-financing of project cost escalation where possible;
• formalised requirements for high quality cost and risk assessment at the business case stage; and
• The introduction of independent appraisal.

1.5Political economy explanations

1.5.1This is the most recent framework for explanaining overruns and delays and addresses the possibility that delays and cost overruns may be the result of deliberate misinformation. In other words, that project planners and promoters misrepresent timescales and cost/benefit projections in order to win favour for the project and get it started. Important work in this area includes Flyvbjerg et al (2004), which deal with the cost side, and Flyvbjerg et al (2005), which deals with the demand side (demand being the factor driving expectations about future benefits).[14]

1.5.2Flyvbjerg et al (2003), in their study of 258 transport infrastructure projects completed between 1927 and 1998, observed that ‘no learning appears to be taking place’ and that cost underestimation and subsequent cost overruns ‘are allowed to continue unchecked decade after decade’.

1.5.3The authors state that we may be in an equilibrium in which ‘strong incentives and weak disincentives for cost underestimation and related escalation may have taught project promoters that cost underestimation pays off.’ Flyvbjerg et al (2004) tested this explanation of cost overruns and, as reported in Flyvbjerg et al (2003), found that cost underestimation is ‘used strategically to make projects appear less expensive than they really are in order to gain approval from decision-makers to build projects’.

1.5.4Most recently, Flyvbjerg, Garbuio and Lovallo have provided a useful framework which seeks to explain optimism bias as a result of ‘delusion’ and deception.’[15] Within the explanation of delusion, the authors include ‘planning fallacy’ and ‘anchoring and adjustment’. Planning fallacy is discussed in 1.4.1 above. ‘Anchoring and adjustment’ refers to the observation that planners find it difficult to move away from initial estimates even if they know they are unrealistic. This may perhaps be due to the negative connotations associated with increases in estimates during the planning phase associated with a reduction in the probability that the project will actually happen.

1.5.5The idea of systematic ‘deception’ is used to explain political and agency sources of cost overruns. Their analysis focuses on the ‘principal-agent problem’ and ‘sources of strategic deception.

1.5.6The principal agent problem arises when funders of infrastructure project (ultimately tax payers) are separated from the project by a chain of intermediaries including contractors, consultants, local governments and departments andnational bodies. The authors identify a number of sources of deception which can occur at each interface of agent and principal including differences in self interest, the presence of asymmetric information, different risk preferences, different time horizons andfinally diffuse or asymmetric accountability. For example politicians may be more concerned with securing approval for flagship projects in the short term, rather than long term value for money for tax payers.

1.6Overcoming Project Optimism bias

1.6.1Flyvberg et al have suggested two approaches to overcoming project optimism bias: correcting incentives and reference class forecasting.[16]

1.6.2Minimising project cost over runs requires all parties to be incentivised to accurately estimate projects and complete projects on time and to budget. For example, all public bodies should be required to contribute to funding overruns. Contractors can be incentivised using a range of project structures such as public-private partnerships and Design, Build, Finance Maintain and Operate (‘DBFMO’) contracts.

1.6.3Reference class forecasting is based on an argument that the most reliable estimate of what a project will cost is what previous projects have actually cost. This approach requires cost distribution data to be gathered on similar projects. The relative position of the project to be forecast within the distribution needs to be considered and used to derive a wholly external view of what the project can be expected to cost.

1.7Conclusion

1.7.1Most of the literature has focused on technical explanations of cost overruns and time delays, typically cataloging a wide range of sources of cost overrun and building a list of key assumptions which planners have got wrong.

1.7.2The more recent work by Professor Flyvberg and his colleagues introduces a new ‘political’ dimension into the analysis of project optimism bias which suggests stakeholders incentives need to be structured to correct biases that might otherwise contribute to delays or overruns. In addition the use of external reference class forecasts is recommended as a means of cross checking the strong forces which have been identified as stifling the preparation of robust bottom-up forecasts.

1.7.3Flyvberg’s new reference class forecasting approach has not yet had time to be rigorously tested in practice. At the very least however, it does provide project funders with a powerful analytical tool to use when appraising projects. It should also provide project planners with useful information which will hopefully help them build and develop more robust estimates as they consider why their own bottom-up estimates are less than those developed using a reference class approach.

[1]Ex Post Evaluation Of Cohesion Policy Programmes 2000-2006, Work Package 10, ‘Efficiency: Unit costs of major projects’ Final Report, 25 October 2009 RGL Forensics, Frontier Economics, Aecom

[2]See Flyvbjerg, Bent, Mette Skamris Holm and Soren Buhl (2003), ‘How common and how large are cost overruns in transport infrastructure projects?’Transport Reviews, Vol. 23 No. 1

[3]See footnote 1

[4] See Flyvbjerg, Bent (2005), ‘Policy and Planning for Large Infrastructure Projects: Problems, Causes, Cures’, World Bank Policy Research Working Paper 3781, World Bank, WashingtonD.C.

[5] See Flyvbjerg, Bent et al (2004), ‘What Causes Cost Overrun in Transport Infrastructure Projects?’ Transport Reviews, Volume 24, No. 1, January and Flyvbjerg, Bent et al (2008), ‘Comparison of Capital Costs per Route-Kilometre in Urban Rail’, EJTIR, 8, no.1.

[6]ECORYS Transport (2005), ‘Ex post evaluation of a sample of projects co-financed by the Cohesion Fund (1993-2002)’, Report to the European Commission, DG Regio, Brussels.

[7]Nichols, Mike (2007), ‘Review of Highways Agency’s Major Roads Programme’, Report to the Secretary of State for Transport, London,

[8]UK National Audit Office (2007), ‘Estimating and Monitoring the Cost of Building Roads in England’, UK Stationary Office.

[9] See Bordat, C. et al (2004), ‘An Analysis of Cost Overruns and Time Delays of INDOT Projects’, Joint Transportation Research Programme, Indiana Department of Transportation.

[10] EX POST EVALUATION OF COHESION POLICY PROGRAMMES 2000-2006, WORK PACKAGE 10, ‘Efficiency: Unit costs of major projects’ FINAL REPORT, 25 October 2009 RGL Forensics, Frontier Economics, Aecom

[11] See Kahneman, D. (1973), Attention and effort, Englewood Cliffs, NJ: Prentice-Hall. See also Kahneman, D. (2003), ‘Maps of bounded rationality: A perspective on intuitive judgment and choice’, in T. Frangsmyr (Ed.), Les Prix Nobel 2002 [Nobel Prizes 2002]. Stockholm, Sweden: Almquist & Wiksell International. See also Kahneman, D. and D. Lovallo (1993) ‘Timid choices and bold forecasts: A cognitive perspective on risk-taking’, Management Science, 39, 17-31.

[12]Flyvbjerg, Bent in association with COWI (2004), ‘Procedures for Dealing with Optimism Bias in Transport Planning: Guidance Document’, for the British Department for Transport, London.

[13] The method is now a requirement for all large transport infrastructure projects seeking government funding in the UK and has also been endorsed by the American Planning Association.

[14] See footnotes 3 and 6.

[15]Delusion and Deception in Large Infrastructure Projects: Two Models for Explaining and Preventing Executive Disaster, Bent Flyvbjerg, Massimo Garbuio, Dan Lovallo, California Management Review Winter 2009

[16] See footnote 17