Adjusting for Optimism Bias in FRS PFI Projects
Guidance Note
Adjusting for Optimism Bias in FRS PFI Projects
Guidance Note
March 2007
Department for Communities and Local Government: London
On 5th May 2006 the responsibilities of the Office of the Deputy Prime Minister (ODPM) transferred to the Department for
Communities and Local Government.
Department for Communities and Local Government
Eland House Bressenden Place London SW1E 5DU
Telephone: 020 7944 4400
Website: www.communities.gov.uk
© Crown Copyright 2007
Copyright in the typographical arrangement rests with the Crown.
This publication, excluding logos, may be reproduced free of charge in any format or medium for research, private study or for internal circulation within an organisation. This is subject to it being reproduced accurately and not used in a misleading context. The material must be acknowledged as Crown Copyright and the title of the publication specified.
Any other use of the contents of this publication would require a copyright licence. Please apply for a Click-Use Licence for core material at www.opsi.gov.uk/click-use/system/online/pLogin.asp or by writing to the Office of Public Sector Information, Information Policy Team, St Clements House, 2-16 Colegate, Norwich NR3 1BQ.
Fax: 01603 723000 or email: .
If you require this publication in an alternative format please email
Communities and Local Government Publications
PO Box 236
Wetherby
West Yorkshire
LS23 7NB
Tel: 0870 1226 236
Fax: 0870 1226 237
Textphone: 0870 1207 405
Email:
or online via the Communities and Local Government website: www.communities.gov.uk
March 2007
Product Code 06PPI04375
Contents
Chapter 1 Purpose 4
Chapter 2 Introduction 5
Chapter 3 Why Does Optimism Bias Arise? 6
Chapter 4 Optimism Bias Evidence Base 7
Chapter 5 The Case Study Evidence 8
Chapter 6 Applying Optimism Bias 11
Appendix References and Sources of Further Guidance 17
3
Adjusting for Optimism Bias in FRS PFI Projects
Chapter 1. Purpose
1.1 The purpose of this Guidance Note is to help appraisers and project staff involved in FRS PFI appraisals understand (i) what optimism bias is, and (ii) how to assess it in projects. The Guidance Note considers:
· What is optimism bias;
· The optimism bias evidence base; and
· How to apply optimism bias.
4
Adjusting for Optimism Bias in FRS PFI Projects
Chapter 2. Introduction
2.1 HM Treasury’s Green Book (2003) introduced the concept of optimism bias and required that it should be taken into account in project appraisal and evaluation.
2.2 It is recognised that risk and optimism bias are closely linked and that good risk management can reduce optimism bias. However, as guidance on project risk identification, assessment and management is provided elsewhere1 it is not covered in this Guidance Note.
2.3 Optimism bias is the term used to describe the demonstrated, systematic tendency for project appraisers to be overly optimistic about project costs, duration and benefits
(outputs and receipts/income). In other words it is the systematic tendency to view things in an overly positive light. It can arise in relation to any aspect of a project but it particularly applies to:
· costs (capital and revenue);
· works’ duration; and
· benefits delivery (outputs and outcomes).
2.4 To redress this tendency HM Treasury’s Supplementary Guidance on Optimism Bias
(http://www.hm-treasury.gov.uk/media/885/68/GreenBook_optimism_bias.pdf) recommends that project appraisers should make explicit adjustments to the estimates of project costs, benefits, and duration based on empirical data to inform project decisions. Adjusting for optimism bias should help agencies to answer key questions such as:
· By how much can we allow benefits to fall short of expectations, if the proposal is to remain worthwhile? How likely is this to happen?
· How much can operating costs increase, if the proposal is to remain worthwhile? How likely is this to happen?
· What will be the impact on benefits if operating costs are constrained?
1 HM Treasury Green Book Ch5 and Annex 4; Orange Book and Risk website http://www.hm- treasury.gov.uk/documents/public_spending_and_services/risk/pss_risk_index.cfm; OGC Successful Delivery Toolkit – Management of Risk
5
Adjusting for Optimism Bias in FRS PFI Projects
Chapter 3. Why does optimism bias arise?
3.1 Projects are inherently risky and the risks increase with the size and complexity of the project. Studies have found that there are deep-seated causes of optimism bias and that they can be broadly split into two categories: technical and institutional.
Technical
· Risks and uncertainties associated with forecasting costs, income etc;
· Changes in project scope; and
· Poor project management.
Institutional
· The desire to see projects happen;
· Institutional pressures; and
· The decision-making process, etc.
3.2 In one influential study, the authors argued that “political-institutional factors have in the past created a climate where only a few actors have a direct interest in avoiding optimism bias.”2 However, it is the norm in projects, rather than the exception, that unplanned events do occur and experienced project managers should consider the effect of these when appraising projects.
3.3 The challenge is therefore to:
· Make explicit allowance for optimism bias in appraisal in a proportional and cost effective way;
· Consider whether a project represents value for money once an allowance for optimism bias is included; and
· Be aware of and work to reduce the causes of both technical and institutional optimism bias.
2 DfT (2004) Procedures for Dealing with Optimism Bias in Transport Planning, Bent Flyvbjerg in association with COWI.
6
Adjusting for Optimism Bias in FRS PFI Projects
Chapter 4. Optimism bias evidence base
4.1 This Guidance Note is based on research commissioned by the Office of the Deputy Prime Minister3. The work examined a subset of the existing FRS PFI projects. It also considered wider-scale evaluation studies and involved a series of practitioner consultations and workshops.
4.2 The Guidance Note provides tailored estimates of optimism bias uplifts for project costs and duration for FRS PFI projects.
3 The responsibilities of the Office of the Deputy Prime Minister (ODPM) were taken over by the newly created Department for
Communities and Local Government on 5 May 2006.
7
Adjusting for Optimism Bias in FRS PFI Projects
Chapter 5. The case study evidence
5.1 Full details of the anonymised case study evidence can be found in the accompanying research report. This includes summaries of overruns and analyses of their causes. Here we provide summaries of key lessons to emerge for both costs and bid duration.
5.2 For costs, the case study evidence, interviews and workshops enabled the project team to identify some of the key drivers contributing towards inaccurate forecasting at both the OBC and FBC stages. These issues can be categorised under five broad headings:
· Approvals and understanding the procurement process;
· Land and property issues;
· Benchmarking capital and revenue costs;
· The role of advisers; and
· Gauging market interest.
5.3 For the approvals process, it is recognised that all of the schemes considered in the case studies were Round 1 or Pathfinder projects. Hence project managers could be forgiven for underestimating the time and cost of complying with the PFI project procurement route. Nonetheless, delays in acquiring permission, for whatever reason, did have knock-on effects on the accuracy of project cost estimates. This is because build timetables had to be revised, financial details rebased to account for inflation, and discounted cashflow analysis re-profiled to account for administrative delays. One project manger noted that a one-month delay in securing approval for their scheme added a potential £100,000 to project costs.
5.4 During interviews there were additional behavioural problems noted by some representatives. These related to the political dynamics associated with securing approval, and a systemic tendency to underestimate costs to secure political sanction. The problem is that practitioners do not accept that this tendency constitutes optimism bias. Because of this nobody assumes ownership of the problem and cost underestimates are left unchallenged.
5.5 For land and property issues, there seems to have been a wholesale misunderstanding of the complexities involved in securing or disposing of assets. Extra delays and costs were incurred because of overly optimistic assumptions over land availability, securing the correct planning permissions, and about miscalculations over land values in a changing market place. During workshop sessions with senior project managers the point was made that while it is difficult for a public body to acquire land without a PFI partner engaged on the project, it is difficult to engage with the private sector while land issues remain outstanding.
5.6 In terms of benchmarking revenue and capital costs various techniques had been employed by authorities, but they usually involved some form of estimate based on
8
Adjusting for Optimism Bias in FRS PFI Projects
performance criteria and historical cost at similar institutions. Visits to authorities who had already begun the PFI process allowed for more realistic estimates to be made, but the number of comparator schemes was small. Generally, project managers consulted with during the course of the research were keen to see a greater sharing of experiences and cost information so that this could inform the appraisal cycle. Ideally, such information should be available to authorities before the preferred bidder was selected.
5.7 In general, the sooner some form of market testing can occur, the more accurate appraisal estimates are likely to be. In one example, not included in the case study evidence, a ghost bid was commissioned early in the procurement cycle. This allowed costs to be informed by private sector experience and led to the development of additional benchmarks. The cost of the ghost-bid (£60,000 to £70,000) was seen to be proportionate to the benefits that this additional intelligence gave the project team.
5.8 The role of advisers is critical to the accuracy of various appraisal assumptions. There was general agreement amongst project officers that financial advisers need to be embedded in project teams so that assumptions are challenged on a regular basis from the beginning of the appraisal cycle. Where external advisers are used, they need to engage with the service deliverers and operational managers, using workshops and other techniques to force involvement. Critical to the success of this process is to have a tightly focused steering group or master consultant to manage the inputs of various experts throughout the organisation. For recent applications, many of the lessons from other PFI schemes have been documented and publicised. This was a luxury that was not available to the Pathfinder and Round 1 project leaders.
5.9 The next major difficulty affecting the accuracy of appraisal assumptions was the inability of public sector professionals to accurately gauge potential private sector interest in a PFI proposal. In two of the case studies this led to delay and significant re- scoping of the project to engender market interest. This again emphasises the importance of getting an early view of private sector interest either through soft-market testing or some other procurement mechanisms.
5.10 Taking the FRS signed schemes as a whole, PFI procurement takes an average of 2 years
8 months from the submission of OBC to commercial closure. However, there is large variation between schemes, with the quickest taking just 18 months, while the longest has taken over 6 years. Partnerships UK were commissioned by the Government to take undertake a review of PFI projects with new capital assets of less than £20m. The research showed that ‘procurement times ranged from 14 months to 5 years, with an average duration of two and a half years’. As CapGemini made clear, this is in line with the procurement time taken for the FRS schemes.
5.11 Almost all the projects (including the four case studies) underestimated how long the procurement process would take. Often, this meant that expenditure had to be re- profiled between different financial years, affecting the discounted cash flow analysis and requiring recalculation and further discussion with central government.
5.12 When independent consultants examined the FRS PFI programme as a whole in 2005, they identified five key reasons for unforeseen delays in the procurement process:
· Delays in arranging planning permission;
9
Adjusting for Optimism Bias in FRS PFI Projects
· Inadequacies within the fire authorities procurement processes, including poor preparation of business cases;
· Time taken to draw-up complex contracts;
· Time taken for banks to carry out due diligence; and
· Delays due to slow decision making centrally or poor communication between
Government departments.
5.13 All of these tendencies were seen in one or more of the case studies. In particular there seems to have been a systematic failure to appreciate the complexities surrounding property acquisition and planning.
5.14 One other interesting issue to emerge from the case studies is that once commercial closure was achieved, further project delays were very limited. In effect, following engagement of the PFI partner, the risk transfer mechanism seemed to ensure that on- site construction was effectively expedited.
10
Adjusting for Optimism Bias in FRS PFI Projects
Chapter 6. Applying optimism bias
6.1 This Guidance provides bespoke advice for those senior managers and project staff engaged in appraising FRS PFI projects, and is based on the case study evidence cited above.
6.2 HM Treasury’s Green Book Supplementary Guidance4 on optimism bias contains evidence from a study undertaken by Mott MacDonald that researched optimism bias on large capital and engineering projects. Mott MacDonald developed a methodology for calculating optimism bias on capital costs and works duration, and provided tables of suggested uplifts to be applied to duration and cost estimates.
6.3 It has become clear that since the Green Book was published, there has been disquiet amongst practitioners and funders over the applicability of using generalised optimism bias uplifts in project appraisal. In all areas, including fire appraisal and civil resilience, there is a demonstrable need for bespoke guidance on techniques and benchmarks.
6.4 PFI offers an alternative delivery mechanism for which bespoke appraisal guidance is already available. This guidance covers all projects and programmes for which a PFI alternative is to be considered. The key issue in such circumstances is whether or not projects procured through a PFI offer value for money against a Public Sector Comparison (PSC).