Preparation of Papers for CSER 2009

7th Annual Conference on Systems Engineering Research 2009 (CSER 2009)

Better Management of Development Risks: Early Feasibility Evidence

Barry Boehm1 andJo Ann Lane2

1University of Southern California, USA,

2 University of Southern California, USA,

LoughboroughUniversity – 20th - 23rd April 2009

7th Annual Conference on Systems Engineering Research 2009 (CSER 2009)

Abstract

The Incremental Commitment Model (ICM) organizes more rapid and thorough concurrent systems engineering and acquisition processes in ways that provide points at which they can synchronize and stabilize, and at which their risks of going forward can be better assessed and fitted into a risk-driven stakeholder resource commitment process.In particular, its concurrent activities of Understanding Needs, Envisioning Opportunities, System Scoping and Architecting, Feasibility Evidence Development, and Risk/Opportunity Assessment enable projects to focus specifically on their system constraints and environments and on opportunities to deal with them. This paper describes in detail the content, preparation, and role of feasibility evidence at key decision points in the system development cycle and how this can be used to effectively identify and manage risks throughout the development cycle. The feasibility evidence is not intended to assess just a single sequentially developed system definition element, but rather to assess the consistency, compatibility, and feasibility of several concurrently-engineered elements. To make this concurrency work, a set of anchor point milestone reviews are performed to ensure that the many concurrent activities are synchronized, stabilized, and risk-assessed at the end of each phase using developer-produced, expert-validated feasibilityevidence.

Keywords –anchor point milestones, feasibility evidence, Incremental Commitment Model

LoughboroughUniversity – 20th - 23rd April 2009

7th Annual Conference on Systems Engineering Research 2009 (CSER 2009)

1Introduction

This paper describes in detail the content, preparation, and role of feasibility evidence at key decision points in the system development cycle and how this can be used to effectively identify and manage risks throughout the development cycle. The feasibility evidence is not intended to assess just a single sequentially developed system definition element, but rather to assess the consistency, compatibility, and feasibility of several concurrently-engineered elements. To make this concurrency work, a set of anchor point milestone reviews are performed to ensure that the many concurrent activities are synchronized, stabilized, and risk-assessed at the end of each phase. Each of these anchor point milestone reviews is focused on developer-produced and expert-validated evidence, documented in a Feasibility Evidence Description (FED) document, to help the key stakeholders determine whether to proceed into the next level of commitment.

The FED is based on evidence from simulations, models, or experiments with planned technologies and increasingly detailed analysis of development approaches and projected productivity rates. The parameters used in the analyses should be based on measured component performance or on historical data showing relevant past performance, cost estimation accuracy, and actual developer productivity rates. The FED is not an optional appendix, but a first-class deliverable subject to earned value management and independent expert review. It has been used effectively from small web-services projects to ultra-large systems of systems [1,2,3,4,5].

This paper also provides experience-based types of evidence that should be evaluated at each commitment review, criteria for evaluating the evidence,and alternative courses of action based upon the results of the evaluation: risk level is acceptable—proceed ahead;number or level of risks are too high and need further mitigation before proceeding—do more work; or risk is unacceptable—rescope or discontinue program.

2Developing and Using Feasibility Evidence for Life Cycle Commitment Milestones

Success-Critical Stakeholders (SCSHs) for systems under development have the authority and the responsibility to exercise accountable governance of their organization’s resources when committing such resources to the next phase of system development. In many cases, when using processes other than the ICM, these stakeholders are presented with insufficient information upon which to make a responsible decision.

As a result, many projects proceed into system development with little idea of whether their project requirements, architectures, plans, budgets, and schedules are feasible and consistent or not. This is one of the major reasons for the relatively low (20-30%) success rates for successful on-time, in-budget system deliveries in the commercial Standish Reports and government GAO reports.

The ICM provides guidance for avoiding such system overruns and shortfalls by using APs and FEDs. The main distinction of using APs and FEDs is to provide evidence of feasibility as part of the system definition’s core deliverables, not just assertions, assumptions, or aspirations placed in optional appendices.

APs and FEDs emphasize that independent experts should evaluate the evidence supplied by developers. Any shortfalls in evidence should be identified as sources of project risk. These sources of project risk need to be covered by risk mitigation plans.

APs and FEDs are an integral part of the ICM discussed in [4,6]. However, they are applicable to models other than the ICM. A good approximation to their use may be obtained by adding FEDs to the content of traditional waterfall or V-model milestones such as System Requirements Reviews (SRRs), System Functional Reviews (SFRs), and Preliminary Design Reviews (PDRs).

3Nature of FEDs and AP Milestones

FEDs are evidence provided by the developer and validated by independent experts that, if the system is built to the specified architecture it will:

Satisfy the specified operational concept and requirements, including capability, interfaces, level of service, and evolution
Be buildable within the budgets and schedules in the plan
Generate a viable return on investment
Generate satisfactory outcomes for all of the SCSHs
Identify shortfalls in evidence as risks, and cover them with risk mitigation plans

A FED does not assess a single sequentially developed system definition element, but the consistency, compatibility, and feasibility of several concurrently-engineered elements. To make this concurrency work, a set of AP milestone reviews are performed to ensure that the many concurrent activities are synchronized, stabilized, and risk-assessed at the end of each phase. Each of these AP reviews is focused on developer-produced and expert-validated evidence, documented in the FED, to help the key stakeholders—the SCSHs—determine whether to proceed into the next level of commitment. Hence, they are called Commitment Reviews.

The FED is based on evidence from simulations, models, or experiments with planned technologies and increasingly detailed analysis of development approaches and project productivity rates. The parameters used in the analyses should be based on measured component performance or on historical data showing relevant past performance, cost estimation accuracy, and actual developer productivity rates.

A shortfall in feasibility evidence indicates a level of program execution uncertainty and a source of program risk. It is often not possible to fully resolve all risks at a given point in the development cycle, but known, unresolved risks need to be identified and covered by risk management plans, including the necessary staffing and funding to address them. The nature of the evidence shortfalls, the strength and affordability of the risk management plans, and the stakeholders’ degrees of risk acceptance or avoidance will determine their willingness to commit the necessary resources to proceed. A program with risks is not necessarily bad, particularly if it has strong risk management plans. A program with no risks may be high on achievability, but low on ability to produce a timely competitive advantage.

A program more familiar with a sequential waterfall or V-model can achieve most of the effect of a FED-based anchor point milestone review by adding a FED to the set of artifacts to be developed and reviewed for adequacy and satisfaction at a System Requirements Review (SRR), System Functional Review (SFR), or Preliminary Design Review (PDR). In principle, some guidance documents indicate that such feasibility evidence should be produced and reviewed. But since the evidence is not generally specified as a developer deliverable and is vaguely defined, it is generally inadequate as a basis for stakeholder commitments.

4Problems Encountered without a FED

In the early 1980s, a large government organization contracted with TRW to develop an ambitious information query and analysis system. The system would provide more than 1,000 users, spread across a large building complex, with powerful query and analysis capabilities for a large and dynamic database.

TRW and the customer specified the system using a classic sequential-engineering waterfall development model. Based largely on user need surveys, an oversimplified high-level performance analysis, and a short deadline for getting the To-Be-Determineds out of the requirements specification, they fixed into the contract a requirement for a system response time of less than one second.

Subsequently, the software architects found that subsecond performance could only be provided via a highly customized design that attempted to anticipate query patterns and cache copies of data so that each user’s likely data would be within one second’s reach (a 1980’s precursor of Google). The resulting hardware architecture had more than 25 super-midicomputers (an earlier term for a computer with performance and capacity between a minicomputer and a mainframe) busy caching data according to algorithms whose actual performance defied easy analysis. The scope and complexity of the hardware-software architecture brought the estimated cost of the system to nearly $100 million, driven primarily by the requirement for a one-second response time.

Faced with this unattractive prospect (far more than the customer’s budget for the system), the customer and developer decided to develop a prototype of the system’s user interface and representative capabilities to test. The results showed that a four-second response time would satisfy users 90 percent of the time. A four-second response time, with special handling for high-priority transactions, dropped development costs closer to $30 million. Thus, the premature specification of a 1-second response time neglected the risk of creating an over-expensive and time-consuming system development. Fortunately, in this case, the only loss was the wasted effort on the expensive-system architecture and a 15-month delay in delivery (see Figure 1). More frequently, such rework is done only after the expensive full system is delivered and found still too slow and too expensive to operate.

Figure 1 - Problems Encountered without a FED: 15-Month Architecture Rework Delay.

5Problems Avoidable with FED

Had the developers been required to deliver a FED showing evidence of feasibility of the one-second response time, they would have run benchmarks on the best available commercial query systems, using representative user workloads, and would have found that the best that they could do was about a 2.5-second response time, even with some preprocessing to reduce query latency. They would have performed a top-level architecture analysis of custom solutions, and concluded that such 1-second solutions were in the $100 million cost range. They would have shared these results with the customer in advance of any key reviews, and found that the customer would prefer to explore the feasibility of a system with a commercially-supportable response time. They would have done user interface prototyping and found much earlier that the four-second response time was acceptable 90% of the time.

As some uncertainties still existed about the ability to address the remaining 10% of the queries, the customer and developer would have agreed to avoid repeating the risky specification of a fixed response time requirement. The customer and developer would instead define a range of desirable-to-acceptable response times, with an award fee provided for faster performance. They would also have agreed to reschedule the next milestone review to give the developer time and budget to present evidence of the most feasible solution available, using the savings over the prospect of a $100 million system development as rationale. This would have put the project on a more solid success track over a year before the actual project discovered and rebaselined itself, and without the significant expense that went into the unaffordable architecture definition.

For example, the customer and developer would negotiate response time ranges. They would say two-second response time desirable, four-second response time acceptable with some two-second special cases. Next they would benchmark commercial system add-ons to validate their feasibility. Finally, they would present solution and feasibility evidence at AP milestone review. The result would be an acceptable solution with minimal delay.

6AT&T Experience with AP Reviews

AT&T and its spinoffs (Telcordia, Lucent, Avaya, regional Bell companies) have been successfully using versions of AP reviews since 1988, see Figure 2. On average, there has been 10% savings per reviewed project, with substantially larger savings on a few reviewed projects. More detail on their Architecture Review experience is in J. Maranzano et. al., “Architecture Reviews: Practice and Experience,” IEEE Software, March/April 2005 [7].

USC has used APs and FEDs to achieve a 92% on-time, in-budget, satisfied-client success rate on over 50 fixed-schedule campus e-service projects[2,4]. AP Reviews are also an integral part of the Rational Software Process, with many successful implementations [8].

Figure 2 - AT&T Experience with AP Reviews.

7ICM Phases, Anchor Points, and Decision Options

Figure 3 shows how the ICM spans the life cycle process from concept exploration to operations. Each phase

LoughboroughUniversity – 20th - 23rd April 2009

7th Annual Conference on Systems Engineering Research 2009 (CSER 2009)

Figure 3 – ICM Detailed Activity Descriptions.

LoughboroughUniversity – 20th - 23rd April 2009

Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDsVersion 0.5

includes a number of concurrent activities that are synchronized and stabilized with a FED-based anchor point milestone review at points corresponding with the major DoD acquisition milestones. At each anchor point, there are 4 options, based on the assessed risk of the proposed system. Some options involve go-backs. These options result in many possible process paths.

The life cycle is divided into two stages: Stage I of the ICM (Definition) has 3 decision nodes with 4 options/node, culminating with incremental development in Stage II (Development and Operations). Stage II has an additional 2 decision nodes, again with 4 options/node.

One can use ICM risk patterns to generate frequently-used processes with confidence that they fit the situation. Initial risk patterns can generally be determined in the Exploration phase. One then proceeds with development as a proposed plan with risk-based evidence at the VCR/Material Development Decision milestone, adjusting in later phases as necessary.

Risks associated with the system drive the life cycle process. Information about the risks (feasibility assessments) supports the decision to proceed, adjust scope or priorities, or cancel the program. The downward arrows at the end of each phase provide built-in off-ramps for clearly infeasible programs. The horizontal arrows provide additional options to skip phases or activities if the risk of doing so is negligible. Figure 4 shows the corresponding commitment reviews in the recent DoDI 5000.02.

Figure 4 – The Defense Acquisition Management System as in DODI 5000.02.

8Key Point: Need to Show Evidence

It was a significant step forward for DoD to progress from having schedule-based reviews, where the review took place at the scheduled time whether the project was ready or not, to event-based reviews, where the review would not take place until its artifacts—e.g., functional requirements—were completed.

However, many event-based reviews focused on hastily-produced artifacts that had not been validated for compatibility or feasibility, such as the example in Figure 1. Thus, it is another significant step forward to progress from event-based reviews to evidence-based reviews.

The major difference in the example project proceeding without and with an FED was that the need to provide evidence of feasibility requires the project to develop and exercise benchmarks and prototypes that expose both infeasible solutions and unnecessarily ambitious requirements.

A FED needs to be more than just traceability matrices and PowerPoint charts. Evidence can include results of:

Prototypes: of networks, robots, user interfaces, COTS interoperability
Benchmarks: for performance, scalability, accuracy
Exercises: for mission performance, interoperability, security
Models: for cost, schedule, performance, reliability; tradeoffs
Simulations: for mission scalability, performance, reliability
Early working versions: of infrastructure, data fusion, legacy compatibility
Previous experience
Combinations of the above

Not only does the evidence need to be produced, but it needs to be validated by independent experts. These experts need to determine the realism of assumptions, the reperesentativeness of scenarios, the thoroughness of analysis, and the coverage of key off-nominal conditions. The risk of validating just nominal-case scenarios is shown in the next section.