Reducing Estimation Uncertainty with Continuous Assessment: Tracking the “Cone of Uncertainty”
Pongtip Aroonvatanaporn, Chatchai Sinthop, Barry Boehm
Center for Systems and Software Engineering
University of Southern California
Los Angeles, CA 90089
{aroonvat, sinthop, boehm}@usc.edu
ABSTRACT
Accurate software cost and schedule estimations are essential especially for large software projects. However, once the required efforts have been estimated, little is done to recalibrate and reduce the uncertainty of the initial estimates. To address this problem, we have developed and used a framework to continuously monitor the software project progress and readjust the estimated effort utilizing the Constructive Cost Model II (COCOMO II) and the Unified CodeCount Tool developed by the University of Southern California. As a software project progresses, we gain more information such as complexity, architecture resolution, and people capability as well as the actual source lines of code developed and effort spent. This information is then used to assess and re-estimate the effort required to complete the remainder of the project. As the estimations of effort grow more accurate with less uncertainty, the quality and goal of project outcome can be assured within the available resources. The paper thus also provides and analyzes empirical data on how projects evolve within the familiar software “cone of uncertainty.”
Categories and Subject Descriptors
D.2.9 [Management]: Cost estimation, Life cycle, Time estimation
General Terms
Management, Measurement, Economics
Keywords
Cost Estimation, Uncertainty
1. INTRODUCTION
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
ASE’10, September 20–24, 2010, Antwerp, Belgium.
Copyright 2010 ACM 1-58113-000-0/00/0010…$10.00.
Having accurate estimations of the effort and resources required to develop a software project is essential in determining the quality and timely delivery of the final product. For highly precedented project and experienced teams, one can often use “yesterday’s weather” estimates of comparable size and productivity to produce fairly accurate estimates of project effort. More generally, though, the range of uncertainty in effort estimation decreases with accumulated problem and solution knowledge within a “cone of uncertainty” defined in [1] and calibrated to completed projects in [2]. To date, however, there have been no tools or data that monitor the evolution of a project’s progression within the cone of uncertainty.
In this paper, we describe a framework used to continuously assess the project status and progress throughout the project’s life cycle and report on the results of the experiment. The study was conducted on a graduate level software engineering course at the University of Southern California (USC). In the two-semester team project based course sequence CSCI577ab [3], students learn to use best software engineering practices to develop software systems for real clients. They adopt the Incremental Commitment Model (ICM) [5][6] to develop the software project, and in most cases, the students have little to no industry experience and little project management knowledge. In order for the ICM process to be usable by the software engineering course, it has been tailored to aim the process at software development as well as reducing the scope to fit the 2-semester, or 24-week, time schedule for an 8-person development team (6 on-campus and 2 off-campus students).
In addition, the teams utilize the Constructive Cost Model (COCOMO II) [2] to estimate the resources required to complete the projects. Since the schedule of class is fixed, the projects’ scopes are adjusted accordingly in order to fit the time frame and the available resources. Once the project estimation has been performed to show that the software project can be completed within 24 weeks by 8 developers, it serves as a basis of commitment for the stakeholders and the team proceeds through the ICM process life cycle to develop the project. However, due to the lack of experience and necessary management skills, the estimates calculated by the student teams often turn out to be inaccurate, resulting in either over or under estimations.
Our goal is to develop a routine, semi-automated assessment framework that helps reduce uncertainties of the software project estimation as the project progresses through its life cycle. The assessment framework integrates the Unified Code Count tool (UCC) developed by USC with the COCOMO II estimation model to quickly generate information to analyze the team’s performance and estimations. This is similar to the concepts of [14], which shows that frequent assessment of the project status help improve the team as well as the final product of the project. We apply this concept to assess the efforts spent on the project and compare with the current progress to predict the effort required to complete the project. This information is then used to evaluate the current project estimations and adjust the estimation parameters as necessary. This will eventually enable the actual and estimated effort to converge. The assessment framework allows the team to validate the direction of the project, while increasing the project understanding as well.
The key benefits of achieving a convergence between actual and estimated efforts are as follows:
· It allows the development team to improve planning and management of project resources and goals. As the estimations become more and more accurate with reduced uncertainties, the team can ensure that the scope of the project is feasible within the available resources and time.
· With better progress monitoring, the quality of the product can be controlled closely as the team are certain with the amount of work required to complete.
· It helps the client to better understand the actual project progress. When the project underestimates, it usually gives the wrong perception to the client that the project is closer to being finished than it actually is. On the other hand, when it overestimates, the project appears to be far from being finished.
This paper is organized into 9 sections. Following the introduction, we discuss about the problem that occurs due to inaccurate estimations and the motivations for the study of the framework in the next section. We then discuss about some of the related works and their shortfalls in section 3. Sections 4 and 5 present the model of the framework and our setup of the study to perform the experiment. Section 6 discusses the analysis of the study performed. Section 7 discusses about the potential threats to the validity of our studies and how we minimize their effects. Finally, the paper concludes in section 8 including with our plans for future work and follows by references in section 9.
2. Problem and Motivation
The main motivation behind the development of the assessment framework is derived from the well-know software “cone of uncertainty” problem.
Figure 1: The Cone of Uncertainty [2]
Figure 1 shows the accuracy of software sizing and estimation by phases. The level of estimation uncertainties are high during the initial estimations due to lack of data and experience. As long as the projects are not re-assessed or the estimations not re-visited, the cones of uncertainty are not effectively reduced [1].
2.1 Imprecise Project Scoping
Based on the observations from the projects of CSCI577 Software Engineering course, many projects end up with either significantly overestimating or underestimating the effort required to complete the project within the available time and resources. This often causes the projects to end with undeliverable products and unsatisfied stakeholders.
Since we adopt the schedule as independent variable (SAIV) development paradigm [7] due to the fixed amount of time available for class, the project scopes are to be adjusted to compensate for the available resources (the adjustments of estimations and project scoping may differ in other practices and other classes of applications). Due to inaccurate estimations, many teams end up with inappropriate project scoping because the estimations state that they either do not have enough resources to complete the project or that the scope of the project can be expanded to cover additional requirements.
When the projects begin with the initial overestimations, the teams are required to re-negotiate with the clients to reduce the size of the projects. This often result in the clients needing to throw away some of the critical core capabilities of the project, thus, losing some of the expected benefits they had hoped from the completed project.
On the other hand, when projects underestimate the resources, they tend to overshoot the goals that the project can achieve. As the project progresses to the end of its life cycle, the team may start to realize that the remainder of the project is more than they can manage to complete. When this happens, one scenario is they try to satisfy the client by attempting to complete the project as quickly as possible, while the quality of the project greatly suffers from this attempt and result in higher long-term maintenance costs. Another scenario is they end up delivering a project that is not complete; thus, leaving the client with an unusable product.
2.2 Project Estimations Not Revisited
During the initial estimation for the software project to be developed, the teams often do not have sufficient data to carefully analyze and perform the necessary predictions. These missing information include aspects that are specified in the COCOMO II cost drivers [2]. In most cases, the project estimation turns into a constant value at the time that the project enters the development phase. This means that regardless of how well the project progresses or how capable the programmers actually are, the project estimations remain constant.
The only estimations that are done for the project are based on the calculations with insufficient information. Once the projects proceed through the development phase, the status and progress of the projects are not assessed and re-assessed by the team in order to analyze the accuracy of the initial estimates. Although in the ICM process, the project status maybe reviewed by the stakeholders during the major milestones, the team never performs minor assessments throughout the project life cycle [14]. There are significant numbers of uncertainties at the beginning of the project as there are instability in requirements and many directions that the project proceed on.
2.3 Manual Assessments are Tedious
The tasks of manually assessing the project progress are tedious and discouraging to the team due to the amount of effort required and complexity. In order to collect enough information to have a useful assessment data, the teams often need to perform various surveys and reviews to determine how well the team had performed in the previous iterations [14].
Furthermore, to accurately report the progress of the software development project, the teams are required to carefully count the number of source lines of code (SLOC) they have developed, analyze the logical lines of code, and compare that to the estimates that they had performed initially. These tasks require significant amount of effort to collect the necessary information to evaluate the initial estimations performed for the project and to identify how well the team is actually performing. This discourages the team from constantly performing assessments of the project status due to tedious and complex work.
2.4 Limitations in Software Cost Estimation
Regardless of what software cost estimation technique is used, there is little that the technique can compensate for the lack of information and understanding of the software to be developed. As clearly shown in [1], until the software is delivered, there exists a wide range of software products and costs that can turn into the final outcome of the software project. In addition to the fact that the initial estimations lack the necessary information to achieve accurate estimates as mentioned in section 2.2, the software design and specifications are prone to changes throughout the project life cycle as well, especially in an agile software engineering environment.
The main goal of our assessment framework is to reduce this cone of uncertainty during the development phase by continuously monitoring the project’s progress and productivity and readjust the estimation parameters accordingly based on the assessed information. As the project progresses through its life cycle, the uncertainties are constantly reduced by all the information gained. We want to utilize the benefits from these information to help ensure the quality and accurate estimations of the final product.
3. Related Work
The most thorough and balanced coverage of software estimation methods is “Estimating Software-Intensive Systems” [19]. More recent updates, including discussions of expert-judgment vs. parametric-model estimation strengths and weaknesses, are [12] and [13]. A good treatment of agile estimation is [8].
Early treatments of software estimation uncertainty include the the PERT sizing method in [17] and the wideband Delphi estimate distributions in [2] and the accuracy-vs.-phase chart in [1], calibrated in [2], and termed the “cone of uncertainty” in [15]. Most commercial estimation models now include capabilities to enter input uncertainties, run a number of random-sample Monte Carlo estimates, and produce a cumulative probability distribution estimate of the probability that the actual cost will exceed a given budget [11].
The best available summary of software project tracking methods is “Practical Software Measurement” [16]. A good early treatment is “Controlling Software Projects” [9]. Tracking progress vs. estimated budget and schedules via Earned Value Management (EVM) systems is covered well in [10]. A business-value extension of EVM is provided in [4].
4. Model
The framework that we developed introduces a semi-automated method to help rapidly assess the project status and progress based on the effort spent and the number of SLOC. Figure 2 provides an overview of the assessment framework. The assessment framework takes the actual SLOCs of each module, readjusts them with the Requirements Evolution and Volatility (REVL) parameter, and then converts them to effort in person-months (PM) and in number of hours using the COCOMO II model. After the data of actual SLOC, or the effort spent on the project, is ready, the estimated total size and total effort can be calculated by comparison with the estimated percent completed of those modules.