Achieving maximum benefits from formal reviews/inspections - strategies and case studies
Fran O’Hara
Insight Consulting
114 Granitefield, Dun Laoghaire,
Co. Dublin, Ireland
Phone/fax: +353-1-2854510
E-mail:
Abstract
Formal reviews/inspections are currently a ‘hot topic’ with many references and case studies detailing, in quantitative terms, the benefits that can be achieved. They have been practiced for over twenty years and yet there are many organisations not using them to anywhere near their full potential. Indeed, many examples exist where introducing a review/inspection process and/or maintaining an effective process have failed. This paper analyses some case studies from the real world of dubious success (and sometimes even failure) and focuses on some of the key issues which caused problems. The aim is point out the pitfalls and highlight the key elements in achieving maximum benefits from introducing and maintaining a formal review/inspection process. A strategy to deploy the process is presented which encapsulates some of the lessons learnt.
1. Introduction
What are the key elements of a cost-effective formal process?
There are many implementations of review and inspection from quite informal approaches through to the most formal and rigorous. ‘Formal’ for the purpose of this paper is a process which meets or exceeds the requirements of Peer Reviews as defined in the SEI’s Capability Maturity Model [CMM, 1993]. As such, processes based on Fagan’s Inspection technique [Fagan, 1976] and the Gilb/Graham [Gilb, 1993] approach most certainly meet and in most cases greatly exceed these requirements. In simple terms, ‘formal’ therefore implies, at a minimum, the following:
· the goal of the process is to rigorously identify and fix defects (especially major/critical defects)
· a secondary goal is to prevent defects
· reviews/inspections are planned explicitly in the project schedule
· review leaders/facilitators and reviewers receive specific training
· a shared understanding of the work products under review is obtained
In terms of the process activities, the following is implied:
· an entry check is performed to ensure the material is ready for review
· reviewers/inspectors spend time individually checking the work product using appropriate parent or ‘source’ documents to help find defects. These work products may be code, design documents, requirements documents, project plans, test plans, procedures, etc. An overview meeting has usually been held to discuss optimum checking rates, to assign individual reviewers additional checking roles, to focus the reviewers on criteria/rules to be checked, etc.
· a meeting is held to log the issues that have been found. Discussion is kept to a minimum. New issues are identified.
· the author investigates issues and fixes any problems.
· data on the review process, the work products, etc. is collected and used for example to determine the effectiveness of the review/inspection process, the optimum checking rates, the quality of work product, etc.
· if defect prevention is a main goal then some activity relating to defect causal analysis should take place (see for example the process brainstorming meeting concept [Gilb, 1993]). If defect prevention is a secondary goal then capturing improvement suggestions as related issues during logging meetings may suffice.
· etc.
Notice the above description advocates some discussion rather than zero discussion at logging meetings. This is so in order to help achieve the process goal of establishing a shared understanding of the work product under review (and to train/educate). However, any discussion should be strictly limited and controlled to facilitate clarification of issues only. Solutions should not be discussed. If this goal were not required then zero discussion would be advocated. The point is to link the process activities to the required process goals.
There is still much confusion around as to the difference between peer reviews, inspections, walkthroughs, technical reviews, etc.. The main difference between these approaches lies in the purpose, level of formality and the amount of effort required. Formal inspections [Fagan, 1976, 1986], [Gilb, 1993] have greatest formality and incorporate some statistical quality improvement and defect prevention. Informal reviews and walkthroughs, on the other hand, may focus more on consensus/buy-in and training respectively rather than rigorously trying to identify and fix defects. An in-depth discussion is beyond this paper (and has been provided elsewhere [Gilb,1993],[Knight, 1993]). A key point to remember is that there should be an explicit relationship between the process activities and the goals of the process. For example, if defect prevention is not an initial goal of the process then the related defect causal analysis activities are not necessary, resulting in a simplified process.
What are the benefits of such a process?
The main benefits are time saving, improved quality and reduced costs. This is obtained:
· by finding more defects
· by finding them early in the development lifecycle
· by preventing defects
The improvement in quality as a result of finding more defects before your customer does is obvious. The fact that they are found early is the key point associated with the prevention/early detection principle of modern best practices[1]. It is well accepted that the earlier one finds a defect in the development lifecycle, the cheaper it will be to fix. Figure 1 shows an example of the relative cost of fixing defects in the development lifecycle[2]. Costs associated with the maintenance phase are also reduced because of a higher quality product with fewer defects and better documentation to support it. [Gilb, 1993] provides many specific examples and quotes the following as typical results from organisations which have introduced inspection:
· net productivity increase of 30% to 100% (less rework so productivity improves)
· net timescale reductions of 10% to 30%
· test execution costs and timescales reduced by a factor of 5 to 10 since there are fewer defects left to find
· reduction in maintenance costs by one order of magnitude
· early or on-time delivery of systems (no defect-correction backlash at systems integration time, since most defects are treated earlier)
The time savings are obvious from the above discussion in terms of reduced rework, etc. but ultimately the prevention of defects by designing quality in to the work products will yield the greatest time saving.
Another benefit is to help build effective technical teams with an associated enhancement in the flexibility of individuals to take over each other’s work when required. This point is often as significant a point as the direct benefits described above. Formal reviews/inspections are a form of on-the-job training where a broad knowledge of the work products and associated quality issues is rapidly learned by less experienced participants. They are invaluable as a form of communication especially when different functional areas such as development and test are brought together to review a work product such as a requirements specification. Work products improve as a result of a review but they also tend to improve before review simply because they are going to be reviewed. The development of a quality culture in general is often strongly linked to the introduction of formal reviews/inspections.
Capers Jones [Jones, 1996] recently reported that formal reviews/inspections had the highest empirical evidence of success with almost no failures. They ranked top position in structured methods along with JAD (Joint Application Development) and QFD (Quality Function Deployment) in terms of Return on Investment (RoI). The industry data collected showed an RoI of 10 at four years! This figure is backed up by many examples including Thorn EMI in [Gilb, 1993]. Even informal reviews had a RoI of 4 at four years according to Jones’ data.
Figure 1. The relative cost of fixing defects depending on where they are found during the development lifecycle.
2. What are the practical experiences?
Four case studies are outlined below. The first two relate to the introduction of the process and the second two are concerned with issues and improvements surrounding an existing process.
3. Case study - Schaffner Ltd.
Background
This company is a market leader in the manufacture of Power Supply Automated Test equipment and EMC Systems and Instruments. The Irish division has eight full-time software engineers and a further twelve that are involved in both software and hardware. As with many traditionally hardware-oriented companies, they are becoming increasingly oriented towards developing software, in this case Windows applications. A business goal therefore was to gain more business through the expansion of their software development activities. To achieve this it was recognised that more structure was required in their software development process and Peer Reviews was chosen as one of the process areas on which to focus.
A prior initiative in the area of code reviews failed. This was reported to have been due to a lack of training and experience with reviews, which led to informality, and a general lack of review leadership. Reviews degenerated into trivial comments and soon were deemed a waste of time.
They re-launched their improvement initiative at the start of 1995 and focused on a number of areas most notably project management[3]. It was estimated at the time that more than half a person year was being lost each year to fixing operational defects. In January 1996 they received funding for a focused process improvement project under the ESPITI TRI-SPIN program (a case study is available [Schaffner, 1996]). The topic of the 6 month project was peer reviews in the areas of code, design, requirements and test documentation. The purpose was to pilot a peer review process with an aim of reducing operational defects by 25%. It was also hoped that reviews would help drive documentation improvements in the areas such as testing which would be the subject of review.
Implementation
The process improvement was project managed by the quality manager with the Peer Reviews project a sub-project in the overall improvement program. Peer Review implementation took the form of a 1-day training workshop followed by seven days consultancy support over the six months to help with the implementation of the process and improve its effectiveness.
Results
The pilot project was viewed as a success. Despite some cultural resistance that had to be overcome, the quality manager noted ‘One of our very first reviews saved us considerable grief and money’. In quantitative terms the following was achieved at the end of the 6 month pilot:
· they achieved a review efficiency of 1 defect/hr i.e. an average of one defect found for every hour spent either preparing for or attending the review meeting.
· an overall benefit to cost of 2:1 was obtained i.e. for every hour spent with reviews, two hours was saved on subsequent rework
Lessons Learnt
The following were the key lessons learnt:
· project management improvement is a pre-requisite to engineering improvement (see for example the Level 2 processes which come before Peer Reviews on Level 3 of the CMM - figure 2.). The progress with the improvement in project management had been less than anticipated so that when the externally funded Peer Review process was being piloted, its implementation was often hampered by resource/schedule issues. This point was made earlier but despite the non-ideal situation, a benefit to cost of 2:1 was achieved.
· management commitment and support at all levels is crucial.
· experienced practitioner support and just in time training are key issues when introducing the Peer Review process. The process is very much a people-centered process and requires implementation support after formal training (especially to help review leaders do their job effectively). Also, a time lag between the formal training and the actual piloting was not ideal and reduced the effectiveness of the training.
· ‘with reviews the cultural issues are probably a bigger part of things than pure techniques’. All of the cultural related activities embedded in an SPI lifecycle model such as the SEI’s IDEAL lifecycle model [IDEAL, 1997] need to be applied to overcome the inherent resistance that such a people-centered process can create.
· short improvement cycles such as this 6 month pilot can be quite effective at getting some tangible improvements underway.
Figure 2. The SEI’s Capability Maturity Model: Key Process areas assigned to process categories. Note Peer Reviews has been elevated to a Key Process Area in the engineering category at level 3.
4. Case Study - I.T. Department Bank X
A review process closely based on Fagan Inspections was introduced into the I.T. department of an Irish Bank. The department consisted of about 200 people. The Software Quality Assurance manager pioneered the process introduction and personally performed internal presentations on inspections. Supporting material including process descriptions/definition, templates, etc. was provided to all the individuals who attended the presentation.
Results
The following points summarise what happened:
· overall the process introduction was a failure although there were a few pockets of isolated success where the people involved adopted the process. Elsewhere the process fell into disuse.
· Only trivial issues were found at these inspections, data on inspections was not returned to the process owner (which was SQA), buy-in was generally poor, the process was viewed as ‘too formal with too many forms’, etc.
· The I.T. department are now looking to introduce a more practical approach to formal reviews/inspections
Lessons learnt
· The process which was introduced was too formal for the culture and level of process maturity which existed at the time in the department
· There was insufficient initial training (especially practical exercises) and a lack of on-going experienced practitioner support. The initial presentations only lasted a few hours.
· A number of generic software process improvement principles were not addressed when trying to introduce the process
· Improvement needs to be managed as a project with clear goals, detailed planning, sufficient tracking, etc. This was not the case.
· There was no improvement lifecycle of phases and activities (see for example the ‘IDEAL’ approach from the SEI [IDEAL, 1997]. The pilot phase with a set of clear measures was especially lacking.
· There was a lack of involvement of the I.T. staff in general in the process definition. As a result no tailoring/customisation occurred and there was a lack of ‘ownership’ of the process by the staff. It was essentially viewed as an ‘SQA process’. There was also a fear of the metrics being introduced – again buy-in had not been achieved re their use.