STQE Magazine Volume 2 Issue 6, Nov/Dec 2000. Bug Report Department.
Page 1 of 7
KEY POINTS:
- Finding common types of bugs
- Preventing requirements problems
- Improving your processes and products
Analyzing Requirements Bugs
Take a closer look at your bugs—you might improve your products.
By Otto Vinter and Søren Lauesen
Why is it that we seem to make the same mistakes over and over again? It can’t be that we don’t want to know why—that we don’t want to find ways to prevent our mistakes, and ultimately to improve the quality of our products. Surely we do.
Analysis of bug reports from previous projects, for example, tells us about our most frequent errors, and can help us improve. But very few companies spend the time to analyze bugs from completed projects.
At Brüel & Kjær ( have learned to use bug reports to improve our software development process. Analyzing this data has made us aware of major problem areas and how to prevent them—and clearly showed that an insufficient requirements engineering process was the most significant problem in our development process.
Before we started analyzing bug reports, the software development process at our company—a leading manufacturer of high-precision measurement instruments for sound and vibration measurement applications—was considered unsatisfactory. Too many projects had schedule overruns, and products were often shipped with too many bugs. Even when task forces were appointed to improve the quality, we were still finding bugs in the field. It was the general opinion that this was caused by insufficient testing before release.
Before rushing to conclusions, however, we decided that we would analyze bug reports from previous projects to see whether we could find which types of bugs were the most frequent. These analyses put us in a better position to determine how such bugs might be prevented, and thus implement focused solutions.
We’d like to show you how we analyzed the bugs, what we improved, and the benefits we achieved.
Our Defect Analysis Approach
A bug report at Brüel & Kjær can contain anything from a serious defect to a suggested improvement (See sidebar, “Sample Problem Reports”). Most projects had numerous bug reports, so we only analyzed an equally spaced sample (e.g., every fourth report). This way the correct distribution over time could still be maintained.
We sorted the problem reports according to the specific developer related to them (typically, the one who had fixed the bugs) and analyzed the reports in an interview with that developer. Without the participation of these developers it would have been impossible to arrive at the correct conclusions.
To classify bugs, we chose a taxonomy proposed by Boris Beizer (See sidebar, “Beizer’s Bug Taxonomy”). This taxonomy suited our purposes very well, considering that it was developed in the mid 1980s and contains some categories we don’t see with modern-day programming languages and tools. We extended the taxonomy with just a few new categories (e.g., creating a subcategory for problems with third-party software).
In our group we had studied the taxonomy and knew it well, but we did not expect developers to know the taxonomy prior to the interview; rather, they learned it as the interview progressed and as we classified their bug reports. Once everyone was familiar with the Beizer taxonomy, categorizing each bug took about five minutes.
Defect Analysis Results
The first project we ran through a defect analysis clearly showed us that we were wrong in our assumption about the causes for the bugs. Our testing process definitely needed improvement, but the largest cause for bugs stemmed from our requirements engineering process.
We compared the results of our bug categorization with Beizer’s statistics. We found many differences between our results and his—the most important being our relatively high number of problems with requirements. Close to a quarter of our bugs were in his Category One (Requirements and Features), while Beizer reported only 8% in this category. In examining the requirements’ subcategories, we found that the majority of our problems with requirements were caused by missing or incomplete requirements (Beizer’s subcategory 13xx). This subcategory accounts for a little more than half of all our bugs in Category One. Another major category of ours was related to requirements changing during implementation and use (Beizer’s subcategory 16xx), which accounted for a little over one third of our bugs in Category One.
We looked into categories beyond Beizer’s main “requirements” category and discovered that there were many subcategories which could also be considered as related to requirements. Most notable were subcategories concerned with developers or testers misunderstanding the requirements.
When we included all those subcategories related to requirements, we found they covered 51% of all our bugs. Requirements were not only our largest problem—they represented the majority of our bugs.
The primary issues seemed to be not in what was actually written in the requirements specification document, but rather in what was not known or tacitly assumed to be known about the requirements. This is demonstrated in Figure 1, in which we grouped the requirements-related categories into: Missing, Changed, Misunderstood, and Other.
We then performed a closer analysis of the requirements-related bugs. For this we used a classification by quality factor —similar to the one defined in the ISO 9126 Standard (See sidebar, “Quality Factor Classification”). We also classified the bugs according to the source of the error, and according to which interface the bug occurred on.
From the quality factor analysis we found that usability issues dominated (64%). Functionality issues (e.g., demand for new or different features)—which we originally thought would be the major requirements problems—actually represented a much smaller percentage (22%). From the interface analysis we found that problems with understanding and cooperating with third-partysoftware packages and circumventing faults in them were more frequent (28%). Other issues accounted for 13%. (Note that the sum of these figures adds up to more than 100% because one bug may involve more than one issue. The sample bug reports in the sidebar demonstrate some of these issues.)
Finally, in order to find and introduce effective prevention techniques in our requirements engineering process, we performed a cost/benefit analysis of a number of potential prevention techniques—and derived from this an optimum set of techniques for our requirements engineering methodology. And because of our findings, we focused on usability problems and early verification and validation techniques, rather than correctness and completeness of requirements documents.
The Requirements Engineering Improvement Action
We then introduced two prevention techniques from our new requirements engineering methodology into a real-life project. In doing so, we concentrated on the techniques that addressed requirementselicitation issues:
- Scenarios: Relate demands to use situations. Write down short descriptions for each known use situation. Explain the work environment, purpose of the work, and the people working. Describe the essential tasks in each scenario. Tasks should have a goal, a beginning, and a termination.
- Navigational Prototype Usability Test, Daily Tasks: Check that the users are able to use the system for daily tasks based on a prototype of the user interface with dynamic navigation and static screens. Test the prototype with users simulating daily tasks, revise the design, test it again, and so on until the result is acceptable.
We trained the team in these techniques. They wrote scenarios and performed usability tests themselves, and then they wrote the requirements specification in the conventional way. The development and release of the product then followed the existing practices in our company.
When we analyzed the bug reports from this project, we saw significant improvement: an overall reduction in bug reports of 27%, compared to projects of roughly equal size, and a 72% reduction in usability issues per screen! We were surprised to find not only a reduction in bugs related to requirements issues, but also reductions in other bug categories. We explain this derived effect on other bug types by the fact that most of the developers achieved a deep understanding of the users’ domain from describing use situations (scenarios) and taking part in the usability tests.
This invariably leads to a reduced uncertainty and indecision among the developers on what features to include, and how they should be implemented and work. It was clear from our interviews with the developers, for example, that the project had been accomplished with less stress than usual.
However, the impact of these techniques on the perceived quality of the released product—in the eyes of the customer—is even greater than the prevention of bugs. Describing use situations (scenarios) captures the most important demands from a user/customer perspective. Usability tests on very early prototypes verify that the concepts derived from these descriptions still match the users’ needs and can be readily understood by them in their daily use situations.
In following the product for eighteen months after its release, we were happy to find that it steadily sold more than twice the number of copies than its predecessor—this in spite of its being aimed at a much smaller market niche, and the price of the new product being much higher.
In the end, analysis of bug reports has helped us to not only find the most important development process to improve; it’s also helped us improve our products. As a testament to their usefulness, the requirements techniques introduced on this project are now used as standard practice in our company.
###
AUTHOR BIOS
Otto Vinter (), a consultant and seminar lecturer, has managed IT process improvement actions for seven years, and managed development projects for thirty years.
Søren Lauesen (), a professor at the IT University of Copenhagen and researcher into the requirements engineering process, also served as professor at Copenhagen Business School acting as consultant for Brüel & Kjær.
###
SIDEBARS
Sample Problem Reports
D1:When a 3D picture is rotated or zoomed, the annotations on coordinate axes, grid surfaces, etc. are also rotated and zoomed. This can make the annotations unreadable.
Explanation: The 3D package works on bitmaps when it moves and zooms. Defining the picture, however, is performed with separate commands for axes and annotations. It is a mistaken expectation of the specification of the third-party software and also a usability issue.
D137:The measurement sequence should be Z-shape, S-shape, or Free shape. However, I cannot find a “Free shape” button. Two buttons say Z-shape and S-shape.
Explanation: When both buttons are out, it means Free-shape. Although the developer understood the tacit requirement for choosing between the alternatives, he was wrong about how to deal with it. It is a usability issue.
Beizer’s Bug Taxonomy
There are several taxonomies for bug classification, but the most detailed one is by Boris Beizer in Software Testing Techniques, second edition, Van Nostrand Reinhold, 1990 (now: Coriolis Press). Beizer’s taxonomy (pages 466-476) splits bugs into nine main categories, which are further detailed in up to four more levels.
For example, under category 1xxx (Requirements and Features), category 16xx covers changes to requirements, 162x changed features, and 1621 that the change is an addition of a feature. The taxonomy is complemented with extensive statistical material on bug distribution in the different categories.
Main categories
- Requirements and Features: Bugs related to requirements as specified or as implemented
- Functionality as Implemented: Requirements are (assumed) correct, but implementation is wrong
- Structural Bugs: Bugs related to the component’s structure (the code)
- Data: Bugs in the definition, structure, or use of data
- Implementation: Typographical bugs, standards violations, and bugs in the documentation
- Integration: Bugs related to interfaces between components or to external interfaces
- System and Software Architecture: Bugs affecting the entire software system (e.g., architecture or system generation)
- Test Definition or Execution Bugs: Bugs in the definition, design, and execution of tests or test data
- Other Bugs, Unspecified
Quality Factor Classification
We classified the requirements related bugs according to the quality factor involved. We used McCall, J.A., P.K.Richards, G.F.Walthers, Factors in Software Quality, Vol I, II, III, US Rome Air Development Center Reports, 1977. The quality factors are a variant of the quality characteristics stated in the ISO 9126 standard (Information technology, Software product evaluation, Quality characteristics and guidelines for their use), but can be easily mapped onto those.
1xx Reliability.
100 Correctness (obviously wrong): System does not follow specification.
110 Robustness: System fails when encountering unexpected input.
120 Functionality (missing, obscure): Demand for another functionality.
130 Interoperability:Demand for another functionality in interaction with other systems.
2xx Usability.
201 Usability, perception: User cannot clearly see the information.
202 Usability, performance: Too many steps needed to carry out the task..
203 Usability, learning: User cannot find out how to carry out the task.
204 Usability, standard conformance: User interface does not follow specific styles or guidelines.
205 Usability, fail safe & feedback: User gets insufficient feedback or cannot recover from mistakes.
299 Usability, other: E.g. dubious user suggestions.
400 Performance: System responds too slowly.
600 Portability: Problems in porting to other platforms.
700 Maintainability: Problems in maintaining system.
###
FIGURES
###
WEBINFOLINK CONTENTS ():
Our reports on defect analysis of requirements issues (the PRIDE project):
- Lauesen S. (1999): Software Requirement: Styles and Techniques, Samfundslitteratur, Denmark, ISBN 87-593-0794-3.
- Lauesen S., O.Vinter (2000): Preventing Requirement Defects, Proceedings of the Sixth International Workshop on Requirements Engineering: Foundations of Software Quality, REFSQ 2000.
- Vinter O. (1998): Improved Requirements Engineering Based on Defect Analysis, Proceedings of the Second International Software Quality Week Europe, Software Research, San Francisco, USA. (
- Vinter O., S. Lauesen & J. Pries-Heje (1999): A Methodology for Preventing Requirements Issues from Becoming Defects. ESSI Project 21167. Final Report, Brüel & Kjær Sound & Vibration Measurement A/S, DK-2850 Nærum, Denmark. (
- Vinter O., J. Pries-Heje (1999): Rapport fra forbedringsaktivitet: Kravspecifikationer.(Danish), Brüel & Kjær Sound & Vibration Measurement A/S, DK-2850 Nærum, Denmark. (
- Vinter O., J. Pries-Heje (2000): Scenarios, Prototyping & Usability Tests Improve Products & Requirements, Proceedings of the Eighths European International Conference on Software Testing Analysis and Review, EuroSTAR 2000. EuroSTAR Conferences, Galway, Ireland.
Bug Classifications:
There are several taxonomies for bug classification but the most detailed one is by Boris Beizer. The taxonomy is complemented with extensive statistical material on bug distribution in the different categories. The statistics are based on 16,209 bugs from projects with a total of 6,877 million lines of code.
- Beizer B. (1990): Software Testing Techniques. Second Edition, Van Nostrand Reinhold New York (now: Coriolis Press).
- Chillarege, R., I. Bhandari, J. Chaar, M. Halliday, D. Moebus, B. Ray & M. Wong (1992): Orthogonal Defect Classification – a Concept for In-process Measurement, IEEE Transactions on Software Engineering, Vol. 18(11), pp. 943-956, November 1992.
- IEEE Standards (1995): Guide to Classification for Software Anomalies, IEEE Std. 1044.1-1995 (Unapproved Draft).
Quality Factors:
We classified the requirements related bugs according to quality factors. McCall is the originator of this classification. Since then, the ISO 9126 standard has emerged and is the recommended way to perform this classification. The quality factors we used can be easily mapped onto the ISO9126 characteristics.
- McCall, J.A., P.K.Richards, G.F.Walthers (1977): Factors in Software Quality, Vol I, II, III, US Rome Air Development Center Reports.
- International Organization for Standardization (1991): Information Technology – Software Product Evaluation – Quality Characteristics and guidelines for their use, ISO 9126.
Our reports on defect analysis leading to improvement of the testing process (the PET project):
Prior to the improvement of the requirements engineering process that we have reported in the acrticle, we used defect analysis on another project. This was at the time when we thought that testing was our major problem. However as reported, once we analyzed the bug reports we found that requirements issues dominated. The testing process also needed improvement, so we completed an improvement action on this (the PET project).
The techniques we selected to improve the testing process were: Static and dynamic (coverage) analysis. The tool LDRA Testbed ( was installed to support these techniques. The baseline project that we introduced the techniques on had just been released in a trial version. We were allowed time to improve the quality of this product before it was released in the final (production) version. We experienced a 46% improvement in testing efficiency (bugs found per person hour) as we raised the branch coverage of all units to above 85%.
The improved (production) version of the baseline product was then released and tracked for the same number of weeks we had measured on the existing (trial) version after its release, so that we were able to evaluate the effect of the experiment on problem reports. The team received 75% fewer error reports than for the trial-release version of the product. Of those error reports 70% were found to be related to requirements e.g. to bugs that could not have been found through static and dynamic analysis. This once more confirmed the need for us to improve the requirements process.
Our reports on this project are:
- Vinter O. (1996): Experience-Driven Process Improvement Boosts Software Quality, Proceedings of the ninth International Software Quality Week, Software Research, San Francisco, USA. (
- Vinter O., P.-M. Poulsen, K. Nissen & J.M. Thomsen (1996): The Prevention of Errors through Experience-Driven Test Efforts. ESSI Project 10438. Final Report, Brüel & Kjær A/S, DK-2850 Nærum, Denmark. (
- Vinter O., P.-M. Poulsen, K. Nissen, J.M. Thomsen & O. Andersen (1996): The Prevention of Errors through Experience-Driven Test Efforts, DLT Report D-259, DELTA, DK-2970 Hørsholm, Denmark.
- Vinter O. (1997): How to Apply Static and Dynamic Analysis in Practice, Proceedings of the tenth International Software Quality Week, Software Research, San Francisco, USA. (
Acknowledgements: