Software Testing: Management & Tools

Software Testing: Management & Tools

Chris Spriegel

Software testing is the cornerstone behind producing quality software products. With the advent of new technologies such as Java, it has become a necessity to test a single software product on multiple operating system platforms, as well as multiple run-time engines. Thus software engineering has incurred new quality assurance challenges. This presentation will cover the current state of software testing, the management challenges faced with a comprehensive test plan, and new software testing tools that allow automation of the test process.

(E.W. Dijkstra) "Testing can determine the presence of bugs, never their absence"

Testing begins within the initial stages of requirements planning. It must be present throughout the entire development process in order to provide critical improvements in software output. Testing because of its importance must be an integral part of the software life-cycle and must be structured according to the type of product, environment, and language.

The first document produced regarding tests would be the test strategy. This document outlines various levels of tests and how these will build up to a final function system test. Overseeing the testing development should be a test manager. Many corporations will define a testing team that's sole purpose in life is the testing of software products. This, however, at the academic level is probably one of your acquaintances, or even yourself, wearing another hat. Another point I would like to cite before diving into the bulk of the presentation is testing reuse. The American software industry has become bogged down with reinventing the wheel. Many corporations have in-house testing, building test cases by hand, or creating homegrown test tools. All of these create immense overhead in the development process, and draw negative attention to the testing process.

OBJECT ORIENTED w/ TESTING

Testing has become difficult with object oriented due to dependency and the dynamic nature of the object oriented languages. The major problem with the OO languages with regards to testing is what is referred to as the "ripple effect", meaning you fix one problem and then another one appears. The solution that is offered by testing industries is to test the single unit, or class, integrate it with other classes, then test the overall unit. This process of coupling classes together is referred to as integration testing; the former testing strategy is referred to as unit testing. If you can imagine the extreme amount of over head this would encompass, you can see why different methods are necessary.

TYPES OF TESTS

I. Regression Testing

(IEEE 1990) "Selective retesting [of software] to detect faults introduced during modification of a system or system component, to verify modifications have not caused adverse efforts, or to verify that a modified system or system component still meets specified requirements"

What does this mean in simple terms? Basically, regression testing answers the question: "Does it still work after I fixed it?" Included in this category of testing is when and how test suite will be run, what tools will be utilized, how developers will be notified of the current state of testing.

II. Stress Testing

Stress testing tests the software product fulfills the requirements even under extreme conditions. This area of testing gets extreme attention from mission critical organizations such as the military when dealing with advanced weaponry.

a. limit verification : tests how well a program handles input values outside of a defined legal range. Suppose for example that you have a program that was designed to accept alphanumeric characters. In order to stress tests this software, an individual could input various control controls or escape codes that were not part of the original subset of acceptable input.

b. iterative testing : testing a program under several runs, or iterations, of use. This strategy will verify if the software will run stable over time. A single occurrence of the software may run fine once, but may crash if called upon again. Some examples of components that need testing in this area would be variable allocations within memory, screen updates, and other dynamic components.

III. Unit Testing

Unit testing is coverage testing for each function, module, or class. This area of testing is by far the most time consuming and impractical technique. This area of testing is slowly being replaced by a combination of "black-box" and "white-box" testing. However, for mission critical software packages, unit testing is essential.

TEST TOOLS

Testing tools fall into one of two categories: static test tools and dynamic test tools.

1. Static Test Tools: take a software product and analyze the underlying code that comprises the software. This type of test does not involve actual input and output; rather it takes a symbolic approach to testing. These tools do not test the actual execution of the software.

a. flow analyzers : ensure consistency in data flow from input to output

b. path tests : try to find unused code and code with contradictions

c. coverage analyzers : ensure all logic paths are tested

d. interface analyzers : examine effects of passing variables and data between modules

2. Dynamic Test Tools: test the software system under actual execution with "live" data.

a. test drivers : input data into a module under test (simulators that mimic unavailable items of hardware or software)

b. test beds : display simultaneously source code along with executing program (show variables' values)

c. emulators : programmable stimulus and response facilities are used to emulate parts of the system not yet developed

d. mutation analyzers : seed intentional errors into code to test fault tolerance of system

AUTOMATED TESTING

Manual testing efforts tend to find the bulk of errors toward the end of the release effort, where it is effort more expensive to fix. Automated tools can catch errors earlier in the development process where it is easier to fix. Tools depend on the level of support management provides, no support with the best tools will not work. There are obstacles to automated testing, too. Some examples that automated testing will often not catch are exception handlers, interrupt handlers, and boundary conditions, such as February 29.

According to a Japanese software industry, "A survey of 200 Japanese software development groups ranked software test tools as having the highest productivity return on investment (ROI), ahead of standard methodology, CASE, any software development tool, and reviews and inspections (Bezier 1990, pg 449).

Before I discuss automation software tools available, I would like to stress the importance of integrating software testability into the products. As a developer, you want to be able to set/get state of any object contained within your product, be able to save/view logs, view network traffic, and log events. One personally example I would like to share with you is a recent execution of Netscape Navigator I had. I ran the program as I had in the past, but it crashed during the execution. This crash loaded a driver program that saved the current state of the crashed program. . . It then asked me for permission to send this data back to the Netscape Navigator developer team.

CATEGORIES OF TESTING TOOLS

1. Requirements analysis and design testers: These testing tools aid in the defining a testing approach based on requirements and design information. Johnson recommends that test implementation lags behind requirements by 1 step because early requirements are often incorrect. These tools determine if tests cover the entire requirement specification. They also give you an idea of what code has been completely tested, and what code has not been touched. Using this type of testing tool will enable managers to determine if they would like to ship the software even when it contains errors.

2. Regression Test Tools : The Software Engineering Institute (SEI) cites that regression testing is the second most common weakness in any software process (Humphrey, 1990). Regression testing often gives over looked because it is tedious and extremely time consuming. The user interface is basically run by the testing program. A good regression tool allows the use of a keyboard and mouse, and will create a set of batch procedures based on recurring actions by the user. A good example of a regression tool is WinRunner from Mercury Interactive Corp. It allows for a scripting language that supports regression, stress, and test reuse. I will cover the actual application at the end of this presentation because of its extensive capabilities and use in the real world.

3. Test-Reuse Tools. These tools integrate testing into the design and requirements process. In order to utilize this type of testing, you need to be at a SEI Level II. Some preparatory steps an organization can implement before using test-reuse tools are using test scripting languages, test browser, and configuration management tools.

4. Test Generators. This type of tool will create test cases based on the requirement documents. No longer will an organization have to spend hours and hours dreaming up test cases. Using these tools will allow the development team to trace test cases back the requirements phase, linking each test case with a specific feature outlined in the requirements phase. The only drawback with this type of tool is its dependency on good requirements.

5. Test Analyzers. These test tools generate test cases to cover all decisions or logical paths within code, or they verify how much coverage a test suite covers. These test tools provide meaningful object oriented metrics such as depth of inheritance trees

(DIT) or coupling between classes.

6. Complete CAST (Computer Aided Software Testing) Tools. Thusfar, no product falls into this category. This testing tool would cover all of the above and ultimately would be the solution to Testing.

THE REAL WORLD

1. Test incident report. These reports document actual bugs, who is responsible for fixing the bugs, and the testing that follows.

Show example of a test incident report.

2. Real world example.

Discuss Mercury Interactive "The Complete Testing Solution for Java Based Applications"