An Investigation into Measuring and Communicating the Reliability of Software Components

John D. McGregor

Dept. of Computer Science

Clemson University

Juddith Stafford

Dept of Computer Science

Tufts University

Il-Hyung Cho

Dept. of Computer Science

Clemson University

Abstract

Much of the research on component-based software engineering assumes that each component has a single service. This simplifies analyses but it is a significant departure from actual components. This paper reports on an investigation of the feasibility of using design constructs as a means of treating several methods as a single unit. Grouping the services of a component into a few sets satisfies the goal of simplicity while still providing the designer with a more realistic model.

Introduction

The reliability of a software system expresses the probability that the system will be able to perform as expected when requested. This probability is interpreted in the context of an expected pattern of use and a specified time frame. A system’s reliability might be reported as:

System X has a reliability of .99 when used in normal

mode during an eight- hour period of operation.

Normal mode refers to a specific system mode that is defined by a pattern of inputs to the system. This may include specific selections from menus or certain characteristics of data in the input files.

The reliability of a component expresses the probability that the component will be able to perform as expected when requested. Most components have multiple entry points just as most systems have multiple menu selections. It is not as easy to identify the expected pattern of use for a component as it is for a system, particularly one that is being developed for reuse across a number of systems.

The goal of this research is to define a reliability measurement technique that supports the prediction of assembly reliabilities from the reliabilities of the components that form the assembly.

The hypothesis of this investigation is that the role a component plays in a specific protocol is a useful unit upon which to base a reliability measure. We use protocol here in the sense that it is used in the real-time extension to UML [TBA] and in our previous work [TBA]. A protocol is a sequence of service invocations between a set of components. A protocol accomplishes a specific task through the interaction of these components. We begin our definition of component reliability by measuring the ability of a component to perform its role in a protocol as expected.

A fundamental assumption in this work is that components are designed to satisfy roles in various standards or design patterns. We assume that no component is developed in the absence of some rationale for its existence. That rationale might be the role that a component would play in implementing a standard such as a communication protocol. A component might be designed to implement a standard encryption/decryption algorithm.

The contribution of this paper is a method for measuring and communicating the reliability of a component that is useful to developers of components that are intended for use by other developers. The method provides a technique for decomposing the specification of the component into logical pieces about which it is possible to reason. The method then provides a context in which the reliabilities of these logical pieces can be combined to match the use a user defines for the component.

In the next section we provide background on components and reliability sufficient to understand the investigation. The following two sections define the method and provide a detailed example respectively. We conclude with a look at how this approach fits into a compositional reliability prediction method.

Background

Reliability

Composition of Components

Components in Context

The reliability measurement technique uses the concepts - roles, and protocols - defined in the real-time extension to the Unified Modeling Language (UML) [UML]. These concepts are generally applicable and have now been incorporated into architecture description languages such as Wright [Wright].

Components, referred to as capsules in the extension, realize their interfaces through ports. Ports participate in protocols. “… a protocol that defines the valid flow of information between connected ports of capsules.” [UML] Each port plays a particular role in a collaboration between capsules.

Architecture description languages also use the concepts of roles, and protocols. A connection between two components is “specified as a collection of protocols that characterize each of the participant roles in an interaction and how these roles interact” [Wright]

These concepts will be used in the definition of our reliability measurement method.

Component Reliability Measurement Method

The Component Reliability Measurement Method (CRMM) is a technique for making empirical measurements of the reliability of a software component. The technique does not need access to the source code. The technique uses “role” as the basic unit of reliability measurement.

Method Definition

The definition of reliability given in the previous section has three significant parts: a probability, an operational profile that defines context, and a duration of operation defined in some natural units.

Natural units – The reliability of a complete system is usually defined in terms of execution time. The duration of the reliability testing is then chosen to correspond to some time period of interest, such as an eight-hour workday. Further, the execution time can be translated into calendar time if that is a more meaningful level for the analysis. This can be used to evaluate a system that has a natural period of a week’s worth, a month’s worth or even a year’s worth of operation.

For an individual component, the time of execution of a specific service may be very brief. Some components may only execute for a very brief time during a lengthy system execution time. Associating a duration of eight hours with a component that only executed a few milliseconds during that period is not representative. When the component is placed in a system that invokes it more often, its contribution to the system reliability may vary markedly from that observed in the original system.

The reliability of an individual component could be expressed in terms of number of individual invocations but this seems too fine-grained to be useful. Architects are unlikely to know exactly how many times a service of a component will be invoked and thus would not carry out their analysis of a system at that level. CRMM reports reliability values based on the probability that a protocol will be executed successfully. For example, a banking transaction might require several invocations of the decryption service of the security component, but the architect is concerned that the transaction is either completed or not. The exact number of invocations of the decryption algorithm is not important to the analysis.

Operational profile – Reliability is an operational concept. A piece of software’s reliability value will vary depending upon how that software is used. Suppose a component has five services, four of which always succeed and one of which always fails. If all services are equally likely to be used, the system will have an 80% reliability over an aggregate of many uses. But if the service that always fails is never used, the system now has a 100% reliability.

An operational profile describes the frequency with which each element is invoked. The typical component’s public set of services is large and diverse. Some applications may never use some of the services while those same services may be heavily used when the component is deployed in another application. CRMM creates an operational profile for each role in each protocol in which the component is designed to participate. The relative frequency for the services in a role is determined by the number of times a service is used by that role to complete one cycle of the protocol.

A system may have many operational profiles corresponding to different users and it may have many profiles for one user if there are different modes of the system. Similarly, a component may be able to play many roles in many protocols and thus an operational profile is created for each role. CRMM reports a reliability value for each role.

Probability of successful execution -- The probability of successful execution is measured by repeatedly operating a system according to the selected operational profile for the specified unit of time. The reliability is computed by measuring the percentage of those executions that terminate successfully. For a system a reliability value is reported for each operational profile. If certain critical or heavily-used profiles correspond to lower than acceptable reliability values, the system may be modified to improve those values.

For a component that will be used in a variety of contexts, a large number of reliability values should be reported so that the architect can compute the effective reliability that will be observed in a particular system. For example, a component is provided that plays a role in each of five protocols. For each role, there is a reliability value. The architect intends to use the component in three of its five roles. The architect determines the relative frequency with which each role will be invoked from the context of the deployment.

The effective reliability of the component in a particular deployment is computed by combining the relative frequencies for each of the roles and the measured reliabilities for each of the roles. Assume the reliabilities for the 5 roles are .9, .91, .92, .93, and .94. If the architect intends to use the first, third, and fifth roles with equal frequency, the computation would be:

.333*.9 + .333*.92 + .333*.94 = .92

The reliability portion of a component datasheet would contain the role reliabilities. (This assumes that role descriptions are also available as part of the component documentation.)

Role / Reliability
A / .9
B / .91
C / .92
D / .93
E / .94

The architect’s reliability analysis worksheet would include each component defined in the architecture and a computation that looks something like this:

Role / Reliability / Relative Frequency / Contribution
A / .9 / .333 / .30
B / .91 / 0 / 0
C / .92 / .333 / .31
D / .93 / 0 / 0
E / .94 / .333 / .31
Component Reliability / .92

The CRMM Process

The CRMM measures and communicates reliability values for a component. In this section we present a detailed outline for applying CRMM given a component and its documentation.

  1. Establish the context for the measurements

Empirical measurements are subject to error. One source of error is the sampling of parameter values for the services that are invoked during reliability testing. Multiple measurements are made and the results aggregated to provide an accurate value. The number of measurements that are required depends upon how much the measurements vary.

Establish the confidence level you wish to have in the accuracy of the reliability value. The typical value is 95% or higher. The reliability value will be reported as an interval of values. This confidence interval is sufficiently large that we have 95%, or other selected value, confidence that the real reliability value is within the interval. Since no one cares if the actual reliability is greater than believed, a one-tailed interval is used.

  1. Identify the specific roles in which the component is designed to participate.

This identification comes from the documentation of the component. The designer may have listed the roles and identified the services that participate in those roles. The reliability test plan identifies each of the roles and for each role the services that implement the role.

Establish an operational profile for each role. The profile determines how the test suite should invoke services of the role. Each test case will be one complete cycle of the role. In that cycle some of the services may be invoked multiple times. The profile captures the number of times each service is invoked. For cases such as a transaction manager where some methods may be called an indefinite number of times, this becomes another sampling variable.

If the designer has not been explicit about the roles, one alternative position is to use the explicit types interfaces that the component may implement.

  1. Construct a reliability test suite for each role.

For each role, a test script is created that obeys the constraints of the role. The constraints usually include a specific ordering of the service invocations defined in the role. Other constraints include the types for the parameters on each service. Although we assume no access to the source code, an analysis of each parameter type leads to a set of partitions of values that are of that type. Each partition contains values for which we believe the component will behave the same.

The test suite should include test cases that (1) test each possible ordering of service invocations and that (2) adequately sample over the parameter spaces for each service. These two conditions provide a means of measuring how completely the test suite covers the component roles. If the initial test suite is not sufficient to achieve a reliability measure in which we have sufficient confidence, the test designer searches for additional orderings of invocations or for parameter values that are likely to exercise new sections of the component. The most likely source of new test cases is combinations of parameters.

  1. Apply each test suite to the component in an appropriate environment

The goal of reliability testing is to evaluate the ability of the software being measured to perform as expected. Sufficient environment is needed to support the component in its operation but the environment should be the minimum necessary for that support.

  1. Analyze the results and compute the reliabilities

Each test run is evaluated and marked as either passed or failed. Each failure case is evaluated to determine whether the environment produced the failure or whether the component is responsible. Those runs in which the component performed as expected are discarded.

  1. Expand test suite to reach the level of confidence desired.

Once the analysis is completed, if the level of confidence has not been reached, additional test cases may be created and executed. The coverage criteria given in step 3 provide direction on how to effectively expand the test suite to cover additional ground.

If the test results do not support a reliability value that is satisfactory, the component might be returned for further development or you may decide not to acquire the component.

Two types of confidence

The use of a confidence interval is intended to measure how certain we are about the measurement we have made. It does not necessarily describe how confident we are that we have measured the correct quantity. For example, we might choose parameters for three test cases, all of which result in the same ten lines of code being executed. There is very close agreement among the three measures so our confidence of the accuracy of the measurement will be high even though choosing other parameter values might have led to a very different result. It is the variance amount the measurements that give confidence.

The use of an operational profile for a role is intended to indicate how completely the reliability test suite covers the functionality. The operational profile ensures that the correct quantity is being measured. If portions of the software are not executed, it is because the role does not use that software and it is not pertinent to the measurement.

CRMM uses both the operational profile and a confidence interval to report the reliability for a component role. In this way we have a high confidence that the measurement is an accurate value for the correct quantity.

Example of applying the Method

To illustrate CRMM we selected an SSH client component. The component provides both SSH1 and SSH2 protocols. Many system administrators have eliminated their SSH1 servers and only accepted SSH2 connections. To report the reliability of this component by combining SSH1 and SSH2 behaviors would not serve the component user as well as reporting the reliabilities of the two capabilities separately.

We measured the reliability of the two protocols separately.

  1. A communication protocol must be highly accurate and we need to have great confidence that it is that accurate. We chose to be 99% confident in the reliability measurement.
  2. The component under test has two distinct roles. The first is to participate in an SSH1 connection. The second is to participate in an SSH2 connection. In each of those roles the component supports the same actions: connection, authentication, and encryption.

SSH1, SSH2

http, sock4, sock5 proxy types

  1. Test frames and test cases were coded in Java. The test frames were constructed from sample telnet and ftp applications include with the SSH component library. The test frames were instrumented to allow assessment of information flowing to and from the SSH component.
  1. The tests were executed using the skeleton telnet and ftp applications.
  1. Each test run was evaluated. Each test was only marked “failed” if the error occurred in the SSH component and not in the sample application code.
  1. The desired level of confidence was achieved using the initial test suite. There was no need to expand the test suite and reapply.

Conclusions and Future Work

References