| Abstract |
An increase in the number of software programs failing suitability and effectiveness requirements during DoD Initial Operational Test and Evaluation (IOT&E) has resulted in a mandate from the Office of the Secretary of Defense (OSD), Director, Operational Test & Evaluation (DOT&E) to manage software reliability growth within a program's development and test lifecycles. The intent is to allow Program Managers (PMs) to better determine whether the current system state is sufficient for a product release, or if the release date should be deferred if existing software issues require resolution and corrective actions. The purpose of this paper is to develop a methodology for applying software reliability growth models (SRGM) to track and predict software reliability growth by applying categorizations of software usage in DoD systems. DOT&E defines three types of software usage in DoD systems: (1) Hybrid systems containing a combination of software, hardware, and human interfaces, but critical functionality is a combination of hardware and software sub systems, i.e., complicated ground combat vehicles, aircraft, and ships, (2) Net centric systems consisting of both hardware and software, but the critical functionality is software centric with hardware being highly reliable and/or redundant, i.e., the C4ISR concept of Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance, and (3) Space systems, principally satellites. As defined in IEEE/AIAA P1633, Recommended Practice on Software Reliability, a software reliability model specifies the general form of the dependence of the failure process on the principal factors that affect it: fault introduction, fault removal and the operational environment. Reliability assessments in test and operational phases may follow the same Failure Reporting, Analysis and Corrective Action System (FRACAS) procedure, but there a few differences. During the test phase, software faults are fixed when the corresponding software failures are detected, depending on severity of the failure effects. As a result of design improvements, reliability growth (e.g., decreasing failures over time) should be observed. However, in the operational phase, correcting a software fault may involve delays in providing the new software release or software patch to the customers' sites. Therefore, the realization of reliability growth in the customer application may not be immediate. Software reliability modeling is done to: (1) estimate the execution time during test required to meet a specified reliability objective, and (2) identify the expected reliability of the software when the product is released. The three general classes of software reliability prediction models we consider are: (1) Exponential non-homogeneous Poisson process (NHPP) models, (2) non-exponential "standard" NHPP models, and (3) Bayesian models. The NHPP-based models are more commonly used because of their simplicity, convenience, and tractability. In spite of their simplicity, they often provide estimates having a good fit to the actual failure data. The other two classes of model are used less often, usually after experimentation has shown that the NHPP-based model predictions do not fit the actual data well. The non-exponential NHPP models, for example, assume that the earlier discovered faults have a greater impact on reducing the failure intensity than those encountered later. They also assume that there is no upper bound on the number of total failures. These two attributes make these models more likely to produce accurate estimates in environments in which the software is undergoing change in addition to defect repair. Other types of software reliability models, such as the Schneidewind model and Shick-Wolverton model, which are NHPP-based models, and the Littlewood-Verrall model, a Bayesian model, will also be assessed and compared in the paper. We also examine the following important data collection issues: • Assuring that failures are counted consistently for each component of the system and for each testing phase. Usually this includes a stipulation that failures are only counted once during testing, even if they occur more than once. This is to maintain consistency with the assumption that underlying faults are repaired. • Assuring that all failures are recorded and properly categorized. The authors' experience indicates that failures are not always recorded. In addition, if software failures are tracked in a problem reporting system that also tracks other types of failures (e.g., hardware, procedural), software failures may be incorrectly classified if the problem closure process does not ensure the failures are correctly labeled. • Assuring that the dates and times at which the failures were observed are correctly recorded. This is particularly important if the SRGMs being used are based on times between failures. |
