Software Complexity Factor in Software Reliability Assessment

Transcription

1 Software Complexity Factor in Software Reliability Assessment Meng-Lai Yin, Ph.D., ECE department, California Polytechnic University, Pomona Jon Peterson, Raytheon Company, Fullerton Rafael R. Arellano, Raytheon Company, Fullerton Key Words: Software reliability, Early-stage prediction, Software complexity metrics, McCabe s cyclomatic complexity. SUMMARY & CONCLUSIONS A straightforward early-stage software reliability assessment method was proposed and applied to large-scale, software-intensive programs (Ref. 1). In this method, software size (in KSLOC) was used as the primary complexity factor, due to the limitation of detailed information available during the early stage of a program. Now that failure data are available from those programs, our next step is to evaluate and refine the method so that better assessment can be provided for future programs. In this paper, the performance of the early-stage method is addressed by comparing with other failure-data-based models. Since our major concern in the original method is the use of software size as the only complexity factor, the complexity issue is probed. The conclusions of this study are two folded. (1) The performance of the early-stage method is compatible with that of other failure-data based models, and (2) The early-stage method can be improved by adding the consideration of a functional complexity indicator n, which is derived from the McCabe s cyclomatic complexity (Ref. 2, 5). 1. INTRODUCTION Early-stage software reliability estimation is essential to program trade studies during the proposal and design stages. However, detailed information regarding the software product being analyzed is typically not available at that time. Although software size (a commonly used indicator for software complexity) does represent some aspects of software complexity (Ref. 2), it does not provide the necessary fidelity for a software reliability assessment. This observation triggers the study presented here. 2.1 The Early-Stage Method 2. BACKGROUND A Non-Homogeneous Poisson Process (NHPP) is used to describe the software failures behavior. Basically, this method assumes that the failure times between any two consecutive software failures are exponentially distributed and independent. The occurrence of a software failure is assumed immediately followed by the removal of the corresponding fault. The NHPP described above is formed by a sequence of non-negative, real-valued, infinite random variables X 1, X 2, X 3, X i, Each of these random variables represents the time between two consecutive software failures. An exponential distribution is assumed; hence a constant failure rate can be applied for each X i. However, each X i has its own corresponding density function, and is referred to as λ(i), where λ(i) is a function of the number of software faults existing in the code during the time X i is of concern. This model is similar to the one supplied in the Jelinski-Moranda model (Ref. 4). That is, λ(i) is proportional to the number of remaining faults existing in the software. In our case, λ(i) is presented as (r-i) λ, where r is the number of remaining faults, i.e., the faults that exist when the software is first delivered, and i is the number of faults that have been removed. Note that the expected duration to experience a certain number of failures, say i, is E[X 1 + X X i ], which can be computed as E[X 1 ] + E[X 2 ] + + E[X i ]. If we know the time for removing i software faults is T, then λ can be calculated by solving the following equation (assuming an exponential distribution): 1/λ() +1/λ(1) + +1/λ(i-1) = T (1) When implementing this early-stage assessment method, certain experience-based assumptions must be made in order for the above equation to be solvable. In essence, the values for r (the number of remaining faults), i (the number of faults removed during duration T), and the duration T (the period of time for i software failures to occur, starting from the time that the software is ready to be delivered) must be assumed. Keep in mind that no empirical failure data is available at the time this assessment is made. Therefore only very preliminary information is used to assess the values of r, i and T. These include the SEI (Software Engineering Institute) CMM (Capability Maturity Model) level (Ref. 2) and the estimated size of the executable code. 2.2 The Performance of the Method The original method considers the software size as the only complexity factor and estimates the time to the next failure for software products. A sample estimated the first 15 time between failure (TBF) compared to the actual measured the first fiftieth time between failure is shown in Figure 1. It can be seen from this sample there is a significant difference in TBF after 25 faults have been removed. An RAMS /4/$ IEEE

2 interpretation is that after 25 software faults were removed, most of the software faults that precipitate failures during the normal operations have been detected and removed thus it will take a much longer time to experience the rarely occurred faults. Within this paper, we refer to the situation when software reaches this state as reaching the steady state. In this study, four different CSCIs (Computer Software Configuration Items) from a large-scale software program are used. These CSCIs demonstrate the same phenomenon, i.e. they experienced a significant change between the measured TBF and the predicted TBF. However, this occurs after a different number of faults were removed for the different CSCIs. When operational failure data was available, the software reliability prediction tool CASRE (Ref. 3, 4) was applied. As a comparison, the differences between the actual TBF and the CASRE estimated MTBF, which is based on actual failure data, are shown in Figure 2. The CASRE (Computer-Aided Software Reliability Estimation) tool was developed by the JPL (Ref. 3) that provides the capability of estimating software reliability based on several models. In this particular case, the Littlewood-Verrall reliability growth model from the CASRE tool has the best fit. From Figures 1 and 2, it is obvious that the early-stage software reliability estimation method is as good as the failure-data-based CASRE model before the software reaches the steady state. Moreover, the time where the significant difference occurs is consistent with the two estimation methods, i.e. when 25 software faults had been removed. Note that the early-stage method is much simpler and is not based on any failure data. This preliminary result encouraged us to pursue further. From the preliminary results, two interesting questions were raised. First, how many software faults have to be removed before the software reaches its steady state? Secondly, when a piece of software reaches its steady state, what is its expected time between failures? In the following, we show that software complexity is closely related to the answers of these two questions. Actual TBF vs. Early-stage predicted TBF TBF (as a function of faults rem oved ) (hours) Predicted MTBF Actual TBF Q ty. O f R e m o v e d S oftw are F a u lts Figure 1. Performance of the early-stage software reliability estimation method Time Between Failures (hours) Differences between CASRE prediction outputs and the Actual TBF number of faults removed Figure 2. Differences between the actual TBF and the Littlewood-Verrall model predicted MTBF RAMS /4/$ IEEE

3 3. THE COMPLEXITY CONCERNS 3.1 Size is Not Enough The CSCI numbers and their executable size in KSLOCs are listed in Table 1. When plotting the average TBF during the steady state and the CSCI size, we notice that, in general, the smaller the CSCI, the longer the TBF. However, there is one exception, as shown in Figure 3. Stead y State TB F (hours) Table 1. CSCIs Sizes CSCI # , 15, 1, 5, S iz e v s. ac tu a l s tea d y s ta te T B F Figure 3. Size vs. Steady-State TBF for the four CSCIs When the size of the software is larger, one would expect more bugs and hence more failures would be experienced before the software reaches its steady state. In Figure 3, although the size of the CSCI 4 (73.8 KCSLOC) is about half of that for CSCI 2 with size KSLOC, the steady-state TBF of the two is about the same. #Faults Removed Size vs. # Faults Removed to reach Steady State Figure 4. Size vs. # Faults Removed for Reaching Steady State Figure 4 shows the sizes of the four CSCIs with the number of faults that need to be removed to reach the steady state. From this figure, we can see that although CSCI 2 and CSCI 4 have comparable number of faults to be removed in order to reach the steady state (i.e., 25 and 26), their sizes are quiet different (152.8 and 73.8 KSLOC). The above observation implicates that the size alone cannot tell the whole story. 3.2 The Functional Complexity Factor n Several software complexity metrics are surveyed in Ref. 2, including the lines of code (size) and McCabe s cyclomatic metric (Ref. 5). Not all the existing software complexity metrics are suitable for early-stage reliability assessment, due to the limited information available at the early stage. The McCabe s cyclomatic metrics was originated from the exploration of the number of paths for testing purposes. Since the cyclomatic metric is mainly used for testability and understandability purposes, it is defined at the code level. Nevertheless, when applying it to high-level system description, the result seems promising. The original formula for obtaining the metric is M= e- n+2p, where e is the number of edges from the graph theory s standpoint, n is the number of nodes and p is the number of independent (unconnected) parts in the graph. Figure 5 shows the directed graph for an if-then-else program structure. n3 e3 e4 n1 n4 Figure 5. The directed graph of an if-then-else program structure As seen in the figure, the number of nodes (n) is 4, the number of edges (e) is also 4, and since this graph is connected, the value of p is 1. Therefore the cyclomatic number M is , which is 2. From the testability point of view, this means that two paths are necessary to complete the testing process. We applied this concept to a much higher level, i.e. the CSCI level, and analyzed the four CSCIs. Thus, n is the number of sub-modules included in one CSCI, e is the number of communication paths between the n sub-modules. After analyzing the Software Requirement Specifications (SRS) for the four CSCIs, it was realized that all the submodules within each CSCI communicate with each other. In other words, the number of sub-modules n determines the value of e, and p is always 1. Hence, it is sufficient enough to represent the McCabe complexity metric by the value of n. In this paper, n is referred to as the functional complexity indicator. e1 e2 n2 RAMS /4/$ IEEE

4 Figure 6 represents the relationship of the size and the value of n for the four CSCIs studied. As shown in this figure, CSCI 4 has a relatively large number of n (i.e., 28). Recall that CSCI 4 has a relatively small TBF (Figure 3) Functional Complexity Indicator vs. Size Functional Complexity Indicator n Figure 6. The Relationship between the Functional Complexity Indicator n and the Size Figure 6 indicates that the functional complexity indicator affects the software s TBF. This finding encourages us about applying the functional complexity indicator in early-stage software reliability prediction. Especially, since an estimate of the number of sub-modules for a CSCI is an information that is usually available during the early stage. 3.3 The Effects of n Two attributes are of particular interests. We want to explore the relationship between the functional complexity factor n and the number of faults removed for a piece of software to reach the steady state; also, the relationship between n and the steady state TBF is essential for a program. After a thorough analysis, it is found that, for the four CSCIs studied, n 2 Size demonstrates a positive relationship with the number of software faults removed to reaching the steady state, as Figure 7 shows. n n ( n 1) = n 2 /2. Note that in our analysis, all four 2 2 CSCIs form complete graphs. In this paper, the number of faults removed is related to the number of failures that occurred. Moreover, we have assumed a one-to-one mapping between software failures and software faults. In order for a piece of software to reach its steady state, all failures that occur during normal operation must have occurred. If the locations of software failures were distributed evenly, then, the number of possible paths within a program would determine the number of failures that have to occur in order to reach the steady state. In our cases, the number of paths is determinable by the number of submodules that exist in a CSCI, and it is proportional to n 2. On the other hand, as stated before, the size of the software also has impact on the number of failures that occurred. This explains the positive effects of n 2 Size on this feature (i.e., the number of software faults need to be removed before reaching the steady state). Another interesting topic is the relationship between the functional complexity indicator and the steady state TBF. It turns out that a positive relationship exists between the TBF and the functional complexity indicator n, as Figure 8 shows. Actual TBF in Steady State (hours) Functional Complexity Indicator vs. Time Between Failures in Steady State ,24 13,2 5,4 5, Functional Complexity Indicator n Number of Software Faults Removed 9 25 The Effects of n^2*size n^2 *Size /1 Figure 7. The Effects of n^2 * Size An explanation of this is from a graph s point of view. Since McCabe s cyclomatic number is a function of n and e (number of egdes), for a complete graph with n nodes, e is 61 Figure 8. Actual TBF in Steady State and the Functional Complexity Indicator n This observation is interesting in that the TBF was determined solely by the size in our original model. This study shows that the functional complexity indicator is closely related to the steady state TBF. In a way, the more modules are subdivided for a CSCI (i.e., the higher n), the more communications are required among those sub-modules. In other words, the communications among software modules contribute to software failures. 4. FUTURE WORK In this study, the software complexity issues are explored with regard to software reliability prediction. The data assures the performance of the original model when compared to the performance of the CASRE tool. More importantly, this study indicates that more complexity factors should be considered in RAMS /4/$ IEEE

5 early stage prediction. In essence, a combination of software size and the functional complexity indicator demonstrates better performance than using the size alone. Currently more failure data is being collected, and further research in this area is being conducted. ACKNOWLEDGMENT The authors appreciate Raytheon s software process for the time-consuming software failure data collection. REFERENCES 1. M.L. Yin et al, An Adaptive Software Reliability Prediction Approach, 23 rd Annual Software Engineering Workshop, Dec. 2-3, 1998, NASA/Goddard Space Flight Center, Greenbelt, Maryland. 2. Stephen H. Kan, Metrics and Models in Software Quality Engineering, Addison Wesley, 2 nd edition, A.P. Nikora, CASRE User s Guide, Jet Propulsion Laboratories, August Michael R. Lyu (editor), Handbook of Software Reliability Engineering, McGraw-Hill, T.J. McCabe, A Complexity Measure, IEEE Transactions on Osftware Engineering, Vol.2, No.4, Dec. 1976, pp BIOGRAPHIES Meng-Lai Yin Electrical and Computer Engineering Dept. California State Polytechnic University, Pomona 381 West Temple Avenue, Pomona, CA 91768, USA Tel: , Fax: myin@csupomona.edu Meng-Lai Yin is an associate professor at the ECE department, California State Polytechnic University at Pomona. She has more than 13 years industrial experiences in the field of dependability, performability and safety analysis. She received her Ph.D. and MS degree in Computer Science from University of California at Irvine, in 1995 and 1989, respectively. She also holds a Masters degree in Electrical and Computer Engineering from National Cheng-Kung University, Taiwan, in Her research interests include software reliability, maintainability, and performability. Jon R. Peterson Raytheon Company Bldg. 676, M/S Z Hughes Drive, Fullerton, CA 92834, USA Tel (Work): , Fax: , jpeterson3@west.raytheon.com J. R. Peterson has more than 19 years experiences in modeling and analysis. He is currently a senior staff engineer for Raytheon, working on the DDX, Swiss FLORAKO and the WAAS FAA Programs. His duties include hardware systems reliability modeling and software systems reliability modeling. Mr. Peterson holds a BS degree in Mathematics from North Dakota State University and a MS degree in Applied Statistics, also from North Dakota State University. Rafael Arellano Raytheon Company Bldg. 676, M/S Z Hughes Drive, Fullerton, CA 92834, USA Tel: (714) , Fax: (714) rrarellano@raytheon.com Rafael R. Arellano is a Systems Engineering Manager who has been with Hughes/Raytheon for 2 years. He is responsible for directing Reliability, Maintainability, and Availability technical activities and cost and schedule management for advanced projects ranging from Undersea Warfare to Satellite Navigation Systems. He holds a BS in Physics from California State University, Fullerton. One of his biggest technical motivations is to apply technology and concepts developed by his team into new or existing products. RAMS /4/$ IEEE