On the Role of Weibull-type Distributions in NHPP-based Software Reliability Modeling

Transcription

1 International Journal of Performability Engineering Vol. 9, No. 2, March 2013, pp RAMS Consultants Printed in India On the Role of Weibull-type Distributions in NHPP-based Software Reliability Modeling XIAO XIAO and TADASHI DOHI Department of Information Engineering, Graduate School of Engineering, Hiroshima University, Kagamiyama, Higashi-Hiroshima , JAPAN (Received on February 27, 2012, revised on December 04, 2012) Abstract: The non-homogeneous Poisson processes (NHPPs) based software reliability models (SRMs) have gained much popularity in actual software testing phases to assess the software reliability, the number of remaining software faults and the software release scheduling. It is well known that the Weibull distribution plays an important role in reliability applications because of its flexibility in being able to represent various patterns of failure rate functions. In this paper, we introduce some recent generations of Weibull distribution to represent the underlying software fault-detection time distribution of the NHPP-based SRMs. We study the effectiveness of Weibull-type distributions in software reliability modeling through goodness-of-fit test and prediction analysis. Keywords: software reliability, NHPP, Weibull distribution, goodness-of-fit test, prediction analysis. 1. Introduction One class of non-homogenous Poisson processes (NHPPs) based software reliability models (SRMs) is concerned with modeling the number of software faults detected in testing phases. Since the NHPP-based SRM is characterized by its mean value function, which is proportional to the cumulative distribution function of software fault-detection time, the well-known Goel-Okumoto NHPP-based SRM [2] can be derived by assuming the exponential distribution as the software fault-detection time distribution. By selecting the traditional Weibull distribution function as the software fault-detection time distribution, a generalized Goel-Okumoto NHPP-based SRM [3] with S-shaped growth curve of the detected software faults was proposed in Recently, Xiao and Dohi [7] *Corresponding author s dohi@rel.hiroshima-u.ac.jp 123

2 124 Xiao Xiao and Tadashi Dohi developed ED-NHPP-based SRMs by applying the equilibrium distribution (ED) to the underlying fault-detection time distribution. They studied four underlying fault-detection time distributions, including the exponential and Weibull distribution, and concluded that the ED-NHPP-based SRMs outperformed the corresponding existing ones in many data sets from the perspective of goodness-of-fit and prediction performance. This motivates us to study the effectiveness of other underlying fault-detection time distributions. It is well known that the Weibull distribution plays an important role in reliability applications because of its flexibility in being able to represent various patterns of failure rate functions. Based on the traditional Weibull distribution function, many generalizations and modifications have been proposed ([5]). Pham and Lai [6] gave a discussion on some Weibull models that appeared since In this paper, we introduce recent generations of Weibull-type distributions to represent the underlying software fault-detection time distribution of the NHPP-based SRMs. We study the effectiveness of Weibull-type distributions in software reliability modeling through data analysis with real software development project data. 2. NHPP-based Software Reliability Modeling Suppose that the initial number of software faults is a Poisson random variable with mean >0, and software faults are detected at independent and identically distributed (i.i.d.) random times. Let denote the cumulative distribution function of the software fault-detection time, then the stochastic point process {, 0} is called an NHPP if its probability mass function is of the following form: Pr = = exp, (1)! where denote the number of software faults detected by testing day. = E = is the mean value function of NHPP and means the expected cumulative number of software faults detected by testing day. From this modeling framework, almost all NHPP-based SRMs can be derived by choosing the software fault-detection time distribution. For instance, if =1 exp, then the resulting NHPP-based SRM becomes the Goel-Okumoto SRM [2]. The commonly used technique for parameter estimation is the maximum likelihood (ML) method. Let denote the vector of model parameters in the mean

3 On the Role of Weibull-type Distributions in NHPP-based Software Reliability Modeling 125 value function = ;. Suppose that software fault count data,,,,,, are available, where and are the -th testing day and the cumulative number of software faults detected by, respectively. Then, the log likelihood function of the NHPP with n software fault count data is given as a function of, by = ln ; ; ; ln!. (2) Then the ML estimate of, say, is given by the solution of argmax. 3. NHPP-based SRMs with Weibull-type Distributions The Weibull-type distribution plays a significant role in reliability applications, such as human life and electronic devices. In addition to these, Pham and Lai [6] discussed some recent Weibull-related studies that appeared since 2004 and briefly summarized the reliability function and the characteristic of the generalized Weibull distributions. Here we offer a review of the distributions mentioned in reference [6] and show the cumulative distribution function =1 instead in Table 1, where the distributions are classified as GP1, GP2 and GP3 by the number of free parameters. The Author of each distribution is kept the same with that in reference [6], so that the reader can find the details of each distribution therein. In this paper, we focus on studying the role of these Weibull-type distributions in software reliability modeling. Under the modeling framework introduced in the last section, we derive 12 NHPP-based SRMs by assuming the Weibull-type distributions as the software fault-detection time distribution. For convenience sake, we name the SRMs based on the distributions in GP1, GP2 and GP3, as M2-1 through M2-5, M3-1 through M3-5 and M4-1 through M4-2, respectively. Note that M2-2 is the well-known Goel-Okumoto SRM [3] where the traditional Weibull distribution function is assumed as the software fault-detection time distribution. Generally, it can be considered that 3- and 4-parameter Weibull-type distributions are developed in order to gain much more flexibility or to be able to capture the non-monotonic behavior of the underlying failure rate function. Therefore, it is of great interests to know which group is more appropriate to software reliability modeling. In

4 126 Xiao Xiao and Tadashi Dohi numerical study, we compare the goodness-of-fit and predictive performance of the derived 12 NHPP-based SRMs, especially taking notice on the differences among 2-, 3- and 4-parameter Weibull-type distributions. Additionally, we calculate the quantitative reliability evaluation measure such as software reliability. Table 1: Generalized Weibull Distributions Group Author Model Gompertz (1825) 1 exp 1 exp >0, < < M2-1 Weibull (1951) 1 exp, >0) M2-2 GP1 Smith and Bain (1975) 1 exp 1 exp, >0) M2-3 Jiang and Murthy (2001) exp / a, >0 M2-4 Bebbington, Lai and Zitikis (2006) 1 exp exp /, >0 M2-5 Slymen and Lachenbruch (1984) 1 exp exp + M3-1 Mudholkar and Srivastava (1993) 1 exp /, >0, 0 M3-2 GP2 Marshall and Olkin (1997) 1 / a,, >0 M3-3 / Xie, Tang and Goh (2002) 1 exp 1 exp /,, >0 M3-4 Lai, Xie and Murthy (2003) 1 exp exp, >0, 0) M3-5 GP3 Xie and Lai (1995) 1 exp / /,,, >0) M4-1 Nadarajah and Kotz (2005) 1 exp exp 1, >0,, 0) M Numerical Study 4.1 Goodness-of-fit Test and Prediction Analysis We use 4 real project data sets, DATA3, DATA7, DATA8 and DATA14, which are cited from Lyu [4]. They consist of 46, 535, 481 and 266 software fault counts, respectively, and are renamed as DS1 through DS4 in this paper. We estimate model parameters by means of the ML method, and calculate the information criteria AIC and BIC as well as MSE. They are of the following forms: = 2 +2, (3) = 2 + ln, (4) = /, (5) where denotes the maximum log likelihood, is the number of model parameters and the software test terminates at -th testing day. Additionally, we take

5 On the Role of Weibull-type Distributions in NHPP-based Software Reliability Modeling 127 place the Kolmogorov-Smirnov (K-S) test with two significance levels (5% and 1%). If the K-S test is accepted, it means that the SRM assumed fits to the underlying data. Table 2 presents the goodness-of-fit test results for all data sets. Since M2-4 and M2-5 are not accepted in the K-S test with significance level 5% (1%) in 3 (1) and 4 (3) data sets, we compare AIC, BIC and MSE of the other 10 SRMs. It can be seen that M2-2 (Goel SRM [3]) only outperforms the other GP1 members in DS3. On the other hand, M2-3 seems better because it performs best among GP2 in 2 data sets regardless of evaluation measures. M3-1 and M4-1 are the best in each group, respectively, when taking notice on AIC, BIC and MLL. Furthermore, we found that the most frequent pattern was (M3-1 M4-1 M2-3) when ranking them by AIC, BIC and MLL, while M4-1 came to the top in case of MSE. This is not a surprising result because MSE measures the distance between estimates and real data, so the more parameters the model has, the nearer the distance is. By contrast, AIC/BIC also takes account of model dimension. Therefore, it can be concluded that 3-parameter Weibull-type distributions are more appropriate to software reliability modeling. Next, we examine the predictive performance of the 12 NHPP-based SRMs, where two prediction measures are used: predictive log likelihood (PLL) and predictive least squares error (PLS). The PLL is defined as the logarithm of likelihood function with future data at an observation point, and the PLS is the residual sum of errors between the mean value function and the future data from an observation point. Table 3 presents the prediction analysis results at observation point 50% and 90% of a whole data set. It is checked that GP2 provides better prediction performance in DS1 and DS3 in spite of observation points. GP1 outperforms GP3 especially at observation point 90%. 4.2 Quantitative Software Reliability We evaluate the quantitative software reliability, which is the probability that the software system does not fail during a specified time interval after release. Suppose that the software test terminates at the -th testing day, and the product is released at the same time to the user or market. Then, the software reliability for the operational period, is defined by =exp +. (6)

6 128 Xiao Xiao and Tadashi Dohi Table 2: Goodness-of-fit Test AIC BIC MSE MLL AIC BIC MSE MLL DS1 DS2 M M M M M M M M M M M M DS3 DS4 M M M M M M M M M M M M

7 On the Role of Weibull-type Distributions in NHPP-based Software Reliability Modeling 129 Table 3: Predictive Performance PLL50 PLSE50 PLL90 PLSE90 PLL50 PLSE50 PLL90 PLSE90 DS1 DS2 M M M M M M M M M M M M DS3 DS4 M M M M M M M M M M M M However, the calculation of software reliability in this way is not very helpful to the practitioner because drops to 0 immediately in most cases. To overcome this deficiency, Fujii et al. [1] proposed a software reliability inference. Here we use a

8 130 Xiao Xiao and Tadashi Dohi similar method to provide the practitioners with a useful measure for decision makings. Assume that the last software fault was detected on the -th testing day, and there exists the zero-count fault period. Under the condition that no fault is detected in (, with an additive testing period, the conditional software reliability can be defined as =exp +, (7) where = + and denotes the operational period. This assumption can be accepted in an intuitive way because it is considered that the product is more reliable with a fault-free period before release. In Table 4, case = 0 and case = 14 correspond to the software reliability calculated by Equation (6) and (7), respectively. This table shows the probability that the system does not fail by the 360-th day from. It can be seen that, when = 0 (without fault-free period), this probability decreases to less than 0.1 one year after release in DS1 and DS4. On the other hand, it remains more than 0.8 in most cases when = 14 (with a 2-week- fault-free period). Among the 12 NHPP-based SRMs, M2-4 provides the most pessimistic evaluation, and the ones based on 4-parameter Weibull-type distributions give a relatively optimistic evaluation in all the data sets. 5. Conclusions In this paper we introduced 12 Weibull-type distributions to the NHPP-based software reliability modeling, by assuming them as the underlying software fault-detection time distribution. In the numerical examples with 4 real software fault data sets, we compared the goodness-of-fit and predictive performance of the 12 NHPP-based SRMs, and found that 3-parameter Weibull-type distributions are more appropriate to software reliability modeling. Additionally, we also calculated the quantitative software reliability, with a fault-free period introduced. This method has been shown much useful for the practitioners through the numerical studies performed here.

9 On the Role of Weibull-type Distributions in NHPP-based Software Reliability Modeling 131 Table 4: Software Reliability = 0 = 14 DS1 DS2 DS3 DS4 DS1 DS2 DS3 DS4 M M M M M M M M M M M M References [1]. Fujii, T, T. Dohi, H. Okamura, and T Fujiwara. A Software Accelerated Life Testing Model. Proceedings of the 16th Pacific Rim International Symposium on Dependable Computing (PRDC2010), Tokyo, Japan, December 13-15, 2010; 85-92, IEEE CS Press. [2]. Goel, A. L., and K. Okumoto. Time-dependent Error-detection Rate Model for Software Reliability and other Performance Measures. IEEE Transactions on Reliability, 1979; R-28 (3): [3]. Goel, A. L. Software Reliability Models: Assumptions, Limitations and Applicability. IEEE Transactions on Software Engineering, 1985; SE-11 (12): [4]. Lyu, M. R. (Ed.). Handbook of Software Reliability Engineering, McGraw-Hill, New York, [5]. Murthy, D. N. P., M. Xie and R. Jiang, Weibull Models, Wiley-Interscience, New Jersey, 2004.

10 132 Xiao Xiao and Tadashi Dohi [6]. Pham, H. and C. D. Lai. On Recent Generalizations of the Weibull Distribution, IEEE Transactions on Reliability, 2007; R-56 (3): [7]. Xiao, X., and T. Dohi, NHPP-Based Software Reliability Models Using Equilibrium Distribution, IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 2012; E95-A (5): pp Xiao Xiao received the B.Sc. (Engineering) and M.Sc. (Engineering) from Hiroshima University, Japan, in 2008 and 2010, respectively. She is now working as a Ph.D. candidate in Graduate School of Engineering, Hiroshima University. She is a Student Member of IEEE, ACM, IEICE and ORSJ. Tadashi Dohi received the B.Sc. (Engineering), M.Sc. (Engineering) and Ph.D. (Engineering) from Hiroshima University, Japan, in 1989, 1991 and 1995, respectively. Since 2002, he has been working as a Full Professor in the Department of Information Engineering, Graduate School of Engineering, Hiroshima University. He is a Regular Member of ORSJ, JSIAM, IEICE, REAJ and IEEE.