Revisiting the performance of mixtures of software reliability growth models

Transcription

1 Revisitig the performace of mixtures of software reliability growth models Peter A. Keiller 1, Charles J. Kim 1, Joh Trimble 1, ad Marlo Mejias 2 1 Departmet of Systems ad Computer Sciece 2 Departmet of Egieerig Maagemet ad Systems Egieerig Howard Uiversity George Washigto Uiversity Washigto D.C Washigto D.C Abstract I this paper, we evaluate the performace of a mixture of software reliability growth models (Super models) agaist the origial base models. The base models chose are four well kow ohomogeeous Poisso process (NHPP) growth models proposed i literature. We use the method of maximum likelihood for fittig these models to data. The Super models are based o the above base models plus a selectio criterio based o past predictio measures. Four Poisso type reliability models ad four Super models are evaluated based o their performace o three data sets. The performace is judged by the relative error of the predicted umber of failures over future time itervals relative to the umber of failures evetually observed durig the iterval. Coclusios are draw based o the aalysis of the results ad the use of the selectio of models. Key Words: Software reliability, Maximum Likelihood Method, Nohomogeeous Poisso process 1. Itroductio Software has ow become a itegral part i idustrial, fiacial, commercial, health, defese ad real time safety critical systems. I a 2002 press release the Natioal Istitute of Sciece ad Techology (NIST) estimated that 80% of developmet costs ivolve idetifyig ad correctig bugs. It is estimated that costs related to software failure amouted to over 70 millio US dollars. With the the icrease of complexity of software systems. maagig the quality of software i projects has ever bee more apparet The use of Software Relaibility Growth Models (SRGMs) ca address this maagemet problem by allowig the user to as accurately as possible predict the curret ad future reliability of the software o test. Research usig NHPP based software models have show that these models do perform creditably whe compared to expert opiio [1]. The problem that arise is that the estimates of these models however ted to either cosistetly overestimate or uderestimate the quality of the software [2]. I this research we propose the use of four commoly refereced NHPP models ad four mixture of models or Super models. This paper is a cotiuatio of the work doe by Keiller ad Miller [3]. Our reliability growth models o test iclude several of the well kow models i literature together with additioal models that we call 'Super models'. These Super models are based o a set of the usual reliability growth models plus a selectio criterio, which idetifies oe of the set to use for predictios at each poit of time; the selectio criterio is based o the 'quality-of-past predictio' measures. The object is to idetify the best models or approaches to be use durig the test phase. 2. Nohomogeeous Poisso Process A Poisso process is a coutig process characterized by its mea value fuctio; ad the cumulative umber of software failures up to time t, N(t), ca be described by the ohomogeeous Poisso process (NHPP). We also kow that the 'reliability growth process' ca be represeted as a coutig process {N(t),0 t} where {N(t) = max{i: T i t},0 t}, ad T i s are radom variables ad t i s are real scalars. The process is observed for t: 0 t t c where t c is the curret time ad (x 1,x 2,x 3, x c ) is a sequece of iterfailure times- x i = t i+1 t i. If we let N(t) be the umber of failures occurrig i a arbitrary iterval of time t, the for this coutig process {N(t),0 t} modeled by the NHPP, N(t) follows a Poisso distributio with parameter M(t). M(t) is called the mea value fuctio ad describes the expected cumulative umber of failures i (0,t). [3] Xie [4] ad Lyu[5] outlie the followig iformatio about the NHPP. The probability that N(t) is a give iteger is expressed by (M (t)) P(N(t) = ) = e -M (t), = 0,1,2,3,... (1)! The fuctio λ (t) which is called the istataeous

2 failure itesity is defied as P(N(t +Δt)- N(t) > 0) λ (t)= lim (2) Δ t-> 0+ Δ t If you are give λ (t), the mea fuctio M(t) = E{N(t)} satisfies t M (t)= λ(s)ds. 0 Reversig the process, if we kow M(t), the failure itesity at time t ca be obtaied as dm (t) λ (t)= (4) dt Also, the ohomogeeous Poisso processes have a idepedet Poisso umber of failures i disjoit itervals give by: P((N(t + s)- N(t)) = ) = e -(M(t+s)-M(t))! (3) (M (t+ s)- M (t)) 0 t, 0 s; =0,1,2,3, (5) By usig differet odecreasig fuctios M(t), we get differet NHPP models. The reliability fuctio at time t 0 is : -(M(t+t ) - M( t0)) 0 R(t t0) = e (6) = P(N(t 0 + t) N(t 0 )=0) The estimatio of the ukow parameters i M(t) is obtaied by usig either the method of maximum likelihood or the method of least squares. The likelihood fuctio for the NHPP model with mea value fuctio M(t) is the followig L( 1, 2,...,k) = k M(si)) i (M( si-1) - exp(m( si-1) - M(si)) = (7) i 1 i! where i deotes the umber of faults detected i time iterval ( si-1, si), ad 0<s 0 < s 1 < s 2 <.< s i, i 0, are the total moitored test times of the software. The parameters i M(t) are estimated by maximizig the likelihood fuctio. Takig the atural log of the likelihood fuctio, we use umerical algorithms to solve for the parameters [6]. 3. Software Reliability Models o Test 3.1 Base Models I this study the followig four parametric families of NHPPs, characterized by their mea fuctios: M1 Logarithmic M 1 (t) = log t 0< M2 Pareto M 2 (t) = t 0<,0< M3 Expoetial M 3 (t) = e - t 0 M4 Geeral Power M 4 (t) = ( t -1< are selected to represet oe group of models o test (the simple models). Model M1 represets the Musa-Okumoto model [7]; model M2 represets the Littlewood NHPP model [8]; ad model M3 represets the Goel-Okumoto model [9]. M4, the 'Geeral Power' curve arises aturally whe cosiderig order statistics of idepedet but oidetically expoetially distributed failure times. Musa ad Okumoto [7] have promoted the use of the NHPP models i software reliability growth modelig. Miller [10] also gives strog theoretical justificatio for usig the NHPP. 3.2 Super Models We cosider four Super models. Keiller ad Miller [3] defies a Super model as a set of parametric reliability growth models ad a selectio criterio; for a give software failure data set ad for a give time, the selectio criterio chooses the parametric model i the set that is to be used for makig predictios of future failure behavior. As time passes for a give data set, a give super model may chage its choice of parametric family to use for predictios. Our procedure for a Super model is as follows: usig maximum likelihood estimatio, we fit all four of the parametric models (M1-M4). Next, the selectio criterio picks oe parametric class based o the fitted models. The curret fitted model of the chose class is used for makig predictios at the curret time. We cosider four pure quality-of-predictio measures ( 8): the u-plot, the y-plot, the prequetial likelihood, ad the maximum likelihood values. Usig these criteria requires fittig each of the four NHPP models (M1-M4) after each failure ad calculatig the predictive distributio ad desity of the time util ext failure. To summarize, our four super models are all based o four NHPP models (M1-M4). The selectio criteria for the four super models are: M5 U-plot M6 Y-plot M7 Prequetial likelihood M8 Maximum likelihood The four additioal mixture of models were developed usig dyamic selectio amog models based o goodess-of-fit ad quality-of-predictio criteria such as

3 the u-plot, y-plot, prequetial likelihood ad the maximum likelihood values. Figure 3.1c (Data set TV2005) 3. Failure Data Collectio The failure data used i this research is take from the research by Almerig et. al [11]. The data is take from three software developmet projects for high-ed TV sets cotaiig several millio lies of code. The data sets related to the projects are referred to as TV2003, TV2004, ad TV2005. To give the reader a rough idea of the data, it is preseted i a aggregate form i Table 1. The total cumulative time for each data set is split ito 10 equal itervals ad the cumulative umber of failures occurrig up to each of the 10 elapsed poits is show. From this table, it is possible to costruct very rough plots of the reliability growth. Figures 3.1a -3.1c show the relatioship betwee the cumulative iterfailure times agaist the % test times (100% of the total test time of the software) of the tested data sets. Figure 3.1a (Data set TV2003) Table 1 Summary of Failure Data SYSTEM (# DEFECTS) 100 TV TV TV Performace Measures The performace measures are recapped from previous work [3 ]. 4.1 Quality of Predictio Figure 3.1b (Data set TV2004) Let x i+1 be give as the iterfailure time at time s i+1 (i.e. x i+1 = s i+1 - s i ). The first quality-of-predictio measure is the u- plot : it is well kow that U=F x (X) has a uiform distributio o the iterval [0,1]. Usig this fact, if i+1 ( ) is the true distributio of X i+1 the u i+ 1 = i+1 (x i+1 ) will be a observatio from U[0,1] distributio. Thus the empirical distributio formed from the u s should be closed to U[0,1]. If we observe o + failures, startig to make predictios after the o th failure, the plot of {(u o+i, i/(+1)), i = 1,2,3,.,} is the u-plot. The maximum deviatio of the u-plot from the idetity fuctio is a measure of quality-of- predictio. The secod measure of quality-of predictio is the yplot : if the predictive distributios are good the u s should look like a radom sequece of idepedet U[0,1] variables, ad log (1-u) s like expoetial radom variates. I this case, let y i = i j 1 log(1-u o+j ) j 1 log(1-u o+j i=1,2,3,., ad plot the pairs {y i,, i/(+1)), i= 1,2,3,..,}. If the predictive distributios are good, this plot should be close to the idetity fuctio. A quatitative measure

4 of the quality-of-predictio is the maximum deviatio betwee the y-plot ad the idetity fuctio. The third measures of quality-of-predictio is the prequetial likelihood: based o Dawid s geeralizatio of likelihood to a sequetial situatio, we have PL = i 1 o+i; where i+1 (s) = d i+1 (s) i 0 ds sum of the relative errors (RE), ad the sum of the relative errors squared (RESQ) are ormalized by averagig over the umber of predictios for the model. 5. Fidigs Table 2 shows the compariso rakig of the Base models (M1-M4) for each of the datasets o test usig the u-plot, y-plot, PL, ad ML performace measures. Tables 3a-3c show the Base models chose by Super Models 5-8 for each of the datasets o test. Table 2 Rakig of Base Models For compariso purposes, the best predictive system should have the largest prequetial likelihood. For a detailed discussio of these measures of quality-ofpredictio, see Brocklehurst [12]. These three measures give a dyamic real-time evaluatio of how well a give parametric model has doe predictig iterfailure times up to the preset. Brocklehurst [12] argues that it would seem logical to calculate the ext predictio usig the model which has performed best up to the preset o the particular software failure data set uder cosideratio. These three measures give a basis for makig this choice. 4.2 Goodess-of-Fit Measures The performace of the model is ivestigated for each data set to see how well it ca predict the umber of ew failures maifestig durig fiite future time itervals. The experimet is desiged as a four step process. For each data set D k, where k =1,..., 3: (1) The MLE's of the model parameters are determied based o the first d data poits of data set D k. (2) The fitted model is the used to predict the umber of failures i [s d, k,s,k ] where l sl,k xm,k ad x m,k is the mth data poit of m 1 data set D k. (3) The model is evaluated usig (i) predicted errors e = ˆ (ii) relative errors re ˆ (iii) absolute relative errors re (iv) relative error squared (re ) 2 where ˆ ( ) is the predicted (actual) umber of failures i [s 10,k,s,k ]. (4) For each data set, the aalysis (1)-(3) is performed for d = 10,..., k-1. The value te was selected arbitrarily but is large eough to eable meaigful estimatio ad small eough to allow for meaigful evaluatio of the procedure. The sum of the absolute relative errors (ARE), the DATASETS AND BASE MODELS ON TEST TV2003 TV2004 TV2005 M1 M2 M3 M4 M1 M2 M3 M4 M1 M2 M3 M4 U-PLOT Y-PLOT PL ML Table 3a Base Models Chose by Super-Models M5-M8 (SYSTEM ID: TV2003) SYSTEM ID: TV2003 U-PLOT:M5 M2 M2 M2 M2 M2 M2 M2 M2 M2 Y-PLOT:M6 M1 M1 M2 M2 M2 M1 M3 M3 M3 PL:M7 M4 M3 M1 M4 M4 M1 M3 M3 M3 ML:M8 M2 M4 M4 M3 M1 M3 M4 M4 M4 Table 3b Base Models Chose by Super-Models M5-M8 (SYSTEM ID: TV2004) SYSTEM ID: TV2004 U-PLOT M3 M3 M3 M4 M3 M3 M3 M3 M3 Y-PLOT M4 M3 M3 M4 M4 M3 M3 M3 M3 PL M4 M4 M4 M4 M4 M4 M3 M3 M3 ML M3 M3 M3 M3 M3 M3 M3 M3 M3 Table 3c Base Models Chose by Super-Model M5-M8 (SYSTEM ID: TV2005) SYSTEM ID: TV2005 U-PLOT M2 M3 M3 M1 M1 M1 M1 M1 M1 Y-PLOT M4 M4 M4 M4 M4 M4 M4 M2 M2 PL M4 M4 M4 M4 M4 M4 M4 M4 M4 ML M3 M3 M3 M2 M3 M3 M3 M3 M3

5 Tables 4a-4c show the rakig of models (RE, ARE, ad RESQ performace measures) for the datsets o test Table 4a Rakig of models M1-M8 (SYSTEM ID: TV2003) SYSTEM ID: TV2003 MODELS ON TEST RE ARE RESQ Table 4b Rakig of models M1-M8 (SYSTEM ID: TV2004) SYSTEM ID: TV2004 MODELS ON TEST RE ARE RESQ Table 4c Rakig of models M1-M8 (SYSTEM ID: TV2005) SYSTEM ID: TV2005 MODELS ON TEST RE ARE RESQ Coclusio I this experimet four NHPP models (M1-M4) ad four Super models (M5-M8) are ivestigated. The experimet uses 3 data sets. We oticed that there were o sigificat performace improvemets whe usig the Super models (M5-M8) compared to the Base models. We had expected oe of the simple models (M1-M4) to be truly a superior fittig model ad expected the model to be cosistetly chose by the Super models (M5-M8). This did ot happe. We expect to cotiue this study i a more cotrolled experimet based o Mote Carlo data ad usig model recalibratio methods[12].. 7. Ackowledgemet The research was supported by Grat # NRC REFERENCES [1] Keiller, P.A., ad Miller, D.R., O the use ad performace of software reliability growth models, i Reliability Egieerig ad System Safety, Special Issue o Software, Vol 32 Nos 1&2, 1991, [2] Keiller, P.A. ad Mazzuchi, T. A., Ivestigatig a Specific Class of Software Reliability Growth Models. Proceedigs of the Aual Reliability ad Maitaiability Symposium, 2002, [3] Keiller, P.A. ad Mazzuchi, T. A., Improvig the Predictability of the Musa-Okumoto Software Reliability Growth Model. The First Iteratioal Software Assurace Certificatio Coferece (ISACC 99),1999; B1-3. [4] Xie, M., Software Reliability modellig, World Scietific, Sigapore,1991. [5] Lyu, M., Software Reliability Egieerig Hadbook, IEEE Computer Press, [6] Nelder, J. A. & Mead, R., A simplex method for fuctio miimizatio., Computer Joural, 7 (1965) [7] Musa, J. D. ad Okumoto, K., A Logarithmic Poisso executio time model for software reliability measuremet, IEEE Proceedigs of the 7th Iteratioal Coferece o Software Egieerig, 1984, [8] Abdala-Ghaly, A.A., Cha, P.Y. ad Littlewood, B., Evaluatio of competig reliability predictios, IEEE Trasactios o Software Egieerig, SE-12, 1986, [9] Goel, A. L. ad Okumoto, K., Time-depedet errordetectio rate model for software reliability ad other performace measures,ieee Trasactios o Reliability, R-28, 1979, [10] Miller, D. R., Expoetial order statistic models of software reliability growth, CR-3909, Natioal Aeroautics ad Space Admiistratio, July 1985, (Abridged versio: IEEE Trasactio o Software Egieerig, SE-12, 1986, 12-24). [11] Vicet Almerig et al., Usig Software Reliability Growth Models i Practice, IEEE Software November/December 2007 (vol. 24 o.6) pp [12] Brocklehurst, S., Cha, P.Y., Littlewood, B., ad Shell, J., Recalibratig Software Reliability Models, IEEE Trasactio of Software Egieerig, Vol. 16, No. 4, 1990