Parameter Estimation of Hyper-Geometric Distribution Software Reliability Growth Model by Genetic Algorithms

Transcription

1 Parameter Estimation of Hyper-Geometric Distribution Software Reliability Growth Model by Genetic Algorithms Takashi MINOHARA Department of Computer Science Takushoku University, Tokyo,, Japan minohara@cs. t akushoku-u. ac. jp Yoshihiro TOHMA Department of Information and Communication Engineering Tokyo Denki University, Tokyo, Japan tohma@c.dendai.ac.jp Abstract Usually, parameters in software reliability growth models are not known, and they must be estimated by using observed failure data. Several estimation methods have been proposed, but most of them have restrictions such as the existence of derivatives on evaluation functions. On the other hand, genetic algorithms(ga) provide us with robust optimization methods in many fields. In this paper, we apply GA to the parameter estimation of hyper-geometric distribution software reliability growth model. Experimental result shows that GA is eflective in the parameter estimation and removes restrictions from software reliability growth models Introduction At the test-and-debug phase of the software development, software reliability models are used to estimate the software quality measures. One of the important measures is the number of faults residual at the beginning of the test-and-debug phase. It is useful for deciding reasonably when the program can be released to service. Several software reliability growth models( SRGMs) are proposed [I] [] [] [] to estimate the number of residual software faults. These models have unknown parameters, and they must be estimated by using observed failure data. Usually calculus-based estimation methods such as maximum-likelihood and least square sum are applied to the parameter estimation. However they impose restrictions to SRGMs in most cases. To maximize the likelihood function or minimize the square sum error function, for example, the continuity and the existence of derivatives of the evaluation function are required. Genetic algorithms(ga) are searching techniques based on the mechanism of natural evolution of species, and have been used for solving optimization problems in many fields [][]. Fundamentally, GA are free from restrictive assumptions on the continuity, the existence of derivatives, the unimodality, and etc. It looks very attractive to use GA for estimating the parameter of SRGMs. In this paper, we apply GA to the parameter estimation of the hyper-geometric distribution software reliability growth model (HGDM) [l] []. There are criticisms against the sensitivity function w(i) of the HGDM such that w(i) + when i -+. By using GA for the parameter estimation, we can take into consideration the boundary of w(i) within a finite range of i together with the fitness of the estimated parameters. Hyper - geometric distribution software reliability growth model In this section, we briefly review the HGDM. At the beginning of the test-and-debug phase, m initial faults are resident in the program. Test operations performed for a unit scale (an hour, a day, a week,...) are called test instances, and they are denoted by t(i), i =,,.. e. The sensitivity factor, w(i) represents how many faults are detected by the application of the test instance t(i). Since some of the faults detected by t(i) may have been detected previously, the number of faults newly discovered by t(i) is not necessarily equal to w(i). Considering the application of a test instance t(i), let N(i) be the number of faults newly discovered by t(i) and C(i - ) be the number of faults already discovered by t(l), t(), -, t(i - ). With the assumption such that new faults will not be inserted in the program when king is performed and w(z) faults which responds to t(i) are those taken rather randomly out of m initial faults, we can formulate the probability Prob( x(i) I m, w(i), C(i-)) that N(i) is equal to x(i) as follows. -/ $. IEEE

2 Prob( x(i) I m, w(i), C(i - ) ) = m!. '- C(i - ) -- ( ;) ) ( z) ; xl : : w; i- (G> where C(i - ) = cx(k),c(o) =,and z(k) is an k=l observed instance of N(k). The expected value of N(i) denoted by N(i) is.- N(i) = {m.- C(i - I)}- m In order to predict the growth of C(i) in advance, we estimate C(i:) by the following E[C(i)] [] []. (Hereafter, the estimate for a p will be generally denoted by Ebl of evaluation for each chromosome. Then, some operations will be performed on these selected candidates to produce new population of the next generation. This process will be repeated until a population satisfactory in terms of the evaluation measure is obtained. GA can be applied to solve practical problems by using a way of encoding potential solutions in a chromosome and an appropriate function of evaluating fitness of a chromosome to the required target of the problem. The actual process of applying GA is as follows: Initialize a population of chromosomes. Evaluate the fitness function of each chromosome in the population. Produce probabilistically the next population of chromosomes from current population by applying the genetic operators. If time is over, stop and return the best chromosome; otherwise, go to. We describe how genetic operations can be used in our parameter estimation hereafter.. Population of chromosomes The value of sensitivity factor w(i) and the number of initial fau.t~ m are not known, and therefore they must be estimated. Various functions of w(i) have been proposed as given in Table [] (. Parameters to be Table : Various w(i) fiinctions of the HGDMs. A population is a constant number Np of chromosomes. We use bit strings to encode chromosomes and assigned bits each to the parameters, E[m], E[a], E[b], and EbLT], respectively( Fig. ). Initial chro- lo.....*~ oo**** **.* estimated are m, a, b, and PLT, respectively. Note that the number of parameters in any functions is at most four. Parameter estimation by genetic algorit hms GA simulate the progress of generation of a species, chromosomes here. It deals with a group of chromosomes, called a population, of a generation. From the population of the current generation, GA produce new population of the next generation. That is, GA select probabilistically candidate chromosomes from the current population according to an appropriate measure. Fitness function A chromosome gives a set of parameter values. We evaluate the chromosome by the errors between the observed and the estimated test-and-debug data. Two types of the errors can be considered, the difference in the number of newly discovered faults and the difference in the cumulative number of faults. Thus, the sum of the squares of these two types of the errors are:

3 EFI n Selection Crossover & Mutation ( Chromosome Chromosome Chromosome ' i= Chromosome M /"I n Chromosome Chromosome Chromosome l-gzgz- fk where n is the total number of test instances. Fundamentally, genetic algorithms are methods to get the maxim of some fitness function, while our object is to minimize the errors. We use the linear scaling[] to map the error function EF to a fitness function FF: FF = QI. EF +/ () Coefficient QI and / are calculated for every generation of chromosomes by the following principle.. The minimum fitness FF,;, must not be negative.. The average fitness FF,,, is equal to the average error EF,,,.. If the above rules are satisfied, the maximum fitness FFmaZ: is twice as much as the average fitness FF,,,, otherwise FF,,, is scaled as much as possible. When we use Eq.(a) or Eq.(d) in Table as sensitivity function w(i), we check the value of w(i). If w(z) of a chromosome exceeds the boundary [, m], the fitness value of the chromosome is set to zero so that it never be selected as a parent of next generation. This boundary check is performed at a test instance where almost all faults are discovered. To get the value of z for such a test instance, the approximate equation Eq.() is used with estimated parameters. For example, considering.% of initial faults, we can obtain the value of i by solving the following equation Eq.().. Genetic operators We use the conventional genetic operators to produce new generation of population of chromosomes from current generation. The progress of generations is made by two stages as shown in Fig.. In the first stage, parents from which the next generation will be emerged are probabilistically selected from current generation. We use the roulette wheel selection technique where any chromosome j is selected in proportion to their fitness FFj. In the second stage, selected parents are mated to produce their offsprings. We apply the uniform crossover to mated pairs with probability PCTOSS. Fig. illustrates the uniform crossover which produces Chromosome Np Generation (t) IChromosome Np'l Generation (t+l) Figure : Generation of chromosomes ParentA U I Mask(random bit pattern) I ChildA I.:O:-: :.b:.:.:o:.: Child B.;.:.:.::.:. ):(.;: ::!::... Figure : Uniform crossover Before mutation After mutation flipped Figure : Bit mutation two children by randomly selecting which parent contributes its bit value to which child. If crossover is not applied, the children is left as exact copies of their parents. In some cases the one-point crossover is used. However, there is a schema theory such that the onepoint crossover cannot combine some desirable properties encoded on chromosomes in the current generation to produce superior offsprings.[]. We used the uniform crossover, because it can avoid the above drawback. We tried the one-point crossover but it gave poorer performance than the uniform crossover in our experiments. We also apply the bit mutation. After the crossover, the bit mutation flips the value of a randomly selected bit of the chromosome with probability Pmutate as shown in Fig.. In addition to the above genetic operation, we take the elitist policy so that the best fit chromosome in current generation can survive(make a copy of itself) in the next generation. Production of new generations is iterated a constant N,,, times, and then the best fit chromosome in the last generation gives the result of estimation.

4 Experimental result on pseudo data We made experimental evaluations, applying our method of GA to pseudo data sets. The pseudo data sets were generated by computer simulation so that they follow the HGDM with m and w(i) given a priori. We generated ten pseudo data sets for each of sensitivity functions Eq.(a), Eq.(b), and Eq.(c) of Table. People may be interested in how many data obtained during earlier stage from the beginning of testand-debug phase can give satisfactory estimates for parameters. Therefore, we carried out the experiment of estimation, changing a of the number of earlier pseudo data generated from the beginning. Table sliows the condition of genetic operation. We also tried other different GA conditions, but no significant improvement of the performance was obtained, and the GA operation was not sensitive to the selection of condition values in our experiments. Because of the probabilistic nature of GA, parameter values estimated by GA may differ in some cases, even though the same pseudo data set with the same data are taken. Therefore, we conducted the experiment ten times for each data and took the average over these experiments ( times for each generated pseudo data sets). We also calculated the standard deviation. - Table and shows E[m] estimated by EF and EF, respectively. The pseudo data are generated for w(i) of Eq.(a) in Table. Parameter values given a - priori are m ==,a =.,b =. In these tables E[m] denotes the average value of E[m] and c(e[m]) denotes the standard deviation of E[m]. For the purpose of comparison, the calculus-based estimation, that is the combination of the linear multiple regression analysis and Newton's method[l], was made to the same data -- used in Table and. Table shows the estimated E[m]. For one of the ten generated pseudo data, the estimation was failed because of floating-point exception, when the % data was taken. From Table and we see that the parameter estimation by GA shows a satisfactory performance. Note that, in many cases, errors in the fourth column are smaller than those in the fifth column of these table. Although tlhe error values in the estimation by GA are as small as in the case of the calculus-based method, - Chromosomes in population Max&" generation IW'utation - rate Crossover rate Np =lo N,,, =ZOO Pmutate=. PC,,,, =a Table : Estimation of E[m] by EF using the pseudo data for w(i) of Eq.(a) data I EF for EF estimated for given % % % % % % % % % % data % % % % % % % % % % E[m] a(e[m]) p ~ param'. ~ Table : Estimation of E[m] by EF using the pseudo data for w(i) of Eq.(a) EF for estimated param... EF for given param.. Table : Estimation of E[m] by the calculus-method using the pseudo data for w(i) of Eq.(a) data I % % % % % % % % % % EF for estimated param. A EF for estimated param ~ the standard deviation of parameter values estimated by GA are larger than that by calculus-based method. This difference may be attributed to the way of choosing initial parameter values in the two methods. While GA starts its estimation with randomly selected multiple initial points, the calculus-based method begins with the last C[i] of the data and find a optimal point

5 Table : the average of the last C[i] for each data data I % % % % % Clil I.....,, data Clil % % % % %..... near start point. Table shows the average of the last C[i] for each data. When the last C[i] is far from the initial number of faults m, the calculus-based method may fail its estimation. From Table, and, note that when only the data obtained during very early stage of test-and-debug phase are used, GA gives better results than those by the calculus-based method. We speculate that GA can avoid to fall into local minimum conditions by the nature of search with multiple initial points and the probabilistic way of selection, while the calculus-based method may be more sensitive to the initial condition with which the calculation begins. Computation time depends on the number of chromosomes Np, the number of progress of generation N,,, and the number of test instances. Each experimental results have been obtained within a few seconds by a workstation of loomhz RSC(SONY NWS- UA). It is rather comparable with the calculusbased estimation method. Application to real observed data We applied our method of GA to two sets of real test-and-debug data. Table shows the estimated number of initial faults E[m] for the real test-anddebug data No. in Table. From the result of % data, we see the performance of GA rather similar to the case of the pseudo data sets. In this case: the calculus-based method failed to get the value of E[m]. On the other hand, when % data were used, EF gives worse result than that obtained by the calculus-based method. However, the value of EF calculated with the estimated parameters by the calculus-based method(.) is larger than that by GA(.). Therefore, we cannot say readily that the method by GA is worse than the calculus-based one. This worse result by GA might be caused by the natures of data observed. 'This error is square sum of the difference between theoretical value of N(i) (Eq.() ) and probabilistically generated pseudo data. This column is to show the fluctuation of probabilistically generated pseudo data. This error is square sum of the difference between theoretical value of C(i) and probabilistically generated pseudo data. Marked values are calculated over the successful estimations Table : Estimation of E[m] for the real data. data % % % % % % % % % % EF for w(i) of Eq.(d) EF for w(i) of Eq.(d) calculusbased method FAILED Table : Estimation of E[m] for the real data. data % % % % % % % % % % EF for w(i) of Eq.(d) EF for w(i) of Eq. (d) calculusbased method..... FAILED.... Table shows the estimated initial number of faults E[m] for the real test-and-debug data No. in Table. Note again that when % of data were used, the calculus-based method failed. When all data were used in Table and Table, the both methods of GA and the calculus-based gave almost the same estimates. Conclusion In this paper, we have proposed a new parameter estimation method for the hyper-geometric distribution software reliability growth model using genetic algorithms. Compared to the calculus-based method, it imposes little restriction on evaluation function in making the estimation. Further, the validity check for the sensitivity function, w(i), can be incorporated into the evaluation of fitness. The experiment showed that the proposed method gave the satisfactory result. Note that when only earlier % pseudo data were used, the estimate E[m] obtained by GA is closer to the true value than that obtained by the calculus-based method. Further, GA give an estimate even in the case, where the calculus-based method fails to estimate the parameters. These results suggest that GA approach may be a more stable method to get the estimates.

6 qtj-?& jhl rest E(;) : Real u(i) : Nun.ea - st- g e itan(?tect faults observed!r of test workers involved a iq,bsl - red q - We have applied GA only to the HGDM in this paper. However, the proposed method could be applied to the other software reliability growth models such as the non-homogeneous poisson process model. Currently, no all-round software reliability growth model exists and one needs to select a suitable model among many candidates. By encoding parameters of different models in a chromosomes, multiple models could be concurrently evaluated and the best fit model could be selected. This approach may be a research issue in the future. References [I] Y. Tohma, K. Tokunaga, S. Nagase, and Y. Murata, Structural approach to the estimation of the number of residual software faults base on the hyper-geometric distribution, IEEE Transactions on Software Engineering, vol. SJ-, no., pp. -, Mar.. [] A. L. Goel and K. Okumoto, Time-dependent error detection rate model for software relibility and other performance measures, IEEE Trans. Reliability, vol. R-, no., pp. -, Aug.. [] S. Yamada, M. Ohba, and S. Osaki, S-shaped software reliability growth models and their applications, Tat: rest E(i) Real.(ij : Num : Real test- - Stan - e. - % - eteci faults observed?r of test workers involved,ta. ob: LOO LO LO LO LO iq.vec q IEEE Trans. Reliability, vol. R-, no., pp. -, Oct.. [] J. D. Musa, A. Iannino, and K. Okumoto, Software Reliability: Measurement, Prediction, Application, McGraw-Hill,. [] D. E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley,. [] L. Davis, Ed., Handbook of Genetic Algorighms, Van Nostrand Reinhold,. [] Y. Tohma, R. Jacoby, Y. Murata, and M. Yamamoto, Hyper-geometric distribution model to estimate the number of residual software faults, in COMPSA C, Sept., pp. -. [] R. Jacoby and Y. Tohma, The hyper-geometric distribution software reliability growth model (HGDM) : Precise formulation and applicability, in COMP- SAC, Oct., pp. -. [] R. Hou, S. Kuo, and Y. Chang, Applying various learning curves to hyper-geometric distribution software reliability growth model, in ISSRE,. [lo] Y., Tohma and Y. Matsunaga, Application of hypergeometric distribution model to the hardware debugging process, Computer Systems Science Fd Engineering, vol., no., pp. -, Jan..