Software Reliability Moeling an Cost Estimation Incorporating esting-effort an Efficiency Chin-Yu Huang, Jung-Hua Lo, Sy-Yen Kuo, an Michael R. Lyu -+ Department of Electrical Engineering Computer Science & Engineering Department National aiwan University he Chinese University of Hong Kong aipei, aiwan Shatin, Hong Kong sykuo@cc.ee.ntu.eu.tw lyu@cse.cuhk.eu.hk Abstract Many stuies have been performe on the subject of software reliability, but few explicitly consier the impact of software testing on the reliability process. his paper presents two important issues on software reliability moeling an software reliability economics: testing effort an efficiency. First, we will iscuss on how to exten the logistic testing-effort function into a general form. he generalize logistic testing-effort function has the avantage of relating the work profile more irectly to the natural flow of software evelopment. herefore, it can be use to escribe the actual consumption of resources uring software evelopment process an get a conspicuous improvement in moeling testing-effort expenitures. Furthermore, we will incorporate the generalize logistic testing-effort function into software reliability moeling an its fault-preiction capability is evaluate through four numerical experiments on real ata. hen, we will aress the effects of automate techniques or tools on increasing the efficiency of software testing. New testing techniques will usually increase test coverage. We propose a moifie software reliability cost moel to reflect these effects. From the simulation results, we obtain a powerful software economic policy which clearly inicates the benefits of applying new automate testing techniques an tools uring software evelopment process.. Introuction When computer applications permeate our aily life, reliability becomes a very important characteristic of a computer system. In moern society, computer-controlle an computer-embee systems heavily epen on the Sy-Yen Kuo is with A& Labs-Research, New Jersey. He is on leave from National aiwan University, aipei, aiwan. correct performance of software. Software reliability is one of the most important features for a critical system which can affect human s life. herefore, it is necessary to measure an control the reliability of a software system. A number of Software Reliability Growth Moels (SRGMs) have been propose [, 6]. Among these moels, Goel an Okumoto consiere an NHPP as the stochastic process to escribe the fault process []. Yamaa et al. [-] moifie the G-O moel an incorporate the concept of testing-effort in an NHPP moel to get a better escription of the software fault phenomenon. Later, we [7-8] also propose a new software reliability growth moel with the logistic testing-effort function. In this paper, we exten the logistic testing-effort function to a generalize form. he generalize logistic testing-effort function has the avantage of relating a work profile more irectly to the natural structure of the software evelopment. herefore, it can be use to pertinently escribe the resource consumption uring the software evelopment process an get a conspicuous improvement in moeling the istribution of testing-effort expenitures. In general, we will have more confience in the measure software reliability with more software tests. Unfortunately, testing with ineffective or reunant test cases may lea to excessive cost. o avoi such phenomenon, we nee to know when to stop testing. One alternative is to restrict the test ata such that testing will stop when the os of etecting aitional faults (estimate by SRGMs) are very low. But this may not be realistic since testers typically want to test for all possible valuable failure ata, even the cost of testing is significant. Okumoto an Goel [] first iscusse the software optimal release policy from the cost-benefit viewpoint an propose a software reliability cost moel. It was shown that the optimal software release time can be obtaine base on a cost criterion when minimizing the total expecte cost. Recently, many papers iscusse such optimal software release time problem base on the cost-reliability relationship [4-6, 8-,, 8-9,, 4]. In
fact, to etect aitional faults uring the test phase of a software evelopment process, the testers or ebuggers may use some new automate tools or methos that are just iscovere an become available. hese tools, techniques or methos can greatly help the evelopers an testers to create tests an eliminate some reunant test cases. As time progresses, they can etect aitional faults uring testing, which saves the greater expense of correcting faults uring the operational phase. hese approaches have improve software testing an prouctivity recently, allowing project managers to maximize software reliability. Hence the extra cost trae-off base on new techniques an tools can be consiere in software reliability cost moel an viewe as the investment require to improve long-term competitiveness an to spee up the software prouct release in the commercial market. In this paper, we propose a new reliability cost moel that provies a means of assessing whether the software cost is uner control an the software quality is improving with time. he methos we propose allow the software testers an software quality assurance (SQA) engineers to ecie when the software is likely to be of aequate quality for release.. Relationship between SRGM an testing-effort function In this section we propose a set of generalize software reliability growth moels incorporating testing-effort functions. he mathematical relationship between reliability moels an testing effort expenitures is explicitly escribe in etail. Numerical results are given to illustrate the avantage of this new approach.. Software reliability moeling escriptions.. Review of SRGM with Logistic testing-effort function A typical software reliability moel is base on the following assumptions []:. he fault removal process is moele by a Non Homogeneous Poisson Process (NHPP).. he software system is subject to failures at ranom times cause by manifestation of remaining faults in the system.. he mean number of faults etecte in the time interval (t, t+ t] to the current testing-effort is proportional to the mean number of remaining faults in the system at time t. 4. he proportionality is a constant over time. 5. esting effort expenitures are escribe by a Logistic testing-effort function. 6. Each time a failure occurs, the fault that cause it is immeiately remove an no new faults are introuce. Base on the thir assumption, we obtain the following ifferential equation: m( r [ a m( ] () t Solving the above ifferential equation uner the bounary conition m()= (i.e., the mean value function m( must be equal to zero at time ), we have m(=a(-exp[-r(w(-w())])=a(-exp[-rw (]) () where m( is the expecte mean number of faults etecte in time (, t], is the current testing-effort consumption at time t, a is the expecte number of initial faults, an r> is the error etection rate per unit testing-effort at time t. Eq. () is an NHPP moel with mean value function consiering the testing-effort consumption. From the above escription, we know that represents the current testing-effort consumption (such asvolume of test cases, human power, CPU time, an so on) at time t uring the software testing/ebugging phase. he consume testing-effort can inicate how effective the faults are etecte in the software. herefore, this function plays an important role in moeling software reliability an it can be escribe by ifferent istributions. From the stuies in [-5, 4], several testing-effort pattern expressions, such as Exponential, Rayleigh, an Weibull-type curves, can be applie. Moreover, we [7-8] propose a Logistic testing-effort function to escribe the possible test effort patterns, in which the current testing-effort consumption is NA exp[ t] NA () Aexp[ t] t t (exp[ ] Aexp[ ]) where N is the total amount of testing effort to be eventually consume, isthe consumption rate of testing-effort expenitures, an A is a constant. he cumulative testing effort consumption of Logistic testing-effort function in time (, t] is W ( N A exp[ t] (4) an t W ( ) (5) Besies,the testing effort reaches its maximum value at time ln A tmax (6)
.. A generalize Logistic testing-effort function From the previous stuies in [7-8], we know that the Logistic testing-effort function (i.e. the Parr moel [4]) is base on a escription of the actual software evelopment process an can be use to escribe the work profile of software evelopment. In aition, this function can be use to consier an evaluate the effects of possible improvements on software evelopment methoology, such as top-own esign or stepwise refinement. herefore, if we relax some assumptions when eriving the original Parr moel an take into account the structure evelopment effort, we obtain a generalize Logistic testing-effort function as: / ( ( ) / ) W t N (7) t Ae where is a structuring inex with a large value for moeling well-structure software evelopment efforts, an is a constant. If =, the above equation becomes N W ( t Ae (8) If is viewe as a normalize constant an the above equation is reuce to Eq. (4). Similarly, if =, we have N W ( t Ae (9) Similarly, if we set, we get a more generalize an plain solution for escribing the cumulative testing effort consumption in time (, t]: N W ( () t Ae In this case,the testing effort reaches its maximum value at time A t max ln( ) / (). Numerical examples.. Numerical example he first ata set is from Ohba [7] where the testing time is measure in CPU hours. he Maximum Likelihoo Estimation an Least Squares Estimation are use to estimate the parameters of Eq. (), Eq. (4), an Eq. (), an we substitute the calculate normalizing value for. he estimate values of parameters for the generalize logistic testing-effort function are liste in able. From able, =.66 is the real estimate value for the first ata set an the other possible values of are pre-calculate. Figure epicts the fitting of the estimate current testing effort by using generalize logistic testing-effort function, in which we fin that the peak work rate occurs when about half of the work on the project has been one. his phenomenon can be interprete as that in a well-structure software evelopment environment, the slope of the testing-effort consumption curve may grow slowly initially, but a compensating reuction will happen later. able shows the estimate values of parameters by using ifferent SRGMs an two comparison criteria, Accuracy of Estimation (AE) an Mean of Square Fitting Faults (MSF) [7-8]. he smaller MSF an AE inicate fewer number of fitting faults an better performance. From able, we know that when the value of varies from to, both MSF an AE will be less than other existing SRGMs; therefore, it is conceivable that the propose moel has a better gooness-of-fit. able : Parameters of generalize logistic testing-effort function for the first ata set. N A 54.864.4.67 5.7 4.64.8889.5 5.78 5.8.7 49. 6..5876.5 48.7768 49.67.584.66 48.68 8.5.544 48.69 88..444.5 47.857 579.9.99 4 47.656 489..657 4.5 esting Effort(CPU Hours) 4 K= K= K=.66 K= Actual.5 5 7.5.5 5 7.5 ime(weeks) Figure : Observe/estimate testing-effort vs. time for the first ata set. able : Comparison results for the first ata set. Moel a r AE(%) MSF
Propose 94.76.47.6 8.9 Moel (=) (=.5) 84.77.457 7.46 4. (=) 77.57.4785 5.5.4 (=.6) 69.9.5955.8.7 (=) 67.89.595.75 5.9 (=.5) 4.87.998 5. 8.76 (=4) 44.46.986 5.76 889. (=4.5) 46.4.97 6. 95.8 G-O Moel 76..68.9 9.8 G-O with 565.5.9659 57.9.9 Weibull fun G-O with Rayl. Fun. 459.8.76 8. 68.4 G-O with 88.5.78.5 4.66 Exp. fun. Inflection S Moel 89..9549 8.69.5 Delaye S Moel 74.5.9765 4.48 68.67 Exp. Moel 455.7.676 7.9 6.9 Delaye S.8.45 6.9 798.49 Moel with Ray. fun. S-Shape Moel with logistic fun. 8.6.4 5.54 4.79 HGDM 87.7 NA 8. 8. HGDM with 87.79 NA 8. 8. linear factor HGDM with 85. NA 7.56.4 Exp. factor Musa Log. Poisson NA 7... Numerical example he secon ata set is cite from Musa et al. [4-5]. he software were teste for weeks (5. CPU Hours were use) an 6 faults were etecte. he Maximum Likelihoo Estimation an Least Squares Estimation are use to estimate the parameters of the Eq. (), Eq. (4), an Eq. () an we substitute the calculate normalizing value for. he estimate values for the parameters are liste in able. In fact, from able, =.77 is the real estimate value for the secon ata set an other possible values of are pre-calculate. Figure epicts the fitting of the estimate current testing effort by using generalize logistic testing-effort function. able 4 shows the estimate values of parameters an the comparison results between the observe an the estimate values obtaine by the other SRGMs. Similarly, smaller AE an MSF inicate less fitting errors an better performance. We fin that when the value of varies from.5 to 4.5, both MSF an AE will be less than other existing SRGMs. Hence, we still can conclue that the propose moel is goo enough to give a more accurate escription of resource consumption uring the software evelopment phase an gives a better fit in this experiment. able : Parameters of generalize logistic testing-effort function for the secon ata set. N A 9.95 464.89.4955 8.5 9.9.4447.77 8.5 4584.8.977.5 8.458 655.7 8.464 7845.574 7.566 597.65.5 7. 7489.957 4 6.5 9559.8695 4.5 esting Effort(CPU Hours).5.5.5.5 K= K=.77 K= Actual 5 5 ime(weeks) Figure : Observe/estimate testing-effort vs. time for the secon ata set. able 4: Comparison results for the secon ata set. Moel a r AE (%) MSF Propose Moel (=) 8.6.4598 6.58 6.4 ( =.77) 4..796 5.5 5.79 ( =.5) 9.9.459 5.96 5.4 ( =) 4.55.46 4.9 7.6 ( =) 47.88.7.7 9.7 ( =.5) 54.44.8975 8. 8.58 ( =4) 6.5.7747.7.8 ( =4.5) 68.65.76.49 6.58 G-O Moel 4..46 4.9 48. G-O with 866.94.964 6. 89.4 Rayleigh fun. Exp. Moel 7..56 7. 9.66 Delay S-shape Moel 7.96.9644 6.6 45.5
Delaye S 688.59.976 66.7 5.9 with Exp. fun. Delaye S with Logistic function. 7.49.6 6.86 7.7.. Numerical example he thir set of real ata is the pattern of iscovery of faults in the software that supporte Space Shuttle flights SS, SS, SS4 at the Johnson Space Center []. he system is also a real-time comman an control application. A weekly summary of software test hours an the faults of various severity iscovere is given in []. he cumulative number of iscovere faults up to thirty-eight weeks is 7. Similarly, the Maximum Likelihoo Estimation an Least Squares Estimation are use to estimate the parameters of the Eq. (), Eq. (4), an Eq. (), an we substitute correct normalizing value for. he estimate values of parameters for the generalize logistic testing-effort function are liste in able 5. In fact, from able 5, =.56 is real estimate value for this ata set an the other possible values of are pre-calculate. Figure epicts the fitting of the estimate current testing effort by using generalize logistic testing-effort function. able 6 shows the estimate values of parameters by using ifferent SRGMs an the comparison criteria. herefore, the estimation results of iniviual moels show that the propose moel gives the better AE. able 5: Parameters of generalize logistic testing-effort function for the thir ata set. N A 88.88.557.98884 66. 8.74.96.56 664.54.765.848.5 57.8 8.66.744 57.67.44.68964.5 46. 5.5.65575 46.8 9..666.5 esting Effort(CPU Hours) vs. time for the thir ata set. able 6: Comparison results for the thir ata set. Moel a r AE (%) Propose Moel (=) 4.5.97 75.5 (=.56) 4.84.98 6.465 (=.5) 4.686.98 7.95 (=) 9.65.78 7.944 (=.5) 7.7.8 7.666 G-O Moel 597.887.9 78.87 G-O with Rayleigh 45.7.758 8.66 Function..4 Numerical example 4 he fourth set of real ata is the pattern of iscovery of faults by homa in []. he ebugging time an the number of etecte faults per ay are reporte. he cumulative number of iscovere faults up to twenty-two ays is 86 an the total consume ebugging time is 9 CPU hours. All ebugging ata are use in this experiment. Similarly, we can estimate each parameter by the Maximum Likelihoo Estimation an Least Squares Estimation in the propose SRGM an they are shown in able 7. In fact, from able 7, =.76 is real estimate value for this ata set an the other possible values of are pre-calculate. Figure 4 epicts the fitting of the estimate current testing effort by using generalize logistic testing-effort function. able 8 shows the estimate values of parameters by using ifferent SRGMs an the comparison criteria. herefore, in this ata set, we conclue that our propose moel gets a reasonable preiction in estimating the number of software faults an fits this ata set better than others. able 7: Parameters of generalize logistic testing-effort function for the fourth ata set. N A 99.98 8.9.5746 95.645 9.68.4856.5 95...556.76 94.9 89.6.9665 9. 6..8.5 8 6 4 K= K=.56 K= K= Actual ime(weeks) Figure : Observe/estimate testing-effort
esting Effort(CPU Hours) 5.5 7.5 5.5 K= K=.76 K= K=.5 Actual 5 5 ime(weeks) Figure 4: Observe/estimate testing-effort vs. time for the fourth ata set. able 8: Comparison results for the fourth ata set. Moel a r AE (%) Propose Moel 88.89.959 55.5 (=) (=.5) 88.699.8548.77 (=.76) 9.54.77 9.45 (=) 9.478.75 8.494 (=.5) 9.66.7478 8.564 G-O Moel 7.7.55445 5. HGDM 88..68. Optimal release time incorporating test efficiency In the last section we escribe a generalize approach to incorporate testing effort into software reliability moels. In this section we will ientify the efficiency of testing an stuy its impact on software reliability. In particular, we iscuss how to incorporate testing efficiency into reliability moels an how to etermine the optimal software release time.. Impact of new tools/techniques on software testing efficiency As soon as software coing is complete, the necessary but expensive testing phase starts. During the testing phase, the evelopers will nee to make a software reliability evaluation an etermine when to stop testing. If the results meet the requirement specifications an the reliability criteria are also satisfie, then the software prouct is reay for release. herefore, ajusting specific parameters in an SRGM an aopting the corresponing actions appropriately can help to achieve the goal of etermining the software release time. Several approaches can be applie. For example, we have iscusse the applications of testing-effort control an management problem in our previous stuy [7]. Using the propose methos, we can easily control the moifie consumption rate of testing-effort expenitures an etect more faults in a specifie time interval. his means that the evelopers an testers can evote their time an resource to complete their testing tasks base on well-controlle expenitures. In aition to controlling the testing-effort expenitures, we can achieve a given operational quality at a specifie time by introucing new automate testing tools an techniques. hat is, through the aoption of new testing techniques an tools, we can etect an remove more aitional faults (i.e. those faults that are not easily expose uring the testing phase). hese new methos, however, will impose extra software evelopment cost. For example, professional experts can help evelopers to assess the original software evelopment process, to meet their quality goals, an to reuce risks. In general, these external personnel can offer efficient an effective approaches to test planning, moule-level unit testing, or testing strategies. Moreover, many automate testing tools an techniques are available to increase test coverage an replace traitional manual testing. he benefits of applying new techniques an tools inclue increase software quality, reuce testing costs, improve release time to market, repeatable test steps, an improve testing prouctivity [5,, 5, ]. Consequently, it is esirable that these experts an automate testing tools/techniques can greatly help the evelopers in etecting aitional faults that are ifficult to fin uring regular testing an usage, in ientifying an correcting faults effectively, an in improving their software evelopment processes. An important step towar these new approaches, then, is to offer enough information about these approaches to software evelopers an reliability engineers. Before aopting the automate techniques an tools, we shoul get quantitative information from the inustrial ata relative to these methos past performance (i.e. the previous testing experience), or get qualitative information from the evaluation on the methos attributes. Basically, these methos past performance shoul be evaluate in etermining whether they will be successful in managing reliability growth []. In aition, they can be evaluate by performing various simulations base on actual ata sets. Finally, the test team s capacity in applying these techniques an tools an the relate operational profiles also play an important role. We iscuss how the software reliability moeling process can inclue these testing methos, an how a new optimal software release time problem can be formulate an solve.. Optimal software release time problem Okumoto an Goel [] first iscusse the software optimal release policy from the cost-benefit viewpoint. he total cost of testing-effort expenitures at time, C(), can be expresse as [-, 7, 9-,, 8-9, 4]:
C ( ) C m( ) C [ m( ) m( )] C LC x) x C ( ) C( ). his phenomenon usually occurs () infrequently, but if it can really shorten the testing perio where LC =software life-cycle length uner the same software reliability requirements, we may still consier applying the new techniques. By C =cost of correcting an error uring testing ifferentiating Eq. () with respect to the time we have: C =cost of correcting an error uring operation C =cost of testing per unit testing-effort expenitures. C( ) C ( ) C (( P) m( )) C From the work by B. Boehm [6], we know C >C as (( P) m( )) C ) (6) C is usually an orer of magnitue greater than C. In orer to etect aitional faults uring testing, the testers If we let Eq. (6) be equal to zero an use the mean value an ebuggers may use new automate tools or techniques. function in Eq. (), we can get a finite an unique solution he cost trae-off of these new tools an techniques, for the etermination of an optimal software release therefore, shoul be consiere in the software cost moel, time problem base on the new cost criterion. incluing their expenitures an benefits. Consequently, From Eq. (6), if we let C (+P)= C an C (+P)= C, we moify the overall software cost moel as follows [4]: then we have C( ) C ( ) C ( P) m( ) C [ m( ) ( P) LC C( ) C ( ) C m( ) C m( ) m( )] C x ) x () C ) where C () is the cost function for eveloping an acquiring the automate tools an techniques that etect an (7) aitional fraction P of faults uring testing. If the mean value function is given in Eq. (), we obtain We note that the cost for eveloping an acquiring new tools or techniques, C (), oes not have to be a constant uring the testing. Moreover, the testing cost for C () can be parameterize an estimate base on actual ata. From our experience, we foun that C () may have ifferent forms as time progresses, which epens on the characteristics of a tool s performance, testing effort expenitures, effectiveness, an so on. We can formulate this cost function as simple linear functions or simple non-linear functions. In general, the longer the software is teste, the more the testing cost C (). Uner the cost-benefit consierations, the automate tools or techniques will pay for themselves if C ( ) C( ) (4) hat is, m( ) C [ m( ) m( )] C x x C LC ) C ( ) C ( P) m( ) C [ m( LC ) ( P) m( )] C x) x Rearranging the above equation, we obtain C ) P m( ) ( C ) (5) ( C Eq. (5) is use to ecie whether the new automate tools or techniques are effective or not. If C () is low enough or if the new methos are effective in etecting aitional faults, this investment is worthwhile. Usually appropriate automate tools or techniques are best selecte epening on how thoroughly failure ata are collecte an faults are categorize [5]. Sometimes incorporating new automate tools an techniques into a software evelopment process may introuce excessive, that is, C( ) C ( ) C ar ) exp[ rw ( )] C ar ) exp[ rw ( )] C ) (8) Without loss of generally, we consier several possibilities for C () in orer to interpret the cost consumption: () C () is a constant. () C () is proportional to the testing-effort expenitures. () C () is exponentially relate to the testing-effort expenitures. A. C ()= C, s ; C ()=, < s C( ) ) [ ( C C ) ar exp[ r(( W ( ) W ( ))] C ] (9) Since > for, C ( ) if ( C C ) ar exp[ r( W ( ) W ())] C () he left-han sie in Eq. () is a monotonically ecreasing function of. Here we let s be the starting time of aopting new techniques/tools. If ( C s C, then C ) ar exp[ r( W ( ) W ())] ( C C ) ar exp[ r( W ( LC ) W ())] C for s < < LC. herefore, the optimal software release time = s since
C ( ) for s << LC. Similarly, if ( C s C an C ) ar exp[ r( W ( ) W ())] ( C C ) ar exp[ r( W ( LC ) W ())] C, there exists a finite an unique solution satisfying Eq. (). hat is, A ln minimizes C() () N where C C N ln ar r C A since C ( ) for s << an C( ) > for << LC. If ( C C ) ar exp[ r( W ( LC ) W ())] C, then ( C C ) ar exp[ r( W ( ) W ())] C for s << LC. herefore, the optimal software release time = LC since C ( ) for s < < LC. heorem : Assume C ()= C (constan, C >, C >, C >, C >, C >C, we have CASE ( C C ) ar exp[ r( W ( s ) W ())] C an ( C C ) ar exp[ r( W ( LC ) W ())] C : there exists a finite an unique solution satisfying Eq. () an the optimal software release time is =. CASE ( C C ) ar exp[ r( W ( s ) W ())] C : = s. CASE ( C C ) ar exp[ r( W ( LC ) W ())] C : = LC. B. C ( ) C C s t, s ; C ()=, < s where C is an nonnegative real number that inicates the basic cost of aopting new techniques/tools, an s is the start time of aopting new techniques/methos. C( ) C ) C ar ) exp[ rw ( )] C ar ) exp[ rw ( )] C ) w ) [( C C ) ar exp[ r(( W ( ) ())] ( W C C ] () Since > for, C ( ) if ( C C ) ar exp[ r( W ( ) W ())] C C () As the left-han sie in Eq. () is a monotonically ecreasing function of, therefore, if ( C s C C an C ) ar exp[ r( W ( ) W ())] ( C C ) ar exp[ r( W ( LC ) W ())] C C, there exists a finite an unique solution satisfying Eq. (). where A ln minimizes C() (4) N C C N ln ar r C C A heorem : Assume C ( ) C C s t, C,C >, C >, C >, C >, C >C, we have CASE ( C C ) ar exp[ r( W ( s) W ())] C C an ( C C ) ar exp[ r( W ( LC ) W ())] C + C : there exists a finite an unique solution satisfying Eq. () an the optimal software release time is =. CASE ( C C ) ar exp[ r( W ( s) W ())] C C : = s. CASE ( C C ) ar exp[ r( W ( LC) W ())] C C : = LC. C. < s, s C ( ) C ( C s m C ( ) C m ) ( C C s m ;C ()=, ar ) exp[ rw ( )] C ar ) exp[ rw ( )] C ) w [( C C ) ar exp[ rw ( )] C + ( t s m C m ( C ] Because > for, C ( ) if
P ( ) [( C C ) ar exp[ rw ( ] C m C ( m ] s C (5) he left-han sie in Eq. (5) is a monotonically ecreasing function of. herefore, if ( C C ) ar exp[ r( W ( s ) W ())] C an P( LC )<C, it means that there exists a finite an unique solution satisfying Eq. (5), which can be solve by numerical methos. It is note that C ( ) for s an C( ) for >. hus, = minimizes C() for < LC. Similarly, we can get the following theorem. heorem : m Assume C ( ) C ( C s, C, C >, C >, C >, C >, C >C, we have CASE ( C C ) ar exp[ r( W ( s ) W ())] C an P( LC )<C : there exists a finite an unique solution satisfying Eq. (5) an the optimal software release time is =. CASE ( C C ) ar exp[ r( W ( s ) W ())] C : = s. CASE P( LC )>C : = LC. D. C ( ) C C (exp[ m ( ) ] ) s w t t, s ; C ()=, < s. C( ) C m ) exp[ m t] C ar ) s exp[ rw ( )] C ar ) exp[ rw ( )] C ) w {( C C ) ar exp[ rw ( )] ( C C m exp[ m t] } s Since > for, C ( ) if ( ) ( C C ) ar exp[ rw ( ] C m exp[ m t] C s (6) he left-han sie in Eq. (6) is a monotonically ecreasing function of. herefore, if ( C C ) ar exp[ r( W ( s ) W ())] Cm C an Q( LC ) <C, it means that there exists a finite an unique solution satisfying Eq. (6), which can be solve by numerical methos [6]. It is note that C ( ) for s an C( ) > for >. hus, = minimizes C() for < LC. Similarly, we can get the following theorem. heorem 4: Assume C ( ) C C (exp[ m ( ) ] ) s w t t, C >, C >, C >, C >, C >, C >C, we have CASE ( C C ) ar exp[ r( W ( s) W ())] Cm C an Q( LC )<C : there exists a finite an unique solution satisfying Eq. (6) an the optimal software release time is =. CASE ( C C ) ar exp[ r( W ( s ) W ())] Cm C : = s. CASE Q( LC )>C, : = LC.. Numerical example We have consiere several ifferent cases of minimizing the software cost in which the new automate tools an techniques are introuce uring testing. Due to the limitation of space, we choose Eq. () as the testing-effort function for a software evelopment project. Other logistic testing-effort functions with ifferent values can be similarly applie base on the same proceure. From the previously estimate parameters for the first ata set in able, we get N=48.7768, A=49.67, =.584, =.66, a=69.9, r=.5955. We further set C =$, C =$ per error, C =$5 per error, C =$ per unit testing-effort expenitures, an LC = weeks. We will consier the following two types of cost function C (): m. ( ) C ( C C s. C ( ) C C (exp[ m ( ) ] ) s w t t Here we assume C =$, s =9, LC =, an m=, that is, C ( ) 9 t. From heorem, the relationship of the cost optimal release time with ifferent P is given in able 9. From able 9, we fin that if the P value is larger, the optimal release time is larger an the total expecte software cost is smaller. his reflects that when we have better testing performance, we can etect
more latent faults through aitional techniques an tools. herefore, we can shorten testing time an release software soon. Compare with the estimate values of traitional software cost moel (i.e. Eq. ()) where =4.88, C()=479.66, we can see that in able 9, same optimal release time is achieve when P=. (i.e., =4.89), then C()= 4.9. It means that the C() is smaller than C() with equal optimal release time; that is, the assumption C ( ) C( ) is satisfie. Besies, the Operational Quality Inex (OQI) is increase from 89.5% to 98.6% [7]. Similarly, the relationships of the optimal release time with various P values base on ifferent cost functions are shown in able -4. From these tables we conclue the following facts: ) As P increases, the optimal release time increases but the total expecte software cost C() ecreases. his is because we can etect more faults an reuce the cost of correcting faults uring operational phase. ) Uner the same P value an with ifferent cost m functions (such as ( ) C ( C or C s C ( ) C C (exp[ m ( ) ] ) s w t t ), the larger the cost function is, the smaller the optimal release time is. However, the ifference in estimating the total expecte software cost is insignificant. able 9: Relationship between the cost optimal release time, C(), an P base on the cost function () C 9 t P C() P C(). 9.78 5574.5.7.854 46.69..6 544.5.8.4464 445.94..887 554.74.9.7 49..4.67 594.77. 4.89 4.9.5.965 494.6. 6.6 969.6.6.9747 4774.4 able : Relationship between the cost optimal release time, C(), an P base on the cost function C () (. 9 P C() P C(). 9.465 5574.88.7 9.68 46.6. 9. 545.96.8 9.7589 446.84. 9.669 556.98.9 9.895 4..4 9.4 597.9..58 44.69.5 9.47 498.8..96 98.95.6 9.589 4779.57 able : Relationship between the cost optimal release time, C(), an P base on the cost function C () (exp[ 9 t] ) P C() P C()..447 5565.5.7. 46.4..589 545..8.68 444...844 544.4.9.747 479..4.96 58.7. 4.866 47.88.5.99 49.94. 4.68 956.67.6.69 476.8 able : Relationship between the cost optimal release time, C(), an P base on the cost function 9 C () (exp[. t] ) P C() P C()..8548 5565.97.7.878 46.5..59 545.7.8.994 444.68..77 545.9.9.6 48.96.4.84 584.98..49 4.8.5.5 494.5..54 96.5.6.664 476.96 able : Relationship between the cost optimal release time, C(), an P base on the cost function C () 5 (exp[ 9 t] ) P C() P C(). 9.775 557..7.58 46.99. 9.8669 54.6.8.9 4454.9. 9.957 55..9.479 494..4.46 59.6..5568 44..5.8 49.6..69 974.6.6. 477.84 able 4: Relationship between the cost optimal release time, C(), an P base on the cost function 9 C () 5 (exp[. t] ) P C() P C(). 9.545 557..7 9.997 466.49. 9.65 54.9.8. 4456.85. 9.684 554.54.9.594 497.7.4 9.7499 595...75 47.45.5 9.847 495.6..745 977.68.6 9.8779 4776.8 4. Summary an conclusions
In this paper we stuy the impact of software testing effort an efficiency on the moeling of software reliability, incluing the reliability measure an the cost for optimal release time. We propose a generalize logistic testing-effort function which relates work profile irectly to the natural flow of software evelopment. his function is use to escribe the actual consumption of resources uring software testing which provies more accurate information for reliability moeling purpose. We also escribe the effects of applying new tools an techniques for increase efficiency of software testing an stuie the relate optimal software release time problem from the cost-benefit viewpoint. New reliability problems are formulate to incorporate software testing effort an efficiency. Finally, numerical examples are provie to emonstrate these new approaches. 5. Acknowlegments We woul like to express our gratitue for the support of the National Science Council, aiwan, R.O.C., uner Grant NSC 87-PC-E--7. he work escribe in this paper was also partially supporte by a grant from the Research Grant Council of the Hong Kong Special Aministrative Region (Project No. CUHK44/99E). he authors are please to thank Professor Y. K. Malaiya for his many constructive an insightful suggestions for improving the etails of this manuscript. Besies, we also thank several anonymous referees for their critical review an comments. References [] S. Yamaa, J. Hishitani, an S. Osaki, "Software Reliability Growth Moel with Weibull esting Effort: A Moel an Application," IEEE rans. on Reliability, Vol. R-4, pp. -5, 99. [] S. Yamaa, H. Ohtera, an H. Narihisa, "Software Reliability Growth Moels with esting Effort", IEEE rans. on Reliability, vol. R-5, No., pp. 9-, April 986. [] S. Yamaa an S. Osaki, " Cost-Reliability Optimal Release Policies for Software Systems, IEEE rans. on Reliability, Vol. 4, No. 5, pp. 4-44, 985. [4] J. D. Musa, A. Iannino, an K. Okumoto (987). Software Reliability, Measurement, Preiction an Application. McGraw Hill. [5] J. D. Musa (999). Software Reliability Engineering: More Reliable Software, Faster Development an esting. McGraw-Hill. [6] M. E. Helaner, M. Zhao, an N. Ohisson, "Planning Moels for Software Reliability an Cost," IEEE rans. on Software Engineering, Vol. 4, No. 6, pp. 4-44, June 998. [7] C. Y. Huang, J. H. Lo an S. Y. Kuo, "A Pragmatic Stuy of Parametric Decomposition Moels for Estimating Software Reliability Growth," Proceeings of the 9th International Symposium on Software Reliability Engineering (ISSRE 98), pp. -, Nov. 4-7. 998, Paerborn, Germany. [8] C. Y. Huang, S. Y. Kuo an I. Y. Chen, "Analysis of a Software Reliability Growth Moel with Logistic esting-effort Function," Proceeings of the 8th International Symposium on Software Reliability Engineering (ISSRE 97), pp. 78-88, Nov. 997, Albuquerque, New Mexico. U.S.A. [9] R. H. Huo, S. Y. Kuo, an Y. P. Chang, "Optimal Release imes for Software Systems with Scheule Delivery ime Base on HGDM," IEEE rans. on Computers, Vol. 46, No., pp. 6-, Feb. 997. [] R. H. Huo, S. Y. Kuo, an Y. P. Chang, "Optimal Release Policy for Hyper-Geometric Distribution Software Reliability Growth Moel," IEEE rans. on Reliability, Vol. 45, No. 4, pp. 646-65, Dec. 996. [] K. Okumoto an A. L. Goel, "Optimum Release ime for Software Systems Base on Reliability an Cost Criteria, Journal of Systems an Software, Vol., pp. 5-8, 98. [] M. R. Lyu (996). Hanbook of Software Reliability Engineering. McGraw Hill. [] S. R. Dalal an C. L Mallows, "When Shoul One Stop esting Software, " Journal of the American Statistical Association, Vol. 8, No. 4, pp. 87-879, September 988. [4] F. N. Parr, "An Alternative to the Rayleigh Curve for Software Development Effort," IEEE rans. on Software Engineering, SE-6, pp. 9-96, 98. [5] M. Lipow, "Preiction of Software Failures, " Journal of Systems an Software, Vol., pp. 7-75, 979. [6] B. Boehm (98). Software Engineering Economics. Prentice-Hall, Englewoo Cliffs, NJ. [7] M. Ohba, " Software Reliability Analysis Moels, IBM J. Res. Develop., Vol. 8, No. 4, pp. 48-44, July 984. [8] Y. W. Leung, "Optimum Software Release ime with a Given Buget, Journal of Systems an Software, Vol. 7, pp. -4, 99. [9] S. Yamaa, H. Narihisa, an S. Osaki, " Optimum Release Policies for a Software System with a Scheule Delivery ime, Int. J. of Systems Science, Vol. 5, pp. 95-94, 984. [] J. Farquhar an A. Mosleh, "An Approach to Quantifying Reliability-Growth Effectiveness," Proceeings Annual Reliability an Maintainability Symposium, pp. 66-7, 995. [] S. S. Gokhale, M. R. Lyu, an K. S. rivei, "Software Reliability Analysis Incorporating Fault Detection an Debugging Activities, Proceeings of the 9th International Symposium on Software Reliability Engineering (ISSRE 98), pp. -, November 4-7 998, Paerborn, Germany. [] P. N. Misra, "software reliability analysis, IBM Systems Journal, Vol., No., pp. 6-79, 98. [] Y. ohma, R. Jacoby, Y. Murata, an M. Yamamoto, "Hyper-Geometric Distribution Moel to Estimate the Number of Resiual Software Faults, Proc. COMPSAC-89, Orlano, pp. 6-67, 989. [4] C. Y. Huang, S. Y. Kuo, an M. R. Lyu, " Optimal Software Release Policy Base on Cost, Reliability an esting Efficiency," he wenty-hir Annual International Computer Software an Applications Conference (COMPSAC 99), October 7-9, 999, Phoenix, Arizona,
U.S.A. (accepte for publication) [5] M. R. Lyu an A. Nikora, " Using Software Reliability Moels More Effectively, IEEE Software, pp. 4-5, July 99. [6] Xie, M., Software Reliability Moeling, Worl Scientific Publishing Company, 99.