Bayesia approach to reliability modellig for a probability of failure o demad parameter BÖRCSÖK J., SCHAEFER S. Departmet of Computer Architecture ad System Programmig Uiversity Kassel, Wilhelmshöher Allee 71, D-34121 Kassel GERMANY j.boercsoek@ui-kassel.de schaefer@ui-kassel.de www.rs.eecs.ui-kassel.de Abstract: The testig of safety-critical software systems yields ofte a small amout of errors. The reliability aalysis is therefore subject to restrictios, e.g. because of high variaces. The use of Bayesia statistics icorporates additioal data ito the reliability aalysis to overcome these restrictios. A model is developed that ca be used for a aalysis of software with low demad mode. The model is discussed ad evaluated. Key-Words: software reliability, bayesia reliability, reliability model, static aalysis, PFD 1 Itroductio 1.1 Motivatio I a ormal software-developmet process it is ecessary to calculate reliability metrics like failure rate, failure itesity, reliability or availability. For safety-related or safety-critical software it is madatory to calculate these parameters. Otherwise the software caot be released due to regulatios. These calculatios require data from the softwaredevelopmet process. The required data is obtaied i the test process as failure data, i.e. the failure occurreces ad times are recorded. The failure data is the used to compute the failure rate ad the depedet reliability parameters usig differet models. For safety-related software these parameters are based o scarce data, because the testig of safetyrelated software yields very few failures if the developmet process was carried out accordig to the safety stadards. Therefore the calculated reliability parameters ca have a high variace ad are ot trustworthy eough to be take ito accout. 1.2 Requirig additioal iformatio Cosequetly it is ecessary to acquire additioal iformatio, to reduce the statistical variace ad to make the reliability parameters more trustworthy. Extra data ca come from differet areas ad phases of the software developmet process. This ca iclude experiece ad parameters from past or similar projects or additioal iformatio directly gaied from the source code of the software, e.g. through formal methods a correctess proof ca be made. This is usually ot practicable for whole software systems, because the proof methods work oly uder certai circumstaces or with special restrictios [3]. The use of extra data from past or similar projects is also problematic, because small chages i the project ca lead to a very differet behaviour regardig reliability. The additioal iformatio that is used i this paper is derived from static aalysis. It is a procedure that ca automatically aalyse the sourcecode of the software, without actual executio, i regard to certai properties, e.g. security issues, deadlocks, memory leaks or value aalysis. Failures are the result of oe or more errors i a program state. A error is usual geerated from a variable that has a out of rage value, e.g. the Ariae V lauch failure was caused by a overflow of a program variable [5]. Value aalysis tries to detect the possibility of these kids of errors, so that variables ca be made safe, i.e. a overflow is made impossible through programmig techiques [1]. Not all program variables ca be made safe i this regard, either because of performace issues, additioal checks cost processig time, or because of the high costs of time ad moey. The extra data that is used is therefore umber of variables i the program that are safe, i.e. these variables ca ifer o errors ad therefore o failures, ad the umber of variables that are usafe, ISBN: 978-960-474-349-0 270
i.e. these variables ca potetially be the cause of a failure. 2 Reliability parameter The reliability parameters that are regarded i this paper are depedet o discrete probability parameters. Oe of the most importat reliability parameter accordig to the IEC 61508 stadard [4] is the so called probability of failure o demad (PFD). It is defied as the mea probability to perform a desiged fuctio o demad. The PFD is used for compoets with a low demad mode. A fuctio has a low demad mode, whe it is used oly scarce iside a system, i.e. it is executed for oly a fractio of the time compared to the surroudig system. Therefore it is appropriate to model this parameter as the result of a discrete radom process, i which the calculated parameter describes the probability the success or the failure of the fuctio with low demad mode. Safety Itegrity Level PFD SIL1 10-1 10-2 SIL2 10-2 10-3 SIL3 10-3 10-4 SIL4 10-4 10-5 Table 1: SILevel with their accordig PFD values 3 Bayesia reliability 3.1 Basics Addig extra iformatio or a-priori iformatio ito a model is usually doe with the use of Bayesia statistics. I reliability aalysis this is called Bayesia reliability. The classic approach to reliability aalysis produces parameters of a probability fuctio with its expected values ad its variace. A cofidece iterval has to be estimated for these parameters, i.e. the probability that the iterval boudaries are trustworthy i regard to the parameters. The advatage of the Bayesia approach is that it gives the credible iterval, i.e. the probability that the calculated reliability parameter lies i the iterval. This gives additioal certaity i the calculated values for the reliability parameters. Bayesia reliability uses two sets of iformatio: λ describes the reliability parameters that have to be calculated. This is for example the failure rate, failure itesity or the probability of failure o demad. The set D is the data that is available after testig. It cosists usually of failure occurreces ad times. These sets are used i the followig relatio: f ( D ) ) g ( D) (1) 0 f ( D ) ) d The reliability parameters for the model are calculated i the posteriori distributio λ D). It calculates the parameters with the highest probability give the recorded failure times. The additioal iformatio that is gaied from the value aalysis is icorporated ito the model through the prior distributio λ). The prior distributio is the iitial estimate of the parameters before ay failure times are collected. The probability that the give iitial estimate of the reliability parameters ca produce the collected data is calculated i the likelihood fuctio f(d λ). The deomiator of equatio (1) is called the margial distributio ad it represets a ormalizig factor for the posteriori distributio. A complete theoretical backgroud of Bayesia reliability is give i [2]. 4 Bayesia modellig I Bayesia statistics it is ecessary to specify a probability model. Oly the the posterior distributio ca be calculated. The model has the to be evaluated to cofirm that the Bayesia model yields usable results. The model is based o the chose likelihood fuctio ad the chose prior fuctio. The form of the prior distributio ca be derived from the chose likelihood, so that the resultig posterior distributio is i the same family of probability distributios. The model is the mathematical more coveiet, if the model is producig soud results. 4.1 Modellig of PFD For a PFD calculatio software is aalysed that has oly discrete rus. A fuctio that performs safety measures, e.g. emergecy shutdow, is i low demad mode ad therefore a PFD calculatio ca be applied. This is iterpreted as the probability that this software fuctio will fail if a safety measure has to be performed. Each demad to the software fuctio ca be see as Beroulli trial. A sum of rus is performed by the software fuctio ad the iterestig parameter is the umber of usuccessful rus, i.e. the umber of failures k. The Bayesia parameter λ i equatio (1) ca the be iterpreted as parameter of these Beroulli trials. It is iterpreted as the probability that a evet occurs. I this case the evet is defied as the iability to perform the iteded fuctio, i.e. a failure. The ISBN: 978-960-474-349-0 271
umber of evets k that occur ad the umber of trials represet the collected data D. A umber of idepedet ad idetical Beroulli trials are called a biomial distributio. The umber of trials is the amout of rus. The collectio of data D is doe by testig or simulatio of the software fuctio times. It will result i k failures of the software ad -k successful rus of the software. This gives a likelihood fuctio i the form of a biomial distributio, with a ukow parameter λ: k k f ( k ) (1 ) (2) k Figure 1 shows two biomial distributios with differet estimatios for λ. Fig. 1: Two examples of a biomial distributio The ukow parameter λ ca be estimated with the use of a prior distributio. The prior is chose because of mathematical cosideratios ad because it has to fit the uderlyig model. The model should also be useful whe o prior iformatio is available. The prior distributio should reflect that. For easy iterpretatio of the posterior parameters the prior distributio is chose so that it belogs to the same family as the posterior distributio, it is the called a cojugate prior i respect to the likelihood. The beta distributio ca be used as a cojugate prior for the biomial distributed likelihood fuctio i equatio (3): ( a a1 b1 ) Be( a, (1 ) (3) ( a) ( The parameters a ad b are both shape parameters. The beta distributio is defied for 0 <= λ <= 1, which makes sese because the PFD is a probability. The beta distributio ca take very differet forms ad is therefore very flexible ad powerful i modellig. Figure 2 shows four beta distributios with differet shape parameters. Fig. 2: Four examples of a beta distributio The most iterestig cases are the first ad the last forms, i relatio to the leged, of the example figure. The first form has the parameters a=1 ad b=1. It is a special case i which the beta distributio reduces to the uiform distributio. It ca be iterpreted as the complete absece of ay prior iformatio. The probability for every λ betwee zero ad oe is therefore equal, before ay data is collected. The example with the parameters a=2 ad b=10 is useful because it has the desired form for the PFD, it is cocetrated aroud small values for the PFD. If there is extra iformatio before the data collectio which ca give the prior distributio this form, the the posterior distributio has a similar form. It is absolute ecessary to check if the form of the prior distributio is soud i regard to the collected data. Buildig the posterior distributio from the biomial likelihood ad the beta prior yields a probability distributio of the followig form: k k (1 ) Be( a, k a, b, k, ) 1 (4) k k (1 ) Be( a, dp 0 k This is a beta distributed probability of the form: ( a k a1 bk 1 a, b, k, ) (1 ) ( k a) ( b k)) a, b, k, ) Be(( k a),( b k)) The PFD depeds o the two parameters of the prior distributio a ad b, ad o the umber of evets k (5) ISBN: 978-960-474-349-0 272
ad trials, i.e. failures that are collected through the testig or simulatio phase. 4.1.1. Parameter iterpretatio The iterpretatio of the parameter a ad b of the prior distributio is as follows. If the beta distributio is used as cojugate prior, with the biomial likelihood, the parameter a ad b describe the umber of evets of the Beroulli trial. I this case a is the umber of failures o demad ad b is the umber of successes o demad. The prior data is collected through static value aalysis ad is comprised of the umber of overall variables i the software code ad the umber of safe or usafe variables. The basic reasoig is, if there is o prior iformatio, i.e. the umber of safe variables is ot kow ad all variables have to be regarded as usafe variables, the the prior distributio has to reflect that absece of iformatio i the form of a uiform distributio. The beta distributio is flexible eough to support this (see form oe i figure 2) with the parameters a=1 ad b=1. The argumet for choosig the parameters is that with o prior iformatio ad with o data from actual rus o iformatio about the successful performace of the software is possible. The successful ru of a software demad has the the same probability as the failure of a software demad. This is reflected i the values of the parameters a ad b, which represet the umber of failures ad the umber of successes. The values of a ad b is 1, so that for every successful ru there is a usuccessful ru. The prior data has o iformatio about the umber of rus, because it is derived from static aalysis, which is performed without actual ruig the software. Istead the relatio of usafe ad safe variables is iterpreted as relatio of usuccessful ad successful rus. The parameter a ca the be iterpreted as the umber of usuccessful rus ormalised to 1 ad b is the umber of successful rus i relatio to a. This relatio is a result of the relatio of usafe variables to overall variables, i.e. if half of the variables are cosidered safe this gives a relatio of 1:2 ad the values of the parameters of the prior distributio are a=1 ad b=2. This model makes a costrait i regard to the actual umber of uses of the variables withi the software. Differet variables are used a differet umber of times. Some variables are used i every ru ad some are used ot oce i a actual ru. Because of the metioed costrait every variable has the same probability of beig used. I a actual ru of the software the umber of variables that are executed are i direct relatio to the umber of usafe ad safe variables, if this relatio is 1:2 the oly half as much usafe variables are used tha safe variables. Therefore the prior probability for a failure o demad has the same relatio ad ca be used i the prior distributio. This model has several advatages. It is easy to use. The prior distributio is easy to calculate. The form of the prior distributio fits the actual data. I the special case of absece of iformatio the results the results of the posterior distributio differ very little from the result if oly the likelihood distributio is used to calculate the reliability parameter. 5 Evaluatio of the model The model has to be checked to test that the delivered results of the model have reasoable meaig. I the followig sectio differet case are assumed ad the results are calculated ad their fitess iterpreted. The cases do ot use real world data. Istead oly sythetic data is used to check if the model theoretically produces meaigful results. Before the model is used i real world applicatios the model testig has to be exteded with real world data. 5.1. Applicatio I this case it is assumed the aalysed software is used i a eviromet that has a requiremet for the safety itegrity level (SIL) 2. It ca be see i table 1 that SIL2 requires a PFD reliability parameter betwee 10-2 ad 10-3. It is assumed that the testig of the software is doe 1000 times (=1000) ad that 5 failure are couted (k=5). If a biomial distributio for the PFD parameter is assumed that parameter ca be calculated with a maximum likelihood estimatio L(λ). The maximum likelihood estimatio (MLE) maximises the fuctio L(λ) i regard to the parameter λ ad gives thereby the most likely value for λ give the recorded data: L( ) xi L( ) (1 ) xi ˆ i1 x i1 i f ( x ) k i xi (6) ISBN: 978-960-474-349-0 273
Fig. 3: Plot of the MLE of the test data Figure 3 shows the resultig ML estimatio, with a maximum at λ=0.005 which correspods to the expected mode for this data k/=0.005. By addig differet prior iformatio it is evaluated, if the model yields meaigful results. The first examied case is the absece of additioal iformatio, which gives a prior beta distributio of the form Be 1_prior (1,1). The a value aalysis is performed ad the result gives relatio of 1:20 of usafe variables to overall variables. This relatio is further improved with two additioal cases with the relatios of 1:50 ad 1:100. The resultig prior distributios are therefore Be 1_prior (1,1), Be 2_prior (1,20), Be 3_prior (1,50) ad Be 4_prior (1,100). The posterior distributio for these priors ca easily be calculated with equatio (5) ad the give data k=5 ad =1000. The resultig posterior distributios are Be 1_post (6,996), Be 2_post (6,1015), Be 3_post (6,1045) ad Be 4_post (6,1095) ad are show i figure 4. 5.1.1. Discussio These posteriori distributios have the same form as the uderlyig likelihood fuctio of figure 3. This should be the case, because the mai source of the parameter estimatio is still the collected data, which is represeted by the likelihood fuctio. The prior iformatio should therefore ot alter the form of the likelihood. It also should ot skew or shift the likelihood too much. I that sese all of the above posterior distributios are suitable for the collected data. The advatage that the Bayesia model gives is that from the posterior distributio credible itervals ca easily be obtaied. The credible iterval describes the probability that a calculated parameter lies i a certai iterval. The example distributios have 95% credible itervals of CI 1_post (0.0022,0.0116), CI 2_post (0.0022,0.0114), CI 3_post (0.0021,0.011) ad CI 4_post (0.002,0.0106). The parameter that was estimated directly from the likelihood fuctio has a aalogous iterval the cofidece iterval, which has the problem that there are differet methods to calculate the iterval ad the meaig of the iterval is slightly differet, compared to the credible iterval. The cofidece iterval describes the probability of the iterval boudaries i cotrast to the probability of the reliability parameter. The 95% cofidece iterval of the example i figure 3 is CI LF (0.0006,0.0094). Here the extra iformatio makes a differece. The itervals for the posterior distributio are smaller tha the iterval i the likelihood fuctio, which meas the actual parameter is more trustworthy. The reliability parameter itself is ot much chaged. This makes sese as the mai source of iformatio is the test data, ad the prior iformatio should ot shift that parameter much. I compariso the calculated modes, the value with the highest probability, are Mo 1_post (0.005), Mo 2_post (0.0049), Mo 3_post (0.0048) ad Mo 4_post (0.0045). For the posterior distributio without prior iformatio the mode is the same as the mode of the likelihood fuctio. Fig. 4: Plot of the posterior distributios 5.1.2. Special case The developmet of safety critical software is doe very thorough ad accordig to a set of rules ad stadards. Therefore the software is ofte i a very mature state, whe the data collectio, the testig process, begis ad it is possible that o failures occur. But the software caot be regarded as free of errors, because that is a assumptio that is too optimistic. The resultig distributio for the likelihood fuctio has the followig form: ISBN: 978-960-474-349-0 274
mathematical results, e.g. the cofidece or credible itervals. A improvemet is achieved for the special case that o data is collected. The traditioal models fail to produce meaigful results i this case. The model used i this paper ca be used to overcome this deficit. Fig. 5: Plot of the MLE with k=0 The importat statistical characteristics for the biomial distributio with the calculated parameter λ=0 are the mode ad the expected mea, which are both 0. The software has tha to be iterpreted as free of errors, which caot be used i real projects. The posterior distributio for that case, with the same prior distributios as above, yield similar distributio forms: 6 Future Work Additioal models ca be developed ad evaluated, both for discrete data ad for cotiuous data. It has to be examied if differet prior distributios or likelihood fuctios yield better results. Data from real software projects has to be collected ad applied to the model, to see if the promisig results hold uder real world coditios. The umber of uses of safe ad usafe variables has to be collected ad has to be icorporated ito the model to get a more accurate relatio for the prior distributio. It has to be examied if additioal prior iformatio is easy to obtai ad ca icorporated ito this model to icrease the usefuless of this model, especially for the special case, whe o failure data could be collected. Fig. 5: Plot of the posterior distributios with k=0 The modes of all four distributios are also 0 ad therefore ot helpful. But for the expected meas of all these distributios applies λ!=0. These distributios regard the software as ot free of errors ad are there more usable. Refereces [1] Blachet B. et al. 2003. A static aalyzer for large safety-critical software, Proceedigs of the ACM SIGPLAN 2003 coferece o Programmig laguage desig ad implemetatio, Jue 09-11, 2003, Sa Diego: ACM [2] Hamada, M.S. et. Al. 2008. Bayesia Reliability, New York: Spriger [3] Heziger, T.A. et al. 2003 Software verificatio with Blast. I SPIN'03, Workshop o Model Checkig Software, 2003 [4] Fuctioal Safety of Electrical / Electroic / Programmable Electroic Safety-related Systems (IEC 61508), Iteratioal Electrotechical Commissio: Iteratioal Electrotechical Commissio, 2010 [5] Lios, J.L. 1997 Flight 501 Failure, Report by the Iquiry Board 6 Coclusio The described model demostrates good prospects. The calculatios give meaigful results that ca be easily iterpreted ad used. The use of additioal iformatio icreases the trust i the reliability calculatios, which is also displayed i the ISBN: 978-960-474-349-0 275