A Proactive Fault-detection Mechanism in Large-scale Cluster Systems

Transcription

1 A Poactve Fault-detecto Mechasm age-scale Cluste Systems Wu pg Meg Da Gao We ad Zha Jafeg Isttute of Computg Techology Chese Academy of Sceces Bejg Cha {wlp md gw Gaduate School of the Chese Academy of Sceces Bejg Cha Abstact To mpove the whole depedablty of lage-scale cluste systems a ole fault detecto mechasm s poposed ths pape. Ths mechasm ca detect the fault tme befoe ode fals ad eables the poactve fault maagemet. The poposed mechasm s summazed as follows: Fst the dyamc chaactestcs of cluste system ug omal actvty ae bult usg Tme Sees Aalyss methods. Secod the fault detecto pocess s mplemeted by compag the cuet ug state of cluste system wth omal ug model. The fault alam decso s made mmedately whe the cuet ug state devates the omal ug model. The expemet esults show that ths mechasm ca detect the fault cluste system good tme.. Itoducto The depedablty ad avalablty ae the majo challeges fo the lage-scale cluste systems. The eseaches elated to cluste system depedablty ae dvded to two categoes appoxmately. Oe s Hgh Avalablty HA techque based o edudat compoets ad ams at povdg cotuous sevce. Whe oe of the odes fals the fal-ove pocedues wll beg mmedately ad the applcatos ug o faled ode ae mgated to aothe edudat ode[]. The othe s based o the pedcto mechasm. The faults ae detected tme ad the coectly poactve mateace pogam s mplemeted befoe ode fals. Ths method avods the costs of uplaed dowtme of odes. Ths method s amed as Poactve Fault Maagemet seveal elated eseaches[]. The key questo fo Ths wok s suppoted by the Natoal 863 Hgh-Tech Pogam of Cha No. 4AA66 ad the 5th key poject of Cha No. 4BA8B9-. poactve fault maagemet s how to detect the fault system quckly that's effcet fault detecto mechasm. I fact the fault detecto mechasm has bee appled to some msso-ctcal ad safety-ctcal felds such as the acaft fault detecto[4] ad uclea powe plat fault detecto[5]. Now two elated woks compute systems ae dscussed. Oe s detecto of softwae agg pheomea ad the othe s SMART techology fo had dsk falue pedcto. Softwae agg[6] s oe kd of softwae fault whch s a pheomeo that the state of the softwae system degades o cashes wth tme. The measuemet-based ejuveato appoach deals wth detecto of the exstece of softwae agg ad pedctg agg-elated falues. I[7] the opeatg system esouce usage ad system actvty data ae collected at egula tevals fom etwoked UNIX wokstatos. A statstcal ted detecto techque s appled to the collected data to detect the exstece of agg ad the estmated tme to exhausto s calculated usg a o-paametc slope estmato techque. I[8] the dyamc chaactestc of web seve s bult usg tme-sees ARMA model ad the ARMA model s used to detect agg ad estmate esouce exhausto tme. To mpove the elablty of had dsk the SMART[9] Self-Motog Aalyss ad Repotg Techology falue pedcto system s cuetly mplemeted dsk dves. The SMART ca detect the fault had dves befoe dsk fals. The eseaches elated to SMART focus o the fault detecto ad falue pedcto techques. Fo example [9][] the geeal famewok s to detect aomales o vaatos fom omal behavo usg a ak-sum ull hypothess test; [] the fault detecto pocess s mplemeted usg dffeet statstcal tests based o ave Bayesa classfes. Both softwae agg detecto ad SMART methods focus o pat of system o pat of fault models. I ths pape we gve a poactve fault detecto mechasm fo lage-scale cluste systems fom system vew. System motog s a ecessay compoet fo lage-scale cluste systems. Fo example the system motog BlueGee/ s accomplshed though a combato of I/O ode ad sevce ode fuctoalty[]; fo /6/$. 6 IEEE

2 Dawg4A[3] the system motog fucto s accomplshed by the detecto ad data bullet sevce[4] the cluste opeatg system Phoex[5]. The pefomace paametes elated to system actvty whch s collected by system moto at egula tevals ae vaous such as the usage of CPU ad Memoy o Swap the voltage of odes the otate speed of fa ad the tempeatue of odes. Whe faults occu cluste system the actual obsevato of those paametes by system moto wll devate fom the omal ug model. So ole aalyss the state fomato deved fom cluste motog system ca atta the symptom of fault. Cuetly the aalyss of state fomato s smple theshold detecto method: f oe of the pefomace paametes volates the acceptable theshold the fault maagemet system wll cosde that the fault occued cluste system ad sed out a alam. Fo example a alam wll be set to admstato whe the load of CPU exceeds 9%. The met of theshold detecto s smple to mplemet ad appopate whe the sgals ae statc o slowly vayg. I cluste system the pefomace paametes ae vayg wth tme so theshold detecto method s appopate. Ths pape poposed a poactve ole fault detecto mechasm based o the actvty data collected by cluste system moto. The actvty data descbes the system ug state. Fst the omal ug models ae bult usg tme sees aalyss o collected data whe cluste us omally. Secod the dffeece betwee a paamete s pedcted value by omal ug model ad ts dectly obseved value s temed a esdual sees. All paametes esdual sees should be whte ose sees whe the cluste system s behavg omally. So the esdual sees gves the symptom of faults. I the ed the kelhood Rato Test I the emade RT fo shot o the esdual sees s used to detect faults. The pape s ogazed as follows. Secto gves the fault detecto pocedue ovevew. The ma algothms cludg modelg the cluste omal actvtes ad the RT method ae gve secto 3. I secto 4 we peset the expemet esults ad secto 5 cocludes wth futue wok.. Fault detecto pocedue The fault detecto pocedue s llustated fgue. Supposed that oe applcato exclusve use of the odes that t us o whe oe job maked AppM s ug o oe ode maked N wthout faults maked <N AppM> the pefomace paamete S s sampled at egula tevals ad the tme sees SN t s fomed. The omal ug model of <N AppM S> s bult usg the tme sees model of SN t. The pocedue of buldg the tme sees model s dscussed secto 3. detal. Fgue. The ole fault detecto pocedue Node N Nomal ug models Cluste system moto f Actvty data Resdual sees The fault detecto fucto f Result of f? f Fault alam The tme sees model ca pedct the futue value of paamete S. Afte buldg the omal ug models of evey <N AppM S> we popose a ole fault detecto method based o Slde Wdow. Defe the wdow W<o p > t-l+ <o p > t l fo evey <N AppM S> whee the tple <o p > t-l+ deotes the actual obseved value o the pedcted value p by the omal ug model ad the esdual o-p at t-l+ tme ad <o p > t deotes the actual obseved value o the pedcted value p by the omal ug model ad the esdual o-p at t tme; l s the legth of W. Now the fault detecto pocedue s descbed as follows: Fst step: omal ug model selecto. Select the omal ug model maked M based o tple <N AppM S> whee N s ode ame AppM s job ame ad S s oe of the pefomace paametes. Secod step: wdow W moves fowad oe pace. At t tme the wdow s W<o p > t-l+ <o p > t l. At t+ t tme the wdow moves fowad oe pace ad s W<o p > t+ t-l+ <o p > t+ t l whee p t+ t s the pedcted value by model M o t+ t s the obseved value collected fom system moto the esdual at t+ t tme s t+ t o t+ t p t+ t. Thd step: fault detecto. Rl t s oe l-legth esdual sees W<o p > t+ t-l+ <o p > t+ t l. Usg Rl t as the put of fault detecto fucto f ad calculate the esult of f. If f oe fault alam decso s made. Othewse cotue the fault detecto pocess ad etu to the secod step. The defto of fault detecto fucto s gve 3.3.

3 Fouth step: f AppM s ove ad aothe ew job s load etu to the fst step. 3. The ma algothms Fom above aalyss the key questo s buldg the omal ug models of evey job o evey ode ad usg the esdual sees betwee obseved value ad pedcted value of pefomace paametes as the symptom of fault detecto. Based o the tme sees aalyss techque the esdual sees should be a whte ose sees ad the dstbuto of the esdual sees follows the omal dstbuto N. If fault occus cluste system the esdual sees wll devate fom the whte ose sees ad the dstbuto of the esdual sees does t follow the omal dstbuto N. The RT method s used to costuct the fault detecto fucto f. 3. Modelg the omal actvtes Whe job AppM s ug o ode N wthout faults oe of the pefomace paametes maked S such as usage of CPU ad Memoy etwok badwdth etc s sampled at egula tevals by the cluste motog system. The sample cycle s t ad oe tme sees maked SN t s fomed whee N s the legth of sees. The dyamc model s bult usg tme sees aalyss o SN t ad ths model descbes the omal ug model of <N AppM S>. The followg gves the detaled pocedue of buldg the omal ug model of <N AppM S> usg tme sees aalyss o SN t. Fst step: Data Pepocess. Fo some upedctable easos SN t s ot a statoay tme sees. To mpove the accuacy of fault detecto two data pepocess pocedues ae ecessay fo SN t. Oe s movg aveage smoothg method ad the othe s computg the zeo aveage of SN t. Movg aveage smoothg: the -pot movg aveage of SN t s S N-+ t ad s + s. j j Zeo aveage: suppose µ s the aveage of tme sees S N-+ t that s µ s N +. The zeo aveage of S N-+ t s ZN-+ t whee evey sample s z s µ. Secod step: buldg the ARIMA model. ZN-+ t s t a statoay tme sees geeally. Thee s ted t. Hece we wll fst emove the ted by seveal dffeecg pocedues. Defe the oe pace dffeecg opeatos s z t z t -z t- -Bz t whee B s the backshft opeato so the d-pace dffeecg s d z t -B d z t. The d-pace dffeecg of ZN-+ t s XN-+-d t whee the sample of ew sees s d j x d d t z t C z t j t > d j d Fo each d3 calculate the Auto Coelato Fucto ACF ad Patal Auto Coelato Fucto PACF of XN-+-d t espectvely tll the XN-+-d t ca be egaded as a statoay tme sees Whe the ACF of the tme sees s quckly deceasg ths s a dcato that the tme sees s statoay. Buldg the ARMAp q model fo XN-+-d t: p q x t j j x t a t j a t ϕ whee p s autoegessve ode of the model q s the movg aveage ode of the model a t s a whte ose sees wth mea zeo ad vaace ad s ad s ae costats. If q the model educes to ARp. Ad f p the model p educes to MAq. Suppose ϕ B ϕ B ad q B B. So the above model ca be descbed as: Bx t Ba t The key step s detemg the ode of the model that s the value of p q. The ACF ad PACF povde mpotat fomato of p q. Table gves the behavo of the ACF ad PACF fo ARp MAq ad ARMAp q models. Table. ACF ad PACF fo AR MA ARMA ARp MAq ARMApq ACF Tals off Cuts off afte lag q Tals off PACF Cuts off afte lag p Tals off Tals off Fo ARp ad MAq the value of p o q ca be detemed by the cuts off of PACF o ACF. Fo ARMAp q two key pots to deteme p q ae: The ACF of ARMAp q begs decay at lag q. The PACF of ARMAp q decays statg at lag p. Afte deteme the ode of ARMAp q the values of s ad s ae estmated by east Squaes method. Fom ad the ARIMA model of ZN-+ t s B d z t Ba t t N-+ 3 Thd step: the omal ug model. Fom the aalyss of fst ad secod steps the omal ug model of <N AppM S> s descbed as follows: ϕ B d s t µ B at t N-+ 4 whee s t s the sample of pefomace paamete S at tme t. Fo d s d s t µ t fomula 4 ca be descbed as: ϕ B d s t B at t N-+ 5

4 The model 5 gves the omal actvtes fo <N AppM S>. If AppM s ug o ode N wthout faults the pefomace paamete S should follow model 5 ad the esdual sees {a t } betwee actual obseved sees ad pedcted sees should be whte omal ose. Othewse S wll devate model 5 ad the esdual sees {a t } wll ot be whte omal ose aymoe. So the fault detecto questo cluste system s tasfomed as testg whethe the esdual sees s whte omal ose o ot. We use RT method to vefy whethe the esdual sees s whte omal ose o ot. 3. kelhood Rato Test Fo a -legth esdual sees R t the ull hypothess ad alteatve hypothess ae defed as: H : R t s whte omal ose wth mea zeo cluste system s ug omally; H : The mea of R t s ot zeo thee ae faults cluste system. The level of sgfcace P{eject H H } s the pobablty of type I eo the pobablty of ejectg ull hypothess that s tue ad s the false-alam ato. Assume oe tme sees whose samples come fom omal dstbuto N whee s mea ad s vaace paamete space s { ; < < < < }. So the above hypothess test ca be descbed as follows: H : R t~ N ad > H : R t~ N ad > The subset { ; < < } of s the paamete space fo H. It s a composte vesus composte test ad we use RT method to do ths. The lkelhood fuctos of R t o ad ae: exp. ; π ω Ω exp. ; π x et ωˆ max ad Ωˆ max that s exp ˆ e π π ω Ω exp ˆ π e π whee. So the lkelhood ato s ˆ ˆ + Ω ω ad the test wll eject H f s small. The ctcal ego of fo H s. I aothe wods we wll eject H ad accept that thee ae faults cluste system whe. The value of s calculated as: + that s c 6 The left of fomula 6 s a adom vaable maked t ad t s a t-dstbuto wth - degees of feedom. Fo the gve ad the sgfcace level the value of c ca be foud the t-table based o P[ t c; H ]. So c +. Fo the esdual sees R t we eject H f. Othewse we accept H. 3.3 The fault detecto fucto Fo the esdual sees Rl t wdow W<o p > t+ t-l+ <o p > t+ t l we use RT method to test whethe Rl t s whte omal ose wth mea zeo o ot. Fo Rl t wdow W<o p > t+ t-l+ <o p > t+ t l defe the fucto > + f f t m t whee ad ae defed secto 3.. To get d of the acto of ose we toduce the pesstece checkg paamete w to the Fault Detecto Fucto FDF. The FDF s defed as s w s m s f +. If f oe fault alam decso s made. It s obvously that oe fault alam s tggeed ff the umbe of cotuous m s w. 4. Expemet

5 I ths secto we wll pove the utlty of ou poactve fault detecto mechasm by expemetal method. I the expemets the fault jecto s used to smulate the fault evomet. 4. Buldg the omal ug models The expemetal setup ths pape cossts of fou odes ode ode ode3 ode4 whose cofguatos ae dual P3 GHz CPU G Memoy 8GB had dsk ad ug ux.4.-8smp keel. Fst thee MPI jobs App App App3 ae ug o these odes wthout faults tu ad the ug tme s secods espectvely. At the same tme the cluste motog system collects the pefomace paametes of evey ode at egula tevals ad the sample cycle s 5 secods. Based o the coss coelato aalyss we select 9 pmay pefomace paametes fom about 8 pefomace paametes of ux opeatg system ad these 9 pefomace paametes ae show table. We buld the omal ug models of evey oe of 9 pefomace paametes ad use these models as the bass of fault detecto. Fo each <N AppM S> whee N {ode ode ode3 ode4} ad AppM {App App App3} ad S s oe of pefomace paametes table we buld the omal ug model usg the collected data. The follow gves a example: whe App s ug o ode wthout faults the omal ug model of actvepg s bult usg the collected data. The pocess of buldg the omal ug model of <ode App actvepg> s gve Fgue : a s the ogal sample data the umbe s 48; b s the data afte 3-pot movg aveage ad zeo aveage the aveage µ the fot 3 sample data ae goed ad the umbe of emade data s 45; c s the ACF k 5 fo b data the ACF of b s ot quckly deceasg so b s ot oe statoay tme sees; d s the data afte oe pace dffeecg of b; e ad f ae the ACF k ad PACF k of d espectvely accodg to table the d ca be modeled Table. Pefomace paametes Num Name Descpto Fle_sz Numbe of used fle hadles. Iode_sz Numbe of used ode hadles. 3 Supe_sz Numbe of supe block hadles allocated by the keel. 4 Dquot_sz Numbe of allocated dsk quota etes. 5 Rtsg_sz Numbe of queued RT sgals. 6 actvepg Numbe of actve ecetly touched pages memoy 7 Txpck_ps Total umbe of packets tasmtted pe secod 8 kbmemused Amout of used memoy klobytes. Ths does ot take to accout memoy used by the keel tself. 9 kbswpused Amout of used swap space klobytes. Cswch_ps Total umbe of cotext swtches pe secod. Ruq_sz Ru queue legth umbe of pocesses watg fo u tme davg_oe System load aveage fo the last mute 3 Pecetage_use Pecetage of CPU utlzato that occued whle excutg at the use level. 4 Pecetage_system Pecetage of CPU utlzato that occued whle excutg at the system level. 5 Fmpg_ps Numbe of memoy pages feed by the system pe secod. A egatve value epesets a umbe of pages allocated by the system a page has a sze of 4 kb o 8 kb accodg to the mache achtectue. 6 Iteupt_ps Total umbe of teupts eceved pe secod. 7 Tps Total umbe of tasfes pe secod that wee ssued to the physcal dsk. A tasfe s a I/O equest to the physcal dsk. Multple logcal equests ca be combed to a sgle I/O equest to the dsk. 8 Txe_ps Total umbe of eos that happeed pe secod whle tasmttg packets. 9 Txdop_ps Numbe of tasmtted packets dopped pe secod because of a lack of space lux buffes.

6 as ARMA64 ad B.49B B B B B B -6 ; B+.7579B B B B -4 ; g s the esdual sees whose mea s zeo ad vaace s 58; The QQ plot h shows that the esduals g s whte omal ose. I the ed the omal ug model of <ode App actvepg> ca be bult as B actvepg t Ba t : whee actvepg t s the sample of actvepg at tme t. The omal ug models of each <N AppM S> ae bult by the same meas. These models ae used fault detecto ad the esults of fault detecto ae show secto 4.. Fgue. Buldg the omal ug model of <ode App actvepg>.3 x 5 The Ogal sample data 5 The Sample data pocessed actvepg actvepg a The ogal sample data b The data afte pepocess Sample Autocoelato Fucto ACF 8 The Sample data oe degee dffeecg 6 actvepg.8 4 Sample Autocoelato ags b The ACF of b d Data afte -pace dffeecg of b Sample Autocoelato Fucto ACF Sample Patal Autocoelato Fucto PACF.8 Sample Autocoelato.5 Sample Patal Autocoelatos ags e The ACF k of d ags f The PACF k of d

7 The Resdual sees betwee Obseved data ad Pedcted value 8 Stadad Nomal Q-Q Plot of Sample Data 8 esduals Quatles of Resdual sees lags g The esduals Stadad Nomal Dstbuto h The QQ plot fo g 4. Results of fault detecto The Resdual sees betwee obseved value ad Pedcted value 8 Resduals 6 Oce the omal ug models ae bult the fault detecto pocess ca be mplemeted by compag the cuet ug state wth the omal ug models. Dug the expemet we wll ecod the tme whe the FDF s ad the tme whe the fault s jected to system. The dffeece betwee these two tmes s the fault detecto latecy. The latecy epesets the sestvty of fault detecto mechasm fo faults. The fault jecto techque s used to smulate the eal-wold faults cluste system. Fst u App ad ject memoy faults o ode. The jecto tme s the 3-th lag 5 secods afte App begs. Fgue 3 gves the fault detecto esults based o the bthe esdual sees Fault Alam Idex Fault Detecto Results.6 Fgue 3. Fault detecto esults by omal ug model of <ode App kbmemused> The kbmeused obseved data ad pedcted value x 6.3 Obseved value Pedcted value c The fault detecto fucto a kbmemused sees omal ug model of <ode App kbmemused>: a the blue s the actual obseved value ad gee s the pedcted value by the omal ug model; b s the esdual sees betwee obseved value ad pedcted value; c s the esults of fault detecto fucto ad the pesstece

8 checkg paamete w5 the sze of slde wdow. Fom c the fault detecto fucto s fom 43-th lag o. So the fault detecto latecy s 3 lags 565 secods. Secod u App ad ject I/O faults o ode. The faults ae jected whe App begs. Fgue 4 gves the fault detecto esults based o the omal ug model of <ode App Iteupt_ps>: a the blue s the actual obseved value ad gee s the pedcted value by the omal ug model; b s the esdual sees betwee obseved value ad pedcted value; c s the esults of fault detecto fucto ad the pesstece checkg paamete w5 the sze of slde wdow. Fom c the fault detecto fucto s fom 7-th lag o ad the fault detecto latecy s 7 lags 86 secods. The esults of fst ad secod expemets show the valdty of ou fault detecto mechasm. But the fault detecto latecy s hgh about mutes that's to say afte the fault exst about te mutes the Fgue 4. Fault detecto esults by omal ug model of <ode App Iteupt_ps> 5 The Iteupt-ps obseved data ad pedcted value Fault Detecto Results Fault Alam Idex c The fault detecto fucto fault alam just bee sed out. The ma cause fo the hgh latecy s the sze of slde wdow ad the ext expemet esults gve the elato betwee latecy ad the sze of slde wdow. I the last expemet u App3 ad ject I/O faults o ode3. The faults ae jected whe App3 begs. Nomal ug model of <ode3 App3 Cswch_ps> s B s t a t whee B.5439B - Fgue 5. The esdual sees by omal ug model of <ode3 App3 Cswch_ps> 5 The Cswch-ps Obseved data ad Pedcted value Obseved value Pedcted value Obseved value Pedcted value a Iteupt_ps sees Sample lags a Cswch_ps sees The Resdual sees betwee obseved value ad Pedcted value The Resdual sees betwee obseved value ad Pedcted Value 8 8 Resduals b The esdual sees b The esdual sees

9 ad s used to fault detecto. The fault detecto esults fo dffeet sze of wdows 46 ad w5 show the mpact of o fault detecto latecy. I fgue 5 a shows the obseved value blue ad the pedcted value gee; b s the esdual sees. Table 3 gves the fault detecto esults fo dffeet sze of wdows 46 ad w5. The esults show that the fault detecto latecy deceases whle the sze of wdow ceases. The ma easo s that the RT method s moe sestve wth a lage sample space. Table 3. The fault detecto latecy fo dffeet sze of wdows The sze of wdow Fault detecto latecy lags lags 5 secods Cocluso I lage-scale cluste system the fault detecto befoe ode falue s mpotat. Ths pape poposes a poactve ole fault detecto method usg the fomato collected fom cluste motog system. Fst buldg the cluste omal ug models of pefomace paametes usg the actvty data whe cluste us wthout faults; Secod the dffeece betwee actual obseved value ad pedcted value by omal ug model s esdual ad use the esdual tme sees as the symptom of fault; Whe the cluste system s behavg omally all the esdual sees ae whte ose sees; I the ed usg the RT method to test whethe the esdual sees ae whte ose sees ad the fault detecto fucto f s costucted based o the esults of RT ad the pesstece checkg paamete w. If f oe fault alam decso s made. The expemet esults show that the method ths pape ca detect the fault system good tme ad povdes the ablty to poactve fault maagemet. But the method ths pape s just a fault detecto ad ot fault dagoss. If abomal behavo s detected the ext step s locatg the accuate cause of fault. I the futue wok we wll pay moe atteto o two questos: Extact the moe sestve symptom of fault cluste system. I ths pape oly usg the esdual sees of pefomace paametes as the symptom of faults. I the futue we wll select the moe sestve symptom fo faults ad educe the fault detecto latecy educe the computato complexty at same tme. Ole fault dagoss method. Fdg the oot cause of fault ad gve the coectly healg advce to system admstato. 6. Refeeces []. Gao We. The desg ad aalyss method of hgh avalablty the seve cosoldato system: [Ph.D. dssetato]. Bejg: Isttute of Computg Techology Chese Academy of Sceces.. []. Vttoo Castell Rchad E. Hape Phlp Hedelbege Steve W. Hute Ksho S. Tved Kalyaaama Vadyaatha Wllam P. Zegget: Poactve maagemet of softwae agg. IBM Joual of Reseach ad Developmet 45: 3-33 [3]. R. Sahoo A. Ole I. Rsh M. Gupta J. Moea S. Ma R. Vlalta A. Svasubamaam. Ctcal Evet Pedcto fo Poactve Maagemet age-scale Compute Clustes. I Poceedgs of the ACM SIGKDD Iteatoal Cofeece o Kowledge Dscovey ad Data Mg pages August 3. [4]. D. Dasgupta K. KshaKuma D. Wog M. Bey: Negatve Selecto Algothm fo Acaft Fault Detecto. 3d Iteatoal Cofeece o Atfcal Immue Systems Cataa Scly.Italy Septembe [5]. J. Wesley Hes D. W. Mlle ad B. K. Hajek. Megg Pocess Models Wth Neual Netwoks fo Nuclea Powe Plat Fault Detecto ad Isolato The poceedgs of the 9th Powe Plat Dyamcs Cotol ad Testg Symposum Koxvlle TN 995. [6]. Y. Huag C. Ktala N. Koletts N.D. Fulto. Softwae ejuveato: aalyss module ad applcatos.

10 Poc. 5th Iteatoal Symposum o Fault-Toleat Computg Jue 995 pp [7]. S. Gag A. va Moosel K. Vadyaatha ad K. Tved A Methodology fo Detecto ad Estmato of Softwae Agg Poceedgs of the 9th Iteatoal Symposum o Softwae Relablty Egeeg Padebo Gemay Novembe 998 pp [8]. e Kalyaaama Vadyaatha Ksho S. Tved: A Appoach fo Estmato of Softwae Agg a Web Seve. Poc. It'l Symp. o Empcal Softwae Eg. pp.9 IEEE CS Pess. [9]. Hughes G.F. Muay J.F. Keutz-Delgado K. ad Elka C. Impoved Dsk Dve Falue Wags. IEEE Tasactos o Relablty Septembe. [].J. F. Muay G F Hughes Had Dve Falue Pedcto Usg No-paametc Statstcal Methods coautho Iteatoal Cofeece o Atfcal Neual Netwoks Istabul Jue [].Geg Hamely ad Chales Elka. Bayesa appoaches to falue pedcto fo dsk dves. I Poceedgs of the eghteeth teatoal cofeece o mache leag pages -9. Moga Kaufma Sa Facsco CA. [].J. Moea "System Maagemet The BlueGee/ Supecompute" Poc. It'l. Paallel ad Dstbuted Pocessg Symposum. os Alamtos CA IEEE Compute Socety. 3 p. 8. Apl 3. [3].Top 5 Supecopute Stes. System Ifo: Dawg 4A Opteo. GHz Myet [4].Che Y Meg Da Zha Ja-Feg Ze Ng Desg ad mplemet of fedeato based tadg sevce Compute Egeeg ad Applcatos 4. [5].Meg Da Zha Jafeg Wag e Tu Bbo Zhag Zhhog. Fully tegated cluste opeatg system: Phoex. Joual of Compute Reseach ad Developmet 4.6.