The Impact of Product Development on the Lifecycle of Defects

Transcription

1 The Impac of Produc Developmen on he Lifecycle of Rudolf Ramler Sofware Compeence Cener Hagenberg Sofware Park 21 A-4232 Hagenberg, Ausria ABSTRACT This paper invesigaes he defecs of a large embedded sofware developmen projec over a period of abou wo years. I describes how sofware developmen and produc developmen are organized in parallel branches. By mapping he defecs repored on produc developmen branches o he releases on he main line of sofware developmen, he paper shows he impac of he produc developmen sraegy on he defec lifecycle in sofware developmen. Caegories and Subjec Descripors D.2.9 [Sofware Engineering]: Managemen life cycle, sofware configuraion managemen, sofware qualiy assurance (SQA). General Terms Managemen, Measuremen. Keywords Produc developmen, defec lifecycle. 1. INTRODUCTION Sofware is he core par of more and more producs. I is embedded in elecommunicaion equipmen, medical insrumens, household appliances, enerainmen and mulimedia devices, auomoive componens, and indusrial machinery. While sofware is one of he criical success facors ha deermines he producs funcionaliy and qualiy, i is he produc as a whole ha cusomers and users value. They may no even be aware abou he sofware embedded in he producs hey buy and use. Hence, many companies have o embed sofware developmen ino produc developmen, and so heir produc sraegy defines and consrains how he sofware is developed and evolved. In his paper we invesigae he developmen of a large embedded sofware sysem in he conex of produc developmen. The research objecive of his paper is o measure he impac of he produc developmen sraegy on he defecs in he sofware sysem. We herefore observe he major poins in he lifecycle of every defec he poin were he defec is inroduced, repored, and resolved in erms of he sofware release and we race back he defecs repored in produc developmen o he corresponding releases on he main line of sofware developmen. The paper is srucured o reflec he seleced poins of observaion: Defec inroducion, defec deecion, and defec resoluion in sofware developmen and in produc developmen. Thus, afer an overview of he sudied projec in Secion 2, we discuss he hree poins of observaion for he main line of sofware developmen in Secion 3 and for produc developmen in Secion 4. In Secion 5 we analyze he measured impac of produc developmen on he main line of sofware developmen. Secion 6 summarizes and concludes he paper. 2. PROJECT BACKGROUND The observaions in his paper are based on he analysis of he developmen of an embedded sofware sysem. The sofware sysem is he basis for a diverse range of producs, which inegrae he sofware sysem wih hardware and peripherals from differen manufacurers. Figure 1 depics he sofware sysem s archiecure. I consiss of a runime kernel and drivers o conrol he underlying hardware layer, as well as a number of applicaions providing funcionaliy for he user. On op of he sofware sysem a generic user inerface suppors basic user ineracion and he branding and cusomizaion of he producs for differen marke segmens and regions in he world. Generic User Inerface Sofware Sysem App. 1 App. 2 App. 3 App. n Runime Kernel & Drivers Permission o make digial or hard copies of all or par of his work for personal or classroom use is graned wihou fee provided ha copies are no made or disribued for profi or commercial advanage and ha copies bear his noice and he full ciaion on he firs page. To copy oherwise, or republish, o pos on servers or o redisribue o liss, requires prior specific permission and/or a fee. DEFECTS'08, July 20, 2008, Seale, Washingon, USA. Copyrigh 2008 ACM /08/07...$5.00. Real-ime Operaing Sysem Hardware & Firmware Figure 1: Sofware Sysem Archiecure 21

2 Repored defecs Releases on he main line Figure 2: Accumulaed number of defecs for releases on he main line of developmen The sofware sysem encompasses more han 700 KLOC C++ code. I has evolved o his size over hree o four years of developmen wih an increase of abou 200 KLOC per year. In his ime no only addiional feaures were added bu also new hardware plaforms and peripherals were suppored as well as various realime operaing sysems. Technically, sofware developmen and produc developmen are organized using differen branches in he version conrol sysem. Sofware developmen is conduced on he main line, producs branch off from he main line and are evolved on parallel branches. In his sudy we invesigaed he evoluion of he sofware sysem by analyzing he defecs repored for abou 130 releases of he sofware sysem and heir impac on 14 consecuive releases on he main line (Figure 2), which capure abou wo years of ongoing developmen. The daa used in his sudy is rerieved from he issue reposiory and he release daabase. The issue reposiory conains more han resolved issue repors. Besides deailed informaion abou he naure of he issue, a repor conains a link o he release in which he issue has been found. Issue repors concern eiher defecs (68 percen) or enhancemens (32 percen). In his sudy we invesigae only defecs, for which he repor describes a failure of he sofware sysem and conains deails abou he faul in he code when resolved. Release daabase: The evoluion of he sofware sysem and he aciviies in conex of produc developmen are reconsruced from combining he daa from he issue reposiory wih he release daabase. The release daabase records abou 130 releases in oal, on he main line of sofware developmen and on he differen branches. In addiion, he release daabase associaes he resolved issues o one or more releases in which he issue has been fixed, including he files ha have been changed. 3. DEFECT LIFECYCLES IN THE MAIN LINE OF SOFTWARE DEVELOPMENT The main line of sofware developmen proceeds in ieraions of approximaely eigh weeks. Every ieraion ends wih he release of a labeled version on he main branch. Per ieraion abou 50 o 100 defecs are repored and abou he same number of defecs are resolved. The majoriy of he defecs are repored by he cenral esing deparmen, which ess every release while developmen coninues wih he nex release. In he following we describe he hree major phases of he defec lifecycle in he main line of sofware developmen. 3.1 Defec Deecion deeced on he main line of developmen are repored for releases on he main line. Figure 2 displays he cumulaive defec occurrence over ime for he 14 analyzed releases. The majoriy of he observable curves show a similar paern, a small iniial S- shape behavior. The number of repored defecs varies and, furhermore, he majoriy of he curves overlap in ime, since esing of one release may coninue for several ieraions. Ye, he resuling shor cycles of defec deecion in Figure 2 can clearly be aribued o he releases on he main line of developmen. 3.2 Defec Resoluion Defec repors are assigned o developers for resoluion. In our sudy we only ake defecs ino accoun for which a fix had been applied. Measuring he lifecycle of a defec from he release, in which he defec has been repored (R r ), o he release, in which he defec has been fixed (R f ) see Figure 3 shows, ha abou half (56 percen) of he defecs are resolved wihin one ieraion and he fixes are made available as par of he nex release. 89 percen of all defecs are fixed wihin 2 ieraions. The remainder of he repored defecs is fixed wihin a mos 10 releases. 3.3 Defec Inroducion Due o he delay beween he error, i.e., he even ha led o he inroducion of he defec, and he deecion of he defec, he defec may be presen in releases previous o he repored release (Figure 3). The release he defec has been inroduced (R i ) is he firs release conaining he defec. As an approximaion for R i, we raced back all repored defecs along he line of releases by he following rule: A defec is presen in releases where all he files changed during a fix remain unchanged, going backward from he fixed release R f, excluding he changes for he fix. This rule has been opimized o avoid false posiives, i.e. racing a defec o a release ha does no acually conain his defec, as he crierion for erminaing he back- 22

3 racing is a change o any of he involved files alhough his change is no necessarily he cause of he defec. For 21 percen of he defecs repored on he main line of developmen i is possible o race back he poin hey had been inroduced o a release R i before he repored release R r. Compuing he delay in number of ieraions from inroducion o reporing for defecs where R i < R r showed, ha 52 percen of hese defecs were repored in he nex release, 85 percen are repored wihin 3 ieraions, and all defecs are repored wihin a mos 10 ieraions. R i R r defec repored Figure 3: Releases conaining defecs (marked black) 4. DEFECT LIFECYCLE IN PRODUCT DEVELOPMENT In his secion we pu sofware developmen in he conex of produc developmen. While he sofware sysem is developed on he main line of developmen (or main branch) as described in he previous secion, sofware developmen in conex of produc developmen akes place in separae branches. On a produc branch, a seleced release of he main branch is advanced unil i mees he produc s requiremens. Inegraion branches are used o inegrae a sable release of he produc branch wih a produc s arge hardware. Figure 4 gives an overview of he differen branches, which are described in he following. Main line of developmen. The sofware sysem evolves along he main line of developmen. In every release new feaures are added, new hardware devices and new peripherals are suppored, new user inerface elemens are implemened, and he inegraion wih he real-ime operaion sysem is improved. Produc developmen, however, proceeds asynchronously from sofware developmen, usually a a faser pace, since he marke is highly compeiive and produc lifecycles are shor. Thus, produc developmen does no accommodae for developing hardware or sofware from scrach. Insead, produc developmen is dominaed by he procuremen and inegraion of hird pary componens and he cusomizaion for differen price segmens and he needs of R f inegraion branch produc branch main line of developmen Figure 4: Branches in produc developmen local markes. Ofen, a family of producs is developed, based on a single plaform bu wih differen markeable feaures. Produc branches. Sofware developmen in he conex of produc developmen branches off from he main line when a consisen feaure se and hardware suppor has been reached. On he produc branch, he sofware sysem is sabilized and alhough no inended bu occasionally necessary he feaure se is evolved furher unil i mees he produc requiremens. Fixes and enhancemens are merged back o he main line, where in he meanime developmen moves forward o he nex generaion of producs. Inegraion branches. When he sofware sysem is sable and complee, i is inegraed wih he hardware of he final produc. Inegraion is a major sep and requires close cooperaion wih produc developers, hardware developers, and suppliers of hird pary componens. I is herefore organized in a separae projec and a separae inegraion branch is creaed. In inegraion projecs, new releases of he sofware sysem are usually produced in weekly ieraions. Ofen here are several inegraion projecs and branches, one for every produc of a produc family. Inegraion is also he las resor o find and fix defecs. Once he produc has been released on he marke, i is usually no feasible o collec defec repors from end-users or o updae he sofware inegraed wih he produc. Thus, all relevan sofware defecs have o be found and fixed before he produc is released. Field defecs do no play a major role for sofware developmen any more, even hough hey are carefully moniored and cared by produc managemen. 4.1 Defec Deecion Over he analyzed period of wo years of developmen, wo produc developmen cycles can be observed. Figure 5 depics he number of defecs repored per monh over hese wo years for (a) he main line of developmen, (b) produc branches, and (c) inegraion branches. The wo produc developmen cycles appear as wo major peaks in he rend of he overall number of defecs repored. The apparen wo long cycles are a resul of he overlying produc developmen sraegy, which defers esing and sabilizaion for produc specific requiremens unil hey become available. A he ime of he firs release of he firs produc (marked as P1 in Figure 5) he number of defecs repored for he main line of developmen rapidly dropped as many resources from developmen were shifed o suppor he sabilizaion on he produc branch. Thus, he oal number of repored defecs sill increased. Once he sofware sysem had been sabilized on he produc branch, inegraion for he produc sared wih relaively few addiional defecs repored and developmen on he main line regained speed. For he second produc (marked as P2 in Figure 5) addiional resources where involved and, hus, developmen on he main line coninued comparaively unaffeced. Sill, he oal number of defecs increased significanly due o he work on he produc branch and in his case, where several producs of a produc family had o be suppored also on he inegraion branches. 23

4 Sum of defecs repored per monh Inegraion branch Produc branch Main branch P1 P Figure 5: repored on he main line, produc branch, and inegraion branch per monh The main purpose of produc and inegraion branches is he sabilizaion of he sofware sysem, i.e., finding and fixing of defecs in he ligh of specific produc requiremens and hardware configuraions. Hence, he defecs repored on produc and inegraion branches are deeced in addiion o wha is coninually found and fixed on he main line of developmen. On average, over he oal ime period observed, 45 percen of all defecs were repored on produc and inegraion branches. A he peak of he developmen of he firs produc, 90 percen of all defecs repored were due o produc and inegraion branches; 75 percen a he peak of he developmen of he second produc. R i defec repored R r R f 4.2 Defec Resoluion In he sudied projec i is a sric policy ha all changes have o be merged o he main line of developmen. Hence, developmen on he main line does no coninue unaffeced by he work on produc and inegraion branches. Besides he fixes for he defecs repored for a paricular release on he main line, addiional fixes are applied o he main line (R f in Figure 6) due o defecs repored and fixed on parallel branches (R f in Figure 6). 4.3 Defec Inroducion The defecs repored on produc and inegraion branches have eiher been inroduced already in he main line of developmen, before branching, or in he course of sabilizaion due o a defecive fix. As all changes are merged back o he main line, he defecs repored on produc and inegraion branches are presen on he main line in any case. The approach used o approximae he defecive releases on he main line is illusraed in Figure 6. In a firs sep, he rule described in Secion 3.3 was applied o race he defec back on he produc or inegraion branch, for which i had been repored, considering he files changed during he fix. When in racing back he saring poin of he branch had been reached, he rule was applied recursively for he release of he branch below, which had been used o creae he branch. In a second sep, if defec inroducion had no ye been raced back o a release on he main branch, he defec was mapped o he chronologically nex release on he main branch, making use of he fac ha all changes are merged o he main branch. The so idenified release R i is he earlies release on he main line conaining he defec, unless an earlier release could be idenified by racing he defec back on he main line from release R f. R i 5. IMPACT AND IMPLICATIONS In his secion we show he impac, he defecs repored on produc and inegraion branches have on he releases on he main line of developmen. The impac is measured in erms of he increase in he number of defecs inroduced per release, he number of defecs fixed per release, and he number of open defecs per release on he main line. Our findings are summarized in Table 1. The values given in he able are calculaed as he percenage of he oal number of inroduced, repored, fixed or open defecs due o defecs repored on branches. In release 4, for example, 82 percen of he all defecs inroduced in ha paricular release were raced back from defecs repored on a produc or inegraion branches. The firs column of he able, defecs inroduced, shows an increase in hese numbers for all releases. In half of he releases, he share of defec inroducions found via repors on branches was even more han 50 percen. For defecs repored on branches here is no meaningful mapping o defecs repored for releases on he main line. So, for his poin in he defec lifecycle, here is no direcly observable impac and he second column is herefore zero for all releases. The hird column, defecs fixed, shows he number of fixes merged o a release on he main line due o defecs repored on branches. For he firs hree releases here are simply no parallel R f Figure 6: Mapping defecs from branches o he main line 24

5 or earlier branches, so no fixes had been merged. In seven of he eleven remaining releases, he share of defec fixes merged from branches is larger han 50 percen. open is he number of defecs repored for his release plus he defecs from earlier release, which have no ye been fixed. When analyzing he defecs of a paricular release, e.g., when consrucing defec models based on he release s properies, he acual number of defecs presen, i.e. open, in his release is imporan. The las column in Table 1 shows he impac of defecs repored on branches on he number of open defecs. For example, in release 4 abou wo hird (67 percen) of he open defecs were repored on branches and mapped o he main line. Table 1: Impac of defecs from branches on he main line Release inroduced repored fixed open 1 3% 0% 8% 2 5% 0% 8% 3 58% 0% 40% 4 82% 57% 67% 5 44% 80% 38% 6 23% 64% 24% 7 15% 0% 26% 19% 8 19% 3% 22% 9 23% 5% 28% 10 56% 12% 48% 11 72% 32% 62% 12 80% 57% 69% 13 63% 70% 67% 14 65% 70% 62% Figure 7 furher analyzes he observed increase in he number of open defecs by comparing he number of open defecs per release, showing boh, he share resuling from he branches and from he main line. The disribuion of open defecs due o branches and main line can be explained by he produc developmen cycles as depiced in Figure 5. presen per release due o branches due o main line Releases Figure 7: Impac of defecs repored on branches on open defecs per release on he main line 6. SUMMARY AND CONCLUSIONS In his paper we invesigaed he defecs of a large embedded sofware developmen projec over a period of abou wo years. We described how sofware developmen and produc developmen are organized as parallel branches, consiuing he main line of sofware developmen, produc developmen, and inegraion. By mapping he defecs repored on branches o he releases on he main line, we were able o show he significan impac of he produc developmen sraegy on he lifecycle of he defecs on he main line of sofware developmen. Branching and merging in sofware developmen has been subjec o several sudies (e.g., [3, 4]) and some approaches and ools (e.g., [1, 5]) ake he evoluion of a sofware sysem across branches ino accoun. The novely in our work is he mapping of he defecs repored on produc and inegraion branches o releases on he main line of sofware developmen, including measuring he impac of defecs repored on branches for he main line. Furhermore, in his paper we were able o explain he measured impac as a resul of he pursued produc developmen sraegy. From our observaions and findings we can conclude ha defecs repored on branches have o be mapped o releases on he main line in order o obain a complee and realisic saus of he sofware sysem s defecs. Our observaions furher imply ha here is a laency caused by defec inroducion on he main line and defec deecion on branches during sabilizaion (bug-fix ime [2]). Thus, when analyzing releases on he main line, he ime frame has o be seleced large enough o encompass all defecs on branches which have an impac on he analyzed releases. We herefore currenly sudy how long defecs remain open o deermine his laency and he corresponding impac of he produc developmen sraegy. 7. ACKNOWLEGEMENTS This work was funded by he Ausrian COMET Program. The auhor also hanks Prof. Giancarlo Succi and Prof. Barbara Russo from he Free Universiy of Bolzano-Bozen for heir suppor. 8. REFERENCES [1] Fischer, M., Pinzger, M., and Gall, H Populaing a Release Hisory Daabase from Version Conrol and Bug Tracking Sysems. Inernaional Conference on Sofware Mainenance (ICSM 03), Amserdam, Neherlands. [2] Kim, S. and Whiehead, Jr., E. J How Long Did I Take To Fix Bugs?. Workshop on Mining Sofware Reposiories, ICSE 2006, Shanghai, China. [3] Perry, D.E., Siy, H.P., and Voa, L.G Parallel Changes in Large Scale Sofware Developmen: An Observaional Case Sudy. Inernaional Conference on Sofware Engineering (ICSE 98), Kyoo, Japan. [4] Williams C.C. and Spacco J.W Branching and merging in he reposiory. Workshop on Mining Sofware Reposiories, ICSE 2008, Leipzig, Germany. [5] Zimmermann, T., P. Weibgerber, S. Diehl, and Zeller, A Mining version hisories o guide sofware changes. Inernaional Conference on Sofware Engineering (ICSE 04), Edinburgh, UK. 25