Mining Edge-Weighted Call Graphs to Localise Software Bugs

Transcription

1 Mining Edge-Weighted Call Graphs to Loalise Software Bugs Frank Eihinger, Klemens Böhm, and Matthias Huer Institute for Program Strutures and Data Organisation (IPD), Universität Karlsruhe (TH), Germany Astrat. An important prolem in software engineering is the automated disovery of nonrashing oasional ugs. In this work we address this prolem and show that mining of weighted all graphs of program exeutions is a promising tehnique. We mine weighted graphs with a omination of strutural and numerial tehniques. More speifially, we propose a novel redution tehnique for all graphs whih introdues edge weights. Then we present an analysis tehnique for suh weighted all graphs ased on graph mining and on traditional feature seletion shemes. The tehnique generalises previous graph mining approahes as it allows for an analysis of weights. Our evaluation shows that our approah finds ugs whih previous approahes annot detet so far. Our tehnique also doules the preision of finding ugs whih existing tehniques an already loalise in priniple. 1 Introdution Software quality is a ig onern in industry. Almost any software displays at least some minor ugs after eing released. Suh ugs inur signifiant osts. A lass of ugs whih is partiularly hard to handle is nonrashing oasional ugs, i.e., failures whih lead to faulty results with some ut not with any input data. Nonrashing ugs in general are already hard to find. This is eause no stak trae of the failure is availale. With oasional ugs, the situation is even more diffiult, as they are harder to reprodue. Developers usually try to find and fix ugs y doing an in-depth ode review along with testing and lassial deugging. Sine suh reviews are very expensive, there is a need for tools whih loalise piees of ode that are more likely to ontain a ug. Researh in the field of software reliaility has een extensive, and various tehniques have een developed for loating ugs. Stati tehniques require a large ug and version history dataase, whih is not always availale. Dynami tehniques using instrumentation often have a poor runtime ehaviour. Another dynami tehnique is the analysis of all graphs. Suh a graph reflets the invoation struture of a partiular program exeution. Without any further treatment, aallgraphisarooted ordered tree. Themain() method 1 of a program usually 1 In this paper, we use method interhangealy with funtion. W. Daelemans et al. (Eds.): ECML PKDD 2008, Part I, LNAI 5211, pp , Springer-Verlag Berlin Heidelerg 2008

2 334 F. Eihinger, K. Böhm, and M. Huer is its root, and all methods invoked diretly are its hildren. Figure 1(a) is an astrat example of suh a all graph. Reent work [1, 2] deploys graph mining tehniques on all graphs for ug loalisation. [2] then derives a ranking of methods whih are most proale to ontain a ug. Generating suh a ranking is not trivial. For instane, follow up ugs need to e identified. [2] does not identify follow up ugs at all, and [1] only generates a aktrae-like struture whih helps the programmer to find follow up ugs. Graph mining is a relatively new disipline in data mining, and innovative algorithms have een developed in reent years [3, 4, 5, 6]. Various algorithms deal with the prolem of mining frequent sugraphs, i.e., disovering all sugraphs whih are frequent in a set of graphs. A diffiulty with graph mining on raw all graphs is that the algorithms do not sale. Therefore, redution tehniques are developed and applied first. Suh tehniques are not ovious: They involve a trade-off etween loss of information and the size of the resulting graphs. An important piee of information inluded in the raw all graphs is the all frequeny of all methods. The redution tehniques in [1, 2] loose this information. But it eases detetion of ugs whih affet the numer of invoations of a method, all frequeny affeting ugs. Various reasons for suh ugs exist, e.g., wrongly speified onditions. Note that it is not only loop onditions whih ause these ugs, ut any wrongly speified ondition leading to method alls within in a loop. This is eause ranhes taken within a loop or not affet the frequeny of a ertain method all. As iterations are elementary in programming languages, a wide range of ugs is all frequeny affeting. To find suh ugs, we take all frequenies into aount and analyse their differenes in orret and failing exeutions. Graph mining researh has foused on strutural and ategorial tehniques, and the various graph miners availale target at different kinds of graphs. Almost all algorithms handle ategorial data in node and edge laels. However, little attention has gone into the analysis of quantitative information, and no algorithm is availale for mining weighted graphs. Sine we want to analyse all graphs where weighted edges represent all frequenies, we must ome up with a solution. Further, finding a suitale omination of all graph redution, graph mining and the analysis of all frequenies is hallenging. In this work, we use onventional mining tehniques for unweighted graphs in onjuntion with feature seletion algorithms to analyse numerial edge weights. More speifially, we first trae program exeutions and lassify them as orret or failing using a test orale. We for our part do this y omparing exeution results to a fault-free referene. These orret results are typially availale, as test suites providing suh information are widely used in quality assurane [7]. We represent the program traes as all graphs and redue these graphs y deleting multiple method alls aused y iterations and introduing edge weights representing all frequenies. We then mine the redued graphs efore taking the edge weights into aount: By applying an entropy ased feature seletion algorithm to the weights of the different edges, we alulate the likelihood of oth every method invoation as well as of every method ontaining a ug. For a final

3 Mining Edge-Weighted Call Graphs to Loalise Software Bugs 335 ranking, we omine these likelihoods with another sore ased on strutural properties of the graph mining results. The rationale is that this ranking is given to a software developer who an do a ode review of the suspiious methods. In other words, solving the prolems mentioned so far requires innovations at different levels of analysis. Our ontriutions are as follows: Redution of software all graphs. We propose using a new variant of redued all graphs and present a tehnique to aomplish this redution. The redued graphs keep muh information, while they are relatively small. For example, [2] would redue one all graph from our evaluation from 9,946 to 50 edges. With our tehnique, there are just 31 edges, while keeping more information and, ultimately, giving way to etter results. Mining weighted graphs. We present an approah whih omines numerial analysis of edge weights with onventional graph mining. Our approah distinguishes etween ourrenes of edges whih appear more than one within a sugraph. This allows for a detailed analysis. To our knowledge, a tehnique whih analyses weights in the postproessing of graph mining has not een desried efore. This partiular ontriution is not limited to all graph analysis, ut an e transferred to any domain where weighted graphs are present. Comination of numerial and strutural tehniques. There exist frequent sugraphs whih our oth in all graphs of orret and of failing exeutions, as well as frequent sugraphs ourring in failing exeutions only. We analyse the edge weights for sugraphs from the first lass, whereas we generate purely strutural evidene from sugraphs from the seond lass. Our approah then omines these different kinds of evidene, and we demonstrate its usefulness. Evaluation. We show that our approah is partiularly well suited to disover all frequeny affeting ugs. Unlike previous tehniques, it also detets follow up ugs. With regard to ugs existing tehniques an already loalise, our approah doules the preision of finding them. Furthermore, it finds ugs on the granularity of method invoations instead of the level of methods. In this work, we do not redue reursive alls and onentrate on iterations. The redution of reursions is not ovious and is eyond the sope of this study. Paper outline: Setion 2 reviews related work. Setion 3 presents an overview of graph mining with all graphs. Setion 4 disusses graph redution tehniques. Setion 5 desries how to alulate proailities of ontaining ugs ased on redued graphs. Setion 6 features an evaluation. Setion 7 onludes. 2 Related Work A lot of researh has een done in the field of software reliaility. Approahes range from stati ode analysis and mining of software repositories and ug dataases [8, 9, 10] to dynami program verifiation. The latter fous on the data flow [11, 12] or, like all all graph ased tehniques, on the ontrol flow [13, 14]. In the following, we will first disuss the appliation of data mining tehniques

4 336 F. Eihinger, K. Böhm, and M. Huer in this ontext ug loalisation is just one appliation. Then we onentrate on two graph mining ased approahes [1, 2] whih are most related to our work. Finally, we desrie some related work in the area of mining weighted strutures. 2.1 Mining Software Metris and Invariants [8] maps post-release failures from a ug dataase to defets in stati soure ode. Using standard omplexity measures from software engineering, the soure ode is mined with regression models, whih an then predit post-release failures for new software entities. A similar study uses deision trees to predit failure proailities [9]. [10] uses regression tehniques to predit the likelihood of ugs ased on stati usage relationships etween software omponents. All approahes mentioned require a large olletion of ugs and version history. Dynami program sliing [11] gives hints whih parts of a program might have ontriuted to a faulty exeution, without ranking the loations in question. This is done y disovering all statements that atually affet the variales involved. Advaned tehniques like [12] perform dynami data flow foused analysis y instrumenting the soure ode to gain program invariants. These are used as features of orret and failing exeutions whih are analysed with regression tehniques. This leads to potentially faulty piees of ode. A similar approah, ut with a fous on the ontrol flow, is [13]. It instruments ondition statements and alulates a ranking ased on its evaluation frequenies. The instrumentation ased approahes mentioned either suffer from poor runtime ehaviour or miss ugs if only sampled parts of the software are instrumented. [14] is a tehnique whih uses traing and visualisation. It relies on a simple ranking of program omponents ased on the information whih omponents are exeuted more often in failing program exeutions. This ranking serves as a asis for more sophistiated rankings in [2] as well as in our approah. 2.2 Call Graph Based Fault Detetion The approah from Liu et al. [1]: This first study whih applies graph mining tehniques to dynami all graphs onsiders so alled software ehaviour graphs. These are redued all graphs, augmented with some temporal information. Setion 4 will provide more information on these graphs. The ehaviour graphs representing orret and failing program exeutions are mined with a variant of the CloseGraph algorithm [5]. This step results in frequent sugraphs whih are used as inary features haraterising a program exeution: A oolean feature vetor represents every exeution. In this vetor, every element indiates if a ertain sugraph is inluded in the orresponding ehaviour graph. Using those feature vetors, a support vetor mahine is learned whih deides if a program exeution is orret or failing. More preisely, for every method, two lassifiers are learned: one ased on ehaviour graphs inluding the respetive method, one ased on graphs without it. If the preision rises signifiantly when adding graphs ontaining a ertain method, this method is deemed morelikely to ontain a ug. Experiments with five out of 130 ugs from the Siemens Programs [15]

5 Mining Edge-Weighted Call Graphs to Loalise Software Bugs 337 demonstrate good lassifiation performane, ut do not evaluate the preision of the ug loalisation. Furthermore, the authors do not generate a ranking of methods suspeted to ontain a ug. As we do so, our approah an not e ompared diretly. However, in Set. 6, we ompare the redution tehniques. The approah from Di Fatta et al. [2]: In this work, a redution tehnique is again applied to the raw all trees first (see Set. 4 for details). The next step is similar to the one desried efore: A olletion of redued all graphs representing orret and failing program exeutions is analysed with graph mining. The authors use the tree miner FREQT [16] to find all frequent sutrees. The all trees analysed are large and lead to salaility prolems of the algorithm. Hene the authors limit the size of the sutrees searhed to values up to 4. Then the authors identify whih resulting sugraphs are frequent in the set of failing program exeutions, ut not frequent in the set of orret ones. This set of sugraphs is alled speifi neighourhood. For all methods invoked within sugraphs whih are part of the speifi neighourhood, a proaility of ontaining a ug is alulated ased on support values. Like [1], [2] does not put attention on all frequenies. 2.3 Susumption Frequent sugraph mining is a generalisation of previous strutural knowledge disovery tehniques suh as mining of itemsets, sequenes and trees [17]. Early work [18] in the area of itemsets has introdued the prolem of weighted struture mining. Itemsets an e seen as graphs onsisting of nodes only. In [18], these nodes are weighted and are disretised during preproessing. Corresponding tehniques are applied to graphs in transportation networks [19] and image analysis [20]. Suh disretisation leads to a loss of information, as we will disuss in Set In [21], we have already analysed tuples of weights of sequenes as a postproessing step of sequene mining. This allows for a more detailed analysis of weights in different strutural ontexts. This urrent work is in the field of weighted struture mining as well it analyses edge weights susequent to a graph mining step. [22] is a preliminary and muh shorter version of this paper. It diretly omines its results with those of [2] in order to find a wider range of ugs. It requires two ostly graph mining steps. This urrent work avoids this. 3 Call Graph Mining Overview Before we fous on redution and ranking tehniques in Setions 4 and 5, we now give an overview of the proedure of loalising ugs with graph mining. Note that [2] follows this general proedure as well. Algorithm 1 first assigns a lass (orret, failing) to every program trae (Line 4), using a test orale. Then every trae is redued (Line 5), whih leads to smaller all graphs. Tehniques to do so are disussed in Set. 4. Now frequent sugraphs are mined (Line 7). For this step, several algorithms, e.g., tree mining or graph mining in different variants,

6 338 F. Eihinger, K. Böhm, and M. Huer Algorithm 1. Generi graph mining ased ug loalisation proedure 1: Input: a olletion of program traes t T 2: G = // initialise a olletion of redued graphs 3: for all traes t T do 4: assign a lass {orret, failing} to t 5: G = G {redue (t)} 6: end for 7: SG = frequent sugraph mining(g) 8: alulate P (m) for all methods m; ased on SG an e used. The last step is alulating a likelihood of ontaining a ug. This an either e fine granular for every method invoation or, more oarse grained, for every method (as shown in Line 8). The alulation of the likelihood is ased on the frequent sugraphs mined and failitates a ranking of the methods, whih an then e given to the software developer. 4 Call Graph Redution In the related work on all graph mining (see Set. 2.2), two different approahes exist whih lead to redued all graphs (as speified in Line 5 of Algorithm 1). The redution tehnique used in [1] projets every node representing the same method in the all graph to a single node in the redued graph. We all this tehnique total redution. Note that this may give way to the existene of loops and limits the size of the graph (in terms of nodes) to the total numer of methods in the program analysed. As an example, the raw ordered all tree in Fig. 1(a) would result in the redued graph displayed in Fig. 1(). In addition to the redution, so alled temporal edges are inserted etween all methods whih are exeuted onseutively and are invoked y the same method. This tehnique integrates the temporal order from the raw ordered all trees into the graph representations. Tehnially, temporal edges are direted edges having another lael, e.g., temporal, ompared to other edges whih are laelled, say, all. Figure 1() serves as an example of a graph using the redution tehnique and temporal edges (dotted), alled software ehaviour graph. This redution is rather severe, e.g., from several millions of nodes to several hundreds. It allows graph a a a a a a (a) () () (d) (e) (f) Fig. 1. Variants of redued all graphs

7 Mining Edge-Weighted Call Graphs to Loalise Software Bugs 339 mining ased ug loalisation even with larger software projets. In ontrast, muh information aout the program exeution is lost. This onerns frequenies of the exeution of methods as well as information on different strutural patterns within the graphs. In partiular, the information in whih ontext a ertain sustruture is exeuted is lost (see Set. 5.2). Furthermore, the temporal edges inrease the size of the graphs signifiantly. An inreased preision of fault detetion y using temporal edges has not een evaluated. The approah in [2] keeps more information. It omits sustrutures of susequent exeutions, whih are invoked more than twie in a row from the same node. See Fig. 1(d) for an example. This redution ensures that many equal sustrutures alled within a loop do not lead to all graphs of an extreme size. In ontrast, the information that some sustruture is exeuted several times is still enoded in the graph struture, ut without exat numers. Compared to [1], muh more information aout a program exeution is kept, ompromised y a all graph whih is generally muh larger. For example, graphs are redued from several millions of nodes to several ten thousand nodes. In our approah, we try to overome the shortomings of oth approahes and try to keep most of the information availale. We redue sutrees exeuted iteratively y deleting all ut the first one and inserting the all frequenies as edge weights. This makes the graphs relatively small and keeps a lot of information. An example is given in Fig. 1(e). The introdution of edge weights allows for a detailed analysis. If, e.g., a ug is hidden in a loop ondition, this might lead to hundreds of iterations of the loop, ompared to just a few in orret program exeutions. Note that, with oth previous graph representations, the graph of the orret and of the failing exeution is redued to exatly the same struture in this ase. In our approah, the edge weights would e signifiantly different. Analysis tehniques an then disover this. level 1 a level 1 a level 1 a level 3 level 2 d d level level 3 d 2 1 d 2 level 2 2 level 3 d 3 (a) () () Fig. 2. A raw all tree, its first and seond transformation step For our graph redution approah, whih implements the funtionality speified in Line 5 of Algorithm 1, we organise the all tree into n horizontal levels. The root node is in level 1, all other nodes are in susequent levels, inreasing with the distane to the root. See Fig. 2(a). A naïve approah to redue our example all tree in Fig. 2(a) would e to start at level 1withNodea. There, one would find two hild sutrees having a different struture one ould not merge anything. Therefore, we work level y level, starting from level n 1, as

8 340 F. Eihinger, K. Böhm, and M. Huer Algorithm 2. Sutree redution algorithm 1: Input: a all tree organised in n levels 2: for level = n 1 to 1 do 3: for eah node in level do 4: merge all idential hild-sutrees of node, sum up orresponding edge weights 5: end for 6: end for desried in Algorithm 2. In our example in Fig. 2(a), we have to start in level 2. The left node has two different hildren. Thus, nothing an e merged there. In the right, the two hildren are merged y adding the edge weights of the merged edges, yielding the tree in Fig. 2(). In the next level, level 1, we proess the root node a. Here, the struture of the two sueeding sutrees is the same. We merge them, resulting in the tree in Fig. 2(). Our redution tehnique oviously is not lossless. The size of the resulting graphs would e prohiitive. With large piees of software, graph mining may not sale with the size of the all graphs, even if the redution tehnique applied is effetive. To deal with this prolem, it seems promising to use graphs of oarser granularity: Instead of using methods as nodes in all graphs, it is possile to, say, use lasses as a oarser astration. Suh all graphs would have lasses as nodes and inter lass method alls as edges. Oviously, suh a oarser astration would lead to less detailed ug loalisation as well. 5 The Ranking Framework So far, we have disussed how to redue all graphs. Now we will desrie our framework for deriving a ranking of potentially uggy method invoations (edges) and methods (nodes) from suh graphs. Before we fous on the individual omponents in the following susetions, we give an overview of our framework. At first, we apply frequent sugraph mining to the redued all graphs, without onsidering the weights for now (Set. 5.1). This orresponds to Line 7 in Algorithm 1. We then partition the set of frequent sugraphs just mined and onsider two sets separately: (1) the set of sugraphs whih our in oth orret and failing exeutions SG f and (2) the set of sugraphs whih only our in failing exeutions SG 2 f.weusesg f to uild a ranking ased on the differenes in edge weights in orret and failing exeutions (Set. 5.2). This ranking an e ased on method invoations or on individual methods. As sugraphs in SG f are never ontained in orret exeutions, it is not possile to analyse differenes ased on SG f. In ontrast, SG f provides ruial information aout the graph strutures in failing exeutions. Therefore, we derive a sore ased on the support in SG f (Set. 5.3) and omine it with the edge weight ased one mentioned aove (Set. 5.4). This omination is in Line 8 in Algorithm 1. 2 In preliminary experiments, we have also evaluated the influene of sugraphs whih our in orret exeutions only. It has turned out that suh graphs do not help to loalise ugs.

9 Mining Edge-Weighted Call Graphs to Loalise Software Bugs Graph Mining Step After having redued the all graphs gained from orret and failing program exeutions, we searh for frequent losed sugraphs SG in the graph dataset G using the CloseGraph algorithm [5]. For this step, we employ the ParSeMiS graph mining suite 3. Closed mining redues the numer of graphs in the result set signifiantly and inreases the performane of the mining algorithm (studying its effets on result quality is eyond the sope of this artile). Furthermore, the usage of a general graph mining algorithm instead of a tree miner allows for omparative experiments with other graph redution tehniques (see Set. 6). After the graph mining step, we partition SG and derive the set of sugraphs whih our in orret and failing exeutions SG f and the set of sugraphs whih our in failing exeutions only SG f. 5.2 Entropy Based Soring We now fous on frequent sugraphs whih our in oth orret and failing exeutions (SG f ). Our goal is to develop an approah whih disovers whih edge weights of all graphs from a program are most signifiant to disriminate etween orret and failing. To do so, one possiility is to onsider different edge types, e.g., edges having the same alling method m s (start) and the same method alled m e (end). However, edges of one type an appear more than one within one sugraph and, of ourse, in several different sugraphs. Therefore, we analyse every edge in every suh loation, whih we refer to as a ontext. To speify the exat loation of an edge in its ontext within a ertain sugraph, we do not use the method names, as they may our more than one. Instead, we use a unique id for the alling node (id s ) and another one for the method alled (id e ). All ids are valid within its sugraph. To sum up, we referene an edge in its ontext in a ertain sugraph sg with the following tuple: (sg, id s, id e ). A ertain ug does not affet all method alls (edges) of the same type, ut method alls of the same type in the same ontext. Therefore, we assemle a feature tale with every edge in every ontext as olumns and all program exeutions (represented y their redued all graphs) as rows. The tale ells ontain the respetive edge weights. The following tale serves as an example: a a a (sg 1, id 1, id 2 ) (sg 1, id 1, id 3 ) (sg 2, id 1, id 2 ) Class g failing g orret The first olumn orresponds to the first sugraph (sg 1 ) and the edge from id 1 (method a) toid 2 (method ). The seond olumn orresponds to the same sugraph (sg 1 ) ut to the edge from id 1 (method a)toid 3 (method ). The third olumn represents an edge from id 1 to id 2 in the seond sugraph (sg 2 ). Note 3

10 342 F. Eihinger, K. Böhm, and M. Huer that method ours twie in sg 1 and that ids have different meanings in sg 1 and sg 2. The last olumn ontains the lass orret or failing. The rows orrespond to all redued all graphs g 1,..., g n G availale. If a ertain sugraph is not ontained in a all graph, the orresponding ells have value 0, like g 2 whih does not ontain sg 1. Graphs (rows) an ontain a ertain sugraph not just one, ut several times at different loations. In this ase, we use aggregates in the orresponding ells of the tale. As minimum values would ignore ugs resulting in inreased numers and maximum values would ignore ugs leading to redued numers, we use the average. In the example, sg 2 is emedded at two loations in g 1. In one loation the edge from id 1 to id 2 has the weight 6, in the other one weight 8. The tale struture desried allows for a detailed analysis of edge weights in different ontexts within a sugraph. All following steps in this susetion are desried in Algorithm 3. After assemling the tale, we employ a standard feature seletion algorithm to sore the olumns of the tale and thus the different edges. We use an entropy ased algorithm from the Weka data mining suite [23] whih alulates the information gain InfoGain [24] (with respet to the lass of the exeutions, orret or failing) for every olumn (Line 2 in Algorithm 3). The information gain is a value etween 0 and 1 whih we interpret as a likelihood of eing responsile for ugs. Columns with an information gain of 0, e.g., the edges always have the same weights in oth lasses, are disarded immediately (Line 3 in Algorithm 3). a a d d e f e f (a) () Fig. 3. Follow up ugs Call graphs of failing exeutions frequently ontain ug-like patterns whih are aused y a preeding ug. We all suh patterns follow up ugs and remove them from our ranked list of features. Figure 3 illustrates a follow up ug: (a) represents a ug free version, () ontains a ug in method a where it alls method d. Here, this method is alled 20 times instead of twie. Following our redution tehnique, this leads to the same (or a proportional) inrease in the numer of alls in method d. In our entropy ased ranking, the edges d e and d f inherit the sore from a d if the saling of the weights is proportional. Thus, we interpret these two edges as follow up ugs and remove them from our ranking. More formally, we remove edges if the edge leading to its diret parent within the same sugraph has the same entropy sore (Line 4 in Algorithm 3). In ase of more than one ug in a program, this way of follow up ug detetion might

11 Mining Edge-Weighted Call Graphs to Loalise Software Bugs 343 Algorithm 3. Proedure to alulate P e (m s,m e )andp e (m) 1: Input: a set of edges e E representing edges in their ontext, e =(sg, id s, id e) 2: assign every e E its information gain InfoGain 3: E = E \{e e.infogain =0} 4: // remove follow up ugs: E = E \{e p : p E,p.sg = e.sg,p.id e = e.id s,p.infogain = e.infogain} 5: E (ms,m e) = {e e E e.id s.lael = m s e.id e.lael = m e} 6: P e(m s,m e) = max e E (ms,me) (e.infogain) 7: E m = {e e E e.id s.lael = m} 8: P e(m) = max e E m (e.infogain) not find all suh ugs, ut preliminary experiments have shown that it does detet ommon ases effiiently. We leave aside the pathologial ase that this tehnique lassifies a real ug as follow up ug. This is aeptale, sine the proaility of a ertain entropy value is the same for every ug. Therefore, it is very unlikely that two unrelated ugs lead to exatly the same entropy value, whih would lead to a false positive lassifiation. Now we alulate likelihoods of ontaining a ug for every method invoation (desried y a alling method m s and a method alled m e ). We all this sore P e (m s,m e ) as it is ased on entropy. To do so, we first determine sets E (ms,m e) of edges e E for every method invoation in Line 5 of Algorithm 3. In Line 6, we use the max() funtion to alulate P e (m s,m e ), the maximum of all edges (method invoations) in E (ms,m e). This is neessary, as in general there are many edges in E with the same method invoation. This is eause an invoation an our in different ontexts. With the max() funtion, we assign every invoation the sore from the ontext ranked highest. Lower sores for the same invoation are less important, and we ignore them. At this point, the ranking does not only provide the sore for a method invoation, ut also the sugraphs where it ours and the loations within it. This information might e important for a software developer. We report this information additionally. As we also want to ompare our results to those of [2] whih does not provide information on the invoation level, we also alulate P e (m) for every alling method m in Lines 7 and 8 of Algorithm 3. The explanation is analogous to the one of the alulation of P e (m s,m e )inlines5and6. This susetion has presented a tehnique relying on the analysis of edge weights in different ontexts. As we will see in the evaluation (Set. 6), the onsideration of different ontexts is key for good results. As ontexts are defined ased on the sugraphs mined, suh a differentiated analysis is only possile susequent to graph mining, ut not during preproessing (see Set. 2.3). 5.3 Strutural Soring Our entropy ased soring (Set. 5.2) annot detet ugs whih are not all frequeny affeting, as it analyses all frequenies only. At the same time, it

12 344 F. Eihinger, K. Böhm, and M. Huer does not onsider sugraphs whih our in failing exeutions only (SG f ). As some ugs result in suh sugraphs, these are essential to detet ugs as well. Therefore, we alulate the sore P s (m) for individual methods ased on the support in SG f. This sore is another likelihood of ontaining a ug, as it refers to the frequeny of method invoations in failing exeutions. P s (m) =supp(m, SG f ) (1) where supp(m, SG f ) is the fration of graphs in SG f ontaining a node m. 5.4 Comination Now we alulate the overall likelihood P (m) of ontaining a ug for every method m, ased on the average of the normalised values for P e (m) (see Set. 5.2) and P s (m) (see Set. 5.3). Normalisation is neessary: While oth values are in the [0, 1]-range, their maximum an e very different. Normalisation keeps us from overemphasising one of the two rankings. P (m) is the asis for the ranking of all methods m, whih is used to loate ugs: P (m) = P e (m) 2max (P e(n)) + P s (m) 2max (P s(n)) n t T n t T (2) where n is a method in a program trae t in the olletion of all traes T. 6 Evaluation To evaluate ug loalisation tehniques, the Siemens Programs [15] are often used [1, 2, 13] as a referene suite of C programs artifiially instrumented with different ugs. More speifially, it usually is just a small suset of this enhmark whih is used. For example, [2] just onsiders three of the seven Siemens Programs, [1] only five different ugs out of 130 availale in total. As most ug detetion tehniques are limited to ertain lasses of ugs, these tehniques annot find every element of a standard suite of ugs. Our approah, as well as the two most related ones [1, 2], fous on nonrashing oasional ugs. As we rely on Java software, we use a well known Java diff tool (taken from [25]) and instrument it with fourteen different ugs 4.Todoso,wehaveexamined the Siemens Programs and have identified five types of ugs whih are most frequent within them. Our programs ontain these ugs and also the kinds of ugs used in [1]. Most of these ugs are all frequeny affeting. The Siemens Programs mostly ontain ugs in single lines and just a few programs with more than one ug. To mimi the Siemens Programs as lose as possile, we have instrumented only two out of fourteen versions (Bugs 7 and 8) with more than one ug. We give an overview of the kinds of ugs used in the following tale: 4 We provide the software versions ontaining the ugs used at eihi/papers/eihinger08mining/

13 Mining Edge-Weighted Call Graphs to Loalise Software Bugs 345 Version Desription Bug 1, Bug 10 Wrong variale used Bug 2, Bug 11 Additional or-ondition Bug 3 >= instead of!= Bug 4, Bug 12 i+1 instead of i in array aess Bug 5, Bug 13 >= instead of > Bug 6 > instead of < Bug 7 A omination of Bug 2 and Bug 4 (in the same line) Bug 8 i+1 instead of i in array aess + additional or ondition Bug 9, Bug 14 Missing ondition Every version has een exeuted exatly 100 times with different input data. The results have een lassified as orret or failing exeutions with a test orale ased on a ug free referene version. Based on this data, we arry out four experiments: 1. The onventional method ranking, following [2], inluding its graph redution tehnique, using the same method soring and the same mining parameters for support and maximum sugraph size. 2. The total redution method ranking, the total redution from [1] with edge weights representing all frequenies (see Fig. 1(f)) together with the omined soring (Set. 5.4). 3. The entropy-ased method ranking, our redution tehnique (Set. 4) together with the entropy ased soring (Set. 5.2) ut without the omination. 4. The omined method ranking (Set. 5.4), our redution tehnique (Set. 4) together with the entropy ased soring (Set. 5.2) and the strutural soring (Set. 5.3). To keep the omparison foused, we leave aside the temporal order inside the all graphs. We have found graphs with temporal edges as used in [1] too large (in terms of edges) to e mined effiiently. Nevertheless, our study is fair sine we leave aside that temporal information with all alternatives. All experiments produe ordered lists of methods. A software developer doing a ode review would start with the top ranked method in suh a list. The maximum numer of methods whih have to e heked to find the ug is, therefore, the line numer of the faulty method in the ranked list. Sometimes two or more susequent lines have the same sore. As the intuition is to ount the maximum numer of methods whih have to e heked, all lines with the same sore have the numer of the last line with this sore. In order to evaluate the auray of the results, the line of the ranking where the first instrumented ug is found needs to e identified. If the first instrumented ug is, e.g., reported in the third line, this is a fairly good result. A software developer only has to do a ode review of maximally three out of 25 methods from our target program. Furthermore, in the entropy-ased and the omined method ranking, there usually is more information availale where a ug is loated within a method and in whih ontextitappears.thus,ouromparison is onservative, i.e., it does not demonstrate the full apailities of our approah.

14 346 F. Eihinger, K. Böhm, and M. Huer We present the results (the numer of the first line in whih a ug is found) of the four experiments for all fourteen ugs in the following tale. For versions with two ugs, we onentrate on finding the first of the two ugs ontained. Value 25 refers to a ug whih is not disovered with the respetive approah. Experiment \ Bug Version Numer Conventional method ranking (1) Total redution method ranking (2) Entropy-ased method ranking (3) Comined method ranking (4) Comparing the results from (1) and (3), the entropy-ased approah almost always performs as good or etter than the onventional one. This shows that analysing numerial all frequenies is adequate to loate ugs. Bugs 2, 9 and 13 illustrate that oth approahes alone annot find ertain ugs. Bug 9 an not e found y omparing all frequenies (3), as a ondition has een modified whih now always leads to the invoation of a ertain method. Therefore, the all frequeny is always the same (not all frequeny affeting). Bugs 2 and 13 are not found with the purely strutural onventional approah (1). Both are typial all frequeny affeting ugs: Bug 2 is in a ondition inside a loop and leads to more invoations of a ertain method. In Bug 13, a modified for-ondition slightly hanges the all frequeny of a method inside the loop. With the redution tehnique used in (1), this leads in Bug 2 and Bug 13 to the same graph struture oth with orret and with failing exeutions. Thus, it is not possile to identify strutural differenes. To ease presentation, we first desrie the omined approah (4) efore we disuss (2). Experiment (4) is intended to take important strutural information into aount as well and therefore to improve the results from (3). We do ahieve this goal: We retain the already good results from (3) in nine ases and improve them in five. In partiular, (4) finds Bug 9, whih is not possile with (3) alone. Therefore, the omination of numerial and strutural tehniques turns out to e superior. When alulating averages of the improvement of ertain approahes in the following, we leave aside Bugs 2, 9 and 13. This is eause high values would have a too strong influene on the results. Note that this lets our approah look somewhat worse, as it does find all three ugs. Bugs 7 and 8 illustrate that our approah analyses versions with more than one ug suessfully as well. In (4), oth ugs are ranked at Position 1 and 2. This is not worse than versions with one ug only. Experiment (2) is intended to evaluate the graph redution tehnique in [1]. Exept for the graph redution tehnique, the approah is idential to the one in (4). In almost all ases, (2) performs worse than (4). This onfirms that our graph redution tehnique is reasonale and that it is worth to keep more strutural information than the total redution does. Summing up, our experiments show that weighted graph mining on all graphs redued with our method is appropriate for preise software ug loalisation. Even if previous approahes are ale to detet all frequeny affeting ugs, our

15 Mining Edge-Weighted Call Graphs to Loalise Software Bugs 347 approah an detet them with a muh higher degree of preision. This is eause it expliitly analyses all frequenies. Furthermore, our approah finds ugs in settings with more than one ug and doules the preision of [2] on average. 7 Conlusions In this work we have addressed the prolem of loalising nonrashing oasional software ugs. This loalisation is important as suh ugs are hard to detet manually and ause signifiant osts. Our approah is dynami and ontrol flow entred as it relies on all graphs of program exeutions. We have presented a novel tehnique to redue suh graphs. It keeps the size of the resulting graphs relatively small while keeping more important information. In partiular, it introdues edge weights representing all frequenies. As none of the reently developed graph mining algorithms analyse weighted graphs, we have developed a omined approah whih does so. It onsists of onventional frequent sugraph mining and susequently soring of numerial edge weights using an entropy ased algorithm. Our experiments do not just show a douled preision of ug loalisation. They also show that our approah detets ugs whih previous approahes an not find in priniple. We demonstrate that the numerial information kept with our all graph redution tehnique is important for good results. We have shown that our omination of strutural and numerial mining tehniques is key for preise loalisations. Future work will address reursive method invoations. Another diretion is mining of all graphs with onstraint ased and approximative tehniques. Referenes [1] Liu, C., Yan, X., Yu, H., Han, J., Yu, P.S.: Mining Behavior Graphs for Baktrae of Nonrashing Bugs. In: Pro. of the 5th Int. Conf. on Data Mining (SDM) (2005) [2] Di Fatta, G., Leue, S., Stegantova, E.: Disriminative Pattern Mining in Software Fault Detetion. In: Pro. of the 3rd Int. Workshop on Software Quality Assurane (SOQUA) (2006) [3] Borgelt, C., Berthold, M.R.: Mining Moleular Fragments: Finding Relevant Sustrutures of Moleules. In: Pro. of the 2nd Int. Conf. on Data Mining (ICDM) (2002) [4] Yan, X., Han, J.: gspan: Graph-Based Sustruture Pattern Mining. In: Pro. of the 2nd Int. Conf. on Data Mining (ICDM) (2002) [5] Yan, X., Han, J.: CloseGraph: Mining Closed Frequent Graph Patterns. In: Pro. of the 9th Int. Conf. on Knowledge Disovery and Data Mining (KDD) (2003) [6] Nijssen, S., Kok, J.N.: A Quikstart in Frequent Struture Mining Can Make a Differene. In: Pro. of the 10th Int. Conf. on Knowledge Disovery and Data Mining (KDD) (2004) [7] Harrold, M.J., Gupta, R., Soffa, M.L.: A Methodology for Controlling the Size of a Test Suite. ACM Transations on Software Engineering and Methodology (TOSEM) 2(3), (1993)

16 348 F. Eihinger, K. Böhm, and M. Huer [8] Nagappan, N., Ball, T., Zeller, A.: Mining Metris to Predit Component Failures. In: Pro. of the 28th Int. Conf. on Software Engineering (ICSE) (2006) [9] Kna, P., Pinzger, M., Bernstein, A.: Prediting Defet Densities in Soure Code Files with Deision Tree Learners. In: Pro. of the Int. Workshop on Mining Software Repositories (MSR) at ICSE (2006) [10] Shröter, A., Zimmermann, T., Zeller, A.: Prediting Component Failures at Design Time. In: Pro. of the 5th Int. Symposium on Empirial Software Engineering (2006) [11] Korel, B., Laski, J.: Dynami Program Sliing. Information Proessing Letters 29(3), (1988) [12] Lilit, B., Aiken, A., Zheng, A.X., Jordan, M.I.: Bug Isolation via Remote Program Sampling. ACM SIGPLAN Noties 38(5), (2003) [13] Liu, C., Yan, X., Han, J.: Mining Control Flow Anormality for Logi Error Isolation. In: Pro. of the 6th Int. Conf. on Data Mining (SDM) (2006) [14] Eagan, J., Harrold, M.J., Jones, J.A., Stasko, J.: Tehnial Note: Visually Enoding Program Test Information to Find Faults in Software. In: Pro. of the Symposium on Information Visualization (INFOVIS) (2001) [15] Huthins, M., Foster, H., Goradia, T., Ostrand, T.: Experiments on the Effetiveness of Dataflow- and Controlflow-Based Test Adequay Criteria. In: Pro. of the 16th Int. Conf. on Software Engineering (ICSE) (1994) [16] Asai, T., Ae, K., Kawasoe, S., Arimura, H., Sakamoto, H., Arikawa, S.: Effiient Sustruture Disovery from Large Semi-strutured Data. In: Pro. of the 2nd Int. Conf. on Data Mining (SDM) (2002) [17] Han, J., Cheng, H., Xin, D., Yan, X.: Frequent Pattern Mining: Current Status and Future Diretions. Data Mining and Knowledge Disovery 15(1), (2007) [18] Srikant, R., Agrawal, R.: Mining Quantitative Assoiation Rules in Large Relational Tales. In: Pro. of the Int. Conf. on Management of Data (SIGMOD) (1996) [19] Jiang, W., Vaidya, J., Balaporia, Z., Clifton, C., Banih, B.: Knowledge Disovery from Transportation Network Data. In: Pro. of the 21st Int. Conf. on Data Engineering (ICDE) (2005) [20] Nowozin, S., Tsuda, K., Uno, T., Kudo, T., Bakir, G.: Weighted Sustruture Mining for Image Analysis. In: Pro. of the Conf. on Computer Vision and Pattern Reognition (CVPR) (2007) [21] Eihinger, F., Nauk, D.D., Klawonn, F.: Sequene Mining for Customer Behaviour Preditions in Teleommuniations. In: Pro. of the Workshop on Pratial Data Mining at ECML/PKDD (2006) [22] Eihinger, F., Böhm, K., Huer, M.: Improved Software Fault Detetion with Graph Mining. In: Proeedings of the 6th Int.Workshop on Mining and Learning with Graphs (MLG) at ICML (2008) [23] Witten, I.H., Frank, E.: Data Mining: Pratial Mahine Learning Tools and Tehniques with Java Implementations. Morgan Kaufmann, San Franiso (2005) [24] Quinlan, J.R.: C4.5: Programs for Mahine Learning. Morgan Kaufmann, San Franiso (1993) [25] Darwin, I.F.: Java Cookook. O Reilly, Seastopol (2004)