Are the Classes that use Exceptions Defect Prone?

Transcription

1 Are the Classes that use Exceptions Defect Prone? Cristina Marinescu LOOSE Research Group Politehnica University of Timişoara, Romania ABSTRACT Exception hling is a mechanism that highlights exceptional functionality of software systems. Currently many empirical studies point out that sometimes developers neglect exceptional functionality, minimizing its importance. In this paper we investigate if the design entities (classes) that use exceptions are more defect prone than the other classes. The results, based on analyzing three releases of Eclipse, show that indeed the classes that use exceptions are more defect prone than the other classes. Based on our results, developers are advertised to pay more attention to the way they hle exceptions. Categories Subject Descriptors D.2.4 [Software Engineering]: Software/Program Verification General Terms Measurement, Languages, Reliability 1. INTRODUCTION By enabling software developers to separate the source code that deals with unusual situations from the code that supports normal processing, exception hling mechanisms are intended to make it easier for developers to conceptualize build robust software systems. Since exception hling is not the main feature to be implemented, it does not receive so much attention [5] sometimes developers neglect it they use the ignore-for-now approach tend not to invest time in implementing code for proper hling unless it helps with debugging [14]. There are many empirical studies which show that different deviations from best programming practices correlate with defects 1 [5, 7]. Since among object-oriented systems we encounter many situations where exception hling is not properly hled, we investigate if the classes that use exceptions exhibit a higher significant number of bugs than the classes which do not make use of exceptions. 1 bugs in the source code. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage that copies bear this notice the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission /or a fee. IWPSE-EVOL 11, September 5 6, 2011, Szeged, Hungary. Copyright 2011 ACM /11/09...$ The paper is structured as follows: in Section 2 we relate our empirical study with existing works. In Section 3 we emphasize the process to which we extract the involved data in this study. In the first part of the next section (Section 4) we present the addressed research question. We continue with a brief description of the employed statistical tests followed by the presentation of the performed steps within our study. We end the section by pointing out the results of the study. The threats to the validity are presented in Section 5. In the last section (Section 6) we summarize the results address some hints towards the future work. 2. RELATED WORK To our best knowledge a study which correlates the usage of exceptions in the source code with the defects encountered in a system has not been done before. But there are several works which address different problems related with the usage of exceptions in this section we present some of them. Different approaches like the ones from [6, 13] that help developers to establish the flow of exceptions in object-oriented programs exist. Although our work did not focus on this topic, we want to emphasize that analyzing how exceptions are used within objectoriented source code is an important issue in the field. In [5] the authors analyze the fault proneness of the well-known return-code idiom for dealing with special situations in the source code. Their findings show that an idiomatic approach is indeed fault prone. In [4] is proposed an approach for replacing the return-code idiom this is done using the structured exception hling mechanism. With a slight increase of the source code size the approach almost completely removes the return-code idiom the control flow becomes simpler. The system under observation is further reengineered in order to make use of aspect-oriented programming in order to implement exception hling. Although some benefits were gained, as the author claims, using aspect-oriented programming for removing return idioms may not result in great benefit. In this context we consider that a similar study, like the one presented in this paper, which analyze the defect proneness among source code which make use of aspect-oriented programming for hling exceptions would support researchers developers. 3. DATA COLLECTION In this work we inspect the correlations (if any) between classes that use exceptions the defects these classes exhibit on three releases (2.0, ) of Eclipse 2. The mentioned releases of Eclipse were heavily used within various empirical studies like the 2

2 post number of post-release defects the type exhibits Name of Data filename typename type nthrows ncatches pre post Description location of the type (relative path) name of the type 0 - normal class 1 - abstract class 2 - interface total number of exceptions the contained methods throw within the throws clause total number of exceptions the contained methods catch number of pre-release defects the type exhibits number of post-release defects the type exhibits Figure 1: The Structure of the Extracted Data. ones from [17, 7]. Extracting the required data involve two steps we dedicate next a separate section for each performed step. 3.1 Extracting entities from the source code In order to extract the design entities (e.g., methods, classes, thrown exceptions) from the source code we use an enhanced version of the IPLASMA 3 4 [8] environment. IPLASMA is an integrated environment for quality analysis of object-oriented software systems that includes support for different phases of analysis from model extraction (including scalable parsing for C++, Java lately C#) up to high-level metrics-based analysis or detection of code duplication. This environment relies on the MEMORIA [12] meta-model which specifies the main entities which are extracted from the source code. We create within this environment a new analysis which provides for each class having the same name as the file it belongs to the first five values described in Figure 1. These values are stored within a csv (comma-separated values) file which is further processed by the R environment [11]. We performed an investigation we discover that the entities were captured from the source code with a precision near 100% a recall of 94%. More details about this investigation are presented in [9]. 3.2 Extracting defects The defects were extracted from the CVS version archive of Eclipse Bugzilla bug tracking system using the approach from [17]. The authors of the approach provides the Eclipse bug data set freely available, we use the number of pre-release postrelease defects (number of defects reported six months before after release) provided by this data set. We integrate from the Eclipse bug data set the numbers of prerelease post-release defects into the last two columns of the file whose structure in described in Figure 1. The correspondence between the data extracted by IPLASMA the data related to defects is established based on the value of the filename column. We take into account only the entities whose type is 0 or 1 (i.e., interfaces were excluded from this study). 4. CHARACTERISTICS OF THE EMPIRI- CAL STUDY The goal of this case study is to make an examination related to the classes that throw catch exceptions defect proneness of those classes. The quality focus is the defect proneness of classes making use of exceptions. The perspective of this case study is 3 Integrated Platform for software modelling analysis. 4 a previous version can be downloaded from that both researchers developers get knowledge about the correlation between the usage of exceptions in the source code the defect proneness of the classes. The context is to analyze three versions of Eclipse. 4.1 Research Question Are the classes which use exceptions more defect prone than the classes which do not use exceptions? 4.2 Employed Statistical Tests In order to answer the mentioned research question we employ the Chi-Square test we compute the Odds Ratios for each contingency table. The Chi-Square Test, as it is presented in [15], evaluates if within the underlying population represented by the sample in a contingency table (rxc), the observed cell frequencies are different from the expected frequencies. The tested hypothesis is that the two involved dimensions are independent of one another (i.e., no correlations among them is found). Odds indicate how much likely is for an event to occur as opposed to not occur. The Odds Ratio is the ratio of the odds of an event occurring in one group to the odds of it occurring in another group. If we encounter a value greater than 1 for the Odds Ratio, it means that the probability that an event to occur into the first group is higher than to occur into the second group. 4.3 Conducting the Empirical Study The structure of the two-dimensions contingency table used for computing the value for the Chi-Square test consists of two dimensions: Exceptions. The Exceptions dimension is the row dimension (e.g., independent) the is the column dimension (e.g., dependent). The two categories which composed the Exceptions dimension are: do not catch exceptions a class is mapped to this category if it does not throw catch exceptions from the bodies of its methods catch exceptions a class belongs to this category if it throws or catches at least an exception from the bodies of its methods. The two categories which composed the dimension are: defects the class reveal at least a defect Do not reveal defects the class does not reveal a defect. Since we investigate both pre-release post-release defects, we create two contingency tables for each analyzed versions of Eclipse (2.0, ). For each version the first contingency table (Figure 2a, Figure 3a, Figure 4a) stores the observations related to the pre-release defects, while the second contingency table (Figure 2b, Figure 3b, Figure 4b) stores the observations related to the post-release defects. Each value encountered in each contingency table represents the number of classes we observe as falling into the corresponding categories. For example, Figure 2a shows that Eclipse 2.0 has 946 classes that do not catch exceptions pre-release defects (the upper left value associated to the Cell O 11). From the contingency tables we can notice that each version of Eclipse contains around 44% of classes throwing or catching exceptions. 4.4 Results of the Empirical Study We consider the Chi-Square test of independence as being the most appropriate test to employ. The hypothesis that is evaluated is

3 related to the independence of the two dimensions from the contingency table. Null Hypothesis: H 0 : o ij = ɛ ij, where o ij represents the observed frequency of Cell ij, while ɛ ij represents the expected frequency of Cell ij in the underlying population of classes. Considering the sample, it means that the observed frequency is equal to the expected frequency for each cell. For the contingency table revealing pre-release defects for Eclipse 2.0 (Figure 2a) the expected values are presented in Figure 2c. Each expected value from Figure 2c was computed according to the formula E ij = (O i.)(o.j ), n where (O i.) represents the sum of observations in the row where the cell appears, while (O.j) represents the sum of observations in the column where the cell appears. Alternative Hypothesis: H 1 : o ij ɛ ij. This formula states that in the underlying population of classes the sample represents, the observed frequency for at least one cell is different than the expected frequency. With respect to the sample it means that the observed frequency is not equal to the expected frequency for at least one cell. For each of the contingency tables we compute the values of the Chi-Square test χ 2 using the R Project for Statistical Computing [11]. Since all p-values are less than a 0.05 level of significance (α=0.05), we consider we have enough evidence to reject the null Hypothesis. Consequently, the two dimensions of the contingency table are not independent (i.e., some correlations exist among them). Next, based on the values of the observed expected frequencies, we establish the way (positive or negative) in which the involved dimensions are correlated. A trait from the row dimension is positively correlated with a trait from the column dimension if the observed frequency is greater than the expected frequency. Based on this fact we discuss below each type of the existing inferred correlation.. Each version of Eclipse reveals a negative correlation between classes that do not catch exceptions pre-release defects (946< , 722< , 1130< ). Thus, classes having this trait were less likely to exhibit this type of defects. Classes that catch exceptions were more likely to pre-release defects (1245> , 1068>839.62, 1420> ).. Comparing the observed values from the contingency tables with the expected values, we establish the same types of correlations as within the pre-release defects. Conclusions. The performed test reveals that if a class throws or catches exceptions, it is more likely to exhibit defects. We consider it is important to mention that the provided results do not allow us to draw the conclusion that exceptions are the cause of encountering more defects within; we only provided evidence about an existing positive correlation. In order to bring a stronger evidence related to the correlation between the classes that use exceptions defects, we calculate the Odds Ratio for the last category, with respect to the first row category. The computations of the Odds Ratio firstly require the computation of the Odds. For example, the odds for a class revealing pre-release defects in the first condition ( do not catch exceptions) is computing by dividing 946 to In Figure 5 we present the values of the Odds for each contingency table associated with the three analyzed versions of Eclipse. We can notice that the probability for a class to reveal defects is lower than one half (Odds values lower than 1) for almost all conditions. The Odds Ratio is computed by dividing two values of the Odds Do not throw catch Exceptions Throw or Throw or Eclipse2.0 Eclipse RATIO Eclipse2.0 RATIO Eclipse Eclipse2.1 Eclipse Figure 5: The Values of the Odds. RATIO Eclipse2.1 RATIO Eclipse Figure 6: The Values of the Odds-Ratio. Eclipse3.0 Eclipse RATIO Eclipse3.0 RATIO Eclipse computing for a contingency table. We compute the Odds Ratio for the last category with respect to the first category (see Figure 6). The Odds Ratio for a class in the second condition ( catch exceptions) revealing pre-release defects versus a class in the first condition ( do not catch exceptions) is 2.40 (1.1517/0.4787) for Eclipse 2.0, 2.03 for Eclipse for Eclipse 3.0. The Odds Ratio for a class in the same condition revealing post-release defects is 2.98 for Eclipse 2.0, 2.45 for Eclipse for Eclipse 3.0. These results show us that: the chances that a pre-release defect occur in the group of classes that catch exceptions are 1.98 to 2.40 times higher than the chances a pre-release defect occur in the group of classes that do not catch exceptions. the chances that a post-release defect occur in the group of classes that catch exceptions are 1.69 to 2.98 times higher than the chances a post-release defect occur in the group of classes that do not catch exceptions. Conclusions. The values of the Odds Ratio indicate increased chances for the classes that throw or catch exceptions to exhibit defects compared to the classes that do not catch exceptions. 5. THREATS TO VALIDITY In this section we present the threats to validity associated to our empirical study, following the guidelines from [16]. Construct validity. This type of threats are connected to the extent the operational measures for the concepts being studies were established correctly [16]. Within the case study presented in this paper they are mainly related to the errors performed during the data extraction. The possible errors are due to the extraction of: design entities from the source code. We consider they were extracted with a high degree of precision recall. defects from the CVS version repository Bugzilla bug tracking system. We use the free Eclipse bug data set containing the defects involved in the study presented in [17].

4 ECLIPSE PRE ECLIPSE PRE ECLIPSE POST ECLIPSE POST Row Sums throw Sums (a) Contingency Table for. Do not (b) Contingency Table for (c) Expected Values for. Do Do not not (d) Expected Values for. ECLIPSE PRE ECLIPSE PRE Figure 2: Eclipse Contingency Tables Expected Values. ECLIPSE POST ECLIPSE POST Row Sums not throw Sums Sums (a) Contingency Table for. Do not (b) Contingency Table for (c) Expected Values for. Do Do not not or (d) Expected Values for. ECLIPSE PRE ECLIPSE PRE Figure 3: Eclipse Contingency Tables Expected Values. ECLIPSE POST ECLIPSE POST Row Sums throw Sums Sums (a) Contingency Table for. Do not (b) Contingency Table for (c) Expected Values for. Do Do not not or (d) Expected Values for. Figure 4: Eclipse Contingency Tables Expected Values.

5 Consequently, all the encountered threats to the construct validity we can find in the mentioned study are also found in our study. We use only non parametric statistical tests all the assumptions required by the used tests were satisfied. Internal validity. This aspect of validity is related to the causal relations that are inferred. Since our study is an exploratory one, this aspect is out of interest. External validity. The reported results are obtained by analyzing three versions of the well known Eclipse open source software system. This system is the subject of many empirical studies (e.g., [7] [17] [10]), like the previous studies, we do not suggest generalizing our research results to other systems unless further case studies are performed. Reliability validity. This aspect concerns the fact that a later investigator that conduct the same case study like the one presented here should obtain the same results, consequently, reach the same conclusions. We consider we provided enough information about the conducted study in order to let it be replicated. The source code of Eclipse is freely available, as well as the number of bugs it exhibits. The structure of the information extracted from the source code as well as the approach according to it has been extracted are presented in this paper. 6. CONCLUSIONS AND FUTURE WORK In this paper we present an empirical study performed upon three releases of Eclipse that provides evidence about a positive correlation between the classes that use exceptions the defects those classes exhibit. We show that the classes which use exceptions are more likely to exhibit defects (, as well as ) than the classes which do not use exceptions. We consider our finding may be correlated with an improper usage of exceptions in the source code. Consequently, a further step is to answer the following research question: Are classes which hle improperly the exceptions more defect prone than classes which do not hle improperly the exceptions? In this context an important question is Which are the improper usages of exceptions we should consider? Some static analyses related to an improper usage of exceptions in the source code are embedded by different tools for source code analysis (e.g., [1, 2]). For example, Check- Style provides a check called RedundantThrows [2] that verifies for redundant exceptions declared in throws clauses such as duplicates, unchecked exceptions or subclasses of another declared exception. We believe the list of the relevant analyses we should take into consideration is open for discussions. Last but not least we intend to replicate this study against other systems in order to see if the results obtained in this study can be generalized. Acknowledgments. I would like to thank Thomas Zimmermann, Rahul Premraj Andreas Zeller for making available the Eclipse bug data from [17] within the PROMISE data set [3]. I also owe a lot to Radu Marinescu, Ioana Verebi George Ganea for contributing, over the years, to the high degree of scalability IPLASMA reveals now. Last but not least I would like to deeply thanks Mihai Balint for spending almost an entire Sunday in order to make possible the extraction of data from Eclipse. 7. REFERENCES [1] FindBugs - find bugs in Java programs, last published: [2] CheckStyle - write Java code that adheres to a coding stard, last published: [3] G. Boetticher, T. Menzies, T. Ostr. PROMISE Repository of empirical software engineering data, West Virginia University, Department of Computer Science, [4] Magiel Bruntink. Reengineering idiomatic exception hling in legacy C code. In Proceedings of the 12th European Conference on Software Maintenance Reengineering (CSMR). IEEE Computer Society, [5] Magiel Bruntink, Arie van Deursen, Tom Tourwé. Discovering faults in idiom-based exception hling. In Proc. International Conference on Software Engineering (ICSE). ACM Press, [6] Chen Fu Barbara G. Ryder. Exception-chain analysis: ing exception hling architecture in java server applications. In Proceedings of the 29th international conference on Software Engineering, ICSE 07, pages , Washington, DC, USA, IEEE Computer Society. [7] Wei Li Raed Shatnawi. An empirical study of the bad smells class error probability in the post-release object-oriented system evolution. Journal of Systems Software, 80, July [8] C. Marinescu, R. Marinescu, P.F. Mihancea, D. Raţiu, R. Wettel. iplasma: An integrated platform for quality assessment of object-oriented design. In Proc. IEEE International Conference on Software Maintenance (ICSM Industrial Tool Volume), Budapest, Hungary. IEEE Computer Society Press, [9] Radu Marinescu Cristina Marinescu. Are the clients of flawed classes (also) defect prone? In 11th IEEE International Working Conference on Source Code Analysis Manipulation. IEEE Computer Society, [10] N. Nagappan, A. Zeller, T. Zimmermann, K. Herzig, B. Murphy. Change bursts as defect predictors. In IEEE 21st International Symposium on Software Reliability Engineering (ISSRE), [11] R Development Core Team. R: A Language Environment for Statistical Computing. R Foundation for Statistical Computing, Web page: http: //findbugs.sourceforge.net, ISBN [12] D. Raţiu. Memoria: A Unified Meta-Model for Java C++. Master Thesis, "Politehnica" University of Timişoara, [13] Martin P. Robillard Gail C. Murphy. Static analysis to support the evolution of exception structure in object-oriented systems. ACM Transactions on Software Engineering Methodology, 12, [14] Hina B. Shah, Carsten Gorg, Mary Jean Harrold. Understing exception hling: Viewpoints of novices experts. IEEE Transactions on Software Engineering, 99(RapidPosts): , [15] David J. Sheskin. Hbook of Parametric Nonparametric Statistical Procedures, 4th edition. Chapman&Hall/CRC, [16] Robert K. Yin. Case Study Research: Design Methods., 3rd edition. SAGE Publications, [17] Thomas Zimmermann, Rahul Premraj, Andreas Zeller. Predicting defects for Eclipse. In Third International Workshop on Predictor Models in Software Engineering. IEEE Computer Society, 2007.