Software Fault Prediction using Machine Learning Algorithm Pooja Garg 1 Mr. Bhushan Dua 2

Transcription

1 IJSRD - International Journal for Scientific Researc & Development Vol. 3, Issue 04, 2015 ISSN (online): Software Fault Prediction using Macine Learning Algoritm Pooja Garg 1 Mr. Busan Dua 2 1,2 Department of Computer Science and Engineering 1,2 DIET, Karnal, Haryana, India Abstract Software fault prediction indicates te likeliood of software fault at an early stage of software development process and ence it will be easier to identify and correct tem and also reduce faults tat would occur at later stages. Tis will improve te overall quality of te software product. In te recent years, several macine learning tecniques wic uses examples of faulty and non-faulty modules to build prediction models. Software metric ave been used as input to tese macine learning tecniques to represent te software modules. Support Vector Macines (SVM) is te main algoritm wic as been used for classification of faulty and non-faulty modules. But prior to using SVM, A few data pre-processing metods as been proposed suc as bootstrapping, clustering and random projection. Tese metods are needed because of certain problems wit metric datasets suc as class imbalance, noise, small dataset size and ig dimension. It as sowed by experiments tat wen software metrics dataset is preprocessed and transformed using above tecniques, te performance of SVM in predicting faulty and non-faulty modules is better. Te accuracy measure used for comparing performances of different models was F-measure. F-measure as been used because of its robustness to class imbalance. For some models F-measure as increased by about 40%, wic is very encouraging. Experimental results sows tat te proposed approac worked better tan existing approac using MATLAB and Weka programming. Key words: Support Vector Macines (SVM) Bootstrapping, Clustering, Random Projection, Software Metrics, Metrics datasets I. INTRODUCTION A. Software Metrics: Software metrics are various measurements of software. Measurements are useful because tey are repeatable wic means same measurement value will be reported by different persons for a particular object. Metrics are being increasingly used for quantitative and qualitative analysis of software. Tey give software professionals te ability to evaluate software process. One of te simplest and earliest metric is Lines of Code (LOC). LOC as te name implies means number of text lines in a software program. It gives a good measure of size of software. LOC metrics can be different depending on wat tey count. Oter metrics are Halstead metrics, McCabe s metric suit, Node Count, Condition count, Edge count, Branc count, Call pairs, Error count Etc. B. Metrics Datasets: A major callenge in developing software quality prediction models is to find sources of metrics data. Datasets wic contains tese metrics and fault information are not available widely. Altoug tere are many organizations were tis kind of data is available internally for its management, but tese are not publicly available. NASA MDP datasets are one suc publicly available datasets wic is widely used in software quality modelling. C. NASA MDP Metrics Datasets NASA MDP datasets are available from a total of 12 different NASA projects (Promise Data, 2012).Cleaned versions of datasets denoted by D are used (Sepperd et al., 2013). Dataset from eac project is given a name. Te name, description and programming language for eac of tese 12 datasets is given below (Jianget al., 2008): JM1: Real-time predictive ground system written CM1: NASA spacecraft instrument written KC1: Storage management for receiving and processing ground data written in C++. PC1: Fligt software for eart orbiting satellite written KC3: Storage management for ground data written in Java. MW1: Zero gravity experiment related to combustion written MC2: Video guidance system written MC1: Combustion experiment of a space suttle written in bot C and C++. PC2: Dynamic simulator for attitude control systems written PC3: Fligt software for eart orbiting satellite written PC4: Fligt software for eart orbiting satellite written PC5: Fligt software for eart orbiting satellite written D. Support Vector Macines: Support Vector Macines (SVM) is te one of te most widely used supervised macine learning algoritm (Hsuet al., 2003). SVM tries to map te data so tat data of different class is separated by a yper plane tat as maximum distance from nearest training data point of all classes. SVM always cooses a yper plane tat maximizes tis margin. It is matematically proved tat by maximizing te margin of separation on training data, it reduces te complexity of SVM and gives better performance on unseen data. Te optimized yper plane is te one tat minimize straining error and as maximum margin of separation. Te basis of SVM is linear discriminant functions. SVM classification function like linear discriminant functions is of form Parameters to be learnt are. If tere are only 2 classes and patterns are linearly separable ten in linear discriminant functions, function tat is learn is suc tat for patterns of one class and for patterns of oter class. Parameters ave to be learned in suc a way tat above inequalities is followed for all training examples in case of linearly separable case. Now, consider two yperplane at equal margins from ; and. Tese planes are cosen suc tat training All rigts reserved by

2 Software Fault Prediction using Macine Learning Algoritm examples tat are nearest to yperplane falls on above 2 yperplanes only (Duda et al., 2012). Figure 4 gives a clear illustration ow yperplane will look like in 2- d plane. Fig. 1: Support Lines Caracterizing te Margin Figure 1 sows training patterns from two classes and labeled as O and X respectively. Te two lines on eiter side of middle line are called as support yperplanes (in tis case of 2-d data, yperplane is a line). Te distance between yperplane and decision boundary is given by: were, f(x) =1. Similarly distance between yperplane and decision boundary is also. So total distance between te 2 supporting yperplanes is. SVM algoritm is noting but to maximize tis distance i.e. maximizing te margin between supporting yperplanes. In oter words, task is to find a w tat minimizes. For simplicity in calculation, is replaced by. Constraint is tat all X class examples sould satisfy condition and all O class examples sould satisfy. In oter words, minimum margin of all examples sould be at least from decision boundary. Lets denotes te output lable were for all X class examples and for all O class examples and denoted number of examples. Ten te SVM problem can be summarized as follows: Minimize (1) Subject to: ; SVM needs only tose examples tat fall on supporting yperplanes to learn te function. Tese training examples are called as supporting vectors. Tese examples (or vectors) are sufficient to learn bot w and b. If data is not linearly separable, ten decision boundary could be nonlinear. Figure 4 sowed an example of SVM in linearly separable case. Training examples of positive class is denoted by solid circle and negative class by ollow circle. Margin of a linear classifier is te widt of boundary (yellow strip) tat can be increased before itting a point. A linear classifier for above example and its margin is sown in Figure 5(a). (a) (b) Fig. 2: (a) A linear classifier and its margin in yellow. (b) Linear classifier wit te maximum margin Linear SVM (Moore, 2001). SVM is te maximum margin classifier as discussed above. Te maximum margin will be someting like as sown in Figure 2(b). SVM tries to make margin between supporting vectors and decision boundary as large as possible. For above example, tere are tree support vectors for te maximum margin decision boundary as sown in Figure 5(b). Te boundaries of te yellow strip are te supporting yperplanes (or lines in tis case). E. Resampling and Bootstrapping: Resampling is a tecnique of generating new patterns from te available set of patterns of a class (Wolter, 2007). Tis is a popular way of reducing sparseness of data. If te training set size is small due to reasons like unavailability of enoug patterns or ig cost of collecting patterns ten Pattern Syntesis is used to generate new patterns to add to te training set. Resampling is broadly divided into te following two types: Model based Instance based Model based resampling derives a model from te training set of patterns and uses tis model for generating new patterns. Te derived model could be a Hidden Markov Model (HMM) and or it could be a probability distribution model (Viswanatet al., 2005). Te advantage is tat artificial patterns can easily be generated once a model is in and. But if during te derivation of te model some wrong assumptions were made ten te patterns generated may be erroneous. Computational time for deriving te model can be proibitive. Moreover once te model is derived tis can be used to classify te patterns directly instead of using it to generate more patterns and ten use some oter tecnique to classify. In Instance based resampling, tere is no need to derive any model. By using te existing pattern, new patterns are derived instantaneously using some properties of te data. Te advantage of instance based resampling is tat tere is no need to any assumptions and it s very fast compared to generating a probability model first (Viswanatet al., 2005). Unlike probability distribution model, number of generated patterns is finite. Bootstrapping is one suc tecnique and is popularly used in statistics for resampling (Davison, 1997). In a popular implementation of bootstrapping, for every pattern its r-nearest neigbors are found from among te training set patterns of te same class and a weigted average of te neigbors is taken(hamamoto et al., 1997). Suc an average pattern is called a bootstrap pattern. Tis procedure can be repeated many times to generate more patterns. Tere are many variations of tis basic All rigts reserved by

3 Software Fault Prediction using Macine Learning Algoritm bootstrapping tecnique like avoiding te same pattern s use more tan once and taking weigts differently etc. Bootstrapping as been used in conjunction wit nearest neigbor classifiers only. In tis dissertation te researcer uses a novel metod wic is to use patterns tat are generated using bootstrapping instead of original software metrics training set and investigates weter tis could increase prediction accuracy of SVM for te software fault proneness prediction problem. F. Clustering: Clustering is an unsupervised learning tecnique were te given patterns are clustered based on some criteria like distance between points (Duda et al., 2012). It is te process of partitioning a set of patterns. For example, consider te following collection of nine caracters: C.a A, e c * E Te tree clusters after portioning are:, *aec C A E In te above example, size is used as te basis for clustering. Te tree clusters consist of smallest sized, lower case and upper case letters. Tus clustering is a tecnique to divide data into smaller groups of data based on some similarity features. Most commonly used similarity measure in clustering is distance. Given a set of points te aim is to divide te points into clusters based suc tat points are close to eac oter in a cluster. A popular algoritm based on inter-cluster similarity is K-means clustering algoritm in wic data points are divided into k clusters. Eac point is closer to te mean of its cluster tan te mean of oter clusters (Kanungoet al., 2000). An example is given below. Suppose tere are following 9 points in 3-D plane. { } Let te number of clusters be 3;. In above example distance between any two points in a cluster is less tan distance between any two points in different clusters. For example, distance between is 2.5 unit and distance between is around 6 units. G. Random Projection If we reduce te dimensionality of te input data by some tecnique tat preserves certain properties in te input space like distance (dissimilarity measure) or angles (similarity measure) between data points, ten we will be andling lower dimensional data. Moreover te minimum number of examples m required is also lower in te lower dimensional space. Random projection is widely used tecnique to reduce dimension. Random projection preserves all pair wise distances to witin a factor of were ( ) Let R be a random matrix used for projecting d- dimensional vectors onto k-dimensional space. If we ave n d-dimensional patterns ten A will be and B will be matrix It can be sown tat te time taken to construct te random matrix is and performing projection takes time. II. LITERATURE SURVEY Coosing te appropriate macine learning algoritm is te most important task before building any software quality model. A study to analyze various macine learning algoritms for software quality prediction on NASA MDP data sets as been done by few autors in 2008 (Vandecruyset al., 2008). Tey ave used 70% data for training and 30% for testing. SVM as also been tried. All macine learning algoritms tat tey analyzed were found to give close results. Tey also proposed a new algoritm AntiMiner+ wic was giving sligtly better results (Vandecruyset al., 2008). One major issue in te software metrics datasets is te class imbalance (Japkowicz, 2000). Class imbalance means instances of one class in muc more tan instances of oter class. In case of software metrics datasets, most of te modules are non-faulty. Hence, tere is large imbalance in number of non-faulty and faulty modules. Class imbalance is found in many datasets and not just software metrics datasets (Guo et al., 2008). In metrics dataset most of te software modules are not defective. Because of tis imbalance, problems arise in learning instance-based macine learning classifiers (Seiffertet al., 2007). Since tere is imbalance in number of software modules instances wit defects and number of instances wit no defects, accuracy of te classifier is not te best possible measure due to bias. Because of te bias, if classifier tries to increase accuracy it will also increase false alarm rate. Increase in false alarm rate will mean software modules wic are not defective would be predicted as defective. Companies ten ave to again test te modules for defects wen tere is none. Tis will increase time and cost. A researc was done to use resampling tecniques to overcome class imbalance (Pelayo& Dick, 2007). A resampling tecnique known as SMOTE wic oversamples te minority class data was used. Results of te researc were encouraging. For few selected datasets, tere was around 25% improvement in accuracy. In one anoter researc, empirical evaluation of five resampling tecniques was done to overcome class imbalance (Afzal et al., 2012). Bootstrapping gave better performance compared to oter resampling metods for four datasets but overall tere was no significant improvement. Combination of different classifiers for predicting software quality was also tried to improve performance (Tosun et al., 2008). Autors ave used tree classifiers and decision was taken on te majority vote. Tis means if two of te tree classifiers predict one ting ten tat will be te overall result. Te classifiers autors used were simple Naïve Bayes, Artificial Neural Networks (ANN) and Voting feature intervals (VFI). He applied te combination of tree classifiers on NASA MDP datasets. Tere was no significant improvement in performance. But on non-nasa datasets, performance improved sligtly. Te algoritm was able to predict 75% of defected modules wile using Naïve Bayes would ave given only 70% detection rate (Tosun et al., 2008). SVM was also used for software defect prediction. One suc paper evaluated SVM capability for fault All rigts reserved by

4 Software Fault Prediction using Macine Learning Algoritm proneness prediction and compared its performance wit oter classifiers. (Elis&Elis, 2008). Te results were at least as good as for oter classifiers for all datasets wit sligtly better results for couple of datasets. One more study analyzed te empirically validate te relation between software metrics and fault proneness using SVM as te learning algoritm (Sing et al., 2009). Te findings stated tat tere were few metrics wic were found to ave relation to fault proneness. For macine learning applications, data transformation tecniques could elp to acieve better performance. For software quality models also, autors ave tried to use tese transformation tecniques (Jiang et al., 2007). Tey ave used log normalization, discretization and teir combination for filtering. After filtering tey used te data on various macining learning classifiers. Performances of Naïve Bayes improved after pre-processing using discretization. Oter tan tat classifiers performance did not improve muc by pre-processing data using log normalization and discretization. So te important finding is tat tis transformation did not succeed in improving te performance of classifiers. Oter finding was tat te random forest performed better overall tan oter classifier (Jiang et al., 2007). Logistic regression was also used for fault prediction on a small data set (Denaroet al., 2002). Te data set was from antenna network configuration. Performance of logistic regression was good on tat data set wit True Positive Rate (TPR) just below 90%. TPR is te fraction of examples belonging to positive class tat are correctly classified. Tere was also one study performed on te analysis of NASA MDP data sets (Koru &Liu, 2005). It was found tat large sized software modules contain more defects. Tat is of course expected. Tey also noted tat many data sets are small sized. Hence metrics values will also be small. Tis will result in small variance between data sets of two classes and ence classification becomes difficult. To prove tis, tey decided to divide sets of instances of modules into different sizes. Different classifiers were used on eac set and te result was tat modules wit large size performed better tan smaller sized modules(koru &Liu, 2005). Selecting best subset of metrics is a good way to eliminate unwanted and redundant metrics. Tis process is called as feature selection. Principle Component Analysis (PCA) is a popular algoritm to select best subset of features and as been tried for software metrics datasets also (Menzieset al.,2003). Autors ave used PCA to remove inter-correlation among te metrics. Tis was done by transforming te metrics set to a smaller dimension set. Number of metrics was reduced to 5-6 from original 20 metrics. Results were good and better performance was acieved. Anoter major issue in software metrics data is noise and outliers (Seiffertet al., 2007). In te context of macine learning, noise means corruption, redundancy or any oter imperfection in data tat may impact te learning algoritm s performance. Noise may appear because a faulty module was not detected in testing and ence during metrics collection it was labeled as non-faulty. If in a dataset, tere are one or more software metrics tat sows very little variation across te faulty and non-faulty models, ten tese would also be considered as noise. Anoter example of noise would be two or more metrics sowing nearly identical variation across all instances (Kosgoftaaret al., 2005). Tis would result in redundant information from some metrics. Next is te problem of outliers. Outliers are related to measurement errors. An instance in dataset is termed as outlier if values of its features are very different from oter instances from te same class. A researc paper based on Random Forests sowed more robustness to noise and outliers (Guo et al., 2004).Autors ave used trade off between accuracy and recall using a parameter called cut-off. Tey also used Random Forests for feature selection and selected five best features. But tat did not improve te performance of te classifier. Te model s performance was also compared wit oter macine learning classifiers. It was found tat Random Forests give sligtly better performance (Guo et al., 2004). III. PROPOSED WORK Te proposed algoritm is described below: Two variations of bootstrapping were used to pre-process te metrics data. In te first metod nearest neigbors of eac dataset example were found in te input space. In te second metod nearest neigbors were found in te kernel space. Bot te algoritms are given below altoug only algoritm 1 will be used in tis dissertation. A. Bootstrapping in Input Space Algoritm 1:Bootstrapping in Input Space Output: Bootstrapped metrics data set Bootstrapping in Kernel Space Finding neigbors in te kernel space makes more sense. Using kernel function, faulty and non-faulty modules are linearly separated by projecting metrics into iger dimension space. Bootstrapping in te kernel space reduces te Dia and increases te Margin in te kernel space. Neigbors in te kernel space were found using kernel trick as follows: Te distance between two points and is ( ) Algoritm 2: Bootstrapping in Kernel Space All rigts reserved by

5 Software Fault Prediction using Macine Learning Algoritm Output: Bootstrapped metric data set B. Clustering Clustering can also be used as a data syntesis tecnique. Clustering is an unsupervised learning tecnique were te given training examples are clustered based on some criteria like distance between points. Clustering as been used in association wit SVMs before (Finley&Joacims,2005). A notable example is CBSVM (Yu et al., 2003). But te impact of clustering to control te VC dimension of SVM as not been discussed in te literature earlier. Te researcer presents an approac to reduce te VC dimension of SVM using clustering. Here cluster centers replace te original metrics data sets. Training of SVM takes place on tese cluster centers instead of te original metrics data set. Tis can be tougt of as a training data syntesis tecnique were te syntetic data examples are te cluster centers. It can be sown tat clustering reduces te VC dimension of SVM. Wen te data examples belonging to a class are clustered and te entire dataset is replaced by tese cluster centers, te Margin increases and te diameter decreases. Te training examples responsible for decreasing te margin or increasing te diameter are tose wic lie at te boundaries. Wen metrics data set is clustered tese points are sifted towards te centre of te metrics data set. Tey are not present temselves in te final data set (te cluster centres form te final data set)but te centre to wose cluster tese points belong is influenced by eac point. Tus if te cluster as a boundary data example, ten it will pull te cluster centre towards te boundary but oter inner data examples also pull te centre. Tus every data example as its influence on te centre, and te sape of te boundary is not lost due to clustering. Te boundary just srinks and all te data examples wic lie outside tis new boundary are effectively removed. Most probably tese outside data examples are noise. Tus clustering takes care of noisy data removal. Clustering brings out te true boundary of te data set by filtering out te noisy data. If required, we can eliminate points in small size clusters from te data sets. Wen te training data is clustered, te cluster centroid is average of all data examples belonging to tat cluster. If te size of every cluster is more tan one ten tis average pattern lies closer to te centroid of te entire training set tat te fartest pattern in te training set. Wen te cluster centres replace te training set te new Dia and Margin are smaller tan te corresponding values of te original training set. Let te number of clusters in eac class after clustering be k. Let te cluster centres be in class 1 and in class 2. Let te points belonging to cluster be and similarly for oter clusters in class 2. Ten we ave clusters centres. Let te new Dia and Margin be Now we sow tat In te case of Margin too, te proof follows in similar fasion. IV. EXPERIMENT RESULT Te improved approac of using bootstrapping and clustering wit SVM as many advantages compared to previous approac: A. Experiment: First normalization is done on all datasets. Ten outliers are removed using Weka Outlier removal filter. Ten for 10- fold cross validation, dataset is broken into 10 equal subsets. 9 sets are used for training and rest one for testing. Before dividing datasets, randomize option is first selected in Weka to randomize te data. Te training set s minority class, i.e. faulty modules are oversampled by using SMOTE. Ten bootstrapping is applied to te oversampled training set separately for eac class. Bootstrapping will give output dataset for eac class wic is te combined to make a single set before applying SVM. After tat six SVM models wit RBF kernels are built by varying gamma and cost factor. Ten models are testing on test data. Procedure is repeated for all 10 folds. An example of using SMOTE in Weka is sown in Figure 3. Te dataset is one fold on CM1 after outlier removal. Before applying SMOTE, number of defective modules was 30 as sown in Figure 3(a). Te effect of applying SMOTE on CM1 is sown in Figure 3(b). SMOTE percentage used was 300% wic increase te number of defective modules to 120. Table 1 gives te SMOTE percentage for eac dataset used in tis experiment. Fig. 3(a) Fig. 3(b) Fig. 3: (a)smote filter in Weka. (b) Effect of SMOTE on CM1 dataset wit percentage parameter as 300. Dataset SMOTE Percentage CM1 300 JM1 100 KC1 200 KC3 150 All rigts reserved by

6 Software Fault Prediction using Macine Learning Algoritm MC1 300 MC2 0 MW1 300 PC1 300 PC2 300 PC3 250 PC4 300 PC5 300 Table 1: SMOTE percentage for different datasets Finally an example of comparison of Cyclomatic Density values before and after bootstrapping is sown below by Figure 4 for CM1 dataset. Fig. 4: Design Complexity Metric (a) before bootstrapping b) after bootstrapping As can been seen by Figure 17 tat standard deviation as come down wic is expected wile taking any kind of averages. V. CONCLUSION By using different filtering tecniques and various preprocessing and transformation tecniques like bootstrapping, clustering, random projection, large number of fault prediction problem are solved using te SVM(software vector macine) Algoritm and te datasets used are te popular NASA Metric Data Program datasets. From te result te conclusions are: 1) Tere is very significant improved in fault prediction of models built after using bootstrapping and clustering as data transformation. 2) Performance of fault proneness prediction model increases wen a subset of metrics based on a correlation measure is selected instead of all metrics for building te model. prediction?. Journal of Empirical Software Engineering, vol. 32, no. 2, pp [5] Guo, L., Ma, Y., Cukic, B., & Sing, H. (2004).Robust prediction of fault-proneness by random forests. Software Reliability Engineering, 2004.ISSRE t International Symposium on IEEE, pp [6] Koru, A. G., & Liu, H. (2005).An investigation of te effect of module size on defect prediction using static measures.acm SIGSOFT Software Engineering Notes.ACM, vol. 30, no. 4, pp [7] Lanza, M., Marinescu, R., &Ducasse, S. (2006). Object-oriented metrics in practice. Heidelberg: Springer, pp [8] Jiang, Y., Cukic, B., & Menzies, T. (2007). Fault prediction using early lifecycle data. Software Reliability, 2007.ISSRE'07.Te 18t IEEE International Symposium on. IEEE, pp [9] Kotsiantis, S. B. (2007). Supervised Macine Learning: A Review of Classification Tecniques. Informatica, vol. 31, pp [10] Elis, K. O., &Elis, M. O. (2008). Predicting defectprone software modules using support vector macines. Journal of Systems and Software, vol. 81, no. 5, pp [11] Gondra, I. (2008). Applying macine learning to software fault-proneness prediction. Journal of Systems and Software, vol. 81, no. 2, pp [12] Guo, X., Yin, Y., Dong, C., Yang, G., & Zou, G. (2008).On te class imbalance problem. Natural Computation, 2008.ICNC'08.Fourt International Conference on. IEEE, vol. 4, pp [13] Catal, C., &Diri, B. (2009). A systematic review of software fault prediction studies. Expert systems wit applications, vol. 36, no. 4, pp [14] Pressman, R. S. (2010) Software Engineering: A Practitioner's Approac. 8t Ed. New York: McGraw- Hill. REFERENCES [1] Japkowicz,N.(2000). Te class imbalance problem: Significance and strategies. Proc. of te Int l Conf. on Artificial Intelligence. [2] Denaro, G., Morasca, S., &Pezzè, M. (2002).Deriving models of software fault-proneness. Proceedings of te 14t international conference on Software engineering and knowledge engineering.acm, pp [3] Dijkstra, E. W. (2002). Go to statement considered armful. Software pioneers. Springer Berlin Heidelberg, pp [4] Menzies, T., Ammar, K., Nikora, A., & Stefano, S. (2003). How simple is software defect All rigts reserved by