An Optimized Analogy-Based Project Effort Estimation

Transcription

1 (IJACSA) Internatonal Journal of Advanced Computer Scence and Applcatons, An Optmzed Analogy-Based Project Effort Estmaton Mohammad Azzeh Faculty of Informaton Technology Appled Scence UnverstyAmman, Jordan POBOX 166 Yousef Elshekh Faculty of Informaton Technology Appled Scence Unversty Amman, Jordan Marwan Alsed Faculty of Informaton Technology Appled Scence Unversty Amman, Jordan Abstract Despte the predctve performance of Analogy- Based Estmaton (ABE) n generatng better effort estmates, there s no consensus on: (1) how to predetermne the approprate number of analoges, (2) whch adjustment technque produces better estmates. Yet, there s no pror works attempted to optmze both number of analoges and feature dstance weghts for each test project. Perhaps rather than usng fxed number, t s better to optmze ths value for each project ndvdually and then adjust the retreved analoges by optmzng and approxmatng complex relatonshps between features and reflects that approxmaton on the fnal estmate. The Artfcal Bees Algorthm s utlzed to fnd, for each test project, the approprate number of closest projects and features dstance weghts that are used to adjust those analoges efforts. The proposed technque has been appled and valdated to 8 publcally datasets from PROMISE repostory. Results obtaned show that: (1) the predctve performance of ABE has notceably been mproved; (2) the number of analoges was remarkably varable for each test project. Whle there are many technques to adjust ABE, Usng optmzaton algorthm provdes two solutons n one technque and appeared useful for datasets wth complex structure. Keywords Cost Estmaton; Effort Estmaton by Analogy; Bees Optmzaton Algorthm I. INTRODUCTION Analogy-Based Estmaton (ABE) has preserved popularty wthn software engneerng research communty because of ts outstandng performance n predcton when dfferent data types are used [1, 15]. The dea behnd ths method s rather smple such that the new project s effort can be estmated by reusng efforts about smlar, already documented projects n a dataset, where n a frst step one has to dentfy smlar projects whch contan the useful predctons [15]. The predctve performance of ABE reles sgnfcantly on the choce of two nterrelated parameters: number of nearest analoges and adjustment strategy [8]. The goal of usng adjustment n ABE s twofold: (1) mnmzng the dfference between a new project and ts nearest analoges, and (2) producng more successful estmates n comparson to orgnal ABE [2]. If the researchers read the lterature on ABE, they wll encounter large number of ABE models that use varety of adjustment strateges. Those strateges suffer from common problems such as they are not able to produces stable results when appled n dfferent contexts as well as they use fxed number of analoges for the whole dataset [1]. Usng fxed number of analoges has been proven to be unsuccessful n many stuatons because t depends heavly on expert opnon and requres extensve expermentaton to dentfy the best k value, whch mght not be predctve for ndvdual projects [2]. The am of ths work s therefore to propose a new method based on Artfcal Bees Algorthm (BA) [14] to adjust ABE by optmzng the feature smlarty coeffcents that mnmzes dfference between new project and ts nearest projects, and predctng the best k number of nearest analoges. The paper s structured as follows: Secton 2 ntroduces an overvew to ABE and adjustment methods. Secton 3 presents the proposed adjustment method. Secton 4 presents research methodology. Secton 5 shows obtaned results. Fnally the paper ends wth our conclusons. II. RELATED WORKS ABE method generates new predcton based on assumpton that smlar projects wth respect to features descrpton have smlar efforts [8, 15]. Adjustment s a part of ABE that attempts to mnmze the dfference between new observaton ( ê ) and each nearest smlar observaton ( e ), then reflects that dfference on the derved soluton n order to obtan better soluton ( e t ). Consequentally, all adjusted solutons are aggregated usng smple statstcal methods such 1 k as mean ( et k eˆ 1 ). In prevous study [17] we nvestgated the performance of BA, on adjustng ABE and fndng best k value for the whole dataset. Ths model showed some mprovements on the accuracy, but on the other sde t dd not solve the problem of predctng the best k value for each ndvdual project. In addton the soluton space of BA was a challenge because there was only one common weght for all nearest analoges. The used optmzaton crteron (.e. MMRE) was problematc because t was proven to be based towards underestmaton. For all these reason and snce we need to compare our proposed model wth valdated and replcated models, we excluded ths model from comparson later n ths paper. Ths paper thereby attempts to solve abovementoned lmtatons. In lterature there s a sgnfcant number of adjustment methods that have been documented and replcated n prevous studes. Therefore we selected and summarzed only the most wdely used strateges. Walkerden and Jeffery proposed Lnear Sze Adjustment () [16] based on the sze extrapolaton. Mendes et al. [12] proposed Multple Lnear Feature Extrapolaton () to nclude all related sze features. 6 P a g e

2 (IJACSA) Internatonal Journal of Advanced Computer Scence and Applcatons, Jorgenson et al. [6] proposed Regresson Towards the Mean () to adjust projects based on ther productvty values. Chu and Huang [4] proposed another adjustment based on Genetc Algorthm () to optmze the coeffcent αj for each feature dstance based on mnmzng performance measure. Recently, L et al. [10] proposed the use of Neural Network () to learn the dfference between projects and reflects the dfference on the fnal estmate. Further detals about these methods and ther functons can be found n [1]. Indeed, the most mportant questons to consder when to use such methods s how to predct the best number of nearest analoges (k). In recent years varous approaches have been proposed to specfy ths number such as: 1) fxed number selecton (.e. k=1, 2, 3 etc) as n studes of [7, 11, 12, 16], 2) Dynamc selecton based on clusterng as n study of [2, 17]. 3) Smlarty threshold based selecton as n studes of [5, 9]. Generally, these studes except [2] use the same k value for all projects n the dataset whch does not necessarly produce best performance for each ndvdual project. On the other hand, the certan problem wth [2] s that t does not nclude adjustment method but t predcts the best k value based on the structure of dataset. III. THE PROPOSED METHOD () The proposed adjustment method starts wth Bees Algorthm n order to fnd out, for each project: (1) the feature weghts (w), and (2) the best k number of nearest analoges that mnmze mean absolute error. The search space of BA can be seen as a set of n weght matrxes where the sze of each matrx (.e. soluton) s k m. That means each possble soluton contans weght matrx wth dmenson equvalent to the number of analoges (k) and number of features (m) as shown n Fgure 1. The number of rows (.e. k) and weght values are ntally generated by random. Each row represents m weghts for one selected analogy and accordngly w j 1 j 1. In each run the algorthm selects the top k nearest analoges based on the number of k weghts n the search space. Then each selected analogy s adjusted wth correspondng weghts taken from the matrx w as shown Eq.1. The algorthm contnues searchng untl the value of Mean Error (.e. MR 1 k k j 1 j ) between new project and ts k analoges s mnmzed. The optmzed k value and weght matrx are then appled to Eqs. 1, 2 and 3 to generate new estmate. The new ntegraton between ABE wth BA wll be called Optmzed Analogy Based Estmaton (hereafter ). Fg. 1. w11 w21 w wk1 w12 w22 wk 2 w1 m w2m wkm Weght Matrx for one soluton n the search space 1 m j w f - f j j ( tj j) m 1 (1) eˆ e j (2) k k r eˆ e 1 1 t k (3) 1 The settng parameters for AB have been found after performng senstvty analyss on the employed datasets to see the approprate values. Table I shows BA parameters, ther abbrevatons and ntal values used n ths study. Below we brefly descrbe the process of BA n fndng best k values and the correspondng weghts for each new project. The algorthm starts wth an ntal set of weght matrxes generated after randomly ntalzng k for each matrx. The solutons are assessed and sorted n ascendng order after they are beng evaluated based on MR. The best from 1 to b solutons are beng selected for neghborhood search for better solutons, and form new patch. Smlarly, a number of bees (nsp) are also recruted for each soluton ranked from b+1 to u, to search n the neghborhood. The best soluton n each patch wll replace the old best soluton n that patch and the remanng bees wll be replaced randomly wth other solutons. The algorthm contnues searchng n the neghborhood of the selected stes, recrutng more bees to search near to the best stes whch may have promsng solutons. These steps are repeated untl the crteron of stop (mnmum MR) s met or the number of teraton has fnshed. TABLE I. BA PARAMETERS Parameter Descrpton Value q dmenson of soluton (number of features +1) n represents sze of ntal solutons 100 u b number of stes selected out of n vsted stes number of best stes out of s selected stes nep number of bees recruted for best b stes 30 nsp Number of bees recruted for the other selected stes ngh ntal sze of patches (ngh) 0.05 IV. METHODOLOGY A. Datasets The proposed model has been valdated over 8 software effort estmaton datasets come from companes of dfferent ndustral sectors [3]. The datasets characterstcs are provded n Table II whch shows that the datasets are strongly postvely skewed ndcatng many small projects and a lmted number of outlers. It s mportant to note that all contnuous features have been scaled and all observaton wth mssng values are excluded P a g e

3 (IJACSA) Internatonal Journal of Advanced Computer Scence and Applcatons, TABLE II. DESCRIPTIVE STATISTICS OF THE DATASETS Dataset Feature Sze Effort Data Mn Max Mean Skew Albrecht Kemerer Nasa Desharnas COCOMO Chna Maxwell Telecom B. Performance measures A key queston to any estmaton model s whether the predcatons are accurate, the dfference between the actual effort ( e ) and the predcted effort ( ê ) should be as small as possble because large devaton wll have opposte effect on the development progress of the new software project [13]. Ths secton descrbes several performance measures used n ths research as shown n Table III. Although some measures such as MMRE, MMER have been crtczed as based to under and over estmatons, we nsst to use them because they are wdely used n commentng on the success of predctons [13]. TABLE III. ERROR MEASURES Error Measure Name Equaton e eˆ Magntude Relatve Error MRE e Mean Magntude MMRE N 1 Relatve Error MRE Medan Magntude MdMRE medan ( MRE) Relatve Error Mean Magntude of Error e Relatve to the estmate eˆ MMER N 1 eˆ Mean Balanced Error 1 e eˆ MBER N (MBRE) mn, ˆ e e 100 N Predcton Performance 1 f MRE 0.25 pred l N 1 0 otherwse Interpretng these error measures wthout any statstcal test can lead to concluson nstablty, therefore we used wnte-loss algorthm [8] to compare the performance of to other estmaton methods. We frst check f two methods method ; method j are statstcally dfferent accordng to the Wlcoxon test. If so, we update wn ; wn j and loss ; loss j after checkng whch one s better accordng to the performance measure at hand; otherwse we ncrease te and te j. The performance measures used here are MRE, MMRE, MdMRE, MMER, MBER and Pred 25. Algorthm 1 llustrates the wn-teloss algorthm [8]. Also, the Bonferron-Dunn test s used to perform multple comparsons for dfferent models based on the absolute error to check whether there are dfferences n populaton rank means among more than populatons. Algorthm 1. Pseudocode of wn-te-loss algorthm betweenmethod and method jbased on performance measure E [8] 1: Wn =0,te =0,loss =0 2: Wn j=0,te j=0;loss j=0 3: f Wlcoxon (MRE(method ), MRE(method j), 95) says they are the same then 4: te = te + 1; 5: te j = te j + 1; 6:else 7: f better(e(method ), E(method j)) then 8: wn = wn + 1 9: loss j = loss j : else 11: wn j = wn j : loss = loss :end f 14: end f V. RESULTS Ths secton presents performance fgures of aganst varous adjustment technques used n constructng ABE models. Snce the selecton of the best k settng n s dynamc, there was no need to pre-set the best k value. In contrast, for other varants of adjustment technques there was necessarly fndng the best k value that almost fts each model, therefore we appled dfferent k settngs from 1 to 5 on each model as suggested by L et al. [9] and the settng that produces best overall performance has been selected for comparson wth other dfferent models. Tables IV, V, VI, VII and VIII summarze the resultng performance fgures for all nvestgated ABE models. The most successful method should have lower MMRE, MdMRE, MMER, MBER and hgher Pred 25. The obtaned results suggest that the produced accurate predctons than other methods wth qute good performance fgures over most datasets. TABLE IV. MMRE RESULTS Dataset Albrecht Kemerer Desharnas COCOMO Maxwell Chna Telecom Nasa TABLE V. PRED 25 RESULTS Dataset Albrecht Kemerer Desharnas COCOMO Maxwell Chna Telecom Nasa TABLE VI. MDMRE RESULTS Dataset Albrecht Kemerer Desharnas COCOMO Maxwell Chna Telecom Nasa P a g e

4 (IJACSA) Internatonal Journal of Advanced Computer Scence and Applcatons, However, these fndngs are ndcatve of the superorty of BA n optmzng k analoges and adjustng the retreved project efforts, and consequentally mprove overall predctve performance of ABE. Also from the obtaned results we can observe that there s evdence that usng adjustment technques can work better for datasets wth dscontnutes (e.g. Maxwell, Kemerer and COCOMO). Note that the result s exactly the searchng for the best k settng result as mght be predcted by the researchers mentoned n the related work. We speculate that pror Software Engneerng researchers who faled to fnd best k settng, dd not attempt to optmze ths k value wth adjustment technque tself for each ndvdual project before buldng the model. TABLE VII. MMER RESULTS Dataset Albrecht Kemerer Desharnas COCOMO Maxwell Chna Telecom Nasa TABLE VIII. MBRE RESULTS Dataset Albrecht Kemerer Desharnas COCOMO Maxwell Chna Telecom Nasa Furthermore, two results worth some attenton whle we are carryng ths experment: Frstly, the general trend of predctve accuracy mprovements across all error measures, overall datasets s not clear ths certanly depends on the structure of the dataset. Secondly, there s no consstent results regardng the sutablty of for small datasets wth categorcal features (as n Maxwell and Kemerer datasets) but we can nsst that s stll comparable to n terms of MMRE and Pred 25 and have potental to produce better estmates. In contrast, showed better performance than for the other two small datasets (NASA and Telecom) that do not have categorcal features. To summarze the results we run the wn-te-loss algorthm to show the overall performance. Fgure 3 shows the sum of wn, te and loss values for all models, over all datasets. Every model n Fgure 2 s compared to other fve models, over sx error measures and eght datasets. Notce n Fgure 2 that except the low performng model on, the te values are n band. Therefore, they would not be so nformatve as to dfferentate the methods, so we consult wn and loss statstcs to tell us whch model performs better over all datasets usng dfferent error measures. Apparently, there s sgnfcant dfference between the best and worst models n terms of wn and loss values (n the extreme case t s close to 119). The wn-te-loss results offer yet more evdence for the superorty of over other adjustment technques. Also the obtaned wn-te-loss results confrmed that the predctons based on model presented statstcally sgnfcant but necessarly accurate estmatons than other technques. Two aspects of these results are worth commentng: 1) The was the bg loser wth bad performance for adjustment. 2) technque performs better than whch shows that usng sze measure only s more predctve than usng all sze related features. We use the Bonferron-Dunn test to compare the method aganst other methods as shown n Fgure 3. The plots have been obtaned after applyng ANOVA test followed by Bonferron test. The ANOVA test results n p-value close to zero whch mples that the dfferences between two methods are statstcally sgnfcant based on AR measure. The horzontal axs n these fgures corresponds to the average rank of methods based on AR. The dotted vertcal lnes n the fgures ndcate the crtcal dfference at the 95% confdence level. Obvously, the methods generated lower AR than other methods over most datasets except for small datasets. For such datasets, all models except generated relatvely smlar estmates but wth preference to that has smaller error. Ths ndcates that adjustment method s far less prone to ncorrect estmates. Fg. 2. Wn-Te-Loss Results for all Models VI. CONCLUSIONS AND FUTURE WORK Ths paper presents a new adjustment technque to tune ABE usng Bees optmzaton algorthm. The BA was used to automatcally fnd the approprate k value and ts feature weghts n order to adjust the retreved k closest analoges. The results obtaned over 8 datasets showed sgnfcant mprovements on predcton accuracy of ABE. We can notce that all models rankng can change by some amount but has relatvely stable rankng accordng to all error measure as shown n Fgure 2. Future work s planned to study the mpact of usng ensemble adjustment technques. VII. ACKNOWLEDGEMENT The authors are grateful to the Appled Scence Prvate Unversty, Amman, Jordan, for the fnancal support granted to ths research project. 9 P a g e

5 (IJACSA) Internatonal Journal of Advanced Computer Scence and Applcatons, groups have mean ranks sgnfcantly dfferent from (a) Albrecht dataset groups have mean ranks sgnfcantly dfferent from (b) Kemere dataset groups have mean ranks sgnfcantly dfferent from (c) Desharnas dataset groups have mean ranks sgnfcantly dfferent from (d) COCOMO dataset groups have mean ranks sgnfcantly dfferent from (e) Maxwell dataset groups have mean ranks sgnfcantly dfferent from (f) Chna dataset The mean ranks of groups and are sgnfcantly dfferent (g) Telecom dataset Fg. 3. Bonferron-Dunn test multple comparson test over all datasets REFERENCES [1] M. Azzeh, A replcated assessment and comparson of adaptaton technques for analogy-based effort estmaton, Journal of Emprcal Software Engneerng vol. 17, pp , [2] M. Azzeh, Y. Elshekh, Learnng Best K analoges from Data Dstrbuton for Case-Based Software Effort Estmaton, The Seventh Internatonal Conference on Software Engneerng Advances (ICSEA 2012), pp , [3] G. Boettcher, T. Menzes, T. Ostrand, PROMISE Repostory of emprcal software engneerng data repostory, West Vrgna Unversty, Department of Computer Scence The mean ranks of groups and are sgnfcantly dfferent (h) NASA dataset [4] N. H. Chu, S. J. Huang, The adjusted analogy-based software effort estmaton based on smlarty dstances, Journal of System and Software, Vol. 80, pp , [5] A. Idr, A. Abran, T. Khoshgoftaar, Fuzzy Analogy: a New Approach for Software Effort Estmaton, 11th Internatonal Workshop n Software Measurements, pp , [6] M. Jorgensen, U. Indahl, D. Sjoberg, Software effort estmaton by analogy and regresson toward the mean, Journal of System and Software, vol. 68, pp , [7] C. Krsopp, E. Mendes, R. Premraj, M. Shepperd, An emprcal analyss of lnear adaptaton technques for case-based predcton, Internaton 10 P a g e

6 (IJACSA) Internatonal Journal of Advanced Computer Scence and Applcatons, conference on Case-Based Reasonng Research and Development, pp , 2003 [8] E. Kocagunel, T. Menzes, A. Bener, J. Keung, Explotng the Essental Assumptons of Analogy-based Effort Estmaton, Journal of IEEE transacton on Software Engneerng, vol. 38, pp , [9] J. Z. L, G. Ruhe, A. Al-Emran, M. Rchter, A flexble method for software effort estmaton by analogy, Journal of Emprcal Software Engneerng, vol. 12, pp , [10] Y. F. L, M. Xe, T. N. Goh, A study of A study of the non-lnear adjustment for analogy based software cost estmaton, Journal of Emprcal Software Engneerng, vol. 14, pp , [11] U. Lpowezky, Selecton of the optmal prototype subset for 1-nn classfcaton, Pattern Recog. Letters, vol. 19, pp , [12] E. Mendes, N. Mosley, S. Counsell, A replcated assessment of the use of adaptaton rules to mprove Web cost estmaton, Internatonal Symposum on Emprcal Software Engneerng, pp , [13] I. Myrtvet, E. Stensrud, M. Shepperd, Relablty and valdty n comparatve studes of software predcton models, Journal of IEEE Transacton on Software Engneerng, vol. 3, pp , [14] D. T. Pham, A. Ghanbarzadeh, E. Koç, S. Otr, S. Rahm, M. Zad, The Bees Algorthm A novel tool for complex optmsaton problems, Proceedngs of the 2nd Vrtual Internatonal Conference on Intellgent Producton Machnes and Systems, pp , [15] M. Shepperd, C. Schofeld, Estmatng software project effort usng analoges, Journal of IEEE Transacton on Software Engneerng, vol. 23, pp , [16] F. Walkerden, D. R. Jeffery, An emprcal study of analogy-based software effort Estmaton, Journal of Emprcal Software Engneerng, vol. 4, pp , [17] M. Azzeh, "Adjusted case-based software effort estmaton usng bees optmzaton algorthm. Internatonal conference on Knowlege-Based and Intellgent Informaton and Engneerng Systems. Sprnger BerlnHedelberg, pp , P a g e