Should Duration and Team Size be Used for Effort Estimation?

Transcription

1 Should Duraton and Team Sze be Used for Effort Estmaton? Takesh Kakmoto 1, Masateru Tsunoda 2, and Akto Monden 3 Abstract Project management actvtes such as schedulng and project progress management are mportant to avod project falure. As a bass of project management, effort estmaton plays a fundamental role. To estmate software development effort by mathematcal models, varables whch are fxed before the estmaton are used as ndependent varables. Some studes used team sze and project duraton as ndependent varables. Although they are sometmes fxed because of the lmtaton of human resources or busness schedule, they may change by the end of the project. For nstance, when delvery s delayed, actual duraton and estmated duraton s dfferent. So, although usng team sze and project duraton may enhance estmaton accuracy, the error may also lower the accuracy. To help practtoners to select ndependent varables, we analyzed whether team sze and duraton should be used or not, when we consder the error ncluded n the team sze and the duraton. In the experment, we assumed that duraton and team sze nclude errors when effort s estmated. To analyze nfluence of the errors, we add n% errors to duraton and team sze. As a result, usng duraton as an ndependent varable was not very effectve n many cases. In contrast, usng maxmum team sze as an ndependent varable was effectve when the error rate s equal or less than 50%. Key words software effort predcton, project management, productvty, estmaton error 1 Takesh Kakmoto Department of Electrcal and Computer Engneerng, Natonal Insttute of Technology, Kagawa College, Takamatsu, Japan. e-mal: kakmoto@t.kagawa-nct.ac.jp 2 Masateru Tsunoda Department of Informatcs, Knda Unversty, Hgashosaka, Japan. e-mal: tsunoda@nfo.knda.ac.jp 3 Akto Monden Graduate School of Natural Scence and Technology, Okayama Unversty, Okayama, Japan. e-mal: monden@okayama-u.ac.jp

2 2 1 Introducton As recent software systems grow n sze and complexty, project management actvtes such as staffng, schedulng and project progress management are becomng ncreasngly mportant to avod project falure (cost overrun and/or delayed delvery). As a bass of project management, effort estmaton plays a fundamental role; therefore, accurate effort estmaton s vtal to organzaton s proftablty. To date, varous estmaton models that use past projects hstorcal data have been proposed [2][23][25]. One of the most commonly used estmaton models s a lnear regresson model, whch represents the relatonshp between the dependent varable (.e. effort) and ndependent varables such as functonal sze, archtecture, programmng language, and so on. Analogy based estmaton [23] s one of major estmaton methods, and many proposals and case studes have been reported [8][9][20][27][31]. Analogy based estmaton selects projects (neghborhood projects) whch are smlar to the estmated project from past project dataset, and estmates effort based on smlar projects effort. One of the advantages of analogy based estmaton s that estmaton results are comprehensble for estmators such as project managers [31], because they can confrm neghborhood projects used for estmaton. To estmate software development effort by mathematcal models, varables whch are fxed before the estmaton are used as ndependent varables. Effort s estmated on the early phase of projects,.e., after basc desgn phase. For example, archtecture and programmng language are fxed after the phase, and they are often used as ndependent varables of estmaton models. In contrast, varables whch are not fxed before the estmaton cannot be used as the ndependent varables. Some studes used team sze [7][16] and project duraton [1][12][13] as ndependent varables. However, they are not always fxed before the estmaton. They are occasonally fxed after estmaton. For example, when estmated effort s 9 person-months, duraton s set as 9 and team sze s set as 3. Sometmes, they are fxed because of the lmtaton of human resources or busness schedule. But they may change by the end of the project. For nstance, when delvery s delayed, actual duraton (fxed after project s fnshed) and estmated duraton (value nput to the model) s dfferent. Generally, estmaton model s made based on the actual duraton of past projects, and therefore nput value (estmated effort) would nclude errors, as shown n Fgure 1. Therefore, although usng team sze and project duraton may enhance estmaton accuracy, the error may also lower the accuracy. The goal our study s to help practtoners to select ndependent varables when they buld effort estmaton models. So, we analyzed whether team sze and duraton should be used or not, when we consder the error ncluded n the team sze and the duraton. To clarfy the purpose of the analyss, we set followng research questons: RQ1: Is duraton effectve to mprove estmaton accuracy? RQ2: Is team sze effectve to mprove estmaton accuracy?

3 3 Past projects Sze Effort Team Sze (Actual) Past project A Past project B Past project C Past project D Estmaton model Buld Input Estmaton target project Sze Effort Team Sze (Estmated) Target project V Actual team sze s 8 (Fxed at the end of the project). Fgure 1 An example of the error ncluded n team sze RQ3: At what error rate s estmaton accuracy negatvely affected? Secton 2 explans effort estmaton methods used n the experment. Secton 3 descrbes the expermental settng, and secton 4 shows results of the experment. Secton 5 explans related work, and Secton 6 concludes the paper. 2 Effort Estmaton Methods 2.1 Multple Lnear Regresson Model The multple lnear regresson model s wdely used when estmatng software development effort mathematcally. The model s bult based on ordnary least squares. When the effort s denoted as y, and ndependent varables such as software sze are denoted as x 1, x 2,, x k (k s the number of ndependent varables), y s explaned as: y = β 0 + β 1x 1 + β 2x β kx k + ε (1) In the equaton, β 0 s an ntercept, β 1, β 2,, β k are partal regresson coeffcents, and ε s an error term. As a rule of thumb, to buld a proper model usng lnear regresson analyss, t s needed that the number of data ponts s fve to ten tmes larger than the number of ndependent varables.

4 4 Table 1 Dataset used on analogy based effort estmaton Varable1 Varable2 Varablej Varablel p1 m11 m12 m1j m1l p2 m21 m22 m2j m2l p m1 m2 mj ml pk mk1 mk2 mkj mkl When buldng the model, log-transformaton s appled to enhance the accuracy of the model [10]. Ths s because the dstrbutons of some varables such as effort and software sze are log-normal dstrbuton. 2.2 Analogy Based Estmaton The orgn of analogy based estmaton s CBR (case based reasonng), whch s studed n artfcal ntellgence feld. Shepperd et al. [23] appled CBR to software development effort estmaton. CBR selects a case smlar to current ssue from accumulated past cases, and apples soluton of the case to the ssue. CBR assumes smlar ssues can be solved by smlar soluton. Analogy based estmaton assumes neghborhood (smlar) projects (For example, development sze and used programmng language s smlar) have smlar effort, and estmates effort based on neghborhood projects effort. Although ready-made estmaton models such as COCOMO [2] can make estmaton wthout stored software project dataset, analogy based estmaton cannot estmate wthout t. It s a weak pont of analogy based estmaton, but t can be overcome by usng publc dataset. Analogy based estmaton uses k l matrx shown n Table I. In the matrx, p s -th project, m j s j-th varable. That s, each row denotes a data pont (.e., a project), and each columns denotes a metrc. We presume p a s estmaton target project, and s the estmated value of m ab. Procedures of analogy based estmaton consst of the three steps descrbed below. Step 1: Snce each varable has dfferent range of value, ths step makes the ranges [0, 1]. The value m j, normalzed the value of m j s calculated by: mˆ ab j mnm j m mnm m m' j (2) max j j In the equaton, max(m j) and mn(m j) denote the maxmum and mnmum value of m j respectvely. The equaton s one of the commonly used methods to normalze the range of a value [26]. Step 2: To fnd projects whch are smlar to estmated project p a (.e., dentfyng neghborhood projects), smlarty between p a and other projects p s calculat-

5 ed. Varables of p a and p are used as elements of vectors, and cosne of the vectors are regarded as smlarty. Smlarty sm(p a, p ) between p a and p s calculated by: sm p, p a m' aj avgm' j m ' j avgm' j jm a M m' aj avgm' j m' j avgm' j jm a M jm a M In the equaton, M a and M are set of varables measured n project p a and p respectvely. avg(m, j) s average of -th varable. The range of sm(p a, p ) s [-1, 1]. Step 3: The estmated effort of project p a s calculated by actual effort of k neghborhood projects. Whle average of neghborhood projects effort s generally used, we adopt sze adjustment method, whch showed hgh estmaton accuracy n some studes [9][20][31]. Estmated value mˆ ab s calculated by: mˆ ab b k nearestprojects amp( p, p ) a m amp p, p sm p, p a k nearestprojects fpa fp a sm p, p a In the equaton, fp a and fp are software sze of project p a and p respectvely. Sze adjustment method assumes effort s s tmes (s s real number greater than 0) larger when software sze s s tmes larger. The method adjusts effort of p based on rato of target project s sze fp a and neghborhood project s sze fp. (3) (4) (5) 5 3 Experment 3.1 Datasets We used the ISBSG [6], Ktchenham [11], and Desharnas datasets [5]. Nomnal scale varables were transformed nto dummy varables (e.g. f the varable has n categores, t s transformed nto n - 1 dummy varables). We removed dummy varables when the number of cases whch correspond wth the category was very small. The ISBSG dataset s provded by the Internatonal Software Benchmark Standard Group (ISBSG), and t ncludes project data collected from software development companes n 20 countres [6]. The dataset (Release 9) ncludes 3026 projects that were carred out between 1989 and 2004, and 99 varables were recorded. The ISBSG dataset ncludes low qualty project data (Data qualty ratngs are also ncluded n the dataset). We extracted projects based on the prevous study [15] (Data qualty ratng s A or B, functon pont was recorded by the IFPUG method, and so on). Also, we ex-

6 6 Table 2 Varables of ISBSG Dataset Varable Scale Descrpton FP Rato Unadjusted functon pont Effort Rato Summary work effort (hour) Duraton Rato Actual duraton of project Maxmum team sze Language type Development type Development platform Rato Rato Maxmum number of personnel who engaged the project 3GL (second-generaton programmng language), 4GL, and others Nomnal New development, enhancement, and others Nomnal Md range, man frame, and others Table 3 Varables of Ktchenham Dataset Varable Scale Descrpton FP Rato Adjusted functon pont Effort Rato Actual development effort (hour) Duraton Rato Actual duraton of project Development type Nomnal Development, perfectve, and others Table 4 Varables of Desharnas Dataset Varable Scale Descrpton FP Rato Unadjusted functon pont Effort Rato Actual development effort (hour) Duraton Rato Actual duraton of project Adjustment factor Rato Adjustment factor of functon pont TeamExp Interval Experence of team (measured n years) ManagerExp Interval Experence of manager (measured n years) Language Nomnal type1, type2, and others cluded projects that ncluded mssng values (lstwse deleton). As a result, we used 196 projects. The varables used n our experment are shown n Table 2. They are almost same as the prevous study [15] except for duraton and maxmum team sze. The Ktchenham dataset ncludes 145 projects of a software development company, shown by Ktchenham et al. n ther study [11]. We selected 135 projects

7 that do not nclude mssng values. Three varables shown n Table 3 were chosen as the ndependent varables, and nadequate varables for effort estmaton (e.g. estmated effort by a project manager) were elmnated. Development type was transformed nto dummy varables. The Desharnas dataset ncludes 88 projects of 1980 s, collected from a Canadan company by Desharnas [5]. The dataset s avalable at the PROMISE Repostory [3]. We used 77 projects that do not have mssng values. Varables shown n Table 4 were used as ndependent varables, and development year were not used. Also, the adjusted functon pont, the number of transactons, and the number of enttes were not used to avod multcollnearty. Programmng language was transformed nto dummy varables whch reflects dfferent development envronments Evaluaton crtera To evaluate the accuracy of effort estmaton, we used the conventonal metrcs such as AE (Absolute Error), MRE (Magntude of Relatve Error) [4], and BRE (Balanced Relatve Error) [21]. Especally, MRE s wdely used to evaluate the effort estmaton accuracy [31]. When x denotes actual effort, and xˆ denotes estmated effort, each crteron s calculated by the followng equatons: AE x xˆ (6) x xˆ MRE (7) x x xˆ, x xˆ 0 BRE x x xˆ, x xˆ 0 xˆ A lower value of each crteron ndcates hgher estmaton accuracy. Intutvely, MRE means error relatve to actual effort. However, MRE have bases for evaluatng under estmaton [14]. The maxmum MRE s 1 even f an extreme underestmate occurs (For nstance, when the actual effort s 1000 person-hour, and the estmated effort s 0 person-hour, MRE s 1). So we employed BRE whose evaluaton s not based as s both MRE [22], and we evaluated the classfed models based on manly BRE (MRE were used for reference). We dd not use Pred(25) [4] whch s sometmes used as an evaluaton crteron, because Pred(25) s based on MRE and t has also a bas for evaluatng under estmaton. (8)

8 8 Dataset Learnng data Team Sze Sze Effort (Actual) Past project A Past project C Test data (Team sze error: 25%) Team sze Team Sze Sze Effort (Includng error) (Actual) Past project B Past project D Used as values of ndependent varables Fgure 2 Injectng errors nto values of an ndependent varable 3.3 Procedure of Experment In the experment, we assume that duraton and maxmum team sze nclude errors when effort s estmated. Ths s because they are not fxed when effort s estmated (.e., they are estmated values). At the end of the project, actual values of them may be dfferent from the estmated values. To analyze nfluence of the errors, we add n% errors to duraton and maxmum team sze. We set n as 0%, 25%, 50%, 100%, and 200%. The defnton of the error rate s same as BRE. Fgure 2 s an example of the procedure. Dataset s dvded nto learnng data and test data. Only team sze on the test data ncludes the error. In the fgure, team sze on test data ncludes 25% errors. We generated new values of team sze ncludng the errors, and used t when effort s estmated. We made the followng models n the experment, usng analogy based estmaton and multple lnear regresson analyss. A) Models wthout duraton and maxmum team sze B) Models wth duraton C) Models wth maxmum team sze D) Models wth duraton and maxmum team sze On model A, ndependent varables do not nclude duraton and maxmum team sze. Model B ncludes duraton as one of ndependent varables. In the same way, model C and D have ndependent varables. We call the model A as baselne, and evaluated other models wth the baselne. Model C and D were made when ISBSG dataset s used. Snce only ISBSG dataset ncludes maxmum team sze.

9 9 Table 5 Relatonshp to Effort and Productvty Dataset Varable Effort Productvty Desharnas Duraton Ktchenham Duraton ISBSG Duraton ISBSG Max. team sze We evaluated accuraces of models by dfferences of crtera from a baselne model. Therefore, postve values mean estmaton accuraces were mproved from the baselne model, and negatve values mean estmaton accuraces got worse. We appled 5-fold cross valdaton to dvde the dataset nto ft datasets and test datasets. The ft datasets were used to buld the models, and the test datasets were used to evaluate the models. Logarthmc transformaton and varable selecton was appled when multple regresson models were bult. The number of neghborhoods was set as 5 when analogy based estmaton was appled. 4 Results 4.1 Prelmnary analyss As prelmnary analyss, we analyzed the relatonshp of duraton and team sze to effort and productvty. If the relatonshp s strong, usng duraton and team sze as ndependent varables s expected to enhance estmaton accuracy. Productvty was calculated by FP (functon pont) dvded by effort. Strength of the relatonshp was evaluated usng Spearman's rank correlaton coeffcent. The result s shown n Table 5. The relatonshp between duraton and productvty was weak on the three datasets, although the relatonshp between duraton and effort was not weak. The result suggests that usng duraton as an ndependent varable s not very effectve to enhance estmaton accuracy. In contrast, strength of the relatonshp between maxmum team sze and productvty was moderate on ISBSG dataset. So, usng maxmum team sze as an ndependent varable s expected to enhance estmaton accuracy.

10 Estmaton accuracy of analogy based estmaton Table 6 shows estmaton accuracy of the models when analogy based estmaton was used. In the table, top row of each dataset shows the accuracy of the model A (.e., the baselne), and other rows do the dfference from the baselne. Boldface n the table ndcates the accuracy s mproved usng duraton and maxmum team sze as ndependent varables. Evaluaton of model B (usng duraton): On Desharnas dataset, even the error rate s 0%, mprovement of the accuracy was very small. Specfcally, mprovement of average AE and average BRE were very small, and medan AE and medan BRE got slghtly worse. When the error rate of duraton was equal or less than 50%, the negatve nfluence to estmaton accuracy was small. On Ktchenham dataset, average and medan BRE were mproved when the error rate was less than 100%. However, average and medan AE got worse even the rate was 0%. On ISBSG dataset, estmaton accuracy got worse on most cases. Therefore, when effort s estmated by analogy based estmaton, usng duraton as an ndependent varable s not effectve but sometmes negatvely affects to estmaton accuracy. Evaluaton of model C and D (usng maxmum team sze): When maxmum team sze was used as an ndependent varable on ISBSG dataset (model C), t was effectve to mprove estmaton accuracy. Except for medan MRE, estmaton accuracy was mproved on most cases, when the error rate was equal or less than 50%. When both maxmum team sze and duraton were used (model D), medan AE and average BRE got worse. Ths would be because duraton was negatvely affected to the accuracy. So, usng maxmum team sze as an ndependent varable s effectve when the error rate s equal or less than 50%, and effort s estmated by analogy based estmaton. 4.3 Estmaton accuracy of multple regresson analyss Table 7 shows estmaton accuracy of the models when multple regresson analyss was used. The structure of the table s same as Table 6. Evaluaton of model B (usng duraton): On Desharnas dataset, average AE, MRE, and BRE were slghtly mproved, when the error rate was equal or less than 50%. In contrast, medan AE, MRE, and BRE got worse. On Ktchenham dataset, estmaton accuracy was mproved when the error rate was equal or less than 50%. Especally, the mprovement of average BRE was about 10%. On ISBSG dataset, usng duraton dd not affect estmaton accuracy very much, when the error rate was equal or less than 50%. Overall, usng duraton as ndependent varable dd not negatvely affected when multple regresson analyss was used, and sometmes postvely affected when the error rate was equal or smaller than 50%. Evaluaton of model C and D (usng maxmum team sze): When maxmum team sze was used as an ndependent varable on ISBSG dataset (model C), t was

11 effectve to mprove estmaton accuracy. Estmaton accuracy was mproved on 11

12 12 most cases, when the error rate was equal or less than 50%. Also, when both max-

13 mum team sze and duraton were used (model D), estmaton accuracy was mproved when the error rate s equal or less than 25%. When the error rate was 0%, the estmaton accuracy of model D was better than the model C. However, then the rate s 25%, the accuracy was almost same. Therefore, usng maxmum team sze as an ndependent varable s effectve when the error rate s equal or less than 50%, but addng duraton as an ndependent varable does not mprove estmaton accuracy unless the error rate s very small Summery of the results Usng duraton as an ndependent varable (model B) was not very effectve n many cases. Estmaton accuracy was explctly mproved only when multple regresson analyss was used on Ktchenham dataset. So, the answer of RQ1 s No. In contrast, usng maxmum team sze as an ndependent varable (model C) was effectve when the error rate s not very large (equal or less than 50%). So, the answer of RQ2 s Yes. To know the error rate, duraton and maxmum team sze should be estmated and recorded, and we can calculate the rate when the data s accumulated. When the error rate s equal or more than 100%, the estmaton accuracy got worse n many cases. So, the answer of RQ3 s 100% and more. Overall, nfluence of duraton, maxmum team sze and the error rate to estmaton accuracy was not very dfferent between analogy based estmaton and multple regresson analyss. So, the nfluence would not be very dfferent even when other estmaton models are used. 5 Related Work In our past studes, we focus on error ncluded n ndependent varables such as dfference between estmated team sze and actual team sze. Study [29] proposed an estmaton method based on stratfcaton of team sze, and analyzed the nfluence of the error of team sze. Also, study [28] proposed an estmaton method based on productvty and proposed new method to absorb the nfluence of the error of the estmated productvty. However, study [29] used team sze as a categorcal varable, and not used as a rato scale varable. Also, study [28] used productvty, but not used team sze as an ndependent varable. Therefore, our past studes [28][29] dd not clarfy the effect of team sze and duraton to estmaton accuracy. There are many studes whch analyzed the relatonshp between project attrbutes such as duraton and productvty. For example, Maxwell et al. [17] and Premraj et al. [23] analyzed an nfluence of busness sector for productvty, usng

14 14 Fnnsh software development project dataset collected by Software Technology Transfer Fnland (STTF). Lokan et al. [16] showed productvty by busness sector usng dataset of Internatonal Software Benchmarkng Standards Group (ISBSG). In these studes, projects for manufacturng have the hghest productvty, and projects for bankng/insurance have the lowest productvty. Also, relatonshp of team sze and duraton to productvty was analyzed n some studes [18][30]. In the study [30], team sze showed strong relatonshp to productvty, and duraton was weak relatonshp to productvty. Dataset used n the study s Japanese cross-company dataset, and t s not ISBSG dataset. Therefore, our analyss result has external valdty to some extent. 6 Conclusons In ths study, we evaluated the effect of usng project duraton and maxmum team sze as an ndependent varable on effort estmaton models. We assume that duraton and maxmum team sze nclude errors when effort s estmated. Ths s because they are not fxed on the pont. To analyze nfluence of the errors, we add n% errors to duraton and maxmum team sze. We set n as 0%, 25%, 50%, 100%, and 200%. We used ISBSG dataset, Ktchenham dataset, and Desharnas datasets n the experment. To estmate effort, analogy based estmaton and multple lnear regresson analyss were used. Our fndngs nclude the followngs: Usng duraton as an ndependent varable was not very effectve n many cases. Usng maxmum team sze as an ndependent varable was effectve when the error rate s not very large (equal or less than 50%). When the error rate s equal or more than 100%, the estmaton accuracy got worse n many cases. Influence of duraton, maxmum team sze, and the error rate to estmaton accuracy was not very dfferent between analogy based estmaton and multple regresson analyss. The nfluence of maxmum team sze was evaluated only one dataset. To enhance the relablty of the results, we wll analyze the nfluence n other dataset. Acknowledgment Ths research was partally supported by the Japan Mnstry of Educaton, Scence, Sports, and Culture [Grant-n-Ad for Scentfc Research (C) (No. 16K00113)]

15 References M. Azzeh, D. Neagu, and P. Cowlng, Fuzzy grey relatonal analyss for software effort estmaton, Emprcal Software Engneerng, vol.15, no.1, pp.60-90, B. Boehm, Software Engneerng Economcs, Prentce Hall, G. Boettcher, T. Menzes, and T. Ostrand, PROMISE Repostory of emprcal software engneerng data, West Vrgna Unversty, Department of Computer Scence, S. Conte, H. Dunsmore, and V. Shen, Software Engneerng, Metrcs and Models, Benjamn/Cummngs, J. Desharnas, Analyse Statstque de la Productvte des Projets Informatque a Parte de la Technque des Pont des Functon, Master Thess, Unversty of Montreal, Internatonal Software Benchmarkng Standards Group (ISBSG), ISBSG Estmatng: Benchmarkng and research sute, ISBSG, R. Jeffery, M. Ruhe, and I. Weczorek, Usng Publc Doman Metrcs To Estmate Software Development Effort, In Proceedngs of the Internatonal Symposum on Software (METRICS), pp.16-27, J. Keung, B. Ktchenham, and R. Jeffery, Analogy-X: Provdng Statstcal Inference to Analogy-Based Software Cost Estmaton, IEEE Transactons on Software Engneeng,vol.34, no.4, pp , C. Krsopp, E. Mendes, R. Premraj, and M. Shepperd, An Emprcal Analyss of Lnear Adaptaton Technques for Case-Based Predcton, In Proc. of Internatonal Conference on Case-Based Reasonng, pp , B. Ktchenham, and E. Mendes, Why comparatve effort predcton studes may be nvald, In Proceedngs of Internatonal Conference on Predctor Models n Software Engneerng (PROMISE), art.4, p.5, B. Ktchenham, S. Pfleeger, B. McColl, and S. Eagan, An Emprcal Study of Mantenance and Development Estmaton Accuracy, Journal of Systems and Software, vol. 64, no. 1, pp , Y. L, M. Xe, and T. Goh, A study of the non-lnear adjustment for analogy based software cost estmaton, Emprcal Software Engneerng, vol.14, no.6, pp , J. L, and G. Ruhe, Analyss of attrbute weghtng heurstcs for analogybased software effort estmaton method AQUA+, Emprcal Software Engneerng, vol.13, no.1, pp.63-96, 2008.

16 C. Lokan, What Should You Optmze When Buldng an Estmaton Model? In Proceedngs of Internatonal Software Metrcs Symposum (METRICS), pp. 34, Como, Italy, Sep C. Lokan, and E. Mendes, Cross-company and sngle-company effort models usng the ISBSG Database: a further replcated study, In Proceedngs of the Internatonal Symposum on Emprcal Software Engneerng (ISESE), pp , Sep C. Lokan, T. Wrght, P. Hll, and M. Strnger, Organzatonal Benchmarkng Usng the ISBSG Data Repostory, IEEE Software, vol.18, no.5, 2001, pp K. Maxwell, and P. Forselus, Benchmarkng Software Development Productvty, IEEE Software, vol.17, no.1, pp.80-88, K. Maxwell, L. Wassenhove, and S. Dutta, Software Development Productvty of European Space, Mltary, and Industral Applcatons, IEEE Transactons on Software Engneerng, vol.22, no.10, pp , E. Mendes, S. Martno, F. Ferrucc, and C. Gravno, Cross-company vs. sngle-company web effort models usng the Tukutuku database: An extended study, Journal of Systems and Software, vol.81, no.5, pp , E. Mendes, N. Mosley, and S. Counsell, A Replcated Assessment of the Use of Adaptaton Rules to Improve Web Cost Estmaton, In Proc. of the Internatonal Symposum on Emprcal Software Engneerng (ISESE), pp , Y. Myazak, M. Terakado, K. Ozak, and H. Nozak, Robust Regresson for Developng Software Estmaton Models, Journal of Systems and Software, vol. 27, no. 1, pp. 3 16, K. Mølokken-Østvold, and M. Jørgensen, A Comparson of Software Project Overruns-Flexble versus Sequental Development Models, IEEE Transacton on Software Engneerng, vol. 31, no. 9, pp , R. Premraj, M. Shepperd, B. Ktchenham, and P. Forselus, An Emprcal Analyss of Software Productvty over Tme, In Proceedngs of Internatonal Software Metrcs Symposum (METRICS), pp.37, M. Shepperd, and C. Schofeld, Estmatng software project effort usng analoges, IEEE Transacton on Software Engneerng, vol. 23, no. 12, pp , K. Srnvasan, and D. Fsher, Machne learnng approaches to estmatng software development effort, IEEE Transacton on Software Engneerng, vol. 21, no. 2, pp , 1995.

17 26. K. Strke, K. Eman, and N. Madhavj, Software Cost Estmaton wth Incomplete Data, IEEE Transactons on Software Engneerng, vol.27, no.10, pp , A. Tosun, B. Turhan, and A. Bener, Feature weghtng heurstcs for analogy-based effort estmaton models, Expert Systems wth Applcatons, vol.36, no.7, pp , M. Tsunoda, A. Monden, J. Keung, and K. Matsumoto, Incorporatng Expert Judgment nto Regresson Models of Software Effort Estmaton, In Proceedngs of Asa-Pacfc Software Engneerng Conference (APSEC), pp , M. Tsunoda, A. Monden, K. Matsumoto, and A. Takahash, Software development effort estmaton models stratfed by productvty factors, SEC journal, pp.58-67, (n Japanese) 30. M. Tsunoda, A. Monden, H. Yadohsa, N. Kkuch, and K. Matsumoto, Software Development Productvty of Japanese Enterprse Applcatons, Informaton Technology and Management, vol.10, no.4, pp , F. Walkerden, and R. Jeffery, An Emprcal Study of Analogy-based Software Effort Estmaton, Emprcal Software Engneerng, vol.4, no.2, pp ,