Error Detection and Impact-Sensitive Instance Ranking in Noisy Datasets

Transcription

1 Error Detecton and Impact-Senstve Instance Ranng n osy Datasets Xngquan Zhu, Xndong Wu, and Yng Yang Department of Computer Scence, Unversty of Vermont, Burlngton VT 05405, USA {xqzhu, xwu, yyang}@cs.uvm.edu Abstract Gven a nosy dataset, how to locate erroneous nstances and attrbutes and ran suspcous nstances based on ther mpacts on the system performance s an nterestng and mportant research ssue. We provde n ths paper an Error Detecton and Impact-senstve nstance Ranng (EDIR) mechansm to address ths problem. Gven a nosy dataset D, we frst tran a benchmar classfer T from D. The nstances, that cannot be effectvely classfed by T are treated as suspcous and forwarded to a subset S. For each attrbute A, we swtch A and the class label C to tran a classfer AP for A. Gven an nstance I n S, we use AP and the benchmar classfer T to locate the erroneous value of each attrbute A. To quanttatvely ran nstances n S, we defne an mpact measure based on the Informaton-gan Rato (IR). We calculate IR between attrbute A and C, and use IR as the mpact-senstve weght of A. The sum of mpact-senstve weghts from all located erroneous attrbutes of I ndcates ts total mpact value. The expermental results demonstrate the effectveness of our strateges.. Introducton The goal of nductve learnng s to form generalzatons from tranng nstances such that the classfcaton accuracy on prevously unobserved nstances s maxmzed. Ths maxmum accuracy s usually determned by two most mportant factors: () the qualty of the tranng data; and (2) the nductve bas of the learnng algorthm. Gven a specfc learnng algorthm, t s obvous that ts classfcaton accuracy depends vtally on the qualty of the tranng data. Bascally, the qualty of a real-world dataset depends on a number of ssues (Wang et al. 200), but the source of the data s a crucal factor. Data entry and acquston are nherently prone to errors. Unless an organzaton taes extreme measures n an effort to avod data errors the feld error rates are typcally around 5% or more (Orr 998; Maletc & Marcus 2000). There have been many approaches for data preprocessng (Maletc & Marcus 2000; Wang et al. 200) and nose handlng (Bansal et al. 2000; Brodley & Fredl 999; Gamberger et al. 999; Kubca et al. 2003; ttle & Rubn 987; u et al. 2002; Teng 999; Wu 995; Zhu et al. 2003) to enhance the data qualty, where the enhancement s acheved through nose elmnaton, mssng value predcton, or nosy value correcton. Bascally, a general nose handlng mechansm conssts of three mportant steps: () nosy nstance dentfcaton; (2) erroneous attrbute detecton; and (3) error treatment. For errors ntroduced by mssng attrbute values, the frst two steps are trval, because the nstance tself wll explctly ndcate whether t contans nose or not (e.g, a? represents a mssng attrbute value). Therefore, technques for ths type of attrbute Copyrght 2004, Amercan Assocaton for Artfcal Intellgence ( All rghts reserved. nose handlng manly focus on predctng correct attrbute values (whch s called mputaton n statstcs) by usng a decson tree (Shapro 987) or other mechansms (ttle & Rubn 987). If an nstance contans an erroneous value, how to dstngush ths error becomes a challengng tas, because such a nosy nstance lely acts as a new tranng example wth valuable nformaton. Teng (999) proposed a polshng mechansm to correct nosy attrbute values by tranng classfers for each attrbute. However, ths correctng procedure tends to ntroduce new nose when correctng the suspcous attrbutes. A recent research effort from Kubca & Moore (2003) employed probablstc models to model nose and data generaton, n whch the traned generatve models are used to detect suspcous nstances n the dataset. A smlar approach was adopted n Schwarm & Wolfman (2000) where a Bayesan model s adopted to dentfy erroneous attrbute values. Unfortunately, snce real-word data rarely comply wth any generatve model, these model-based methods stll suffer from a common problem of nose correcton: ntroducng new errors durng data correcton. Meanwhle, researchers from statstcs have also put sgnfcant efforts to locate and correct problematc attrbute values. Among varous solutons from statstcs, the Felleg-Holt edtng method (Felleg & Holt 976) s the most representatve one. To dentfy and correct errors, ths approach taes a set of edts as nput, where the edts ndcate the rules that attrbute values should comply wth. For example age < 6 should not come wth Martal status = Marred. The Felleg-Holt method has the advantage that t determnes the mnmal number of felds to change (located errors) so that a record satsfes all edts n one pass through the data. Ths approach has been extended to many edtng systems, such as SPEER at the Census Bureau (Greenberg & Petunas 990). Unfortunately, the most challengng problem of the Felleg-Holt method s to fnd a set of good edts, whch turns to be mpossble n many stuatons. All methods above are effcent n ther own scenaros, but some mportant ssues are stll open. Frst of all, due to the fact that nose correcton may ncur more troubles, such as gnorng outlers or ntroducng new errors, the relablty of these automatc mechansms s questonable, especally when the users are very serous wth ther data. On the other hand, for real-world datasets, dong data cleansng "by hand" s completely out of the queston gven the amount of human labor and tme nvolved. Therefore, the contradcton between them rases a new research ssue: how to ran nstances mpacts, so that gven a certan amount of expenses (e.g, processng tme), the data manager can maxmze the system performance by puttng prorty on nstances wth hgher mpacts. In ths paper, we provde an error detecton and mpactsenstve nstance ranng system to address ths problem. Our expermental results on real-world datasets wll demonstrate the effectveness of our approach: wth datasets from the UCI repostory (Blae & Merz 998), at any nose level (even 50%), our system shows sgnfcant effectveness n locatng erroneous attrbutes and ranng suspcous nstances. 378 EARIG

2 2. Proposed Algorthms Our EDIR system conssts of two major steps: Error Detecton and Impact-senstve Ranng. The system flowchart s depcted n Fg., and the procedures are gven n Fgs. 2 and 3. osy Dataset D Calculate Informaton-gan Ratos Suspcous Instances Subset S Impact-senstve Weght for Each Attrbute Impact-senstve Ranng Erroneous Attrbute Detecton Overall Impact Value for Each Suspcous Instance Error Detecton Impact-senstve Ranng and Recommendaton Fg.. The flowchart of EDIR system Procedure: EDIR () Input: Dataset D; Output: Raned suspcous nstances. Parameter: K, # of maxmal changes for one nstance (). S φ; EA φ; IR φ (2). Tran benchmar classfer T from D. (3). For each nstance I n D (4). If {!CorrectClassfy(I, T) or!cover(i, T)} (5). S S {I } (6). For each attrbute A (7). Calculate Informaton-gan Rato (IR ) between A and C (8). IR IR {IR } (9). Swtch A and C to learn AP and rule set AR (0). Evaluate accuracy of each rule n AR on D. (). For each nstance I n S (2). EA φ; A V φ; CV φ (3). For each attrbute A of I (4). Calculate A V and confdence CV from AP (5). A V A V { A V }; CV CV { CV } (6). For (=; <=K; ++) (7). If {ErrorDetecton(I,, A V, CV, T, &EA )} (8). EA ocated erroneous attrbutes (9). EA EA {EA } (20). Brea (2). ImpactSenstveRanng (S, IR, EA) // see Secton 2.2 Fg. 2. Error Detecton and Impact-senstve Ranng CV, T, &EA ) Procedure: ErrorDetecton (I,, A V ~, (). EA φ; Ω [] φ ; Chgum 0; Iteraton 0; CV[] φ (2). Do { ~ ~ AV AV ;.., AV AV ; A,.. A R; A,.. A Ω[]; A R.. A ; A j (3). If {CorrectClassfy (I, T)} (4). Ω [ Chgum] { A,..., A } (5). CV [Chgum]= CV CV (6). Chgum++ (7). Restore old values of (8). } whle { Iteracton ( ) } A,... A ; Iteraton++; (9). If {Chgum = 0} Return (0) (0). Else (). EA Ω [l]; l=arg{max(cv [j]), j=, 2.., Chgum} (2). Return () Fg. 3. Erroneous attrbute detecton 2.Error Detecton 2.. Suspcous Instance Subset Constructon Our frst step of error detecton s to construct a subset S to separate suspcous nstances from the dataset D. We frst tran a benchmar classfer T from D, and then use T to evaluate each nstance n D. The subset S s constructed by usng the followng crtera, as shown n steps (3) to (5) of Fg. 2.. If an nstance I n D cannot be correctly classfed by T, we forward t to S. 2. If an nstance I n D does not match any rule n T, we forward t to S too. The proposed mechansm reles on the benchmar classfer (whch s also mperfect) traned from the nosy dataset to explore nosy nstances. Our expermental analyss has suggested that although the benchmar classfer s not perfect (actually we may never learn a perfect classfer), t can stll be relatvely relable to detect some nosy nstances Erroneous Attrbute Detecton Gven an nstance I n S, assume I contans attrbutes A, A 2,.., A and one class label C. Further assume each attrbute A has V possble values. We denote the aggregaton of all attrbutes by R. To locate erroneous attrbutes from I (where an erroneous attrbute means the attrbute has an ncorrect value), we adopt an Attrbute Predcton (AP) mechansm, as shown n Fgs. 2 and 3. Bascally, Attrbute Predcton uses all other attrbutes A,.. A j, A (j ) and the class label C to tran a classfer, AP, for A (usng nstances n D). Gven an nstance I n S, we use AP to predct the value of attrbute A. Assume I s current value of A s AV, and the predcted value from AP s V A ~. If V AV and A ~ are dfferent, t mples that A of I may possbly contan an ncorrect value. Then we use the benchmar classfer T (whch s relatvely relable n evaluatng nosy nstances) to determne whether the predcted value from AP maes more sense: If we change AV to A ~, and I can be correctly classfed by T, t V wll ndcate that the change results n a better classfcaton. We therefore conclude that attrbute A contans an erroneous value. However, f the change stll maes I ncorrectly classfed by T, we wll leave A unchanged and try to explore errors from other attrbutes. If the predcton from each sngle attrbute does not conclude any error, we wll start to revse multple attrbute values at the same tme. For example, change the values of two attrbutes A (from AV to A ~ ) and A j (from V AV to j V j A ~ ) at the same tme, and then evaluate whether the multple changes mae sense to T. We can teratvely execute the same procedure untl a change maes I correctly classfed by T. However, allowng too many changes n one nstance may actually have the error detecton algorthm mae more mstaes. Currently, the EDIR system allows to change up to K (K 3) attrbutes smultaneously to locate errors. If all the changes above stll mae I ncorrectly resolved by T, we wll leave I unprocessed. Wth the proposed error detecton algorthm, one mportant ssue should be resolved n advance: whch attrbute to select f multple attrbutes are found to contan errors. Our soluton n solvng ths problem s to maxmze the predcton confdence whle locatng the erroneous attrbutes, as shown n Fg. 3. When learnng AP for each attrbute A, we use a classfcaton rule algorthm, e.g., C4.5rules, to learn an Attrbute EARIG 379

3 predcton Rule set (AR ). Assumng the number of rules n AR s r M, for each rule AR, r=,..m, n AR, we evaluate ts accuracy r ( AR ) on dataset D. Ths accuracy wll ndcate the confdence r that AR classfes the nstance. In our system, we use C4.5rules (Qunlan 993) to learn the rule set AR, so the accuracy value has been provded wth each learned rule. When adoptng the rule set AR to predct the value of A, we use the frst ht mechansm (Qunlan 993), whch means we ran the rules n AR n advance, and classfy nstance I by ts frst coved rule n AR. Meanwhle, we also use the accuracy of the selected rule as the confdence ( AC ) of AP n predctng A of I. Gven an nstance I, assume the predcted values for each attrbute are AV ~,.., A V ~ respectvely, wth the confdences for each of them denoted by AC,.., AC. We frst set to to locate an erroneous attrbute by usng Eq. (). If ths procedure does not fnd any erroneous attrbute, we ncrease the value of by and repeat the same procedure, untl we fnd erroneous attrbutes or reaches the maxmal allowable changes for one nstance (K). ~ ~ () EA = { A,., A }; arg{max{ }; (,.,, ) = } AC CorrectClassfy AV l AV T,.., l { A,. A., } R ;.. ; A j j A R j 2..3 Valdty Analyss Our algorthm above swtches each attrbute A and class C to learn an AP classfer, then uses AP to locate erroneous attrbutes. There are two possble concerns wth the algorthm: () swtchng A and C to learn classfer AP does not mae much sense to many learnng algorthms, because attrbute A may have very lttle correlaton wth other attrbutes (or not at all); and (2) when the predcton accuracy from AP s relatve low (e.g., less than 50%), does the algorthm stll wor? Our expermental results n Secton 3 wll ndcate that even the predcton accuracy from all AP classfers are relatvely low, the proposed algorthm can stll provde good results. As we can see from Fg. 3, the predcton from each AP classfer just provdes a gude for the benchmar classfer T to evaluate whether a change maes the classfcaton better or not. The predcton from AP won t be adopted unless T agrees that the value predcted by AP wll mae the nstance correctly classfed. In other words, the proposed mechansm reles more on T than on any AP. Even f the predcton accuracy from AP s 00%, we won t tae ts predcton unless t gets the support from T. Therefore, a low predcton accuracy from AP does not have much nfluence wth the proposed algorthm. However, we obvously prefer a hgh predcton accuracy from each AP. Then the queston comes to how good AP could be wth a normal dataset? Actually, the performance of AP s nherently determned by correlatons among attrbutes. It s obvous that f all attrbutes are ndependent (or condtonally ndependent gven the class C), the accuracy of AP could be very low, because no attrbute could be used to predct A. However, t has often been ponted out that ths assumpton s a gross over-smplfcaton n realty, and the truth s that the correlatons among attrbutes extensvely exst (Fretas 200; Shapro 987). Instead of tang the assumpton of condtonal ndependence, we tae the benefts of nteractons among attrbutes, as well as between the attrbutes and class. Just as we can predct the class C by usng the exstng attrbute values, we can turn the process around and use the class and some attrbutes to predct the value of another attrbute. Therefore, the average accuracy of AP classfers from a normal dataset usually mantans a reasonable level, as shown n Tab Impact-senstve Ranng To ran suspcous nstances by ther mpacts on the system performance, we defne an mpact measure based on the Informaton-gan Rato (IR). We frst calculate IR between each attrbute A and class C, and tae ths value as the mpact-senstve weght (IW) for A. The mpact value for each suspcous nstance I, Impact(I ), s then defned by Eq. (2), whch s the sum of the mpact-senstve weghts of all located erroneous attrbutes n I, IR ; If A contans error (2) Impact( I ) = IW( A, I ); IW( A, I ) = = 0 otherwse The Informaton-gan Rato (IR) s one of the most popular correlaton measures used n data mnng. It was developed from nformaton gan (IG) whch was ntally used to evaluate the mutual nformaton between an attrbute and the class (Hunt et al. 966). The recent development from Qunlan (986) has extended IG to IR to remove the bas caused by the number of attrbute values. Snce then, IR has become very popular n constructng decson trees or explorng correlatons. Due to the space lmt of the paper, we omt the technque on calculatng IR. Interested readers may refer to Qunlan (986; 993) for detals. Wth Eq. (2), nstances n S can be raned by ther Impact(I ) values. Gven a dataset wth attrbutes, the number of dfferent mpact values of the whole dataset D s determned by Eq. (3), f we allow the maxmal number of changes for one nstance to be K. For example, a dataset D wth =6 and K=3 wll have C(D) equal to 4. It seems that C(D) s not large enough to dstngush each suspcous nstance, f the number of suspcous nstance n D s larger than 4 (whch s lely a normal case). However, n realty, we usually wor on a bunch of suspcous nstances rather than a sngle one to enhance the data qualty. Therefore t may not be necessary to dstngush the qualty of each nstance, but to assgn suspcous nstances nto varous qualty levels. Gven the above example, we can separate ts suspcous nstances nto 4 qualty levels. From ths pont of vew, t s obvous that ths number s large enough to evaluate the qualty of nstances n S. K (3) C ( D ) = l = l 3. Expermental Evaluatons 3. Experment Settngs The majorty of our experments use C4.5, a program for nducng decson trees (Qunlan 993). To construct the benchmar classfer T and AP classfers, C4.5rules (Qunlan 993) s adopted n our system. We have evaluated our algorthms extensvely on datasets collected from the UCI data repostory (Blae & Merz 998). Due to the sze restrctons, we wll manly report the results on two representatve datasets: Mons-3 and Car, because these two datasets have relatvely low attrbute predcton accuraces, as shown n Tab.. If our algorthms acheve good results on these two datasets, they can possbly have good performances on most real-world datasets. In Tab., AT represents the average predcton accuracy for attrbute A (from AP ). For Soybean, Krvsp and Mushroom datasets, we only show the results of the frst sx attrbutes. We also report summarzed results from these three datasets n Tab. 5. For most of the datasets that we used, they don t actually contan much nose, so we use manual mechansms to add attrbute 380 EARIG

4 nose, where error values are ntroduced nto each attrbute wth a level x 00%, and the error corrupton for each attrbute s ndependent. To corrupt each attrbute A wth a nose level x 00%, the value of A s assgned a random value approxmately x 00% of the tme, wth each alternatve value beng approxmately equally lely to be selected. Wth ths scheme, the actual percentage of nose s always lower than the theoretcal nose level, as sometmes the random assgnment would pc the orgnal value. ote that, however, even f we exclude the orgnal value from the random assgnment, the extent of the effect of nose s stll not unform across all components. Rather, t s dependent on the number of possble values n the attrbute. As the nose s evenly dstrbuted among all values, ths would have a smaller effect on attrbutes wth a larger number of possble values than those attrbutes that have only two possble values (Teng 999). In all fgures and tables below, we only show the nose corrupton level x 00%, but not the actual nose level n each dataset. Tab.. The average predcton accuracy for each attrbute DataSet AT (%) AT 2 (%) AT 3 (%) AT 4 (%) AT 5 (%) AT 6 (%) Mons Car Soybean Krvsp Mushroom Suspcous Subset Constructon To evaluate the performance of the suspcous subset (S) constructon, we need to assess the nose level n S. Intutvely, the nose level n S should be hgher than the nose level n D, otherwse the exstence of S becomes useless. Also, the sze of S should be reasonable (not too large or too small), because a subset wth only several nstances has a very lmted contrbuton n locatng erroneous attrbutes, even f all nstances n S are nose. To ths end, we provde the followng measures: () S/D, whch s the rato between the szes of S and D; (2) I_S and I_D, whch are the nstance-based nose levels n S and D, as defned by Eq. (4); and (3) A _S and A _D, whch represent the nose levels for attrbute A n S and D, as defned by Eq. (5). I_X= # erroneous nstances n X / # nstances n X (4) A _X= # nstances n X wth error n A / # nstances n X (5) We have evaluated our suspcous subset constructon at dfferent nose levels and provded the results n Tab. 2. Bascally, the results ndcate that n terms of S/D, the proposed algorthm n Secton 2. has constructed a suspcous subset wth a reasonable sze. When the nose level ncreases from 0% to 50%, the sze of Tab. 2. Suspcous subset constructon results S also proportonately ncreases (from 4.7% to 4.07% for the Car dataset), as we have antcpated. If we tae a loo at the thrd column wth nstance-based nose, we can fnd that the nose level n S s always hgher than the nose level n D. For the Car dataset, when the nose corrupton level s 0%, the nose level n S s about 30% hgher than the nose level n D. Wth Mons-3, the nose level n S s sgnfcantly hgher than D. It ndcates that wth the proposed approach, we can construct a reasonable sze of the subset concentratng on nosy nstances for further nvestgaton. The above observatons conclude the effectveness of the proposed approach, but t s stll not clear whether S equally captures nose from all attrbutes or more focuses on some of them. We therefore evaluate the nose level on each attrbute, whch s shown from columns 5 to 0 n Tab. 2. As we can see, most of the tme, the attrbute nose level n S (A _S) s hgher than the nose level n D (A _D), whch means the algorthm has a good performance on most attrbutes. However, the algorthm also shows a sgnfcant dfference n capturng nose from dfferent attrbutes. For example, the nose level on attrbutes 2 and 5 n Mons-3 (n S) s much hgher than the attrbute nose level n the orgnal dataset D. The reason s that these attrbutes have sgnfcant contrbutons n constructng the classfer, hence the benchmar classfer T s more senstve to errors n these attrbutes. Ths s actually helpful for us to locate erroneous attrbutes: f nose n some attrbutes has less mpact wth the system performance, we can smply gnore them or put less effort on them. 3.3 Erroneous Attrbute Detecton To evaluate the performance of our erroneous attrbute detecton algorthm, we defne the followng three measures: Error detecton Recall (ER), Error detecton Precson (EP), and Error detecton Recall for each Attrbute (ERA ). Ther defntons are gven n Eq. (6), where n, p and d represent the number of actual errors, the number of correctly located errors and the number of located errors n A respectvely. p ER = ERA, ERA = ; p EP = EPA, EPA = (6) = n = d We provde the results n Tab. 3, whch are evaluated at three nose levels (from 0% to 50%). As we can see from the thrd column (EP) of Tab. 3, the overall error detecton precson s pretty attractve, even wth the datasets (Mons-3 and Car) that have low attrbute predcton accuraces. On average, the precson s mantaned at 70%, whch means most located erroneous attrbutes actually contan errors. We have also provded the expermental results from other three datasets n Tab. 5. All these results prove the relablty of the proposed error detecton algorthm. Dataset Mons- 3 Car ose evel S/D Instance ose AT ose AT 2 ose AT 3 ose AT 4 ose AT 5 ose AT 6 ose I_D I_S A _D A _S A 2 _D A 2 _S A 3 _D A 3 _S A 4 _D A 4 _S A 5 _D A 5 _S A 6 _D A 6 _S 0% % % % % % EARIG 38

5 Obvously, havng a hgh precson s only a partal advantage of the algorthm. A system that predcts only one error s lely useless, even f ts precson s 00%. We therefore evaluate the error detecton recall (ER), as shown on columns 4 to 0 n Tab. 3. On average, the algorthm can locate about 0% of errors. Meanwhle, for dfferent attrbutes, the ERA values vary sgnfcantly. For example, when the nose level s 30%, the ERA for Mons-3 s about 5.4% whch s much less than the value (about 24%) of attrbute 2. If we go further to analyze the correlatons between attrbutes and the class, we can fnd that the proposed algorthm actually has a good performance n locatng errors for some mportant attrbutes. For example, among all attrbutes n Mons-3, attrbute 2 has the hghest correlaton wth class C, and ts recall value (ERA 2 ) also turns out to be the hghest. Just loong at the ER values may mae us worry that the algorthm has mssed too much nose (90% of errors). We d le to remnd the readers that from the data correcton pont of vew, havng a hgher precson s usually more mportant, because an algorthm should avod ntroducng more errors when locatng or correctng exstng errors. Actually, the 0% located errors lely brng more troubles than others (because they obvously cannot be handled by the benchmar classfer T). Our expermental results n the next subsecton wll ndcate that correctng ths part of the errors wll mprove the system performance sgnfcantly. Tab. 3. Erroneous attrbute detecton results Data Set Mon s-3 Car ose evel EP ER ERA ERA 2 ERA 3 ERA 4 ERA 5 ERA 6 0% % % % % % Impact-senstve Instance Ranng We evaluate the performance of the proposed mpact-senstve nstance ranng mechansm n Secton 2.2 from three aspects: tranng accuracy, test accuracy and the sze of the constructed decson tree. Obvously, t s hard to evaluate the ranng qualty nstance by nstance, because one nstance lely does not mpact too much wth the system performance. We therefore separate the raned nstances nto three ters, each ter consstng of 30% of nstances from the top to the bottom of the ranng. Intutvely, correctng nstances n the frst ter wll produce a bgger mprovement than correctng nstances n any of the other two ters, because nstances n the frst ter have more negatve mpacts (wth relatvely larger mpact values), so does the second ter to the thrd ter. Accordngly, we manually correct the nstances n each ter (because we now whch nstance was corrupted, the manual correcton has a 00% accuracy), and compare the system performances on the corrected dataset and the orgnal dataset. We evaluate the system performance at three nose levels and provde results n Tab. 4, where Org means the performance (tranng, test accuracy and tree sze) from the orgnal dataset (D), and Fst, Snd and Thd ndcate the performance of only correctng nstances n the frst, second and thrd ter respectvely. From Tab. 4, we can fnd that at any nose level, correctng nstances n the frst ter always results n a better performance than correctng nstances n any of the other two ters, so does the second ter to the thrd ter. For example, wth the Car dataset at 30% nose level, correctng nstances at the frst ter wll acheve 0.7% and 2.3% more mprovements wth the test accuracy than correctng nstances n the second and thrd ters. In terms of the decson tree sze, the mprovement s even more sgnfcant. It proves that our system provdes an effectve way to automatcally locate errors, and ran them by ther negatve mpacts (danger levels). So we can put more emphass on nstances n the top ter than those n the bottom ter. When comparng the performances from each ter wth the performance from the orgnal dataset, we fnd that correctng recommended nstances has a sgnfcant mprovement, especally when the nose level goes hgher. For example, wth the Car dataset at 30% nose level, correctng nstances at any ter wll contrbute a 2.% (or more) mprovement wth the test accuracy. Ths also proves the effectveness of our EDIR system n enhancng the data qualty, even n hgh nose-level envronments. When usng EDIR as a whole system for nose detecton and data correcton, we d le to now ts performance n comparson wth other approaches, such as random samplng. We therefore perform the followng experments. Gven a dataset D, we use EDIR to recommend α% of nstances n D for correcton. For comparson, we also randomly sample α% of nstances n D for correcton. We compare the test accuraces from these two mechansms (because the comparson of the test accuraces s lely more objectve), and report the results n Fg. 4 (b) to Fg. 4 (g), where Org, EDIR and Rand represent the performances from the orgnal dataset, the corrected dataset by EDIR and the random samplng mechansm respectvely. We have evaluated the results by settng α 00% to four levels: 5%, 0%, 5% and 20%. In Tab. 5, we provde summarzed results from other three datasets by settng α 00% to 5%. The results from Fg. 4 ndcate that when the value of α ncreases, the performances of EDIR and Rand both get better. Ths does not surprse us, because when recommendng more and more nstances for correcton, we actually lower the overall nose level n the dataset, therefore better performances could be acheved. However, the nterestng pont s that when comparng EDIR and Rand, we can fnd that EDIR always has a better performance than Rand. Tab. 4. Impact-senstve nstance ranng results DataSet Mons- 3 Car ose Tranng Accuracy (%) Test Accuracy (%) Tree Sze evel Org Fst Snd Thd Org Fst Snd Thd Org Fnt Sed Thd 0% % % % % % EARIG

6 Actually, most of the tme, the results from EDIR by recommendng 5% of nstances are stll better than usng Rand to recommend 20% of nstances. For example, when EDIR recommends 5% of nstances for correcton, the test accuracy for the Car dataset at 0% nose level s 88.5%, whch s stll better than the results (88.%) of usng Rand to recommend 20% of nstances. The same concluson can be drawn from most other datasets. It ndcates that EDIR has a sgnfcantly good performance n locatng and recommendng suspcous nstances to enhance the data qualty Test Acc. Org Test Acc. EDIR Test Acc. Rand (a) Meanng of each curve n Fg. 4 (b) to Fg. 4 (g) (b) Mons-3 (0%) (c) Mons-3 (30%) (d) Mons-3 (50%) (e) Car (0%) (f) Car (30%) (g) Car (50%) Fg. 4. Expermental comparsons of EDIR and random samplng approaches from three nose levels: 0%, 30% and 50% In Fgs. 4 (b) to (g), the x-axs denotes the percentage of recommended data (α) and the y-axs represents the test accuracy. Tab. 5. Expermental summary from other three datasets (α=0.05) Data Set Soyb ean Krvs p Mush room ose evel EP ER Test Accuracy Tree Sze Org EDIR Rand Org EDIR Rand 0% % % % % % % % % Conclusons In ths paper, we have presented an EDIR system, whch automatcally locates erroneous nstances and attrbutes and rans suspcous nstances accordng to ther mpact values. The expermental results have demonstrated the effectveness of our proposed algorthms for error detecton and mpact-senstve ranng. By adoptng the proposed EDIR system, correctng the nstances wth hgher rans always results n a better performance than correctng those wth lower rans. The novel features that dstngush our wor from exstng approaches are threefold: () we provded an error detecton algorthm for both nstances and attrbutes; (2) we explored a new research topc on mpactsenstve nstance ranng, whch can be very useful n gudng the data manager to enhance the data qualty wth mnmal expenses; and (3) by combnng error detecton and mpactsenstve ranng, we have constructed an effectve data recommendaton system. It s more effcent than the manual approach and more relable than automatc correcton algorthms. Acnowledgement Ths research has been supported by the U.S. Army Research aboratory and the U.S. Army Research Offce under grant number DAAD References Bansal,., Chawla, S., & Gupta, A., (2000), Error correcton n nosy datasets usng graph mncuts, Techncal Report, CMU. Blae, C.. & Merz, C.J. (998), UCI Repostory of machne learnng databases. Brodley, C.E. & Fredl, M.A. (999), Identfyng mslabeled tranng data, J. of Artfcal Intellgence Research, : Greenberg, B. & Petunas, T. (990), SPEER: structured programs for economc edtng and referrals, Amercan Statstcal Asso., Proc. of the 990 Secton on Survey Research Methods. Felleg, I. P. & D. Holt, (976), A systematc approach to automatc edt and mputaton, Journal of the Amercan Statstcal Assocaton, vol.7, pp Fretas, A., (200), Understandng the crucal role of attrbute nteracton n data mnng, AI Revew, 6(3): Gamberger, D., avrac,., & Groselj C. (999), Experments wth nose flterng n a medcal doman, Proc. of 6 th ICM, CA. Hunt, E. B., Martn, J. Stone, P. (966), Experments n Inducton. Academc Press, ew Yor. Kubca, J. & Moore A., (2003), Probablstc nose dentfcaton and data cleanng, Proc. of ICDM, F, USA. ttle, R.J.A. & Rubn, D.B. (987), Statstcal analyss wth mssng data, Wley, ew Yor. u, X., Cheng, G., & Wu, J. (2002), Analyzng outlers cautously, IEEE Trans. on TKDE, 4: Maletc J. & Marcus A. (2000), Data cleansng: Beyond ntegrty analyss, Proc. of Informaton Qualty, pp Orr, K., (998), Data qualty and systems theory, CACM, 4 (2):66-7. Qunlan, J.R. (986). Inducton of decson trees. Machne earnng, (): Qunlan, J.R. (993). C4.5: programs for machne learnng, Morgan Kaufmann, San Mateo, CA. Schwarm S. & Wolfman S., (2000), Cleanng data wth Bayesan methods, Techncal Report, Unversty of Washngton. Shapro A. (987), Structured nducton n expert systems, Addson-wesley. Teng M. T., (999), Correctng nosy data, Proc. of 6 th ICM. Wang R., Zad M., & ee Yang, (200), Data qualty, Kluwer. Wu, X. (995), Knowledge acquston from database, Ablex Pulshng Corp., USA. Zhu, X., Wu, X. & Chen Q. (2003). Elmnatng class nose n large datasets, Proc. of 20 th ICM, Washngton D.C., USA. EARIG 383