A Comparative Study of Outlier Detection Procedures in Multiple Linear Regression

Transcription

1 Proceedgs of the Iteratoal MultCoferece of Egeers ad Computer Scetsts 009 Vol I IMECS 009, March 18-0, 009, Hog Kog A Comparatve Study of Outler Detecto Procedures Multple Lear Regresso Pmpa Ampathog, Prachoom Suwattee Abstract Outler detecto methods multple lear regresso are revewed. Eght statstcs for outler detecto have bee vestgated ad compared. It s foud from Mote Carlo smulato that Mahalaobs dstace ( MD ) detfers the presece of outlers more ofte tha the others for small, medum ad large sample szes wth dfferet percetages outlers the regressors ad both the regressors ad the depedet varable. The ext best statstcs for the detecto are Hat matrx ( h ),Cook s square dstace ( CD ) ad DEFFIT dstace. As for the depedet varable outler, Cook s square dstace ( CD ) ad PRESS resdual ( r () ) perform better tha the others. Idex Terms Multple lear regresso, Outlers, Outler detecto, Resduals. 1. INTRODUCTION Lear models are commoly used to study the fuctoal relatoshp betwee a depedet varable ad regressors. Usually, ordary least- squares (OLS) method s appled to the sample data to obta the ftted lear model or lear regresso equato of the depedet varable y o the regressors X1, X,, X p, p 1. However, sometmes the samples mght cota outlers the X s values, the Y s values, or both X s ad Y s values. I that case, the OLS estmates of the regresso coeffcets are o loger precse estmates. The presece of outlers wll have some effects o the results of the statstcal ferece cocerg the models. It s mportat for the data aalyst to be able to detfy outlers the samples f they exst so that approprate measures mght be take. Cosder a geeral lear model of the form y = Xβ + ε, (1) where y s a 1 vector of observed values of the depedet or respose varable, X a p matrx of p predctors or regressors, β a p 1 vector of ukow parameters, ad ε a 1vector of errors. If ε follow a Mauscrpt receved Jauary 14, 009. Ths work was supported part by the U.S. Departmet of Commerce uder Grat ICDMA_65 Ampathog, P., s wth the Natoal Isttute of Developmet Admstrato, School of Appled Statstcs, Bagkok, Thalad, (e-mal address: pm_pmpa@hotmal.com). Suwattee, P., s wth the Natoal Isttute of Developmet Admstrato, School of Appled Statstcs, Bagkok, Thalad, (e-mal address: prachoom@as.da.ac.th). ormal N(0, σ I) assumptos, the the OLS or the maxmum lkelhood (ML) estmates of β tur out to be the best lear ubased estmates (BLUE) of β accordg to the Gauss-Markov theorem. If the ormalty ad depedece codtos do ot hold, the the OLS or ML estmates of β may tur out to be arbtrarly bad. Whe the sample data cota outlers, alteratve approach to the problem should be appled to obta better ft of the models or more precse estmates of β. Actually, dfferet ways to aalyze the data wth outlers have bee suggested, usg robust regresso methods, by may statstcas, for example, Maroa, R.A. [1], Cambell, N.A. [4], Huber, P.J. [8], Lopuhaa, H.P. ad Rousseeuw, P.J. [11], Kafard, F. ad Swallow, W. [9], Had, A.S. ad Smooff, J.S. [7], Atkso, A.C. [1], Barett, V. ad Lews, T. [], Woodruff, D.L. ad Rocke, D.M. [5], Sebert, D.M. [], ad Ra, M. ad Atkso, A.C. [16]. So detecto of outlers regresso s very mportat ad should be study more carefully. Ths paper wll revew ad compare dfferet methods of outler detecto.. METHODS OF OUTLIER DETECTION IN REGRESSION I the lterature, there are may methods of detecto of outlers multple lear regresso. They may be classfed to two groups, amely graphcal ad aalytcal methods. 1.1 Graphcal methods. For graphcal methods, we detfy the presece of outlers by the shape of the plot or the graph of observed data or resduals. Varous plots ad graphs are avalable for the purpose Scatter Plot. Observed data pots ( xj, y ), = 1,,, for each j = 1,,, p are plotted. The scatter plot of the observed data pots wth oe or more sample pots stadg apart from the majorty dcate the presece of outlers..1. Normal Probablty Plot. For a radom sample of sze, the resduals, e ˆ = y y, where y ˆ comes from a OLS ftted equato ( yˆ ˆ = xβ ), where [ x1, x,, xp] for each = 1,,, are calculated ad raked as e (1) < e () < < e ( ). The e () s are plotted agast the cumulatve probablty ( 0.5) p =, 1,,, =. The ormal probablty plot wth pots depart from a straght le dcates the presece of outlers..1.3 The Boxplot. Whe the resduals e, = 1,,, are plotted the form of a box-ad-whsker plot. The box part ISBN: IMECS 009

2 Proceedgs of the Iteratoal MultCoferece of Egeers ad Computer Scetsts 009 Vol I IMECS 009, March 18-0, 009, Hog Kog coverg are the ter-quartle rage. If the whskers are too log, the the presece of outler s dcated..1.4 Resdual Plots. The resduals e (or the scaled 1/ 1/ resduals d /? = e σ, r = e / σ(1 h) or t = e / S(1 h), wth σ = MSE h s the th dagoal elemet of the hat matrx ˆ, H X( X ' X) X ' S = [( p) MSE e /(1 h )]/( p) may = ad be plotted agast the ftted value yˆ or each regressor varables, X j, = 1,,, for each j = 1,,, p. Extreme pots the resdual plots dcate the exstece of outlers the sample.. Aalytcal Methods. There are may statstcal values computed from the sample data that ca be used to detfy the exstece of outlers. To detfy the exstece of oe or more outlers the sample eght statstcs have bee suggested by dfferet authors...1 Stadardzed Resduals. To detfy the exstece of outlers the stadardzed resduals d = e / MSE, () = 1,,, are computed. A Large stadardzed resduals ( d > 3) dcates the exstece of outlers (Motgomery, D. C., et al. [15])... Studetzed Resduals. For each resdual e ˆ = y y, compute the stadardzed resduals or r e / MSE(1 h ) =, (3) ( ) r = e / MSE 1 (1/ ) ( Xj X) / S +, (4) aga r > 3 dcates that e s a outler, = 1,,, (Motgomery, D. C., et al. [15])...3 PRESS Resduals. For each varable observato X, = 1,,, ad j = 1,,, p compute the j predcto error or the PRESS resduals e = y yˆ, (5) where yˆ s the ftted value of the th respose based o observatos deletg the th observed values. The PRESS resduals may be computed from the hat matrx ad the resdual e = y yˆ as r() e /(1 h) =, (6) th = 1,,, where h s the dagoal elemet of H = X( X ' X) X '. If r () > 3 the the th observato s detfed as outlers (Motgomery, D. C., et al. [15])...4 The Hat Matrx. May authors 1 use the value of h, the th dagoal elemet of H = X( X ' X) X ' to dcate 1 The book by Rousseeuw, P.J. ad Leroy, A.M., o pages 0, determe potetally fluetal pot by the most authors are Hoagl ad Welsh (1978), Hederso ad Vellema (1981), Cook ad Wesberg (198), Hockg (1983), Paul (1983), ad Steves (1984). outlers. For h > p/ ( Rousseeuw, P.J. ad Leroy, A.M., [19], p. 0), the th observato s detfed as outler...5 Cook s Square Dstace. Cook s square dstace of ut th s a measure base o the square of the maxmum dstace betwee the OLS estmate based o all pots ˆβ ad the estmate obtaed whe the th pot, say ˆ β (). Cook ad Wesberg 3 suggest examg cases wth CD > 0.5 ad that case where CD > should always be studed. Ths dstace measure ca be expressed a geeral form CD? X X? p ˆ = ( β β)'( ' )( β β)/ σ, (7) = 1,,,. However, substtutg CD statstc may also be rewrtte as CD = ( e / p)( h /(1 h)) all of whch are related to the full data...6 R-Studet. A commo way to model a outler s the mea shft outler model. However, the R-studet statstc wll be more sestve to ths pot. A formal testg procedure for outlers detecto based o R-studet s gve by t = e / ˆ( σ ) (1 h), (8) = 1,,, where t > t( α / ), ( p 1) dcates the exstece outlers. Ths s referred to as a estmate of MSE based o a data set wth the th observato removed. The estmate of MSE, so obtaed from the th observato s ˆ σ = [( p) MSE e /(1 h )]/[ ( p+ 1)], (9) ()..7 DEFFIT Dstace. For each observato compute y? y () or ( he )/(1 h) whch tells how much the predcted value y ˆ, at the desg pot x would be affected f the th case were deleted. The stadardzed verso of DEFFIT s DEFFIT h e h 1/ ( )/( σ (1 )) =, (10) = 1,,,. Belsley, Kuh ad Welsch 4 suggested that ay observato for whch DEFFIT > / p / warrats atteto for outlers...8 Mahalaobs Dstace. The measure the leverage by meas of MD (Mahalaobs dstace), where = 1,,, where = 1 MD ( μ μ) σ ( μ μ)' ( 1)[ h 1/ ] = =, (11) ( μ μ)'( μ μ). If MD p μ = 1/ ( μ ) ad σ = 1/( 1)* = 1 χ 1,0.95 > where χ p 1,0.95 percetle of a ch-square dstrbuto wth p s the 95 th degrees of Belsley, D. A.,Kuh, E. ad Welsch, R.E.S., Regresso Dagostcs : Idetfyg Ifluetal Data ad Source of Collearty, New York: Joh Wley & Sos, Cook, R.D., Detecto of fluetal observatos regresso, Techometrcs, Vol. 19, 1977, pp Cook, R.D. ad Wesberg, S., Resduals ad Ifluece regresso, Lodo : Chapma & Hall, ISBN: IMECS 009

3 Proceedgs of the Iteratoal MultCoferece of Egeers ad Computer Scetsts 009 Vol I IMECS 009, March 18-0, 009, Hog Kog freedom the there s a outler ( Rousseeuw, P.J. ad Leroy, A.M., [19], pp. 4). 3. COMPARISON OF THE METHODS FOR OUTLIER DETECTION Oe thousad data sets are geerated from the model y = β0 + β1x e, = 1,,, where all regresso coeffcets are fxed β j = 1, for each = 1,,, ad j = 1,,, p ad the errors are assumed to be depedet. The p explaatory varables xj R are sampled depedetly from a N (0,1). The sample data sets are geerated uder (p=3 ad p=4) regressors ad the sample szes are small szes (=10), medum szes (=0, ad =30), ad large szes (=50, ad =100), wth dfferet percetage of outlers. From table 1, the best X s outler detecto are h, CD, DEFFIT ad MD method perform better tha other methods. The performace of MD ad h method are hghest values of outler detecto (00) hgh percetage of X s outlers ad every sample szes. For the low percetage of X s outlers CD method performs much better tha other methods ad CD method has hgh values of detecto outlers whe percetage of X s outlers are decreased (0.97) ad small szes [Fg. 1(a)]. Table. The Values of Statstcs for Detecto of Outlers by Szes ad Percetage of Y s Outlers wth Three Szes Outlers d r r () h CD t DEFFIT MD The comparso of eght detecto statstcs s carred out by the followg steps: 1) Geerato of the data wth certa percetage of X s outlers, Y s outlers ad both X s ad Y s outlers ad dfferet sample szes (small, medum ad large). ) Each statstc s computed from each of the 1,000 replcatos. 3) Make comparso of detecto of outlers by coutg the umber of tmes that each statstc ca detfy outlers. The varato comparso of eght outler detecto methods provdes a dcato of the sestvty of the methods. 4. COMPARISON RESULTS 4.1 Results for Three The computatos of detecto of outlers gve the best of outler detecto methods for dfferet sample szes ad the percetages of outler from 1,000 replcatos. The results of statstcs of eght outler detecto methods are as followg; Table 1. The Values of Statstcs for Detecto of Outlers by Szes ad Percetage of X s Outlers wththree Szes Outlers d r r () h CD t DEFFIT MD From table, the best of Y s outler detecto s r () method. The performace of r () method s hghest values of the test (00) large szes ad hgh percetage of Y s outlers. Wth small szes CD method s performs much better tha other methods, ad the values of the detecto outlers whe the low percetage of Y s outlers s (0.53). Wth large szes the performace of r ad d methods have hgh values of detecto outler whe hgh percetages of Y s outler are (00) [Fg. 1(b)]. Table 3. The Values of Statstcs for Detecto of Outlers by Szes ad Percetage of both X s ad Y s Outlers wth Three Szes Outlers d r r () h CD t DEFFIT MD ISBN: IMECS 009

4 Proceedgs of the Iteratoal MultCoferece of Egeers ad Computer Scetsts 009 Vol I IMECS 009, March 18-0, 009, Hog Kog From table 3, the best of both X s ad Y s outler detecto are h ad MD methods. The performaces of h ad MD method are hghest values of the detecto outlers (00) other sample szes ad percetage of outlers. Wth small szes, the outler detecto of t method has a hgh value of the test (0.978). Wth large szes, the outler detecto of the r () ad CD methods are hgh values of detecto outler whe the hgh percetages of outlers. The performace of hghest values of the test s (00) [Fg. 1(c)]. Table 4. The Values of Statstcs for Detecto of Outlers by Szes ad Percetage of X s Outlers wth Four Szes Outlers d r r () h CD t DEFFIT MD (a) X's Outlers value of outler detecto h() CD() DEFFIT() MD() %Outler/sample sz (b) (c) value of outler detecto value of outler detecto Y's outlers %Outler/sample sz Both X's ad Y's Outlers d() r r() CD() h() CD() DEFFIT() MD() From table 4, the best X s outler detecto are h, CD ad MD method perform sgfcatly better tha the other methods. The performace of MD ad h methods are hghest values of detecto outler (00) hgh percetage of X s outlers ad every sample szes. For the low percetage of X s outlers CD method performs much better tha the other method ad CD method has hgh values of the test whe percetage of X s outlers are decreased (0.983) ad small szes [Fg. (a)]. Table 5. The Values of Statstcs for Detecto of Outlers by Szes ad Percetage of Y s Outlers wth Four Szes Outlers d r r () h CD t DEFFIT MD %Outler/sample sz Fgure.1 A Comparso of Statstcs for Detecto of Outlers by Szes wth Three (a) X s Outlers; (b) Y s Outlers; (c) Both X s ad Y s Outlers. 4. Results for Four The tables gve the best of outler detecto methods for dfferet sample szes ad the percetages of outler from 1,000 replcatos. The results of statstcs of eght outler detecto methods are as followg; From table 5, the best of Y s outler detecto s r () method. The performace of r () method s hghest values of detecto outler (00) large szes ad hgh percetage of Y s outlers. Wth small szes CD method s performs much ISBN: IMECS 009

5 Proceedgs of the Iteratoal MultCoferece of Egeers ad Computer Scetsts 009 Vol I IMECS 009, March 18-0, 009, Hog Kog better tha others method ad the values of detecto outler whe low percetage of Y s outlers are (0.50). Wth large szes the performace of r ad d methods have hgh values of the test whe hgh percetages of Y s outler are (00) [Fg. (b)]. Table 6. The Values of Statstcs for Detecto of Outlers by Szes ad Percetage of both X s ad Y s Outlers wth Four Szes Outlers d r r () h CD t DEFFIT MD dstace ( CD ) ad DEFFIT. The DEFFIT method has more the values of detecto outler whe less tha outlers. Although show r (), CD ad MD methods are clearly favorable to outler detecto methods, gve our methods success the detfcato of outlers. They ca also be cosdered for use estmato. Oe ca estmate the regresso coeffcets wth outlers by applyg the robust regresso. The estmato method s applyg a dow weghg approach would be worthwhle. (a) X's Outlers value of outler detecto h() CD() DEFFIT() MD() (b) %Outler/sample sz Y's outlers value of outler detecto %Outler/sample sz d() r r() CD() From table 6, the best of both X s ad Y s outlers detecto are h ad MD methods. The performaces of h ad MD methods are hghest values of detecto outler (00) other sample szes ad percetage of outlers. Wth small szes, the outler detecto of t method has a hgh value of detecto outler (0.999). Wth large szes, the outler detecto of the r () ad CD methods are hgh values of the test whe the hgh percetages of outlers. The performace of hghest values of the test s (00) [Fg. (c)]. (c) value of outler detecto Both X's ad Y's Outlers %Outler/sample sz r() h() CD() MD() 5. CONCLUSION AND RECOMMENDATIONS The results from the Mote Carlo smulato show the eght dfferet methods for detectg outlers. The best of Y s outlers are r () ad CD methods. Ths s mportat r () perform better tha CD, because r () maly show hgh values of the detecto outler of every the sample szes ad the percetages of Y s outlers. The ext best statstcs for detecto are d ad r methods. They have good outler detecto whe large sample szes ad hgh the percetage of Y s outlers. The h ad t methods have values of the detecto outler wth small sample szes, but compromsed outler detecto whe the large sample sze ad the percetages of outlers are creased. The best of X s ad both X s ad Y s outlers s MD method. It has the hghest values of detecto outler whe the presece the sample szes are small, medum ad large szes. The ext best statstcs for the detecto are Hat matrx ( h ), Cook s square Fgure. A Comparso of Statstcs for Detecto of Outlers by Szes wth Four (a) X s Outlers; (b) Y s Outlers; (c) Both X ad Y s Outlers. REFERENCES [1] Atkso, A.C., I: Plots, Troasformatos, ad Regresso: A Itroducto to Graphcal Methods of Dagostc Regresso Aalyss, Oxford : Claredo Press, [] Barett, V. ad Lews, T., Outlers Statstcal Data, 3rd ed. UK : Wley, Chcester, [3] Brkes, D. ad Dodge, Y., Alteratve Methods of Regresso, New York : Joh Wley & Sos, [4] Cambell, N.A., Robust Procedures Multvarate Aalyss I: Robust Covarace estmato, Appl. Stat. Vol. 9, 1980, pp [5] Had, A. S., Idetfyg Multple Outlers Multvarate Data, J. Roy. Statst. Soc. Ser B. Vol. 54, 199, pp [6] Had, A. S., A Modfcato of a Method for the Detecto of Outlers Multvarate s, J. Roy. Statst. Soc. Ser B. Vol. 56, 1994, pp [7] Had, A.S. ad Smooff, J.S., Procedures for the Idetfcato of Multple Outlers Lear Models, J. Ammer. Statst. Assoc. Vol. 88, 1993, pp ISBN: IMECS 009

6 Proceedgs of the Iteratoal MultCoferece of Egeers ad Computer Scetsts 009 Vol I IMECS 009, March 18-0, 009, Hog Kog [8] Huber, P.J., Robust Statstc, New York : Joh Wley & Sos, [9] Kafard, F. ad Swallow, W., A Mote Carlo Comparso of Fve Procedures for Idetfyg Outlers Lear Regresso, Commu. Statst. Part A Theory Methods. Vol. 19, 1990, pp [10] Kossk, A. S., A procedure for the detecto of multvarate outlers, Computatoal Statstcs ad Data Aalyss. Vol. 9, 1998, pp [11] Lopuhaa, H.P. ad Rousseeuw, P.J., Breakdow Pot of Affe Equvarat Estmators of Maultvarate Locato ad Covarace Matrce, Techcal Report, Faculty of Mathematcs ad Iformatcs, Netherlads: Delft Uversty of Techology, [1] Maroa, R.A., Robust M-estmates of Multvarate Locato ad Scatter, A. Stat. Vol. 4, 1976, pp [13] Maroa, R. A., Mart, D. R. ad Yoha, V. J., Robust Statstcs - Theory ad Methods, New York: Joh Wley & Sos, 006. [14] McKea, J. W., Sheather, S. J. ad Hettmasperger, T. P., Robust ad Hgh-Breakdow Fts of Polyomal Models, Techometrcs. Vol. 36, 1994, pp [15] Motgomery, D. C., Peck, E. A., ad Vg, G. G., Itroducto to Lear Regresso Aalyss, 3rd ed. New York: Joh Wley & Sos, 003. [16] Ra, M. ad Atkso, A.C., Robust Dagostc Data Aalyss: Trasformatos Regresso, Techometrcs. Vol. 44, 000, pp [17] Rya, T.P., Moder Regresso Methods, New York: Joh Wley & Sos, [18] Rousseeuw, P.J., Least Meda of Squares Regresso, J. Ammer. Statst. Assoc. Vol. 79: 1984, pp [19] Rousseeuw, P.J. ad Leroy, A.M., Robust Regresso ad Outler Detecto, New York : Joh Wley & Sos, [0] Rousseeuw, P.J. ad Zomere, B.C.V., Umaskg Multvarate Outlers ad Leverage Pots, J. Ammer. Statst. Assoc. Vol. 85, 1990, pp [1] Rousseeuw, P.J. ad Dresse, K. V., A Fast Algorthm for the Mmum Covarace Determat Estmator, Techometrcs. Vol. 41, 1999, pp [] Sebert, D.M., Idetfyg Multple Outlers ad Ifluetal Subsets: A Clusterg Approach, AZ: Upublshed Dssertato, Arzoa State Uversty, [3] Se, A. ad Srvastava, M., Regresso Aalyss: Theory, Methods, ad Applcatos, New York: Sprger-Verlag, [4] Wsowsk, J. W., Motgomery, D. C. ad Smpso, J. R., A Comparatve Aalyss of Multple Outler Detecto Procedures the Lear Regresso Model, Computatoal Statstcs ad Data Aalyss. Vol. 6, 001, pp [5] Woodruff, D.L. ad Rocke, D.M. Computable Robust Estmateo of Multvarate Locato ad Shape Hgh Dmeso Usg Compoud Estmator, J. Ammer. Statst. Assoc.Vol. 89, 1994, pp [6] You, J., A Mote Carlo comparso of several hgh breakdow ad effcet estmators, Computatoal Statstcs ad Data Aalyss. Vol. 30, 1999, pp ACKNOWLEDGMENT Oe of us would lke to thak Professor Prachoom Suwattee for helpful dscusso of ths materal, ad commets o a earler draft of ths paper. I partcularly thak Dr. Yosapol Rathamart ad Mr. Sutt Hopetrugruag for valuable suggesto ad support, ad Mss Supapor Tuporptuk for kdly provdg the SAS program. The remag errors belog to the author. ISBN: IMECS 009