JMASM42: An Alternative Algorithm and Programming Implementation for Least Absolute Deviation Estimator of the Linear Regression Models (R)

Transcription

1 Journal of Modern Appled Statstcal Methods Volume 15 Issue 2 Artcle JMASM42: An Alternatve Algorthm and Programmng Implementaton for Least Absolute Devaton Estmator of the Lnear Regresson Models (R) Suraju Olany Ogundele Federal Unversty of Petroleum Resources, Effurun, Delta State, Ngera., ogundele.olany@fupre.edu.ng J. I. Mbegbu Department of Mathematcs, Unversty of Benn, Benn-Cty, Edo State, Ngera. C. R. Nwosu Department of Statstcs, Nnamd Azkwe Unversty, Awka, Anambra State, Ngera. Follow ths and addtonal works at: Part of the Appled Statstcs Commons, Socal and Behavoral Scences Commons, and the Statstcal Theory Commons Recommended Ctaton Ogundele, Suraju Olany; Mbegbu, J. I.; and Nwosu, C. R. (2016) "JMASM42: An Alternatve Algorthm and Programmng Implementaton for Least Absolute Devaton Estmator of the Lnear Regresson Models (R)," Journal of Modern Appled Statstcal Methods: Vol. 15 : Iss. 2, Artcle 45. DOI: /jmasm/ Avalable at: Ths Algorthms and Code s brought to you for free and open access by the Open Access Journals at DgtalCommons@WayneState. It has been accepted for ncluson n Journal of Modern Appled Statstcal Methods by an authorzed edtor of DgtalCommons@WayneState.

2 JMASM42: An Alternatve Algorthm and Programmng Implementaton for Least Absolute Devaton Estmator of the Lnear Regresson Models (R) Cover Page Footnote *Correspondent Author E-mal/Phone Number: Erratum Ths paper was orgnally publshed n JMASM Algorthms & Code wthout ts enumeraton, JMASM42. Ths algorthms and code s avalable n Journal of Modern Appled Statstcal Methods: vol15/ss2/45

3 Journal of Modern Appled Statstcal Methods November 2016, Vol. 15, No. 2, do: /jmasm/ Copyrght 2016 JMASM, Inc. ISSN An Alternatve Algorthm and R Programmng Implementaton for Least Absolute Devaton Estmator of the Lnear Regresson Models Suraju Olany Ogundele Federal Unversty of Petroleum Resources Effurun, Ngera. J. I. Mbegbu Unversty of Benn Benn-Cty, Ngera C. R. Nwosu Nnamd Azkwe Unversty Awka, Ngera We propose a least absolute devaton estmaton method that produced a least absolute devaton estmator of parameter of the lnear regresson model. The method s as accurate as exstng method. Keywords: Lnear regresson model, least absolute devaton (LAD), equaton of a lne, R statstcal programmng and algorthm Introducton Regresson s a statstcal methodology that s use to relate a varable of nterest, whch s called the dependent varable or response varable, to one or more predctors (ndependent/regressors) varables. The objectve of regresson analyss s to buld a regresson model or predcton equaton that helps us to descrbe, predct and control the dependent varable on the bass of the ndependent varable(s). When we predct the dependent varable for a partcular set of values of the ndependent varables, we wsh to place a bound on the error of predcton. The goal s to buld a regresson model that produces an error bound that wll be small enough to meet our needs. In the smple lnear regresson model, the Populaton Regresson Functon (PRF) s gven by: y x (1) 0 1 Mr. Ogundele s an Assstant Lecturer n the Department of Mathematcs / Computer Scence. Emal hm at: ogundele.olany@fupre.edu.ng. J. I. Mbegbu s a Professor n the Department of Mathematcs. C. R. Nwosu s an Assocate Professor n the Department of Statstcs. 755

4 ALGORITHM AND CODE FOR LAD ESTIMATOR In ths model there s only one factor x to explan y. All the other factors that affect y are jontly captured by the error term denoted by ε. We typcally refer to y as the endogenous or dependent varable and x as the exogenous or ndependent varable. The dea of the regresson model s to estmate the populaton parameters, β0 and β1 from a gven sample. The Sample Regresson Functon (SRF) s the sample counterpart of the populaton regresson functon (PRF). Snce the SRF s obtaned for a gven sample, a new sample wll generate dfferent estmates. The SRF, whch s an estmaton of the PRF s gven by: yˆ ˆ ˆ x (2) 0 1 Equaton (2) s used to calculate the ftted value y ˆ for y when x = x. In the SRF ˆ 0 and ˆ 1 are estmators of the parameters β0 and β1. For each x we have an observed value (y) and a ftted value y called the resdual ˆ gven by: ˆ. The dfference between y and y ˆ s ˆ y yˆ (3) The ordnary least squares (OLS) method s the most wdely used method of parameter estmaton. The OLS crtera s to mnmze the sum of squared error of predcton y yˆ 2 ˆ (4) 2 OLS regresson yelds estmates for the parameters that have the desrable property of beng mnmum varance unbased estmators (Chatterjee & Had, 2006). Ordnary least squares estmaton places certan restrctve assumptons on the random component n the model, the errors of predcton. OLS estmaton assumes, among others, that the errors of predcton are normally dstrbuted, wth a common error varance at all levels of X [ε ~ N (0, σ 2 )]. The normalty assumpton s frequently untenable n practce. Volaton of ths assumpton s often manfested by the presence of outlers n the observed data (Nevtt & Tam, 1998). Thus data contanng outlyng values may reflect non-normal error dstrbutons wth heavy tals or normal error dstrbutons contanng observatons 756

5 OGUNDELE ET AL. atypcal of the usual normal dstrbuton wth larger varance than the assumed σ 2. It s well demonstrated that outlers n the sample data heavly nfluence estmates usng OLS regresson, sometmes even n the presence of one outler (Rousseeuw & Leroy, 1987). If the assumpton that the uncertantes (.e., errors) n the data are uncorrelated and normally dstrbuted are vald for the data at hand, then for most quanttatve experments, the method of least squares s the "best" analytcal technque for extractng nformaton from a set of data. The method s best n the sense that the parameters determned by the least squares analyss are normally dstrbuted about the true parameters wth the least possble standard devatons (Wolberg, 2006). However, the assumpton of the general applcablty of the normal law of errors has been under attack from the very begnnng of the development of lnear regresson and, n partcular, the least squares analyss hnges crtcally on the exstence of the second moment of the error dstrbuton. Thus, f t must assumed the error dstrbuton follows, for nstance, a Cauchy dstrbuton or any longtaled dstrbuton havng no fnte second moment, then the elegant arguments made n favour of the least squares regresson estmators become nvald, and thus, t may become mandatory to look for other crtera to fnd best estmators for the lnear regresson model (Glon & Padberg, 2002). For stuatons n whch the underlyng assumptons of OLS estmaton are not tenable, the choce of method for parameter estmaton s not clearly defned. Thus, the choce of estmaton method under non-deal condtons has been a long-standng problem for methodologcal researchers (Nevtt & Tam, 1998). The hstory of ths problem s lengthy wth many alternatve estmaton methods havng been proposed and nvestgated (Brkes & Dodge, 1993). Robust estmaton refers to the ablty of a procedure to produce hghly nsenstve estmates to model msspecfcatons. Hence, robust estmates should be good under wde range of possble data generatng dstrbutons. In the regresson context, under normalty wth dentcally and ndependently dstrbuted errors, the least squares s the most effcent among the unbased estmaton methods. However, when the normalty assumpton not feasble, t s frequently possble to fnd estmaton methods that are more effcent than the tradtonai least squares. Ths occurs when the data generatng process has fat tals resultng to several outlers compared to the normal dstrbuton. In these cases the least squares becomes hghly unstable and sample dependent because of the quadratc weghtng, whch makes the procedure very senstve to outlyng observatons (Pynnonen & Salm, 1994). Examples of ths type of robust estmaton are Huber 757

6 ALGORITHM AND CODE FOR LAD ESTIMATOR M-estmaton, the method of Least Medan of Squares, and the method of Least Absolute Devatons (LAD). The robust LAD estmator s nvestgated n the present study and so we provde a bref descrpton of the method. LAD was developed by Roger Joseph Boscovch n 1757, nearly 50 years before OLS estmaton (see Brkes & Dodge, 1993 for a revew and hstorcal ctatons). In contrast to OLS estmaton whch defnes the loss functon on the resduals as Σe 2, LAD fnds the slope and Y ntercept that mnmze the sum of the absolute values of the resduals, Σ e. Although the concept of LAD s not more dffucult than the concept of the OLS estmaton but due to computatonal dffcultes n obtanng LAD estmates and lack of exact samplng theory based on such estmates, the LAD method lay n the background and the LS method became popular (Rao & Toutenburg, 1999). Snce there are no exact formulas for LAD estmates, an algorthm s used to teratvely obtan the estmate of the parameters. Methodology Propose Method Usng the LAD crteron, the model y ˆ ˆ ˆ 0 1x should be constructed by two pars of data ponts that yeld the mnmum sum of absolute devaton, our approach s to nvestgate the sum of absolute devaton generated by all possble dfferent combnatons of data ponts and then select the two data ponts that produced the least absolute devaton to fnd the least absolute devaton estmator. The two ponts (x, y) and (xj, yj) yeld the followng system of equatons: y y j ˆ ˆ x 0 1 ˆ ˆ x 0 1 j (5) The soluton of equaton (5) yeld the value of ˆ 0 and ˆ 1, for the selected par of data. Subttutng the value of ˆ 1 and ˆ 0 obtaned from (5) nto (1), we have y ˆ ˆ x (6) ˆ

7 OGUNDELE ET AL. (6) s then used to determne the sum of absolute devaton of all other data ponts from the lne jonng (x, y) and (xj, yj) by substtutng for x ( = 1, 2,, n) of all data ponts and calculatng n y ˆ y. (7) 1 Ths procedure wll be repeated for all other combnatons of data ponts and the two data ponts that yelded the least absolute devaton determne the least absolute devaton estmator. The presence of personal computers make t possble to evaluate repeated process usng any programmng language of choce. The R program s employed for all calculatons, and the program yelds the least absolute devaton estmate for the data. Algorthm for Smple Lnear Regresson INPUT: Observatons of x and y as vectors X and Y. OUTPUT: Slope and ntercept. Step 1. Set = 1 and j = 2 Step 2. Whle (n 1); j n; j, do steps 3 to 4 Step 3. Select the pars (x, y) and (xj, yj) from the data and calculate the values of 1 and ˆ 0 from the system of equatons gven by (5). Step 4. For = 1 to n, determne the estmated value of ˆ substtutng for (x, y) n (6) and calculate AbsDev, j y yˆ n 1 y y by Step 5. Determne the mnmum value among all AbsDev (, j) and select the two data ponts that produced the mnmum value. Step 6. Prnt out the values of ˆ 1 and ˆ 0 that correspond to the two selected data ponts. Step 7. If > (n 1) and j > n, then OUTPUT; stop. 759

8 ALGORITHM AND CODE FOR LAD ESTIMATOR Step 8. Set = + 1; j + 1 and go to step 2. Step 9. OUTPUT Method faled after > (n 1) and j > n". Ths algorthm s mproved upon for multple lnear regresson and can be scaled to accommodate the numbers of ndependent varables present n the data. We provde the algorthm and the R program for the smplest form of the multple regresson wth two ndependent varables Algorthm for Multple Lnear Regresson INPUT: Observatons of x1, x2 and y as vectors X1, X2 and Y. OUTPUT: Intercept, frst parameter and second parameter. Step 1. Set = 1, j = 2 and k = 3 Step 2. Whle (n 2); j (n 1); k n; j k, do steps 3 to 4 Step 3. Select the pars (x, y), (xj, yj) and (xk, yk) from the data and calculate the values of ˆ 2, 1 and ˆ 0 from the system of equatons y ˆ ˆ x ˆ x y ˆ ˆ x ˆ x j j 2 2 j y ˆ ˆ x ˆ x k 0 1 1k 2 2k (8) Step 4. For = 1 to n, determne the estmated value of y y ˆ by substtutng for (x1, x2, y) n yˆ ˆ ˆ x ˆ x (9) and calculate AbsDev, j, k y yˆ n 1 Step 5. Determne the mnmum value among all AbsDev (, j, k) and select the three data ponts that produced the mnmum value. 760

9 OGUNDELE ET AL. Step 6. Prnt out the values of ˆ 2, ˆ 1 and ˆ 0 that correspond to the three selected data ponts. Step 7. If > (n 2), j > (n 1) and k > n, then OUTPUT; stop. Step 8. Set = + 1; j = j + 1; k = k + 1 and go to step 2. Step 9. OUTPUT Method faled after (n 2), j (n 1) and k n". Results The R program desgned by applyng ths algorthm s presented n the Appendx. Applcaton of the R program wrtten for the algorthm to the data n Table 1 yelded the same results wth the teratve method by Brkes and Dodge (1993). The best two data ponts are gven to be at (x5, y5) and (x14, y14). The least absolute devaton estmate of the model parameters are ˆ LAD ˆ LAD Then, the LAD regresson lne s yˆ x Table 1. Brth Rate Data Country Brth Rate (y) Urban Percentage (x) Canada Costa Rca Cuba Domncan Republc El Salvador Guatemala Hat Honduras Jamaca Mexco Ncaragua Panama Trndad-Tobago Unted States

10 ALGORITHM AND CODE FOR LAD ESTIMATOR The R program wrtten for the multple regresson mplementaton of the proposed method was appled to fnd the least absolute devaton estmate of the parameter of subset of the supervsor data (Chatterjee & Had, 2006) whch ncludes Y, X1 and X2. The data s presented n Table 2. The best three data ponts are gven to be at (x8, y8), (x9, y9) and (x21, y21). The program gves the least absolute devaton estmate of the parameters as ˆ LAD ˆ LAD ˆ LAD Table 2. Subset of Supervsor Data Y X1 X

11 OGUNDELE ET AL. Concluson The proposed method produced a least absolute devaton estmate that s the same as the one provded by the teratve method by Brkes and Dodge (1993) and other exstng methods References Brkes, D., & Dodge, Y. (1993). Alternatve methods of regresson. New York, NY: John Wley & Sons Inc. Chatterjee, S., & Had, A. S. (2006). Regresson analyss by example. New York, NY: John Wley & Sons Inc. Glon A., & Padberg, M. (2002). Alternatve methods of lnear regresson. Mathematcal and Computer Modellng, 35(3/4), do: /S (01) Nevtt, J., & Tam, H. P. (1998). A comparson of robust and nonparametrc estmators under the smple lnear regresson model. Multple Lnear Regresson Vewponts, 25, Pynnonen, S. & Salm, T. (1994). A report on least absolute devaton regresson wth ordnary lnear programmng. The Fnnsh Journal of Busness Economcs, (1), R Core Team (2013). R: A language and envronment for statstcal computng. R Foundaton for Statstcal Computng, Venna, Australa. Retreved from Rao, R. C. & Toutenburg, H. (1999). Lnear models: Least squares and alternatves (2 nd ed.). New York, NY: Sprnger-Verlag, Inc. Rousseeuw, P. J. & Leroy, A. M. (1987). Robust regresson and outler detecton. New York, NY: John Wley & Sons. Wolberg, J. (2006). Data analyss usng the method of least squares. Berln Hedelberg: Sprnger-Verlag. 763

12 ALGORITHM AND CODE FOR LAD ESTIMATOR Appendx # Smple Lnear Regresson # Least Absolute Devaton Estmator (LAD) Y<-c(16.2,30.5,16.9,33.1,40.2,38.4,41.3,43.9,28.3,33.9,44.2,28,24.6,16) X<-c(55,27.3,33.3,37.1,11.5,14.2,13.9,19,33.1,43.2,28.5,37.7,6.8,56.5) p<-1 n<-length(y) Const<-rep(1,(p+1)) l<-0;abserror<-0;esterror<-0;veclad<-c();vecj<-c();veck<-c();vecx<-c();vecy<c();vecb0<-c();vecb1<-c() TralModel1<-functon(X,Y,B0,B1){(abs(Y-(B0+(B1*X)))) LAD1<-functon(n,X,Y,B0,B1){ for ( n 1:n){X<-X[];Y<-Y[] EstError<-TralModel1(X,Y,B0,B1) AbsError<-AbsError+EstError EstLAD<-AbsError Kstart<-2 for (j n 1:(n-1)){xj<-X[j];yj<-Y[j]; for (k n Kstart:n){xk<-X[k];yk<-Y[k] f(k==n){kstart<-kstart+1 Vecx<-c(xj,xk) Vecy<-c(yj,yk) A<-cbnd(Const,Vecx) Det<-round(det(A),4) f(det!=0){ Beta<-solve(A,Vecy) B0<-Beta[1] B1<-Beta[2] LADEst<-LAD1(n,X,Y,B0,B1) l<-l+1 VecLAD[l]<-c(LADEst) Vecj[l]<-c(j) Veck[l]<-c(k) VecB0[l]<-c(B0) VecB1[l]<-c(B1) 764

13 OGUNDELE ET AL. LADLoc<-sort.nt(VecLAD,ndex.return=TRUE) ParLoc<-LADLoc$x[1] Par1<-Vecj[ParLoc] Par2<-Veck[ParLoc] Intercept<-round(VecB0[ParLoc],5) Slope<-round(VecB1[ParLoc],5) Label1<-"Best two data ponts for LAD estmate are";label2<-"and" Label3<-"The ntercept of the LAD regresson s" Label4<-"The slope of LAD regresson s" Label1;Par1;Label2;Par2;Label3;Intercept;Label4;Slope # Multple Lnear Regresson # Least Absolute Devaton Estmator (LAD) Y<c(43,63,71,61,81,43,58,71,72,67,64,67,69,68,77,81,74,65,65,50,50,64,53,40,63,66, 78,48, 85,82) X1<c(51,64,70,63,78,55,67,75,82,61,53,60,62,83,77,90,85,60,70,58,40,61,66,37,54,77, 75,57, 85,82) X2<c(30,51,68,45,56,49,42,50,72,45,53,47,57,83,54,50,64,65,46,68,33,52,52,42,42,66, 58,44, 71,39) p<-2 n<-length(y) Const<-rep(1,(p+1)) l<-0;abserror<-0;esterror<-0;veclad<-c();vecb0<-c();vecb1<-c();vecb2<c();vecx1<-c();vecx2<-c();vecy<-c();vecj<-c() Veck<-c();Vecv<-c() TralModel2<-functon(Y,X1,X2,B0,B1,B2){(abs(Y-(B0+(B1*X1)+(B2*X2)))) LAD2<-functon(n,Y,X1,X2,B0,B1,B2){ for ( n 1:n){X1<-X1[];X2<-X2[];Y<-Y[] 765

14 ALGORITHM AND CODE FOR LAD ESTIMATOR EstError<-TralModel2(Y,X1,X2,B0,B1,B2) AbsError<-AbsError+EstError EstLAD<-AbsError Kstart<-1;Vstart<-2;EndCount<-3 for (j n 1:(n-2)){x1j<-X1[j];x2j<-X2[j];yj<-Y[j]; Kstart<-Kstart+1 for (k n Kstart:(n-1)){x1k<-X1[k];x2k<-X2[k];yk<-Y[k] f(vstart<n){ Vstart<-Vstart+1 else{ EndCount<-(EndCount+1) Vstart<-EndCount for (v n Vstart:n){x1v<-X1[v];x2v<-X2[v];yv<-Y[v] Vecx1<-c(x1j,x1k,x1v) Vecx2<-c(x2j,x2k,x2v) Vecy<-c(yj,yk,yv) A<-cbnd(Const,Vecx1,Vecx2) Det<-round(det(A),4) f(det!=0){ Beta<-solve(A,Vecy) B0<-Beta[1] B1<-Beta[2] B2<-Beta[3] LADEst<-LAD2(n,Y,X1,X2,B0,B1,B2) l<-l+1 VecLAD[l]<-c(LADEst) VecB0[l]<-c(B0) VecB1[l]<-c(B1) VecB2[l]<-c(B2) Vecj[l]<-c(j) Veck[l]<-c(k) Vecv[l]<-c(v) 766

15 OGUNDELE ET AL. LADLoc<-sort.nt(VecLAD,ndex.return=TRUE) ParLoc<-LADLoc$x[1] FrstPont<-Vecj[ParLoc] SecondPont<-Veck[ParLoc] ThrdPont<-Vecv[ParLoc] Constant<-VecB0[ParLoc] Beta1<-VecB1[ParLoc] Beta2<-VecB2[ParLoc] Label1<-"The frst LAD data pont s";label2<-"the second data pont s";label3<-"the thrd data pont s" Label4<-"The constant of LAD regresson model s";label5<-"the frst parameter s" Label6<-"The second parameter s" Label1;FrstPont;Label2;SecondPont;Label3;ThrdPont;Label4;Constant;Label5;Be ta1;label6;beta2 767