Weak Dependence on Initialization in Mixture of Linear Regressions

Transcription

1 Proeedings of the International MultiConferene of Engineers and Computer Sientists 8 Vol I IMECS 8, Marh -6, 8, Hong Kong Weak Dependene on Initialization in Mixture of Linear Regressions Ryohei Nakano and Seiya Satoh Abstrat This paper examined how initial values of a learning algorithm influene the performane of mixture of linear regressions (MoLR). MoLR employs the EM algorithm as a learning algorithm. Our experiments used two kinds of artifiial data and one real dataset. Experiments using artifiial data showed MoLR disovered the original lines from data ontaining Gaussian or t-distribution noise. Moreover, almost all our experiments showed the best solution an be easily found with any initialization method. This suggests MoLR may have rather weak dependene on initialization. Index Terms mixture model, mixture of regressions, EM algorithm, lustering, information riterion I. INTRODUCTION This paper examines how initial values of the learning, here the EM algorithm [], influene the final performane of mixture of linear regressions (MoLR). First, mixture of regressions (MoR) is briefly reviewed. Given multivariate data, onsider a regression problem, that is, try to explain the behavior of a target variable using explanatory variables. It may happen that any single regression funtion annot explain well. There an be several reasons for this: a lak of explanatory variables, the intrinsi randomness of the target variable, and so on. When data arise from heterogeneous ontexts, it is reasonable to introdue MoR. There an be two types of the mixture. One is hard mixture and the other is soft mixture. In the former eah data point is exlusively lassified into one of the lasses, while in the latter eah data point probabilistially belongs to every lass. Hard mixture has been alled regression lustering [], [] or luster-wise (linear) regression []. Soft mixture has been simply alled mixture of regressions. Researh on hard MoR has been less popular, while soft MoR has the solid bakground developed as finite mixture models. Finite mixture models provide a flexible tool for modeling data that arise from heterogeneous populations. The book by MLahlan and Peel [] ontains a omprehensive review of finite mixture models. Leish gave a general framework for MoR in the R statistial omputing environment [6]. Huang, Li, and Wang investigated MoR by employing kernel regression [7]. The book by Bishop [8] ontains mixture models inluding MoLR. Next, how a learning method for MoLR depends on the initialization is briefly reviewed. As a learning method, hard MoLR usually employs Späth s exhange algorithm [], while soft MoR usually employs the EM algorithm This work was supported by Grants-in-Aid for Sientifi Researh (C) 6K. R. Nakano is with the Department of Computer Siene, Chubu University, Matsumotoho, Kasugai 87-8, Japan. nakano@s.hubu.a.jp S. Satoh is with National Institute of Advaned Industrial Siene and Tehnology, --7 Aomi, Koto-ku, Tokyo, -6, Japan. seiya.satoh@aist.go.jp []. It is known that the suess of the Späth s exhange algorithm depends heavily on the initial onfiguration []. Qian and Wu [] proposed a method to generate an initial onfiguration for hard MoLR. On the other hand, the loal optimality of the EM algorithm is well known, and in general its performane depends heavily on its initial values. To overome its optimality, several EM variants have been proposed, suh as DAEM [9] and SMEM []. Thus, the initial onfiguration for the EM algorithm may play a large role in soft MoLR. This paper examines how initial values of the EM algorithm influene the performane of soft MoLR. Setion explains the framework of soft MoLR, and Setion desribes initialization and model seletion. In the initialization we propose a new method to generate regressionoriented initial values for MoLR, and in model seletion we explain information riterion to evaluate the desirability of MoLR andidates. Then Setion desribes our experiments performed to examine how initial values of the EM algorithm influene the performane of MoLR using two kinds of artifiial data and one real dataset. II. MIXTURE OF LINEAR REGRESSIONS (MOLR) Following Bishop [8], this setion explains the framework of soft MoLR, whih employs the EM algorithm [] as a learning method. We onsider a mixture of C linear regression models. Let x = (x,, x K ) T be explanatory variables, and y denotes a target variable. Sine we onsider a onstant term in eah regression funtion, we extend a vetor of explanatory variables as x = (, x,, x K ) T. We assume the value of y is generated by adding a Gaussian noise to a value of linear regression funtion f(x w ) with w as its weight vetor. A linear regression funtion of lass is defined as follows. f(x w ) = w T x () Although Bishop assumes the ommon error variane (equivalently, a preision parameter) for all lasses, we introdue individual variane for eah lass to enhane the fitting apability of the model. That is, for lass, an error ε follows the Gaussian with mean and variane σ. ε N (, σ ) () Class is a latent variable and annot be observed. The target variable y follows the following distribution. y N (f(x w ), σ ) () Let π be the mixing oeffiient of lass. Then, the density of omplete-data is desribed as follows. p(y, θ ) = π g (y f(x w ), σ ) () ISBN: ISSN: (Print); ISSN: (Online) IMECS 8

2 Proeedings of the International MultiConferene of Engineers and Computer Sientists 8 Vol I IMECS 8, Marh -6, 8, Hong Kong Here g(u m, s ) denotes a density funtion where u follows one-dimensional Gaussian with mean m and variane s. ) g(u m, s (u m) ) = exp ( π s s () The density of inomplete-data is written as follows. p(y θ) = C p(y, θ ) = = C π g (y f(x w ), σ ) (6) = Here θ is a vetor omprised of all parameters, while θ is a vetor of lass parameters. θ = (θ T,, θ T,, θ T C) T, θ = (π, w T, σ ) T (7) Now the posterior probability is written as follows, whih indiates the probability that y belongs to lass. P ( y, θ) = p(y, θ) p(y, θ) (8) Given data D = {(x, y ), =,, N}, the inompletedata log likelihood is defined as below. L(θ) = N = log p(y θ) (9) When we employ the EM algorithm to estimate MoLR parameters, the Q-funtion to maximize is shown as below. Here θ (t) denotes the estimate at the t step of the EM algorithm, and let f f(x w ). Q(θ θ (t) ) = P ( y, θ (t) ) log p(y, θ) = P (t) log(π g (y f, σ )) = ( P (t) log π log(π) log σ (y f ) ) σ () In the above, we use the following for brevity. P ( y, θ (t) ) = π(t) where g (t) π(t) g (t) g (t) () g (y f (t), σ (t) ) () When we maximize the Q-funtion, we use the Lagrange method sine there is an equality onstraint π =. The Lagrangian funtion an be written as follows with λ as a Lagrange multiplier. ( ) J = Q(θ θ (t) ) λ π () The neessary ondition for a loal maximizer is shown below. J π = J w = J σ = /π λ = () (y f ) f σ = () w ( σ + (y f ) ) = (6) σ Sine we have λ = N from eq.() and the equality onstraint, a new estimate of π is given below. π (t+) = P (t) (7) N From eq.(6) a new estimate of σ is given below. (σ ) (t+) = (y f ) / (8) From eq.() we obtain a new estimate of w by solving the following. P (t) (y f ) f = (9) w Here in this paper we onsider linear regression models as mixture elements. So by substituting eq.() into the above, we have the following. P (t) (y w T x ) x = () This formula is transformed in suession as below. P (t) (w T x ) x = P (t) y x x ( x T w ) = ( x x T )w = y x y x () Here data of explanatory variables is expressed as the following N (K + ) matrix, and data of a target variable is expressed as the following N vetor. X = ( x,, x,, x N ) T () y = (y,, y,, y N ) T () For eah lass, we onsider N N diagonal matrix S (t) = diag(p (t) ), whose -th diagonal element is P (t). Then eq.() an be written as follows. ( X T S (t) X) w = X T S (t) y () When there is the inverse matrix of the above equation, we have the following new estimate of weights. w (t+) = ( X T S (t) X) XT S (t) y () III. INITIALIZATION AND MODEL SELECTION A. Initialization of MoLR We onsider two methods for the initialization of MoLR. One is random initialization (R init), where eah data point is randomly assigned to one of the lasses. The other is a new method whih lassifies data points into lusters taking into aount the height of target values. The method is alled height-sensitive initialization (HS init). The proedure goes as follows. First, perform linear regression using all data points and lassify them into upper, lower, and remaining groups. Then the upper group is grouped into two sets U and U, and the lower group is also grouped into L and L. The remaining group R onsists of data points near the regression plane. Finally, by ombining these five sets, we make initial onfigurations of MoLR. The point of ISBN: ISSN: (Print); ISSN: (Online) IMECS 8

3 Proeedings of the International MultiConferene of Engineers and Computer Sientists 8 Vol I IMECS 8, Marh -6, 8, Hong Kong this method is to separate data into upper, middle, and lower groups; suh separation is expeted be effetive to make good initial onfigurations of MoLR. HS initialization making lasses step. Perform linear regression using all data points. step. If a data point is more than tolerane δ higher than the regression plane, it is assigned to the set U, and if the point is more than δ lower than the plane, it is assigned to L, and if it is near the plane, it is assigned to R. step. U is grouped into two sets U and U, and L is grouped into L and L as well. We employ Kmeans [] for this lustering. step. By ombining four sets exept seemingly less relevant R, we make the following 7 initial onfigurations. No. : {U, U } {L, L } No. : {U, L } {U, L } No. : {U, L } {U, L } No. : {U } {U, L, L } No. : {U } {U.L, L } No.6 : {L } {U, U, L } No.7 : {L } {U, U, L } HS initialization making lasses steps, and are the same as the above. step. By ombining all five sets with (,,) pattern, we make the initial onfigurations suh as {R} {U, U } {L, L } and {R} {U, L } {U, L }. B. Model Seletion In the ontext of MoR, we onsider many andidates of mixture models; thus, we need a riterion to evaluate the desirability of eah andidate and to selet the best mixture model. For this purpose we make use of information riterion. Although many information riteria have been proposed so far, we employ the Bayesian information riterion [], beause stably showed good performane in our experiments on MLP model seletion []. was proposed for regular models, but it also works rather well for singular models suh as multilayer pereptrons (MLPs). Let p(x w) be a learning model with parameter vetor w. Given data {x, =,, N}, the log-likelihood is defined as follows: N L N (w) = log p(x w). (6) = Let ŵ be a maximum likelihood estimator. is obtained as an estimator of free energy F (D) shown below. Here p(d) is alled evidene and p(w) is a prior distribution of w. F (D) = log p(d), (7) N p(d) = p(w) p(x w) dw (8) = is derived using the asymptoti normality and Laplae approximation. = L N (ŵ) + M log N N = log p(x ŵ) + M log N (9) = an be alulated using only one point estimator ŵ. Here M is the number of parameters. As another standpoint, total sum of squares (TSS) indiates how muh variation the data have, residual sum of squares (RSS) indiates the disrepany between the data and the estimates, and explained sum of squares (ESS) indiates how well a regression model represents the data. These quantities satisfy the relation TSS = ESS + RSS. Thus, ESS/TSS indiates goodness of fit for a regression model. IV. EXPERIMENTS We performed experiments to examine how initial values of the EM algorithm influene the performane of MoLR using two kinds of artifiial data and one real dataset. As artifiial data, we utilized Qian-Wu s one-dimensional data []. Datasets were generated using the following two linear funtions: for eah dataset 7 and points were generated for y and y respetively with one of two kinds of noise added. y = + 8x, y = + x () Moreover, R init was repeated times for eah dataset. A. Experiment using Artifiial Data In artifiial data, standard Gaussian noise N(, ) was added to the lines shown in eq.(). We generated five datasets for artifiial data. Figure shows one dataset, depiting two original lines and data points original data (QW-KC-N-S) - - Fig.. Two Original Lines and Qian-Wu Data Generated with Gaussian N(,) Noise and Seed For all five datasets, riterion for lasses (C=) was smaller than those for C=. Figure shows the best results of our soft MoLR of C= for Fig. dataset. We an see two obtained lines fit well to data points. Table I ompares the original parameters with parameters obtained by our soft MoLR and hard MoLR by Qian-Wu []. The table shows both soft and hard MoLRs found good estimates. Moreover both HS and R inits for soft MoLR got muh the same results. Figure shows the best results of soft MoLR of C= for Fig. dataset. Three lines somehow over all data points. Figures and show histograms of values obtained by soft MoLR using HS init for C= and C=, while Figs. ISBN: ISSN: (Print); ISSN: (Online) IMECS 8

4 Proeedings of the International MultiConferene of Engineers and Computer Sientists 8 Vol I IMECS 8, Marh -6, 8, Hong Kong MOR results ( lasses) init onfig:6 Histram of with QW_N_S_HS_C Fig.. Data with HS Init,, and Gaussian N(,) Noise Histogram for QW Fig.. Histram of with QW_N_S_HS_C Histogram for QW Data with HS Init,, and Gaussian N(,) Noise Histram of with QW_N_S_R_C Histram of with QW_N_S_R_C Fig.. Two Lines of the Best MoLR Model of for Qian-Wu Data Generated with Gaussian N(,) Noise and Seed TABLE I ORIGINAL AND OBTAINED PARAMETERS w w Original (, 8) T (, ) T soft MoLR(HS, C=) (.8, 8.) T (.876,.77) T soft MoLR(R,C=) (.9, 8.) T (.876,.77) T Qian-Wu(C=) [] (., 8.) T (.76,.) T 6 and 7 show histograms of obtained by MoLR using R init. These figures show that the best solution of C= was better than that of C=. Moreover, for the best model (C=), HS init found the best solution times out of 7, and R init found the best solution 8 times out of. This means for the best model the best solution an be easily found with any initialization. B. Experiment using Artifiial Data In artifiial data, t-distribution noise t() was added to the lines shown in eq.(). We generated five datasets for artifiial data. Figure 8 shows one dataset, depiting two original lines and data points. Compared with Fig., data points are more widely sattered around the lines. For five datasets, the true model C= was seleted four times out of five. In Qian-Wu [] inorret C= was more Fig. 6. Data with R Init,, and Gaussian N(,) Noise Histogram for QW Fig Histogram for QW Data with R Init,, and Gaussian N(,) Noise frequently seleted than C=. Figure 9 shows the best results of our soft MoLR of C= for Fig. 8 dataset. We an see muh the same lines as original were obtained. Table II ompares the original parameters with parameters obtained by soft MoLR. The table shows soft MoLR found rather exat estimates. Both HS and R inits reahed exatly the same results. TABLE II ORIGINAL AND OBTAINED PARAMETERS w w Original (, 8) T (, ) T soft MoLR(HS, C=) (.66, 8.) T (.869,.) T soft MoLR(R, C=) (.66, 8.) T (.869,.) T Figure shows the best results of soft MoLR of C= for Fig. 8 dataset. Two lines out of three almost overlap eah other. Figures and show histograms of values obtained MOR results ( lasses) init onfig: original data (QW-KC-T-S) Fig.. Three Lines and the Best MoLR Model of for Qian-Wu Data Generated with Gaussian N(,) Noise and Seed Fig. 8. Two Original Lines and Qian-Wu Data Generated with t-dist t() Noise and Seed ISBN: ISSN: (Print); ISSN: (Online) IMECS 8

5 Proeedings of the International MultiConferene of Engineers and Computer Sientists 8 Vol I IMECS 8, Marh -6, 8, Hong Kong MOR results ( lasses) init onfig: Histram of with QW_T_S_R_C Fig.. Data with R Init,, and t- dist t() Noise histogram for QW Fig.. Histram of with QW_T_S_R_C histogram for QW Data with R Init,, and t- dist t() Noise Fig. 9. Two Lines of the Best MoLR Model of for Qian-Wu Data Generated with t-dist t() Noise and Seed MOR results ( lasses) init onfig: Fig.. Three Lines of the Best MoLR Model of for Qian-Wu Data Generated with t-dist t() Noise and Seed by our MoLR using HS init, while Figs. and show obtained by MoLR using R init. These figures show that the best solution of C= was surely better than that of C=. Moreover, for the best model (C=), HS init found the best solution times out of 7, and R init found the best solution times out of. Again this means that for the best model we an easily find the best solution with any initialization. numerial explanatory variables and the number of data points is 77 (N = 77). As a powerful single regression model, we employ multilayer pereptron (MLP) and a learning method alled SSF (singularity stairs following) [], []. SSF suessively learns MLP models to find very exellent solutions making good use of singular regions. SSF guarantees monotoni derease of training errors, thus is very suited to model seletion. Figure shows values for Abalone dataset by SSF. The figure indiates J =7 is the best model. at J =7 was,6. Multiple orrelation oeffiient at J =7 is.79; thus, the oeffiient of determination is.6, whih is not so high. Fig J for Abalone by SSF Histram of with QW_T_S_HS_C histogram for QW Fig.. Fig.. Data with HS Init,, and t-dist t() Noise C. Experiment using Real Data Histram of with QW_T_S_HS_C histogram for QW Data with HS Init,, and t-dist t() Noise As real data we used Abalone dataset from UCI Mahine Learning Repository. We seleted this beause any single regression funtion annot fit well. The dataset has seven MSE tr Fig J Training MSE for Abalone by SSF Figure 6 shows training MSE (mean squared error) of Abalone dataset by SSF. We an see training error dereased ISBN: ISSN: (Print); ISSN: (Online) IMECS 8

6 Proeedings of the International MultiConferene of Engineers and Computer Sientists 8 Vol I IMECS 8, Marh -6, 8, Hong Kong monotonially. MSE at J =7 was.69, while minimum MSEs of best MoLR (C=) and best MoLR (C=) were.69 and.8 respetively. This indiates MoLR (C=) ould fit better than MoLR (C=), and a mixture of two or three linear regressions ould fit better than a single MLP with SSF. Table III ompares values and goodness of fit ESS/TSS for different regression models and different initialization methods. We an see the init methods of MoLR made little differene. Moreover, MoLR (C=) had smaller than MoLR (C=), and MoLR had smaller than MLP with SSF. MoLR (C=) showed the highest goodness of fit, higher than very powerful single nonlinear regression model MLP with SSF. TABLE III FOR ABALONE DATASET Model & Method ESS/TSS MoLR (C=) with HS Init 6.79 MoLR (C=) with R Init 6.78 MoLR (C=) with HS Init.798 MoLR (C=) with R Init.796 MLP (J =7) with SSF 6.6 Figures 7 and 8 show histograms of obtained by MoLR using HS init, while Figs. 9 and show obtained by MoLR using R init. These figures show that the best solution of C= was better than that of C=. Moreover, for the best model (C=), HS init found the best solution 7 times out of, and R init found it times out of. Again this means that for the best model (C=) we an easily find the best solution with any initialization... Histram of with Abalone_HS_C Fig. 7. histogram for Fig. 8. histogram for Abalone Data with HS Init and Abalone Data with HS Init and Histram of with Abalone_R_C 7 6 Histram of with Abalone_HS_C Histram of with Abalone_R_C regressions (MoLR). Our experiments used two kinds of artifiial data and one real dataset. Experiments using artifiial data showed soft MoLR suessfully disovered the original lines from data ontaining Gaussian or t-distribution noise. Experiments using real dataset showed soft MoLR had smaller and higher goodness of fit than single MLP. This shows the potential of MoLR. Moreover, almost all our experiments showed the best solution an be found with more than % using any of two initializations. This may suggest soft MoLR may have rather weak dependene on initialization. In the future we plan to investigate more to verify the plausibility of this tendeny and further extend MoLR to nonlinearity. REFERENCES [] A. P. Dempster, N. M. Laird, and D. B. Rubin, Maximum-likelihood from inomplete data via the EM algorithm, J. Royal Statist. So. Ser. B, vol. 9, pp. 8, 977. [] NCSS, Regression lustering, NCSS Statistial Software Doumentation, Teh. Rep. Chapter 9, pp. 7,. [] G. Qian and Y. Wu, Estimation and seletion in regression lustering, European Journal of Pure and Applied Mathematis, vol., no., pp. 66,. [] H. Späth, Algorithm 8: A fast algorithm for lusterwise linear regression, Computing, vol. 9, pp. 7 8, 98. [] G. J. MLahlan and D. Peel, Finite mixture models. John Wiley & Sons,. [6] F. Leish, FlexMix: A general framework for finite mixture models and latent lass regression in R, Journal of Statistial Software, vol., no. 8, pp. 8,. [7] M. Huang, L. Runze, and W. Shaoli, Nonparametri mixture of regression models, Journal of the Amerian Assoiation, vol. 8, no., pp. 99 9,. [8] C. M. Bishop, Pattern reognition and mahine learning. Springer, 6. [9] N. Ueda and R. Nakano, Deterministi annealing EM algorithm, Neural Networks, vol., no., pp. 7 8, 998. [] N. Ueda, R. Nakano, Z. Ghahramani, and G. E. Hinton, SMEM algorithm for mixture models, Neural Comput., vol., no. 9, pp. 9 8,. [] Y. Linde, A. Buzo, and R. M. Gray, An algorithm for vetor quantizer design, IEEE Trans. on Communiations, vol. 8, no., pp. 8 9, 98. [] G. Shwarz, Estimating the dimension of a model, Annals of Statistis, vol. 6, pp. 6 6, 978. [] S. Satoh and R. Nakano, How new information riteria WAIC and W worked for MLP model seletion, in Pro. of 6th International Conf. on Pattern Reognition Appliations and Methods (ICPRAM), 7, pp.. [], Fast and stable learning utilizing singular regions of multilayer pereptron, Neural Proessing Letters, vol. 8, no., pp. 99,. [], Multilayer pereptron learning utilizing singular regions and searh pruning, in Pro. Int. Conf. on Mahine Learning and Data Analysis,, pp Fig. 9. histogram for Fig.. histogram for Abalone Data with R Init and Abalone Data with R Init and V. CONCLUSION This paper examined how initial values of the EM algorithm influene the performane of soft mixture of linear ISBN: ISSN: (Print); ISSN: (Online) IMECS 8