Learning Robust Locality Preserving Projection via p-order Minimization

Transcription

1 Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence Learning Robust Locality Preserving Projection via -Order Minimization Hua Wang, Feiing Nie, Heng Huang Deartment of Electrical Engineering and Comuter Science Colorado School of Mines, Golden, Colorado 84, USA Deartment of Comuter Science and Engineering University of Texas at Arlington, Arlington, Texas 769, USA Abstract Locality reserving rojection (LPP) is an effective dimensionality reduction method based on manifold learning, which is defined over the grah weighted squared l -norm distances in the rojected subsace. Since squared l -norm distance is rone to outliers, it is desirable to develo a robust LPP method. In this aer, motivated by existing studies that imrove the robustness of statistical learning models via l -norm or not-squared l -norm formulations, we roose a robust LPP (rlpp) formulation to minimize the -th order of the l -norm distances, which can better tolerate large outlying data samles because it suress the introduced biased more than the l -norm or not squared l -norm minimizations. However, solving the formulated objective is very challenging because it not only non-smooth but also non-convex. As an imortant theoretical contribution of this work, we systematically derive an efficient iterative algorithm to solve the general -th order l -norm minimization roblem, which, to the best of our knowledge, is solved for the first time in literature. Extensive emirical evaluations on the roosed rlpp method have been erformed, in which our new method outerforms the related state-of-the-art methods in a variety of exerimental settings and demonstrate its effectiveness in seeking better subsaces on both noiseless and noisy data. Introduction Dimensionality reduction (DR) algorithms seek to reresent the inut data in their lower-dimensional intrinsic subsace/sub-manifold, in which irrelevant features are runed and inherent data structures are more lucid. In the early ages, DR algorithms assume that the inut data objects are homogeneous but not relational, and thereby are devised to deal with a set of attributes in the format of fixed length vectors. For examle, Princial Comonent Analysis (PCA) (Jolliffe 5) attemts to maximize the covariance among data oints, while Linear Discriminant Analysis (LDA) (Fukunaga 99) aims at maximizing the class searability. In recent years, manifold learning has motivated many DR algorithms using airwise similarities between To whom all corresondence should be addressed. This work was artially suorted by NSF-IIS 7965, NSF-IIS 3675, NSF-IIS 3445, NSF-DBI 35668, NSF-IIS Coyright c 5, Association for the Advancement of Artificial Intelligence ( All rights reserved. data objects, either comuted from data attributes or directly obtained from exerimental observations. These algorithms, such as ISOMAP (Tenenbaum, De Silva, and Langford ), Locally Linear Embedding (LLE) (Roweis and Saul ), Lalacian Eigenma (Belkin and Niyogi ), Locality Preserving Projection (LPP) (Niyogi 4), and discriminant Lalacian embedding (Wang, Huang, and Ding b), etc., generally assume that the observed data are samled from an underlying sub-manifold which is embedded in a high-dimensional observation sace. Due to this reasonable assumtion, manifold based learning methods have achieved great success to solve roblems in a large number of real-world alications. Among these manifold learning based rojection methods, LPP has been widely acceted because it is able to learn a linear rojection from the original data, such that it can be easily alied to not only training data but also out-of-samle data oints. In this aer, we address the issue of robustness of LPP in the resence of outlier samles, which is defined as the data oints that deviates significantly from the rest majority of the data oints. In classical LPP aroach, the learning objective is formulated using the squared distance in the rojected subsace, which, same as other least square based learning objectives in statistical learning, could be significantly influenced by outlying observations. That is, the traditional LPP formulation becomes inaroriate at contaminated data sets, because large errors squared dominate the sum. Many revious works have been done to imrove the robustness of the linear dimensionality reduction methods via using the l -norm minimizations or not-squared l -norm minimizations (Baccini, Besse, and Falguerolles 996; Ding et al. 6; Gao 8; Ke and Kanade 5; Kwak 8; Wright et al. 9; Nie et al. ; ; Wang, Nie, and Huang 4). The key motivation of these rior works lies in that the not-squared l -norm distance or error functions, say x i x j for the two vectors x i and x j, can generally better tolerate the biases caused by the outlying data samles, esecially when the outlier data samles are far away from the normal data distributions. Following the same intuition, we recognize that the distance with lower orders, e.g., x i x j where < <, can achieve the same goal for robustness with otentially better results (Wang et al. 3), because the bias caused by outlying data samles can be further suressed when (< ) is selected 359

2 as a very small number. Based on this recognition, we roose a new LPP objective using the -th order distance, and call the resulted objective as the -order minimization roblem. Because the learned rojection by our new method is robust against outlier data samles, we call it robust locality reserving rojection (rlpp) method, which is interesting from a number of ersectives as following. Our new objective is defined over the -th ( < ) order of the l -norm distance, which is more general and makes the traditional LPP a secial case of our new method when =. Same as other l -norm or not-squared l -norm based learning objectives, our new method is robust against outlier data samles, which is articularly true when <. The smaller the value of is, the better robustness our new method can achieve. Desite its clear intuitions and nice theoretical roerties, the resulted objective of the roosed rlpp method is difficult to solve, because it is not only non-smooth but also non-convex. To solve the roblem, we roose an efficient iterative solution algorithm, whose convergence is rigourously roved. Extensive emirical studies have been erformed to evaluate a variety asects of the roosed rlpp method, which clearly demonstrate the effectiveness of the roosed method on not only noiseless data sets but also noisy data sets with outlier samles, esecially for the latter case. Motivation of the Proosed Problem Suose we have n data oints {x,,x n R d },we construct a grah using the data with the similarity matrix S R n n. The Lalacian matrix L is defined as L = D S, where D is a diagonal matrix with the i-th diagonal element as j S ij. Recently, Locality Preserving Projection (LPP) and its variants have been successfully alied for dimensionality reduction. The rojection matrix W R d m (m<d) in LPP is obtained by solving the following roblem : min S ij W T x i W T x j. () The objective in Eq.() can be written as tr(w T XLX T W ). Thus the otimal solution W to roblem () is the m eigenvectors of (W T XDX T W ) W T XLX T W corresonding to the m smallest eigenvalues. The basic idea of LPP is to reserve the neighborhood relationshi between data oints. Secifically, as shown in Eq.(), LPP tries to find a embedded subsace such that the distances of data airs which are neighbors in the original sace are minimized. In ractice, to ensure the learned rojection W is shiftinvariant, D in Eq.() should be L d = D D T D, or the training data X must be centered (Nie et al. 9; Nie, Cai, and Huang 4) The squared distances used in Eq.() do not tolerate large value of distance, thus makes the distances in the embedded subsace tend to be even, i.e., not too large but also not too small. Therefore, the squared distances used in LPP would makes the method can not find the otimal subsace such that most of the distances of local data airs are minimized but a few of them are large. In this aer, we roose to solve the following roblem to find the otimal subsace: min S ij W T x i W T x j. () where <. Obviously, LPP is a secial case of the roosed new method when =. More imortantly, by setting, the method will focus on minimizing most of the distances of local data airs. Although the motivation of Eq. () is clear, it is a nonsmooth objective and difficult to be solved efficiently. Thus, in the next section, we will introduce an iterative algorithm to solve the roblem (). We will show that the original weight matrix W would be adatively re-weighted to cature clearer cluster structures after each iteration. Otimization Algorithm to the Proosed Method The Lagrangian function of the roblem () is n L(W )= S ij W T x i W T x j Tr(Λ(W T XDX T W I)). Denote a Lalacian matrix L = D S, where S is a reweighted weight matrix defined by S ij = S ij W T x i W T x j, (4) D is a diagonal matrix with the i-th diagonal element as j S ij. Taking the derivative of L(W ) w.r.t W, and setting the derivative to zero, we have: (3) L(W ) W = X LX T W XDX T W Λ=, (5) which indicates that the solution W is the eigenvectors of ( XDX T ) X LX T. Note that ( XDX T ) X LX T is deendent on W, we roose an iterative algorithm to obtain the solution W such that Eq. (5) is satisfied. The algorithm is guaranteed to converge to a local otimum, which will be roved in the next subsection. The algorithm is described in Algorithm. In each iteration, L is calculated with the current solution W, then the solution W is udated according to the current calculated L. The iteration rocedure is reeated until converges. From the algorithm we can see, the original weight matrix S is adatively re-weighted to minimize the objective in Eq. () during the iteration. 36

3 Inut: Training data X R d n. The original weight matrix S R n n. D is a diagonal matrix with the i-th diagonal element as j S ij. Initialize W R d m such that W T XDX T W = I ; while not converge do. Calculate L t = D t S t, where S ij = S ij W T x i W T x j, D is a diagonal matrix with the i-th diagonal element as j ( S) ij ;. Udate Q. The columns of the udated Q are the first m eigenvectors of ( XDX ) T X T LX corresonding to the first m smallest eigenvalues ; end Outut: W R d m. Algorithm : The algorithm to solve the roblem (). Convergence Analysis To rove the convergence of the Algorithm, we need the following lemmas: Lemma For any scalar x, when <, we have x x +. Proof: Denote f(x) =x x +, then we have f (x) =(x ), (6) and f ( ) (x) = x 4. (7) Obviously, when x>and <, then f (x) and x =is the only oint that f (x) =. Note that f() =, thus when x>and <, then f(x). Thus f(x ), which indicates x x +. Lemma For any nonzero vectors v, v, when <, the following inequality holds: v v v v v v. (8) Proof: ) ( ) v v ( v v + v v v ( ) v v v v v v v, where the first inequality is true according to Lemma. Now we have the following theorem: Theorem The Algorithm will monotonically decrease the objective of the roblem () in each iteration, and converge to a local otimum of the roblem. Proof: Suose the udated W is W. According to the ste in the Algorithm, we know that W =arg min Tr(W T X LX T W ) =arg min S ij W T x i W T x j. (9) Note that S ij = S ij W T x i W T x j,sowehave S ij W T x i W T x j W T x i W x j n S ij W T x i W T x j W T x i W T x j. () According to Lemma, we have S ij W T x i W x j S ij W T x i W T x j W T x i W x j n S ij W T x i W T x j S ij W T x i W T x j W T x i W x j. () Summing Eq. () and Eq. () in the two sides, we arrive at S ij W T x i W x j n S ij W T x i W T x j. () Thus the Algorithm will monotonically decrease the objective of the roblem () in each iteration t until the algorithm converges. In the convergence, the equality in Eq. () holds, thus W and L will satisfy Eq. (5), the KKT condition of roblem (). Therefore, the Algorithm will converge to a local otimum of the roblem (). Exerimental Results In this section, we emirically study the roosed robust locality reserving rojection (rlpp) method, where our goal is to examine its robustness under the conditions when data outliers are resent. Data Descritions We evaluate the roosedmethods on five widely used benchmark data sets in machine learning studies. The data descritions are summarized in Table. The first two data sets are obtained from the UCI machine learning data reository. For the CMU PIE (Face Pose, Illumination, and Exression) face data set, all the face images are resized to 3 3 following standard comuter vision exerimental conventions to reduce the misalignment effects. For the two document data sets, following revious studies, for Reuters578 data set, we remove the keywords aearing less than 5 times and end u with 599 features; for TDT corus data set, we remove the keywords aearing less than times, and end u with 357 features, resectively. Study of the Parameter of the Proosed Method The roosed method has only one arameter, i.e., in Eq. (), which controls to which extent we suress the bias introduced by the outlier data samles. Thus, we first evaluate its imacts to the learned rojections. 36

4 Table : Data sets used in our exeriments. Data set Number Dimension Classes Coil 44 4 Vehicle PIE face Reuters TDT corus Figure : Clustering accuracy in the learned rojected saces by the roosed method vs. on the Coil data set. Exerimental setus. We exeriment with the Coil data set. An imortant roerty of the Coil data set is that the ictures in the data set were taken reeatedly for one same object from different viewing angles. As a result, the images for the same object indeed reside on an intrinsical manifold. Our goal is to cluster the object images in the rojected sace learned by the roosed method, by which we examine whether the manifold structures can be discovered such that the clustering erformance can be imroved. We vary of the roosed objective in the range of. to to study its imacts to the clustering erformance. We erform the clustering using the K-means clustering method in the rojected subsaces, where K is set to the true cluster numbers. The clustering erformances with different arameter settings measured by the clustering accuracies are reorted in Figure. To alleviate the randomness due to the initializations for the K-means clustering method, we reeat the exeriment at each arameter setting for 5 times and reort the average clustering accuracy in Figure. Exerimental results. From Figure we can see that, smaller leads to better clustering accuracy, i.e., the rojected subsace learned by our new method with smaller can better find the intrinsic data structures, which clearly confirms the correctness to use -order minimization to learn the locality reserved rojections. We also notice that when is very small, e.g., when =.and =., the clustering erformances is not as good as that when =.3. This can be attributed that, when is too small, the distance measurement will be comromised, i.e., the relative difference between different distances become smaller. In the extreme case, when, the distance between any data airs will turn to be the same. Theoretically, should take a small value to imrove the robustness of the roosed objective against outlier data samles; meanwhile should not be too small to invalidate the distance measurement in the Euclidean sace. Uon the results in Figure, emirically, we set =.3 in all our subsequent exeriments, unless otherwise stated. Convergence Study of the Solution Algorithm Because the roosed rlpp method emloys an iterative solution algorithm, an imortant issue is its convergence roerty. We have theoretically roved the convergence of the algorithm, and now we emirically study the convergence roerty of the roosed iterative algorithm. The objective values of our algorithm on the five data sets in each iteration are lotted in the sub-figures of Figure, which show that the objective values of our algorithm kee to decrease along with iterative rocesses, which is erfectly in accordance with our earlier theoretical analysis. Moreover, the algorithm tyically converges to the asymtotic limit within 7 iterations, which demonstrates that our solution algorithm is very comutationally efficient. As a result, our new algorithm scales well to large-scale data sets and adds its value for ractical use. Uon these exerimental results, emirically, we select a stoing threshold of 5 in all our following exeriments, which is sufficient to achieve satisfactory results in terms of convergence. Exerimental Results on Benchmark Noiseless Data Exerimental setus. When we construct the data grah only using data similarity, the roosed rlpp method is an unsuervised method. Thus we comare it against the following unsuervised dimensionality reduction methods: () rincial comonent analysis (PCA) (Jolliffe 5), () robust rincial comonent analysis (rpca) (Wright et al. 9) (this method is the most recently ublished robust PCA method with better erformance than others (Gao 8; Ke and Kanade 5; Ding et al. 6; Kwak 8)), (3) locality reserving rojections (LPP) (Niyogi 4) which is the non-robust counterart of the roosed method, and (4) the Kernel Lalacian Embedding (KLE) method (Wang, Huang, and Ding a). In addition, as a baseline, we also reort the clustering results by (5) the K- means clustering method in the original feature sace. For PCA, we reduce the dimensionality of the inut data such that 9% of data variance is reserved. For rpca, following (Wright et al. 9), we set λ = d /. We emirically select the reduced dimensionality of LPP method to be c where c is the number of clusters of a data set, and use the codes ublished by the authors (Niyogi 4). The KLE method is one of the most recent unsuervised learning method by integrating attribute data and airwise similarity data, and has demonstrated state-of-the-art dimensionality reduction erformance. We imlement the KLE method following its original work (Wang, Huang, and Ding a). For the roosed method, as well as the KLE method and the LPP method, we construct the nearest-neighbor grah for each data set and set the neighborhood size for grah construction as following (Niyogi 4). Excet for rpca, 36

5 3 4 (a) Yale face. 3 4 (b) Vehicle. 3 4 (c) PIE face. 3 4 (d) Reuters (e) TDT. Figure : vs. the objective value of the roosed rlpp method. once the rojection matrix is obtained by a dimensionality reduction method, K-means clustering method is used to cluster the data oints in the rojected subsace, where we set K as the true class numbers. Because rpca method does not roduce the rojection matrix but the rojected data, we use its immediate outut for clustering. For our method, we exeriment with two different setting, where we set the arameter to be.3 and. Note that, when =, the roosed method is exactly the traditional LPP method roosed in (Niyogi 4); when =or =.3, the roosed method is exected to suress the imacts of the outlier data samles. Because the results of the K-means clustering algorithm deend on the initialization, to reduce the statistical variety, we indeendently reeat the clustering rocedures in the rojected subsaces learned by all comared methods for 5 times with random initializations, and then we reort the results corresonding to the best objective values. The clustering erformance measured by clustering accuracy are reorted in Table. Exerimental results. From the exerimental results in Table we can see that our method consistently outerforms all other comared methods, which demonstrate the effectiveness of our methods in discovering the inherent manifold structures of the inut data and thereby imroving the clustering erformance. Most imortantly, as exected, when is smaller, the clustering erformance of the roosed method is better, which again demonstrate the effectiveness of using -order minimization in learning LPP subsace. Robustness Against Outlier Samles Exerimental setus. Because the main advantage of the roosed rlpp method lies in its robustness against outlier data samles, we further evaluate the it on noisy data with outlier samles. To emulate the outlier samles, given the inut data matrix X, we corrut it by a noise matrix M whose element are i.i.d. standard Gaussian variables. Then we carry out the same rocedures as those in the revious subsection for rojection learning on X + δm, where δ = nf X F M F and nf is a given noise factor. We set nf =.in all our studies. We comare our new rlpp method against the same unsuervised dimensionality reduction methods as before and reort the clustering results in Table 3. Exerimental results. First, the roosed rlpp method is consistently better than all other comared methods on all five exerimental data sets, which demonstrate that our new method is able to effectively learn a subsace to imrove the clustering erformance on noisy data with outlier data samles. Second, although the imrovements by our method over the cometing methods on the original data without noise are mediocre as shown in Table in the last subsection, the imrovements by our new method on the contaminated data with outlier data samles in this subsection are considerably large. For examle, on the Coil data set with outlier samles, our new rlpp method imroves the clustering accuracy over the baseline PCA method by 4.49% = (.7.56)/.56. In contrast, the imrovement of clustering accuracy on the same data set by our method over the PCA method under the noiseless condition is only 3.44% = (.75.66)/.66. The same observations can be seen on all the other exerimental data sets, which show that the roosed method has better caability to learn a more effective subsace for clustering on contaminated data, and confirms its robustness against outlier data 363

6 Table : Clustering accuracy of the comared methods on the four benchmark data sets without noise. Data PCA rpca LPP KLE rlpp ( = ) rlpp ( =.3) Coil Vehicle PIE face Reuters TDT corus Table 3: Clustering accuracy of the comared methods on the four benchmark data sets with outlier data samles. Data PCA rpca LPP KLE rlpp ( = ) rlpp ( =.3) Coil Vehicle PIE face Reuters TDT corus samles. Finally, we also notice that when takes smaller value, the roosed method can achieve better clustering results, which rovide one more concrete evidence to suort the usefulness of the -order minimization when learning a LPP subsaces. In summary, the roosed method is able to achieve better clustering erformance on not only noiseless data but also noisy data with outlier samles, which is articularly true for the latter case. These observations are consistent with our motivations to theoretically formulate our new objective in that the -order minimization can alleviate the negative imacts of outliers data samles. Conclusions We roosed a robust LPP method based on the -th order of l -norm distance, which formulated a non-smooth nonconvex minimization roblem. The new objective imoses the -th order l -norm distance when comuting the airwise data affinities, which makes the resulted objective very robust against outlier data samles. However, the new objective brings the much more challenging otimization roblem than that in traditional LPP. To solve the roblem, we introduced an efficient iterative algorithm and rovided the rigorous theoretical analysis on the convergence of our algorithm. The new algorithm is easy to be imlemented and fast to converge in ractice, because we have closed form solution in each iteration. We erformed extensive exeriments on both noiseless and noisy data, and all results have clearly shown that the roosed method is more effective and robust to outlier samles than traditional methods. References Baccini, A.; Besse, P.; and Falguerolles, A Al -norm ca and a heuristic aroach. In Ordinal and symbolic data analysis. Sringer Belkin, M., and Niyogi, P.. Lalacian eigenmas and sectral techniques for embedding and clustering. In NIPS, volume 4, Ding, C.; Zhou, D.; He, X.; and Zha, H. 6. R -ca: rotational invariant l -norm rincial comonent analysis for robust subsace factorization. In Proceedings of the 3rd international conference on Machine learning, ACM. Fukunaga, K. 99. Introduction to statistical attern recognition. Access Online via Elsevier. Gao, J. 8. Robust l rincial comonent analysis and its bayesian variational inference. Neural comutation (): Jolliffe, I. 5. Princial comonent analysis. Wiley Online Library. Ke, Q., and Kanade, T. 5. Robust l norm factorization in the resence of outliers and missing data by alternative convex rogramming. In Comuter Vision and Pattern Recognition, 5. CVPR 5. IEEE Comuter Society Conference on, volume, IEEE. Kwak, N. 8. Princial comonent analysis based on l-norm maximization. Pattern Analysis and Machine Intelligence, IEEE Transactions on 3(9): Nie, F.; Xiang, S.; Song, Y.; and Zhang, C. 9. Orthogonal locality minimizing globality maximizing rojections for feature extraction. Otical Engineering 48():7 7. Nie, F.; Huang, H.; Cai, X.; and Ding, C. H.. Efficient and robust feature selection via joint l, -norms minimization. In Advances in Neural Information Processing Systems, Nie, F.; Wang, H.; Huang, H.; and Ding, C.. Unsuervised and semi-suervised learning via l -norm grah. In IEEE International Conference on Comuter Vision (ICCV ), IEEE. Nie, F.; Cai, X.; and Huang, H. 4. Flexible shift-invariant locality and globality reserving rojections. In Machine 364

7 Learning and Knowledge Discovery in Databases. Sringer Niyogi, X. 4. Locality reserving rojections. In Neural information rocessing systems, volume 6, 53. Roweis, S. T., and Saul, L. K.. Nonlinear dimensionality reduction by locally linear embedding. Science 9(55): Tenenbaum, J. B.; De Silva, V.; and Langford, J. C.. A global geometric framework for nonlinear dimensionality reduction. Science 9(55): Wang, H.; Nie, F.; Cai, W.; and Huang, H. 3. Semisuervised robust dictionary learning via efficient l, +- norms minimization. In 3 IEEE International Conference on Comuter Vision (ICCV 3), Wang, H.; Huang, H.; and Ding, C. a. Multi-label feature transform for image classifications. In ECCV. Sringer Wang, H.; Huang, H.; and Ding, C. H. b. Discriminant lalacian embedding. In AAAI. Wang, H.; Nie, F.; and Huang, H. 4. Robust distance metric learning via simultaneous l-norm minimization and maximization. In Proceedings of the 3st International Conference on Machine Learning (ICML 4), Wright, J.; Ganesh, A.; Rao, S.; Peng, Y.; and Ma, Y. 9. Robust rincial comonent analysis: Exact recovery of corruted low-rank matrices via convex otimization. In Advances in neural information rocessing systems,