Manifold Regularized Transfer Distance Metric Learning

Transcription

1 HAIBO SHI, YONG LUO, CHAO XU, YONGGANG WEN: MTDML 1 Manifold Regularized Transfer Distance Metric Learning Haibo Shi 1 shh@pku.edu.cn Yong Luo 2 yluo180@gail.co Chao Xu 1 xuchao@cis.pku.edu.cn Yonggang Wen 2 ygwen@ntu.edu.sg 1 Key Laboratory of Machine Perception (MOE), Cooperative Medianet Innovation Center, School of EECS, Peking University, Beijing, China 2 School of Coputer Engineering, Nanyang Technological University, Singapore Abstract The perforance of any coputer vision and achine learning algoriths are heavily depend on the distance etric between saples. It is necessary to e xploit abundant of side inforation like pairwise constraints to learn a robust and reliable distance etric. While in real world application, large quantities of labeled data is unavailable due to the high labeling cost. Transfer distance etric learning (TDML) can be utilized to tackle this proble by leveraging different but certain related source tasks to learn a target etric. The recently proposed decoposition based TDML (DTDML) is superior to other TDML ethods in that uch fewer variables need to be learned. In spite of this success, the learning of the cobination coefficients in DTDML still relies on the liited labeled data in the target task, and the large aounts of unlabeled data that are typically available are discarded. To utilize both the inforation contained in the source tasks, as well as the unlabeled data in the target task, we introduce anifold regularization in DTDML and develop the anifold regularized transfer distance etric learning (MTDML). In particular, the target etric in MTDML is learned to be close to an integration of the source etrics under the anifold regularization thee. That is, the target etric is soothed along each data anifold that is approxiated by all the labeled and unlabeled data in the target task and each source etric. In this way, ore reliable target etric could be obtained given the liited labeled data in the target task. Extensive experients on the NUS-WIDE and USPS dataset deonstrate the effectiveness of the proposed ethod. 1 Introduction Distance etric is critical in a nuber of achine learning and pattern recognition algoriths [4, 4, 11, 12, 17]. For instance, the siple k-nearest neighbor (knn) classifier is soeties superior to other well designed classifier with a suitable distance etric [4, 14]. Also in unsupervised settings such as clustering, a proper distance etric will help find a plausible patterns that eaningful to users. c The copyright of this docuent resides with its authors. It ay be distributed unchanged freely in print or electronic fors. Pages DOI:

2 2 HAIBO SHI, YONG LUO, CHAO XU, YONGGANG WEN: MTDML In order to learn a robust distance etric to reveal data siilarity, we need a large aount of side inforation [15, 16, 17]. These side inforation is usually cast in a for of pairwise constraint that indicates weather saples are siilar or not. However, abundant labeled saples is unavailable in practice due to the high labeling cost. A natural solution to tackle this proble is to utilize the large aount of unlabeled data [5, 8, 9]. For exaple, Hoi et al. [5] proposed a Laplacian regularized etric learning algorith for iage retrieval. Recently, any works focus on transfer distance etric learning (TDML), which ais to derive a reliable target etric by leveraging auxiliary knowledge fro soe different but related tasks [10, 18, 19]. For exaple, the target etric is learned in [18] by iniizing the log-deterinant divergence between the source etrics and target etric. Zhang and Yeung [19] proposed to learn the task relationships in transfer etric learning, and therefore, allow odeling of negative and zero transfer. DTDML [10] is superior to both of these two ethods in that the nuber of variables to be learned is reduced significantly, by representing the target etric as a cobination of soe "base etrics". However, in the learning of the cobination coefficients, only the source etrics pre-trained on the labeled data in the source tasks and liited labeled data in the target task are utilized. But the large aount of unlabeled data in the target task are discarded. Therefore, in this paper, we propose anifold regularized transfer distance etric learning (MTDML), which could ake full use of both the label inforation in the source tasks and all the available (labeled and unlabeled) data in the target task. In particular, a source etric is learned by utilizing the abundant labeled data in a certain source task. Then a data adjacency graph is constructed by applying a learned source etric on all the data in the target task. This leads to ultiple graphs, and we integrate the to reveal the geoetric structure of the target data distribution, which is assued to be supported on a low-diensional anifold. Finally, the Laplacian of the integrated graph is forulated as a regularization ter to sooth the target etric. In this way, both the large aount of label inforation and the abundant unlabeled data in the target task are utilized in the learning of the target etric. The learned target etric not only is close to an integration of the source etrics, but also respects the data distribution of the target task. We therefore obtain ore reliable solutions given the liited side inforation. In the optiization, the "base etric" cobination coefficients and the source graph Laplacian integration weights are learned alternatively until converge. We conduct experients on NUS-WIDE, which is a challenge web iage annotation dataset and USPS, a handwritten digit classification dataset. The results confir the effectiveness of the proposed MTDML. 2 Manifold Regularized Transfer Distance Metric Learning In this section, we propose our anifold regularized transfer distance etric learning (MT- DML) odel. The following are soe notations that used throughout this paper: Let D = {(xi l,xl j,y i j)} nl i, j=1 denotes the labeled training set for the target task, wherein x i, x j R d and y i j = ±1 indicates xi l and xi l are siilar/dissiilar to each other. The nuber of labeled saples n l is very sall, and thus we assue there are large aount of n u unlabeled data {xi u,xu j }, as well as different but related source tasks with abundant labeled training data D p = {(x pi,x p j,y pi j )} n p i, j=1, p = 1,...,.

3 HAIBO SHI, YONG LUO, CHAO XU, YONGGANG WEN: MTDML Background and Overview of MTDML In distance etric learning (DML), a etric is usually learned to iniize the distance between the data fro the sae class and axiize their distance otherwise. This leads to the following loss function for learning the etric A: Φ(A) = x i x j 2 A µ y i j =1 = tr(s A) µtr(d A) y i j = 1 x i x j 2 A where d A (x i,x j ) = x i x j A = (x i x j ) T A(x i x j ) is the distance between two data points x i and x j, tr( ) is the trace of atrix, µ is a positive trade-off paraeter. Here, S and D are given by, S = D = (x i,x j ) S (x i,x j ) D (x i x j )(x i x j ) T, (1) (x i x j )(x i x j ) T (2) The loss function (1) above is widely used in [5, 13, 18]. A regularization ter Ω(A) = A 2 F can be added in (1) to control the odel coplexity. However, when the nuber of labeled data n l is sall, especially less than the nuber of variables to be estiated in etric atrix A, such a siple regularization is often insufficient to control the odel coplexity. Transfer distance etric learning (TDML) ethods [10, 18, 19] tackle this proble by leveraging auxiliary knowledge fro soe different but related source tasks. The recently proposed decoposition based TDML (DTDML) [10] algorith is superior to the previous TMDL approaches in that uch fewer variables are needed to be learned. Given the source tasks, we learn corresponding etrics A p R d d, p = 1,..., independently. Considering that any etric A can be decoposed as A = Udiag(θ)U T = d i=1 θ iu i u T i, DTDML proposed to learn a cobination of soe base etrics to approxiate the optial target etric. The base etrics can be derived fro the source etrics or soe randoly generated base vectors. For exaple, we can apply singular value decoposition (SVD) to A p and obtain a set of source eigenvectors U p = [u p1...u pd ]. Then the base etric is calculated as B p j = u p j u T p j and this leads to d base etrics for source tasks. Based on this idea, the forulation of DTDML is given by arg in β,θ Φ(β,θ) + γ A 2 A A S 2 F + γ B 2 β γ C 2 θ 2 1 (3) where Φ( ) is soe pre-defined convex loss, A = d r=1 θ ru r u T r, and A S = p=1 β pa p is an integration of the source etrics. Although the liited labeled saples in the target task and the auxiliary source etrics are effectively utilized in proble (3) by siultaneously iniizing the losses Φ(β,θ) and the divergence between A S and A, the large aount of unlabeled data in the target task are discarded. Therefore, we propose to utilize anifold regularization [2] to take advantage of all the given labeled and unlabeled inforation in a unified etric learning fraework. The diagra of the proposed MTDML is shown in Figure. 1. Each source task learns a corresponding source etric independently. These etrics are used to construct several graphs by using the labeled and unlabeled data in target task. The different graph Laplacians are

4 HAIBO SHI, YONG LUO, CHAO XU, YONGGANG WEN: MTDML 4 Source etrics Source doains cat dog d og A1 A Manifold M aniifolld IIntegration ntegratiion zebra L1 horse e L Target doain L A* Preserve Locality Label Inforation Result etric R Unlabeled data U Labeled d data diag g(q ) Source eigenvectors or rando bases A Figure 1: Diagra of the proposed MTDML algorith. weightedly cobined as an integrated Laplacian atrix L, which is used to preserve locality in the feature space and approxiate the data distribution. Meanwhile, the target etric A = Udiag(θ )U T is represented by a group of base etrics that generated fro decoposition of source etrics. By incorporating the liited label inforation, and soothing the target etric A along the data anifold that is approxiated with the integrated L, we learn the base etric cobination coefficients and the graph Laplacian integration weights siultaneously, and finally, obtain the result etric. 2.2 Proble Foralization Manifold regularization fraework [2] iplies the geoetry of the intrinsic data probability distribution is supported on the low-diensional anifold. The Laplacian of the adjacency graph coputed in an unsupervised anner using Laplacian Eigenap[1] with both labeled and unlabeled saples. The data anifold can be approxiated with the graph Laplacian. Moreover, the distance easure is a key point for graph Laplacian construction. Since both the integrated source etric AS and target etric A are derived fro the sae feature space and related tasks, these two etrics should be siilar. Rather than explaining this siilarity by siply iniizing the least squares difference in DTDML, we forulate it as a soothing penalty ter. Based on the obtained source etric A p, we construct an adjacency graph Wp by using all the labeled and unlabeled data in the target task. This leads to ultiple graphs Wp, p = 1,...,. Considering between two saples the target etric A, distance T can be further written as, da (xi, x j ) = (xi x j ) A(xi x j ) = (xi x j )T PPT (xi x j ) with

5 HAIBO SHI, YONG LUO, CHAO XU, YONGGANG WEN: MTDML 5 P R d d. As a consequence, it is equivalent to learn the target etric A and the linear apping P. Following the anifold regularization principle, we can sooth P along the data anifold [2, 18], which is approxiated by the Laplacian of the graph W p. By suing over all the different graphs {W p } p=1, we obtain the following regularizer for the apping P as well as the etric A, i.e., Ω(A) = 1 2 = p=1 p=1 β p ( Px i Px j 2 W p (i, j)) i, j β p tr(xl p X T PP T ) = tr(xlx T A) (4) where L = p=1 β pl p, is the integrated graph Laplacian, and each L p = D p W p. Here, D p is a diagonal atrix with the entity D pii = nl +n u j=1 W pi j. In this way, target etric A is not only close to an integration of the source etrics, and also sooth along the data anifold. This leads to lower odel coplexity copared with DTDML, and thus better generalization ability for etric learning. By introducing the regularizer (4) in (1), and adopting the decoposition based etric learning strategy in [10], we obtain the following optiization proble for our MTDML: arg in β,θ tr(s A) µtr(d A) + γ Atr(XLX T A) + γ B 2 β γ C s.t. i=1 2 θ 2 2 β i = 1,β i 0,i = 1,..., (5) where γ A,γ B,γ C are positive trade-off paraeters. By the use of the learned θ, we can easily construct A = d r=1 θ r u r u T r as optial distance etric for next step classification. 2.3 Optiization The solution of proble (5) can be obtained by alternately solving two sub-proble, which correspond to the iniization w.r.t β = [β 1,...,β ] T and θ = [θ 1,...,θ d ] T respectively until convergence. For fixed β, the optiization proble with respect to θ is forulated as: arg in θ n i=1 θ i tr((s µd + γ A XLX T ) (u i u T i )) + γ C 2 θ 2 2 (6) Equation above can be further written in a copact for: θ = argin θ T h + γ C θ 2 θ 2 2 (7) where h = [h 1,...,h n ] with each h i = tr((s µd + γ A XLX T )(u i u T i )). Since (6) is a convex proble and has close for solution, we can get θ efficiently.

6 6 HAIBO SHI, YONG LUO, CHAO XU, YONGGANG WEN: MTDML With fixed θ, the optiization proble with respect to β is forulated as: arg in β γ A i=1 s.t. i=1 β i tr(xl i X T A) + γ B 2 β 2 2 β i = 1,β i 0,i = 1,..., where A is derived fro optiized θ obtained last step. Siilarly, (8) can be written as: β = argin β T g + γ B β s.t. i=1 2 β 2 2 β i = 1,β i 0,i = 1,..., where the constant ter has been oitted, g = [g 1,...,g ] with each g i = γ A tr(xl i X T A). (9) is a standard quadratic prograing proble with linear constraints. Typically, it can be solved by adopting coordinate descent algorith. In each iteration, we select two eleents β i and β j for updating while the others fixed. Due to the constraint i=1 β i = 1, the suation of β i and β j will not change after current iteration. Therefore, we obtain the following updating rule: β i = γ B(β i + β j ) + (h j h i ) 2γ B β j = β i + β j β j To satisfied β i 0, we set βi = 0 and β j = β i + β j if γ B (β i + β j ) + (h j h i ) < 0. We iteratively traversal over all pairs of eleents in β with (10) until the object function in (5) does not decrease. In su, the two group of coefficients are learning alternately by this two sub-proble until convergence. 3 Experient Evaluation In this section, experients are conducted to validate the effectiveness of the proposed MT- DML on two popular datasets, a challenge real-world web iage annotation dataset and a handwritten iage dataset. For coparison purpose, we also evaluate the following ethods: RDML[6] A regularized distance etric learning ethod. The regularization ter, A 2 F, used to control odel coplexity. An online algorith is deonstrated to be efficient and effective for solving the proble. In this paper, we conduct an aggregation strategy which is siply applying RDML on the training set that consists of data fro both source and target tasks. This ethod also serves as a baseline for regularized etric learning. LRML[5] A sei-supervised distance etric learning ethod by utilizing unlabeled data. The LRML algorith is forulated as Seidefinite Progra (SDP) and solved by convex optiization techniques. This ethod serves as a baseline for sei-supervised etric learning. (8) (9) (10)

7 HAIBO SHI, YONG LUO, CHAO XU, YONGGANG WEN: MTDML 7 L-DML & M-DML[18] A transfer etric learning ethod that utilizes the auxiliary knowledge by adopting Bregan divergence as well also anifold regularization. The entries in target etric are estiated also by solving a SDP proble and we reipleent the algorith by the toolbox, SDPT3 solver. TML[19] A transfer etric learning algorith proposed recently. Relationship between the source and target tasks is learned for transfer by solving a second-order cone prograing (SOCP). An online algorith is developed for target etric s optiization. STML[19] A etric learning odel based on the TML. It is provided as sei-supervised extension in the sae paper with [19]. DTDML[10] A transfer etric learning ethod learned by decoposition based ethod. Our ethod is built on top of DTDML and is further iproved by exploiting useful inforation contained in the unlabeled data. With etric learned by above algoriths, we apply 1-Nearest Neighbor classifier on the test set and report the average classification accuracy. 3.1 Web Iage Annotation We first utilize the well-known natural iage dataset NUS-WIDE[3] to justify the significance of the proposed ethod for learning a reliable distance etric, especially under the data deficiency condition. This dataset contains 269, 648 iages and corresponding features. Our experients use features extracted by SIFT [7] descriptor with 500-D bag of visual words, which is available on the hoepage 1 of the dataset. To perfor transfer eaningfully, we select 12 anial concepts: bear, bird, cat, cow, dog, elk, fish, fox, horse, tiger, whale and zebra. For each concept, 100 saples are randoly picked fro the dataset. In this setting of experient, we randoly select 6 concepts as source tasks and target task requires annotation of all others. It fors as a ulti-class proble. The pair constraints are labeled as siilar if they are fro the sae class, otherwise dissiilar. Table 1: Average Annotation Perforance on NUS-WIDE Dataset Method Split 1 Split 2 Split 3 RDML L-DML M-DML TML DTDML MTDML The source etrics are trained by using the RDML ethod with all available data in the source tasks. For the target task, we divide the dataset by 6 labeled training saples for each class and 300 for testing, others as unlabeled data for sei-supervised setting. All trade-off paraeters are deterined epirically by adopting grid search. The overall classification 1

8 8 HAIBO SHI, YONG LUO, CHAO XU, YONGGANG WEN: MTDML result is reported in Table We perfor 3 different rando splits of the concept set and run the progra for 5 iterations and calculate the average accuracy. We then show that the perforance of different sei-supervised distance etric learning ethod w.r.t the nuber of labeled saples in each class. The result shows in Figure. 2. MTDML always perfors the best copared with other sei-supervised etric learning ethods. In particular, our ethod harvests a 6.5% iproveent on the average over M- DML when only 4 labeled saples are used in each class Accuracy MTDML STML 0.28 MDML LRDML # labeled training saples Figure 2: Coparison with other Sei-supervised Distance Metric Learning ethod. 3.2 Handwritten Iage Classification USPS 2 is a handwritten digit dataset, which contains 7,291 saples. Each saple is constructed by an iage of size in raw pixels. These raw iages are treated as features of d = 256 diension. Here, we conduct the sae experient settings with DTDML for coparison, i.e. nine tasks: 0/6, 0/8, 1/4, 2/7, 3/5, 4/7, 4/9, 5/8 and 6/8, each corresponding to a binary classification task. While training, we choose one of the nine tasks as target task and others as source tasks. Also, the source etrics for our ethod are obtained by RDML and all trade-off paraeters are set epirically with grid search. We present the average perforance over all settings of MTDML with soe related distance etric learning ethod. The result shows in Figure. (3). Fro the result, we observe that: 1) Coparing to RDML, our ethod achieves a better classification perforance which shows that source etrics and unlabeled data greatly help with learning target etric. 2) Our algorith is superior than LRML which ay due to exploiting inforation in source etrics as well as the finding of a bad local inia in LRML optiization steps. 3) The perforance of all copared ethods tends to be iproved while training saples increasing. MTDML outperfors other ethods and especially stable under training data deficiency condition. 2

9 HAIBO SHI, YONG LUO, CHAO XU, YONGGANG WEN: MTDML RDML LRML DTDML MTDML Coparison with baseline ethod 0.94 Average Accuracy #Training saples in each class Figure 3: USPS dataset in coparison with soe baseline ethod. 4 Conclusion This paper proposed the anifold regularization transfer distance etric learning odel, MTDML. By integrating source and target tasks with a data geoetric constraint, MTDML learns a robust and reliable etric fro both labeled saples and unlabeled saples. Moreover, our algorith can be solved quite efficiently under an alternative optiization fraework. Fro the experiental validation on the web iage annotation and handwritten iage classification task, MTDML outperfors other transfer etric learning and sei-supervised etric learning ethods, especially under training data deficiency conditions. Acknowledgeents This research was partially supported by grants fro NBRPC 2011CB302400, NSFC , 2015BAF15B00. References [1] Mikhail Belkin and Partha Niyogi. Laplacian eigenaps for diensionality reduction and data representation. Neural coputation, 15(6): , [2] Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geoetric fraework for learning fro labeled and unlabeled exaples. The Journal of Machine Learning Research, 7: , [3] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus-wide: a real-world web iage database fro national university of singapore. In Proceedings of the ACM international conference on iage and video retrieval, page 48. ACM, 2009.

10 10 HAIBO SHI, YONG LUO, CHAO XU, YONGGANG WEN: MTDML [4] Matthieu Guillauin, Thoas Mensink, Jakob Verbeek, and Cordelia Schid. Tagprop: Discriinative etric learning in nearest neighbor odels for iage autoannotation. In International Conference on Coputer Vision, pages IEEE, [5] Steven CH Hoi, Wei Liu, and Shih-Fu Chang. Sei-supervised distance etric learning for collaborative iage retrieval. In IEEE Conference on Coputer Vision and Pattern Recognition, pages 1 7. IEEE, [6] Rong Jin, Shijun Wang, and Yang Zhou. Regularized distance etric learning: Theory and algorith. In Advances in neural inforation processing systes, pages , [7] David G Lowe. Distinctive iage features fro scale-invariant keypoints. International journal of coputer vision, 60(2):91 110, [8] Yong Luo, Dacheng Tao, Bo Geng, Chao Xu, and Stephen J Maybank. Manifold regularized ultitask learning for sei-supervised ultilabel iage classification. IEEE Transactions on Iage Processing, 22(2): , [9] Yong Luo, Dacheng Tao, Chang Xu, Chao Xu, Hong Liu, and Yonggang Wen. Multiview vector-valued anifold regularization for ultilabel iage classification. IEEE Transactions on Neural Networks and Learning Systes, 24(5): , [10] Yong Luo, Tongliang Liu, Dacheng Tao, and Chao Xu. Decoposition-based transfer distance etric learning for iage classification. IEEE Transactions on Iage Processing, 23(9): , [11] Yong Luo, Tongliang Liu, Dacheng Tao, and Chao Xu. Multi-view atrix copletion for ulti-label iage classification. IEEE Transactions on Iage Processing, 24(8): , [12] Yong Luo, Dacheng Tao, Kotagiri Raaohanarao, Chao Xu, and Yonggang Wen. Tensor canonical correlation analysis for ulti-view diension reduction. IEEE transactions on Knowledge and Data Engineering, page DOI: , [13] Hua Wang, Feiping Nie, and Heng Huang. Robust distance etric learning via siultaneous l1-nor iniization and axiization. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages , [14] Kilian Q Weinberger, John Blitzer, and Lawrence K Saul. Distance etric learning for large argin nearest neighbor classification. In Advances in neural inforation processing systes, pages , [15] Eric P Xing, Michael I Jordan, Stuart Russell, and Andrew Y Ng. Distance etric learning with application to clustering with side-inforation. In Advances in neural inforation processing systes, pages , [16] Liu Yang and Rong Jin. Distance etric learning: A coprehensive survey. Michigan State Universiy, 2, [17] Liu Yang, Rong Jin, and Rahul Sukthankar. Bayesian active distance etric learning. arxiv preprint arxiv: , 2012.

11 HAIBO SHI, YONG LUO, CHAO XU, YONGGANG WEN: MTDML 11 [18] Zheng-Jun Zha, Tao Mei, Meng Wang, Zengfu Wang, and Xian-Sheng Hua. Robust distance etric learning with auxiliary knowledge. In IJCAI, pages , [19] Yu Zhang and Dit-Yan Yeung. Transfer etric learning with sei-supervised extension. ACM Transactions on Intelligent Systes and Technology (TIST), 3(3):54, 2012.