MULTI-VIEW ANCHOR GRAPH HASHING

Transcription

1 MULTI-VIEW ANCHOR GRAPH HASHING Saehoon Km 1 and Seungjn Cho 1,2 1 Department of Computer Scence and Engneerng, POSTECH, Korea 2 Dvson of IT Convergence Engneerng, POSTECH, Korea {kshkawa, seungjn}@postech.ac.kr ABSTRACT Mult-vew hashng seeks compact ntegrated bnary codes whch preserve smlartes averaged over multple representatons of objects. Most of exstng mult-vew hashng methods resort to lnear hash functons where data manfold s not consdered. In ths paper we present mult-vew anchor graph hashng (MVAGH), where nonlnear ntegrated bnary codes are effcently determned by a subset of egenvectors of an averaged smlarty matrx. The effcency behnd MVAGH s due to a low-rank form of the averaged smlarty matrx nduced by mult-vew anchor graph, where the smlarty between two ponts s measured by two-step transton probablty through vew-specfc anchor (.e. landmark) ponts. In addton, we observe that MVAGH suffers from the performance degradaton when the hgh recall s requred. To overcome ths drawback, we propose a smple heurstc to combne MVAGH wth localty senstve hashng (LSH). Numercal experments on CIFAR-10 dataset confrms that MVAGH(+LSH) outperforms the exstng mult- and sngle-vew hashng methods. Index Terms Anchor graphs, hashng, mult-vew learnng 1. INTRODUCTION Hashng seeks a hash functon to embed hgh-dmensonal data nto a smlarty-preservng low-dmensonal Hammng space such that an approxmate nearest neghbor of a gven query can be found wth sub-lnear tme complexty [2,15]. Classc approach on hashng (localty senstve hashng [2]) s to compute a hash functon purely n randomzed manner, where random projectons followed by roundng are used to generate bnary codes such that the two smlar examples have the same bnary codes. Snce the performance s not satsfed when short bnary codes are used [12], multple hash tables or longer bnary codes should be employed n practce. Learnng to hash seeks to the compact smlarty-preservng bnary codes, whch can be categorzed nto unsupervsed and (sem- )supervsed paradgms. Spectral hashng (SH) [14, 15] s a representatve unsupervsed hashng method, where the Laplace-Beltram egenfunctons of manfolds are used to determne bnary codes. Iteratve quantzaton (ITQ) [3] tres to rotate PCA-embedded data by an orthogonal matrx n order to mnmze the quantzaton error caused by mappng the embedded data nto a bnary hypercube. Semantc hashng [11] s the earlest supervsed hashng, explotng deep networks to learn a non-lnear mappng between nput data and bnary codes. Practcal usefulness of semantc hashng s lmted due to tme-consumng tranng tme. Supervsed hashng wth a reasonable tranng tme has been proposed, where a hash functon s determned sequentally yeldng very short dscrmnatve codes [8]. Sem-supervsed hashng [13] mnmzes the emprcal error nduced by the volaton of parwse constrants (must-lnk and cannot-lnk), and prevents from over-ftted hash functons usng unlabeled data. Recently, learnng to hash for mult-vew (or mult-modal) data has been developed. [4, 6, 17] seeks an ntegrated bnary codes whch preserves an averaged smlarty, extendng spectral hashng for mult-vew data. [6] explots a pre-defned averaged smlarty (or dentty matrx) to compute vew-specfc bnary codes whch concatenates nto an ntegrated code. [17] uses a lnear sum of vew-specfc smlarty matrces as an averaged smlarty, where an ntegrated bnary code s drectly computed. [4] seeks an ntegrated bnary code n a sequental manner n order to de-correlate bnary codes, where an averaged smlarty s computed by α-average of vew-specfc dstance matrces. [18] proposes a generatve model, where ntra-vew and nter-vew smlartes are generated gven by vew-specfc bnary codes. Besdes [4, 6, 17], [18] can adopt non-vectoral nputs, but the algorthm s not scalable and the performance s degraded when code length s ncreased. In ths paper, we present mult-vew anchor graph hashng (MVAGH), where an non-lnear ntegrated bnary code s determned by a subset of egenvectors of an averaged smlarty nduced by mult-vew anchor graph. Mult-vew anchor graph keeps the averaged smlarty n a low-rank form, where the egenvectors can be computed effcently. Note that hashng wth an anchor graph [9] has been already popular for unsupervsed hashng, but the extenson for mult-vew data has never been studed. Snce MVAGH computes non-lnear bnary code n an effcent way, t has a clear advantage than the prevous mult-vew hashng, where [4, 6, 17] tres to compute a lnear hash functon and [18] s not scalable and needs label nformaton. More specfcally, [4, 6, 17] consders a lnear hash functon to preserve the par-wse smlarty n Eucldean space. However, f the data le on the embedded low-dmensonal manfold, the lnear hash functon cannot well capture the data smlarty, because the examples measured large dstance n Eucldean dstance can be smlar n the manfold. [18] consders a non-lnear hash functon, but t s not scalable because the smlarty matrx s kept explctly. We observe that MVAGH suffers from the performance degradaton when the hgh recall s requred. To overcome ths drawback, we propose a smple heurstc to combne MVAGH wth localty senstve hashng (LSH). 2. MULTI-VIEW ANCHOR GRAPH HASHING In ths secton, we descrbe mult-vew anchor graph hashng (MVAGH), explotng mult-vew anchor graph to approxmate the averaged smlarty. Before dscussng MVAGH, we clarfy our notaton. Suppose that {x } N =1 s a set of N objects, where each object x s represented by K dfferent vew-specfc examples by {x (1),..., x (K) }. We defne the ntegrated bnary code matrx as

2 Y = {y } N =1 { 1, +1} r N, where y s the bnary code assocated wth x and r s the code length. MVAGH borrows the spectral hashng formulaton n order to seek an ntegrated bnary code matrx Y { 1, +1} r N whch preserves the averaged smlarty S : 1 arg mn Y 2 N =1 j=1 N Sj y y j 2 2, subject to Y 1 N = 0, 1 N Y Y = I r, (1) where y s the Eucldean norm of y and 1 N R N s the vector of all ones. Snce y y s always r, the objectve functon s equvalent to arg max Y 1 2 tr(y S Y ). Ignorng the bnary constrant, the soluton of wth the constrants of (1) s the largest egenvectors of the smlarty matrx. As n anchor graph hashng [9], f the smlarty matrx s low-rank approxmated, the egenvectors are effcently computed and the generalzed egenfunctons are also easly computed for a novel nput. Therefore, the natural queston s how to construct the low-rank approxmaton of the averaged smlarty nduced by mult-vew data Mult-Vew Anchor Graph x 1 x 2 x 3 (a) s 1 s 2 s 3 Fg. 1. Random walk vew of smlarty graph approxmated by anchor graph (a) and mult-vew anchor graph (b) when the number of vews s two. If more than two vews, the mult-vew anchor graph can be developed lke (b). Anchor graph [7] s a low-rank approxmaton of neghborhood graph (such as k-nn and ɛ-graph), where the smlarty between data ponts are measured by a small number of anchor ponts. Fg. 1 (a) represents the bpartte graph, where the left vertces are data ponts and the rght ones anchor ponts. Data ponts and ther assocated two-nearest anchor ponts are connected. The smlarty of two ponts are measured by two-step transton probablty through anchor ponts. We observe that the smlarty of two ponts s greater than zero only when they share the same anchor pont, whch means that the smlarty matrx approxmated by anchor graph s emprcally sparse and preserves the data localty. The anchor ponts should be selected to suffcently cover the data dstrbuton. In practce, the cluster centers by a few teraton of k-means are enough to be anchor ponts. Though anchor graph s successfully appled to unsupervsed hashng [9], the extenson for mult-vew data has not studed. We frst defne mult-vew anchor graph, where the dfferent contrbuton x1 x2 x3 (b) s 1 s 2 s 3 of each vew s well combned to approxmate the averaged neghborhood graph. Snce the data dstrbuton mght be dfferent for each vew, we select anchor ponts for each vew 1, and k-th vew specfc anchor ponts denoted as {µ (k) j specfc smlarty between the example x (k) Z (k) j = k(x (k), µ (k) j ) K k=1 m [] k(x(k), µ (k) m ) } M j=1. Defne the k-th vew and µ (k) j as, j [], (3) where [] contans the ndces of the l-nearest anchors of x (k) (usually l = 3), and k(, ) s any kernel functon. The average smlarty between the objects x and x j s denoted as S j = p(x j x ). When the number of vews s two, we consder a tr-partte graph n Fg. 1 (b) to approxmate the average smlarty by mult-vew anchor graph, where the smlarty between the two objects, x and x j, are two-step transton probablty through vewspecfc anchor ponts: p(x j x ) = where p(x j µ (k) m ) = K k=1 m=1 M p(x j µ (k) m )p(µ(k) m x), (4) Z (k) jm Nj=1 Z (k) jm and p(µ (k) m x ) = Z (k). We defne that Λ (k) s a dagonal matrx whose m-th dagonal entry s N j=1 Z(k) jm and [Z(k) ] m = Z (k) m. Usng ths notaton, p(xj x) = [ K k=1 Z(k) Λ (k) 1 Z (k) ] j. Lettng Z = [Z (1) Z (K) ] and Λ s dag(λ (1)... Λ (K) ), the averaged smlarty can be low-rank approxmated by mult-vew anchor graph: S Z Λ 1 Z. Mult-vew anchor graph can be nterpreted as lnear sum of vewspecfc anchor graph, where the normalzaton term n (3) s the lnk between the vew-specfc ones. The egenvectors of S are obtaned by the egen-problem soluton of small matrx A = Λ 1/2 Z Z Λ 1/2. Let the r-largest egenvalues Σ = dag(σ 1,..., σ r) assocated wth the egenvectors V = [v 1,..., v r] of A. Now, the transpose of egenvectors of S denoted as Ỹ s computed by Ỹ = NΣ 1/2 V Λ 1/2 Z = NW Z, (5) where W = Λ 1/2 V Σ 1/2. The bnary codes can be computed by roundng at zero: Y = sgn(ỹ ), where sgn functon operates on each element. However, the relaxed soluton s not satsfed, and we adapt several heurstcs to fnd the better soluton (snce the orgnal soluton s NP-hard, we have to use some reasonable heurstcs for the better soluton). In spectral clusterng, spectral roundng [16] has been wdely used to refne the soluton, where a rotaton matrx s estmated to mnmze the quantzaton error nduced by dscretzng the contnuous relaxed soluton. Recently, the smlar dea has been successfully appled to learn a hash functon, whch s known as teratve quantzaton (ITQ) [3]. The objectve functon of ITQ s presented as arg mn R,Y Y R Ỹ 2 F, s.t. Y 1 N = 0 N, R R = RR = I r, (6) 1 For vew-specfc anchor ponts, we apply k-means (10 teratons) nto each vew m

3 where R and Y are teratvely estmated to mnmze the quantzaton error as n [3, 16]. ITQ orgnally consders the PCA embedded space, but spectral embedded space s also well suted to ITQ, because we emprcally observe that the bnary codes become more compact and reflect the better data smlarty (Fg. 2). Fg. 2 (a) represents tr(y S Y ) over teratons, where we observe that the smlarty between bnary codes more reflects the orgnal data smlarty by ITQ. Fg. 2 (b) means 1 n Y Y I r 2 F over teratons, where we observe that the bnary codes are more de-correlated by ITQ. the smlarty between data and bnary code 2.8 x teraton (a) De correlaton of bnary codes teraton Fg. 2. Durng ITQ teraton, the bnary codes more preserve the data smlarty (a) and are more de-correlated (b). CIFAR-10 dataset s used Out-of-Sample Extenson A subset of egenvectors of mult-vew anchor graph s used to determne the bnary codes of tranng data. We can compute analytcally the bnary code of a novel pont by usng Nyström method as n [9]. Lemma 1 represents the analytc hash functon of a novel pont. Lemma 1. Gven m vew-specfc anchor ponts {µ (k) j } M j=1 for k = 1,..., K and any example x, defne a mappng z : R D R M as follows z(x) = [z(x(1) ),..., z(x (K) )] K, k=1 z(x(k) ) where z(x (k) ) = [δ 1k(x (k), µ (k) 1 ),..., δmk(x(k), µ (k) m )] and δ {1, 0}. δ j = 1 ff anchor µ (k) j s one of s-nearest anchors of sample x (k) accordng to the kernel functon k(, ). The bnary code of a novel pont, y, s expressed as where W s defned n (5). y = sgn(r W z(x)), The proof of ths lemma s straghtforward from Theorem 1 n [9]. Algorthm 1 represents a pseudo-code for MVAGH Mult-Vew Anchor Graph Hashng + LSH Mult-vew anchor graph hashng conssts of the two steps: (1) project the data onto the largest egenvectors of mult-vew anchor graph, and roundng at zero to produce bnary codes. Snce the ntrnsc dmenson s usually low, the smlarty between bnary codes wth large code sze mght not well reflect the data smlarty. We observe that the precson s decreased wth large code sze when the hgh recall s requred. As n Fg. 3, when the hgh recall s requred (the number of returned example s large), the smlarty nduced by bnary codes turns (b) Algorthm 1 Mult-Vew Anchor Graph Hashng (MVAGH) Input: For each k-th vew, tranng data are X (k) = [x (k) 1 x (k) n ] R m n, anchor ponts are {s (k) j } M j=1, and test data s x (k) R m for k = 1,, K. Bnary code length s r. Output: Bnary code y assocated wth the test data {x (k) } K k=1. 1: Compute the smlarty between the example and anchor pont as [Z (k) ] m usng the equaton 3. 2: Let Z = [Z (1),, Z (K) ] and Σ = [Σ (1),, Σ (K) ]. 3: Apply egenvalue decomposton to A = V ΣV, where A = Λ 1/2 Z Z Λ 1/2. 4: Ŷ = NW Z, where W = Λ 1/2 V Σ 1/2. 5: R 0 s a random rotaton matrx. 6: for = 1,..., 50 do 7: Y = sgn(r 1Ŷ ). 8: R = MM, where Y Ŷ = MΩ M by SVD. 9: end for 10: Return k-bt bnary code: y = sgn(r 50W z(x)), where z(x) s defned n Lemma 1. precson 5 5 precson at 50 precson at 200 precson at 500 precson at 2000 Fg. 3. Precson of MVAGH over the code sze when the number of returned examples s changed. to be ncorrect. Ths observaton leads us to propose a smple heurstc to ncrease retreval performance when code sze s large, where the bnary codes from MVAGH and localty-senstve hashng (LSH) are concatenated. We compute the meanngful bnary code from MVAGH by choosng r a largest egenvectors, where r a should be small (n practce r a = 32 s suffcent). If we want to use r 1bts bnary code and r 1 > r a, we frst compute Y r a = sgn(r Ỹ ), where Ỹ s transpose of r a largest egenvectors from mult-vew anchor graph. The remanng (r 1 r a)bts are generated by LSH, where Y reman = sgn(m R Ỹ ) and M s generated from N(0, I). The fnal bnary code s obtaned by concatenatng nto Y r a and Y reman, yeldng Y = [Y r a ; Y reman]. Snce the r a largest egenvectors reveal the ntrnsc data structure, we expect that the bts generated by LSH are meanngful. We denote ths heurstc as MVAGH + LSH, and secton 3 shows that ths method outperforms state-of-the art methods over all bts. 3. NUMERICAL EXPERIMENTS In ths secton, we use CIFAR-10 [5] contans 60,000 mages wth 10 labels. We form a query set by randomly choosng 1,000 mages and construct a tranng set usng the rest of mages. All experments are repeated fve tmes to compute the mean and standard devaton for error bars. We use GIST [10] and HOG [1] descrptors to produce two vews of each mage. For GIST, we use Gabor flter wth 8 orentatons and 3 scales, leadng to 384-dmensonal vector. For HOG,

4 5 5 5 (GIST) (HOG) CHMIS MVH CS SU MVSH (a) (b) (c) AGH ITQ RMMH Fg. 4. Precson of MVAGH+LSH when sngle feature s used, or two features combned by mult-vew anchor graph (a). Comparson between MVAGH+LSH nto mult-vew (b) and sngle-vew (c) hashng methods. 6 9 precson precson at 50 precson at precson at 500 precson at Fg. 5. Precson of MVAGH-LSH over the number of returned mage (left), and the effects of the parameter r a (rght). we compute the mage gradents of non-overlappng wndows, where the orentaton of gradents s quantzed nto 8 bns and normalzed wth 4 dfferent metrcs, and 36 4-by-4 non-overlappng wndows yeld 1152-dmensonal vectors. We compare our proposed method (MVAGH + LSH) and the recent mult-vew hashng methods: mult-vew hashng [6], CHMIS [17], and SU-MVAGH [4] (Fg. 4 (b)). We also compare our method nto the state-of-the art sngle-vew hashng methods: anchor graph hashng (AGH), teratve quantzaton (ITQ), random maxmum margn hashng (RMMH) (Fg. 4 (c)). For sngle-vew hashng methods, mult-vew data are concatenated nto sngle representaton. As a performance measure, we use Hammng rankng [9, 13] where the rankngs between a query and data ponts are decded by Hammng dstance. If there exsts a te n the Hammng dstance, we break t randomly. We calculate the precson at the top 500 examples. For the parameter of MVAGH, we select 500 anchor ponts for each vew, s = 3, and r a = 32. We use Gaussan kernel: exp( 1 x y 2 σ 2), where σ s set the medan of parwse dstances of data 2 ponts. We choose the best parameters for the compared methods. Fg. 4 summarzes the comparson between MVAGH + LSH nto varous state-of-the-art hashng methods, where (a) means mult-vew hashng clearly mproves the precson than usng sngle descrptor, (b) and (c) suggest that our MVAGH + LSH outperforms the exstng mult and sngle-vew hashng methods. Fg. 5 shows that the performance of MVAGH-LSH s not decreased when code length s large, and the effects of the parameter r a, whch means that f we choose longer r a than 32, the performance s decreased. Fg. 6 represents the example of retreval result, where the leftmost mage s a query and the 20-nearest neghbors are dsplayed. The ncorrect retreved mages are marked by red rectangle. We can see that the Fg. 6. Retreval results on CIFAR-10. Leftmost mage s a query and the 20-nearest mages are dsplayed, and the ncorrect ones are marked by the red rectangle. precson s ncreased when the two features are used together, and MVAGH s superor to the sngle-vew hashng methods. 4. CONCLUSIONS We have presented mult-vew anchor graph hashng (MVAGH) where non-lnear ntegrated bnary codes are determned by a subset of egenvectors of an averaged smlarty matrx. The underlyng dea was based on mult-vew anchor graph to keep the averaged smlarty n a low-rank form, where the smlarty between two ponts s measured by two-step transton probablty through vew-specfc anchor ponts. We have also presented a heurstc to mprove the performance of MVAGH by combnng t wth LSH, demonstratng ts hgh performance over exstng methods. Acknowledgments: Ths work was supported by NIPA-MSRA Creatve IT/SW Research Project, ITRC Program (NIPA-2012-H ), POSTECH Rsng Star Program, and NRF WCU Program (R ).

5 5. REFERENCES [1] N. Dalal and B. Trggs, Hstograms of orented gradents for human detecton, n Proceedngs of the IEEE Internatonal Conference on Computer Vson and Pattern Recognton (CVPR), San Dego, CA, [2] A. Gons, P. Indyk, and R. Motawan, Smlarty search n hgh dmensons va hashng, n Proceedngs of the Internatonal Conference on Very Large Data Bases (VLDB), [3] Y. Gong and S. Lazebnk, Iteratve quantzaton: A procrustean approach to learnng bnary codes, n Proceedngs of the IEEE Internatonal Conference on Computer Vson and Pattern Recognton (CVPR), Colorado Sprngs, CO, [4] S. Km, Y. Kang, and S. Cho, Sequental spectral learnng to hash wth multple representaton, n Proceedngs of the European Conference on Computer Vson (ECCV), Frenze, Italy, [5] A. Krzhevsky and G. E. Hnton, Learnng multple layers of features from tny mages, Computer Scence Department, Unversty of Toronto, Tech. Rep., [6] S. Kumar and R. Udupa, Learnng hash functons for crossvew smlarty search, n Proceedngs of the Internatonal Jont Conference on Artfcal Intellgence (IJCAI), Barcelona, Span, [7] W. Lu, J. He, and S. F. Chang, Large graph constructon for scalable sem-supervsed learnng, n Proceedngs of the Internatonal Conference on Machne Learnng (ICML), Hafa, Israel, [8] W. Lu, J. Wang, R. J, Y. G. Jang, and S. F. Chang, Supervsed hashng wth kernels, n Proceedngs of the IEEE Internatonal Conference on Computer Vson and Pattern Recognton (CVPR), Provdence, Rhode Island, USA, [9] W. Lu, J. Wang, S. Kumar, and S. F. Chang, Hashng wth graphs, n Proceedngs of the Internatonal Conference on Machne Learnng (ICML), Bellevue, WA, [10] A. Olva and A. Torralba, Modelng the shape of the scene: A holstc representaton of the spatal envelope, Internatonal Journal of Computer Vson, vol. 42, no. 3, pp , [11] R. Salakhutdnov and G. Hnton, Semantc hashng, n Proceedng of the SIGIR Workshop on Informaton Retreval and Applcatons of Graphcal Models, [12] A. Torralba, R. Fergus, and Y. Wess, Small codes and large mage databases for recognton, n Proceedngs of the IEEE Internatonal Conference on Computer Vson and Pattern Recognton (CVPR), Anchorage, Alaska, [13] J. Wang, S. Kumar, and S. F. Chang, Sem-supervsed hashng for scalable mage retreval, n Proceedngs of the IEEE Internatonal Conference on Computer Vson and Pattern Recognton (CVPR), San Francsco, CA, [14] Y. Wess, R. Fergus, and A. Torralba, Multdmensonal spectral hashng, n Proceedngs of the European Conference on Computer Vson (ECCV), Frenze, Italy, [15] Y. Wess, A. Torralba, and R. Fergus, Spectral hashng, n Advances n Neural Informaton Processng Systems (NIPS), vol. 20. MIT Press, [16] S. X. Yu and J. Sh, Multclass spectral clusterng, n Proceedngs of the Internatonal Conference on Computer Vson (ICCV), [17] D. Zhang, F. Wang, and L. S, Composte hashng wth multple nformaton sources, n Proceedngs of the ACM SIGIR Conference on Research and Development n Informaton Retreval (SIGIR), Bejng, Chna, [18] Y. Zhen and D. Y. Yeung, A probablstc model for multmodal hash functon learnng, n Proceedngs of the ACM SIGKDD Conference on Knowledge Dscovery and Data Mnng (KDD), Bejng, Chna, 2012.