arxiv: v1 [cs.cv] 29 Aug 2017

Transcription

1 Mult-vew Low-rank Sparse Subspace Clusterng Mara Brbć and Ivca Koprva arxv: v [cs.cv] 9 Aug 07 Rudjer Boskovc Insttute, Zagreb, Croata August 07 Abstract Most exstng approaches address mult-vew subspace clusterng problem by constructng the affnty matrx on each vew separately and afterwards propose how to extend spectral clusterng algorthm to handle mult-vew data. Ths paper presents an approach to mult-vew subspace clusterng that learns a jont subspace representaton by constructng affnty matrx shared among all vews. Relyng on the mportance of both low-rank and sparsty constrants n the constructon of the affnty matrx, we ntroduce the objectve that balances between the agreement across dfferent vews, whle at the same tme encourages sparsty and low-rankness of the soluton. Related low-rank and sparsty constraned optmzaton problem s for each vew solved usng the alternatng drecton method of multplers. Furthermore, we extend our approach to cluster data drawn from nonlnear subspaces by solvng the correspondng problem n a reproducng kernel Hlbert space. The proposed algorthm outperforms state-of-the-art mult-vew subspace clusterng algorthms on one synthetc and four real-world datasets. Introducton In many real-world machne learnng problems the same data s comprsed of several dfferent representatons or vews. For example, same documents may be avalable n multple languages [] or dfferent descrptors can be constructed from the same mages []. Although each of these ndvdual vews may be suffcent to perform a learnng task, ntegratng complementary nformaton from dfferent vews can reduce the complexty of a gven task []. Mult-vew clusterng seeks to partton data ponts based on multple representatons by assumng that the same cluster structure s shared across vews. By combnng nformaton from dfferent vews, mult-vew clusterng algorthms attempt to acheve more accurate cluster assgnments than one can get by smply concatenatng features from dfferent vews. In practce, hgh-dmensonal data often resde n a low-dmensonal subspace. When all data ponts le n a sngle subspace, the problem can be set as fndng a bass of a subspace and a low-dmensonal representaton of data ponts. Dependng on the constrants mposed on the lowdmensonal representaton, ths problem can be solved usng e.g. Prncpal Component Analyss (PCA) [4], Independent Component Analyss (ICA) [5] or Non-negatve Matrx Factorzaton (NMF) [6, 7, 8]. On the other hand, data ponts can be drawn from dfferent sources and le n a unon of subspaces. By assgnng each subspace to one cluster, one can solve the problem by applyng mara.brbc@rb.hr vca.koprva@rb.hr

2 standard clusterng algorthms, such as k-means [9]. However, these algorthms are based on the assumpton that data ponts are dstrbuted around centrod and often do not perform well n the cases when data ponts n a subspace are arbtrarly dstrbuted. For example, two ponts can have a small dstance and le n dfferent subspaces or can be far and stll le n the same subspace [0]. Therefore, methods that rely on a spatal proxmty of data ponts often fal to provde a satsfactory soluton. Ths has motvated the development of subspace clusterng algorthms [0]. The goal of subspace clusterng s to dentfy the low-dmensonal subspaces and fnd the cluster membershp of data ponts. Spectral based methods [,, ] present one approach to subspace clusterng problem. They have ganed a lot of attenton n the recent years due to the compettve results they acheve on arbtrarly shaped clusters and ther well defned mathematcal prncples. These methods are based on the spectral graph theory and represent data ponts as nodes n a weghted graph. The clusterng problem s then solved as a relaxaton of the mn-cut problem on a graph [4]. One of the man challenges n spectral based methods s the constructon of the affnty matrx whose elements defne the smlarty between data ponts. Sparse subspace clusterng [5] and lowrank subspace clusterng [6, 7, 8, 9] are among most effectve methods that solve ths problem. These methods rely on the self-expressveness property of the data by representng each data pont as a lnear combnaton of other data ponts. Low-Rank Representaton (LRR) [6, 7] mposes low-rank constrant on the data representaton matrx and captures global structure of the data. Low-rank mples that data matrx s represented by a sum of small number of outer products of left and rght sngular vectors weghted by correspondng sngular values. Under assumpton that subspaces are ndependent and data samplng s suffcent, LRR guarantees exact clusterng. However, for many real-world datasets ths assumpton s overly restrctve and the assumpton that data s drawn from dsjont subspaces would be more approprate [0, ]. On the other hand, Sparse Subspace Clusterng (SSC) [5] represents each data pont as a sparse lnear combnaton of other ponts and captures local structure of the data. Learnng representaton matrx n SSC can be nterpreted as sparse codng [,, 4, 5, 6, 7]. However, compared to sparse codng where dctonary s learned such that the representaton s sparse [8, 9], SSC s based on selfrepresentaton property.e. data matrx stands for a dctonary. SSC also succeeds when data s drawn from ndependent subspaces and the condtons have been establshed for clusterng data drawn from dsjont subspaces [0]. However, theoretcal analyss n [] shows that t s possble that SSC over-segments subspaces when the dmensonalty of data ponts s hgher than three. Expermental results n [] show that LRR msclassfes dfferent data ponts than SSC. Therefore, n order to capture global and the local structure of the data, t s necessary to combne low-rank and sparsty constrants [, ]. Mult-vew subspace clusterng can be consdered as a part of mult-vew or mult-modal learnng. Mult-vew learnng method n [4] learns vew generaton matrces and representaton matrx, relyng on the assumpton that data from all the vews share the same representaton matrx. The mult-vew method n [5] s based on the canoncal correlaton analyss n extracton of two-vew flter-bank-based features for mage classfcaton task. Smlarly, n [6] the authors rely on tensorbased canoncal correlaton analyss to perform mult-vew dmensonalty reducton. Ths approach can be used as a preprocessng step n mult-vew learnng n case of hgh-dmensonal data. In [7] low-rank representaton matrx s learned on each vew separately and learned representaton matrces are concatenated to a matrx from whch a unfed graph affnty matrx s obtaned. The method n [8] reles on learnng a lnear projecton matrx for each vew separately. Hgh-order dstancebased mult-vew stochastc learnng s proposed n [9], to effcently explore the complementary characterstcs of mult-vew features for mage classfcaton. The method n [40] s applcaton or-

3 ented towards mage rerankng and assumes that mult-vew features are contaned n hypergraph Laplacans that defne dfferent modaltes. In [4] authors propose mult-vew matrx completon algorthm for handlng mult-vew features n sem-supervsed mult-label mage classfcaton. Prevous mult-vew subspace clusterng works [4, 4, 44, 45] address the problem by constructng affnty matrx on each vew separately and then extend algorthm to handle mult-vew data. However, snce nput data may often be corrupted by nose, ths approach can lead to the propagaton of nose n the affnty matrces and degrade clusterng performance. Dfferent from the exstng approaches, we propose mult-vew spectral clusterng framework that jontly learns a subspace representaton by constructng sngle affnty matrx shared by mult-vew data, whle at the same tme encourages low-rank and sparsty of the representaton. We propose Mult-vew Low-rank Sparse Subspace Clusterng (MLRSSC) algorthms that enforce agreement: () between affnty matrces of the pars of vews; () between affnty matrces towards a common centrod. Opposed to [5, 40, 46], the proposed approach can deal wth hghly heterogeneous mult-vew data comng from dfferent modaltes. We present optmzaton procedure to solve the convex dual optmzaton problems usng Alternatng Drecton Method of Multplers (ADMM) [47]. Furthermore, we propose the kernel extenson of our algorthms by solvng the problem n a Reproducng Kernel Hlbert Space (RKHS). Expermental results show that MLRSSC algorthm outperforms state-of-the-art mult-vew subspace clusterng algorthms on several benchmark datasets. Addtonally, we evaluate performance on a novel real-world heterogeneous mult-vew dataset from bologcal doman. The remander of the paper s organzed as follows. Secton gves a bref overvew of the lowrank and sparse subspace clusterng methods. Secton ntroduces two novel mult-vew subspace clusterng algorthms. In Secton 4 we present the kernelzed verson of the proposed algorthms by formulatng subspace clusterng problem n RKHS. The performance of the new algorthms s demonstrated n Secton 5. Secton 6 concludes the paper. Background and Related Work In ths secton, we gve a bref ntroducton to Sparse Subspace Clusterng (SSC) [5], Low-Rank Representaton (LRR) [6, 7] and Low-rank Sparse Subspace Clusterng (LRSSC) [].. Man Notatons Throughout ths paper, matrces are represented wth bold captal symbols and vectors wth bold lower-case symbols. F denotes the Frobenus norm of a matrx. The l norm, denoted by, s the sum of absolute values of matrx elements; nfnty norm s the maxmum absolute element value; and the nuclear norm s the sum of sngular values of a matrx. Trace operator of a matrx s denoted by tr( ) and dag( ) s the vector of dagonal elements of a matrx. 0 denotes null vector. Table summarzes some notatons used throughout the paper.. Related Work Consder the set of N data ponts X = { x IR D} N that le n a unon of k > lnear subspaces = of unknown dmensons. Gven the set of data ponts X, the task of subspace clusterng s to cluster data ponts accordng to the subspaces they belong to. The frst step s the constructon of the affnty matrx W IR N N whose elements defne the smlarty between data ponts. Ideally, the affnty matrx s a block dagonal matrx such that a nonzero dstance s assgned to the ponts

4 Table : Notatons and abbrevatons Notaton N k v n v D (v) X (v) IR D(v) N IR N N C IR N N W IR N N X = UΣV T Φ(X (v) ) K (v) IR N N Defnton Number of data ponts Number of clusters Vew ndex Number of vews Dmenson of data ponts n a vew v Data matrx n a vew v Representaton matrx n a vew v Centrod representaton matrx Affnty matrx Sngular value decomposton (SVD) of X Data ponts n a vew v mapped nto hgh-dmensonal feature space Gram matrx n a vew v from the same subspace. LRR, SSC and LRSSC construct the affnty matrx by enforcng low-rank, sparsty and low-rank plus sparsty constrants, respectvely. Low-Rank Representaton (LRR) [6, 7] seeks to fnd a low-rank representaton matrx C IR N N for nput data X. The basc model of LRR s the followng: mn C C s.t. X = XC, () where the nuclear norm s used to approxmate the rank of C and that results n the convex optmzaton problem. Denote the SVD of X as UΣV T. The mnmzer of equaton () s unquely gven by [6]: Ĉ = VV T. () In the cases when data s contamnated by nose, the followng problem needs to be solved: mn C The optmal soluton of equaton () has been derved n [8]: X XC F +λ C. () Ĉ = V (I λ Σ )V T, (4) where U = [U U ], Σ = dag(σ Σ ) and V = [V V ]. Matrces are parttoned accordng to the sets I = { : σ > λ } and I = { : σ λ }. Sparse Subspace Clusterng (SSC) [5] requres that each data pont s represented by a small number of data ponts from ts own subspace and t amounts to solve the followng mnmzaton problem: mn C s.t. X = XC, dag(c) = 0. (5) C The l norm s used as the tghtest convex relaxaton of the l 0 quas-norm that counts the number of nonzero elements of the soluton. Constrant dag(c) = 0 s used to avod trval soluton of representng a data pont as a lnear combnaton of tself. 4

5 If data s contamnated by nose, the followng mnmzaton problem needs to be solved: mn C X XC F +λ C s.t. dag(c) = 0. (6) Ths problem can be effcently solved usng ADMM optmzaton procedure [47]. Low-Rank Sparse Subspace Clusterng (LRSSC) [] combnes low-rank and sparsty constrants: mn C +λ C s.t. X = XC, dag(c) = 0. (7) C In the case of the corrupted data the followng problem needs to be solved to approxmate C: mn C X XC F +β C +β C s.t. dag(c) = 0. (8) Once matrx C s obtaned by LRR, SSC or LRSSC approach, the affnty matrx W s calculated as: W = C + C T. (9) Gven affnty matrx W, spectral clusterng [, ] fnds cluster membershp of data ponts by applyng k-means clusterng to the egenvectors of the graph Laplacan matrx L IR N N computed from the affnty matrx W. Mult-vew Low-rank Sparse Subspace Clusterng In ths secton we present Mult-vew Low-rank Sparse Subspace Clusterng (MLRSSC) algorthm wth two dfferent regularzaton approaches. We assume that we are gven a datasetx = { X (),X (),...,X (nv)} of n v vews, where each X () = { x () } j IR D() N s descrbed wth ts own set of D() j= features. Our objectve s to fnd a jont representaton matrx C that balances trade-off between the agreement across dfferent vews, whle at the same tme promotes sparsty and low-rankness of the soluton. We formulate jont objectve functon that enforces representaton matrces { C (),C (),...,C (nv)} across dfferent vews to be regularzed towards a common consensus. Motvated by [4], we propose two regularzaton schemes of the MLRSSC algorthm: () MLRSSC based on parwse smlartes and () centrod-based MLRSSC. The frst regularzaton encourages smlarty between pars of representaton matrces. The centrod-based approach enforces representatons across dfferent vews towards a common centrod. Standard spectral clusterng algorthm can then be appled to the jontly nferred affnty matrx.. Parwse Mult-vew Low-rank Sparse Subspace Clusterng We propose to solve the followng jont optmzaton problem over n v vews: mn C (),C (),...,C (nv) n v v= ( β +β ) + v,w n v,v w s.t. X (v) = X (v), dag( ) = 0, v =,...n v, λ (v) C (w) F (0) where IR N N s the representaton matrx for vew v. Parameters β,β and λ (v) defne the trade-off between low-rank, sparsty constrant and the agreement across vews, respectvely. 5

6 In the cases where we do not have a pror nformaton that one vew s more mportant than the others, λ (v) does not dependent on a vew v and the same value of λ (v) s used across all vews v =,...,n v. The last term n the objectve n (0) s ntroduced to encourage smlartes between pars of representaton matrces across vews. Wth all but one fxed, we mnmze the functon (0) for each ndependently: mnβ +β +λ (v) w n v,v w C (w) F s.t. X (v) = X (v), dag( ) = 0. By ntroducng auxlary varables, C(v), C(v) and A (v), we reformulate the objectve: mn β,c(v),c(v),a(v) +β +λ (v) w n v,v w C (w) F s.t. X (v) = X (v) A (v), A (v) = dag( ), A(v) =, A(v) = (). () The augmented Lagrangan s: L ( { } =,A(v),{Λ (v) } 4 ) = = β +β +λ (v) (v) C C (w) F + µ X (v) X (v) A (v) F + µ +tr [ Λ (v) +tr [ Λ (v) w n v,w v A (v) +dag( ) F + µ T( X (v) X (v) A (v))] +tr [ Λ (v) T( A (v) )] +tr [ Λ (v) 4 A (v) F + µ 4 T( A (v) +dag( T( A (v) )], A (v) ))] F () where {µ > 0} = are penalty parameters that need to be tuned and {Λ(v) } 4 = are Lagrange dual varables. To solve the convex optmzaton problem n (), we use Alternatng Drecton Method of Multplers (ADMM) [47]. ADMM converges for the objectve composed of two-block convex separable problems, but here the terms, C(v) and do not depend on each other and can be observed as one varable block. Update rule for A (v) at teraton k+. Gven { } =,{Λ(v) } 4 = at teraton k, the matrx A (v) that mnmzes the objectve n equaton () s updated by the followng update rule: A (v) = [ µ X (v)t X (v) +(µ +µ +µ 4 )I ] ( µ X (v)t X (v) +µ +µ +µ 4 +X (v)t Λ (v) Λ (v) Λ (v) Λ (v) ) (4) 4. The update rule follows straghtforwardly by settng the partal dervatve of L n equaton () wth respect to A (v) to zero. Update rule for at teraton k+. Gven A (v) at teraton k+ and Λ (v) at teraton 6

7 k, we mnmze the objectve n equaton () wth respect to mnl ( = mn = mn = mn,a(v),λ (v) β β β ) + µ + µ + µ A (v) A (v) A (v) F +tr[ Λ (v) F +tr[ Λ (v) + Λ(v) µ F, : T( A (v) )] T( A (v) )] Λ (v) + F µ (5) From [48], t follows that the the unque mnmzer of (5) s: ( = Πβ A (v) + Λ(v) ), (6) µ µ where Π β (Y) = Uπ β (Σ)V T performs soft-thresholdng operaton on the sngular values of Y and UΣV T s the sknny SVD of Y, here Y = A (v) +µ Λ(v). π β(σ) denotes soft thresholdng operator defned as π β (Σ) = ( Σ β) + sgn(σ) and t + = max(0,t). Update rule for at teraton k+. Gven A (v) at teraton k+ and Λ (v) at teraton k, we mnmze the L n equaton () wth respect to : mnl (,A(v),Λ (v) ) = mn β + µ = dag( ). A (v) + Λ(v) µ The mnmzaton of (7) gves the followng update rules for matrx [49, 50]: ( = πβ A (v) + Λ(v) ) µ µ F (7) = dag( ), (8) where π β denotes soft thresholdng operator appled entry-wse to ( A (v) +µ ) Λ(v). Update rule for at teraton k+. GvenA (v) at teratonk+ andλ (v) 4, w n v,w v C(w) at teraton k, we mnmze the objectve n equaton () wth respect to mnl (,A(v),Λ (v) ) 4 = mn + µ 4 λ (v) w n v,w v A (v) C (w) F F +tr[ Λ (v) 4 The partal dervatve of L n equaton () wth respect to : L = [ λ (v) (n v )+µ 4 ] C (v) λ (v) 7 w n v,w v : (9) T( A (v) )]. C (w) µ 4 A Λ (v) 4. (0)

8 Settng the partal dervatve n (0) to zero: = [ λ (v) (n v )+µ 4 ] ( λ (v) w n v,w v C (w) +µ 4 A (v) +Λ (v) ) 4. () Update rules for dual varables {Λ (v) } 4 = at teraton k +. Gven A(v),{ } = at teraton k+, dual varables are updated wth the followng equatons: Λ (v) = Λ (v) ( +µ X (v) X (v) A (v) ) Λ (v) = Λ (v) ( +µ A (v) Λ (v) = Λ (v) ( +µ A (v) ) Λ (v) 4 = Λ (v) 4 +µ 4 ( A (v) ) ). () If data s contamnated by nose and does not perfectly le n the unon of subspaces, we modfy the objectve functon as follows: mn C (),C (),...,C (nv) n v v= + ( X (v) X (v) F +β C (v) +β ) v,w n v,v w λ (v) C (w) F s.t. dag( ) = 0, v =,...n v. () Update rule for A (v) at teraton k + for corrupted data. Gven { } =,{Λ(v) } 4 = at teraton k, the matrx A (v) s obtaned by equatng to zero partal dervatve of the augmented Lagrangan of problem (): A (v) = [ X (v)t X (v) +(µ +µ +µ 4 )I ] ( X (v) T X (v) +µ +µ +µ 4 Λ (v) Λ (v) Λ (v) ) (4) 4. Update rules for { } = and dual varables {Λ(v) } 4 = are the same as n (6), (8), (), (), respectvely. These update steps are then repeated untl the convergence or untl the maxmum number of teraton s reached. We check the convergence by verfyng the followng constrants at each teraton k: A (v) ǫ, A (v) ǫ, A (v) ǫ and (v) A k A (v) k ǫ, for v =,...,n v. After obtanng representaton matrx for each vew { C (),C (),...,C (nv)}, we combne them by takng the element-wse average across all vews. The next step of the algorthm s to fnd the assgnment of the data ponts to correspondng clusters by applyng spectral clusterng algorthm to the jont affnty matrx W = C avg + C avg T. Algorthm summarzes the steps of the parwse MLRSSC. Due to the practcal reasons, we use the same ntal values of {µ } 4 =, ρ and µmax for dfferent vews v and update {µ } 4 = after the optmzatons of all vews. However, t s possble to have more general approach wth dfferent ntal values of {µ } 4 =, ρ and µmax for each vew v, but ths sgnfcantly ncreases the number of varables for optmzaton. The problem n (0) s convex subject to lnear constrants and all ts subproblems can be solved exactly. Hence, theoretcal results n [5] guarantee the global convergence of ADMM. The 8

9 computatonal complexty of Algorthm so(tn v N ), wheret s the number of teratons, n v N s the number of vews and N s the number of data ponts. In the experments, we set the maxmal T to 00, but the algorthm converged before the maxmal number of teratons s exceeded (T 5 0). Importantly, the computatonal complexty of spectral clusterng step s O(N ), so the computatonal cost of the proposed representaton learnng step s Tn v tmes hgher.. Centrod-based Mult-vew Low-rank Sparse Subspace Clusterng In addton to the parwse MLRSSC, we also ntroduce objectve for the centrod-based MLRSSC whch enforces vew-specfc representatons towards a common centrod. We propose to solve the followng mnmzaton problem: mn C (),C (),..,C (nv) n v v= ( β +β +λ (v) C F) s.t. X (v) = X (v), dag( ) = 0, v =,...n v, (5) where C denotes consensus varable. Algorthm Parwse MLRSSC Input: X = {X (v) } nv v=, k, β, β, {λ (v) } nv v=,{µ,} 4 =,µmax,ρ Output: Assgnment of the data ponts to k clusters : Intalze: { = 0} =, A(v) = 0, {Λ (v) = 0} 4 =, =,...,n v : whle not converged do : for v = to n v do 4: Fx others and update A (v) by solvng (4) n the case of clean data or (4) n the case of corrupted data 5: Fx others and update by solvng (6) 6: Fx others and update by solvng (8) 7: Fx others and update by solvng () 8: Fx others and update dual varables Λ (v),λ(v),λ(v) 4 by solvng () and also Λ (v) n the case of clean data 9: end for 0: Update µ = mn(ρµ,µ max ), =,...,4 : end whle : Combne { C (),C (),...,C (nv)} by takng the element-wse average : Apply spectral clusterng [] to the affnty matrx W = C avg + C avg T Ths objectve functon can be mnmzed by the alternatng mnmzaton cyclng over the vews and consensus varable. Specfcally, the followng two steps are repeated: () fx consensus varable C and update each, v =,...,n v whle keepng all others fxed and () fx,v =,...,n v and update C. By fxng all varables except one, we solve the followng problem: mn β +β +λ (v) C F s.t. X (v) = X (v), dag( ) = 0. (6) Agan, we solve the convex optmzaton problem usng ADMM. We ntroduce auxlary varables 9

10 , C(v), C(v) and A (v) and reformulate the orgnal problem: mn β,c(v),c(v),a(v) +β +λ (v) C F s.t. X (v) = X (v) A (v), A (v) = dag( ), A(v) =, A(v) = The augmented Lagrangan s: L ( { } =,A(v),{Λ (v) } 4 ) = = β +β +λ (v) (v) C C F + µ X (v) X (v) A (v) F + µ A (v) +dag(c ) F + µ A (v) C F + µ 4 A (v) F +tr[ Λ (v) T (X (v) X (v) A (v) ) ] +tr [ Λ (v) T (A (v) )] +tr [ Λ (v) T (A (v) +dag( )] +tr [ Λ (v) T 4 (A (v) )].. (7) Update rule for at teraton k+. Gven A (v) at teraton k+ and C, Λ (v) 4 at teraton k, mnmzaton of the objectve n equaton (8) wth respect to leads to the followng update rule for : = ( λ (v) ) ( +µ 4 λ (v) C +µ 4 A (v) +Λ (v) ) 4. (9) Update rule for C. By settng the partal dervatve of the objectve functon n equaton (5) wth respect to C to zero we get the closed-form soluton to C : (8) C = nv v= λ(v) nv v= λ(v). (0) It s easy to check that update rules for varables A (v),, C(v) and dual varables {Λ (v) } 4 = are the same as n the parwse smlartes based mult-vew LRSSC (equatons (4), (6),(8) and ()). In order to extend the model to the data contamnated by addtve whte Gaussan nose, the objectve n (5) s modfed as follows: n v mn C (),...,C (nv) v= X (v) X (v) F +β C (v) +β +λ (v) C F s.t. dag( ) = 0, v =,...n v. () Compared to the model for clean data, the only update rule that needs to be modfed s for A (v), whch s the same as n parwse MLRSSC gven n equaton (4). In centrod-based MLRSSC there s no need to combne affnty matrces across vews, snce the jont affnty matrx can be drectly computed from the centrod matrx.e. W = C + C T. Algorthm summarzes the steps of centrod-based MLRSSC. The computatonal complexty of Algorthm s the same as the complexty of Algorthm. 4 Kernel Mult-vew Low-rank Sparse Subspace Clusterng The spectral decomposton of Laplacan enables spectral clusterng to separate data ponts wth nonlnear hypersurfaces. However, by representng data ponts as a lnear combnaton of other data 0

11 Algorthm Centrod-based MLRSSC Input: X = {X (v) } nv v=, k, β, β, {λ (v) } nv v=,{µ,} 4 =,µmax,ρ Output: Assgnment of the data ponts to k clusters : Intalze: { = 0} =, C = 0, A (v) = 0, {Λ (v) = 0} 4 =, =,...,n v : whle not converged do : for v = to n v do 4: Fx others and update A (v) by solvng (4) n the case of clean data or (4) n the case of corrupted data 5: Fx others and update by solvng (6) 6: Fx others and update by solvng (8) 7: Fx others and update by solvng (9) 8: Fx others and update dual varables Λ (v),λ(v),λ(v) 4 by solvng () and also Λ (v) n the case of clean data 9: end for 0: Update µ = mn(ρµ,µ max ), =,...,4 : Fx others and update centrod C by solvng (0) : end whle : Apply spectral clusterng [] to the affnty matrx W = C + C T ponts, the MLRSSC algorthm learns the affnty matrx that models the lnear subspace structure of the data. In order to recover nonlnear subspaces, we propose to solve the MLRSSC n RKHS by mplctly mappng data ponts nto a hgh dmensonal feature space. We defne Φ : X F to be a functon that maps the orgnal nput space X to a hgh (possbly nfnte) dmensonal feature space F. Snce the presented update rules for the corrupted data of both parwse and centrod-based MLRSSC depend only on the dot products X (v),x (v) = X (v)t X (v),v =,...,n v, both approaches can be solved n RKHS and extended to model nonlnear manfold structure. Let Φ(X (v) ) = { Φ(x (v) ) F } N = denote the set of data ponts X(v) = { x (v) IR D} N = mapped nto hgh-dmensonal feature space. The objectve functon of parwse kernel MLRSSC for data contamnated by nose s the followng: s: mn C (),C (),...,C (nv) n v v= + ( Φ(X (v) ) Φ(X (v) ) F +β C (v) +β ) v,w n v,v w λ (v) C (w) F s.t. dag( ) = 0, v =,...n v. Smlarly, the objectve functon of centrod-based MLRSSC n feature space for corrupted data mn C (),C (),...,C (nv) n v v= ( Φ(X (v) ) Φ(X (v) ) F +β C (v) +β +λ (v) C F s.t. dag( ) = 0, v =,...n v. Snce A (v) s the only varable that depends on X (v), the update rules for { } = varables {Λ (v) } 4 = reman unchanged. ) () () and dual

12 Update rule for A (v) at teraton k+. Gven { s updated by the followng update rule: } =,{Λ(v) } 4 = at teraton k, the A(v) A (v) = [ Φ(X (v) ) T Φ(X (v) )+(µ +µ +µ 4 )I ] [ Φ(X (v) ) T Φ(X (v) )+µ +µ +µ 4 Λ (v) Λ (v) Λ (v) ] (4) 4. Substtutng the dot product Φ(X (v) ),Φ(X (v) ) wth the Gram matrx K (v), we get the followng update rule for A (v) : A (v) = [ K (v) +(µ +µ +µ 4 )I ] [ K (v) +µ +µ +µ 4 Λ (v) Λ (v) Λ (v) ] (5) 4. Update rule for A (v) s the same n parwse and centrod-based versons of the algorthm. 5 Experments In ths secton we present results that demonstrate the effectveness of the proposed algorthms. The performance s measured on one synthetc and three real-world datasets that are commonly used to evaluate the performance of mult-vew algorthms. Moreover, we ntroduce novel real-world mult-vew dataset from molecular bology doman. We compared MLRSSC wth the state-of-theart mult-vew subspace clusterng algorthms, as well as wth two baselnes: best sngle vew LRSSC and feature concatenaton LRSSC. 5. Datasets We report the expermental results on synthetc and four real-world datasets. We gve a bref descrpton of each dataset. Statstcs of the datasets are summarzed n Table. UCI Dgt dataset s avalable from the UCI repostory. Ths dataset conssts of 000 examples of handwrtten dgts (0-9) extracted from Dutch utlty maps. There are 00 examples n each class, each represented wth sx feature sets. Followng experments n [45], we used three feature sets: 76 Fourer coeffcents of the character shapes, 6 profle correlatons and 64 Karhunen-Love coeffcents. Reuters dataset [5] contans features of documents avalable n fve dfferent languages and ther translatons over a common set of sx categores. All documents are n the bag-of-words representaton. We use documents orgnally wrtten n Englsh as one vew and ther translatons to French, German, Spansh and Italan as four other vews. We randomly sampled 00 documents from each class, resultng n a dataset of 600 documents. -sources dataset s news artcles dataset collected from three onlne news sources: BBC, Reuters, and The Guardan. All artcles are n the bag-of-words representaton. Of 948 artcles, we used 69 that are avalable n all three sources. Each artcle n the dataset s annotated wth a domnant topc class. Prokaryotc phyla dataset contans 55 prokaryotc speces descrbed wth heterogeneous mult-vew data ncludng textual data and dfferent genomc representatons [5]. Textual data conssts of bag-of-words representaton of documents descrbng prokaryotc speces and s consdered

13 as one vew. In our experments we use two genomc representatons: () the proteome composton, encoded as relatve frequences of amno acds () the gene repertore, encoded as presence/absence ndcators of gene famles n a genome. In order to reduce the dmensonalty of the dataset, we apply prncpal component analyss (PCA) on each of the three vews separately and retan prncpal components explanng 90% of the varance. Each speces n the dataset s labeled wth the phylum t belongs to. Unlke prevous datasets, ths dataset s unbalanced. The most frequently occurrng cluster contans speces, whle the smallest cluster contans 5 speces. Synthetc dataset was generated n a way descrbed n [4, 54]. 000 ponts are generated from two vews, where data ponts for each vew are generated from two-component Gaussan mxture models. Cluster means and covarance matrces for vew are: µ () = ( ), Σ () = ( 0.5; 0.5.5), µ () = ( ), Σ () = (0. 0; 0 0.6), and for vew are: µ () = ( ), Σ () = (0. 0; 0 0.6), µ () = ( ), Σ () = ( 0.5; 0.5.5). Table : Statstcs of the mult-vew datasets Dataset Samples Vews Clusters UCI Dgt Reuters sources 69 6 Prokaryotc 55 4 Synthetc Compared Methods and Parameters We compare parwse MLRSSC, centrod-based MLRSSC and kernel extensons of both algorthms (KMLRSSC) wth the best performng state-of-the-art mult-vew subspace clusterng algorthms, ncludng Co-regularzed Mult-vew Spectral Clusterng (Co-Reg) [4], Robust Mult-vew Spectral Clusterng (RMSC) [44] and Convex Sparse Mult-vew Spectral Clusterng (CSMSC) [45]. Moreover, we also compare MLRSSC algorthms wth two LRSSC baselnes: () best sngle vew Low-rank Sparse Subspace Clusterng (LRRSC) [] that performs sngle vew LRSSC on each vew and takes the ndvdual vew that acheves the best performance, and () feature concatenaton LRRSC that concatenates features of each ndvdual vew and performs sngle-vew LRSSC on the jont vew representaton. Co-regularzed mult-vew SC has a parameter α that we vary from 0.0 to 0.05 wth step 0.0 [4]. We choose λ n RMSC from the set of the values: {0.005,0.0,0.05,0.,0.5,,5,0,50,00}, as tested n [44]. Parameter α n CSMSC s chosen from {0,0 } and parameter β from {0,0 4,0 5 } [45]. For all these algorthms the standard devaton of Gaussan kernel used to buld smlarty matrx s set to the medan of the parwse Eucldean dstances between the data ponts [4, 44, 45]. The number of teratons of the Co-Reg SC s set to 00, but t converged wthn less than 0 teratons. The number of teratons of the CSMSC s set to 00 [45] and of the RMSC to 00, as set n the avalable source code provded by the authors. All other parameters of these algorthms are set to values based on the respectve source codes provded by ther authors. For LRSSC and MLRSSC we frst choose penalty parameterµfrom the set of values{0,0,0, 0 4 } wth fxed β, β and λ (v). We set the same value µ for all constrants (µ, =,...,4), but one can also optmze µ for each of the constrants. In each teraton we update µ to be ρµ wth fxed ρ of.5 and tll the maxmal value of µ (set to 0 6 ) s reached. For sngle-vew LRSSC ρ s set to

14 . Low-rank parameter β s tuned from 0. to 0.9 wth step 0. and sparsty parameter β s set to ( β ). Consensus parameter λ s tuned from 0. to 0.9 wth step 0.. It s also possble to use dfferent λ (v) for each vew v, but snce we dd not have any pror nformaton about the mportance of vews, we use the same λ = λ (v) for each vew v. For all datasets we use the varant adjusted for the corrupted data, except for the UCI dgt dataset. In the kernel extenson of MLRSSC, we use Gaussan kernel and optmze standard devaton for each vew separately n range {0.5,, 5, 0, 50} tmes the medan of the parwse Eucldean dstances between the data ponts, whle holdng other parameters fxed. Best sgma for parwse MLRSSC was also used for centrod MLRSSC wthout further optmzaton. The maxmum number of teratons s set to 00 and the convergence error tolerance to ǫ = 0 for lnear MLRSSC and ǫ = 0 5 for kernel MLRSSC. We tune the parameters of each algorthm and report the best performance. All compared methods have k-means as the last step of the algorthm. Snce k-means depends on the ntal cluster centrod postons and can yeld dfferent soluton wth dfferent ntalzatons, we run k-means 0 tmes and report the means and standard devatons of the performance measures. We evaluate clusterng performance usng fve dfferent metrcs: precson, recall, F-score, normalzed mutual nformaton (NMI) and adjusted rand ndex (Adj-RI) [55]. For all these metrcs, the hgher value ndcates better performance. 5. Results Table compares the clusterng performance of the MLRSSC wth other algorthms on four realworld datasets and one synthetc dataset. Results ndcate that MLRSSC consstently outperforms all other methods n terms of all tested measures. On all fve datasets, MLRSSC mproves performance to a large extent whch demonstrates the mportance of combned low-rank and sparsty constrants. More specfcally, the average NMI of the MLRSSC s hgher than the second best method by 7%, 9%, 4%, % and % on the -sources, Reuters, UCI dgt, Prokaryotc and synthetc datasets, respectvely. Smlar mprovements can also be observed when usng other metrcs for measurng clusterng performance. Parwse and centrod-based MLRSSC perform comparably, except on Prokaryotc dataset where parwse MLRSSC s sgnfcantly better than the centrod-based MLRSSC, except n recall. When comparng lnear MLRSSC wth the kernel MLRSSC, lnear MLRSSC performs better on -sources and Reuters datasets. Kernel MLRSSC outperforms lnear MLRSSC on UCI Dgt, Prokaryotc and synthetc datasets, although the dfference on the UCI Dgt dataset s not sgnfcant. However, ths comes wth the cost of tunng more parameters for computng the kernel. Better performance of lnear MLRPPSC on -sources and Reuters datasets s not suprsng, snce these datasets are very sparse (more than 95% values are zeros) and have a large number of features, much hgher than the number of data ponts. On the other hand, UCI Dgt, Prokaryotc and especally synthetc datasets have dense lower-dmensonal feature vectors and beneft from the projecton to a hgh-dmensonal feature space. 5.4 Parameter Senstvty MLRSSC trades-off low-rank, sparsty and consensus parameters: β, β and λ (v), respectvely. In ths secton, we test the effect of these parameters on the performance of the MLRSSC. In all experments, we set the sparsty parametar β to β,.e. the hgher value of a low-rank parameter leads to the lower value of a sparsty parameter and vce versa. Ths depends on whether the problem beng solved requres explotng more global or the local structure of the data. 4

15 Table : Performance of dfferent algorthms on fve mult-vew datasets. The mean and standard devaton of 0 runs of k-means clusterng algorthm wth dfferent random ntalzatons are reported. Dataset Method F-score Precson Recall NMI Adj-RI Best Sngle Vew LRSSC (0.09) (0.05) 0.54 (0.058) (0.04) (0.04) Feat Concat LRSSC (0.048) 0.59 (0.0) 0.57 (0.078) 0.5 (0.05) (0.054) Co-Reg Parwse 0.46 (0.00) (0.049) 0.47 (0.0) 0.59 (0.06) 0.5 (0.0) Co-reg Centrod (0.0) 0.55 (0.05) (0.05) 0.54 (0.06) 0.70 (0.045) -sources RMSC (0.0) 0.55 (0.04) 0.45 (0.06) 0.57 (0.04) 0.0 (0.045) CSMSC 0.48 (0.06) 0.58 (0.056) (0.07) 0.58 (0.06) 0.5 (0.09) Parwse MLRSSC (0.049) (0.05) 0.69 (0.056) (0.05) (0.060) Centrod MLRSSC (0.04) (0.055) 0.69 (0.05) (0.0) (0.05) Parwse KMLRSSC 0.54 (0.05) 0.69 (0.0) 0.48 (0.0) 0.59 (0.00) 0.44 (0.09) Centrod KMLRSSC (0.045) 0.6 (0.049) 0.50 (0.044) 0.5 (0.0) 0.49 (0.056) Best Sngle Vew LRSSC 0. (0.00) 0. (0.007) 0.57 (0.09) 0.45 (0.008) 0.9 (0.005) Feat Concat LRSSC 0.47 (0.005) 0.9 (0.00) 0.84 (0.0) 0.8 (0.006) 0.04 (0.008) Co-Reg Parwse 0.7 (0.009) 0.44 (0.06) 0.40 (0.0) 0.00 (0.04) 0. (0.07) Co-reg Centrod 0.6 (0.07) 0. (0.0) (0.00) 0.9 (0.04) 0. (0.0) Reuters RMSC 0.6 (0.09) 0.5 (0.0) 0.4 (0.0) 0.97 (0.08) 0.7 (0.05) CSMSC 0.65 (0.005) 0.7 (0.00) 0.40 (0.04) 0.95 (0.00) 0.0 (0.008) Parwse MLRSSC 0.48 (0.0) 0.89 (0.04) (0.09) 0.90 (0.08) 0.00 (0.0) Centrod MLRSSC 0.4 (0.00) 0.95 (0.0) 0.48 (0.05) 0.94 (0.05) 0.06 (0.07) Parwse KMLRSSC 0.49 (0.0) 0.45 (0.08) (0.06) 0.80 (0.08) 0. (0.07) Centrod KMLRSSC 0.46 (0.0) 0.40 (0.08) 0.44 (0.05) 0.7 (0.06) 0.07 (0.07) Best Sngle Vew LRSSC 0.70 (0.0) (0.0) (0.07) (0.00) (0.08) Feat Concat LRSSC (0.08) 0.67 (0.046) 0.78 (0.0) 0.75 (0.0) 0.66 (0.04) Co-Reg Parwse (0.057) 0.67 (0.068) 0.78 (0.047) 0.79 (0.06) (0.065) Co-reg Centrod (0.067) 0.75 (0.08) (0.050) 0.78 (0.0) 0.76 (0.075) UCI dgt RMSC 0.74 (0.070) 0.78 (0.080) (0.06) (0.040) 0.7 (0.079) CSMSC (0.045) 0.75 (0.069) 0.86 (0.05) 0.89 (0.09) (0.05) Parwse MLRSSC 0.80 (0.048) (0.070) (0.07) 0.85 (0.0) 0.80 (0.054) Centrod MLRSSC 0.85 (0.047) 0.89 (0.066) (0.07) (0.0) 0.87 (0.05) Parwse KMLRSSC 0.87 (0.06) (0.078) 0.86 (0.0) (0.07) (0.07) Centrod KMLRSSC (0.04) 0.80 (0.065) 0.86 (0.09) (0.00) 0.8 (0.048) Best Sngle Vew LRSSC (0.057) 0.55 (0.06) 0.64 (0.00) 0. (0.06) 0.80 (0.05) Feat Concat LRSSC (0.054) 0.54 (0.05) (0.09) 0.8 (0.09) 0.75 (0.057) Co-Reg Parwse (0.0) (0.0) 0.98 (0.0) 0.86 (0.0) 0. (0.0) Co-reg Centrod (0.00) (0.00) 0.86 (0.0) 0.96 (0.08) 0.06 (0.0) Prokaryotc RMSC (0.07) (0.08) 0.69 (0.0) 0.5 (0.04) 0.98 (0.044) CSMSC 0.46 (0.06) (0.04) 0.9 (0.06) 0.69 (0.0) 0.06 (0.0) Parwse MLRSSC 0.59 (0.06) 0.64 (0.00) (0.06) 0. (0.00) 0.45 (0.06) Centrod MLRSSC (0.08) 0.50 (0.04) (0.4) 0.0 (0.08) 0.58 (0.0) Parwse KMLRSSC 0.59 (0.056) 0.75 (0.068) (0.048) 0.47 (0.09) 0.98 (0.08) Centrod KMLRSSC 0.58 (0.070) 0.7 (0.079) 0.49 (0.06) 0.44 (0.046) 0.84 (0.00) Best Sngle Vew LRSSC 0.64 (0.000) (0.000) (0.000) 0.8 (0.000) 0.5 (0.000) Feat Concat LRSSC 0.68 (0.000) 0.68 (0.000) 0.68 (0.000) 0.8 (0.000) 0.64 (0.000) Co-Reg Parwse (0.000) 0.67 (0.000) (0.000) 0.60 (0.000) 0.95 (0.000) Co-reg Centrod (0.000) 0.60 (0.000) (0.000) 0.9 (0.000) 0.74 (0.000) Synthetc RMSC 0.75 (0.000) 0.75 (0.000) 0.75 (0.000) 0.8 (0.000) 0.40 (0.000) CSMSC 0.70 (0.000) 0.79 (0.000) 0.7 (0.000) 0.66 (0.000) (0.000) Parwse MLRSSC (0.000) (0.000) (0.000) 0.94 (0.000) 0.79 (0.000) Centrod MLRSSC (0.00) (0.00) (0.00) 0.96 (0.00) 0.80 (0.004) Parwse KMLRSSC 0.74 (0.000) 0.74 (0.000) 0.74 (0.000) 0.85 (0.000) (0.000) Centrod KMLRSSC 0.74 (0.000) 0.74 (0.000) (0.000) 0.88 (0.00) (0.000) Fgure shows how the NMI metrcs changes wth dfferent values of low-rank parameter β for both parwse and centrod-based MLRSSC, whle keepng λ (v) parameter fxed. On the -sources, 5

16 Reuters and UCI Dgt, MLRSSC algorthm outperforms the second best algorthm regardless of the choce of β. On the Prokaryotc dataset, parwse MLRSSC performs comparably to RMSC, but agan the algorthm s nsenstve to the β parameter. On the other hand, centrod-based MLRSSC lags behnd on ths dataset wth respect to NMI measure, but consstently mproves ts performance wth the hgher values of β. Next, we vary consensus parameter λ (v) and keep the low-rank parameter β and sparsty parameter β fxed. Fgure shows the performance of the MLRSSC wth respect to NMI measure for dfferent values of λ (v). Smlarly as when varyng β parameter, the MLRSSC performs consstently better than other algorthms regardless of the choce of λ (v). Agan, the only excepton s the centrod-based MLRSSC on the Prokaryotc dataset. These results prove that MLRSSC s pretty stable regardless of the choce of ts parameters β, β and λ (v), as long as the parameters are chosen n an approprate range. Fgure : The performance of the MLRSSC w.r.t. NMI measure when varyng low-rank parameter β and keepng consensus parameter λ (v) fxed. Sparsty parameter β s set to β. Blue lne shows the best performng algorthm besdes MLRSSC, among the algorthms lsted n Table. PMLRSSC stands for parwse MLRSSC and CMLRSSC for centrod-based MLRSSC. Fgure : The performance of the MLRSSC w.r.t. NMI measure when varyng consensus parameter λ (v) and keepng low-rank parameter β and sparsty parameter β fxed. Blue lne shows the best performng algorthm besdes MLRSSC, among the algorthms lsted n Table. 5.5 Computatonal Tme and Convergence In order to check how computatonal tme of the MLRSSC scales wth the ncrease of the number of data ponts, we perform experments on the UCI dgt dataset and compare MLRSSC wth other algorthms. Computatonal tme depends on the number of teratons and convergence condtons. We use the same number of teratons and error tolerance as when comparng performance of the algorthms. Fgure shows the computatonal tme averaged over 0 runs as a functon of the number 6

17 of data ponts. Fgure demonstrates that MLRSSC s more effcent than CSMSC. Compared to Co-Reg SC and RMSC, the better performance of MLRSSC comes wth a hgher computatonal cost. Fgure : Average computatonal tme n seconds as a functon of the number of data ponts, measured on the UCI Dgt dataset. For the Co-Reg and MLRSSC algorthm tmes for parwse regularzaton are shown, but they are smlar for the centrod regularzaton. The dfference between Co-Reg and RMSC can not be seen on ths scale, so these two algorthms are shown together. Fgure 4 demonstrates the behavor of convergence condtons for parwse MLRSSC. For ease of llustraton, the errors are normalzed and summed across vews. It can be seen that on all four real-world datasets, the algorthm converges wthn 0 teratons. Centrod MLRSSC exhbts very smlar behavor. Fgure 5 shows objectve functon value for both parwse and centrod MLRRSC wth the respect to number of teratons. Fgure 4: Sum of normalzed errors across vews for parwse MLRSSC. Behavor s very smlar for centrod MLRSSC. 7

18 Fgure 5: Objectve functon value wth the respect to number of teratons for parwse and centrod MLRSSC. 6 Concludng Remarks In ths paper we proposed mult-vew subspace clusterng algorthm, called Mult-vew Low-rank Sparse Subspace Clusterng (MLRSSC), that learns a jont subspace representaton across all vews. The man property of the algorthm s to jontly learn an affnty matrx constraned by sparsty and low-rank. We defned optmzaton problems and derved ADMM-based algorthms for parwse and centrod-based regularzaton schemes. In addton, we extended the proposed MLRSSC algorthm to nonlnear subspaces by solvng the related optmzaton problem n reproducng kernel Hlbert space. Expermental results on mult-vew datasets from varous domans showed that proposed algorthms outperforms state-of-the-art mult-vew subspace clusterng algorthms. Hgh computatonal complexty presents serous drawback of spectral clusterng algorthms. In the future work, we plan to explore how to mprove the effcency of the proposed approach to be applcable to large-scale mult-vew problems. Moreover, we may consder how to extend the MLRSSC algorthm to handle ncomplete data. Acknowledgements Ths work has been supported by the Croatan Scence Foundaton grant IP (Structured Decompostons of Emprcal Data for Computatonally-Asssted Dagnoses of Dsease) and by the Croatan Scence Foundaton grant HRZZ-96 (Descrptve Inducton). References [] M. Amn, N. Usuner, C. Goutte, Learnng from multple partally observed vews - an applcaton to multlngual text categorzaton, n: Advances n Neural Informaton Processng Systems, 009, pp [] A. Vedald, V. Gulshan, M. Varma, A. Zsserman, Multple kernels for object detecton, n: Proceedngs of the Internatonal Conference on Computer Vson (ICCV), 009. [] C. Xu, D. Tao, C. Xu, A survey on mult-vew learnng, arxv preprnt arxv: [4] I. Jollffe, Prncpal Component Analyss, Sprnger Verlag,

19 [5] P. Comon, Independent component analyss, a new concept?, Sgnal Processng 6 () (994) [6] D. D. Lee, H. S. Seung, Learnng the parts of objects by nonnegatve matrx factorzaton, Nature 40 (999) [7] T. Lu, D. Tao, On the performance of Manhattan nonnegatve matrx factorzaton, IEEE Transactons on Neural Networks and Learnng Systems 7 (9) (06) [8] J. Lu, C. Wang, J. Gao, J. Han, Mult-vew clusterng va jont nonnegatve matrx factorzaton, n: Proceedngs of the 0 SIAM Internatonal Conference on Data Mnng, SIAM, 0, pp [9] J. Macqueen, Some methods for classfcaton and analyss of multvarate observatons, n: 5-th Berkeley Symposum on Mathematcal Statstcs and Probablty, 967, pp [0] R. Vdal, Subspace clusterng, IEEE Sgnal Processng Magazne 8 () (0) [] J. Sh, J. Malk, Normalzed cuts and mage segmentaton, IEEE Transactons on Pattern Analyss and Machne Intellgence (8) (000) [] A. Y. Ng, M. I. Jordan, Y. Wess, On spectral clusterng: Analyss and an algorthm, n: Advances n Neural Informaton Processng Systems 4, 00, pp [] M. Flppone, F. Camastra, F. Masull, S. Rovetta, A survey of kernel and spectral methods for clusterng, Pattern Recognton 4 () (008) [4] U. von Luxburg, A tutoral on spectral clusterng, Statstcs and Computng 7 (4) (007) [5] E. Elhamfar, R. Vdal, Sparse subspace clusterng: Algorthm, theory, and applcatons, IEEE Transactons on Pattern Analyss and Machne Intellgence 5 () (0) [6] G. Lu, Y. Ln, Zhouchen annd Yu, Robust subspace segmentaton by low-rank representaton, n: 6th Internatonal Conference on Machne Learnng, 00, pp [7] G. Lu, Z. Ln, S. Yan, J. Sun, Y. Yu, Y. Ma, Robust recovery of subspace structures by low-rank representaton, IEEE Transactons Pattern Analyss and Machne Intellgence 5 () (0) [8] P. Favaro, R. Vdal, A. Ravchandran, A closed form soluton to robust subspace estmaton and clusterng, n: CVPR, 0, pp [9] R. Vdal, P. Favaro, Low rank subspace clusterng (LRSC), Pattern Recognton Letters 4 (04) [0] K. Tang, R. Lu, Z. Su, J. Zhang, Structure-constraned low-rank representaton, IEEE Transactons Neural Networks Learnng Systems 5 () (04) [] K. Tang, D. B. Dunsonc, Z. Sub, R. Lub, J. Zhanga, J. Dong, Subspace segmentaton by dense block and sparse representaton, Neural Networks 75 () (06) [] B. A. Olshausen, D. J. Feld, Emergence of smple-cell receptve feld propertes by learnng a sparse code for natural mages, Nature 8 (658) (996)

20 [] M. Aharon, M. Elad, A. Brucksten, K-SVD: An algorthm for desgnng overcomplete dctonares for sparse representaton, IEEE Transactons on Sgnal Processng 54 () (006) 4 4. [4] M. Elad, M. Aharon, Image denosng va sparse and redundant representatons over learned dctonares, IEEE Transactons on Image Processng 5 () (006) [5] J. Maral, M. Elad, G. Sapro, Sparse representaton for color mage restoraton, IEEE Transactons on Image Processng 7 () (008) [6] J. Maral, F. Bach, J. Ponce, G. Sapro, Onlne learnng for matrx factorzaton and sparse codng, Journal of Machne Learnng Research (00) [7] M. Flpovć, I. Koprva, A comparson of dctonary based approaches to npantng wth an emphass to ndependent component analyss learned dctonares, Inverse problems and magng 5 (4) (0) [8] W. Lu, Z. J. Zha, Y. Wang, K. Lu, D. Tao, p-laplacan regularzed sparse codng for human actvty recognton, IEEE Transactons on Industral Electroncs 6 (8) (06) [9] J. Yu, Y. Ru, D. Tao, Clck predcton for web mage rerankng usng multmodal sparse codng, IEEE Transactons on Image Processng (5) (04) [0] E. Elhamfar, R. Vdal, Clusterng dsjont subspaces va sparse representaton, n: 00 IEEE Internatonal Conference on Acoustcs, Speech and Sgnal Processng, 00, pp [] B. Nashatkon, R. Hartley, Graph connectvty n sparse subspace clusterng, n: CVPR, 0, pp [] Y.-X. Wang, H. Xu, C. Leng, Provable subspace clusterng: When LRR meets SSC, n: Advances n Neural Informaton Processng Systems 6, 0, pp [] Y. Lu, X. L, C. Lu, H. Lu, Structure-constraned low-rank and partal sparse representaton wth sample selecton for mage classfcaton, Pattern Recognton 59 (C) (06) 5. [4] C. Xu, D. Tao, C. Xu, Mult-vew ntact space learnng, IEEE Transactons on Pattern Analyss and Machne Intellgence 7 () (05) [5] X. Yang, W. Lu, D. Tao, J. Cheng, Canoncal correlaton analyss networks for two-vew mage recognton, Informaton Scences 85 (C) (07) 8 5. [6] Y. Luo, D. Tao, K. Ramamohanarao, C. Xu, Y. Wen, Tensor canoncal correlaton analyss for mult-vew dmenson reducton, IEEE Trans. on Knowl. and Data Eng. 7 () 4. [7] C. Hong, J. Yu, J. Wan, D. Tao, M. Wang, Multmodal deep autoencoder for human pose recovery, IEEE Transactons on Image Processng 4 () (05) [8] J. Yu, X. Yang, F. Gao, D. Tao, Deep multmodal dstance metrc learnng usng clck constrants for mage rankng, IEEE Transactons on Cybernetcs (99) (07). [9] J. Yu, Y. Ru, Y. Y. Tang, D. Tao, Hgh-order dstance-based multvew stochastc learnng n mage classfcaton, IEEE Transactons on Cybernetcs 44 () (04)

21 [40] J. Yu, Y. Ru, B. Chen, Explotng clck constrants and mult-vew features for mage rerankng, IEEE Transactons on Multmeda 6 () (04) [4] Y. Luo, T. Lu, D. Tao, C. Xu, Multvew matrx completon for multlabel mage classfcaton, IEEE Transactons on Image Processng 4 (8) (05) [4] A. Kumar, P. Ra, H. Daume, Co-regularzed mult-vew spectral clusterng, n: Advances n Neural Informaton Processng Systems 4, 0, pp [4] B. Cheng, G. Lu, J. Wang, Z. Huang, S. Yan, Mult-task low-rank affnty pursut for mage segmentaton, n: 0 Internatonal Conference on Computer Vson, 0, pp [44] R. Xa, Y. Pan, L. Du, J. Yn, Robust mult-vew spectral clusterng va low-rank and sparse decomposton, n: Proceedngs of the Twenty-Eghth AAAI Conference on Artfcal Intellgence, AAAI Press, 04, pp [45] C. Lu, S. Yan, Z. Ln, Convex sparse spectral clusterng: Sngle-vew to mult-vew, IEEE Transactons on Image Processng 5 (6) (06) [46] J. Yu, B. Zhang, Z. Kuang, D. Ln, J. Fan, prvacy: Image prvacy protecton by dentfyng senstve objects va deep mult-task learnng, IEEE Transactons on Informaton Forenscs and Securty (5) (07) [47] S. Boyd, N. Parkh, E. Chu, B. Peleato, J. Ecksten, Dstrbuted optmzaton and statstcal learnng va the alternatng drecton method of multplers, Foundatons and Trends n Machne Learnng () (0). [48] J.-F. Ca, E. J. Candes, Z. Shen, A sngular value thresholdng algorthm for matrx completon, SIAM Journal on Optmzaton 0 (4) (00) [49] D. L. Donoho, De-nosng by soft-thresholdng, IEEE Transactons on Informaton Theory 4 () (995) [50] I. Daubeches, M. Defrse, C. De Mol, An teratve thresholdng algorthm for lnear nverse problems wth a sparsty constrant, Communcatons on Pure and Appled Mathematcs 57 () (004) [5] M. Hong, Z.-Q. Luo, On the lnear convergence of the alternatng drecton method of multplers, Mathematcal Programmng 6 () (07) [5] D. D. Lews, Y. Yang, T. G. Rose, F. L, RCV: A new benchmark collecton for text categorzaton research, Journal of Machne Learnng Research 5 (004) [5] M. Brbć, M. Pškorec, V. Vduln, A. Krško, T. Šmuc, F. Supek, The landscape of mcrobal phenotypc trats and assocated genes, Nuclec Acds Research 44 () (06) [54] X. Y, Y. Xu, C. Zhang, Mult-vew EM Algorthm for Fnte Mxture Models, Sprnger Berln Hedelberg, 005, pp [55] C. D. Mannng, P. Raghavan, H. Schütze, Introducton to Informaton Retreval, Cambrdge Unversty Press, New York, NY, USA, 008.