Fast Robust Non-Negative Matrix Factorization for Large-Scale Human Action Data Clustering

Transcription

1 roceedngs of the Twenty-Ffth Internatonal Jont Conference on Artfcal Intellgence IJCAI-6) Fast Robust Non-Negatve Matrx Factorzaton for Large-Scale Human Acton Data Clusterng De Wang, Fepng Ne, Heng Huang Department of Computer Scence and Engneerng Unversty of Texas at Arlngton Abstract Human acton recognton s mportant n mprovng human lfe n varous aspects. However, the outlers and nose n data often bother the clusterng tasks. Therefore, there s a great need for the robust data clusterng technques. Nonnegatve matrx factorzaton ) and Nonnegatve Matrx Tr-Factorzaton NMTF) methods have been wdely researched these years and appled to many data clusterng applcatons. Wth the presence of outlers, most prevous /NMTF models fal to acheve the optmal clusterng performance. To address ths challenge, n ths paper, we propose three new and NMTF models whch are robust to outlers. Effcent algorthms are derved, whch converge much faster than prevous methods and as fast as K-means algorthm, and scalable to large-scale data sets. Expermental results on both synthetc and real world data sets show that our methods outperform other and NMTF methods n most cases, and n the meanwhle, take much less computatonal tme. Introducton Human acton recognton s very useful n mprovng human lfe n varous aspects, lke smart home applcatons, rehabltaton, human-computer nteracton, etc. To recognze human acton s helpful to know one s ntent, such that an ntellgent system could make correspondng reactons. However, n the real world lfe, the data are not as clean as we want, and there mght be many outlers and nose. For example, t s common that part of the human body s shelded behnd an obstacle, whch makes the human acton dffcult to recognze. Therefore, robust data mnng technques should be developed n order to acheve good performance. Non-negatve matrx factorzaton ) and nonnegatve matrx tr-factorzaton NMTF) are models whch am to factorze a matrx nto non-negatve matrces wth mnmum reconstructon error. It has been wdely researched To whom all correspondence should be addressed. Ths work was partally supported by US NSF-IIS 7965, NSF-IIS 32675, NSF-IIS 34452, NSF-DBI , NIH R AG4937. and used n varous knds of applcatons, lke document coclusterng [L and Dng, 26], computer vson [Lee and Seung, 999], socal network analyss [Huang et al., 23], bonformatcs [Wang et al., 22; 24b], knowledge transfer [Wang et al., 2; 25] and many others. Dfferent knds of and NMTF models are proposed these years. The standard factorze a non-negatve matrx nto two non-negatve matrces. [Dng et al., 25] dscussed the equvalence relatonshp between K-means, and spectral clusterng. Later, [Dng et al., 2] proposed two models: sem- and convex. In many stuatons, the data matrx may contan negatve elements. In such stuaton, t s not sutable to enforce the non-negatve constrants on model factors. Dfferent from standard, sem- relaxed the non-negatve constrant on models, allowed one factor to be mx-sgned. In convex, the author restrcts that the centrod to be a lnear combnaton of samples. Such model has better nterpretablty, however, the performance may not be as good as other models [L and Dng, 26]. [Dng et al., 26] proposed to add orthogonalty constrants n the cluster ndcator factor G. Ths property wll enforce the soluton of G to have a clear cluster structure, whch s desrable for usng for clusterng. It s ponted out n [Gu et al., 2] that the orthogonal constrant can prevent trval soluton of models. [Gu and Zhou, 29] combnes Laplacan graph based algorthms wth to enforce the smoothness on data manfolds. [Kong et al., 2; Gao et al., 25] proposed robust models whch are robust to outlers. Most prevous works can not acheve good clusterng results when there exst outlers n the data. resence of outlers s very common n real world applcatons. For example, n human acton recognton, t s common that part of the human body s shelded behnd an obstacle, whch makes the human acton dffcult to recognze. Therefore, robust data mnng technques should be developed n order to acheve good performance. In vew of the above consderatons, we propose three and NMTF models whch converge as fast as K-means, and are robust to outlers. Notatons: Gven a matrx 2 R n m, ts -th row and j-th column are denoted by.,.j, respectvely. The `r,p -norm of a matrx s defned as: 24

2 kk r,p = n = m j= x j r ) p r ) p. Accordng to ths defnton, we can get the followng norms: v n u m n kk 2, = t x 2 j = x ) 2 = kk = j= n = j= = m x j 2) v u n m kk F = kk = t x 2 j 3) = j= 2 New Fast Algorthms for Robust and NMTF Models 2. Motvatons models often have the followng form of objectve functons, subjectng to dfferent knds of constrants. mn F,G FGT 2 s.t. F,G 4) F where 2 R d n + s a data matrx wth d features and n samples. Such models have close relaton to K-means clusterng as ponted out n [Dng et al., 25]. F 2 R d c + can be vewed as cluster centrods, and G 2 R n c + can be vewed as clusterng ndcator matrx. Many prevous work [Dng et al., 2; 26; Lee and Seung, 2; Kong et al., 2] use a soft ndcator matrx G,.e., elements n G are contnuous values. Algorthms for those models are based on matrx multplcatve updatng rules. However, these algorthms converge very slowly, often need hundreds or even thousands of teratons before convergence. Consderng the computatonal cost of large matrx multplcaton, such algorthms usually take a long tme to converge. In addton, an addtonal postprocessng step need to be performed to get the fnal clusterng labels usng the soft ndcator matrx G. On the other hand, outlers are very common n real world data problems. For example, n mage recognton problems, there maybe dfferent extent of nose due to dfferent condtons of llumnatons, vewponts, wearngs, etc. Also, human measurement and recordng errors may also ncur many outlers. revous models usng Frobenus norm as loss measurement can not get good clusterng results f there exst outlers/nose n the data. Ths s because the loss term wll be domnated by outlers, thus the loss of other normal data ponts wll be dsregarded. We wll show ths usng a synthetc data set n the expermental secton. Therefore, developng robust models to handle data wth outlers s mportant. To address the above challenges, n ths paper, we propose three new fast robust and NMTF models. 2.2 New Fast Robust and NMTF Models To make the /NMTF models robust to outlers, nstead of usng Frobenus norm, we propose to use the `2, -norm and `-norm as loss measurements. The new robust and fast models am to mnmze the followng objectve functons: mn F,G2Ind FGT 5) mn F,G2Ind FGT 6) 2, mn F 2Ind,G2Ind,S FSGT 7) where G 2 Ind or F 2 Ind ndcates that G and F are ndcator matrces,.e. g j =f x belongs to class j, and g j =otherwse. There s only one element can be nonzero n each row of bnary ndcator matrx. Note that n ths way, the constrant G T G = I n orthogonal [Dng et al., 26] s automatcally satsfed. In addton, clusterng labels are drectly obtaned wthout the need for further postprocessng as n prevous works. Whle F ndcates that F s a non-negatve matrx wth all elements greater or equal to zero. In the tr-factorzaton model 7), snce both F and G are restrcted to be bnary ndcators, S s ntroduced to absorb the magntude n the orgnal data matrx. Snce our methods are robust and fast /NMTF models, we call the three models as L,, and, respectvely. `2, -norm and `-norm are often used for enforcng sparsty when appled to regularze parameter matrx [Ne et al., 2; Wang et al., 24a; 23], and achevng robustness to outlers when appled to loss functon [Kong et al., 2]. Laplacan Nose Interpretaton for L: We show the probablstc motvaton of our model from Laplacan nose dstrbuton. Suppose x s an observed data pont contamnated by an addtonal nose : x = + 8) where s the unobservable true data, n clusterng t s the clusterng centrod,.e. = FG T., G. 2 Ind. s the nose. Suppose the nose s drawn from Laplacan dstrbuton wth zero mean, then we have: px )= 2b exp kx k ) 9) b where b s the scale parameter of Laplacan dstrbuton. Suppose we have an observed data set =[x,x 2,...,x N ], the maxmum lkelhood estmate MLE) of should be: max log N N px ) ) max = b kx = k ) N N mn kx k kx = F,G.2Ind = FG. k N k FGk ) F,G2Ind = Therefore, the MLE under Laplacan nose assumpton s equvalent to the L model. In the algorthm secton, we wll show that the soluton of cluster centrods F s exactly fndng the sample medans, whch concdes wth the MLE of the locaton parameter of Laplacan dstrbuton. Smlar to the Laplacan nose nterpretaton of L, t s also not dffcult to see that: the 25

3 model can also be nterpreted from a Laplacan dstrbuted nose, except s replaced by FSG. nstead of FG. ; the model can be nterpreted from a Gaussan dstrbuted nose. For space reason, we do not show the detal dervaton. The constrants n the objectve functons nvolves combnatoral search over the soluton space of F and G, whch s very hard to optmze. revous works always solve the relaxed verson to make G have contnuous values. In the followng secton, effcent algorthms are proposed for solvng the objectve functons wth bnary ndcator constrants. 3 Optmzaton Algorthms We frst ntroduce a lemma whch wll be used n the followng optmzaton algorthms. Lemma. Consderng the followng objectve functon: mn z a ) z The optmal soluton of z s the medan value of a. Ths lemma can be easly obtaned by settng the dervatve of Eq. to zero: sgnz a )= 2) where sgnx) =f x>, sgnx) = f x<, and sgnx) =f x =. Ths condton can be satsfed f and only f z takes the medan value of a. 3. Effcent Algorthm for roblem 5) Gven an ntal guess of F and G, we teratvely update the model parameters. When G s fxed, problem 5) becomes: mn F FGT 3) Snce `-norm can be decoupled through rows and columns, the above problem can be reduced to: F. F k F k G Ṭ k k j F k G jk = The above problem can be decoupled as: for 8, k, solvng: mn j F k 4) F k G jk = Accordng to Lemma, the optmal soluton of F K can be effcently obtaned by fndng the medan values of samples belong to the k-th cluster. When F s fxed, G can be effcently optmzed by assgnng label to the cluster wth nearest centrod,.e.: j = arg mn k. F.k k g j = k 5) otherwse 3.2 Effcent Algorthm for roblem 6) When G s fxed, usng reweghted method, problem 6) can be reduced to: mn F Tr FGT )D FG T ) T ) 6) where D 2 R n n s a dagonal matrx whose dagonal elements are: D = 2. FG T 7). Settng the dervatve w.r.t. F to zero, we get: F = DGG T DG) 8) When F s fxed, G can be obtaned by assgnng labels to the cluster wth the nearest centrod: j = arg mn k. F.k k 2 g j = k 9) otherwse 3.3 Effcent Algorthm for roblem 7) When F and G are fxed, problem Eq. 7) becomes: S FSG T S S S kl F.k G Ṭ l k,l k,l F k =,G jl = S kl j Ths problem can be decoupled as: 8k, l, solvng: S kl j 2) S kl F k =,G jl = Accordng to Lemma, S can be effcently optmzed by fndng the medan values. When S and F are fxed, G can be obtaned by: j = arg mn k. FS.k k g j = k 2) otherwse When S and G are fxed, F can be obtaned by: j = arg mn. S k. G T f j = k otherwse 22) Therefore, all the three models can be effcently solved by teratvely update F and G. Unlke prevous works whch need to ntalze G usng the clusterng results of K-means, our algorthms works good wth random ntalzaton of G. In practce, we can repeat the algorthm many tmes usng dfferent random ntalzatons, and then take the result wth the mnmum objectve value. Note that our algorthms use bnary ndcator matrx nstead of soft ndcator matrx. Therefore, the clusterng labels can be obtaned drectly by assgnng labels to the cluster wth nearest centrod. Therefore, our algorthms avod the ntensve computaton of matrx multplcaton. More mportantly, prevous /NMTF works converges slowly, often needs hundreds or even thousands teratons before convergence. Our algorthms converge very fast, usually n just tens of teratons, and take much less computatonal tme. 26

4 3.4 Convergence Analyss In ths secton, we prove the convergence of our algorthms. To begn wth, followng are some smple defntons that wll be used. Jf,f )= x f s f) 23) where x s the -th data, f s the cluster centrod of the prevous teraton, f s the new cluster centrod, s f) denotes the clusterng assgnment of the -th data usng the cluster centrod of prevous step. Obvously, when f = f = f t, Ef t ) = Jf t,f t ) s exactly the objectve value of L n Eq. 5) n the t-th teraton. Accordng to Lemma : the optmum soluton of f s the medan value of samples, and we have Jf,f ) apple Jf,f). In addton, snce by defnton s f ) s the best cluster assgnment usng cluster centrod f, we have: Ef ) Jf,f ) = Jf,f ) Jf,f ) = x fs f ) x fs f) apple 24) So we have: Ef ) Ef) = Ef ) Jf,f )+Jf,f ) Jf,f) apple Jf,f ) Jf,f) apple 25) Therefore, our algorthm for solvng problem 5) monotoncally decreases the objectve value at each teraton untl t reaches the optmum soluton where f = f. In addton, the KKT condton s satsfed. So the algorthm converges to a local mnmum. The convergence proof of the other two algorthms follows a smlar procedure. For space reasons, we omt the detaled proof. 4 Expermental Results 4. Experment Settngs In order to valdate the effectveness of the proposed and NMTF methods, we compare our methods wth some related methods as followng: ) Standard ) [Lee and Seung, 2]: solves the objectve functon n Eq. 4). 2) Orthogonal NMTF Orth) [Dng et al., 26]: factorzes a matrx nto three non-negatve components, and each column of the soft ndcator matrces F and G) are requred to be orthogonal. 3) Sem [Dng et al., 2]: allows the bass matrx F n standard to be mx-sgned 4) Convex Conv) [Dng et al., 2]: restrcts the bass matrx F nto a lnear combnaton of orgnal data ponts. 5) Robust R) [Kong et al., 2]: replaces the loss measurement n standard from Frobenus norm to `2, -norm, whch makes the model robust to outlers. The dfference between our methods and R s that: our methods Table : Average dstance from the centrods for normal data blue and red ponts n fgure a)), outlers, and all data. normal data outlers all data Our methods use a hard ndcator, whch s more dffcult to solve, but has clearer clusterng nterpretaton snce they reduce to drectly assgn cluster labels to samples at each updates of ndcator matrx. Ths approach s better for clusterng tasks. In addton, the algorthm s also much faster as we wll show emprcally n the experment secton. All the above models are solved usng a multplcatve updatng rules, whch converges slowly and nvolves ntensve matrx computatons. K-means serves as a baselne n the clusterng performance comparson. The above comparson methods need to be ntalzed wth the clusterng results of K-means, snce the objectve functons are non-convex, and the fnal clusterng results are senstve to ntalzatons. A lttle perturbaton s needed for preventng these models from stckng at the K-means solutons. As suggested n prevous researches [Kong et al., 2; Dng et al., 26], the ntalzaton of ndcator matrx s set as G =G k +.2, where G k s the result of the K-means algorthm. The soluton of these comparson methods s then get by teratvely update F and G usng the multplcatve rule n Table n [L and Dng, 26]. The teraton number s set as 5. As for our methods, the ntalzaton s set as the followng: for and L, rather than usng the result of K-means, we just randomly ntalze the cluster ndcator matrx G n case of the model stuck at the soluton of K-means. We can see n the experments that our objectve converges good on a local mnmum even wth random ntalzaton. For, the ntalzaton of G and F s usng the clusterng results of the data dmenson and feature dmenson, smlar to the strategy used n tradtonal methods. 4.2 Experments on Synthetc Data We compare our methods wth standard on a synthetc data set wth outlers. As shown n Fgure a), the data set contans two normal clusters blue ponts and red ponts) drawn from two gaussan dstrbutons wth mean -6, ) and 6,), respectvely. Each cluster contans data ponts. Three outlers black ponts) were added far away from these two clusters. Fgure b) shows the clusterng results usng standard algorthm. Because standard s not robust to outlers, the orgnal blue/red ponts n one cluster are splt nto two clusters. Other comparson methods follow the same pattern as standard,.e. splt the orgnal cluster nto two clusters. Fgure c) shows the clusterng results usng the proposed three methods. Our methods are robust to outlers, therefore, the correct clusterng structure s recovered. Table reports the average dstance of data ponts from the correspondng cluster centrods. We can see that the dstance of standard algorthm for outlers s a lttle smaller than 27

5 a) Orgnal data wth outlers 5 b) Clusterng results wth standard 5 nn c) Clusterng results wth our three methods Fgure : Clusterng performance on synthetc data. Blue ponts and red ponts are normal data drawn from two gaussan dstrbutons. Black ponts are outlers. Magenta ponts are computed cluster centrods. Accuracy _L OrthNMTF Sem Conv R k means NMI _L OrthNMTF Sem Conv R k means purty _L OrthNMTF Sem Conv R k means Objectve value 2.5 x _L DIGIT HumanEvaYouTube KTH UCF a) Accuracy.2. DIGIT HumanEvaYouTube KTH UCF b) NMI.2. DIGIT HumanEvaYouTube KTH UCF c) purty Number of teratons d) DIGIT Fgure 2: a)-c): Clusterng performance comparson d): objectve value versus number of teratons. Table 2: Computatonal tme n seconds) comparson. Averaged over repettons. L OrthNMTF Sem Conv R K-means DIGIT HumanEva YouTube KTH UCF our methods, snce t pays more attenton to outlers. The dstances of our methods for normal data and for all data are much more smaller than standard. 4.3 Data Set Descrptons In ths secton, we wll present emprcal results to evaluate the proposed and NMTF approaches. Fve real world data sets are used to evaluate the effectveness of our method. Dgt data set s a publc data set hosted n UCI Machne Learnng Repostory. Ths data set conssts of handwrtten dgts from to 9, and each dgt s a class. HumanEva data set 2 contans samples from two subjects wth 5 samples per subject. There are fve types of motons: boxng, walkng, throw-catch, joggng and gesturng. YouTube data set 3 contans human actvtes: soccer lujg/youtube Acton dataset.html jugglng, swngng, tenns swngng, to name a few. Fgure 3 show some sample mages wth boundng acton labels n YouTube data set. UCF data set 4 s an extenson to the YouTube data sets. It contans 5 actons consstng of realstc vdeos n YouTube. Ths data set s very hard to recognze due to large varatons n camera poston, pose, llumnaton condtons, vewpont, and cluttered background. KTH data set 5 contans sx type of human actons: walkng, joggng, runnng, boxng, hand wavng and hand clappng. These actons are performed by 25 subjects n four dfferent scenaros: outdoor, outdoor wth scale varatons, outdoor wth dfferent clothes, and ndoors. All samples are taken n homogeneous background, so that t s more close to natural envronments. Ths settng add nose n the sample, thus, t s more dffcult to recognze the actons

6 Fgure 3: Sample mages from the webste of the YouTube data set. 4.4 Clusterng erformance and Convergence Speed Comparson We use accuracy, NMI normalzed mutual nformaton), and urty as measurements of the clusterng performance. Snce all the methods are senstve to ntalzatons, we repeat each algorthm tmes, and the clusterng results wth the mnmum objectve value s recorded n Fgure 2. We can see that: L and acheve the best performance on most data sets n terms of both accuracy, NMI and purty. The performance of OrthNMTF and R s also pretty good on some data sets. Fgure 2 d) shows the convergence of the proposed algorthms. For space lmtaton, only one of the fve data sets are shown n the paper. We can see that all the three algorthms converge fast, usually n about 5 teratons. Table 2 shows the comparson of computatonal tme. The algorthms are run on a Dell desktop wth double 7 Cores and 6GB memory. For and L, we just use random ntalzaton n the experments. For all of the other methods, K-means result s used as ntalzaton snce t s suggested by prevous research. The tme consumpton of K-means ntalzaton s not consdered for all of these methods. We can see that the L algorthm converges very fast, whch s often comparable or even faster than K- means. Thus, our algorthm s scalable to large data. The computatonal tme of and s also much less than other compared and NMTF methods. 5 Conclusons We proposed three new and NMTF models whch are robust to outlers to mprove the human acton clusterng tasks. Effcent algorthms are derved, whch converge as fast as the standard K-means algorthm, and thus are scalable to large-scale data sets. Expermental results on both synthetc and real world data sets show that our methods outperform other exstng and NMTF methods n most cases, and n the meanwhle, take much less computatonal tme. References [Dng et al., 25] C. Dng,. He, and H.D. Smon. On the equvalence of nonnegatve matrx factorzaton and spectral clusterng. In SDM, pages 66 6, 25. [Dng et al., 26] Chrs Dng, Tao L, We eng, and Haesun ark. Orthogonal nonnegatve matrx t-factorzatons for clusterng. In roceedngs of the 2th ACM SIGKDD nternatonal conference on Knowledge dscovery and data mnng, pages ACM, 26. [Dng et al., 2] Chrs Dng, Tao L, and Mchael I Jordan. Convex and sem-nonnegatve matrx factorzatons. attern Analyss and Machne Intellgence, IEEE Transactons on, 32):45 55, 2. [Gao et al., 25] Hongchang Gao, Fepng Ne, Wedong Ca, and Heng Huang. Robust capped norm nonnegatve matrx factorzaton. 24th ACM Internatonal Conference on Informaton and Knowledge Management CIKM 25), pages 87 88, 25. [Gu and Zhou, 29] Quanquan Gu and Je Zhou. Neghborhood preservng nonnegatve matrx factorzaton. In roceedngs of the Brtsh machne vson conference, 29. [Gu et al., 2] Quanquan Gu, Chrs Dng, and Jawe Han. On trval soluton and scale transfer problems n graph regularzed nmf. In roceedngs of the nternatonal jont conference on artfcal ntellgence, 2. [Huang et al., 23] Jn Huang, Fepng Ne, Heng Huang, Ycheng Tu, and Yu Le. Socal trust predcton usng heterogeneous networks. ACM Transactons on Knowledge Dscovery from Data TKDD), 74):7: 7:2,

7 [Kong et al., 2] Deguang Kong, Chrs Dng, and Heng Huang. Robust nonnegatve matrx factorzaton usng l2,-norm. In ACM Conference on Informaton and Knowledge Management CIKM 2), 2. [Lee and Seung, 999] Danel D Lee and H Sebastan Seung. Learnng the parts of objects by non-negatve matrx factorzaton. Nature, 46755):788 79, 999. [Lee and Seung, 2] D.D. Lee and H.S. Seung. Algorthms for nonnegatve matrx factorzaton. In NIS, pages , 2. [L and Dng, 26] Tao L and Chrs Dng. The relatonshps among varous nonnegatve matrx factorzaton methods for clusterng. In Data Mnng, 26. ICDM 6. Sxth Internatonal Conference on, pages IEEE, 26. [Ne et al., 2] F. Ne, H. Huang,. Ca, and C. Dng. Effcent and robust feature selecton va jont l2, -norms mnmzaton. Advances n Neural Informaton rocessng Systems, 23:83 82, 2. [Wang et al., 2] Hua Wang, Heng Huang, and Chrs Dng. Cross-language web page classfcaton va jont nonnegatve matrx tr-factorzaton based dyadc knowledge transfer. In Annual ACM SIGIR Conference 2, pages , 2. [Wang et al., 22] Hua Wang, Heng Huang, Chrs Dng, and Fepng Ne. redctng proten-proten nteractons from multmodal bologcal data sources va nonnegatve matrx tr-factorzaton. The 6th Internatonal Conference on Research n Computatonal Molecular Bology RECOMB), pages , 22. [Wang et al., 23] De Wang, Fepng Ne, Heng Huang, Jngwen Yan, Shannon L Rsacher, Andrew J Saykn, and L Shen. Structural bran network constraned neuromagng marker dentfcaton for predctng cogntve functons. In Informaton rocessng n Medcal Imagng, pages Sprnger Berln Hedelberg, 23. [Wang et al., 24a] De Wang, Fepng Ne, and Heng Huang. Unsupervsed feature selecton va unfed trace rato formulaton and k-means clusterng track). In Machne Learnng and Knowledge Dscovery n Databases, 24. [Wang et al., 24b] Hua Wang, Heng Huang, and Chrs Dng. Correlated proten functon predcton va maxmzaton of data-knowledge consstency. The 8th Internatonal Conference on Research n Computatonal Molecular Bology RECOMB 24), pages 3 325, 24. [Wang et al., 25] Hua Wang, Fepng Ne, and Heng Huang. Large-scale cross-language web page classfcaton va dual knowledge transfer usng fast nonnegatve matrx tr-factorzaton. ACM Transactons on Knowledge Dscovery from Data TKDD), ), 25. 2