Human Action Recognition Using Dynamic Time Warping Algorithm and Reproducing Kernel Hilbert Space for Matrix Manifold

Transcription

1 IJCTA, 10(07), 2017, pp Internatonal Scence Press Closed Loop Control of Soft Swtched Forward Converter Usng Intellgent Controller 79 Human Acton Recognton Usng Dynamc Tme Warpng Algorthm and Reproducng Kernel Hlbert Space for Matrx Manfold Wanhyun Cho* Sangkyoon Km** and Soonyoung Park** Abstract : In ths paper, we propose new human acton recognton rule usng the Dynamc Tme Warpng (DTW) algorthm and the Reproducng Kernel Hlbert Space (RKHS) for matrx manfold Frst, we have revewed the Dynamc Tme Warpng algorthm and the RKHS wth postve defnte kernel functon Second, we defne RKHS for a manfold of covarance feature matrx extracted from each frame mage of human acton vdeo, and then we calculate the dstance between two sequences of feature metrcs gven from two human acton vdeos usng a postve defnte kernel functon Thrd, we recognze human acton by applyng a classfer whch s the combnaton of DTW and k-nearest neghbours (k-nn) method to the dstance matrx gven by two sequence of feature matrces Fnally, we assess a performance of the proposed method usng KTH human acton data set Expermental results reveal that the proposed method performs very well on ths data set Keywords : Human acton classfcaton, Dynamc tme warpng, Reproducng kernel Hlbert space, K-nearest neghbours method, KTH dataset 1 INTRODUCTION The automatc recognton and localzaton of human actons n vdeo sequence has become an mportant research topc n computer vson, whch are very useful for varous applcatons Among those, several key applcatons are ncludng advanced human-computer nteracton (HCI), asssted lvng, physcal therapy, moves, 3DTV & anmatons, gesture-based nteractve game, sport moton analyss, smart envronments, survellance, and vdeo annotaton [1-3] Here, the human acton recognton task are largely conssts of three areas whch are mage representaton, human acton recognton, and software developments Frst, we have revewed several research papers for human behavour recognton usng the DTW method Blackburn and Rbero [4] demonstrated the applcablty of Isomap for dmensonalty reducton n human moton recognton And they show how an adapted dynamc tme warpng algorthm can be successfully used for matchng moton patterns of embedded manfolds Sempena et al [5] bult feature vector from jont orentaton along tme seres that nvarant to human body sze They appled the dynamc tme warpng to the resulted feature vector to recognze varous human actons Huu et al [6] presented a human acton recognton method usng dynamc tme warpng and votng algorthms on 3D human skeletal models Gong and Medon [7] proposed a Spatal-Temporal Manfold model to analyze nonlnear multvarate tme seres wth latent spatal structure and apply t to recognze actons n the jonttrajectores space Second, several research papers have been publshed for usng the reproducng kernel Hlbert space (RKHS) theory on the human acton recognton Danafar et al [8] ntroduced two novel approaches * Dept of Statstcs, Chonnam Natonal Unversty, South Korea E-mal : whcho@chonnamackr ** Dept of Electroncs Engneerng, Mokpo Natonal Unversty, South Korea E-mal : narcss76@mokpoackr, sypark@mokpoackr

2 80 Wanhyun Cho, Sangkyoon Km and Soonyoung Park outperformng state of the art algorthms They are an unsupervsed nonparametrc kernel based method explotng the Maxmum Mean Dscrepancy test statstc, and a supervsed method based on Support vector Machne wth a characterstc kernel specfcally talored to hstogram-based nformaton Gadon et al [9] modeled actons as tme seres of per-frame representatons and proposed a kernel specfcally talored for the purpose of acton recognton Harand et al [10] proposed to embed the Grassmann manfolds nto reproducng kernel Hlbert spaces and then tackle the problem of dscrmnant analyss on such manfolds In ths paper, we propose new human acton recognton rule, whch s a combnaton of the Dynamc Tme Warpng (DTW) algorthm and the Reproducng Kernel Hlbert Space (RKHS) for matrx manfold Frst, we extract the covarance feature matrx from each mage of vdeo sequence representng human acton Next, we embed these feature matrces nto RKHS by usng a postve defnte kernel functon, and then we compute the dstance on RKHS between two feature matrces Fnally, we recognze the human actons by usng DTW and K-nearest neghbours based on derved RKHS dstances for two sequence of mages 2 HUMAN ACTION RECOGNITION Here, we have revewed the Dynamc Tme Warpng technque and RKHS theory Next, we suggest new human acton recognton method usng theoretcal backgrounds of these methods A Dynamc Tme Warpng Frst, DTW algorthm has earned ts popularty by beng extremely effcent as the tme-seres smlarty measure whch mnmzes the effects of shftng and dstorton n tme by allowng elastc transformaton of tme seres n order to detect smlar shapes wth dfferent shapes [11] Gven two tme seres X = (x 1,, x N ), and Y = (y 1,, y M ) represented by the sequences of values, DTW yelds optmal soluton n the O(NM) tme whch could be mproved further through dfferent technques such as mult-scalng Algorthm starts by buldng the dstance matrx C N M representng all parwse dstances between X and Y Ths dstance matrx called be local cost matrx for the algnment of two sequences X and Y: C = {c j : c j =, (, j) = (1, 1),, (N, M)} N M Once the local cost matrx bult, the algorthm fnds the algnment path whch runs through the lowcost areas- valleys on the cost matrx Ths algnment path (or warpng path) defnes the correspondence of an element x X to y j Y followng the boundary condton whch assgns frst and last elements of X and Y to each other Formally speakng, the algnment path bult by DTW s a sequence of ponts p = (p 1,, p K ) wth p l = (p, p j ), (, j) [1 : N] [1 : M] for l [1 : K] whch must satsfy to the followng crtera: Boundary condton, Monotoncty condton, Step sze condton The cost functon assocated wth a warpng path computed wth respect to the local cost matrx wll be: c p (X, Y) = K å l = 1 cx (,y ) The warpng path whch has a mnmal cost assocated wth algnment called the optmal warpng path We wll denote ths path P * To fnd ths path P *, DTW employs the Dynamc Programmng based algorthm wth complexty only O(NM) B Reproducng Kernel Hlbert Space Second, we wll also brefly revew how to construct a RKHS form any postve defnte functon and ts propertes [12-13] We frst defne nner product nl ml

3 Human Acton Recognton Usng Dynamc Tme Warpng Algorthm and Reproducng Kernel Hlbert Space Defnton 1 Let H be a vector space over A functon <, > H : H H s sad to be an nner product on f t must satsfy the followng condtons: 1 Symmetry :, H = <, H,, H 2 B-lnearty : H = <, H, (, ) H,,, H, H, 3 Postve defnte:, H 0, and, H = 0 f and only f = 0 Now, we defne kernel functon, and we menton several propertes of postve defnte kernel related wth Hlbert space Defnton 2 Let X be a non-empty set A functon k : X X s a postve defnte kernel f k s symmetrc, and k satsfy the followng condton: åå cc jk( x,x j )³ 0, for all (c 1,, c m ) m and (x 1,, x m ) X m j Frst, the Gaussan kernel functon s postve defnte s equvalent wth a kernel functon s negatve defnte Theorem 3 Let be a nonempty set and k : X X be a kernel The kernel exp ( k(x, y)) s postve defnte for all > 0 f and only f the kernel k s negatve defnte Second, a negatve defnte kernel presents the squared norm of a Hlbert space under some condtons stated by the followng theorem Theorem 4 Let X be a nonempty set and k : X X be a negatve defnte kernel Then, there exsts a Hlbert space and a mappng :X H such that k(x, y) = (x) (y) h(x) h(y) where h : X s a functon whch s nonnegatve whenever k s Furthermore, f k(x, x = 0) for all x X, then h = 0 Thrd, the followng lemma says the relatonshp between a feature map and kernel functon Lemma 5 Let X be a nonempty set, H be an nner product space, and : X H be a functon Then, a functon k : X X defned by k(x, y) (x) (y) H s negatve defnte Next, we note that every nner product n Hlbert space s a postve defnte functon, and more generally Lemma 6 Let H be any Hlbert space, X be a non-empty set and a map : X H Then, k(x, y) (y) H s a postve defnte functon Now, we defne a reproducng kernel functon and a reproducng kernel Hlbert space Defnton 7 Let H be a Hlbert space of -valued functons defned on a non-empty set X A functon k : X X s called a reproducng kernel of H, and s a reproducng kernel Hlbert space (RKHS), f a functon k satsfes 1 Reproducng Property : (x) = ( ), k(, x) H, x X, H å 2 k span H : H = { (): () = k (,x ),x ÎC} Fnally, we can have a very fundamental results Theorem 8 (Moore-Aronszajn) Every postve defnte kernel k s assocated wth a unque RKHS H From the results of consderaton untl now, we can see the followng results In general, the degree of smlarty between two patterns n the pattern recognton problem s represented by the dstance between the two patterns But, the feature vectors that extracted from each of the patterns belong to the non-lnear manfold Therefore, t s not approprate to apply the computer vson algorthms whch are prmarly developed for data ponts lyng n Eucldean space drectly to ponts on the non-lnear manfold To overcome ths problem, one could thnk of embeddng a pont n the manfold nto a pont n the hgh dmensonal reproducng kernel Hlbert space (RKHS) usng kernel methods Here, accordng to Mercer s theorem, the postve defnte kernel can transform a pont n manfold nto a pont n RKHS 81

4 82 Wanhyun Cho, Sangkyoon Km and Soonyoung Park Formally, the pattern X n manfold X s frst mapped nto a functon space H usng feature map : X H, and then postve defnte kernel k : X X s defne on space X X Hence, to compare two patterns X and X n a manfold, we have to usng an nner product (x), (x) n a functon space But, to avod workng n the potentally hgh-dmensonal functon space, we can replace an nner product (x), (x) n a functon space as a postve defnte kernel functon k(x, x) defned on a pattern space X by mean of the kernel trck : (x), (x) = k(x, x) In concluson, n order to compare the degree of smlarty between two patterns n manfold X, we have to calculate the dstance x x 2 of the two patterns By the way, ths can be approxmated by the dstance (x) (x) 2 n the feature space Ths can also be computed usng the kernel trck as the followng equaton: (x) (x) 2 = (x) (x), (x) (x) = (x) (x) (x) (x) 2(x), (x) = k(x, x) k(x, x) 2k(x, x) Fnally, to compare two patterns, we only compute the postve defnte kernel functon C Human Acton Recognton We wll suggest new algorthm that can recognze the human acton usng a combnaton of DTW and RKHS In our acton recognton system, we frst express two sequences of feature matrces gven from two vdeos of human actons as two tme seres of feature matrces X and Y: X = [X 1,, X, X T1 ], and Y = [Y 1,, X j, X T2 ], where each feature matrces X and Y j are gven from (, j) th frame mages of two vdeo data X and Y Here, we wll use a feature matrx X or Y j gven from each frame mage as the covarance matrx descrptor C X or C Y defned as followng manner Defnton 9 Gven a regon of nterest R X or R Y of each frame mage I X or I Y, let X d or Y d, for = 1, N be feature vectors form R X or R Y Then, the covarance matrx descrptor C X or C Y s respectvely defned as C X = 1 ( x x)( x x), N 1 N 1 C X = ( x X)( x X) N 1 where x or y s the mean vector, and N s the number of pxels n regon R X or R Y Here, we note that the covarance matrx descrptor becomes to be a set of symmetrc postve defnte (SPD) matrces Sym d, and also they les on a connected Remannan manfold The DTW algorthm s appled to fnd the warpng path satsfyng the condtons mnmzng the warpng cost Here, the warpng path reveals the smlarty matchng between two tme seres nput data Moreover, the smlarty between two nput acton matrces s computed by usng the Eucldean dstance between the covarance matrx descrptor C X or C Y But, snce two matrxes belong to the Remannan manfold as mentoned on above, we have to embed a pont n the manfold nto a pont n the hgh dmensonal reproducng kernel Hlbert space (RKHS) usng kernel methods In ths case, we wll consder the metrc on the SPD matrces Sym d that can be used to defne postve defnte Gaussan kernel In partcular, we focus on the log-eucldean dstance whch s a true geodesc dstance on the Remannan manfold Sym d Under the log-eucldean framework, the geodesc dstance between X = C x and Y j = C yj for X, Yj Sym d, can be expressed as, =1 T

5 Human Acton Recognton Usng Dynamc Tme Warpng Algorthm and Reproducng Kernel Hlbert Space d g (X ) = log (X ) log (Y j ) F, where log ( ) denotes the usual matrx logarthm operators, and F denotes the Frobenus matrx norm nduced by the Frobenus matrx nner product F Here, the log-eucldean dstance defnes a true geodesc dstance that has proven an effectve dstance measure on Sym d Moreover, t yelds a postve defnte Gaussan kernel as mentonng n the followng Theorem Theorem 10 Let (Sym d, d F ) be a metrc space and defne k : Sym d Sym d be defne by k(x ) exp ( d g 2 (X )/2 2 ) Then, k s a postve defnte kernel for all > 0 f and only f there exsts an nner product space Z and a functon log : Sym d Z such that d g (X ) = log (X ) log (Y j ) F Therefore, n order to calculate the dstance matrx for two tme seres data requred by the DTW algorthm, we calculate the dstances between every correspondng two component matrces of two tme seres data by usng a symmetrc postve defnte kernel functon as the followng form d g (X ) = log (X ) log (Y j ) F = k(x, X ) k(y j ) 2k(X ) = 2(1 exp( Tr(log(X ) T log (Y j ))/2 2 )) And then, by applyng DTW algorthm wth an obtaned dstance matrx, we fnd an algnment warpng path wth mnmum cost, and computed the mnmum dstance for ts warpng path Fnally, n order to recognze or classfy a testng human acton, we use k-nn method Frst, we compute the mnmum warpng dstances between a testng acton sample and all tranng acton samples by usng DTW Second, we select k nearest neghbourhood samples that have the smaller dstances wth testng sample Thrd, we count the number k, = 1,, C of tranng sample that belong to -th class C among a selected k nearest neghbourhood tannng samples, and then we calculate the posteror classfcaton rates, p(c testng sample ) = k / k, = 1, C Therefore, f the class C k has the maxmum of posteror classfcaton rates, we classfy a testng sample nto the class C k 3 EXPERIMENTAL RESULTS A KTH Dataset In order to evaluate the performance of the proposed method, we used the KTH human acton dataset Ths dataset contans 25 people performng sx acton classes, namely: walkng, runnng, joggng, hand wavng, boxng, and hand clappng Each vdeo sequence contans one actor performng an acton In order to tran the proposed model, we used the KTH human acton dataset Ths dataset conssted of 384 vdeo sets n whch 16 people performed 6 acton classes a total of 4 tmes each Each frame mage of a gven vdeo conssted of a black and whte mage wth ( ) resoluton To test the performance of our model, we have also used another dataset of 216 vdeo sequences consstng of 9 people performng 6 dfferent human behavoral tasks a total of 4 tmes each B Procedure of Human Acton Recognton Fgure 1 shows the overall processng of the proposed human acton recognton system Ths system goes through three steps as follows In the frst step, we computed the covarance descrtors usng the followng feature vectors x gven from each mage of vdeo representng some human acton: 2 2 I x x = x, y I x, I y, Ix I y, I xx, I yy, arctan I y where x, y are pxel postons, and I x, I y are ntensty dervatves The covarance matrx for an mage path of arbtrary sze s a (8 8) symmetrc postve defnte matrx And we transform a vdeo sequence for 83

6 84 Wanhyun Cho, Sangkyoon Km and Soonyoung Park gven human acton nto a sequence of covarance matrces In the second step, we apply DTW algorthm to matchng two tme sequences of covarance matrces representng two human actons In ths case, we use a postve defnte kernel functon to compute the dstance between two covarance matrces In the thrd step, to recognze a gven human acton, we use k-nn algorthm for the mnmum warpng dstances obtaned by DTW algorthm That s, we select k nearest neghborhood samples that have the smaller dstances wth testng sample We count the number k, = 1, C of tranng sample that belong to the class C among selected k nearest neghborhood tannng samples And then we classfy a testng sample nto the class C k that has the largest number Vdeo Sequence Feature Extracton for vdeo Covarance Feature Matrx C, =1,2,,T (C 1, C 2,, C T) Transformaton SPD Inner Product K(C,C), X j R RKHS embeddng (C 1), (C 2),, (C n) Recognton Human Acton k-nearest Neghbors k p(c D) =, k = 1, C k* = max p(c D) DTW Matchng C = (C 1, C ), T1 D = (D 1, D ), T2 d g (C, D j) = log(c ) log(d ) F DTW(C, D) = argw mn Q q 1 w ( ( q), jq ( ) C Recognton Result Fgure 1: Procedure of Human Acton Recognton Table 1 shows the classfcaton rates for the sx human behavors obtaned by the proposed method represented n matrx form From the results n Table 1, we noted that sx human behavors can be manly dvded nto two categores wth smlar behavor The frst category of smlar actons ncludes boxng, hand-clappng, and hand-wavng The second category of smlar behavor ncludes joggng, runnng, and walkng Here, we can see, a lttle msclassfcaton happenng a lttle between the actons belongng to the same category Consequently, we note that the correct classfcaton rate of the proposed method totally appears to be 8465% on average Table 1 Classfcaton rate Classf caton rate Boxng Hand clappng Hand wavng Joggng Runnng Walkng Boxng Hand-clappng Hand-wavng Joggng Runnng Walkng

7 Human Acton Recognton Usng Dynamc Tme Warpng Algorthm and Reproducng Kernel Hlbert Space 4 CONCLUSION In ths paper, we propose new human acton recognton method usng the Dynamc Tme Warpng (DTW) algorthm and a symmetry postve defnte kernel functon defned on the Reproducng Kernel Hlbert Space (RKHS) Frst, we computed the covarance feature matrces from human acton vdeo Second, we calculated the dstance matrx between two sequences of covarance feature matrces usng a symmetry postve defnte kernel functon Thrd, we recognzed the testng acton vdeo by applyng the DTW and k-nn method for both tranng samples and testng sample To evaluate a performance of the proposed method, we conducted the experment usng KTH data Expermental results show that our method performs very well on publc vdeo datasets more than others Our future work wll extend the proposed method to other vdeo recognton problems such as 3D human acton recognton, gesture recognton, and survellances system 5 ACKNOWLEDGMENT Ths work was supported by Korea Research Foundaton ( ) 6 REFERENCES 1 M B Holte, C Tran, M M Treved, and T B Moeslund, Human Pose Estmaton and Actvty Recognton from Mult-Vew Vdeos: Comparatve Exploratons of Recent Developments, IEEE Journal of selected topcs n sgnal processng, vol 6(5), pp , X Xu, J Tang, X Zhang, X Lu, H Zhang, and Y Qu, Explorng Technques for Vson Based Human Actvty Recognton: Methods, Systems and Evaluaton, Sensors, vol 13, pp , G Cheng, Y Wan, A N Saudagar, K Namudur, and B P Buckles, Advances n Human Acton Recognton: A Survey, Proceedng of Computer Vson and Pattern Recognton, Jan 2015, pp J Blackburn and E Rbero, Human Acton recognton Usng Isomap and Dynamc Tme Warpng, LNCS 4814, pp , S Sempena, N U Mauldev, and P R Aryan, Human Acton Recognton Usng Dynamc Tme Warpng, 2011 Internatonal Conference on Electrcal Engneerng and Informatcs, July, 2011, Bandung, Indonesa 6 P C Huu, L Q Khanh, and L Thanh Ha, Human Acton Recognton Usng Dynamc Tme Warpng and Votng Algorthm, Journal of Scence: Comp Scence & Com Eng, vol 30(3), pp 22-30, D Gong and G Medon, Dynamc Manfold Warpng for Vew Invarant Acton Recognton, Proc Of the IEEE 13th Internatonal Conference on Computer Vson (ICCV 2011), Barcelona, Span, Nov S Danafar, A Gust, J Shmdhuber, Novel Kernel Based Recognzers of Human Actons, EURASIP Journal on Advances n Sgnal Processng Specal ssue on vdeo analyss for human behavor understandng, vol 2010, Feb Gadon, Z Harchaou, and C Schmd, A tme seres kernel for acton recognton, Proc of BMVC 2011 Brtsh Machne Vson Conference, Aug 2011, Dundee, Unted Kngdom 10 M T Harand, C Sanderson, S Shraz, and B C Lovell, Kernel Analyss on Grasmann Manfolds for Acton recognton, Preprnt submtted to Pattern Recognton Letters 11 P Senn, Dynamc Tme Warpng Algorthm Revew, Informaton and Computer Scence Department, Unversty of Hawa at Manoa, Honolulu, USA, December, S Jayasumana, R Hartley, M Salzmann, and M Harand, Kernel Methods on Remannan Manfolds wth Gaussan RBF Kernels, arxv: v2 [cscv] 17 Mar Y Wu, Y Ja, P L, J Zhang, and J Yuan, Manfold Kernel Sparse Representaton of Symmetrc Postve Defnte Matrces and Its Applcaton, IEEE Transactons on Image Processng, vol 24(11), Nov