Action Recognition Using Completed Local Binary Patterns and Multiple-class Boosting Classifier

Transcription

1 Acton Recognton Usng ompleted Local Bnary Patterns and Multple-class Boostng lassfer Yun Yang, Baochang Zhang, Lnln Yang School of Automaton Scence and Electrcal Engneerng Behang Unversty Beng, hna hen hen Department of Electrcal Engneerng he Unversty of exas at Dallas Rchardson, X, USA Wankou Yang School of Automaton Southeast Unversty Nanng, hna Abstract hs paper, for the frst tme, ntroduces a multple-class boostng scheme (MBS) to combne depth moton maps (DMMs) and completed local bnary patterns (LBP) for acton recognton. DMMs derve from proectng depth frames onto three orthogonal artesan planes (front, sde and top) and characterze the moton energy of an acton, on whch the LBP features are further extracted. And then a new mult-class boostng method s used and leads to an effectve decson-level classfer. Extensve experments on the MSRActon3D and MSRGesture3D datasets ndcate that the proposed MBS method acheves new state-of-the-art results.. Introducton Human acton recognton has been an actve research topc n computer vson. It has a wde range of applcatons ncludng human computer nteracton, moton sensng based gamng, ntellgent survellance and asssted lvng. Early research focuses on vdeo sequences captured by RGB vdeo cameras [-4]. In [], bnary moton-energy mages (MEI) and moton-hstory mages (MHI) are used to represent where moton has occurred and characterze human actons. In [2], a low computatonal-cost volumetrc acton representaton from dfferent vew angles s utlzed to obtan hgh recognton rates. In [3], the noton of spatal nterest ponts s extended to the spato-temporal doman based on the dea of the Harrs nterest pont operator. he results show robustness to occluson and nose. In [4], moton descrptor based on optcal flow measurements n a spato-temporal volume s ntroduced to deal wth low resoluton mages. Despte researchers have made great progress n the past decades, robust acton recognton n varous real world condtons s stll a challengng task. Wth the development of RGBD cameras, especally Mcrosoft Knect, more recent research works on human acton recognton have been carred out usng depth mages [7-9]. ompared wth the conventonal RGB cameras, depth cameras provde depth nformaton whch s robust to changes n lghtng condtons. In [5], a bag of 3D ponts correspondng to the nodes n an acton graph are used to recognze human acton from depth sequences. An actonlet ensemble model s proposed n [6] and the developed local occupancy patterns features are shown to be robust to nose and nvarant to translatonal and temporal msalgnments. In [7], Hstograms of Orented Gradents (HOG) computed from Depth Moton Maps (DMMs) are used to capture body shape and moton nformaton from depth mages. In [7], hen et al. combnes Local bnary pattern (LBP) and the extreme learnng machne (ELM), whch acheves the best performance on ther own datasets. From the lterature revew, we know that the dscrmnatve feature extracton and classfer desgn play key roles n the performance mprovement. o enhance the robustness of the exstng features, n ths paper, DMMs and completed local bnary patterns (LBP) [0] are frst used to extract dscrmnatve features from depth mages. And then a mult-class boostng scheme s proposed to combne the ELM base classfers. More specfcally, two man contrbutons of our work are summarzed as follows:. he combnaton of LBP features and DMMs s, for the frst tme, employed to represent human acton. LBP s a completed model based on LBP and acheves sgnfcant mprovement for rotaton nvarant texture classfcaton. Dfferent from prevous work focusng on feature-level combnaton, we pay attenton to ts applcaton to acton recognton based on the decson-level combnaton. 2. We propose a mult-class boostng algorthm to mprove the performances of the ELM classfer []. hree knds of ELM base classfers on the LBP features are obtaned, whch s further combned together to mprove the classfcaton performance. Inspred from the mult-class SVM [4] and AdaBoost methods [2], we propose a multple-class boostng scheme (MBS) by addng an nequalty constrant derved from [4] nto the boostng obectve. he extensve expermental results valdate the proposed method. he rest of the paper s organzed as follows. Secton 2

2 descrbes the detals of DMMs-based LBP features. Secton 3 ntroduces the ELM classfer and the proposed mult-class boostng method. Expermental results are gven n Secton 4. Some concludng remarks are n Secton Feature extracton from depth mages 2.. Depth moton maps In a depth mage, the pxel values ndcate the dstances between the surface of an obect and a depth camera, therefore provdng 3D structure nformaton of a scene. Yang et al. [7] proposes to proect each depth frame n a depth sequence onto three orthogonal artesan planes to make use of the 3D structure and shape nformaton. hen et al. [8] modfes the procedure of generatng DMMs to reduce the computatonal complexty. More specfcally, each 3D depth frame generates three 2D proected maps accordng to front (f), sde (s), and top (t) vews,.e. map f, map s, and map t. hen the depth moton maps are calculated accordng to N + DMM{ f, s, t} = map{ f, s, t} map{ f, s, t}, () = where represents frame ndex and N s the total number of frames n a depth sequence. In ths paper, we use the method n [8] to generate DMMs due to ts computatonal effcency DMMs-based LBP features LBP [9] s an effectve texture descrptor has been used n varous mage processng and computer vson applcatons [20]. Gven a center pxel t c, ts neghborng pxels are equally spaced on a crcle of radus r ( r > 0 ) wth the center at t c. If the coordnates of t c are (0,0) and m neghbors {} m t = 0 are consdered, the coordnates of t are ( rsn(2 π m), rcos(2 π m)). he LBP s computed by thresholdng the neghbors {} m t = 0 wth the center pxel t c to generate an m -bt bnary number. he resultng LBP for t c can be expressed n decmal form as follows: m m mr, c = c = = 0 = 0 (2) LBP ( t ) s( t t )2 s( d )2, where d = ( t tc) s the dfference between the center pxel and each neghbor, sd ( ) = f d 0 and sd ( ) = 0 f d < 0. he LBP only uses the sgn nformaton of d whle gnorng the magntude nformaton. However, the sgn and magntude are complementary and they can be used to exactly reconstruct the dfference d. In the LBP scheme [0], the mage local dfferences are decomposed nto two complementary components: the sgns and magntudes (absolute values of d,.e., d ). Fgure shows an example of the sgn and magntude components of the LBP extracted from a sample block. Note that 0 s coded as - n LBP [see Fgure (c)]. wo operators, namely LBP-Sgn (LBP_S) and LBP-Magntude (LBP_M), are used to code these two components. LBP_S s equvalent to the tradtonal LBP operator. he LBP_M operator s defned as follows: m, u c LBP _ M mr, = p( d, c)2, p( u, c) = = 0 0, u < c (3) where c s a threshold that s set to the mean value of d from the whole mage. he LBP-enter part whch codes the values of the center pxels also has dscrmnant nformaton. It s coded as: LBP _ mr, = p( tc, c), (4) where p s defned n (3) and the threshold c s set as the average gray level of the whole mage. Fgure : (a) 3 3 sample block; (b) the local dfferences; (c) the sgn component of LBP; and (d) the magntude component of LBP. In our feature extracton method, DMMs are frst generated from a depth sequence, and then the DMMs are dvded nto several overlapped blocks. Each component (LBP_S, LBP_M, and LBP_) of the LBP operator s appled to each block from the three DMMs and hstograms of all blocks are concatenated to form a sngle composte feature vector. herefore, three LBP hstogram feature vectors denoted by DMM-LBP_S, DMM-LBP_M, and DMM-LBP_ are obtaned. 3. Decson-level classfer fuson based on a multple-class boostng scheme 3.. Extreme learnng machne ELM [] s an effcent learnng algorthm for sngle-hdden-layer feed-forward neural networks (SLFNs). Let y = [ y,..., y,..., ] k y be the class to whch a sample belongs, where yk {, } ( k ) and s the number of classes. Gven n tranng samples { x, } n y =, M where x and y, a sngle hdden layer neural network havng L hdden nodes can be expressed as 2

3 L β h( w x + e) = y, =,..., n, (5) = where h() s a nonlnear actvaton functon (e.g., Sgmod functon), β denotes the weght vector connectng M the th hdden node to the output nodes, w denotes the weght vector connectng the th hdden node to the nput nodes, and e s the bas of the th hdden node. he above n equatons can be wrtten compactly as: Hβ = Y, (6) L n where β = [ β ;...; βl ], Y = [ y ;...; yn ], and H s the hdden layer output matrx. A least-squares soluton ˆβ of (6) s found to be ˆ β = HY, (7) where H s the Moore-Penrose generalzed nverse of matrx H. he output functon of the ELM classfer s I fl( x) = h( x) β = hx ( ) H + HH Y, (8) ρ where ρ s a regularzaton term. he label of a test sample s assgned to the ndex of the output nodes wth the largest value. In our experments, we use a kernel-based ELM (KELM) wth a radal bass functon (RBF) kernel Multple-class boostng scheme Ensemble learnng s an effectve approach to obtan hgh classfcaton performance. Ensembles have more flexblty and can reduce problems related to over-fttng of the tranng data. As one of the outstandng types of ensembles, Boostng nvolves ncrementally buldng an ensemble by tranng each new model nstance to emphasze the tranng nstances that prevous models msclassfed. In ths paper, we concentrate on AdaBoost [2] whch s the frst practcal boostng algorthm and ntroduce a new mult-class boostng method. Let h ( x ) denotes the th base classfer, a boostng algorthm seeks for a convex lnear combnaton m Gx ( ) = ωh( x), (9) = = where ω s weght coeffcent of the th weak classfer. AdaBoost has proved to be equvalent to mnmze the exponental loss functon [3]: N mn exp( yg ( x)), st.. ω 0. (0) ω he logarthmc functon log( ) s a strctly monotoncally ncreasng functon and t s easy to calculate the mnmum value of a non-exponental functon. AdaBoost s equal to solve [3]: N mn log( exp( ygx ( ))), st.. ω 0, ω= /. () ω = he constrant ω= / avods enlargng the soluton ω by an arbtrary large factor to make the cost functon approach zero n the case of separable tranng data. In [4], rammer and Snger propose to construct multclass predctors wth pecewse lnear bound. onsderng smplcty and effcency of lnear functon, we use the followng rule for multple-class classfcaton, n x ( ) = arg max{ ω x}. (2) = And then we heurstcally propose the followng lnear obectve functon as: max( ω x ω x), (3) Where m. Fnally, we propose a mult-class boostng method as: m m (4) 2 f( ω, ω ) =ω x ω x+ log( exp( z)) +λ ω. he method mentoned above s the proposed MBS approach. Once we obtan the probablty output of three ELM classfers utlzng DMM-LBP_S, DMM-LBP_M, and DMM-LBP_ feature vectors, we conduct the MBS method to obtan the weght coeffcents. Our method utlzes the nformaton derved from DMMs and LBP operators, to mprove the performance of the ELM base classfers. We solve our obectve based on the MALAB toolbox. 4. Expermental results We evaluate our acton recognton method on the MSRActon3D [5] and MSRGesture3D [5] datasets whch consst of depth sequences captured by RGBD cameras. Our method s then compared wth the exstng methods. 4.. MSRActon3D dataset he MSRActon3D dataset [5] ncludes 20 actons performed by 0 subects. Each subect performs each acton 2 or 3 tmes. he resoluton of each depth mage s wo dfferent expermental settngs are used to evaluate our method. Settng - he same expermental settng reported n [5] s followed. Specfcally, the actons are dvded nto three subsets as lsted n able. For each subset, three dfferent tests are performed. In test one, /3 of the samples are used for tranng and the rest for testng; n test two, 2/3 of the samples are used for tranng and the rest for testng; n the cross subect test, one half of the subects (, 3, 5, 7, 9) are used for tranng and the rest for testng. m 3

4 Acton set (AS) Acton set 2 (AS2) Acton set 3 (AS3) Horzontal wave (2) Hgh wave () Hgh throw (6) Hammer (3) Hand catch (4) Forward kck (4) Forward punch (5) Draw x (7) Sde kck (5) Hgh throw (6) Draw tck (8) Joggng (6) Hand clap (0) Draw crcle (9) enns swng (7) Bend (3) wo hand wave () enns serve (8) enns serve (8) Forward kck (4) Golf swng (9) Pckup throw (20) Sde boxng (2) Pckup throw (20) able : hree subsets of actons for the MSR-Acton3D dataset. Settng 2 - he same expermental setup n [24] s used. A total of 20 actons are employed and one half of the subects (, 3, 5, 7, 9) are used for tranng and the remanng subects are used for testng. o facltate a far comparson, we set the same parameters of DMMs and blocks as noted n [7]. We fx m=4 and r= for the LBP operator n terms of classfcaton performance and computatonal complexty. he results shown n able 2 clearly demostrate the effectveness of our method. In test one, our method acheves 00% recognton accuracy n AS3 and comparable results to the hghset accuraces n AS and AS2. In test two, our method reaches 00% recognton accuracy for all the three subsets. In the cross subect test, whch s a challengng settng due to the large nter-class varatons of dfferent tranng and testng subects, the MBS method acheves the hghest average recognton accuracy. he comparson results of settng 2 are llustrated n able 3. Our approach acheves the hghest recognton accuracy. It should be noted that DMM-LBP-DF s also a decson-level fuson method, the recognton accuracy of our method s more than 2% hgher over DMM-LBP-DF whch demonstrates the effectveness of our proposed mult-class boostng fuson scheme. Method Accuracy (%) DMM-HOG [7] 85.5 Random Occupancy Patterns [6] 86.5 DMM-LBP-FF [7] 9.9 DMM-LBP-DF [7] 93.0 HON4D [25] 88.9 Actonlet Ensemble [6] 88.2 Depth ubod [26] 89.3 Rahman et al. [27] 88.8 Vemulapall et al. [23] 89.5 MBS (Ours) 95.2 able 3: Recognton accuracy (%) compared wth prevous methods on the MSRActon3D dataset MSRGesture3D dataset he MSRGesture3D dataset [5] conssts of 2 gestures defned by Amercan Sgn Language (ASL). It contans 333 depth sequences. he same parameters reported n [7] are used here for the szes of DMMs and blocks. able 4 shows the recognton results of our method as well as other exstng methods on the MSRGesture3D dataset. As can be seen from able 4, decson level fuson approach (DMM-LBP-DF) acheves the closest recognton accuracy to our method. hs ndcates that the DMMs model and texture features contrbute more effect on gesture recognton than the decson-level fuson. However, both our method and DMM-LBP-DF outperform other methods consderably. Method Accuracy (%) Random Occupancy Patterns [6] 88.5 HON4D [25] 92.5 Rahman et al. [27] 93.6 DMM-HOG [7] 89.2 DMM-LBP-FF [7] 93.4 DMM-LBP-DF [7] 94.6 Edge Enhanced DMM [28] 90.5 Kurakn et al. [5] 87.7 MBS (Ours) 94.7 able 4: Recognton accuracy (%) compared wth prevous methods on the MSRGesture3D dataset. Method est one est two ross subect AS AS2 AS3 Average AS AS2 AS3 Average AS AS2 AS3 Average L et al. [5] DMM-HOG [7] HOJ3D [2] haaraou et al. [22] Vemulapall et al. [23] DMM-LBP-FF [7] DMM-LBP-DF [7] Occupancy Patterns [24] MBS (Ours) able 2. omparson of recognton accuraces (%) of our method and prevous methods on the MSRActon3D dataset usng settng. 4

5 5. oncluson In ths paper, we propose an effectve feature descrptor and a novel decson-level fuson method for acton recognton. hs feature descrptor combnes depth moton maps (DMMs) and completed local bnary patterns (LBP) for an effectve acton representaton of a depth vdeo sequence. In decson-level fuson, we add the nequalty constrants derved from mult-class Support Vector Machne to modfy the general AdaBoost optmzaton functon. Kernel-based extreme learnng machne (KELM) classfers are utlzed as the basc classfers of AdaBoost. he expermental results on two benchmark datasets (MSRActon3D and MSRGesture3D) demonstrate that our method outperforms the state-of-art methods. Acknowledgement We acknowledged the support of the Natural Scence Foundaton of hna, under ontracts and , and the Program for New entury Excellent alents of the Unversty of Mnstry of Educaton of hna. References [] A. Bobck, and J. Davs. he recognton of human movement usng temporal templates. IEEE ransactons on Pattern Analyss and Machne Intellgence, 23(3): , 200. [2] A. Iosfds, A. efas, and I. Ptas. Mult-vew acton recognton based on acton volumes, fuzzy dstances and cluster dscrmnant analyss. Journal of Sgnal Processng, 93(6): , 203. [3] I. Laptev. On Space-me Interest Ponts. Journal of omputer Vson, 64(2): , [4] A. A. Efros, E.. Berg, G. Mor, and J. Malk. Recognzng acton at a dstance. In IV, pages , [5] W. L, Z. Zhang, and Z. Lu. Acton recognton based on a bag of 3D ponts. In VPR Workshops, pages 9 4, 200. [6] J. Wang, Z. Lu, Y. Wu, and J. Yuan. Mnng actonlet ensemble for acton recognton wth depth cameras. In VPR, pages , 202. [7] X. Yang,. Zhang, and Y. an. Recognzng actons usng depth moton maps-based hstograms of orented gradents. In AM Multmeda, pages , 202. [8]. hen, K. Lu, and N. Kehtarnavaz. Real tme human acton recognton based on depth moton maps. Journal of Real-me Image Processng, 203. [9]. Oala, M. Petkänen, and. Mäenpää. Multresoluton gray-scale and rotaton nvarant texture classfcaton wth local bnary patterns. IEEE ransactons on Pattern Analyss and Machne Intellgence, 24(7):97 987, [0] Z.-H. Guo, L. Zhang, and D. Zhang. A completed modelng of local bnary pattern operator for texture classfcaton. IEEE ransactons on Image Processng, 9(6): , 200. [] G.-B. Huang, H. Zhou, X. Dng, and R. Zhang. Extreme learnng machne for regresson and multclass classfcaton. IEEE ransactons on Systems, Man, and ybernetcs, Part B: ybernetcs, 42(2):53 529, 202. [2] Y. Freund, and R. E. Schapre. Experments wth a New Boostng Algorthm. Internatonal onference on Machne Learnng, pages 48 56, 996. [3] M. ollns, R.E. Schapre, and Y. Snger. Logstc Regresson, AdaBoost and Bregman Dstances. Machne Learnng, 48(): , [4] K. rammer, and Y. Snger. On the algorthmc mplementaton of multclass kernel-based vector machnes. Journal of Machne Learnng Research, 2(2), 200. [5] A. Kurakn, Z. Zhang, and Z. Lu. A real tme system for dynamc hand gesture recognton wth a depth sensor. In EUSIPO, pages , 202. [6] J. Wang, Z. Lu, J. horowsk, Z. hen, and Y. Wu. Robust 3d acton recognton wth random occupancy patterns. In EV, pages , 202. [7]. hen, R. Jafar, and N. Kehtarnavaz. Acton Recognton from Depth Sequences Usng Depth Moton Maps-Based Local Bnary Patterns. In WAV, pages , 205. [8]. hen, R. Jafar, and N. Kehtarnavaz. Improvng Human Acton Recognton Usng Fuson of Depth amera and Inertal Sensors, IEEE ransactons on Human-Machne Systems, 45():5 6, 205. [9]. hen, N. Kehtarnavaz, and R. Jafar. A Medcaton Adherence Montorng System for Pll Bottles Based on a Wearable Inertal Sensor. IEEE Engneerng n Medcne and Bology Socety, pages , 204. [20] W. L,. hen, H. Su, and Q. Du. Local Bnary Patterns for Spatal-Spectral lassfcaton of Hyperspectral Imagery. IEEE ransactons on Geoscence and Remote Sensng, 53(7): , 205. [2] L. Xa,.-. hen, and J. Aggarwal. Vew nvarant human acton recognton usng hstograms of 3d onts. In VPR Workshops, pages 20 27, 202. [22] A. A. haaraou, J. R. Padlla-López, P. lment-pérez, and F. Flórez-Revuelta. Evolutonary ont selecton to mprove human acton recognton wth rgb-d devces. Expert Systems wth Applcatons, 4(3): , 204. [23] R. Vemulapall, F. Arrate, and R. hellappa. Human acton recognton by representng 3d human skeletons as ponts n a le group. In VPR, 204. [24] A. W. Vera, E. R. Nascmento, G. L. Olvera, Z. Lu, and M. F. ampos. On the mprovement of human acton recognton from depth map sequences usng space-tme occupancy patterns. Pattern Recognton Letters, 36:22 227, 204. [25] O. Orefe and Z. Lu. Hon4d: Hstogram of orented 4d normals for actvty recognton from depth sequences. In VPR, pages , 203. [26] L. Xa and J. Aggarwal. Spato-temporal depth cubod smlarty feature for actvty recognton usng depth camera. In VPR, pages , 203. [27] H. Rahman, A. Mahmood, D. Q. Huynh, and A. Man. Real tme acton recognton usng hstograms of depth gradents and random decson forests. In WAV, pages , 204. [28]. Zhang and Y. an. Edge enhanced depth moton map for dynamc hand gesture recognton. In VPR Workshops, pages ,