Human Action Recognition Using Discriminative Models in the Learned Hierarchical Manifold Space

Transcription

1 Human Acton Recognton Usng Dscrmnatve Models n the Learned Herarchcal Manfold Sace Le Han, We Lang*, nxao Wu and Yunde Ja School of Comuter Scence and Technology, Beng Insttute of Technology 5 South Zhongguancun St. Beng, 0008, PR Chna {hanle,langwe,wuxnxao,ayunde}@bt.edu.cn Abstract A herarchcal learnng based aroach for human acton recognton s roosed n ths aer. It conssts of herarchcal nonlnear dmensonalty reducton based feature extracton and cascade dscrmnatve model based acton modelng. Human actons are nferred from human body ont motons and human bodes are decomosed nto several hysologcal body arts accordng to nherent herarchy (e.g. rght arm, left arm and head all belong to uer body). We exlore the underlyng herarchcal structures of hgh-dmensonal human ose sace usng Herarchcal Gaussan Process Latent Varable Model (HGPLVM) and learn a reresentatve moton attern set for each body art. In the herarchcal manfold sace, the bottom-u cascade Condtonal Random Felds (CRFs) are used to redct the corresondng moton attern n each manfold subsace, and then the fnal acton label s estmated for each observaton by a dscrmnatve classfer on the current moton attern set.. Introducton Human acton recognton has recently attracted ncreasng nterest from comuter vson and attern recognton communty. In artcular, t has a wde range of romsng alcatons, e.g. ntellgent nterface, systems for entertanment, survellance and securty [, ]. Almost all researches on recognton of human actons manly amed at daly human actons such as walkng and umng [8], sttng and lyng [5]. For ether the entre human body or most body arts, these actons are obvously dfferent n both D aearances and 3D ont ostons. However, many human actons are only dstnct n one or few body arts and t mght be dffcult to dstngush these actons from each other n ths case. So the recognton of subtle human actons, such as walkng, marchng, drbblng a basketball wth rght arm, walkng wth stff arms, *The corresondng author walkng wth wld legs, s a challengng task on whch we wll concentrate n ths aer. Generally, there are two crucal roblems nvolved n ths recognton task. One s extractng useful moton features such as ostons for each body arts. We do not address ths roblem but assume that such ostons are avalable, ether from a Moton Cature (MoCa) system [9] or from an automatc ose trackng system. Even f accurate 3D ostons are avalable, acton recognton s also dffcult due to the hgh dmensonalty of ose sace whch not only ncreases comutatonal comlexty but may also hdes key features of the actons. In ths aer, we use 3D ont oston traectores as the nut moton sequences. Based on hysologcal human body decomoston, a herarchcal manfold sace for human motons s learned usng herarchcal latent varable sace analyss. Moreover, a set of moton atterns for each body art s learned to reresent the tycal actons erformed by corresondng body art. The other roblem s modelng reference motons. Human actons evolve dynamcally over tme, and they are often subtle. Human actons can haen at varous tmescales and may exhbt long-range deendences. Furthermore, the transton between smle actons naturally has temoral segments of ambguty and overla. One of the most common aroaches for human acton recognton s to use Hdden Markov Model (HMM) or ts varants [5, 6]. However, a strong assumton that observatons are condtonally ndeendent s usually made n such generatve models n order to ensure comutatonal tractablty. Ths restrcton makes t dffcult or mossble to accommodate long-range deendences among observatons or multle overlang features of the observatons at multle tme stes. Condtonal random felds (CRF) and ts varants use an exonental dstrbuton to mode the entre sequence gven the observaton sequence [7, 8, 9, 0, ] whch avods the ndeendence assumton between observatons and makes t ossble to ncororate both overlang features and long-range deendences nto the model. CRFs and most ts varants can estmate a label for each observaton, so t s cometent for the task of recognzng contnuous human acton from unsegmented moton sequence. In ths aer, we ntroduce a cascade /08/$ IE

2 CRF to recognze dfferent moton atterns for both the entre body and each body art n the learned herarchcal manfold sace, and then a dscrmnatve classfer s used to redct the fnal acton label for each observaton. Consequently, our aroach can be cometent for contnuous human acton recognton.. Related Work Smnchsescu et al. [8] aled dscrmnatve models to classfy human moton actons (.e. walkng, umng, etc). They used a lnear-chan CRF based on moton cature data or mage descrtors. Hdden CRF (HCRF) has been used for arm and head gesture recognton from segmented sequences [9]. For caturng both extrnsc dynamcs and ntrnsc sub-structure, Morency et al. [] mroved HCRF by assocatng a dsont set of hdden states to each class label. Then nference on ths LDCRF (Latent-Dynamc CRF) can be effcently comuted usng belef roagaton. Obvously, features used as observatons n recognton model should be smle, ntutve and relable to extract wthout manual labor. Feature tyes that used n roosed methods can be dvded nto two categores: features based on D mage sequences, such as slhouettes [0], key frames [], otcal flow [3], temoral temlates [4]; features based on 3D moton streams, such as ont angel/oston traectores [3, 5]. 3D features have many advantages over features based D mage sequences as the deendence on vewont, llumnaton and occluson. Wang et al. [0] used smle slhouette observatons and exlored the underlyng structure of the artculated acton sace by KPCA. Factoral CRF was used to model temoral sequences n multle nteractng ways and ncrease ont accuracy by nformaton sharng. Lv et al. [3] regard the evoluton of one 3D ont oston as a channel wth corresondng weght. Human actons are reresented and recognzed usng sato-temoral moton temlates whch conssts of a set of moton channels. As stated n secton one, we also use accurate ont ostons and consder motons as traectores of ont ostons. However, the dmensonalty of orgnal human ose sace s too hgh. Fortunately, the subset of allowable confguratons n orgnal ose sace s generally restrcted by human bomechancs and traectores n the allowable subsace can be exected to le on a low-dmensonal manfold. Moreover human body has ts nherent herarchy and many actons mght be only dfferent n few body arts. So we decomose human body nto sx arts and construct a tree-based herarchcal model wth three layers (as shown n Fg.). The herarchcal manfold sace s learned by Herarchcal Gaussan Process Latent Varable Model (HGPLVM). Rather than learnng a sngle manfold sace for the orgnal ose sace, manfold subsaces on the lower layers have determned meanng and manfold subsaces on the hgher layers (.e. mddle nodes and root node n the tree-based model) can model not only the moton dynamc of corresondng body art but also the moton constrants among ther chld manfold subsaces. So the herarchcal manfold sace s more reasonable and skllful for reresentng the ntrnsc of human moton. In artcular, we ntroduce a cascade CRF for modelng and recognzng the moton atterns of each body art n the corresondng manfold subsace. Recognton rocesses from the bottom layer to the to one. Outut of the CRF nstances n the chld manfold subsaces s also regard as a art of the nut of the CRF nstance n arent manfold subsace. Then the dscrmnatve classfer, SVM, s used to redct the acton label for the current observaton. lower body (a) human body abdomen uer body rght leg left leg rght arm head left arm (b) Fg.. (a) the skeleton used n our aroach s the default skeleton n the CMU MoCa database. The numbers n arentheses ndcate the number of DOFs for the ont drectly above the labeled body node n the knematc tree. (b) decomoston of skeleton for herarchcal modelng. By ths searaton we can

3 model the manfold sace of moton for each body art and exress them ndeendently. 3. Feature Extracton and Moton Pattern Informatve features as observatons are crucal for the erformance of recognton model. We erform herarchcal dmensonalty reducton technology that can learn more effectve reresentaton for human moton. 3.. Gaussan Process Latent Varable Model Gaussan rocess latent varable model (GPLVM) s a fully robablstc, non-lnear, latent varable model that generalzes rncal comonent analyss [6]. Gven hgh-dmensonal data sace Y wth the dmensonalty D, the urose s to estmate the latent varable x for each tranng data y Y. The kernel matrx K s the core of the GPLVM model. We use the Radal Bass Functon (RBF) kernel because t smoothly nterolates the latent sace. The RBF kernel functon takes the form: γ T k( x, x ) = α rbf ex( ( x x ) ( x x )) () + α bas + β δ where k( x, x ) s the element n -th row and -th column of the kernel matrx K, α rbf controls the scale of the outut functons, γ s the nverse wdth arameter, α bas s the varance of the kernel bas, δ denotes the Kronecker delta. The scale k ( x, x ) models the roxmty between two onts x and x. GPLVM otmzaton s the rocess of learnng the kernel arameters θ = ( α rbf, β, α bas, γ ) and latent varables x. We maxmze the osteror P({ x }, θ ) { y}, whch corresonds to mnmzng the followng obectve functon: N D T L = ln K + tr( K YY ) + x () wth resect to the kernel arameter and latent varables. The otmzaton rocess s realzed through the Scaled Conugate Gradent (SCG) method. 3.. Herarchcal Manfold Sace We use HGPLVM [7], a herarchcal tree based model, to learn the herarchcal latent varables sace of hand moton. Most exstent tree based models tycally assume that all the nodes of the tree were observed, but the tree structure of HGPLVM refers to a herarchy of latent varables and only the leaf nodes need to be observed. It means that the latent varables of arent nodes are only constructed from latent varables of ther chld nodes. So a arent manfold sace can reresent the correlaton of motons n ts chld manfold subsace. The algorthm for otmzaton of the herarchcal manfold sace roceeds as follows:. Intalze each leaf node s latent varable set usng rncal comonent analyss (PCA) on the corresondng observaton dataset.. Intalze latent varable sets of all the arent nodes (.e. mddle nodes and root node) usng PCA on the concatenated latent varables of ther chld nodes. 3. Otmze ontly the arameters of the kernel matrces for each Gaussan rocess model and the latent varable ostons. In Fg. we show the results of mang moton sequence wth fve smlar actons nto the herarchcal manfold sace. Fg.. One of the herarchcal manfold saces n our exerments. To llumnate the ont robablty dstrbuton reresented by our herarchcal model, we assume a smle tree-based model wth three nodes (one arent node and two chld nodes). Y and Y are observatons assocated wth two chld nodes. and are the latent varable sets of the chld nodes, s the latent varable set of the arent node. The ont robablty dstrbuton s gven by ( Y, Y ) = ( Y ) ( Y ) (3) (, ) d However, the margnalzaton s not tractable and MAP solutons are adoted to fnd the values of the latent

4 varables. It means the maxmzaton of log (,, Y, Y ) = log ( Y ) + log ( Y ) + log (, 3.3. Moton Pattern Many human actons can be dfferent due to only one or few body arts and t means there must be some unversal moton atterns for each body art shared by dfferent actons, e.g., human actons such as walkng, drbblng a basketball wth rght arm and walkng wth stff arms have smlar leg motons. Accordng to the results of herarchcal manfold learnng, we can dscover the nherent moton atterns for each body art. (.e. n our exerments each leg has three moton atterns and the left arm has also three moton atterns but the rght arm has four moton atterns). Then each body art can have a moton attern set and each human acton s reresented by a set of moton atterns from all the body arts. Fg.3 shows fve nstances of moton attern n the to manfold subsace. (a) walkng (c) drbblng wth rght arm ) (b) marchng (4) (d) walkng wth stff arms (e) walkng wth wld legs Fg. 3. Fve nstances of moton attern reresented n the to manfold sace. Snce dfferent actons have dfferent sequence lengths, we aly tme war technology to arrange multle nstances to the same length. For each acton n every manfold subsace, gven N latent varable sequences,,..., where {,,..., } = x x xt s a sequence N of T latent varables and x reresents the d -dmenson latent varable at tme of nstance. Normalze the manfold traectores to the same length T and calculate the mean of N traectores by least squares aroxmaton: N T = arg mn x x + ε. (5) = = = { x, x,..., xt } s vewed as a moton attern n the corresondng manfold subsace. 4. Human Acton Recognton Human acton recognton can be regard as the redcton of a class label for each observaton n a moton sequence. Based on the herarchcal manfold learnng, we ntroduce a tree-based CRF model to recognze all the moton atterns for body arts. The toology of ths tree model deends on the human body decomoston and every node has ts own observaton sequence. The outut of each CRF s the label of corresondng moton attern and the nut of the mddle nodes also nclude the outut of CRFs n ther chld nodes. Then human actons can be dvded nto several moton atterns n dfferent manfold subsace. In the leaf nodes of the herarchcal manfold sace, we use general lnear-chan CRF to estmate the moton attern labels. The CRF s formulated as follows: assume s to be an acton label belongng to the ossble label set L and o s an observaton vector. Let S = { s t } and O = { o t }, t =,...,T so that S can be thought as a label sequence of an observed sequence O. Let C = {{ S c, O c}} be the set of clques n the undrected model G, CRF defnes the condtonal robablty of the state sequence (or the moton attern label sequence) gven the observed sequence as P = Φ S O θ ( S O) ( c, c ) (6) Z( O) where O) = Φ c C Z ( ( S, O ) S c C c c s a normalzaton factor over all label sequence. Φ s a otental functon as T Φ( Sc, Oc ) = ex λ n fn ( Sc, Oc, t) (7) t= n where { f n } s a set of features and they are mght be fn ( st, st, O, t) for each label transton and g n ( st, O, t) for each label. The model arameters θ = { λ n } are a set of real weghts, one weght er feature. For each CRF model, N gven a set of tranng samles {, S } =, model O arameters can be estmated by otmzng the followng condtonal log-lkelhood functon: Ω( θ ) = log θ ( S O ). (8) For CRF nstances used n the manfold subsace of

5 mddle nodes and root node, the nut s the combnaton of corresondng observaton sequence and the moton attern label sequences whch are oututted by CRFs used n ther chld manfold subsaces, then new observed sequence s m O arent = { O,{ S } = } where m s the number of chld nodes of current arent node. Usng above cascade recognton model, a set of moton attern labels for the current observaton s acqured. Then we use the traned dscrmnatve classfer, SVM [8], to estmate the current acton label. (a) walkng 5. Exerment Results We choose fve tycal and subtle human actons (Fg.4 shows walkng, marchng, drbblng a basketball wth rght arm, walkng wth stff arms and walkng wth wld legs) from CMU MoCa dataset [9] to rove our aroach. Walkng s the most common daly acton and all the fve actons are dfferent from each other due to one or few lmbs actons, e.g., drbblng a basketball wth rght arm s only dfferent wth walkng on the rght arm. 39 MoCa sequences n our dataset contan 85 frames. We manually segment these sequences such that each segment contans a whole course of one acton. In total we have 57 acton segments. The dstrbuton of these segments n each acton class s not unform. Walkng has 69 segments whle drbblng a basketball wth rght arm has only segments. The average number s 3. The lengths of these segments are also dfferent, rangng from 4 to 60 frames. The average s 39 frames. Snce sequences n orgnal MoCa dataset are samled at 0 frames er second, we samle at 30 frames er second. Each moton attern s normalzed to 35 frames. We dvde the dataset nto three arts, such as D, D and D 3 and otonally choose one art to learn the herarchcal CRF models, choose another to redct the sequences of moton attern label, whch are used to tran the SVM classfer. The last art s regarded as the test set. As ths arrangement, we erform dfferent exerments sx tmes and calculate the average recognton rates. Fg.5 shows the recognton rate of moton atterns (erformed by our cascade CRFs) n every manfold subsace, and Table. shows the recognton rate of each acton label (erformed by the traned SVM classfer). We also test CRFs wth dfferent wndow sze (.e. w=0 or ) and the results ndcate that sgnfcant mrovement erformance s obtaned when use wndow sze one, but we note that the erformance mrovement s not always along wth the ncrease of the wndow sze. In our extensve exerments, we fnd that the cometent recognton erformance s already acheved by settng wndow sze wth one. (b) marchng (c) drbblng a basketball wth rght arm (d) walkng wth stff arms (e) walkng wth wld legs Fg.4. Fve subtle acton sequences used n our exerments and the actons are all from CMU MoCa dataset. We can note that Walkng and Walkng wth stff arms are even too dffcult to be recognzed by human eyes.

6 References Fg.5. recognton rates of moton atterns n each manfold subsace w = 0 w = walkng march drbblng wth rght arm walkng wth stff arms walkng wth wld legs average Table.. Recognton rates of dfferent human actons 6. Concluson and Future Work In ths aer, we roose a learnng-based aroach to reresent and recognze 3D human actons usng herarchcal latent varable analyss and dscrmnatve models. By hysologcal decomoston of human body, the learned herarchcal manfold sace can reresent more ntrnsc moton of human actons. Each acton conssts of a set of moton atterns of all the body arts and each moton attern s a reresentatve moton of a body art n ts manfold subsace. A bottom-u dscrmnatve recognton aroach s erformed as moton atterns are redcted by a cascade CRF and the fnal acton labels are estmated usng a SVM classfer. Usng accurate 3D data allevates the comlexty of human acton recognton from the moton recovery roblem, whch also moses one lmtaton on the roosed aroach. Our future work ncludes extendng the algorthm to usng real mage data and exlorng herarchcal recognton models wth feedback. [] J. Aggarwal and Q. Ca. Human Moton Analyss: A Revew. CVIU, 73(3):48-440, 999. [] D. Gavrla. The Vsual Analyss of Human Movement: A Survey. CVIU, 73():8-98, 999. [3] F. Lv, R. Nevata and M. W. Lee. 3D Human Acton Recognton Usng Satal-temoral Moton Temlates. HCI/ICCV 005, LNCS 3766,. 0-30, 005. [4] L. Rabner. A tutoral on Hdden Markov Models and selected alcatons n seech recognton. Proc. IEEE, 77():57-86, 989. [5] T. Mor, Y. Segawa, M. Shmosaka and T. Sato. Herarchcal Recognton of Daly Human Actons Based on Contnuous Hdden Markov Models. Proceedngs of the 6 th Internatonal Conference on Automatc Face and Gesture Recognton, May 004, Seoul, Korea. [6] N. Nguyen, D. Phung, S. Venkatesh, and H. Bu. Learnng and detectng actvtes from movement traectores usng the herarchcal hdden Markov models, CVPR, 005. [7] J. Lafferty, A. McCallum and F. Perera. Condtonal random felds: robablstc models for segmentng and labelng sequence data. ICML, 00. [8] C. Smnchsescu, A. Kanaua, Z. L and D. Metaxas. Condtonal models for contextual human moton recognton. ICCV, 005. [9] S. Wang, A. Quatton, L. Morency, D. Demrdan and T. Darrell. Hdden condtonal random felds for gesture recognton. CVPR, 006. [0] L. Wang and D. Suter. Recognzng Human Actvtes from Slhouettes: Moton Subsace and Factoral Dscrmnatve Grahcal Model. CVPR, 007. [] L. Morency, A. Quatton and T. Darrell. Latent-Dynamc Dscrmnatve Models for Contnuous Gesture Recognton. CVPR, 007. [] S. Carlsson and J. Sullvan. Acton recognton by shae matchng to key frames. Worksho on Models versus Exemlars n Comuter Vson, 00. [3] A. Efros, A. Berg, G. Mor and J. Malk. Recognzng acton at a dstance. ICCV, 003. [4] A. Bobck and J. Davs. The recognton of human movement usng temoral temlates. PAMI, 3(3):57-67, 00. [5] N. Nguyen, D. Phung, S. Venkatesh, and H. Bu. Learnng and detectng actvtes from movement traectores usng the herarchcal hdden Markov models, CVPR, 005. [6] Nel D. Lawrence. Gaussan Process Latent Varable Models for Vsualsaton of Hgh dmensonal Data. NIPS, 004. [7] Nel D. Lawrence. Herarchcal Gaussan Process Latent Varable Models. Proceedngs of the 4 rd Internatonal Conference on Machne Learnng (ICML 07), Corvalls, USA, 007. [8] C.-C. Chang and C.-J. Ln. LIBSVM: a lbrary for suort vector machnes, 00. Software avalable at [9] CMU Grahcs Lab Moton Cature Database. htt://moca.cs.cmu.edu/.