Coarse-to-fine markerless gait analysis based on PCA and Gauss-Laguerre decomposition

Transcription

1 Coarse-to-fne markerless gat analyss based on PCA and Gauss-Laguerre decomposton M. Goffredo *, M. Schmd, S. Conforto, M. Carl, A. Ner and T. D Alesso Dept. Appled Electroncs, Unv. ROMA TRE, Va della Vasca Navale 84, I Roma ITALY ABSTRACT Human movement analyss s generally performed through the utlzaton of marker-based systems, whch allow reconstructng, wth hgh levels of accuracy, the trajectores of markers allocated on specfc ponts of the human body. Marker based systems, however, show some drawbacks that can be overcome by the use of vdeo systems applyng markerless technques. In ths paper, a specfcally desgned computer vson technque for the detecton and trackng of relevant body ponts s presented. It s based on the Gauss-Laguerre Decomposton, and a Prncpal Component Analyss Technque (PCA) s used to crcumscrbe the regon of nterest. Results obtaned on both synthetc and expermental tests provde sgnfcant reducton of the computatonal costs, wth no sgnfcant reducton of the trackng accuracy. Keywords: Markerless, Gat analyss, HMA, PCA, Gauss-Laguerre, moton analyss 1. INTRODUCTION Marker-based stereophotogrammetrc systems are the most common nstruments through whch human moton can be analysed n dagnostc and rehabltaton contexts 1. They make t possble to capture the coordnates of a set of markers, ether actve or passve, whch are fxed on the body segments the spatal trajectores of whch are to be measured,3. Marker systems show hgh levels of accuracy, as demonstrated n several studes 4,5. Ther accuracy s however counteracted by some drawbacks: actve marker based systems need a power supply, and the presence of wres between markers lmts the span of ther applcatons. Passve marker-based systems bear a lengthy set-up as well, and the presence of external devces could cause patents to feel uneasy. More recently, some authors tred to come up to the requrements n terms of tme- and cost-effectveness, by applyng technques derved from computer vson to track relevant elements wthout usng markers,.e. Markerless. Thus, markerless moton analyss performs the actvty of studyng and estmatng human moton by applyng computer vson technques 6,7, ether on the trackng phase or on the pose estmaton process, wthout fxng any knd of mplement on the body surface. Today the nterest n moton capture grows exponentally, not only n the clncal feld: the potental applcatons of moton analyss encompass advanced user nterfaces, sport studes, smart survellance systems, vrtual realty worlds, teleconferencng and telerehabltaton. The great nterest n moton estmaton s assented by the number of papers publshed n ths area. Aggarwal 7 revews exstng moton estmaton technques and compares approaches that use shape models wth those that do not. More recently, the survey by Gavrla 6 provdes a general ntroducton to the topc wth a specal focus on dfferent applcatons. The estmaton of human movement s deeply nfluenced by the applcaton feld: the present work deals wth an nnovatve method that allows estmatng human moton wth hgh levels of accuracy and low computatonal costs. Ths s acheved by applyng the Gauss Laguerre Decomposton (GLD) 8 algorthm 9 on regons that are dynamcally determned. GLD has been already used for rotaton nvarant sgnature extracton n the feld of pattern recognton 8,10,11 and has been employed for several mage processng applcatons such as restoraton and classfcaton 1. Ths approach has been * goffredo@unroma3.t, phone ; fax ;

2 proved sutable for a large varety of applcatons, such as mage archve exploraton, data mnng, map regstraton for telesensng and optcal telemetry. It has been also expermented n a smpler form for computer-asssted reassembly of fragmented frescoes n the context of a research program for artwork recovery 13. Moreover, t has been employed for moton estmaton n vdeo sequences robust wth respect to rotaton and zoom 14. Results obtaned from these applcatons have been proven satsfactory and have allowed applyng the method for analysng human gestures. Ths method, on the other hand, presents a hgh computatonal cost that requres to be reduced especally when nsertng the core of the moton estmaton n pont trackng durng long vdeo sequences. To ths objectve, when dealng wth subsequent frames n a sequence, the expanson n Gauss-Laguerre polynomals s performed only n a lmted regon of the mage centred on the pxel to track: the sze of ths area s a key parameter that deeply nfluences both the computatonal costs and the accuracy n the moton estmaton technque. Settng a small research area reduces the computatonal costs but could be nsuffcent to nclude the moton vector between consecutves frames. On the other hand, a bgger research area could deeply ncrease the processng tme. The method proposed n ths work thus ams at adaptng the sze of the research area durng the analyss of the vdeo sequences by usng the Prncpal Component Analyss (PCA) 15. The dynamc settng of the regon where GLD s appled s the soluton for accurate moton estmaton wth low computatonal costs. In lterature, PCA analyss methods have been manly adopted n mage compresson and face recognton 16. In the proposed approach, PCA analyss s used to dentfy regons presentng sgnfcant moton rate and to adapt the sze and the shape of the search area wth the moton drecton.. METHODOLOGY The proposed method s based on a coarse to fne procedure that ntally selects regons charactersed by movement n the scene and then apples a moton estmaton algorthm based on GLD to ths area. The coarse estmaton of the areas charactersed by moton s accomplshed by applyng the Prncpal Component Analyss Technque on fve consecutve frames of the analysed vdeo. The fne procedure estmates the moton vector by applyng GLD on the regon selected n the prevous step. Fgure 1 shows a non formal flow dagram of the proposed algorthm: the frames of the vdeo sequences are analysed frst by the PCA Block, to determne the sze, shape, and poston of the Adapted Research Area, ARA; the GLD Block s then performed n ARA., to estmate the moton vector mv. F F +1 PCA ARA GLD mv Fgure 1: Graphcal representaton of the proposed moton estmaton algorthm The PCA s appled on fve consecutve frames after the transformaton of each frame F nto a one-dmensonal sgnal s obtaned by juxtaposng each row of the frame: a matrx, s then bult by consderng the fve consecutve sgnals (s, s +1,, s +4 ), undergoes Prncpal Component Analyss, to obtan the correspondng matrx, composed of the fve prncpal components pc j, j+1,,,5, formed by transformng the orgnal data nto the space of the prncpal components. Each of them s then rearranged to retreve the orgnal shape of the frame, and each mage n the space of the prncpal components shows unque characterstcs. Fgure shows the graphcal representaton of the applcaton of PCA. Expermental results evdenced that the thrd prncpal component s ntrnscally tuned to show the range of moton on the fve consdered frames,.e. allows to extract regons charactersed by movement, that are sgnfcantly dstnct from the statc ones. Fgure 3 shows an example of the components obtaned from the frst fve frames of the sequence dsk used n the synthetc tests.

3 pc 1 s F pc F+1 T s +1 T -1 F + pc 3 F +3 s + PCA F +4 pc 4 s +3 PC 1 PC PC 3 PC 4 PC 5 s +3 pc 5 Fgure : Graphcal representaton of the Prncpal Component Analyss appled on fve consecutve frames Fgure 3: PCA components calculated on the frst fve consecutve frames of the sequence dsk The thrd component s charactersed by a unform background where the pxels that are charactersed by movement are put n evdence. The largest connected regon whose pxels led outsde the range [µ-σ, µ+σ] of the entre PC 3 was then extracted, and underwent medan flterng wth wndow sze [7,7] pxel and morphologcal operatons of dlaton and eroson. Fgure 4 shows these operatons on the thrd component of the prevous fgure. a b c d Fgure 4: Example of the regon selecton from PC 3 (a. PC 3, b. after thresholdng, c. after medan flterng and dlaton, d. after eroson) Research Area ARA Fgure 5: Research area of a selected pxel p (frst frame)

4 The latest regon s thus used to buld the ARA, defned by the vertcal, horzontal, left and rght extremtes calculated as the boundares of the whte regon wth respect to the selected pxel p. The lengths of ARA sdes are then weghted wth a factor that depends on the estmated moton of p n the prevous analysed frames. Fgure 5 shows ARA for the selected pxel at the frst frame of the sequence dsk. In ths way a coarse estmaton of moton drecton can be obtaned and the frst step of the proposed method allows estmatng the area where the movement s present. The fne estmaton of the selected pont trajectory s then evaluated by applyng the Gauss Laguerre Decomposton only n ths area: the frames are represented by means of a set of Crcular Harmonc Functons (CHF), called Gauss-Laguerre (GL) functons. In general, CHFs are polar separable complex functons wth harmonc angular shape, suted for generatng rotaton nvarant sgnatures closely related to moment nvarants 17. The CHFs of the GL type employed are obtaned by dfferentaton of a Gaussan functon, and consttute a complete famly of orthogonal functons wth respect to the Gaussan one. Expandng an mage n GL functons around each pont yelds a redundant representaton, called Gauss Laguerre Transform (GLT) 18,19, whch s actually the polar form of the Hermte Transform 0. Snce CHFs are ntrnscally "tuned" to mage features lke edges, lnes, equangular forks, orthogonal crosses wthout regard to ther orentatons, they are good canddates for pont matchng applcatons. In the mplementaton here adopted, the mage s thus transformed wth a seres expanson fndng a sum of prmtve mages that present the dfferent characterstcs of the orgnal mage (Wavelet Transform). If I(r) s the orgnal mage, W S (a,b) s the Wavelet transform of I(r), wrtten as: 1 * r b W I ( a, b) = I, ψ a, b = I ( r) ψ d r a a where a and b are complex factors (scale and translaton factors respectvely) and Ψ* s the Mother Wavelet. In the Gauss Laguerre expanson, the Mother Wavelet used to transform the sgnal s the CHF of the GL type, wth the followng form: ( ) k ( ) ( ( ) ( ) ) n n 1/ n / k! + n n πr jnθ L k r, θ = 1 π r Lk ( πr ) e e ( n + k)! (n) where L k are the Laguerre Polynomals wth order n and degree k. Once the transformaton s employed, the moton estmaton s obtaned by defnng a Lkelhood Map that takes nto account the GL Expansons of the dfferent frames. In many cases, hgh accuracy n the localzaton estmate s requred, so that optmum solutons have to be consdered. Drect evaluaton of the optmal Maxmum Lkelhood (ML) soluton entals, n prncple, the exhaustve search of the lkelhood functonal maxma n a four-dmensonal parameter space consttuted by poston, sze and orentaton. The collecton of ML maxma for each pont defnes the so-called Gauss Laguerre Lkelhood Map, whose absolute maxmum ndcates the pattern locaton along wth the assocate scale and orentaton. The essental pont of ths method s that the true ML soluton can be approxmated by GLT truncaton, and that few GLT coeffcents are most often suffcent to obtan unambguous and relable estmates. The proposed method s partcularly useful for analysng movements wth tme varant velocty: n ths case, the sze of the research area s automatcally modfed wth respect to the coarse movement estmaton of fve successve frames. The qualty of the overall method has then been tested on dfferent vdeos: at frst on synthetc vdeo tests and then expermentally on real mage sequences recorded durng gat trals. 3. SYNTHETIC TESTS The proposed method has been tested on synthetc vdeo sequences charactersed by controlled movements wth dfferent drectons and veloctes. The sequence here presented exhbts a dsk wth a chess texture that has been moved on a compact background. The object moton s charactersed by 8 dfferent drectons along the prncpal axes and the prncpal dagonals. The dsk veloctes have been selected between 30 and 43 pxels/s, followng a snusodal law. The vdeo sequence has a spatal resoluton of 4x4 pxels and a tme resoluton of 30 frames/s. 1

5 The proposed moton estmaton method has been appled on 0 ponts on the dsk. The vdeo sequence has been analysed by applyng the GLD n two dfferent ways: at frst by settng a fxed sze for the research area (Fxed Research Area, FRA, 10x10 pxels) and then by applyng the coarse pre-processng to adapt the area sze, and shape, thus obtanng the Adapted Research Area (ARA). The trajectores of the 0 selected ponts have been determned wth hgh levels of accuracy, and the Root Mean Square Error (RMSE) has been calculated by comparng the Eucldean velocty vectors: ( ) ( ) ( ) ( ) E E E = + s x y R R R = + s x y RMSE trajectory = N R E ( s ) s where N s the overall number of frames n the vdeo sequence, x and y represent the projectons of the trajectory on the X and Y axs, the apex R ndcates the real trajectory and the apex E ndcates the estmated trajectory. Table 1 shows RMSE values obtaned by comparng the real dsk trajectory and the estmated ones. = 1 RMSE trajectory (pxel) FRA (10x10 pxels) 1.05 ARA 0.60 N Table 1: Comparson between trajectores RMSE for the Fxed Research Area (FRA), and PCA-based Adapted Research Area (ARA) The proposed method shows hgher levels of accuracy respect to the applcaton of GL decomposton of fxed research areas centred on the selected ponts. The coarse estmaton, n fact, lmts the set of canddates-pxels where the Gauss Laguerre Lkelhood Map s evaluated and, therefore, the possblty to estmate ncorrect movements. Fgure 6 shows the dmenson of the research area durng the vdeo sequence. The movement mposed to the dsk s composed by 8 dagonal trajectores and 8 horzontal and vertcal movements. The frst part of the moton, dagonal drectons, produces a larger research area. The pseudo-snusodal trend s obtaned because of the dsk s snusodal moton law (perod of 30 frames). It results clear that the overall lower value of ARA sgnfcantly lowers processng tme. Fgure 6: research areas over tme

6 In order to evaluate the extenson of the auto-shaped research area durng the vdeo sequence, a parameter, called Medum Areas Dfference (MAD), has been defned as follow: MAD = N = 1 where A p s the area bult on the projectons of the movement between the pont at the frame and ts real poston at the frame +1. The varable A a s the porton of the research area between the pont at the frame and the vertex of the refers to the movement drecton (see fgure 7). The square root of the dfference between A a and A p allows obtanng a parameter wth unt of measure of pxel. Research Area Pont frame A a N A p = A p = A a Pont frame +1 Fgure 7: the parameter MAD Table shows the results of the applcaton of the proposed method n terms of MAD: the auto-shape research area fts the movement n the correct drecton wth a smaller number of extra pxels. MAD (pxel) FRA (10x10 pxels) 7.05 ARA 3.99 Table : Comparson between MAD The proposed moton estmaton algorthm has been appled on a synthetc vdeo sequence, charactersed by movements wth dfferent velocty and drectons. Results of the PCA-based ARA shows both hgher levels of accuracy and lower computatonal costs (processng tme < 10 s per frame on a 1. GHz processor) than wth FRA (processng tme < 18 s per frame). 4. EXPERIMENTAL RESULTS In the human moton context, gat analyss represents a typcal repettve gesture, where the leg movements are charactersed by dfferent phases. The gat cycle, n fact, s defned as the set of movements and events that take place between two successve ntal contacts wth the ground: the stance phase and the swng phase 1. The movement of ponts on a leg durng gat, therefore, presents phases wth dfferent veloctes. Sgnfcant results have been obtaned by analysng vdeo sequences from a commercal camera (30 fps, x4 pxels). The sequences show a human gat: the moton drecton s orthogonal to the camera plane. In ths experment, specfc attenton has been dedcated on the junctons of knee and ankle and on the foot lmt. The applcaton of the proposed method has allowed shapng the research area followng the gat cycle of the legs. Fgure 8 shows the thrd prncpal

7 component for sx typcal shots extracted from the vdeo sequence and the correspondng frames where the selected ponts and the research areas are shown. The applcaton of the proposed method allows correctly estmatng the trajectores of the selected ponts. The foot plantar- dors-flexon angle has also been estmated from the estmated trajectores of the selected ponts for a more complete knematc evaluaton. Fgure 9 shows the estmated tme seres. frame n 18 frame n 1 1 frame n frame n frame n 1 frame n Fgure 8: Thrd prncpal component for sx typcal shots extracted from the gat-vdeo sequence and the correspondng frames wth the selected ponts and the research areas.

8 degree frames Fgure 9: Ankle angle 5. CONCLUSIONS Am of ths work s to defne a novel method to analyse human gat by estmatng relevant ponts knematc wthout the applcaton of markers. In the proposed contrbuton, the use of PCA as an nstrument to speed up the search process has been dscussed, and the mathematcal model and the related optmal Bayesan soluton based on the maxmzaton of the lkelhood functonal n the Gauss Laguerre doman has been detaled. Performance of the proposed method has been llustrated by results on both n smulated and n vvo trals. Synthetc tests on several ponts have allowed to evaluate the accuracy of moton estmaton and to quantfy the mprovement n terms of both accuracy and processng tme thanks to the PCA-based adaptaton of the research area. Promsng results have been obtaned by applyng the proposed markerless moton estmaton technque to real vdeo sequences recordng human gat gestures. In ths case, n fact, the trajectores of ponts placed on the knee and ankle junctons and on the foot have been estmated and the use of PCA has allowed adaptng the research area followng the step cycle characterstc of the gat gesture. In order to analyse the knematcs of the gat, the ankle angle has also been estmated from the trajectores of the selected ponts. The results obtaned wth the proposed method are encouragng and the possblty of developng more complex moton estmaton technques can hopefully enhance the accuracy, thus ncludng motons that do not necessarly le on the camera plane. REFERENCES 1 A. Cappello, A. Cappozzo, U. Della Croce, A. Leardn, Bone poston and orentaton reconstructon usng external markers, Fve Dmensonal Analyss of Human Locomoton, London, 1997

9 I. Soderkvst I, P.A. Wedn, Determnng the movements of the skeleton usng well-confgured markers, J Bomech, 1993, 6, M.P. Kabada, H.K. Ramakrshnan, M.E. Wooten, Measurement of lower extremty knematcs durng level walkng, J Orthop Res, 1990, 8, Y. Ehara, H. Fujmoto, S. Myazaky, S. Tanaka, S. Yamamoto, Comparson of the performance of 3D camera systems, Gat Posture, 1995, 3, Y. Ehara, H. Fujmoto, S. Myazaky, M. Mouvhmaru, S. Tanaka, S. Yamamoto, Comparson of the performance of 3D camera systems II, Gat Posture, 1997, 5, D.M. Gavrla, The Vsual Analyss of Human Movement: A Survey, Comp Vs Imag Underst, 1999, 73, J.K. Aggarwal, Q. Ca, Human Moton Analyss: A Revew, Comp Vs Imag Underst, 1999, 73, H. H. Arsenault, Y. Sheng, Propertes of the crcular harmonc expanson for rotaton nvarant pattern recognton Appl. Opt., 1986, 5, M. Carl, F. Coppola, A. Ner, Translaton, Orentaton and Scale Estmaton Based on Laguerre-Gauss Crcular Harmonc Pyramds, Image Processng: Algorthms and Systems, Proc. SPIE Vol. 4667, Electronc Imagng,, San Jose, Calforna USA. 10 Y. Sheng, J. Duvernoy, Crcular-fourer radal-melln transform descrptors for pattern recognton J Opt Soc Amer, 1986, 3, R. Mukundan, S. H. Ong, P. A. Lee, Image analyss by Tchebchef moments, IEEE Trans Imag Proc, 1, 10, G. Jacovtt, A. Ner, Bayesan removal of codng-block artefacts n the harmonc-angular flters features doman, Proc. SPIE, 1995, 1195, Avalable onlne: 14 M. Carl, G. Jacovtt, A. Ner, Pattern recognton based on the Gauss-Laguerre transform, Proc. SPIE, 0, 409, J. E. Jackson, A user's gude to prncpal components, John Wley and Sons, 1991, A. Levy, M. Lndenbaum, Sequental Karhunen Loeve Bass Extracton and ts Applcaton to Images, IEEE Trans Imag Proc, 0, 9, E. P. Smoncell, A rotaton-nvarant pattern sgnature Proc. 3rd Int. Conf. Image Processng, ICIP 1996, 3, G. Jacovtt, A. Manca, A. Ner, Hypercomplete crcular harmonc pyramds, Proc. SPIE, 1996, 85, ,. 19 G. Jacovtt, A. Ner, Multresoluton crcular harmonc decomposton, IEEE Trans Sgn. Proc, 0, 48, J. B. Martens, Local orentaton analyss n mages by means of the Hermte transform, IEEE Trans Image Proc, 1997, 6, V. T. Inman, H. J. Ralston, F. Todd, Human Walkng, Wllam and Wlkns, Baltmora, 1981.