Detection Tracking and Recognition of Human Poses for a Real Time Spatial Game

Transcription

1 Deecion Tracking and Recogniion of Human Poses for a Real Time Spaial Game Feifei Huo, Emile A. Hendriks, A.H.J. Oomes Delf Universiy of Technology The Neherlands f.huo@udelf.nl Pascal van Beek, Remco Velkamp Urech Universiy The Neherlands phbeek@sudens.cs.uu.nl Absrac In his paper, we presen an approach o deec, rack people, and recognize poses. The deeced poses are used for conrolling a real ime spaial game. In he people deecion, racking and pose recogniion sysem, body pars such as he orso and he hands are segmened from he whole body and racked over ime. The 2D coordinaes of hese body pars are used as he inpu of a pose recogniion sysem. By ransferring disance and angles beween he orso cener and he hands ino classifier feaure space, simple classifiers, such as he neares mean classifier, are sufficien for recognizing predefined key poses. The oupu of he classifier, ha is he idenificaion of he pose, is used o conrol color and acions of he virual acor. The posiion of he virual acor is seered by he deeced posiion of he user in he image. Keywords: Pose Recogniion, Spaial Game, Real Time. 1 INTRODUCTION Nowadays video-based applicaions have become more and more widespread [1]. A well-known video-based applicaion is man-machine ineracion, in which people can use heir facial expressions, gesures and poses o conrol e.g. virual acors or (serious) games. The essenial ingredien for an effecive man-machine ineracion experience is ha he sysem indicaes is level of undersanding of he user s movemen. Therefore, human moion analysis plays an imporan role in man-machine ineracion. Generally, here are wo approaches o obain he movemen of human body. One approach is marker-based, in which users need o wear specific sui wih sensors on i. These sensors are used o capure he moion of differen body pars. The oher approach is vision-based, in which users are oally free of any obrusive sensors. The movemen of users is analyzed from he recorded video daa. Compared wih he firs approach, he second one may have less accuracy of reading moion informaion. However, i is more convenien and friendly o users, especially for gaming applicaions. Therefore in his paper, we propose a vision-based people deecion, racking, and pose recogniion sysem. I direcly uses he capured video frame as inpu, hen gives he 2D posiion and pose of he people if here are people appearing in he scene. The posiion and pose informaion is conneced o a spaial game sysem, and used as he conrol command of he spaial game. The remainder of he paper is organized as follows. In Secion 2, we give a brief inroducion of previous researches. The mehodology of he proposed approach is described in Secion 3. In Secion 4, we show he spaial game applicaion. A he end, he conclusion is drawn in Secion 5. 2 PREVIOUS RESEARCHES Alhough here has been published a significan number of research papers on human body racking and pose recogniion, many research issues sill remain o be solved. In [2] he body pars are reliably labeled and locaed by 2D conour shape analysis. Tracking performance is significanly increased by aking color ino accoun. The limiaion of his mehod is ha all he body pars need o be segmened alhough hey may no be needed for cerain applicaions. In [3], an approach o esimae 3D human pose wih muliple cameras is proposed. I can deal wih cluered scenes and self-occlusion under some consrains. Bu he processing ime for pose esimaion is abou 45

2 seconds per frame, which is no suiable for real-ime applicaions. In [4] an exemplar based approach is used o localize and recognize human poses. However, he pose deecor is only learn from walking poses, so hey can no deec people wih oher poses han walking. In his paper, we presen an approach for real-ime people deecion, racking, and pose recogniion, which can handle a variey of poses and is fas enough o seer a real ime game. The oupu of he pose classifier is used for conrolling appearance and acions wihin his spaial game. Fig.1 gives he flowchar of he proposed sysem. 3 METHODOLOGY Fig.1. The flowchar of he proposed sysem MOTION EXTRACTION In an indoor scenario, cameras are usually a fixed locaions, so we do no need o consider moion of he scene for foreground obec exracion. Therefore, background subracion is a quie suiable mehod in his case. Compared o emporal differencing, he background subracion will no only give he edges bu he whole silhouee of he moving obecs. The firs sep of he background subracion is o build up a background image, which should no include any foreground obecs. In our implemenaion, he background image is buil by using a mixure of Gaussian model in [5]. This mehod is also robus o cluered scenes. The background image is updaed by using curren frames, in order o deal wih he change of lighing condiions. Afer he background image is obained, he pixelwise difference beween he curren frame and he background model is used o classify each pixel as eiher foreground or no. A difference image D beween an inpu image F and he background image B shows only he moving obec regions, while he saionary background is suppressed (see eq.1). D = F B (1) A foreground binary image is obained by using he difference image Di, and a hreshold T d (see eq.2). T d can be obained experimenally and is found ou o be no very sensiive for differen indoor scenes. Fig.2 shows he inpu image F and is foreground binary image R. R 1 = 0 if D else T d (2) (a) (b) Fig.2. Exracion of foreground binary image: (a) inpu image F, (b) foreground binary image R.

3 3.2. TORSO AND HAND SEGMENTATION Afer we obain he moion-based human body blob, nex sep is o segmen differen body pars. This can be based on he selecion of various feaures, such as shape, edge, silhouee, conour and color of human s body. We use a 2D silhouee model o deec he orso and use skin color for he deecion of hands Torso segmenaion Afer he foreground binary image is buil, we use a geomerical characerisic of persons o deec hem. The prior knowledge of human geomerical srucure ha we use is a head-shoulder-upperbody model, shown in Fig.3. The parameers in his 2D human shape model are posiion and scale, so his model is described as P = ( x, y, scale). If he scale parameer in he 2D model is deermined, he widh and lengh of human upper body can also be derived according o he shape characerisic of persons. Since he recorded video daa is from one single camera, he posiion of he human is given by 2D coordinaes, ha is x and y coordinae. In he real 3D world, his mehod can be easily exended o muliple view approaches by fusing he 2D daa derived from synchronized cameras ino 3D real coordinaes. Afer he definiion of he human model, he problem comes up wih how o use his 2D model o deermine if here is a person in he image or no. Generally he inuiive soluion is exhausively searching he image for he defined model. Alhough here are only hree parameers in his model, x y coordinae and scale, i sill needs a large amoun of calculaions o find he global opimal soluion. In he siuaion of more han one person appearing in he image simulaneously, his exhausive search is oo far away from real ime applicaions. So i is impracical o use his model direcly for emplae maching. However, his deecion and racking problem can be seen as a sae esimaion problem. To solve saisical mehods, such as paricle filers, can be used, which is especially suiable for non-gaussian and muli-model siuaions. In a paricle filer, he probabiliy of deecion of a person in he image is represened as he finess coefficien of he defined 2D shape model. The definiion of his finess coefficien is he same as [6]. The emplae used o calculae he finess coefficien is shown in Fig.3 (b). I is composed of wo regions: foreground region F and background region B, which is surrounding region F. (a) Fig.3. (a) One sample in a paricle filer, (b) wo dimensional human model o calculae finess coefficien [6]. In order o consruc he probabiliy, he finess coefficien should be a value beween 1 and 0. This requiremen is saisfied by using he following definiion (see eq.3). (3) 1 F B, if F > B ω = Area( F) 0, oherwise Where Area( F ) = Area( B), F is he summaion of he pixel values in region F, B is a summaion of he pixel values in region B. Hopefully foreground pixels wih a value 1 will fall ino region F as much as possible and background pixels wih a value 0 are only included in region B. Under his opimal siuaion he finess coefficien will be 1, which means he foreground binary image saisfy he 2D model perfecly. We use his finess coefficien ω as he probabiliy of a person presen in he image. The larger he value of he finess coefficien is, he higher he probabiliy of a person exising in he image. By fiing his 2D model on he foreground binary image, people s head and orso can be segmened from oher pars. (b)

4 3.2.2 Hand segmenaion In addiion o he 2D model menioned in for human s orso deecion and racking, foreground pixels are furher segmened ino skin-color and non-skin-color regions. A skin color model in he RGB color-space is used o selec skin color pixels on he foreground image. This human skin color model is similar o he model in [7]. If foreground pixels mapping ino he RGB color-space saisfy he following condiions in eq.4 [7], hey will be considered as skin-color pixels. People s face and orso region are excluded from skin color deecion by masking he head and orso region esimaed from B π π G π π B π π arcan( ) <, arcan( ) <, arcan( ) < (4) R 4 8 R 6 18 G 5 15 Afer he skin color pixels are seleced, wo pos-processing seps are used o ge rid of false posiive deecions. The firs sep is o delee regions wih a very small size, which are impossible o be hand regions. In he second sep, a moion mask is inroduced o exclude regions which are far away from previous hand locaions. I limis he movemen of hands wihin a cerain bounding box. From he remaining wo larges blobs, we calculae he ceners of graviy and use hem o represen he posiion of he hands FEATURE SPACE CONSTRUCTION The inpu of he proposed pose recogniion sysem are 2D posiions of he orso cener and he hands. However, we ransfer hem ino normalized feaure space and rain he classifier in his new feaure space. The reason is ha he pose recogniion sysem should be scene invarian. Tha is, no maer where he person is in he scene, or how far he person is from he cameras, he predefined key poses should be recognized. Therefore he feaure space is buil by using angles and relaive posiions beween hands and orso cener. We consruc he following 6 feaure F = c, c, c,, c : componens, denoed as { } se l l r r l ( x2 x2) ( y c1 =, 2 y2) ( x c2 =, 2 x2) ( y c3 =, 2 y2) y c4 =, 2 y2 c5 = arcan l s s s s x x r, 2 2 c6 = arcan r 2 2 x2 x2 y y (5) l l r r Here ( x2, y 2), ( x2, y2) and ( x2, y2 ) are he 2D posiions of he orso cener, lef hand and righ hand. s is he scale parameer in 2D model. The classifier will be rained and esed on his 6D feaure space F POSE CLASSIFICATION The key poses are designed for gaming conrol, so hey should be easy for users o remember and perform. We also choose he number of he poses no oo high o make i easier for users. In our sysem, we defined nine poses in oal, as shown in Fig.4. From op row o boom row and lef o righ, hese nine poses are labeled as pose1 o pose9. In order o build a classifier, we manually labeled he frames conaining he nine poses ino nine classes. For each pose, he samples are seleced from differen persons. Our experimenal daa se conains 1515 samples of 9 pose ypes (classes) and 6 feaures. On average, each pose class is represened by 170 samples. The performances of several saisical classifiers wih differen complexiies are compared. Specifically, we evaluaed he neares mean classifier (NMC), he linear classifier (LDC) and he quadraic classifier (QDC) assuming normal densiies and he non-parameric Parzen classifier [8]. We observe ha he simples mehod (NMC) provides comparable performance o more complex classifiers which need an exra dimensionaliy reducion sep o avoid he curse of dimensionaliy. We conclude ha he exraced feaures are informaive and do no require use of more complex classifiers. Therefore, we chose a simple classifier, 10-neares neighbourhood classifier (NNC), for he pose classificaion par of our sysem. I is easy o implemen and also benefis from he compuaion poin of view. se

5 Fig.4. Predefined key poses. From op row o boom row and lef o righ, hese nine poses are labeled as pose1 o pose RESULTS AND DISCUSSION The proposed sysem is implemened in C++ wih OpenCV libraries [9]. The processing ime for each frame is seconds, including background subracion, body pars segmenaion and pose recogniion. We esed he sysem wih more han 20 users. We also chose he differen indoor environmen wih various seings. Some of he experimenal resuls are shown in Fig.5. An online video demo is also available [10]. People s head and orso are indicaed wih yellow and red recangles. Wihin his 2D model, he locaion of head op, head cener, orso cener, orso boom and boh shoulder can be esimaed, which are presened by yellow and red cross. People s lef and righ hand are marked wih blue and green cross separaely. The oupu of he pose recogniion sysem is an ineger, he number shown in Fig.5. I gives he indicaion which pose user is performing. This ineger and he 1D (horizonal) posiion of he user will be used as he conrol command of a spaial game. We evaluaed pose classifiers using cross-validaion approaches. The average error for all he poses is 6% by using NMC. The resul shows ha here is a clear separaion beween pre-defined poses. We also calculae he confusion marices of he 9-class pose classifier (NMC). The resuls are promising. Mos of he poses can be recognized very well. More deails abou he pose classificaion can be found in [11].

6 Fig.5. Resuls from people segmenaion and pose recogniion. 4. SPATIAL GAME APPLICATION 4.1. IMPLEMENTATION As an applicaion for he pose recogniion sysem we implemened a spaial game, based on he proposal of Phong in [12]. This is a variaion of he game Pong [13] in which he player conrols a ba o bounce off balls. In Phong he player conrols a chameleon which has o bounce off phoons, see in Fig.6. The posiion of he chameleon is deermined by he player s posiion in fron of he camera. The phoons can have 6 differen colors: red, blue, green, yellow, cyan and magena. The chameleon can change ino each of hese colors when he player adops he appropriae pose. (a) Fig.6. (a) Color and posiion of he chameleon are conrolled by pose and posiion of he player, (b) he ongue of he chameleon is also conrolled by a pose o cach flies [12]. (b)

7 When he phoon his he ceiling, i changes color. When he phoon is bounced off while he chameleon has he wrong color, he conrols flip. Lef becomes righ and vice versa. When he chameleon has he righ color while bouncing he phoon off, he score and speed of he phoon is increased. When he chameleon misses he phoon he ground is heaed up. Afer 4 misses he ground is oo ho and he game is over. A random momens a bug flies ino he scene which can be eaen by he chameleon when he player adaps o he eaing pose. This will increase he score and he ground will cool down. The game is implemened using he graphics engine Ogre [14]. The inpu for he game is given by he pose recogniion sysem. The pose recogniion sysem and he spaial game are wo separae applicaions which communicae via sockes [15]. Therefore, i is possible for he wo applicaions o run on differen compuers and communicae hrough a nework. The pose recogniion sysem sends wo ypes of daa o he spaial game in every ime sep. I sends an ineger ha represens a pose (1-9) and an ineger represening he 1D-locaion of he player. Whenever he spaial game receives his daa, i updaes he posiion of he chameleon according o he posiion ineger and i carries ou he acion belonging o he pose index ha was send. These acions consis of 6 poses for changing he chameleon ino he 6 differen colors, 1 pose is for eaing he bug, 1 pose is for saring he game and 1 pose is o pause he game. Fig.7 gives a screen sho of user playing he game. On he lef side is he inerface of he game, which shows he level, bounces, hea and score of he player. The hree windows on he righ side are he resuls from vision-based analysis. From op o boom, hey are original image, resuls from body pars segmenaion and pose recogniion, and foreground binary image RESULTS AND DISCUSSION In our firs es runs i became clear ha he sensiiviy of he pose recogniion o deec he change of poses gave a problem for he game player. Whenever he player needs o change from one pose o anoher here could be a differen pose adoped ha is in beween hese wo poses. When his happens he color of he chameleon in he game is shorly changed ino an unwaned color. This problem has been overcome by using a couner whenever a new pose is adoped. The new pose has o be adoped for 4 consecuive ime seps unil is corresponding acion is carried ou. This adusmen improved he playabiliy of he game as he user feels having a beer conrol of he chameleon. We did encouner a shor delay in handling he players inpu. The delay is caused by he image processing ime. This is mosly noiced wih updaing he chameleon s posiion by he player s acual locaion, bu he delay is oo small o acually cause gameplay problems. Fig.7. Spaial game inerface. On he lef side is he inerface of he game, which shows he level, bounces, hea and score of he player. The hree windows on he righ side are he resuls from vision-based analysis. From op o boom, hey are original image, resuls from body pars segmenaion and pose recogniion, and foreground binary image.

8 The implemenaion of he Phong game showed ha he gameplay of he spaial game is ineresing. As a nex sep i is good o reduce he delay o a minimum. Afer his improvemen i is ineresing o creae a more complex game wih an elaborae user inerface. 5 CONCLUSION In his paper, we described a real-ime compuer vision based applicaion. The proposed sysem is composed of wo pars. The firs par is video-based people deecion, racking and pose recogniion sysem. I direcly uses he capured video frame as inpu, hen gives he 2D posiion and pose of he people if here are people appearing in he scene. The posiion and pose informaion is conneced o he second par, a spaial game sysem, and used as he conrol command of he game. This pose-driven spaial game is a real ime man-machine ineracion wihou obrusive sensors. I shows he possibiliy of a new way of ineracions in novel compuer games and enerainmen. The combinaion of compuer vision research and a pracical applicaion is quie useful. I allows us o direcly es if he proposed algorihm saisfies cerain requiremens, in a specific applicaion environmen. Fuure work will include improving he robusness of he sysem (e.g. beer skin color deecion, more robus feaure deecion) and developing muliple-user applicaions. One of he challenges will be o solve he occlusion problem if users are allowed o move freely. ACKNOWLEDGMENTS This research has been suppored by he GATE (Game Research for Training and Enerainmen) proec, funded by he Neherlands Organizaion for Scienific Research (NWO) and he Neherlands ICT Research and Innovaion Auhoriy (ICT Regie). The spaial game is based on he proposal of Berend Berendsen. REFERENCES [1] Moeslund, T. B. Hilon, A. and Kruger, V. (2006). A survey of advances in vision-based human moion capure and analysis. In Compuer Vision and Image Undersanding, vol. 104, pages , [2] Wren, C. Azarbayean A. Darrell, T. and Penland, P. (1997). Pfinder: real-ime racking of he human body. In IEEE Transacion on Paern Analysis and Machine Inelligence, vol.19, no.7, pages , [3] Gupa, A. Mial, A. Davis, L. S. (2008). Consrain inegraion for efficien muliview pose esimaion wih self-occlusion. In IEEE Transacions on Paern Analysis and Machine Inelligence, pages , [4] Rogez, G. Rihan, J. Ramalingam S. and ec, (2008). Randomized rees for human pose deecion. In Proceedings IEEE Compuer Sociey Conference on Compuer Vision and Paern Recogniion, IEEE, June [5] Sauffer, C. and Grimson, W. (1999). Adapive background mixure models for real-ime racking. In Proceedings IEEE Compuer Sociey Conference on Compuer Vision and Paern Recogniion, vol. II, pages , [6] Miciloa, A. (2005). Deecion and racking of humans for visual ineracion. In PhD. Disseraion, School of Elecronics and Physical Sciences, Universiy of Surrey, [7] Porikl F.M. and Tuzel, O. (2003). Human body racking by adapive background models and mean-shif analysis. In IEEE Inernaional Workshop on Performance Evaluaion of Tracking and Surveillance, [8] PRTools oolbox hp://prools.org and PRSD Sudio hp://prsdsudio.com sofware packages. [9] hp://opencv.willowgarage.com/wiki/ [10] hp://prsysdesign.ne/index.php/hml/blog_commens/embedding_classifi [11] Huo, F. Hendriks, E.A. Paclik, P. and Oomes, A.H.J. (2009). Markerless human moion capure and pose recogniion. In Inernaional Workshop on Image Analysis for Mulimedia Ineracive Services (WIAMIS), 2009.

9 [12] Berendsen, B. (2008). Tracking and 3D body model fiing using muliple cameras. Maser s hesis, UU, Deparmen of Compuer Science, INF/SCR , [13] hp:// [14] hp:// [15] hp://en.wikipedia.org/wiki/inerne_socke