Occlusion-Free Hand Motion Tracking by Multiple Cameras and Particle Filtering with Prediction

Transcription

1 58 IJCSNS Inernaional Journal of Compuer Science and Nework Securiy, VOL.6 No.10, Ocober 006 Occlusion-Free Hand Moion Tracking by Muliple Cameras and Paricle Filering wih Predicion Makoo Kao, and Gang Xu Deparmen of Media Technology, Risumeikan Universiy, Kusasu, Japan Summary This paper proposes a new echnique o simulaneously esimae he global hand pose and he finger ariculaion imaged by muliple cameras. Tracking a free hand moion agains a cluered background is a difficul ask. The firs reason is ha hand fingers are self-occluding and he second reason is he high dimensionaliy of he problem. In order o solve hese difficulies, we propose using calibraed muliple cameras and a he same ime improving search efficiency by prediced paricle filering. Therefore our mehods can cope wih boh rapid global hand moion and self-occlusion. We also add predicion o paricle filering so ha more paricles are generaed in areas of higher likelihood, which reduces search cos significanly. The effeciveness of our mehod is demonsraed by racking free hand moions in real image sequences. Key words: Ariculaed hand racking, Moion, Gesure recogniion, Paricle filering, Moion capure. Inroducion Recenly, hand gesure recogniion and hand moion racking have become imporan issues in he field of human-compuer ineracion. Many vision-based approaches have been proposed [1]-[7] [14] [17]. The hand racking mehods by vision can be divided ino wo caegories. One is appearance-based, and he oher is model-based. In he appearance-based mehods, mapping beween image feaures and hand pose is esablished firs, and hand pose esimaion is formulaed as an image daabase indexing problem, where he closes maches for an inpu hand image are rerieved from a large daabase of synheic hand images []. The problem wih he appearance-based mehod is he requiremen of a very large daabase. In conras, he model-based mehods use an ariculaed hand model. The hand pose a he curren frame is esimaed from he curren image inpu and previous pose. The problem of using a hand model is he high dimensionaliy. The high dimensionaliy causes an exponenially high compuaional cos. Paricle filering is one of he mos successful objec racking algorihms [9] [10]. However, o keep racking correcness especially for rapid moions, i needs a large number of paricles. Anoher problem wih he model-based approach is self-occlusion. While a hand moves freely, pars of he hand change from being visible o being invisible, and hen becoming visible again. Previously proposed echniques avoid his problem by resricing hand moions o only hose ha are fronal o he camera [1] [3]-[5] [17]. To overcome his resricion, we propose o use muliple pre-calibraed cameras, so ha pars invisible in one camera are sill visible in a leas anoher camera. While his is he righ approach o he self-occlusion problem, more observaions pu more burdens on he already busy compuer. This moivaes furher improvemen of search efficiency. Alhough Rehg and Kanade [14] proposed o use muliple cameras for finger racking, hand moion in his paper is much more complicaed and we need o design and implemen more efficien mehod. Since i is infeasible o mainain dense sampling in high dimensional sae spaces, wo mehods have been proposed o solve hese problems. One is o reduce he sae dimensionaliy and he oher is o improve sampling and o make beer predicion. We reduced he dimensions of he hand moion space by PCA and furher perform independen componen analysis (ICA) o exrac local feaures as ICA basis vecors [1]. To improve sampling efficiency, Rui e al. propose Unscened Paricle Filer (UPF) [13]. The UPF uses he unscened Kalman filer o generae sophisicaed proposal disribuions ha seamlessly inegrae he curren observaion, hus grealy improving he racking performance. This mehod needs o esablish a sysem dynamics model. As for our 6-DOF problem, i is even hard o esablish he sysem dynamics model. Deuscher e al. propose annealed paricle filering which is modified for searches in high dimensional sae spaces [11]. I uses a coninuaion principle, based on annealing, o inroduce he influence of narrow peaks in he finess funcion, gradually. I is shown o be capable of recovering full ariculaed body moion efficienly. However, he experimen is done agains a black background. Bray e al. propose smar paricle filering which combines he Sochasic Mea-Descen (SMD), based on gradien descen wih paricle filering [17]. Their 3D hand racking resul is robus and accurae. However, hey need deph Manuscrip received Ocober 18, 006. Manuscrip revised Ocober 5, 006.

2 `IJCSNS Inernaional Journal of Compuer Science and Nework Securiy, VOL.6 No.10, Ocober maps generaed by a srucured ligh 3D sensor, which are no available in real ime. We propose o add predicion o paricle filering. Parameers in he nex frame are prediced and more paricles are accordingly generaed for areas of higher likelihood. The mehod is sraighforward bu proven o very effecive in significanly reducing search cos. This paper is organized as follows: Secion describes hand model, and briefly shows how is dimension is reduced by ICA. Secion 3 presens racking by muliple cameras. Secion 4 presens paricle filering wih predicion. Secion 5 presens experimenal resuls. Conclusion is given in Secion 6. We proposed an ICA-based represenaion of hand. Hand Model and Dimension Reducion by ICA In our sudy, a hand is rendered in OpenGL using spheres, cylinders, and recangular parallelepiped. A hand can be described in his way: he base is a palm and five fingers are aached o he palm. Each finger has 4 DOF. of 4 DOF correspond o he meacarpophalangeal join (MP) and is abducion (ABD). The oher DOF correspond o he proximal inerphalangeal join (PIP) and he disal inerphalangeal join (DIP). I is shown in Fig. 1. Therefore, our hand model has 0 DOF. In addiion, o represen he posiion and orienaion of a hand, we need 6 more parameers, 3 for posiion and 3 for orienaion. In oal, he hand model has 6 DOF. Figure. The ICA-based represenaion of hand ariculaion. (a)-(e) ICA basis moion, respecively. ariculaion [1]. I compresses he dimensionaliy of hand ariculaed moion very efficienly. Each ICA basis vecor represens a finger moion. Thus he 0-DOF hand model can be represened by 5 DOF using his represenaion. Ariculaed hand moion is learned from he raining daa capured by a daa glove. Firs, we perform PCA o reduce he dimensionaliy. Then we perform ICA o exrac local finger moions. Each local finger moion corresponds o a paricular finger moion as shown in Fig.. The oal dimension reduces o 11, wih 5 for finger ariculaion, and 6 for global hand moion. 3. Tracking by Muliple Cameras (a) (b) Figure 1. (a) Hand model wih he name of each join, and he degrees of freedom (DOF). (b) Hand model rendered in OpenGL. We perform camera calibraion so ha he inrinsic parameers and posiions and orienaions of he cameras recovered [15]. Once he cameras are calibraed, a hand model is projeced ono he images, and he projeced images are compared wih real observaions so ha he parameers of he hand model can be esimaed. Since calibraed cameras do no increase unknown parameers, more images do no mean more parameers. They merely bring more informaion. In our currenly experimens, we use wo cameras looking a he hand, wih he wo cameras separaed by roughly 90 degrees. This brings a grea improvemen over using a single camera, and is sufficien in handling occlusions.

3 60 IJCSNS Inernaional Journal of Compuer Science and Nework Securiy, VOL.6 No.10, Ocober 006 Figure 4. (a) Inpu image (b) Disance map of edge observaion (c) Exraced silhouee Figure 3. Relaion beween wo cameras. 3.1 Relaion Beween he Hand Model and Two Cameras The relaion beween wo cameras is drawn as follows. The 3D coordinae sysem cenered a opical cener of camera 1 is X. The 3D coordinae sysem cenered a opical cener of camera is X. As depiced in Fig. 3, he roaion marix and he ranslaion vecor from he coordinae sysem of camera 1 o he coordinae sysem of camera are Rc, c. Then he relaion beween he wo coordinae sysems is given by X= R X +. (1) The 3D coordinae sysem of hand model is X m. The roaion marix and he ranslaion vecor from he coordinae sysem of hand model o he coordinae sysem of camera 1 are Rw, w. Then he relaion beween he wo coordinae sysems is given by X = R X+. () c m w w From (1) and (), we can ransform X m o X and X, and hen projec he hand model ono he images. 3. Observaion Model We employ edge and silhouee informaion o evaluae he hypoheses. For edge informaion, we employ he Chamfer disance funcion []. Firs, we perform Canny edge deecion o he inpu image. In he resul image of edge deecion, he edge pixels are black and oher pixels are whie. Then, a each pixel, we calculae he disance from each pixel o he closes edge poin by using disance ransformaion. If he disance is over a hreshold, he c Figure 5. Areas of silhouee measuremens. Black areas are he corresponding areas. (a) ai ao, (b) am ao, (c) ai am. Noe ha a I is he silhouee of inpu image, a M is he silhouee of hand model, and a O is he silhouee of overlap. disance is se o he hreshold. A disance map of he inpu image is obained. Fig. 4 (b) shows an example of disance map. Then we projec he edge of he hand model ono he disance map. We add all disances along he edge poins of he projeced hand model, and calculae he average of disances. Then he likelihood from he edge informaion is ( avadis) pedge( z x ) exp, (3) σ edge where avadis is he average of disances. In order o exrac he silhouee of a hand region, we conver image color space from RGB o HSV (hue, sauraion and brighness). Then he skin color region is exraced by using a hreshold. Fig. 4 (c) shows an exraced silhouee. We calculae subracions of he area of silhouee. The hree calculaed subracion resuls are a shown in Fig. 5. The subracions of I ao and

4 `IJCSNS Inernaional Journal of Compuer Science and Nework Securiy, VOL.6 No.10, Ocober am ao are used o measure he similariy of he hand a a posiion. The subracion of I M is used o measure he similariy of hand finger pose. Then likelihoods from he silhouee informaion are ( ai a ) O psilio ( z x ) exp (4) σ silio p p ( am a ) O ( z x ) exp (5) σ silmo silmo ( ai a ) M ( z x ) exp. (6) σ silim silim Thus he final likelihood is p( z x ) p p p p. (7) edge silio silmo silim When we use muliple cameras, he likelihood is n pi z i= 1 pz ( x ) ( x ), (8) where n is he number of cameras. 4. Paricle Filering wih Predicion 4.1 Paricle Filering The paricle filering algorihm [16] [18] is a sequenial Mone Carlo mehod. The algorihm is powerful in approximaing non-gaussian probabiliy disribuions. Paricle filering is based on sequenial imporance sampling and Bayesian heory. Wih paricle filering, coninuous disribuions are approximaed by discree random sample ses, which are composed of weighed paricles. The paricles represen hypoheses of possible soluions and he weighs represen likelihood. There are hree main seps in he algorihm: resampling, diffusion, and observaion. The firs sep selecs he paricles for reproducion. In his sep, paricles ha have heavier weighs are more likely o be seleced. Heavy-weigh paricles generae new ones, while lighweigh paricles are eliminaed. The second sep diffuses paricles randomly. A par of space ha is more likely o have a soluion has more paricles, while a par of space ha is less likely o have a soluion has fewer paricles. The hird sep measures he weigh of each paricle Figure 6. One ime-sep in paricle filering. There are 3 seps, resampling-diffusion-observaion. according o an observaion densiy. Fig. 6 shows a picorial descripion of paricle filering. 4. Paricle Filering wih Predicion The classical paricle filering requires an impracically large number of paricles o follow rapid moions and o keep racking correc. I becomes a serious problem when he racking arge has a high dimensional sae space like hand racking. In order o ackle his problem, we propose using predicion o generae beer proposal disribuions. According o he Bayes rule, he hand pose of he curren frame x can be esimaed from he prior hand pose x 1 as p( x z1: ) p( z x) p( x z1: 1), (9) where p( x z ) = p( x x ) p( x z ), (10) x 1: : 1 1 z is he observaion of he curren frame, pz ( x ) is he likelihood disribuion and p( x x 1) is he ransiion probabiliy disribuion. (9) can be inerpreed as he equivalen of he Bayes rule: p( x z) p( z x) p( x ). (11) In paricle filering, he sequence of probabiliy disribuions is approximaed by a large se of paricles. Therefore, how o propagae he paricles efficienly in areas of higher likelihood significanly affecs racking resuls. The paricles are defined as follows: in order o represen a poseriori p( x z ) 1:, we employ a imesamped sample se, denoed ( n { s ), n= 1, L, N}. The sample se is weighed by he observaion densiy

5 6 IJCSNS Inernaional Journal of Compuer Science and Nework Securiy, VOL.6 No.10, Ocober 006 Figure 7. Tracking resul by wo cameras. (a) Camera 1 view. (b) Camera view. The projecion of hand model s edge is drawn on he images by red lines. The CG models are examples of some corresponding hand models. π = pz ( x = s ), where he weighs ( n) ( n) ( n) normalized so ha π ( n) = N π are 1. Then he sample se ( n) ( n) { s, π } represens he poseriori p( x z1: ). The sample se of he poseriori is propagaed from ( n) ( n) { s 1, π 1} which represens p( x 1 z1: 1) as shown in Fig. 6. The ransiion probabiliy disribuion p( x x 1) affecs p( x 1: 1), which in urn affecs p( x z1: ). z

6 `IJCSNS Inernaional Journal of Compuer Science and Nework Securiy, VOL.6 No.10, Ocober Figure 8. (a) Resul 1 by single camera. (b) Resul by single camera. (c) Resul by wo cameras. In paricle filering, p( x x 1) is modeled by a dynamical model. The simples dynamical model [1] [4]- [6] [11] [17] is s = s + B, (1) ( n) ( n) 1 where B is a mulivariae Gaussian disribuion wih covariance P and mean 0. However, his simple dynamical model does no propagae he paricles efficienly and many paricles are wased in areas of lower likelihood. Figure 9. (a) Tracking resul wihou predicion. (b) Tracking resul wih predicion. To overcome hese difficulies, we simply use he firs-order approximaion of Taylor series expansion for predicion: ( n) ( n) ( n) s 1 s = s 1 + Δ + B. (13) We also ried o use he second-order approximaion of Taylor series expansion ( n) ( n) ( n) ( n) s 1 1 s 1 s = s 1 + Δ + Δ + B. (14)

7 64 IJCSNS Inernaional Journal of Compuer Science and Nework Securiy, VOL.6 No.10, Ocober 006 Figure 10. Trajecory of he roaion around Y axis (uni: degrees). However, he racking ges rapped in local minima. The reason is ha he second derivaive canno be esimaed accuraely due o noise. 5. Experimenal Resuls The performance of our mehod was esed by using real image sequences. A movie (avi) file of he resuls is available on he web a hp:// We manually iniialize he hand model o mach roughly wih he hand a he firs frame. Then he algorihm auomaically racks he hand while i moves freely. 5.1 Tracking Rapid Moions agains a Cluered Background Fig. 7 shows he racking resul of our mehod. The sequences include rapid moion, large roaions angle agains a camera, occlusions and a cluered background. The experimen was run using paricles per frame. The racking correcly esimaed hand posiion and moion hroughou he sequence. In he following subsecions, we compare our mehod o he mehod by a single camera, and he mehod wihou predicion. 5. Tracking by a Single Camera If we use only a single camera, he racking resul becomes Fig. 8 (a) and (b). We ried he experimen by a single camera o wo image sequences respecively. Fig. 8 (c) is he resul by wo cameras. In Fig. 8 (a), a frame 50, he hand orienaion is slighly incorrec and hen he error becomes larger, finally, a frame 60, he hand orienaion is compleely incorrec. In Fig. 8 (b), a frame 50, he hand orienaion is incorrec and hen he error becomes larger, finally a frame 60, he racking esimaed ha he hand fingers exis a he hand wris posiion. From he resuls, we can see ha occlusion is a severe problem for racking by a single camera bu is no a problem for muliple cameras. 5.3 Tracking Wihou Predicion In he nex experimen, we compared he mehods wih predicion and wihou predicion. Fig. 9 (a) is he resul wihou predicion and Fig. 9 (b) is ha wih predicion. In Fig. 9 (a), a frame 50, he hand orienaion is slighly incorrec, and hen he error becomes larger and finally, a frame 60, he racking esimaed ha he hand is upside down comparing wih he real hand. 5.4 The Number of Paricles We also did experimens wih differen numbers of paricles per frame in order o wach how many paricles are suiable for he racking. We show he rajecory of he roaion around Y axis in Fig. 10. We did he experimen wih 500 paricles, 3000 paricles, paricles and paricles. The resuls have dramaic change when we increase he number of paricles from 500 o And he resuls only have sligh change when we increase he number of paricles from o Therefore, is he opimized number of paricles for his hand moion.

8 `IJCSNS Inernaional Journal of Compuer Science and Nework Securiy, VOL.6 No.10, Ocober Conclusions In his paper we proposed an ariculaed hand moion racking by muliple cameras. This mehod is useful for gesure recogniion. Tracking a free hand moion agains a cluered background was unachievable in previous mehods because hand fingers are self-occluding. To improve search efficiency, we proposed adding predicion o paricle filering so ha more paricles are generaed in areas of higher likelihood. The experimenal resuls show ha our mehod can correcly and efficienly rack he hand moion hroughou he image sequences even if hand moion has large roaion agains a camera. References [1]M. Kao, Y. W. Chen, and G. Xu, Ariculaed Hand Tracking by PCA-ICA approach, IEEE Proc. FG06, pp , 006. [] B. Senger, A. Thayananhan, P. H. S. Torr, and R. Cipolla, Filering using a ree-based esimaor, Proc. ICCV03, pp , Nice, France, Ocober 003. [3] H. Zhou and T. S. Huang, Tracking Ariculaed Hand Moion wih Eigen Dynamics Analysis, Proc. ICCV03, pp , Nice, France, Ocober 003. [4] Y. Wu, John Lin, and T. S. Huang, Capuring naural hand ariculaion, Proc. ICCV01, pp.46-43, Vancouver, July 001. [5] Y. Wu, John Lin, and T. S. Huang, Analyzing and Capuring Ariculaed Hand Moion in Image Sequences, IEEE Trans. PAMI, Vol. 7, No. 1, December 005. [6] W. Y. Chang, C. S. Chen, and Y. P. Hung, Appearance- Guided Paricle Filering for Ariculaed Hand Tracking, Proc. CVPR05, Vol. 1, pp.35-4, June 005. [7] J. Lee and T. Kunii, Model-based analysis of hand posure, IEEE Compuer Graphics and Applicaions, 15: pp.77-86, Sep [8] M. Isard and A. Blake, Conour racking by sochasic propagaion of condiional densiy, Proc. of European Conf. on Compuer Vision, pp , Cambridge, UK, [9] M. Isard and A. Blake, CONDENSATION-condiional densiy propagaion for visual racking, In. J.Compuer Vision, [10] A. Blake and M. Isard, Acive Conours. Springer, London, [11] J. Deuscher, A. Blake, and I. Reid, Ariculaed Body Moion Capure by Annealed Paricle Filering, Proc. CVPR00, Vol., pp , 000. [1] C. Sminchisescu and B. Triggs, Covariance Scaled Sampling for Monocular 3D Body Tracking, Proc. CVPR01, Vol. 1, pp , 001. [13] Y. Rui and Y. Chen, Beer Proposal Disribuions: Objec Tracking Using Unscened Paricle Filer, Proc. CVPR01, Vol., pp , 001. [14] J. Rehg and T. Kanade, Model-Based Tracking of Self Occluding Ariculaed Objecs, Proc. ICCV95, pp , [15] J. Y. Bougue, MRL-Inel Corp. Camera Calibraion Toolbox for Malab, available a hp:// [16] P. M. Djuric, J. H. Koecha, J. Zhang, Y. Huang, T. Ghirmai, M. F. Bugallo, and J. Miguez, Paricle filering, IEEE Signal Processing Magazine, pp.19-38, Sep [17] M. Bray, E. K. Meier, and L. V. Gool, Smar Paricle Filering for 3D Hand Tracking, IEEE Proc. FG04, pp , 004. [18] M. Sanjeev Arulampalam, S. Maskell, N. Gordon, and T. Clapp, A Tuorial on Paricle Filers for Online Nonlinear/Non-Gaussian Bayesian Tracking, IEEE Trans. Signal Processing, Vol. 50, No., pp , Feb 00. Makoo Kao received he B.S. and M.S. degrees in Compuer Science from Risumeikan Universiy in 001 and 003, respecively. He is currenly a Ph.D. suden in Risumeikan Universiy. His research ineress include saisical paern recogniion, compuer vision, and machine learning. Gang Xu is a professor of compuer science, Risumeikan Universiy. He received his Ph.D. degree from Osaka Universiy in 1989, and since hen he has held eaching and research posiions in Osaka Universiy, Risumeikan Universiy, Harvard Universiy, Microsof Research Asia and Moorola Ausralian Research Cenre. He also founded 3D MEDiA Company Limied in 000, specialized in 3-dimensional image processing and phoogrammery. He auhored and co-auhored 3 books in compuer vision.