A Fast Stereo-Based Multi-Person Tracking using an Approximated Likelihood Map for Overlapping Silhouette Templates

Transcription

1 A Fas Sereo-Based Muli-Person Tracking using an Approximaed Likelihood Map for Overlapping Silhouee Templaes Junji Saake Jun Miura Deparmen of Compuer Science and Engineering Toyohashi Universiy of Technology Toyohashi, Japan {saake, Absrac This paper describes a mehod of racking muliple persons wih occlusions using sereo. Many previous sereo-based sysems rack each person separaely and do no explicily handle such occlusions. We previously developed an accurae, sable racking mehod using overlapping silhouee emplaes which considers how persons overlap in he image. However, because he mehod uses a paricle filer, a lo of processing ime is needed for esimaing each paricle s likelihood by comparing many emplaes wih he image. In his paper, we propose a new mehod which can decrease he number of image comparison by using an approximaed likelihood map based on kernel densiy esimaion. Experimenal resuls show ha he proposed mehod is able o reduce he processing ime grealy wihou dropping he racking performance. Index Terms person racking; paricle filer; sereo camera; silhouee emplaes; kernel densiy esimaion; I. INTRODUCTION Following a specific person is an imporan ask for service robos. Visual person following in public spaces enails racking of muliple persons by a moving camera. There have been a lo of works on person deecion and racking using various image feaures and classificaion mehods [1] [4]. Many of hem, however, use a fixed camera. In he case of using a moving camera, foreground/background separaion is an imporan problem. This paper deals wih deecion and racking of muliple persons which will be used on a mobile robo. Laser range finders are widely used for person deecion and racking by mobile robos [5], [6]. Image informaion such as color and exure is, however, someimes necessary for person segmenaion and/or idenificaion. Omnidirecional cameras are also used [7], [8], bu heir limied resoluions are someimes inappropriae for analyzing complex scenes. Sereo is also popular in moving objec deecion and racking [9] [11]. In hese works, however, occlusions beween people are no handled. Ess e al. [12], [13] proposed o inegrae various cues such as appearance-based objec deecion, deph esimaion, visual odomery, and ground plane deecion using a graphical model for pedesrian deecion. Alhough heir mehod exhibis a nice performance for complicaed scenes, i is sill cosly o be used for conrolling a real robo. Some mehods o rack muliple objecs by using paricle filer are proposed [14] [16]. In hese mehods, racking of muliple ineracing arges is realized by adding a probabilisic exclusion principle. They deal wih he case where mos par of each arge is visible alhough small occlusions occur very ofen. In our case, since he images are aken from a camera on a mobile robo, complee occlusions frequenly occur. To rack overlapping persons sably, we proposed a mehod using overlapping silhouee emplaes [17], [18], which considers how persons overlap in he image. The remainder of his paper is organized as follows. Secion II describes our previous mehod [17], [18] using overlapping silhouee emplaes. In Secion III, we propose an improvemen of speeding up processing by using an approximaed likelihood map based on kernel densiy esimaion. Secion IV presens experimenal resuls, and finally Secion V presens conclusions and fuure work. II. MULTI-PERSON TRACKING USING STEREO A. Person racking based on disance informaion To rack persons sably wih a moving camera, we use deph emplaes [17], which are he emplaes for human upper bodies in deph images (see Fig. 1). We currenly use hree emplaes wih differen direcion of body. We made he emplaes from he deph images where he arge person was a 2 [m] away from he camera. A deph emplae is a binary emplae, he foreground and he background value are adjused according o he saus of racks and inpu daa. For a person being racked, his/her prediced scene posiion is available from he sae variable (see Sec. II-B). We hus se he foreground deph of he emplae o he prediced deph of he head of he person. Concerning he background deph, since i may change as he camera moves, we esimae i on-line. We make he deph hisogram of he curren inpu deph image and use he Kh percenile as he background deph (currenly, K =90). 392

2 Lef Fig. 1. Fron Righ Deph emplaes Fig. 3. (a) Inpu image Fig. 2. (b) Deph image Deecion example using deph emplaes Fig. 4. For a deph emplae T (x, y) of H W pixels and he deph image ID (x, y), he dissimilariy d is calculaed as follows. 1 [T (p, q) ID (x+p, y+q)]2. d= HW p q We use he hree emplaes simulaneously and ake he one wih he smalles dissimilariy as a maching resul. Figure 2 shows an example of deecion using he deph emplaes. Three recangles in he deph image are deecion resuls wih he hree emplaes, and he one wih he smalles dissimilariy is shown in bold line. Even when he direcion of he body changed, i is possible o deec person sably by using he muliple emplaes. B. Esimaion of 3D posiion using paricle filer Figure 3 illusraes he coordinae sysems aached o our mobile robo and sereo sysem. In he robo coordinae sysem, he person s posiion a ime is defined as (X, Y, Z ). The sae variable x is defined as x Definiion of coordinae sysems = [ X Y Z X Y ]T, where X and Y denoe velociies in he horizonal plane. We assume he verical posiion is consan. The sae equaion is given by x+1 = F x + w, where marix F is se o represen a linear uniform moion. We esimae he 3D posiion of each person by using paricle filer. The likelihood L of each paricle is calculaed based on he dissimilariy d described in Sec. II-A. d2 1 exp 2, L = 2σ 2πσ where he sandard deviaion σ is se up experienially. Person s posiion is calculaed by he weighed average of paricles. We use an OpenCV implemenaion of paricle filer. Procedure of racking using an overlapping silhouee emplae C. Muli-person racking using overlapping silhouee emplaes To rack overlapping persons sably, we make overlapping silhouee emplaes [18] which consider he overlap of persons in he image. Each person which is isolaed from oher persons is independenly racked by using N paricles. When wo persons, say A and B, approach each oher and an overlap occurs, a new combined sae vecor x for boh persons is B made from he respecive ones xa and x. xa. = x xb Since he number of paricles of each person is N, he oal number of combined sae is N N. To reduce he calculaion cos of emplae maching for he combinaions, we use only paricles wih large likelihood values among each person s paricles. We se he number of each person s paricles o N = 100, and he number of combined paricles o N = The iniial likelihood of each combined paricle is se as produc of heir likelihood values L = LA LB. The sae equaion is as follows. F 0 w x+1 = x +. w 0 F Figure 4 shows he procedure of racking using an overlapping silhouee emplae. The emplae of each combined paricle is made in consideraion of he saes of wo persons. The relaive posiion in he image coordinaes is calculaed, and he individual emplae of he person near he camera is overwrien on he emplae of he far person. The values for he foregrounds and he background are se similarly o he case of one foreground case in Sec. II-A. Only one emplae among hree (see Fig. 1), corresponding o he esimaed movemen direcion is used for reducing he number of combined emplaes. The overlapping silhouee emplae is mached o he deph image, and he likelihood L is calculaed. 393

3 (a) Previous mehod Fig. 6. Four facors of silhouee emplae (b) Proposed mehod based on kernel densiy esimaion Fig. 5. Esimaion of each paricle s likelihood When he disance beween persons A and B exceeds a hreshold, he sae variable x is separaed ino x A and x B using he N paricles wih he larges likelihood values among N paricles. Our previous work realized an accurae and robus racking using overlapping silhouee emplaes [18]. However, he processing speed is abou 300 [ms/frame] for esimaing every paricle s likelihood by comparing many emplaes wih he image. In Secion III, we propose a new mehod which can decrease he number of image comparison by using an approximaed likelihood map based on kernel densiy esimaion. III. APPROXIMATION OF LIKELIHOOD MAP BASED ON KERNEL DENSITY ESTIMATION Figure 5 shows he ouline of our proposed mehod. The previous mehod needs a lo of processing ime because he combined emplae is compared wih he image for all paricles. Therefore, we propose a new mehod which selecs a par of paricles, compares heir emplaes wih he image, and makes an approximaed likelihood map by using he comparison resuls based on kernel densiy esimaion. The oher paricles likelihoods are hen esimaed by using he approximaed likelihood map. As shown in Figure 6, he silhouee emplae is made based on four facors: wo persons dephs X A and X B, horizonal disance Y A Y B, and heigh difference Z A Z B. In his paper, we consider he disance Y A Y B and assume ha deph X A, X B, and heigh difference Z A Z B are consan under a shor occlusion. The posiion ( X N, Ȳ N, Z N ) and ( X F, Ȳ F, Z F ) denoe he near and far person s posiion which are calculaed by he weighed average of paricles. As shown in Figure 7(a), we examine he change of he evaluaion value when horizonal posiions Y N and Y F change; his is equivalen o examining he change of he disance Y A Y B and he emplae s horizonal posiion. The search range is he near person s posiion ±300 [mm] and he far person s posiion ±900 [mm] as follows: X N X F X N Y N Z N = Ȳ N + δy N Z N, X F Y F Z F = Ȳ N + δy F Z F where δy N =[ 300, 300],δY F =[ 900, 900]. To deec sably he occluded far person who appears from he side of he near person, he far person s search range is se o boh sides of Ȳ N. Figure 7 (b) shows he evaluaion value of emplae maching. Each person s posiion is sampled a inervals of 30 [mm]. This likelihood map is considered a complee reference able ha indicaes he evaluaion value for each combinaion of person posiions. I is confirmed ha he evaluaion value a he correc posiion is he highes and falls as he posiion shifs. However, i needs a lo of processing ime o make his reference able because 1281 (= 21 61) emplaes are mached. To reduce he processing ime, we approximae a likelihood map by a less number of emplae maching. A firs, A cerain number of paricles are seleced wih large likelihood values a he previous frame and heir emplaes are mached wih he image o obain he evaluaion values (Fig. 7(c)). We se he number of seleced paricles o 30 or 50 in he experimen. The near and he far person s posiions and he likelihood of paricle i are denoed as (Xi N,YN i,zi N), (XF i,yf i,zi F ) and L i, respecively. Using a kernel densiy esimaion wih a Gaussian disribuion, an approximaed likelihood map is made as follows. L ( δy N,δY F ) = L i exp i { (δy N μ N i )2 2σ N 2 (δy F μ F i )2 2σ F 2, }, where he averages of disribuions are μ N i = Yi N Ȳ N, μ F i = Yi F Ȳ N. The sandard deviaions σ N and σ F are se empirically based on he size of acual person and he overlap of wo persons (σ N =40[mm], σ F = 100 [mm]). Moreover, he coefficien par of disribuion is omied because only 394

4 (a) Inpu image and search area (b) Reference able and evaluaion value Fig. 7. (c) Evaluaion value of 30 paricles Approximaion of likelihood map baed on kernel densiy esimaion racking is almos he same as he mehod (a). The posiional error slighly grows in he verical direcion because we consider he heigh difference is consan. For example, he far person s posiion shifs downward in Figure 8 #119. (c) When he approximaed likelihood map is used, he processing ime is reduced grealy while keeping a similar level of success rae. In he case of 50 paricles seleced as reference poins, he success rae is almos he same as he mehod (a) and (b). When 30 paricles were used, he success rae fell a lile. I is necessary o examine he paricle selecion sraegy o make he likelihood map more effecively. he raio is necessary for he calculaion of each paricle s likelihood. Approximaed likelihood maps are shown in Figure 7(d). Similar shapes o he reference ables of Figure 7(b) are obained. Each paricle s likelihood is esimaed by using on his approximaed likelihood map insead of maching beween he emplae and he inpu image. We realize a faser racking by using his approximaed likelihood map which is made by a less number of emplae maching. IV. E XPERIMENTAL RESULT We have implemened he proposed mehod wih a Bumblebee2 sereo camera (by Poin Grey Research) and a noe PC (Core2Duo, 3.06 [GHz]). The processed image size is Because he mehod does no consider he change in deph and heigh of each person, we used es daa ses in which persons roughly move in parallel o he image plane. Figure 8 is a resul of racking using off-line images aken a 10 [fps]. Each pair of circles wih whie edges shows a racking resul by using he combined sae variable. Blue lines show search area of likelihood map (see Fig. 7). Even when one person is occluded by he oher, each person s posiion can be esimaed sably. Table I shows he comparison of racking resuls for 140 occlusion cases (14 daa ses were esed en imes respecively). The number of oal frames is = Each es daa se is he off-line images in which wo person approach each oher, inersec, and hen depar. Each person s posiion (ground ruh) in each frame was given manually. We couned success cases where every person was racked correcly a all frames and calculaed he success rae. The averages of he 2D posiional error and he ime of processing a frame were calculaed for only he success daa ses. (a) Our previous work [18] realized a sable racking. However, i akes a lo of processing ime o mach he emplae wih each 625(= 252 ) paricle. (b) When he reference able (see Fig. 7(b)) is used, 1281(= 21 61) emplaes are mached. The success rae of (d) Approximaed likelihood map Figure 9 is an example of he racking resul when he persons dephs change. The racking fails because we considered only he change of horizonal posiions and assumed ha he dephs and he heighs are consan. In he fuure, we should expand dimensions for deph and heigh, and realize a sable racking in a more general siuaion. V. C ONCLUSION This paper has described a mehod of racking muliple persons by using overlapping silhouee emplaes. We proposed a mehod which can decrease he number of image comparison by using an approximaed likelihood map based on kernel densiy esimaion. We have reduced he number of emplae maching o abou 1/10 wihou degrading he racking performance, and he oal processing ime o abou 100 [ms/frame]. This processing speed is pracical for a person following robo. However, we considered only he change of horizonal posiions and assumed ha he dephs and he heighs are consan. In he fuure, we should expand dimensions for deph and heigh, and realize a sable racking in a more general siuaion. To make he approximaed likelihood map more effecively, i is necessary o examine he paricle selecion sraegy. Moreover, i is necessary o deal wih he overlapping of hree persons or more. 395

5 Fig. 8. Fig. 9. Experimenal resul of racking Example of racking failure because of change in deph TLE I C OMPARISON OF TRACKING RESULTS racking mehod emplae maching processing ime (maching) processing ime (oal) (a) previous mehod [18] (b) using reference able (c) proposed mehod 625 [imes/frame] 1281 [imes/frame] 50 [imes/frame] 30 [imes/frame] 247 [ms/frame] 475 [ms/frame] 20 [ms/frame] 11 [ms/frame] 328 [ms/frame] 555 [ms/frame] 101 [ms/frame] 93 [ms/frame] ACKNOWLEDGMENT A par of his research is suppored by Kuraa Memorial Hiachi Science and Technology Foundaion, NEDO Inelligen RT Sofware Projec, and JSPS KAKENHI R EFERENCES [1] P. Viola, M.J. Jones, and D. Snow, Deecing pedesrians using paerns of moion and appearance, In. J. Compuer Vision, vol. 63, no. 2, pp , [2] N. Dalal and B. Briggs, Hisograms of oriened gradiens for human deecion, IEEE Conf. Compuer Vision and Paern Recogniion, pp , [3] B. Han, S. W. Joo, and L. S. Davis, Probabilisic fusion racking using mixure kernel-based Bayesian filering, 11h In. Conf. Compuer Vision, [4] S. Munder, C. Schnorr, and D. M. Gavrila, Pedesrian deecion and racking using a mixure of view-based shape-exure models, IEEE Trans. ITS, vol. 9, no. 2, pp , [5] D. Schulz, W. Burgard, D. Fox, and A. B. Cremers, People racking wih a mobile robo using sample-based join probabilisic daa associaion filers, In. J. Roboics Research, vol. 22, no. 2, pp , [6] N. Belloo and H. Hu. Mulisensor daa fusion for join people racking and idenificaion wih a service robo, IEEE In. Conf. Roboics and Biomimeics, pp , [7] H. Koyasu, J. Miura, and Y. Shirai, Realime omnidirecional sereo for obsacle deecion and racking in dynamic environmens, IEEE/RSJ In. Conf. Inelligen Robos and Sysems, pp , posiional error success rae [8] M. Kobilarov, G. Sukhame, J. Hyams, and P. Baavia, People racking and following wih mobile robo using omnidirecional camera and a laser, IEEE In. Conf. Roboics and Auomaion, pp , [9] D. Beymer and K. Konolige, Real-ime racking of muliple people using coninuous deecion, 7h In. Conf. Compuer Vision, [10] A. Howard, L. H. Mahies, A. Hueras, M. Bajracharya, and A. Rankin, Deecing pedesrians wih sereo vision: safe operaion of auonomous ground vehicles in dynamic environmens, In. Symp. Roboics Research, [11] D. Calisi, L. Iocchi, and R. Leone, Person following hrough appearance models and sereo vision using a mobile robo, VISAPP Workshop on Robo Vision, pp , [12] A. Ess, B. Leibe, and L. V. Cool, Deph and appearance for mobile scene analysis, 11h In. Conf. Compuer Vision, [13] A. Ess, B. Leibe, K. Schindler, and L. V. Cool, A mobile vision sysem for robus muli-person racking, IEEE Conf. Compuer Vision and Paern Recogniion, [14] J. MacCormick and A. Blake, A probabilisic exclusion principle for racking muliple objecs, 7h In. Conf. Compuer Vision, pp , [15] Z. Khan, T. Balch, and F. Dellaer, An MCMC-based paricle filer for racking muliple ineracing arges, 8h European Conf. Compuer Vision, pp , [16] D. Tweed and A. Calway, Tracking many objecs using subordinaed Condensaion, Briish Machine Vision Conf., pp , [17] J. Saake and J. Miura, Robus sereo-based person deecion and racking for a person following robo, ICRA Workshop on People Deecion and Tracking, [18] J. Saake and J. Miura, Sereo-Based Muli-Person Tracking using Overlapping Silhouee Templaes, 20h In. Conf. on Paern Recogniion, pp ,