Human Tracking in Thermal Catadioptric Omnidirectional Vision

Transcription

1 Proceedng of the IEEE Internatonal Conference on Informaton and Automaton Shenzhen, Chna June 20 Human Tracng n Thermal Catadoptrc Omndrectonal Vson Yazhe Tang, Youfu L, Tanxang Ba, Xaolong Zhou Department of Manufacturng Engneerng and Engneerng Management Cty Unversty of Hong Kong Kowloong, Hong Kong,Chna yztang2@student.ctyu.edu.h & meyfl@ctyu.edu.h Zhongwe L Department of Materal Scence and Engneerng Huazhong Unversty of Scence and Technology Wuhan, Hube Provnce, Chna zhongwl@ctyu.edu.h Abstract - We propose to explore a novel tracng system for human tracng n thermal catadoptrc omndrectonal (TCO) vson, whch s able to realze the survellance n all-weather and wde feld of vew condtons. In contrast, prevous human tracng system manly focuses on tracng n conventonal magng system. In ths paper, the proposed tracng method adopts the classfcaton posteror probablty of Support Vector Machne (SVM) to relate the observaton lelhood of partcle flter for effcent tracng. However, prevous wors only employ the fnal output label of SVM for classfcaton. Due to no exstng TCO vson dataset avalable n publc, we establsh a dataset ncludng TCO vdeos and extracted human samples to tran the classfer and test the proposed tracng method. Moreover, we adjust tracng wndow dstrbuton of partcle flter to ft the characterstc of catadoptrc omndrectonal vson whch s the sze of target n omn-mage depends on the dstance between target mage and the center of catadoptrc omndrectonal mage. Fnally, the expermental results show that our proposed tracng method has a stable and good performance n TCO vson tracng system. Index Terms Tracng, Thermal, Catadoptrc omndrectonal vson. I. INTRODUCTION Human tracng s a hot research topc n computer vson area, whch conssts of the man part of survellance system. In modern socety, the automatc survellance system becomes more and more popular n a large varety of applcatons. However, the prevous survellance system [] manly focuses on the conventonal vson system, whch has a lmted feld of vew and depends on the llumnaton. In ths paper, we try to explore a novel tracng system for human tracng n thermal catadoptrc omndrectonal (TCO) vson to realze survellance under all-weather and wde feld of vew condtons. The proposed novel survellance system can offer a much wder feld of vew and ndependent of llumnaton, whch permt to acqure more nformaton from the envronment. In order to overcome the llumnaton restrcton, the thermal camera s employed for t can detect the warmblooded anmal (Human) n any llumnaton condton, no matter day and nght, ran and fog. Furthermore, the prce of thermal camera has reduced dramatcally, so t becomes popular n cvlan applcaton over the recent years. To acheve a bg feld of vew, the catadoptrc omndrectonal sensor s able to meet our requrement, whch can reflect the surroundng lght (360 degree n horzon) nto a sngle camera. So, the hardware of proposed survellance system manly conssts of a thermal camera, a catadoptrc omndrectonal mrror and a stand (Fg. ). In addton, a representatve of TCO mage s gven n Fg. 2. For the merts of the proposed survellance system, t could have a wde range of applcatons, such as survellance n the arport, securty n the mltary, wld anmal conservaton and so on. Fg.. Confguraton of TCO system. Fg. 2. Representatve of TCO mage. Tracng n TCO vson has some bg challenges due to only lmted features can be used n thermal magery and severe nherent dstorton n catadoptrc omndrectonal vson. To the best of the author's nowledge, there are very lmted wors relate to detecton and tracng n TCO vson can be referenced by our wor. Prevously, researchers dd some excellent wors on detecton and tracng n thermal vson. Even tracng n thermal magng s dffcult compare to tradtonal vsble magng but drven by ts mert of llumnaton ndependent, and decreasng prce of t, more and more researchers began to focus on thermal magery vson. Recently, there have a flurry of wors on human detecton and tracng n thermal vson. [2] employed Support Vector Machne (SVM) for classfcaton and use of Kaman flter //$ IEEE 97

2 ntegrate mean shft to trac n thermal magery. In [3], the author presented a two-stage template-based method ntegrates wth an Adaboosted classfer for pedestran detecton. [4] ntegrated SVM [5][6] and Hstogram of Orented Gradent (HOG) [7] to detect the pedestran n thermal magery. In [8], the author used a generalzed expectaton-maxmzaton (EM) algorthm to separate nfrared mages nto bacground and foreground layers and ncorporate wth SVM for classfcaton, then present a graph matchng-based method to meet the tracng purpose. In ths decade, more researchers began to concentrate on omndrectonal vson tracng and detecton for ts wde feld of vew. In [9], a fsheye omndrectonal tracng system s ntroduced, the author used of optcal flow to detect the target and employed color feature based ernel partcle flter (KPF) to realze the sngle target tracng n omndrectonal vson. [0] presented a catadoptrc omndrectonal survellance system whch use of mult-bacground modelng and dynamc thresholdng to mae a outdoor tracng n clutter to spot the snper n the battlefeld. [][2] used of partcle flter ncorporates wth color feature to realze the tracng n catadoptrc omndrectonal vson. [3] ntroduced a catadoptrc omndrectonal pedestran recognton system for vehcle automaton, whch proposed a method of boosted cascade of wavelet-based classfers combned wth a subsequent texture-based neural networ. [3][4] unwarped the catadoptrc omndrectonal mage to the panoramc mage frst, and then mae the human recognton. However, I prefer to mae tracng n the orgnal catadoptrc omndrectonal mage rather than panoramc mage due to unwarp the omnmage nvolve nterpolaton whch s tme consumng. In addton, unwarppng has the rs to splt the target that locates at the border of the panoramc mage. In ths paper, we manly address the tracng n TCO vson. So, the proposed system s ntalzed manually n detecton phase. Due to only lmted features can be adopted n thermal magery, the HOG [7] feature wll be employed as descrptor for contour encodng. Le the most of tradtonal methods [4] [7] [5], our proposed method adopts SVM to classfy the extracted HOG feature n the TCO mage. More mportant s the proposed tracng method ntends to tae advantage of the SVM classfcaton posteror probablty to relate the observaton lelhood of partcle flter for effcent tracng n TCO vson (Fg. 3). For the reason of no exstng TCO vson dataset n publc, a dataset ncludng TCO vdeos and extracted human samples s formed. The extracted samples are used to tran the SVM classfer, and TCO vdeos are employed for testng. In addton, polar coordnate s adopted to ft the catadoptrc omndrectonal vson. Moreover, the tracng wndow dstrbuton of partcle flter s adjusted to ft the characterstc of catadoptrc omndrectonal vson whch s the sze of target n omn-mage depend on the dstance between target mage and center of catadoptrc omndrectonal mage. The paper s structured as follows. In Secton II, present the proposed tracng method whch conssts of partcle flter theoretcal foundaton, SVM classfcaton posteror probablty, extracted TCO samples and tracng wndow dstrbuton of partcle flter based on catadoptrc omndrectonal vson. Secton III presents and dscusses the expermental results. Fnally, the concluson wll be followed n Secton IV. Input Image Extracted HOG SVM Classfcaton Posteror Probablty Fg. 3. The framewor of proposed method Partcle Flter II. PROPOSED TRACKING ALGORITHM Tracng n TCO vson s dffcult due to there are several bg challenges have to be overcome, such as lmted features n thermal mage and nherent dstorton n catadoptrc omndrectonal vson. In ths secton, we propose to ntroduce a novel tracng method whch s very sutable to apply n TCO vson. The proposed method ntegrates SVM wth partcle flter and maes some relevant adjustments to ft the catadoptrc omndrectonal vson for effcent tracng. The detal of algorthm s shown as followng. A. Partcle Flter Partcle flter [6]-[8] s also nown as Sequental Monte Carlo methods (SMC), and t has been wdely used for ts advantage of nonlnear/non-gaussan. In Bayesan framewor, the partcle flter s recursvely obtan the state x at tme, gven the avalable observatons z z, z, : = 2, z up to tme p x z at tme - s avalable, the. Suppose posteror ( : ) posteror ( x z ) p : can be obtaned recursvely by predcton and update. The predcton stage maes use of the probablstc system transton model p ( x x ) to predct the posteror probablty of tme nstant. Predcton step: ( x z ) p( x x ) p( x z ) p dx () : = : At tme nstant, an observaton z s avalable, the posteror p ( x z : ) at tme nstant can be obtaned through update pror by Bayes s rule. Update step: ( x z ) : ( z x ) p( x z: ) p( z z ) p p = (2) : Where p ( z x ) s the observaton lelhood that nfluences the weght dstrbuton of each partcle. In ths paper, the observaton lelhood can be obtaned from the SVM classfcaton posteror probablty. At tme nstant, each partcle represents a hypothetcal state x, wth correspondng weght w. So, the prncple of partcle flter s to utlze fnte dscrete N weghted partcles 98

3 { x, w },2,, N = to approxmate the contnuous posteror dstrbuton. Of course, a large number of partcles can approach to the true dstrbuton wth a greatest degree. However, the computatonal cast s ncreased as the N ncreasng. So, that s a trade-off problem n the practcal stuaton. B. SVM Classfcaton Posteror Probablty SVM [9] [20] s a very useful algorthm for pattern classfcaton. It can dscrmnate the unlabeled samples correctly based on a traned model. Suppose the tranng sample set s provded as,, l s the number of the tranng samples. The SVM wors by maxmzng the margn between two classes n feature space as follows [20]: N T mn J( w, b, ξ) = w w + C ξ (3) 2 Subject to the constrants: = T y[ w ϕ( x ) + b] ξ, 0 Where s the slac varable, C s the penalty factor, ϕ( ) s the mappng from nput space to feature space. By tang the Lagrangan of (3), the orgnal problem can be derved as: l l max α αα yy(, ) x x (4) j j = 2 =, j= classfed by SVM, the result of (6) expresses the extent of extracted feature be postve (class ). That s p p =P(y = f) represents the probablty of sample x to be classfed nto postve set (class ). In contrast, p n =-P(y = f) s the probablty of negatve set (class -). So, the acheved SVM classfcaton posteror probablty can be employed to obtan observaton model p( z x = x ) as shown n the followng equaton: p 2 ( z x ) exp( d ) w w λ (7) ( z x ) p (8) Where s covarance and d s defned by (9). The obtaned observaton lelhood ( ) p z x s used to relate weght w of the partcles n (8). If the extracted sample s classfed nto postve set by SVM, the proposed algorthm calculates the dstance d. Oppostely, the algorthm gnores the result f the extracted sample s classfed nto negatve set. The obtaned d expresses the dstance between canddate sample and standard postve sample. In other word, f dstance d gettng small, the sample has a hgher probablty to be postve and the weght gettng to bg. (9) D. Extracted TCO Samples l = Subject to the constrants: α y = 0 α 0 Where ( x, x) = ϕ( x) ϕ( x ) s a ernel functon, and =,2, l, s the Lagrange coeffcents whch s obtaned from (4). The hyperplane functon can be expressed as: (a) Postve TCO samples; m f( x) = α y x x + b (5) = T Where m s the number of support vectors. Further, the obtaned hyperplane functon (5) can be used to derve the classfcaton posteror probablty [2] defned as follow: Py ( = f) = + exp (6) + ( Af B) It expresses the posteror probablty of ths sample be classfed nto class (postve set). Oppostely, -P(y= f) s the probablty of sample be classfed nto class - (negatve set). C. Lelhood As aforementoned, f an extracted local feature be (b) Negatve TCO samples; Fg. 4. Extracted TCO samples. In tradtonal detecton and tracng communty, some publc sample datasets are avalable. But for human tracng n TCO vson, there are no exstng datasets can be utlzed untl now. So, we have to manually extract the human (foreground) samples n TCO mages wth a tedous and tme consumng process. Also, the negatve samples are obtaned automatcally from a set of TCO mages not contanng human. Some extracted samples are shown n Fg. 4. The formed TCO dataset can be used to tran the classfer, test the proposed tracng method and further research on detecton and tracng n TCO. 99

4 E. Tracng Wndow Dstrbuton n TCO vson Tracng n TCO vson s dfferent wth tradtonal tracng system due to nherent dstorton caused by lght reflecton of convex catadoptrc mrror. So, the effectve omn-mage s the mage on catadoptrc mrror that s crcular. Naturally, the polar coordnate s employed for TCO tracng system, and the center of omn-mage s algnng wth the orgn of the polar coordnate. The orgn O (0, 0) of the mage XY coordnates s located on the top left corner. The center O (0, 0) of the polar coordnate can be obtaned (Fg. 5). Fg. 5. Polar coordnate n TCO tracng system. In catadoptrc omndrectonal vson system, the sze of target S n the mage vares wth the dstance D between target n mage and orgn of polar coordnate (Fg. 5). When the target approachng to the catadoptrc omndrectonal sensor from far to near n world coordnate, the sze of target S gettng to bg and dstance D gettng to small. That s to say, the sze of target S n mage depends on the dstance D. Based on forementoned characterstc of catadoptrc omndrectonal vson, the partcle flter can dstrbute the tracng wndow more reasonable. The sze of tracng wndow S s changng proportonally wth the dstance between partcle and center of omn-mage. In practcal stuaton, the real sze of tracng wndow s depend on the system confguraton, such as the curvature of catadoptrc mrror, the orentaton of catadoptrc mrror, the heght of catadoptrc mrror from ground and so on. In ths paper, we adopt HOG feature as the contour descrptor of target. We use a 2 by 2 cell array to form a bloc, each cell conssts of 4 pxels and the 9-bn hstogram of gradent magntude at each orentaton s computed. So, each bloc forms a 36-dmensonal vector. Fnally, the proposed tracng method s formed. The SVM classfcaton posteror probablty s ntegrated wth the partcle flter, and the tracng wndow dstrbuton based on characterstc of catadoptrc omndrectonal vson s proposed. Accordng to above, the proposed tracng method should able to trac the human n TCO vson effcently. The detal experments are show below. III. EXPERIMENT In the begnnng of ths paper, the confguraton of TCO system has been ntroduced. In ths secton, the experment results of proposed tracng method, gray-level nformaton based partcle flter and gray-level nformaton based onlne adaptve partcle flter are compared. Through test n a number of dfferent experments, the effectveness, stablty and robustness of proposed tracng algorthm are verfed. ) Proposed method s the proposed tracng method whch utlzes the SVM classfcaton posteror probablty to mpact the observaton lelhood of partcle flter. 2) G-PF s the partcle flter based on gray-level nformaton, whch has a fxed reference template. 3) A-PF s the onlne adaptve partcle flter based on gray-level nformaton, whch updates the reference template on tme nstead of fxed template. For a far comparson, we set the same parameters for above three tracng methods, such as partcles number N, covarance and so on. The dynamc models of the three tracng methods are random walng models whch represent as x = x + v, where v s a zero-mean Gaussan random varable. For Proposed method, the SVM classfer has been well traned wth 500 postve samples and 900 negatve samples. Due to lmted space, we just show three expermental results wth two shot n the daytme and one n the nghttme. In ths paper, t s should be noted that we just test sngle target to dscuss the feasblty of the Proposed method n TCO vson. After that, mult-targets tracng of TCO vson wll be extended n our future wor. In experment I, a person wals around the TCO-sensor wth a short tme occluson n the daytme. At the begnnng, three tracers are ntalzed respectvely, and they traced the target successfully (Fg. 6). After several frames later, A-PF tends to drft whch results n lost target at Frame 49 fnally (Fg. 7). Ths s not dffcult to understand snce A-PF s easy mpacted by the bacground although t good at adaptve to the changng of target. Moreover, only lmted gray-level nformaton can be adopted by A-PF n thermal mage nstead of three channels RBG features n normal vsble mage. In ths experment, only Proposed method and G-PF survve to trac the human target successfully to the end. Although both of A-PF and G-PF are based on gray-level nformaton, the reason of dfferent results s A-PF eeps update the template to adapt the change of target that easy tend to drft but G-PF eep the fxed template that s effectve when the dstracton from bacground s lmted. In addton, we set a short tme occluson to test reacton senstvty of the tracng methods when target recover from the occluson. As shown n the Fg. 8, the Proposed method can recover from the occluson n tme when target appear at Frame 332. In contrast, G-PF cannot capture the target sharply. Due to SVM classfer s well traned, the human target can be detected qucly even f the tracng wndow s not well center on the target. Instead, A-PF/G-PF requres relatve restrct template match. If the partcles cannot well land on the target, A-PF/G-PF has rs to fal. Especally n the stuaton of low number of partcles N (N=30 n our experment), the tracng performance of 00

5 Proposed method s much better than A-PF/G-PF n TCO vson. In experment II, a person wals from near to far relatve to the TCO-sensor n the daytme. Three tracers are well ntalzed as experment I and they trac the human target well n the frst few frames. Several frames later, A-PF begn to drft and lose the target completely at the Frame 03 due to accumulated changng of adaptve template (Fg. 9). In the meantme, Proposed method and G-PF stll wor on tracng well. After a few frames later, Proposed method and G-PF lose the target when the target dsappear at the dstant place for a whle. However, the Proposed method can capture the target mmedately when t appears agan because the partcles of Proposed method loo around the target at the nearby place where t dsappeared (Fg. 0). In contrast, G-PF loses the target forever due to the partcles of G-PF have drft to the wrong place where has smlar gray-level dstrbuton as target. Fg. 6. Experment I, Frame. From left to rght: Proposed method, G-PF, A-PF Fg. 7. Experment I, Frame 49. From left to rght: Proposed method, G-PF, A-PF Fg. 8. Experment I, Frame 332. From left to rght: Proposed method, G-PF, A-PF Fg. 9. Experment II, Frame 03. From left to rght: Proposed method, G-PF, A-PF Fg. 0. Experment II, Frame 202. From left to rght: Proposed method, G-PF, A-PF 0

6 Fg.. Experment III, Frame 32. From left to rght: Proposed method, G-PF, A-PF Fg. 2. Experment III, Frame 362. From left to rght: Proposed method, G-PF, A-PF In experment III, a person wals around the TCO-sensor n the nghttme, and system also well ntalzed at the begnnng. As prevous two experments, A-PF loses target agan at Frame 32 (Fg. ). At Frame 362, G-PF also loses the target completely wthout occluson and dsappearance but caused by bacground dstracton (Fg. 2). The reason of G- PF lost target s due to very smlar gray-level feature dstrbuton of surroundng bacground wth reference template at Frame 362. Fnally, only Proposed method tracs the human target well and survve tll the end of experment. In all experments, Proposed method not only tracs the human target successfully but well focus on center of target and very lmted bas occur. In contrast, G-PF and A-PF often happen to bas and drftng n some dffcult stuatons, whch cause to lose target n the end. In summary, the above three experments can effectve verfy the Proposed method has a good performance on tracng the human target n TCO vson. IV. CONCLUSION In ths paper, we present a novel human tracng system n TCO vson. The proposed tracng method maes an effcent utlzaton of SVM classfcaton posteror probablty to relate the observaton lelhood of partcle flter for tracng purpose. Moreover, the proposed tracng approach has acheved a good performance on tracng the human n TCO vson through a varety of experments. Fnally, we wll extend our research to mult-target tracng n TCO vson for real TCO survellance system n the future. REFERENCES [] P. Vola, M. J. Jones, and D. Snow. Detectng pedestrans usng patterns of moton and appearance. In Proc. ICCV, Volume 2, pages , 2003 [2] F. Xu, X. Lu, K. Fujmura, Pedestran detecton and tracng wth nght vson, IEEE Trans. Intell. Transport. Syst. 6 (2005) 63-7 [3] J. Davs, M. Kec, A two-stage approach to person detecton n thermal magery, Proc. Worshop on Applcatons of Computer Vson, 2005, IEEE OTCBVS WS Seres Bench [4] F. Suard, A. Raotomamonjy, A. Bensrhar, Pedestran detecton usng nfrared mages and hstograms of orented gradents, Intellgent Vehcles Symposum 2006, June 3-5, Toyo, Japan [5] C. J. C. Burges, A Tutoral on Support Vector Machnes for Pattern Recognton, Data Mnng and Knowledge Dscovery, 2, 2-67, 998 [6] V. Vapn. Statstcal Learnng Theory. Wley, 998 [7] N. Dalal, B. Trggs, Hstogram of Orented Gradents for Human Detecton, CVPR, volume 2, pp [8] C. X. Da, Y. F. Zheng, X. L, Layered representaton for pedestran detecton and tracng n nfrared magery, Computer Vson and Image Understandng. 06 (2007) [9] S. Y. Yang, G. W. Mn and C. Zhang, Tracng unnown movng targets on omndrectonal vson, Vson Research, 49 (2009), pp [0] T. E. Boult, X. Gao, R. Mcheals, M. Ecmann, Omn-drectonal vsual survellance, Image Vs. Comput. 22, (2004), pp [] J. C. Bazn, K. J. Yoon, I. Kweon, C Demonceaux, P Vasseur, Partcle flter approach adapted to catadoptrc mages for target tracng applcaton, IEEE, BMVC, [2] O. A Jame and B.C Eduardo, Omndrectonal vson tracng wth partcle flter, ICPR [3] W. Schulz, M. Enzwler and T. Ehlgen, Pedestran recognton from a movng catadoptrc camera, Proceedngs of the 29th DAGM conference on Pattern recognton 2007, pp [4] A. L. C Barcza, J. O. Jr and V. G. Jr, Face tracng usng a hyperbolc catadoptrc omndrectonal system, Res. Lett. Inf. Math. Sc, 2009 (3), pp55-67 [5] W. L. Ye, H. P. Lu, F. C. Sun and M. Gao, Vehcle tracng based on co-learnng partcle flter, IROS, 2009, pp [6] M. S. Arulampalam, S Masell, N Gordon and T Clapp, A Tutoral on partcle flters for onlne nonlnear/non-gaussan Bayesan tracng, IEEE Transactons on Sgnal Processng, (50), 2002, pp [7] A. Doucet, J.F.G. de Fretas and N.J.Gordon, An ntroducton to sequental Monte Carlo methods, n Sequental Monte Carlo Methods n Practce, A. Doucet, J.F.G.de Fretas and N.J.Gordon, Eds. New Yor: Sprnger-Verlag, 200 [8] M. Isard, A. Blae, Condensaton-condtonal desty propagaton for vsual tracng, IJCV, vol.29,no.,998, pp.5-28 [9] V. Vapn, The Nature of Statstcal Learnng Theory, Sprng-Verlag, New Yor, 995 [20] C. J. C. Burges, A Tutoral on Support Vector Machnes for Pattern Recognton, Data Mnng and Knowledge Dscovery, 2, 2-67, 998 [2] J. C. Platt. Probablstc outputs for support vector machnes and comparsons to regularzed lelhood methods. n Advances n Large Margn Classfers, Alexander J. Smola, Peter Bartlett, Bernhard Scholopf, Dale Schuurmans, eds., MIT Press, (999) 02