Object Tracng under Heavy Occluson based on Extended Kalman Flter, Partcle Flter, and Color Matchng Youngsung Soh, Mudasar Qadr, Had Raja, Yongsu Hae, Intae Km, Mal M. Khan, and Tayyab Wahab soh@mju.ac.r, mudasar.alwar@gmal.com, nothan111@gmal.com, wse_sunys@nate.com, t@mju.ac.r, malm2003@yahoo.com, and tayyabwahab90@gmal.com Myongj Unversty, Yongn, South Korea Abstract Object tracng s useful n areas such as survellance, access control, mltary applcatons, and health care. Objects may be human, vehcle, and other enttes. A huge number of approaches have been proposed n ths feld. Dfferent methods assume dfferent envronments and adopt varous technques rangng from determnstc to probablstc approaches. Excellent surveys can be found n [1]. In ths paper, we proposed a method for tracng objects by explotng probablstc power of trac flters ncludng extended Kalman flter and partcle flter n the presence of heavy occluson. We also use color matchng to mprove on tracng results. We appled the method to real world vdeos and obtaned successful results. Index Terms Object tracng, Extended Kalmanfter, Partcle flter, Color matchng. I. INTRODUCTION Object tracng plays an mportant role n many areas such as survellance, traffc montorng, health care, etc. To trac objects, objects should be extracted frst. There are many ways n segmentng objects from the mage. They nclude, bacground dfferencng, frame dfferencng, optcal flow based method, mult resoluton method, etc. After objects are extracted, they should be traced. Jalal et al. [1] broadly categorze tracng methods nto top down and bottom up approaches. Top down approaches usually assume the presence of external nput to ntalze the tracng process. Mean shft method and ts varous varants are examples of top down approaches. In bottom up approaches, objects are extracted frst by any segmentaton algorthm and traced by mappng objects n between consecutve frames. Most of these approaches suffer when dffcultes such as nose, llumnaton change, and camera jtterng occur. Jalal et al. [1] mentoned that wavelet s a promsng tool to overcome these dffcultes. There s yet another major dffculty that maes tracng hard. When there are occlusons, the characterstcs of objects changes dynamcally. Jalal et al. [1] ntroduced three types of occluson. They are self-occluson, nterobject occluson, and object-bacground occluson. In ths paper, we propose a new method for tracng objects when there are heavy occlusons. We bascally allow all three nds of occlusons, but most of them are nter-object occlusons. When dfferent objects are occluded and then separated, there occur merge and splt of objects. Here the term splt does not mean the splt of a sngle object nto 20 several peces. Rather t means that the group of objects breas nto smaller groups of objects or ndvdual object. The proposed method utlzes the combnaton of extended Kalman flter (EKF), partcle flter(pf), and color matchng(cm) dependng on merge and splt scenaros. Ths paper s organzed as follows. Chapter II revews the related wors for tracng under heavy occlusons. Chapter III descrbes the proposed method n detal. The expermental results are shown n Chapter IV. Fnally Chapter V concludes the paper. II. RELATED WORKS Many methods have been proposed for multple object tracng under heavy occluson. Marceno et al. [2] proposed multple object tracng method under heavy occluson based on Kalman flter (KF) and shape matchng. They argued f an object retans constant speed, KF alone wors. If speed constrant s volated, shape matchng should also be used. The shape they used s a bnary sub mage of ndvdual object and match score s computed by smple correlaton. Snce ther method s based on lnear KF, t has an nherent lmtaton for nonlnear moton and the occluson n ther test data does not seem heavy. Lu et al. [3] used color appearance model nstead of whole object appearance model to support occluson cases. They combne R, G, and B color components wth varous lnear coeffcents to get feature set for target and non-target objects. To select relable features, they apply onlne feature selecton method based on Ada Boost proposed n [4]. After selectng relable features, they extract regons havng dscrmnatve power measured by log lelhood of selected features. Fnally they appled PF to trac regons. They tested ther method on staton hallway sequences n PETS2006 database provded by Unversty of Readng n U.K. They tested four dfferent varatons of the algorthm and showed that the one wth feature selecton capablty and regon based feature update and tracng performed best. Yang et al. [5] proposed a tracng method that can handle merges and splts of movng target. Frst, objects are segmented based on two level bacground mantenance model. Two levels are pxel level and frame level. Morphology s used to get fnal objects. Second, the presence of merges and splts are checed. If there s a merge, a group s created and s traced as f t s a sngle target. If splt s detected, feature correspondence s
conducted to fnd the match. They consdered three classes of features whch are moton, appearance, and color. Among them, color was chosen due to ts mmunty to scene dynamcs. They appled the method to a few publc databases and reported good results. Pan et al. [6] proposed the content based tracng scheme where they performed content-adaptve progressve occluson analyss to combne spato-temporal context, target, and moton constrants. To trac occluded target, they used varant-mas template matchng. To prevent template from deteroraton, drftnhbtve mased Kalman appearance flter was ntroduced. They adopted a local best match authentcaton algorthm to handle complete occluson. Here, represents state vector, a process lnear functon, and a process nose vector. Represents observaton vector, a observaton lnear functon, and a measurement nose vector. Human and vehcles are common objects consdered n tracng. In real world stuatons, they do not necessarly follow lnear moton. They change speed and drecton ether smoothly or abruptly, thus volatng the lnearty. To overcome ths dffculty, EKF [9,10] was ntroduced as n (2). III. THE PROPOSED METHOD Fg. 1 shows the bloc dagram of the proposed method. Gven two consecutve frames Ft and Ft+1, objects n two frames are segmented by bacground subtracton. Next the number of blobs and the sze of blobs n two frames are compared to detect the presence of merges and splts. Dependng on the stuatons, one of three possble scenaros s selected and performed. Those scenaros consst of three basc components. They are extended Kalman flter(ekf), partcle flter(pf), and color matchng(cm). By approprately combnng basc components, merges and splts can be handled and objects are traced successfully. We explan each component n turn n detal along wth the bacground subtracton method we adopted. Fg 1.Bloc dagram of the proposed method Here and are nonlnear functons of state varables and evolutons. EKF s performed through two steps: predcton and correcton. Fg. 2 shows the detal of these two steps. The measurement z s the external nput and state estmate s the fnal output. In order to derve the state estmate by predcton and correcton, several parameters ( A, H, Q, R ) are requred. A and H are df dh dx and dx respectvely and they are extracted from the nonlnearty x ˆ f ( 1 ) and h( ) H. Q s the covarance matrx of the process and R represents nose characterstcs n the measurement. As n KF, EKF fnds the estmate from the measurement z. The predcton step computes predcted value of estmate and predcted error covarance. The correcton step calculates Kalman gan and updates estmate and error covarance. Predcton A. Bacground Subtracton Bacground subtracton s used to extract objects. We used spato-temporal Gaussan mxture model(stgmm) proposed by Soh et al. [7]. STGMM s based on Gaussan mxture model (GMM) [8]. STGMM consders both spatal and temporal varatons of the mage contents, whereas GMM taes nto consderaton only temporal varatons. STGMM performed far better than GMM especally when there are bacground dynamcs such as swayng tree branches, flutterng flags, and sea waves. B. Extended Kalman Flter Kalman flter (KF) s frequently used to model lnear dynamcs by lnear evoluton functons as n (1). (1) Project state ahead x f (, u,0) ˆ 1 (2) Project the error covarance ahead T T P 1 A P A WQW Correcton Intal estmates and P 21
(1) Compute the Kalman Gan T T T 1 K P H ( H P H V RV ) (2) Update estmate wth measurement ˆ ˆ ( ( ˆ x x K z h x,0)) (3) Update the error covarance P ( 1 K H ) P 3. Resamplng Step - Resample wth replacement N partcles: - from the set: - accordng to the normalzed mportance weghts, 4. Set - Proceed to the Importance Samplng step, as the next measurement arrves. Fg 2. A complete operaton of EKF wth equatons C. Partcle Flter PF s a sequental Monte Carlo method for onlne learnng based on a Bayesan framewor. It has many other names such as bootstrap flter, condensaton tracer, etc. It mplements recursve Bayesan flter by Monte Carlo samplng where t represents the posteror densty by a set of random partcles wth accompanyng weghts. Estmates are computed by generated samples and weghts. There are many nds of PFs n the lterature. We choose to use sequental mportance resamplng (SIR) PF. To explan the algorthm, we use the notatons used n Latec [11] that are the extended verson of Keth Copsey n Pattern and Informaton Processng Group, DERA Malvern. The mechancs of SIR PF s llustrated n Fg. 3. D. Color Matchng As Yang et al. [5] ponted out, color s an mportant feature that can dscrmnate dfferent objects. In case of human, we extract color nformaton of torso and leg parts as n Fg. 4. The numbers on both sdes of blac box represent the postonal ratos when the box heght s set to 1 measured from top to bottom. Thus the porton of the box from 0.25 to 0.375 belongs to torso and that from 0.625 to 0.75 belongs to leg. These postonal ratos were obtaned by manually dvdng the body parts of accumulated object data and tang the average. Fg. 4 Torso and leg parts of human body Fg. 3 SIR PF process The algorthm conssts of four steps. They are ntalzaton, mportance samplng, resamplng, and teraton steps, and are descrbed below [11]. 1. Intalzaton - - For sample - and set 2. Importance samplng step - For sample and set - For compute the mportance weght - Normalze the mportance weght, To match the color smlarty between two objects n consecutve frames, color hstograms are bult and compared. Here we use HSI representaton of color snce, unle RGB, chromatcty and ntensty components are well separated. We use only H and S to buld hstograms. Gven hstograms of objects under comparson, there are several ways to measure the dstance between them. Equatons (3) to (7) show some of the dstance metrcs usng color hstograms. Here and represent two color hstograms under comparson. Hstogram ntersecton mnh Q( ), H ( ) H ( HQ, H ) mn HQ( ), H ( ) Eucldean dstance L2 ( HQ, H ) HQ( ) H ( ) 2 (4) (3) 22
Bhattacharyya dstance B ( H Q, H ) ln H Q ( ) H ( ) M ( HQ, H ) Matusta dstance Dvergence H Q ( ) D( H Q, H ) H Q ( ) H ( ) ln (7) H ( ) We choose to use Bhattacharyya dstance snce t s nown to perform best. Equatons(8) to (10) show the Bhattacharyya dstance we defned for our applcaton. HQ ( ) 2 H ( ) (5) (6) (8) breas nto two blobs, one havng three and the other havng one object. Snce there s only one blob havng multple objects, we use EKF+CM combnaton. Frstly, EKF predcts the locaton and then CM dentfes the object havng ID 1. Snce there remans a bg blob havng multple objects, all other IDs were allocated to that blob. (9) (10) Here, and represent hue components of two hstograms under comparson each havng 180 bns, and and are saturaton components each havng 256 bns. s the weghtng factor and s the fnal Bhattacharyya dstance. Snce s the dstance, smlarty can be obtaned by, where s a small constant to prevent dvdng by zero. IV. EPERIMENTAL RESULTS Test vdeo for the experment was captured n outdoor envronment. Humans and vehcles are movng. Four human objects are experencng heavy occluson, n our case, many nds of merges and splts. As depcted n Fg. 1, the proposed algorthmruns n three modes. They are merge, splt, and none. Fg. 5 shows the tracng nstance where merge has occurred. Among four ndvdual humans, two havng IDs 0 and 2 were merged nto a sngle blob. Merged blob has label 2/0 to ndcate that objects labeled 0 and 2 are n the same blob. Here we use the combnaton of EKF and CM. EKF was used for locaton predcton and CM was used to dentfy blobs havng IDs 1 and 3. All the objects were traced correctly. Fg. 6 Splt example1 Fg. 7 depcts the tracng nstance where another type of splt has occurred. In ths case, four n a group breas nto two groups of two objects. Snce there s no blob havng a sngle object, CM cannot be used. PF s appled to fnd whch objects are n whch blob. To do that, we explot partcles wth assocated lelhood values as weghts that were computed usng color hstograms. When we compute lelhood value we use the equatons (8) to (10) provded for CM. In the rght mage of Fg. 7, partcles for four objects were dsplayed n dfferent colors. All the objects were traced correctly. By analyzng the partcle locatons n merged blobs, we even can fnd out the relatve postonng of merged objects nsde a blob. Fg. 7 Splt example2 Fg. 8 shows the tracng nstance where none has occurred. In ths case, only EKF s suffcent to trac all the objects. All four objects were traced successfully. Fg. 5 Merge example Fg. 6 shows the tracng nstance where splt has occurred. A sngle blob havng four ndvdual humans Fg. 8 No merge and splt example 23
V. CONCLUSIONS Object tracng plays an mportant role n many areas such as survellance, access control, health care, etc. Many approaches were proposed assumng varous envronments. One of the major dffcultes for tracng s occluson. In ths paper, we proposed a new method for tracng under heavy occluson. Dependng on the occluson scenaro that can happen n real world, the proposed method used varous combnatons of EKF, PF, and CM. We tested the method wth real world vdeo and obtaned successful results. Le many other tracng algorthms, the proposed method assumes that objects to be traced have dfferent colors. Dffculty s expected when objects wth a smlar color nteract n nearby locatons. Ths s ntended for future research. ACKNOWLEDGMENT Ths wor (Grant No.C0210325) was supported by Busness for Cooperatve R&D between Industry, Academy, and Research Insttute funded by Korea Small and Medum Busness Admnstraton n 2014. REFERENCES [1] A. S. Jalal and V. Sngh, The state-of-the-art n vsual object tracng, Informatca, vol.36, pp.227-248, 2012. [2] L. Marceno, M. Ferrar, L. Marchesott, and C. S. Regazzon, Multple object tracng under heavy occluson by usng Kalman flters based on shape matchng, IEEE Internatonal Conf. on Image Processng, pp.iii-341-iii-344, 2002. [3] T. Lu and P. u, Relable multple object tracng under heavy occluson, Internatonal Symposum on Intellgence Informaton Processng and Trusted Computng, pp.88-92, 2010. [4] Y. J. Yeh and C. T. Hsu, Onlne Selecton of Tracng Features Usng AdaBoost, IEEE Trans. Crcuts and Systems for Vdeo Technology, vol. 19, no. 3, pp. 442-446, Aprl 2009. [5] T. Yang, Q. Pan, J. L, and S. Z. L, Real-tme Multple Objects Tracng wth Occluson Handlng n Dynamc Scenes, IEEE Internatonal Conf. Computer Vson and Pattern Recognton, vol. 1, pp.970-975, 2005. [6] J. Pan, B. Hu, and J. Q. Zhang, Robust and Accurate Object Tracng under Varous Types of Occlusons, IEEE trans. On Crcuts and Systems for Vdeo Technology, vol. 18, ssue 2, pp.223-236, 2007. [7] Y. Soh, Y. Hae and I. Km, Spato-temporal Gaussan Mxture Model for Bacground Modelng, IEEE Internatonal Symposum on Multmeda (ISM), pp. 360-363, Dec. 2012. [8] C. Stauffer and W. Grmson, Adaptve bacground mxture models for real-tme tracng, IEEE Internatonal Conference on Computer Vson and Pattern Recognton, pp.246-252, 1999. 24 [9] Juler, S.J.; Uhlmann, J.K., "Unscented flterng and nonlnear estmaton, Proceedngs of the IEEE, pp. 401 422, 2004. [10] Brano Rstc, Sanjeev Arulampalam, Nel Gordon, Beyond the Kalman Flter, Artech House, 2004. [11] https://www.google.co.r/?gfe_rd=cr&e=qrfhua2omw6m gwm5ihqba&gws_rd=ssl#newwndow=1&q=partcle+flt er+powerpont. ABOUT AUTHORS Youngsung Sohgot BS n electrcal engneerng n 1978 from Seoul Natonal Unversty n Seoul, Korea. He obtaned MS and PhD n computer scence from the Unversty of South Carolna n Columba, South Carolna, USA n 1986 and 1989, respectvely. object tracng, stereo vson, and parallel algorthms for mage processng. Mudasar Qadr got BS n computer scence n 2011 from FAST-NU Unversty n Peshawar, Pastan. He entered a Master course n nformaton and communcaton engneerng n Myongj Unverstyn 2013. object tracng, stereo vson and parallel algorthms for mage processng. HadRaja got BS n electrcal engneerng n 2010 from Govt. Unversty n Lahore, Pastan. He entered a Master course n nformaton and communcaton engneerng n Myongj Unverstyn 2012.. object tracng, stereo vson and parallel algorthms for mage processng. YongsuHae got BS and MS n nformaton and communcaton engneerng from Myongj Unversty n Yongn, Korean 2009 and 2012, respectvely. He entered a PhD course n nformaton and communcaton engneerng n Myongj Unversty n Yongn, Korean 2012. object tracng, stereo vson, and parallel algorthms for mage processng.
Intae Km receved BS and MS n electroncs engneerng from Seoul Natonal Unversty n Seoul, Korea n 1980 and 1984 respectvely. He obtaned PhD n electrcal engneerng from Georga Insttute of Technology n Atlanta, Georga, USA n 1992. Hs research nterest ncludes pattern recognton, mage processng and smart grd area. Mal M. Khan got BS n electrcal engneerng n 2011 from Govt. College Unversty n Lahore, Pastan. He entered a Master course n nformaton and communcaton engneerng n Myongj Unversty n 2012. vehcle to grd, Kalman flterng, bacground modelng and mult-vewpont tracng. TyyabWahab got BS n telecommuncaton engneerng n 2012 from Natonal Unversty of Computer and Emergng Scences n Peshawar, Pastan. He entered a Master course n nformaton and communcaton engneerng n Myongj Unversty n 2013. vehcle to grd, Kalman flterng, partcle flterng and mult-vewpont tracng. 25