Video Object Tracking Based On Extended Active Shape Models With Color Information

Transcription

1 CGIV'2002: he Frst Frst European Conference Colour on Colour n Graphcs, Imagng, and Vson Vdeo Object rackng Based On Extended Actve Shape Models Wth Color Informaton A. Koschan, S.K. Kang, J.K. Pak, B. Abd, and M. Abd Imagng, Robotcs, and Intellgent Systems Laboratory, Unversty of ennessee Knoxvlle, ennessee Abstract rackng and recognzng non-rgd objects n vdeo mage sequences are complex tasks of ncrea sng mportance to many applcatons. For the trackng of such objects n a vdeo sequence e.g. "actve shape models" can be appled. he exstng actve shape models are usually based on ntensty nformaton and they do not consder color nformaton. However, actve shape models are senstve to outlers, especally n the case of partal object occlusons. In ths paper, we present an extenson of the actve shape model for color mages and we examne to what extent the use of color nformaton can contrbute to the soluton of the outler problem. Introducton he problem of trackng people and recognzng ther actons n vdeo sequences s of ncreasng mportance to many applcatons. 1,2,3 Examples nclude vdeo survellance, human computer nteracton, and moton capture for anmaton, to name a few. Specal consderatons for dgtal mage processng are requred when trackng objects whose forms change between two frames. For example, pedestrans n a road scene belong to ths class of objects denoted as non-rgd objects. For the trackng of non-rgd objects n a vdeo sequence, actve shape models (ASMs) could be appled. he exstng actve shape models usually do not consder color nformaton. In ths paper, we present several extensons of the actve shape model for color mages usng dfferent color adapted objectve functons. rackng and recognzng non-rgd objects n vdeo mage sequences are complex tasks. Usng color nformaton as a feature to descrbe a movng object or person can support these tasks. he use of fourdmensonal templates for trackng objects n color mage sequences was suggested n Ref. 4. However, f the observaton s accomplshed over a long perod of tme and wth many sngle objects, then both the memory requrements for the templates n the database and the tme requrements for the search of a template n the database ncrease. In contrast to ths, ASMs represent a compact model for whch the form varety and the color dstrbuton of an object class are taught n a tranng phase. 5 Several systems use skn color nformaton for trackng faces and hands. 6,7,8 he basc dea s to lmt the search complexty to one sngle color cluster (representng skn color) and to dentfy pxels based on ther membershp to ths cluster. Several problems affect these approaches. Frst, skn colors cannot be unquely defned and, n addton, a person cannot be dentfed when seen from behnd. Here trackng clothes nstead of skn s more approprate. 9 Second, color dstrbutons are senstve to shadows, occlusons, and changng llumnatons. Addressng the problem occurrng wth shadows and occlusons, Lu and an 10 assume that the only movng objects n the scene are persons. hs assumpton does not hold for many applcatons. Most of the approaches mentoned above cannot be easly extended to mult-colored objects other than persons. In ths paper, we present a general technque to track colored non-rgd objects (ncludng persons). A very effcent technque for the recognton of colored objects s color ndexng 11 Based on comparsons between color dstrbutons, an object n the mage s assgned to an object stored n a database. hs technque usually needs several vews of the object to be recognzed, whch s not always ensured when trackng people n a road scene, for example. Furthermore, color ndexng partly fals wth partal occlusons of the object. Actve shape models do not need several vews of an object, snce by usng energy functons they can be adapted to the slhouette of an object represented n the mage. However, the outler problem, whch can occur partcularly wth partal object occluson, represents a dffculty for these models. In the followng, an extenson of the actve shape models for color mages s presented. We examne to what extent the use of color nformaton can contrbute to the soluton of the outler problem, especally n the case of occlusons. Actve Shape Models For trackng a human target n vdeo, detectng the shape and poston of the target s the fundamental task. Snce the shape of a human object s subject to deformaton and random moton n the two-dmensonal mage space, ASM s one of the best-suted approaches n the sense of both accuracy and effcency. ASM falls nto the category of deformable shape models wth a pror nformaton about the object. ASMbased object trackng models the contour of the slhouette of an object, and the set of model parameters s used to algn dfferent contours n each mage frame

2 CGIV'2002: he Frst Frst European Conference Colour on Colour n Graphcs, Imagng, and Vson More specfcally, an ASM-based trackng algorthm conssts of the followng steps: () landmark ponts assgnment, () prncpal component analyss (PCA), () model fttng, and (v) local structure modelng. Landmark Ponts Gven a frame of nput vdeo, sutable landmark ponts should be assgned on the contour of the object. Fgure 1 shows manually selected, 42 landmark ponts on the contour of the human object. Good landmark ponts should be consstently located from one mage to another. In a two-dmensonal mage, we represent n landmark ponts by the 2n vector as x = [ x 1,, xn, y1,, y n ]. (1) Varous automatc, systematc ways of obtanng landmark ponts were dscussed n Ref. 12. Fgure 1. A human object wth 42 landmark ponts (n=42). Prncpal Component Analyss A set of n landmark ponts represents the shape of the object. Fgure 2 shows a set of 56 dfferent shapes, called a tranng set. Fgure 2. ranng set of 56 shapes (m=56). Although each shape n the tranng set s n the 2ndmensonal space, we can model the shape wth a reduced number of parameters usng the prncpal component analyss (PCA) technque. Suppose we have m shapes n the tranng set, such as x, = 1,,m. he PCA algorthm s as follows. PCA Algorthm 1. Compute the mean of the m sample shapes n the tranng set. m 1 x = x. (2) m = 1 2. Compute the covarance matrx of the tranng set. m 1 S = ( x x)( x x). (3) m 1 = 3. Construct the matrx - = φ φ φ ], (4) [ 1 2 t where φ,=1,,t represent egenvectors of S correspondng to t largest egenvectors. 4. Gven Φ and x, each shape can be approxmated as where x x +, (5) -E - b = ( x x). (6) In step 3 of the PCA algorthm, t s determned so that the sum of t largest egenvalues s greater than 98% of the sum of all egenvalues. In order to generate plausble shapes, we need to evaluate the dstrbuton of b. o constran b to plausble values we can ether apply hard lmts to each element b or constran b to be n a hyper-ellpsod. he nonlnear verson of ths constrant s dscussed n Ref. 13. Model Fttng We can fnd the best pose and shape parameters to match a shape n the model coordnate frame, x, to a new shape n the mage coordnate frame, y, by mnmzng the followng error functon E = ( y Mx) W ( y Mx), (7) where M represents the geometrc transformaton of rotaton θ, translaton t, and scale s. For nstance, f we apply the transformaton to a sngle pont, denoted by [x,y], we have x cosθ snθ x t x M = s +. (8) y snθ cosθ y t y After the set of pose parameters, {θ,t,s} are obtaned, the projecton of y nto the model coordnate frame s gven as 1 x p = M y. (9) Fnally, the model parameters are updated as b - = p ( x x). (10) Modelng a Local Structure A statstcal, deformable shape model can be bult by landmark pont s assgnment, PCA, and model fttng steps. In order to nterpret a gven shape n the nput mage based on the shape model, we must fnd the set of parameters that best match the model to the mage. If we assume that the shape model represents boundares and strong edges of the object, a profle across each landmark pont has edge-lke local structure

3 CGIV'2002: he Frst Frst European Conference Colour on Colour n Graphcs, Imagng, and Vson Let g, =1,,n, be the normalzed local profle across the I-th landmark pont, and g and S g the correspondng mean and covarance, respectvely. he nearest profle can be obtaned by mnmzng the followng Mahalanobs dstance between the sample and the mean of the model as = s g s f ( g ) ( g g) S ( g g). (11) s In practce, we used a mult-resoluton ASM technque because t provdes a wder range for the nearest profle search. Extendng ASMs to Color Image Sequences In gray scale mage processng, the objectve functons are determned along the normals for a representatve pont n the gray value dstrbuton. hs procedure can be extended to color mages by frst computng objectve functons separately for each component of the color vectors. Afterwards, a "common" mnmum has to be determned by analyzng the resultng mnma that are computed for each sngle color component. One way of dong ths conssts of selectng the smallest mnmum n the three color components as a canddate. If, however, one of the three color channels contans an outler (compare Fgure 3), ths outler mght be selected as a mnmum. Experments and Results wo frames of an ndoor color mage sequence were used to determne the best searchng method. he test mages are shown n Fgure 4. For ths experment, 57 shapes were used as the tranng set for PCA, and a 7 pxel-wde profle was used for each landmark pont n three RGB color channels. After the modelng step, we got three profle models for each color channel and a shape model. he purpose of the frst experment was to evaluate the performance of dfferent combnatons of color models. he used termnologes are summarzed n able 1. Fgure 4. est mages wth ntal ponts for the 57th mage and the 7th mage. Fgure 3. Example of objectve functons for three color components wth an outler n the red component. Another procedure conssts of selectng the average of the absolute mnma n all three color components. However, outlers n one color channel also lead n ths case to a wrong result. Furthermore, the average value may represent a value that corresponds wth none of the regarded energy functons. One way to overcome ths problem s to use the medan of the absolute mnma n the three color channels as a canddate. hereby the nfluence of outlers n the mnma of the objectve functons s mnmzed. However, further false values may arse durng the algnment of the contours. In the next secton we wll further address the queston f a contrast-adaptve optmzaton may mprove the ASM performance. For every sngle landmark pont we wll select the color channel wth the hghest contrast and mnmze the correspondng objectve functon. able 1. ermnologes he result usng the ntensty mage wth the Intensty ntensty profle. he result usng the color mage wth the R Red profle. G he result wth the Green profle. B he result wth the Blue profle. he result usng the color mage after selectng the mnmum of the mnma of the Mnmum Mahalanobs dstance n the three color channels. Medan he result wth the medan of the mnma. Mean he result wth the mean of the mnma. he result usng the ntensty mage wth the adaptve profle model that s modeled wth Adaptve the strongest edge among three color channels for each pont. he ntal landmark ponts were manually placed as shown n Fgure 4. Hll, aylor, and Cootes 5 suggested a genetc algorthm that determnes the "best" form parameters from a randomly specfed set of ntal values. So far we dd not examne ths algorthm due to ts computatonal complexty. We argue that a manual defnton of the form parameters s sutable for our purpose snce the ntal form has only to be determned once for a class of smlar-shaped objects. Our goal s to track persons and to gnore other movng objects

4 CGIV'2002: he Frst Frst European Conference Colour on Colour n Graphcs, Imagng, and Vson Furthermore, we defned a maxmum shft between two mage frames for an object to be tracked. hs lmtaton s due to a reducton of the computng tme and does not restrct the algorthm n general. he maxmum shft parameter depends on the sze of the object, the dstance between the camera and the object, the velocty of the object, and the movng drecton of the object. For example, for trackng a person on an arport we can predct the maxmum sze of a person, the maxmum velocty of a walkng or runnng person, and the mnmum dstance between the camera and a person. o lmt the movng drecton of a person, we can further assume that only a few persons mght move towards a camera that s mounted on a wall. In our nvestgaton we lmted the maxmum shft to 15 pxels for the herarchcal approach. Both herarchcal and non-herarchcal methods were tested for the mage shown n Fgure 4 because ts ntal contour was set smaller than the real object. On the other hand, only the non-herarchcal method was tested n Fgure 4. In the herarchcal approach, level 0 represents the orgnal gven resoluton, level 1 the half-szed resoluton, and level 2 the quarter-szed resoluton. hree dfferent levels are shown n Fgure 5. We performed 5 teratons n level 2, another 5 teratons n level 1, and fnally 10 teratons n level 0. For the non-herarchcal approach we performed 10 teratons. he herarchcal approach helps to enlarge the search regons and shows a better search result than the nonherarchcal approach. he model fttng error for each experment s summarzed n able 2. he result of the herarchcal approach to Fgure 4 s shown n Fgure 6. he result of the nonherarchcal approach s shown n Fgure 7. he medan method gves the best results n the sense of both vsual and the objectve error measurements. Results usng the R, G, and B color channels show worse fttng than those method usng ntensty. able 2 summarzes error measurements of dfferent methods gven n able 1. able 2. he sum of dstance between the estmated ponts by the dfferent searchng methods and the manually assgned ponts. Intensty R G B NH (57 th ) HR (57 th ) NH (7 th ) Mnmum Medan Mean Adaptve NH (57 th ) HR (57 th ) NH (7 th ) (c) Fgure 5. hree dfferent resolutons used n the herarchcal approach: level 2, level 1, and (c) level 0. (c) (d) (e) Fgure 6. Herarchcal search results of the 6 dfferent methods for the 57th mage: ntensty, mnmum, (c) medan, (d) mean, and (e) adaptve. he second experment used an outdoor sequence. We appled the ASM to each of the outdoor mage frames and selected the mean, the mnmum, and the medan of the mnma n the objectve functons for searchng. he results for selectng the medan of the mnma are shown n Fgure 8. he ASM gves good results, even though the object s partally occluded by the bench

5 CGIV CGIV'2002: 2002: he FrstFrst European European Conference Conference on Colour on Colour n Graphcs, Graphcs,Imagng, Imagng,and andvson Vson (c) (d) (e) Fgure 7. Non-herarchcal search results of sx dfferent methods for the 7th mage: ntensty, mnmum, (c) medan, (d) mean, and (e) adaptve. Concluson A technque was presented for recognzng and trackng a movng object or person n a vdeo sequence. For ths the objectve functon for actve shape models was extended to color mages. We evaluated several dfferent approaches for defnng an objectve functon consderng the nformaton from the sngle components of the color mage vectors. hs trackng technque does not requre a statc camera (except to ntalze the landmark ponts for the object to be recognzed). he medan computaton of the mnma n the energy functons proved favorable n our ndoor and outdoor experments. In general the error n fttng an ASM to the real contour of an object was lower when usng color nformaton than when just usng ntensty nformaton. Furthermore, we showed that the fttng error can be further reduced when applyng a herarchcal approach nstead of a non-herarchcal one to the mages. he performance of the algorthm was rather robust regardng partal object occlusons. he problem of outlers n the objectve functons could be partly solved by the evaluaton of color nformaton. One way to further enhance these results mght be a refned analyss of the objectve functons, where the neghbors of one pont are also consdered. hereby the number of outlers can be further reduced. (c) (d) Fgure 8. Search results for an outdoor sequence usng the non-herarchcal approach for the 1st frame, the19th frame, (c) the 27th frame, and (d) the 33rd frame

6 CGIV'2002: he Frst Frst European Conference Colour on Colour n Graphcs, Imagng, and Vson However, the trackng of a person becomes rather dffcult f the mage sequence contans several movng persons wth smlar shape. In ths case, a technque exclusvely based on the contour of a person wll have dffcultes n trackng a selected person and the task may fal f the person s partally occluded. On the other hand, a technque exclusvely evaluatng the colors of a movng person (or object) may also fal. Any colorbased tracker can lose the object t s trackng due, for example, to occluson or changng lghtnng condtons. o overcome the senstvty of a color-based tracker to changng lghtnng condtons, the color constancy problem has to be solved at least n parts. hs s a nontrval and computatonally costly problem that can n general not be solved n vdeo real-tme. Another soluton to the problem mentoned above could consst of a weghted combnaton of a form-based trackng technque usng, for example, ASMs and a color-based trackng technque usng, for example, color ndexng. By applyng such a combnaton technque to mage sequences we mght be able to dstngush between a) objects of smlar colors but wth dfferent forms and b) objects of dfferent colors but wth smlar forms. Acknowledgements hs work was supported by the Unversty Research Program n Robotcs under grant DOE-DE-FG02-86NE37968, by the DOD/ACOM/NAC/ARC Program, R , and by FAA/NSSA Program, R /49. Furthermore, the authors acknowledge the help of Klaus Curo of U Berln, Germany. References 1. R. Plänkers and P. Fua, rackng and modelng people n vdeo sequences, Comp. Vson and Image Understandng 81, pg (2001). 2. S. J. McKenna, Y. Raja, and S. Gong, rackng colour objects usng adaptve mxture models, Image and Vson Computng 17, pg (1999). 3. I. Hartaoglu, D. Hartwood, and L. S. Davs, W4: Realtme survellance of people and ther actvtes, IEEE rans. on PAMI 22, pg (2000). 4. S. A. Brock-Gunn, G. R. Dowlng, and. J. Ells, rackng usng colour nformaton, Proc. ICARCV 94, pg (1994). 5.. F. Cootes, D. H. Cooper, C. J. aylor, and J. Graham, Actve Shape Models - her tranng and applcaton, Comp. Vson and Image Understandng 61, pg (1995). 6. Y. L, A. Goshtasby, and O. Garca, Detectng and trackng human faces n vdeos, Proc. ICPR 00 vol. 1, pg (2000). 7. F. Marqués and V. Vlaplana, Face segmentaton and trackng based on connected operators and partton projecton, Pattern Recognton 35, pg (2002). 8. D. Comancu and V. Ramesh, Robust detecton and trackng of human faces wth an actve camera, Proc. Vsual Survellance 2000, pg (2000). 9. H. Roh, S. Kang, and S.-W. Lee, Multple people trackng usng an appearance model based on temporal color, Proc. ICPR 00 vol. 4, pg (2000). 10. W. Lu and Y.-P. an, A color hstogram based people trackng system, Proc. ISCAS 2001 vol. 2, pg (2001). 11. M. J. Swan and D. H. Ballard, Color ndexng, Int. Journ. of Comp. Vson 7, pg (1991). 12. Q. an, N. Sebe, E. Loupas, and. S. Huang, Image retreval usng wavelet-based salent ponts, Journ. of Electronc Imagng, 10 (4), pg (2001). 13. P. Sozou,. F. Cootes, C. J. aylor, and E. D. Mauro, A nonlnear generalzaton of pont dstrbuton models usng polynomal regresson, Image and Vson Computng 12 (5), pg (1995). 14. A. Hll, C. J. aylor, and. F. Cootes. A generc system for mage nterpretaton usng flexble templates, Proc. ECCV`94, pg (1994). Bography Andreas Koschan receved hs Dplom (M.S.) n Computer Scence and hs Doktor-Ing. (Ph.D.) n Computer Engneerng from echncal Unversty Berln, Germany n 1985 and 1991, respectvely. Currently he s a Research Assocate Professor at the Unversty of ennessee, Knoxvlle. Hs work has focused prmarly on color mage processng and 3D computer vson ncludng stereo vson and laser range fndng technques. He s a coauthor of two textbooks on 3D mage processng and he s a member of the IS& and the IEEE