Evaluaion and Improvemen of Region-based Moion Segmenaion Mark Ross Universiy Koblenz-Landau, Insiue of Compuaional Visualisics, Universiässraße 1, 56070 Koblenz, Germany Email: ross@uni-koblenz.de Absrac Several approaches of moion segmenaion were published in he las years, bu an evaluaion of hese differen approaches is missing up o now. Here we evaluae differen mehods of moion segmenaion o opimize our moion esimaion sysem [5] based on a n:m maching of color regions. We compare four differen neighborhood checking mehods and hree differen moion similariy ess in combinaion. For his purpose a qualiy measure was developed wih a hand segmenaion as a ground ruh. This measure conains boh he posiive moion segmenaion error and he negaive one. Moreover a new, efficien approach for checking neighborhood relaions beween image regions is presened and also evaluaed. Keywords: moion segmenaion, neighborhood analysis, rajecory comparison, moving camera, evaluaion, ri-sae hand segmenaion 1 Inroducion The differen echniques of moion segmenaion can be devided in wo groups: sysems wih saic camera and sysems wih moving camera. A saic camera allows o pariion he images ino foreground and background. Changes in successive images are deeced as foreground whereas saic areas are deeced as background [1]. Obvious hese echniques fail for moving cameras as he complee image migh be in change. In his aricle echniques for moving camera are regarded as a saic camera is a special case of his more general case. 8. In. Workshop on Vision, Modeling, and Visualizaion VMV 2003, München, Erl e al (Hrsg.) Many promising moion segmenaion approaches wih moving camera firs use a segmenaion or clusering 1 of he images ino regions which are homogeneous in color [2], [3], [4], [5], [9] for solving he correspondence problem by using regions as feaure. Following displacemen vecors of each region can be compued e.g. as displacemen beween ceners of graviy of corresponding regions. 2. These displacemen vecors are used o updae regions moion rajecory and/or o calculae heir moion predicion. The las sep of such a complex image processing chain is o combine neighbored 3 image regions wih similar moion as one moion objec. 4. For his ask - he so called moion segmenaion - a sysem has o check boh he moion similariy and a neighborhood relaions beween regions. An oher approach insead of regarding neighborhoods is o use a priori applicaion knowledge - bu for developing an universal moion segmenaion ool his approach is no analyzed here. This aricle describes differen mehods for analyzing moion similariy and neighborhood checking mehods and compares hem. For his purpose a qualiy measure based on a ri-sae hand segmenaion as ground ruh is developed. As processing chain he n:m maching algorihm [5] is used: afer a CSC color segmenaion [6] feaures of each color segmen were calculaed and used for solving he correspondence problem. The goal of n:m maching is o deermine hose subses 1 Segmenaion creaes coniguous regions whereas clusering parially creaes disconiguous regions. 2 [5] uses a correlaion of regions boundary for compuing displacemen vecors. 3 neighbored needn necessarily mean ouch direc 4 Noe ha his ask is in case of saic camera much more easier because of a possible parinioning ino foreground and background. VMV 2003 Munich, Germany, November 19 21, 2003
of wo ses of color regions ha are bes corresponding o each oher. 2 Topological Relaions In his secion hree familiar mehods for analyzing opological neighborhood relaions are briefly described (ch. 2.1-2.3) and a new approach is presened (ch. 2.4). 2.1 Disance of graviy ceners In [4] no direc neighborhood relaion bu opological nearness is used. This is deermined by he euclidian disance of ceners of graviy ( x i, ȳ i), i {1, 2} of wo regions R 1, R 2 and a hreshold D which conrols he maximum disance beween wo neighbored regions. R 1, R 2 are neighbored, iff p ( x1 x 2) 2 + (ȳ 1 ȳ 2) 2 < D, D R. (1) R 1 R 2 R 3 R 4 D Figure 1: Problem of predefining D: R 3 and R 4 are neighbored, R 1 and R 2 are no neighbored, alhough hey are ouching each oher The choice of he hreshold D needs he knowledge abou he applicaion and he image resoluion, i.e. how large he regions usually are. If here are almos very large regions he hreshold mus be high, because he disances of he graviy ceners are large, oo. Figure 1 illusraes he problem of predefining he maximum cener disance D. This approach is accouned wih he reques of deecing parially masked objecs as a single objec, for example a moving car behind a ree. Bu his requires very accurae knowledge abou he scene, in his example he hickness of he ree mus be known for no maching differen cars driving in a row wih same velociy as one moving objec. 2.2 Overlapping of Bounding Boxes The overlapping of bounding boxes is used as neighborhood check in [2]. Le (x i,min, x i,max, y i,min, y i,max) be he bounding box of region R i. Then region R 1 is neighbored wih region R 2 iff x 1,min 0.5 x 2,max + 0.5 x 2,min 0.5 x 1,max + 0.5 y 1,min 0.5 y 2,max + 0.5 y 2,min 0.5 y 1,max + 0.5. (2) The advanages of his approach are is low compuaional coss and ha here is no need of a priori knowledge of he scene. Bu i deecs oo many neighborhood relaions and reduces he performance of he following processing seps, e.g. similariy check of moion rajecories. 2.3 Overlapping of Convex Hulls In [5] R 1, R 2 are defined as neighbors, iff p min (x1 x 2) 2 + (y 1 y 2) 2 < T R. (x i,y i ) R i (3) The big advanage of his definiion is he fac, ha no only regions are neighbored which direcly ouch each oher, bu all wih is minimum disance is small enough. In a moion segmenaion ask in vehicle guidance a small T of 4 pixels has shown very good resuls. As i is oo cosly calculaing he disances from each pixel of one region o each pixel of he oher region (complexiy O(n 2 )) he neighborhood checking algorihm of [5] uses a fas approximaion of he convex hull wih a regular polygon wih 24 edges. The polygons are enlarged abou 1 T pixels in each 2 direcion (see [8] for deails). If any corner of one of he polygons is inside he oher polygon, he wo color segmens are neighbored. This mehod has he same drawback as he bounding box check: non-convex regions will ge oo many neighbors. 2.4 Boundary Based Neighborhood Analysis For exending our moion esimaion [5] sysem from vehicles o more complex objecs wihou convex boundaries, like moving people, a new approach of neighborhood analysis was developed. In
order o handle arbirary boundaries, we consider each boundary pixel 5 of he regions and sore he labels 6 of is neighbor pixels while generaing he chain-code. Therefor each region R i has a boolean array A i of neighborhood relaions wih he size of he number of labels. All he fields in hese arrays are iniialized as logical false. While regarding he boundary of R i each found label l k leads o a rue enry in he array A i a posiion k. Two regions R u, R v wih he labels l u, l v are opologically neighbored iff A u[l v] = rue A v[l u] = rue. (4) Noe ha i is no enough o check only one parial erm of (4) because of possible enclosures. In he case, ha region R u encloses region R v A u[l v] is false and A v[l u] is rue. In n:m maching clusers of regions are mached wih oher clusers. The informaion of all neighbors of a cluser C is received as A C[k] = _ A i[k], k N. (5) i,r i C 3 Moion Similariy Tess We consider a moion vecor as he 2d-displacemen of he cener of graviy of a region (or in case of a n:m maching a cluser of regions) of wo successive frames. The robusness of a moion segmenaion can be increased by regarding no only he newes moion vecor bu also a moion rajecory v(n) which is he series of he las n moion vecors v R 2. In he following a brief descripion of hree common mehods for checking moion similariy is given. 3.1 Similar Predicion in [7] as: v 1 v 2 v 1 + v 2 < T L (6) cos 1 v 1,v 2 < v 1 v 2 Tα wih hreshold T L [0, 1] for checking similariy of vecors lenghs and hreshold T α [0, π] for conrolling maximum angle beween similar vecors. 3.2 Similar Trajecories An oher approach [5] is o compare he hole rajecories v(n) R 2, 0 n < N. In order o olerae local fauls, he rajecories are divided ino overlapping secions of hree successional displacemen vecors of which each a linear approximaion v i = 2i+2 X n=2i v(n), 0 i < N 2 1 (7) is calculaed. Two rajecories are similar, iff heir corresponding linear approximaions v i are similar using he crierion (6). In order o limi he coss, no more han he laes five pairs of linear approximaions were regarded. 3.3 Correlaion of Trajecories In [2] he correlaion of moion rajecories v 1(n), v 2(n), 0 n < N is defined as 1 l 1 l 2 l 1 +l 2 k 1,2 k1,1 k 2,2 wih k i,j = N 1 P (v i(n) v i) T (v j(n) v j), l i = N 1 P n=0 n=0 v i(n), and v i = 1 n N 1 P n=0 v i(n). (8) Two regions are similar in moion, iff correlaion of heir rajecories is greaer han a cerain hreshold T C [0, 1]. A moion predicion can be calculaed as a weighed sum of he moion vecors of a rajecory. Two regions are similar in moion iff heir predicions v 1, v 2 R 2 are similar. The similariy is defined 5 The regions boundary are hose pixels belonging o his region and ouch a pixel of an oher region or are laying on he image boundary. 6 The resul of he previously done color segmenaion
4 Qualiy 4.1 Tri-Sae Hand Segmenaion as Ground Truh In order o compare he qualiy of differen neighborhood checking mehods, a hand segmenaion was performed. The sequence was aken ou of a moving car and show an overaking car, which is he objec of ineres. The problem is o find he pixelexac boundary of his objec. Because of floaing color changes he soluion is no unique. Moreover, he shadow of an objec is moving wih he same velociy, so - wihou using objec knowledge - i will be deeced ogeher wih he car as one moion objec, alhough his is usually no waned. In order o solve his problem we segmened he images ino hree regions: foreground F, background B, and undefined U. This is equivalen o a segmenaion of he images ino wo non disjunc classes, which means ha foreground and background are overlapping in he non unique pixels - he undefined region U. The ri-sae hand segmenaion is performed as following: he pixels of an image were assigned o F and B. These wo regions were eroded wih a very small srucure elemen (5 pixel: ), so ha a wo pixel wide region beween F and B is generaed. We define his region as U. Finally he shadow of he car was assigned o U. Fig. 2 and Fig. 4 show some examples of his hand segmenaion. 4.2 Qualiy Measure In order o evaluae he accuracy of he differen mehods of auomaic segmenaion, we define a qualiy measure, ha compares he resuling image wih he hand segmenaion. Le F H denoe he foreground 7 defined by hand segmenaion and F A denoe he foreground deeced by auomaic moion segmenaion. Tha we define c = F H F A number of correc pixels, e + = F H F A posiive error, and e = F H F A negaive error. The qualiy q is calculaed as q = c c + e + + e [0, 1] (9) 7 Foreground is he moving objec of ineres. wih he propery e + + e 0 c > 0 q 1 e + 0 e 0 q 0. (10) Obviously boh, an increasing posiive error and an increasing negaive error, lead o a decreasing qualiy. Also a decreasing number of correcly classified pixels lead o a decreasing qualiy. Wih = c+e he rue number of pixels we ge q = c c + e + + e = e + e +. (11) The dependency of in (11) leads o a scaling invariance of he qualiy measure, so changing he image resoluion doesn change he value of qualiy measure. Posiive and negaive errors are no weighed equal. A posiive/negaive error of 1 pixel causes a change in qualiy of: q + = + 1 q = 1 = 1 1 + 1 = 1 1 (12) (13) Imagine a moion objec wih 100 pixels. In he case of a posiive error of 50 pixels he qualiy is 100 = 0.6 6, in he case of a negaive error of 100+50 also 50 pixels he qualiy is 100 50 = 0.5. Bu if 100 e +, e his problem disappears. An alernaive qualiy measure which weighs boh errors equal is q al = + e + + e = 1 e + e + + e e + + e + + e.(14) However we prefer q as qualiy measure, because q al canno become zero bu in he wors case for an image wih N pixels q al = + (N ) {z } e + + {z} e = + N. (15) Fig. 3 and Fig. 5 shows same racking resuls wih is respecive correc number of pixels c, posiive and negaive error e +, e, and qualiy q.
5 Resuls 5.1 Qualiy Comparision Table 1: Resuling qualiy values of combinaions of neighborhood checking mehods wih moion similariy funcions mehod A B C graviy disance D = 25 85 % 75 % 71 % bounding box 86 % 70 % 72 % convex hull T = 4 86 % 71 % 73 % boundary based 86 % 69 % 72 % A B C similar predicions similar rajecories correlaed rajecories The rajecory based approaches (similar rajecories and rajecory correlaion) need more moion informaion (longer rajecories) for a moion segmenaion han he predicion based approach. Boh [5] and [2] use only rajecories consising of a leas 3 moion vecors for he moion segmenaion. Thus, he qualiy in he firs 3 images 8 is zero, because no moion segmenaion could be done. This also reduces he average 9 qualiy. In conras o ha a moion predicion can be calculaed from rajecories consising of only one single moion vecor. Depending on parameers he predicion similariy mehod showed already qualiies of 70% o 80% in he firs image. 5.2 Performance Comparision Table 2 shows he average number of neighborhood relaions in a sequence of vehicle guidance. Assuming ha he boundary based approach finds he correc neighborhoods all he oher mehods find oo many - or in case of convex hull, T = 0 oo few - neighbors. The convex hull wih T = 0 can find all neighbors, because ouching polygons were no recognized as a neighborhood bu only overlapping 10 polygons. Assuming ha processing ime for moion analysis grows linear wih number of neighborhoods he 8 For simplificaion we equae number of frames and number of moion vecors alhough calculaing rajecories of lengh n uses n + 1 frames. 9 The average wihou firs hree frames is abou 5% greaer han Table 2: Average number of neighborhoods. Absolue and relaive o boundary based mehod. mehod absolue relaive graviy disance D = 50 1247.4 578% D = 25 577.5 268% bounding box 237.1 110% convex hull T = 4 298.8 138% T = 0 174.6 81% boundary based 215.8 100% boundary based neighborhood analysis shows bes performance. The average processing ime was abou 200 ms on a sandard PC, so he n:m maching sysem runs a 5 fps (image size is 340 275). Noe ha abou half of he ime is needed for he CSCsegmenaion. So real ime processing a 25 fps can be nearly reached by reducing he image resoluion o he half (170 138) as i speeds up a abou facor 4. For reducing he coss regions wih oo small area aren regarded in n:m maching phase. An oher approach could be combining small regions wih heir neighboring regions. For example a small region R i can be combinded wih region R j when R j is he only neighbor of R i. Then i mus be an inclusion. 6 Conclusions and Fuure Work In his aricle we described differen echniques for analyzing moion similariy and neighborhood relaions beween color regions. These echniques were successfully applied in an efficien n:m maching sysem. I has been shown ha differen moion similariy funcions lead o varying racking resuls. Alhough in each case he main par of he moving objec has been racked over he sequence he qualiy values vary significan. The approach using similar moion predicion has shown he bes qualiy values independen of a cerain neighborhood checking mehod. In conras o his differen neighborhood checking mehods do no vary he racking qualiy releaverage over all frames. 10 A leas one corner of a polygon mus lay inside he oher polygon or ono is boundary.
van. Alhough he boundary based neighborhood analysis doesn show beer bu similar qualiy values han he oher mehods we prefer i because of is beer performance. Bu he beer accuracy of his neighborhood analysis should show increasing qualiy values for racking non-convex objecs in naural sequences. For example a group of people walking in same direcion could be separaed beer in differen moving objecs. To ge more informaion abou he described echniques we inend o make more (imeconsuming) hand segmenaions of sequences in paricular of oher applicaions. 7 Acknowledgemens Thanks o Prof. Dierich Paulus, Prof. Luz Priese, Dirk Balhasar, Sahla Bouaour, Delev Droege, Vinh Hong, and Parick Surm for consrucive criics and proof-reading and Guido Schwab for he work of hand segmenaion. References [1] Tim Ellis and Ming Xu: Objec Deecion and Tracking in an Open and Dynamic World. Proceedings 2nd IEEE In. Workshop on PETS, Kauai, Hawaii, USA, December 9, 2001 [2] Bernd Heisele and Werner Rier: Obsacle Deecion based on Color Blob Flow. Proc. IEEE Conf. of he Inelligen Vehicles Symposium, pp. 282-286, 1995 [3] Bernd Heisele, U. Kreßel, and Werner Rier: Tracking non-rigid, moving objecs based on color cluser flow. Proceedings IEEE Conf. of Compuer Vision and Paern Recogniion, pp. 257-260, 1997 [4] B. Melzer, A. Miene, and Th. Hermes: Bewegungsanalyse in Bildfolgen auf Basis eines n:m-machings von Farbregionen. Tagungsband 8. Workshop Farbbildverarbeiung 2002, pp. 103-110, Ilmenau, 10.-11. Oc. 2002 [5] Volker Rehrmann: Objec Oriened Moion Esimaion in Color Image Sequences. ECCV 1998, Vol. I, LNCS 1406, pp. 704-719, 1998 [6] Volker Rehrmann and Luz Priese: Fas and Robus Segmenaion of Naural Color Scenes. Proc. of 3rd Asian Conf. on Compuer Vision, Special Session on Advances in Color Vision, Vol. I, pp 598-606. Springer Verlag, 1998 [7] Marin Rohhaar: OOMECS Objec-oriened Moion Esimaion in Color Image Sequences. Diploma Thesis, Universiy Koblenz, Koblenz,1996 [8] Rainer Schian: Auomaische Bildauswerung zur dynamischen Schielwinkelmessung bei Kleinkindern und Säuglingen. Disseraion, Universiy Koblenz, Koblenz, 1999 [9] Bern Schiele: Model-Free Tracking of Cars and People based on Color Regions. Proceedings 1s IEEE In. Workshop on PETS, Grenoble, France, March 31, 2000 [10] Homepage of Image Recogniion Lab Labor Bilderkennen of Universiy Koblenz: www.uni-koblenz.de/ lb/
a) Firs Frame 24707.ppm a) Firs Frame 001.ppm b) Median Frame 24816.ppm b) Median Frame 049.ppm c) Las Frame 25002.ppm Figure 2: Example frames of sequence 1 lef: Original, righ: ri-sae hand segmenaion B: ligh, F : medium, U: dark c) Las Frame 099.ppm Figure 4: Example frames of sequence 2 Lef: Original Righ: ri-sae hand segmenaion B: ligh, F : medium, U: dark a) 24707.eps b) 24804.eps correc e + e qualiy a) 3796 0 908 81 % b) 3950 588 596 77 % Figure 3: Tracking resuls in sequence 1 Green: approximaion of convex hull Table: correc pixels, pos./neg. error, and qualiy a) 037.eps b) 045.eps correc e + e qualiy a) 1999 3 191 91 % b) 1103 4 1156 49 % Figure 5: Tracking resuls in sequence 2 Green: approximaion of convex hull Table: correc pixels, pos./neg. error, and qualiy