Color Image Segmentation Based on Adaptive Local Thresholds

Transcription

1 Color Iage Segentaton Based on Adaptve Local Thresholds ETY NAVON, OFE MILLE *, AMI AVEBUCH School of Coputer Scence Tel-Avv Unversty, Tel-Avv, 69978, Israel E-Mal * : llero@post.tau.ac.l Fax nuber: Abstract The goal of stll color age segentaton s to dvde the age nto hoogeneous regons. Obect extracton, obect recognton and obect-based copresson are typcal applcatons that use stll segentaton as a low-level age processng. In ths paper we present a new ethod for color age segentaton. The proposed algorth dvdes the age nto hoogeneous regons by local thresholds. The nuber of thresholds and ther values are adaptvely derved by an autoatc process, where local nforaton s taken nto consderaton. Frst, the watershed algorth s appled. Its results are used as an ntalzaton for the next step, whch s teratve ergng process. Durng the teratve process regons are erged and local thresholds are derved. The thresholds are deterned one-by-one at dfferent tes durng the ergng process. Every threshold s calculated by local nforaton on any regon and ts surroundngs. Any statstcal nforaton on the nput ages s not gven. The algorth s found to be relable and robust to dfferent knd of ages. Key words: Local thresholds, Iage segentaton, Hoogenety, Splttng, Mergng.

2 1. Introducton Iage segentaton parttons an age nto non-overlappng regons. A regon s defned as a hoogeneous group of connected pxels wth respect to a chosen property. There are several ways to defne hoogenety of a regon that are based on a partcular obectve n the segentaton process. For nstance, t ay be easured by color, gray levels, texture, oton, depth of layers, etc. Overlaps aong regons are not pertted, thus, each pxel belongs only to a sngle regon. Two neghborng regons should be erged f the new cobned regon s hoogeneous. Consequently, each regon s antcpated to be as large as possble under ts certan characterzaton. Then, the total nuber of regons s reduced. Snce segentaton defnton s nforal, t s very dffcult to propose a seantcs to easure the qualty of a gven segentaton, unless the segentaton s goal s well defned. Iage segentaton has a varety of purposes. For exaple, segentaton plays an portant role n the feld of vdeo obect extracton [1],[],[3]. Snce hoogeneous regons correspondent to eanngful obects (whch are ostly nhoogeneous), any of the vdeo obect extracton algorths frst partton the age nto hoogeneous regons, and then, n order to extract the ovng obect, the regons are erged accordng to teporal nforaton of the sequence. In age copresson [6],[7],[8], the nput age s dvded nto regons that should be separately copressed snce better copresson s acheved as long as the regons are ore hoogeneous. Trackng systes that are regon-based [9],[10],[11] utlze the nforaton of the entre obect's regons. They track the hoogenous regons of the obect by ther color, lunance or texture. Then, a ergng technque that s based on oton estaton s used n order to obtan the coplete obect n the next frae. Iage segentaton s also used n obect recognton systes [4],[5]. Many of these systes partton the obect to be recognzed nto sub-regons and try to characterze each separately n order to splfy the atchng process. Autoatc segentaton n stll age has been nvestgated ([1],[13]) by any researchers fro dverse felds of scences. The exstng segentaton ethods can be dvded nto the followng an approaches: Hstogra-based ethods, boundary based ethods, regon-based ethods, hybrd-based ethods and graph-based technques. Hstogra-Based ethods: Most of the hstogra based algorths deal wth gray level ages, whch are represented as one densonal hstogra. The range of ntenstes s assued to be constant. The hstogra s consdered as beng a probablty densty functon

3 of a Gaussan and the segentaton proble s reforulated as a paraeter estaton followed by pxel classfcaton [13]. However, color ages are usually represented by three-densonal (3-D) bands as GB or soe transforaton of GB. Hence, selectng global thresholds n 3-D hstogras s a dffcult task. In order to deal wth 3-D color hstogras, soe technques [16], [17] were developed to proect the 3-D color space onto -D or even 1-D surface, and analyze the obtaned surface for the segentaton process. Other technques [14],[15] transfored the 3-D hstogra nto a bnary tree such that each node represents a dfferent range for GB key values. The nuber of the GB ponts that are represented by each node s transfored to the key value of the node. Later, Cheng and Sun [18] extended the general 3-D hstogras to hoogeneous doan hstogra. They defned hoogenety as a coposton of two coponents: standard devaton and dscontnuty of the ntenstes, and used the hoogenety hstogra to detect unfor regons. Then, for any regon the tradtonal hstogra, based on Hue color feature, s constructed and hstogra analyss s perfored. However, ost of the hstogra based ethods were found to feet specfc ages. These ethods acheved reasonable perforance when the nput s characterzed wthout nose and wth sall nuber of regons. Moreover, the nuber of potental segentaton classes n the age s usually assued to be known beforehand. Boundary-based ethods: These ethods search for pxels that le on a regon boundary (or at the boundary between two regons). These pxels are called edges []. An edge s characterzed by a sgnfcant local change n age ntenstes. Edges are detected by lookng at neghborng pxels. The basc assupton s that the change n pxels values between neghborng pxels nsde a regon s not as sgnfcant as the change n pxels values on the regons boundary. When the dfference between two regons grows, the change becoes bgger and the edge becoes stronger. Soetes weak edges should be detected as strong edges and n other tes they should not. Consequently, not all the detected edges create closed curves, whch are necessary to separate between regons. Therefore, soe type of post/pre-processng technques, such as [19],[0],[1] are requred for groupng the detected edges nto a connected surfaces to represent the regon. In [3] an EdgeFlow ethod was presented that s based on the edge drectons rather than the edge energy. They detected the regons boundares by dentfyng a flow drecton at each pxel locaton that pont to the closest boundary. Then, t follows by detecton of the locatons that encounter two opposte drectons of edge flow. However, the an drawback of any 3

4 boundary based ethods s the over-segentaton result, whch s not always correctly reflects the age nature. egon-based ethods: These ethods gather slar pxels accordng to soe hoogenety crtera [4],[5] and [6]. They are based on the assupton that pxels, whch belong to the sae hoogeneous regon, are ore alke than pxels fro dfferent hoogeneous regons. The splt-and-erge or the regon-growng technques are exaples for such ethod [7],[8],[30],[31],[3]. The regon growng algorth ntally defnes each pxel as a regon. Then, t scans the age fro left to rght and fro top to botto and copares the current pxel wth ts neghborng regons that were already scanned. If the pxel s suffcently slar to one of ts adacent regons t s added to that regon. If t s not close enough to any of the, then t s stll defned as a dfferent regon. On the contrary, the splt-and-erge technques [9] ntally assue that the age s coposed of one regon. It splts nhoogeneous segent nto four rectangular segents and erge four adacent regons f they are found to be slar. When no regon can splt and no four adacent regons can erge the algorth s ternated. Two an drawbacks characterze these technques. They are both strongly dependent on global pre-defned hoogeneous crtera thresholds whle the regon growng technque depends also on ntal segents, whch s the frst pxel/segent, that s frst to be scanned and on the order of the process. Hybrd-based ethods: These ethods prove the segentaton result by cobnng the above ethods for segentaton. Many of the hybrd technques cobne the regonbased ethod wth the boundary-based ethod. Soe used the cobnaton of the hstogra-based wth the regon-based ethods. The hybrd technque for segentaton s very coon snce t reles on wde nforaton as global (hstogra) and local (regons and boundares). An exaple of an hybrd technque was presented n [35], whch ntegrated between regons based and boundary based ethods. Frst a splt-and-erge algorth s perfored n order to ntally segent the age. Then, the contours of the obtaned regons are refned usng the edge nforaton. Later, the watershed algorth [33],[34] was presented. It begns wth a boundary based ethod to get gradent agntude. Then, regons are produced by a regon growng technque. In [36], K. Hars et al presented a segentaton algorth usng the watershed algorth [33] and regons ergng. They appled the watershed transfor to ntal partton the age nto prtve regons. The output of the watershed used as an nput for herarchcal (botto-up) regon ergng process, whch produced the fnal segentaton. 4

5 Graph-Based technque: Specal graph algorths have been adopted for segentaton. These algorths typcally construct a graph n whch the nodes represent the pxels n the age and arcs represent affntes between nearby pxels. The age s segented by nzng the weght, whch s assocated wth cuttng the graph nto subgraphs. In a spler verson, the weght s the su of the affntes across the cut [41]. Other versons noralze ths weght by dvdng t by the overall area of the segents [38] or by a easure derved fro the affntes between nodes wthn the segents [39],[40]. Noralzng the weght of a cut prevents over-segentaton of the age. In [37] a fast, ultscale algorth for age segentaton was ntroduced. The algorth uses algebrac ultgrd (AMG) solvers to fnd an approxate soluton to noralzed cut easures n te that s lnear n the age sze. It detects the segents by applyng a process of recursve coarsenng n whch the sae nzaton proble s represented wth fewer and fewer varables producng an rregular pyrad. We propose a new ethod for stll color age segentaton, whch s based on adaptve autoatc dervaton of local thresholds through an teratve procedure, where local nforaton s taken nto consderaton. The algorth s coposed of two an steps. Intally, the age s dvded nto a large nuber of regons usng the applcaton of watershed algorth. The egon Adacency Graph (AG) s the data structure we use to represent the age partton. The second step s an teratve process, n whch regons are erged and local thresholds are derved. The order n whch the ergng process takes place s based on Kruskal s algorth [43] for fndng nu spannng tree n a graph. Durng the ergng process we follow the changes of each regon and save the changes of the regons characterstcs. By analyzng these changes we dentfy where durng the ergng process each regon becoes nhoogeneous. Then, local thresholds are derved. To cancel the erge that produces nhoogeneous regon, all the erges are canceled oneby-one fro the end untl that erge s reached. The two regons, whch ths erge refers to, are dentfed as non-ergeable regons, and are consdered as fnal regons. The algorth nether assues any gven paraeters nor any gven thresholds. The nuber of thresholds and ther values are known only when the process s ternated. The segentaton result s the partton of the age, whch s obtaned by the fnal regons. The rest of the paper s organzed as follows. Secton descrbes the proposed ergng ethodology and ts relaton to the nu spannng tree algorth. In secton 3 we descrbe the core process that assgns local thresholds based on local consderaton of each regon. Secton 4 presents all the steps of the segentaton algorth and analyzes the 5

6 overall te coplexty. Experental results are gven n secton 5 and we conclude ths paper n secton 6.. Mergng Methodology The proposed algorth pre-segents the age (secton.1) usng the watershed algorth, whch generates an over-segentaton output. Next a ergng process s appled. The ergng process deals wth the followng three ssues: () the dsslarty between regons (secton.), () MST constructon (secton.4). () coputaton of local thresholds (secton 3).1 Intal Segentaton Usng the Watershed Algorth The nput to the watershed algorth s a gray-scale gradent age. Thus, we frst convert our nput (color age) I nto a gray level age. Then, Canny edge detecton [0] s appled to get ts gradent agntude age, denoted by I G. The gradent age s consdered as a topographc relef. Each pxel s value (gray level) stands for the evaluaton at that pont. The algorth defnes catchent basns and das. Each catchent basn, whch s assocated wth a nu M s a set of connected pxels such that a drop of water fallng fro any pxel that belongs to ths catchent basn, fallng down untl t reaches the nu M. On ts way down the drop passes only through pxels that belong to ths catchent basn. Das are watershed lnes. They are pxels that separate dfferent catchent basns. A drop fallng fro one sde of a da reaches the nu of one catchent basn whle a drop fallng fro ts other sde reaches the nu of another catchent basn. Catchent basns and das are llustrated n Fgure 1. Fgure 1: Catchent basns and das. Low and heght level pxels are llustrated by the lower and upper red arrows, respectvely. 6

7 We apply the Vncent and Solle [33] verson of the watershed algorth, whch s based on erson sulaton: the topographc surface s ersed fro ts lowest alttude untl water reaches all pxels. The algorth contans two steps: sortng and floodng. At the frst step the pxels are sorted n an ncreasng order accordng to ther ntenstes. Then, at the floodng step the pxels are scanned by the sortng order to construct catchent basns. Each catchent basn s assgned a dfferent label. At a pont where water coes fro dfferent catchent basns, da s constructed. At the end of the process a tessellaton of the age nto catchent basns (by ther labels) s produced. Fgure deonstrates the watershed result. Fgure a s the nput age. Fgure b s ts gray level age. The gradent age after applyng Canny edge detector s shown n Fgure c and the segentaton, that s generated by the watershed algorth s shown n Fgure c. The output of the watershed algorth s segentaton of I G nto a set of n nonoverlappng regons. Snce these regons are gong to be erged durng the next ergng process we denote the by, = 1,..., n, = 1,..., M, where n s the nuber of regons, and M s the nuber of erges of durng the ergng process. 0, = 1,..., n s the set of ntal regons, whch s the output of the watershed algorth before the teratve ergng process of the second step starts. (a) (b) (c) (d) Fgure :(a) The nput color age. (b) Its gray-level age. (c) The gradent age after applyng Canny edge-detector. (d) The segentaton result by the watershed algorth. 7

8 . Dsslarty Measure between egons To deterne the ergng order dsslarty functon between any two neghborng regons, and, denoted by ( ) f,, s defned. The functon s based on two coponents: color and edges. The Hue coponent of the HSV color space [4] s used for the color coponent snce t less nfluenced by changes n llunaton such as shade and shadow. The ean value of the hue coponent of a gven regon by ( ) h s denoted µ. The gradent agntude s used as another source of local nforaton for the second coponent. We denote by (, ) µ the ean gradent between and, G whch s calculated fro the gradents aong the shared pxels between the two regons. The values of the pxels n I G are used to get the agntude of the gradents. Let set of pxels whch are the boundary of and. (, ) µ s defned as G B be the (, ) I ( x y) G, ( x y ) BIJ =, µ G (1) B where B denotes the nuber of pxels at B. The dsslarty functon s defned as where d ( ( ) µ ( ) value of h ( ) w d( µ ( ), µ ( ) + w ( ) f =, () h, 1 h h µ G µ, s the dfference between the ean value of and the ean ( µ ( ), µ ( ) n{ µ ( ) µ ( ), (360 µ ( ) ( ) )} d = µ (3) h h h h h h and w 1 and w are predefned constant coeffcents. The dsslarty functon s ostly based on the hue color space rather than the gradent agntude, thus, w 1>> w. Based on experents on dfferent ages w 1 and w set to 0.8 and 0., respectvely. 8

9 .3 AG Data Structure The regon adacency graph (AG) s the data structure that s beng used to represent the partton of the age. The AG that represents the set of regons undrected graph G = ( V, E). V { 1,,..., n} and e( ) E 0, = 1,..., n s an = such that each regon s represented by a node,, f, V and the regons and are adacent. Snce the ergng process s based on G, each edge s assgned a weght. The weght of an edge ( ) value of ( ) e, s the f,, calculated by Eq.(). An exaple of an age that contans sx regons wth ts correspondng AG s shown n Fgure 3. The sx regons are represented by sx nodes and the edges correspond to the neghborng lst. For nstance, the four edges that are connected to node 1, e ( 1, ), e ( 1,3), e ( 1,4 ) and e ( 1,5 ) regons:, 3, 4 and 5., represent ts four adacent (a) (b) Fgure 3: (a) Sx parttons of the age and (b) ts correspondng AG..4 On the elaton between MST and the Mergng Process The ergng process s based on Kruskal s algorth [43] for fndng a nu spannng tree (MST). Let G ( V, E) of I where the weght of ( ) = be the AG, whch represents the ntal segentaton e, s the value of ( ) f,. A spannng tree of G s defned as a connected acyclc subgraph that spans all ts nodes. Every spannng tree of G has n 1 edges where V = n. When each edge has a weght, a nu spannng tree s a spannng tree of G that has the lowest total weght of ts edges easured as the su of the weghts of the edges n the spannng tree. Kruskal s algorth generates the nu spannng tree, denoted as T, fro scratch by addng one edge at a te. Intally, the edges of G are sorted n a non-decreasng order 9

10 of ther weghts. Then, the edges n the sorted lst are exaned one by one and checked whether addng the edge that s currently beng exaned creates a cycle wth the edges that were already added to T. If t does not, t s added to T. Otherwse, t s dscarded. The process s ternated when T contans n 1 edges. At the end of the process T s the nu spannng tree of G. We apply Kruskal s algorth on G whle focusng on the process tself. The process that constructs the MST s the process that erges regons as descrbed below: - Addng ( ) e, to T represents the erge of ts two correspondng nodes (ts two regons) and. - Addng the edge wth the nu weght one-by-one n an ncreasng order to T (usng the sorted lst) s equvalent to the erge of the two ost slar regons. - When an edge s reected because t creates a cycle n T, no erge s perfored because ts two regons have already been erged nto one regon. - At the end, when T spans all the nodes, all the regons were erged nto one regon, and the ergng process s ternated. Fgure 4 s an exaple of a weghted graph G. The black thck edges n Fgure 4b are the edges of ts nu spannng tree. After ts edges were sorted, the MST constructon, whch represents a ergng process, proceeds as follows: e ( 4,6) wth weght 10 s added frst to T snce t has the lowest weght aong the edges n the sorted lst, hence, 4 and are erged. Second, (,3) 6 e s added to T, hence, and 3 are erged. The thrd edge s e ( 3,4), hence, 3 (that already erged wth ) s erged wth 4 (that was already erged wth 6 ). The fourth edge s e ( 1,5), hence, 1 and 5 are erged. e (,4) s exaned next. Snce e (,4) creates a cycle wth e (,3) and e (,3) that have already been chosen to be n T, t s reected. Dscardng e (,4) eans that no erge takes place snce, 3, 4 and 6 have already been erged nto one regon. Next, ( 1,4 ) e s added, hence, 1 s erged wth 4. Then, the constructon of the nu spannng tree s ternated snce T contans fve edges that span ts sx nodes. Hence, all the regons have been erged nto one regon and no further erge s possble. 10

11 (a) (b) Fgure 4: (a) The source graph. (b) Its nu spannng tree s ndcated by the black bold edges. Durng the MST constructon process T s not necessarly connected. Mnal forests ay be created. Growng forests, and not only one tree, s equvalent to ergng regons at dfferent locatons n the age. Not only one regon ay grow but any, unlke the tradtonal regon-growng ethod where pxels or regons are scanned n a predefned order and the generated regons are eerged and expanded fro one locaton. After any erge of any two regons a new regon s generated. As a result, there s new nforaton about ths regon and ts new surroundngs whle the prevous nforaton becoes rrelevant. Therefore, we have to follow the changes of the regons durng the process and use the current updated nforaton. Let that s generated by the erge of neghbors of the new regon wth, where ( ) ( v v f, v ) for every v N( ) Snce the values of ( v v f, ) for every N( ) = be the new regon and let N( ) N( ) N( ) = be the N s the neghborng regons of. Then s recoputed by the new odfed nforaton. v are the weghts of the correspondng edges n G, the edges and the sorted lst have to be updated to nclude the new values after ( v ) f, has been updated. v Thus, the coplete ergng process s based on Kruskal's algorth for MST constructon wth the followng odfcaton: When an edge, say e (, ) T has less than 1 n edges the sorted lst s updated. ( v ) v v, s added to T and f, s calculated for any new regon and ts neghbors, and then the weghts of the edges 11

12 k l { e( k) e(, l) N( ), N( ), k, l }, are updated. These are the edges that are connected to k l and the edges that are connected to generated durng any erge. Two edges, e (, ) and ( u v). Parallel edges ay be e,, are consdered to be parallel f u v v already erged wth (or wth ) and already erged wth (or wth u u u v, respectvely). Snce only one of the parallel edges ay be added to T, one of the edges s assgned the cost of the dsslarty functon and the other s assgned " ". The edge, whose cost s assgned to be " ", s added to the end of the sorted lst and wll not be exaned. Hence, " " ndcates that no further consderaton whether to add t to T s requred. Fgure 5 llustrates an updatng process. When the frst edge e ( 4,6) s added to T, e ( 3,4), e (,4) and ( 1,4 ) 1 4 e have to be updated. f ( ) 4, f ( ) 3 4 and f ( ) are recalculated and the new values, 61, 50 and 8 are the new edge costs, respectvely. e (,3) s the next edge wth the nu weght that s added to T. Four edges have to be updated: e (,1), e (,4), e ( 3,1 ) and e ( 3,4). Snce e (,1) and e ( 3,1 ) becoe parallel, we assgn to one of the the new value 73 and the other edge s assued to have " ". The sae s done for e (,4) and e ( 3,4). The weght of e ( 3,4) s 4 and the weght of e (,4) s " ". 1, 4, 4 v 3, 4 (a) (b) (c) Fgure 5: Updatng edges durng the applcaton of the MST constructon. (a) The source graph. (b) Addng 4,6 e,3 to T. e ( ) to T. (c) Addng ( ) 1

13 3. Fndng Adaptve Local Thresholds Although the descrpton of the ergng process s copleted, t should be decded when the process has to be ternated. In other words, t s unknown how to deterne whch regons should not be erged and when. We descrbe here an autoatc procedure that derves local thresholds by followng the changes of each regon durng the applcaton of the ergng process. These thresholds wll be the ndcaton whether or not a certan regon should be erged. Hence, these thresholds generate the fnal segentaton. 3.1 The Need for Local Inforaton Snce we consder the segentaton process to be a local operaton we can assue that not all the local erges wll be ternated sultaneously. The use of one global threshold s nsuffcent because dfferent regons are usually separated fro ther surroundngs at dfferent tes durng the process wth dfferent thresholds. However, there are soe cases, where one global threshold s suffcent. The exaple n Fgure 6 descrbes an exceptonal stuaton where a good segentaton can be obtaned by a sngle global threshold. Ths s possble snce the age contans one obect, whch s hoogenous n ts colored texture, and so s ts surroundng background. In ths case, one threshold was used n the ergng process. The process was ternated when the weght of the exaned edge was hgher than the chosen threshold, whch was set to be 100. The segentaton result s deonstrated n Fgure 6b. Snce n ost of the cases the age contans ore than two hoogenous regons, t s obvous that t s dffcult to predct whether one global threshold can handle a gven nput age. (a) (b) Fgure 6: (a) The source age. (b) Segentaton result obtaned by the global threshold

14 Fgure 7 llustrates the reason why local thresholds are needed nstead of usng one global threshold. Fgure 7a s the source age. Fgure 7b s the output of the watershed algorth. Fgure 7c s the result after usng global threshold, t=0. Fgure 7d s the result after usng global threshold, t=30. In Fgure 7c all the regons are hoogenous and can grow. However, as the threshold ncreases to 30, regons such as the face and the sofa, whch are consdered vsually as hoogenous, are stll over-segented, whle the regon, whch s ndcated by the yellow arrow, s nhoogeneous. Thus, the constructon of that regon should be ternated at t<30 n order to prevent the erge of the two dfferent hoogenous regons: the an s acket and the sofa. (a) (b) (c ) (d) Fgure 7: (a) Orgnal nput age. (b) After the applcaton of the watershed algorth. (c) After the copleton of the ergng process usng one global threshold t=0. (d) After the ergng process usng one global threshold t=30. The calculaton of local thresholds wll be based on local nforaton, whch s related to the regons and ther surroundngs, snce regons are affected by ther surroundngs. The dependency between regons and ther surroundngs, whch causes the sae regon n dfferent surroundngs to be vsually dfferent, can be sply deonstrated. In Fgure 8a the yellow ellptc obect s clearly seen and t s well separated fro ts background whle the sae yellow ellptc obect wth dfferent background s alost nvsble n Fgure 8b. 14

15 (a) (b) Fgure 8: The sae brght ellptc obect appears dfferently due to ts dfferent backgrounds. 3. Coputaton of a Local Adaptve Threshold We present an autoatc ethod that calculates adaptve local thresholds. The ethod s based on local propertes of the regons durng the ergng process. Proposton 1: A sgnfcant change n the hoogenety of a gven regon occurs durng a erge that generates nhoogeneous regon. At ths erge, local threshold s deterned. The dentfcaton of hoogenety s anly based on color space. We use the V coponent of the HSV color space to calculate the varance of the hoogenety of a gven regon. Let ( ) µ be the ean V value of v and let V ( x, y) be the value of V n locaton ( y) x,. The varance of any regon n = 1,..., after ts th erge s defned as: 1 ( ) = ( V ( x, y) µ v ( ) σ (4) ( x, y) where s the sze of. We defne the change n the hoogenety of after the th erge to be 1 ( ) = σ ( ) σ ( ) σ. (5) Let J be the set of σ, = 1,...,. K local axus of ( ) M J {(, ( ) ( ) ( ) ( ) ( )} 1 + σ > σ & σ > σ 1 = σ. (6) 15

16 Although no statstcal nforaton on the age s gven, local nforaton on any regon 1,..., n = s obtaned fro σ ( ) the erges n J represent sgnfcant transtons of. Snce the varance s a easure for hoogenety, durng the ergng process. Gven that the ergng process begns wth over-segentaton of hoogeneous regons and the regons are erged untl one regon s left, every regon becoes nhoogeneous at a dfferent erge operaton. Hence, we argue that local axu n J that satsfes: where β s the ean value of σ ( ) ( ) β becoes nhoogeneous at the frst σ > (7) at every J defned as Due to the unque behavor of σ ( ) J ( ) the local axus that refer to the erges n whch plots n Fgure 9 llustrate the behavor of σ ( ) β = 1 σ K. (8) (see Fgure 9) the defnton of β enables to reect s stll hoogenous. The three of three dfferent regons of the an s shrt (Fgure 7b), whch reflects the changes n the hoogenety. The plots descrbe the values of σ ( ) as a functon of the nuber of the erges. The green arrow n each plot ponts to the frst local axu, aong all the local axus, that satsfes Eq. (7). In ths erge, the shrt regon s erged wth another regon (the brght background) and becoes nhoogeneous. Snce these three regons were erged nto one regon, the values of the plots n Fgure 9a and Fgure 9b are equal fro the ffth erge of Fgure 9a and the frst erge of Fgure 9b. In addton, fro the thrd erge of the regon n Fgure 9c and fro the nnth erge of Fgure 9b (or the th 13 erge of Fgure 9a) the values σ ( ) of the plots n Fgure 9c and n Fgure 9b are equal. Moreover, the dentfcaton of the erge that generates nhoogeneous regon, whch s equal too, s ndependent on whch regon (aong all the regons that copose the hoogenous regon, the an s shrt) we exane. 16

17 (a) (b) (c) Fgure 9: The representaton of σ ( ) x-axs s the nuber of erges. The y-axs s σ ( ) of three dfferent regons of the an s shrt (fro Fgure 7b). The. The green arrow n each plot ponts to the frst local axu that satsfes Eq.7, aong all the local axus, whch ndcates the erge where the regon becoes nhoogeneous. The erge that generates nhoogeneous regon has to be canceled. Assue that and are the two regons that by ther erge nhoogeneous regon was generate. A local threshold s derved and ts value s the value of ( ) f, of that erge. Because of the erge order any other erge of wth any of ts neghbors wll generates nhoogeneous regon. Therefore, ths threshold prevents these two regons fro beng erged durng the proceedng operatons. As was entoned, an teratve process s appled n order to derve the thresholds. More precsely, any teraton obtans a sngle threshold. Let s = 1,..., K be the ndex of the teratons nuber. K s currently unknown snce the nuber of thresholds (teratons) s unknown. Let t s be the threshold of the assocated wth t s. Durng the s th teraton and let t s be the erge that s s th teraton regons are erged accordng to the ergng process untl one regon s left (except fro fnal regons that are dscussed bellow). For every regon { 1 M },..., :, 1,..., n = we get fro that process a ap { 1,..., M } { 1,..., M } 1 n L for every { 1,..., } to M L : =,..., (9) where M s the total nuber of erges n the current teraton. L ( ) = eans that the th erge of s ts th erge aong all the M erges For exaple, = 5 and ( ) = 7 L eans that the ffth erge of 5 s the 7 th erge aong all the M 17

18 erges of all regons. For every regon let J, be the frst local axu that satsfes Eq.(7). The t s erge of the current threshold s defned to be s = 1,..., n { L ( )} t = n. (10) At the s th teraton, t s refers to the frst erge aong all the erges n the current teraton, that generates nhoogeneous regon. If were erged at the t erge, the value of ( ) s and are the two regons that f, of that erge s assgned to be the value of t s. Snce t s prevents the erge of and, all the erges fro the fnal erge to the t s erge have to be canceled one-by-one. Ths process s called a regresson process, and t wll be dscussed n secton 3.3. When the t s erge s reached and canceled durng the regresson process, and are arked as fnal regons, and denoted as * and *. They wll rean unerged. Hence, durng the next teraton, the ergng process proceeds and all the regons, except the fnal regons, are erged nto one regon, and the next threshold t s+ 1 wll be derved. Gven that the edges n the sorted lst are exaned except the edges ( k ) are edately reected, snce addng the edge ( k ) * and * are fnal, all { e k = or = }, that e, to T eans that ether or are beng erged. As a result, T at that pont s not a tree but contans three forests: 1. and ts connected nodes.. * and ts connected nodes. 3. All the others nodes. Note that T s a spannng forest such that by addng edges fro ( k ) { e k = or = } *, a nu spannng tree s generated. The teratve process, whch conssts of ergng process, dervaton of local threshold and regresson process, s ternated when no regons to be erged are left and all the regons are arked as fnal regons. 3.3 The egresson Process Durng the regresson process we reove edges fro T. Ths s done n a reverse order to the order they were added: edges are reoved fro botto to top. Addng ( ) eans that and are erged to create ; reoval of ( ) e, to T e, fro T eans a splt 18

19 of nto two dfferent regons and. Hence, all the edges that were added to T are saved n the order they were added. We save these edges n a ergng tree denoted by MT. The root of MT represents the sngle regon that was obtaned at the end of the ergng process and all ts leaves correspond to the nodes of G. At the begnnng MT contans only leaves, and t s constructed fro ts leaves to the root. Each erge operaton creates a new node. Ths node represents the ancestor of the two regons that were already erged. Fgure 10 llustrates the MT constructon, whch s done n parallel to the constructon of the MST of G (the correspondng graph s presented n Fgure 5a). Fgure 10a s the MT at the begnnng of the ergng process. Only the leaves that correspond to the nodes of G are ncluded. The new node n Fgure 10b, whch ponts to ts two 4 6 descendants and 4, was created when ( 4,6) 6 e (see Fgure 5a) was added to T. The 3 new node n Fgure 10c, that ponts to ts two descendants and, was created when e (,3) was added to T. The fnal for of MT s presented n Fgure 10d. MT s constructed by two operatons: 3 MTAdd( ( ) e, ) : Generates a new node whch s the father of and. Saves ( ) e,. Updates LastP: LastP ponts to. MTLast() : eturns the node ponted by LastP and reoves t fro MT. where LastP s a ponter to the last node that was generated and has to be updated when MTAdd( e (, ) ) or MTLast() are nvoked. The operaton MTAdd( e (, ) ) s nvoked when ( ) e, s added to T durng the ergng process. When the ergng process s ternated MT s fully constructed and the regresson process can be appled. Throughout the regresson process, MTLast() s called untl t returns the regon that corresponds to the current threshold. The regon returns by MTLast(), s splt nto ts two descendants: and ( ( ), that e, s reoved fro T ). When MTLast() returns a node that corresponds to the last threshold, the regresson 19

20 process s ternated. Its two descendants are arked as fnal regons. Snce the regresson process s appled n each teraton, MT s constructed n parallel to the ergng process and t used n every regresson process. When the teratve process s ternated, the unerged nodes n MT represent the "fnal regons". When the edge ( ) k p { e(, k) e(, p) ( ) ( )} k N p N e, s added to T, the set of edges E, =, s updated wth the new weghts of the edges. When ( ) e, s reoved fro T, the weghts of the edges n E, are reassgned wth the orgnals weghts (whch are the weghts before ( ) the reconstructon of G by the erge of MTAdd( ( ) e, was added to T ). Therefore, and requred that E, wll be saved by e, ) procedure. Parallel edges, whose weghts are " ", are not saved snce they are not gong to take part n any future ergng procedures. (a) (b) (c) (d) Fgure 10: The MT constructon process that corresponds to the MST constructon of the graph n Fgure 5a. (a) MT at the begnnng. Only the leaves are ncluded. The new node n (b), whch ponts to ts two 4 6 descendants and e 4,6 (see Fgure 5a) was added to T. The new node n (c), 4, was created when ( ) 3 that ponts to ts two descendants and MT. 6, was created when (,3) 3 e was added to T. (d) The fnal 0

21 3.4 Fro local thresholds to adaptve algorth The coputaton of local thresholds s an autoatc process, whch coes fro the cobnaton of local consderaton and the proposed technque. As we have already entoned, the nuber of thresholds s known only when the process s ternated. Usng local nforaton does not necessary assocate wth autoatc procedure. At [18] for exaple, local nforaton s consdered to defne hoogenety hstogra. Then, peakfndng algorth s eployed to dentfy the ost sgnfcant peaks. Snce t uses predefned constant thresholds, sgnfcant global peaks are detected. As a result the obtaned regons are dvded n the next step. The use of local nforaton n every step of our proposed ethod s actually a study process of the age. The nput to the ergng process (the over-segentaton generated by the watershed algorth) and the order of the erges (of nu spannng tree constructon) enable to nvestgate the growng regons: the nforaton s saved, updated and exaned. At any teraton, where sngle threshold s derved, a specfc regon s regarded. The fnal segentaton of ths regon s defned (by 'fnal regon') whle the exanaton process of the other regons of the age contnues slarly. As a result, ths autoatc study process derves adaptve thresholds, whch produce adaptve segentaton. Hence, the proposed algorth s an adaptve algorth. Snce good results are obtaned by adaptve algorths our algorth perfors well on dfferent knd of ages; low contrast ages (regons) are segented (defned) as well as heght contrast ages (regons). 4. Ipleentaton and Coplexty In ths secton we descrbe the flow of the algorth followed by te coplexty analyss. 4.1 Algorthc Ipleentaton Notaton: I s the nput color age, s = 1,..., K s the ndex of the derved thresholds (set to 1). The set, = 1,,... n. = 1,,..., M, represent the regons durng the ergng process where M s the nuber of erges of the regon of the current teraton. 1,,...,, = n s the n fnal regons, whch s the algorth output. 1

22 Process: 1. Apply Canny edge detector on the gray level age of I. Its output I G s the age gradents.. Apply the watershed algorth on I G to get an ntal partton of I. The set 0, = 1,..., n are the age partton after the applcaton of the watershed algorth. 3. Construct a AG, denoted by G, to represent the partton of I. 4. Merge regons: a. Merge the regons nto one regon (except the fnal regons ). The ergng order s based on Kruskal s MST algorth, usng the dsslarty functon ( ) f, (Eq.()). b. Construct the ergng tree denoted by MT. c. For any regon save the set J of ( ), = 1,..., M K local axus of σ that were calculated usng Eq.(6). For every J save ts L ( ). 5. Calculate the s th threshold : a. For any regon calculate β by Eq.(8) and then detect the frst J that satsfes Eq.(7). b. For any regon use the ap L n Eq.(9) to detect the t s erge (defned by Eq.(10)), whch s assocated wth the t s threshold. Assue that and are the regons that correspond to t s. c. t = f (, ). s 6. Apply the regresson process: reove all the erges fro botto to top, usng the MT data structure, untl the erge of and, whch s related to the last threshold, s reached. 7. and are defned as fnal regons and denoted by and, respectvely.

23 8. If regons to be erged rean, set s = s + 1 and repeat step 4. Otherwse, ternate the process and let 1,,...,, = n be the segentaton result. 4. Coplexty Analyss The overall te coplexty of the algorth s O( N K E log E ) + where E s the nuber of edges n G, N s the age sze and K s the nuber of teratons. Next we analyze the coplexty for the ost expensve steps n the algorth. The te coplexty of the watershed algorth s lnear n the age sze N. A dscusson on ts coplexty s gven n [33]. Any teraton contans the ergng process, the dervaton of the current threshold and regresson. The ergng process takes ( E E ) O log operatons, whch s the su of the followng three procedures: 1. Sortng the edges of G requres ( E E ). Updatng the sorted lst requres ( nc E ) O log operatons. O log operatons: If C s the axu degree n G then, when an edge s added to T, O ( C) edges n the sorted lst are updated. If the sorted lst s pleented by a heap data structure, the update can be done n ( C E ) erges we get ( nc E ) O log operatons. Hence, for the worst case, whch contans n 1 O log operatons. 3. The constructon of MT requres ( nc) O operatons: Snce n 1 edges are added to T, MT s bult n n 1 operatons. Each operaton follows by addng a new node, whch s done n O () 1 operatons. Savng the edges of the erged regons requres O ( C) operatons. The total coplexty of the above three procedures s O ( E log E + nc log E + nc) E = O( nc) the coplexty s ( E E ) O log.. Snce When the ergng process s copleted, the current threshold s derved n O ( n) operatons: β s calculated n O ( n) operatons, whch s the nuber of regons that have been erged n the worst case (the frst teraton). Then, for every regon we scan all ts erges to detect the erge J that s the frst erge that satsfed Eq.(7). For the n 3

24 regons ( n ) operaton. We reduce ( n ) O operatons are requred. Fnally, usng Eq.(10) s O to ( n) t s derved n O ( n) O as follows: For every regon we save the set J and β value such that a drect access to J σ takes O () 1 operatons. In addton, we use an array of sze M such that the par ( ),, where and and L ( ) =. For exaple 31 and ts ( ) th entry, = 1,..., M, represents the are the two regons that are erged such that L ( ) = = and ( 5,8) eans that the 31 th erge aong all the erges s the ffth erge of and s the eghth erge of. Then, by one scan of ths array t s s derved. Hence, t s un-necessary to fnd the frst local axu of every regon that frst satsfes Eq.(7). Snce M n the worst case s O ( n), one scan of the array requres O ( n) operatons. () 1 In each step durng the regresson process the last new regon s deleted fro MT n O operatons. Snce the worst case deands a reoval of n 1 prevous erges and update of O ( C) edges (n the heap) durng each step then, the total nuber of operatons for the regresson s ( nc E ) process takes O( K E log E ) O log. If the nuber of teratons s K, then the whole teratve operatons. By addng to t the te coplexty of the watershed transfor, the overall coplexty s O( N K E log E ) +. Soe exaples and further dscusson on executon te are gven at the experental results secton (secton 5.). 4

25 5. Experental esults 5.1 Step-by-Step Executon of the Algorth Fgure 11, Fgure 13 and Fgure 14 deonstrate step-by-step the nteredate results of the segentaton process. Dfferent types of ages wth dfferent hoogenous areas were chosen n order to deonstrate the advantages of usng local thresholds. Fgure 11 deonstrates step-by-step the results durng the applcaton of the algorth on Clar vdeo sequence (Fgure 11a). Fgure 11b s the over-segentaton generated by the applcaton of the watersheds algorth. Fgure 11c shows the result after the detecton of the frst local threshold t 1, whch generates the fnal regons 1 and shows the result after fndng the second threshold t, that generates the regons 4. Fgure 11e, shows the result after fndng t 3 that generates the regons. Fgure 11d 5 and 3 and 6. The regons that are ponted by the red arrows n Fgure 11e are not assocated wth any threshold. Snce they are surrounded by fnal regons they rean unerged. (a) (b) (c) (d) Fgure 11: Interedate results of the segentaton process on the nput age(a). (e) represents the fnal segentaton output. (e) 5

26 The values of σ ( ) of the three regons, whch are assocated wth the three thresholds (Fgure 11), are represented by the three dfferent plots n Fgure 1. The green arrow n each plot ponts to the erge, fro whch the threshold s derved. Ths erge s the frst that satsfes Eq.(7), aong all the local axus, that generates an nhoogeneous regon. Snce every threshold s derved n a dfferent teraton, each plot represents the erges n dfferent teraton. (a) (b) (c) Fgure 1: The values of σ ( ) (Fgure 11). The x-axs s the nuber of erges. The y-axs s σ ( ) ponts to the erge, fro whch the threshold s derved. of the three regons, whch are assocated wth the three thresholds. The green arrow n each plot Fgure 13 and Fgure 14 deonstrate step-by-step the result of the segentaton process of two dfferent ages. The nput ages are Fgure 13a and Fgure 14a. The outputs fro the algorth (that are bounded by the green borders) are gven by Fgure 13f and Fgure 14f. Fgure 13a was segented nto 16 dfferent regons. Fgure 14a was segented nto 1 dfferent regons. Note that not all the results fro all the teratons are gven. Only four arbtrary teratons were pcked. 6

27 (a) (b) (c) (d) (e) (f) Fgure 13: Step-by-step results of the segentaton process that operates on the nput age (a). (e) s the fnal segentaton output. (a) (b) (c) (d) (e) (f) Fgure 14: Step-by-step results of the segentaton process that operates on the nput age (a).(e) s the fnal segentaton output. 7

28 5. Fnal esults Segentaton results of varety of ages are llustrated n Fgure 15 and Fgure 16. The ages are characterzed by dfferent color hoogenety. Snce the varance of the perforance of the algorth s low, ages wth low contrast regons are segented as well as ages wth heght contrast regons. The left ages are the nput and the rght ages are the fnal segentaton results. (a) (b) (c) (d) Fgure 15: Fnal results after the applcaton of the segentaton algorth. 8

29 (a) (b) (c) (d) Fgure 16: Fnal results after the applcaton of the segentaton algorth. The proposed algorth was pleented n C++ prograng language on a Pentu MHz coputer. Table 1 shows executon tes (see coplexty analyss n secton 4.) of the algorth on dfferent ages (presented n ths secton) and the an varables of the 9

30 algorth: the age sze (N), the nuber of regons (n) that s generated by the watershed and the nuber of thresholds/teratons (K) of the an teratve process. The values of n are llustrated for the values of E, whch s ncluded at the coplexty analyss, snce E = O( nc) and experentally the average axal value of C found to be 9. Note that although any threshold defnes two 'fnal regons' the nuber of regons of the segentaton n any age s not necessary K due to the order n whch fnal regons are generated. For exaple see the regons that are ponted by the red arrows n Fgure 11e. egardng the executon te of the algorth, whch depends also on the nuber of regons (n), the three plots n Fgure 17 llustrate the nuber of regons at any teraton durng the teratve process of three dfferent ages (Fgure 16a, Fgure 16c and Fgure 16d). It s clear that the aor reducton at the nuber of regons s accoplshed by the frst teraton. For exaple, n Fgure 17a at the begnnng of the teratve process the nuber of regons n s After the frst teraton t reduces to 83, after the second teraton t reduces to 71 etc'. Ths s ustfed fro the fact that the nput to the teratve process s the over-segentaton generated by the watershed algorth. Thus, the over segentaton s sgnfcantly reduced by the frst teraton, whle the nuber of regons s gradually decreased durng the next teratons. As a result, aong all the teratons, the executon te of the frst teraton s affected by the over-segentaton. Based on experents on large nuber of dfferent ages the coeffcents w 1 (for the color coponent) and w (for the gradent agntude coponent) of the dsslarty functon (Eq.()) set to 0.8 and 0., respectvely. Snce the thresholds derved adaptvely, the segentaton result s not senstve to w 1 and w. Dfferent values of w 1 and w have an affect only on the growng regons durng the ergng process. However, as long as the regons grow and becoe hoogenous (and the gradents becoe stronger) the dfferentaton between w 1 and w s neglgble snce colors and gradents usually depend on each other. Fgure 18 llustrates the growng regon for three dfferent values of w 1 and w : w 1=0.8 and w =0., w 1=0. andw =0.8, and equal values, w 1 = w =0.5 (Fgure 18b, Fgure 18c and Fgure 18d, respectvely). The exaples are taken fro the frst teraton, at soe arbtrary step (when the frst experent s stopped at soe rando step the value of the dsslarty functon s exeplfed and used for the next two cases). At the three cases soe dfferent regons exst; ore regons caused by weak edges are generated as long as 30

31 w ncreases. However, the sae segentaton result (Fgure 18e) s obtaned at the three experents. Iage age sze N (K) total te (N) (seconds) Fgure 11a 35 x Fgure 13a 35 x Fgure 14a 35 x Fgure 15a 56 x Fgure 15b 15 x Fgure 15c 56 x Fgure 15d 55 x Fgure 16a 375 x Fgure 16b 56 x Fgure 16c 303 x Fgure 16d 116 x Table 1: Experental results of dfferent ages: Iage sze (N), nuber of regons (n) generated by the watershed, the nuber of thresholds (K) and executon te. (a) (b) (c) Fgure 17: (a), (b) and (c) llustrate the nuber of regons (y-axs) after any teraton (x-axs) of three dfferent ages: Fgure 16a, Fgure 16c and Fgure 16d, respectvely. The aor reducton at the nuber of regons, whch s accoplshed by the frst teraton at all the exaples, represents the reducton of the over segentaton. 31

32 (a) (b) (c) (d) (e) Fgure 18: The sae segentaton (e) s obtaned although dfferent values for of w 1 and w are used. The growng regons at the frst teraton at soe arbtrary step are llustrated: (a) the source age. (b) w 1 =0.8 w =0. (c) w 1 =0. w =0.8 (d) w 1 = w = Conclusons In ths paper we propose a new approach to color age segentaton. The algorth ntegrates edges and regon-based technques whle local nforaton s consdered. The local consderaton enables to derve local thresholds adaptvely such that any threshold s assocated wth a specfc regon. As a result, the qualty of the segentaton s proved. The algorth s coposed of two stages. In the frst stage, the watershed algorth s appled. Its segentaton result s represented by AG data structure and s used as an ntalzaton for the next stage. An teratve process that derves the thresholds s the second stage. Any teraton conssts of a ergng process, dervaton of threshold and regresson process. Durng the ergng process attrbutes of hoogenety of each regon are saved n order to dentfy when nhoogeneous regons are generated. Then a threshold, whch s assocated wth the frst erge that generates nhoogeneous regon, s derved. The nuber of thresholds s autoatcally deterned durng the process, whch s also autoatcally ternated. The output of the algorth s the fnal regons that are deterned by the thresholds. The algorth s robust for large varety of color ages. 3

33 eferences [1] Den Wang. "Unsupervsed vdeo segentaton based on watersheds and teporal trackng". IEEE Transactons on Crcuts and Systes for Vdeo Technology, Vol. 8, No. 5, Septeber 1998 [] Ha Gao, Wan-Ch Su, and Chao-Huan Hou. "Iproved technques for autoatc age segentaton". IEEE Transactons on Crcuts and Systes for Vdeo Technology, Vol. 11, No. 1, Deceber 001. [3] F. Dufaux, F. Moschen and A. Lppan. "Spato-teporal segentaton based on oton and statc segentaton". IEEE Proc. Int. Conf. Iage Processng 95, Washngton, DC October [4] P. Suetens, P. Fua, and A. J. Hanson, Coputatonal strateges for obect recognton, ACM Coput. Surv., vol. 4, pp. 5 61, Mar [5] P. Besl and. Jan, Three-densonal obect recognton, ACM Coput. Surv., vol. 17, pp , Mar [6] M. Kunt, M. Benard, and. Leonard, ecent results n hghcopresson age codng, IEEE Trans. Crcuts Syst., vol. 34, pp , Nov [7] Kael Belloulata and Janusz Konrad, "Fractal Iage copresson wth regonbased functonalty", IEEE transactons on age processng, Vol. 11, No. 4, Aprl [8] Hayder adha, Martn Vetterl, ccoardo Leonard. "Iage copresson usng bnary space parttonng trees", IEEE transactons on age processng, Vol. 5, No. 1, Deceber [9] Ercan Ozyldz, Nls Krahnst-over, aeev Shar "Adaptve texture and color segentaton for trackng ovng obects" Pattern ecognton , 00. [10] D.S. Yang, H.I. Cho, Movng obect trackng by optzng odels, Proceedngs of the Internatonal Conference of Pattern ecognton, Brsbane, Australa, 1998, pp [11]. Murreta-Cd, M. Brot, N. Vandapel, "Landark dentfcaton and trackng n natural envronent", IEEE Internatonal Conference on Intellgent obots and Systes, Vctora, B.C., Canada, 1998, pp [1] N. Pal and S. Pal, A revew on age segentaton technques, Pattern ecognt., vol. 6, pp , [13]. M. Haralck and L. G. Shapro, Survey: Iage segentaton technques, Coput. Vs. Graph. Iage Process., vol. 9, pp ,