Unsupervised hierarchical image segmentation with level set and additive operator splitting

Transcription

1 Pattern Recognton Letters 26 (2005) Unsupervsed herarchcal mage segmentaton wth level set and addtve operator splttng M. Jeon a, M. Alexander a, W. Pedrycz b, N. Pzz a, * a Insttute for Bodagnostcs, Natonal Research Councl, 435 Ellce Avenue, Wnnpeg, MB, Canada R3B 1Y6 b Department of Electrcal and Computer Engneerng, Unversty of Alberta, Edmonton, AB, Canada Receved 28 September 2004 Avalable onlne 18 December 2004 Abstract Ths paper presents an unsupervsed herarchcal segmentaton method for mult-phase mages based on a sngle level set (2-phase) method and the sem-mplct addtve operator splttng (AOS) scheme whch s stable, fast, and easy to mplement. The method successvely segments mage subregons found at each step of the herarchy usng a decson crteron based on the varance of ntensty across the current subregon. The segmentaton contnues untl a specfed number of levels has been reached. The segmentaton nformaton for sub-mages at each stage s stored n a tree data structure, and s used for reconstructng the segmented mages. The method avods the complcated governng equatons of the mult-phase segmentaton approach, and appears to converge n fewer teratons. The method can easly be parallelzed because the AOS scheme decomposes the equatons nto a sequence of one dmensonal systems. Ó 2004 Elsever B.V. All rghts reserved. Keywords: Image processng; Herarchcal segmentaton; Varatonal PDE; Level set methods; Addtve operator splttng 1. Introducton * Correspondng author. Tel.: ; fax: E-mal addresses: moon-gu.jeon@nrc-cnrc.gc.ca (M. Jeon), murray.alexander@nrc-cnrc.gc.ca (M. Alexander), pedrycz@ ee.ualberta.ca (W. Pedrycz), nc.pzz@nrc-cnrc.gc.ca, pzz@ cs.unmantoba.ca (N. Pzz). Segmentaton s the mportant technque for detectng objects and analyzng mages n the computer vson and mage processng felds. Segmentaton methods fall nto several categores; we wll consder the followng: (1) hstogram analyss, (2) regon growng, (3) edge detecton, and (4) partal dfferental equatons (PDE)-based methods. For a more comprehensve dscusson, see the references (Cheng et al., 2001; Jahne, 2002; Russ, 2003). The hstogram analyss method (Russ, 2003) segments an mage based on the dstrbuton of ts ntensty and a pre-defned set of thresholds. Ths method does not mae use of spatal structural /$ - see front matter Ó 2004 Elsever B.V. All rghts reserved. do: /j.patrec

2 1462 M. Jeon et al. / Pattern Recognton Letters 26 (2005) nformaton, so t s effectve for smple mages that dsplay only small amounts of structure. Edge-based segmentaton methods (Jahne, 2002) fnd the edges of objects n the mage, and use ths edge nformaton to reconstruct complete boundares for the prncpal objects n the mage. It s based on the fact that the poston of an edge s gven by an extreme of the frst-order dervatve or a zero crossng n the second-order dervatve. Its man drawbac s that the true edges are fragmented by nose n the mage. Sobel and Canny algorthms belong to ths group. Regon growng (Gonzalez and Woods, 2002) s one of the most popular segmentaton methods. It segments an mage by splttng the mage nto smaller regons and mergng them nto larger ones wth -means, -nearest neghborhood or some other clusterng method. Mergng or groupng crtera may be based on crtera such as homogenety, proxmty, color, gray level or texture. To some extent t solves the nose problem related to the edge detecton method, and out-performs hstogram methods n almost all stuatons. However, ths method stll has several dsadvantages. Among them, ts hgh computatonal complexty s the most serous. Many approaches use some combnaton of these and other methods. For example, the watershed segmentaton algorthm uses a combnaton of regon growng and edge-based approaches, producng more stable segmentaton results (Gonzalez and Woods, 2002). The PDE and varatonal based methods (Morel and Solmn, 1995) are the most recently developed of the methods for mage segmentaton. The man advantage s that the theory behnd the concept and the soluton technques are well establshed n other felds such as physcs and mechancs (Aubert et al., 2002). The snae (Kass et al., 1987), the gradent vector flow (Xu and Prnce, 1998) and the level set method (Osher and Sethan, 2003), are typcal of the methods belongng to the class of PDE-based methods. The level set method s one of the most powerful tools for capturng boundares or tracng nterfaces n mage processng and materal scence. To trac nterfaces, the functon descrbng the nterfaces s defned mplctly by a partal dfferental equaton, and solved numercally. The method of actve contours wthout edges (Chan and Vese, 2001) s based on a varatonal prncple. The objectve functonal s the one formulated by Mumford and Shah (1989), and s wrtten n terms of the level set functon (whch determnes the contour). It ncorporates the length of nterfaces, and varaton of the ndvdual pxel ntenstes nsde and outsde the nterface. The level set functon s found as a soluton of the correspondng Euler Lagrange equatons, and s obtaned numercally usng a gradent descent method (Strwerda, 1989). These authors also developed mult-level set models (Vese and Chan, 2002) for mult-phase segmentaton based on the actve contour (pecewse constant case) and the Mumford Shah model (pecewse smooth case). Ther models can segment 2 n phases of the mage, where n s the number of level set functons. Thus, the mult-phase C V model evolves more regons than necessary whenever the number of regons s not a power of two. In ths case, the redundant regons are empty. For example, a 5-phase mage requres 3 level set functons, whch produces 8 sub-mages of whch 3 are empty. Therefore, although ths method stll produces correct results, t does so at the expense of unnecessary and tme-consumng computaton. Tsa et al. (2001) have employed a herarchcal approach to releve the complexty of the multphase level set methods based on the Mumford Shah model. Our method s smlar to thers but dffers wth respect to user nterventon and soluton technque. Whle ther method requres user nterventon at each stage of segmentaton, ours automatcally selects a subregon usng ntensty varatons as the decson crteron, and employs a mult-scale analyss wth wavelets for fast processng. The paper s organzed as follows. The next secton revews the basc level set 2-phase segmentaton model developed by Chan and Vese (2001) and the addtve splttng operator developed by Wechert et al. (1998). Secton 3 descrbes our unsupervsed herarchcal segmentaton algorthm n detal, and then provdes some test results n Secton 4. Fnally, conclusons and future research drectons are presented.

3 M. Jeon et al. / Pattern Recognton Letters 26 (2005) Level set formulaton and addtve operator splttng Our method s based on the model of actve contours wthout edges for two-phase problems by Chan and Vese (2001) (C V model), where the energy functonal nvolves the length of nterfaces, area (volume for 3-D), and two fttng terms, wrtten as Z EðC 1,C 2,/Þ ¼l dð/þjr/jdx XZ þ m Hð/Þdx X Z þ 1 ju 0 ðxþ C 1 j 2 Hð/Þdx X Z þ 2 ju 0 ðxþ C 2 j 2 ð1 Hð/ÞÞdx, X where X s the mage doman, / represents an mplct functon of the nterface, C 1 and C 2 represent the average ntenstes of nsde and outsde nterfaces, respectvely, H the Heavsde functon and d s the Drac functon: dð/þ ¼ dhð/þ. d/ The postve parameters l, m, 1 and 2 are fxed, and defne the relatve weghtngs gven to the varous terms n the above functonal. Mnmzng wth respect to ts parameters (C 1,C 2,/) gves the followng three Euler Lagrange equatons over the doman X: jr/j lr C 1 ð/þ ¼ C 2 ð/þ ¼ r/ jr/j m 1 ðu 0 C 1 Þ 2 þ 2 ðu 0 C 2 Þ 2 ¼ 0; ð1þ R u0 ðxþhð/ðxþþdx R Hð/ðxÞÞdx, ð2þ R u0 ðxþð1 Hð/ðxÞÞÞdx R ð1 Hð/ðxÞÞÞdx : ð3þ Eq. (1) can be solved ether drectly by an optmzaton technque (Chan et al., 1995), or by transformng t to an evoluton equaton (Rudn et al., 1992). The former method s rarely used, and the latter, whch we use n ths paper, formulates the evoluton equaton wth ntal and boundary condtons as the gradent descent method, as follows: o/ ot ¼jr/j lr r/ 1 ðu 0 C 1 Þ 2 jr/j þ 2 ðu 0 C 2 Þ 2 ; ð4þ /ðx,0þ ¼/ 0 ðxþ, ð5þ o/ðxþ ¼ 0, x 2 ox: boundary of mage: ð6þ on Followng (Chan and Vese, 2001), we replace j$/j by the approxmaton to the delta functon d h ð/þ ¼ o 1 ð1 þ 2 o/ 2 p tan 1 ð / ÞÞ whch leads to more h effectve mplementaton: o/ ot ¼ d hð/þ lr r/ jr/j 1 ðu 0 C 1 Þ 2 þ 2 ðu 0 C 2 Þ 2 : ð7þ o/ As the tme t!1,! 0, whch gves the ot soluton to the Euler Lagrange equaton (1). Usual choces of 1 and 2 are 1, m s 0, and the parameter l controls the sze of captured objects. Wth small l small objects are captured, and wth larger l only larger objects are captured. The above analyss can easly be extended to multchannel mages (Chan et al., 2000). The system equatons of mult-phase level sets can be derved by addng more level sets as varatonal parameters n the energy functonal E (Vese and Chan, 2002). However, as the number of level sets ncreases, the system of governng equatons becomes more complcated, and ther numercal soluton more dffcult to obtan on account of slow convergence of the teratons. An explct scheme s the most popular for solvng Eq. (7), but due to the Courant Fredrechs Lewy (CFL) condton whch asserts that the numercal waves should propagate at least as fast as the physcal waves (Osher and Fedw, 2003), t requres very small tme steps and therefore a large number of teratons. Instead, we apply the sem-mplct addtve operator splttng (AOS) scheme by Wechert et al. (1998), whch remans

4 1464 M. Jeon et al. / Pattern Recognton Letters 26 (2005) numercally stable for large tme steps and converges n fewer teratons. We brefly revew ts basc dea; more detals can be found n (Wechert et al., 1998). The AOS scheme splts the m-dmensonal spatal operator nto a sum of m one-dmensonal space dscretzatons. The update of each grd pont nvolves only two neghbors n each dmenson, thus reducng the system to a set of trdagonal equatons. The numercal soluton of each trdagonal system requres O(3N ) floatng pont operatons where N s the number of pxels along the th coordnate axs, and there are N N such systems for each =1,...,m, gvng a total of O(3Nm) float operatons per teraton, where N ¼ P m ¼1 N s the total number of pxels n the mage. The Thomas algorthm (Wechert et al., 1998) s used to solve the underlyng trdagonal system, resultng n a very fast and parallelzable algorthm. Ths s possble snce the data for each trdagonal system s completely ndependent. Let and represent tme and spatal ndces, respectvely. Let f ¼ 1, then f jr/j s the value of f at spatal (grd) pont and tme. At grd pont, the 1- dmensonal sem-mplct dscretzaton of equaton (7) s / þ1 / dt ¼ d h ð/ Þ f þ fþ1 2h / þ1 þ1 /þ1 h 2 f þ f 1 2h / þ1 / þ1 1 h 2! þ F, where F =[ 1 (u 0, C 1 ) 2 2 (u 0, C 2 ) 2 ], whch can be rearranged as / þ1 ¼ / þ dt h 2 ðb/þ1 þ1 a/þ1 þ c/ þ1 1 þ F Þ, where dt = tme step, h = space step, and a ¼ d h ð/ Þ f 1 þ 2f þ fþ1, 2 b ¼ d h ð/ Þ f þ fþ1, 2 c ¼ d h ð/ Þ f þ f 1 : 2 1 To avod a sngularty of, t s slghtly perturbed (Osher and Fedw, 2003) jr/j 1 jr/j 1 q ffffffffffffffffffffffffffffffffffffffffffff, ðr/ x Þ 2 þ where s a small number (10 7 whch s used n our mplementaton). 3. Unsupervsed herarchcal segmentaton Ths secton descrbes an unsupervsed herarchcal segmentaton algorthm usng the actve contour model. As mentoned above, the C V segmentaton model wors well for two-phase problems. Unle (Chan and Vese, 2001), we now extend t to mult-phase problems. We use only one level set equaton. Suppose we want to segment the mage nto three phases. After the frst step, the one-level set method captures one phase as an object, and ts complement as bacground. The ntensty varaton across the captured object and ts complementary bacground s compared, and the one havng the larger varaton becomes the target for the next step of the herarchy. The regon of smaller ntensty varaton s then replaced wth a unform ntensty (already computed as the parameter C 1 or C 2 ) equal to the average ntensty of the regon havng the larger ntensty varaton. Ths effectvely excludes the regon of smaller ntensty varaton from beng re-segmented at the next step of the herarchy. The above procedure s repeated on the resultng mage. Note that ths mage has the same dmensons as the ntal nput mage, so that at each step the segmentaton s carred out on the same sze of mage. After ths second step, the mage wll be parttoned nto 3 phases. The entre herarchcal procedure can be represented as a bnary tree (Fg. 1), wth regons represented as nodes and chldren of each node beng created by a sngle step of the one-level set segmentaton procedure. The set of leaves of ths tree consttutes the fnal segmentaton of the orgnal mage nto a number of phases equal to the number of leaves. Even though the methodõs use of the entre mage at each segmentaton step results n redundant computaton, t has the advantage of usng a smple boundary (namely, the boundary ox of the full-sze mage) at each step. Now we defne a decson crteron for branchng at a node. Defnton 1 (Intensty Varaton). Let I denote a gven mage, and S 1,S 2,...,S n denote a partton

5 M. Jeon et al. / Pattern Recognton Letters 26 (2005) phases 3 phases 4 phases 5 phases (A) (B) (C) (D) Fg. 1. Bnary tree structure of herarchcal segmentaton. of I, formed by segmentaton usng the sngle level set method. Then the ntensty varaton across S s defned as VarðS Þ¼ 1 X M ðiðx,y M Þ C Þ 2, ¼1 where C (defned n Secton 2) represents the average ntensty of sub-mage S, and M represents the number of pxels n the sub-mage S. We note that the defned ntensty varaton measures the devaton from homogenety of the subregon of the mage. Thus, the sub-mage wth larger varaton can be further segmented nto more phases, and wll be the next target for segmentaton. Based on the above reasonng we can segment mult-phase mages herarchcally wth just a sngle level set functon as descrbed n Algorthm 1 where the only data a user should provde are the number of phases of the gven mage and the ntal contour. Algorthm 1 (unsupervsed herarchcal segmentaton). nos: gven number of segmentaton phases u 0 : gven mage p 0 : ntal contour Img u 0 L: left chld (sub)mage R: rght chld (sub)mage for =1:nos do p p 0 [p,l,r] level_set_solver(p,img) Fnd a leaf (subi) wth smallest ntensty varaton (Defnton 1). Img subi and replace ts complement by the C value belongng to subi end for To llustrate ths algorthm, we show n Fg. 1 an example of a herarchy of a 5-phase mage segmentaton. Each leaf contans nformaton,.e. the ntensty varaton and pxel ndces on one submage. The root (gven mage) branches nto two nodes of sub-mages (A), and the left sub-mage s chosen for branchng by our selecton crtera of the ntensty varaton. After fnshng segmentaton of the left node, we now have three leaves of sub-mages (B). If the number of phases or clusters requred s 3, then we stop here snce three leaf nodes are decded. Otherwse, the algorthm compares the varatons of all current leaves to decde whch one should be segmented, and repeats branchng untl the requred number of leaves are produced ((C) and (D)). Some test results are gven n the next secton. 4. Experment Our algorthm s appled to the segmentaton of two artfcal mages and a real magnetc resonance (MR) bran mage. In all cases, the ntal condtons for the 0-level set are chosen as a large number of small crcles coverng the mage doman, whch gves fast convergence to the soluton contour.

6 1466 M. Jeon et al. / Pattern Recognton Letters 26 (2005) Fg. 2. Two-phase nosy artfcal mage segmentaton: (A) nosy 2-phase mage, (B) ntal 0-contour, (C) 2 phases segmentaton. 1 For nterpretaton of colour please refer the web verson of ths artcle. Frst, a smple 2-phase nosy mage s tested wth our method for robustness to nose. The robustness property s nherted from Eq. (7) (Chan and Vese, 2001). In Fg. 2, the frst mage s the orgnal one, and the second dsplays the mult-crcle ntal condton of the 0-level set. The thrd mage shows the successful segmentaton of the target object. It was found that as the sgnal-to-nose rato decreases, the algorthm taes a greater number of teratons to converge. Alternatvely, we could choose a larger tme step to reduce the number of teratons, whch cannot be acheved by the explct scheme (see Secton 2). The second test was carred out wth three phases on an artfcal nosy mage. The frst fgure n the top row of Fg. 3 contans an ntal contour (red crcle) 1 on the gven mage. The second fgure shows that the sngle level set method segments frst phase objects enclosed by the green contour, but cannot capture objects whose ntensty s close to that of bacground. In other words, at the frst level of the herarchy they are treated as the bacground. Our algorthm computes the ntensty varatons of the bacground and the captured objects, and decdes that the bacground contanng the objects should be segmented at the next stage. The thrd fgure shows the successful segmentaton of the remanng objects (enclosed by the red contour) from the bacground at the next stage of segmentaton. To see the effectveness of our algorthm, we compare t wth the three common algorthms, Sobel and Canny n the Image Processng Toolbox of Matlab (Mathwors, 2003), and gradent vector flow (GVF) (Xu and Prnce, 1998); the results are shown n the bottom row of Fg. 3. Sobel cannot recognze objects whose ntensty s close to the bacground, and Canny captures both objects, but s heavly corrupted by the nose n the mage: ths maes the correct analyss of the mage very dffcult. GVF can successfully segment an ndvdual object, but s not able to smultaneously segment dsjont objects (as shown by the red contour enclosng the brght segments n the lower rght mage n Fg. 3). Also as wth Sobel, GVF s unable to capture objects closer n ntensty to the bacground. Our fnal example s the mult-phase segmentaton of an MR bran mage n Fg. 4. The bran s composed of cerebrospnal flud (CSF), gray matter (GM), whte matter (WM), sull and scalp, whch appears as the brght outer rng. In practce, the mportant objects are CSF, GM and WM, but for the demonstraton of mult-phase segmentaton we try to segment all 5 phases. The frst fgure of the top row of Fg. 4 s the gven MR bran mage, and the second fgure showng the ntal contour of multple crcles. At the frst stage of segmentaton, most phases are captured as one object (green contour n the thrd fgure of the frst row), but some parts whose ntenstes are close to the bacground are not. Snce the ntensty varaton of the captured object s greater than the bacground, the subregon nsde the nterface s processed at the

7 M. Jeon et al. / Pattern Recognton Letters 26 (2005) Fg. 3. Top row: from left, ntal contour, 2 phases segmentaton, 3 phases segmentaton, bottom row: Sobel, Canny, GVF segmentaton. next stage ((B) of Fg. 1 and the lower left of Fg. 4). The procedure s repeated untl all 5 phases are segmented. Segmentaton of each phase s represented wth the dfferent coloured contour n Fg. 4. Notce that at the fnal stage t segments the sub-mage whch was chosen as the bacground at the frst stage. See the tree structure n Fg. 1, whch corresponds to ths example. 5. Dscusson and concluson Our method extends a sngle level-set (2-phase) method to a mult-phase method by usng a sngle level-set herarchcally. At each step of ths herarchy, the bacground or unselected sub-mage s replaced wth the average ntensty of the selected sub-mage for the nput of the next stage of segmentaton. Test results show ths strategy wors very well, s robust to nose, and captures very smallscale detals of mult-phase mages. In the comparson wth segmentaton of three phases n a nosy mage, we found that whle Sobel does not capture some phases, Canny captures even spurous objects, and GVF has a problem wth segmentng a group of objects. Our method overcame all the drawbacs of three three methods. For herarchcal segmentaton the method employs a smple crteron of comparng ntensty varatons wthn the current segmentaton and ts complement. The component wth the smaller ntensty varaton s replaced by the mean ntensty of ts complement, and the algorthm proceeds to segment the resultng mage as before. In ths way, complex boundary condtons are avoded at the expense of a larger computatonal doman. That s, the sze of nput for each stage s the same as that of the frst stage, even though the actual doman s reduced n sze. The prncpal contrbuton of our research s that, wth a smple decson crteron and a herarchcal approach, the sngle level set segmentaton method successfully extends to the mult-phase

8 1468 M. Jeon et al. / Pattern Recognton Letters 26 (2005) Fg. 4. Top row: gven mult-phase MR bran Image, multple ntal contours, 2 phases segmentaton (green), bottom row: 3 phases segmentaton (red), 4 phases segmentaton (yellow), 5 phases segmentaton (blue) (for nterpretaton of colour please refer the web verson of ths artcle). segmentaton method. For fast and convenent processng, our algorthm wll be parallelzed and ncorporated nto the Scopra system (Demo et al., 2002) whch was developed by the Insttute for Bodagnostcs, NRC Canada, and whch provdes many optmzed data structures, mathematcal lbrares, vsualzaton tools, and GUIs. More mprovement on computatonal tme can be acheved by mult-scale analyss wth wavelets for each stage of segmentaton. Acnowledgement We than the referees for ther suggeston for mprovng the clarty and content of the paper. Ths wor was fnancally supported by NSERC (Natural Scence and Engneerng Research Councl, Canada). References Aubert, G., Kornprobst, P., Mathematcal Problems n Image Processng. Sprnger, New Yor. Chan, T.F., Golub, J.H., Mulet, P., A Nonlnear Prmal- Dual Method for Total Varaton-Based Image Restoraton. CLA Tech. Report, Sept Chan, T.F., Sandberg, B.Y., Vese, L.A., Actve contours wthout edges for vector-valued mages. J. Vsual Commun. Image Represenat. 11 (2), Chan, T.F., Vese, L.A., Actve contours wthout edges. IEEE Trans. Image Process. 10 (2),

9 M. Jeon et al. / Pattern Recognton Letters 26 (2005) Cheng, H.D., Jang, X.H., Sun, Y., Wang, J., Color mage segmentaton: Advances and prospects. Pattern Recognton 34, Demo, A., Pzz, N., Somorja, R., Scopra a system for the analyss of bomedcal data. Proceedngs of IEEE Canadan Conference on Electrcal and Computer Engneerng, Wnnpeg, Canada, May pp Gonzalez, R.C., Woods, R.E., Dgtal Image Processng. Prentce Hall. Jahne, B., Dgtal Image Processng. Sprnger. Kass, M., Wtn, A., Terzopoulos, D., Snaes: Actve contour models. 1st Internatonal Computer Vson Conference. Mathwors. Image Processng Toolbox UserÕs Gude, Morel, J.M., Solmn, S., Varatonal methods n mage segmentaton. In: Progress n Nonlnear Dfferental Equatons and ther Applcatons. Brhauser, Basel. Mumford, D., Shah, J., Optmal approxmaton by pecewse smooth functons and assocated varatonal problem. Commun. Pure Appl. Math. 42, Osher, S., Fedw, R., Level Set Methods and Dynamc Implct Surfaces. Sprnger, New Yor. Osher, S., Sethan, J.A., Fronts propagatng wth curvature-dependent speed: Algorthms based on Hamlton Jacob formulatons. Journal of Computatonal Physcs 79, Rudn, L.I., Osher, S., Fetam, E., Nonlnear total varaton based nose removal algorthms. Physca D 60, Russ, J.C., The Image Processng Handboo. CRC Press. Strwerda, J.C., Fnte Dfference Schemes and Partal Dfferental Equatons. CRC Press. Tsa, A., Yezz, A., Wllsy, A.S., Curve evoluton mplementaton of the Mumford Shah functonal for mage segmentaton, denosng, nterpolaton, and magnfcaton. IEEE Trans. Image Process. 10 (8), Vese, L.A., Chan, T.F., A multphase level set framewor for mage segmentaton usng the Mumford and Shah model. Int. J. Comput. Vson 50 (3), Wechert, J., Romeny, B., Vergever, M.A., Effcent relable schemes for nonlnear dffuson flterng. IEEE Trans. Image Process. 7 (3), Xu, C., Prnce, J.L., Snaes, shapes, and gradent vector flow. IEEE Trans. Image Process. 7 (3).