GRAPH CUT OPTIMIZATION FOR THE MUMFORD-SHAH MODEL

Transcription

1 GRAPH CUT OPTIMIZATION FOR THE MUMFORD-SHAH MODEL Noha El Zehiry 1,2, Steve Xu 2, Prasanna Sahoo 2 and Adel Elmaghraby 1 1 Deartment of Comuter Engineering and Comuter Science, University of Louisville, USA. 2 Deartment of Mathematics, University of Louisville, USA. ABSTRACT In this aer, we introduce a Grah Cut Based Level Set (GCBLS) formulation that incororates grah cuts to otimize the curve evolution energy function resented earlier by Chan and Vese. We resent a discrete form of the level set energy function, rove that it is grah-reresentable, and minimize it using grah cuts. The major advantages of this formulation include the existence of global minimum and its insensitivity to initialization. Numerical imlementations show that minimizing the energy function in this non-iterative manner imroves the seed of the algorithm dramatically. This makes it more aealing to real time alications such as object tracking and image guided surgery. Yet, all the advantages of using level sets methods will still be reserved. KEY WORDS: GrahCuts,MumfordShah,imagesegmentation. 1. INTRODUCTION Image segmentation is one of the most imortant and critical tasks in the field of comuter vision. Snakes and deformable models has been introduced by Kass, Witkin and Terzoulos [1], after that several edge based models [2, 3, 4, 5, 6] and region based models [7, 8] have been introduced to solve the image segmentation roblem. All the aforementioned models aim at caturing or extracting the contours of the different distinct objects in the image, and hence rovide a ixel labeling that assign a secific class for each ixel. It is idle to disute that these techniques made a breakthrough and contributed a whole lot in the roblem of ixel labeling, nevertheless, they suffer from three major disadvantages; First, they find the nearest local minimum to the initialized contour, in other words, they are very sensitive to initialization. Secondly, the mathematical formulation of most of them deends, in one way or another, on the gradient which highly affects the robustness to noise and finally, they can not handle toology changes. Several attemts have been done to overcome the aforementioned drawbacks and various aroaches have emerged. On the to of these contributions was the minimization of the Mumford and Shah function develoed by Chan and Vese [9]; they introduced a level set formulation for the Mumford- Shah functional that converted the roblem into a mean curvature flow roblem just like the active contours but the results outerformed the classical active contours because the stoing term did not deend on the gradient of the image which reduces the deendence on clear edges, segmentation results in [9] illustrates the robustness to noise and toology changes. Tsai et al. [10] introduced indeendently a very similar work to [9]. The work of Tsai et al. differs in the algorithmic details, secially when trile oints and multile regions were handled. In addition, [10] resented imressive results for several image segmentation roblems, image denoising and interolation. The resentation of the level set formulation of the Mumford Shah functional was another major breakthrough in image segmentation that result in the develoment of subsequent techniques that used [9, 10] as a solid infrastructure for novel segmentation algorithms and then handled other asects of the roblem, such as incororating shae riors [11, 12, 13]. Many other authors studied the minimization of the Mumford Shah functional and other issues related to the images segmentation roblems such as including an exectation minimization framework, incororating rior robabilities and shae riors, imroving the seed and reducing the comlexity of the imlementation. It will be very difficult to discuss all the revious contributions in this toic but we refer the reader to [11, 12]. In this aer, we resent a novel attemt to imrove the seed and accuracy of the minimization of the Mumford Shah functional by using grah cut otimization. The dynamic labeling associated with the grah cut minimization will imrove the seed of the imlementation, and the fact that grah cuts solve for global minimum rather than a local one will imrove the accuracy of the algorithm and make it much less sensitive to initialization. The aer is organized as follows: Section 2 will review the related work and develo the relationshi between revious ublications and our aer. section 3 will introduce the mathematical formulation, Section 4 will resent the exerimental results and comare them with the currently existing techniques, and then conclusion and future work will be discussed in section RELATED WORK 2.1. General FrameWork As reviously discussed, level set formulations of the Mumford- Shah model use gradient descent to minimize the energy. And due to the reviously mentioned drawbacks of the gradient descent, the basic idea of this aer is to relace the

2 gradient descent otimization by grah cuts otimization. Again, we would like to emhasize that the use of grah cuts will enable us to find a global minimum rather than a local one, and the dynamic labeling that distinguishes grah cut techniques will tremendously imroves the seed of the imlementation [14]. Although grah cuts have been widely used in image rocessing roblems in the last decade [15, 16], u to our knowledge, no one has studied minimizing the Mumford Shah functional or energy functions that are formulated using level sets by using grah cuts. Xu et al. [17] incororated grah cuts with classical active contours for object segmentation, the aer showed imressive results but the authors outlined in their aer that their aroach could not handle toology changes which ushed us to think about using level sets instead of classical active contours to overcome this drawback. Also, the tutorial resented in ECCV 2006 by Boykov, Cremers and Kolmogrov to connect level sets with grah cuts was very insiring and was a major motivation for us to study the combination of both of them. The idea was very challenging to imlement; the challenges lie in formulating the Mumford-Shah functional in a discrete framework that can be incororated with grah cuts. This ste would have never been easy till Kolmogrov et al. [18, 19] studied the metrics that can be aroximated by geo-cuts and exlained the relationshi between the differential and integral reresentations of the different metrics. The second challenge was that even after obtaining the discrete framework, there is no guarantee that the energy function obtained is grah reresentable. Even if it is, the roblem of finding the maing between the image of interest and the grah remains an obstacle. In other words, how can we construct the grah that will minimize the energy function under consideration? Fortunately, in 2004, Kolmogrov and Zabih [18]investigated this roblem and came out with a neat result that checks whether the function can be reresented by a grah or not and how to construct the grah and minimize the function. To summarize, we will combine all the reviously mentioned results to imlement our idea. Our aroach is outlined as follows. 1. Level set formulation of the Mumford Shah functional will be reintroduced in a discrete framework based on [9, 20]. 2. We will introduce the roof that the resulting energy function is grah reresentable and can be minimized using grah cuts based on [18]. In the next two subsections, we will review the necessary background in both level sets and grah cuts Active Contour without Edges As we have discussed reviously, there were several studies that investigated the minimization of the Mumford Shah functional, in this aer we adot the formulation resented in [9]. Recall that the roblem was evolving a curve C that reresent the boundary of an oen subset ω of. Let u(x, y) be the image of interest, c 1 and c 2 are the mean value of the image intensities inside and outside the contour C, resectively. The authors resented the following energy function; F (c 1, c 2, C) = µ Length(C) + ν Area(inside(C)) + λ 1 u(x, y) c 1 2 dxdy (1) + λ 2 incide(c) outside(c) u(x, y) c 2 2 dxdy They reresented the level set function φ : R, such that; ω = {(x, y) ɛ Φ(x ) > 0} ω = {(x, y) ɛ Φ(x ) < 0} (2) C = ω = {(x, y) ɛ Φ(x ) = 0} Then using the heaviside ste function H(φ) and the dirac delta function δ(φ), the energy function was rewritten as: F (c 1, c 2, φ) = µ δ(φ(x, y)) φ(x, y) dxdy + ν H(φ(x, y))dxdy + λ 1 u(x, y) c 1 2 H(φ(x, y))dxdy + λ 2 u(x, y) c 2 2 (1 H(φ(x, y)))dxdy (3) Using the gradient descent, F can be minimized with resect to c 1 and c 2 u(x, y)h(φ(x, y))dxdy c 1 (φ) = H(φ(x, y))dxdy (4) c 2 (φ) = u(x, y)(1 H(φ(x, y)))dxdy (1 H(φ(x, y)))dxdy. (5) It results in the eicewise smooth aroximation u(x, y) = c 1 H(φ(x, y)) + c 2 (1 H(φ(x, y))) Cut Metrics Aroximating the length of the contour using grah reresentation was an essential ste in our roosed framework. This roblem has been studied earlier by Kolmogrov and Boykov [21, 20]. The authors construct a grid grah and assign the weights to the edges of the grah such that the cost of the cut aroximates the length of the contour. It has been done using Cauchy Crofton formula. Cauchy Crofton formula imlies that drawing sufficiently large number of straight lines in all direction from 0 to 2π and counting the number of oints of intersections of the lines and the contour of interest aroximates the length of the contour. The mathematical formulation of the Cauchy Crofton formula is as follows; n L dl = 2 C E (6) L 183

3 where C E is the Euclidean length and n L is the number of times that the line L intersects the contour C Therefore, dl = dρdθ 0 < ρ < and 0 < θ < 2π. (7) L n L = π 0 n L (ρ, θ) dρdθ. (8) By reasonably artitioning the set [0,π] R, Boykov and Zabih used the following discrete formula to aroximate the length of the contour; C E = 1 2 = k k n k δ 2 θ k e k n k w k ; w k = δ2 θ k e k Figure 1 shows an examle for families of 4 lines 1 (9) (10) F (c 1, c 2, φ, x 1,..., x n ) = µ δ(φ(x, y)) φ(x, y) dxdy + ν H(φ(x, y))dxdy + λ 1 u(x, y) c 1 2 x + λ 2 u(x, y) c 2 2 (1 x(12) ) c 1 and c 2 can be reresented in discrete form in the same manner: c 1 = c 2 = u(x, y)x, (13) x u(x, y)(1 x ) (1 x. (14) ) Now we need to reresent the first term (length of the contour) in a discrete form, We can do this using the aroximation exlained in 2.3 and we will discard the second term for now for the sake of simlifying our discussion. That is similar to [9], we set ν=0. We will use an 8 neighborhood system, illustrated in figure 2, in aroximating the length of the contour and we will set ρ to 1 unit in all direction, Let w1, w2, w3 and w4 be the edge costs assigned to e1, e2, e3 and e4 resectively. Notice that e1 = e3 = 1 and e2 = e4 = 2, w 1 = π 8, w 2= π 8, w 2 3= π 8 and w 4= π 8 2 Fig. 1. Four families of arallel line at angles 0, 45,90 and 135 Kolmogorov and Zabih stated that all F 2 class functions are grah reresentable as long as the function is regular. Details of F 2, regularity and grah construction can be found in [18] and a generalized roof for F k can be found in [22]. 3. MATHEMATICAL FORMULATION 3.1. Discrete Reresentation Having resented the related work and the necessary background, this section will introduce the discretization of the energy function, rove the regularity of it and declare the grah construction and minimization of the energy. Define a binary variable x for each ixel = (x, y) such that; { 1, φ() > 0; x = (11) 0, otherwise. Then, the last 2 terms in equation 3 can be easily discretized as follows; 1 Figures 1 and 2 are coied from Cut Metrics and Geometry of Grid Grahs with ermission from both authors Fig. 2. Neighborhood system of size 8 To calculate n k, we introduce a detector function d(,q) that detects whether the line connecting the two ixels and q intersects the contour or not. It is quite clear that the line intersects the contour if and only if x and x q have different labels, hence the detector function can be exressed as: d(, q) = x (1 x q ) + x q (1 x ) (15) Combining this with (9), the contour length can be rewritten as C E =,qɛe k w k (x (1 x q ) + x q (1 x )). (16) 184

4 The final discrete from of the energy function is exressed as F (x 1,...x n ) = λ 1 u(x, y) c 1 2 x + λ 2 u(x, y) c 2 2 (1 x ) (17) + µ 3.2. Grah Reresentation,q e k w k (x (1 x q ) + x q (1 x )) In this subsection, we will rove that the energy function in (17) can be minimized using grah cuts. The results in [18] stated that any F 2 class function of n binary variables reresented as; E(x 1,..., x n ) = E (x ) + <q(x, x q ) (18) is grah reresentable if and only if it is a regular function, i.e it satisfies the condition E,q (0, 0) + E,q (1, 1) E,q (0, 1) + E,q (1, 0). (19) The corresondence between (18) and our discrete formulation (17) shows that our energy function is an F 2 class function with E (x ) = λ 1 u(x, y) c 1 2 x + λ 2 u(x, y) c 2 2 (1 x ) (20) E,q (x, x q ) = (x (1 x q ) + x q (1 x ))w q = (x + x q 2x x q )w q, (21) where w q is the edge weight of the edge joining ixels and q (Notice that each ixel have a corresonding vertex on the grah). The verification of the regularity is retty straight forward and follows immediately from the definition of E,q. Since E,q has a nonzero value if and only if x x, we have E,q (0, 0) = E,q (1, 1) = 0 and E,q (0, 1) and E,q (1, 0) are always ositive because w k is always ositive. Having roved the regularity of the roosed energy function, we can construct the grah in the same way introduced [18] and solve for the labels (x 1,x 2,...,x n )the minimize our energy function 2, and then the new labeling obtained by the grah cut minimization is used to udate c 1 and c 2 using (13) and (14) Ste by ste algorithm The roosed algorithm is summarized in the following stes: 1. Initialize the contour C anywhere in the image, initialization of x, changes from 1 to n where n is the number of ixels in the image of interest follows. 2. Calculate c 1 and c 2 using equations (13) and (14). 2 Check rdz/grahcuts.html for details 3. Calculate the energy of each ixel using equation (20) and the interaction energy between each ixel and its neighbors in the 8 neigborhood system using (21). 4. Construct the grah using the construction described in [18] and solve for the new labeling; x 1 n that minimizes the total energy. 5. Recalulate c 1 and c 2 according to the new labeling and reeat 1-4 till the energy remains constant which will also reflect on stability in the values of c 1 and c EXPERIMENTAL RESULTS This section introduces exerimental results for several images: Some of them are synthetic images used to test the robustness of the algorithm to noise and toology changes. We have alied the algorithm on some of the images ublished in the [9]. We have also imlemented the original level set formulation introduced in [9] and alied it on the chosen images on the same machine (2GHz Intel Core Duo, 2GB RAM) to comare the seed of our algorithm and investigate whether grah cut otimization reduces the rocessing time or not. The arameters were chosen as follows: all images were resized to 256*256, λ 1 =λ 2 =1 and µ= , and the initialization for all the synthetic images is C = (x 128) 2 + (y 128) Figure 3 shows that the algorithm is robust to noise and that it can detect toology changes that made the initialized circle slit an detect different shaes inside the image and detect the donut shae.this result emhasizes that the algorithm still reserves all the advantages of the original level set imlementation.figure 4 shows the effectiveness of the algorithm in detecting the objects defined by grouing and shows the imortance of the regularization; the transitions from to and from to show how the riles of the contours of the black circles have been dramatically suressed due to the minimization of the contour length. Insensitivity to Initialization Figures 5, 6 and 7 illustrate the stability of the grah cut otimization technique and the insensitivity to initialization. Although the three intializations used are comletely different, the algorithm returns the exact same minimum and the exact same contour each time. On the contrary, when we used the same initialization and alied the classical level set imlementation, the algorithm result in different solutions. When the initialization was not at all overlaing with the mass, the algorithm failed to cature the mass and the contour ket shrinking till it disaeared. Figure 8 shows the two solutions where the algorithm converged to a solution. Imrovement in the rocessing time To comare the efficiency of our algorithm (seed-wise) relative to active contour without edges [9], we have fixed the model arameters of both imlementations and alied 185

5 Fig. 3. Noise and Toology - Illustration of robustness to toology changes and noise Initialization, (b,c) Intermediate integrations of the evolution, Final results after 9 iterations, cu=2203µsec Fig. 5. Mammogram /Initialization 1 - Detecting the edge of a mass in a mammogram: the initial contour is included inside the mass Initialization C = (x 128) 2 + (y 128) 2 20, (b,c) Intermediate integrations of the evolution, Final results after 5 iterations, cu=1041µsec. Fig. 4. Grouing - Illustration of robustness to toology changes and grouing objects Initialization, (b,c) Intermediate integrations of the evolution, Final results after 4 iterations, cu=1121musec them on all the revious images. Table 1 rovides the number of iterations and the total cu time for each of the images shown in figures CONCLUDING REMARKS AND DISCUSSION The aer resented a novel algorithm to minimize the Mumford Shah functional using grah cuts. The starting oint was the level set formulation of the Mumford shah model introduced by Chan and Vese, we have introduced a discrete form of the aforementioned model, roved the regularity of the resulting energy function and minimized it using grah cuts. The novelity of the algorithm lies in relacing the gradient descent minimization by the grah cut minimization. The results have shown that this relacement resulted in a more stable and robust imlementation. The results were comletely indeendent of the initialization of the contour. We Fig. 6. Mammogram / Initialization 2 -Detecting the edge of a mass in a mammogram: the mass is included inside the the initial contour Initialization C = (x 128) 2 + (y 128) 2 114, (b,c) Intermediate results of the evolution, Final results after 13 iterations, cu=2724 µsec. comared the seed of our algorithm relative to the seed of the traditional imlementation that uses gradient descent. Using the same model for both methods, we found out that our algorithm is much faster. We lan to extend this work by introducing a grah cut otimization for multihase segmentation roblems. REFERENCES [1] Michael Kass, Andrew Witkin, and Demetri Terzooulos, Snakes: Active contour models, in Int. J. of Comuter Vision, IJCV, January 1987, vol. 1, [2] L. D. Cohen, On active contour models and balloons, Comuter Vision, Grahics, and Image Processing., vol. 53, no. 2, ,

6 Table 1. Comarison of the seed between the our discrete imlementation and the classical imlementation introduced in [9] Grah Cuts Gradient Descent Image Number of Iterations TotalCPU time Number of Iterations Total CPU time Toology µsec msec Grouing µsec msec Mammogram 1/Initialization µsec msec Mammogram 2/Initialization µsec msec Fig. 7. Mammogram / Initialization 3 - Detecting the edge of a mass in a mammogram: Initialization is far from the mass Initialization C = (x 150) 2 + (y 220) 2 20, (b,c) Intermediate interations of the evolution, Final results after 13 iterations, cu=3475 µsec. Fig. 8. Detecting the edge of a mass in a mammogram Initial Contour is entirely inside the mass, cu= ms Initial contour is outside the mass, cu= ms [3] S. Kichenassamy, A. Kumar, P. Olver, A. Tannenbaum, and A. Yezzi, Conformal curvature flows: From hase transitions to active vision, in ICCV, [4] Ravi Malladi, James A. Sethian, and Baba C. Vemuri, Shae modeling with front roagation: A level set aroach, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no. 2, , [5] A.A. Amini, T.E. Weymouth, and R.C. Jain, Using dynamic rogramming for solving variational roblems in vision, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 9, , [6] Anthony J. Yezzi, Satyanad Kichenassamy, Arun Kumar, Peter J. Olver, and Allen Tannenbaum, A geometric snake model for segmentation of medical imagery., IEEE Trans. Med. Imaging, vol. 16, no. 2, , [7] N. Paragios and R. Deriche, Geodesic active contours for suervised texture segmentation, in CVPR, [8] Remi Ronfard, Region based strategies for active contour models., International Journal of Comuter Vision, vol. 13, no. 2, , October [9] T. F. Chan and L. A. Vese, Active contours without edges, IEEE Trans. Img. Process.,, vol. 10, no. 2, , [10] Andy Tsai, Anthony J. Yezzi, and Alan S. Willsky, Curve evolution imlementation of the mumford-shah functional for image segmentation, denoising, interolation, and magnification., IEEE Trans. Img. Process., vol. 10, no. 8, , [11] Daniel Cremers, Timo Kohlberger, and Christoh Schnörr, Nonlinear shae statistics in mumford-shah based segmentation., in ECCV (2), 2002, [12] D. Cremers, N. Sochen, and C. orr, Towards recognitionbased variational segmentation using shae riors and dynamic labeling, [13] T. Chan and Wei Zhu, Level set based shae rior segmentation, in CVPR, [14] Willim J. Cook, William H. Cunningham, William R. Pulleyblank, and Alexander Schrijver, Combinatorial Otimization, Wiley Interscience, [15] Yuri Boykov, Olga Veksler, and Ramin Zabih, Fast aroximate energy minimization via grah cuts, in ICCV (1), 1999, [16] Olga Veksler, Image segmentation by nested cuts, in CVPR, 2000, [17] Ning Xu, Ravi Bansal, and Narendra Ahuja, Object segmentation using grah cuts based active contours, CVPR, vol. 02,. 46, [18] Vladimir Kolmogorov and Ramin Zabih, What energy functions can be minimizedvia grah cuts?, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 2, [19] Yuri Boykov, Vladimir Kolmogorov, Daniel Cremers, and Andrew Delong, An integral solution to surface evolution des via geo-cuts., in ECCV (3), 2006, [20] Vladimir Kolmogorov and Yuri Boykov, What metrics can be aroximated by geo-cuts, or global otimization of length/area and flux, in ICCV 05, [21] Y. Boykov and V. Kolmogorov, Comuting geodesics and minimal surfaces via grah cuts, in ICCV, [22] D. Freedman and P. Drineas, Energy minimization via grah cuts: Settling what is ossible, 2005,. II: