Using Game Theory for Image Segmentation

Transcription

1 Using Game Theory for Image Segmentation Elizabeth Cassell Sumanth Kolar Alex Yakushev 1 Introduction 21st March 2007 The goal of image segmentation, is to distinguish objects from background. Robust segmentation of structures is indeed required for a variety of image analysis applications, including robot vision and biomedical image processing. The digital images that we discuss in this paper are represented as matrix y of intensity levels. We will assume that the intensity level is a fraction between zero and one, and the intensity of a pixel at (i, j) is given by y i,j. Our goal in segmentation, is to produce a segmentation matrix x, where x i,j can take on only a xed number of dierent values between zero and one, corresponding to the number of segmentation levels. For example, in the case of two levels, x i,j {0, 1}. 2 Related Work There are a number of approaches for image segmentation, the common ones include: threshold techniques, boundary nding methods, region-based techniques, and connectivity-preserving relaxation methods. We look at integrating two of these approaches, boundary nding based and region-based techniques using game theory. 2.1 Region-based method A region-based method usually proceeds as follows: the image is partitioned into connected regions by grouping neighboring pixels of similar intensity levels. Adjacent regions are then merged under some criterion involving homogeneity or sharpness of region boundaries. A common approach is to minimize an objective function of the form E = (y i,j x i,j ) 2 + λ (x i,j x is,j s ) 2 (1) i,j i,j i s,j s Where i s and j s are indices in the neighborhood of pixel x i,j. The justication for this objective function is simple. The rst summation is trying to minimize 1

2 Figure 1: An example of simple image thresholding on a noisy image. Original image is on the left (SNR 0.67), the thresholded image is on the right. the dierence between the classication and the pixel intensity. Thus, in the two-level case, we would want the pixels where y i,j is small (i.e. the pixel is dark) to correspond to x i,j = 0 (black). Ignoring the second term (setting λ = 0) would correspond to a simple threshold method. (See gure 1) The second part of equation 1, seeks to minimze the dierence between classications of neighboring pixels, essentially to minimize the region boundary. As can be seen from gure 1, in only considering the intensity value we have misclassied a huge number of noisy pixels. Enforcing the smoothness constraint would force, for example, a single black pixel surrounded by white pixels to become white. There is a number of implementations of region-based methods. We cannot do an exhaustive search, since even in the two-level case there are 2 MN possibilities (where M and N are the image width and height.) One approach is called pyramid-linking, initially suggested by Brut, referenced from Jäne [2] and Morel et. al. [4]. The idea is rst to construct a pyramid from the image in the following manner: each level of the pyramid has half the pixels of the previous level, and each higher-level pixel has the average intensity of a 4x4 pixel block of the previous pixel. Each pixel could be related to any one of 16 pixels on the level below. We link a pixel on a higher level in such a way that minimizes our energy function from (1). We will end up with a tree, that we could now easily split according to the number of segmentation levels required. Another approach is called region growing. The idea is to start from a seed, which could be a single pixel, placed inside the region of interest. We then take all the pixels on the boundary of the seed, and compute the value of E if each of those pixels was in the region. We will accept all pixels for which the value E falls below a certain threshold, and repeat the process until no pixels can be accepted. You can see the output of this method in gure 2. 2

3 Figure 2: An example of using region segmentation only 3

4 Figure 3: Example of the Canny edge-detection lter on an image of a canine heart. (a)original image. (b) Detected edges 2.2 Edge-based methods A common edge-based approach is to nd the edges of the object of interest using image gradient information. One way is to simply look at the image gradients, and nd places where the gradient has a peak. This procedure is usually implemented as a simple lter. Figure 3 shows an example of one such lter the Canny lter. The gure shows that since intensity information is lost, it is very dicult to nd the region of interest in these images. Many edges are broken due to obstruction, blurring, or noise, while shadows introduce additional, unwanted edges. A much more advanced approach involves deformable contours. The idea is to start with a large contour, approximately surrounding the area of interest. Then we use the gradient information to deform the contour in such a way, that it lies along points with largest gradient, while maintain smooth curvature. An excellent implementation of this method was done by Li et. al. [3]. Their results are shown in gure 4. Though they produce a very good t, their implementation is well suited to only very small images, the example shown in gure 4, for example, is of a 64x80 image. The Matlab code they provided took 10ms per iteration on the image in gure 4. When given a larger 300x300 image, it took 160ms per iteration. Note that it takes around 800 iterations to determine the contour completely. Additionally, many of these approaches suer from problems of poor localization and oversegmentation. Overstringent criteria create fragmentation; lenient ones overlook blurred boundaries and merge unrelated regions. Region based segmentation is not likely to give us precise information regarding the shape and location, boundary nding may not be feasible if some of the structures are not well located. Clearly, neither boundary nding nor region-based segmentation alone is not likely to provide us with all the necessary answers. 4

5 Figure 4: An example of an implementation of deformable contours by Li et. al. [3] Figure 5: Flow Diagram for game theoretic image segmentation 5

6 2.3 Game-theoretic integration Our solution comes on the basis of the work done by Chakraborty and Duncan [1]. We adapt a parallel game-theoretic decision-making procedure as shown in the gure to image segmentation problem. The game is being played out by a set of decision makers (or players) which in our case, will correspond to the two segmentation modules that we want to integrate. The segmentation problem discussed here can be formulated as a two player game. If p 1 is the set of strategies of player 1, and p 2 is the set of strategies of player 2, then each player is trying to minimize their payo function F i (p 1, p 2 ). The goal is then to nd the Nash Equilibrium of the system (p 1, p 2 ), such that F 1 (p 1, p 2 ) F 1 (p 1, p 2 ); F 2 (p 1, p 2 ) F 2 (p 1, p 2 ) We will try to move toward the Nash Equilibrium iteratively, taking k as time index we can view the game as [1] p 1 k+1 = arg min p 1 P 1 F 1 (p 1, p 2 k); p 2 k+1 = arg min F 2 (p 1 k, p 2 ) Chakraborty and Duncan[1] give a proof of a theorem, which shows that there always exists a NE solution if F 1 and F 2 are of the following form: F 1 (p 1, p 2 ) = f 1 (p 1 ) + αf 21 (p 1, p 2 ) F 2 (p 1, p 2 ) = f 2 (p 2 ) + βf 12 (p 1, p 2 ) where α and β are some scaling constants, F 1 is bounded in p 1 P 1, F 1 is continuously second-order dierentiable in p 1 P 1, and a closed neighborhood u 1 P 1 such that F 1 is strongly convex in u 1. Given the above assumptions, Chakraborty and Duncan[1] provide this theorem on the existence of a Nash Equilibrium solution: Theorem: For the above-given structure of F 1 (p 1, p 2 ) and F 2 (p 1, p 2 ), there exists a locally stable Nash Equilibrium solution, i.e., for any p 1 U 1 P 1 and p 2 U 2 P 2, the sequence of rational choices generated by the parallel decision-making process converges and the limit point is a Nash equilibrium solution if α and β satisfy the following condition: [ ( α 1 2 p 1 p 1 f 1(p 1 ) + [ ( β 1 2 p 2 p 2 f 2(p 2 ) + ) 1 ( ) ] 2 p 1 p 1 f 21(p 1, p 2 2 ) p 1 p 2 f 21(p 1, p 2 ) 2 p 2 p 2 f 12(p 1, p 2 ) ) 1 ( ) ] 2 p 2 p 1 f 12(p 1, p 2 ) < 1 (2) The proof is given in Appendix A of Chakraborty and Duncan[1], which proves a Nash Equilibrium exists under the given conditions. In our case, the objective functions are as follows: 6

7 For the region-based module (player 1), F 1 (p 1, p 2 ) = min [y i,j x i,j ] 2 + λ 2 (x i,j x i 1,j ) 2 + (x i,j x i,j 1 ) 2 (3) x i,j i,j i,j +α (x i,j u) 2 + (x i,j v) 2 (4) (i,j) A p (i,j) Ā p where y is the intensity of the original image, x is the segmented image given by p 1, u is the intensity of the image inside the contour given by p 2, and v is the intensity outside the contour given by p 2. The rst term is trying to minimize the dierence between the pixel intensity values and the found region, as well as enforce continuity. The second term is trying to match the region and the contour. The objective function for player 2 (the boundary nding module) is as follows: F 2 (p 1, p 2 ) = arg max p [M gradient (I g, p) + βm region (I r, p)] (5) where p is the parameterization of the contour given by p 2, I g is the gradient image, and I r is the region segmented image obtained from x and p 1. M gradient is a measure of the match between the gradient image, I g, and the contour given by p. M region is a measure of the match between the region-segmented image I r and the contour, p. Chakraborty and Duncan show that equations 4 and 5 comply with the convexity requirements of the theorem. 3 Implementation Our implementation somewhat simplies Chakraborty and Duncan's approach, however, having made this simplication relieves us of the requirement of the prior shape information. The implementation is detailed in the sections below. 3.1 Region-Based Module We used the region growing approach discussed in section 2.1, gure 6 illustrates the algorithm. Though it requires an initial seed, the seed only needs to be a single pixel within the region of interest. At each iteration we look at the neighboring pixels and compute the value of E from equation 1. The pixels for which this value is less than a threshold, will be accepted into the region. The choice of the threshold depends on the characteristics of the image and the region of interest. While automatic threshold selection could be possible, it is still highly dependent on the image type [5]. Since we worked with a variety of images from dierent domains, we usually chose the threshold manually. 7

8 Figure 6: Illustration of the region growing algorithm. (a) Original image. (b) Seed Image. (c) Pixels queried at the rst iteration are marked with a question mark. (d) Pixels accepted at the rst iteration are checked. (e) Pixels queried at the second iteration. (f) Pixels accepted at the second iteration are checked, pixels marked with an x are rejected. 8

9 Figure 7: Illustration of the boundary nding module - Closing operation on a binary image 3.2 Boundary Finding Module The main work of the boundary nding module is done via a closing operation A B = (A B) B, which is a combination of dilation followed by erosion. While either set A or B can be thought of as an "image", A is usually considered as the image and B is called a structuring element. The structuring element is to mathematical morphology what the convolution kernel is to linear lter theory. Dilation A B is the set of all points generated by obtaining the reection of B about its origin and then shifting this reection by x. Dilation, in general, causes objects to dilate or grow in size; erosion, however which performs the morhological dual of dilation, causes objects to shrink. The amount and the way that they grow or shrink depend upon the choice of the structuring element. The closing operation uses the image gradient to identify pixels regions in the region segmentation, where holes/valleys need to be lled. This morphological operation identies regions of smoothness and lls them in case a hole ( black pixels instead of white ones) is present in those regions. However, if there are sharp pixel dierences over a span of pixels, the holes are not lled. This results is a smoother region as illustrated in Figure 7. Because our implementation does not use any prior shape information, no initialization is needed for this module. 4 Results Our rst experiment was on a synthetic image with added noise. We compared the output generated using our game theoretic integration method with the output obtained using our region based segmentation module alone. We also tested our method with real images from a variety of domains. As you can see from the gures, our integrated game-theoretic image segmentation method 9

10 Figure 8: Synthetic image example. (a) Original image with noise added. (b) Output of region-based segmentation using game-theoretic integration. (c) Output of region-based segmentation module alone. (d) Seed image. produces a contour that ts the original image signicantly better than the output produced from the region-based method alone. 5 Conclusions We have presented a successful game-theoretic approach to the image segmentation problem. Our approach produces better results than each of the individual modules, and it signicantly more robust to noise. Future work in this area could include improving the region based and boundary nding algorithms, for example by creating a region based algorithm that learns the seed instead of having to provide it. 10

11 Figure 9: Example using an image of a split papaya. (a) Original image. (b) Boundary of the fruit interior found using the region-based segmentation module alone, without using game theoretic integration. (c) Output using gametheoretic integration. Figure 10: Example using an image of a mushroom. (a) Original image. (b) Output using game-theoretic integration. 11

12 Figure 11: Example using an MR image of the brain. (a) Original image. (b) Output using game-theoretic integration. The black contour outlines the boundary of the corpus callosum. References [1] Amit Chakraborty and James S. Duncan. Game-theoretic integration for image segmentation. In Pattern Analysis and Machine Intelligence, IEEE Transaction on., pages IEEE, [2] Bernd Jähne. Digital Image processing. Springer, Berlin, 6th edition, [3] Chunming Li, Chenyang Xu, Changfeng Gui, and Martin D. Fox. Level set evolution without re-initialization: A new variational formulation. In Pattern Analysis and Machine Intelligence, IEEE Transaction on. IEEE, [4] Jean-Michel Morel and Sergio Solimini. Variational Methods in Image Segmentation. Birkhäuser, Boston, [5] Memet Sezgin and Bülent Sankur. Survey over image thresholding techniques and quantitative performance evaluation. Journal of Electronic Imaging, 13(1):146168,