Honours Project Report. Genetic Algorithms: Colour Image Segmentation

Transcription

1 Honours Project Report Genetic Algorithms: Colour Image Segmentation Keri Woods Supervised by: Audrey Mbogho Department of Computer Science University of Cape Town 2007

2 Abstract Image segmentation has great importance in many image processing applications, and yet no general image segmentation exists. Image segmentation is complicated task, often with many parameters needing to be tuned to get good results. This report researches and discusses the concepts of image segmentation, genetic algorithms and segmentation evaluation. Due to the flexibility of genetic algorithms and their ability to effectively explore large search spaces, it may be viable to use them to improve existing image segmentation methods. A region merging algorithm was implemented and evaluated using quantitative means. These segmentation results were compared to those of two other image segmentation methods: region growing and watershed segmentation. Our region merging method was shown to produce average results. A genetic algorithm was implemented in an attempt to improve the segmentation results by evolving the segmentation parameters. A fitness function needing neither human input nor a ground truth segmentation comparison was proposed. The results on the effect of the genetic algorithm on the performance of the genetic algorithm are inconclusive. However, even if the genetic algorithm does offer an improvement, it has the major drawback of running very slowly. There are many possible way of improving this system, and it seems to be an area where much research is needed. i

3 Contents Abstract List of Figures i iv 1 Introduction Project Description Motivation Project Scope Limited Scope Background and Related Work General Overview Image Segmentation Colour Segmentation Methods of Image Segmentation Difficulties with Image Segmentation Segmentation Evaluation Genetic Algorithms Applications of Genetic Algorithms for Image Segmentation Fitness Function Conclusion System Design and Implementation Region Merging Image Segmentation Region Merging Module Genetic Algorithm Module Chromosome Structure Reproduction Selection and Termination Fitness Function ii

4 4 Experimental Design Evaluation of Region Merging Image Segmentation Effect of Genetic Algorithm on Segmentation Quality Hypothesis Method Modification of Genetic Algorithm Parameters Purpose Method Results Region Merging Genetic Algorithm Conclusion Alteration of Genetic Algorithm Parameters Future Work 38 7 Conclusions 39 iii

5 List of Figures 3.1 Module Interaction UML Diagram Module Interaction RegionList Class Region Class Genetic Algorithm Module Parameter Details Chromosome Structure Image Segmentation Example Comparison of Segmentation Methods using RMSE This table shows the comparison of the RMSE values using genetic algorithms vs. traditional algorithms The Effect of Scale Parameter on Segmentation Results iv

6 Chapter 1 Introduction 1.1 Project Description In this project we review image segmentation methods and look into the use genetic algorithms for colour image segmentation. We base the segmentation part of our implementation on a region merging method described in Baatz and Schape[8]. For our genetic algorithm, we follow a later improvement by Feitosa et al[22] which evolves the parameters for region merging to give visibly better results. The key to this improvement is a well-chosen fitness function, which captures the requirements of good segmentation without needing a comparison with a ground truth segmented image. However, Feitosa s fitness function relies on manual (user-based) segmentation to test the success or failure of the genetic algorithm. Our approach, in contrast, does not need user feedback, making it useful to real-world, automated segmentation applications. 1.2 Motivation Image segmentation is an important process and its results are used in many image processing applications. However, despite its importance, there doesn t to seem to be any general method of image segmentation that works well on all images[42]. Image segmentation involves a lot of uncertainty, often with many parameters that need to be tuned to provide optimal results. For example, the Phoenix image segmentation method has 14 adjustable parameters[12]. This large number of parameters create a very large search space. Colour images have even more information than grey-scale images, and this information can be used to create higher quality segmentation. It does, however, increase the complexity of the problem. A way of handling the large search space is to use a directed search method, such as genetic algorithms. Genetic algorithms have many qualities that make them well suited to the problem of image segmentation, such as the ability to forego a local optimum to reach a global 1

7 optimum and the ability to efficiently find an optimal solution from within a large search space[9]. Genetic algorithms could allow an image segmentation process that usually requires manual input to become unsupervised. Genetic algorithms have been used to successfully colour segment images[12]. Due to their flexibility, it seems feasible to be able to use them to come up with a general segmentation method. 1.3 Project Scope The main aims of this investigation are: 1. Research image segmentation 2. Research genetic algorithms, especially those used as part of an image segmentation process 3. Research segmentation evaluation 4. Implement a method of segmenting colour images 5. Implement a genetic algorithm to improve the segmentation algorithm implemented 6. Evaluate whether the use of the genetic algorithm is an improvement over the original segmentation method It will be advantageous if the segmentation method using the genetic algorithm is able to segment general images and enables unsupervised colour image segmentation Limited Scope Due to time limitations, some of the aims mentioned in the project proposal were not met. These include: 1. Only one segmentation method was implemented and experimented with, rather than comparing a number of different ones 2. The genetic algorithm implemented wasn t compared with those developed by others 3. No experimentation on genetic algorithm parameters was done 2

8 The reasons for these aims not being completed include: 1. Difficulties in the implementation of the region merging segmentation, including memory leaks 2. Proposal objectives being too ambitious 3. Slow running times of the region merging-genetic algorithm system 3

9 Chapter 2 Background and Related Work In this section, we start by looking at the concept and importance of image segmentation and mention the requirements for good image segmentation. The implication of using colour in image segmentation is explored and methods for image segmentation are briefly discussed. Problems with existing image segmentation methods are mentioned. The evaluation of image segmentation is briefly discussed. Genetic algorithms are then introduced and their suitability for use in image segmentation is examined. We explore various applications of genetic algorithms to the problem of image segmentation. Genetic algorithm fitness functions that have been used by others with image segmentation as discussed. Finally, the feasibility of the use of genetic algorithms for general colour image segmentation is considered and design issues for such an algorithm are discussed. 2.1 General Overview Image segmentation is an important process and its results are used in many image processing applications. However, there is no general way to successfully segment all images[42]. Colour images have more information than grey-scale images, and this information can be used to create higher quality segmentation. It does, however, increase the complexity of the problem. A way of handling this complexity is to use a directed search method, such as genetic algorithms. Genetic algorithms, which mimic the process of evolution, have many qualities that make them well suited to the problem of image segmentation, such as the ability to forego a local optimum to reach a global optimum[9] and the ability too efficiently find an optimal solution from within a large search space[6]. The main uses of genetic algorithms in image segmentation are for the modification of parameters in existing segmentation algorithms and pixel-level segmentation[21]. Various algorithms that successfully apply genetic algorithms to image segmentation 4

10 have been developed. Though these results are promising, none of them solve this open problem. 2.2 Image Segmentation Image segmentation is the process of dividing an image into homogeneous regions. This is equivalent to finding the boundaries between the regions. Segmentation is the first step for many higher level image processing and computer vision operations, including shape recognition, medical imaging[24], locating objects in satellite images[33], face detection[4] and road sign recognition[48] Colour Segmentation Until recently, most image segmentation has been performed on grey-scale images. Processing colour images requires much more computation than the processing of grey-scale ones, but now with the increasing speed and decreasing cost of computation, colour image processing has been much researched in the last decade[33][19][15][42]. Colour images contain far more information than monochrome images. Each pixel in a colour image has information about brightness, hue and saturation. There are many models to represent the colours, including RGB (red, green, blue), CMY (cyan, magenta, yellow), HSV (hue, saturation, intensity), YIQ, HSI and many others. Several colour spaces have been used for image segmentation[19] and no general advantage of one colour space has yet been found[42]. Many of the colour image segmentation algorithms are derived from methods of grey-scale image segmentation. However, colour creates a more complete representation of an image and exploiting this fact can result in a more reliable segmentation. Specialised techniques suited to the nature of colour information have been devised[33][19] Methods of Image Segmentation Image segmentation is an old and important problem, and there are numerous image segmentation methods. Most of these methods were developed to be used on a certain class of images and therefore aren t general image segmentation methods[9]. Bhanu and Lee[10] divide the image segmentation algorithms into three major categories: 1. Edge Based 5

11 2. Region Based 3. Clustering Based 4. Model Based 1 Edge Based Techniques Edge detection involves the detection of boundaries between different regions of the image. These boundaries correspond to discontinuities between pixels of the chosen feature (eg colour, texture, intensity). Region Based Techniques Region splitting is an image segmentation method whereby pixels are classified into regions. Each region corresponds to a range of feature values, with thresholds being the delimiters. The choice of these thresholds is very important, as it greatly affects the quality of the segmentation. This method tends to excessively split regions, resulting in over segmentation[10]. Region growing joins neighbouring pixels with similar characteristics to form larger regions. This continues until the termination conditions are met. Most of the region growing algorithms focus on local information, making it difficult to get good global results. This method tends to excessively add to regions, resulting in under segmentation[10]. Region merging recursively merges similar regions. It is similar to region growing, except that two whole regions are combined, rather than one region combining with individual pixels[14]. Region splitting and merging tries to overcome the weaknesses of region growing and region splitting by combining the two techniques. Initially the image is divided into arbitrary regions. Region splitting and region merging occur until the termination conditions are met[10]. The Phoenix image segmentation algorithm is a region splitting method for segmentation that has been widely used and tested on colour images. It uses histogram analysis, thresholding and connected component analysis to partially segment the image. Each region then has the same process applied to it recursively, until termination conditions are met and the image is fully segmented. The algorithm uses 17 parameters, 14 of which are adjustable[12][30]. 1 This isn t discussed here as it is used to match particular objects and so isn t relevant to general image segmentation 6

12 Clustering Based Techniques Clustering separates the image into various classes without any prior knowledge. This method is based on the assumption that objects within each class should have a high degree of similarity, while those in different classes should be dissimilar. It is considered an unsupervised image segmentation technique[33][40] Difficulties with Image Segmentation Image segmentation is easy when objects have distinct colours and are well separated, but can be a problem if there are many complex objects with less distinct colour. Gradual variation in colour[41], illumination[43], shading[49] and textures[40] are also possible problems. A brute force method of dealing with image segmentation would be to enumerate all possible partitions of the image and evaluate each one. This creates an extremely large search space, and so this method is not feasible[9]. Even once a segmentation method has been chosen, there are usually many parameters that need to be tuned to create high quality segmentation. For most methods, it is not feasible to perform an exhaustive search of these parameters. Despite the many methods for image segmentation, there is no general algorithm that works well for all images. Because of the wide variety of images, a general algorithm needs to be adaptable. Only then can a segmentation algorithm cope with a wide variety of images[10]. Many adaptive methods have been used for image segmentation, including genetic algorithms[11], neural networks[20], self-adaptive regularisation[47], ant colony optimization[37], fuzzy clustering[2] and simulated annealing[51]. 2.3 Segmentation Evaluation Methods of evaluating image segmentation are divided into two main categories[50]: 1. Analytical Methods 2. Empirical Methods Analytical methods look at the actual segmentation algorithm itself, rather than its results, while empirical methods evaluate the segmentation algorithm by looking at its results. In this case, it is the empirical methods that we are more interested in. Empirical methods can be further sub divided into: 7

13 1. Goodness Methods 2. Discrepancy Methods Goodness methods evaluate the quality of the segmentation by looking at its desirable properties, and don t compare it to any other segmentation. It is this type of method that we wish to use as a fitness function for the genetic algorithm, because they don t need any prior knowledge of the image segmentation or ground truth segmented image and so are advantageous for an unsupervised algorithm. Discrepancy methods compare the segmentation results to an ideally segmented image, to see how much its segmentation differs from the target segmentation. It is this class of method that we will use to evaluate the results of our segmentation algorithm during experimentation. 2.4 Genetic Algorithms Genetic algorithms are an optimization technique that can be used in image segmentation [11]. It mimics natural selection, allowing an algorithm to adapt. Solutions are represented by a population of individual chromosomes, usually represented as binary strings. A chromosome is made up of genes, each of which can represent a particular characteristic. Each individual in the population is evaluated and given a fitness score based on how well they solve the particular problem. The higher the individual s fitness score, the greater their probability of breeding. Breeding creates the next generation through crossover and mutation. Crossover combines the chromosome of two individuals, creating a new individual which is unlike either of the parents. Mutation, which occurs only a small percent of the time, randomly alters a new individual s chromosome. Since the more optimal individuals have a greater chance of breeding, the population tends to evolve and reach an optimal solution[45][5][23]. Genetic algorithms have been used to solve a wide variety of problems, including numerical and combinatorial optimisation, circuit design and cellular automata rule design[23]. In image processing, genetic algorithms have successfully been used for feature extraction, object recognition[44], knowledge based segmentation[36] and image classification[18]. Image segmentation is easily and naturally formulated as on optimisation problem. It can either be seen as finding the optimal segmentation amongst all candidate segmentations, or as finding the optimal parameters for an existing image segmentation algorithm. In both cases, this creates an extremely large search space, indicating the use of genetic algorithms[6]. Genetic algorithms are advantageous in that they are able to forego local optima in an attempt to reach the global optimum[9]. This makes them far less likely to 8

14 get caught in a local optimum than deterministic optimization techniques, such as local hill-climbing and gradient descent. Though more computationally expensive than these methods, genetic algorithms are less computationally expensive than exhaustive searches and other adaptive techniques, such as simulated annealing, which is theoretically guaranteed to find a global optimum[35]. While, genetic algorithms cannot guarantee finding a global optimum, they usually give a good approximation. This makes genetic algorithms a good compromise between accuracy and computational intensity. Many image segmentation problems have large search spaces but need only an approximate global optimum. In this case, genetic algorithms using a directed search have proven useful[10]. A disadvantage of genetic algorithms is that they can take a long time to converge. Though this is the case, they are still much more efficient than performing an exhaustive search[23] [51]. Many images, particularly natural scenes, are complex and noisy. A characteristic of genetic algorithms is their effectiveness and robustness in dealing with uncertainty, insufficient information and noise. Combined with the fact that no matter how it is posed, the image segmentation problem involves a very large search space, making genetic algorithms well suited to the problem[10][3]. One of the major challenges for designing genetic algorithms is defining a fitness function. The only information available to the population of chromosomes is the result of the fitness function evaluated every generation. This makes an appropriately defined fitness function essential for successful genetic algorithms. In the context of image segmentation, the fitness function should evaluate the resulting segmentation. There is, however, no generally accepted unsupervised method of evaluating image segmentation[44] Applications of Genetic Algorithms for Image Segmentation Farmer and Shugars[21] divide the genetic algorithms used for image segmentation into two major classes: 1. Parameter selection, where genetic algorithms are used to modify the parameters of an existing image segmentation method to improve its output. 2. Pixel-level segmentation, where genetic algorithms are used to perform region labelling. Most image segmentation methods have many parameters that need to be optimised, and therefore the first method is used more often[11]. Many such methods are 9

15 discussed, as well as a few methods employing pixel-level segmentation. Modified genetic algorithms and hybrid genetic algorithms have also been used for segmentation. Parameter Modification Most image segmentation methods have many parameters, constants and thresholds that need to be adjusted to produce optimal segmentation results. This creates a very large search space. Since the parameters typically interact in complex and nonlinear ways, an analytic solution is not generally possible. With a reasonable amount of computation, genetic algorithms are able to find good approximations of a global optimum within a large search space. They are therefore well suited to problems involving parameter optimization. Most of the applications of genetic algorithms to image segmentation involve the optimisation of various parameters[44][11][42][40]. Bhanu et al[11] pose image segmentation as an optimisation problem. They define a general segmentation method, whereby genetic algorithms are applied to the parameters of various well known image segmentation methods. They advocate the use of genetic algorithms to adapting the parameters of knows segmentation methods in order to be applicable to general images. They used outdoor colour imagery and adapted 4 parameters of the Phoenix segmentation algorithm with genetic algorithms. They had successful results, producing high quality image segmentation with a reasonable amount of computation. Even though they perform well on outdoor scenes, these algorithms have not been proved to be able to cope with general images. The fact that these algorithms can be modified to adapt the parameters of other segmentation methods makes this method very promising. Feitosa et al[22] adopt a very similar approach and use genetic algorithms to modify the parameters of a region merging segmentation algorithm. They use a fitness function that measures the similarity of resulting segments to a target segmentation provided by a user. Though computation is straight forward and intuitive, manual segmentation is still necessary beforehand. This method can easily be adapted to modify parameters of other segmentation methods. Zingaretti el al[52] propose using genetic algorithms in unsupervised colour image segmentation. This is another case of parameters of an existing image segmentation method being tuned by genetic algorithms. A key difference in this method is that it performs multi-pass thresholding. Different thresholds are adapted during each pass of genetic algorithms. An important advantage of this method over the previous one is that segmentation is performed totally unsupervised, without any manual segmentation. It also doesn t rely on any prior information regarding the type of image that is being processed or the task for which the segmentation results will be used. This approach successfully segmented a wide variety of images, with 10

16 the exception of images that were highly textured. Pignalberi et al[39] use genetic algorithms for the optimisation of parameters in an image segmentation algorithm. In this case, they focused on range images, where a pixel is coloured depending on the distance between the object and a sensor. This method segments out surfaces of 3D objects, but could be applied to segmentation of 2D images. Pixel-Level Segmentation In pixel-level segmentation, genetic algorithms are used for region labelling. A pixel is classed in a particular region depending on its characteristics[21]. Peng el al[38] take this approach. Each pixel in the image is represented by a chromosome, which is a region label. Ramos and Muge[40] reformulate image segmentation as a clustering problem and use genetic algorithms to find the optimal clusters. The chromosomes encode which pixels are members of which regions. The major disadvantage of this method is that the number of clusters must be given as an input, and so can t be an unsupervised, general approach. Chun and Yang[16] take a similar approach, but use a fuzzy fitness function. Modified Genetic Algorithms Gong and Yang[25] represent the image and the segmentation results by quadtrees. In a similar way to Zingaretti el al[52], they define a two pass system, genetic algorithms being used for optimization in both passes. In the first pass, genetic algorithms are used to minimise an energy function. In the optional second pass, a parameter defining how coarse or fine the segmentation is modified by genetic algorithms to obtain optimal segmentation results. The chromosomes encode the quadtrees, making it inefficient to apply the usual crossover and mutation operations. To cope with this, a new crossover method and three mutation methods are defined. Aoyagi and Tsuji[7] use modified genetic algorithms for pixel-level segmentation. They approach image segmentation as a feature clustering problem and like Gong and Yang[25] use an energy function as a fitness function. They found it difficult to get ideal segmentation using traditional genetic algorithms, and so introduced four special types of mutation. They also propose a new method for creating individuals of the population. Hybrid Genetic Algorithms Grenfenstette[26] mentions that genetic algorithms can be combined with local search techniques, creating a high performance search algorithm. The following 11

17 are examples of successful implementation of hybrid genetic algorithms for image segmentation. Genetic algorithms have been combined with other evolutionary methods to tackle the image segmentation problem. Bhandarkar and Zhang[9] combine genetic algorithms with simulated annealing based techniques - which mimic the physical process of cooling - to approach the problem of grey-scale image segmentation. They do this is an attempt to get rid of the weaknesses of each class. This resulted in better performance than traditional genetic algorithms. Zhijun et al[51] combined genetic algorithms and simulated annealing to evolve a neural network, resulting in less computation than if genetic algorithms were used on there own. Melkemi et al[35] combine genetic algorithms with extremal optimization to perform pixel-level segmentation Fitness Function As mentioned previously, the choice of an appropriate fitness function is very important. It is the fitness function that is largely responsible for the quality of the image segmentation obtained. The fitness function is what the algorithm "aims" to optimise. When dealing with image segmentation, one needs a fitness function that indicates how well the image has been segmented. Feitosa et al[22] calculate fitness by quantifying the difference between the segmentation obtained using the individual s parameters and a target segmentation. The target segmentation is obtained by manual segmentation, making this a discrepancy method of evaluation. This works fine for experimental purposes - it shows that genetic algorithms can be used to improve the quality of segmentation. However, this fitness function makes the segmentation program of little practical use. If one needs to segment the image beforehand, no benefit is obtained from the program. Many other implementations also used discrepancy methods for calculating fitness, including [29][39]. These methods aren t relevant for use as a fitness function in this implementation, as we need an unsupervised method of image segmentation. Instead of using comparing the segmentation to a target segmentation, Zingaretti el al[52] compare it to an edge map obtained by applying a Roberts edge operator to the image. This makes the application more useful, but using an edge map as "perfect" segmentation means that their program tries to obtain this segmentation. It doesn t seem that segmentation better than that obtained with the edge operator can be obtained, and using an edge filter is much more efficient. As discussed previously, good image segmentation meets certain requirements:[33][9][10][27] 1. Every pixel in the image belongs to a region 12

18 2. A region is connected: any two pixels in a particular region can be connected by a line that doesn t leave the region 3. Each region is homogeneous with respect to a chosen characteristic. The characteristic could be syntactic (for example, colour, intensity or texture) or based on semantic interpretation 4. Adjacent regions can t be merged into a single homogeneous region 5. No regions overlap One needs a fitness function that quantifies the quality of segmentation and it would be ideal if a fitness function could evaluate the segmentation based on the above segmentation requirements. 2.5 Conclusion The use of genetic algorithms in image segmentation shows promising results. Genetic algorithms are a commonly used approach to optimising the parameters of existing image segmentation algorithms The major decisions are choosing a method of segmentation to which genetic algorithms will be applied, finding a fitness function that is a good measure of the quality of image segmentation and finding a meaningful way to represent the chromosomes. 13

19 Chapter 3 System Design and Implementation This system is composed of two modules: 1. Region Merging Module 2. Genetic Algorithm Module The Region Merging Module performs a region merging method of image segmentation on the input image. The Genetic Algorithm Module optimises the region merging parameters. varies the region merging parameters, in an attempt to improve the segmentation. The Region Merging Module can be totally independent of the Genetic Algorithm Module, with the parameters being set manually rather than letting them be altered by the genetic algorithm. The Genetic Algorithm Module only calls the Region Merging Module to calculate the fitness value. The interaction between the modules is shown below in Figure 3.1: The Region Merging Module receives and stores the input image. Whenever the Genetic Algorithm Module requires a fitness value, it sends the segmentation parameters that it is currently working with to the Region Merging Module, requesting a fitness value. The Region Merging Module uses these parameters, segments the image, evaluates the fitness and sends the fitness value to the Genetic Algorithm Module. This process is repeated until the genetic algorithm terminates. The Genetic Algorithm Module then sends the parameters that corresponded to the best fitness value to the Region Merging Module. The Region Merging Modules uses these parameters to segment the image and outputs the results of this segmentation. These modules are loosely coupled, with the region merging module being able to be replaced with a different segmentation method that needs its parameters tuned. The fitness function can also be easily replace with different fitness function. This 14

20 Input Image Region Merging Module Send Parameters (Request Fitness) Return Fitness Final Parameters Genetic Algorithm Module Output Result Figure 3.1: Module Interaction allows the system to be more adaptable when experimenting with the use of genetic algorithms for image segmentation. This system was implemented in C++, using Microsoft Visual Studio Image handling was done using the CImg Library[46], which is an open source C++ toolkit for image processing. This library is very easy to set up, is well documented and easy to use. It also offers much useful functionality for image processing. A genetic algorithm library, GALib[1] was used for the implementation of the Genetic Algorithm Module. It is well-documented, is very easy to use and offers many different options for customising and configuring your genetic algorithm. This implementation uses genetic algorithms modify the parameters of a region merging algorithm. This particular region merging method was designed by Baatz and Schape[8]. Feitosa et al[22] used genetic algorithms to modify the parameters of this algorithm in order to optimise the segmentation obtained. This implementation follows Feitosa et al s ideas, except uses a different fitness function. The problem with the fitness function used by them is that it relies on manual segmentation as measure of good segmentation, which makes the segmentation method only useful to prove that a genetic algorithm can be used to improve segmentation. This means that their implementation isn t useful as a segmentation application. The article[22] didn t mention key implementation details, leaving implementation open to interpretation. 15

21 3.1 Region Merging Image Segmentation The idea behind region merging[8][22] is to divide the image up into many small regions. Adjacent regions can be merged together, based on whether certain criteria are met. This continues until no regions meet the merging criteria. This particular method of region merging looks at the heterogeneity of the various regions in the image when deciding whether or not regions will be merged. The goal of the merging procedure is to minimise the weighted heterogeneity. There are two main classes of heterogeneity: spectral heterogeneity and spatial heterogeneity. The spectral heterogeneity measures how varied the colour within the region is. The spatial heterogeneity measures the deviation of the region from a compact, smooth shape. This is separated into compactness and smoothness components. Compactness looks at the ratio of the perimeter to the square root of the area, while smoothness looks at the ratio of the perimeter to the length of a bounding box of the region, parallel to the image borders. Initially, each pixel is initialised as a region. Regions are then visited in turn, to decide whether that region will merge with another region. The order in which regions are visited is discussed later. When considering whether a region is going to be merged, each of the regions adjacent to the region are considered. A fusion factor between the region and each particular adjacent region is calculated. The fusion factor takes into account the various heterogeneity components: F usionfactor = w colour h colour + (1 w colour ) h shape where: h colour is the spectral heterogeneity component h shape is the spatial heterogeneity component w colour is the weight given to the spectral heterogeneity component, which gives its relative importance in comparison to the spatial heterogeneity component A colour heterogeneity of each of the colour channels (red, green and blue) is calculated separately, and added to get the spectral heterogeneity component (modified from the version presented in the paper): h colour = Σ c w c (n Obj3 σ cobj3 n Obj1 σ cobj1 n Obj2 σ cobj2 ) where: Obj1 is the region selected for merging Obj2 is the region adjacent to Obj1 Obj3 is the result of merging Obj1 and Obj2 16

22 c is the particular colour channel being considered σ c is the standard deviation of the pixels in the colour channel c 1 The spatial heterogeneity component is a combination of compactness and smoothness components: h shape = w compact h compact + (1 w compact h smooth ) where: w compact is the weight given to the compactness component relative to the smoothness component h compact is the compactness component h smooth is the smoothness component The formula for the compactness component is: h compact = n Obj3 l Obj3 nobj3 (n Obj1 l Obj1 nobj1 + n Obj2 l Obj2 nobj2 ) where: l is the perimeter of the region The formula for the smoothness component is: h smooth = n Obj3 l Obj3 b Obj3 (n Obj1 l Obj1 b Obj1 + n Obj2 l Obj2 b Obj2 ) where: b is the perimeter of the region s bounding box Once the fusion factor for each adjacent region has been calculated, the region with the minimum fusion factor is considered. If its fusion factor is less than a certain threshold, then that adjacent region will be merged to the region being initially considered. The threshold value used here to the square of a scale parameter. This is known as a scale parameter, as adjusting this will result in differences in the resulting region size. The larger the scale parameter, the more merges that will happen, resulting in larger regions. This process is then repeated, with another region being selected and the fusion factor between it and its neighbouring regions being calculated. This process is repeated until no more regions can be merged. Baatz[8] mentions that one of the problems with their method of region merging is that the order in which regions are visited in an attempt to merge them makes a difference in the segmentation results. They recommend that when choosing a sequence of starting regions that sequential merges should be distributed as far from each other as possible. 1 1 In this case, σ = N 1 ΣN i=1 x2 i xbar2 1 = N 1 ΣN i=1 (x i x) 2 where N is the number of pixels, x i is the pixel value and x is the average value of the pixels 17

23 They mention that an approximate solution is to handle each region in a random sequence, but this isn t optimal. They didn t give details of the method they actually used. We tried to select random regions and process them, but found that processing regions in order 2 was more successful. This also provides a deterministic result, which makes it easier for a genetic algorithm to work out. If the fitness generated by the segmentation algorithm isn t deterministic, it is difficult for the genetic algorithm to optimise it Region Merging Module The system only depends on the CImg[46] class, with its accompanies cimg_library namespace, as well as standard C++ templates and functions. We make broad use of the Standard Template Library (STL) to create containers for the point, edge and region data structures. In each case, we use the container that minimises the segmentation algorithm s running time. The system is composed of: 1. class Region 2. class RegionList 3. struct Point Their interactions can be seen in the UML diagram, shown below in Figure 3.2. RegionList and Region are the two main classes, with Point being just a simple struct. Along with this, various typedefs were defined: typedef unsigned long int Label typedef std::map < Label, Region > RMap typedef cimg_library::cimg < Label > RImage typedef cimg_library::cimg < Label > Image typedef std::multimap < Label, Region > EdgeMap typedef std::pair < Label, Region > EdgePair typedef std::list < P oint > PointList 2 Initially, the pixels are given region labels left to right, top to bottom. Region ordering is an ordering of these regions, with regions that no longer exist being ignored 18

24 std::multimap std::list std::map cimg_library:: cimg RegionList Region RImage rimage; Image image; RMap* regions; unsigned int imagewidth; unsigned int imageheight; float wred; float wgreen; float wblue; float wcolour; float scale; float wcompact; RegionList( ); ~RegionList(); getsize (); mergeall (); mergerandom (); saveresults(); saveregionmap(); addregion (); getregion (); removeregion (); getfusionfactor (); gethcompact (); gethsmooth (); gethcolour (); gethshape (); trymerge ( ); coordtolabel (); RImage* rimage; Image* image; PointList* pixels; EdgeMap* edges; Label label; unsigned int maxx; unsigned int maxy; unsigned int minx; unsigned int miny; Label sumr; Label sumg; Label sumb; float averager; float averageg; float averageb; unsigned int area; unsigned int perimeter; Region(); ~Region(); void addedge (); unsigned int getarea (); Label getlabel (); float getaverager (); float getaverageg (); float getaverageb (); unsigned int getperimeter (); unsigned int getperimetercombined (); unsigned int getperimeterbb (); unsigned int getperimeterbbcombined (); float getstddev (); float getstddevcombined (); void splice ( ); EdgeMap* getedges (); PointList* getpixels (); void removealledgesto ( ); void changealledgesto ( ); Point int x int y Point( int a, int b) Figure 3.2: UML Diagram 19

25 Input Image Region Merging Module Send Parameters (Request Fitness) Return Fitness Final Parameters Genetic Algorithm Module Output Result Figure 3.3: Module Interaction RegionList Class The RegionList class represents the collection of all the regions in the segmented or partially segmented image. The structure of the class is shown in Figure 3.4. The main components of this class are: 1. RImage rimage 2. Image image 3. RMap* regions The Image object, represents the original image, represented in RGB colour space. Image is a type of CImg object and allows access to the red, green and blue components of each pixel. The RImage object (region map) is also a type of CImg image object, with the same dimensions as the original image. The segmentation algorithm working on a collection of regions must often select and adjust a specific region. In this case, the STL map data structure is the natural choice (RMap). As a container, a map has the advantage of a logarithmic size complexity for key-based insertions, deletions, and searching. To allow keyed access to our collection of regions, we assign a unique integer (Label) to each region based on its initial position in the starting image. Subsequently, when merging two regions, the target region will keeps its key and adjusts all neighbouring regions to reflect this change. In this way, all possible changes to the region collection, like adding, removing, or altering region contents, and searching for a region, run in logarithmic complexity in the size of collection. 20

26 RegionList RImage rimage; Image image; RMap* regions; unsigned int imagewidth; unsigned int imageheight; float wred; float wgreen; float wblue; float wcolour; float scale; float wcompact; RegionList( const char* filename, float r, float g, float b, float col, float s, float com ); ~RegionList(); int getsize (); void mergeall (); bool mergerandom (); void saveresults( const char* filename ); void saveregionmap( const char* filename ); void addregion ( const Label& label, const Point& p ); Region* getregion ( const Label& label ); void removeregion ( const Label& label ); float getfusionfactor ( Region* region1, Region* region2 ); float gethcompact ( Region* region1, Region* region2 ); float gethsmooth ( Region* region1, Region* region2 ); float gethcolour ( Region* region1, Region* region2 ); float gethshape ( float hcompact, float hsmooth ); bool trymerge ( Region* r ); Label coordtolabel( unsigned int x, unsigned int y ); Figure 3.4: RegionList Class 21

27 Region RImage* rimage; Image* image; PointList* pixels; EdgeMap* edges; Label label; unsigned int maxx; unsigned int maxy; unsigned int minx; unsigned int miny; Label sumr; Label sumg; Label sumb; float averager; float averageg; float averageb; unsigned int area; unsigned int perimeter; Region( Label lbl, const Point& p, RImage* regionimage, Image* originalimage ); ~Region(); void addedge ( Label lbl, Region* otherregion ); unsigned int getarea (); Label getlabel (); float getaverager (); float getaverageg (); float getaverageb (); unsigned int getperimeter (); unsigned int getperimetercombined ( const Region* otherregion ); unsigned int getperimeterbb (); unsigned int getperimeterbbcombined ( const Region* otherregion ); float getstddev ( unsigned int channel ); float getstddevcombined ( const Region* otherregion, unsigned int channel ); void splice ( Region* otherregion ); EdgeMap* getedges (); PointList* getpixels (); void removealledgesto ( Label lbl ); void changealledgesto ( Label lbloldregion, Region* newregion ); Figure 3.5: Region Class The image height and width are stored, as they are accessed very frequently and their storage prevents numerous function calls to the Image and RImage objects. All the other attributes are segmentation parameters and are passed in as arguments to the constructor. These are either set manually or are set by the genetic algorithm. Region Class The Region class represents a region in a segmented or partially segmented image. The structure of the class is shown in Figure 3.5. The main components of the Region class are: 1. PointList* pixels 22

28 2. EdgeMap* edges 3. Label label Pixels represent the pixels making up the image. They are Points, stored as a list (PointList). A list was used because insertion by pushing on to the front of the list runs in constant time. Deletion of pixels only occurs when all the pixels in the region are deleted 3 are deleted, so at each pixel delete takes constant time. (Deletion runs in linear time if the algorithm wasn t already iterating through the list.) Edges are stored as a multimap (EdgeMap). A boundary pixel in a region can point to one or more other regions. However, edges are stored separately from pixels, since actual pixel positions are unimportant when using edges to check connectivity. Insertion and find are both logarithmic time operations on a multimap. When erasing, all the elements with the same key are deleted. This is an logarithmic time operation, plus linear time is required for deleting n equal elements. Regions are assigned unique integer values (Label) based on their initial positions in the image. Pointers to both the original image and the region map are included as this class needs access to them. All the other attributes are stored, and updated every time the region is altered. These values are needed for the fusion factor calculation. It is worth using a little extra memory to store these values and save the numerous computations that would be required otherwise. 3.2 Genetic Algorithm Module The Genetic Algorithm Module is a simple module, with much of the functionality being provided by the GALib library[1]. Its basic behaviour and interaction with the Region Merging Module is shown in Figure 3.6. The genetic algorithm is represented by a GASimpleGA object, provided by GALib. This class is used for genetic algorithms with non-overlapping populations, and inherits from the general GAGeneticAlgorithm class Chromosome Structure Each chromosome is represented by a GABin2DecGenome object (again provided by GALib) and inherits from the general GAGenome class. This type of genome stores its genes in binary, but converts them into decimal values to be used in the fitness calculation. 3 Regions are only every merged or destroyed 23

29 Genetic Algorithm Module Initialize Population Region Merging Module Select Indiviuals for Mating request fitnes return fitness Mate individulal to produce offspring Mutate Offspring Insert Offspring into Population Are stopping criteria satisfied Select Indiviuals for Mating Figure 3.6: Genetic Algorithm Module 24

30 Range Precision No. of No. of Values Bits scale wcolour wcompact wred wgreen wblue Figure 3.7: Parameter Details In the Genetic Algorithm Module, a chromosome represents the various segmentation parameters of the region merging algorithm. These are: 1. Scale parameter 2. Weight given to the spectral heterogeneity 3. Weight given to the compactness component relative to the smoothness component 4. Weight of the red channel 5. Weight of the green channel 6. Weight of the blue channel Each of these is encoded in binary, as a separate gene. The scale parameter can vary from 0 to 100, while all the other parameters are vary from 0 to 1. The scale parameter and the weight of the various colour bands each have a precision of 0.1, while spectral and compactness weights have a precision of All the weight parameters are encoded as positive integers, with the maximum value of 1 divided by the particular precision factor (0.01 0r 0.1). To access the actual fractional values of the weights, one converts the parameter from binary to decimal and then multiplies it by the precision factor. To encode each gene in binary, one looks at the number of possible values that parameter can take on. For example, the scale parameter can take on (100 10)+1 = 1001 different values. The binary representation of that gene therefore has to be able to encode 1001 different values is then encoded using the same number of bits that would be used to encoded the smallest power of 2 greater than This is 1024 = 2 10, and so log = 10 bits is used to encode this. As can be see from figure, there are about = different parameter combinations! It would be infeasible to try each of these combinations to find the ideal combination of them, indicating the use of a genetic algorithm. The only other complication is that the sum of the weights of the colour bands needs to add up to 1, so their values are normalised. To get the normalised value 25

31 Scale wcolour wcompact wred wgreen wblue 10 bits 7 bits 7 bits 4 bits 4 bits 4 bits Figure 3.8: Chromosome Structure of of band i, one takes the weight of band i and divides it by the sum of all three band weights Reproduction In this implementation, one point crossover is used. In one point crossover, a random position on the chromosome is select and the two individuals exchange the genes on one side of that point. The mutation used is flip mutation, which is the typical mutation operation used on binary strings: bits in the chromosome are flipped with a given probability. Elitism is used, with the best individual in each generation being carried over to the next generation Selection and Termination Roulette wheel selection is used to select individuals to take part in reproduction. In roulette wheel selection, an individual s probability of taking part in reproduction is proportional to their fitness. In this case, the fitness values are normalised: in this fitness evaluation (see the following section later), a low fitness corresponds to a good quality segmentation. The fitness values values are adjusted so that individuals with low fitness value end up with a high adjusted fitness, which corresponds to a high probability of taking place in reproduction. Termination takes place when the maximum number of generations is reached. This was chosen to be the termination criterion because then one can be sure that the genetic algorithm will terminate. On the other hand, if convergence was used as the termination criterion, it could continue for very many generations with the population never converging. Also, because of the large amount of computation required for each fitness computation - an image has to be both segmented and evaluated - one wants to make sure not too many generations are evolved. The image segmentation parameters don t have to be exact to get a good segmentation result, so it isn t worth waiting for the population to converge. 26

32 3.2.4 Fitness Function As mentioned previously, one would like a fitness function that evaluates the following requirements of good image segmentation: 1. Every pixel in the image belongs to a region 2. A region is connected 3. Each region is homogeneous 4. Adjacent regions can t be merged into a single homogeneous region 5. No regions overlap Requirements 1, 2 and 5 are always obtained when using this region merging technique, therefore the important characteristics to quantify is that each region is homogeneous and that adjacent regions can t be merged into one homogeneous region. Various different fitness function were. Liu[31] proposed goodness method to be used for the quantitative evaluation of the performance of image segmentation, which could be used as a fitness measure. The evaluation function F on segmented image I is defined as: F (I) = N 2 RΣ R i=1 e 2 i Ai where: N 2 is the size of the image R is the total number of regions A i is the area of region i e i is the colour error, Euclidean distances in colour space between the values of the pixels in region i in the original image and the values of the pixels in the segmented image, defined as: e i = Σ A i j=1 (C1 j R C2 j R) 2 + (C1 j G C2 j G) 2 + (C1 j B C2 j B) 2 where: R, G and B are the colour channels C1 j is the colour of pixel j in the original image C2 j is the assigned colour of the same pixel in the segmented image Lower values of F (I) correspond to better segmentation. F evaluates the results both locally and globally. R is global measure and penalises over-segmentation. The e2 i Ai large colour error. portion is a local measure which penalises small regions or regions with a Borsotti et al[13] propose an improvement, which was implemented: Q(I) = N RΣ R 2 i=1 [ 1+logA i + ( R(A i) A i ) 2 ] e 2 i 27

33 We propose an alternative fitness function 4, aiming to minimise weighted colour heterogeneity. The colour heterogeneity is based on Feitosa s[22] concept. E(I) = ((Σ R i=1h i ) + 1) R where: E(I) is the fitness evaluation of segmented image I R is the number of regions in the image H i is the colour heterogeneity of the i th region: H i = A i (σ R + σ G + σ B ) where: A i is the area of region i σ j is the standard deviation of colour channel j H i is the weighted colour heterogeneity of region i. It is weighted by the size of the region so that larger regions contribute more to the overall heterogeneity. The colour heterogeneity of all the regions are added together to get the total heterogeneity of the image. 1 is added to this as when each region has area of 1, the standard deviation of that image is zero. When the regions are initialised, they are have an area of 1 pixel, and so would each have a standard deviation of zero. If the 1 wasn t added, the fitness of the initial segmentation would be 0, meaning its the ideal segmentation, which of course isn t the case. The sum of the colour heterogeneities as multiplied by R to penalise small regions. This is done because smaller regions tend to have a smaller standard deviation, for example a single pixel has a standard deviation of 1. When evaluating this method against the segmentation requirements, one can see that it is a good measure of the fact that each region is homogeneous. The fact that adjacent regions can t be merged isn t evaluated directly in this measure, but multiplying by the number of regions indirectly evaluates this: if two adjacent regions are very similar and are merged, their combined standard deviation doesn t increase much (ie A i σ i + A j σ j ) is not much greater than (A i + A j ) (σ i + σ j )), the total colour heterogeneity is multiplied by 1 less 5. This results in a lower fitness value. This indirectly implies that if fitness is at a minimum, adjacent regions can t be merged into a homogeneous regions, otherwise the fitness value would decrease - which is not possible if the fitness value is the minimum one. 4 Once again, a low fitness value corresponds to good segmentation 5 Because the number of regions decreases 28

34 Chapter 4 Experimental Design The main areas that need to be tested are: 1. Evaluation of region merging image segmentation 2. Segmentation improvement provided by genetic algorithms 3. Improvement of genetic algorithm by altering genetic algorithm parameters 4.1 Evaluation of Region Merging Image Segmentation Various images from the Berkeley Image Dataset[34] will be chosen to be segmented. These represent a wide range of images, from those that are easy to segment to those that are much more difficult to segment, such as a camouflaged animal. These results will be evaluated subjectively. For example, it can be seen whether or not a camouflaged animal is segmented out. Dorin Comaniciu[17] has images on his website with the corresponding successfully segmented images. These segmented images will be used as ground truths when evaluating our system. It would have been better if images manually segmented by humans were used as the ground truth, as they represent "ideal" segmentation. However, no manually segmented images in the correct format were found. The region merging algorithm will be run on each of the images to produce the resulting segmented images. One needs to find an objective way of evaluating the performance of this segmentation in comparison to the ground truth images. This evaluation will be done using ImageMagick s[32] compare function, using the MSE (mean squared error) option. The difference between the values of each pixel of the ground truth segmentation and the corresponding pixel in the segmented image that is being evaluated. This value is then squared, to get rid of negative values 29

35 and penalise more heavily where the pixel value difference is large. These values are then summed over the whole image to get the MSE. The root of this is taken to get, the RMSE (root mean squared error). This value is then divided by the number of pixels in the image to obtain the normalised RMSE, an evaluation measure that is size independent. This same procedure will be repeated with the region growing and watershed algorithms implemented by Marco Gallotta. The results of all three of these image segmentation algorithms will be compared. 4.2 Effect of Genetic Algorithm on Segmentation Quality Hypothesis The use of the genetic algorithm will improve the quality of the image segmentation Method Each image to be evaluated will be segmented using the region merging algorithm. The segmentation parameters will be tweaked to attempt to obtain the best results. The best results will be evaluated using a normalised RMSE, as discussed above, and recorded. Due to the fact that the results from genetic algorithms aren t deterministic, the images will be segmented five number times 1 by the genetic algorithm version. Each run will be evaluated and the average score for each image recorded. The average score for each algorithm type will calculated and the results will be compared. The population size used in the genetic algorithm should be made as large as computation time allows. The other genetic algorithm parameters should be tweaked to try and obtain the best possible results. Note of the values used should be made, so that the experiment can be repeated was an arbitrary choice. The more times the segmentation is run, the more accurate the results, so it should be repeated as many times as is feasible 2 It may be preferable to perform the next part of the experimentation first, examining the effect of modifying the parameters of the genetic algorithm, to determine the optimal parameters 30

36 4.3 Modification of Genetic Algorithm Parameters Purpose The main purpose of this section of the experimentation is to see what effect changing the genetic algorithm parameters and other factors - such as reproduction operators and termination criteria - will have on the quality of the segmentation results. This knowledge can then be used to find the optimal settings for the genetic algorithm Method Various settings can be modified, such as: 1. Population size 2. Chance of mutation taking place 3. Termination criteria 4. Number of individuals passing unchanged to the next generation 5. Crossover operators used 6. Mutation operators used 7. Selection method 8. Whether elitism is used and how many individuals are passed unchanged to the next generation The setting to be examined is chosen. If it is a numerical value, it will be tested at equally spaced intervals throughout its range or part of its range, otherwise the various options are tried. It is very important that only one of these factors is changed at a time, so that one knows what is causing the change in the segmentation quality (if any change). The segmentation will be performed 5 3 times will each option or value, and the... measure calculated. The average of these values will be taken as the result at that particular value or option. If the parameter being set is numerical, the results will be graphed and analysed, if not the various options will be compared. Conclusions will be made about what options produce the best segmentation results. 3 again an arbitrary choice 31

37 Chapter 5 Results 5.1 Region Merging The segmentation program outputs two result images. The first is the segmented image, with each region being shown in the average colour of all its pixels. The second is the region map, where each region is given a different value, and therefore a different shade in the image. This is done to make the distinction between regions clearer. If two adjacent regions have the same or very similar average values, the fact that they weren t merged (and probably should have been) wouldn t show up in the segmented image. An example of the results are shown in Figure 5.1. The results of the comparison of the region merging implementation with the region growing and watershed algorithms is shown in Figure 5.2. As can by seen, this implementation had rather average results. The watershed method performed the best, and ran much faster than either of the other algorithms. 5.2 Genetic Algorithm Two different fitness functions were implemented. Q(I), as defined previously, could be subjectively seen to perform badly, appearing to produce worse segmentation results than the original region merging algorithm. No further evaluation using this fitness function was performed. E(I) subjectively seemed to perform better than the original region merging algorithm. To confirm or refute this, further evaluation was performed. This evaluation was objective, being deterministic and calculated by the computer. The results obtained are shown below in Figure 5.3: As can be seen, the genetic algorithm only showed an improvement over the original region merging implementation for one of the images. Reasons for this could include 32

38 (a) (b) (c) Figure 5.1: Image Segmentation Example (a) Original Image (b) Segmented Image (c) Image Map 33

39 Watershed Region Merging Region Growing House 5597 Hand Woman Figure 5.2: Comparison of Segmentation Methods using RMSE House Hand Non GA GA GA GA GA GA GA avg Woman Figure 5.3: This table shows the comparison of the RMSE values using genetic algorithms vs. traditional algorithms a poorly chosen fitness function, using a population size that is too small to offer enough variation for the individuals to evolve or not running the genetic algorithm for enough generations for the population to evolve sufficiently. It may seem totally infeasible to find the ideal parameter combination for the region merging algorithm. However, the fact that there are 10 10, different parameter combinations is deceptive. From experimenting with varying the parameters, it was found 1 that the only parameter that made a significant different was the scale parameter. Even the scale parameter only had any great effect on the segmentation when altered by a large amount. A feasible solution would be to give each colour channel an equal weighting, setting wred, wgreen and wblue all to 0.33; letting compactness and smoothness have the same influence, setting wcompact to 0.5. The effect of changing the scale parameter is shown below in Figure 5.4. The effect of changing the scale parameter by 10 isn t that great. Suppose, to be on the safe side, we allow the scale parameter to vary by increments of 5. This means that there are only 20 different combinations of parameters to try - which should be slightly faster than running 2 generations of a genetic algorithm with a population size of 10. With a bit of prior knowledge, one doesn t even have to try all of this limited range of scale factor. If the various objects in the image appear distinct (such as the flowers shown below), one can set the scale parameter high. This high scale parameter will mean that more region merging occurs. This means that small colour variations, such as shadows or reflections, within an object hopefully won t be recognised as separate objects, while the separate objects are distinct enough from each other not to be merged together. However, if one is looking to distinguish objects of similar 1 These are just general observations, and haven t been formally tested 34

40 (a) (b) (c) Figure 5.4: The Effect of Scale Parameter on Segmentation Results (a) Original Image (b) Scale parameter 30 (c) Scale parameter 70 35