Columbia University. Electrical Engineering Department. Fall 1999

Transcription

1 Columbia University Electrical Engineering Department Fall 1999 Report of the Project: Knowledge Based Semantic Segmentation Using Evolutionary Programming Professor: Shih-Fu Chang Student: Manuel J. Reyes.

2 Project: Knowledge-Based Semantic Image Segmentation Using Evolutionary Programming. Professor: Shih-Fu Chang. Student: Manuel Reyes. INTRODUCTION The Visual Apprentice is a system developed in the Image and Advanced TV Laboratory, which uses a Model-Based Classification of Visual Information for Content-Based Retrieval for still images and video.[1] Given the importance of objects and their parts in classification of visual information[2], the system bases its framework on an objectdefinition hierarchy of the following levels: (1) region; (2) perceptual; (3) object-part; (4) objects and (5) scene, as can be seen in figure 1[1]. During training the user selects the parameters of the segmentation, creates the class-definition hierarchy by defining the labels to be used from level 2 to level 5 and labels the regions in each training image/video frame according to the hierarchy. Level 4: Object Object Level 3: Object-part Object-Part1 Object-Part2 Object-Part3 Level 2: Perceptual Area Perceptual Area-1... Perceptual Area-n Level 1: Region Region-1... Region-n Figure 1: Every node in the tree has a conceptual meaning, but also corresponds to a set of connected pixels in the image. Using the information obtained during training, the system computes a feature vector for every element in the tree. These feature vectors are used to generate a set of fuzzy classifiers at different levels. As can be observed in figure 2 (scene with one object and two objectparts), classification follows the bottom up order of the object definition, the multiple level classifier structure is not identical to the object definition hierarchy; it has only one region classifier as a descendant to each object-part classifier. The first step in the classification of a new image/video is its automatic segmentation. This is done using the segmentation parameters provided by the user during the class definition and training phase. Once the input image/video is segmented, a region level classifier is applied, finding the grades of membership of each region to a particular object-part, so the regions that are more likely to be part of a particular object-part are found. Then adjoining groups of those regions are found. A perceptual area classifier is applied to that set of groups, which find the best combinations. Those combinations are in

3 turn combined to form prospective object-part, which form objects that form scenes.[1] Object-Part 1 Region Classifier Object-Part 2 Region Classifier Perceptual area 1 Perceptual area 2 Perceptual area 3 Perceptual area 1 Perceptual area 2 Object-Part 1 Object-Part 1 Object Scene Figure 2: Multiple Level classifiers models. Each classifier acts as a black box passing its results to the next level, thus allowing the incorporation of different learning and grouping strategies. Each box represents a classifier and arrows indicate the propagation of classification information.[1] The region classifier assigns a degree of membership to each region with respect to a particular object-part. Among the regions with membership value different than zero, a classification between strong(s) and weak(w) members is made (figure 3). Groups of these regions should be made to find the prospective perceptual areas. The search over the space of possible groups is done using Evolution Algorithms where the strong and weak candidates form the initial population. a) b) Figure 3: a) Segmented input image, b) Classified regions.

4 INITIAL PROPOSAL The objective of the project is to improve the search of the best groups of regions to find the prospective perceptual areas and to extend this improvement to the search of Object-Parts and Objects. The first step in the classification of a new image/video is its automatic segmentation; this is done using the segmentation parameters provided by the user during the training phase. Unfortunately, these parameters are not always the best ones for the new image/video and we could find regions that could be part of more than one object or that cover part of the object but also part of some other things not define in the object-definition hierarchy. An extreme case can be seen in the Figures 4 and 5. In figure 5, we can observe the automatic segmentation results of figure 4; there we can see that, there is a region that is part of the face, the hair and the hand of the woman. Figure 4: Original Image This part of the region corresponds to the hair of the women. This part of the region corresponds to the face of the women. This part of the region corresponds to the hand of the women. Figure 5: Segmented Image

5 In order to alleviate this problem, I suggested to subdivide the regions obtained from the automatic segmentation in to sub regions and then find the best groups of sub regions to obtain potential perceptual areas. The subdivision of the regions should be done using the feature vector calculated by the system for each one of the regions. The search of these groups was proposed to be done using evolution programming. We would use the segmented image in figure 3 to explain the proposed procedure. After classification the strong and weak regions would be numerated starting by number 1, and then subdivided in a certain number of sub regions (The subdivision processes implemented would be discuss later). These processes could be observed in Figure 6. a) b) Figure 6: a) Strong and weak regions numerated; b) Regions subdivided in sub regions, note that each region is divided in different patterns and in different number of sub regions. PROJECT DEVELOPMENT, IDEAS, IMPLEMENTATION AND RESULTS. The existing code to make the automatic segmentation[3] and to compute the feature vector for each region was first carefully studied. The understanding of how the regions are represented by the algorithm is crucial for the development of this project. In order to become more familiar with the representation of regions, I decided to work first in the search of group of regions, because each group is another region itself and it should be represented in the same form as in the case of a single region. The philosophy of the search algorithm is the same, if it is use with regions or with sub regions, so I decided to implement the search algorithm with regions first, later I will explain the subdivision processes implemented. Although, originally, I proposed to use evolution programming to realize the search, I decided first to work with a traditionally search algorithm such as Greedy Search. Because of it would be easier to manage the region representation in this algorithm than in an evolution one.

6 Greedy Search Algorithm In the greedy search algorithm, the following terms are used, state: defines a particularly situation of the system; initial state: initial situation in the system; goal state: the desired situation; operator: is used to denote the description of an action in terms of which state will be reached by carrying out the action in a particular state, when it is applied to a state, it generates a different set of states called successors of the state, this process is known as the expansion of a state; and finally an heuristic function, which provides a measure of how close is a particularly state to the goal one[4]. The algorithm starts with the initial state, then the initial state is expanded to get a new set of states, the heuristic function is applied to the whole state space to determine which is the closer state to the goal one, then that state is expanded, the algorithm determines again which is now the best state and expands it, and so on. The algorithm stops when it reaches a state with a particular heuristic value (goal state test), after a predefined number of expanded states or when there is not any improvement in the heuristic function in the newest successors[4]. It is important to notice that the program should keep in memory the whole state space to determine which state is the best one, it also has to keep track of which states have been already expanded and be careful to avoid repetition of states. In this project, a state is a particular group of regions (sub regions), the initial state is formed by all the regions classified as strong or weak. The operator will eliminate one of the regions of the state, so when a state with n regions is expanded, n different successors with n-1 regions are generated. The heuristic function utilized in this case is the perceptual area classifier A graphic chart of the algorithm for the segmented image of figure 6 b) can be observed in figure 7. The heuristic function used in this example is the roundness of the region. In the segmentation algorithm used the extension of a region is represented by a mask, which occupied a lot of memory. If the regions represented by every state were represented in the same way, the amount of memory needed to keep the whole state space would be enormous. So I represent the states using an id number, which is formed concatenating the number of each one of the original regions that the state contains, in the figure 7 some of these id numbers can be observed. The mask of the region will be computed only when the heuristic function is going to be applied to the corresponding state, but the program doesn t keep the mask in memory. Using the id number, I avoid the repetition of states. As we can see in figure 7, the algorithm would find the best group of regions (Id: ) for this particular heuristic function, but it would also expand it, after this expansion the algorithm would not be able to find another state with a highest heuristic value than the one found for state So, I used the following criteria, if after 5 consecutive state expansions the best value of the heuristic value doesn t improve, the algorithm would stop.

7 Initial state 1 9 Id: First Expansion.... Id: Id: Id: Id: Second Expansion Fifth Expansion.... Id: Id: Third Expansion.. Id: Id: Fourth Expansion.. Id: Id: Figure 7: Example of the greedy search algorithm implemented, the order of the state expansions is determined by the heuristic function, the id number of the states represents the original regions that the state contains, this id number is used to avoid the repletion of states. The evolution program would use the same representation of states than in the greedy search algorithm, but instead of expanding states, it would mutate them. Although theoretically this implies a big difference, I think that in the practice it is not that much, so I decided to work in the subdivision of the regions rather than in the implementation of the evolution-programming search. Subdivision of Regions The subdivision of regions is made in order to improve the results of the automatic segmentation applied to any new input image/video. Analyzing different segmented images; I found basically three types of segmentation errors as can be seen in figure 8. Type I, all the regions of the object-part cover other portions of the image; type II, some regions are totally enclosed by the object, but others cover also other portions of the image; type III, a single region covers the object-part and also other portion of the image.

8 Object-Part a) b) c) Figure 8: Segmentation errors a) Type I, b) Type II c) Type III. I decided to work first with the third case; an example of this case can be seen in figure 9. The region that corresponds to the face covers also part of the background of the image. a) b) Figure 9: a) Original image, b) Segmented image. First Approach The image segmentation process creates three feature maps from the original image: edge map, color map and motion field. Color map is the major feature map, it is generated by first converting the original image into the CIE L*u*v* color space and then quantizing pixels to a limited number of colors using a clustering based method. The edge map is a binary mask where edge pixels are set to 1 and non-edge-pixels are set to 0. It is generated by applying the Canny edge detection algorithm. The segmentation algorithm fusions color and edge information to improve the segmentation process[]. As I mentioned before in this case the region covers the object and also other portions of the image. Part of the problem is that the object part and the other portions of the image covered by the region

9 are merged into a single quantization level during the generation of the color map. My first approach consists in reapplied the segmentation process only to the bounding box of the selected region. Because of I am using only the local information, the quantization levels in the generation of the color map would change and then the selected region would be re-segmented. Then the new regions obtained in the position of the selected region would be considered as the sub regions. This approach would not work if the distribution of the colors in the bounding box were similar to the distribution in the whole image. To prevent this situation, I developed two alternatives; the bounding box contains the selected region and part of the background, so if I change the original background to black or white, the distribution of color in the bounding box would change significantly. In figure 10, we can observed the images containing the bounding box of the face region of figure 9 using a) the original background; b) white background and c) black background. a) b) c) Figure 10: Image containing the bounding box of the face region of figure 9 using a) the original background, b) white background and c) black background. Once the image with the bounding box is segmented, the masks of the new regions are mapped to the mask of the original selected region, having then the desired subdivision. The results of the subdivision of the selected region in the whole segmented image, for the three cases can be observed in figure 11. a) b) c) Figure 11: Mapping in the entire segmented image, the results of the resegmentation of the image of the bounding box using a) original background, b) white background and c) black background.

10 Another two examples of this approach can be observed in figures 12 and 13. a) Original image b) Segmented image c) d)

11 e) Figure 12: Mapping in the entire segmented image, the results of the resegmentation of the image of the bounding box using c) original background, d) black background and e) white background. a) Original Image b)segmented image c)

12 d) e) Figure 13: Mapping in the entire segmented image, the results of the resegmentation of the image of the bounding box using c) original background, d) black background and e) white background. As we can see, the results using different backgrounds differ for each example. I thought in two forms for the automation of the background s choice: One is to estimate the color distribution of the input image, then calculate the color distribution of the image of the bounding box using the three different backgrounds and then use the background for which the color distribution would be more different than the color distribution of the input image. The other form was to observe the color features and the information about which background produced a better re-segmentation for each image in a training set (I used 30 images) and compute a decision tree to decide which background to use for a new image using its color features as the decision factor. But after discussing these ideas with Alex Jaimes, we agreed that for both cases the automation would be very dependant of the segmentation algorithm and that this would subtract flexibility to the system, so I decided to work in another more general approach for the subdivision of the regions. Comments about the first approach. Even though, the first approach I used for the subdivision resolves only a particular type of error and that it is dependant of the segmentation algorithm I thought in some applications were it could be used. Suppose that we want to train the system to recognize a set of objects that normally are grouped together in the image and that there are also some other objects in the image. If we want to segment every one of the objects of our interest we would need to choose the parameters of the segmentation in the training phase such that the segmentation would be accurate enough to segment all the objects in the image, this would produce a lot of regions including the ones of the objects in which we are not interested on. These undesired regions are going to be classified consuming time during the classification phase. Now if we train the system using a coarse segmentation such that the group of objects of our interest would be included in a single region, the total number of regions in the segmented image would be reduced. If the

13 system is trained to recognized the region that include the group of objects of our interest, a local segmentation could be use in that region to segment the region in the different objects it contains and then apply a second classification for the regions of the desired objects. For example in figure 14, we may want to recognize the batter, the catcher and the umpire, if we use an accurate segmentation, the pitcher, his body parts and some persons in the public could be segmented as well, having in this case to many regions to classify, but if we tune the system with the segmentation parameters that would segment the group of desired objects as a single one, the resulting number of regions would be considerable small. Then, the local segmentation could be performed using the same parameters or using a finer segmentation. Objects of interest Figure 14 The second idea developed in this project for the subdivision of the regions is discuss in the next pages.

14 Second Approach. After observing hundred of images a second idea came along. Frequently in segmented images two or more different parts of the image are united into a single region by a few number of pixels. A critical example can be seen in figure 15. a) b) Figure 15: a) Original Image, b) Segmented image: the regions of the faces are united by a relative small number of pixels. One of the features that the present system calculates is the maximum line (or distance) of connected pixels in the region. As it can be observed in Figure 16, the minimum is estimated using the minor axis of the ellipse that best fits the region in the orientation of the maximum line. Maximum line Figure 16: Maximum line of the region of interest of figure 15 b)

15 There is a function in the present system that calculates the largest line of the region in the horizontal direction. In order to find the maximum line the region is rotated using 16 different angles ( θ = 22.5 ), the largest horizontal line is calculated for each rotation and the largest one is the maximum line (For now on, I would use the term distance to mean a line of connected pixels in the horizontal direction). Now, what I want is to find the points of the region were a relative small number of pixels unite comparative bigger portions of the region and then divide the region in those points. This means that the distance of the region before and after those points is larger than the distance in those points, then those points would correspond to a relative minimum of the function of the region distance. (The graphics of the distance function for the region of figure 16 for different rotation angles can be observed in the appendix A of this report), an example for the region of figure 16 rotated by an angle of 90 can be observed in figure 17. Another relative minimum The distance after point X is bigger than the one in point X The distance before point X is bigger than the one in point X X, Relative minimum of the distance Figure 17: Region of figure 17 rotated by an angle of 90, the point X corresponds to a relative minimum of the distance for this region. More than relative minimum could be found for a particular rotated region. To find the relative minimum of the function distance of each rotated region, I compute the distance function using the existing function that calculates connected horizontal lines in the region, then I find its derivative and using the first derivative criteria I find the relative minimums. From this relative minimums, I find the absolute minimum and this point is marked as a potential point of division. The original function uses 16 different angles ( θ = 22.5 ), completing a rotation interval from 0 to 360, but to calculate the minimum we only need 8 different rotation angles ( θ = 22.5 ) completing a rotation interval from 0 to 180, because the distance functions in the interval from 180 to 360 are exactly the reflection in time of the ones in the first interval and then, they have exactly the same minimums, as it can be observed in the graphics of the appendix A. For some angles of rotation, the absolute minimum of the region distance is not significantly smaller than the largest distance found for the same rotated region, in those cases the absolute minimum would not be used as a potential point of division. For this project the discrimination criteria is the ratio between the absolute minimum and the largest distance, if it is not smaller than 0.5, that particular absolute minimum is eliminated as a potential point of division.

16 The suggested subdivision for the region in figure 19 using the significant absolute minimums found for all the rotations can be seen in figure 18. As we can see a better division could be done if I would permit more than one point of division per rotation. This is one of the open issues in this approach. This point would be also marked if I would use more than one point for each rotation Another example can be observed in figure 20. a) b) c) Figure 20: a) original image b) Region in which the program was used c) proposed division of the region. This approach can be used only in regions were significant minimum of the distance functions exists, for example for a round region this

17 approach cannot be used. Even though, I presented two examples of the third kind of errors, this approach could be used in any region as long as a significant minimum of the distance exists. These values (The minimums) would be computed as a part of the feature vector of the regions. Comments about the second approach For this approach, I only find the potential points of division, but I didn t have time to divide the region and to generate the masks of the subsequent sub regions. As I commented before, one of the open issues for this approach is the number of minimums to use for each rotation, I have only work with the absolute minimum but, it could be extended to work with more than one. Another detail is the discrimination factor for significant minimums. One detail I have not mentioned before is that, frequently a region has more than one connected line for a given horizontal position, when I found this case, I decided to use the largest one for the computation of the minimums, but different calculation could be done increasing the performance of the region division. Final Comments. The greedy search algorithm developed in the first part of the project can be used with the sub regions that can be obtained using the first approach to divide the regions, this is not the case for the second approach where some work has to be done to apply the search algorithm to the sub regions that may be obtained from this approach. I think that the main issue in this project is the division of the regions; this is a very challenging and not trivial problem with a lot of variants. I think that, it is an open problem where a lot of different ideas could be implemented. References [1] A. Jaimes and S.-F. Chang, Model-Based Classification of Visual Information for Content-Based Retrieval, Storage and Retrieval for Image and Video Databases VII, IS&T/SPIE99, San Jose, CA. January 1999 [2] A. Jaimes and S.-F. Chang, Syntax, Semantics and Visual Information: A Conceptual Representation for Indexing, ADVENT project technical report No , Columbia University, January [3] D. Zhong and S.-F. Chang, Video Object Model and Segmentation for content Based Video Indexing, 1997 IEEE International Symposium of Circuits and Systems, June [4] Stuart Russell and Peter Norvig, Artificial Intelligence, A Modern Approach, Prentice Hall, 1995.

18 APPENDIX A