Prewitt. Gradient. Image. Op. Merging of Small Regions. Curve Approximation. and

Transcription

1 A RULE-BASED SYSTEM FOR REGION SEGMENTATION IMPROVEMENT IN STEREOVISION M. Buvry, E. Zagrouba and C. J. Krey ENSEEIHT - IRIT - UA 1399 CNRS Vision par Calculateur A. Bruel 2 Rue Camichel, Toulouse - France author name@enseeiht.fr 1. INTRODUCTION The principal goals of image segmentation are to extract the main features of the images and to decrease the data necessary for the image description. But it is usually admitted that it does not exist a universal method for image segmentation and this is the reason why the algorithms are only adapted to a domain or even to an application. In stereovision, the following step consists in matching the features yielded by the segmentation. Most of the time, the segmentation and the matching processes are considered as two independent processes and the results of the matching is highly dependent on the results of the segmentation. The aim of this work is to develop a method to improve the region segmentation of images by considering each image separately and taking into account the results of the matching process. The method is carried out in several steps. First, an initial region segementation is computed by using a Split-and-Merge algorithm cooperating with an edge extractor. Then, a rule-based system is used in order to improve the initial region segmentation. In the second step of the method, the regions of the images are matched by an iterative algorithm ; only the reliable matches are performed and for this reason, numerous regions are left unmatched. Then, these regions are treated by another rule-based system by comparing the homologous regions on each image. Both the image segmentation and the matching results are improved at the same time making easier the 3-D reconstruction of the facets corresponding to the matched regions. 2. FORMAL DEFINITION OF REGIONS A region R is dened as a maximal homogeneous and connected subset of the image I. It is also necessary to dene a function that can evaluate the homogeneity of a region. [1] denes the region segmentation of an image I as a partition P = (R1; R2; ::: R n ) with the following properties : 1. I = n[ i=1 R i 2. 8 i 2 f1::ng; R i connected 3. 8 i 2 f1::ng; 8 j 2 f1::ng; i 6= j; R i \ R j = ; 4. 8 i 2 f1::ng; H ( R i ) = T rue 5. 8 i 2 f1::ng; 8 j 2 f1::ng; i 6= j; R i and R j adjacent; H ( R i [ R j ) = F alse H is a boolean function dening the region homogeneity. Each region is a subset of pixels with closed boundary. But generally, the boundaries are wrongly localized due to the homogeneity criterion and therefore are not very reliable. 3. OUR REGION GROWING ALGORITHM Our algorithm takes into account the edges which are supposed to be detected more precisely and more reliably than the region boundaries.

2 Image Median Filter Modified Image Prewitt Op. Gradient Image Edge Extraction Edge Image REGION GROWING ALGORITHM Split-and-Merge Regions Image Merging of Small Regions and Curve Approximation Modified Regions Rules Application Final Regions The rst process is a median lter in order to eliminate the noise on the image. Then, after having computed the gradient image with a Prewitt Operator, an edge extractor provides the edge segmentation. These two processes are detailed in [2]. The rst part of the Region Growing Algorithm has two steps and is based on the Split-and-Merge algorithm [5] : the rst step splits the image until obtaining a partition P such as all the regions satisfy a homogeneity criterion ; the second step merges adjacent regions by considering the edge segmentation and a new homogeneity criterion. Then, the small regions are eliminated and the boundaries of the regions are described by a curve approximation. Finally, a rule-based system improves the segmentation. The rules are classied in two categories : rules dealing with the regions with common horizontal or vertical boundaries without physical meaning, and rules dealing with the remaining small regions Image Splitting A homogeneity criterion P Ampl is used to determinate if an area R i must be splitted into four dierent sub-areas with the same size. R i split?! 8 >< >: 4[ j=1 R i R ij The homogeneity criterion P Ampl is dened as follows : if P Ampl ( R i ) = F alse otherwise P Ampl (R i ) = T rue, Max f(x; y)? Min f(x; y) 6 1 (x; y) 2 R i (x; y) 2 R i This process is repeated recursively on every new area and ends when all the areas satisfy the homogeneity criterion Image Merging After the previous step, the image is divided in n homogeneous areas according to the criterion P Ampl. The Merging process selects two adjacent regions belonging to dierent areas. If the region resulting from the merging of these two regions does not meet the homogeneity criterion P Mean, or, if one or several edges are detected on the common boundaries, then the two regions are not merged. Otherwise, the two regions are merged. Let C = (C1; C2; : : : ; C l ), the set of the edges provided by the edge extractor. P Mean ( R i [ R j ) = T rue 9 C k 2 C = C k CommonBoundary( R i ; R j ) AND R i ; R j neighboring regions >= >; merge?! R i [ R j

3 We have tested several homogeneous criteria taking into account the amplitude, the mean and the variance of the regions. The experiments have shown that the following criterion is the most ecient : j Mean ( R i )? Mean ( R j ) j 6 2 ) P Mean ( R i [ R j ) = T rue Then, two processes are executed to provide the initial segmentation : the rst one eliminates the many small regions located near each signicant boundary ; the second one is a curve approximation. Figure 1: Initial image and Split-and-Merge results Figure 2: Small regions elimination and curve approximation 4. A RULE-BASED SYSTEM IMPROVING THE SEGMENTATION The segmentation algorithm creates many horizontal or vertical boundaries without physical meaning. Therefore we dened a rule-based system dealing with this type of boundaries. The gradient mean of each boundary is used in order to decide if the boundary has a physical reality.

4 4.1. Some Denitions Let F k a boundary dened by its extremities E1(x1; y1) and E2(x2; y2), R i and R j two regions, p and q two pixels. F k HV Boundary, F k horizontal or vertical R j 2 Neigh ( R i ), 9 p 2 R j ; 9 q 2 R i = p and q adjacent F k 2 Boundary ( R i ), F k boundary of R i ( R i ; R j ) = number of common pixels belonging to the common boundaries Gradient ( F k ) = mean of the gradient on F k The rules analyze dierent characteristics of the regions and their boundaries : the gradient mean along the boundaries, the boundary length, the region surface... The rules need various thresholds to determine the wrong regions. We want to nd only the regions with extreme characteristics : very small regions, very long boundaries, very weak gradient... This is the reason why the values of the thresholds can be easily determined. All the results presented here have been obtained with the same thresholds Regions with Common HV Boundaries Let R i a reference region and let R j a region belonging to Neigh ( R i ). r Rule 1 PMean ( R i [ R j ) = T rue 8 F k = F k 2 Boundary ( R i ) \ Boundary ( R j ); F k HV Boundary AND Gradient ( F k ) weak =) R i and R j are merged The rule 1 merges the pairs of regions such as all the boundaries are only HV Boundaries with a weak gradient. The resulting region must satisfy the P Mean criterion. r Rule 2 IF 9 F k = F k 2 CommonBoundary ( R i ; R j ) AND 8 >< >: P Mean ( R i F k HV Boundary F k long Gradient ( F k ) weak [ R j ) = T rue =) R i and R j are merged The rule 2 merges the pairs of regions which have a long common HV Boundary and the resulting region must satisfy the homogeneity property. r Rule 3 8 >< >: P Mean ( R i [ R j ) = T rue 8 F k = F k 2 Boundary( R i ) and F k 2 Boundary( R j ) 8 < : F k small OR Gradient ( F k ) weak 9 F k = F k 2 Boundary( R i ) and F k 2 Boundary ( R j ) F k HV Boundary =) R i and R j are merged The rule 3 merges two regions R i and R j if all the common boundaries are either small or have weak gradient values, if R i and R j have at least a common HV Boundary and if the resulting region satises the homogeneity property.

5 4.3. Small Regions The homogeneity criterion is not considered for these rules because we admit that if the surface of a region is very small compared to the surface of the other one, the mean value is not signicant. r Rule 4 R i small R j big R i R j 9 = ; =) R i and R j are merged The rule 4 merges the small regions which are included in big regions ; we consider that the small regions come from noise. r Rule 5 8 >< >: R i small R j big and R j 2 Neigh ( R i ) 8 R k 2 Neigh ( R i ); R k big 8 R k 2 Neigh ( R i ); ( R k ; R i ) < ( R j ; R i ) =) R i and R j are merged The rule 5 merges small regions neighboring only big regions. The small region is merged with the big region with the longest common boundary Regions with Common Perpendicular HV Boundaries The boundaries, which are exactly perpendicular, have a weak probability to exist really ; this kind of boundaries can come from a default of our region segmentation. This rule takes into account the boundaries with an exactly null scalar product ( and not to within ). r Rule 6 9 F1 = F1 2 Boundary ( R i ) and F1 2 Boundary ( R j ) and 9 F2 = F2 2 Boundary ( R i ) and F2 2 Boundary ( R j ) such as P Mean ( R i [ R j ) = T rue F1 and F2 HV Boundaries F1 and F2 perpendicular Gradient ( F1 ) weak Gradient ( F2 ) weak 9 >= >; =) R i and R j are merged The rule 6 merges two regions R i and R j if they have two common and perpendicular HV Boundaries with a weak gradient. The resulting region must satisfy the homogeneity criterion. 5. RESULTS OF THE REGION SEGMENTATION We can see with the results ( gure 3 ) that the boundaries are correctly localized, the number of the small regions is not important, and for many regions, the assumption \a region corresponds to the 2-D projection of an object face" is true.

6 Figure 3: Application of the rules 1 to 6 6. MATCHING ALGORITHM AND IMPROVEMENT OF THE SEGMENTATION The second step of our method takes into account the results of the matching algorithm [6] which considers the initial segmentation produced by the rst step of the method. The matching algorithm [6] looks for the regions meeting very selective radiometric and geometric constraints. Then the matching process is extended to the neighboring regions by using a property of adjacency and by relaxing the constraints. Figure 4: Results of the matching algorithm The algorithm of the stereoscopic improvement presented below deals with unmatched regions and is carried out in three steps : classication of unmatched regions, generation of new couples of homologous regions, merge of single regions.

7 Left Region Segmentation Couples of Homologous Regions Right Region Segmentation Classification of Unmatched Regions Matchable Regions Single Regions Unmatchable Regions Generation of New Couples Final Couples of Homologous Regions Merge of Single Regions Final Left Segmentation Final Right Segmentation 7. CLASSIFICATION OF UNMATCHED REGIONS The region growing process produces a dierent segmentation for each image : some regions appear on an image but do not exist on the other image, or, one region corresponds to several smaller regions on the other image. The aim of this analyzis is to classify the unmatched regions. This classication is based on the results of the matching process presented previously Method For a given region R we look for its adjacent regions, their homologous and the neighboring regions of these homologous Denitions Let R a given region, we dene successively the following sets : A(R) = Neigh ( R ) : the set of adjacent regions of R. Then A(R) is splitted in two subsets UA(R) and MA(R) such as : { UA(R) = fr i = R i 2 A(R) and R i is an unmatched regiong { MA(R) = fr i = R i 2 A(R) and R i is a matched regiong If MA(R) is not empty, we dene the set of the homologous regions of MA(R) by : { HMA(R) = f Homologous(R i ); 8 R i 2 MA(R) g Finally, we associate to the region R a set S R of possible homologous regions of R dened by : { S R = Property R i \ 2 HMA(R) UA(R i ) If there is a homologous region R for the region R, the region R must not belong to HMA(R) because of the uniqueness constraint fullled by the matching process. In fact, the regions of HM A(R) are already matched and cannot be matched twice.

8 Therefore, R must be unmatched and, because of the adjacency constraint, R must be adjacent to all the regions of HM A(R) since their respective homologous are adjacent to R. Therefore, we can say that : such a region R exists if and only if R 2 S R Example Left Image Right Image R 4 2 R * MA(R) = f 1, 2, 3 g UA(1 0 ) = f R ; 4 0 ; 5 0 g UA(2 0 ) = f R ; 5 0 g UA(3 0 ) = f R ; 4 0 g HMA(R) = f 1', 2', 3' g 9 = ; ) S R = f R g 7.2. Region Classication For every unmatched region R, we compute the set S R. Then, three classes of unmatched regions are carried out : Unmatchable region (M A(R) = ;) If M A(R) is an empty set, it means that the region R is surrounded only by unmatched regions ; in this case, nothing can be concluded and the region R remains unmatched. This class of regions corresponds to areas where the segmentation process yields dierent results on each image Matchable region (S R 6= ;) If S R is not an empty set, we can try to nd a region R belonging to S R meeting the best some requirements in order to be matched to the region R. The matching algorithm is described in x Single region (S R = ;) If S R is an empty set, we can deduce that the region R is a default of segmentation which appears only in one of the two images and can be corrected. The single regions are merged to regions already matched according to the algorithm described in x9.. Consequently, both the segmentation and the matching are improved. 8. GENERATION OF NEW COUPLES OF HOMOLOGOUS REGIONS This step deals with the matchable regions. In fact, we try to match the region R with the best region R 2 S R according to some requirement based on geometric criteria Denitions Circumscribing rectangle We dene the circumscribing rectangle as the minimum rectangle surrounding the region and with the edges parallel to the coordinate axes :

9 Y l R l Y r R r X l X r Under the hypothesis that the two cameras are closely located, we assume that we increase the similarity between the two regions R l and R r by minimizing the value of the function : E1 ( R l ; R r ) = 1? f1 ( R l ; R r ); where f1 ( R l ; R r ) = 1 2 r ( X l ; X r ) r ( Y l ; Y r ) with r ( x ; y ) = Min(x; y) Max(x; y) Compactness and surface function The compactness of a region R is an attribute insensitive to scale change and dened by : C(R) = P erimeter(r)2 Surface(R) We dene the function E2 by : E2 ( R l ; R r ) = 1? 1 2 r ( C ( R l ) ; C ( R r ) )? 1 2 r ( Surface( R l ) ; Surface( R r ) ) Moment invariant function We use the moment invariant 1 of a region R. Let 1 = where pq is the normalized central moment of order p+q. 1 is invariant to translation, rotation and scale changes. We dene the function E3 ( R l ; R r ) by : E3 ( R l ; R r ) = 1? r ( 1 ( R l ) ; 1 ( R r ) ) 8.2. Choice of the Homologous of a Matchable Region For a given matchable region R, we consider its associated set S R of possible candidates. Then, according to the above information, we can obtained the best candidate R as the solution of the following minimization problem : R Min Cost(R; R ) = i E i ( R ; R ) i=1 2 S R The weights i (i=1,2,3) must be assigned depending on the inuence of each term. In practice, we use : i = 1 8 i 2 f 1; 2; 3 g 3X 9. MERGE OF SINGLE REGIONS This process consists in merging the single regions to regions already matched in order to increase the similarity between the homologous regions of a couple. The single regions are classied in two dierent classes : the regions with only one adjacent region ( x 9.1. ) and those with several adjacent regions ( x 9.2. ). Then, we dene three rules which are applied sequentially.

10 9.1. Single Regions with One Adjacent Region r Rule 7 Let R a single region : IF A(R) = f A g THEN Merge R to A The rule 7 merges the single regions which are included in another region already matched or which are located on the border of the image. Generally, the internal region is small and the rule 7 merges the internal region to the surrounding one. This segmentation modication is justied mostly by topologic considerations. Left Image Right Image R A A R is merged to A 9.2. Single Regions with Several Adjacent Regions This class contains the single regions with several adjacent regions. If R is a single region and if the set MA(R) contains more than one element, R is merged to one of the regions belonging to MA(R). Example : Left Image Right Image A R B A B In this example, the surface of the region A [ R is more similar to the surface of its homologous region A 0 and the rectangle criterion is unchanged. If R was merged to B, the rectangle criterion would not be satised any longer and the dierence of surface with B 0 would be greater. So, we dene the rule 8 by : Let R a single region : r Rule 8 IF MA(R) = f A, B g AND E1 ( A [ R, A 0 ) 6 E1 ( B [ R, B 0 ) THEN Merge R to A ELSE Merge R to B The single regions which have not been merged by the two previous rules may be merged to a region A of MA(R) by using a cost function. The cost function is based on local properties of the regions. We formulate this as a minimization problem (P) using the same cost function as in x 8.2., and the solution of the problem is the region A we are looking for. r Rule 9 (P) : Min j Cost ( A ; A 0 )? Cost ( A [ R ; A 0 ) j A 2 MA(R) A 0 2 Homologous(A) IF (P) has a solution A, THEN the region R is merged to A.

11 9.3. Results This process yields a better segmentation of images in which spurious regions created by the growing process have disappeared. The resulting regions are more similar to their homologous and they give a better representation of the object faces. This algorithm has been tested on images taken by the robot of INRIA. 60% of the total number of regions are matched (80% of the image surface is matched). Then, the single region merging process improves the segmentation and adds 2% to 10% of matched surface depending on the initial segmentation (numerous small regions detected). Figure 5: Final segmentation and matching 10. CONCLUSION In this paper, we have presented a method to improve the region segmentation of images. The method relies on the use of two rule-based systems. The rst rule-based system performs the improvement of the segmentation by dealing with each image independently. The second rule-based system takes advantage of a cooperation between the matching and the segmentation processes ; it classies the unmatched regions and merges the single regions to regions already matched. The similarity between the two segmentations is increased ; neverthless, the regions that cannot be treated are the regions which have not been detected correctly (bad lighting conditions for example) or the regions located where the results of the segmentation are very dierent on each image. References [1] D.H. BALLARD, C.M. BROWN \Computer Vision", Prentice Hall Incorporation, 1982 [2] Z. HAMROUNI \Connaissances et Raisonnement en Vision de Bas-Niveau", PhD INP-ENSEEIHT, [3] D. MARR, \Vision : A computational investigation into human representation and processing of visual information", W.H. Freeman and Company. San Francisco, [4] O. MONGA, B. WROBEL \Segmentation d'images : vers une Methodologie", Traitement du Signal, V4-3, pp , [5] T. PAVLIDIS, H. HOROWITZ \Picture Segmentation by a Directed Split-And-Merge Procedure", 2nd IJCPR, [6] E. ZAGROUBA, C. KREY \Region Matching by Adacency Propagation in Stereovision", Proc. 2nd ICARCV'92, Singapour, September 1992.