Qualitative Spatial Relations using Arrangements for Complex Images M. Burge and W. Burger Johannes Kepler University, Department of Systems Science Computer Vision Laboratory, A-4040 Linz, Austria burge@cast.uni-linz.ac.at July 2, 1997 Abstract A new spatial relation called arrangements has been previously proposed (H.M. Tagare et al., \Arrangement: A Spatial Relation Between Parts for Evaluating Similarity of Tomographic Section," IEEE PAMI, 17 (9), pp. 880{893, 1995) to describe how embedded parts in an image are surrounded by their neighbors. Arrangements can be derived directly from the sequence of Voronoi cells bordering an embedded part in an image. It has been show that it is possible to compare any two arrangements, caused by the embedding of the same parts, by use of the Diagonal Exchange Operator and the Voronoi Flower diagram. However, the algorithms previously proposed is practical only for very small sets of embedded parts because of both the expensive operation of computing the prerequisite area Voronoi tessellation and the exponential search complexity (in terms of the number of edges in the Voronoi tessellation) required to compute the distance metric. We present a new algorithm for computing arrangements eciently for complex images containing a large number of embedded parts. Motivated by the new algorithm, we propose the use of arrangements for the indexing and retrieval of complex technical diagrams which may contain many similar parts. 1 INTRODUCTION In the rst part of the paper the theoretical foundation for arrangements is presented. The Voronoi diagram is introduced and the necessary generalization for machine vision to area based attractors is shown. The basic theory allowing for the comparing two Voronoi diagrams with the same parts is shown. It is based upon using a series of applications of the diagonal exchange operator to convert each diagram into it's canonical Voronoi ower diagram. After establishing the theoretical foundation for the comparison of the diagrams, a method for computing the distance metric based on the exchange tree is shown. This method is shown to be exponential in complexity and therefore unsuited for our usage. A new algorithm for the computation of arrangements for real world images with many parts is presented in the second part of the paper. Motivated by the new algorithm the nal section discusses the retrieval of complex technical diagrams based upon their arrangements.
Figure 1: An Area Voronoi Diagram of a typical GIS Scene generated from the ISPRS Working group III/3 Avenches data set. 2 ARRANGEMENTS 2.1 Voronoi Diagram Voronoi neighborhoods provide an intuitively appealing denition of the neighborhood of an embedded part, namely, the neighborhood which corresponds to that portion of the Euclidean plane which is closer to any given part than to any other. The advantages of dening neighborhoods in this way has been shown by Ahuja[1] in 1982 and more recently by Okabe, Boots, and Sugihara[2]. To achieve similar results in the general case, one solution is to use a generalized Voronoi diagram which is suited to arbitrarily shaped image elements and still exhibits the desired properties of the normal Voronoi tessellation. Generalized Voronoi diagrams are dened by generalizing ordinary Voronoi diagrams with respect to either the distance measure (e.g. Euclidean, Manhattan), the space (e.g. two dimensional Cartesian, three dimensional spherical), or the generator set (e.g. point, area). The point based Voronoi tessellation[2] is unsuited to image understanding as the useful primitives are not the individual pixels corresponding to geometric points, but groups of pixels corresponding to segments, arcs, ellipses, or polygonal objects. Rather a generalized Voronoi tessellation (which we will call the area Voronoi tessellation) using an Euclidean distance measure, a two dimensional Cartesian space, and an area generator set is a good choice. O(n log n) algorithms for certain generalized tessellation have been presented by several authors including Yap[3] for curves, and Mayya and Rajan[4] for polygons, and the approximated area based Voronoi tessellation for generic shapes of Burge and Monagan[5]. 2.2 Area Based Voronoi Diagram We use the denition of the area Voronoi tessellation presented by Okabe Boots, and Sugihara[2], which is as follows. Given that A x ; : : : ; A n are parts and that p and q are locations in the image, we can dene the distance, d a (p; A i ), from p to A i as: 2
d a (p; A i ) = min q2a i d (p; q) (1) where this represents the minimum Euclidean distance from p to any location in A i. Using this d a, they dene the area Voronoi region, V a (A i ), as the set of locations from which the distance to A i is less than or equal to the distance to any other areas: V a (A i ) = fpjd a (p; A i ) d a (p; A j ) ; j 6= i; j = 1; : : : ; ng: (2) For brevity we will let N i = V a (A i ), and the area Voronoi tessellation is then the set V = fn x ; : : : ; N i g. An example of an area Voronoi diagram can be seen in Figure 1 generated from the ISPRS Working group III/3 Avenches data set. 2.3 Mathematical Foundation for Comparing Arrangements Tagare et al. [6] have previously laid the mathematical foundation for the comparison of two Voronoi diagrams, V x and V y with the same parts. They have shown that one can convert V x into V y using a series of applications of the diagonal exchange operator, call this sequence x, to convert rst V x into it's ower, F x, and then V y to it's ower, F y. They have show that when V x and V y have the same parts, then their ower diagrams are equivalent, F x = F y, and the sequence to convert V x into V y is the combination of the sequence, x, to convert V x into it's ower and the inverse of the sequence,?1 y, to convert V y into it's ower, that is: x (V x ) = F x = F y = y (V y ) x (V x ) = y (V y )?1 y ( x (V x )) = V y Listing the adjacent cells of a given cell in counterclockwise order produces the sequence termed the arrangement of N 0. Given a section of a Voronoi diagram with the following arrangements: N0 N1 N2 Before N3 N 0 : N 2 ; N 1 N 1 : N 0 ; N 2 ; N 3 N 2 : N 3 ; N 1 ; N 0 N 3 : N 1 ; N 2 N0 N1 N2 After N3 N 0 : N 2 ; N 3 ; N 1 N 1 : N 0 ; N 3 N 2 : N 3 ; N 0 N 3 : N 1 ; N 0 ; N 2 Figure 2: Application of the Diagonal Exchange Operator to a Voronoi conguration. 3
In Figure 2 on the left side the initial conguration of a section of a Voronoi diagram is shown, and it's arrangement. The right side shows the resulting gure and it's arrangement after the application of the diagonal exchange operator. The operator works locally, that is after application of the operator only the relation between the four parts are aected while the rest of the diagram remains the same. Another characteristic of the operator is that it is reversible, applying it twice to the same edge results in the original diagram, or more generally applying it an even number of times results in no change and an odd number of times is equivalent to applying it once. If a sequence of exchanges,, applied to V 1 results in V 2 then application of the inverse of this sequence,?1, to V 2 results in V 1. It is not obvious the order in which to apply the diagonal exchange operator to convert V 1 into V 2. Tagare shows that such a sequence does exist, and gives a constructive algorithm for nding it. This involves the used of the ower diagram, which is a canonical representation of a Voronoi diagram with n parts, he shows that any Voronoi diagram of n parts can be reduced using the diagonal exchange operator to the canonical diagram of size n. 2.4 Exchange Tree Tagare proved constructively that their exists a sequence of diagonal exchange operators,, to convert any two Voronoi diagrams, V 1 and V 2, with the same parts into one another, but this sequence does not meet the criterion of a distance, notably it is not necessarily minimum length. To use the diagonal exchange operator as a distance measure, d(v x ; V y ), one must be able to compute the size of the minimum length sequence between V x and V y. The diagonal exchange tree was proposed by Tagare[6] to compute this distance. It is a tree rooted at V x with each exchangeable edge of V x represented by a branch from V x to the node consisting of the new Voronoi diagram formed by the application of the diagonal exchange operator to that edge of V x, the tree continues recursively for each of these new Voronoi diagrams. Since every possible sequence of operators exists as a path in the tree, then it is guaranteed that V y exists in the tree. The distance measure can be computed using a breadth rst search of the tree to nd the number of levels from V x to V y in the tree. Even with some simple pruning rules arising from the nature of the diagonal exchange operator, this tree will be exponential on the order of the number of edges of V x. As the only way to nd V y in the tree is by breadth rst search, and since the tree grows exponentially, the search time will also be exponential. The exchange tree can not be used for computing the distance, d(v x ; V y ), when the number of parts, and consequentially the number of edges, is at all large. As an example the number of parts in Tagare's examples[6] is never larger than 5 and no more than one part is allowed to be dierent between the sets. As a result a new method of calculating the distance metric for a large number of parts must be found. 4
Figure 3: Computing Arrangements from the point set representation of the parts. The left shows the complete Delaunay triangulation while the right shows the nal Area Voronoi tessellation after application of rule (3) to remove certain Delaunay triangles, arrangements are calculated directly at this stage. 3 COMPUTING ARRANGEMENTS OF MANY PARTS To compute arrangements for complex images containing many parts it suces to compute an approximation of the area Voronoi tessellation which is topologically correct. One of two methods are commonly used to create an approximation to the area Voronoi tessellation. The rst method is to represent each part of the image as a point, typically it's centroid, and to then calculate the point tessellation based on this point list. This results in a poor approximation of the area tessellation but is useful in some domains, particularly those in which the parts are circular. The second method is to sample the edges of the parts and to compute the point tessellation based upon this point set. Then each Voronoi edge is examined to determine if it intersects a part, and if so, it is removed. This is an expensive process; the representation of the Voronoi edges, e.g. as a list of points, segments, or curves, determines the complexity of the process. If for example we represent the Voronoi edges with a sequence of points, then we must examine each point of each edge for an intersection with each part. The algorithm we present in this paper improves on this by rst constructing the Delaunay triangulation, the dual of the Voronoi tessellation, and concurrently removing those Delaunay edges which would give rise to Voronoi edges which would intersect with the parts. The test for removal of these edges is performed in constant time. We begin with an image in which each pixel is labeled as either belonging to an embedded part or to the background. We allow embedded parts to exhibit any shape but require that they be contiguous. Each part is sampled along its boundary at key points to obtain a list of representative boundary points. Using these points, the Delaunay triangulation is computed. The Delaunay triangulation of the points is then be processed to remove those Delaunay quads (two adjacent Delaunay triangles) which originate from the same part. Intuitively, 5
these are the Delaunay triangles which are within an part, and therefore would not be present in the area Voronoi tessellation. More specically, the Delaunay triangles are removed if the following rule evaluates to true: ((C(V a ) = C(V b )) ^ (C(V b ) = C(V c ))) _ ((C(V b ) = C(V d )) ^ (C(V d ) = C(V c ))) (3) where, V a 6= V b 6= V c 6= V d and given a vertex V x of a Delaunay triangle, C(V x ) is a function which returns the label of the part upon which the vertex is located. We have observed that as the visual complexity of the image elements increases, the overall percentage of Delaunay triangles that we can remove increases, for images such as those shown, an average of 36% of the Delaunay triangles can be removed using rule (3), resulting in more ecent computation of the Area Voronoi tessellation. The algorithm creates an approximation to the true tessellation which is dependent upon the sampling rate selected, with higher rates corresponding to better approximations. Even at relatively low sampling rates, the topology of the created tessellation is correct and sucient for computing arrangements. For details of the algorithm and analysis of the complexity please see Burge and Monagan's 1995 paper on the construction of Area Voronoi Diagrams.[5]. At this point arrangements can be calculated directly from the area Voronoi diagram of the parts. For each part, the Voronoi cells adjacent to it listed in a counter clockwise order are the arrangements of that part. 4 ARRANGEMENTS FOR IMAGE RETRIEVAL Given a set of technical diagrams in which the parts have been previously labeled we wish to retrieve diagrams containing a certain combination, or assembly, of parts. Due to industrial standardization technical diagrams of mechanical devices contain many of the same parts, eg. bolts, gaskets, nuts, tubes, and so forth. Important assemblies are made up of combinations of standardized parts in dierent arrangements. The retrieval of all diagrams containing a specic assembly is a common problem in industry, and one can not simply query the system for all diagrams containing instances of 10 bolts, 2 gaskets, 3 tubes and a distributor. Arrangements can be used as a method of retrieval when not only the parts, but the spatial arrangement of them is important. The problem of nding the arrangement corresponding to an assembly in a large technical diagram is similar to that of \word spotting" in speech recognition. In Figure 4 the \word" we wish to spot is shown in the middle. The \word" in the same arrangement can be found in both the left and right images. In the right image it is in a dierent orientation, but still in the same arrangement. Partial matches can also be found, such as in the lower left hand corner of the right image. It should be pointed out that we are not searching based on the shape of the Voronoi cells, which we have found in previous research[7] to be too variable 1, but on the arrangement of the cells. We are currently experimenting with a dynamic programming algorithm we have developed based upon evidence collection[8] and partial string matching[9]. 1 Note for instance the shape of the Voronoi cell of the large pipe in both images, it of course varies based upon the spatial arrangement and shape of it's neighbors. 6
Section (A) Assembly Section (B) N0 N1 N2 N3 N3 N0 N2 N1 Arrangement of Labeled Parts N 0 : N 1 ; N 2 N 1 : N 2 ; N 0 N 2 : N 0 ; N 1 ; N 3 N 3 : N 2 Arrangement of Labeled Parts N 0 : N 1 ; N 2 N 1 : N 2 ; N 0 N 2 : N 3 ; N 0 ; N 1 N 3 : N 2 Figure 4: Retrieving technical diagrams based on similar assemblies. Both Right and Left sections contain the assembly shown in the Middle. The arrangements for the assembly in each section are shown underneath, note that string rotations within an arrangement, as in N 2, are considered equivalent. 5 CONCLUSION The mathematical basis for the comparison of images with the same parts using arrangements was summarized. It was shown that the previously developed algorithm for computing the distance metric has exponential complexity in the number of edges of the Voronoi diagram and is therefore unsuited for use in images with many parts. The problem of computing arrangements for images with many parts was introduced and a new algorithm to solve it was presented. Searching for assemblies of parts in collections of complex technical diagrams was shown as a motivation for the usage of arrangements of many parts. 6 BIOGRAPHY Mark Burge received the BS degree from Ohio Wesleyan University in 1990 and the MS degree from The Ohio State University in 1993, both in computer science. In 1993 he joined the computer science department of The Swiss Federal Institute of Technology in Zurich and is currently a research assistant at the Johannes Kepler University in Austria. Wilhelm Burger received the MS degree from the University of Utah in 1985 and the PhD from Johannes Kepler University in Linz, Austria, in 1992. He is the head of the computer vision group within the Dept. of Systems Science at Johannes Kepler University. This work was supported in part by the Austrian Science Foundation (FWF) under grant S7002. 7
References [1] N. Ahuja. Dot pattern processing using voronoi neighborhoods. IEEE PAMI, 4(3):336{343, 1982. [2] A. Okabe, B. Boots, and K. Sugihara. Nearest neighbourhood operations with generalized Voronoi diagrams: a review. International Journal of Geographical Information Systems, 8(1):43{71, January-February 1994. [3] C. Yap. An O(n log n) algorithm for the voronoi diagram of a set of simple curve segments. Discrete Computational Geometry, 2:365{393, 1987. [4] N. Mayya and V.T. Rajan. Voronoi diagrams of polygons: A framework for shape representation. In CVPR94, pages 638{643, 1994. [5] M. Burge and G. Monagan. Using the voronoi tessellation for grouping words and multipart symbols in documents. In VISION GEOMETRY IV, volume 2573, pages 116{123, San Diego, California, July 9-14 1995. SPIE. Online at http://www.cast.uni-linz.ac.at/vision/papers/visgeo-95/ [6] H. Tagare, F. Vos, C. Jae, and J. Duncan. Arrangement: A spatial relation between parts for evaluating similarity of tomographic section. IEEE Trans. Pattern Analysis and Machine Intelligence, 9(17):880{893, 1995. [7] M. Burge and G. Monagan. Extracting words and multi-part symbols in graphics rich documents. In 8th ICIAP, Lecture Notes in Computer Science, pages 533{538, San Remo, Italy, September 13{16 1995. IAPR, Springer Verlag. Online at http://www.cast.uni-linz.ac.at/vision/papers/iciap-95/ [8] A.K. Jain and R. Homan. Evidence-based recognition of 3D objects. PAMI, 10(6):783{802, November 1988. [9] D. Sanko and J. Kruskal. Time Warps, String Edits, and Macromolecules: The Theory and practice of sequence comparison. Addison-Wesley, Inc., Reading, Massachusetts, 1983. 8