High Expressive Spatio-temporal Relations

Size: px

Start display at page:

Download "High Expressive Spatio-temporal Relations"

Todd Marshall
5 years ago
Views:

1 From: FLAIRS-02 Proceedings. Copyright 2002, AAAI ( All rights reserved. High Expressive Spatio-temporal Relations R. CHBEIR, Y. AMGHAR, A. FLORY LISI INSA de Lyon 20, Avenue A. Einstein F VILLEURBANNE - FRANCE Phone: (+33) Fax: (+33) {rchbeir, amghar, flory}@lisi.insa-lyon.fr Abstract The management of images remains a complex task that is currently a cause for several research works in various domains such as GIS, medicine, etc. In line with this, we are interested in this work with the problem of image retrieval that is mainly related to the complexity of image representation. In this paper, we address the problem of relation representation and we propose a novel method for identifying and indexing relations between salient objects. Our method is based on defining several intersection sets for each feature like shape, time, etc. Relations are then computed using a binary intersection matrix between sets. Several types of relations can be calculated using our method in particular spatial and temporal relations. We show how our method is able to provide high expressive power and homogeneous representation to relations in comparison to the traditional methods. Introduction During the last decade, a lot of work has been done in information technology in order to integrate image retrieval in the standard data processing environments. Image retrieval is involved in several domains [(Yoshitaka and Ichikawa 1999), (Rui, Huang, and Chang 1999), Grosky 1997), (Smeulders, Gevers, and Kersten, 1998)] such as Geographic Information Systems, Medicine, Surveillance, etc. where queries criteria are based on different types of features. Principally, spatio-temporal relations are very important in image retrieval and need to be considered in several domains. In medicine, for instance, the spatial data in surgical or radiation therapy of brain tumors is decisive because the location and the temporal evolution of a tumor has profound implications on a therapeutic decision (Chbeir and Favetta 2000). Hence, it is crucial to provide a precise and powerful system to express spatio-temporal relations required in image retrieval. In this paper, we address this issue and propose a novel method that can easily compute several types of relations with better expressions. The rest of this paper is organized as follows. In section 2, we present the related work. Section 3 is devoted to define Copyright 2002, American Association for Artificial Intelligence ( All rights reserved. our method for calculating relations. Finally, conclusions and future orientations are given in section 4. Related Work Relations between either salient objects, shapes, points of interests, etc. have been widely used in image indexing such as R-tree (Guttman 1984), R+-tree (Sellis, Roussopoulos, and Faloutsos 1987), R*-tree (Beckmann 1990), hb-tree (Lomet and Salzberg 1990), ss-tree (White and Jain 1996), TV-tree (Lin, Jagadish, and Faloutsos 1994), 2D-String (Chang, Shi, and Yan 1987), 2D-G String (Chang and Jungert 1991), 2D-C+ String (Huang and Jean 1994], θr-string (Gudivada 1995], σ-tree (Chang and Jungert 1997), etc. Temporal and spatial relations are the most used relations in image indexing. To calculate temporal relations, Allen relations (Allen 1983) are often used. Allen proposes 13 temporal relations (Before, After, Meet, Touched By, Started by, Overlapped By, Start With, Started With, Finish wish, Finishes, Contain, During, Equal) in which 6 are symmetrical. On the other hand, three major types of spatial relations are generally proposed in image representation (Egenhofer, Frank, and Jackson 1989): Metric relations: measure the distance between salient objects (Peuquet 1986). For instance, the metric relation far between two objects A and B indicates that each pair of points A i and B j has a distance grater than a certain value δ. Directional relations: describe the order between two salient objects according to a direction, or the localisation of salient object inside images (El-kwae and Kabuka 1999). In the literature, fourteen directional relations are considered: north, south, east, west, north-east, north-west, south-east, south-west, left, right, up, down, front and behind. Topological relations: describe the intersection and the incidence between objects [(Egenhofer and Herring 1991), (Egenhofer 1997)]. Egenhofer (Egenhofer and FLAIRS

2 Herring 1991) has identified six basic relations: Disjoint, Touch, Overlap, Cover, Contain, and Equal. In spite of all the proposed work to represent complex visual situations, several shortcomings exist in the methods of spatial relation computations. For instance, Figure 1 shows two different spatial situations of three salient objects that are described by the same spatial relations in both cases, in particular topological relations: a1 Touch a2, a1 Touch a3, a2 Touch a3; and directional relations: a1 Above a3, a2 Above a3, a1 Left a2. F i (or inferior): contains elements of F - that do not belong to any other intersection set and inferior to F elements on the basis of i dimension. F i (or superior): contains elements of F - that do not belong to any other intersection set and superior to F elements on the basis of i dimension. F i F F Ω F i Dimension i Figure 1: Two different spatial situations. In this paper, we present our method able to provide a high expressive power to represent several types of relations in particular spatial and temporal ones. Proposition Our proposal represents an extension of the 9-Intersection model (Egenhofer and Herring 1991). It provides a general method for computing not only topological relations but also other types of relations in particular temporal and spatial relations with higher precision. The idea is to identify relations in function of two values of a feature such as shape, position, time, etc. The shape feature gives spatial relations, the time feature gives temporal relations, and so on. To identify a relation between two values of the same feature, we propose the use of an intersection matrix between several sets defined in function of each feature: Definition Let us first consider a feature F. We define its intersection sets as follows: The interior F Ω : contains all elements that cover the interior or the core of F. In particular, it contains the barycentre of F. The definition of this set has a great impact on the other sets. F Ω may be empty ( ). The boundary F: contains all elements that allow determining the frontier of F. F is never empty ( ). The exterior F - : is the complement of F Ω F. It contains at least two elements (the minimum value) and (the maximum value). F - can be divided into several disjoint subsets. This decomposition depends on the number of the feature dimensions. For instance, if we consider a feature of one dimension i (such as the acquisition time of a frame in a video), two intersection subsets can be defined (Figure 2): Figure 2: Intersection sets of one-dimensional feature. If we consider a feature of 2 dimensions i and j (as the shape of a salient object in a 2D space), we can define 4 intersection subsets: F i F j (or inferior): contains elements of F - that do not belong to any other intersection set and inferior to F Ω and F elements on the basis of i and j dimensions. F i F j : contains elements of F - that do not belong to any other intersection set and inferior to F Ω and F elements on the basis of i dimension, and superior to F Ω and F elements on the basis of j dimension. F i F j : contains elements of F - that do not belong to any other intersection set and superior to F Ω and F elements on the basis of i dimension, and inferior to F Ω and F elements on the basis of j dimension. F i F j (or superior): contains elements of F - that do not belong to any other intersection set and superior to F Ω and F elements on the basis of i and j dimensions. More generally, we can determine intersection sets (2 n ) of n dimensional feature. In addition, we use a tolerance degree in the feature intersection sets definition in order to represent separations between sets and to provide a simple certainty parameter for the end-user. For this purpose, we use two tolerance thresholds: Internal threshold ε i : defines the distance between F Ω and F, External threshold ε e : defines the distance between subsets of F -. To calculate relation between two features, we establish an intersection matrix of their corresponding feature intersection sets. Matrix cells have binary values: 0 whenever intersection between sets is empty, 1 otherwise For one-dimensional feature (such as the acquisition time), the following intersection matrix is used to compute relations between two values A and B: 472 FLAIRS 2002

3 R(A, B) = Figure 3: Intersection matrix of two values A and B on the basis of one-dimensional feature. For two-dimensional feature (such as the shape) of two values A and B, we obtain the following intersection matrix: A Ω A Ω B A Ω B1 Figure 4: Intersection matrix of two values A and B on the basis of two-dimensional feature. Temporal relations To calculate temporal relations, we define the following intersection sets for each interval 1 T (Figure 5): T Ω : represents the interval T without initial instant t i and final instant t f. It is situated between t i +ε i and t f -ε i. T: contains both initial instant t i and final instant t f. ε i : represents the internal tolerance threshold that separates T Ω from T,. ε e : represents the external tolerance threshold that separates T from both T and T. T : is the set of values strictly anterior to t i -ε e. T : is the set of values strictly posterior to t f +ε e. T A Ω A Ω B A Ω B A Ω B A A B A B A B A A B A B A B A A B A B A B T i A A B A B1 B1 B1 B1 B1 T T Ω A Ω B1 ε e ε i ε i Time T Figure 5: Graphical representation of temporal interval T. Using our method, 33 (32+1) temporal relations between two intervals are identified instead of 13 (Allen 1983). We show here below 7 of these principal temporal relations (non symmetrical) between two intervals T 1 and T 2 : TR 1 : The interval T 1 starts and ends before T 2 : 1 In this paper, we do not consider intersections sets between instants. A B1 B1 B1 B1 B1 t f A Ω A ε e T A Ω B1 A B1 B1 B1 B1 B1 TR2: The interval T 1 starts before T 2 and ends almost before T 2. T 1 T 2 TR 3 : The interval T 1 ends when T 2 starts: TR 4 : The interval T 1 finishes almost after the beginning of T 2 : TR 5 : The interval T 1 starts before the beginning of T 2 and finishes during T 2 : TR 6 : The interval T 1 starts before the beginning of T 2 and finishes almost before the end of T 2 : TR 7 : The interval T 1 starts almost before T 2 and finishes during T 2 : Our method leads also to identify 5 temporal relations between instants instead of 3 traditional relations. This shows that our method provides a higher expressive power than traditional methods. FLAIRS

We obtain 10 (2+2 3 ) intersection sets that conduct to 10x10 intersection matrix cells in order to calculate spatio-temporal relation between two features. P Ω : represents the surface of P.

4 Spatial relations In this paper, we consider only spatial relations calculated between polygonal shapes. Intersections sets of a polygon P are defined as follows (Figure 6): and 2 dimensions for 2D shape s. Intersections sets are defined exactly as time and shape features. We obtain 10 (2+2 3 ) intersection sets that conduct to 10x10 intersection matrix cells in order to calculate spatio-temporal relation between two features. P Ω : represents the surface of P. It contains all points situated in the internal part of P. P: is the frontier of P. P x P y : contains all points that do not belong to P Ω nor to P, where coordinates are inferior to P x P y : contains all points that do not belong to P Ω nor to P, where coordinates are superior to P x P y : contains all points that do not belong to P Ω nor to P, where abscises are strictly inferior to barycentre s and ordinates are strictly superior to P x P y : contains all points that do not belong to P Ω nor to P, where abscises are strictly superior to barycentre s and ordinates are strictly inferior to R(a1, a2) Situation 1 R(a1, a3) Situation y F i F j F i F j F Ω F i F j F F i F j R (a1, a2) R (a1, a3) Figure 6: Intersection sets of polygonal shape. Using our method, we are able to provide a high expression power to spatial relations that can be applied to describe images and formulate complex visual queries in several domains. For example, for Figure 1 that shows two different spatial situations between three salient objects a1, a2, and a3, our method expresses the spatial relations as shown in Figure 3. The relations R(a1, a2) and R (a1, a2) are equal but the relations R(a1, a3) and R (a1, a3) are clearly distinguished. Similarly, we can express relations between a2 and a3 in both situations (Figure 7). Moreover, our method allows combining both directional and topological relation into one binary relation, which is very important for indexing purposes. There are no directional and topological relations between two salient objects but only one spatial relation. Hence, we can propose a 1D-String to index images instead of 2D-Strings (Chang, Shi, and Yan 1987). Discussion Using our method, we can provide other type of relations. For instance, we can calculate spatio-temporal relations into one representation. For that, we define a new feature with 3 dimensions: 1 dimension for time representation, x Figure 7: Identified spatial relations using our method. Conclusion In this paper, we presented our method to identify relations in image representation. We used spatial and temporal relations as a support to show how relations can be powerfully expressed within our method. This work aims to homogenize, reduce and optimize the representation of relations. We currently experiment its integration in our prototype MIMS (Chbeir, Amghar, and Flory 2001). However, in order to study its efficiency and limits, our method requires more intense experiments in complex environment where great number of feature dimensions and salient objects exist. Furthermore, we will examine it in other type of media like videos. References Allen J.F., Maintaining knowledge about temporal Intervals, Communications of the ACM, 1983, Vol. 26, N 11, P Beckmann N., The R*-tree: an efficient and robust access method for points and rectangles, SIGMOD Record, 1990, Vol. 19, N 2, P FLAIRS 2002

5 Chang S. K., Jungert E., Pictorial data management based upon the theory of symbolic projections, in Journal of Visual Languages and Computing, 1991, Vol. 2, N 3, P Chang S.K., Jungert E., Human- and System-Directed Fusion of Multimedia and Multimodal Information using the Sigma-Tree Data Model, Proceedings of the 2 nd International Conference on Visual Information Systems, 1997, San Diego, P Chang S.K., Shi Q.Y., Yan C.W., Iconic Indexing by 2-D Strings, IEEE-Transactions-on-Pattern-Analysis-and- Machine-Intelligence, 1987, Vol. PAMI-9, N 3, P Chbeir R., Amghar Y., Flory A., A Prototype for Medical Image Retrieval, International Journal of Methods of Information in Medicine, Schattauer, Issue 3, 2001, P Chbeir R. and Favetta F., A Global Description of Medical Image with a High Precision ; IEEE International Symposium on Bio-Informatics and Biomedical Engineering IEEE-BIBE'2000, Washington D.C., USA, (2000) November 8th-10 th, P Egenhofer M. and Herring J., Categorising Binary Topological relationships between Regions, Lines, and Points in Geographic Databases, A Framework for the Definition of Topological Relationships and an Algebraic Approach to Spatial Reasoning within this Framework, Technical Report 91-7, National center for Geographic Information and Analysis, University of Maine, Orono, Egenhofer M., Query Processing in Spatial Query By Sketch, Journal of Visual Language and Computing, Vol 8 (4), 1997, P Egenhofer M., Frank A., and Jackson J., A Topological Data Model for Spatial Databases, Symposium on the Design and Implementation of Large Spatial Databases, Santa Barbara, CA, Lecture Notes in Computer Science, July 1989, Vol. 409, P El-kwae M.A. and Kabuka M.R., A robust framework for Content-Based Retrieval by Spatial Similarity in Image Databases, ACM Transactions on Information Systems, vol. 17, N 2, avril 1999, P Grosky W.I., Managing Multimedia Information in Database Systems, Communications of the ACM, 1997, Vol. 40, No. 12, P Gudivada V.N., Spatial similarity measures for multimedia applications, Proceedings of SPIE Storage and Retrieval for Still Image and video Databases III, 1995, San Diego/La Jolla, CA, USA, Vol. 2420, P Guttman A., "R-trees: a dynamic index structure for spatial searching", SIGMOD Record, 1984, Vol. 14, N 2, P Huang P.W. and Jean Y.R., Using 2D C+-Strings as spatial knowledge representation for image database management systems, Pattern Recognition, 1994, Vol. 27, N 9, P Lin K.I., Jagadish H. V., Faloutsos C., The TV-tree: An index structure for high dimensional data, Journal of Very Large DataBase, 1994, Vol. 3, N 4, P Lomet D.B., Salzberg B., The hb-tree: A multiattribute indexing method with good guaranteed performance, ACM Transactions on Database Systems, 1990, Vol. 15, N 4, P Peuquet D. J., The use of spatial relationships to aid spatial database retrieval, Proc. Second Int. Symp. on Spatial Data Handling, Seattle, 1986, P Rui Y., Huang T.S., and Chang S.F., Image Retrieval: Past, Present, and Future, Journal of Visual Communication and Image Representation, 1999, Vol. 10, P Sellis T., Roussopoulos N., Faloutsos C., the R+-tree: a dynamic index for multidimensional objects, Proceedings of the 13 th International Conference of VLDB, 1987, Brighton, England, P Smeulders A.W.M., Gevers T., Kersten M.L., Crossing the Divide Between Computer Vision and Databases in Search of Image Databases, Visual Database Systems Conf., Italy, 1998, P White D.A., Jain R., Similarity Indexing with the SStree, Proceedings of the 12 th International Conference on Data Engineering, 1996, New Orleans, Louisiana, P Yoshitaka A. and Ichikawa T., A Survey on Content- Based Retrieval for Multimedia Databases, IEEE Transactions on Knowledge and Data Engineering, vol. 11, No. 1, 1999, P FLAIRS

DERIVING SPATIOTEMPORAL RELATIONS FROM SIMPLE DATA STRUCTURE

DERIVING SPATIOTEMPORAL RELATIONS FROM SIMPLE DATA STRUCTURE Ale Raza ESRI 380 New York Street, Redlands, California 9373-800, USA Tel.: +-909-793-853 (extension 009) Fax: +-909-307-3067 araza@esri.com