Characterizing Trajectories of Moving Objects Using Natural Language Path Descriptions

Transcription

1 Characterizing Trajectories of Moving Objects Using Natural Language Path Descriptions Elisabeth André, Guido Bosch, Gerd Herzog, Thomas Rist SFB 314, Project VITRA, Universität des Saarlandes D Saarbrücken, Germany Abstract The topic of this paper 1 is the analysis of the semantics of the particular spatial relations 'along' and 'past' which are used to characterize the path of moving objects. The German dialogue system CITYTOUR, which answers questions about the spatial relations of objects in a scene, is presented by means of a simple dialogue example. The representational prerequisites of the static and the dynamic objects, required for computational analysis are presented. Two concrete predicate functions testing whether a dynamic object moves along or moves past a static object are described in detail. This paper appeared in: Proc. of the 7th ECAI, volume 2, pp. 1 8, Brighton, UK, This report describes work done in the project VITRA, which is part of the SFB 314 Research Program of the Science Foundation (DFG) on AI and Knowledge-Based Systems. 1

2 1 Introduction The development of knowledge-based systems for natural language descriptions of image sequences includes the problem of defining a computational semantics of natural language expressions which describe spatial relations. Aspects of this problem have been discussed in theoretical studies e.g. by Boggess [1978], Herskovits [1980], and Dirven [1981]. In systems like NAOS, Neumann [1984], HAM-RPM, von Hahn et al. [1980], GEOSYS, Fürnsinn et al. [1984], SWYSS, Hußmann and Schefe [1984], or MERCATOR, Davis [1984], many aspects of spatial relations have already been implemented. Though path expressions have been discussed from a linguistic point of view, they have not yet been implemented in a system like those mentioned above. Our interests on this topic arise from implementing the CITYTOUR system, André et al. [1985], which is a German question-answering system that simulates a fictitious sightseeing tour through an interesting part of a particular city. Fig. 1 shows an exam- Figure 1: The basic windows of the CITYTOUR system ple of the domain of discourse: a part of a city with static objects (e.g. buildings) and dynamic objects (e.g. pedestrians). A special dynamic object is the sightseeing bus which is graphically represented as a big dot. The questioner is assumed to be sitting in that bus. Thus the conversational partner is himself part of the scene under discussion. Therefore CITYTOUR's answers can take into account the current position of 2

3 the observer. In this respect CITYTOUR differs from the systems mentioned above. The dialogue window shown in Fig. 1 contains a short typical dialogue example: The first question Liegt die Sparkasse hinter dem Rathaus? (Is the bank behind the town hall?) produces the negative answer Nein, das kann man nicht sagen. (No, you can't say that) because the front side (cf. the bold edges in Fig. 1) of the reference object Rathaus is decisive for the system's answer. If the deictic expression von hier aus (from here) is added to the first question, CITYTOUR considers the passenger's position instead of the front side of the reference object and returns the answer Ja, die Sparkasse befindet sich direkt hinter dem Rathaus von hier aus. (Yes, the bank is directly behind the town hall from here.). In addition to some static spatial relations (e.g. in front of, behind, left, right, between) and simple dynamic relations (e.g. move to the front side of, move to the back side of, move to the left side of, move to the right side of), the focus of our research was the computational semantics of path prepositions. In this paper we present two new algorithms which decide whether the path of a moving object can be described in German by the path prepositions entlang (along) or vorbei (past). 2 Representational Prerequisites The most important prerequisite for evaluating path expressions is that the static objects are not simply represented as centroids. Thus in the CITYTOUR system the representation of large static objects also includes a closed polygon, a functional property called prominent front (cf. the bold edge of the polygon as shown in Fig. 2) and a delineative rectangle which is oriented on the object's prominent front. Dynamic objects (e.g. pedestrians) are simplified and represented as centroids. The movement of a dynamic object is represented by a trajectory, a list of pairs, each of which consists of the location of the object and a time-marker. For example, the trajectory of a pedestrian may be the list ((P t1 t 1 )(P t2 t 2 ) :::: (P t i t i) ::::), where P t i denotes the position of the pedestrian at the time t i (cf. Fig. 3). We use the tuple (< relident < object 1 ::: < object n ) to denote a spatial relation, where rel ident is a predicate and object 1 ::: object n are its arguments. For example the utterance The man walked along the street will be associated with the tuple (along man11 street22). 3 The Relation vorbei (past) Frequently the relation vorbei (past) can be combined with other relations like behind, in front of, left of and so on, e.g. the man is passing the left side of the post-office. These relations restrict the semantics of the relation vorbei (past). The following 3

4 Figure 2: Different representation forms of a static object Figure 3: The trajectory of a dynamic object algorithm defines a predicate function which decides whether two objects are in the relation vorne vorbei (passing in front of). By means of the delineative rectangle we compute the four half-planes (L, H, V, R), as shown in Fig. 4. The delineative rectangle has been computed according to the given prominent front of the object. We will show how we can find out whether a dynamic object passes in front of a static object. Some general definitions follow: SO = set of static objects DO = set of dynamic objects object size = arithmetic mean of the sides of delineative rectangle 4

5 Figure 4: Dynamic object passing the right side and the back side of a static object The trajectory T of a dynamic object is written in terms of the sequence T = ((P 1 t 1 )(P 2 t 2 ) ::: (P i t i ) ::: (P n;1 t n;1)(p n t n )) where 1 i n 2 N P i 2 R 2 is the position of an object 2 DOat the time-markert i 2 N: The function trajectory maps from DO to the set of all trajectories. L, R, H, V R 2 according to Fig. 4. We define a core interval of a dynamic relation as the minimal time interval [t begin t end ], for which the sub-trajectory ((P begin t begin ):::(P end t end )) satisfies the predicate vorne vorbei(obj1 Obj2), where Obj1 2 DO and Obj2 2 SO. Definition of the predicate vorne vorbei (passing in front of): [t begin t end ] is a core interval for which the predicate vorne vorbei (passing in 5

6 front of) is satisfied () 1. (P begin 2 R ^ P end 2 L) _ (P begin 2 L ^ P end 2 R), i.e., the dynamic object passes the static object from the right to the left or the reverse. 2. 8t i 2 ]t begin t end [: (P i =2 L S R) ^ (P i 2 V ), i.e., the dynamic object is exactly in front of the static object between t begin and t end. 3. 8i 2 N : t begin < t i < t end : distance1(p i Obj) <k 1 (Obj), where distance1(p i Obj) is the distance of the point P i from the delineative rectangle of the static object Obj, and k 1 (Obj) is an object-dependent threshold, i.e., the distance between the static object and the dynamic object does not exceed an object-dependent threshold. Experiments with the CITYTOUR system have shown that for k 1 values in the range of 2 object size yield reasonable results. Analogous definitions for the remaining cases rechts vorbei, links vorbei, hinter vorbei (passing right of, passing left of, passing behind of) are easy to formulate. 4 The Relation entlang (along) The analysis of the dynamic relation entlang (along) shows that the exact shape of the trajectory has to be taken into account to a higher degree than in the analysis of the relation vorbei (past). As a first approach to the modeling of a predicate entlang (along) the following definition has been developed. Definition of the predicate entlang (along): [t begin t end ] is a core interval for which the predicate entlang (along) is satisfied () 1. 8i 2]t begin t end [: distance2(p i Obj) <k 2 (Obj) where distance2(p i Obj) is the minimal distance between the point P i and the static object Obj, (i.e. the polygonal representation of the object), and k 2 (Obj) is an object-dependent threshold, i.e., the distance between the dynamic and the static object must not exceed an object-dependent threshold. 2. The direction of the movement is either clockwise or counterclockwise with regard to the orientation of the object (the polygonal representation implemented in LISP implies an orientation). 3. P end;1 distance3(p i=begin i P i+1 ) k 3 (Obj) where distance3(p j P k ) is the Euclidian distance between the two points P j and P k, and k 3 (Obj) is an 6

7 object-dependent threshold, i.e., the dynamic object has to cover a distance which exceeds the object-dependent threshold k [t t ] [t begin i endj begin t end ] for which (1) to (3) hold, i.e., there is no sub-trajectory that satisfies the predicate entlang (along). Figure 5: Trajectory of an object moving along the church2 For k 2 we found that values in the range of 0:3 object size yield reasonable results, whereas k 3 has been set to 0:75 object size. 5 Concluding Remarks For an implementation of a generalized algorithm testing the relation vorbei without further spatial specification the use of one of the four predicates vorne vorbei, hinten vorbei, rechts vorbei, links vorbei (to pass the front side of, back side of, right side of, left side of) proves to be insufficient. Fig. 6 shows an example where a human observer would decide The man passed the church, but none of the four cases vorne vorbei, hinten vorbei, rechts vorbei, links vorbei is applicable. In some cases the predicate entlang (along) is satisfied, although it does not agree with natural language use. If the static object is full of nooks and crannies, it seems reasonable to use a more complicated algorithm which compares the course of the trajectory with the edges of the object. In our opinion the calculation of the applicability of the predicate entlang (along) has to consider the shape of the static object to a greater extent than the calculation of the predicate vorbei (past). This has the following consequences for our algorithms: Whereas it is sufficient for the calculation of the predicate vorbei (past) to utilize the delineative rectangle of the static object, the calculation of the applicability of the predicate entlang (along) starts out from the polygonal representation of an object (cf. distance1, distance2 in the algorithms presented). Moreover the threshold k 1 for the predicate vorbei is more generous than k 2 for the predicate entlang (along). Therefore 7

8 Figure 6: Passing that will not be matched by the algorithm trajectory1 shown in Fig. 7 satisfies our definitions for the predicate entlang (along) and the predicate vorbei (past), whereas trajectory2 only satisfies the latter. 6 Future Work Currently CITYTOUR can only test whether a predicate is satisfied or not. That means that all arguments of the tupels are instantiated. Problems arise dealing with partially instantiated tupels, e.g. (past, man1,?building), (?predicate, man1, church21), where? marks variables. For the latter case an algorithm has to be developed which selects the most suitable predicate from a set of predicates. It could be used to decide whether an object passes a building or moves along it following its contours. In addition, the concept of degree of applicability of a predicate can be introduced, which expresses the extent to which a spatial relation is appropriate to describe a given object configuration. In CITYTOUR the degree of applicability has already been implemented for static relations. In surface structure the degree of applicability is expressed by linguistic hedges, such as 'directly' and 'immediately'. Further attention should be paid to the question of whether the algorithms developed for a special discourse world can be transferred to another domain. The question whether the chosen types of representation (centroid, polygon, deliniative rectangle) are sufficient, deserves particular interest. In the project VITRA, problems occuring when generating present-tense descriptions of motion will be examined in detail. In this case we have to consider uncom- 8

9 Figure 7: Comparison between the applicability of the predicates along and past pleted movements which will require expectation-driven heuristics for recognizing events before they are finished. 7 Technical Notes The Citytour system is implemented on a SYMBOLICS 3600 LISP machine. The program is written in FUZZY and ZetaLISP. The response time for a typical question is about five seconds. Acknowledgements We would like to thank Prof. Dr. Wolfgang Wahlster and Gudula Retz-Schmidt for their insightful comments and suggestions throughout the development and the writing of this paper. 9

10 References E. André, G. Bosch, G. Herzog, T. Rist. CITYTOUR Ein nürlichsprachliches Anfragesystem zur Evaluierung räumlicher Präpositionen. Abschlußbericht zum fortgeschrittenenpraktikum prof. dr. w. wahlster, wintersemester 1984/85, Fachbereich Informatik, Univ. des Saarlandes, L. C. Boggess. Computational Interpretation of English Spatial Prepositions. Technical Report T-75, Coordinated Science Laboratory, Univ. of Illinois, E. Davis. Representing and Acquiring Geographic Knowledge. Ph.D. thesis, Computer Science Department, Yale Univ., New Haven, CT, R. Dirven. Spatial Relations in English. In: G. Radden, R. Dirven, eds., Anglistik und Englischunterricht. Kasusgrammatik und Fremdsprachendidaktik, pp , WVT Wissenschaftlicher Verlag, Trier, M. Fürnsinn, M. N. Khenkhar, B. Ruschkowski. GEOSYS Ein Frage-Antwort- System mit räumlichem Vorstellungsvermögen. In: C.-R. Rollinger, ed., Probleme des (Text-) Verstehens, Ansätze der künstlichen Intelligenz, pp , Niemeyer, Tübingen, A. Herskovits. On the Spatial Uses of Prepositions. In: Proc. of the 18th ACL, pp. 1 5, Philadelphia, PA, M. Hußmann, P. Schefe. The Design of SWYSS, a Dialogue System for Scene Analysis. In: L. Bolc, ed., Natural Language Communication with Pictorial Information Systems, pp , Hanser/McMillan, München, B. Neumann. Natural Language Description of Time-Varying Scenes. Report 105, Fachbereich Informatik, Univ. Hamburg, W. von Hahn, W. Hoeppner, A. Jameson, W. Wahlster. The Anatomy of the Natural Language Dialogue System HAM-RPM. In: L. Bolc, ed., Natural Language Based Computer Systems, pp , Hanser/McMillan, München,