Motion Trajectory Clustering for Video Retrieval using Spatio-Temporal Approximations

Transcription

1 Motion Trajectory Clustering for Video Retrieval using Spatio-Teporal Approxiations Shehzad Khalid and Andrew Naftel School of Inforatics, University of Manchester, PO Box 88, Sackville Street, Manchester M60 1QD, United Kingdo Abstract. A new technique is proposed for clustering and siilarity retrieval of video otion clips based on spatio-teporal object trajectories. The trajectories are treated as otion tie series and represented either by least squares or Chebyshev polynoial approxiations. Trajectory clustering is then carried out to discover patterns of siilar object otion behaviour. The coefficients of the basis functions are used as input feature vectors to a Self-Organising Map which can learn siilarities between object trajectories in an unsupervised anner. Encoding trajectories in this anner leads to efficiency gains over existing approaches that use point-based flow vectors to represent the whole trajectory as input vector. Experients on two different otion datasets vehicle tracking and pedestrian surveillance - deonstrate the effectiveness of our approach. Applications to otion data ining in video surveillance databases are envisaged. 1 Introduction An increasing nuber of systes are now able to capture and store data about object otion such as those of huans and vehicles. This has acted as a spur to the developent of sophisticated content-based visual data anageent techniques. General purpose tools are now urgently required for otion data search and retrieval, discovery and grouping of siilar otion patterns, detection of anoalous behaviour, otion understanding and prediction. Much of the recent research focus has been on representation schees for otion indexing and retrieval [1]-[11]. This work presupposes the existence of soe lowlevel tracking schee for reliably extracting object-based otion trajectories. A description of relevant tracking algoriths is not within the scope of this paper but recent surveys can be found in [12], [13]. The literature on trajectory-based otion understanding and pattern discovery is less ature but advances using learning Vector Quantization (LVQ) [14], Self-Organising Feature Maps (SOMs) [15], [16], hidden Markov Models (HMMs) [17], and fuzzy neural networks [18] have been reported. Most of these techniques attept to learn high-level otion behaviour patterns fro saple trajectories using point-based flow vectors as input to the learning phase. In this paper, we show how to circuvent this requireent by proposing a

2 trajectory classification approach based on a siple spatio-teporal representation schee used for otion indexing and retrieval. Related work within the teporal database counity on approxiation schees for indexing tie series data is highly relevant to the paraeterisation of object trajectories. However, coputer vision researchers have been slow to adopt this work. For exaple, spatioteporal trajectories have been successfully odelled using discrete Fourier transfors (DFT) [19], wavelet transfors (DWT) [20], adaptive piecewise constant approxiation (APCA) [21], and Chebyshev polynoials [22], to nae but a few. In this paper, we ai to apply tie series indexing of spatioteporal trajectories to the proble of trajectory classification and show how to learn otion patterns by using the indexing schee as an input feature vector to a neural network learning algorith. The reainder of the paper is organized as follow. We review soe relevant background aterial in section 2. In section 3, we present our trajectory odelling approach. The algorith for learning trajectories is then presented in section 4 within the fraework of a self-organising ap. This is applied in the context of clustering otion trajectories and experiental results for both vehicle and pedestrian object tracking databases are reported in section 5. The paper concludes with a discussion of the advantages of our proposed technique over copeting approaches and outlines further work. 2 Review of Previous Work Motion trajectory descriptors are known to be useful candidates for video indexing and retrieval schees. Previous work has sought to represent oving object trajectories through piecewise linear or quadratic interpolation functions [1], [2], otion histogras [4] or discretised direction-based schees [3], [8], [9]. Flexible spatioteporal representations using piecewise polynoials were proposed by Hsu [6], although consistency in applying trajectory-splitting across query and searched trajectories can be probleatic. Affine and ore general spatioteporal invariant schees for trajectory retrieval have also been presented [5], [7], [10]. The iportance of selecting the ost appropriate trajectory odel and siilarity search etric has received relatively scant attention [11]. In addition to polynoial odels, a wide variety of basis functions have been used to approxiate object trajectories [19]-[22]. Efficient indexing schees can then be constructed in the coefficient space of the basis functions. These have been copared with respect to search pruning power, CPU and I/O efficiency costs [21], [22]. It is surprising to find that any of these candidate spatioteporal trajectory indexing schees have not yet been applied to the proble of otion data ining and trajectory classification. Recent work has either used probabilistic odels such as HMMs [17] or point-based trajectory flow vectors [14], [16], [18] as a eans of learning patterns of otion activity. Flow vectors consist of spatial coordinates augented by instantaneous object velocities and optionally accelerations. These can be noralised to account for variation in trajectory lengths.

3 The contribution of this paper is to show that a trajectory-encoding schee based on input feature vectors consisting of basis function approxiation coefficients can be used to learn otion patterns. Hence, clustering and classification processes can be carried out effectively in the coefficient space. 3 Trajectory Representation The output of a otion tracking algorith is usually a set of (noisy) 2-D points (x i, y i ) representing the object s otion path over a sequence of n fraes, where i = 1,,n. In this case, the representative point is taken to be the centroid of the object s iniu bounding rectangle. The otion trajectory can be considered as two separate 1- diensional tie series, <t i, x i > and <t i, y i >, the horizontal and vertical displaceent against tie where t 1 < < t n. We consider two alternative trajectory odels, Least Squares (LS) and Chebyshev polynoial approxiations. LS polynoials are suitable for odelling siple otion trails, e.g. vehicles oving soothly along highways, whilst Chebyshev approxiations are ore appropriate for odelling coplex spatioteporal trajectories such as pedestrian tracking exhibiting stop-start and looping otions. 3.1 Least Squares Polynoials The trajectory can be approxiated by a polynoial P (t) of degree < n as x y] P ( t) = a0 + a1t + K+ at [ (1) The x, y displaceents are odelled as independent polynoials P x, P y in t. Note that separate 1-D trajectories are created for each spatial coordinate [x y]. The unknown 2(+1) coefficients {a xi, a yi }, i = 0,..., can be deterined using LS by iniising the function E with respect to a 0, a 1,... n 2 0, a1, K, a ) = {[ xi yi ] ( a0 + a1t + K ati )} i= 1 E ( a + The otion trajectories are thus indexed by vector of coefficients {(a x0,,a x ),( a y0,,a y )}. (2) 3.2 Chebyshev Polynoials Alternatively, a spatioteporal trajectory can be approxiated by a function f(t) expressed as a weighted su of Chebyshev polynoials C k (t) up to degree, defined as

4 [ x y ] = f ( t ) b k C k k= 0 (3) 1 t where ( t) = cos( k cos ( )) and C k 1 0 = f ( t k ) k= 1 b 2, bi = f ( tk ) Ci ( tk ) k= 1 (4) for t [-1,1] and i = 1,...,. The k roots of C k (t) are given by t j for 1 j k. Ipleentation details can be found in [23]. Occasionally it is necessary to approxiate the otion trail (spatial trajectory shape) in the xy plane. In this case, we replace t by x or y in one of the above equations depending on the choice of principal axis [6]. This would only be worthwhile if all trajectories can be aligned with the sae principal axis. 3.3 Siilarity Search Metric A Euclidean distance is used as the basis for coparing the siilarity of otion trajectories. Each polynoial produces a vector of coefficients which can be used to index a 2-diensional spatioteporal trajectory. Given two trajectories Q and S, we can index these by a vector of 2(+1) coefficients Q { q0,..., q } T T S { s,..., } q, s are q = [ q, q ] and s i [ sxi, syi ] = and = 0 s, where i i i xi yi = (i = 0,..., ). A Euclidean distance function (ED) on the coefficient space can be expressed as ED( Q, S) = ( q i s i ) i= 0 r r 2 = 2 ( qxi sxi ) + ( q yi s yi ) i= 0 2 (5) 4 Learning Trajectory Patterns Using Self-Organizing Maps Self-organised aps (SOMs) have been previously used for otion trajectory classification [15], [16] with trajectories encoded as flow vectors. This step can be replaced by the proposed coefficient indexing schee. A SOM discovers the underlying structure of otion trajectory data through unsupervised learning. 4.1 Network Model The architecture chosen for the SOM is very siple with a layer of input neurons connected directly to a single 1-diensional output layer. Each input neuron is con-

5 nected to every output neuron with the connection represented by a weight vector. A siilar architecture was used in [16] for learning vehicle trajectories as a eans for accident prediction. In a SOM network, physically adjacent output nodes encode the patterns in the trajectory data that are siilar and, hence, it is known as a topology-preserving ap. Consequently, siilar object trajectories are apped to the sae output neuron. The nuber of input neurons is deterined by the size of the feature vector which in this case relates to the selected nuber of coefficients for the basis functions. The degree of the polynoial can be chosen by setting a threshold on the axiu deviation of the approxiation fro the data or ean-squared error. The nuber of output neurons represents the nuber of groups in the trajectory data and this is selected anually. 4.2 Learning Algorith The algorith used to cluster the trajectories differs slightly fro the original SOM proposed by Kohonen [24]. The nuber of output neurons is initially set to a higher value than the desired nuber of clusters which we wish to produce. After training the network, clusters representing the ost siilar patterns are erged until the cluster count is reduced to the required nuber. The weights are initialised to linearly spaced values lying within the range of input values. Neighbourhood size is initially set to cover over half the diaeter of the output neurons. Let B be the input feature vector representing the set of trajectory basis function coefficients, and W the weight vector associated to each output neuron. The learning algorith coprises the following steps: 1. Deterine the winning output node k (indexed by c) such that the Euclidean distance between the current input vector B and the weight vector W k is a iniu aongst all output neurons, given by the condition B W ( t) B W ( t) k (6) c k 2. Train the network by updating the weights. A subset of the weights constituting a neighbourhood centred around node c are updated using W ( t + 1) = W ( t) + α ( t) η( k, c)( B W ( t)) (7) k k where η(k, c) = exp( r k r c 2 / 2σ 2 ) is a neighbourhood function that has value 1 when k=c and falls off with distance r k r c between nodes k and c in the output layer, σ is a width paraeter that is gradually decreased over tie and t is the training cycle index. 3. Decrease the learning rate α(t) linearly over tie. 4. After a pre-deterined nuber of training cycles, decrease the neighbourhood size. k

6 5. At the end of the training phase, erge the ost siilar cluster pairs until the desired nuber of groupings is achieved. Clusters are erged by calculating the weighted ean of the weights associated with each neuron taking into account the nuber of input saples allocated to the cluster. Assuing W a and W b are the weight vectors associated with output neurons representing the ost siilar clusters, and, n are the nuber of saple trajectories apped to these neurons respectively, a new weight value W ab for the erged cluster can be calculated as W ab Wa + nwb = + n (8) 5 Experients We now present soe results to indicate the effectiveness of the proposed trajectory clustering technique. The algorith was first tested on a highway traffic surveillance sequence. The video sequence was recorded with a stationary caera having a resolution of 176 x 144 pixels and a video capture rate of 15 fraes/second. The individual trajectories of 284 vehicles were extracted using the PTMS tracking algorith [25] illustrated in Fig. 1(a). This dataset was chosen due to its siple underlying structure and the fact that cluster visualisation could be easily interpreted in ters of vehicle lane classification. Given the unifority of the vehicles trajectory shape, it was decided to fit a least squares polynoial of degree 3, in the for x = P(y) to the otion trail. The coefficient vectors {a i0,a i1,a i2,a i3 } for each saple trajectory i were input to the SOM training network. Initially, the trajectories were grouped into 10 clusters (output neurons). Siilar clusters were then erged in a hierarchical fashion until just 4 clusters reained. The results can be seen in Fig. 1(b)-(e). Notice that a nuber of trajectories represent vehicle lane changes and these are usually the ost distant fro the cluster centre. By setting a threshold on the distance between the final adjusted weight vector and each saple trajectory, anoalous trajectories can be detected as shown in Fig. 1(f). It should be noted that soe anoalies result fro tracking errors and vehicle occlusions. Given that labelled ground truth was available, the effect on classification accuracy of partitioning the trajectory saples into training and test data was investigated. The training set was obtained by selecting saples at rando fro each labelled group. The classification errors are shown in Table 1 for different sized training/test data. The SOM achieves 100% classification accuracy for odest sized training sets deonstrating the robustness of using polynoial coefficients as inputs to the learning algorith. In the second exaple, we evaluate the perforance of the trajectory clustering algorith using the CAVIAR visual tracking database [26]. The database consisted of hand annotated video sequences of oving and stationary people and are intended to provide a testbed for bencharking vision understanding algoriths. Seantic de-

7 scriptions of the target object behaviours and otion have been generated and stored in XML files. These have been parsed these to extract ground truth-labelled object trajectories. The dataset contains 102 trajectories as shown in Fig. 2. (a) (b) (c) (d) (e) Fig. 1. Clustering of vehicle trajectories using SOM network. (a) Motion trajectories extracted using vehicle tracking. (b)-(e) Trajectories clustered into groups based on lane ebership. (f) Anoalous trajectories obtained by setting threshold on distance fro final weight vector. Since this database contains ore coplex trajectories, these are odelled spatioteporally using Chebyshev approxiations. Teporal shift and scale invariance has been used to noralise the tie series datasets[6]. Polynoials of degree 8 and above provide adequate odel fidelity to the data generating input feature vectors with 18 coefficients (9 for each spatial coordinate). Orthogonality properties ensure (f)

8 that high order approxiations ( > 4) do not yield oscillatory polynoials. We initially train a SOM network with 25 output neurons and then reduce these to 12 using the aggloerative clustering ethod described in section 4.2. The resulting trajectory groups are shown in Fig 3. Qualitatively siilar otion trajectory patterns appear to have been grouped together quite successfully. Table 1. Vehicle trajectory classification errors for SOM training based on highway lane ebership. Saple trajectories have been partitioned into different sizes of training and test datasets. Size of training set Size of test set Correct classification (%) onwards Fig. 2. Background scene containing database of ground truth labelled object trajectories. 6 Discussion and Conclusion This paper presents a neural network learning algorith for classifying spatioteporal trajectories. Global features of otion trajectories are represented well by polynoial approxiations and this is apparent in the cluster visualizations. Using coefficients of basis functions as input feature vectors to a neural network learning algorith offers an efficient alternative to the use of flow vectors for trajectory classification. A current disadvantage is the handling of partial trajectory atching which is unsuited to a polynoial-based representation. One possibility is to paraeterise the trajectory length and augent the feature vector with additional entries. In future work we would like to extend this approach to the autonoous detection of anoalous trajectories and prediction of unusual otion behaviour.

9 Fig. 3. Clustering of spatioteporal object trajectories fro CAVIAR database into 12 distinct groups using SOM network. References 1. S-F. Chang, W. Chen, H.J. Meng, H. Sundara, and D. Zhong, A fully autoated contentbased video search engine supporting spatioteporal queries, IEEE Trans. Circuits Syst. Video Technol., vol. 8, no. 5, pp , Sept S. Jeannin and A. Divakaran, MPEG-7 visual otion descriptors, IEEE Trans. Circuits Syst. Video Technol., vol. 11, no. 6, pp , June S. Dagtas, W. Ali-Khatib, A. Ghafor, and R.L. Kashyap, Models for otion-based video indexing and retrieval, IEEE Trans. Iage Proc., vol. 9, no. 1, pp , Jan Z. Aghbari, K. Kaneko, and A. Makinouchi, Content-trajectory approach for searching video databases, IEEE Trans. Multiedia, vol. 5, no. 4, pp , Dec F. Bashir, A. Khokhar, and D. Schonfeld, Segented trajectory-based indexing and retrieval of video data, in Proc. IEEE Int. Conf. Iage Processing, Spain, 2003, pp

10 6. C-T. Hsu and S-J. Teng, Motion trajectory based video indexing and retrieval, in Proc. IEEE Int. Conf. Iage Processing, pt 1, 2002, pp F. Bashir, A. Khokhar, and D. Schonfeld, A hybrid syste for affine-invariant trajectory retrieval, in Proc. MIR 04, 2004, pp C. Shi and J. Chang, Content-based retrieval using trajectories of oving objects in video databases, in Proc. IEEE. 7th Int. Conf. Database Systes for Advanced Applications, 2001, pp C. Shi and J. Chang, Trajectory-based video retrieval for ultiedia inforation systes, in Proc. ADVIS, LNCS 3261, 2004, pp Y. Jin and F. Mokhtarian, Efficient video retrieval by otion trajectory, in Proc. BMVC 04, S. Khalid and A. Naftel, Evaluation of atching etrics for trajectory-based indexing and retrieval of video clips, in Proc. IEEE WACV, Colorado, USA, Jan L. Wang, W. Hu, and T. Tan, Recent developents in huan otion analysis, Pattern Recognition, vol. 36, no. 3, pp , W. Hu, T. Tan, L. Wang, and S. Maybank, A survey on visual surveillance of object otion and behaviors, IEEE Trans. Systes, Man & Cybernetic, Part C, vol.34, no.3, pp , August N. Johnson and D. Hogg, Learning the distribution of object trajectories for event recognition, Iage Vis. Coput., vol. 14, no. 8, pp , J. Owens and A. Hunter, Application of the self-organising ap to trajectory classification, in Proc. IEEE Int. Workshop Visual Surveillance, pp , W. Hu, X. Xiao, D. Xie, T. Tan, and S. Maybank, Traffic accident prediction using 3-D odel-based vehicle tracking, IEEE Trans. Vehicular Tech., vol. 53, no. 3, pp , May J. Alon, S. Sclaroff, G. Kollios, and V. Pavlovic, Discovering clusters in otion tieseries data, in Proc. IEEE CVPR, June W. Hu, D. Xie, T. Tan, and S. Maybank, Learning activity patterns using fuzzy selforganizing neural networks, IEEE Trans. Systes, Man & Cybernetic, Pt. B, vol. 34, no. 3, pp , June C. Faloutsas, M. Ranganathan, and Y. Manolopoulos, Fast subsequence atching in tieseries databases, in Proc. ACM SIGMOD Conf., 1994, pp K. Chan and A. Fu., Efficient tie series atching by wavelets, in Proc. Int. Conf. Data Engineering, Sydney, March 1999, pp E. Keogh, K. Chakrabarti, M. Pazzani, and S. Mehrota, Locally adaptive diensionality reduction for indexing large tie series databases, in Proc. ACM SIGMOD Conf., 2001, pp Y. Cui and R. Ng, Indexing spatio-teporal trajectories with Chebyshev polynoials, in Proc. ACM SIGMOD Conf., Paris, June 2004, pp W.H. Press, B.P. Flannery, S.A. Teukolsky, and W.T. Vetterling, Nuerical Recipes in FORTRAN: The Art of Scientific Coputing, 2nd ed. Cabridge, England: Cabridge University Press, T. Kohonen, Self-Organizing Maps, 2 nd ed. New York: Springer-Verlag, 1997, vol J. Melo, A. Naftel, A. Bernardino, and J. Santos-Victor, Viewpoint independent detection of vehicle trajectories and lane geoetry fro uncalibrated traffic surveilllance caeras, in Proc. Int. Conf. Iage Anal. Recog. (ICIAR 04), Porto, Portugal, 2004, pp CAVIAR: Context aware vision using iage-based active recognition. (2004, Jan. 10). [Online]. Available: