Querying Complex Spatio-Temporal Sequences in Human Motion Databases

Transcription

1 Querying Complex Spatio-Temporal Sequences in Human Motion Databases Yueguo Chen #, Shouxu Jiang, Beng Chin Ooi #, Anthony KH Tung # # School o Computing, National University o Singapore Computing, Law Link, Singapore 759 {chenyueg, ooibc, atung}@compnusedusg School o Computer Science and Technology, Harbin Institute o Technology Harbin, China 5 jsx@hiteducn Abstract Content-based retrieval o spatio-temporal patterns rom human motion databases is inherently nontrivial since inding eective distance measures or such data is diicult These data are typically modelled as time series o high dimensional vectors which incur expensive storage and retrieval cost as a result o the high dimensionality In this paper, we abstract such complex spatio-temporal data as a set o rames which are then represented as high dimensional categorical eature vectors New distance measures and queries or high dimensional categorical time series are then proposed and eicient query processing techniques or answering these queries are developed We conducted experiments using our proposed distance measures and queries on human motion capture databases The results indicate that signiicant improvement on the eiciency o query processing o categorical time series (more than, times aster than that o the original motion sequences) can be achieved while guaranteeing the eectiveness o the search I INTRODUCTION Content-based retrieval o spatio-temporal patterns rom human motion databases is inherently nontrivial since inding eective distance measures or such data is diicult Deining similarities (or distances) on such data is diicult because o three reasons: ) it s hard to extract eective numerical eatures rom them such that an eective distance can be aggregated based on the pair-wise dierences o the eatures; ) the transormed data (eatures) are typically in high dimensional space, while the similarities between two such sequences may only exist in some subspace o the overall spatio-temporal space The simple aggregation o distances may not take such partial similarity into account; 3) matching between the sequences might not be perect and variation along the spatial and temporal dimensions must be taken into account Many existing studies [], [], [3] model such data as high dimensional trajectories where each rame or segment o the human motion is represented as a numerical vector in the track o the trajectories This essentially converts the data into a high dimensional time series and the distance between any pair sequence is then measured based on the distance o the two time series However, existing studies on the distance measures o time series [4], [5], [], [7] have been ocused on time series with low dimensionality (usually -dimensional to 3-dimensional trajectories) They cannot be easily applied to high dimensional time series because o the ollowing three reasons ) the eiciency o warping distances or highdimensional time series cannot be well improved by simply extending existing lower bounding techniques [], [8] due to the well-known curse o dimensionality ) the eiciency o warping distances will be urther weakened by the numerous pair-wise distance computation o high dimensional numerical vectors 3) the numerical similarities obtained rom distance aggregation may not be eective enough, especially when large variations (eg variations in shiting, scaling, directions and rotations) exist along the spatio-temporal dimensions Instead o measuring numerical similarities on such complex spatio-temporal data, a logic-oriented approach has recently been proposed in [9] to perorm discrete eature extraction on human motion capture data as a means to improve the eiciency o content-based retrieval on motion capture data In this technique, a number o categorical geometric eatures are extracted rom snapshots o the numerical motion capture data and sequences o extracted categorical eatures are used to describe the change o logical states over time An example is shown in Figure The eiciency is achieved through binary operations on high dimensional categorical time series The advantages o extracting discrete eatures rom complex spatio-temporal data include: ) variations within spatiotemporal patterns are partially handled through discrete transormation; ) similarities obtained rom the categorical states are intuitive, and thereore, provide operational interaces or users to speciy the subspace within which overall similarities are measured; 3) the categorical ormat o the data reduces the storage and computational cost o complex spatio-temporal data As such, the model o high dimensional categorical time series provides a signiicant probe on eicient and eective content-based retrieval o complex spatio-temporal patterns Most studies on data management o time series have been ocused on time series o numerical vectors [], [], [5], [], [8] These techniques cannot be applied directly to categorical time series The studies on managing categorical time series are however still limited to one dimensional data such as gene sequences [], [3] To overcome this limitation, our study

2 3 4 d d Fig Three snapshots o d dimensional categorical vectors in a categorical time series A local geometric eature (dimension d) o distance between eet is marked in red here will ocus on high dimensional categorical time series In this paper, we propose new distance measures and query types or high dimensional categorical time series, which are used in our PIPA system (a database engine or interactive media, or supporting eective matches o multimedia objects) Eicient query processing techniques are developed to handle the proposed queries Our contributions can be summarized as ollow: We propose the sketch query, which allows users to retrieve complex spatio-temporal patterns by speciying partial dimensions on some key rames Incremental sketch matching technique is proposed to speed up the sketch query processing We propose to segment high dimensional categorical time series using convexes, and measure the distance o two categorical time series in block-wise Eicient early abandoning technique is proposed to accelerate the subsequence matching on clip queries Extensive experiments on real-lie data were conducted The results show that our solutions are not only eicient, but also yielding acceptable accuracy or content-based retrieval o spatio-temporal data The rest o the paper is organized as ollows Section introduces some related work Section 3 gives the problem deinition Section 4 and Section 5 introduce the distance deinitions, queries, and related query processing techniques o sketch query and clip query respectively The experimental studies are shown in Section, and conclusions are drawn in Section 7 II RELATED WORK A Distances between Time Series The distance between any two time series is essentially computed rom the aggregation o pair-wise dierence o data items within them Traditionally, the Euclidean distance is used to measure the distances between time series o the same length Warping distances such as DTW [4] and EDR [4] have been proposed to measure distances o time series with arbitrary lengths The optimal alignments o data items between two time sequences are obtained by repeating some data items so that the lengths o two sequences can be the same The distance is calculated by inding the best warping path in the distance matrix using dynamic programming, which has a complexity o O(mn) (m and n are the lengths o time series) Lower bounds o warping distances [], [8] have been proposed to prune real warping distance computation Although they can be extended to high dimensional time series [5], the achieved lower bounds will be quite loose This is because the distance between a high dimensional data item and a high dimensional bounding item (with an extension in each dimension [5]) will be much less than the real distance o two high dimensional data items without the bound A more common way to match time series is subsequence matching, where a short time series is compared to a long time series Subsequence matching attempts to locate matching subsequences within a long time series to a short time series [5], and it is very useul in content-based retrieval o spatiotemporal patterns It is however more complex than ull sequence matching since the positions and lengths o the matching subsequences need to be detected Some eicient subsequence matching algorithms [5], [] have been proposed recently They detect matching subsequences in a dynamic programming manner based on warping distance such as DTW [] and SpADe [5] B High Dimensional Time Series Content-based retrieval based on high dimensional numerical time series is widely used in multimedia applications [], [], [3], [7] One typical example is the retrieval o motion capture data rom a large motion database, which is very useul in computer animation Many approaches or matching high dimensional numerical time series are based on numerical similarities, in which the high dimensional time series data are typically decomposed into a number o trajectories with low dimensionality (-dimensional to 3-dimensional) [7], [8] Warping distances such as DTW distance are preerred to match those low dimensional trajectories to handle time shiting and scaling o time series However, due to the variations o shiting, scaling, direction and rotation along the amplitude and temporal dimensions o sequences, the numerical similarities obtained rom the distance aggregation may be ar rom the logical similarities [9], [8] based on human views To address this problem, eective geometric eatures have been subsequently proposed to map snapshots o numerical sequences into discrete states [9], [9] It has been shown that using logic-oriented categorical eatures not only signiicantly improves the eiciency o se-

3 quences matching but also achieves high accuracy in contentbased retrieval o human-motion data Studies on managing categorical time series have been ocused on one dimensional gene sequences [], [3] To the best o our knowledge, there has been little study on query processing o high dimensional categorical time series C Building Categorical Time Series Discretization o time series has been studied in [], [] However, they are ocused on discretizing time series o low dimensional data In [9], one high dimensional categorical eature vector is extracted rom one rame o human motion data by thresholding the logic-oriented geometric relations Each logic-oriented geometric relation is actually a unction o spatial relationship o some local dimensions, and it is robust to variations within local dimensions; or example, the angle between humerus and radius, the distance between two eet Discretization is applied to map a geometric relation expression to a small number o categorical states This is very intuitive as people are sensitive to state dierence instead o the quantitative dierence in logical similarities Discretization partially handles variations within numerical time series as small variation does not result in a change in the discretized states Hysteresis-like thresholding can be applied to avoid random luctuations around the thresholds [9] III PRELIMINARIES A Problem Deinition A categorical time series instance S is represented as S[ : T ] S[t], the snapshot o S at time point t, isad-dimensional categorical vector For those categorical vectors extracted rom complex spatio-temporal data, d is typically very large, eg, more than 4 In this paper, we ocus on inding the k nearest neighbors (ie knn queries) o a given query based on some distance measure as it is a very common query in the domains we are studying The general knn query can be described as ollow: given a small integer k, a distance unction d(, ), a query example Q and a high dimensional categorical time series database DB, the knn query retrieves a set M consists o k time series R DB such that or any time series R M and any time series R DB M, d(r,q) d(r,q) Dierent distance measures o time series result in dierent query processing solutions or the knn queries B Distances Between Categorical Vectors A commonly used distance measure o categorical vectors is Hamming distance The Hamming distance o two categorical vectors and, denoted as H(, ), is deined as the number o non-matching dimensions o two eature vectors and are said to be matched in dimension i i i = i I H(, ) =, then and are said to be perectly matched When using Hamming distance to measure the similarities o categorical vectors, users may only be interested in a subset o eatures by speciying some dimensions that must be matched and some dimensions which can be ignored Thereore, instead o matching eature vectors in all dimensions, a more operable way is to match them on a user speciied subspace We denote the ull dimensional space as Ω={,,d}, and the subspace as π Ω The Hamming distance on the subspace π can be deined as: Deinition (π-hamming distance): Given two eature vectors and,theπ-hamming distance o and, denoted as H π (, ) is deined as the number o unmatched dimensions o two eature vectors in subspace π The π-hamming distance ignores all the dimensions in Ω π Hamming distance is a speciic case o π-hamming distance when π =Ω Based on the basic distance measures o categorical vectors, a binary decision on the matches o two categorical eature vectors and can be made We say and are απ-match, denoted as = απ, i H π (, ) α Besides speciying the subspace, users may also speciy some possible states on some chosen dimensions This can be done by giving a σ clause A σ selection clause consists o a number o speciied eatures σ = {i,,i m } Ω For each eature j σ, a number o matched states σ j = {s j,,s jp } are speciied (typically p =) Given a selection clause σ, a eature vector is a σ-match, denoted as σ, i i σ, i σ i Categorical time series are extracted rom semantic eatures Thereore, it is quite convenient or the users to describe a σ clause by speciying some constraints on the categorical eature vectors The Hamming distance, subspace π and σ clause orm three basic elements o queries on categorical time series IV SKETCH QUERIES A Deinition A very common query on high dimensional time series is the sketch query, in which users describe a number o snapshots (by speciying some π and σ clauses) and the system returns those matching subsequences which it the userspeciied snapshots well For example, in animation design, users may describe their queries by giving some sketches o the starting pose, the ending pose and some intermediate poses o an action These sketches can be some σ clauses or some examples o snapshots, which are typically not adjacent to each other Formally, a sketch ϕ = {σ,, απ} describes the snapshots users want to retrieve It consists o a σ clause, an example o categorical vector and a απ-match iltering threshold on A eature vector is a ϕ-match, denoted as ϕ, i σ and = απ The distance o toasketchϕ is deined as: { d( Hπ (,ϕ)=,) i ϕ; + Otherwise It is obvious that d(,ϕ) α i is a ϕ-match Note that σ, α, and π are optional in a sketch ϕ When α is not present, it is assumed that no απ iltering is conducted on

4 ϕ-match, ie, α =+ ; I and π are not present, the sketch ϕ is simply a σ clause, ie, ϕ σ In this case, d(,ϕ)=i ϕ When σ is not present, the sketch ϕ is simply a απ iltering The ϕ-distance is a π-hamming distance o to i = απ Deinition (γ-distance): Given a time series R and m sketches γ = {ϕ,,ϕ m }, the γ-distance o time series R is deined as, d(r, γ) = min(σ m i= d(r[t i],ϕ i )), where <t i+ t i, and is the constraint on the temporal gaps between two consecutive qualiied snapshots In this deinition, R[t i ] (i =,,m) are snapshots o eature vectors within R They are called qualiied snapshots i the constraints <t i+ t i (or i =,,m ) are satisied The γ-distance o R is the minimum o the sum o ϕ-distances o m qualiied snapshots among all combinations o qualiied snapshots within R Based on the γ-distance, we deine the knn sketch query on categorical time series as ollows: Deinition 3 (knn sketch query): Given a categorical time series dataset DB, a sketch query γ and a small integer k, the knn sketch query retrieves a set M consisting o k time series R DB such that or any time series R M and any time series R DB M, d(r,γ) d(r,γ) B ϕ-match Query Processing The basis o a sketch query is the single sketch match, ie, ϕ-match A simple ϕ-match query retrieves all eature vectors which are ϕ-match in the time series database The sketch ϕ is used to describe the eature vectors o interest to the users The ϕ-match query can be simply processed by perorming σ-match and π-hamming distance computation on each eature vector within the database A naïve way to achieve this is to linearly scan all eature vectors However, it is obviously expensive since there are typically millions o categorical vectors in the database There are two ways o organizing categorical eature vectors in the time series database One way is to ignore the orders o the eature vectors within sequences so that the same vectors in various positions o various time series can be summarized as one vector A pointer list recording the positions o a eature vector is required so that the positions o the eature vectors can be traced back to the time series Another way to organize the vectors is to store them based on the orders o vectors and sequences containing the vectors, so that ast consecutive retrieval o eature vectors or a time series can be done The beneit o the ormer approach is that it saves both the memory and retrieval cost as the duplications o the eature vectors are removed within the whole database However, since the vectors in this approach are stored in a random order, the retrieval o clips o time series is not convenient Since queries on time series require consecutive scan on the eature vectors within time series, we choose the latter strategy in our solution There are many similar eature vectors, which may only dier by a ew bits to each other, especially or those neighboring eature vectors A good way to improve the eiciency o ϕ-match processing is to develop some lower bounding techniques on high dimensional categorical vectors, so that a tight lower bound o ϕ-distance o similar consecutive vectors can be eiciently calculated We propose to use a convex to bound a number o similar categorical vectors The convex provides a lower bound o ϕ-distances o all vectors within it Thereore, eicient pruning can be achieved by using the convex A d-dimensional categorical eature vector is actually a word o d bits in memory, where d d as one eature may consume more than one bit Figure shows an example o the storage structure o a eature vector, where one eature i is mapped to bits si to ei d Fig 3 Binary structure o a eature vector Given a number o eature vectors,, n, the convex o,, n includes two d -dimensional binary vectors, c = (c,c ) c is deined as c = i,j i j, which appears one at positions where,, n share common binary values c is deined as c =, which is the irst eature vector recording the common binary values shared by,, n in the common positions represented by c Based on the convex, the lower bound o π-hamming distance o another eature vector to c is LB(,c) =D π (c ( c )) D π ( ) = ei i π j=s i j, which is a unction checking the number o non-zero values o d eatures within the subspace π It is guaranteed that LB(,c) H π (, i ), i =,,n Figure 3 shows an example o a convex obtained rom three consecutive eature vectors, and 3 Here, π is the subspace over dimensions o d, d, d 3, d 5 and d 7 The lower bound LB(,c) is calculated based on the number o dimensions (in π) having ones (in red o the last word) In this example, it is 3 A convex c is a σ-match i c j (cj σj ) is true, j σ That means those common binary values in convex c must satisy the σ clause It is guaranteed that all eature vectors are not σ-matches i their convex c is not a σ-match However, a eature vector may not be a σ-match even though its convex is a σ-match Thereore, the σ-match o a convex actually provides a lower bound on the σ-matches o the vectors it contains Those bits with zeros in c are called lexible bits as the s s e s e d d

5 π d d d 3 d 4 d 5 d d 7 3 c c c ( c ) Fig 3 An example o a convex convex allows variance in these bits However, the eectiveness o convex will be aected by the number o lexible bits in c, denoted as ρ O course, the smaller the number o lexible bits, the tighter the convex, and the more accurate the convex is in lower bounding ϕ-distances However, ρ should not be too small because small ρ limits the number o the eature vectors that could be bounded by a convex We build the convexes by dividing categorical time series into a number o segments Each segment contains a number o consecutive eature vectors, which are packaged into a convex As shown in Algorithm, the convexes are built incrementally The irst eature vector is chosen as the c o a convex The convex expands until the number o lexible bits surpasses the threshold ρ A number o convexes will be learned rom a sequence by iteratively implementing the above procedure Algorithm Convex building : Input: R, a categorical time series instance, R[ : T ] : Input: ρ, the maximial number o lexible bits 3: Output: C, the convexes o eatures vectors in R 4: i = 5: while i T do : c = R[i] //the irst vector within a convex 7: c = //a vector o ones in all dimensions 8: i ++ 9: while i T do : c = c c R[i] //enclose one vector : i NumberOZeros(c ) ρ then : aconvexc = c 3: else 4: aconvexc = c 5: Cadd(aconvex) //generate a convex : break 7: i ++ C Sketch Query Processing In content-based retrieval o complex spatio-temporal patterns, a single snapshot is not enough to describe a pattern perectly Thereore, a number o sketches are used in a γ sketch query, to retrieve subsequences consisting o ϕ- match snapshots to the sketches in γ A sketch query γ can be processed by executing multiple ϕ-match processing and joining the results o multiple ϕ-matches based on the temporal gap constraints (speciied in Deinition ) knn queries can be urther conducted based on the γ-distances o matching subsequences generated rom the spatial join We note that a multi-way join requires linear scan o the eature vectors (or convexes) multiple times However, due to the spatial constraints over consecutive ϕ-match snapshots, the number o potential matching subsequences to γ may be quite small ater the irst ϕ-match or the irst ew joins o ϕ-matches Thereore, subsequent ϕ-matches and spatial joins may not be necessary since the potential matching subsequences are already reduced to a very small number It will be more eicient i the multi-way spatial join can be processed incrementally, so that some unnecessary linear scans and ϕ-match joins can be pruned Figure 4 shows the incremental approach o sketch query processing Given a sketch query γ = {ϕ,,ϕ 5 }, ϕ is irst processed by a linear scan o convexes and vectors in the database A number o snapshots which are ϕ -match are retrieved ater the linear scan Next, or ϕ clause, we may not need to retrieve all ϕ -match snapshots by linear scan over all the convexes and vectors i the number o retrieved ϕ -match snapshots is limited The potential matching subsequences can be detected by checking the successive qualiied snapshots o those ϕ -matches Thereore, or each matching snapshot ϕ ( R), we try to detect successive qualiied snapshots in R by only considering the temporal region rom t to t + By doing this sketch by sketch, the matching subsequence is ound cumulatively The number o potential matching subsequences drops signiicantly in the incremental process As a result, most regions are not required to be scanned during the subsequent ϕ-match processing Those matching snapshots within the untouched regions are actually pruned rom the computation The incremental sketch query processing preorms γ distance computation and ϕ-match on the ly, providing an eective pruning approach or the ϕ- match processing ϕ ϕ ϕ ϕ ϕ a matching subsequence * * * * * * * * * * * * * * * * * * * * * * cumulated * ϕ matches * ϕ Fig 4 pruned matches Incremental sketch matching Along with the incremental sketch query processing, the γ-distance o a time series is also calculated incrementally t

6 The best itting combination o matching snapshots can be detected by attaching a cumulative distance to each detected ϕ- match The cumulative distance o a matching snapshot can be accumulated rom the cumulative distance o its predecessor, ie, cd( i )=cd( i predecessor)+d( i,ϕ j ) The predecessor k o a ϕ-match snapshot i satisies ollowing conditions: ), i ϕ j, k ϕ j ), k t < i t and i t k t + k is said a potential predecessor o i i it satisies conditions and 3), k has the lowest cumulative distance among all potential predecessors o i For the example shown in Figure 5, all ϕ distances o matching snapshots are given in the brackets The γ-distance o subsequence rom to is 7 t ϕ5 * () ϕ4 * 5 () ϕ3 * 4 (3) ϕ () * * 3 () ϕ () * Fig 5 Computing γ-distance V CLIP QUERIES A Distances Between Categorical Time Series The distance between time series is calculated rom the aggregation o pair-wise dierences o data items within two time series, regardless o the actual distance used For categorical time series, the pair-wise distance is actually the π-hamming distance o two categorical vectors However, the neighboring categorical vectors within one time series are typically identical or quite similar to each other as they are extracted rom consecutive rames o spatio-temporal data, especially when the spatio-temporal data is in high sampling rate or low changing rate The naive way o pair-wise distance aggregation on those near-duplicative vectors may accrue some disadvantages: ), the distance o time series will be mainly determined by those sub-regions where intensive duplications exist, while these regions do not provide too much semantics inormation since the changing rate o time series is slow over these sub-regions ), the large number o the resultant distance aggregation (pair-wise) is not eicient; 3), the duplicated aggregation o π-hamming distance caused by duplications o eature vectors does not make too much sense when local time scaling is present as in many cases o complex spatiotemporal data Many studies in content-based retrieval o videos avoid the near-duplication problem by segmenting video sequences and extracting key rames rom the video sequences The similar idea can be applied to high dimensional categorical time series by applying the convex technique Following the Algorithm, a categorical time series can be partitioned into a number o segments/blocks The similarities o vectors within one block are controlled by the threshold o lexible bits ρ Upon measuring distances o categorical time series, instead o using pair-wise distance aggregation, we propose to use block-wise distance aggregation Given two convexes c and c, the block-wise distance between c and c is deined as d(c,c ) = D π ((c c c c ) c c c c ) The block-wise distance o two time series is then deined as the aggregation o block-wise distances o convexes within the two time series B Clip Queries We next look at a clip query which retrieves the similar categorical time series instances (or subsequences o time series) o a given query instance o time series Depending on the matching sequences, clip queries can be classiied into ull clip queries and subsequence clip queries In ull clip queries, the query clip is compared to the whole time series in the database while in subsequence clip queries, the query clip may be matched to any subsequence o the time series within database It is obvious that subsequence clip queries are more complex than ull clip queries Full clip queries have been well studied Subsequence clip queries on the other hand are more useul since users may only provide a query example o a short clip, and the matching clips are most likely to be subsequences within the long time series instances Consequently, we are more interested in subsequence clip queries Given a query clip Q and a time series R, we deine the τ-distance between R and Q as: Deinition 4 (τ-distance): d τ (R, Q) = min d(r,q), where R is any subsequence o R, and d(r,q) is a ull clip block-wise distance between R and Q Note that distance unction d(, ) can be any distance measure o time series However, we propose to use DTW distance as it handles temporal shiting and scaling between two time series Moreover, it is widely used as a distance measure o time series Based on τ-distance, the knn clip query is deined as: Deinition 5 (knn clip query): Given a categorical time series dataset DB, a clip query Q and a small integer k, the knn clip query retrieves a set M consisting o k time series R DB such that or any time series R M and any time series R DB M, d τ (R,Q) d τ (R,Q) C Clip Query Processing Note that a time series instance R within DB may be much longer than the query clip Q The number o subsequences o R will be extremely large as subsequences o various lengths must be allowed in order to handle temporal shiting and scaling between the matching subsequences and Q Due to the properties o high dimensionality and categorization, indexing techniques on DTW distance [], [8] cannot be applied to high dimensional categorical time series It will be very expensive i all subsequences are extracted rom R, and linearly compared to Q based on the ull clip block-wise distances A better solution or matching subsequences is to use the dynamic programming approach or linear scan [5], [] For

7 the query Q with m blocks (convexes) and time series R with n blocks, a matrix o n m is used to calculate the τ-distance between R and Q The τ-distance d τ (R, Q) is determined by the best matching subsequence in R It is calculated column by column in the distance matrix Within each column, we accumulate distances rom bottom to up, based on the block-wise distances o two convexes According to DTW distance, the cumulative distance at position [i, j] is aggregated as d[i, j] = min(d[i, j ],d[i,j],d[i,j ]) + d(rc i,qc j ), with d[i, ] = d(rc i,qc ) Thereore, d[i,j] is calculated beore d[i,j] i i <i and d[i, j ] is calculated beore d[i, j ] i j <j Note that the dominant computational component o τ- distance is rom the numerous block-wise distance computation and the accumulating o distances in the distance matrix To improve the eiciency o τ-distance computation, we propose an early abandoning technique to avoid some distance computation within the distance matrix Given a pruning threshold ε (achieved in the early stage o knn query and reined in the later stage), the bound o a column i, denoted as h i, is deined as: i i =; max h i = j<hi, d[i,j] ε j + Else i d[i, h i ] >ε; max j<m, d[i,j] ε j + Else i d[i, m] ε; min j>hi,d[i,j]>ε j Otherwise For each column i, the distances are irst accumulated rom d[i, ] to d[i, h i ]Id[i, h i ] > ε, the distances within column i will not be accumulated urther Otherwise, the accumulation continues until d[i, j] >ε The correctness o early abandoning by the bound can be proven by induction Assume that all cumulative distances d[i,j] >ε, where j h i Given a cumulative distance d[i, l] > ε, where l h i, d[i, l +] must be larger than ε as d[i,l] and d[i,l+] are also larger than ε Incrementally, d[i, m] can be proven to be larger than ε Thereore, all cells ater row l in the column i can be pruned out or distance aggregation As shown in Figure, the bound helps to prune the real blockwise distance computation in the regions where there are no matching subsequences as the distances will quickly surpass ε in these regions m Fig Matching subsequence t H (t, j) π Illustration o early abandoning in subsequence clip queries For the given example shown in Figure, suppose the 5 4 pruning threshold o τ-distance is ε = 7 The cells where the bounds o columns locate are colored with blue In the the current column o time t, three block-wise distances (4, 5, ) are calculated The distance stops rom aggregation at the third row since the row position reaches h t =3and d[t, 3] = >ε The bound o current column is updated as h t =where a cumulative distance o d[t, ] = 8 exists VI EXPERIMENTAL STUDY A Datasets We generate the high dimensional categorical time series rom the real-lie dataset o CMU motion capture database [] Two thousands o human motion sequences o more than subjects are used in our experiments, with the length o sequences varied rom to,948 The original motion sequences are 3-dimensional trajectories o 3 joints o human bodies Thereore, the size o original dataset is 3G Based on the geometric eatures proposed in [9], 4 geometric relation eatures are extracted in our experiments Among them, there are 9 pair-wise distance eatures (eg, hand to hand distance), angle eatures (eg, the angle between emur and tibia) and 5 plane eatures (eg, whether let hand is beore the body) A binary vector o 5 dimensions is generated rom one snapshot o 4 dimensional geometric relation eatures by categorizing these eatures In our experiments, each eature is categorized into at most three discrete states As a result, a categorical time series dataset o 4M is obtained Thereore, the compression ratio o categorical time series to the original motion sequences is around : Due to its small size, once the whole transormed vector database is read into the memory, it could be kept in memory without urther rereerencing On the other hand, the original database has to be read in and the pages are paged out i there are no buer pages available Depending on the buering strategy, pages may have to be read in or each scan Our experimental platorm is a PC o Pentium4 3G CPU with G RAM B ϕ-match Query Processing To show the eectiveness o categorizing high dimensional time series, we retrieve some pairs o matching categorical snapshots by using π-hamming distance Figure 7 shows 3 examples o matching snapshot pairs which have zero π- Hamming distance We can see that these pairs o snapshots are quite similar Some o the matching pairs have certain variation on detailed poses However, the categorical vectors ignore small magnitude o variations, keeping those snapshots exactly matched One o the advantages o extracting discrete eatures rom complex spatio-temporal data is to achieve eiciency on matching spatio-temporal data We compare the eiciency o ϕ-match (π-match) on categorical vectors, with range queries (smallε such that similar selectivity is achieved to ϕ-match) on original motion sequences (both disk-based and in-memory) The average computational time is shown in Table I As expected, the results clearly show that ϕ-match on categorical vectors are much aster than range queries on original motion

8 Computational cost (ms) 8 4 φ-match without convexes φ-match with convexes απ-match threshold α Fig 8 Eiciency o ϕ-match queries o eature vectors contained by a convex Consequently, the perormance o convex scan will be dropped The results in Figure 9 shows the impact o ρ on the eiciency o ϕ-match query processing In our experiments, convexes achieve better pruning perormance when ρ =7 Fig 7 Pairs o matching snapshots sequences, due to the eicient binary operations and small size o categorical vectors Moveover, many results o the range queries on original motion sequences are ar rom logical similarities due to the dierent variations in human motion While, the results o ϕ-match are quite matched Original Original Categorical disk-based in-memory vectors t(sec) 3 TABLE I EFFICIENCY ON SNAPSHOT MATCHING To evaluate the perormance o convexes, we adjust the matching threshold α in ϕ-match query, and compare the eiciency o ϕ-match queries on categorical vectors with and without convexes in Figure 8 The number o lexible bits on building convexes is capped at ρ =8 The results show that ϕ-match o high dimensional categorical vectors with convexes is more eicient than that without convexes when the matching threshold α is small The perormance o ϕ- match with convexes drops when α is enlarged, as the convex is not eective enough or pruning large distances However, in practice, the matching threshold in απ-match is very small (eg, less than ) The eiciency o ϕ-match with convex supports will be better than direct ϕ-match on categorical vectors To urther study the perormance o convexes, we adjust lexible bit threshold ρ In practice, ρ should be limited to a small number compared to d (dimensions o binary vectors) so that the derived convex can be eectively used in pruning ϕ-distances However large number o convexes will be generated i ρ is set too small as it limits the number Computational cost (ms) 8 4 ρ = 4 ρ = 8 ρ = ρ = ρ = απ-match threshold α Fig 9 Impact o ρ on the eiciency o ϕ-match query processing with convexes C Sketch Query Processing We evaluate the eectiveness and eiciency o the sketch query using sketches rom one time series o a swing action in gol(time series 4 ) As shown in Figure, 5 key snapshots are extracted rom one query example Assuming a user is interested in all the swing actions o goling in the database, he may pose a sketch query consisting o these key snapshots (which may already exist or be obtained rom ϕ- match queries) We use the 8π-match iltering on each sketch in the γ query The results o nearest neighbors o the γ sketch query are shown in Table II The irst 9 nearest neighbors o a γ sketch query contain all swing actions o gol (time series 4-4, 4 is excluded as it orms the sketch query) in the database Moreover, the γ distances o these matching actions are no more than 7, which are much more less than those o the other actions (more than Videos o all time series used in our experiments are available at

9 5) Thereore, the sketch query in our example eectively distinguishes matching sequences rom unmatched sequences Note that some sketch queries may not be good enough to distinguish sequences, however, they can be modiied by reining the matching threshold α, speciying π and σ, or providing more sketches Rank Results Distances Rank Results Distances TABLE II RESULTS OF A KNN SKETCH QUERY We compare the eiciency o sketch query processing o naive spatial join with that o incremental spatial join The απ-match threshold α in the γ sketch query is adjusted rom to From the results shown in Figure, we can see that two sketch query processing approaches are quite eicient (no more than seconds) Comparatively, the incremental spatial join is more eicient than the naive spatial join, especially when α is small The smaller α ilters more snapshots in ϕ- match As a result, many regions o the incremental spatial join are pruned in the last several linear scans o ϕ-match This is why the computational cost o the incremental spatial join is around /5 o that o the naive spatial join, as only one pass o linear scan is required or most o unmatched sequences The pruning eect o the incremental spatial join will be more prominent i the number o sketches in a γ sketch query is larger Computational cost (ms) Naive spatial join Increamental spatial join 4 8 Snapshot matching threshold α Fig Eiciency o sketch queries D Clip Query Processing We now compare the accuracy and eiciency o pairwise distance and block-wise distance on categorical time series over a number o clip queries The accuracy o one nearest neighbor classiication has been widely used as an important measure o eectiveness o time series distance measures in many studies [4], [5], [3] Thereore, we use it as the benchmark or accuracy comparison 7 time series o moderate lengths (with 3 snapshots) are randomly chosen as query instances We do not use too many query examples as the class labels o the time series are not given explicitly, and many actions within long time series need to be checked manually The average results o one nearest neighbor queries are shown in table III We can see that block-wise distance is much more eicient than the pair-wise distance due to the summarization o categorical vectors The accuracy o block-wise distance however can also be better than pairwise distance when the lexible bit threshold ρ is small For example, compared to pair-wise distance, an eiciency o 3 times aster is achieved by block-wise distance when ρ =4, while the accuracy is still better than that o pair-wise distance We should note that some error rates are generated because there are actually no matching sequences o the query clips ρ Pair-wise 4 8 Error rate 8% 53% 53% 7% 3% 3% Time (sec) TABLE III COMPARISON OF ACCURACY AND EFFICIENCY ON ONE NEAREST NEIGHBOR CLIP QUERIES To improve the eiciency subsequence matching, we apply the proposed early abandoning technique on τ-distance computation, and test the eiciency o clip queries in Figure The query clip is an action o kicking ball in soccer We perorm several knn queries by adjusting the parameter k Figure shows that the less the k, the more eiciency can be achieved by the early abandoning technique because more pruning power is achieved when the pruning threshold ε is smaller We also conduct a 4NN query o this query instance on the original motion sequences, which consumes more than, seconds Comparatively, the categorical time series achieves more than, times aster Moreover, the results o 4NN query on original sequences are ar rom accuracy due to spatial variations Computational cost (sec) Fig 5 5 Without early abandoning With early abandoning 4 8 Parameter k Example o the eectiveness o early abandoning in clip query VII CONCLUSIONS In this paper, we have addressed the problem o query processing on high dimensional categorical time series, and

10 Fig Snapshot examples o a γ sketch query (a swing in gol) to the best o our knowledge, this paper is the irst to address the issue We show that the transormation o complex spatio-temporal data to high dimensional categorical time series reduces the size o the database signiicantly Eective distance measures on categorical time series are developed to perorm content-based retrieval o complex spatio-temporal data eiciently and eectively We deine the basic distance measures o high dimensional categorical time series, γ-distances and τ-distances, and subsequently propose the sketch queries and clip queries o the categorical time series We also propose eicient query processing techniques on these queries Experiments on the proposed distance measures and queries were conducted on a real-lie spatio-temporal dataset, the CMU motion capture database [] The results demonstrate the eiciency and eectiveness o our proposed distance measures and queries The proposed techniques signiicantly improve the eiciency o knn queries on original motion sequences up to, times while guaranteeing the accuracy o the search In conclusion, query processing based on categorical time series provides an eicient and eective means or content-based retrieval o complex spatio-temporal sequences ACKNOWLEDGMENT This work was supported in part by the National Grant Fundamental Research 973 Program o China (Grant No CB33), the Key Program o National Natural Science Foundation o China (Grant No 533) [9] M Müller, T Röder, and M Clausen, Eicient content-based retrieval o motion capture data ACM Trans Graph, vol 4, no 3, pp 77 85, 5 [] R Agrawal, C Faloutsos, and A N Swami, Eicient similarity search in sequence databases in FODO, 993, pp 9 84 [] Y Cai and R T Ng, Indexing spatio-temporal trajectories with chebyshev polynomials in SIGMOD Conerence, 4, pp 599 [] S Burkhardt, A Crauser, P Ferragina, H-P Lenho, E Rivals, and M Vingron, Q-gram based database searching using a suix array (quasar) in RECOMB, 999, pp [3] C Meek, J M Patel, and S Kasetty, Oasis: An online and accurate technique or local-alignment searches on biological sequences in VLDB, 3, pp 9 9 [4] D J Berndt and J Cliord, Using dynamic time warping to ind patterns in time series in KDD Workshop, 994, pp [5] T M Rath and R Manmatha, Lower-bounding o dynamic time warping distances or multivariate time series in University o Massachusetts Amherst Technical Report MM-4, [] Y Sakurai, C Faloutsos, and M Yamamuro, Stream monitoring under the time warping distance in ICDE, 7, pp 4 55 [7] Y Sheikh, M Sheikh, and M Shah, Exploring the space o a human action in ICCV, 5, pp [8] A Veeraraghavan and A K R Chowdhury, The unction space o an activity in CVPR (),, pp [9] M Müller and T Röder, Motion templates or automatic classiication and retrieval o motion capture data, in Proceedings o the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, [] E J Keogh, J Lin, and A W-C Fu, HOT SAX: Eiciently inding the most unusual time series subsequence in ICDM, 5, pp 33 [] V Megalooikonomou, Q Wang, G Li, and C Faloutsos, A multiresolution symbolic representation o time series in ICDE, 5, pp 8 79 [] CMU Graphics Lab Motion Capture Database, [3] E J Keogh and S Kasetty, On the need or time series data mining benchmarks: a survey and empirical demonstration in KDD,, pp REFERENCES [] S-C S Cheung and A Zakhor, Eicient video similarity measurement with video signature IEEE Trans Circuits Syst Video Techn, vol 3, no, pp 59 74, 3 [] L Kovar and M Gleicher, Automated extraction and parameterization o motions in large data sets ACM Trans Graph, vol 3, no 3, pp , 4 [3] H T Shen, B C Ooi, and X Zhou, Towards eective indexing or very large video sequence database in SIGMOD Conerence, 5, pp [4] L Chen, M T Özsu, and V Oria, Robust and ast similarity search or moving object trajectories in SIGMOD Conerence, 5, pp 49 5 [5] Y Chen, M A Nascimento, B C Ooi, and A K H Tung, Spade: On shape-based pattern detection in streaming time series in ICDE, 7, pp [] E J Keogh, Exact indexing o dynamic time warping in VLDB,, pp 4 47 [7] E J Keogh, T Palpanas, V B Zordan, D Gunopulos, and M Cardle, Indexing large human-motion databases in VLDB, 4, pp [8] Y Zhu and D Shasha, Warping indexes with envelope transorms or query by humming in SIGMOD Conerence, 3, pp 8 9