USC Real-time Pattern Isolation and Recognition Over Immersive Sensor Data Streams

Transcription

1 Real-time Pattern Isolation and Recognition Over Immersive Sensor Data Streams Cyrus Shahabi and Donghui Yan Integrated Media Systems Center and Computer Science Department, University of Southern California 1

2 Outline Motivation Target Application Related Work Weighted-sum SVD The Ridge-Climbing Heuristic Performance Evaluation Conclusion & Future Work 2

3 Motivation With traditional DBMS Dataset is persistent It operate on the entire dataset In many recent applications Data (continuous data streams, CDS) constantly change or continuously arriving (e.g., web clickstreams, IP packets) Challenges: Queries must be answered based on limited amount of information Queries need to be answered in real time Query results may need to be aggregated from different sources (e.g., the yahoo web page with the maximum number of hits) 3

4 Target Application: Immersive Interaction (Beyond Keyboard & Mouse) Recognition System command Play Run Stop Zoom-In Zoom-Out Immersive environment DB of Labeled Patterns Immersidata 1) multidimensional, 2) spatiotemporal, 3)continuous data stream, 4) large, 5)noisy Immersive Environments allow a user to become immersed within an augmented or virtual reality environment in order to interact with people, objects, places, and databases [ACM-ITP 02]. Immersidata Data acquired from user s interaction with the immersive environment [CIDR 03]. 4

5 Immersive Sensor Data Streams <S i, x, y, z, t, v> 5

6 Focus: American Sign Language (ASL) Recognition Extra challenges (to CDS) A hand sign is composed by a sequence of data samples A sequence for one sign has no fixed length The data rates of sensors are fixed Data need to be tightly integrated and hence are high dimensional An example statement in American Sign Language (ASL) I like yellow shoes Two problems (chicken & egg-problem) with interdependent solutions should be addressed Isolate signs Recognize the isolated sign 6

7 Isolation/Recognition Processes 1. User makes ASL signs w/ a glove Isolation module: -Ridge-Climbing Heuristic Acquisition Module Immersidata Database 2. Sensor values sampled over time 4. Isolate the streams I : 0.9 Like: 0.2 I : 0.5 Like: Accumulated similarity values Recognition module: -Weighted-Sum SVD 7

8 Related Work Research efforts on querying over CDS DBMS support, e.g., Stream [Babu&Widom 2001], Fjords [Madden&Frankin, 2002] Query processing and data mining issues [Hulten 2001; Guha 2000] Pattern recognition in one dimensional time series [Gao&Wang 2002] No studies on pattern recognition over aggregated and high dimensional CDS Distance measures for the similarity queries Euclidean distance :dimensionality curse Discrete Wavelet transform : our dataset are no correlated on Discrete Fourier transform sensor dimension at any given time 8

9 SVD Background The idea of SVD is based on the following theorem of linear algebra: m n If matrix X R, then there exist columnorthonormal matrices U and V such that X = U A V m r r n r r where U R and V R, and A R is a diagonal matrix A = diag( a1, a2,..., a p ) such that a a a p T 9

10 Weighted-Sum SVD Each data sequence could be represented as a matrix, where the columns (r) are the sensors and hence their # is fixed The similarity metric of two data sequences is defined on the square matrices To eliminate the effect that the number of rows (i.e., the time dimension) in the two matrices are different (i.e., multiply the matrix by its transpose matrix) 10

11 Weighted-Sum SVD Problem: Obtain the similarity of input sequence and the pattern square q 11 q 1r q r1 q rr SVD decompose e 1, e 2,, e r cw 1 cw 2 cw r cw 1 +cw cw r =1 c 1 c 2 c r weight dw 1 dw 2 e 1 e 2 e r dw r dw 1 +dw dw r =1 square p 11 p r1 p 1r p rr SVD decompose f 1, f 2,, f r d 1 d 2 d r f 1 f 2 f r 11

12 Weighted-Sum SVD Problem: Obtain the similarity of input sequence and the pattern e 1 f cw 1 dw 1 e e 1, e 2,, e cw 1 f2 2 2 r f 1, f 2,, f dw 2 r cw r e r dw r f r θ θ 1 2 = = r i= 1 r i= 1 cw i dw ( e f ) i i ( e f ) i i i The similarity of input sequence and the pattern =min(t 1, T 2 ) 12

13 The Ridge-Climbing Heuristic Designed based on Mutual information [Resa, 1961] Observation: if the sequence B should be detected as K, the accumulated similarity value between B and K will be monotonically increasing up to some time instance and then starts decreasing Accumulated similarity value 13

14 The Ridge-Climbing Heuristic Procedure: Compute the accumulated similarity values (ASVs) between the input sequence and all vocabulary sequences Keep track of all ASVs For each vocabulary sequence, check whether the ASV is monotonically increasing, and whether a maximum is reached Yes: put this vocabulary into the candidates pool Choose the vocabulary from the candidates pool with biggest maximal value Isolate the recognized stream 14

15 The Ridge-Climbing Heuristic Assume the database only has three vocabulary sequence, like, yellow, and I. Input sequence like Maximum is reached! Isolate! Reset the ASVs like yellow I ASVs ASVs ASVs time time time 15

16 Experimental Setup Training Collect data from ten different people Each perform ten different color signs Obtain sign representatives Testing User performed ten color sign combinations, e.g., Green-Orange-Yellow 16

17 Compare similarity metric Yellow Orange Black Red Green Yellow Orange Black Red Green

18 Sign Combination 18

19 Sign Combination Yellow-Black Stream Black-Blue Stream 19

20 Conclusion & Future Work Conclusion Propose weighted-sum SVD to measure the similarity of immersive sequences Apply in immersive interaction application Effectively handle high-dimensional data Propose the Ridge-Climbing Heuristic for tackling the problem of sequence separation over CDS Experiments demonstrated this method is an appropriate mathematical abstraction Future work Explore techniques for computing SVD incrementally Evaluate the effectiveness of different similarity metrics 20

21 Thank You! 21

22 References [Babu&Widom 2001]S. Babu and J. Widom. Continuous Queries Over Data Streams. ACM SIGMOD Record, 30(3), Sep [Guha 2000 ] S. Guha, N. Mishra, R. Motwani, and L. O Callaghan. Clustering data streams. In Proc. of the 2000 Annual Symp. On Foundations of Computer Science. Nov [Hulten 2001]G. Hulten, L. Spencer, and P. Domingos. Mining time-changing data streams. In Proc. Of the 2001 ACM SIGKDD Int. Conf. On Knowledge Discovery and Data Mining, Aug [Madden& Frankin 2002] S. Madden and M. J. Frankin. Fjording the Stream: An Architecture for Queries over Streaming Sensor Data. ICDE Conference, San Jose, Feb [Gao&Wang 2002] L. Gao, X. Sean Wang. Continually Evaluating Similarity-Based Pattern Queries on a Streaming Time Series. In Proc. Of the ACM SIGMOD Conf. On Management of Data, 2002 [Reza 1961] F. M. Reza. An Introduction to Information Theory, McGraw-Hill,