DATA STREAMS: MODELS AND ALGORITHMS

Size: px

Start display at page:

Download "DATA STREAMS: MODELS AND ALGORITHMS"

Karin Robertson
6 years ago
Views:

1 DATA STREAMS: MODELS AND ALGORITHMS

3 DATA STREAMS: MODELS AND ALGORITHMS Edited by CHARU C. AGGARWAL IBM T. J. Watson Research Center, Yorktown Heights, NY Kluwer Academic Publishers Boston/Dordrecht/London

4 Contents List of Figures List of Tables Preface xi xv xvii 1 An Introduction to Data Streams 1 Charu C. Aggarwal 1. Introduction 1 2. Stream Mining Algorithms 2 3. Conclusions and Summary 6 References 7 2 On Clustering Massive Data Streams: A Summarization Paradigm 9 Charu C. Aggarwal, Jiawei Han, Jianyong Wang and Philip S. Yu 1. Introduction The Micro-clustering Based Stream Mining Framework Clustering Evolving Data Streams: A Micro-clustering Approach Micro-clustering Challenges Online Micro-cluster Maintenance: The CluStream Algorithm High Dimensional Projected Stream Clustering Classification of Data Streams: A Micro-clustering Approach On-Demand Stream Classification Other Applications of Micro-clustering and Research Directions Performance Study and Experimental Results Discussion 36 References 36 3 A Survey of Classification Methods in Data Streams 39 Mohamed Medhat Gaber, Arkady Zaslavsky and Shonali Krishnaswamy 1. Introduction Research Issues Solution Approaches Classification Techniques Ensemble Based Classification Very Fast Decision Trees (VFDT) 46

5 vi DATA STREAMS: MODELS AND ALGORITHMS 4.3 On Demand Classification Online Information Network (OLIN) LWClass Algorithm ANNCAD Algorithm SCALLOP Algorithm Summary 52 References 53 4 Frequent Pattern Mining in Data Streams 61 Ruoming Jin and Gagan Agrawal 1. Introduction Overview New Algorithm Work on Other Related Problems Conclusions and Future Directions 80 References 81 5 A Survey of Change Diagnosis Algorithms in Evolving Data Streams 85 Charu C. Aggarwal 1. Introduction The Velocity Density Method Spatial Velocity Profiles Evolution Computations in High Dimensional Case On the use of clustering for characterizing stream evolution On the Effect of Evolution in Data Mining Algorithms Conclusions 100 References Multi-Dimensional Analysis of Data Streams Using Stream Cubes 103 Jiawei Han, Y. Dora Cai, Yixin Chen, Guozhu Dong, Jian Pei, Benjamin W. Wah, and Jianyong Wang 1. Introduction Problem Definition Architecture for On-line Analysis of Data Streams Tilted time frame Critical layers Partial materialization of stream cube Stream Data Cube Computation Algorithms for cube computation Performance Study Related Work Possible Extensions Conclusions 122 References Load Shedding in Data Stream Systems 127

6 Contents vii Brian Babcock, Mayur Datar and Rajeev Motwani 1. Load Shedding for Aggregation Queries Problem Formulation Load Shedding Algorithm Extensions Load Shedding in Aurora Load Shedding for Sliding Window Joins Load Shedding for Classification Queries Summary 146 References The Sliding-Window Computation Model and Results 149 Mayur Datar and Rajeev Motwani 0.1 Motivation and Road Map A Solution to the BasicCounting Problem The Approximation Scheme Space Lower Bound for BasicCounting Problem Beyond 0 s and 1 s References and Related Work Conclusion 164 References A Survey of Synopsis Construction in Data Streams 169 Charu C. Aggarwal, Philip S. Yu 1. Introduction Sampling Methods Random Sampling with a Reservoir Concise Sampling Wavelets Recent Research on Wavelet Decomposition in Data Streams Sketches Fixed Window Sketches for Massive Time Series Variable Window Sketches of Massive Time Series Sketches and their applications in Data Streams Sketches with p-stable distributions The Count-Min Sketch Related Counting Methods: Hash Functions for Determining Distinct Elements Advantages and Limitations of Sketch Based Methods Histograms One Pass Construction of Equi-depth Histograms Constructing V-Optimal Histograms Wavelet Based Histograms for Query Answering Sketch Based Methods for Multi-dimensional Histograms Discussion and Challenges 200 References 202

7 viii DATA STREAMS: MODELS AND ALGORITHMS 10 A Survey of Join Processing in Data Streams 209 Junyi Xie and Jun Yang 1. Introduction Model and Semantics State Management for Stream Joins Exploiting Constraints Exploiting Statistical Properties Fundamental Algorithms for Stream Join Processing Optimizing Stream Joins Conclusion 230 Acknowledgments 231 References Indexing and Querying Data Streams 237 Ahmet Bulut, Ambuj K. Singh 1. Introduction Indexing Streams Preliminaries and definitions Feature extraction Index maintenance Discrete Wavelet Transform Querying Streams Monitoring an aggregate query Monitoring a pattern query Monitoring a correlation query Related Work Future Directions Distributed monitoring systems Probabilistic modeling of sensor networks Content distribution networks Chapter Summary 257 References Dimensionality Reduction and Forecasting on Streams 261 Spiros Papadimitriou, Jimeng Sun, and Christos Faloutsos 1. Related work Principal component analysis (PCA) Auto-regressive models and recursive least squares MUSCLES Tracking correlations and hidden variables: SPIRIT Putting SPIRIT to work Experimental case studies Performance and accuracy Conclusion 286 Acknowledgments 286

8 Contents ix References A Survey of Distributed Mining of Data Streams 289 Srinivasan Parthasarathy, Amol Ghoting and Matthew Eric Otey 1. Introduction Outlier and Anomaly Detection Clustering Frequent itemset mining Classification Summarization Mining Distributed Data Streams in Resource Constrained Environments Systems Support 300 References Algorithms for Distributed Data Stream Mining 309 Kanishka Bhaduri, Kamalika Das, Krishnamoorthy Sivakumar, Hillol Kargupta, Ran Wolff and Rong Chen 1. Introduction Motivation: Why Distributed Data Stream Mining? Existing Distributed Data Stream Mining Algorithms A local algorithm for distributed data stream mining Local Algorithms : definition Algorithm details Experimental results Modifications and extensions Bayesian Network Learning from Distributed Data Streams Distributed Bayesian Network Learning Algorithm Selection of samples for transmission to global site Online Distributed Bayesian Network Learning Experimental Results Conclusion 326 References A Survey of Stream Processing 333 Problems and Techniques in Sensor Networks Sharmila Subramaniam, Dimitrios Gunopulos 1. Challenges The Data Collection Model Data Communication Query Processing Aggregate Queries Join Queries Top-k Monitoring 341

9 x DATA STREAMS: MODELS AND ALGORITHMS 4.4 Continuous Queries Compression and Modeling Data Distribution Modeling Outlier Detection Application: Tracking of Objects using Sensor Networks Summary 347 References 348 Index 353

Data Streams Models and Algorithms

Data Streams Models and Algorithms ADVANCES IN DATABASE SYSTEMS Series Editor Ahmed K. Elmagarmid Purdue Universify West Lafayette, IN 47907 Other books in the Series: SIMILARITY SEARCH: The Metric Space