An Adaptive Framework for Multistream Classification

Transcription

1 An Adaptive Framework for Multistream Classification Swarup Chandra, Ahsanul Haque, Latifur Khan and Charu Aggarwal* University of Texas at Dallas *IBM Research This material is based upon work supported by

2 Data Stream Classification Time t s Model Mohammad M. Masud, Jing Gao, Latifur Khan, Jiawei Han, Bhavani M. Thuraisingham: A Practical Approach to Classify Evolving Data Streams: Training with Limited Amount of Labeled Data. ICDM 2008:

3 Data Stream Classification Time t+1 Label Time t Time t+1 s Model

4 Data Stream Analytics Label t+1 Label Evaluation t t+1 s Model Concept Drift detection Ahsanul Haque, Latifur Khan, Michael Baron, Bhavani M. Thuraisingham, Charu C. Aggarwal: Efficient handling of concept drift and concept evolution over Stream Data. ICDE 2016: Ahsanul Haque, Latifur Khan, Michael Baron: SAND: Semi-Supervised Adaptive Novel Class Detection and Classification over Data Stream. AAAI 2016:

5 Data Stream Analytics Expensive! Label t+1 Label Evaluation t t+1 s Model Concept Drift detection

6 Data Stream Analytics Semi-supervised or Active Learning t+1 Label Evaluation t t+1 s Model Concept Drift detection

7 Motivation What if we do not find a good training set? Biased training data selection mechanism. Example Scenario Biased Labeled training data Small set of users Unlabeled test data Affects Classifie r Accurac y Population

8 Problem (Multistream Classification) Two types of data stream (independent). Stream Labeled Data t Create training data t t Label Stream Unlabeled Data t+1

9 Problem (Multistream Classification) Two types of data stream (independent). Stream Labeled Data t Create training data t Concept Drift detection t Label Stream Unlabeled Data t+1

10 Potential Applications Domain Adaptation and Transfer Learning over data streams Text Classification Sensor-based location estimation Collaborative filtering

11 Outline Challenges Solution Overview (MSC) Framework Details Empirical Evaluation Conclusion

12 Challenges Leveraging labeled and unlabeled data bias-corrected training set. Asynchronous concept drift in source and target stream. Drift detection Non-Stationary Process Drift 11 Drift 12 Drift correction Stream Domain Stream Time Drift 21 Drift 22

13 Challenges Can the two streams be combined? Data distributions are different. Combination represent same distribution Separate representation has advantages when multiple sources are present.

14 Solution Overview Stream (Unlabeled) Output 2 Non-stationary Domain Stream (Labeled) Class Drift Detection (CDT) 5a Ensemble Update 5b

15 Design Overview Two data streams Stream (Unlabeled) Output 2 Non-stationary Domain Stream (Labeled) Class Drift Detection (CDT) 5a Ensemble Update 5b

16 Design Overview Two data streams To address asynchronous concept drift. Stream (Unlabeled) Non-stationary Domain Stream (Labeled) Output 2 Class Drift Detection (CDT) 5a Ensemble Update 5b

17 Design Overview Two data streams To address asynchronous concept drift. Stream (Unlabeled) Non-stationary Domain Stream (Labeled) Output 2 Class Drift Detection (CDT) 5a Ensemble Update 5b

18 Solution Overview Data in source and target occur simultaneously. Stream (Unlabeled) Output 2 Non-stationary Domain Stream (Labeled) Class Drift Detection (CDT) 5a Ensemble Update 5b

19 Solution Overview Data in source and target occur simultaneously. In the case of source data Stream (Unlabeled) Non-stationary Domain Stream (Labeled) Output 2 Class Drift Detection (CDT) 5a Ensemble Update 5b

20 Solution Overview Data in source and target occur simultaneously. In the case of source data Stream (Unlabeled) Non-stationary Domain Stream (Labeled) Output 2 Class Drift Detection (CDT) 5a Ensemble Update 5b

21 Solution Overview Data in source and target occur simultaneously. In the case of source data, drift detection output used to update source classifier. Stream (Unlabeled) Non-stationary Domain Stream (Labeled) Output 2 Class Drift Detection (CDT) 5a Ensemble Update 5b

22 Solution Overview Data in source and target occur simultaneously. In the case of source data, drift detection output used to update source classifier. Stream (Unlabeled) Non-stationary Domain Stream (Labeled) Output 2 Class Drift Detection (CDT) 5a Ensemble Update 5b

23 Solution Overview Data in source and target occur simultaneously. In the case of target data Stream (Unlabeled) Non-stationary Domain Stream (Labeled) Output 2 Class Drift Detection (CDT) 5a Ensemble Update 5b

24 Solution Overview Data in source and target occur simultaneously. In the case of target data Stream (Unlabeled) Non-stationary Domain Stream (Labeled) Output 2 Class Drift Detection (CDT) 5a Ensemble Update 5b

25 Solution Overview Data in source and target occur simultaneously. In the case of target data Stream (Unlabeled) Non-stationary Domain Stream (Labeled) Output 2 Class Drift Detection (CDT) 5a Ensemble Update 5b

26 Solution Overview Data in source and target occur simultaneously. In the case of target data, drift detection output used to update target classifier. classifier corrects bias between source and target stream at time t. Stream (Unlabeled) Non-stationary Domain Stream (Labeled) Output 2 Class Drift Detection (CDT) 5a Ensemble Update 5b

27 Solution Overview Data in source and target occur simultaneously. In the case of target data, drift detection output used to update target classifier. classifier corrects bias between source and target stream at time t. Stream (Unlabeled) Non-stationary Domain Stream (Labeled) Output 2 Class Drift Detection (CDT) 5a Ensemble Update 5b

28 Typical classifier using training data from source stream. Predict labels of newly occuring source stream data. Bias corrected source stream data for training. Predict labels of newly occuring target stream data.

29 Training : Sampling bias correction via Kernel Mean Matching Minimize mean discrepency between labeled source and unlabeled target distribution. data instance weight: : window : window Matrices of kernel in RKHS:

30 Label Finite dynamic size window for incoming source and target data. Weighted hybrid ensemble Fixed number of classifiers. Contains both source and target classifiers. classifier weight based on classifier error.. : classifier weight based on classifier confidence on unlabeled target data.

31 Concept Drift Detection classifier error window Contain binary values. Follow Bernoulli distribution. classifier confidence window CUSUM-type change point detection to detect change point at element q of window W. Sequential sub-window Likelihood ratio score at point q: Contain confidence value between 0 and 1. Follow Beta distribution. Change point is at q if:

32 Drift Adaptation - Why not train both types of classifiers once a drift is detected on either stream? - Sampling bias correction if target stream has a concept drift. Stream Stream Stream Stream Stream Stream Drift 11 Adaptation not required Adaptation required Drift Drift 3121 Adaptatio n required Drift 32 Case 1 only drift Case 2 only drift Case 3 & drift

33 Empirical Evaluation Dataset # features # classes # instances ForestCover ,438 Real World Sensor ,000 SEA ,000 SynEDC ,816 Synthetic SynRBF@00 2 SynRBF@ , ,686 Divide dataset into and Stream, with bias in source stream data selection according to:

34 Empirical Evaluation SVM as base classifier : Typical multiclass SVM. : Weighted SVM confidence: Distance of test data to hyperplane.

35 Empirical Evaluation Baseline Variants Symbols skmm mkmm-5k srcmsc trgmsc MSC MSC2 Description Single target classifier without update. Single target classifier with update every 5k instances. CPD with source classifier only. No bias correction. CPD with target classifier only. No source drift adaptation. Proposed method with hybrid ensemble. Proposed method with separate source and target ensemble.

36 Results MSC is better MSC2 is better ForestCover MSC baselines also good, but.. Sensor Dataset

37 Results MSC2 is better MSC2 is better Dataset Dataset

38 Conclusion Introduce a new data stream mining setting with bias labeled data Propose a framework to address new challenges of concept drift in this setting. Empirical results achieve significantly better accuracy than baseline. Future work: Multi-source setting and Semi-supervised target stream classification.

39 Thank you Q & A