Large synthetic data sets to compare different data mining methods

Transcription

1 Large synthetic data sets to compare different data mining methods Victoria Ivanova, Yaroslav Nalivajko Superviser: David Pfander, IPVS June 3, 2016

2 Overview 1 Data Mining Overview 2 Support Vector Machines 3 Random Forests 4 Data sets and their main characteristics 5 Conclusions

3 Data mining Data Mining Definition Data mining is the application of specific algorithms for extracting patterns from data. U. Fayyad, G. Piatetsky-Shapiro, P. Smyth

4 Binary classification vector space R n, with vectors x = (x 1, x 2,..x n ) target function f : R n {+1, 1} labeled training set: D = {( x 1, y 1 ), ( x 2, y 2 ), ( x 3, y 1 ),...} R n {+1, 1} a model: ˆf : R n {+1, 1} should be computed, using D such that ˆf ( x) = f ( x) for all x i R n

5 Evaluating Data mining methods what for Choose the right method for a specific task/data set compare the performance of two or more data mining methods find the main advantages/disadvantages of given data mining methods

6 Support Vector Machines SVM - common data mining algorithm introduced by Vapnik and Cortes 1995 classification (binary classification, multiclass extensions), regression maximum margin classifier

7 Linear separable case w - orthogonal to optimal hyperplane w x b 1,if y = 1 w x b 1,if y = 1 the margin width 2 w max 2 w w = min 2 with constraints: y i (w x b) 1, x D w x b = 1 w x b = 1 x 2 ρ x 1 w x b = 0

8 Maximum margin calculation min w 2 s.t. y i (w x b) 1, x D is equivalent (Karush-Kuhn-Tacker conditions) to finding a saddle point of a Lagrange multiplier constrained quadratic programming problem max α 1 n 2 i,j=1 α iα j y i y j xi x j + n i=1 α i s.t. α i 0 and n i=1 α iy i = 0 can be solved with various algorithms only support vectors correspond to non zero α-s now w can be computed w = n i=1 α iy i = 0 bias b is calculated as median from b = w x y

9 Linearly not separable case, soft margin classifier slack variable ξ 1 2 w +C n i ξ i - problem reformulation constraint y i (w x b) 1 ξ C - regularization parameter ρ x 2 x 1 w x b = 0

10 Linearly not separable case, Kernel Trick Data points are mapped into a higher dimensional space every x mapped to Φ(x) Kernel Function k: k x, y = Φ(x), Φ(y) optimization problem reformulation: max α 1 n 2 i,j=1 α iα j y i y j k x i, x j + n i=1 α i s.t. 0 α i C and n i=1 α iy i = 0

11 Random Forests Used to solve different Data Mining problems Based on Ensemble of Decision trees Introduced by Leo Breiman and Adele Cutler, 1995

12 Decision Tree A Decision Tree for classification simple 2x2 chess-data. The are different algorithms to build DT: ID3, C4.5, CART, RI, IndCart, DB-Cart, CHAID or MARS. true X > 0.5 false y 1 Y > 0.5 Y > 0.5 true false true false x O 1 1 2

13 Random Forest principle of operation Random Forest Algorithm build N Decision Trees, to learn each of them randomly chosen part of training set is used. Then each Tree votes. Answer with maximum number of votes is returned by Random Forest. Tree 1 Tree 2 Tree N

14 Comparison of the method s complexity Method time complexity space complexity SVM O(m 3 ) m is the training size O(m 2 ) RF O(M (m n log(n)) m - instances, n attributes, M trees grown O(M 2 h ) h - height of a tree

15 Known problems in data sets Noise Class noise: Misclassification, contradictory examples e.g. duplicate values Attribute noise: Missing/unknown attribute values, incomplete/wrong values Crosstalk Weak vs strong relationships(redundant attributes) Inherent complexity Imbalanced data (one class dominates over the other) Overlapping data Too many attributes/dimensions

16 Not axis parallel data sets Check data set rotation angle 30 degrees 1 Method 2D 30 degrees 2D 1 degree 3D 30 degrees SVM 98.34% 98.37% 90.88% RF 99.24% 96.99% 82.5% points y coordinate x coordinate

17 Dataset class noise 1 Method 2D 1% 2D 2% 2D 5% 5D, 5% SVM 97.71% 96.29% 92.78% 58.0% RF 98.88% 97.83% 94.67% % Number of points y coordinate x coordinate

18 Dataset attribute noise Method 2D, 1K 2D, 10K 3D, 10K SVM 94.5% 94.2% 90.29% RF 95.5% 95.2% 92.15% adding a random number to all points y coordinate x coordinate

19 Dataset linearly not separable I 2 dimensional sphere in a square Method 2D, 10K 3D, 10K 5D, 10K SVM 98.1% 98.73% 97.45% RF 99.01% 96.69% 95.8 % y coordinate x coordinate 1000 points

20 Dataset linearly not separable II 2 dimensional sphere in a sphere Method 2D, 1K 3D, 10K 5D, 10K 10D, 100K SVM 98.5% 97.58% 85.28% 79.21% RF 96.6% 96.74% 90.7% 81.4 % y coordinate x coordinate 1000 points

21 Crosstalk in data datasets with redundant attributes not adding any information to classification Method 1K 1 r.a. 10K with 8 r.a. 100K 16 r.a. SVM 88.4% % 54 % RF 97.2% 99.6% %

22 Conclusions Python scripts for creation of scalable multidimensional data sets Changeable e.g: number of dimensions number of points rotation angle add attribute/class noise add redundant attributes The difficulty of data sets can be adopted for concrete problem/method Synthetic data can be usefull in testing data mining methods More experimental work is needed (testing datasets with other data mining methods e.g. Sparse Grids)

23 Thank You

24 Questions?