k-means Gaussian mixture model Maximize the likelihood exp(

Transcription

1 k-means Gaussian miture model Maimize the likelihood Centers : c P( {, i c j,...,, c n },...c k, ) ep( i c j )

2 k-means P( i c j, ) ep( c i j ) Minimize i c j Sum of squared errors (SSE) criterion (k clusters and n samples) min j k i C j i c j

3 k-means k-means works perfectly when clusters are linearly separable and spherical

4 k-means k-means works perfectly when clusters are linearly separable and spherical

5 k-means SSE criterion doesn t always work

6 k-means What about data which contain arbitrarily shaped clusters of different densities?

7 The Kernel Trick Revisited

8 The Kernel Trick Revisited Map points to feature space using basis function () ( ). ( y) Replace dot product (for similarity computation between points and y) with kernel entry K(, y) Mercer s condition: To epand Kernel function K(,y) into a dot product, i.e. K(,y)= () (y), K(, y) has to be positive semi-definite function, i.e., for any function f() whose is finite, the following inequality holds ddyf ( ) K(, y) f ( y) 0

9 Kernel k-means Minimize sum of squared error: k-means: min n k i j u ij i c j u {0,} u ij ij j k

10 Kernel k-means Minimize sum of squared error: k-means: min n k i j u ij i c j min n k i j u ij Replace with () ( i ) c~ u {0,} u ij ij j k j

11 Kernel k-means Cluster centers: Substitute for centers: n i i ij j j u n c ) ( ~ n i k j ij n i k j ij n l l lj j i u j i u u n c ) ( ) ( ~ ) (

12 Kernel k-means Use kernel trick: n k i j u ij ~ ( i ) c j trace( K) trace( UKU') Optimization problem: min trace ( K) trace ( UKU ') ma trace ( UKU ') K is the n n kernel matri, U is the optimal normalized cluster membership matri

13 Eample k

14 Eample k k-means clusters

15 Eample

16 Eample 3,, ),, ( ), ( ) ' ( ), ( kernel Polynomial z z z y y K z z z 3

17 Eample 3,, ),, ( ), ( ) ' ( ), ( kernel Polynomial z z z y y K z z z 3

18 Eample z z z 3 3,, ),, ( ), ( ) ' ( ), ( kernel Polynomial z z z y y K

19 k-means Vs. Kernel k-means k-means k Kernel k-means

20 Performance of Kernel k-means Evaluation of the performance of clustering algorithms in kernelinduced feature space, Pattern Recognition, 005

21 Limitations of Kernel k-means More comple than k-means Need to compute and store n n kernel matri Appropriate kernel function has to be determined Largest n that can be handled?

22 Limitations of Kernel k-means More comple than k-means Need to compute and store n n kernel matri Appropriate kernel function has to be determined Largest n that can be handled? Intel Xeon E Processor (Q ), Oct-core,.8GHz, 4TB ma memory < million points with single precision numbers May take several days to only compute the kernel matri

23 Big data Volume* Big data comes in one size: large *Defn. due to IBM

24 Data Volume Application Clustering Task Size of data Number of features Document retrieval Gene analysis Image retrieval Earth science data analysis Group documents of similar topics Group genes with similar epression levels Quantize low-level features Derive climate indices

25 Big data Velocity Often time-sensitive, big data must be used as it is streaming

26 Big data Variety Big data etends beyond structured data, including unstructured data of all varieties: tet, audio, video, click streams, log files and more

27 Large Scale Clustering Deals with the first issue related to big data the volume of data Issues: Computational Compleity Hardware Limitations Application Requirements

28 MapReduce Framework

29 How to distribute k-means?

30 How to distribute k-means? Two methods Distribute distance computation

31 k-means Clustering with MapReduce - I Distribute the cost of distance computation Cluster centers maintained in global memory Divide points among map tasks Parallel k-means clustering based on MapReduce, Cloud computing, 009

32 Map function k-means Clustering with MapReduce - I Find the closest center for data point Intermediate output: Closest cluster inde Combine function Partially sum the values of the points assigned to the same cluster, keep track of number of points in the cluster Reduce function Compute new centers from the output of combine function Parallel k-means clustering based on MapReduce, Cloud computing, 009

33 How to distribute k-means? Two methods Distribute distance computation

34 How to distribute k-means? Two methods Distribute distance computation Distribute clustering task

35 k-means Clustering with MapReduce - II Distribute the cost of clustering Map function Cluster the partition into k clusters Intermediate output: Clusters of the partition Reduce function Cluster the cluster centers from the map output to obtain the new centers Fast clustering using MapReduce, KDD, 0

36 k-means Clustering with MapReduce - II No global storage required Approimate solution Clustering error (SSE) < * optimal clustering error Fast clustering using MapReduce, KDD, 0

37 Machine Learning on Mapreduce Mahout scalable implementation of major clustering and classification algorithms on Hadoop Open source Java and Maven based

38 Large Scale Kernel Clustering Data set with 'n' points. K n n When n ~ 0 6 more than TB of memory required, highly epensive computationally

39 Approimate Kernel k-means Low rank approimation Use a small portion of the kernel matri for clustering. (n-m) (n-m) chunk of the kernel matri need not be computed K K B n n n m m m m n = ˆK ' K B Approimate Kernel k-means: Solution to Large Scale Kernel Clustering, KDD, 0

40 Approimate Kernel k-means Cluster centers linear combination of sampled points Approimation error m c j ( ˆ i ji i ) Clustering error m Optimal Clustering error Approimate Kernel k-means: Solution to Large Scale Kernel Clustering, KDD, 0

41 Approimate Kernel k-means

42 Performance of Approimate Kernel k-means

43 Performance of Approimate Kernel k-means MNIST data set (70,000 data points) Kernel calculation Clustering Kernel k-means 54 seconds 3953 seconds Approimate kernel k- means (m=000) 8 seconds 75 seconds About 98% reduction in time Almost the same clustering error as kernel k-means

44 Performance of Approimate Kernel k-means Network Intrusion data set ( > 4 million data points) Kernel k-means not possible on a normal system Requires 64 TB of memory Approimate kernel k-means with just 40 GB memory Approimate kernel k- means (m=000) Kernel calculation Clustering 5 seconds 433 seconds

45 Summary Kernel k-means Performs better than k-means Kernel clustering algorithms, in general are more comple than linear clustering algorithms Large scale clustering Distributed and approimate variants of eisting algorithms required for clustering large data