+ Cluster analysis. a generalization can be derived for each cluster and hence processing is done batch wise rather than individually

Transcription

1 Trasitio 1

2 + Cluster aalysis 2 Provides a quick ad meaigful overview of data Improves efficiecy of data miig by combiig data with similar characteristics so that a geeralizatio ca be derived for each cluster ad hece processig is doe batch wise rather tha idividually Gives a good uderstadig of the uusual similarities that may occur oce the clusterig is complete Provides a really good base for earest eighbourig ordiatio of deeper relatios

3 3 Parallel K-meas: practical experiece Geoveva Vargas-Solar Frech Coucil of Scietific Research, LIG & LAFMIA Labs Javier Espiosa Barceloa Supercomputig Cetre & LAFMIA Lab Motevideo, 22 d November 4 th December, INFORMATIQUE

4 + K-meas 4 P = (x1(p), x2(p), x3(p)...) ad Q = (x1(q), x2(q), x3(q)...). Compute the distace d(p,q) Compute cluster cetroid: the poit whose coordiates correspods to the mea of the coordiates of all the poits i the cluster The data set will have certai items that may ot be related to ay cluster ad that caot be classified uder them, Such poits are referred to as outliers Ofte correspod to the extremes of the data set depedig o whether their values or extremely high or low Objective: obtai a miimal squared differece betwee the cetroid of the cluster ad the item i the dataset

5 + K-meas 5 Objective: obtai a miimal squared differece betwee the cetroid of the cluster ad the item i the dataset Where x i is the value of the item ad c j is the value of the cetroid of the cluster

6 + K-meas steps 6 The required umber of cluster must be chose: K Choose distat ad distict cetroids for each of the chose set of K clusters Cosider each elemet of the give set ad compare its distace to all the cetroids of the K clusters. Based o the calculated distace the elemet is added to the cluster whose cetroid is earest to the elemet The cluster cetroids are re-calculated after each assigmet or a set of assigmets Iterative method ad cotiuously updated

7 7 Map reduced K-meas Prajesh P Achalia, AjaK Koudiya, Sriath N K, MapReduce Desig of K-Meas Clusterig Algorithm, IEEE, 2013

8 + Desig steps 8 Defie ad hadle the iput ad output of the implemetatio. The iput is give as a <key, value> pair key is the cluster cetre value a vector i the data set

9 + Geeral priciple 9

10 Implemetatio 1 10

11 + Pre-requisite 11 Two files: F1: houses the clusters with their cetroids F2: houses the vectors to be clustered The iitial set of cetres is stored i the iput directory of HDFS they form the key field i the <key,value>

12 + Map & Reduce routies 12 Mapper: Computes the distace betwee the give data set ad cluster cetre fed as a <key,value> Keeps track of the cluster to which the give vector is closest Assig the vector to the earest cluster, oce the computatio of distaces is complete Reducer: Recalculates the cetroid Restructures the cluster to prevet creatios of clusters with extreme sizes i.e. cluster havig too less data vectors or a cluster havig too may data vectors Re-writes the ew set of vectors ad clusters to the disk Ready for the ext iteratio

13 13

14 14

15 15 Implemetatio 2

16 Implemetatio 3: Spark platform 16

17 Spark Fast, Iteractive, Laguage-Itegrated Cluster Computig Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Akur Dave, Justi Ma, Murphy McCauley, Michael Frakli, Scott Sheker, Io Stoica UC BERKELEY

18 + Project Goals 18 Exted the MapReduce model to better support two commo classes of aalytics apps: Iterative algorithms (machie learig, graphs) Iteractive data miig Ehace programmability: Itegrate ito Scala programmig laguage Allow iteractive use from Scala iterpreter

19 + Motivatio 19 Most curret cluster programmig models are based o acyclic data flow from stable storage to stable storage Map Reduce Iput Map Output Map Reduce

20 + Motivatio 20 Most curret cluster programmig models are based o acyclic data flow from stable storage to stable storage Map Reduce Beefits of data flow: rutime ca decide where Iput to ru Map tasks ad ca automatically Output recover from failures Map Reduce

21 + Motivatio 21 Acyclic data flow is iefficiet for applicatios that repeatedly reuse a workig set of data: Iterative algorithms (machie learig, graphs) Iteractive data miig tools (R, Excel, Pytho) With curret frameworks, apps reload data from stable storage o each query

22 + Solutio: Resiliet Distributed Datasets (RDDs) 22 Allow apps to keep workig sets i memory for efficiet reuse Retai the attractive properties of MapReduce Fault tolerace, data locality, scalability Support a wide rage of applicatios

23 + Spark Operatios 23 Trasformatios (defie a ew RDD) map filter sample groupbykey reducebykey sortbykey flatmap uio joi cogroup cross mapvalues Actios (retur a result to driver program) collect reduce cout save lookupkey

24 + Outlie 24 Spark programmig model Implemetatio User applicatios

25 + Programmig Model 25 Resiliet distributed datasets (RDDs) Immutable, partitioed collectios of objects Created through parallel trasformatios (map, filter, groupby, joi, ) o data i stable storage Ca be cached for efficiet reuse Actios o RDDs Cout, reduce, collect, save,

26 + Example: Log Miig 26 Load error messages from a log ito memory, the iteractively search for various patters lies = spark.textfile( hdfs://... ) errors = lies.filter(_.startswith( ERROR )) messages = errors.map(_.split( \t )(2)) cachedmsgs = messages.cache() cachedmsgs.filter(_.cotais( foo )).cout cachedmsgs.filter(_.cotais( bar )).cout... Result: full-text scaled search to 1 TB of data Wikipedia i 5-7 sec i <1 sec (vs (vs sec for for o-disk data) Base RDD Trasformed RDD results Work er Driver Actio Block 3 Cache 3 tasks Work er Block 1 Work er Block 2 Cache 1 Cache 2

27 + RDD Fault Tolerace 27 RDDs maitai lieage iformatio that ca be used to recostruct lost partitios Ex: messages = textfile(...).filter(_.startswith( ERROR )).map(_.split( \t )(2)) HDFS File Filtered RDD filter (fuc = _.cotais(...)) map (fuc = _.split(...)) Mapped RDD

28 + Example: Logistic Regressio 28 Goal: fid best lie separatig two sets of poits radom iitial lie target

29 + Example: Logistic Regressio 29 val data = spark.textfile(...).map(readpoit).cache() var w = Vector.radom(D) for (i <- 1 to ITERATIONS) { val gradiet = data.map(p => (1 / (1 + exp(-p.y*(w dot p.x))) - 1) * p.y * p.x ).reduce(_ + _) w -= gradiet } pritl("fial w: " + w)

30 + Logistic Regressio Performace 30 Ruig Time (s) Number of Iteratios 127 s / iteratio Hadoop Spark first iteratio 174 s further iteratios 6 s This is for a 29 GB dataset o 20 EC2 m1.xlarge machies (4 cores each)

31 Spark Scheduler 31 Dryad-like DAGs A: B: Pipelies fuctios withi a stage Stage 1 groupby G: Cache-aware work reuse & locality C: D: map E: F: joi Partitioig-aware to avoid shuffles Stage 2 uio Stage 3 = cached data partitio

32 + Coclusio 32 Spark provides a simple, efficiet, ad powerful programmig model for a wide rage of apps Dowload our ope source release: matei@berkeley.edu

33 Related Work 33 DryadLINQ, FlumeJava Similar distributed collectio API, but caot reuse datasets efficietly across queries Relatioal databases Lieage/proveace, logical loggig, materialized views GraphLab, Piccolo, BigTable, RAMCloud Fie-graied writes similar to distributed shared memory Iterative MapReduce (e.g. Twister, HaLoop) Implicit data sharig for a fixed computatio patter Cachig systems (e.g. Nectar) Store data i files, o explicit cotrol over what is cached

34 34 Let s dive o Spark for executig ad aalyzig K-Meas

35 35