An Introduction to Big Data Analysis using Spark
|
|
- Mervyn Chapman
- 6 years ago
- Views:
Transcription
1 An Introduction to Big Data Analysis using Spark Mohamad Jaber American University of Beirut - Faculty of Arts & Sciences - Department of Computer Science May 17, 2017 Mohamad Jaber (AUB) Spark May 17, / 43
2 Big Data 1 Big Data 2 Apache Spark 3 Distributed File System Mohamad Jaber (AUB) Spark May 17, / 43
3 Big Data Big Data We live in the data age Mohamad Jaber (AUB) Spark May 17, / 43
4 Big Data Big Data - Some Numbers ( ) Storage and Processing Mohamad Jaber (AUB) Spark May 17, / 43
5 Big Data Big Data - Some Numbers ( ) Storage and Processing Facebook hosts more than 240 billion photos, growing at 7 petabytes per month Mohamad Jaber (AUB) Spark May 17, / 43
6 Big Data Big Data - Some Numbers ( ) Storage and Processing Facebook hosts more than 240 billion photos, growing at 7 petabytes per month New York Stock Exchange generates about 4 5 terabytes of data per day Mohamad Jaber (AUB) Spark May 17, / 43
7 Big Data Big Data - Some Numbers ( ) Storage and Processing Facebook hosts more than 240 billion photos, growing at 7 petabytes per month New York Stock Exchange generates about 4 5 terabytes of data per day Google processes 20 petabytes of information per day... Mohamad Jaber (AUB) Spark May 17, / 43
8 Big Data Big Data - Some Numbers ( ) Storage and Processing Facebook hosts more than 240 billion photos, growing at 7 petabytes per month New York Stock Exchange generates about 4 5 terabytes of data per day Google processes 20 petabytes of information per day... Estimation The size of digital universe is 4.4 zettabytes (10 21 ) in 2013 Mohamad Jaber (AUB) Spark May 17, / 43
9 Big Data Big Data - Some Numbers ( ) Storage and Processing Facebook hosts more than 240 billion photos, growing at 7 petabytes per month New York Stock Exchange generates about 4 5 terabytes of data per day Google processes 20 petabytes of information per day... Estimation The size of digital universe is 4.4 zettabytes (10 21 ) in : 44 zettabytes Mohamad Jaber (AUB) Spark May 17, / 43
10 Big Data Simple Java Program to Analyze Data p u b l i c s t a t i c l o n g analyze ( String filename, Analyzer analyzer ) throws IOException { // Read Input BufferedReader reader = new BufferedReader (new FileReader ( filename )); l o n g score = 0; String line = n u l l ; } // Processing w h i l e (( line = reader. readline ())!= n u l l ) { score += analyzer. analyze ( line ); } r e t u r n score ; Mohamad Jaber (AUB) Spark May 17, / 43
11 Big Data Simple Java Program to Analyze Data p u b l i c s t a t i c l o n g analyze ( String filename, Analyzer analyzer ) throws IOException { // Read Input BufferedReader reader = new BufferedReader (new FileReader ( filename )); l o n g score = 0; String line = n u l l ; } // Processing w h i l e (( line = reader. readline ())!= n u l l ) { score += analyzer. analyze ( line ); } r e t u r n score ; Throughput 1GB per hour 10GB data set 10 hours Mohamad Jaber (AUB) Spark May 17, / 43
12 Big Data How can we Improve the Performance? Mohamad Jaber (AUB) Spark May 17, / 43
13 Big Data How can we Improve the Performance? Faster CPU Scale up (vertically) More/Faster memory Scale up (vertically) Increase the number of cores Increase the number of threads Mohamad Jaber (AUB) Spark May 17, / 43
14 Big Data How can we Improve the Performance? Faster CPU Scale up (vertically) More/Faster memory Scale up (vertically) Increase the number of cores Increase the number of threads Increase the number of threads and cores Shared Memory (pthreads) Message Passing (MPI) Mohamad Jaber (AUB) Spark May 17, / 43
15 Big Data How can we Improve the Performance? Faster CPU Scale up (vertically) More/Faster memory Scale up (vertically) Increase the number of cores Increase the number of threads Increase the number of threads and cores Shared Memory (pthreads) Message Passing (MPI) Multi-threaded Throughput 10GB per hour Mohamad Jaber (AUB) Spark May 17, / 43
16 Big Data How can we Improve the Performance? Faster CPU Scale up (vertically) More/Faster memory Scale up (vertically) Increase the number of cores Increase the number of threads Increase the number of threads and cores Shared Memory (pthreads) Message Passing (MPI) Multi-threaded Throughput 10GB per hour 1PB? Fault? Mohamad Jaber (AUB) Spark May 17, / 43
17 What do we Need? Big Data Mohamad Jaber (AUB) Spark May 17, / 43
18 What do we Need? Big Data We need a framework that abstracts away / hides: Mohamad Jaber (AUB) Spark May 17, / 43
19 What do we Need? Big Data We need a framework that abstracts away / hides: Scale Out (horizontally) Parallelization Data distribution Fault-tolerance Load Balancing Mohamad Jaber (AUB) Spark May 17, / 43
20 Apache Spark 1 Big Data 2 Apache Spark 3 Distributed File System Mohamad Jaber (AUB) Spark May 17, / 43
21 Why Spark? Apache Spark Normally, data science and analytics is done in the small, in R/Python/MATLAB, etc. Mohamad Jaber (AUB) Spark May 17, / 43
22 Why Spark? Apache Spark Normally, data science and analytics is done in the small, in R/Python/MATLAB, etc. If your dataset ever gets too large to fit into memory, these languages/frameworks won t allow you to scale Mohamad Jaber (AUB) Spark May 17, / 43
23 Why Spark? Apache Spark Normally, data science and analytics is done in the small, in R/Python/MATLAB, etc. If your dataset ever gets too large to fit into memory, these languages/frameworks won t allow you to scale You have to re-implement everything in some other language or system Mohamad Jaber (AUB) Spark May 17, / 43
24 Why Spark? Apache Spark Normally, data science and analytics is done in the small, in R/Python/MATLAB, etc. If your dataset ever gets too large to fit into memory, these languages/frameworks won t allow you to scale You have to re-implement everything in some other language or system Moreover, there is a massive shift in industry to data-oriented decision making too! data science in the large Mohamad Jaber (AUB) Spark May 17, / 43
25 Why Spark? Apache Spark Normally, data science and analytics is done in the small, in R/Python/MATLAB, etc. If your dataset ever gets too large to fit into memory, these languages/frameworks won t allow you to scale You have to re-implement everything in some other language or system Moreover, there is a massive shift in industry to data-oriented decision making too! data science in the large According to the popular IT job portal, Dice.com, a keyword search for the term Spark Developer showed listings as of 16th December, Mohamad Jaber (AUB) Spark May 17, / 43
26 Why Spark? Apache Spark Spark is More expressive. APIs modeled after Scala collections. Look like functional lists! Richer, more composable operations possible than in MapReduce Efficient. Not only performant in terms of running time... But also in terms of developer productivity! Interactive! Good for data science. Not just because of performance, but because it enables iteration, which is required by most algorithms in a data scientist s toolbox (e.g., machine learning, graph analytics) Mohamad Jaber (AUB) Spark May 17, / 43
27 Scala Quick Tour Apache Spark Scala is a high-level language for the Java VM (object oriented + functional programming) supports interactive shell // declare variables v a r x: Int = 7 v a r x = 7 // type inferred v a l y = "hi" // read - only // Functions d e f square (x: Int ): Int = x * x d e f square (x: Int ): Int = { x*x } d e f announce ( text : String ) { println ( text ) } // Generic Types v a r arr = new Array [ Int ](8) v a r lst = List (1, 2, 3) arr (5) = 7 // processing collections v a l list = List (1, 2, 3) list. foreach (x => println (x)) list. foreach ( println ) // shortcut v a l incmap = list. map (x => x + 2) // same with place holder notation v a l incmap = list. map (_ + 2) v a l f = list. filter (x => x % 2 == 1) v a l f = list. filter (_ % 2 == 1) v a l n = list. reduce ((x,y) => x + y ) v a l n = list. reduce (_ + _) // List is immutable Mohamad Jaber (AUB) Spark May 17, / 43
28 Apache Spark Visualizing Shared Memory Data Parallelism v a l res = jar. map ( jellybean => dosomething ( jellybean )) Shared Memory Data Parallelism Split the data Workers/threads independently operates on the data shared in parallel Combine when done (if necessary) Mohamad Jaber (AUB) Spark May 17, / 43
29 Apache Spark Visualizing Distributed Data Parallelism v a l res = jar. map ( jellybean => dosomething ( jellybean )) Distributed Data Parallelism Split the data over several nodes (machines) Workers/threads independently operates on the data shared in parallel Combine when done (if necessary) Mohamad Jaber (AUB) Spark May 17, / 43
30 Apache Spark Visualizing Distributed Data Parallelism v a l res = jar. map ( jellybean => dosomething ( jellybean )) Distributed Data Parallelism Split the data over several nodes (machines) Workers/threads independently operates on the data shared in parallel Combine when done (if necessary) New concern: Mohamad Jaber (AUB) Spark May 17, / 43
31 Apache Spark Visualizing Distributed Data Parallelism v a l res = jar. map ( jellybean => dosomething ( jellybean )) Distributed Data Parallelism Split the data over several nodes (machines) Workers/threads independently operates on the data shared in parallel Combine when done (if necessary) New concern: now we need to worry about network latency (combining)! Mohamad Jaber (AUB) Spark May 17, / 43
32 Apache Spark Apache Spark Apache Spark is a framework for distributed data processing! Spark implements a distributed data parallel model called Resilient Distributed Datasets (RDDs) Mohamad Jaber (AUB) Spark May 17, / 43
33 Apache Spark Distributed Data Parallel: High-Level Given some large dataset that cannot fit into memory on a single node Mohamad Jaber (AUB) Spark May 17, / 43
34 Apache Spark Distributed Data Parallel: High-Level Chunk up (partition) the data Distribute it over a cluster of machines From there think of your distributed data like a single collection! Mohamad Jaber (AUB) Spark May 17, / 43
35 Apache Spark Distributed Data Parallel: High-Level Chunk up (partition) the data Distribute it over a cluster of machines From there think of your distributed data like a single collection! Example (transform the text of all wiki articles to lowercase) v a l wiki : RDD [ WikiArticle ]... v a l lowerwiki = wiki. map ( article => article. text. tolowercase ) Mohamad Jaber (AUB) Spark May 17, / 43
36 Distribution Apache Spark Distribution introduces important concerns beyond parallelism in the shared memory case (single node/machine) Mohamad Jaber (AUB) Spark May 17, / 43
37 Distribution Apache Spark Distribution introduces important concerns beyond parallelism in the shared memory case (single node/machine) Partial failure: crash failure of a subset of machines Mohamad Jaber (AUB) Spark May 17, / 43
38 Distribution Apache Spark Distribution introduces important concerns beyond parallelism in the shared memory case (single node/machine) Partial failure: crash failure of a subset of machines Latency: certain operations (combining) have a much higher latency than other operations due to network communication Mohamad Jaber (AUB) Spark May 17, / 43
39 Distribution Apache Spark Distribution introduces important concerns beyond parallelism in the shared memory case (single node/machine) Partial failure: crash failure of a subset of machines Latency: certain operations (combining) have a much higher latency than other operations due to network communication Important Latency Numbers Main memory reference Send 2K bytes over 1Gbps network SSD random read Read 1 MB sequentially from memory Read 1 MB sequentially from SSD Read 1 MB sequentially from disk Send packet US Europe US 100ns 20,000ns 150,000ns 250,000ns 1,000,000ns 20,000,000ns 150,000,000ns Mohamad Jaber (AUB) Spark May 17, / 43
40 Apache Spark Big Data Processing and Latency Network communication and disk operations can be very expensive! How do these latency numbers related to big data processing? Mohamad Jaber (AUB) Spark May 17, / 43
41 Apache Spark Big Data Processing and Latency Network communication and disk operations can be very expensive! How do these latency numbers related to big data processing? To answer this question let us discuss Spark s predecessor, Hadoop Mohamad Jaber (AUB) Spark May 17, / 43
42 Apache Spark Big Data Processing and Latency Network communication and disk operations can be very expensive! How do these latency numbers related to big data processing? To answer this question let us discuss Spark s predecessor, Hadoop Hadoop is a widely-used large scale batch data processing framework It is an open source implementation of Google s MapReduce (2004) Mohamad Jaber (AUB) Spark May 17, / 43
43 Apache Spark Big Data Processing and Latency Network communication and disk operations can be very expensive! How do these latency numbers related to big data processing? To answer this question let us discuss Spark s predecessor, Hadoop Hadoop is a widely-used large scale batch data processing framework It is an open source implementation of Google s MapReduce (2004) Ground breaking because of: (1) simplicity (map and reduce); and (2) fault tolerance Fault tolerance is what made it possible for Hadoop MapReduce to scale up to 1000 nodes (recover from node failure) Mohamad Jaber (AUB) Spark May 17, / 43
44 Hadoop MapReduce Apache Spark MapReduce works by breaking the processing into two phases Each phase has key-value pairs as input and output Mohamad Jaber (AUB) Spark May 17, / 43
45 Hadoop MapReduce Apache Spark MapReduce works by breaking the processing into two phases Each phase has key-value pairs as input and output Map: Grab the relevant data from the source and output intermediate (key, value) pairs (local file system - disk) Reduce: Aggregate the results for each unique key of the generated intermediate (key, value) pairs (HDFS) Mohamad Jaber (AUB) Spark May 17, / 43
46 Why Spark? Apache Spark Fault-tolerance in Hadoop MapReduce comes at a cost Between each map and reduce step, in order to recover from potential failures, Hadoop MapReduce shuffles its data and write intermediate data to disk Mohamad Jaber (AUB) Spark May 17, / 43
47 Why Spark? Apache Spark Fault-tolerance in Hadoop MapReduce comes at a cost Between each map and reduce step, in order to recover from potential failures, Hadoop MapReduce shuffles its data and write intermediate data to disk Cons. of Hadoop MapReduce Not efficient to use the same data multiple times (iterative and interactive) Intermediate results written into stable storage Output of reducers written on HDFS Disk I/O, network I/O, [de]serialization Mohamad Jaber (AUB) Spark May 17, / 43
48 Why Spark? Apache Spark Retains fault-tolerance Different strategy handling latency Mohamad Jaber (AUB) Spark May 17, / 43
49 Why Spark? Apache Spark Retains fault-tolerance Different strategy handling latency Achieves this using ideas from functional programming Keep all data immutable and in-memory All operations on data are just functional transformations Fault tolerance is achieved by replaying function transformations over original dataset Mohamad Jaber (AUB) Spark May 17, / 43
50 Why Spark? Apache Spark Retains fault-tolerance Different strategy handling latency Achieves this using ideas from functional programming Keep all data immutable and in-memory All operations on data are just functional transformations Fault tolerance is achieved by replaying function transformations over original dataset Spark has been shown to be 100x more efficient than Hadoop MapReduce while adding even more expressive APIs! Mohamad Jaber (AUB) Spark May 17, / 43
51 Apache Spark Spark vs Hadoop Performance Mohamad Jaber (AUB) Spark May 17, / 43
52 Apache Spark Spark vs Hadoop Popularity According to Google trends, Spark has surpassed Hadoop in popularity! Mohamad Jaber (AUB) Spark May 17, / 43
53 Spark - RDD Apache Spark Spark extends MapReduce model to better support two common classes analytics applications Iterative algorithms (e.g., machine learning, graph) Interactive: efficiently analyze data sets interactively Mohamad Jaber (AUB) Spark May 17, / 43
54 Spark - RDD Apache Spark Spark extends MapReduce model to better support two common classes analytics applications Iterative algorithms (e.g., machine learning, graph) Interactive: efficiently analyze data sets interactively Spark implements a distributed data parallel model called Resilient Distributed Datasets (RDDs) RDDs look just like immutable sequential or parallel Scala collections RDD is big parallel collection that distributed (in-memory or Disk) across the cluster Mohamad Jaber (AUB) Spark May 17, / 43
55 Spark - RDD Apache Spark Spark extends MapReduce model to better support two common classes analytics applications Iterative algorithms (e.g., machine learning, graph) Interactive: efficiently analyze data sets interactively Spark implements a distributed data parallel model called Resilient Distributed Datasets (RDDs) RDDs look just like immutable sequential or parallel Scala collections RDD is big parallel collection that distributed (in-memory or Disk) across the cluster Spark provides high-level APIs in Java, Scala, Python and R Mohamad Jaber (AUB) Spark May 17, / 43
56 RDD Apache Spark An RDD can be created either from a stable storage (e.g., local, HDFS) or through parallel transformation of another RDD (e.g., map, filter) Mohamad Jaber (AUB) Spark May 17, / 43
57 RDD Apache Spark An RDD can be created either from a stable storage (e.g., local, HDFS) or through parallel transformation of another RDD (e.g., map, filter) It is also possible to execute actions on RDD An action returns single values (not collections) as results (e.g., reduce, count, first) Mohamad Jaber (AUB) Spark May 17, / 43
58 RDD Apache Spark An RDD can be created either from a stable storage (e.g., local, HDFS) or through parallel transformation of another RDD (e.g., map, filter) It is also possible to execute actions on RDD An action returns single values (not collections) as results (e.g., reduce, count, first) RDD can be cashed for efficient (later) use Mohamad Jaber (AUB) Spark May 17, / 43
59 Apache Spark Programming with RDDs - Spark Context Spark Context Main entry point to Spark functionality Available in shell as variable sc scala > val rdd = sc. textfile (" input. txt ") Mohamad Jaber (AUB) Spark May 17, / 43
60 Apache Spark Programming with RDDs - Spark Context Spark Context Main entry point to Spark functionality Available in shell as variable sc scala > val rdd = sc. textfile (" input. txt ") Standalone application v a l conf = new SparkConf (). setappname (" Simple Application "). setmaster (" local [*] ") v a l sc = new SparkContext ( conf ) Mohamad Jaber (AUB) Spark May 17, / 43
61 Create RDDs Apache Spark An RDD can be created either from a stable storage (e.g., local, HDFS): v a l conf = new SparkConf (). setappname (" Simple Application "). setmaster (" local [*] ") v a l sc = new SparkContext ( conf ) v a l logdata = sc. textfile (" hdfs :// hadoop - master /a. txt ", 32) // 32 partitions Mohamad Jaber (AUB) Spark May 17, / 43
62 Create RDDs Apache Spark An RDD can be created either from a stable storage (e.g., local, HDFS): v a l conf = new SparkConf (). setappname (" Simple Application "). setmaster (" local [*] ") v a l sc = new SparkContext ( conf ) v a l logdata = sc. textfile (" hdfs :// hadoop - master /a. txt ", 32) // 32 partitions Or, through parallel transformation of another RDD // remove lines containing the word error v a l logdatafilter = logdata. filter (x =>!x. contains (" error ")) // count the number of words per line v a l countdata = logdatafilter. map (x => x. split ("\\ s+"). count (x => true )) // combined v a l countdata = logdata. filter (x =>!x. contains (" error ")). map (x => x. split ("\\ s+"). count (x => true )) Mohamad Jaber (AUB) Spark May 17, / 43
63 Create RDDs Apache Spark An RDD can be created either from a stable storage (e.g., local, HDFS): v a l conf = new SparkConf (). setappname (" Simple Application "). setmaster (" local [*] ") v a l sc = new SparkContext ( conf ) v a l logdata = sc. textfile (" hdfs :// hadoop - master /a. txt ", 32) // 32 partitions Or, through parallel transformation of another RDD // remove lines containing the word error v a l logdatafilter = logdata. filter (x =>!x. contains (" error ")) // count the number of words per line v a l countdata = logdatafilter. map (x => x. split ("\\ s+"). count (x => true )) // combined v a l countdata = logdata. filter (x =>!x. contains (" error ")). map (x => x. split ("\\ s+"). count (x => true )) Or, you can turn a Scala collection into an RDD v a l rdd = sc. parallelize ( Array (1, 2, 3, 4, 5)) Mohamad Jaber (AUB) Spark May 17, / 43
64 Actions on RDDs Apache Spark v a l nums = sc. parallelize ( List (1, 2, 3)) // Retrieve RDD contents as a local collection nums. collect () // => Array (1, 2, 3) // Return first K elements nums. take (2) // => Array (1, 2) // Count number of elements nums. count () // => 3 // Merge elements with an associative function nums. reduce (_ + _) // => 6 -- equivalent to nums. reduce ((x,y) => x + y) // Write elements to a text file nums. saveastextfile (" hdfs :// file. txt ") // loop over all elements nums. foreach ( println ) Mohamad Jaber (AUB) Spark May 17, / 43
65 Apache Spark Lazy Operations and Caching All transformations in Spark are lazy They do not compute their results right away They just remember the transformations applied to some base dataset The transformations are only computed when an action requires a result to be returned to the driver program v a l logdata = sc. textfile (" hdfs :// hadoop - master /a. txt ") v a l logdatafilter = logdata. filter (x =>!x. contains (" error ")) v a l countdata = logdatafilter. map (x => x. split ("\\ s+"). count (x => true )) println ( countdata. count ()) countdata = logdatafilter. map (x => x. split ("\\ s+"). count (x => x!= " mohamad ") ) print ( countdata. count ()) // will repeat from the begining Mohamad Jaber (AUB) Spark May 17, / 43
66 Apache Spark Lazy Operations and Caching All transformations in Spark are lazy They do not compute their results right away They just remember the transformations applied to some base dataset The transformations are only computed when an action requires a result to be returned to the driver program Upon execution an action a result will be computed and intermediate RDDs are stored in RAM (if possible) v a l logdata = sc. textfile (" hdfs :// hadoop - master /a. txt ") v a l logdatafilter = logdata. filter (x =>!x. contains (" error ")) v a l countdata = logdatafilter. map (x => x. split ("\\ s+"). count (x => true )) println ( countdata. count ()) countdata = logdatafilter. map (x => x. split ("\\ s+"). count (x => x!= " mohamad ") ) print ( countdata. count ()) // will repeat from the begining Mohamad Jaber (AUB) Spark May 17, / 43
67 Apache Spark Lazy Operations and Caching All transformations in Spark are lazy They do not compute their results right away They just remember the transformations applied to some base dataset The transformations are only computed when an action requires a result to be returned to the driver program Upon execution an action a result will be computed and intermediate RDDs are stored in RAM (if possible) Executing another action would repeat the reconstruction from the beginning v a l logdata = sc. textfile (" hdfs :// hadoop - master /a. txt ") v a l logdatafilter = logdata. filter (x =>!x. contains (" error ")) v a l countdata = logdatafilter. map (x => x. split ("\\ s+"). count (x => true )) println ( countdata. count ()) countdata = logdatafilter. map (x => x. split ("\\ s+"). count (x => x!= " mohamad ") ) print ( countdata. count ()) // will repeat from the begining Mohamad Jaber (AUB) Spark May 17, / 43
68 Apache Spark Lazy Operations and Caching All transformations in Spark are lazy They do not compute their results right away They just remember the transformations applied to some base dataset The transformations are only computed when an action requires a result to be returned to the driver program Upon execution an action a result will be computed and intermediate RDDs are stored in RAM (if possible) Executing another action would repeat the reconstruction from the beginning However, you can cache some RDDs! v a l logdata = sc. textfile (" hdfs :// hadoop - master /a. txt ") v a l logdatafilter = logdata. filter (x =>!x. contains (" error ")) logdatafilter.cache() v a l countdata = logdatafilter. map (x => x. split ("\\ s+"). count (x => true )) println ( countdata. count ()) countdata = logdatafilter. map (x => x. split ("\\ s+"). count (x => x!= " mohamad ") ) print ( countdata. count ()) // will repeat from the begining Mohamad Jaber (AUB) Spark May 17, / 43
69 Pair RDDs Apache Spark Spark s distributed reduce transformations operate on RDDs of key-value pairs // Scala pair v a l pair = (a, b) pair._1 // => a pair._2 // => b // pets is a Pair RDD pets = sc. parallelize ( Array (("cat ", 1), (" dog ", 1), (" cat ", 2))) Mohamad Jaber (AUB) Spark May 17, / 43
70 Pair RDDs Apache Spark Spark s distributed reduce transformations operate on RDDs of key-value pairs // Scala pair v a l pair = (a, b) pair._1 // => a pair._2 // => b // pets is a Pair RDD pets = sc. parallelize ( Array (("cat ", 1), (" dog ", 1), (" cat ", 2))) Some transformations: reducebykey, join, sortbykey, mapvalues v a l data = sc. textfile (" input. txt ") v a l pairdata = data. map (v => { v a l split = v. split ("\\s+") ( split (0), split (1). toint ) }). cache () // automatically implements combiners v a l rdd1 = pairdata. reducebykey ((x,y) => x + y)) // (" cat ", 3), (" dog ", 1) v a l rdd2 = pairdata. groubbykey () // (" cat ", [1, 2]), (" dog ", [1]) v a l rdd3 = pairdata. sortbykey () // (" cat ", 1), (" cat ", 2), (" dog ", 1) Mohamad Jaber (AUB) Spark May 17, / 43
71 Example: Word Count Apache Spark v a l lines = sc. textfile (" input. txt ") v a l counts = lines. flatmap ( line => line. split ("\\ s+")). map ( word => (word, 1)). reducebykey (_ + _) Mohamad Jaber (AUB) Spark May 17, / 43
72 Apache Spark Example: Simple Linear Regression v a l rdddata = sc. textfile (" input ") v a r teta = Math. random () v a l learningrate = v a r i = 0 v a l iterations = 100 v a l rdddataxy = rdddata. map ( item => { v a l itemsplit = item. split (" ") ( itemsplit (0). todouble, itemsplit (1). todouble ) }). cache () f o r (i < 1 to iterations ) { v a l rddinnergradient = rdddataxy. map ( item => 2 * ( teta * item._1 - item._2 ) * item._1) v a l gradient = rddinnergradient. reduce (( v1, v2) => v1 + v2) teta = teta - learningrate * gradient. todouble } Mohamad Jaber (AUB) Spark May 17, / 43
73 Fault Tolerance Apache Spark One option to do fault tolerance is to replicate the data (into multiple nodes) However, this may drastically affect the performance (disk and network I/O) Mohamad Jaber (AUB) Spark May 17, / 43
74 Fault Tolerance Apache Spark One option to do fault tolerance is to replicate the data (into multiple nodes) However, this may drastically affect the performance (disk and network I/O) Spark uses method called lineage Remember how an RDD it was built from a given source Automatically rebuilt on failure Recompute only lost partitions on failures, that is, no cost if nothing fails! Mohamad Jaber (AUB) Spark May 17, / 43
75 Apache Spark Spark Execution Engine - Stages and Tasks sc. textfile (" hdfs :// master - node / input / data "). map (x => (x (0), x)). groupbykey (). mapvalues (f => f. count (x => true )) Mohamad Jad Mary stage1 (M, Mohamad) (J, Jad) (M, Mary) M, (Mohamad, Mary) J, (Jad) stage2 M, 2 J, 1 Mohamad Jaber (AUB) Spark May 17, / 43
76 Apache Spark Spark Execution Engine - Stages and Tasks stage 1 stage 2 RDD1 map map RDD2 DAGScheduler is the scheduling layer of Spark that implements stage-oriented scheduling RDD3 RDD5 filter RDD4 It transforms a logical execution plan to a physical execution plan (using stages) join Stages are submitted as tasks RDD6 map stage 3 When the result generate is independent of any other data then we can pipeline! RDD7 Mohamad Jaber (AUB) Spark May 17, / 43
77 Spark Execution Flow Apache Spark Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (driver) Mohamad Jaber (AUB) Spark May 17, / 43
78 Spark Execution Flow Apache Spark Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (driver) SparkContext can connect to several types of cluster managers either Spark s own standalone cluster manager, Mesos or YARN > spark - submit -- class path. to. your. Class -- master yarn -- deploy - mode cluster \ [ options ] <app jar > [ app options ] Mohamad Jaber (AUB) Spark May 17, / 43
79 Spark Execution Flow Apache Spark Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (driver) SparkContext can connect to several types of cluster managers either Spark s own standalone cluster manager, Mesos or YARN > spark - submit -- class path. to. your. Class -- master yarn -- deploy - mode cluster \ [ options ] <app jar > [ app options ] Once connected, Spark acquires executors on nodes in the cluster Mohamad Jaber (AUB) Spark May 17, / 43
80 Spark Execution Flow Apache Spark Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (driver) SparkContext can connect to several types of cluster managers either Spark s own standalone cluster manager, Mesos or YARN > spark - submit -- class path. to. your. Class -- master yarn -- deploy - mode cluster \ [ options ] <app jar > [ app options ] Once connected, Spark acquires executors on nodes in the cluster Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to the executors Mohamad Jaber (AUB) Spark May 17, / 43
81 Spark Execution Flow Apache Spark Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (driver) SparkContext can connect to several types of cluster managers either Spark s own standalone cluster manager, Mesos or YARN > spark - submit -- class path. to. your. Class -- master yarn -- deploy - mode cluster \ [ options ] <app jar > [ app options ] Once connected, Spark acquires executors on nodes in the cluster Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to the executors Finally, SparkContext sends tasks to the executors to run Mohamad Jaber (AUB) Spark May 17, / 43
82 More about Spark Apache Spark Modules on top of Spark Spark also supports a rich set of higher-level tools GraphX forgraph processing MLLib for machine learning Spark SQL for structured data processing Spark Streaming Geo and Spatial Spark: geographical and spatial data Mohamad Jaber (AUB) Spark May 17, / 43
83 Distributed File System 1 Big Data 2 Apache Spark 3 Distributed File System Mohamad Jaber (AUB) Spark May 17, / 43
84 Distributed File System Distributed File Systems Client/Server-based Distributed File Systems The actual file service is offered/stored by a single machine Network File System (NFS) Andrew File System (AFS) Client 1 Client 2 Client 3 Server HDD Mohamad Jaber (AUB) Spark May 17, / 43
85 Distributed File System Distributed File Systems Client/Server-based Distributed File Systems The actual file service is offered/stored by a single machine Network File System (NFS) Andrew File System (AFS) Client 1 Client 2 Client 3 Server HDD Cluster-based Distributed File Systems Divide files among tens, hundreds, thousands or tens of thousands of machines Server Server Google File System (GFS - appeared in SOSP 2003) Hadoop Distributed File System (HDFS) Client 1 Client 2 Client 3 HDD Server HDD HDD Server HDD Mohamad Jaber (AUB) Spark May 17, / 43
86 Distributed File System Hadoop Distributed File System (HDFS) HDFS (open source) is inspired by GFS file DataNode DataNode DataNode DataNode DataNode FsImage EditLog metadata Mohamad Jaber (AUB) Spark May 17, / 43
87 Distributed File System HDFS Components in Cluster Mohamad Jaber (AUB) Spark May 17, / 43
88 HDFS Commands Distributed File System HDFS provides a shell like and a list of commands are available to interact with it # format the file system hdfs namenode - format # starts namenode and datanode daemons start - dfs.sh # Operations on the HDFS hdfs dfs <args > # Create directory hdfs dfs - mkdir / input # List Files hdfs dfs -ls / # Transfer and store a data file from local systems to the HDFS hdfs dfs -put / home / jaber / file. txt / input # View the data from HDFS using cat command hdfs dfs -cat / input / file. txt # Get the file from HDFS to the local file system hdfs dfs -get / input / file. txt / home / jaber / Desktop Mohamad Jaber (AUB) Spark May 17, / 43
89 Distributed File System Hadoop - Yarn Cluster Mohamad Jaber (AUB) Spark May 17, / 43
Spark Overview. Professor Sasu Tarkoma.
Spark Overview 2015 Professor Sasu Tarkoma www.cs.helsinki.fi Apache Spark Spark is a general-purpose computing framework for iterative tasks API is provided for Java, Scala and Python The model is based
More informationProcessing of big data with Apache Spark
Processing of big data with Apache Spark JavaSkop 18 Aleksandar Donevski AGENDA What is Apache Spark? Spark vs Hadoop MapReduce Application Requirements Example Architecture Application Challenges 2 WHAT
More informationBig data systems 12/8/17
Big data systems 12/8/17 Today Basic architecture Two levels of scheduling Spark overview Basic architecture Cluster Manager Cluster Cluster Manager 64GB RAM 32 cores 64GB RAM 32 cores 64GB RAM 32 cores
More informationMapReduce, Hadoop and Spark. Bompotas Agorakis
MapReduce, Hadoop and Spark Bompotas Agorakis Big Data Processing Most of the computations are conceptually straightforward on a single machine but the volume of data is HUGE Need to use many (1.000s)
More informationApache Spark is a fast and general-purpose engine for large-scale data processing Spark aims at achieving the following goals in the Big data context
1 Apache Spark is a fast and general-purpose engine for large-scale data processing Spark aims at achieving the following goals in the Big data context Generality: diverse workloads, operators, job sizes
More informationAn Introduction to Apache Spark
An Introduction to Apache Spark Anastasios Skarlatidis @anskarl Software Engineer/Researcher IIT, NCSR "Demokritos" Outline Part I: Getting to know Spark Part II: Basic programming Part III: Spark under
More informationCOSC 6339 Big Data Analytics. Introduction to Spark. Edgar Gabriel Fall What is SPARK?
COSC 6339 Big Data Analytics Introduction to Spark Edgar Gabriel Fall 2018 What is SPARK? In-Memory Cluster Computing for Big Data Applications Fixes the weaknesses of MapReduce Iterative applications
More informationSpark, Shark and Spark Streaming Introduction
Spark, Shark and Spark Streaming Introduction Tushar Kale tusharkale@in.ibm.com June 2015 This Talk Introduction to Shark, Spark and Spark Streaming Architecture Deployment Methodology Performance References
More information08/04/2018. RDDs. RDDs are the primary abstraction in Spark RDDs are distributed collections of objects spread across the nodes of a clusters
are the primary abstraction in Spark are distributed collections of objects spread across the nodes of a clusters They are split in partitions Each node of the cluster that is running an application contains
More information2/26/2017. RDDs. RDDs are the primary abstraction in Spark RDDs are distributed collections of objects spread across the nodes of a clusters
are the primary abstraction in Spark are distributed collections of objects spread across the nodes of a clusters They are split in partitions Each node of the cluster that is used to run an application
More informationRDDs are the primary abstraction in Spark RDDs are distributed collections of objects spread across the nodes of a clusters
1 RDDs are the primary abstraction in Spark RDDs are distributed collections of objects spread across the nodes of a clusters They are split in partitions Each node of the cluster that is running an application
More informationCSE 444: Database Internals. Lecture 23 Spark
CSE 444: Database Internals Lecture 23 Spark References Spark is an open source system from Berkeley Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. Matei
More information2/26/2017. Originally developed at the University of California - Berkeley's AMPLab
Apache is a fast and general engine for large-scale data processing aims at achieving the following goals in the Big data context Generality: diverse workloads, operators, job sizes Low latency: sub-second
More informationCloud Computing 2. CSCI 4850/5850 High-Performance Computing Spring 2018
Cloud Computing 2 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning
More informationHadoop File System S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y 11/15/2017
Hadoop File System 1 S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y Moving Computation is Cheaper than Moving Data Motivation: Big Data! What is BigData? - Google
More informationBeyond MapReduce: Apache Spark Antonino Virgillito
Beyond MapReduce: Apache Spark Antonino Virgillito 1 Why Spark? Most of Machine Learning Algorithms are iterative because each iteration can improve the results With Disk based approach each iteration
More informationBatch Processing Basic architecture
Batch Processing Basic architecture in big data systems COS 518: Distributed Systems Lecture 10 Andrew Or, Mike Freedman 2 1 2 64GB RAM 32 cores 64GB RAM 32 cores 64GB RAM 32 cores 64GB RAM 32 cores 3
More informationAnalytic Cloud with. Shelly Garion. IBM Research -- Haifa IBM Corporation
Analytic Cloud with Shelly Garion IBM Research -- Haifa 2014 IBM Corporation Why Spark? Apache Spark is a fast and general open-source cluster computing engine for big data processing Speed: Spark is capable
More informationLecture 11 Hadoop & Spark
Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem
More informationHadoop 2.x Core: YARN, Tez, and Spark. Hortonworks Inc All Rights Reserved
Hadoop 2.x Core: YARN, Tez, and Spark YARN Hadoop Machine Types top-of-rack switches core switch client machines have client-side software used to access a cluster to process data master nodes run Hadoop
More informationIntroduction to Apache Spark. Patrick Wendell - Databricks
Introduction to Apache Spark Patrick Wendell - Databricks What is Spark? Fast and Expressive Cluster Computing Engine Compatible with Apache Hadoop Efficient General execution graphs In-memory storage
More informationBig Data Infrastructures & Technologies
Big Data Infrastructures & Technologies Spark and MLLIB OVERVIEW OF SPARK What is Spark? Fast and expressive cluster computing system interoperable with Apache Hadoop Improves efficiency through: In-memory
More informationIntroduction to Apache Spark
Introduction to Apache Spark Bu eğitim sunumları İstanbul Kalkınma Ajansı nın 2016 yılı Yenilikçi ve Yaratıcı İstanbul Mali Destek Programı kapsamında yürütülmekte olan TR10/16/YNY/0036 no lu İstanbul
More informationChapter 4: Apache Spark
Chapter 4: Apache Spark Lecture Notes Winter semester 2016 / 2017 Ludwig-Maximilians-University Munich PD Dr. Matthias Renz 2015, Based on lectures by Donald Kossmann (ETH Zürich), as well as Jure Leskovec,
More informationCompSci 516: Database Systems
CompSci 516 Database Systems Lecture 12 Map-Reduce and Spark Instructor: Sudeepa Roy Duke CS, Fall 2017 CompSci 516: Database Systems 1 Announcements Practice midterm posted on sakai First prepare and
More informationCloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018
Cloud Computing and Hadoop Distributed File System UCSB CS70, Spring 08 Cluster Computing Motivations Large-scale data processing on clusters Scan 000 TB on node @ 00 MB/s = days Scan on 000-node cluster
More informationOverview. Prerequisites. Course Outline. Course Outline :: Apache Spark Development::
Title Duration : Apache Spark Development : 4 days Overview Spark is a fast and general cluster computing system for Big Data. It provides high-level APIs in Scala, Java, Python, and R, and an optimized
More informationData-intensive computing systems
Data-intensive computing systems University of Verona Computer Science Department Damiano Carra Acknowledgements q Credits Part of the course material is based on slides provided by the following authors
More informationData Platforms and Pattern Mining
Morteza Zihayat Data Platforms and Pattern Mining IBM Corporation About Myself IBM Software Group Big Data Scientist 4Platform Computing, IBM (2014 Now) PhD Candidate (2011 Now) 4Lassonde School of Engineering,
More informationIBM Data Science Experience White paper. SparkR. Transforming R into a tool for big data analytics
IBM Data Science Experience White paper R Transforming R into a tool for big data analytics 2 R Executive summary This white paper introduces R, a package for the R statistical programming language that
More informationAnnouncements. Reading Material. Map Reduce. The Map-Reduce Framework 10/3/17. Big Data. CompSci 516: Database Systems
Announcements CompSci 516 Database Systems Lecture 12 - and Spark Practice midterm posted on sakai First prepare and then attempt! Midterm next Wednesday 10/11 in class Closed book/notes, no electronic
More informationIn-Memory Processing with Apache Spark. Vincent Leroy
In-Memory Processing with Apache Spark Vincent Leroy Sources Resilient Distributed Datasets, Henggang Cui Coursera IntroducBon to Apache Spark, University of California, Databricks Datacenter OrganizaBon
More informationApache Spark Internals
Apache Spark Internals Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Apache Spark Internals 1 / 80 Acknowledgments & Sources Sources Research papers: https://spark.apache.org/research.html Presentations:
More informationApplied Spark. From Concepts to Bitcoin Analytics. Andrew F.
Applied Spark From Concepts to Bitcoin Analytics Andrew F. Hart ahart@apache.org @andrewfhart My Day Job CTO, Pogoseat Upgrade technology for live events 3/28/16 QCON-SP Andrew Hart 2 Additionally Member,
More informationCS Spark. Slides from Matei Zaharia and Databricks
CS 5450 Spark Slides from Matei Zaharia and Databricks Goals uextend the MapReduce model to better support two common classes of analytics apps Iterative algorithms (machine learning, graphs) Interactive
More informationProgramming Systems for Big Data
Programming Systems for Big Data CS315B Lecture 17 Including material from Kunle Olukotun Prof. Aiken CS 315B Lecture 17 1 Big Data We ve focused on parallel programming for computational science There
More informationAn Introduction to Apache Spark
An Introduction to Apache Spark 1 History Developed in 2009 at UC Berkeley AMPLab. Open sourced in 2010. Spark becomes one of the largest big-data projects with more 400 contributors in 50+ organizations
More informationDistributed Filesystem
Distributed Filesystem 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributing Code! Don t move data to workers move workers to the data! - Store data on the local disks of nodes in the
More informationCloud Computing 3. CSCI 4850/5850 High-Performance Computing Spring 2018
Cloud Computing 3 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning
More informationDistributed File Systems II
Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation
More informationAnalytics in Spark. Yanlei Diao Tim Hunter. Slides Courtesy of Ion Stoica, Matei Zaharia and Brooke Wenig
Analytics in Spark Yanlei Diao Tim Hunter Slides Courtesy of Ion Stoica, Matei Zaharia and Brooke Wenig Outline 1. A brief history of Big Data and Spark 2. Technical summary of Spark 3. Unified analytics
More information15.1 Data flow vs. traditional network programming
CME 323: Distributed Algorithms and Optimization, Spring 2017 http://stanford.edu/~rezab/dao. Instructor: Reza Zadeh, Matroid and Stanford. Lecture 15, 5/22/2017. Scribed by D. Penner, A. Shoemaker, and
More informationMapReduce Spark. Some slides are adapted from those of Jeff Dean and Matei Zaharia
MapReduce Spark Some slides are adapted from those of Jeff Dean and Matei Zaharia What have we learnt so far? Distributed storage systems consistency semantics protocols for fault tolerance Paxos, Raft,
More informationBig Data Analytics with Apache Spark. Nastaran Fatemi
Big Data Analytics with Apache Spark Nastaran Fatemi Apache Spark Throughout this part of the course we will use the Apache Spark framework for distributed data-parallel programming. Spark implements a
More informationIntroduction to Apache Spark
1 Introduction to Apache Spark Thomas Ropars thomas.ropars@univ-grenoble-alpes.fr 2017 2 References The content of this lectures is inspired by: The lecture notes of Yann Vernaz. The lecture notes of Vincent
More informationBig Data Programming: an Introduction. Spring 2015, X. Zhang Fordham Univ.
Big Data Programming: an Introduction Spring 2015, X. Zhang Fordham Univ. Outline What the course is about? scope Introduction to big data programming Opportunity and challenge of big data Origin of Hadoop
More informationData Clustering on the Parallel Hadoop MapReduce Model. Dimitrios Verraros
Data Clustering on the Parallel Hadoop MapReduce Model Dimitrios Verraros Overview The purpose of this thesis is to implement and benchmark the performance of a parallel K- means clustering algorithm on
More informationDATA SCIENCE USING SPARK: AN INTRODUCTION
DATA SCIENCE USING SPARK: AN INTRODUCTION TOPICS COVERED Introduction to Spark Getting Started with Spark Programming in Spark Data Science with Spark What next? 2 DATA SCIENCE PROCESS Exploratory Data
More informationAn Introduction to Apache Spark Big Data Madison: 29 July William Red Hat, Inc.
An Introduction to Apache Spark Big Data Madison: 29 July 2014 William Benton @willb Red Hat, Inc. About me At Red Hat for almost 6 years, working on distributed computing Currently contributing to Spark,
More informationCloud Computing & Visualization
Cloud Computing & Visualization Workflows Distributed Computation with Spark Data Warehousing with Redshift Visualization with Tableau #FIUSCIS School of Computing & Information Sciences, Florida International
More informationSpark and Spark SQL. Amir H. Payberah. SICS Swedish ICT. Amir H. Payberah (SICS) Spark and Spark SQL June 29, / 71
Spark and Spark SQL Amir H. Payberah amir@sics.se SICS Swedish ICT Amir H. Payberah (SICS) Spark and Spark SQL June 29, 2016 1 / 71 What is Big Data? Amir H. Payberah (SICS) Spark and Spark SQL June 29,
More informationCloud, Big Data & Linear Algebra
Cloud, Big Data & Linear Algebra Shelly Garion IBM Research -- Haifa 2014 IBM Corporation What is Big Data? 2 Global Data Volume in Exabytes What is Big Data? 2005 2012 2017 3 Global Data Volume in Exabytes
More informationDistributed Computing with Spark and MapReduce
Distributed Computing with Spark and MapReduce Reza Zadeh @Reza_Zadeh http://reza-zadeh.com Traditional Network Programming Message-passing between nodes (e.g. MPI) Very difficult to do at scale:» How
More informationL3: Spark & RDD. CDS Department of Computational and Data Sciences. Department of Computational and Data Sciences
Indian Institute of Science Bangalore, India भ रत य व ज ञ न स स थ न ब गल र, भ रत Department of Computational and Data Sciences L3: Spark & RDD Department of Computational and Data Science, IISc, 2016 This
More informationResearch challenges in data-intensive computing The Stratosphere Project Apache Flink
Research challenges in data-intensive computing The Stratosphere Project Apache Flink Seif Haridi KTH/SICS haridi@kth.se e2e-clouds.org Presented by: Seif Haridi May 2014 Research Areas Data-intensive
More informationToday s content. Resilient Distributed Datasets(RDDs) Spark and its data model
Today s content Resilient Distributed Datasets(RDDs) ------ Spark and its data model Resilient Distributed Datasets: A Fault- Tolerant Abstraction for In-Memory Cluster Computing -- Spark By Matei Zaharia,
More informationResilient Distributed Datasets
Resilient Distributed Datasets A Fault- Tolerant Abstraction for In- Memory Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin,
More informationApache Spark. CS240A T Yang. Some of them are based on P. Wendell s Spark slides
Apache Spark CS240A T Yang Some of them are based on P. Wendell s Spark slides Parallel Processing using Spark+Hadoop Hadoop: Distributed file system that connects machines. Mapreduce: parallel programming
More informationMassive Online Analysis - Storm,Spark
Massive Online Analysis - Storm,Spark presentation by R. Kishore Kumar Research Scholar Department of Computer Science & Engineering Indian Institute of Technology, Kharagpur Kharagpur-721302, India (R
More information2/4/2019 Week 3- A Sangmi Lee Pallickara
Week 3-A-0 2/4/2019 Colorado State University, Spring 2019 Week 3-A-1 CS535 BIG DATA FAQs PART A. BIG DATA TECHNOLOGY 3. DISTRIBUTED COMPUTING MODELS FOR SCALABLE BATCH COMPUTING SECTION 1: MAPREDUCE PA1
More informationIntroduction to Spark
Introduction to Spark Outlines A brief history of Spark Programming with RDDs Transformations Actions A brief history Limitations of MapReduce MapReduce use cases showed two major limitations: Difficulty
More informationBig Data Analytics. C. Distributed Computing Environments / C.2. Resilient Distributed Datasets: Apache Spark. Lars Schmidt-Thieme
Big Data Analytics C. Distributed Computing Environments / C.2. Resilient Distributed Datasets: Apache Spark Lars Schmidt-Thieme Information Systems and Machine Learning Lab (ISMLL) Institute of Computer
More informationCSC 261/461 Database Systems Lecture 24. Spring 2017 MW 3:25 pm 4:40 pm January 18 May 3 Dewey 1101
CSC 261/461 Database Systems Lecture 24 Spring 2017 MW 3:25 pm 4:40 pm January 18 May 3 Dewey 1101 Announcements Term Paper due on April 20 April 23 Project 1 Milestone 4 is out Due on 05/03 But I would
More informationMapReduce Simplified Data Processing on Large Clusters
MapReduce Simplified Data Processing on Large Clusters Amir H. Payberah amir@sics.se Amirkabir University of Technology (Tehran Polytechnic) Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 1 /
More informationCA485 Ray Walshe Google File System
Google File System Overview Google File System is scalable, distributed file system on inexpensive commodity hardware that provides: Fault Tolerance File system runs on hundreds or thousands of storage
More informationWe consider the general additive objective function that we saw in previous lectures: n F (w; x i, y i ) i=1
CME 323: Distributed Algorithms and Optimization, Spring 2015 http://stanford.edu/~rezab/dao. Instructor: Reza Zadeh, Matroid and Stanford. Lecture 13, 5/9/2016. Scribed by Alfredo Láinez, Luke de Oliveira.
More informationSpark. In- Memory Cluster Computing for Iterative and Interactive Applications
Spark In- Memory Cluster Computing for Iterative and Interactive Applications Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker,
More informationTITLE: PRE-REQUISITE THEORY. 1. Introduction to Hadoop. 2. Cluster. Implement sort algorithm and run it using HADOOP
TITLE: Implement sort algorithm and run it using HADOOP PRE-REQUISITE Preliminary knowledge of clusters and overview of Hadoop and its basic functionality. THEORY 1. Introduction to Hadoop The Apache Hadoop
More informationIntroduction to MapReduce
Basics of Cloud Computing Lecture 4 Introduction to MapReduce Satish Srirama Some material adapted from slides by Jimmy Lin, Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed
More informationa Spark in the cloud iterative and interactive cluster computing
a Spark in the cloud iterative and interactive cluster computing Matei Zaharia, Mosharaf Chowdhury, Michael Franklin, Scott Shenker, Ion Stoica UC Berkeley Background MapReduce and Dryad raised level of
More informationMap-Reduce. Marco Mura 2010 March, 31th
Map-Reduce Marco Mura (mura@di.unipi.it) 2010 March, 31th This paper is a note from the 2009-2010 course Strumenti di programmazione per sistemi paralleli e distribuiti and it s based by the lessons of
More informationLambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL. May 2015
Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL May 2015 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved. Notices This document
More informationTopics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples
Hadoop Introduction 1 Topics Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples 2 Big Data Analytics What is Big Data?
More informationMI-PDB, MIE-PDB: Advanced Database Systems
MI-PDB, MIE-PDB: Advanced Database Systems http://www.ksi.mff.cuni.cz/~svoboda/courses/2015-2-mie-pdb/ Lecture 10: MapReduce, Hadoop 26. 4. 2016 Lecturer: Martin Svoboda svoboda@ksi.mff.cuni.cz Author:
More informationRESILIENT DISTRIBUTED DATASETS: A FAULT-TOLERANT ABSTRACTION FOR IN-MEMORY CLUSTER COMPUTING
RESILIENT DISTRIBUTED DATASETS: A FAULT-TOLERANT ABSTRACTION FOR IN-MEMORY CLUSTER COMPUTING Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin,
More informationData processing in Apache Spark
Data processing in Apache Spark Pelle Jakovits 21 October, 2015, Tartu Outline Introduction to Spark Resilient Distributed Datasets (RDD) Data operations RDD transformations Examples Fault tolerance Streaming
More informationCSE 544 Principles of Database Management Systems. Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark
CSE 544 Principles of Database Management Systems Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark Announcements HW2 due this Thursday AWS accounts Any success? Feel
More informationHDFS: Hadoop Distributed File System. CIS 612 Sunnie Chung
HDFS: Hadoop Distributed File System CIS 612 Sunnie Chung What is Big Data?? Bulk Amount Unstructured Introduction Lots of Applications which need to handle huge amount of data (in terms of 500+ TB per
More informationDistributed Computation Models
Distributed Computation Models SWE 622, Spring 2017 Distributed Software Engineering Some slides ack: Jeff Dean HW4 Recap https://b.socrative.com/ Class: SWE622 2 Review Replicating state machines Case
More informationWe are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info
We are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info START DATE : TIMINGS : DURATION : TYPE OF BATCH : FEE : FACULTY NAME : LAB TIMINGS : PH NO: 9963799240, 040-40025423
More informationSummary of Big Data Frameworks Course 2015 Professor Sasu Tarkoma
Summary of Big Data Frameworks Course 2015 Professor Sasu Tarkoma www.cs.helsinki.fi Course Schedule Tuesday 10.3. Introduction and the Big Data Challenge Tuesday 17.3. MapReduce and Spark: Overview Tuesday
More informationSpecialist ICT Learning
Specialist ICT Learning APPLIED DATA SCIENCE AND BIG DATA ANALYTICS GTBD7 Course Description This intensive training course provides theoretical and technical aspects of Data Science and Business Analytics.
More informationSpark Streaming. Guido Salvaneschi
Spark Streaming Guido Salvaneschi 1 Spark Streaming Framework for large scale stream processing Scales to 100s of nodes Can achieve second scale latencies Integrates with Spark s batch and interactive
More informationTop 25 Big Data Interview Questions And Answers
Top 25 Big Data Interview Questions And Answers By: Neeru Jain - Big Data The era of big data has just begun. With more companies inclined towards big data to run their operations, the demand for talent
More informationDistributed Systems 16. Distributed File Systems II
Distributed Systems 16. Distributed File Systems II Paul Krzyzanowski pxk@cs.rutgers.edu 1 Review NFS RPC-based access AFS Long-term caching CODA Read/write replication & disconnected operation DFS AFS
More informationHadoop/MapReduce Computing Paradigm
Hadoop/Reduce Computing Paradigm 1 Large-Scale Data Analytics Reduce computing paradigm (E.g., Hadoop) vs. Traditional database systems vs. Database Many enterprises are turning to Hadoop Especially applications
More informationDell In-Memory Appliance for Cloudera Enterprise
Dell In-Memory Appliance for Cloudera Enterprise Spark Technology Overview and Streaming Workload Use Cases Author: Armando Acosta Hadoop Product Manager/Subject Matter Expert Armando_Acosta@Dell.com/
More informationClustering Documents. Document Retrieval. Case Study 2: Document Retrieval
Case Study 2: Document Retrieval Clustering Documents Machine Learning for Big Data CSE547/STAT548, University of Washington Sham Kakade April, 2017 Sham Kakade 2017 1 Document Retrieval n Goal: Retrieve
More informationAn Overview of Apache Spark
An Overview of Apache Spark CIS 612 Sunnie Chung 2014 MapR Technologies 1 MapReduce Processing Model MapReduce, the parallel data processing paradigm, greatly simplified the analysis of big data using
More informationCS435 Introduction to Big Data FALL 2018 Colorado State University. 10/24/2018 Week 10-B Sangmi Lee Pallickara
10/24/2018 CS435 Introduction to Big Data - FALL 2018 W10B00 CS435 Introduction to Big Data 10/24/2018 CS435 Introduction to Big Data - FALL 2018 W10B1 FAQs Programming Assignment 3 has been posted Recitations
More informationBig Data Hadoop Developer Course Content. Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours
Big Data Hadoop Developer Course Content Who is the target audience? Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours Complete beginners who want to learn Big Data Hadoop Professionals
More informationBig Streaming Data Processing. How to Process Big Streaming Data 2016/10/11. Fraud detection in bank transactions. Anomalies in sensor data
Big Data Big Streaming Data Big Streaming Data Processing Fraud detection in bank transactions Anomalies in sensor data Cat videos in tweets How to Process Big Streaming Data Raw Data Streams Distributed
More informationClustering Documents. Case Study 2: Document Retrieval
Case Study 2: Document Retrieval Clustering Documents Machine Learning for Big Data CSE547/STAT548, University of Washington Sham Kakade April 21 th, 2015 Sham Kakade 2016 1 Document Retrieval Goal: Retrieve
More informationSpark Over RDMA: Accelerate Big Data SC Asia 2018 Ido Shamay Mellanox Technologies
Spark Over RDMA: Accelerate Big Data SC Asia 2018 Ido Shamay 1 Apache Spark - Intro Spark within the Big Data ecosystem Data Sources Data Acquisition / ETL Data Storage Data Analysis / ML Serving 3 Apache
More informationMap Reduce & Hadoop Recommended Text:
Map Reduce & Hadoop Recommended Text: Hadoop: The Definitive Guide Tom White O Reilly 2010 VMware Inc. All rights reserved Big Data! Large datasets are becoming more common The New York Stock Exchange
More informationDistributed Systems. 22. Spark. Paul Krzyzanowski. Rutgers University. Fall 2016
Distributed Systems 22. Spark Paul Krzyzanowski Rutgers University Fall 2016 November 26, 2016 2015-2016 Paul Krzyzanowski 1 Apache Spark Goal: generalize MapReduce Similar shard-and-gather approach to
More informationPage 1. Goals for Today" Background of Cloud Computing" Sources Driving Big Data" CS162 Operating Systems and Systems Programming Lecture 24
Goals for Today" CS162 Operating Systems and Systems Programming Lecture 24 Capstone: Cloud Computing" Distributed systems Cloud Computing programming paradigms Cloud Computing OS December 2, 2013 Anthony
More informationMapReduce review. Spark and distributed data processing. Who am I? Today s Talk. Reynold Xin
Who am I? Reynold Xin Stanford CS347 Guest Lecture Spark and distributed data processing PMC member, Apache Spark Cofounder & Chief Architect, Databricks PhD on leave (ABD), UC Berkeley AMPLab Reynold
More informationIntroduction to Hadoop. Owen O Malley Yahoo!, Grid Team
Introduction to Hadoop Owen O Malley Yahoo!, Grid Team owen@yahoo-inc.com Who Am I? Yahoo! Architect on Hadoop Map/Reduce Design, review, and implement features in Hadoop Working on Hadoop full time since
More informationFast, Interactive, Language-Integrated Cluster Computing
Spark Fast, Interactive, Language-Integrated Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker, Ion Stoica www.spark-project.org
More information