Distributed Systems. CS422/522 Lecture17 17 November 2014

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Distributed Systems. CS422/522 Lecture17 17 November 2014"

Transcription

1 Distributed Systems CS422/522 Lecture17 17 November 2014

2 Lecture Outline Introduction Hadoop Chord

3 What s a distributed system?

4 What s a distributed system? A distributed system is a collection of loosely coupled nodes interconnected by a communication network. In a distributed system, users access remote resources in the same way they access local resources.

5 What s a distributed system? A distributed system is a collection of loosely coupled nodes interconnected by a communication network. In a distributed system, users access remote resources in the same way they access local resources.

6 Why distributed systems?

7 PC Scalability

8 Scalability PC What if one computer is not enough? - Buy a bigger (server-class) computer

9 Scalability PC Server What if one computer is not enough? - Buy a bigger (server-class) computer

10 Scalability PC Server What if one computer is not enough? - Buy a bigger (server-class) computer What if the biggest computer is not enough? - Buy many computers

11 Scalability PC Server Cluster What if one computer is not enough? - Buy a bigger (server-class) computer What if the biggest computer is not enough? - Buy many computers

12 Scalability

13 Rack Scalability

14 Rack Scalability Network switches (connects nodes with each other and with other racks)

15 Rack Scalability Network switches (connects nodes with each other and with other racks) Many nodes/blades (often identical)

16 Rack Scalability Network switches (connects nodes with each other and with other racks) Many nodes/blades (often identical) Storage device(s)

17 Scalability PC Server Cluster What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center

18 Scalability PC Server Cluster What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center

19 Scalability PC Server Cluster Data center What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center

20 Google Data Center in Oregon

21 Google Data Center in Oregon Data centers (size of a football field)

22 Google Data Center in Oregon Data centers (size of a football field) A warehouse-sized computer - A single data center can easily contain 10,000 racks with 100 cores in each rack (1,000,000 cores total)

23 Google Data Centers World Wide

24 Why distributed systems? Resource sharing - E.g., Napster Computation speedup - E.g., Hadoop Reliability - E.g., S3

25 Why distributed systems? Resource sharing - E.g., Napster Computation speedup - E.g., Hadoop Reliability - E.g., S3

26 Why distributed systems? Resource sharing - E.g., Napster Computation speedup - E.g., Hadoop Reliability - E.g., Amazon S3

27 Why distributed systems? Resource sharing - E.g., Napster Computation speedup - E.g., Hadoop Reliability - E.g., S3 Issues?

28 Lecture Outline Introduction Hadoop Chord

29 Lecture Outline Introduction Hadoop Chord

30 What s the Hadoop A project based on GFS and MapReduce - Distribute data across machines - Try to achieve the reliability and scalability An open-source software framework for big data The Google File System - Distributed storage - Distributed processing

31 What s the Hadoop A project based on GFS and MapReduce - Distribute data across machines - Try to achieve the reliability and scalability An open-source software framework for big data - Distributed storage - Distributed processing

32 Why Hadoop?

33 Hadoop Components

34 Hadoop Components NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

35 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

36 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) Data HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

37 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

38 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) Data Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

39 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) Data Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

40 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

41 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework MapReduce Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

42 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework MapReduce JobTracker TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

43 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework MapReduce JobTracker TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

44 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework MapReduce JobTracker TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

45 HDFS HDFS is responsible for storing data - Data is split into blocks and distributed across nodes - Each block is replicated Implementation of HDFS: - Based on Google s GFS - Offers redundant storage for massive amounts of data

46 Getting Data in/out of HDFS Hadoop API: - Use hadoop fs to work with data in HDFS - hadoop fs -copyfromlocal local_dir /hdfs_dir - hadoop fs -copytolocal /hdfs_dir local_dir

47 HDFS Example NameNode: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, NameNode holds metadata for the data files DataNodes hold the actual blocks - Each block is replicated three times When a client wants to read file 5 3 DataNodes: Store Blocks

48 HDFS Example NameNode: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, bar? Client NameNode holds metadata for the data files DataNodes hold the actual blocks - Each block is replicated three times When a client wants to read file 5 3 DataNodes: Store Blocks

49 NameNode: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, HDFS Example bar? < DataNode IPs, 3, 5 > Client NameNode holds metadata for the data files DataNodes hold the actual blocks - Each block is replicated three times When a client wants to read file 5 3 DataNodes: Store Blocks

50 NameNode: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, HDFS Example < DataNode IPs, 3, 5 > Client NameNode holds metadata for the data files DataNodes hold the actual blocks - Each block is replicated three times When a client wants to read file 5 3 DataNodes: Store Blocks

51 Case Study Policy: Assigning blocks based on disk space tiger frog lion monkey python

52 Case Study Policy: Assigning blocks based on disk space Hadoop tiger frog lion monkey python

53 Case Study Policy: Assigning blocks based on disk space Metadata Hadoop tiger frog lion monkey python

54 Case Study Policy: Assigning blocks based on disk space Metadata Hadoop tiger frog lion monkey python

55 Case Study Policy: Assigning blocks based on disk space Metadata Hadoop tiger frog lion monkey python

56 Case Study Policy: Assigning blocks based on disk space Metadata Hadoop tiger frog lion monkey python gator

57 Recall: Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework MapReduce JobTracker TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

58 MapReduce A method distributing a task across nodes: - Each node processes data stored on that node Consists of two phases: - Map - Reduce

59 MapReduce A method distributing a task across nodes: - Each node processes data stored on that node Consists of two phases: - Map - Reduce Data Map Reduce Result (Mapper Input)

60 Features of MapReduce Automatic parallelization and distribution A clean abstraction for programmers - MapReduce programs are usually written in Java MapReduce abstracts all the housekeeping away from the developer: - Developers concentrate on Map and Reduce functions

61 MapReduce Example: Word Count Count the # of occurrences of each word in a large amount of input data Map(input_key, input_value) { foreach word w in input_value: emit(w, 1); }

62 MapReduce Example: Word Count Count the # of occurrences of each word in a large amount of input data Map(input_key, input_value) { foreach word w in input_value: emit(w, 1); } Input to the Mapper (3414, the cat sat on the mat ) (3437, the aardvark sat on the sofa )

63 MapReduce Example: Word Count Count the # of occurrences of each word in a large amount of input data Map(input_key, input_value) { foreach word w in input_value: emit(w, 1); } Input to the Mapper (3414, the cat sat on the mat ) (3437, the aardvark sat on the sofa ) Output from the Mapper ( the, 1), ( cat, 1), ( sat, 1), ( on, 1), ( the, 1), ( mat, 1), ( the, 1), ( aardvark, 1), ( sat, 1), ( on, 1), ( the, 1), ( sofa, 1)

64 Map Phase Mapper Input the cat sat on the mat the aardvark sat on the sofa

65 Map Phase Mapper Output Mapper Input the cat sat on the mat the aardvark sat on the sofa ( the, 1), ( cat, 1), ( sat, 1), ( on, 1), ( the, 1), ( mat, 1), ( the, 1), ( aardvark, 1), ( sat, 1), ( on, 1), ( the, 1), ( sofa, 1)

66 Reducer After the Map, all the intermediate values for a given intermediate key are combined together into a list

67 Reducer After the Map, all the intermediate values for a given intermediate key are combined together into a list Mapper Output ( the, 1), ( cat, 1), ( sat, 1), ( on, 1), ( the, 1), ( mat, 1), ( the, 1), ( aardvark, 1), ( sat, 1), ( on, 1), ( the, 1), ( sofa, 1) Reducer Input aardvark, 1 cat, 1 mat, 1 on [1, 1] sat [1, 1] sofa, 1 the [1, 1, 1, 1]

68 Reducer Add up all the values associated with each intermediate key: Reduce(output_key, intermediate_vals) { set count = 0; foreach v in intermediate_vals: count += v; emit(output_key, count); }

69 Reducer Add up all the values associated with each intermediate key: Reduce(output_key, intermediate_vals) { set count = 0; foreach v in intermediate_vals: count += v; emit(output_key, count); } Output from the Reducer ( the, 4), ( sat, 2), ( on, 2), ( sofa, 1), ( mat, 1), ( cat, 1), ( aardvark, 1)

70 Map + Reduce Mapping Shuffling Reducing ( the, 1), Mapper Input the cat sat on the mat the aardvark sat on the sofa ( cat, 1), ( sat, 1), ( on, 1), ( the, 1), ( mat, 1), ( the, 1), ( aardvark, 1), ( sat, 1), ( on, 1), aardvark, 1 cat, 1 mat, 1 on [1, 1] sat [1, 1] sofa, 1 the [1, 1, 1, 1] aardvark, 1 cat, 1 mat, 1 on, 2 sat, 2 sofa, 1 the, 4 ( the, 1), ( sofa, 1)

71 Why we care about counting words Word count is challenging over massive amounts of data Fundamentals of statistics often are aggregate functions Most aggregation functions have distributive nature MapReduce breaks complex tasks into smaller pieces which can be executed in parallel

72 Deployment NameNode JobTracker [Determines execution plan] Client TaskTracker Map Reduce DataNode TaskTracker Map Reduce DataNode TaskTracker Map Reduce DataNode

73 Discussions

74 Lecture Outline Introduction Hadoop Chord

75 Lecture Outline Introduction Hadoop Chord

76 Why Chord?

77 Chord

78 Chord

79 Chord

80 Chord

81 Chord start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1)

82 Chord start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0

83 Chord start int. succ. 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0

84 Chord start int. succ. 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0

85 Importance: Complexity! start int. succ. 1 [1,2) 1 2 [2,4) [4,0) start int. succ. 2 [2,3) [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0

86 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0

87 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0

88 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0

89 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0

90 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0

91 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 6 5 [5,7) 6 7 [7,3) 0

92 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 7 [7,0) 0 0 [0,2) 0 2 [2,6) start int. succ. 4 [4,5) 6 5 [5,7) 6 7 [7,3) 0

93 Case Study: Node Leaves start int. succ. 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 7 [7,0) 0 0 [0,2) 0 2 [2,6) start int. succ. 4 [4,5) 6 5 [5,7) 6 7 [7,3) 0

94 Case Study: Node Leaves start int. succ. 1 [1,2) 3 2 [2,4) 3 4 [4,0) start int. succ. 7 [7,0) 0 0 [0,2) 0 2 [2,6) start int. succ. 4 [4,5) 6 5 [5,7) 6 7 [7,3) 0

95 Chord Issues?

MapReduce & BigTable

MapReduce & BigTable CPSC 426/526 MapReduce & BigTable Ennan Zhai Computer Science Department Yale University Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis:

More information

Overview. Why MapReduce? What is MapReduce? The Hadoop Distributed File System Cloudera, Inc.

Overview. Why MapReduce? What is MapReduce? The Hadoop Distributed File System Cloudera, Inc. MapReduce and HDFS This presentation includes course content University of Washington Redistributed under the Creative Commons Attribution 3.0 license. All other contents: Overview Why MapReduce? What

More information

Clustering Lecture 8: MapReduce

Clustering Lecture 8: MapReduce Clustering Lecture 8: MapReduce Jing Gao SUNY Buffalo 1 Divide and Conquer Work Partition w 1 w 2 w 3 worker worker worker r 1 r 2 r 3 Result Combine 4 Distributed Grep Very big data Split data Split data

More information

Cloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018

Cloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018 Cloud Computing and Hadoop Distributed File System UCSB CS70, Spring 08 Cluster Computing Motivations Large-scale data processing on clusters Scan 000 TB on node @ 00 MB/s = days Scan on 000-node cluster

More information

CPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University

CPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University CPSC 426/526 Cloud Computing Ennan Zhai Computer Science Department Yale University Recall: Lec-7 In the lec-7, I talked about: - P2P vs Enterprise control - Firewall - NATs - Software defined network

More information

Distributed Systems CS6421

Distributed Systems CS6421 Distributed Systems CS6421 Intro to Distributed Systems and the Cloud Prof. Tim Wood v I teach: Software Engineering, Operating Systems, Sr. Design I like: distributed systems, networks, building cool

More information

CS60021: Scalable Data Mining. Sourangshu Bhattacharya

CS60021: Scalable Data Mining. Sourangshu Bhattacharya CS60021: Scalable Data Mining Sourangshu Bhattacharya In this Lecture: Outline: HDFS Motivation HDFS User commands HDFS System architecture HDFS Implementation details Sourangshu Bhattacharya Computer

More information

Introduction to Hadoop and MapReduce

Introduction to Hadoop and MapReduce Introduction to Hadoop and MapReduce Antonino Virgillito THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Large-scale Computation Traditional solutions for computing large

More information

Hadoop. copyright 2011 Trainologic LTD

Hadoop. copyright 2011 Trainologic LTD Hadoop Hadoop is a framework for processing large amounts of data in a distributed manner. It can scale up to thousands of machines. It provides high-availability. Provides map-reduce functionality. Hides

More information

Parallel Programming Principle and Practice. Lecture 10 Big Data Processing with MapReduce

Parallel Programming Principle and Practice. Lecture 10 Big Data Processing with MapReduce Parallel Programming Principle and Practice Lecture 10 Big Data Processing with MapReduce Outline MapReduce Programming Model MapReduce Examples Hadoop 2 Incredible Things That Happen Every Minute On The

More information

Lecture 11 Hadoop & Spark

Lecture 11 Hadoop & Spark Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem

More information

Big Data Programming: an Introduction. Spring 2015, X. Zhang Fordham Univ.

Big Data Programming: an Introduction. Spring 2015, X. Zhang Fordham Univ. Big Data Programming: an Introduction Spring 2015, X. Zhang Fordham Univ. Outline What the course is about? scope Introduction to big data programming Opportunity and challenge of big data Origin of Hadoop

More information

A brief history on Hadoop

A brief history on Hadoop Hadoop Basics A brief history on Hadoop 2003 - Google launches project Nutch to handle billions of searches and indexing millions of web pages. Oct 2003 - Google releases papers with GFS (Google File System)

More information

Introduction to MapReduce

Introduction to MapReduce Basics of Cloud Computing Lecture 4 Introduction to MapReduce Satish Srirama Some material adapted from slides by Jimmy Lin, Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed

More information

CCA-410. Cloudera. Cloudera Certified Administrator for Apache Hadoop (CCAH)

CCA-410. Cloudera. Cloudera Certified Administrator for Apache Hadoop (CCAH) Cloudera CCA-410 Cloudera Certified Administrator for Apache Hadoop (CCAH) Download Full Version : http://killexams.com/pass4sure/exam-detail/cca-410 Reference: CONFIGURATION PARAMETERS DFS.BLOCK.SIZE

More information

MI-PDB, MIE-PDB: Advanced Database Systems

MI-PDB, MIE-PDB: Advanced Database Systems MI-PDB, MIE-PDB: Advanced Database Systems http://www.ksi.mff.cuni.cz/~svoboda/courses/2015-2-mie-pdb/ Lecture 10: MapReduce, Hadoop 26. 4. 2016 Lecturer: Martin Svoboda svoboda@ksi.mff.cuni.cz Author:

More information

Top 25 Hadoop Admin Interview Questions and Answers

Top 25 Hadoop Admin Interview Questions and Answers Top 25 Hadoop Admin Interview Questions and Answers 1) What daemons are needed to run a Hadoop cluster? DataNode, NameNode, TaskTracker, and JobTracker are required to run Hadoop cluster. 2) Which OS are

More information

Database Applications (15-415)

Database Applications (15-415) Database Applications (15-415) Hadoop Lecture 24, April 23, 2014 Mohammad Hammoud Today Last Session: NoSQL databases Today s Session: Hadoop = HDFS + MapReduce Announcements: Final Exam is on Sunday April

More information

CS370 Operating Systems

CS370 Operating Systems CS370 Operating Systems Colorado State University Yashwant K Malaiya Fall 2017 Lecture 26 File Systems Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 FAQ Cylinders: all the platters?

More information

TITLE: PRE-REQUISITE THEORY. 1. Introduction to Hadoop. 2. Cluster. Implement sort algorithm and run it using HADOOP

TITLE: PRE-REQUISITE THEORY. 1. Introduction to Hadoop. 2. Cluster. Implement sort algorithm and run it using HADOOP TITLE: Implement sort algorithm and run it using HADOOP PRE-REQUISITE Preliminary knowledge of clusters and overview of Hadoop and its basic functionality. THEORY 1. Introduction to Hadoop The Apache Hadoop

More information

Hadoop/MapReduce Computing Paradigm

Hadoop/MapReduce Computing Paradigm Hadoop/Reduce Computing Paradigm 1 Large-Scale Data Analytics Reduce computing paradigm (E.g., Hadoop) vs. Traditional database systems vs. Database Many enterprises are turning to Hadoop Especially applications

More information

Big Data Analytics. Izabela Moise, Evangelos Pournaras, Dirk Helbing

Big Data Analytics. Izabela Moise, Evangelos Pournaras, Dirk Helbing Big Data Analytics Izabela Moise, Evangelos Pournaras, Dirk Helbing Izabela Moise, Evangelos Pournaras, Dirk Helbing 1 Big Data "The world is crazy. But at least it s getting regular analysis." Izabela

More information

Introduction to HDFS and MapReduce

Introduction to HDFS and MapReduce Introduction to HDFS and MapReduce Who Am I - Ryan Tabora - Data Developer at Think Big Analytics - Big Data Consulting - Experience working with Hadoop, HBase, Hive, Solr, Cassandra, etc. 2 Who Am I -

More information

HADOOP FRAMEWORK FOR BIG DATA

HADOOP FRAMEWORK FOR BIG DATA HADOOP FRAMEWORK FOR BIG DATA Mr K. Srinivas Babu 1,Dr K. Rameshwaraiah 2 1 Research Scholar S V University, Tirupathi 2 Professor and Head NNRESGI, Hyderabad Abstract - Data has to be stored for further

More information

Parallel Data Processing with Hadoop/MapReduce. CS140 Tao Yang, 2014

Parallel Data Processing with Hadoop/MapReduce. CS140 Tao Yang, 2014 Parallel Data Processing with Hadoop/MapReduce CS140 Tao Yang, 2014 Overview What is MapReduce? Example with word counting Parallel data processing with MapReduce Hadoop file system More application example

More information

Your First Hadoop App, Step by Step

Your First Hadoop App, Step by Step Learn Hadoop in one evening Your First Hadoop App, Step by Step Martynas 1 Miliauskas @mmiliauskas Your First Hadoop App, Step by Step By Martynas Miliauskas Published in 2013 by Martynas Miliauskas On

More information

Lecture 30: Distributed Map-Reduce using Hadoop and Spark Frameworks

Lecture 30: Distributed Map-Reduce using Hadoop and Spark Frameworks COMP 322: Fundamentals of Parallel Programming Lecture 30: Distributed Map-Reduce using Hadoop and Spark Frameworks Mack Joyner and Zoran Budimlić {mjoyner, zoran}@rice.edu http://comp322.rice.edu COMP

More information

The MapReduce Framework

The MapReduce Framework The MapReduce Framework In Partial fulfilment of the requirements for course CMPT 816 Presented by: Ahmed Abdel Moamen Agents Lab Overview MapReduce was firstly introduced by Google on 2004. MapReduce

More information

Where We Are. Review: Parallel DBMS. Parallel DBMS. Introduction to Data Management CSE 344

Where We Are. Review: Parallel DBMS. Parallel DBMS. Introduction to Data Management CSE 344 Where We Are Introduction to Data Management CSE 344 Lecture 22: MapReduce We are talking about parallel query processing There exist two main types of engines: Parallel DBMSs (last lecture + quick review)

More information

Hadoop and HDFS Overview. Madhu Ankam

Hadoop and HDFS Overview. Madhu Ankam Hadoop and HDFS Overview Madhu Ankam Why Hadoop We are gathering more data than ever Examples of data : Server logs Web logs Financial transactions Analytics Emails and text messages Social media like

More information

International Journal of Advance Engineering and Research Development. A Study: Hadoop Framework

International Journal of Advance Engineering and Research Development. A Study: Hadoop Framework Scientific Journal of Impact Factor (SJIF): e-issn (O): 2348- International Journal of Advance Engineering and Research Development Volume 3, Issue 2, February -2016 A Study: Hadoop Framework Devateja

More information

Informa)on Retrieval and Map- Reduce Implementa)ons. Mohammad Amir Sharif PhD Student Center for Advanced Computer Studies

Informa)on Retrieval and Map- Reduce Implementa)ons. Mohammad Amir Sharif PhD Student Center for Advanced Computer Studies Informa)on Retrieval and Map- Reduce Implementa)ons Mohammad Amir Sharif PhD Student Center for Advanced Computer Studies mas4108@louisiana.edu Map-Reduce: Why? Need to process 100TB datasets On 1 node:

More information

Cloud Computing. Ennan Zhai. Computer Science at Yale University

Cloud Computing. Ennan Zhai. Computer Science at Yale University Cloud Computing Ennan Zhai Computer Science at Yale University ennan.zhai@yale.edu About Final Project About Final Project Important dates before demo session: - Oct 31: Proposal v1.0 - Nov 7: Source code

More information

Introduction to Data Management CSE 344

Introduction to Data Management CSE 344 Introduction to Data Management CSE 344 Lecture 24: MapReduce CSE 344 - Winter 215 1 HW8 MapReduce (Hadoop) w/ declarative language (Pig) Due next Thursday evening Will send out reimbursement codes later

More information

Systems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2013/14

Systems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2013/14 Systems Infrastructure for Data Science Web Science Group Uni Freiburg WS 2013/14 MapReduce & Hadoop The new world of Big Data (programming model) Overview of this Lecture Module Background Cluster File

More information

Introduction to MapReduce

Introduction to MapReduce Basics of Cloud Computing Lecture 4 Introduction to MapReduce Satish Srirama Some material adapted from slides by Jimmy Lin, Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed

More information

CS November 2017

CS November 2017 Distributed Systems 09r. Map-Reduce Programming on AWS/EMR (Part I) Setting Up AWS/EMR Paul Krzyzanowski TA: Long Zhao Rutgers University Fall 2017 November 21, 2017 2017 Paul Krzyzanowski 1 November 21,

More information

Distributed Systems. 09r. Map-Reduce Programming on AWS/EMR (Part I) 2017 Paul Krzyzanowski. TA: Long Zhao Rutgers University Fall 2017

Distributed Systems. 09r. Map-Reduce Programming on AWS/EMR (Part I) 2017 Paul Krzyzanowski. TA: Long Zhao Rutgers University Fall 2017 Distributed Systems 09r. Map-Reduce Programming on AWS/EMR (Part I) Paul Krzyzanowski TA: Long Zhao Rutgers University Fall 2017 November 21, 2017 2017 Paul Krzyzanowski 1 Setting Up AWS/EMR November 21,

More information

Data Platforms and Pattern Mining

Data Platforms and Pattern Mining Morteza Zihayat Data Platforms and Pattern Mining IBM Corporation About Myself IBM Software Group Big Data Scientist 4Platform Computing, IBM (2014 Now) PhD Candidate (2011 Now) 4Lassonde School of Engineering,

More information

CS 345A Data Mining. MapReduce

CS 345A Data Mining. MapReduce CS 345A Data Mining MapReduce Single-node architecture CPU Machine Learning, Statistics Memory Classical Data Mining Disk Commodity Clusters Web data sets can be very large Tens to hundreds of terabytes

More information

Big Data for Engineers Spring Resource Management

Big Data for Engineers Spring Resource Management Ghislain Fourny Big Data for Engineers Spring 2018 7. Resource Management artjazz / 123RF Stock Photo Data Technology Stack User interfaces Querying Data stores Indexing Processing Validation Data models

More information

MapReduce, Hadoop and Spark. Bompotas Agorakis

MapReduce, Hadoop and Spark. Bompotas Agorakis MapReduce, Hadoop and Spark Bompotas Agorakis Big Data Processing Most of the computations are conceptually straightforward on a single machine but the volume of data is HUGE Need to use many (1.000s)

More information

Big Data 7. Resource Management

Big Data 7. Resource Management Ghislain Fourny Big Data 7. Resource Management artjazz / 123RF Stock Photo Data Technology Stack User interfaces Querying Data stores Indexing Processing Validation Data models Syntax Encoding Storage

More information

Cloud Computing. Hwajung Lee. Key Reference: Prof. Jong-Moon Chung s Lecture Notes at Yonsei University

Cloud Computing. Hwajung Lee. Key Reference: Prof. Jong-Moon Chung s Lecture Notes at Yonsei University Cloud Computing Hwajung Lee Key Reference: Prof. Jong-Moon Chung s Lecture Notes at Yonsei University Cloud Computing Cloud Introduction Cloud Service Model Big Data Hadoop MapReduce HDFS (Hadoop Distributed

More information

Lecture 12 DATA ANALYTICS ON WEB SCALE

Lecture 12 DATA ANALYTICS ON WEB SCALE Lecture 12 DATA ANALYTICS ON WEB SCALE Source: The Economist, February 25, 2010 The Data Deluge EIGHTEEN months ago, Li & Fung, a firm that manages supply chains for retailers, saw 100 gigabytes of information

More information

CS6030 Cloud Computing. Acknowledgements. Today s Topics. Intro to Cloud Computing 10/20/15. Ajay Gupta, WMU-CS. WiSe Lab

CS6030 Cloud Computing. Acknowledgements. Today s Topics. Intro to Cloud Computing 10/20/15. Ajay Gupta, WMU-CS. WiSe Lab CS6030 Cloud Computing Ajay Gupta B239, CEAS Computer Science Department Western Michigan University ajay.gupta@wmich.edu 276-3104 1 Acknowledgements I have liberally borrowed these slides and material

More information

CS370 Operating Systems

CS370 Operating Systems CS370 Operating Systems Colorado State University Yashwant K Malaiya Spring 2018 Lecture 24 Mass Storage, HDFS/Hadoop Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 FAQ What 2

More information

Ghislain Fourny. Big Data 6. Massive Parallel Processing (MapReduce)

Ghislain Fourny. Big Data 6. Massive Parallel Processing (MapReduce) Ghislain Fourny Big Data 6. Massive Parallel Processing (MapReduce) So far, we have... Storage as file system (HDFS) 13 So far, we have... Storage as tables (HBase) Storage as file system (HDFS) 14 Data

More information

Announcements. Parallel Data Processing in the 20 th Century. Parallel Join Illustration. Introduction to Database Systems CSE 414

Announcements. Parallel Data Processing in the 20 th Century. Parallel Join Illustration. Introduction to Database Systems CSE 414 Introduction to Database Systems CSE 414 Lecture 17: MapReduce and Spark Announcements Midterm this Friday in class! Review session tonight See course website for OHs Includes everything up to Monday s

More information

BigData and Map Reduce VITMAC03

BigData and Map Reduce VITMAC03 BigData and Map Reduce VITMAC03 1 Motivation Process lots of data Google processed about 24 petabytes of data per day in 2009. A single machine cannot serve all the data You need a distributed system to

More information

Hadoop-PR Hortonworks Certified Apache Hadoop 2.0 Developer (Pig and Hive Developer)

Hadoop-PR Hortonworks Certified Apache Hadoop 2.0 Developer (Pig and Hive Developer) Hortonworks Hadoop-PR000007 Hortonworks Certified Apache Hadoop 2.0 Developer (Pig and Hive Developer) http://killexams.com/pass4sure/exam-detail/hadoop-pr000007 QUESTION: 99 Which one of the following

More information

Vendor: Cloudera. Exam Code: CCA-505. Exam Name: Cloudera Certified Administrator for Apache Hadoop (CCAH) CDH5 Upgrade Exam.

Vendor: Cloudera. Exam Code: CCA-505. Exam Name: Cloudera Certified Administrator for Apache Hadoop (CCAH) CDH5 Upgrade Exam. Vendor: Cloudera Exam Code: CCA-505 Exam Name: Cloudera Certified Administrator for Apache Hadoop (CCAH) CDH5 Upgrade Exam Version: Demo QUESTION 1 You have installed a cluster running HDFS and MapReduce

More information

Systems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2012/13

Systems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2012/13 Systems Infrastructure for Data Science Web Science Group Uni Freiburg WS 2012/13 MapReduce & Hadoop The new world of Big Data (programming model) Overview of this Lecture Module Background Google MapReduce

More information

Hadoop محبوبه دادخواه کارگاه ساالنه آزمایشگاه فناوری وب زمستان 1391

Hadoop محبوبه دادخواه کارگاه ساالنه آزمایشگاه فناوری وب زمستان 1391 Hadoop محبوبه دادخواه کارگاه ساالنه آزمایشگاه فناوری وب زمستان 1391 Outline Big Data Big Data Examples Challenges with traditional storage NoSQL Hadoop HDFS MapReduce Architecture 2 Big Data In information

More information

Ghislain Fourny. Big Data Fall Massive Parallel Processing (MapReduce)

Ghislain Fourny. Big Data Fall Massive Parallel Processing (MapReduce) Ghislain Fourny Big Data Fall 2018 6. Massive Parallel Processing (MapReduce) Let's begin with a field experiment 2 400+ Pokemons, 10 different 3 How many of each??????????? 4 400 distributed to many volunteers

More information

Announcements. Optional Reading. Distributed File System (DFS) MapReduce Process. MapReduce. Database Systems CSE 414. HW5 is due tomorrow 11pm

Announcements. Optional Reading. Distributed File System (DFS) MapReduce Process. MapReduce. Database Systems CSE 414. HW5 is due tomorrow 11pm Announcements HW5 is due tomorrow 11pm Database Systems CSE 414 Lecture 19: MapReduce (Ch. 20.2) HW6 is posted and due Nov. 27 11pm Section Thursday on setting up Spark on AWS Create your AWS account before

More information

Introduction to Data Management CSE 344

Introduction to Data Management CSE 344 Introduction to Data Management CSE 344 Lecture 24: MapReduce CSE 344 - Fall 2016 1 HW8 is out Last assignment! Get Amazon credits now (see instructions) Spark with Hadoop Due next wed CSE 344 - Fall 2016

More information

Hortonworks PR PowerCenter Data Integration 9.x Administrator Specialist.

Hortonworks PR PowerCenter Data Integration 9.x Administrator Specialist. Hortonworks PR000007 PowerCenter Data Integration 9.x Administrator Specialist https://killexams.com/pass4sure/exam-detail/pr000007 QUESTION: 102 When can a reduce class also serve as a combiner without

More information

CS 61C: Great Ideas in Computer Architecture. MapReduce

CS 61C: Great Ideas in Computer Architecture. MapReduce CS 61C: Great Ideas in Computer Architecture MapReduce Guest Lecturer: Justin Hsia 3/06/2013 Spring 2013 Lecture #18 1 Review of Last Lecture Performance latency and throughput Warehouse Scale Computing

More information

MapReduce. U of Toronto, 2014

MapReduce. U of Toronto, 2014 MapReduce U of Toronto, 2014 http://www.google.org/flutrends/ca/ (2012) Average Searches Per Day: 5,134,000,000 2 Motivation Process lots of data Google processed about 24 petabytes of data per day in

More information

April Final Quiz COSC MapReduce Programming a) Explain briefly the main ideas and components of the MapReduce programming model.

April Final Quiz COSC MapReduce Programming a) Explain briefly the main ideas and components of the MapReduce programming model. 1. MapReduce Programming a) Explain briefly the main ideas and components of the MapReduce programming model. MapReduce is a framework for processing big data which processes data in two phases, a Map

More information

Vendor: Cloudera. Exam Code: CCD-410. Exam Name: Cloudera Certified Developer for Apache Hadoop. Version: Demo

Vendor: Cloudera. Exam Code: CCD-410. Exam Name: Cloudera Certified Developer for Apache Hadoop. Version: Demo Vendor: Cloudera Exam Code: CCD-410 Exam Name: Cloudera Certified Developer for Apache Hadoop Version: Demo QUESTION 1 When is the earliest point at which the reduce method of a given Reducer can be called?

More information

Database Systems CSE 414

Database Systems CSE 414 Database Systems CSE 414 Lecture 19: MapReduce (Ch. 20.2) CSE 414 - Fall 2017 1 Announcements HW5 is due tomorrow 11pm HW6 is posted and due Nov. 27 11pm Section Thursday on setting up Spark on AWS Create

More information

Hadoop File System S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y 11/15/2017

Hadoop File System S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y 11/15/2017 Hadoop File System 1 S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y Moving Computation is Cheaper than Moving Data Motivation: Big Data! What is BigData? - Google

More information

Outline. Distributed File System Map-Reduce The Computational Model Map-Reduce Algorithm Evaluation Computing Joins

Outline. Distributed File System Map-Reduce The Computational Model Map-Reduce Algorithm Evaluation Computing Joins MapReduce 1 Outline Distributed File System Map-Reduce The Computational Model Map-Reduce Algorithm Evaluation Computing Joins 2 Outline Distributed File System Map-Reduce The Computational Model Map-Reduce

More information

Mixing and matching virtual and physical HPC clusters. Paolo Anedda

Mixing and matching virtual and physical HPC clusters. Paolo Anedda Mixing and matching virtual and physical HPC clusters Paolo Anedda paolo.anedda@crs4.it HPC 2010 - Cetraro 22/06/2010 1 Outline Introduction Scalability Issues System architecture Conclusions & Future

More information

Chapter 5. The MapReduce Programming Model and Implementation

Chapter 5. The MapReduce Programming Model and Implementation Chapter 5. The MapReduce Programming Model and Implementation - Traditional computing: data-to-computing (send data to computing) * Data stored in separate repository * Data brought into system for computing

More information

Evaluation of Apache Hadoop for parallel data analysis with ROOT

Evaluation of Apache Hadoop for parallel data analysis with ROOT Evaluation of Apache Hadoop for parallel data analysis with ROOT S Lehrack, G Duckeck, J Ebke Ludwigs-Maximilians-University Munich, Chair of elementary particle physics, Am Coulombwall 1, D-85748 Garching,

More information

Introduction to Hadoop. Owen O Malley Yahoo!, Grid Team

Introduction to Hadoop. Owen O Malley Yahoo!, Grid Team Introduction to Hadoop Owen O Malley Yahoo!, Grid Team owen@yahoo-inc.com Who Am I? Yahoo! Architect on Hadoop Map/Reduce Design, review, and implement features in Hadoop Working on Hadoop full time since

More information

Introduction to MapReduce. Adapted from Jimmy Lin (U. Maryland, USA)

Introduction to MapReduce. Adapted from Jimmy Lin (U. Maryland, USA) Introduction to MapReduce Adapted from Jimmy Lin (U. Maryland, USA) Motivation Overview Need for handling big data New programming paradigm Review of functional programming mapreduce uses this abstraction

More information

A Multilevel Secure MapReduce Framework for Cross-Domain Information Sharing in the Cloud

A Multilevel Secure MapReduce Framework for Cross-Domain Information Sharing in the Cloud Calhoun: The NPS Institutional Archive Faculty and Researcher Publications Faculty and Researcher Publications 2013-03 A Multilevel Secure MapReduce Framework for Cross-Domain Information Sharing in the

More information

Distributed File Systems II

Distributed File Systems II Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation

More information

Introduction to Data Management CSE 344

Introduction to Data Management CSE 344 Introduction to Data Management CSE 344 Lecture 26: Parallel Databases and MapReduce CSE 344 - Winter 2013 1 HW8 MapReduce (Hadoop) w/ declarative language (Pig) Cluster will run in Amazon s cloud (AWS)

More information

Scaling Up 1 CSE 6242 / CX Duen Horng (Polo) Chau Georgia Tech. Hadoop, Pig

Scaling Up 1 CSE 6242 / CX Duen Horng (Polo) Chau Georgia Tech. Hadoop, Pig CSE 6242 / CX 4242 Scaling Up 1 Hadoop, Pig Duen Horng (Polo) Chau Georgia Tech Some lectures are partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John Stasko, Christos Faloutsos, Le

More information

Distributed Computation Models

Distributed Computation Models Distributed Computation Models SWE 622, Spring 2017 Distributed Software Engineering Some slides ack: Jeff Dean HW4 Recap https://b.socrative.com/ Class: SWE622 2 Review Replicating state machines Case

More information

An Introduction to Big Data Analysis using Spark

An Introduction to Big Data Analysis using Spark An Introduction to Big Data Analysis using Spark Mohamad Jaber American University of Beirut - Faculty of Arts & Sciences - Department of Computer Science May 17, 2017 Mohamad Jaber (AUB) Spark May 17,

More information

Data Clustering on the Parallel Hadoop MapReduce Model. Dimitrios Verraros

Data Clustering on the Parallel Hadoop MapReduce Model. Dimitrios Verraros Data Clustering on the Parallel Hadoop MapReduce Model Dimitrios Verraros Overview The purpose of this thesis is to implement and benchmark the performance of a parallel K- means clustering algorithm on

More information

MapReduce Design Patterns

MapReduce Design Patterns MapReduce Design Patterns MapReduce Restrictions Any algorithm that needs to be implemented using MapReduce must be expressed in terms of a small number of rigidly defined components that must fit together

More information

18-hdfs-gfs.txt Thu Oct 27 10:05: Notes on Parallel File Systems: HDFS & GFS , Fall 2011 Carnegie Mellon University Randal E.

18-hdfs-gfs.txt Thu Oct 27 10:05: Notes on Parallel File Systems: HDFS & GFS , Fall 2011 Carnegie Mellon University Randal E. 18-hdfs-gfs.txt Thu Oct 27 10:05:07 2011 1 Notes on Parallel File Systems: HDFS & GFS 15-440, Fall 2011 Carnegie Mellon University Randal E. Bryant References: Ghemawat, Gobioff, Leung, "The Google File

More information

Databases 2 (VU) ( / )

Databases 2 (VU) ( / ) Databases 2 (VU) (706.711 / 707.030) MapReduce (Part 3) Mark Kröll ISDS, TU Graz Nov. 27, 2017 Mark Kröll (ISDS, TU Graz) MapReduce Nov. 27, 2017 1 / 42 Outline 1 Problems Suited for Map-Reduce 2 MapReduce:

More information

CS370 Operating Systems

CS370 Operating Systems CS370 Operating Systems Colorado State University Yashwant K Malaiya Spring 2018 Lecture 25 RAIDs, HDFS/Hadoop Slides based on Text by Silberschatz, Galvin, Gagne (not) Various sources 1 1 FAQ Striping:

More information

TP1-2: Analyzing Hadoop Logs

TP1-2: Analyzing Hadoop Logs TP1-2: Analyzing Hadoop Logs Shadi Ibrahim January 26th, 2017 MapReduce has emerged as a leading programming model for data-intensive computing. It was originally proposed by Google to simplify development

More information

Cloud Computing 2. CSCI 4850/5850 High-Performance Computing Spring 2018

Cloud Computing 2. CSCI 4850/5850 High-Performance Computing Spring 2018 Cloud Computing 2 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning

More information

Introduction to Map/Reduce. Kostas Solomos Computer Science Department University of Crete, Greece

Introduction to Map/Reduce. Kostas Solomos Computer Science Department University of Crete, Greece Introduction to Map/Reduce Kostas Solomos Computer Science Department University of Crete, Greece What we will cover What is MapReduce? How does it work? A simple word count example (the Hello World! of

More information

CSE 444: Database Internals. Lecture 23 Spark

CSE 444: Database Internals. Lecture 23 Spark CSE 444: Database Internals Lecture 23 Spark References Spark is an open source system from Berkeley Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. Matei

More information

CSE 544 Principles of Database Management Systems. Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark

CSE 544 Principles of Database Management Systems. Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark CSE 544 Principles of Database Management Systems Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark Announcements HW2 due this Thursday AWS accounts Any success? Feel

More information

50 Must Read Hadoop Interview Questions & Answers

50 Must Read Hadoop Interview Questions & Answers 50 Must Read Hadoop Interview Questions & Answers Whizlabs Dec 29th, 2017 Big Data Are you planning to land a job with big data and data analytics? Are you worried about cracking the Hadoop job interview?

More information

Hadoop Overview. Lars George Director EMEA Services

Hadoop Overview. Lars George Director EMEA Services Hadoop Overview Lars George Director EMEA Services 1 About Me Director EMEA Services @ Cloudera Consulting on Hadoop projects (everywhere) Apache Committer HBase and Whirr O Reilly Author HBase The Definitive

More information

CSE 344 MAY 2 ND MAP/REDUCE

CSE 344 MAY 2 ND MAP/REDUCE CSE 344 MAY 2 ND MAP/REDUCE ADMINISTRIVIA HW5 Due Tonight Practice midterm Section tomorrow Exam review PERFORMANCE METRICS FOR PARALLEL DBMSS Nodes = processors, computers Speedup: More nodes, same data

More information

Introduction to MapReduce

Introduction to MapReduce Introduction to MapReduce Based on slides by Juliana Freire Some slides borrowed from Jimmy Lin, Jeff Ullman, Jerome Simeon, and Jure Leskovec Required Reading! Data-Intensive Text Processing with MapReduce,

More information

Introduction to MapReduce. Instructor: Dr. Weikuan Yu Computer Sci. & Software Eng.

Introduction to MapReduce. Instructor: Dr. Weikuan Yu Computer Sci. & Software Eng. Introduction to MapReduce Instructor: Dr. Weikuan Yu Computer Sci. & Software Eng. Before MapReduce Large scale data processing was difficult! Managing hundreds or thousands of processors Managing parallelization

More information

Facilitating Consistency Check between Specification & Implementation with MapReduce Framework

Facilitating Consistency Check between Specification & Implementation with MapReduce Framework Facilitating Consistency Check between Specification & Implementation with MapReduce Framework Shigeru KUSAKABE, Yoichi OMORI, Keijiro ARAKI Kyushu University, Japan 2 Our expectation Light-weight formal

More information

Topics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples

Topics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples Hadoop Introduction 1 Topics Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples 2 Big Data Analytics What is Big Data?

More information

ExamTorrent. Best exam torrent, excellent test torrent, valid exam dumps are here waiting for you

ExamTorrent.   Best exam torrent, excellent test torrent, valid exam dumps are here waiting for you ExamTorrent http://www.examtorrent.com Best exam torrent, excellent test torrent, valid exam dumps are here waiting for you Exam : Apache-Hadoop-Developer Title : Hadoop 2.0 Certification exam for Pig

More information

itpass4sure Helps you pass the actual test with valid and latest training material.

itpass4sure   Helps you pass the actual test with valid and latest training material. itpass4sure http://www.itpass4sure.com/ Helps you pass the actual test with valid and latest training material. Exam : CCD-410 Title : Cloudera Certified Developer for Apache Hadoop (CCDH) Vendor : Cloudera

More information

CS 61C: Great Ideas in Computer Architecture (Machine Structures) Warehouse-Scale Computing, MapReduce, and Spark

CS 61C: Great Ideas in Computer Architecture (Machine Structures) Warehouse-Scale Computing, MapReduce, and Spark CS 61C: Great Ideas in Computer Architecture (Machine Structures) Warehouse-Scale Computing, MapReduce, and Spark Instructors: Bernard Boser & Randy H. Katz http://inst.eecs.berkeley.edu/~cs61c/ 11/11/16

More information

Dr. Chuck Cartledge. 4 Feb. 2015

Dr. Chuck Cartledge. 4 Feb. 2015 CS-495/595 Hadoop (part 1) Lecture #3 Dr. Chuck Cartledge 4 Feb. 2015 1/23 Table of contents I 1 Miscellanea 2 Assignment 3 The Book 4 Chapter 1 5 Chapter 2 7 Break 8 Assignment #2 9 Conclusion 10 References

More information