Distributed Systems. CS422/522 Lecture17 17 November 2014
|
|
|
- Delphia Foster
- 5 years ago
- Views:
Transcription
1 Distributed Systems CS422/522 Lecture17 17 November 2014
2 Lecture Outline Introduction Hadoop Chord
3 What s a distributed system?
4 What s a distributed system? A distributed system is a collection of loosely coupled nodes interconnected by a communication network. In a distributed system, users access remote resources in the same way they access local resources.
5 What s a distributed system? A distributed system is a collection of loosely coupled nodes interconnected by a communication network. In a distributed system, users access remote resources in the same way they access local resources.
6 Why distributed systems?
7 PC Scalability
8 Scalability PC What if one computer is not enough? - Buy a bigger (server-class) computer
9 Scalability PC Server What if one computer is not enough? - Buy a bigger (server-class) computer
10 Scalability PC Server What if one computer is not enough? - Buy a bigger (server-class) computer What if the biggest computer is not enough? - Buy many computers
11 Scalability PC Server Cluster What if one computer is not enough? - Buy a bigger (server-class) computer What if the biggest computer is not enough? - Buy many computers
12 Scalability
13 Rack Scalability
14 Rack Scalability Network switches (connects nodes with each other and with other racks)
15 Rack Scalability Network switches (connects nodes with each other and with other racks) Many nodes/blades (often identical)
16 Rack Scalability Network switches (connects nodes with each other and with other racks) Many nodes/blades (often identical) Storage device(s)
17 Scalability PC Server Cluster What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center
18 Scalability PC Server Cluster What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center
19 Scalability PC Server Cluster Data center What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center
20 Google Data Center in Oregon
21 Google Data Center in Oregon Data centers (size of a football field)
22 Google Data Center in Oregon Data centers (size of a football field) A warehouse-sized computer - A single data center can easily contain 10,000 racks with 100 cores in each rack (1,000,000 cores total)
23 Google Data Centers World Wide
24 Why distributed systems? Resource sharing - E.g., Napster Computation speedup - E.g., Hadoop Reliability - E.g., S3
25 Why distributed systems? Resource sharing - E.g., Napster Computation speedup - E.g., Hadoop Reliability - E.g., S3
26 Why distributed systems? Resource sharing - E.g., Napster Computation speedup - E.g., Hadoop Reliability - E.g., Amazon S3
27 Why distributed systems? Resource sharing - E.g., Napster Computation speedup - E.g., Hadoop Reliability - E.g., S3 Issues?
28 Lecture Outline Introduction Hadoop Chord
29 Lecture Outline Introduction Hadoop Chord
30 What s the Hadoop A project based on GFS and MapReduce - Distribute data across machines - Try to achieve the reliability and scalability An open-source software framework for big data The Google File System - Distributed storage - Distributed processing
31 What s the Hadoop A project based on GFS and MapReduce - Distribute data across machines - Try to achieve the reliability and scalability An open-source software framework for big data - Distributed storage - Distributed processing
32 Why Hadoop?
33 Hadoop Components
34 Hadoop Components NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
35 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
36 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) Data HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
37 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
38 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) Data Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
39 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) Data Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
40 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
41 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework MapReduce Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
42 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework MapReduce JobTracker TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
43 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework MapReduce JobTracker TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
44 Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework MapReduce JobTracker TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
45 HDFS HDFS is responsible for storing data - Data is split into blocks and distributed across nodes - Each block is replicated Implementation of HDFS: - Based on Google s GFS - Offers redundant storage for massive amounts of data
46 Getting Data in/out of HDFS Hadoop API: - Use hadoop fs to work with data in HDFS - hadoop fs -copyfromlocal local_dir /hdfs_dir - hadoop fs -copytolocal /hdfs_dir local_dir
47 HDFS Example NameNode: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, NameNode holds metadata for the data files DataNodes hold the actual blocks - Each block is replicated three times When a client wants to read file 5 3 DataNodes: Store Blocks
48 HDFS Example NameNode: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, bar? Client NameNode holds metadata for the data files DataNodes hold the actual blocks - Each block is replicated three times When a client wants to read file 5 3 DataNodes: Store Blocks
49 NameNode: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, HDFS Example bar? < DataNode IPs, 3, 5 > Client NameNode holds metadata for the data files DataNodes hold the actual blocks - Each block is replicated three times When a client wants to read file 5 3 DataNodes: Store Blocks
50 NameNode: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, HDFS Example < DataNode IPs, 3, 5 > Client NameNode holds metadata for the data files DataNodes hold the actual blocks - Each block is replicated three times When a client wants to read file 5 3 DataNodes: Store Blocks
51 Case Study Policy: Assigning blocks based on disk space tiger frog lion monkey python
52 Case Study Policy: Assigning blocks based on disk space Hadoop tiger frog lion monkey python
53 Case Study Policy: Assigning blocks based on disk space Metadata Hadoop tiger frog lion monkey python
54 Case Study Policy: Assigning blocks based on disk space Metadata Hadoop tiger frog lion monkey python
55 Case Study Policy: Assigning blocks based on disk space Metadata Hadoop tiger frog lion monkey python
56 Case Study Policy: Assigning blocks based on disk space Metadata Hadoop tiger frog lion monkey python gator
57 Recall: Hadoop Components Two core components - The Hadoop distributed file system (HDFS) - MapReduce software framework MapReduce JobTracker TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
58 MapReduce A method distributing a task across nodes: - Each node processes data stored on that node Consists of two phases: - Map - Reduce
59 MapReduce A method distributing a task across nodes: - Each node processes data stored on that node Consists of two phases: - Map - Reduce Data Map Reduce Result (Mapper Input)
60 Features of MapReduce Automatic parallelization and distribution A clean abstraction for programmers - MapReduce programs are usually written in Java MapReduce abstracts all the housekeeping away from the developer: - Developers concentrate on Map and Reduce functions
61 MapReduce Example: Word Count Count the # of occurrences of each word in a large amount of input data Map(input_key, input_value) { foreach word w in input_value: emit(w, 1); }
62 MapReduce Example: Word Count Count the # of occurrences of each word in a large amount of input data Map(input_key, input_value) { foreach word w in input_value: emit(w, 1); } Input to the Mapper (3414, the cat sat on the mat ) (3437, the aardvark sat on the sofa )
63 MapReduce Example: Word Count Count the # of occurrences of each word in a large amount of input data Map(input_key, input_value) { foreach word w in input_value: emit(w, 1); } Input to the Mapper (3414, the cat sat on the mat ) (3437, the aardvark sat on the sofa ) Output from the Mapper ( the, 1), ( cat, 1), ( sat, 1), ( on, 1), ( the, 1), ( mat, 1), ( the, 1), ( aardvark, 1), ( sat, 1), ( on, 1), ( the, 1), ( sofa, 1)
64 Map Phase Mapper Input the cat sat on the mat the aardvark sat on the sofa
65 Map Phase Mapper Output Mapper Input the cat sat on the mat the aardvark sat on the sofa ( the, 1), ( cat, 1), ( sat, 1), ( on, 1), ( the, 1), ( mat, 1), ( the, 1), ( aardvark, 1), ( sat, 1), ( on, 1), ( the, 1), ( sofa, 1)
66 Reducer After the Map, all the intermediate values for a given intermediate key are combined together into a list
67 Reducer After the Map, all the intermediate values for a given intermediate key are combined together into a list Mapper Output ( the, 1), ( cat, 1), ( sat, 1), ( on, 1), ( the, 1), ( mat, 1), ( the, 1), ( aardvark, 1), ( sat, 1), ( on, 1), ( the, 1), ( sofa, 1) Reducer Input aardvark, 1 cat, 1 mat, 1 on [1, 1] sat [1, 1] sofa, 1 the [1, 1, 1, 1]
68 Reducer Add up all the values associated with each intermediate key: Reduce(output_key, intermediate_vals) { set count = 0; foreach v in intermediate_vals: count += v; emit(output_key, count); }
69 Reducer Add up all the values associated with each intermediate key: Reduce(output_key, intermediate_vals) { set count = 0; foreach v in intermediate_vals: count += v; emit(output_key, count); } Output from the Reducer ( the, 4), ( sat, 2), ( on, 2), ( sofa, 1), ( mat, 1), ( cat, 1), ( aardvark, 1)
70 Map + Reduce Mapping Shuffling Reducing ( the, 1), Mapper Input the cat sat on the mat the aardvark sat on the sofa ( cat, 1), ( sat, 1), ( on, 1), ( the, 1), ( mat, 1), ( the, 1), ( aardvark, 1), ( sat, 1), ( on, 1), aardvark, 1 cat, 1 mat, 1 on [1, 1] sat [1, 1] sofa, 1 the [1, 1, 1, 1] aardvark, 1 cat, 1 mat, 1 on, 2 sat, 2 sofa, 1 the, 4 ( the, 1), ( sofa, 1)
71 Why we care about counting words Word count is challenging over massive amounts of data Fundamentals of statistics often are aggregate functions Most aggregation functions have distributive nature MapReduce breaks complex tasks into smaller pieces which can be executed in parallel
72 Deployment NameNode JobTracker [Determines execution plan] Client TaskTracker Map Reduce DataNode TaskTracker Map Reduce DataNode TaskTracker Map Reduce DataNode
73 Discussions
74 Lecture Outline Introduction Hadoop Chord
75 Lecture Outline Introduction Hadoop Chord
76 Why Chord?
77 Chord
78 Chord
79 Chord
80 Chord
81 Chord start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1)
82 Chord start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0
83 Chord start int. succ. 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0
84 Chord start int. succ. 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0
85 Importance: Complexity! start int. succ. 1 [1,2) 1 2 [2,4) [4,0) start int. succ. 2 [2,3) [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0
86 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0
87 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0
88 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0
89 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0
90 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 0 5 [5,7) 0 7 [7,3) 0
91 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 4 [4,5) 6 5 [5,7) 6 7 [7,3) 0
92 start int. succ. Node Joins 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ. 2 [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 7 [7,0) 0 0 [0,2) 0 2 [2,6) start int. succ. 4 [4,5) 6 5 [5,7) 6 7 [7,3) 0
93 Case Study: Node Leaves start int. succ. 1 [1,2) 1 2 [2,4) 3 4 [4,0) start int. succ [2,3) 3 3 [3,5) 3 5 [5,1) start int. succ. 7 [7,0) 0 0 [0,2) 0 2 [2,6) start int. succ. 4 [4,5) 6 5 [5,7) 6 7 [7,3) 0
94 Case Study: Node Leaves start int. succ. 1 [1,2) 3 2 [2,4) 3 4 [4,0) start int. succ. 7 [7,0) 0 0 [0,2) 0 2 [2,6) start int. succ. 4 [4,5) 6 5 [5,7) 6 7 [7,3) 0
95 Chord Issues?
MapReduce & BigTable
CPSC 426/526 MapReduce & BigTable Ennan Zhai Computer Science Department Yale University Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis:
Overview. Why MapReduce? What is MapReduce? The Hadoop Distributed File System Cloudera, Inc.
MapReduce and HDFS This presentation includes course content University of Washington Redistributed under the Creative Commons Attribution 3.0 license. All other contents: Overview Why MapReduce? What
Clustering Lecture 8: MapReduce
Clustering Lecture 8: MapReduce Jing Gao SUNY Buffalo 1 Divide and Conquer Work Partition w 1 w 2 w 3 worker worker worker r 1 r 2 r 3 Result Combine 4 Distributed Grep Very big data Split data Split data
Cloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018
Cloud Computing and Hadoop Distributed File System UCSB CS70, Spring 08 Cluster Computing Motivations Large-scale data processing on clusters Scan 000 TB on node @ 00 MB/s = days Scan on 000-node cluster
CPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University
CPSC 426/526 Cloud Computing Ennan Zhai Computer Science Department Yale University Recall: Lec-7 In the lec-7, I talked about: - P2P vs Enterprise control - Firewall - NATs - Software defined network
Distributed Systems CS6421
Distributed Systems CS6421 Intro to Distributed Systems and the Cloud Prof. Tim Wood v I teach: Software Engineering, Operating Systems, Sr. Design I like: distributed systems, networks, building cool
CS60021: Scalable Data Mining. Sourangshu Bhattacharya
CS60021: Scalable Data Mining Sourangshu Bhattacharya In this Lecture: Outline: HDFS Motivation HDFS User commands HDFS System architecture HDFS Implementation details Sourangshu Bhattacharya Computer
Introduction to Hadoop and MapReduce
Introduction to Hadoop and MapReduce Antonino Virgillito THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Large-scale Computation Traditional solutions for computing large
Hadoop. copyright 2011 Trainologic LTD
Hadoop Hadoop is a framework for processing large amounts of data in a distributed manner. It can scale up to thousands of machines. It provides high-availability. Provides map-reduce functionality. Hides
Parallel Programming Principle and Practice. Lecture 10 Big Data Processing with MapReduce
Parallel Programming Principle and Practice Lecture 10 Big Data Processing with MapReduce Outline MapReduce Programming Model MapReduce Examples Hadoop 2 Incredible Things That Happen Every Minute On The
Lecture 11 Hadoop & Spark
Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem
Big Data Programming: an Introduction. Spring 2015, X. Zhang Fordham Univ.
Big Data Programming: an Introduction Spring 2015, X. Zhang Fordham Univ. Outline What the course is about? scope Introduction to big data programming Opportunity and challenge of big data Origin of Hadoop
A brief history on Hadoop
Hadoop Basics A brief history on Hadoop 2003 - Google launches project Nutch to handle billions of searches and indexing millions of web pages. Oct 2003 - Google releases papers with GFS (Google File System)
Introduction to MapReduce
Basics of Cloud Computing Lecture 4 Introduction to MapReduce Satish Srirama Some material adapted from slides by Jimmy Lin, Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed
CCA-410. Cloudera. Cloudera Certified Administrator for Apache Hadoop (CCAH)
Cloudera CCA-410 Cloudera Certified Administrator for Apache Hadoop (CCAH) Download Full Version : http://killexams.com/pass4sure/exam-detail/cca-410 Reference: CONFIGURATION PARAMETERS DFS.BLOCK.SIZE
MI-PDB, MIE-PDB: Advanced Database Systems
MI-PDB, MIE-PDB: Advanced Database Systems http://www.ksi.mff.cuni.cz/~svoboda/courses/2015-2-mie-pdb/ Lecture 10: MapReduce, Hadoop 26. 4. 2016 Lecturer: Martin Svoboda svoboda@ksi.mff.cuni.cz Author:
Top 25 Hadoop Admin Interview Questions and Answers
Top 25 Hadoop Admin Interview Questions and Answers 1) What daemons are needed to run a Hadoop cluster? DataNode, NameNode, TaskTracker, and JobTracker are required to run Hadoop cluster. 2) Which OS are
Database Applications (15-415)
Database Applications (15-415) Hadoop Lecture 24, April 23, 2014 Mohammad Hammoud Today Last Session: NoSQL databases Today s Session: Hadoop = HDFS + MapReduce Announcements: Final Exam is on Sunday April
CS370 Operating Systems
CS370 Operating Systems Colorado State University Yashwant K Malaiya Fall 2017 Lecture 26 File Systems Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 FAQ Cylinders: all the platters?
TITLE: PRE-REQUISITE THEORY. 1. Introduction to Hadoop. 2. Cluster. Implement sort algorithm and run it using HADOOP
TITLE: Implement sort algorithm and run it using HADOOP PRE-REQUISITE Preliminary knowledge of clusters and overview of Hadoop and its basic functionality. THEORY 1. Introduction to Hadoop The Apache Hadoop
Hadoop/MapReduce Computing Paradigm
Hadoop/Reduce Computing Paradigm 1 Large-Scale Data Analytics Reduce computing paradigm (E.g., Hadoop) vs. Traditional database systems vs. Database Many enterprises are turning to Hadoop Especially applications
Big Data Analytics. Izabela Moise, Evangelos Pournaras, Dirk Helbing
Big Data Analytics Izabela Moise, Evangelos Pournaras, Dirk Helbing Izabela Moise, Evangelos Pournaras, Dirk Helbing 1 Big Data "The world is crazy. But at least it s getting regular analysis." Izabela
Introduction to HDFS and MapReduce
Introduction to HDFS and MapReduce Who Am I - Ryan Tabora - Data Developer at Think Big Analytics - Big Data Consulting - Experience working with Hadoop, HBase, Hive, Solr, Cassandra, etc. 2 Who Am I -
HADOOP FRAMEWORK FOR BIG DATA
HADOOP FRAMEWORK FOR BIG DATA Mr K. Srinivas Babu 1,Dr K. Rameshwaraiah 2 1 Research Scholar S V University, Tirupathi 2 Professor and Head NNRESGI, Hyderabad Abstract - Data has to be stored for further
Parallel Data Processing with Hadoop/MapReduce. CS140 Tao Yang, 2014
Parallel Data Processing with Hadoop/MapReduce CS140 Tao Yang, 2014 Overview What is MapReduce? Example with word counting Parallel data processing with MapReduce Hadoop file system More application example
Your First Hadoop App, Step by Step
Learn Hadoop in one evening Your First Hadoop App, Step by Step Martynas 1 Miliauskas @mmiliauskas Your First Hadoop App, Step by Step By Martynas Miliauskas Published in 2013 by Martynas Miliauskas On
Lecture 30: Distributed Map-Reduce using Hadoop and Spark Frameworks
COMP 322: Fundamentals of Parallel Programming Lecture 30: Distributed Map-Reduce using Hadoop and Spark Frameworks Mack Joyner and Zoran Budimlić {mjoyner, zoran}@rice.edu http://comp322.rice.edu COMP
The MapReduce Framework
The MapReduce Framework In Partial fulfilment of the requirements for course CMPT 816 Presented by: Ahmed Abdel Moamen Agents Lab Overview MapReduce was firstly introduced by Google on 2004. MapReduce
Where We Are. Review: Parallel DBMS. Parallel DBMS. Introduction to Data Management CSE 344
Where We Are Introduction to Data Management CSE 344 Lecture 22: MapReduce We are talking about parallel query processing There exist two main types of engines: Parallel DBMSs (last lecture + quick review)
Hadoop and HDFS Overview. Madhu Ankam
Hadoop and HDFS Overview Madhu Ankam Why Hadoop We are gathering more data than ever Examples of data : Server logs Web logs Financial transactions Analytics Emails and text messages Social media like
International Journal of Advance Engineering and Research Development. A Study: Hadoop Framework
Scientific Journal of Impact Factor (SJIF): e-issn (O): 2348- International Journal of Advance Engineering and Research Development Volume 3, Issue 2, February -2016 A Study: Hadoop Framework Devateja
Informa)on Retrieval and Map- Reduce Implementa)ons. Mohammad Amir Sharif PhD Student Center for Advanced Computer Studies
Informa)on Retrieval and Map- Reduce Implementa)ons Mohammad Amir Sharif PhD Student Center for Advanced Computer Studies mas4108@louisiana.edu Map-Reduce: Why? Need to process 100TB datasets On 1 node:
Cloud Computing. Ennan Zhai. Computer Science at Yale University
Cloud Computing Ennan Zhai Computer Science at Yale University ennan.zhai@yale.edu About Final Project About Final Project Important dates before demo session: - Oct 31: Proposal v1.0 - Nov 7: Source code
Introduction to Data Management CSE 344
Introduction to Data Management CSE 344 Lecture 24: MapReduce CSE 344 - Winter 215 1 HW8 MapReduce (Hadoop) w/ declarative language (Pig) Due next Thursday evening Will send out reimbursement codes later
Systems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2013/14
Systems Infrastructure for Data Science Web Science Group Uni Freiburg WS 2013/14 MapReduce & Hadoop The new world of Big Data (programming model) Overview of this Lecture Module Background Cluster File
Introduction to MapReduce
Basics of Cloud Computing Lecture 4 Introduction to MapReduce Satish Srirama Some material adapted from slides by Jimmy Lin, Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed
CS November 2017
Distributed Systems 09r. Map-Reduce Programming on AWS/EMR (Part I) Setting Up AWS/EMR Paul Krzyzanowski TA: Long Zhao Rutgers University Fall 2017 November 21, 2017 2017 Paul Krzyzanowski 1 November 21,
Distributed Systems. 09r. Map-Reduce Programming on AWS/EMR (Part I) 2017 Paul Krzyzanowski. TA: Long Zhao Rutgers University Fall 2017
Distributed Systems 09r. Map-Reduce Programming on AWS/EMR (Part I) Paul Krzyzanowski TA: Long Zhao Rutgers University Fall 2017 November 21, 2017 2017 Paul Krzyzanowski 1 Setting Up AWS/EMR November 21,
Data Platforms and Pattern Mining
Morteza Zihayat Data Platforms and Pattern Mining IBM Corporation About Myself IBM Software Group Big Data Scientist 4Platform Computing, IBM (2014 Now) PhD Candidate (2011 Now) 4Lassonde School of Engineering,
CS 345A Data Mining. MapReduce
CS 345A Data Mining MapReduce Single-node architecture CPU Machine Learning, Statistics Memory Classical Data Mining Disk Commodity Clusters Web data sets can be very large Tens to hundreds of terabytes
Big Data for Engineers Spring Resource Management
Ghislain Fourny Big Data for Engineers Spring 2018 7. Resource Management artjazz / 123RF Stock Photo Data Technology Stack User interfaces Querying Data stores Indexing Processing Validation Data models
MapReduce, Hadoop and Spark. Bompotas Agorakis
MapReduce, Hadoop and Spark Bompotas Agorakis Big Data Processing Most of the computations are conceptually straightforward on a single machine but the volume of data is HUGE Need to use many (1.000s)
Big Data 7. Resource Management
Ghislain Fourny Big Data 7. Resource Management artjazz / 123RF Stock Photo Data Technology Stack User interfaces Querying Data stores Indexing Processing Validation Data models Syntax Encoding Storage
Cloud Computing. Hwajung Lee. Key Reference: Prof. Jong-Moon Chung s Lecture Notes at Yonsei University
Cloud Computing Hwajung Lee Key Reference: Prof. Jong-Moon Chung s Lecture Notes at Yonsei University Cloud Computing Cloud Introduction Cloud Service Model Big Data Hadoop MapReduce HDFS (Hadoop Distributed
Lecture 12 DATA ANALYTICS ON WEB SCALE
Lecture 12 DATA ANALYTICS ON WEB SCALE Source: The Economist, February 25, 2010 The Data Deluge EIGHTEEN months ago, Li & Fung, a firm that manages supply chains for retailers, saw 100 gigabytes of information
CS6030 Cloud Computing. Acknowledgements. Today s Topics. Intro to Cloud Computing 10/20/15. Ajay Gupta, WMU-CS. WiSe Lab
CS6030 Cloud Computing Ajay Gupta B239, CEAS Computer Science Department Western Michigan University ajay.gupta@wmich.edu 276-3104 1 Acknowledgements I have liberally borrowed these slides and material
CS370 Operating Systems
CS370 Operating Systems Colorado State University Yashwant K Malaiya Spring 2018 Lecture 24 Mass Storage, HDFS/Hadoop Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 FAQ What 2
Ghislain Fourny. Big Data 6. Massive Parallel Processing (MapReduce)
Ghislain Fourny Big Data 6. Massive Parallel Processing (MapReduce) So far, we have... Storage as file system (HDFS) 13 So far, we have... Storage as tables (HBase) Storage as file system (HDFS) 14 Data
Announcements. Parallel Data Processing in the 20 th Century. Parallel Join Illustration. Introduction to Database Systems CSE 414
Introduction to Database Systems CSE 414 Lecture 17: MapReduce and Spark Announcements Midterm this Friday in class! Review session tonight See course website for OHs Includes everything up to Monday s
BigData and Map Reduce VITMAC03
BigData and Map Reduce VITMAC03 1 Motivation Process lots of data Google processed about 24 petabytes of data per day in 2009. A single machine cannot serve all the data You need a distributed system to
Hadoop-PR Hortonworks Certified Apache Hadoop 2.0 Developer (Pig and Hive Developer)
Hortonworks Hadoop-PR000007 Hortonworks Certified Apache Hadoop 2.0 Developer (Pig and Hive Developer) http://killexams.com/pass4sure/exam-detail/hadoop-pr000007 QUESTION: 99 Which one of the following
Vendor: Cloudera. Exam Code: CCA-505. Exam Name: Cloudera Certified Administrator for Apache Hadoop (CCAH) CDH5 Upgrade Exam.
Vendor: Cloudera Exam Code: CCA-505 Exam Name: Cloudera Certified Administrator for Apache Hadoop (CCAH) CDH5 Upgrade Exam Version: Demo QUESTION 1 You have installed a cluster running HDFS and MapReduce
Systems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2012/13
Systems Infrastructure for Data Science Web Science Group Uni Freiburg WS 2012/13 MapReduce & Hadoop The new world of Big Data (programming model) Overview of this Lecture Module Background Google MapReduce
Hadoop محبوبه دادخواه کارگاه ساالنه آزمایشگاه فناوری وب زمستان 1391
Hadoop محبوبه دادخواه کارگاه ساالنه آزمایشگاه فناوری وب زمستان 1391 Outline Big Data Big Data Examples Challenges with traditional storage NoSQL Hadoop HDFS MapReduce Architecture 2 Big Data In information
Ghislain Fourny. Big Data Fall Massive Parallel Processing (MapReduce)
Ghislain Fourny Big Data Fall 2018 6. Massive Parallel Processing (MapReduce) Let's begin with a field experiment 2 400+ Pokemons, 10 different 3 How many of each??????????? 4 400 distributed to many volunteers
Announcements. Optional Reading. Distributed File System (DFS) MapReduce Process. MapReduce. Database Systems CSE 414. HW5 is due tomorrow 11pm
Announcements HW5 is due tomorrow 11pm Database Systems CSE 414 Lecture 19: MapReduce (Ch. 20.2) HW6 is posted and due Nov. 27 11pm Section Thursday on setting up Spark on AWS Create your AWS account before
Introduction to Data Management CSE 344
Introduction to Data Management CSE 344 Lecture 24: MapReduce CSE 344 - Fall 2016 1 HW8 is out Last assignment! Get Amazon credits now (see instructions) Spark with Hadoop Due next wed CSE 344 - Fall 2016
Hortonworks PR PowerCenter Data Integration 9.x Administrator Specialist.
Hortonworks PR000007 PowerCenter Data Integration 9.x Administrator Specialist https://killexams.com/pass4sure/exam-detail/pr000007 QUESTION: 102 When can a reduce class also serve as a combiner without
CS 61C: Great Ideas in Computer Architecture. MapReduce
CS 61C: Great Ideas in Computer Architecture MapReduce Guest Lecturer: Justin Hsia 3/06/2013 Spring 2013 Lecture #18 1 Review of Last Lecture Performance latency and throughput Warehouse Scale Computing
MapReduce. U of Toronto, 2014
MapReduce U of Toronto, 2014 http://www.google.org/flutrends/ca/ (2012) Average Searches Per Day: 5,134,000,000 2 Motivation Process lots of data Google processed about 24 petabytes of data per day in
April Final Quiz COSC MapReduce Programming a) Explain briefly the main ideas and components of the MapReduce programming model.
1. MapReduce Programming a) Explain briefly the main ideas and components of the MapReduce programming model. MapReduce is a framework for processing big data which processes data in two phases, a Map
Vendor: Cloudera. Exam Code: CCD-410. Exam Name: Cloudera Certified Developer for Apache Hadoop. Version: Demo
Vendor: Cloudera Exam Code: CCD-410 Exam Name: Cloudera Certified Developer for Apache Hadoop Version: Demo QUESTION 1 When is the earliest point at which the reduce method of a given Reducer can be called?
Database Systems CSE 414
Database Systems CSE 414 Lecture 19: MapReduce (Ch. 20.2) CSE 414 - Fall 2017 1 Announcements HW5 is due tomorrow 11pm HW6 is posted and due Nov. 27 11pm Section Thursday on setting up Spark on AWS Create
Hadoop File System S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y 11/15/2017
Hadoop File System 1 S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y Moving Computation is Cheaper than Moving Data Motivation: Big Data! What is BigData? - Google
Outline. Distributed File System Map-Reduce The Computational Model Map-Reduce Algorithm Evaluation Computing Joins
MapReduce 1 Outline Distributed File System Map-Reduce The Computational Model Map-Reduce Algorithm Evaluation Computing Joins 2 Outline Distributed File System Map-Reduce The Computational Model Map-Reduce
Mixing and matching virtual and physical HPC clusters. Paolo Anedda
Mixing and matching virtual and physical HPC clusters Paolo Anedda paolo.anedda@crs4.it HPC 2010 - Cetraro 22/06/2010 1 Outline Introduction Scalability Issues System architecture Conclusions & Future
Chapter 5. The MapReduce Programming Model and Implementation
Chapter 5. The MapReduce Programming Model and Implementation - Traditional computing: data-to-computing (send data to computing) * Data stored in separate repository * Data brought into system for computing
Evaluation of Apache Hadoop for parallel data analysis with ROOT
Evaluation of Apache Hadoop for parallel data analysis with ROOT S Lehrack, G Duckeck, J Ebke Ludwigs-Maximilians-University Munich, Chair of elementary particle physics, Am Coulombwall 1, D-85748 Garching,
Introduction to Hadoop. Owen O Malley Yahoo!, Grid Team
Introduction to Hadoop Owen O Malley Yahoo!, Grid Team owen@yahoo-inc.com Who Am I? Yahoo! Architect on Hadoop Map/Reduce Design, review, and implement features in Hadoop Working on Hadoop full time since
Introduction to MapReduce. Adapted from Jimmy Lin (U. Maryland, USA)
Introduction to MapReduce Adapted from Jimmy Lin (U. Maryland, USA) Motivation Overview Need for handling big data New programming paradigm Review of functional programming mapreduce uses this abstraction
A Multilevel Secure MapReduce Framework for Cross-Domain Information Sharing in the Cloud
Calhoun: The NPS Institutional Archive Faculty and Researcher Publications Faculty and Researcher Publications 2013-03 A Multilevel Secure MapReduce Framework for Cross-Domain Information Sharing in the
Distributed File Systems II
Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation
Introduction to Data Management CSE 344
Introduction to Data Management CSE 344 Lecture 26: Parallel Databases and MapReduce CSE 344 - Winter 2013 1 HW8 MapReduce (Hadoop) w/ declarative language (Pig) Cluster will run in Amazon s cloud (AWS)
Scaling Up 1 CSE 6242 / CX Duen Horng (Polo) Chau Georgia Tech. Hadoop, Pig
CSE 6242 / CX 4242 Scaling Up 1 Hadoop, Pig Duen Horng (Polo) Chau Georgia Tech Some lectures are partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John Stasko, Christos Faloutsos, Le
Distributed Computation Models
Distributed Computation Models SWE 622, Spring 2017 Distributed Software Engineering Some slides ack: Jeff Dean HW4 Recap https://b.socrative.com/ Class: SWE622 2 Review Replicating state machines Case
An Introduction to Big Data Analysis using Spark
An Introduction to Big Data Analysis using Spark Mohamad Jaber American University of Beirut - Faculty of Arts & Sciences - Department of Computer Science May 17, 2017 Mohamad Jaber (AUB) Spark May 17,
Data Clustering on the Parallel Hadoop MapReduce Model. Dimitrios Verraros
Data Clustering on the Parallel Hadoop MapReduce Model Dimitrios Verraros Overview The purpose of this thesis is to implement and benchmark the performance of a parallel K- means clustering algorithm on
MapReduce Design Patterns
MapReduce Design Patterns MapReduce Restrictions Any algorithm that needs to be implemented using MapReduce must be expressed in terms of a small number of rigidly defined components that must fit together
18-hdfs-gfs.txt Thu Oct 27 10:05: Notes on Parallel File Systems: HDFS & GFS , Fall 2011 Carnegie Mellon University Randal E.
18-hdfs-gfs.txt Thu Oct 27 10:05:07 2011 1 Notes on Parallel File Systems: HDFS & GFS 15-440, Fall 2011 Carnegie Mellon University Randal E. Bryant References: Ghemawat, Gobioff, Leung, "The Google File
Databases 2 (VU) ( / )
Databases 2 (VU) (706.711 / 707.030) MapReduce (Part 3) Mark Kröll ISDS, TU Graz Nov. 27, 2017 Mark Kröll (ISDS, TU Graz) MapReduce Nov. 27, 2017 1 / 42 Outline 1 Problems Suited for Map-Reduce 2 MapReduce:
CS370 Operating Systems
CS370 Operating Systems Colorado State University Yashwant K Malaiya Spring 2018 Lecture 25 RAIDs, HDFS/Hadoop Slides based on Text by Silberschatz, Galvin, Gagne (not) Various sources 1 1 FAQ Striping:
TP1-2: Analyzing Hadoop Logs
TP1-2: Analyzing Hadoop Logs Shadi Ibrahim January 26th, 2017 MapReduce has emerged as a leading programming model for data-intensive computing. It was originally proposed by Google to simplify development
Cloud Computing 2. CSCI 4850/5850 High-Performance Computing Spring 2018
Cloud Computing 2 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning
Introduction to Map/Reduce. Kostas Solomos Computer Science Department University of Crete, Greece
Introduction to Map/Reduce Kostas Solomos Computer Science Department University of Crete, Greece What we will cover What is MapReduce? How does it work? A simple word count example (the Hello World! of
CSE 444: Database Internals. Lecture 23 Spark
CSE 444: Database Internals Lecture 23 Spark References Spark is an open source system from Berkeley Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. Matei
50 Must Read Hadoop Interview Questions & Answers
50 Must Read Hadoop Interview Questions & Answers Whizlabs Dec 29th, 2017 Big Data Are you planning to land a job with big data and data analytics? Are you worried about cracking the Hadoop job interview?
CSE 544 Principles of Database Management Systems. Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark
CSE 544 Principles of Database Management Systems Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark Announcements HW2 due this Thursday AWS accounts Any success? Feel
Hadoop Overview. Lars George Director EMEA Services
Hadoop Overview Lars George Director EMEA Services 1 About Me Director EMEA Services @ Cloudera Consulting on Hadoop projects (everywhere) Apache Committer HBase and Whirr O Reilly Author HBase The Definitive
CSE 344 MAY 2 ND MAP/REDUCE
CSE 344 MAY 2 ND MAP/REDUCE ADMINISTRIVIA HW5 Due Tonight Practice midterm Section tomorrow Exam review PERFORMANCE METRICS FOR PARALLEL DBMSS Nodes = processors, computers Speedup: More nodes, same data
Introduction to MapReduce
Introduction to MapReduce Based on slides by Juliana Freire Some slides borrowed from Jimmy Lin, Jeff Ullman, Jerome Simeon, and Jure Leskovec Required Reading! Data-Intensive Text Processing with MapReduce,
Introduction to MapReduce. Instructor: Dr. Weikuan Yu Computer Sci. & Software Eng.
Introduction to MapReduce Instructor: Dr. Weikuan Yu Computer Sci. & Software Eng. Before MapReduce Large scale data processing was difficult! Managing hundreds or thousands of processors Managing parallelization
Facilitating Consistency Check between Specification & Implementation with MapReduce Framework
Facilitating Consistency Check between Specification & Implementation with MapReduce Framework Shigeru KUSAKABE, Yoichi OMORI, Keijiro ARAKI Kyushu University, Japan 2 Our expectation Light-weight formal
Topics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples
Hadoop Introduction 1 Topics Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples 2 Big Data Analytics What is Big Data?
ExamTorrent. Best exam torrent, excellent test torrent, valid exam dumps are here waiting for you
ExamTorrent http://www.examtorrent.com Best exam torrent, excellent test torrent, valid exam dumps are here waiting for you Exam : Apache-Hadoop-Developer Title : Hadoop 2.0 Certification exam for Pig
itpass4sure Helps you pass the actual test with valid and latest training material.
itpass4sure http://www.itpass4sure.com/ Helps you pass the actual test with valid and latest training material. Exam : CCD-410 Title : Cloudera Certified Developer for Apache Hadoop (CCDH) Vendor : Cloudera
CS 61C: Great Ideas in Computer Architecture (Machine Structures) Warehouse-Scale Computing, MapReduce, and Spark
CS 61C: Great Ideas in Computer Architecture (Machine Structures) Warehouse-Scale Computing, MapReduce, and Spark Instructors: Bernard Boser & Randy H. Katz http://inst.eecs.berkeley.edu/~cs61c/ 11/11/16
Dr. Chuck Cartledge. 4 Feb. 2015
CS-495/595 Hadoop (part 1) Lecture #3 Dr. Chuck Cartledge 4 Feb. 2015 1/23 Table of contents I 1 Miscellanea 2 Assignment 3 The Book 4 Chapter 1 5 Chapter 2 7 Break 8 Assignment #2 9 Conclusion 10 References
Cloudera Exam CCA-410 Cloudera Certified Administrator for Apache Hadoop (CCAH) Version: 7.5 [ Total Questions: 97 ]
![Cloudera Exam CCA-410 Cloudera Certified Administrator for Apache Hadoop (CCAH) Version: 7.5 [ Total Questions: 97 ]](/thumbs/96/126854374.jpg "Cloudera Exam CCA-410 Cloudera Certified Administrator for Apache Hadoop (CCAH) Version: 7.5 [ Total Questions: 97 ]")
s@lm@n Cloudera Exam CCA-410 Cloudera Certified Administrator for Apache Hadoop (CCAH) Version: 7.5 [ Total Questions: 97 ] Question No : 1 Which two updates occur when a client application opens a stream
Map Reduce. Yerevan.
Map Reduce Erasmus+ @ Yerevan dacosta@irit.fr Divide and conquer at PaaS 100 % // Typical problem Iterate over a large number of records Extract something of interest from each Shuffle and sort intermediate
Programming Systems for Big Data
Programming Systems for Big Data CS315B Lecture 17 Including material from Kunle Olukotun Prof. Aiken CS 315B Lecture 17 1 Big Data We ve focused on parallel programming for computational science There