Hadoop/MapReduce Computing Paradigm

Transcription

1 Hadoop/Reduce Computing Paradigm 1

2 Large-Scale Data Analytics Reduce computing paradigm (E.g., Hadoop) vs. Traditional database systems vs. Database Many enterprises are turning to Hadoop Especially applications generating big data Web applications, social networks, scientific applications 2

3 Why Hadoop is able to compete? vs. Database Scalability (petabytes of data, thousands of machines) Flexibility in accepting all data formats (no schema) Efficient and simple faulttolerant mechanism Performance (tons of indexing, tuning, data organization tech.) Features: - Provenance tracking - Annotation management -. Commodity inexpensive hardware 3

4 What is Hadoop Hadoop is a software framework for distributed processing of large datasets across large clusters of computers Large datasets Terabytes or petabytes of data Large clusters hundreds or thousands of nodes Hadoop is open-source implementation for Google Reduce Hadoop is based on a simple programming model called Reduce Hadoop is based on a simple data model, any data will fit 4

5 What is Hadoop (Cont d) Hadoop framework consists on two main layers Distributed file system (HDFS) Execution engine (Reduce) 5

6 Hadoop Master/Slave Architecture Hadoop is designed as a master-slave shared-nothing architecture Master node (single node) Many slave nodes 6

7 Design Principles of Hadoop Scalability to large data volumes Need to process big data Need to parallelize computation across thousands of nodes 1000 s of machines, 10,000 s of disks Cost-efficiency Commodity machines (cheap, but unreliable) Large number of low-end cheap machines working in parallel to solve a computing problem This is in contrast to Parallel DBs (small number of high-end expensive machines) Commodity network 7

8 Design Principles of Hadoop Automatic parallelization & distribution Hidden from the end-user Fault tolerance and automatic recovery Nodes/tasks will fail and will recover automatically Clean and simple programming abstraction Users only provide two functions map and reduce 8

9 Typical Hadoop Cluster Aggregation switch Rack switch 40 nodes/rack, nodes in cluster 1 Gbps bandwidth within rack, 8 Gbps out of rack Node specs (Yahoo terasort): 8 x 2GHz cores, 8 GB RAM, 4 disks (= 4 TB?) Image from

10 Typical Hadoop Cluster Image from

11 Challenges 1. Cheap nodes fail, especially if you have many Mean time between failures for 1 node = 3 years Mean time between failures for 1000 nodes = 1 day Solution: Build fault-tolerance into system 2. Commodity network = low bandwidth Solution: Push computation to the data 3. Programming distributed systems is hard Solution: Data-parallel programming model: users write map & reduce functions, system distributes work and handles faults

12 Hadoop Components Distributed file system (HDFS) Single namespace for entire cluster Replicates data 3x for fault-tolerance Reduce framework Executes user jobs specified as map and reduce functions Manages work distribution & fault-tolerance

13 Hadoop Distributed File System Files split into 128MB blocks Blocks replicated across several datanodes (usually 3) Single namenode stores metadata (file names, block locations, etc) Namenode File Optimized for large files, sequential reads Files are append-only Datanodes

14 How Uses Reduce/Hadoop Google: Inventors of Reduce computing paradigm Yahoo: Developing Hadoop open-source of Reduce IBM, Microsoft, Oracle Facebook, Amazon, AOL, NetFlex Many others + universities and research labs 14

15 Reduce Programming Model Data type: key-value records function: (K in, V in ) list(k inter, V inter ) Reduce function: (K inter, list(v inter )) list(k out, V out )

16 Example: Word Count def mapper(line): foreach word in line.split(): output(word, 1) def reducer(key, values): output(key, sum(values))

17 Word Count Execution Input Shuffle & Sort Reduce Output the quick brown fox the fox ate the mouse the, 1 fox, 1 the, 1 the, 1 brown, 1 fox, 1 quick, 1 Reduce brown, 2 fox, 2 how, 1 now, 1 the, 3 how now brown cow how, 1 now, 1 brown, 1 ate, 1 mouse, 1 cow, 1 Reduce ate, 1 cow, 1 mouse, 1 quick, 1

18 Reduce Execution Details Single master controls job execution on multiple slaves pers preferentially placed on same node or same rack as their input block Minimizes network usage pers save outputs to local disk before serving them to reducers Allows recovery if a reducer crashes Allows having more reducers than nodes

19 An Optimization: The Combiner A combiner is a local aggregation function for repeated keys produced by same map Works for associative functions like sum, count, max Decreases size of intermediate data Example: map-side aggregation for Word Count: def combiner(key, values): output(key, sum(values))

20 Word Count with Combiner Input & Combine Shuffle & Sort Reduce Output the quick brown fox the, 1 brown, 1 fox, 1 Reduce brown, 2 fox, 2 how, 1 the fox ate the mouse the, 2 fox, 1 quick, 1 now, 1 the, 3 how now brown cow how, 1 now, 1 brown, 1 ate, 1 mouse, 1 cow, 1 Reduce ate, 1 cow, 1 mouse, 1 quick, 1

21 Hadoop: How it Works 21

22 Hadoop Architecture Distributed file system (HDFS) Execution engine (Reduce) Master node (single node) Many slave nodes 22

23 Hadoop Distributed File System (HDFS) Centralized namenode - Maintains metadata info about files File F Blocks (64 MB) Many datanode (1000s) - Store the actual data - Files are divided into blocks - Each block is replicated N times (Default = 3) 23

24 Main Properties of HDFS Large: A HDFS instance may consist of thousands of server machines, each storing part of the file system s data Replication: Each data block is replicated many times (default is 3) Failure: Failure is the norm rather than exception Fault Tolerance: Detection of faults and quick, automatic recovery from them is a core architectural goal of HDFS Namenode is consistently checking Datanodes 24

25 -Reduce Execution Engine (Example: Color Count) Input blocks on HDFS Produces (k, v) (, 1) Shuffle & Sorting based on k Consumes(k, [v]) (, [1,1,1,1,1,1..]) Parse-hash Produces(k, v ) (, 100) Reduce Parse-hash Reduce Parse-hash Reduce Parse-hash Users only provide the and Reduce functions 25

26 Properties of Reduce Engine Job Tracker is the master node (runs with the namenode) Receives the user s job Decides on how many tasks will run (number of mappers) Decides on where to run each mapper (concept of locality) Node 1 Node 2 Node 3 This file has 5 Blocks run 5 map tasks Where to run the task reading block 1 Try to run it on Node 1 or Node 3 26

27 Properties of Reduce Engine (Cont d) Task Tracker is the slave node (runs on each datanode) Receives the task from Job Tracker Runs the task until completion (either map or reduce task) Always in communication with the Job Tracker reporting progress Parse-hash Reduce Parse-hash Parse-hash Reduce In this example, 1 map-reduce job consists of 4 map tasks and 3 reduce tasks Reduce Parse-hash 27

28 Key-Value Pairs pers and Reducers are users code (provided functions) Just need to obey the Key-Value pairs interface pers: Consume <key, value> pairs Produce <key, value> pairs Reducers: Consume <key, <list of values>> Produce <key, value> Shuffling and Sorting: Hidden phase between mappers and reducers Groups all similar keys from all mappers, sorts and passes them to a certain reducer in the form of <key, <list of values>> 28

29 Reduce Phases Deciding on what will be the key and what will be the value developer s responsibility 29

30 Example 1: Word Count Job: Count the occurrences of each word in a data set Tasks Reduce Tasks 30

31 Example 2: Color Count Job: Count the number of each color in a data set Input blocks on HDFS Produces (k, v) (, 1) Shuffle & Sorting based on k Consumes(k, [v]) (, [1,1,1,1,1,1..]) Parse-hash Reduce Produces(k, v ) (, 100) Part0001 Parse-hash Reduce Part0002 Parse-hash Reduce Part0003 Parse-hash 31 That s the output file, it has 3 parts on probably 3 different machines

32 Example 3: Color Filter Job: Select only the blue and the green colors Input blocks on HDFS Produces (k, v) (, 1) Write to HDFS Part0001 Each map task will select only the blue or green colors No need for reduce phase Write to HDFS Part0002 Write to HDFS Part0003 That s the output file, it has 4 parts on probably 4 different machines Write to HDFS Part

33 Bigger Picture: Hadoop vs. Other Systems Distributed Databases Computing Model - Notion of transactions - Transaction is the unit of work - ACID properties, Concurrency control Data Model - Structured data with known schema - Read/Write mode Hadoop - Notion of jobs - Job is the unit of work - No concurrency control - Any data will fit in any format - (un)(semi)structured - ReadOnly mode Cost Model - Expensive servers - Cheap commodity machines Fault Tolerance - Failures are rare - Recovery mechanisms - Failures are common over thousands of machines - Simple yet efficient fault tolerance Key Characteristics - Efficiency, optimizations, fine-tuning - Scalability, flexibility, fault tolerance Cloud Computing A computing model where any computing infrastructure can run on the cloud Hardware & Software are provided as remote services Elastic: grows and shrinks based on the user s demand Example: Amazon EC2 33