Welcome to. uweseiler

Transcription

1 Welcome to uweseiler

2 Your Travel Guide Big Data Nerd Hadoop Trainer NoSQL Fan Boy Photography Enthusiast Travelpirate

3 Your Travel Agency specializes on... Big Data Nerds Agile Ninjas Continuous Delivery Gurus Join us! Enterprise Java Specialists Performance Geeks

4 Your Travel Destination YARN App Land Enterprise Land Goodbye Photo HDFS Land Welcome Office YARN Land

5 Stop #1: Welcome Office Welcome Office

6 In the beginning of Hadoop there was MapReduce It could handle data sizes way beyond those of its competitors It was resilient in the face of failure It made it easy for users to bring their code and algorithms to the data

7 Hadoop 1 (007) but it was too low level

8 Hadoop 1 (007) but it was too rigid

9 Hadoop 1 (007) but it was Batch Single App Batch Single App Batch Single App Batch Single App Batch Single App Batch HDFS HDFS HDFS

10 Hadoop 1 (007) but it had limitations Scalability Maximum cluster size ~ 4,500 nodes Maximum concurrent tasks 40,000 Coarse synchronization in JobTracker Availability Failure kills all queued and running jobs Hard partition of resources into map & reduce slots Low resource utilization Lacks support for alternate paradigms and services

11 Hadoop (013) YARN to the rescue!

12 Stop #: YARN Land Welcome Office YARN Land

13 A brief history of Hadoop Originally conceived & architected by the team at Yahoo! Arun Murthy created the original JIRA in 008 and now is the Hadoop release manager The community has been working on Hadoop for over 4 years Hadoop based architecture running at scale at Yahoo! Deployed on 35,000+ nodes for 6+ months

14 Single use system Batch Apps Hadoop 1 Hadoop : Next-gen platform Multi-purpose platform Batch, Interactive, Streaming, Hadoop MapReduce Cluster resource mgmt. + data processing HDFS Redundant, reliable storage MapReduce Data processing YARN HDFS Others Data processing Cluster resource management Redundant, reliable storage

15 Taking Hadoop beyond batch Store all data in one place Interact with data in multiple ways Applications run natively in Hadoop Batch MapReduce Interactive Tez Online HOYA Streaming Storm, Graph Giraph In-Memory Spark Other Search, YARN Cluster resource management HDFS Redundant, reliable storage

16 YARN: Design Goals Build a new abstraction layer by splitting up the two major functions of the JobTracker Cluster resource management Application life-cycle management Allow other processing paradigms Flexible API for implementing YARN apps MapReduce becomes YARN app Lots of different YARN apps

17 Hadoop: BigDataOS Traditional Operating System Hadoop Execution / Scheduling: Processes / Kernel Storage: File System Execution / Scheduling: YARN Storage: HDFS

18 YARN: Architectural Overview Split up the two major functions of the JobTracker Cluster resource management & Application life-cycle management ResourceManager Scheduler NodeManager NodeManager NodeManager NodeManager AM 1 Container 1.1 Container.3 Container.1 NodeManager NodeManager NodeManager NodeManager Container 1. AM Container.

19 YARN: Multi-tenancy I ResourceManager Different types of applications on the same cluster Scheduler NodeManager NodeManager NodeManager NodeManager MapReduce 1 map 1.1 reduce. map.1 Region server reduce.1 nimbus 1 vertex 3 NodeManager NodeManager NodeManager NodeManager HBase Master map 1. MapReduce map. nimbus Region server 1 vertex 4 vertex NodeManager NodeManager NodeManager NodeManager HOYA reduce 1.1 Tez map.3 vertex 1 Region server 3 Storm

20 YARN: Multi-tenancy II ResourceManager Scheduler NodeManager NodeManager NodeManager NodeManager MapReduce 1 map 1.1 Region server reduce.1 reduce. root nimbus 1 map.1 30% 60% 10% NodeManager Different users NodeManager and NodeManager NodeManager Ad- User DWH Hoc HBase Master map 1. MapReduce map. 0% 80% 5% 75% nimbus Region server 1 vertex 4 vertex same cluster Dev Prod Dev Prod NodeManager NodeManager NodeManager NodeManager 60% 40% HOYA reduce 1.1 Tez map.3 Dev1 Dev Region server 3 Storm organizations on the vertex 1 vertex 3

21 YARN: Your own app MyApp YARNClient App specific API Application Client Protocol Application Master Protocol NodeManager ResourceManager Scheduler NodeManager myam Container DistributedShell is the new WordCount! AMRMClient NMClient NodeManager Container Container Management Protocol

22 Stop #3: HDFS Land HDFS Land Welcome Office YARN Land

23 HDFS : In a nutshell Removes tight coupling of Block Storage and Namespace Adds (built-in) High Availability Better Scalability & Isolation Increased performance Details:

24 Horizontally scale IO and storage HDFS : Federation NameNodes do not talk to each other NameNode 1 Namespace 1 NameNode Namespace NameNodes manages only slice of namespace Namespace State Block Map Namespace State Pool 1 Pool Block Map Block Storage as generic storage service Block Pools b3 b b1 b4 b b5 b1 Data Nodes b3 b5 b b4 DataNodes can store blocks managed by any NameNode JBOD JBOD JBOD

25 HDFS : Architecture NFS or Shared state on NFS OR Maintains Block Map and Edits File Take orders only from the Active Active NameNode Block Map Journal Node Edits File Journal Node Journal Node Standby NameNode Block Map Report to both NameNodes Quorum based storage Edits File Simultaneously reads and applies the edits DataNode DataNode DataNode DataNode DataNode

26 HDFS : High Availability Heartbeat ZooKeeper Node ZooKeeper Node ZooKeeper Node Heartbeat ZKFailover Controller Journal Node Journal Node Journal Node ZKFailover Controller Holds special lock znode Active NameNode Block Map Edits File Shared State Standby NameNode Block Map Edits File Monitors health of NN, OS, HW DataNode DataNode DataNode DataNode DataNode Send Heartbeats & Block Reports

27 HDFS : Write-Pipeline Earlier versions of HDFS Files were immutable Write-once-read-many model New features in HDFS Files can be reopened for append New primitives: hflush and hsync Replace data node on failure Read consistency Add new node to the pipeline Data Data Data Writer DataNode 1 DataNode DataNode 3 DataNode 4 Can read from any node and then failover to any other node Reader

28 HDFS : Snapshots Admin can create point in time snapshots of HDFS Of the entire file system Of a specific data-set (sub-tree directory of file system) Restore state of entire file system or data-set to a snapshot (like Apple Time Machine) Protect against user errors Snapshot diffs identify changes made to data set Keep track of how raw or derived/analytical data changes over time

29 HDFS : NFS Gateway Supports NFS v3 (NFS v4 is work in progress) Supports all HDFS commands List files Copy, move files Create and delete directories Ingest for large scale analytical workloads Load immutable files as source for analytical processing No random writes Stream files into HDFS Log ingest by applications writing directly to HDFS client mount

30 HDFS : Performance Many improvements New AppendableWrite-Pipeline Read path improvements for fewer memory copies Short-circuit local reads for -3x faster random reads I/O improvements using posix_fadvise() libhdfs improvements for zero copy reads Significant improvements: I/O.5-5x faster

31 Stop #4: YARN App Land YARN App Land HDFS Land Welcome Office YARN Land

32 YARN Apps: Overview MapReduce Batch Tez DAG Processing Storm Stream Processing Samza Stream Processing Apache S4 Stream Processing Spark In-Memory HOYA HBase on YARN Apache Giraph Graph Processing Apache Hama Bulk Synchronous Parallel Elastic Search Scalable Search

33 YARN Apps: Overview MapReduce Batch Tez DAG Processing Storm Stream Processing Samza Stream Processing Apache S4 Stream Processing Spark In-Memory HOYA HBase on YARN Apache Giraph Graph Processing Apache Hama Bulk Synchronous Parallel Elastic Search Scalable Search

34 MapReduce : In a nutshell MapReduce is now a YARN app No more map and reduce slots, it s containers now No more JobTracker, it s YarnAppmaster library now Multiple versions of MapReduce The older mapred APIs work without modification or recompilation The newer mapreduce APIs may need to be recompiled Still has one master server component: the Job History Server The Job History Server stores the execution of jobs Used to audit prior execution of jobs Will also be used by YARN framework to store charge backs at that level Better cluster utilization Increased scalability & availability

35 MapReduce : Shuffle Faster Shuffle Better embedded server: Netty Encrypted Shuffle Secure the shuffle phase as data moves across the cluster Requires way HTTPS, certificates on both sides Causes significant CPU overhead, reserve 1 core for this work Certificates stored on each node (provision with the cluster), refreshed every 10 secs Pluggable Shuffle Sort Shuffle is the first phase in MapReduce that is guaranteed to not be datalocal Pluggable Shuffle/Sort allows application developers or hardware developers to intercept the network-heavy workload and optimize it Typical implementations have hardware components like fast networks and software components like sorting algorithms API will change with future versions of Hadoop

36 MapReduce : Performance Key Optimizations No hard segmentation of resource into map and reduce slots YARN scheduler is more efficient MR framework has become more efficient than MR1: shuffle phase in MRv is more performant with the usage of Netty nodes running YARN across over 365 PB of data. About jobs per day for about 10 million hours of compute time. Estimated 60% 150% improvement on node usage per day Got rid of a whole 10,000 node datacenter because of their increased utilization.

37 YARN Apps: Overview MapReduce Batch Tez DAG Processing Storm Stream Processing Samza Stream Processing Apache S4 Stream Processing Spark In-Memory HOYA HBase on YARN Apache Giraph Graph Processing Apache Hama Bulk Synchronous Parallel Elastic Search Scalable Search

38 Apache Tez: In a nutshell Distributed execution framework that works on computations represented as dataflow graphs Tez is Hindi for speed Naturally maps to execution plans produced by query optimizers Highly customizable to meet a broad spectrum of use cases and to enable dynamic performance optimizations at runtime Built on top of YARN

39 Apache Tez: Architecture Task with pluggable Input, Processor & Output Task HDFS Input Map Processor Sorted Output Classical Map Task Shuffle Input Reduce Processor HDFS Output Classical Reduce YARN ApplicationMaster to run DAG of Tez Tasks

40 Apache Tez: Tez Service MapReduce Query Startup is expensive: Job launch & task-launch latencies are fatal for short queries (in order of 5s to 30s) Solution: Tez Service (= Preallocated Application Master) Removes job-launch overhead (Application Master) Removes task-launch overhead (Pre-warmed Containers) Hive (or Pig) Submit query plan to Tez Service Native Hadoop service, not ad-hoc

41 MapReduce as Base Apache Tez: The new primitive Apache Tez as Base Hadoop 1 Pig Hive Other MapReduce Cluster resource mgmt. + data processing HDFS Redundant, reliable storage Hadoop MR Pig Hive Tez Execution Engine YARN Cluster resource management HDFS Real time Storm Redundant, reliable storage O t h e r

42 Apache Tez: Performance SELECT a.state, COUNT(*), AVERAGE(c.price) FROM a JOIN b ON (a.id = b.id) JOIN c ON (a.itemid = c.itemid) GROUP BY a.state Existing Hive Parse Query 0.5s Create Plan 0.5s Hive/Tez Parse Query 0.5s Create Plan 0.5s Tez & Hive Service Parse Query 0.5s Create Plan 0.5s Launch Map- Reduce 0s Launch Map- Reduce 0s Submit to Tez Service 0.5s Process Map- Reduce 10s Process Map- Reduce s Process Map-Reduce s Total 31s Total 3s Total 3.5s * No exact numbers, for illustration only

43 Stinger Initiative: In a nutshell

44 Stinger: Overall Performance * Real numbers, but handle with care!

45 YARN Apps: Overview MapReduce Batch Tez DAG Processing Storm Stream Processing Samza Stream Processing Apache S4 Stream Processing Spark In-Memory HOYA HBase on YARN Apache Giraph Graph Processing Apache Hama Bulk Synchronous Parallel Elastic Search Scalable Search

46 Storm: In a nutshell Stream-processing Real-time processing Developed as standalone application Ported on YARN

47 Topology Network of spouts & bolts as the nodes and streams as the edges Storm: Conceptual view Spout Source of streams Spout Tuple List of name-value pairs Spout Tuple Tuple Bolt Stream Unbound sequence of tuples Bolt Bolt Tuple Bolt Consumer of streams Processing of tuples Possibly emits new tuples Bolt Bolt

48 YARN Apps: Overview MapReduce Batch Tez DAG Processing Storm Stream Processing Samza Stream Processing Apache S4 Stream Processing Spark In-Memory HOYA HBase on YARN Apache Giraph Graph Processing Apache Hama Bulk Synchronous Parallel Elastic Search Scalable Search

49 Spark: In a nutshell High-speed in-memory analytics over Hadoop and Hive Separate MapReduce-like engine Speedup of up to 100x On-disk queries 5-10x faster Spark is now a top-level Apache project Compatible with Hadoop s Storage API Spark can be run on top of YARN

50 Spark: RDD Key idea: Resilient Distributed Datasets (RDDs) RDD A11 A1 A13 Read-only partitioned collection of records Optionally cached in memory across cluster Manipulated through parallel operators Support only coarse-grained operations Map Reduce Group-by transformations Automatically recomputed on failure

51 Spark: Data Sharing

52 YARN Apps: Overview MapReduce Batch Tez DAG Processing Storm Stream Processing Samza Stream Processing Apache S4 Stream Processing Spark In-Memory HOYA HBase on YARN Apache Giraph Graph Processing Apache Hama Bulk Synchronous Parallel Elastic Search Scalable Search

53 HOYA: In a nutshell Create on-demand HBase clusters Small HBase cluster in large YARN cluster Dynamic HBase clusters Self-healing HBase Cluster Elastic HBase clusters Transient/intermittent clusters for workflows Configure custom configurations & versions Better isolation More efficient utilization/sharing of cluster

54 HOYA: Creation of AppMaster HOYA Client YARNClient ResourceManager Scheduler HOYA specific API NodeManager HOYA Application Master Container Container NodeManager Container Container NodeManager Container Container

55 HOYA: Deployment of HBase HOYA Client YARNClient ResourceManager Scheduler HOYA specific API NodeManager HOYA Application Master Container HBase Master Container NodeManager Container Region Server Container NodeManager Container Container Region Server

56 HOYA: Bind via ZooKeeper HOYA Client YARNClient ResourceManager Scheduler HOYA specific API HBase Client NodeManager HOYA Application Master Container HBase Master Container NodeManager Container Region Server Container NodeManager Container Container Region Server ZooKeeper

57 YARN Apps: Overview MapReduce Batch Tez DAG Processing Storm Stream Processing Samza Stream Processing Apache S4 Stream Processing Spark In-Memory HOYA HBase on YARN Apache Giraph Graph Processing Apache Hama Bulk Synchronous Parallel Elastic Search Scalable Search

58 Giraph: In a nutshell Giraph is a framework for processing semistructured graph data on a massive scale Giraph is loosely based upon Google's Pregel Both systems are inspired by the Bulk Synchronous Parallel model Giraph performs iterative calculations on top of an existing Hadoop cluster Uses Single Map-only Job Apache top level project since 01

59 Stop #5: Enterprise Land YARN App Land Enterprise Land HDFS Land Welcome Office YARN Land

60 Falcon: In a nutshell A framework for managing data processing in Hadoop Clusters Falcon runs as a standalone server as part of the Hadoop cluster Key Features: Data Replication Handling Data Lifecycle Management Process Coordination & Scheduling Declarative Data Process Programming Apache Incubation Status

61 Falcon: One-stop Shop Data Management Needs Data Processing Replication Retention Scheduling Reprocessing Multi Cluster Mgmt. Tool Orchestration Oozie Sqoop Flume MapReduce Pig & Hive Distcp

62 Falcon: Weblog Use Case Weblogs saved hourly to primary cluster HDFS location is /weblogs/{date} Desired Data Policy: Replicate weblogs to secondary cluster Evict weblogs from primary cluster after days Evict weblogs from secondary cluster after 1 week

63 Falcon: Weblog Use Case <feed description= " name= feed-weblogs xmlns="uri:falcon:feed:0.1"> <frequency>hours(1)</frequency> <clusters> <cluster name= cluster-primary" type="source"> <validity start=" t00:00z" end=" t00:00z"/> <retention limit="days()" action="delete"/> </cluster> <cluster name= cluster-secondary" type="target"> <validity start=" t00:00z" end=" t00:00z"/> <retention limit= days(7)" action="delete"/> </cluster> </clusters> <locations> <location type="data path="/weblogs/${year}-${month}-${day}- ${HOUR}"/> </locations> <ACL owner= hdfs" group="users" permission="0755"/> <schema location="/none" provider="none"/> </feed>

64 Knox: In a nutshell System that provides a single point of authentication and access for Apache Hadoop services in a cluster. The gateway runs as a server (or cluster of servers) that provide centralized access to one or more Hadoop clusters. The goal is to simplify Hadoop security for both users and operators Apache Incubation Status

65 Knox: Architecture Browser Firewall Identity Providers Kerberos SSO Provider Firewall Secure Hadoop Cluster 1 ResourceManager Scheduler NodeManager NodeManager NodeManager NodeManager MapReduce 1 Region server map 1.1 reduce.1 reduce. nimbus 1 map.1 vertex 3 NodeManager NodeManager NodeManager NodeManager DMZ HBase Master nimbus map 1. Region server 1 MapReduce vertex 4 map. vertex NodeManager NodeManager NodeManager NodeManager Ambari Client REST Client Gateway Cluster #1 # #3 HOYA reduce 1.1 Tez map.3 vertex 1 ResourceManager Scheduler Region server 3 Storm Secure Hadoop Cluster NodeManager NodeManager NodeManager NodeManager MapReduce 1 Region server map 1.1 reduce.1 reduce. nimbus 1 map.1 vertex 3 NodeManager NodeManager NodeManager NodeManager HBase Master nimbus map 1. Region server 1 MapReduce vertex 4 map. vertex JDBC Client Ambari / Hue Server NodeManager NodeManager NodeManager NodeManager HOYA reduce 1.1 Tez map.3 vertex 1 Region server 3 Storm

66 Final Stop: Goodbye Photo YARN App Land Enterprise Land Goodbye Photo HDFS Land Welcome Office YARN Land

67 Hadoop : Summary 1. Scale. New programming models & services 3. Beyond Java 4. Improved cluster utilization 5. Enterprise Readiness

68 One more thing Let s get started with Hadoop!

69 Hortonworks Sandbox.0

70 Trainings Developer Training , Düsseldorf , München , Frankfurt Admin Training , Düsseldorf , München , Frankfurt Details:

71 Thanks for traveling with me