Namenode HA. Sanjay Radia - Hortonworks

Transcription

1 Namenode HA Sanjay Radia - Hortonworks

2 Sanjay Radia - Background Working on Hadoop for the last 4 years Part of the original team at Yahoo Primarily worked on HDFS, MR Capacity scheduler wire protocols, compatibility. Hadoop committer and member of the Apache Hadoop PMC Founded Hortonworks with some of my colleagues Currently leading HDFS and working on other technologies Page 2

3 Overview HDFS Availability and Data Integrity what is the record? HA Under Active Development Will be available in 0.23 dot release 3

4 Current HDFS Availability & Data Integrity Simple design, storage fault tolerance Storage: Rely in OS s file system rather than use raw disk Storage Fault Tolerance: multiple replicas, active monitoring Single Namenode Master Persistent state: multiple copies + checkpoints Restart on failure How well did it work? Lost 19 out of 329 Million blocks on 10 clusters with 20K nodes in s of reliability Fixed in 20 and month study: 22 failures on 25 clusters failures per year per cluster Only 8 would have benefitted from HA failover!! (0.23 failures per cluster year) is very robust and can take a lot of abuse is resilient against overload caused by misbehaving apps 4

5 HA Namenode HA Namenode under active development Automatic Failover of Detailed design HDFS-1623 HDFS-1971, 1972, 1973, 1974,1975, 2005, 2064, 1073, 2060 Hadoop-7347, 7524HHaHADOOP-7347 HA is the highest priority 5

6 Approach and Terminology Initial goal is Active-Standby In future, the standby can serve Read requests Terminology Active actively serves the read/write operations from the clients Standby - waits, becomes active when Active dies or is unhealthy Could serve read operations Standby s State may be cold, warm or hot Cold : Standby has zero state (e.g. started after the Active is declared dead. Warm: Standby has partial state: has loaded fsimage & editlogs but has not received any block reports has loaded fsimage and rolled logs and all block reports Hot Standby: Standby has all most of the Active s state and start immediately 6

7 High Level Use Cases Planned downtime Upgrades Config changes Main reason for downtime Unplanned downtime Supported failures Single hardware failure Double hardware failure not supported Some software failures Same software failure affects both active and standby Hardware failure Server unresponsive Software failures Occurs infrequently 7

8 Use Cases Deployment models Single configuration; no failover Active and Standby with manual failover Standby could be cold/warm/hot Active and Standby with automatic failover Hot standby See HDFS-1623 for detailed use cases 8

9 Design Failover control outside Parallel Block reports to Active and Standby (Hot failover) Shared or non-shared state Fencing of shared resources/data Datanodes Shared state (if any) Client failover IP Failover Smart clients (e.g Zookeeper for coordination) 9

10 Failover Control Outside Failover Controller Actions start, stop, failover, monitor, Quorum Service Shared Resources Resources Resources Resources Failover Controller outside Namenode Daemon manages resources All resources modeled uniformly Resources OS, HW, Network etc. Namenode is just another resource Heartbeat with other nodes Quorum based leader election Zookeeper for co-ordination and Quorum Fencing during split brain Prevents data corruption

11 HA with ZooKeeper Heartbeat ZK ZK ZK Heartbeat FailoverController Active FailoverController Standby Monitor Health of. OS, HW Cmds Monitor Health of. OS, HW Active Standby Block Reports to Active & Standby DN fencing: Update cmds from one DN DN DN

12 Sharing the s persistent State (1) Mid term goal (12 months) Move s persistent state to the Datanode Active Standby Store s journal and checkpointed image on Datanodes DN DN DN

13 Sharing the s persistent State (2) Shorter Term (6 months) Active Shared state with single writer (fencing) Shared Storage Approach Standby Active Edit Logs Standby Direct stream to Standby

14 QA Detailed design (HDFS-1623) HDFS-1971, 1972, 1973, 1974,1975, 2005, 2064, 1073, 2060 Hadoop-7347,

15 How Hortonworks Can Help Training and Certification Hortonworks Data Platform and Support Educational Webinars Architecting the Future of Page 15