Ghislain Fourny Big Data for Engineers Spring 2018 7. Resource Management artjazz / 123RF Stock Photo
Data Technology Stack User interfaces Querying Data stores Indexing Processing Validation Data models Syntax Encoding Storage 2
Where we are User interfaces Querying Data stores Indexing Processing Validation Data models Syntax Encoding Storage 3
Last week: MapReduce Input data Map Map Map Map Map Map Map Map Intermediate data (shuffled) Reduce Reduce Reduce Reduce Reduce Reduce Reduce Reduce Output data 4
Hadoop infrastructure (version 1) Namenode /dir/file Datanode Datanode Datanode Datanode Datanode Datanode 5
Hadoop infrastructure (version 1) Namenode + JobTracker /dir/file Datanode + TaskTracker Datanode + TaskTracker Datanode + TaskTracker Datanode + TaskTracker Datanode + TaskTracker Datanode + TaskTracker 6
Responsibilities of the MapReduce JobTracker Resource Management 7
Responsibilities of the MapReduce JobTracker Resource Management Scheduling 8
Responsibilities of the MapReduce JobTracker Resource Management Scheduling Monitoring 9
Responsibilities of the MapReduce JobTracker Resource Management Scheduling Monitoring Job lifecycle 10
Responsibilities of the MapReduce JobTracker Resource Management Scheduling Monitoring Job lifecycle Fault-tolerance 11
Issue 1: scalability M M M M M M M M M M M M < 4,000 nodes < 40,000 tasks 12
Issue 2: bottleneck JobTracker Bottleneck TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker 13 13
Issue 3: Jack of all trades Scheduling Monitoring 14 14
Issue 4: Utilization (task slots) Static (Decide on M/R at configuration time) Fixed-size 15 15
Issue 5: Not fungible Map Reduce 16 16
Issue 5: Not fungible Working at maximum capacity Idle Map Reduce 17 17
kirtchanut / 123RF Stock Photo YARN 18
YARN Yet Another Resource Negotiator 19
YARN Scheduling Application Monitoring management Resource Manager Application Master Application Master Application Master Application Master Application Master 20
Scales more M M M M M M M M M M M M M M M M M M M M M M M M M M M M M M 10,000 nodes 100,000 tasks 21
YARN architecture ResourceManager 22
YARN architecture ResourceManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager 23
YARN architecture ResourceManager Container Container Container NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager 24
Remember... It does ring a bell, doesn't it? 25
Master-slave architecture Master Slave Slave Slave Slave Slave Slave 26
HDFS server architecture Namenode /dir/file1 /dir/file2 /file3 Datanode Datanode Datanode Datanode Datanode Datanode 27
YARN ResourceManager Container Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 28
YARN Client ResourceManager Job Container Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 29
YARN: RM allocates an Application Master Client ResourceManager Job Schedules Container Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 30
YARN: RM allocates an Application Master Client ResourceManager Job Schedules Application Master Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 31
YARN: RM allocates an Application Master Client ResourceManager Job Application Master Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 32
YARN: RM allocates an Application Master Client ResourceManager Job Application Master Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 33
Application Master communicates with containers Application Master Container Container Container Execute Monitor Container 34
kirtchanut / 123RF Stock Photo YARN's Resource Manager 35
Resource Manager Capacity guarantees Cluster Utilization Fairness Service Level Agreements 36
Communication with clients 37
Communication with clients Client Service Application (start, end) Queue information Statistics 38
Communication with clients Client Service Application (start, end) Queue information Statistics Admin Service Refresh the node list Queue configuration 39
Communication with the node managers 40
Communication with the node managers Resource Tracker 41
Communication with the node managers Resource Tracker Liveliness 42
Communication with the node managers Resource Tracker Liveliness Nodes List Manager valid invalid 43
Communication with the application masters 44
Communication with the application masters Application Master Service (registration) 45
Communication with the application masters Application Master Service (registration) Liveliness 46
Communication with the application masters Application Master Service (registration) Liveliness Application Master Service (container requests) 47
Authentication 48
Authentication Application Token 49
Authentication Application Token Container Token 50
Authentication Application Token Application ACL Container Token 51
Pure scheduler Does not monitor tasks. Does not restart upon failure. 52
Scheduling strategies: pluggable scheduler 53
Scheduling strategies: pluggable scheduler FIFO scheduler 54
Scheduling strategies: pluggable scheduler FIFO scheduler 55
Scheduling strategies: pluggable scheduler FIFO scheduler 56
Scheduling strategies: pluggable scheduler FIFO scheduler 57
Scheduling strategies: pluggable scheduler FIFO scheduler 58
Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 59
Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 60
Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 61
Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 62
Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 63
Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 64
Scheduling strategies: pluggable scheduler Fair scheduler 65
Scheduling strategies: pluggable scheduler Fair scheduler 66
Scheduling strategies: pluggable scheduler Fair scheduler 67
Scheduling strategies: pluggable scheduler Fair scheduler 68
Scheduling strategies: pluggable scheduler Fair scheduler 69
Scheduling strategies: pluggable scheduler Fair scheduler 70
Scheduling strategies: pluggable scheduler Fair scheduler 71
Resource container X GB W cores, U GHz Y TB Z MBps 72
kirtchanut / 123RF Stock Photo YARN's Node Manager 73
NodeManager: one per node NodeManager NodeManager NodeManager NodeManager 74
Monitoring Memory CPU Disk Network 75
Reports to ResourceManager Memory CPU ResourceManager Disk Network 76
Container 77
kirtchanut / 123RF Stock Photo YARN's Application Masters 78
Application Master Application Master is per application. 79
Application Master Application Master is application-specific. 80
Framework-specific application masters MapReduce DAG distributed processing Message Passing Interface Graph processing 81
Complexity is moved to the Application Master ResourceManager complexity 82
Application Master ResourceManager negotiates resources 83
Application Master ResourceManager negotiates resources executes and monitors NodeManager 84
Fault tolerance is on the application master 85
Fault tolerance is on the application master 86
Fault tolerance is on the application master relaunch 87
Application-specific monitoring no longer a bottleneck 88
Application Master is not trusted 89
Application Master is not trusted Evil plan to book containers and not use them 90
Summary Separation between scheduling and monitoring 91
Summary Separation between scheduling and monitoring Scalability 92
Summary Separation between scheduling and monitoring Scalability Availability 93
Summary Separation between scheduling and monitoring Scalability Availability Multi-tenancy 94
Forward compatibility with DAGs of tasks 95