Big Data for Engineers Spring Resource Management

Transcription

1 Ghislain Fourny Big Data for Engineers Spring Resource Management artjazz / 123RF Stock Photo

2 Data Technology Stack User interfaces Querying Data stores Indexing Processing Validation Data models Syntax Encoding Storage 2

3 Where we are User interfaces Querying Data stores Indexing Processing Validation Data models Syntax Encoding Storage 3

4 Last week: MapReduce Input data Map Map Map Map Map Map Map Map Intermediate data (shuffled) Reduce Reduce Reduce Reduce Reduce Reduce Reduce Reduce Output data 4

5 Hadoop infrastructure (version 1) Namenode /dir/file Datanode Datanode Datanode Datanode Datanode Datanode 5

6 Hadoop infrastructure (version 1) Namenode + JobTracker /dir/file Datanode + TaskTracker Datanode + TaskTracker Datanode + TaskTracker Datanode + TaskTracker Datanode + TaskTracker Datanode + TaskTracker 6

7 Responsibilities of the MapReduce JobTracker Resource Management 7

8 Responsibilities of the MapReduce JobTracker Resource Management Scheduling 8

9 Responsibilities of the MapReduce JobTracker Resource Management Scheduling Monitoring 9

10 Responsibilities of the MapReduce JobTracker Resource Management Scheduling Monitoring Job lifecycle 10

11 Responsibilities of the MapReduce JobTracker Resource Management Scheduling Monitoring Job lifecycle Fault-tolerance 11

12 Issue 1: scalability M M M M M M M M M M M M < 4,000 nodes < 40,000 tasks 12

13 Issue 2: bottleneck JobTracker Bottleneck TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker TaskTracker 13 13

14 Issue 3: Jack of all trades Scheduling Monitoring 14 14

15 Issue 4: Utilization (task slots) Static (Decide on M/R at configuration time) Fixed-size 15 15

16 Issue 5: Not fungible Map Reduce 16 16

17 Issue 5: Not fungible Working at maximum capacity Idle Map Reduce 17 17

18 kirtchanut / 123RF Stock Photo YARN 18

19 YARN Yet Another Resource Negotiator 19

20 YARN Scheduling Application Monitoring management Resource Manager Application Master Application Master Application Master Application Master Application Master 20

21 Scales more M M M M M M M M M M M M M M M M M M M M M M M M M M M M M M 10,000 nodes 100,000 tasks 21

22 YARN architecture ResourceManager 22

23 YARN architecture ResourceManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager 23

24 YARN architecture ResourceManager Container Container Container NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager 24

25 Remember... It does ring a bell, doesn't it? 25

26 Master-slave architecture Master Slave Slave Slave Slave Slave Slave 26

27 HDFS server architecture Namenode /dir/file1 /dir/file2 /file3 Datanode Datanode Datanode Datanode Datanode Datanode 27

28 YARN ResourceManager Container Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 28

29 YARN Client ResourceManager Job Container Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 29

30 YARN: RM allocates an Application Master Client ResourceManager Job Schedules Container Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 30

31 YARN: RM allocates an Application Master Client ResourceManager Job Schedules Application Master Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 31

32 YARN: RM allocates an Application Master Client ResourceManager Job Application Master Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 32

33 YARN: RM allocates an Application Master Client ResourceManager Job Application Master Container Container NodeManager NodeManager NodeManager NodeManager NodeManager 33

34 Application Master communicates with containers Application Master Container Container Container Execute Monitor Container 34

35 kirtchanut / 123RF Stock Photo YARN's Resource Manager 35

36 Resource Manager Capacity guarantees Cluster Utilization Fairness Service Level Agreements 36

37 Communication with clients 37

38 Communication with clients Client Service Application (start, end) Queue information Statistics 38

39 Communication with clients Client Service Application (start, end) Queue information Statistics Admin Service Refresh the node list Queue configuration 39

40 Communication with the node managers 40

41 Communication with the node managers Resource Tracker 41

42 Communication with the node managers Resource Tracker Liveliness 42

43 Communication with the node managers Resource Tracker Liveliness Nodes List Manager valid invalid 43

44 Communication with the application masters 44

45 Communication with the application masters Application Master Service (registration) 45

46 Communication with the application masters Application Master Service (registration) Liveliness 46

47 Communication with the application masters Application Master Service (registration) Liveliness Application Master Service (container requests) 47

48 Authentication 48

49 Authentication Application Token 49

50 Authentication Application Token Container Token 50

51 Authentication Application Token Application ACL Container Token 51

52 Pure scheduler Does not monitor tasks. Does not restart upon failure. 52

53 Scheduling strategies: pluggable scheduler 53

54 Scheduling strategies: pluggable scheduler FIFO scheduler 54

55 Scheduling strategies: pluggable scheduler FIFO scheduler 55

56 Scheduling strategies: pluggable scheduler FIFO scheduler 56

57 Scheduling strategies: pluggable scheduler FIFO scheduler 57

58 Scheduling strategies: pluggable scheduler FIFO scheduler 58

59 Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 59

60 Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 60

61 Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 61

62 Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 62

63 Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 63

64 Scheduling strategies: pluggable scheduler Capacity scheduler Queue 1 Queue 2 64

65 Scheduling strategies: pluggable scheduler Fair scheduler 65

66 Scheduling strategies: pluggable scheduler Fair scheduler 66

67 Scheduling strategies: pluggable scheduler Fair scheduler 67

68 Scheduling strategies: pluggable scheduler Fair scheduler 68

69 Scheduling strategies: pluggable scheduler Fair scheduler 69

70 Scheduling strategies: pluggable scheduler Fair scheduler 70

71 Scheduling strategies: pluggable scheduler Fair scheduler 71

72 Resource container X GB W cores, U GHz Y TB Z MBps 72

73 kirtchanut / 123RF Stock Photo YARN's Node Manager 73

74 NodeManager: one per node NodeManager NodeManager NodeManager NodeManager 74

75 Monitoring Memory CPU Disk Network 75

76 Reports to ResourceManager Memory CPU ResourceManager Disk Network 76

77 Container 77

78 kirtchanut / 123RF Stock Photo YARN's Application Masters 78

79 Application Master Application Master is per application. 79

80 Application Master Application Master is application-specific. 80

81 Framework-specific application masters MapReduce DAG distributed processing Message Passing Interface Graph processing 81

82 Complexity is moved to the Application Master ResourceManager complexity 82

83 Application Master ResourceManager negotiates resources 83

84 Application Master ResourceManager negotiates resources executes and monitors NodeManager 84

85 Fault tolerance is on the application master 85

86 Fault tolerance is on the application master 86

87 Fault tolerance is on the application master relaunch 87

88 Application-specific monitoring no longer a bottleneck 88

89 Application Master is not trusted 89

90 Application Master is not trusted Evil plan to book containers and not use them 90

91 Summary Separation between scheduling and monitoring 91

92 Summary Separation between scheduling and monitoring Scalability 92

93 Summary Separation between scheduling and monitoring Scalability Availability 93

94 Summary Separation between scheduling and monitoring Scalability Availability Multi-tenancy 94

95 Forward compatibility with DAGs of tasks 95