CONTAINERIZED SPARK ON KUBERNETES. William Benton Red Hat,

Transcription

1 CONTAINERIZED SPARK ON KUBERNETES William Benton Red Hat,

2 BACKGROUND

3 BACKGROUND

4 BACKGROUND

5 BACKGROUND

6 BACKGROUND

7 BACKGROUND

8 BACKGROUND

9 BACKGROUND

10 WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014

11 WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014 Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor

12 WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014 Spark executor Spark executor Networked POSIX FS Spark executor Spark executor Spark executor Spark executor

13 WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014 Mesos Spark executor Spark executor Networked POSIX FS Spark executor Spark executor Spark executor Spark executor

14 WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014 Mesos 1 2 Spark executor Spark executor Spark executor Networked POSIX FS 3 Spark executor Spark executor 4 Spark executor

15 WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014 Mesos 1 1 Spark executor Networked 1 Spark executor POSIX FS 2 2 Spark executor 3 3 Spark executor 3 Spark executor 4 4 Spark executor

16 WHAT OUR SPARK CLUSTER LOOKED LIKE IN 2014 Mesos 1 1 Spark executor Networked 1 Spark executor POSIX FS 2 2 Spark executor 3 3 Spark executor 3 Spark executor 4 4 Spark executor

17 Analytics is no longer a separate workload.

18 Analytics is an essential component of modern datadriven applications.

19 OUR GOALS

20 OUR GOALS

21 OUR GOALS

22 OUR GOALS

23 OUR GOALS git

24 OUR GOALS git

25 OUR GOALS git

26 FORECAST Motivating containerized microservices Architectures for analytics and applications Spark clusters in containers: practicalities and pitfalls Play along at home Future work

27 MOTIVATING MICROSERVICES

28 A microservice architecture employs lightweight, modular, and typically stateless components with well-defined interfaces and contracts.

29 BENEFITS OF MICROSERVICE ARCHITECTURES

30 BENEFITS OF MICROSERVICE ARCHITECTURES

31 BENEFITS OF MICROSERVICE ARCHITECTURES

32 BENEFITS OF MICROSERVICE ARCHITECTURES

33 BENEFITS OF MICROSERVICE ARCHITECTURES

34 BENEFITS OF MICROSERVICE ARCHITECTURES

35 BENEFITS OF MICROSERVICE ARCHITECTURES

36 BENEFITS OF MICROSERVICE ARCHITECTURES 2 + 2

37 BENEFITS OF MICROSERVICE ARCHITECTURES

38 BENEFITS OF MICROSERVICE ARCHITECTURES

39 BENEFITS OF MICROSERVICE ARCHITECTURES

40 BENEFITS OF MICROSERVICE ARCHITECTURES

41 BENEFITS OF MICROSERVICE ARCHITECTURES?

42 BENEFITS OF MICROSERVICE ARCHITECTURES?

43 BENEFITS OF MICROSERVICE ARCHITECTURES?

44 MICROSERVICES AND SPARK executor executor executor executor master

45 MICROSERVICES AND SPARK λ x: x * 2 executor executor executor executor master

46 MICROSERVICES AND SPARK λ x: x * 2 executor executor executor executor master λ x: x * 2 λ x: x * 2 λ x: x * 2 λ x: x * 2

47 MICROSERVICES AND SPARK λ x: x * 2 executor executor executor executor master λ x: x * 2 λ x: x * 2 λ x: x * 2 λ x: x * 2

48 MICROSERVICES AND SPARK λ x: x * 2 executor executor executor executor master λ x: x * 2 λ x: x * 2 λ x: x * 2 λ x: x * 2

49 ARCHITECTURES FOR ANALYTICS AND APPLICATIONS

50 APPLICATION RESPONSIBILITIES events transform databases transform file, object storage transform

51 APPLICATION RESPONSIBILITIES events transform databases transform aggregate file, object storage transform

52 APPLICATION RESPONSIBILITIES events transform databases transform aggregate file, object storage transform models train

53 APPLICATION RESPONSIBILITIES events transform databases transform aggregate file, object storage transform models train

54 APPLICATION RESPONSIBILITIES events transform databases transform aggregate archive file, object storage transform models train

55 APPLICATION RESPONSIBILITIES events transform databases transform aggregate archive file, object storage transform models train

56 APPLICATION RESPONSIBILITIES events transform developer UI web and mobile databases transform aggregate archive file, object storage transform reporting models train

57 APPLICATION RESPONSIBILITIES events transform developer UI web and mobile databases transform aggregate archive file, object storage transform reporting models train management

58 LEGACY ARCHITECTURES

59 CONVENTIONAL DATA WAREHOUSE events

60 CONVENTIONAL DATA WAREHOUSE events transform

61 CONVENTIONAL DATA WAREHOUSE events transform UI

62 CONVENTIONAL DATA WAREHOUSE events transform UI business logic

63 CONVENTIONAL DATA WAREHOUSE events transform UI business logic RDBMS

64 CONVENTIONAL DATA WAREHOUSE events transform UI business logic transaction RDBMS processing

65 CONVENTIONAL DATA WAREHOUSE events transform UI business logic transaction RDBMS processing

66 CONVENTIONAL DATA WAREHOUSE events transform UI business logic transaction processing RDBMS RDBMS

67 CONVENTIONAL DATA WAREHOUSE events transform UI business logic analysis transaction processing RDBMS RDBMS

68 CONVENTIONAL DATA WAREHOUSE events transform reporting UI business logic analysis transaction processing RDBMS RDBMS

69 CONVENTIONAL DATA WAREHOUSE events transform reporting UI business logic analysis transaction processing RDBMS RDBMS

70 CONVENTIONAL DATA WAREHOUSE events transform reporting interactive query business UI analysis logic transaction RDBMS RDBMS processing

71 CONVENTIONAL DATA WAREHOUSE events transform reporting interactive query business UI analysis logic transaction analytic RDBMS RDBMS processing processing

72 HADOOP-STYLE DATA LAKE HDFS HDFS HDFS

73 HADOOP-STYLE DATA LAKE HDFS HDFS HDFS HDFS HDFS

74 HADOOP-STYLE DATA LAKE events HDFS HDFS HDFS HDFS HDFS

75 HADOOP-STYLE DATA LAKE events HDFS HDFS HDFS HDFS HDFS

76 HADOOP-STYLE DATA LAKE events HDFS HDFS HDFS HDFS HDFS compute compute compute compute compute

77 HADOOP-STYLE DATA LAKE events HDFS HDFS compute compute

78 MODERN ARCHITECTURES

79 THE LAMBDA ARCHITECTURE transform (imprecise) analysis events

80 THE LAMBDA ARCHITECTURE transform (imprecise) analysis events DFS

81 THE LAMBDA ARCHITECTURE transform (imprecise) analysis speed layer events DFS

82 THE LAMBDA ARCHITECTURE transform (imprecise) analysis speed layer events transform (precise) analysis DFS

83 THE LAMBDA ARCHITECTURE transform (imprecise) analysis speed layer events transform (precise) analysis DFS batch layer

84 THE LAMBDA ARCHITECTURE transform (imprecise) analysis federate speed layer events transform (precise) analysis DFS batch layer

85 THE LAMBDA ARCHITECTURE transform (imprecise) analysis federate UI speed layer events transform (precise) analysis DFS batch layer

86 THE LAMBDA ARCHITECTURE transform (imprecise) analysis federate UI speed layer serving layer events transform (precise) analysis DFS batch layer

87 THE KAPPA ARCHITECTURE events

88 THE KAPPA ARCHITECTURE events message bus

89 THE KAPPA ARCHITECTURE events message bus queue for raw data topic

90 THE KAPPA ARCHITECTURE events message bus queue for raw data topic transform

91 THE KAPPA ARCHITECTURE events message bus queue for raw data topic queue for preprocessed data topic transform

92 THE KAPPA ARCHITECTURE events message bus queue for raw data topic queue for preprocessed data topic transform analysis

93 THE KAPPA ARCHITECTURE events message bus queue for raw data topic queue for preprocessed data topic queue for analysis results topic transform analysis

94 THE KAPPA ARCHITECTURE events message bus queue for raw data topic queue for preprocessed data topic queue for analysis results topic transform analysis reporting

95 THE KAPPA ARCHITECTURE events message bus queue for raw data topic queue for preprocessed data topic queue for analysis results topic transform analysis reporting end-user UI

96 DATA FEDERATION IN THE COMPUTE LAYER events transform developer UI web and mobile databases transform aggregate archive file, object storage transform reporting models train management

97 DATA FEDERATION IN THE COMPUTE LAYER events transform developer UI web and mobile databases transform aggregate archive file, object storage transform reporting models train management

98 DATA FEDERATION IN THE COMPUTE LAYER events transform developer UI web and mobile databases transform aggregate archive file, object storage transform reporting models train management

99 SIDEBAR: THE MONOLITHIC SPARK ANTIPATTERN Resource manager Cluster scheduler app 1 app 2 Spark executor Shared FS Spark executor Spark executor app 3 app 4 Spark executor Spark executor Spark executor

100 SIDEBAR: THE MONOLITHIC SPARK ANTIPATTERN Resource manager Cluster scheduler app 1 app 2 Spark executor Shared FS Spark executor Spark executor app 3 app 4 Spark executor Spark executor Spark executor

101 ONE CLUSTER PER APPLICATION Resource manager app 1 app 2 app 3 Object stores app 4 app 5 app 6 Databases

102 ONE CLUSTER PER APPLICATION Resource manager app 1 app 2 app 3 Object stores app 4 app 5 app 6 Databases

103 PRACTICALITIES AND POTENTIAL PITFALLS

104 SCHEDULING Kubernetes app 1 app 2 app 3 Object stores app 4 app 5 app 6 Databases

105 SCHEDULING Kubernetes app 1 app 2 app 1 app 3 app 4 app 5 app 6

106 SCHEDULING Kubernetes app 1 app 2 app 1 app 3 app 4 app 5 app 6

107 SCHEDULING Kubernetes app 1 app 2 app 1 app 3 app 4 app 5 app 6

108 SCHEDULING Kubernetes app 1 app 2 app 1 app 3 app 4 app 5 app 6

109 SCHEDULING Kubernetes app 1 app 2 app 1 app 3 app 4 app 5 app 6

110 STORAGE Kubernetes app 1 app 2 app 3 app 4 app 5 app 6

111 STORAGE Kubernetes app 1 app 2 app 3 POSIX filesystem app 4 app 5 app 6

112 STORAGE Kubernetes app 1 app 2 app 3 POSIX filesystem familiar interface app 4 app 5 app 6 interoperability with other programs

113 STORAGE Kubernetes app 1 app 2 app 3 POSIX filesystem familiar interface app 4 app 5 app 6 interoperability with other programs unnecessary semantic guarantees difficult to manage

114 STORAGE Kubernetes HDFS HDFS HDFS app 1 app 2 app 3 HDFS HDFS HDFS app 4 app 5 app 6

115 STORAGE Kubernetes HDFS HDFS HDFS app 1 app 2 app 3 HDFS HDFS HDFS support for legacy app 4 app 5 app 6 Hadoop installations

116 STORAGE Kubernetes HDFS HDFS HDFS app 1 app 2 app 3 HDFS HDFS HDFS support for legacy app 4 app 5 app 6 Hadoop installations inelastic stateful can t collocate compute and data

117 STORAGE Kubernetes app 1 app 2 app 3 object store app 4 app 5 app 6

118 STORAGE Kubernetes app 1 app 2 app 3 object store interoperability app 4 app 5 app 6 fine-grained AC many implementations

119 STORAGE Kubernetes app 1 app 2 app 3 object store interoperability app 4 app 5 app 6 fine-grained AC many implementations consistency model performance (?)

120 STORAGE Kubernetes app 1 app 2 app 3 object store interoperability app 4 app 5 app 6 fine-grained AC many implementations consistency model performance

121 in a cloud native architecture, the benefit of HDFS is actually very small and that is why many cloud-first organizations no longer run HDFS, or only run it as a caching layer for S3. Reynold Xin on Quora (

122 NETWORKING

123 NETWORKING

124 NETWORKING

125 NETWORKING

126 NETWORKING can t access worker web UI (but wait for Spark 2.1!)

127 NETWORKING can t access worker web UI (but wait for Spark 2.1!)

128 NETWORKING can t access worker web UI (but wait for Spark 2.1!)

129 SECURITY

130 SECURITY

131 SECURITY

132 SECURITY $SPARK_HOME/bin/spark-class \ org.apache.spark.deploy.worker.worker \ master:7077 pid root net

133 SECURITY $SPARK_HOME/bin/spark-class \ org.apache.spark.deploy.worker.worker \ master:7077 pid root net

134 SECURITY $SPARK_HOME/bin/spark-class \ org.apache.spark.deploy.worker.worker \ master:7077 pid root /tmp/foo net

135 SECURITY $SPARK_HOME/bin/spark-class \ org.apache.spark.deploy.worker.worker \ master:7077 pid root /tmp/foo net

136 SECURITY

137 SECURITY k8s namespace

138 SECURITY k8s namespace *

139 NEXT STEPS: FUTURE WORK & PLAYING ALONG AT HOME

140 NEXT STEPS Further performance evaluation Better developer experience Improved scheduling of Spark tasks on Kubernetes

141 TRY IT OUT YOURSELF Kubernetes standalone Spark example: Enabling Spark on OpenShift: Native Spark on Kubernetes proposal:

142