RAFT library for Java

Size: px

Start display at page:

Download "RAFT library for Java"

Justin Chase
5 years ago
Views:

1 k8s : h4p

4 RAFT library for Java

5 RAFT library for Java

6 RAFT library for Java

7 RAFT library for Java

22 What is Apache Hadoop

23 What is Apache Hadoop in 60 seconds

26 HDFS

27 HDFS YARN

28 HDFS YARN Mapreduce

29 HDFS YARN Mapreduce Ozone + HDDS

31 Dockerfile FROM frolvlad/alpine-oraclejdk8 ADD hadoop tar.gz /opt WORKDIR /opt/hadoop

33 <configuration> <property> <name>dfs.namenode.rpc-address</name> <value>namenode:9000</value> </property> <property> <name>dfs.datanode.plugins</name> <value>org.apache.hadoop.ozone.hddsdatanodeservice</value> </property> <property> <name>rpc.metrics.percentiles.intervals</name> <value>60,300</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/data/namenode</value> </property> <property> <name>rpc.metrics.quantile.enable</name> <value>true</value> </property> </configuration>

34 version: "3" services: service1: image: apache/imagename hostname: namenode ports: :9870 environment: CONFIGURATION1: value DFS_DIR: /dfs THREAD_NUMBER: 1

35 How to handle configuration? Create a simple launcher script to Create config file from environment variables Start the application

36 version: "3" services: namenode: image: flokkr/hadoop hostname: namenode command: ["hdfs","namenode"] ports: :9870 environment: ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name" CORE-SITE.XML_fs.defaultFS: "hdfs://namenode:9000" HDFS-SITE.XML_dfs.namenode.rpc-address: "namenode:9000" HDFS-SITE.XML_dfs.replication: "1" datanode: image: flokkr/hadoop command: ["hdfs", "datanode"] environment: CORE-SITE.XML_fs.defaultFS: "hdfs://namenode:9000" HDFS-SITE.XML_dfs.namenode.rpc-address: "namenode:9000" HDFS-SITE.XML_dfs.replication: "1" LOG4J.PROPERTIES_log4j.rootLogger: "INFO, stdout" LOG4J.PROPERTIES_log4j.appender.stdout: "org.apache.log4j.consoleappender" LOG4J.PROPERTIES_log4j.appender.stdout.layout: "org.apache.log4j.patternlayout" LOG4J.PROPERTIES_log4j.appender.stdout.layout.ConversionPattern: "%d{yyyy-mm-dd HH:mm:ss} %-5p

37 Configuration management is managing the configuration values

38 ENV (script) n/a n/a

43 Container is the unit of packaging.

44 Launcher script has the power

45 Launcher script

46 Launcher script Create config files from ENV

47 Launcher script Create config files from ENV Wait for the dependency (TCP check)

48 Launcher script Create config files from ENV Wait for the dependency (TCP check) Download additional optional component

49 Launcher script Create config files from ENV Wait for the dependency (TCP check) Download additional optional component Prepare HDFS (format namenode,...)

50 Launcher script Create config files from ENV Wait for the dependency (TCP check) Download additional optional component Prepare HDFS (format namenode,...) Retrieve kerberos/ssl secrets

51 Launcher script Create config files from ENV Wait for the dependency (TCP check) Download additional optional component Prepare HDFS (format namenode,...) Retrieve kerberos/ssl secrets Enable prometheus monitoring (Java agent)

52 Launcher script Create config files from ENV Wait for the dependency (TCP check) Download additional optional component Prepare HDFS (format namenode,...) Retrieve kerberos/ssl secrets Enable prometheus monitoring (Java agent) Show network traffic (Instrumentation with Java agent)

58 Hashicorp stack "do it yourself"

70 Docker network

71 Host network

72 docker host host dns

73 <configuration> <property> <name>dfs.namenode.rpc-address</name> <value>namenode:9000</value> </property> <property> <name>dfs.datanode.plugins</name> <value>org.apache.hadoop.ozone.hddsdatanodeservice</value> </property> <property> <name>rpc.metrics.percentiles.intervals</name> <value>60,300</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/data/namenode</value> </property> <property> <name>rpc.metrics.quantile.enable</name> <value>true</value> </property> </configuration>

83 DIY

85 Consul Yes (script) Restart host netw Nomad.nomad redeploy no yes host netw yes consul yes host

87 Kubernetes "out of the box"

93 +Network! +Storage!! (volume, secrets, configs)

94 DaemonSet

95 ReplicaSet

99 Service Pod

100 StatefulSet

101 DaemonSet

102

103

104

105 Benefits of Hadoop + k8s?

106 Benefits of Hadoop + k8s? Ecosystem Flexibility

107 Example: Monitor Hadoop with Prometheus

108

109

110

111 apiversion: apps/v1beta1 kind: StatefulSet metadata: name: ozone-hdfs-namenode spec: servicename: ozone2-hdfs-namenode replicas: 1 template: metadata: labels: app: ozone spec: containers: - name: hdfs-namenode image: flokkr/ozone:2.1.0 args: ["hdfs","namenode"]

112 apiversion: apps/v1beta1 kind: StatefulSet metadata: name: ozone-hdfs-namenode spec: servicename: ozone2-hdfs-namenode replicas: 1 template: metadata: labels: app: ozone annotations: prometheus.io/scrape: "true" prometheus.io/port: "28942" spec: containers: - name: hdfs-namenode image: flokkr/ozone:2.1.0 args: ["hdfs","namenode"]

113

114

115 Pod

116 Sidecar pattern Pod

117 apiversion: apps/v1beta1 kind: StatefulSet metadata: name: ozone-hdfs-namenode spec: servicename: ozone2-hdfs-namenode replicas: 1 template: metadata: labels: app: ozone annotations: prometheus.io/scrape: "true" prometheus.io/port: "28942" spec: shareprocessnamespace: true containers: - name: hdfs-namenode image: flokkr/ozone:2.1.0 args: ["hdfs","namenode"] - name: jmxpromo image: flokkr/jmxpromo-sidecar

118 configmap helm n/a CNI kubectl helm, yaml yes namespaces yes CNI statefuset DNS no service/ingress

119 Summary

120 Summary Don't buy without checking the label

121 Summary Don't buy without checking the label Hadoop is first class citizen of cloud-native/containerized environments*

122 Summary Don't buy without checking the label Containerization can help a lot to manage our Bigdata clusters Hadoop is first class citizen of cloud-native/containerized environments*

123 Q&A Apache Roadshow EU: Kubernetes + Hadoop + Ozone 13th June, 14:20 Márton (bigdata + containers project) (source) elek@apache.org

124 Image credits Yan Pritzker (CC) Carrie Cizauskas (CC)

Code: Slides:

Code: Slides: Workshop Resources Code: https://github.com/beekpr/public-workshops Slides: https://tinyurl.com/yc2uo3wk Make sure minikube and kubectl is setup (labs/1-setup-cluster.md has some instructions) Kubernetes