From Dev to DevOps: An Unexpected Journey. Luis Angel Vicente Sanchez BigCentech Ltd

Transcription

1 From Dev to DevOps: An Unexpected Journey Luis Angel Vicente Sanchez BigCentech Ltd

2 From Zero to Hero? 2

3 3 What is DevOps? By Devops.png: Rajiv.Pantderivative work: Wylve - This file was derived from Devops.png:, CC BY 3.0,

4 4 What is DevOps? Source Automation Configuration Management Logging Monitoring Alerting

5 5 What is DevOps? Source Automation Infrastructure as Code Logging Monitoring Alerting

6 What is DevOps? 6

7 7 Who am I? 12+ Software Engineer Big fan of functional programming Luis Angel Vicente Sanchez Lately a lot of Big Data

8 What am I doing here? 8

9 What am I doing here? 9

10 An Unexpected Journey 10

11 An Unexpected Journey 11

12 An Unexpected Journey 12

13 An Unexpected Journey 13

14 Deploying to AWS 14

15 Deploying to AWS: Terraform 15

16 16 Deploying to AWS: Terraform modules

17 17 Deploying to AWS: EC2 instances resource "aws_instance" "dse-cassandra" { count = "${var.count}" ami = "${var.ami_id}" key_name = "datalake" instance_type = "${var.instance_type}" availability_zone = "${var.availability_zones[...]}" subnet_id = "${var.subnet_ids[var.availability_zones[...]]}" vpc_security_group_ids = [ "${var.sg_ids}" ] tags = { Name = "${format("%s-%d", var.tag_name, count.index + 1)}" } }

18 18 Deploying to AWS: EBS volumes resource "aws_ebs_volume" "dse-cassandra-ebs-data" { count = "${var.count}" availability_zone = "${var.availability_zones[...]}" size = "${var.data_ebs_size}" type = "${var.data_ebs_type}" tags { Name = "${var.tag_name} - Data Vol" } }

19 19 Deploying to AWS: EBS attachments resource "aws_volume_attachment" "dse-cassandra-ebs-data-attach" { count = "${var.count}" device_name = "/dev/xvdh" volume_id = "${aws_ebs_volume.dse-cassandra-ebs-data.*.id[count.index]}" instance_id = "${aws_instance.dse-cassandra.*.id[count.index]}" force_detach = true }

20 20 Deploying to AWS: Provisioning resource "null_resource" "provision-dse-cassandra" {... connection { user = "${var.ssh_user}" host = "${element(aws_instance.dse-cassandra.*.private_ip, count.index)}" agent = true private_key = "${file("${var.ssh_identity_file}")}" }... }

21 21 Deploying to AWS: Provisioning resource "null_resource" "provision-dse-cassandra" {... provisioner "file" { source = "${var.provisioning_scripts}/dse-cassandra" destination = "/tmp/cassy-up" } } provisioner "file" { source = "${var.provisioning_scripts}/environments/aws/dse_cassandra.sh" destination = "/tmp/cassy-up/dse_cassandra.sh" }...

22 22 Deploying to AWS: Provisioning resource "null_resource" "provision-dse-cassandra" {... provisioner "remote-exec" { inline = [ "echo 'export CASSANDRA_CLUSTER_NAME=${var.cluster_name}'...",... "chmod -R a+x /tmp/cassy-up/*", "sudo /tmp/cassy-up/dse_cassandra.sh" ] } }

23 Kubernetes 23

24 24 Kubernetes: Why? Self-healing Service Discovery Horizontal Scaling Rolling upgrades and rollbacks

25 25 Kubernetes: Resize the cluster KOPS

26 26 Kubernetes: Resize the cluster > kops create cluster --zones eu-west-1a,eu-west-1b,eu-west-1c $CLUSTERNAME > kops edit cluster $CLUSTERNAME > kops update cluster $CLUSTERNAME --yes > kops export kubecfg $CLUSTERNAME

27 27 Kubernetes: Resize the cluster > kops edit ig nodes > export KOPS_FEATURE_FLAGS="+DrainAndValidateRollingUpdate" > kops rolling-update cluster --yes

28 28 Kubernetes: Deploying kubectl Pods Services

29 Kubernetes: What is a Pod? apiversion: v1 kind: Pod metadata: name: nginx labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerport: 80 kubectl create -f 29

30 Kubernetes: What is a Service? apiversion: v1 kind: Service metadata: name: nginx-service labels: app: nginx spec: ports: - port: 80 targetport: 80 selector: app: nginx kubectl create -f 30

31 Kubernetes: What is a Deployment? apiversion: apps/v1beta1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerport: 80 kubectl create -f 31

32 32 Kubernetes: How to debug? 12 Factor App

33 Kubernetes: How to debug? 33

34 34 Kubernetes: How to debug? Node 0 fluentd Node 1 fluentd ES CLUSTER KIBANA Node 3 fluentd

35 Kubernetes: Problems? 35

36 36 Kubernetes: Problems? HEAPSTE R

37 37 Kubernetes: Problems? Node 0 Pod 0 Pod 1 Node 1 Pod 2 Pod 3 HEAPSTER INFLUXDB Node 3 Pod 4 Pod 5 GRAFANA

38 38 Kubernetes: Problems? Node 0 Pod 0 Pod 1 Node 1 Pod 2 Pod 3 TELEGRAF INFLUXDB Node 3 Pod 4 Pod 5 GRAFANA

39 Deployment Pipeline 39

40 Deployment Pipeline: Gitlab CI 40

41 Deployment Pipeline: Stages and Jobs 41

42 42 Deployment Pipeline: Stages Build Test Deploy Build Test Publish Container Deploy (on commit) (on commit) (on tag)

43 43 Deployment Pipeline: Stages stages: - build - test - publish-container - deploy

44 44 Deployment Pipeline: Jobs build: stage: build script: - npm install - npm run build publish-container: stage: publish-container image: heartysoft/docker-builder-aws: script: - chmod +x./ci/* -./ci/publish-container.sh "app-ui" "$CI_COMMIT_TAG" "app/app-ui" only: - tags

45 45 Deployment Pipeline Build (on commit) Test (on commit) Publish Container (on tag) Deploy (on tag)

46 46 Deployment Pipeline: How to deploy? kubect l

47 47 Deployment Pipeline: How to deploy? kubect l

48 Deployment Pipeline: Helm 48

49 49 Deployment Pipeline: Helm Chart mychart charts Chart.yaml templates deployment.yaml _helpers.tpl ingress.yaml NOTES.txt service.yaml values.yaml

50 50 Deployment Pipeline: Chart.yaml apiversion: v1 description: A Helm chart for Kubernetes name: mychart version: 0.1.0

51 51 Deployment Pipeline: values.yaml replicacount: 1 image: repository: nginx tag: stable pullpolicy: IfNotPresent service: name: nginx type: ClusterIP externalport: 80 internalport: 80 resources: {}

52 Deployment Pipeline: templates/service.yaml apiversion: v1 kind: Service metadata: name: {{ template "fullname". }} labels: app: {{ template "name". }} chart: {{.Chart.Name }}-{{.Chart.Version replace "+" "_" }} release: {{.Release.Name }} heritage: {{.Release.Service }} spec: type: {{.Values.service.type }} ports: - port: {{.Values.service.externalPort }} targetport: {{.Values.service.internalPort }} protocol: TCP name: {{.Values.service.name }} selector: app: {{ template "name". }} release: {{.Release.Name }} 52

53 53 Deployment Pipeline: Deploy from filesystem > helm upgrade --install -f dev-values.yaml myapp./mychart

54 54 Deployment Pipeline: Helm Repo repo index.yaml myapp tgz myapp tgz myapp tgz myapp tgz myapp tgz myapp tgz

55 55 Deployment Pipeline: Helmet /

56 56 Deployment Pipeline: Deploy from helm repo > helm upgrade --install --version f dev-values.yaml myapp helmet/mychart

57 57 Deployment Pipeline: Stages Build (on commit) Test (on commit) Publish Container (on tag) Publish Chart (on tag) Deploy (on tag)

58 Questions? 58

59 THANK YOU!