DevOps in the Cloud A pipeline to heaven?! Robert Cowham BCS CMSG Vice Chair

Transcription

1 DevOps in the Cloud A pipeline to heaven?! Robert Cowham BCS CMSG Vice Chair

2 Agenda Definitions, History & Background Cloud intro DevOps Pipelines Docker containers Examples 2

3 Definitions DevOps Agile Continuous Integration Continuous Delivery Continuous Deployment 3

4 Continuous Delivery - Background Continuous Delivery by Jez Humble and David Farley, published in 2011 It builds on pre-existing practices and methods From the Agile manifesto (2001) Our highest priority is to satisfy the customer through early and continuous delivery of valuable software Other influences Lean software development DevOps 4 4

5 DevOps Deploys per Day: Dev and Ops Cooperation at Flickr First Devopsdays event in Belgium 2010 First US Devopsdays #DevOps Twitter hashtag appears 2013 Books start appearing Wins the Marketing War (with CD) 5

6 Impact of CD on Development Continuous Delivery Repeatable process for release If it hurts, do it more frequently Build quality in Done means released Everybody is responsible Continuous improvement Automate almost everything Keep everything in version control Builds: Push: Releases: Code: Teams: Nightly Hourly Every Submit Weekly Daily Hourly Each Change Yearly Quarterly Monthly Daily Home Grown + 3 rd Party + Open Source Small Big Distributed Global 6

7 Organizational structure and DevOps/CD C O O R D I N A T E T E A M S (design, dev, release, devops, etc.) P I P E L I N E 7 Teams Involved Requirements / Business analysis Development Test / QA Configuration Management Project Management Release / Service Transition Operations Customer Support Etc How does this change?

8 Key metric cycle time How long would it take your organization to deploy a change that involved a single line of code? Do you do this on a repeatable, reliable basis? 8 Mary and Tom Poppendieck, Implementing Lean Software Development (2006)

9 Why the Cloud? Hardware Assets vs People s time Flexibility Scalability You don t own infrastructure so encourages better practices Still possible to be sub-optimal about how you do things with wrong mindset! 9

10 Cloud Providers Amazon Microsoft Google The Rest 10

11 Pipelines (and the Cloud) DevOps in the Cloud

12 Maximize pipeline flow Factors which increase flow Automation - Integration / Testing / etc Timely Feedback Smaller packages of release (Micro Services) Release/deployment technologies (Containers etc) Factors which decrease flow Lack of automation Lack of quality Integrating parallel development (after isolated/independent work) 12

13 13 AWS CodePipeline

14 (Some) Tools involved in the Pipeline Agile ALM IDEs QA/Code Review Build/CI Quality Deployment 14

15 Environment Architecture Work with your architecture team Does deployed system work: In the cloud only? On internal servers? Hybrid? Include client applications? Interface with third party interfaces/services? 15

16 Everything in the Cloud/on Premise Source Repo Build Staging / Test Production 16

17 Hybrid Cloud & on Premise Build Staging / Test Source Repo Production 17

18 18 Build / CI Tools

19 19 Jenkins Build Pipeline

20 Demo A Functioning Pipeline QA on port 9091 Prod on port Helix P4D for code and artifacts Helix Swarm for review Jenkins for pipeline Docker image deployed to local QA and Production environments Also to AWS Registry and Elastic Beanstalk

21 Demo pipeline Auto Auto Manual Application checked out, built and checked back in and labeled Artifacts retrieved from Perforce and build into docker container and checked back in Docker container deployed to QA for testing Docker container deployed to production/elastic Beanstalk 21

22 Deployment Automation Deployment challenges Approaches to automation Infrastructure as Code Provisioning (Configuration Management) tools Docker overview and examples 22

23 Environment Composition/Challenges Servers and other resources Client(s)? Interfaces Data feeds Other services Data / Databases Configuration vs Transaction Managing changes to table structures 23

24 Infrastructure as Code Principles Version everything (tracking / rollback) If it hurts, do it more frequently! Changing administrator mindset Don t just login and fix stuff! Advantages Changes are documented Fewer manual mistakes Increases automation 24

25 Docker History Introduced March 2013, at Python Dev Conference by dotcloud Spans virtualization and configuration management Combines application software and OS filesystems in a single image Ship that image between development, test,, production No recompilation/repackaging required Thinner layer than VM technology 25

26 App Containers vs Virtual Machines App A Bins/Libs App B Bins/Libs Virtual Machine App A Docker App B Guest OS Guest OS Bins/Libs Bins/Libs Hypervisor Host OS Server Docker Engine Host OS Server 26

27 27 Docker a shipping container for code

28 Docker Usage in the Cloud/at Scale AWS Elastic Beanstalk EC2 Container Service Google Kubernetes Docker Swarm 28

29 Docker internals Builds on Unix functionality that has years of history Isolation layer (kernel namespaces) Resource isolation (cgroups) Standard interface (libcontainer) 29

30 Data and filesystems AUFS (Another Unified File System) Other options exist BTRFS Copy-on-Write Layers built up, referencing each other Application (W) Jetty (RO) Java JDK (RO) Ubuntu (RO) Docker Engine AUFS Ref to parent 30

31 Docker definitions A Container is a running instance of an image There can be 100-1,000 on a single host An Image is a static snapshot Based on a series of layers Each of which has a unique hash Images are versioned Can be tagged (Ubuntu: or Ubuntu:latest) Can be updated by applying layer dependencies Images can be stored in a repository/registry DockerHub or internal 31

32 Application Dockerfile FROM jpetstore-builder MAINTAINER Robert Cowham # The built JPetstore application is copied in ADD builds/jpetstore.war /opt/jetty/webapps/root.war # Default command to run ensures process stay up CMD /usr/bin/supervisord 32

33 Listing images $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE jetty-with-jpetstore latest 11f730dfb72d 3 days ago MB jpetstore-builder latest 9057f65b019a 5 days ago MB rcowham/jetty-base latest f eb85 5 days ago 679 MB rcowham/puppet-base latest c06fbc3680bb 5 days ago MB ubuntu d46f days ago 188 MB 33

34 Image layers $ docker history rcowham/puppet-base IMAGE CREATED CREATED BY SIZE c06fbc3680bb 5 days ago /bin/sh -c gem install puppet MB 2c37d1c17dfd 5 days ago /bin/sh -c echo "gem: --no-ri 23 B 1c61a64cd407 5 days ago /bin/sh -c apt-get -y install ruby MB daf9412e days ago /bin/sh -c apt-get -y update MB 11e95b days ago /bin/sh -c #(nop) MAINTAINER "r 0 B 97434d46f days ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0 B 34

35 Running containers The local QA job: docker run --name=jpetstore-qa -d -p 9091:8080 jetty-with-jpetstore The local Production job: docker run --name=jpetstore-prod -d -p 9092:8080 jetty-with-jpetstore 35

36 36 AWS Elastic Beanstalk

37 37 Web based architecture

38 38 Team Size

39 A pipeline to DevOps Cloud Heaven! Mindset more than toolset But lots of tools exist! Need good configuration management for a scalable, reliable foundation for DevOps Learn from the Cloud best practices Learn about Docker (containers) Get pizzas for your teams! 39

40 40 Questions?