Enabling Low-friction Continuous Deployment via Workflows and Containers

Transcription

1 Enabling Low-friction Continuous Deployment via Workflows and Containers Jon Pliske Tristan Blease Groupon

2 Groupon?

3 Overview Architectural transformation What we are building Technology in context Tools/techniques Conclusion

4 Where we ve been Monolith Monolithic Service Monolithic application architecture One (giant, painful) deploy Deploy source and assets instead of built and published artifacts Deployed Monolith Deployed Monolith Deployed Monolith Production Hosts Deploying the Monolith

5 Where we are Microservices! Emergent architecture has led To diverse ecosystem Spread the pain, but there s still pain Bespoke deployment model Microservice (Artifacts?) Service A Service B Service C Deployed Service A Deployed Service B Deployed Service C Production Hosts Deploying Microservices

6 Keep going?

7 10+ Deployment frameworks (that we know of)

8 650+ Services in production

9 ~260_ Code deployments last week

10 10+ Deployment frameworks 650+ ~260 Services Code deployments last week

11 Where we want to go Artifact-driven deployments Preserve value of microservices Measurable, auditable builds that produce static artifacts Pipeline first approach Consolidate and centralize build and deployment infrastructure Move teams to shared infrastructure Microservice (Artifacts?) Service A Service B Service C Deployed Service A Deployed Service B Deployed Service C Production Hosts

12 Where we want to go Artifact-driven deployments Preserve value of microservices Measurable, auditable builds that produce static artifacts Pipeline first approach Consolidate and centralize build and deployment infrastructure Move teams to shared infrastructure Artifacts Service A Service B Service C Production Cluster Deployed Service A Deployed Service B Deployed Service C Deployed Service C Deployed Service B Deployed Service A

13 What this new approach gives us Static build artifacts with strong quality assertions and configuration built-in Reduces friction for new services Path to continuous deployment Audit trail that emcompasses entire lifecycle of application artifact (build, test, deploy)

14 Technologies aka The Logo Slide Jenkins Docker Activiti DC/OS Mesos Marathon Avi Vantage Wavefront

15 What We've Built So Far...

16 Build Pipeline Deploy Pipeline Architecture Jenkins Deploy Service Client Requests Wavefront (Metrics/Monitoring) Docker Registry Marathon Service VIP Service ServiceVIP VIP AVI Vantage Build Steps Build BuildStep Step N1 N2 N3 N4 Mesos Cluster Deploy Build Step Steps Build Step Activiti BPM Runtime Environment Nn

17 Build Pipeline Deploy Pipeline Architecture Jenkins Deploy Service Client Requests Wavefront (Metrics/Monitoring) Docker Registry Marathon Service VIP Service ServiceVIP VIP AVI Vantage Build Steps Build BuildStep Step N1 N2 N3 N4 Mesos Cluster Deploy Build Step Steps Build Step Activiti BPM Runtime Environment Nn

18 UAT DC/OS Cluster Staging marathon-conveyor1 DC/OS Cluster Production marathon-conveyor1 DC/OS Cluster marathon-conveyor1 Conveyor API LIVE Three separate DC/OS clusters form pipeline to production A single logical instance of Conveyor API All three environments in Live must always be available to prevent blocking the release pipeline

19 UAT DC/OS Cluster Staging marathon-conveyor1 DC/OS Cluster Production marathon-conveyor1 DC/OS Cluster Conveyor API LIVE Staging Stable DC/OS Cluster marathon-conveyor1 marathon-conveyor2 marathon-conveyor3 Conveyor API UAT Unstable DC/OS Cluster marathon-conveyor1 marathon-conveyor2 Conveyor API marathon-conveyor3 marathon-conveyor1

20 Conveyor deployments deploy <artifact> UAT DC/OS Cluster promote <artifact> staging Staging DC/OS Cluster Conveyor API Conveyor CLI promote <artifact> production Production DC/OS Cluster

21 Conveyor artifact-centric deployment process deploy my-app-023fa04 Artifact UAT Staging Production my-app-023fa04 6/1/2016 3:20PM Artifact UAT Staging Production my-app-023fa04 6/1/2016 3:20PM 6/1/2016 4:20PM Artifact UAT Staging Production my-app-023fa04 6/1/2016 3:20PM 6/1/2016 4:20PM 6/1/2016 5:02PM promote my-app-023fa04 staging promote my-app-023fa04 production deploy command only deploys to UAT promote command deploys an artifact that has already been in the previous environment to the next environment Once an artifact has been deployed to all environments, it becomes available for future use by the rollback command

22 Given a deploy history... Regression discovered in current production release Artifact UAT Staging Production my-app-023fa04 6/1/2016 3:20PM 6/1/2016 4:20PM 6/1/2016 5:02PM my-app-b6afdd3 6/2/2016 2:42PM my-app-633f903 6/2/2016 5:21PM my-app-bd6dbf5 6/3/2016 6:01PM my-app-3c864af 6/4/2016 1:03PM my-app-9ed4995 6/5/2016 5:12PM my-app-c9cf245 6/8/2016 5:12PM my-app d 6/9/2016 3:23PM my-app-7a9007c 6/9/2016 4:02PM my-app-023fa04 6/10/2016 4:12PM 6/3/2016 4:12PM 6/4/2016 3:02PM 6/5/2016 4:13PM 6/9/2016 1:22PM 6/11/2016 1:04PM 6/12/2016 4:04PM

23 Given a deploy history... Available for rollback Regression discovered in current production release Artifact UAT Staging Production my-app-023fa04 6/1/2016 3:20PM 6/1/2016 4:20PM 6/1/2016 5:02PM my-app-b6afdd3 6/2/2016 2:42PM my-app-633f903 6/2/2016 5:21PM my-app-bd6dbf5 6/3/2016 6:01PM my-app-3c864af 6/4/2016 1:03PM my-app-9ed4995 6/5/2016 5:12PM my-app-c9cf245 6/8/2016 5:12PM my-app d 6/9/2016 3:23PM my-app-7a9007c 6/9/2016 4:02PM my-app-023fa04 6/10/2016 4:12PM 6/3/2016 4:12PM 6/4/2016 3:02PM 6/5/2016 4:13PM 6/9/2016 1:22PM 6/11/2016 1:04PM 6/12/2016 4:04PM

24 Given a deploy history... rollback my-app-3c864af production Available for rollback Regression discovered in current production release Artifact UAT Staging Production my-app-023fa04 6/1/2016 3:20PM 6/1/2016 4:20PM 6/1/2016 5:02PM my-app-b6afdd3 6/2/2016 2:42PM my-app-633f903 6/2/2016 5:21PM my-app-bd6dbf5 6/3/2016 6:01PM my-app-3c864af 6/4/2016 1:03PM my-app-9ed4995 6/5/2016 5:12PM my-app-c9cf245 6/8/2016 5:12PM my-app d 6/9/2016 3:23PM my-app-7a9007c 6/9/2016 4:02PM my-app-023fa04 6/10/2016 4:12PM my-app-3c864af 6/3/2016 4:12PM 6/4/2016 3:02PM 6/5/2016 4:13PM 6/9/2016 1:22PM 6/11/2016 1:04PM 6/12/2016 4:04PM 6/14/2016 2:02PM

25 Technology in Context

26 Technical barriers Emergent complexity Networking and Load balancing Metrics Continuous Integration Operations Configuration (modified)

27 Conveyor deployments deploy <artifact> UAT DC/OS Cluster promote <artifact> staging Staging DC/OS Cluster Conveyor API Conveyor CLI promote <artifact> production Production DC/OS Cluster

28 Conveyor CLI Deploy state, service metadata, artifact deployment metadata DC/OS UAT NA1 Emergent complexity Postgres DC/OS Staging NA1 Conveyor API POST /marathon/v2/apps DC/OS Production NA1 NA1 Conveyor API DC/OS Production NA2 psql NA2 Marathon HA leader sends HTTP event callbacks Conveyor API psql EMEA1 DC/OS Production EMEA1 Update load balancing layer with task <ip:port> Multiple clusters, multiple datacenters API availability vs. Mesos/Marathon availability Coordinating infrastructure and API updates Single points of failure can be anything outside of cluster (RDBMS, artifact repositories) Must evolve architecture to solve for high availability Host API on cluster Evaluate off-cluster RBDMS

29 Networking and Load balancing Marathon HA leader sends HTTP event callback Conveyor API Core Routers marathon-conveyor Avi Service Engine BGP VIP advertisement; routes to Service Engine pair DC/OS API Marathon Avi Controller Update Avi with task <ip:port> SSH VIP configuration, task locations, cluster state Zookeeper conveyor-master1-production.emea1 Mesos Masters Directing traffic to ephemeral containers Zero downtime upgrades Network topology constraints Firewalls and security zones conveyor-workern-production.emea1 Mesos Workers Top of Rack Switch (TORS)

30 Metrics Cluster = Service Service Hosts Plan for increased metric volume Distinctive metric scaling profile Ephemeral containers/continuity Host/cluster metric collection model breaks down Might need parallel pipelines Service A Service A Service A Mesos Cluster!= Service Mesos Hosts Services span several hosts Service A A A B C A B A B A A C C Service B Service C

31 Additional technical barriers Continuous integration systems are now in critical path to production Provisioning Service runtime configuration (modified)

32 Cultural barriers Organizational expectations Developer expectations Operator expectations

33 Organizational expectations How do we prioritize the correct set of features and architectural improvements in order to provide incremental value to the organization?

34 Developer expectations How much of my code needs to change to support this model? What does change?

35 Operator expectations How do we manage risk and provide transparency during the rollout?

36 Tools and Techniques

37 Gradual service transition GET /browse/chicago Web Request Routing Browse Service (Conveyor) Browse Service (traditional cluster) service

38 Gradual service transition GET /browse/chicago Web Request Routing Browse Service (Conveyor) Browse Service (traditional cluster) service

39 Gradual service transition GET /browse/chicago Web Request Routing Browse Service (Conveyor) Browse Service (traditional cluster) service

40

41 // Events val startevent = StartEvent() val endevent = EndEvent() val failedendevent = TerminateAllEndEvent() Process DSL // Tasks val approvedeploytask = ServiceTask[ApproveDeployDelegate](...) val startjiraissuetask = ServiceTask[JiraTransitionIssueDelegate]( 'executor -> executor, 'env -> env, 'jiraissueid -> jiraissueid, 'newstate -> jiraissuestateinprogress ) // Gateways val skipdeployapprovalgateway = ExclusiveGateway() val wasdeployapprovedgateway = ExclusiveGateway() val skipjiragateway = ExclusiveGateway() // Sequence flows startevent andthen skipdeployapprovalgateway skipdeployapprovalgateway andthen approvedeploytask when deployapprovalrequired skipdeployapprovalgateway andthen skipjiragateway unless deployapprovalrequired

42 Additional tools/techniques Steering committee - cross-organizational buy-in Dogfood-ing - canaries, internal tooling uses Conveyor pipeline Regular re-provisioning of infrastructure

43 Conclusions and open questions Technology and organizational culture are tightly coupled How do we prioritize the correct set of features and architectural improvements in order to provide incremental value to the organization? How do we bridge the gap between complex, distributed systems and their eventual end users (i.e. developers and operators)? How do we prevent alienating developers from the underlying technology and hardware?

44 Questions? Jon Tristan

45 Thank you! Jon Tristan