Empowering Developers to Take Flight Continuous Integration at Gogo

Transcription

1 Empowering Developers to Take Flight Continuous Integration at Gogo Darin Pantley - Gogo June 26, 2017 Room #304

2 Introduction The Problem The Dream Implementation Lessons Learned

3 Introduction About Gogo Gogo is the inflight internet company At a glance: 17 commercial airline partners 3,000+ commercial aircraft in-service 1,500+ 2Ku backlog 120M+ connectivity sessions to date 85,000+ sessions/day 8,000+ flights/day Gogo equipped flight every 11 seconds 1,300+ employees

4 Introduction Gogo Products and Services Gogo s open architecture is designed to power passenger experience like no other in our industry Flexible passenger experience Entertainment Passenger Connectivity Entertainment/ Messaging Connected Aircraft Services

5 Introduction June 27, 2017 Chicago Coder Conference 5

6 Introduction Infrastructure Requirements Hardware: airplanes, devices, WAPs, satellites, servers, etc. Software: In the air, on the ground Local, off-site, outsourced

7 The Problem

8 The Problem June 27, 2017 Chicago Coder Conference 8

9 The Problem Disc icons from June 27, 2017 Chicago Coder Conference 9 Computer icon made by Panier Avide

10 The Problem Software Engineering: Before Improvements 1. Drink coffee 2. Write some code 3. Compile your changes 4. Run some unit tests (optional) 5. Commit your changes 6. Create a deployable artifact June 27,

11 The Problem Software Engineering: Before Improvements 7. Copy it onto a USB stick 8. Use the elevator to get to the lab 9. Prepare the lab machine obtain a monitor, plug in cables 10. Plug in your USB stick June 27,

12 The Problem Software Engineering: Before Improvements 11. Boot it up! 12. Hold your breath 13. Fix the environment so it looks pretty clean 14. Start manually testing your changes June 27,

13 The Problem Software Engineering: After Improvements 1. Drink coffee better coffee machines 2. Write some code send developers to conferences 3. Compile your changes faster laptops 4. Run some unit tests (optional) not optional 5. Commit your changes do this early in the day 6. Create a deployable artifact lunch time June 27,

14 The Problem Software Engineering: After Improvements 7. Copy it onto a USB stick USB hub 8. Use the elevator to get to the lab work from the lab 9. Prepare the lab machine guard it with your life obtain a monitor, plug in cables 10. Plug in your USB stick SCP onto a running system June 27,

15 The Problem Software Engineering: After Improvements 11. Boot it up! keep it running 12. Hold your breath breathe deeply 13. Fix the environment so it looks pretty clean scripts 14. Start manually testing your changes hiring time June 27,

16 The Problem What s Wrong? It takes too long to release new features Many bugs aren t caught during code reviews It s difficult to obtain a test environment Everything is a manual process Running automated tests is optional We discover bugs too late All of this is expensive June 27, 2017 Chicago Coder Conference 16

17 The Problem Our Infrastructure AWS GitLab Jenkins Artifactory Docker Ansible Vagrant Redis June 27,

18 The Dream

19 The Dream Replace Legacy Processes with Best Practices Distributed Version Control System Agile Development Robust Infrastructure Immutable and scalable Monitoring, metrics, and alerts Version controlled Continuous Integration June 27, 2017 Chicago Coder Conference 19

20 The Dream June 27,

21 The Dream Continuous Integration Automatically triggered by merge requests Automated Building Run static analysis Compile the code Run automated tests in the build environment Automated Packaging Create a deliverable Automated Deployments Run automated tests in the deployed environment If everything worked, let the merge proceed June 27,

22 The Dream Reality: Maybe we can do this later? 1. Our code is in SVN 2. We ve never heard of coupling and cohesion 3. Our tests aren t automated 4. Everyone is busy June 27, 2017 Chicago Coder Conference 22

23 The Dream Reality: Maybe we can do this later? 5. We just ordered more lab equipment 6. Feature development cannot be interrupted 7. Our code must run on physical hardware June 27, 2017 Chicago Coder Conference 23

24 The Dream We can make improvements 1. Our code is in SVN Hard cut-over to Git 2. We ve never heard of coupling and cohesion Componentize what you can now, the rest can wait 3. Our tests aren t automated Provide a way to run them, regardless of whether they exist 4. Everyone is busy Prioritize continuous integration June 27, 2017 Chicago Coder Conference 24

25 The Dream We can make improvements 5. We just ordered more lab equipment Some tests will still require real hardware 6. Feature development cannot be interrupted Avoid major changes near releases 7. Our code must run on physical hardware Virtualize the hardware environment June 27, 2017 Chicago Coder Conference 25

26 The Dream Sell the Dream Diagram out a continuous integration workflow June 27, 2017 Chicago Coder Conference 26

27 The Dream Sell the Dream Diagram out a continuous integration workflow June 27, 2017 Chicago Coder Conference 27

28 The Dream Sell the Dream Diagram out a continuous integration workflow Present your idea to developers and upper management Postpone solution space discussions Agree on the goals Agree on responsibilities Engineering DevOps June 27, 2017 Chicago Coder Conference 28

29 Implementation

30 Implementation Overview from a DevOps perspective 1. JobDSL 2. Jenkins Pipeline Job 3. Jenkins Pipeline Library pre-defined targets DevOps tools June 27, 2017 Chicago Coder Conference 30

31 Implementation Overviewfrom a developer perspective runway/ static-analysis build unit-tests integration-tests smoke-tests fast-functional-tests slow-functional-tests end-to-end-system-tests dsl.groovy version June 27, 2017 Chicago Coder Conference 31

32 Implementation Pipeline Types ci nightly iteration release June 27, 2017 Chicago Coder Conference 32

33 Implementation GitLab Merge Requests June 27, 2017 Chicago Coder Conference 33

34 Implementation Jenkins Pipeline June 27, 2017 Chicago Coder Conference 34

35 Implementation runway / build #!/bin/bash -ex export SHORT_SHA=$(git rev-parse --short HEAD) export VERSION="$(cat runway/version).${build_number}~$short_sha" export FULL_SHA=$(git rev-parse HEAD) export GIT_COMMIT="$FULL_SHA $(git log -1 --pretty=%b head -1)" docker login gogo-docker.jfrog.io -u builder -p hunter2 docker run --rm -w /build -v $(pwd):/build/ gogo-docker.jfrog.io/ buildmachine: /build.sh /build/runway/fs_root $VERSION $GIT_COMMIT June 27, 2017 Chicago Coder Conference 35

36 Implementation runway / dsl.groovy import utilities.continuousintegration import utilities.nightlybuild REPO = 'git@gitserver.gogoair.com:acpu/hal.git' def ci = new ContinuousIntegration(job: pipelinejob('${src_job}_ci')) ci.createjenkinspipeline() def nightly = new NightlyBuild(job: pipelinejob('${src_job}_nightly')) nightly.createjenkinspipeline(repo) June 27, 2017 Chicago Coder Conference 36

37 Implementation ContinuousIntegration.groovy this.job.with { } description("continuous Integration Pipeline") url('git@gitserver.gogoair.com:common/jenkinspipeline.git') scriptpath('continuousintegration.jenkinsfile') June 27, 2017 Chicago Coder Conference 37

38 Implementation Jenkins Pipeline Shared Libraries (root) +- src # Groovy source files +- org +- foo +- Bar.groovy # for org.foo.bar class +- vars +- foo.groovy # for global 'foo' variable +- foo.txt # help for 'foo' variable +- resources # resource files (external libraries only) +- org +- foo +- bar.json # static helper data for org.foo.bar June 27, 2017 Chicago Coder Conference 38

39 Implementation ContinuousIntegration.Jenkinsfile node('ci') { withenv(["pipeline=ci"]) { staticanalysis() buildcomponent() unittests() uploadcomponent() June 27, 2017 Chicago Coder Conference 39

40 Implementation vars / buildcomponent.groovy... dir(component_dir) { }... def target = 'runway/build' if (fileexists(target)) { } sh target June 27, 2017 Chicago Coder Conference 40

41 Implementation Organizing the code Artist Written in Python CLI tool Graffiti Written in Python CLI tool Artifactory GitLab Hosted Graphite June 27, 2017 Chicago Coder Conference 41

42 Lessons Learned

43 Lessons Learned Working with Developers Meet frequently with everyone involved Deliver features incrementally Spread access to tooling Extend internal solutions to cover developer use cases Document answers to questions Agree on solutions to technical debt in existing code June 27, 2017 Chicago Coder Conference 43

44 Lessons Learned Working without Developers Create alternatives to technical debt in existing code Create proof of concept projects Choose between competing technologies Demo solutions to controversial problems Prove that something is possible Working out-of-band allows constant refactoring June 27, 2017 Chicago Coder Conference 44

45 Lessons Learned Hindsight is 20/20 Explicitly define requirements up-front General functionality, documentation, metrics, alerts, Plan your current features with the next features in mind Find users quickly Eat your own dog food Feed the hungry Avoid bus errors June 27, 2017 Chicago Coder Conference 45

46 Lessons Learned Overheard on Slack I can show you how to use Artist to upload. It s easy. Should I merge? The button is red. June 27, 2017 Chicago Coder Conference 46

47 Thank You Empowering Developers to Take Flight Continuous Integration at Gogo Darin Pantley 3:45pm 4:45pm Room #304 June 27,