Thinking Inside the Container: A Continuous Delivery Story Engineering Manager: Riot Games

Transcription

1 Thinking Inside the Container: A Continuous Delivery Story Engineering Manager: Riot Games 1

2 Who s This Guy? 2

3 3

4 A Quick Note on Riot Games 4

5 The Scale of League MORE THAN 67 MILLION MONTHLY ACTIVE PLAYERS MORE THAN 27 MILLION DAILY ACTIVE PLAYERS 7.5 MILLION PEAK CONCURRENT PLAYERS 5

6 1.25 Million Builds a Year 6

7 Thinking Inside the Container 7

8 10,000-14,000 Containers A Week 8

9 120 Jobs An Hour 9

10 A Containerized Build Farm BUILD HOSTS DOCKERHOST JENKINS SSH Docker API SWARM Docker API DOCKERHOST DOCKERHOST DOCKERHOST DOCKERHOST Docker API DRYDOCK CentOS 7.2/Docker cadvisor Docker-GC Container Metrics 4 Core/32GB RAM/120GB LVS 10

11 STORY TIME! (aka. Presenting the Problem) 11

12 3500+ Unique Build Jobs 12

13 650+ Builds an Hour 13

14 90+ Build Slaves 14

15 Expecting 100 More! 15

16 Story Time... Ticket Engineers Build Team 16

17 Story Time... Engineers Build Team 17

18 What Did We Want? TEAMS NEEDED TO MOVE FAST PRODUCTS HAD TO OWN THEIR STACK CONFIGURA TION AS CODE 18

19 Maybe We Want... 19

20 Along Came a Whale (More Story Time!) 20

21 21

22 Oh Look! Another Way to Deploy! 22

23 Docker For Newbz 23

24 24

25 Jenkins Primer 25

26 Jenkins Primer Master Slaves JENKINS Win32 + Java + TeamA Cent7 + Java + TeamB Cent7 + GoLang + TeamB 26

27 Jenkins Primer Master Slaves Build Job Win32 + Java + TeamA JENKINS Win32 + Java + TeamA Cent7 + Java + TeamB Cent7 + GoLang + TeamB 27

28 Jenkins Primer Master Slaves Queue Build Job Win32 + Java + TeamA Build Job Build Job Build Job Build Job JENKINS Win32 + Java + TeamA Cent7 + Java + TeamB Cent7 + GoLang + TeamB 28

29 Jenkins Primer Master Slaves Queue Build Job Win32 + Java + TeamA Build Job Build Job Build Job Build Job JENKINS Win32 + Java + TeamA Cent7 + Java + TeamB Cent7 + GoLang + TeamB 29

30 What Did We Need? CONTAINER RUNNING AS A BUILD SLAVE WAY TO HAVE ENGINEERS PROVIDE DOCKERFILES HOW TO SCALE IT PROFIT? 30

31 A Build Slave Container 31

32 Add a Bit of Secret Sauce... Base Slave Image Universal Riot Tools Product/App Specific Build Stuff 32

33 A Real Example 33

34 Provisioning and Plugins + =? 34

35 Of Whales and Plugins DOCKER PLUGIN MESOS PLUGIN KUBERNETES PLUGIN 35

36 Of Whales and Plugins DOCKER PLUGIN MESOS PLUGIN KUBERNETES PLUGIN 36

37 Quick Look 37

38 Provisioning and Plugins Image Name Label 38

39 Groovy To the Rescue ttps://github.com/maxfields2000/ ockerjenkins_tutorial/tree/master/jenkinsscripts 39

40 We Created A Monster 40

41 We Need to Inspect Our Containers 41

42 We Need to Inspect Our Containers./harbormaster --registry_user "$REGISTRY_USER" --registry_token "$REGISTRY_TOKEN" --jenkins_user $USER -- jenkins_pass $PASSWORD --jenkins --dockerhost $TEST_DOCKER_HOST "$IMAGENAME" "$CLOUDNAME" "$LABELNAME" 42

43 Are We Done? 43

44 But It Needs to Scale! 44

45 If Only... 45

46 If Only... 46

47 But It Needs to Scale! BUILD HOSTS DOCKERHOST JENKINS SSH Docker API SWARM Docker API DOCKERHOST DOCKERHOST DOCKERHOST DOCKERHOST CentOS 7.2/Docker cadvisor Docker-GC Container Metrics 4 Core/32GB RAM/120GB LVS 47

48 Putting It All Together Image REGISTRY Engineer 48

49 Putting It All Together Image Name + Jenkins Label REGISTRY HARBOR MASTER Engineer 49

50 Putting It All Together REGISTRY HARBOR MASTER Engineer Runs Tests 50

51 Putting It All Together REGISTRY HARBOR MASTER SWARM DOCKERHOST DOCKERHOST DOCKERHOST Engineer Image Pulled to Swarm 51

52 Putting It All Together Groovy API Called Jenkins REGISTRY HARBOR MASTER SWARM DOCKERHOST DOCKERHOST DOCKERHOST Engineer 52

53 Build Job Quick Look 53

54 Dockerception 54

55 Where to Build Containers? BUILD HOSTS DOCKERHOST JENKINS SSH Docker API SWARM Docker API DOCKERHOST DOCKERHOST DOCKERHOST DOCKERHOST Docker API DRYDOCK CentOS 7.2/Docker cadvisor Docker-GC Container Metrics 4 Core/32GB RAM/120GB LVS 55

56 Mai Tai s On the Beach Build Engineer 56

57 How Do You Actually Build It? 57

58 How Do You Actually Build It? (engineering.riotgames.com) 58

59 59

60 You Make This Sound Simple. 60

61 Lesson 1 - Docker Isn t Simple DOCKERFILES REQUIRE DECENT SYSTEMS ADMIN KNOWLEDGE DOCKER IMAGES THAT BUILD DOCKER IMAGES IS DOCKERCEPTION DOCKER VOODOO AND BLACK MAGIC QUESTIONS 61

62 Lesson 2 - Containers!= VM s CANNOT MOUNT REMOTE FILE SYSTEMS EPHEMERAL CONTAINERS DON T MAINTAIN STATE HAVE RULES FOR DOCKERFILE VS RUN TIME 62

63 Lesson 3 - Garbage Collection DOCKER BUILD DOCKER RUN DOCKER PULL VOLUME 63

64 Lesson 4 - Maintenance/Failure PULL HOSTS ON/OFFLINE UPDATE ALL IMAGES ROLLING RESTARTS 64

65 Lesson 5 - How Will You Upgrade? 65

66 Lesson 6 - Credentials & Security WHAT SHOULD YOU PUT IN YOUR BASE IMAGE? SSH KEYS IN YOUR REGISTRY? PASSWORDS IN YOUR SOURCE CODE? PASSING EVERYTHING IN AS A ENVIRONMENT VARIABLE? 66

67 67

68 Over 1200 New Build Jobs Created 68

69 30% of All Environments Are Containers 69

70 Environment Change/Create/Fix Tickets Dissappeared 70

71 We No Longer Have Just A Build Team 71

72 QUESTIONS? QUESTIONS? 72