CI/CD @ bol.com
What I ll be talking about 1. 2. 3. 4. 5. About me & bol.com The CI/CD story @ bol.com Current setup Mayfly The future in the cloud
About me Maarten Dirkse @mdirkse on Twitter In IT since 2007 (5 years @ bol.com) Java developer -> CI/CD engineer Bitten by the container bug in 2014 Hobby: local politics
About bol.com Largest online retailer in the Netherlands and Belgium (5.8 million customers, 10+ million products) 55 (and growing) multi-disciplinary teams of 5-8 people Strong Scrum culture (introduced in 2009) 200+ services and apps (SOA, mostly Java + DB backend) Mix of fixed sprint rhythm and CD
The Developer Freedom index
Once upon a build-time...
The situation ca. 2014 4 week release cycle Big-bang release Scrum -> 200 stories per cycle to production Jenkins -> DeployIT -> Schuberg Phillis Shop went offline! Software was thrown over the wall to ops Every team had admin rights on their jenkins jobs
Freedom index: not free Developers couldn t do their own releases at the time of their choosing Releases had to be coordinated with SBP Even some property changes Stack on which apps ran was tightly locked down Developers could go crazy on TST, but could do almost nothing on PRO Endless requests for SSH access to servers which were inevitably denied 2 levels of gatekeeping: ops and SBP On the plus side, they could configure their Jenkins jobs...
Current pipeline
our CD story Man on the Moon to give teams autonomy
How things get to production Build Store Orchestrate Run Deploy
Key aspects TAXP system: custom abstraction over Jenkins jobs No more job admin rights for teams Teams can deploy to PRO at will (have to send notification) TST, ACC environments (ACC is production-like, used for performance tests) No change management process SRT gatekeeper of deploy functionality and new services
Mayfly
r> Genesis of Mayfly te <m as as te <m r> Test <m as r> te as <m Acc Pro te r>
Mayfly idea <branch> <branch> <branch> <branch> <branch> <branch> <branch> <branch> <branch> <branch> <branch> <branch> Pro.........
Mayfly provides per user story: Feature branch in SCM (currently git via Stash) Continuous integration jobs (Jenkins) Isolated, production-like runtime environment (Docker cluster) Automated Definition of Done check Logs & metrics (Logstash, Graphite, Prometheus) Optional user story-specific database (Oracle, PostgreSQL, Mongo)
30% of all commits done via Mayfly
Freedom index: partially free Developers control their releases Developers don t control CI or CD Mayfly offers lots of freedom, until TST
Building and deploying in the cloud
The challenge Build a container-centric, cloud-native CI/CD pipeline that: Is easy to use and get started with Makes it easy to deploy small changes Is fully customizable Can scale to thousands of deploys per day
What about the current stack? Will the CI/CD stack that we use at the moment suffice in the cloud?
The current stack Maintenance nightmare Very expensive docker registry *Not* a deployment tool Jenkins Artifactory Rundeck Puppet Do builds Store artifacts Orchestration of RPM builds and rolling out of artifacts Actually install the new artifact on an existing machine Want immutable infrastructure
The current stack would work, but we can do better by using cloudnative tools
The new stack More developer control & less DPI maintenance Gitlab CI Build docker images Google s concern Google Container Registry Store docker images Actual deployment tool Facilitates immutable infrastructure Spinnaker Kubernetes Deploy dockerized apps Run docker containers
Convention over configuration Convincing over compulsion Opt-out is an option! CI/CD is a product that needs to appeal Iterate on a vision, don t crowdsource the design
Freedom index: free Developers have full control over CI Developers have full control over CD Developers have full control over the stack Well, at least from the kernel up Constraints that do exist are, as much as possible, handled transparently And if opt-out is an option, but comes with many responsibilities
Thanks! Maarten Dirkse