CONTINUOUS INTEGRATION; TIPS & TRICKS

BIO

I DO TECH THINGS

I DO THINGS

BLUE OCEAN

BEEP BEEP

REFACTOR PEOPLE S HOUSES

MY TIPS & TRICKS FOR CI - CI Infrastructure - CI Architecture - Pipeline Architecture - CI Security - CI Debugging - Docker Tips

CI SERVER ARCHIECTURE - Dockerized workers? docker in docker? Can t be good for stability or speed - Availability: Set your executors to 4 per machine (odd numbers are for weirdos) - Speed: If you want pipeline speed, set your executor to 1 per build

PICK YOUR CI TOOL - Pick a CI Product - Jenkins: gives you everything you want - gotta know how to tame the beast - GoCD / DroneCD - for hipsters - it ll die out and you ll be back to Jenkins again anyway - CircleCI / Bitbucket / Gitlab - they re alright

OWN YOUR OWN SLAVE INFRA - Always use your own infra - don t use shared infra - it s slow - it s insecure - you don t control its speed or execution times - you can t SSH into the boxes and do stuff

CLOUD DOCKER BUILD - AVOID - docker hub - quay.io - google s one - slow execution time - queued - slow servers

CONTROL YOUR DOCKER BUILD - Give yourself a 32 core machine - it s cheap - AWS spot instance is like $2 / hour - Run all your docker builds in parallel - easy peasy

CONTROL YOUR DOCKER BUILD

CI INFRASTRUCTURE - Use AWS EC2 Plugin (Best) - Use terraform plugin (AWS Slaves) - Write your own bash to register your slave

CREATE YOUR OWN AMI - put things in it like: - java, docker, docker-compose, - pre-download tools/libraries you need at runtime

CHOOSE YOUR INSTANCE SIZE JENKINS MASTER (dockerized) BIG SLAVE (x1) BIG SLAVE (x1) SMALL SLAVE (x4) BIG SLAVE (x1) BIG SLAVE (x1) SMALL SLAVE (x4) BIG SLAVE (x1) BIG SLAVE (x1) SMALL SLAVE (x4)

WARMING YOUR SLAVES - docker pull commonly used images - docker pull base images (nginx, upstream?) - put some creds on the box, if you really need to

PIPELINE ARCHITECTURE

FAST BUILD INITIALISATION - Skip fetching tags (git) (defer until the end) - Shallow Clone (fast) - Jenkins Optimized Mode

SETUP A BUILD TAG - ASAP

ABSTRACT YOUR PIPELINE STEPS

ABSTRACT YOUR PIPELINE - Use bash - You can re-run the same commands on local or on CI box when debugging

TURN OFF PARTS OF YOUR PIPELINE

CHECKING FOR ON/OFF

SETUP COMMIT HOOKS

INIT

NOTIFY SLACK

INIT

LINTING

BENEFITS OF LINTING - you get early feedback on your code before even thinking about a docker build or npm install - pre-built environments where linting tools exist (docker images) - put your code into the container, run the linter, publish the results

THINGS YOU CAN LINT - terraform validate - spellcheck (check bash scripts) - cloudformation: github.com/awslabs/cfn-python-lint - docker-compose -f docker-compose.yml config - SCSS: github.com/brigade/scss-lint - TypeScript - TSLint - github.com/palantir/tslint - github.com/jakubonderka/php-parallel-lint - Dockerfile: github.com/projectatomic/dockerfile_lint

NODE - ASSET COMPILATION

USING NODE_MODULES?

PARALLEL NODE ING - don t run npm install in a Dockerfile unless you re shipping npm into prod - extract that out to bash scripts that you mount into the container and run for you

COMPILING ASSETS IN PARALLEL

PARALLEL NODE ING

DOCKER BUILDING

DOCKER BUILDING - docker build --pull -t ${BACKEND_IMAGE_FULL}. - always push these to the registry, if all docker builds steps pass, even if test suites fail - you can then pull down the images onto localhost and debug them

UNIT TESTING

UNIT TESTING - don t just run only unit tests - run prep steps for next stage

INTEGRATION / E2E TESTING

PARALLELISATION - parallelise the running of test suites - parallelise the tests within each test suite - show selenium browser scaling - show protractor settings for parallel - show VNC watcher for test runs

POST-PIPELINE RUNS - push docker images to registry - push git tag - push other artifacts - with some logic in place

CI DEBUGGING

POST-PIPELINE RUNS

POST-PIPELINE - DOCKER PUSH

CI DEBUGGING - CLEANUP - Don t wipe the workspace at the end of the pipeline - leave the HDD intact - Beginning of pipeline

CI SECURITY

USING SECURE CREDENTIALS

CHEERS CONSULTING / TRAINING EMAIL: PAUL@TEAM-NEEV.COM TWITTER: @DR4GOONIS 63