Quick Prototyping+CI with LXC and Puppet Ben Kero 2014-05-04
Introduction Ben Kero Release Engineer Responsible for version control systems: CVS, SVN, BZR, Darcs, RCS, Git, Mercurial
Before at Mozilla Dawn of time (2001) to early 2014 All infrastructure deployed with Puppet No documented environment or target for setup Wild-wild west. Write the code. Does it work? Ship it!
Framing the problem Problem: Developers and upstreams need a way to replicate our production environment Problem: Needed a way to make development environment closer to production Requirement: Quick to bring up Requirement: Public, shareable, infinitely reproducible Requirement: Dev environment should be as close as possible to production
Ideas (1/3) Virtual machines (on workstation) KVM/VMWare Workstation Could be closest match to production Expensive to generate up-to-date images Slow, can only run a few at a time Virtual machines (in cloud) Requires internet connection for development/testing Requires IT to manage/audit accounts No/separate public availability
Ideas (2/3) Raw cgroups + CoW (Copy-on-Write) images CoW mechanisms require out-of-tree modules or filesystem of dubious stability Raw cgroups are difficult to use. Bring-up time for devs would be catastrophic Linux-only (Win/Mac devs need to virtualize) Docker Doesn t clean images up after itself Dockerfiles vs Puppet code Fast moving target Linux-only (Win/Mac devs need to virtualize) VMWare Vsphere (Vcloud) Expensive Requires specialized hardware Not free software No offline development Beta status Limited dev access
Ideas (3/3) Amazon Web Services EC2 Can be expensive (or slow) Still a viable alternative for devs No offline development Volunteers can t have access (but could provide AMIs) LXC + Puppet Devs will need access to a Linux host Closely matches our production environment Linux-only (Win/Mac devs need to virtualize)
What are containers? Operating system level hypervisor One kernel, many userlands In Linux these are: CGroups OpenVZ
CGroups (Explained) C(ontrol) groups Feature of the Linux kernel since 2.6.24 Allows resource isolation (CPU, memory, network, devices) Can run a single process or entire system in isolation Similar to chroot, but has other advantages Virtual network interfaces Resource constraints (memory, CPU, more) Separate process table Separate user/group tables
LXC (Explained) Set of convenience commands to facilitate use of CGroups Basic operations: Create Destroy Start Stop Others: Clone, console, (un)freeze, execute Creation: Uses template scripts for creating containers Can handle complex resource setups
Comparison to other dev environment strategies Keeps cost down due to lower resource requirements vs VMs Supported upstream by our vendor and good community support (ML + IRC) Less flexibility (Linux-only, single-kernel, less security) Entirely in vanilla Linux kernel, no out-of-tree patches
Reasons you might choose another container-based solution (Docker) Desire the portability of Dockerfiles Can accept the maintenance of cleaning old instances Developers aren t using Linux as their primary OS
Other container hypervisors OpenVZ Not many active contributors Has better isolation Out-of-kernel patches required to use Some large-scale VPS hosting companies use this
Configuration Management explanation (1/2) Many flavors including Puppet, Chef, Ansible, Salt, etc Approach used in this talk is agnostic For example: Basic puppet pattern 1 class apache { 2 package { httpd : 3 ensure => installed } 4 file { / etc / httpd / httpd. conf : 5 source => " puppet :/// modules / httpd / service " } 6 service { httpd : 7 ensure => running, 8 enable => true } 9 Package [ httpd ] -> File [ httpd ] -> Service [ httpd ] 10 }
= Configuration Management explanation (2/2) Higher-level classes (bricks, metaclasses, etc) Used to set up machine types (such as a web server) Machine types 1 class webserver { 2 include apache 3 include nagios :: nrpe :: webserver 4 include logstash :: webserver } 5 6 node /^ web.*\. dc1 \. example \. com$ / { 7 include webserver 8 }
Initial impressions and roadbumps LXC CentOS template only added very recently Still no template for RHEL6 Kernel audit grumbles Things are fast
Our dev environment offering Custom LXC templates Vagrant scripts to give to devs who would like to replicate Masterless puppet Templates on our public wiki
Custom LXC Templates (1/3) Regular templates create a golden image copy that lives in /var/cache/lxc/ Written in shell, executed on lxc-create Wrote custom LXC templates http://github.com/bkero/lxc-templates
Custom LXC Templates (2/3) Try: Adding extra packages Apply a puppet base class (sometimes bad idea) Pre-install puppet certs Creating multiple containers (!)
Custom LXC Templates (3/3) lxc-centos-hgweb 1 #!/ bin / bash 2 download_ centos () { 3 CACHE_DIR =/ var / cache / lxc /lxc - centos - hgweb 4 INSTALL_ ROOT = $1 5 PKG_ LIST =" yum initscripts rootfiles \ 6 puppet librarian - puppet " 7 8 mkdir $INSTALL_ ROOT 9 if [ -d $CACHE_ DIR ]; then 10 cp - rv $CACHE_ DIR $INSTALL_ ROOT / rootfs ; exit 0; fi 11 add_ yum_ repos 12 yum -- installroot $INSTALL_ ROOT -y \ 13 -- nogpgcheck $PKG_ LIST 14 } 15 apply_ puppet () { 16 cp -rv / data / incoming / hgweb / etc / puppet / modules / 17 librarian - puppet init 18 puppet apply -e include hgweb 19 }
How LXC improves CI Clean vanilla systems every time Allows multiple isolated systems to interact per CI-run Much faster turn-around time Allows devs to easily reproduce CI environment
How LXC improves prototyping Allows testing new code in production OS natively Eases testing of new code in multiple environments More than one or two won t overload the host Programmatic creation of faux-production environment is difficult in VMs, much easier with LXC
Productivity enhancements Things are fast We can write tooling to let developers have (almost) exact replicas of entire production environments* * (Footnote: It s RHEL, so we can t without scary licensing problems) Faster turnaround time Can attract more (external) contributors due to the ease of constructing a dev environment
Holistic Testing Able to spin up multiple containers at once Each container can run a single infrastructure component Containers can interact on a virtual network (Linux Bridge) Can be used to test the end result of a system
Holistic Testing (Example) loadbalancer1.test tcp/80 to webserver1.test webserver1.test tcp/3306 to mysql1.test webserver1.test tcp/6379 to redis1.test webserver1.test tcp/11211 to memcache1.test
Case Study: Socorro Firefox Crash Reporter Infrastructure Processors Collectors Admin hosts Deployment is handled through subclasses of a single puppet module Continuous integration working on codebase Failed CI runs can be replicated by devs
Adoption inside Mozilla
In Review Before: Little to no testing Before: Getting devs a production-esque environment was a lot of manual work and cost Now: Some teams are rolling it out (master documentation on wiki, invitation to copy wiki template) Now: Deployed alongside devs using personal EC2 instances (personal preference)
Demo time!