Managing Configuration Drift and Auditing with Salt Duncan Mac-Vicar P. Director, Data Center Management R&D, SUSE dmacvicar@suse.com
How to manage infrastructure? 2
Sysadmin Alexis #!/bin/bash cat <<EOF server1 server2 EOF while read line ssh -q user1@${line} zypper up done Manages his servers with bash scripts. They reside in`~/bin` Strict ownership and approval process.
New colleague: Devops Adrian /etc/motd: file.managed: - source: salt://common/motd Writes "Configuration Management" recipes and templates They reside in `git`. apache: pkg.installed
The two brains of IT Mode 1 Reliability Waterfall, ITIL Mode 2 Agility Agile, DevOps Conventional Projects New & Uncertain Projects Long-cycle Times (months) Short Cycle (days, weeks) 5
Devops Adrian explains If somebody changes the configuration, I just re-apply it and the tool brings it to the correct state.
Sysadmin Alexis reads: Configuration management (CM) is a systems engineering process for establishing and maintaining consistency of a product's performance, functional, and physical attributes with its requirements, design, and operational information throughout its life. 7
Infrastructure as code Has become a pragmatic way to implement configuration management for IT infrastructure. We know how to manage change of source code. We have the tools and processes for it.
Salt 101 Ports: 4505-4506 Master ØMQ Minion Minion Minion
Salt 101 $ salt * pkg.install foo master $ salt host1 docker.pause c001 $ salt web* cmd.run \ cat /etc/fstab configuration commands results $ salt * state.apply minions
What is a state? /etc/motd: master file.managed: configuration commands results minions - source: salt://common/motd apache: pkg.installed
States state is how Salt calls configuration, in its declarative form.
Non-Compliant system $ salt minion1 state.apply test=true minion1: ---------ID: Function: Result: Comment: Started: Duration: Changes: /etc/motd file.managed None The file /etc/motd is set to be changed 10:06:05.021643 30.339 ms ---------diff: --+++ @@ -1 +1 @@ -Have a lot of fun... +This is my managed motd Summary for minion1 -----------Succeeded: 1 (unchanged=1, changed=1) Failed: 0 -----------Total states run: 1
New trainee $ useradd -r mudserver
Let s run it again $ salt minion1 state.apply test=true minion1: ---------ID: Function: Result: Comment: Started: Duration: Changes: /etc/motd file.managed None The file /etc/motd is set to be changed 10:06:05.021643 30.339 ms ---------diff: --+++ @@ -1 +1 @@ -Have a lot of fun... +This is my managed motd Summary for minion1 -----------Succeeded: 1 (unchanged=1, changed=1) Failed: 0 -----------Total states run: 1
The change was not detected It was not part of the configuration. # We can express... joe: user.present # How to express? any other: user.absent
Disappointed Sysadmin Devops tools focus in creating new systems. Not all change accounted. Is it really Configuration Management. What they call "Configuration Management" is really "Automation". The novelty is more about the declarative approach (vs imperative).
Are Classic IT priorities the same? Detecting Configuration Drift. Auditing Compliance. Documenting infrastructure. Drift Audit Document
Incomplete Configuration States Templates Baseline
Where is the baseline? In configuration management, a "baseline" is an agreed description of the attributes of a product, at a point in time, which serves as a basis for defining change. ~~MIL-HDBK-61
How to define a baseline? How to integrate it with the rest of the configuration?
Snapper (http://snapper.io) snapper is to snapshots what zypper/apt-get/dnf are to packages. First demoed in SUSECon 2011! Main feature of SUSE Linux Enterprise 12
Created by SUSE, available everywhere https://wiki.archlinux.org/index.php/snapper https://apps.fedoraproject.org/packages/snapper https://packages.debian.org/search?keywords=snapper (don't forget to mention btrfs)
Snapper 101 snapper list-configs snapper list snapper create snapper mount <number> snapper status <number1>..<number2> snapper diff <number1>..<number2> [files] snapper undochange <number1>..<number2> [files] YaST and zypper take snapshots automatically In grub menu you can boot old snapshots
YaST2 snapper
Salt and Snapper integration I salt '*' snapper.list_snapshots master salt '*' snapper.get_snapshot salt '*' snapper.create_snapshot salt '*' snapper.undo configuration commands results minions salt '*' snapper.diff
Salt andsnapper Integration $ salt minion2 snapper.run function=file.append args='["/etc/motd", "some text"]' minion2: Wrote 1 lines to "/etc/motd"... pre 21 Thu.. root number post 22 21 Thu... root number salt job 6668 salt_jid=6668 salt job 6668 salt_jid=6668
Salt and Snapper integration $ salt minion2 snapper.diff_jid 6668 minion2: ---------/etc/motd: --- /.snapshots/21/snapshot/etc/motd +++ /.snapshots/22/snapshot/etc/motd @@ -1 +1,2 @@ Have a lot of fun... +some text snapper.undo_jid also works
State module Back to the baseline problem, imagine you could say: # Starting from snapshot #3 baseline: -???? # then... /etc/motd: file.managed: - source: salt://common/motd apache: pkg.installed States Templates Baseline
State module You can! my_baseline: snapper.baseline_snapshot: - number: 20 States Templates - ignore: - /var/log - /var/cache /etc/motd: file.managed: - source: salt://common/motd Baseline
If the somebody adds a new user, a drift against the baseline rule will happen: $ salt minion1 state.apply test=true minion1: ---------ID: my_baseline Function: snapper.baseline_snapshot Result: None Comment: 1 files changes are set to be undone... Changes:... /etc/passwd:... diff: --- /etc/passwd +++ /.snapshots/21/snapshot/etc/passwd @@ -22,5 +22,3 @@ duncan:x:1000:100:duncan Mac-Vicar P.:/home/duncan:/bin/zsh -mudserver:x:167:100::/home/mudserver:/bin/bash ---------ID: /etc/motd... Succeeded: 2 (unchanged=2, changed=2)
Applying states If you apply the state (eg. no `test=true`), the system will be set to the state of the baseline snapshot before applying the rest of the states. $ salt minion1 state.apply Current Baseline state... states
Managing snapshots by number? Creates a snapshot and adds a baseline tag to the userdata property of each snapshot. $ salt '*' snapper.create_baseline Type # single 0 Pre Date Desc userdata current... post 20 19 Sept 26... important=no Sept 26... baseline_tag=baseline... single 22
Baseline tags Audit Co n D ent m cu e n i l se ure fig Ba last_production: snapper.baseline_snapshot: - tag: baseline t rif Do You can move the baseline, without affecting your state. The last tagged snapshot will be used.
Salt Snapper module Already submitted upstream. Will be part of Carbon release. Also available in SUSE Linux Enterprise/SUSE Manager 2015.8.x Salt package Carbon also supports automatic snapshots when applying states https://docs.saltstack.com/en/develop/topics/releases/carbon.html
(about state snapshots) $ salt minion2 snapper.run function=state.apply
Other Resources to Manage Drift
Salt Survey Runner Module Survey groups the returned values in pools of unique results. salt-run survey.diff survey_sort=up "*" cmd.run 'cat /etc/hosts' This tells you which server differs from the others. v2 v1 /etc/hosts /etc/hosts
Salt Package Module salt 'web*' pkg.diff /etc/sudoers Tells the difference between the `/etc/sudoers` of the original package vs the installed one.
Hubble (http://hubblestack.io) Tool Purpose Nova Auditing Framework Pulsar File integrity monitor, security events Nebula Query infrastructure security snapshots Quasar Reporting
Available Nova modules grep (configuration values) iptables (firewall rules) netstat (listening ports) openscap (CVE scan) openssl (cert validation & expiration) pkg (installed packages) service (running services) stat (ownerships & permissions) sysctl (kernel parameters) vulners.com (CVE scan)
Future work
The two brains of IT Mode 1 Reliability Waterfall, ITIL Mode 2 Agility Agile, DevOps Conventional Projects New & Uncertain Projects Long-cycle Times (months) Short Cycle (days, weeks) 44
Bimodal Datacenter Softwar e Defined *: Comput e Storage Network Mode 1 Deployment Networking High Availability Mode 2 Magnum Containers Scaling Monitoring 45
Docker images 46
Motivation You bought into the hype and automated everything with Salt. You have Salt states and templates for all your infrastructure. salt://happy Now you wand to deploy an app into a container and you need a container image... 47
Dockerfiles FROM ubuntu:12.04 RUN zypper -n in foo RUN echo "key=val" > /etc/foo/config 48
Building images foo: pkg.installed salt mybuilder0 dockerng.state myapp mods=myapp /etc/foo/config: file.managed: - source: salt://myapp/foo.config 49
Benefits Reuse states and templates across workloads. Access templates, pillar data (eg. secrets) directly from Salt. Access to Salt modules for configuration. Implemented as ability to run arbitrary Salt modules inside running containers. Auditing 50
Thanks for listening 51
53