Managing Configuration Drift and Auditing with Salt. Duncan Mac-Vicar P. Director, Data Center Management R&D, SUSE

Transcription

1 Managing Configuration Drift and Auditing with Salt Duncan Mac-Vicar P. Director, Data Center Management R&D, SUSE

2 How to manage infrastructure? 2

3 Sysadmin Alexis #!/bin/bash cat <<EOF server1 server2 EOF while read line ssh -q zypper up done Manages his servers with bash scripts. They reside in`~/bin` Strict ownership and approval process.

4 New colleague: Devops Adrian /etc/motd: file.managed: - source: salt://common/motd Writes "Configuration Management" recipes and templates They reside in `git`. apache: pkg.installed

5 The two brains of IT Mode 1 Reliability Waterfall, ITIL Mode 2 Agility Agile, DevOps Conventional Projects New & Uncertain Projects Long-cycle Times (months) Short Cycle (days, weeks) 5

6 Devops Adrian explains If somebody changes the configuration, I just re-apply it and the tool brings it to the correct state.

7 Sysadmin Alexis reads: Configuration management (CM) is a systems engineering process for establishing and maintaining consistency of a product's performance, functional, and physical attributes with its requirements, design, and operational information throughout its life. 7

8 Infrastructure as code Has become a pragmatic way to implement configuration management for IT infrastructure. We know how to manage change of source code. We have the tools and processes for it.

9 Salt 101 Ports: Master ØMQ Minion Minion Minion

10 Salt 101 $ salt * pkg.install foo master $ salt host1 docker.pause c001 $ salt web* cmd.run \ cat /etc/fstab configuration commands results $ salt * state.apply minions

11 What is a state? /etc/motd: master file.managed: configuration commands results minions - source: salt://common/motd apache: pkg.installed

12 States state is how Salt calls configuration, in its declarative form.

13 Non-Compliant system $ salt minion1 state.apply test=true minion1: ID: Function: Result: Comment: Started: Duration: Changes: /etc/motd file.managed None The file /etc/motd is set to be changed 10:06: ms diff: -1 -Have a lot of fun... +This is my managed motd Summary for minion Succeeded: 1 (unchanged=1, changed=1) Failed: Total states run: 1

14 New trainee $ useradd -r mudserver

15 Let s run it again $ salt minion1 state.apply test=true minion1: ID: Function: Result: Comment: Started: Duration: Changes: /etc/motd file.managed None The file /etc/motd is set to be changed 10:06: ms diff: -1 -Have a lot of fun... +This is my managed motd Summary for minion Succeeded: 1 (unchanged=1, changed=1) Failed: Total states run: 1

16 The change was not detected It was not part of the configuration. # We can express... joe: user.present # How to express? any other: user.absent

17 Disappointed Sysadmin Devops tools focus in creating new systems. Not all change accounted. Is it really Configuration Management. What they call "Configuration Management" is really "Automation". The novelty is more about the declarative approach (vs imperative).

18 Are Classic IT priorities the same? Detecting Configuration Drift. Auditing Compliance. Documenting infrastructure. Drift Audit Document

19 Incomplete Configuration States Templates Baseline

20 Where is the baseline? In configuration management, a "baseline" is an agreed description of the attributes of a product, at a point in time, which serves as a basis for defining change. ~~MIL-HDBK-61

21 How to define a baseline? How to integrate it with the rest of the configuration?

22 Snapper ( snapper is to snapshots what zypper/apt-get/dnf are to packages. First demoed in SUSECon 2011! Main feature of SUSE Linux Enterprise 12

23 Created by SUSE, available everywhere (don't forget to mention btrfs)

24 Snapper 101 snapper list-configs snapper list snapper create snapper mount <number> snapper status <number1>..<number2> snapper diff <number1>..<number2> [files] snapper undochange <number1>..<number2> [files] YaST and zypper take snapshots automatically In grub menu you can boot old snapshots

25 YaST2 snapper

26 Salt and Snapper integration I salt '*' snapper.list_snapshots master salt '*' snapper.get_snapshot salt '*' snapper.create_snapshot salt '*' snapper.undo configuration commands results minions salt '*' snapper.diff

27

28 Salt andsnapper Integration $ salt minion2 snapper.run function=file.append args='["/etc/motd", "some text"]' minion2: Wrote 1 lines to "/etc/motd"... pre 21 Thu.. root number post Thu... root number salt job 6668 salt_jid=6668 salt job 6668 salt_jid=6668

29 Salt and Snapper integration $ salt minion2 snapper.diff_jid 6668 minion2: /etc/motd: --- /.snapshots/21/snapshot/etc/motd Have a lot of fun... +some text snapper.undo_jid also works

30 State module Back to the baseline problem, imagine you could say: # Starting from snapshot #3 baseline: -???? # then... /etc/motd: file.managed: - source: salt://common/motd apache: pkg.installed States Templates Baseline

31 State module You can! my_baseline: snapper.baseline_snapshot: - number: 20 States Templates - ignore: - /var/log - /var/cache /etc/motd: file.managed: - source: salt://common/motd Baseline

32 If the somebody adds a new user, a drift against the baseline rule will happen: $ salt minion1 state.apply test=true minion1: ID: my_baseline Function: snapper.baseline_snapshot Result: None Comment: 1 files changes are set to be undone... Changes:... /etc/passwd:... diff: --- /etc/passwd ,5 duncan:x:1000:100:duncan Mac-Vicar P.:/home/duncan:/bin/zsh -mudserver:x:167:100::/home/mudserver:/bin/bash ID: /etc/motd... Succeeded: 2 (unchanged=2, changed=2)

33 Applying states If you apply the state (eg. no `test=true`), the system will be set to the state of the baseline snapshot before applying the rest of the states. $ salt minion1 state.apply Current Baseline state... states

34 Managing snapshots by number? Creates a snapshot and adds a baseline tag to the userdata property of each snapshot. $ salt '*' snapper.create_baseline Type # single 0 Pre Date Desc userdata current... post Sept important=no Sept baseline_tag=baseline... single 22

35 Baseline tags Audit Co n D ent m cu e n i l se ure fig Ba last_production: snapper.baseline_snapshot: - tag: baseline t rif Do You can move the baseline, without affecting your state. The last tagged snapshot will be used.

36 Salt Snapper module Already submitted upstream. Will be part of Carbon release. Also available in SUSE Linux Enterprise/SUSE Manager x Salt package Carbon also supports automatic snapshots when applying states

37 (about state snapshots) $ salt minion2 snapper.run function=state.apply

38 Other Resources to Manage Drift

39 Salt Survey Runner Module Survey groups the returned values in pools of unique results. salt-run survey.diff survey_sort=up "*" cmd.run 'cat /etc/hosts' This tells you which server differs from the others. v2 v1 /etc/hosts /etc/hosts

40 Salt Package Module salt 'web*' pkg.diff /etc/sudoers Tells the difference between the `/etc/sudoers` of the original package vs the installed one.

41 Hubble ( Tool Purpose Nova Auditing Framework Pulsar File integrity monitor, security events Nebula Query infrastructure security snapshots Quasar Reporting

42 Available Nova modules grep (configuration values) iptables (firewall rules) netstat (listening ports) openscap (CVE scan) openssl (cert validation & expiration) pkg (installed packages) service (running services) stat (ownerships & permissions) sysctl (kernel parameters) vulners.com (CVE scan)

43 Future work

44 The two brains of IT Mode 1 Reliability Waterfall, ITIL Mode 2 Agility Agile, DevOps Conventional Projects New & Uncertain Projects Long-cycle Times (months) Short Cycle (days, weeks) 44

45 Bimodal Datacenter Softwar e Defined *: Comput e Storage Network Mode 1 Deployment Networking High Availability Mode 2 Magnum Containers Scaling Monitoring 45

46 Docker images 46

47 Motivation You bought into the hype and automated everything with Salt. You have Salt states and templates for all your infrastructure. salt://happy Now you wand to deploy an app into a container and you need a container image... 47

48 Dockerfiles FROM ubuntu:12.04 RUN zypper -n in foo RUN echo "key=val" > /etc/foo/config 48

49 Building images foo: pkg.installed salt mybuilder0 dockerng.state myapp mods=myapp /etc/foo/config: file.managed: - source: salt://myapp/foo.config 49

50 Benefits Reuse states and templates across workloads. Access templates, pillar data (eg. secrets) directly from Salt. Access to Salt modules for configuration. Implemented as ability to run arbitrary Salt modules inside running containers. Auditing 50

51 Thanks for listening 51

52

53 53