BoKS Puppet module Building the BoKS module http://www.unixerius.nl Rijnstraat 23, 2223 EG, Katwijk
This document is 2016, Unixerius. All diagrams use vector images available in the public domain, acquired from Openclipart.org.
Table of contents 1. The basic framework 5 The building site and version control 5 Generating the module framework 5 Starting the module: init.pp 8 Configuring the module 8 Ensuring that our module gets included 8 2. Installing the BoKS software 9 Approach 1: role-specific packages 9 Approach 2: a generic package 9 How to get the package on the host 9 Puppet package definition 10 Will it work?! 10 3. Expanding upon the basics 11 Installing a bcastaddr file 11 Starting the service 11 4. Going even further 14 Toggling the BoKS SSH daemon 14 Configuring filmon 16 BKSD automatic inactivity timeout 17 Basic management of bokscron 18 5. And now the hard part! 19 Talking to BoKS 19 BoKS host preregistration 20 Adjustments to the installation and configuration classes 20 One big caveat: multiple Puppet runs 23 6. Little bits and bobs 24 Toggling the sysreplace status 24 Randomized root password for new hosts 24 Toggling the pswupdate status 25 Gracefully handling re-installs and upgrades 26 7. Introducing some order 27 Profiles and roles 27 Hiera 28 Forging ahead with Hiera 29 The matter of node keys 30 8. Putting the module to use 34 9. Onwards, into the future 35 Appendix A: modified boksm init script 36 Appendix B: the bokspup wrapper script 38 Appendix C: my building and testing environment 42 Appendix D: how I built my RPM package 44 Appendix E: on keeping clean code 47 3
Introduction In January of 2016 Fox Technologies commissioned Unixerius to build a basic Puppet module to allow for centralized management of BoKS client software. After setting up the Dev/Test environment and after building a basic RPM package we started on building the Puppet module. This document describes the iterative steps of the BoKS Puppet module creation. It s written in a stream-of-activities style, where the chapters and paragraphs follow along as I went and built the module from the ground on up. Thus we will start from the very beginning, with expansion upon expansion leading to a sudden switch of design principles and large overhaul of the code base! Because of this, my explanations may jump to-and-fro a bit and I ll even say a few things which I will later come to realize are wrong. I m learning as I go. One more caveat: while many of the code examples in this document are valid Puppet code they are far from correct. The code will run, but is full of stylistic and syntactical errors. This is alright as long as we re using the snippets to illustrate concepts. But if you re going to work with the BoKS module, please pull it from the Forge! 1 I sincerely hope you will find this paper interesting to read. Thomas Sluyter April 2016. http://forge.puppet.com/tsluyter/boks 1 4
1. The basic framework Puppet Labs provide a lot of good documentation online, including introductory articles for 2 module builders. One very nice starting point is their Module Fundamentals. So let s start setting things up, starting with version control! The building site and version control First you ll need to ensure that you ve created an SSH keypair for your user account and that you ve paired it to your Github account. Refer to appendix C for more details. Login to the Puppet master; in my case I work as root on the Puppet Learning VM. # cd /etc/puppetlabs/code/environments/production/modules # git clone ssh://git@github.com/tsluyter/boks-puppet.git./boks Cloning into './boks'... Warning: Permanently added the RSA host key for IP address '192.30.252.130' to the list of known hosts. Enter passphrase for key '/root/.ssh/id_dsa': remote: Counting objects: 4, done. remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 4 Receiving objects: 100% (4/4), 12.14 KiB 0 bytes/s, done. Because we ll be duplicating that directory name, temporarily move the newly created repository out of the way. mv boks boks-repo Generating the module framework Puppet Labs offer a module creation tool, which starts a questionnaire that helps you build the empty framework of the module. The naming convention for a module is <organization>- <module name>. # puppet module generate unixerius-boks We need to create a metadata.json file for this module. Please answer the following questions; if the question is not applicable to this module, feel free to leave it blank. Puppet uses Semantic Versioning (semver.org) to version modules. What version is this module? [0.1.0] --> Who wrote this module? [unixerius] --> Thomas Sluyter <continued> 2 http://docs.puppetlabs.com/puppet/4.2/reference/modules_fundamentals.html#writing-modules 5
What license does this module code fall under? [Apache-2.0] --> GPLv3 How would you describe this module in a single sentence? --> It installs and configures Fox Technologies BoKS. Where is this module's source code repository? --> https://github.com/tsluyter/boks-puppet Where can others go to learn more about this module? [https://github.com/ tsluyter/boks-puppet] --> Where can others go to file issues about this module? [https://github.com/ tsluyter/boks-puppet/issues] --> ---------------------------------------- { "name": "unixerius-boks", "version": "0.1.0", "author": "Thomas Sluyter", "summary": "It installs and configures Fox Technologies BoKS.", "license": GPLv3", "source": "https://github.com/tsluyter/boks-puppet", "project_page": "https://github.com/tsluyter/boks-puppet", "issues_url": "https://github.com/tsluyter/boks-puppet/issues", "dependencies": [ {"name":"puppetlabs-stdlib","version_requirement":">= 1.0.0" ], "data_provider": null ---------------------------------------- About to generate this metadata; continue? [n/y] --> Notice: Generating module at /etc/puppetlabs/code/environments/production/ modules/boks... Notice: Populating templates... Finished; module generated in boks. boks/readme.md boks/metadata.json boks/manifests boks/manifests/init.pp boks/rakefile boks/tests boks/tests/init.pp boks/spec boks/spec/classes boks/spec/classes/init_spec.rb boks/spec/spec_helper.rb boks/gemfile 6
Now let s make sure all files are in one and the same place! # cd boks; mv *../boks-repo/; mv *.*../boks-repo/ # cd..; rmdir boks; mv boks-repo boks You now have a boks directory containing the combinations between a properly cloned Git repository and a newly generated Puppet module framework. # ls -al boks total 76 drwxr-xr-x 6 root root 4096 Feb 9 20:18. drwxr-xr-x 3 root root 4096 Feb 9 20:18.. -rw-r--r-- 1 root root 240 Feb 9 20:17 Gemfile drwxr-xr-x 8 root root 4096 Feb 9 20:11.git -rw-r--r-- 1 root root 35141 Feb 9 20:11 LICENSE drwxr-xr-x 2 root root 4096 Feb 9 20:17 manifests -rw-r--r-- 1 root root 485 Feb 9 20:17 metadata.json -rw-r--r-- 1 root root 614 Feb 9 20:17 Rakefile -rw-r--r-- 1 root root 3145 Feb 9 20:17 README.md drwxr-xr-x 3 root root 4096 Feb 9 20:17 spec drwxr-xr-x 2 root root 4096 Feb 9 20:17 tests Finally we can sync the newly created, empty framework into Git! You are still in the modules directory of Puppet. cd boks; mv tests examples git add * git commit # git commit [master ddc3c33] Generated empty Puppet module framework using "puppet module generate unixerius-boks". 8 files changed, 190 insertions(+), 2 deletions(-) create mode 100644 Gemfile rewrite README.md (100%) create mode 100644 Rakefile create mode 100644 manifests/init.pp create mode 100644 metadata.json create mode 100644 spec/classes/init_spec.rb create mode 100644 spec/spec_helper.rb create mode 100644 examples/init.pp # git push origin Enter passphrase for key '/root/.ssh/id_dsa': Counting objects: 16, done. Delta compression using up to 2 threads. Compressing objects: 100% (11/11), done. Writing objects: 100% (14/14), 3.89 KiB 0 bytes/s, done. Total 14 (delta 0), reused 0 (delta 0) To ssh://git@github.com/tsluyter/boks-puppet.git 9b6535b..ddc3c33 master -> master 7
Starting the module: init.pp The ring that binds it all together. Right now it s really not exciting! /environments/production/modules/boks/manifests/init.pp: class boks () inherits boks::params { class{'boks::install': -> Class["boks"] Configuring the module Ideally, all configurable information for your module lives outside of the actual module in a place like Hiera. For a quick start, let s begin by adding some information into params.pp file; Hiera will follow later in chapter 7. /environments/production/modules/boks/manifests/params.pp: class boks::params { $package_ensure = 'present' $package_name = boks-client-7.0-1.el6.x86_64.rpm' $service_enable = 'true' $service_ensure = 'running' $service_manage = 'true' $service_name = 'boksm' $boks_etc = '/etc/opt/boksm' $boks_opt = '/opt/boksm' $boks_var = '/var/opt/boksm' $boks_bin = '$boks_opt/bin' $boks_sbin = '$boks_opt/sbin' $boks_lib = '$boks_opt/lib' $nodekey_file = '$boks_etc/nodekey' $bcastaddr_file = $boks_etc/bcastaddr' $boks_master = '10.211.55.60' $boks_replicas = ['10.211.55.61', '10.211.56.61'] Ensuring that our module gets included For our testing purposes, the easiest way to ensure that the BoKS Puppet module gets used for our test hosts, is to edit the site.pp file in our environment s manifests directory. /environments/production/manifests/site.pp: node default { include boks 8
2. Installing the BoKS software We re starting off with the hard part! FoxT currently presents the BoKS software in an installation format that is not really suited for automated installations and configuration management. To clarify, the BoKS installation as provided by FoxT: Comes as an OS and platform specific.tar.gz file, Which needs to be extracted on the target host, Where an installation shell script needs to be run, Which places all files in their relevant locations and Which runs a setup shell script which configures the software as master, replica or client. Most corporate environments have come to expect platform-native package formats, such as RPM (for RedHat derivatives), PKG (for Solaris) and DEB (for Debian derivatives). These package formats are easily integrated with said platform s repository services and also provide lots of useful functionality with regards to file sanity checking and versioning. Many of FoxT s customers are already re-packaging the BoKS software to suit their own needs. Puppet is no different in its needs: it s a state management tool, which expects to have a desired state described to it and which it will enforce. You tell Puppet: I want software X and Y to be installed, configured such-and-so, with services A, B and C running. While it is technically possible to have BoKS extract tar-balls and to run specific scripts, this is not the paradigm of proper Puppet use. It s even possible to simply provide Puppet with a list of hundreds of files and their desired locations, but that would be insanity. And thus, Puppet would very much like you to provide packages for your software. Approach 1: role-specific packages My personal preference is to make role-specific packages for the software that you re packing up. Analogue to how there are packages for an SSH daemon and for an SSH client, for an NTP daemon and an NTP client, for an LDAP server and an LDAP client, etc. Thus, I prefer to make installation packages for BoKS clients, replicas and masters separately. This allows you to install one package, which instantly puts your host in the desired role. To this end I have built boks-client-7.0-1.el6.x86_64.rpm on my dev/test environment. However, as you will read in chapter 5 I have concede that approach #2 below is in fact more suitable for our situation, only to come back on that decision again in chapter 7. Approach 2: a generic package Alternatively, Ken Deschene of FoxT preferred to take a generic approach, building boks-7.0-1.el7.centos.x86_64.rpm. His spec-file was a lot more elegant than my hastily puttogether RPM file. With Ken s package, it is required that your first install the RPM file after which you still have to manually run the setup shell script. It is feasible to trigger a one-off action like this with Puppet. Running setup automatically sets up encryption keys (aka node keys ). How to get the package on the host Generally speaking, any large corporate environment will have their own software repository servers. For example, a RedHat-centric infrastructure will include a local clone of RedHat s installation servers, as well as a local Yum software repository. Packages that are to be installed on servers will then be included in this Yum repository, where each client may easily download it. Unfortunately setting up a Yum server is beyond the scope of this project. 9
In this case, I choose to install the RPM through HTTP. The easiest way to achieve this, was to abuse the web server that s already built into the Learning Puppet VM (see appendix C). The BoKS installation RPMs were placed in /var/www/html/questguide/packages, where they can be downloaded through a URL like http://10.211.55.43/packages/boks-7.0-1.el6.x86_64.rpm. Puppet package definition A separate.pp file will be made for the class(es) pertaining to software installation. For now there is only one defined package: class boks::install inherits boks { package { boks-client-7.0-1.el6.x86_64': ensure => $package_ensure, provider => 'rpm', source => "http://10.211.55.43/packages/boks-client-7.0-1.el6.x86_64.rpm" Will it work?! Yes it does: # rpm -qf /etc/opt/boksm boks-7.0-1.el6.x86_64 # rpm --erase boks-7.0 # rpm -qf /etc/opt/boks file /etc/opt/boksm is not owned by any package # rm -rf /var/opt/boksm /etc/opt/boksm /opt/boksm # puppet agent -t Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for client2.broehaha.nl Info: Applying configuration version '1456173683' Notice: /Stage[main]/Boks::Install/Package[boks]/ensure: created Notice: Applied catalog in 20.80 seconds # rpm -qf /etc/opt/boksm boks-client-7.0-1.el6.x86_64 Initially my old BoKS client package was installed. I removed it with rpm--erase and then applied the Puppet code as defined above. After parsing the module, Puppet downloaded the RPM file from my web server and installed it. 10
3. Expanding upon the basics Now that the software has been installed, let s make sure that it can do something. We ll arrange the most basic configuration and then ensure that the software gets started (and stays running). Installing a bcastaddr file Let s say that you would like to present all your BoKS clients with the same bcastaddr file. This is easily done by providing that file through Puppet. 1. Ensure that the subdirectory files exists in your module directory. 2. Create the desired file as files/bcastaddr. 3. Create a subclass to ensure the presence of the bcastaddr file: boks::config. 4. This subclass should require the aforementioned package to be installed first. 5. Ensure that the params.pp file defines the full path using $bcastaddr_file. 6. Ensure that the new subclass is called by the main class. /environments/production/modules/boks/config.pp: class boks::config inherits boks { file { $bcastaddr_file: ensure => file, owner => 0, group => 0, mode => '0644', source => "puppet:///modules/boks/bcastaddr", /environments/production/modules/boks/init.pp (updated): class boks () inherits boks::params { class{'boks::install': -> class{'boks::config': -> Class["boks"] Starting the service We ll expand the params.pp file to include some variables regarding the BoKS service that it needs to start. We ll then define the service in its own subclass and we ll even tell Puppet to restart the service any time changes are made to bcastaddr and the nodekey file. And yet again we will expand the base class to call the service. /environments/production/modules/boks/params.pp (updated): $service_enable = 'true' $service_ensure = 'running' $service_manage = 'true' $service_name = 'boksm' 11
/environments/production/modules/boks/service.pp: class boks::service inherits boks { if $service_manage == "true" { service { "${service_name": ensure => $service_ensure, enable => $service_enable, subscribe => [ File["${bcastaddr_file"], File[ ${nodekey_file"] ] /environments/production/modules/boks/init.pp (updated): class boks () inherits boks::params { class{'boks::install': -> class{'boks::config': -> class{'boks::service': -> Class["boks"] So what does this yield? [root@client2 ~]# ps -ef grep -i boks root 4180 2093 0 20:19 pts/0 00:00:00 grep -i boks [root@client2 ~]# puppet agent -t Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for client2.broehaha.nl Info: Applying configuration version '1456341610' Error: Could not enable boksm: Execution of '/sbin/chkconfig --add boksm' returned 1: service boksm does not support chkconfig Error: /Stage[main]/Boks::Service/Service[boksm]/ensure: change from stopped to running failed: Could not enable boksm: Execution of '/sbin/chkconfig --add boksm' returned 1: service boksm does not support chkconfig Notice: Applied catalog in 32.25 seconds Well, it s a start. We ve run into a limitation of the way BoKS is currently built: it doesn t play nicely with the RedHat/CentOS modern service tooling. Luckily there are plenty of guides available online 3 4 that can help us with this problem. A hasty rewrite of the BoKS init script gave me the file shown in appendix A. This file was included in the RPM build source, after which the package was rebuilt and uploaded to the web server. As a matter of a quick test before re-appying the BoKS Puppet module, I tested whether the new init script works with CentOS service commands. It does. And now our module works very well! 3 Like this one -> http://www.rpm-based.org/how-to-manage-services-with-chkconfig-and-service Or even better yet -> http://www.tldp.org/howto/highquality-apps-howto/boot.html 4 12
# chkconfig --add boksm # chkconfig --del boksm # /opt/boksm/sbin/boksadm -S Boot -k # ps -ef grep -i boks root 5169 5047 0 20:43 pts/0 00:00:00 grep -i boks # rpm --erase boks-client-7.0 # rm -rf /etc/opt/boksm /var/opt/boksm /opt/boksm # puppet agent -t Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for client2.broehaha.nl Info: Applying configuration version '1456343902' Notice: /Stage[main]/Boks::Install/Package[boks-client-7.0-1.el6.x86_64]/ensure: created Notice: /Stage[main]/Boks::Config/File[/etc/opt/boksm/bcastaddr]/ensure: defined content as '{md59a0018095bf8baf3b937d7782199e078' Info: /Stage[main]/Boks::Config/File[/etc/opt/boksm/bcastaddr]: Scheduling refresh of Service[boksm] Notice: /Stage[main]/Boks::Service/Service[boksm]/ensure: ensure changed 'stopped' to 'running' Info: /Stage[main]/Boks::Service/Service[boksm]: Unscheduling refresh on Service[boksm] Notice: Applied catalog in 52.09 seconds # ps -ef grep -i boks root 16962 1 0 20:58? 00:00:00 boks_init -f /etc/opt/boksm/ boksinit.client < shows more boks processes running > Success! 13
4. Going even further Toggling the BoKS SSH daemon This is a tricky one! Not because changing that one line in $BOKS_etc/ENV is so hard, but because it conflicts with another package and its service! Enabling the BoKS SSH daemon requires that port 22 is unoccupied by the operating system s standard SSH daemon. Thus, for hosts where you d like the BoKS SSH daemon to run, you ll also need to specify a clause that disables the standard SSHD. This must not be done inside the BoKS module! Unfortunately the ssh-module that I ve pulled from the Puppet Forge does not have a parametrized subclass for ssh::server::service. Instead, it s hard-coded to ensure that the service is running. I ll contact the module s author to suggest parametrizing this particular part of his module, but for now I ll hack it manually, as a proof of concept. The ingredients to our recipe are /environments/production/modules/ssh/manifests/server/service.pp (NOT the BoKS module): class ssh::server::service { include ssh::params include ssh::server service { $ssh::params::service_name: ensure => stopped, hasstatus => true, hasrestart => true, enable => true, require => Class['ssh::server::config'], /environments/production/manifests/site.pp: node 'client2.broehaha.nl' { include boks include ssh class { 'boks::boks_sshd' : boks_sshd_set_to => 'on', /environments/production/modules/boks/manifests/boks_sshd.pp: class boks::boks_sshd ($boks_sshd_set_to = 'on') inherits boks { file_line { 'boks_sshd': ensure => 'present', path => "${env_file", line => "BOKS_SSHD=${boks_sshd_set_to", match => '^BOKS_SSHD\=', require => Package["${package_name"], notify => Service["boksm"], 14
The following shows what happens when: 1. You start out with ssh::server::service hard-coded as stopped and boks_sshd_set_to=> on. 2. You ll see that BoKS SSHD is running, and that the vanilla SSH is not. 3. I then toggle the settings to ssh::server::service running and boks_sshd_set_to=> off. 4. You ll see that BoKS SSHD disappears and that the vanilla SSH is started. 5. Toggling it again to disable the vanilla SSH and to start BoKS SSHD. 6. Initially the BoKS SSHD does not start, because Puppet chose to do the BoKS changes before the vanilla SSH modification. However, since boks_init is set to respawn the missing SSHD, it appears in a minute or two. # ps -ef grep -i ssh root 4102 3542 0 07:53 pts/0 00:00:00 /opt/boksm/lib/boks_sshd -D - g120 # puppet agent -t [ ] Notice: /Stage[main]/Boks::Boks_sshd/File_line[boks_sshd]/ensure: created Info: /Stage[main]/Boks::Boks_sshd/File_line[boks_sshd]: Scheduling refresh of Service[boksm] Notice: /Stage[main]/Boks::Service/Service[boksm]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Ssh::Server::Service/Service[sshd]/ensure: ensure changed 'stopped' to 'running' Info: /Stage[main]/Ssh::Server::Service/Service[sshd]: Unscheduling refresh on Service[sshd] Notice: Applied catalog in 88.11 seconds # ps -ef grep -i ssh root 4768 1 0 07:59? 00:00:00 /usr/sbin/sshd # puppet agent -t [ ] Notice: /Stage[main]/Boks::Boks_sshd/File_line[boks_sshd]/ensure: created Info: /Stage[main]/Boks::Boks_sshd/File_line[boks_sshd]: Scheduling refresh of Service[boksm] Notice: /Stage[main]/Boks::Service/Service[boksm]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Ssh::Server::Service/Service[sshd]/ensure: ensure changed 'running' to 'stopped' Notice: Applied catalog in 79.51 seconds # ps -ef grep -i ssh # ps -ef grep boks_init root 5268 1 0 08:00? 00:00:00 boks_init -f /etc/opt/boksm/ boksinit.client # ps -ef grep -i ssh root 5477 5268 0 08:03? 00:00:00 /opt/boksm/lib/boks_sshd -D - g120 15
Configuring filmon Enabling filmon on a client system requires an [on off] toggle in $BOKS_etc/ENV as well as configuration file $BOKS_etc/filmon.conf. /environments/production/manifests/site.pp: node 'client2.broehaha.nl' { include boks class { 'boks::filmon' : filmon_set_to => 'off', filmon_runhours => '23-05', filmon_filesystems => '', /environments/production/modules/boks/manifests/filmon.pp: class boks::filmon ( $filmon_set_to = 'on', $filmon_runhours = '22-06', $filmon_slowdown = '10', $filmon_filesystems = ', ) inherits boks { file_line { 'filmon': ensure => 'present', path => "${env_file", line => "FILMON=${filmon_set_to", match => '^FILMON\=', require => Package["${package_name"], notify => Service["boksm"], 16 file { "${boks_etc/filmon.conf": ensure => file, owner => 0, group => 0, mode => '0644', require => Package["${package_name"], content => template("boks/filmon.erb"), file_line { "filmon_runhours": ensure => present', path => ${boks_etc/filmon.conf", line => "runhours ${filmon_runhours", match => 'runhours ', require => File["${boks_etc/filmon.conf"], file_line { "filmon_slowdown": ensure => present', path => ${boks_etc/filmon.conf", line => "slowdown ${filmon_slowdown", match => 'slowdown ', require => File["${boks_etc/filmon.conf"],
file_line { "filmon_filesystems": ensure => present', path => ${boks_etc/filmon.conf", line => "filesystems ${filmon_run_hours", match => 'filesystems ', require => File["${boks_etc/filmon.conf"], In this case /environments/production/modules/boks/templates/filmon.erb contains a basic version of the filmon.conf configuration file. It has the config and global stanzas as well as some pre-defined trees, directories and files to monitor. For now only the settings from the config stanza have been parametrized, to be adjusted in node-classification. With some more effort it will be possible to adjust this sub-class to more easily add or exclude specific resources for monitoring. BKSD automatic inactivity timeout The settings for bksd are pretty simple to toggle. /environments/production/manifests/site.pp: node 'client2.broehaha.nl' { include boks class { boks::bksd' : bksd_set_to => 'on', bksd_timeout => 'on', bksd_sleep => '30', /environments/production/modules/boks/manifests/bksd.pp: class boks::bksd ( $bksd_set_to = 'on', $bksd_timeout = 'on', $bksd_sleep = '30', ) inherits boks { file_line { 'bksd_set_to': ensure => 'present', path => "${env_file", line => "BKSD=${bksd_set_to", match => '^BKSD\=', require => Package["${package_name"], notify => Service["boksm"], file_line { 'bksd_timeout': ensure => 'present', path => "${env_file", line => "BKSD_TIMEOUT=${bksd_timeout", match => '^BKSD_TIMEOUT\=', require => Package["${package_name"], notify => Service["boksm"], 17
file_line { 'bksd_sleep': ensure => 'present', path => "${env_file", line => "BKSD_SLEEP=${bksd_sleep", match => '^BKSD_SLEEP\=', require => Package["${package_name"], notify => Service["boksm"], Basic management of bokscron /environments/production/manifests/site.pp: node 'client2.broehaha.nl' { include boks class { 'boks::bokscron' : bokscron_set_to => 'on', /environments/production/modules/boks/manifests/bokscron.pp: class boks::bokscron ( $bokscron_set_to = on', ) inherits boks { file_line { 'bokscron': ensure => 'present', path => "${env_file", line => "BOKSCRON=${bokscron_set_to", match => '^BOKSCRON\=', require => Package["${package_name"], notify => Service["boksm"], file { "${boks_etc/bokscron.conf": ensure => file, owner => 0, group => 0, mode => '0644', require => Package["${package_name"], content => template("boks/bokscron.erb"), 18
5. And now the hard part! Talking to BoKS Now that we have a running BoKS client, with toggles for specific parts of the software, there s another big task left: actually using the BoKS client! We ll need to set up a host record and a root account. But how do we do that?! Puppet has no idea about what BoKS is, or how to arrange such things. Which isn t made easier by these facts: All modifications to BoKS occur using shell commands, on the BoKS master server. The Puppet Master will realistically never run on the same system as the BoKS master. BoKS has no API that is remotely accessible. BoKS database is not remotely accessible either. Without opening an API to interact with BoKS we are going to have to scale that wall. We re going to be doing it with a combination of what Puppet calls exported resources and by cleverly running BoKS commands through them. I say cleverly, because Puppet s unless and onlyif parameters on an exec resource trigger on a command s exit code. And BoKS commands almost always exit with a 0, regardless of the actual outcome. # hostadm -Slh master.broehaha.nl master.broehaha.nl 10.211.55.60 REPLICA /home Yes # echo $? 0 # hostadm -Slh pistachio # echo $? 0 So we ll settle for: exec { "Make host record ${hostname": command => ${bokspup --host -a -h ${hostname -i ${ip", unless => ${bokspup --host -l -h ${hostname >/dev/null 2>&1", At this point in time I was still relying on a wrapper script called bokspup (see appendix B) for most BoKS interactions. This wrapper script was present up until release 0.4.0, where it was discarded because I stopped using preregistrations. But you ll read more about that in chapter 7. For now, let s stick with the wrapper. 19
BoKS host preregistration I have to admit that I haven t had much experience in setting up the infrastructure used for BoKS client preregistration, so this was a bit rough in going. After setting up the preregistration record for a new client I was baffled why the client kept saying it was unable to connect to the master, despite a telnet to master.broehaha.nl:6507 worked just fine. The boks_errlog clued me in: err 2016-02-28T08:13:57.757815Z master.broehaha.nl boks_autoregisterd 1976 - - Failed to get host cert/key err 2016-02-28T08:13:57.757896Z master.broehaha.nl boks_autoregisterd 1976 - - ssl_initsess failed As appendix C mentions, I hadn t set up a rootca on my master yet, so I started by running the fccsetup wizard. After that I extracted the certificate for use on the clients, by running bccgethostcert master.broehaha.nl. The resulting PEM file was put into $BOKS_etc/rootca.pem on the client host. This finally let me run the host registration: /opt/boksm/sbin/setup -a -v -A FILE=/etc/opt/boksm/prereg-secret -A MASTER=10.211.55.60 -A CACERT=/etc/opt/boksm/rootca.pem -A USER=puppet -A HOST=client2.broehaha.nl client It s clear that I still have a bit to learn about setting up the certificates that involve BoKS. For now it works though, so moving on! What does this mean for the RPM package? This is where I changed my mind and agreed with Ken Deschene that his approach for the RPM package is better. Why? Because host registration based on a preregistration record requires that you run the setup script; there s no way around it. Thus, I have adjusted the boks-client package to work like Ken s: it was built with the pre-setup files and requires that you run setup afterwards. Adjustments to the installation and configuration classes Now that we have all of that out of the way, the BoKS Puppet classes have changed quite a lot. /environments/production/modules/boks/manifests/init.pp: class boks () inherits boks::params { class{'boks::hostpreregistration': -> class{'boks::install': -> class{'boks::config': -> class{'boks::service': -> class{'boks::rootaccount': -> Class["boks"] The files directory has two new files which will be distributed to all BoKS clients: 1. The rootca certificate of the master server. 2. A shared secret file, for the host registration. One for all clients, one for the master. The init class also has a few additional steps, consisting of a host preregistration before installation and root account creation at the end of it all. How does a host arrange for its own preregistration record on the master server? By having Puppet generate exported resource which will be collected by the Puppet agent on the BoKS master server. 5 Many thanks to my colleague Ger Apeldoorn for pointing me in the right direction for this! His help was indispensable! 5 20
/environments/production/modules/boks/manifests/hostpreregistration.pp: class boks::hostpreregistration () { @@exec { "Hostpreregistration of ${::hostname": command => "${bokspup -m prereg -a -h ${::fqdn -i ${::ipaddress", unless => "${bokspup -m prereg -l -h ${::fqdn", tag => "hostpreregistration", There s the wrapper we talked about before. Unless a valid host preregistration record is found for host ${::fqdn (the Puppet node s fully qualified hostname), a new prereg record is made with the host s full name and IP address. On the BoKS master this is picked up in the master_actions class. On the BoKS client, we ll install the RPM package as before, but here s the new twist that Ken suggested: unless file $BOKS_var/boks_init exists, Puppet will run the setup script. With all the flags shown, the setup script will perform an auto-registration (-a) using the known shared secret (FILE), to the BokS master (MASTER) whose SSL certificate was signed using the rootca (CACERT). The registration is logged as having been done by user puppet (USER) and yet again the host identifies itself as ${::fqdn. /environments/production/modules/boks/manifests/install.pp: class boks::install inherits boks { package { "${package_name": ensure => $package_ensure, provider => 'rpm', source => "${rpm_source/${package_file" file { $prereg_secret: ensure => file, owner => 0, group => 0, mode => '0400', source => "puppet:///modules/boks/prereg-secret", file { $boks_cacert: ensure => file, owner => 0, group => 0, mode => '0400', source => "puppet:///modules/boks/master.broehaha.nl-ca.pem", exec { "setup_client": command => "${boks_sbin/setup -a -v -A FILE=${prereg_secret -A MASTER=$ {boks_master -A CERTFILE=${boks_cacert -A USER=puppet -A HOST=${::fqdn client", creates => "${boks_var/run/boks_init", require => [ Package["${package_name"], File["${prereg_secret"], File["${boks_cacert"], ] 21
Based on the require attributes, you ll note that setup will only run once the package has been installed and once the prereg secret and the rootca files have been installed. And those files are delivered through the same subclass. Finally, on the client side, we set up a root account using the same wrapper from before. /environments/production/modules/boks/manifests/rootaccount.pp: class boks::rootaccount () inherits boks { @@exec { "Root account of ${::hostname": command => "${bokspup -m root -a -h ${::fqdn", unless => "${bokspup -m root -l -h ${::fqdn", tag => "rootaccount", What happens with these exported resources (indicated by the @@) is this: each Puppet node which runs those particular classes will create a resource that is stored in the Puppet database. It s not actually applied on the Puppet node itself. For example, my BoKS client would make the following two resources: @@exec { "Hostpreregistration of client2.broehaha.nl : command => /opt/boksm/lib/bokspup -m prereg -a -h client2.broehaha.nl -i 10.211.55.63, unless => "/opt/boksm/lib/bokspup -m prereg -l -h client2.broehaha.nl", tag => "hostpreregistration", @@exec { "Root account of ${::hostname": command => "/opt/boksm/lib/bokspup -m root -a -h client2.broehaha.nl", unless => "/opt/boksm/lib/bokspup -m root -l -h client2.broehaha.nl", tag => "rootaccount", These resources are then available to any Puppet node which decides to collect them. Which is exactly what happens on the BoKS master! First we ensure that the master server has the bokspup wrapper as well as the shared preregistration secret. # cat manifests/master_config.pp class boks::master_config () inherits boks::params { file { $bokspup: ensure => file, owner => 0, group => 0, mode => '0755', source => "puppet:///modules/boks/bokspup", file { $prereg_secret: ensure => file, owner => 0, group => 0, mode => '0400', source => "puppet:///modules/boks/prereg-secret", 22
And then we make the BoKS master collect all the previously created, exported resources. # cat manifests/master_actions.pp class boks::master_actions () inherits boks::params { Exec << tag=='hostpreregistration' >> Exec << tag=='rootaccount' >> One big caveat: multiple Puppet runs There s a big challenge this leave us with: you cannot control the order in which the various Puppet nodes handle all their actions. On the Puppet agent for the BoKS client everything will proceed swimmingly until the agent runs the setup command. This will fail upon the first run, because the BoKS master has not yet picked up on the exported resource that it needs to use. This is not a problem, because the failed setup will re-run as long as $BOKS_var/run/boks_init hasn t been created. We would love the workflow to proceed as follows: But this isn t an option with vanilla Puppet. And thus the workflow proceeds like this (note that the colored blocks correspond to the colors in the picture above): To sum it up, Puppet will ensure that your BoKS hosts are configured and running correctly, but it will take three runs on the BoKS master and two on the agent. How long this will take fully depends on the way you have configured your Puppet agent to run. These limitations can be solved by using a new feature of Puppet s: Application Orchestration. This is a feature of the commercial Puppet Enterprise and is not available in the free, open source variant. Since we are targeting both versions of Puppet with this module, for now we re stuck working like this. Another possible solution would be to completely do away with preregistration and to use hard coded node keys, which are pulled from Hiera. Those keys would need to be encrypted of course, but that should be doable. We ll be looking into this in chapter 7. 23
6. Little bits and bobs Toggling the sysreplace status In order for BoKS to actively enforce its security rules on a host, a specific mode needs to be set. This is done using the sysreplace command, choosing to either be active ( replace ) or inactive ( restore ). Switching modes leads to a large number of changes on the file system, in configuration files as well as in PAM. The easiest way for us to determine whether BoKS security is activated, is by checking the parameter SSM_ACTIVE in $BOKS_etc/ENV. We want to parametrize BoKS state in the Puppet module, checking whether any changes are needed based on the aforementioned parameter. /environments/production/manifests/site.pp: node 'client2.broehaha.nl' { include boks class { 'boks::sysreplace' : sysreplace_set_to => 'on', /environments/production/modules/boks/manifests/sysreplace.pp: class boks::sysreplace ( $sysreplace_set_to = 'on', ) inherits boks { if $sysreplace_set_to == 'on' { exec { "sysreplace_on": command => "${boks_lib/sysreplace replace", unless => "/bin/grep -i \"SSM_ACTIVE=true\" $env_file", require => Package["${package_name"], else { exec { "sysreplace_off": command => "${boks_lib/sysreplace restore", onlyif => "/bin/grep -i \"SSM_ACTIVE=true\" $env_file", require => Package["${package_name"], Randomized root password for new hosts In our original instance of the root account creation, we do not set a password at all. That s not very safe, so I ve tweaked the aforementioned bokspup script a little bit. It now generates a random password, which is encrypted. The password will never be known to anyone, but at least root won t be accessible with an empty password. /environments/production/modules/boks/files/bokspup.sh (excerpt): PASSHASH=$(head /dev/urandom tr -dc A-Za-z0-9 head -c10 openssl passwd -1 -stdin) mkbks -l ${HOSTNAME:root -u 0 -g 0 -p ${PASSHASH -h "${ROOTHOME" -r "Superuser from BoKS-Puppet" -s "${ROOTSHELL" -P "${ROOTUSERCLASS" 24
Toggling the pswupdate status By default BoKS will manage the local security files on a BoKS client. User accounts, groups, passwords and all are managed in BoKS and put into /etc/passwd, /etc/shadow, /etc/group and so on. If for some reason you do not want this, this process can be disabled. Like the other resources which require actions on the BoKS master server, this class also creates exported resources to make the actual change. /environments/production/manifests/site.pp: node 'client2.broehaha.nl' { include boks class { 'boks::sysreplace' : boks_pswupdate_set_to => 'off', You will see below that I ve stopped using bokspup already. /environments/production/modules/boks/manifests/pswupdate.pp: class boks::pswupdate ( $boks_pswupdate_set_to = "on", ) inherits boks { if $boks_pswupdate_set_to == "on" { @@exec { "pswupdate_on on ${::hostname": command => "${boks_sbin/hostadm -D nopswupdate -h ${::fqdn", unless => "${boks_sbin/hostadm -lh ${::fqdn /bin/grep \"Update passwd file: Yes\"", tag => "pswupdate", else { @@exec { "pswupdate_off on ${::hostname": command => "${boks_sbin/hostadm -A nopswupdate -h ${::fqdn", onlyif => "${boks_sbin/hostadm -lh ${::fqdn /bin/grep \"Update passwd file: Yes\"", tag => "pswupdate", /environments/production/modules/boks/manifests/master_actions.pp (expanded with): class boks::master_actions () inherits boks::params { Exec << tag=='hostpreregistration' >> Exec << tag=='rootaccount' >> Exec << tag=='pswupdate' >> 25
Gracefully handling re-installs and upgrades As per the diagrams in chapter 5 ( One big caveat ) and the v0.3 init.pp, the running of a host s setup scripting is irrevocably linked to the RPM package s installation. Class boks::install includes the clause that runs BoKS setup, as long as ${boks_var/run/boks_init does not exist. /environments/production/modules/boks/manifests/init.pp (version 0.3): class boks () inherits boks::params { class{'boks::hostpreregistration': -> class{'boks::install': -> class{'boks::config': -> class{'boks::service': -> class{'boks::rootaccount': -> Class["boks"] This will cause problems down the road, when we ll re-install the same package, or when we re upgrading to a newer version. Were we to attempt that right now, the Puppet agent would run into a lot of failures because the setup command will fail (the master server will report that the host has already been registered). So we need to break the chain in such a way that boks::install still leads to boks::config, while making boks::clientsetup a new and fallible branch. Or alternatively, we can work towards a situation where host preregistrations are no longer used. Instead we can use pre-defined nodekeys per host, which means we can leave the cumbersome situation from chapter 5 behind us. As Franz Pletz wrote: A second Puppet run should never be an option! Please fix your modules and your manifests! 6 We ll continue in the next chapter! Mayflower Blog, The Puppet anchor pattern in practice, Franz Pletz, 2013 6 26
7. Introducing some order Up to this point we have a Puppet module which can do all manner of nice things for our BoKS infrastructure. It s pretty neat! But at the same time it s also the antithesis of neat (as in tidy): our configuration takes place in at least four locations! The files directory, where the shared secret for pre-registration and the bcastaddr reside. The templates directory, where filmon and bokscron configuration files reside. The params.pp class, where you define default settings and more. The site.pp (or other node classification source), where you can deviate from defaults. What s worse, is that it takes two Puppet runs on a client and two on the master to complete the installation and configuration of a BoKS client! That is very messy indeed and should be limited to one run on each. So let s work towards that! Profiles and roles To quote Gary Larizza: 7 The idea being that Profiles abstract all the technical bits that need to declared to setup a piece of technology, and Roles will abstract all the business logic for what pieces of technology should be installed on a certain class of machine. Basically, you can say that all our frontend infrastructure should have mysql, apache, java, jboss. In this statement, the Role is frontend infrastructure and the Profiles are mysql, apache, java, jboss. I ve taken all those loose lines from the node definitions in the site.pp file and distilled them into basic profiles. To this end, I ve made a dummy module called profile which has a manifests directory like any other module. In the manifests directory, there are two profiles. class profiles::boksclient { include boks include ssh class { 'boks::boks_sshd' : boks_sshd_set_to => 'off', class { 'boks::filmon' : filmon_set_to => 'off', class { 'boks::bksd' : bksd_set_to => 'on', bksd_timeout => 'on', bksd_sleep => '30', class { 'boks::bokscron' : bokscron_set_to => 'on', class { 'boks::sysreplace' : sysreplace_set_to => 'on', class { 'boks::pswupdate' : boks_pswupdate_set_to => 'on', And similarly boksmaster.pp. class profiles::boksmaster { class { 'boks::master_config' : class { 'boks::master_actions' : 7 Case in point: Sh*t Gary says, When to Hiera (aka: How do I module?), Gary Larizza, 2013 27
By moving the heavy lifting of the BoKS setup into profiles, we may now greatly simplify the node definitions in site.pp or even by using roles. /environments/production/manifests/site.pp: node 'client2.broehaha.nl' { include profile::boksclient node 'master.broehaha.nl' { include profile::boksmaster Hiera To quote Simon Young of Puppet Lunch: Hiera allows hierarchical configuration in Puppet, which is difficult to achieve with native Puppet code. Another major advantage is the separation of configuration data from the code, which makes everything generally easier. [ ] This Hiera configuration provides a lot of flexibility. It allows: A global configuration standard for all nodes (useful for enforcing a standard build policy). The ability to override the global configuration for a particular environment The ability to override global and environment configuration for a particular node. 8 To sum it up: Hiera is a piece of software separate from Puppet, which is included with Puppet Enterprise. It helps you build a hierarchical repository of data and configuration parameters. There are many discussions and even more differing opinions on when one should use Hiera and for which purposes. Gary Larizza sums it up pretty well: 9 This leads us back again to What does and DOESN T go in Hiera? I usually say the following: Data in site-specific Hiera datastore Business-specific data (i.e. internal NTP server, per-environment java versions, etc ) Sensitive data Data that you don t want to share with anyone else Data that does NOT go in the site-specific Hire datastore OS-specific data Data that EVERYONE ELSE who uses this module will need to know (paths to config files, package names, etc ) Basically, if I ask you if I can publish your module to the Puppet Forge, and you object because it has business-specific or sensitive data in it, then you probably need to pull that data out of the module and put it in Hiera. Taking Gary s points into account, I conclude that Hiera is a great place to store settings like: Master and replica IP addresses, to build bcastaddr files. Especially if you have multiple BoKS domains! The shared secret for pre-registration. The master s root CA certificate used in pre-registration. The BoKS user class which should be used for newly created root accounts. The BoKS installation package (file) names, as they are always custom. The RPM installation source(s). 8 Puppet Lunch, Using Hiera, Simon Young, 2013 Case in point: Sh*t Gary says, When to Hiera (aka: How do I module?), Gary Larizza, 2013 9 28
Forging ahead with Hiera As per Puppet Labs own documentation, it all starts with hiera.yaml. 10 /etc/puppetlabs/code/hiera.yaml: --- :backends: - yaml :yaml: :datadir: "/etc/puppetlabs/code/environments/%{environment/hieradata" :hierarchy: - "nodes/%{::trusted.certname" - common - defaults In turn this means that we will be using two Hiera levels of information: /etc/puppetlabs/code/environments/production/hieradata/nodes/<fqdn>.yaml, containing the per-host settings. /etc/puppetlabs/code/environments/production/hieradata/common.yaml, which contains the domain-level settings. /environments/production/hieradata/common.yaml: --- boks::sharedsecret: "ks93@wfwiwnal3h101na" boks::bcastaddr: - 10.211.55.61-10.211.55.62-10.211.55.60 /environments/production/hieradata/nodes/client2.broehaha.nl.yaml: --- boks::nodekey: GsWisne3911-rnwana$@ Working with this we can rework a lot of our previous resources to make them more flexible as well as more bug proof (in the case of the shared secret file on the master server). Here s some examples of this refactoring. /environments/production/modules/boks/manifests/config.pp (excerpt): $bcastaddr = hiera( boks::bcastaddr') file { $bcastaddr_file: ensure => file, owner => 0, group => 0, mode => '0644', content => template("boks/bcastaddr.erb"), 10 Puppet Labs, Hiera complete example, 2016 29
/environments/production/modules/boks/templates/bcastaddr.erb: DONT_BROADCAST ADDRESS_LIST <% @bcastaddr.each do ipaddress -%> <%= ipaddress %> <% end -%> /environments/production/modules/boks/manifests/install.pp (excerpt): $sharedsecret = hiera( boks::sharedsecret') file { $prereg_secretfile: ensure => file, owner => 0, group => 0, mode => '0400', content => template("boks/prereg-secret-client.erb"), /environments/production/modules/boks/templates/prereg-secret-client.erb: SECRET=<%= @sharedsecret %> The matter of node keys So We are now back to the topic of the inter-node encryption keys. We ve discussed this subject a few times and it s time to make a proper decision. The way I see it there are two approaches we can take: 1. Use pre-defined node keys, which are pulled from Hiera. 2. Let the nodes perform their node key handshake through host preregistration. The prior is easier and more flexible, but it is also less safe than the latter (because a copy of the node key is available in both Hiera and in the process listing of the puppet account on master and client). But it does allow for a single Puppet run on both the master and client hosts. The latter has so far been very messy (see chapter 5, One big caveat ) and it leads to issues with reinstallation and upgrading of the software because those trigger a failing, second preregistration. Both options are viable and can be built, but personally I prefer to continue work with the prior option. It takes away complexity from our Puppet runs and it will prevent odd situations when the BoKS software needs to be (re)installed. Going this route means one thing: your Puppet master should always be a secure host, but now it has to be really secure! Your Hiera database will contain the keys to all your Unix inter-host security! Protect it with your life! 11 In order to mitigate some of these security concerns we will start using hiera-eyaml. When facing the question of how to store your sensitive configuration data, such as passwords for your MySQL servers, the question is not why you should use Hiera, but how. In the past, the de facto way to store sensitive data in Hiera was to use hiera-gpg. However, with hiera-gpg, the entire Hiera data file had to be encrypted. Recognizing that managing this kind of file over the long term would be a burden, Tom Poulton created hiera-eyaml. This enables Hiera users to encrypt data blocks within the eyaml file separately, making management much easier. 12 11 This of course also goes for the backups, clones and VM disk images. 12 Puppet Labs, Encrypt your data using Hiera eyaml, Terri Haber, 2014 30