Monitoring (with) Puppet. PuppetCamp

Transcription

1 Monitoring (with) Puppet PuppetCamp

2 Kris Buytaert I used to be a Dev, Then Became an Op Chief Trolling Officer and Open Source Everything is an effing DNS Problem Building Clouds since before the bookstore Organising too many confs, #devopsdays, #loadays,... Evangelizing devops

3 Do you know what your children do at 5 am in the morning? Sleeping? Crashing a Party? Why are the cops at your front door? What happened? How long have they been gone?

4 devops = clams Culture (Lean) Automate all the things... Build Automation Test Automation IAC Monitoring, Metrics... Sharing

5 Monitoring is usually an aftertought ENOBUDGET, ENOTIME

6 #monitoringsucks John Vincent A sub movement

7 #monitoringlove #monitoringlove hacksessions #monitorama

8 For years we've tolerated humans to to make structural manual changes to the infrastructure our critical applications are running on. Whilst at the same time demanding those critical applications to go trough rigid test scenarios. Who let this happen?

9 Infrastructure as Code Model our infrastructure A fast reproducable platform Disaster discovery for free

10 Infrastructure as Code Code = Code Version Control Quality Checks Testing Continuous Integration Continous Delivery

11 Infrastructure as Code Core Infrastructure Middleware deployment and integration Automated continuous application deployment Integrated Security enforcement Host, Service and Application Monitoring configured

12 Why #monitoringsucks Manual config (gui) Not in sync with reality Hosts only Services sometimes Appliccation never Chaos

13 But tools do Autodetection Really? Service, Functionalities eg. vhosts etc Single Source of Truth

14 Monitoring 101 Deploy a host, Add it to the monitoring Add collection tools Add check definitions Update the monitoring tool config Restart

15 Collect Metrics 101

16 Collectd Collects Zillion Plugins Nginx,apache, mysql, disk Graphite Carbon Plugin Send metrics to graphite t-collectd

17 Puppet and Graphite Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules! git clone git submodule init git submodule update vagrant up

18 2014 style dashboards

19 Alerting 101

20 Alert all the things And get alertfatigue We are in Germany, Europe It needs to be configurable => Icinga

21 But Monitoring config is Data? If it can generated be generated it's not user content anymore Your computer can generate your config!

22 Stored Configs

23 Collection and Export Export : { Resource << query >>... } Clean out nodes that dissapear puppet node clean

24 Exporting and Collecting

25 Default Puppet Types

26 Puppet-icinga module

27 Monitoring a Vhost

28 Alternative Approaches -external-naginator

29 Deploy a new app Add monitoring Add Real application monitoring Both on infra and on app level

30 Monitoring Puppet & Friends

31 Puppet Runs

32 PuppetMaster

33 PuppetMaster check_command => 'check_tcp!8140', service_description => 'TCP puppet on port 8140', host_name => $::fqdn, use => 'generic-service', contact_groups => $::environment, notification_period => $::icinga::notification_period, notifications_enabled => $::icinga::notifications_enabled, target } => "/etc/icinga/objects/services/${::fqdn}.cfg",

34 PuppetDB

35 PuppetDB(2) check_puppetdb_memory check_puppetdb_processed Nr of reports processed check_puppetdb_populations Java heap memory Resources, nodes, resources per node check_puppetdb_queue Is pgsql down? :)

36 Puppet Dashboard check_command => "check_http!-h ${::fqdn} -p e 200", service_description => 'HTTP PuppetDashboard on port 3000', host_name => $::fqdn, use => 'generic-service', contact_groups => $::environment, notification_period => $::icinga::notification_period, notifications_enabled => $::icinga::notifications_enabled, target } => "/etc/icinga/objects/services/${::fqdn}.cfg",

37 Puppet Dashboard(2) /usr/share/puppet-dashboard/spool

38 PuppetServer

39 #MonitoringSucks Puppetruns break our Icinga boxen Badly Frequently

40 It ain't borken Successful puppet run Successful Icinga reconfigure Disk usage grows Fast

41 Be aware of buckets

42 A Puppet Bug

43 Triggers on Graphs Export Java Metrics JMXTrans Export JMXConfigs Configure NRPE Check Export NagiosCheck Collect JMX Exports on JMXTransNode Graph Em Collect Nagios Configs on Nagios Server

44 Triggers on Graphs

45 Triggers on Graphs

46 Summary Honour your parents Don't manually do what machines can do for you Monitor your puppet infrastructure too! Send Pull Requests Icinga2 module in the works

47 Contact Further Inuits Duboistraat Antwerpen Belgium