Icinga at Hyves.nl. Jeffrey Lensen System Engineer

Transcription

1 Icinga at Hyves.nl Jeffrey Lensen System Engineer

2 Hyves Dutch social network website 3 billion pageviews / month 10M dutch members (17M population) ~7M unique visitors / month (Comscore 09/2011) ~2.3M unique visitors / day photo uploads / day 7M chat messages / day 6Gbps daily outgoing traffic 2

3 Hyves environment 3000 hosts running Gentoo 3 Datacenters 190 types of server functions 160 Employees System Engineering team: 12 Developers: 45 3

4 Back in the day 1 Datacenter 150 servers 4 System Engineers 1 Nagios instance Manual configuration 4

5 Keep up with serverpark growth Popularity required expansion Receiving servers at a time Manual configuration became unmanageable 5

6 Solutions to growth Templates for host and hostgroup configurations Servicechecks defined per hostgroup Automated configuration with scripts (hosts, hostgroups, servicedependencies) Server management database as source Servicedependencies generated based on check_name prefix 6

7 Keep up with more serverpark growth From 1 to 3 datacenters Serverpark grew to 1500 hosts 1 Nagios host isn t enough anymore 7

8 Solutions to more growth Distributed Nagios setup consisting of: 1 Central Nagios server for alerting and webinterface 9 Distributed Nagios servers Required little changes to configuration scripting Distribution based on location and function 8

9 Watching the watchers Monitoring Nagios hosts with Nagios on NOC NOC monitored by one of the Nagios hosts Monitoring all datacenters from HQ 9

10 Distributed Nagios scaling problems Long reloads due to large configuration (mainly Central server) Freezes during large (network) fall-outs -> No alerting! Webinterface could no longer load 10

11 Icinga Switched in November 2010 No more central monitoring server needed Standalone web interface Database backend API Rapid development Painless migration: sed -i s/nagios/icinga/g /etc/nagios/*cfg mv /etc/nagios/* /etc/icinga/ 11

12 Icinga setup 12 Icinga hosts 1 NOC Icinga host service checks hosts 12

13 Icinga setup 2 Icinga-web + database hosts Loadbalanced database and API Easy failover 13

14 Make use of the API: Overview checks Overview checks for hostgroups and services Minimizes alerts during large failures Python script using API Example: python check_monitoring_overview.py --hostgroup webserver --service HTTP,HipHop -w 5% -c 10% All 472 'HTTP', 'HipHop' services for 'mainweb' are OK 14

15 Missing monitoring Is everything that should be monitored, being monitored? Won t realize until it s too late Angry people.. 15

16 Solution: Puppet Puppet is an open-source next-generation server automation tool. It is composed of a declarative language for expressing system configuration, a client and server for distributing it, and a library for realizing the configuration. Modules for each application (Nginx, Postfix, SNMP etc.) Roles based on function as set in server management database Everything is defined in Puppet 16

17 Example: Nginx module class nginx { tag("nginx") package { "nginx": ensure => "latest", category => "www-servers" service { "nginx": enable => true, ensure => running 17

18 Example: Role module class role::webserver inherits role { include nginx 18

19 Using Puppet to generate configs Supports Nagios Exported Resources Exported Resources stored in MySQL backend Define nagios_services in the matching modules 19

20 Include monitoring in NGINX module modules/nginx/manifests/init.pp: class nginx { tag("nginx") { "HTTP $hostname": service_description => "HTTP", check_command => "check_web_http", event_handler => "service_restart!nginx!critical", contact_groups => "sysadmins" 20

21 Predefine defaults in defines.pp $ notifications_enabled = $systemstatus? { operational => "1", fail => "0" Nagios_service { ensure => present, host_name => $hostname.$domain, use => "generic-service", notifications_enabled => $ notifications_enabled, target => "/etc/icinga/puppetgenerated/services/$hostname.cfg", notes => $monitoringhost 21

22 Predefine defaults in defines.pp Nagios_host { ensure => present, host_name => $hostname.$domain, hostgroups => $role, use => "generic-host", alias => $hostname, notifications_enabled => $ notifications_enabled, target => "/etc/icinga/puppetgenerated/hosts/$hostname.cfg", notes => $monitoringhost 22

23 Define host in monitoring module modules/monitoring/manifests/init.pp: class monitoring { "$hostname": address => $ip modules/role/manifests/init.pp: class role { include monitoring 23

24 Retrieving resources class icinga { tag("icinga") Nagios_host << notes == "$hostname" >> { require => File["/etc/icinga/puppetgenerated/hosts"] Nagios_service << notes == "$hostname" >> { require => File["/etc/icinga/puppetgenerated/services"] 24

25 Checking generated configuration class icinga { <snip> exec { "verify new cfg": command => "/usr/bin/icinga -v /etc/icinga/verify-puppetgenerated.cfg", require => Class["get_icinga_puppet_resources"] exec { "mv cfgs": command => "rm -rf /etc/icinga/puppet/*; mv /etc/icinga/puppetgenerated/* /etc/icinga/ puppet/", require => Exec["verify new cfg"] exec { "restart icinga": command => ""/usr/bin/printf '[] RESTART_PROGRAM\n' > /var/icinga/rw/icinga.cmd"", require => [ Exec["mv cfgs"], Service["icinga"] ] 25

26 Problems exporting resources Puppet runs on Icinga hosts took between 10 and 30 minutes! Makes it hard to quickly change monitoring Most time spent retrieving and processing (Nagios) resources 26

27 get_icinga_puppet_resources.py Determined queries used by Puppet Get all resource IDs For each ID get parameter name and value Write to defined file ( target ) Finishes in 15 seconds! 27

28 Retrieving resources ourselves class icinga { <snip> exec { "get_icinga_puppet_resources": command => "/usr/bin/python /usr/local/bin/get_icinga_puppet_resources.py", require => [ File["/etc/icinga/puppetgenerated/hosts"], File["/etc/icinga/puppetgenerated/services"] ] 28

29 Other cool stuff to do with Puppet Generate daemon checks for servers based on config file Generate overview daemon checks using Icinga API 29

30 Retrieve daemons from config modules/role/lib/facter/hyvesfacters.rb: Facter.add("hyves_daemons") do daemons = ["None"] if File::exists?( "/<path_to_config>/daemons.conf" ) daemons = [] daemonarray = [] daemonconf = %x{grep name /<path_to_config>/ daemons.conf for daemon in daemonconf daemon.sub!(/.*\* name:/, '') daemonarray.push(daemon.chomp) end end setcode do daemonarray.uniq end end 30

31 Create services for daemons modules/daemons/manifests/init.pp: class daemons { define add_daemon_check { "$name Daemon $hostname": use => "Daemon-check", service_description => "$name Daemon", check_command => "check_daemon!$name" add_daemon_check { $hyves_daemons: 31

32 Retrieving unique daemons from API require 'net/http' module Puppet::Parser::Functions newfunction(:get_daemons, :type => :rvalue, :docs => "\ This function returns an array of all current daemons, based on the Icinga API ") do args domain = "<icinga-web_url>" url = "/icinga-web/web/api/service/filter[and(service_name%7clike%7c*daemon)]/ columns[service_name]/order[service_name;asc]/authkey=<api_key>/json" response = Net::HTTP.get_response(domain, url) data = response.body results = PSON.parse(data) daemons = Array.new results.each { result daemon = result['service_name'] daemon.sub!(/ Daemon/, '') daemons << daemon daemons.uniq end end 32

33 Create overview services for daemons modules/icinga/manifests/noc.pp: $ daemons = get_daemons() templatefile { "/etc/icinga/puppetgenerated/other/daemons.cfg": template => template("icinga/hyvesdaemons.cfg.erb") hyvesdaemons.cfg.erb: <% daemons.each do daemon -%> define service{ use DaemonOverview-check host_name daemons service_description <%= daemon %> <% end -%> 33

34 Deployment Deploy script to start Puppet runs on all monitoring hosts Reports status of Puppet runs once they re finished Starts Puppet run on NOC monitoring host 34

35 What if a machine doesn t run Puppet? Check to check configuration Retrieve all operational hosts from servermanagent DB Retrieve all hosts from Icinga API Alert if something is missing or notifications are off 35

36 What about failover? Requires puppet run on all server Speed up puppet runs with --noop Redeploy Icinga 36

37 ICL (Icinga CommandLine) Python based script Libraries for access to Icinga API and MK_Livestatus Library for things like translating exit codes, and statuses See host/service status information Control monitoring and alerting Quickly see open problems 37

38 Integration with other tools Integration with server administration script to change status Fail -> disable notifications Operational -> check if everything is OK + enable notifications Deprecated -> disable notification + remove from Puppet DB Integration with failover scripts Deploy monitoring when adding new servers Scripts can check status of hosts and services before continuing 38

39 Demo time 39

40 Plans for the (near) future Upgrade Icinga to 1.6 Clean up ICL and make compatible with Icinga 1.6 Put ICL on GitHub Expose API to developers Trend analysis / integration with Ganglia/Graphite 40

41 Thank you, questions? Puppet: Github: Hyves: 41