How to see what is happening inside your OpenStack using Elastic Stack and Prometheus

Transcription

1 How to see what is happening inside your OpenStack using Eastic Stack and Prometheus

2 Introduction & Agenda About me - Csaba Patyi (csaba@componentsofteu) - Consutant and Instuctor at Component Soft Ltd - 6 years of experience from the Ops side - Mirantis and COA Certs - Linux Foundation Certs Agenda - About Component Soft - Logging with Eastic Stack - Monitoring with Prometheus - (Monasca and Ceiosca)

3 About Component Soft Educationa Services Bash, Per, Python courses Red Hat Linux, Advanced Linux Java, Scaa and C++ courses OpenStack, Docker, Kubernetes, Ceph Software testing methodoogies Apache, Tomcat, MySQL Veritas Storage Foundation and Custer Authorized Orace courses Next Gen Teecom and Networking Consuting Services: OpenStack and Ceph consuting and support services Docker and Kubernetes consuting and support services Open Source Fu Stack consuting and support services Contact Us: Website: Office and training site: 1116 Budapest Fehérvári street Phone: Fax: E-mai:

4 What is OpenStack? - The technica term: Coection of Independent but reated projects provides web UI Dashboard Horizon Teemetry Service Ceiometer measures usage Object Store Swift Network Service Neutron provides connectivity Compute Service Nova provides OS tempate Image Store Gance provides backstore Orchestration Service Heat provides voumes automates creation Bock Storage Cinder Identity Service Keystone provides authentication & authorization

5 Distribution of services neutron-server horizon Controer node(3) neutron-3-agent Network node** cinder gance neutron-2-agent Storage node** swift-account nova-api nova-scheduer Compute node** swift-container swift-object keystone neutron-2-agent Mysq nova-compute swift-proxy RabbitMQ/Qpid

6 Cient VM provisioning in-depth 1 Keystone 16 2 nova-api 5 AMQ hypervisor 1 8 nova-compute 9 Gance Neutron nova-scheduer nova-conductor Cinder 6 4 DB

7 Hit an error? No probem Check the ogs Where are they? Node roe / Project Name Controer Network Compute Swift Storage Horizon 3 Keystone 5 Nova Neutron Gance 2 Cinder 5 1 Ceiometer (+ aodh and gnocchi) Heat 4 Swift 2 1 SUM Controer, 2 Network, 20 Compute, 6 Storage Node == 31 nodes and 312 og fies # OpenStack og fie ocations per service: for i in horizon keystone nova neutron gance cinder ceiometer aodh gnocchi heat swift ; do COUNT=`find /var/og/ egrep $i grep og$ wc -` echo "$i : $COUNT" done;

8 Eastic Stack for the rescue Formay known as ELK stack - Easticsearch Storing and indexing ogs Makes fast search possibe - Logstash transforming incoming ogs and sending it to Easticsearch - Kibana Web interface for the Eastic Stack New modues/pugins: - XPACK (most of them ony in the payed version) - Beats Fiebeat a sma agent appication coecting and sending ogs directy to EasticSearch or Logstash Metricbeat a sma agent appication coecting and sending metrics directy to EasticSearch or Logstash Etc

9 Configuration for fiebeat #/etc/fiebeat/fiebeatym fiebeat: prospectors: - paths: - "/var/og/nova/*og" excude_fies: ['\gz$'] document_type: nova tags: [openstack_service_ogs, nova'] output: ogstash: enabed: true hosts: - ogstash-server:5044 index: fiebeat buk_max_size: 50

10 Reay simpe Logstash config #/usr/share/ogstash/pipeine/ogstashconf input { beats { port => 5044 } } fiter { grok { match => { "message" => "(?m)^%{timestamp_iso8601:date}\s+\d+\s+(?<ogeve>audit CRITICAL DEBUG INFO TRACE WARNING ERR OR)\s(?<modue>\S+)*$" } } if [modue] == "iso8601iso8601" { drop {} } } output { easticsearch { hosts => "easticsearch-server:9200" sniffing => true manage_tempate => fase index => "%{[@metadata][beat]}-%{+yyyymmdd}" document_type => "%{[@metadata][type]}" } }

11 What about mutiine ogs? 1: :42: WARNING neutrondbagents_db [req-2d2958ae-dcaa-40be-b7d3-6e3513a0f2d ] Agent heathcheck: found 5 dead agents out of 13: 2: Type Last heartbeat host 3: Metering agent :13:39 network2openstackoca 4: Loadbaancerv2 agent :13:41 network2openstackoca 5: L3 agent :13:41 network2openstackoca 6: Metadata agent :13:39 network2openstackoca 7: DHCP agent :13:10 network2openstackoca

12 What about mutiine ogs? 1: :29: ERROR neutronpuginsm2driversopenvswitchagentovs_neutron_agent [-] Faied reporting state! 2: :29: ERROR neutronpuginsm2driversopenvswitchagentovs_neutron_agent Traceback (most recent ca ast): 3: :29: ERROR neutronpuginsm2driversopenvswitchagentovs_neutron_agent Fie "fie-ocation", ine 320, in _report_state 4: :29: ERROR neutronpuginsm2driversopenvswitchagentovs_neutron_agent True) 5: :29: ERROR neutronpuginsm2driversopenvswitchagentovs_neutron_agent Fie "another-fie-ocation", ine 88, in report_state 6: :29: ERROR neutronpuginsm2driversopenvswitchagentovs_neutron_agent return method(context, 'report_state', **kwargs) 7: :29: ERROR neutronpuginsm2driversopenvswitchagentovs_neutron_agent Fie "another-another-fie", ine 169, in ca

13 Fiebeat mutiine config exampe #/etc/fiebeat/fiebeatym fiebeat: prospectors: - - paths: - "/var/og/nova/*og" excude_fies: ['\gz$'] document_type: nova mutiinepattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}[0-9]{3} [0-9]+ (ERROR WARNING INFO DEBUG TRACE) [0-9A-Za-z_]+ \[' mutiinenegate: true mutiinematch: after tags: [openstack_service_ogs, nova] output: easticsearch: enabed: true hosts: - ogstash-server:5044 index: fiebeat buk_max_size: 50

14 Kibana Demo

15 Buid test environment for yoursef

16 What about monitoring and aarming? You have to monitor the physica infrastructure - Traditiona: Zabbix Nagios Icinga Zenoss + a few more - New payers Eastic Stack XPACK (commercia icence) Prometheus You have to monitor the virtua infrastructure - Traditiona Ceiometer - New payers Monasca Ceiosca

17 Prometheus an open-source systems monitoring and aerting tookit originay buit at SoundCoud joined the Coud Native Computing Foundation ( in 2016 as the second hosted project after Kubernetes Some of the Features - a muti-dimensiona data mode (time series identified by metric name and key/vaue pairs) - no reiance on distributed storage; singe server nodes are autonomous - time series coection happens via a pu mode over HTTP - targets are discovered via service discovery or static configuration - mutipe modes of graphing and dashboarding support Some of the Components - the main Prometheus server which scrapes and stores time series data - a push gateway for supporting short-ived jobs - specia-purpose exporters (for HAProxy, StatsD, Graphite, etc) - an aertmanager

18 Prometheus Architecture

19 Demo depoyment of Prometheus

20 Prometheus exporters for OpenStack Consu exporter (officia) cadvisor EasticSearch exporter Memcached exporter (officia) MongoDB exporter MySQL server exporter (officia) Node/system metrics exporter (officia) RabbitMQ exporter RabbitMQ Management Pugin exporter Ceph exporter Guster exporter Apache exporter HAProxy exporter (officia)

21 Prometheus configuration goba: scrape_interva: 30s evauation_interva: 30s abes: custer: swarm repica: "1" # Attach these abes to any time series or aerts when communicating with # externa systems (federation, remote storage, Aertmanager) externa_abes: monitor: 'prometheus-swarm' rue_fies: - "aertrues_nodes" - "aertrues_tasks" - "aertrues_service-groups" scrape_configs: - job_name: 'node-exporter' dns_sd_configs: - names: - 'tasksnode-exporter' type: 'A' port: job_name: "node" scrape_interva: 5s static_configs: - targets: [' :9100', ' :9100', ' :9100', ' :9100', ' :9100', ' :9100', ' :9100']

22 Prometheus query exampes Show Overa CPU usage for a server * (1 - avg by(instance)(irate(node_cpu{mode='ide'}[5m]))) HTTP request rate, per second an hour ago - rate(api_http_requests_tota{status=500}[5m] offset 1h) Disk Wi Fi in 4 Hours - predict_inear(node_fiesystem_free[1h], 4*3600)

23 Prometheus Aarm Syntax ALERT <aert name> IF <expression> [ FOR <duration> ] [ LABELS <abe set> ] [ ANNOTATIONS <abe set> ]

24 Prometheus Demo

25 Sources OpenStack og fie ocations per service: Mutiine soutions for LogStash: Grok Pattern names for Logstash: Mutiine soutions for Fiebeat: Mutiine test site for Fiebeat: Docker ELK-stack depoyment exampe: Good ogstash pipeine config exampes: Prometheus main site: Prometheus Docker Swarm usage: Prometheus query exampes:

26 Q & A