COSMOS. Controls Open Source MOnitoring System Project Status Report. Frank on behalf of the COSMOS core team BE-CO Technical Meeting:

Transcription

1 COSMOS Controls Open Source MOnitoring System Project Status Report Frank Frank on behalf of the COSMOS core team BE-CO Technical Meeting: Laura Julien Luigi Felix Sergey

2 Agenda ABACUS review conclusions COSMOS mandate Definition Project Proposal The COSMOS solution Progress Report Conclusions La surprise du chef Julien 2

3 ABACUS Review: Main Conclusions (March 2017) Success and expansion of DIAMON2 Difficult to operate and maintain New paradigm: Responsibility of experts to collect metrics, define alerts and setup dashboards Sysadmin needs to be aligned with IT solution Reuse ITMON components as much as possible OS market extremely rich in the domain of monitoring 3

4 COSMOS Mandate Study and implement a new monitoring system for the CO Infrastructure Context: LS2, new CO systems (IN servers, MasterFIP, WRS, ) Concentrate efforts on our core business Use de-facto standard technologies and OSS 4

5 COSMOS Mandate Objectives: Clarify the scope of the monitoring service Reduce proliferation of solutions and overlap between services Simplify maintenance and configuration/tuning (host/serv., metrics, alerts, ) Provide integrated and coherent user interfaces Phase 1 Targets: CO infrastructure monitoring ready for LS2 (Q4/2018) Main priorities: All CO hosts & hw-modules base checks (ping, system metrics, process/services state) Golden use-cases: QuadServer, JBOD/Raid, crates and field-buses (WRS, Pul.Rep., WFIP) 5

6 Definition: What is COSMOS monitoring? It is not A database A control command system A (smart) diagnostic tool A Logging tool It does Constantly track the system resources Detect fluctuations, slow or failing components Provide health reports for each host and service Collect metrics Notify (by , SMS) Host state Process state Service state Service metrics introspection Acc. equipment check COSMOS Monitoring Diagnostic ping up/down JMS, RDA, SQL, JMX, CMX, FESA, Logging, profiling, search, processing, Offl. analysis, AFT, DocFIP, CANDI, Static configuration Dynamic configuration We know what we are looking for! 6

7 Icinga2 was retained CLI, API web IDO own nagios Processing Full-stack OS solution for monitoring Fits correctly with the new COSMOS paradigm Large user community at CERN (inc. Collectd) Strong and flexible fs-based configuration Multi-user interface Authentication/SSO support Large set of built-in agents Very good documentation syslog, applog, sys., SNMP, IPMI, JMX/CMX, CMW, 7

8 COSMOS Project Proposal The project will be implemented in 2 phases: Phase 1: hosts & services monitoring of the whole BE-CO infrastructure Setup Icinga2 monitoring solution Network distribution (TN/GPN), databases, web interface, GRAFANA, etc. Deploy monitoring for CO golden use cases Status + metrics + notification + dashboards Use Collectd like a DIAMON agent to collect the system metrics cpu, disk, network, memory, interrupts, etc. Generate status and notification from Collectd metrics to Icinga 8

9 Project Proposal Phase 2: System Integration and Extension Extend the Icinga 2 monitoring solution to the complete CO infrastructure Implement advanced requirements: actions/feedback, dependency relations, auto-discovery Review Architecture: Performance, scalability, HA Provide integrated user/expert configuration tools COSMOS and CO services integration (alarms, logging, etc.) DIAMON data access integration (if required) Discuss & open-up the Icinga 2 solution to other groups responsible for monitoring 9

10 CO routing IT firewall rules Solution: COSMOS layout GPN Grafana service IT TN Sys. metrics Collectd server agent agent agent agent IT time series perf.data agent check agent p-check Icinga2 server check check check Icinga2 satellite events IT check check Icinga2web server 10

11 Progress Report: November 2017 Icinga 2: master and web servers (x2 Pro, Dev) available for development and tests Ansible: base roles done and setup enabled, GitLab control IT Services: Large effort done to make the whole chain integrated Databases: COSMOS relies on DB on-demand service with TN/GPN routing (IDO: MySQL, TS: influxdb) Visualisation: COSMOS relies on IT Openshift service for Grafana (data transformation required) Configuration: Model defined. First generation from Ansible and CCDB works well Authentication: LDAP/SSO support for ICINGA2 servers, Grafana setup as multi-tenant service Collectd: Setup and tests done in advance. Linux hosts series accessible from Grafana! Coming soon: Generate Collectd status and notification from time-series to Icinga2 CO checks: Basic host checks: alive, http, etc. SNMP: WR-Switch, Pulse Rep., ELMA crate Coming soon: Quad servers and enclosures (JBOD/RAID systems), Wiener crate Expert/User documentation regularly updated: 11

12 Conclusions COSMOS is progressing well, with respect to the review recommendations Initial developments already addressed many technical queries Full-stack monitoring test successful: Icinga2/Collectd DBs (status/time-series) Grafana Performance and scalability yet to be assessed Ready to welcome the new team member to boost development and complete our work on time 12

13 Questions? 13

14 SPARE 14

15 Solution: Icinga2 checks User is responsible for implementing his Icinga2 checks: A script or binary: returning a status code: >check_http H cwe-513-vol510 w 10 c 20 f follow Status code Host status Service status UP UNSTABLE DOWN/UNREACHABLE DOWN/UNREACHABLE OK WARNING CRITICAL UKNOWN a text output: perf. data (optional): time= s; ; ; size=85997b;;;0 HTTP OK: HTTP/ OK bytes in second response time User is responsible for implementing his dashboards See the demo 15