Open-Falcon A Distributed and High-Performance Monitoring System. Yao-Wei Ou & Lai Wei 2017/05/22

Transcription

1 Open-Falcon A Distributed and High-Performance Monitoring System Yao-Wei Ou & Lai Wei 2017/05/22

2 Let us begin with a little story

3 Grafana PR#3787 [feature] Add Open-Falcon datasource

4 I'm sorry but we will not merge any new datasources unless they are very popular. - Grafana

5

6

7 Open-Falcon,now is one of the most popular monitoring systems in China.

8 About us Laiwei Technical director of DiDi, China. Founder of Open-Falcon software. Core maintainer of Open-Falcon community. Focus on service reliability, DevOps, Cloud computing, etc. Yaowei Director of WiFire oversea R&D center. Core maintainer of Open-Falcon organization. Leads the development of CDN monitoring system in Fastweb. Focus on CDN and Blockchain. 7

9 Outline Motivation Features Architecture Comparison Community 8

10 Motivation(1/3) There are already so many outstanding open source monitoring systems. Why do we reinvent a wheel? And how does it become the most popular monitoring system in China? 9

11 Motivation(2/3) Zabbix difficult to scale out (~2000) Database can grow very large, very quickly if not tuned properly. not very resource friendly: a lot of connections will be made. we have to build several zabbix cluster to deal with the rapid business grow. a bit of a learning curve Tuning and tweaking can be a lot of work more complex to setup hard to extend manually configuration 2015 open sourced by Xiaomi SRE, China, under Apache License Version 2.0 to replace Zabbix 10

12 Motivation(3/3) key factors of an enterprise class monitoring system. Scalability Performance High Availability Flexibility User- Oriented 11

13 Features

14 Scalability Scalable monitoring system is necessary to support rapid business growth. Each module of Open-Falcon is super easy to scale horizontally. Supports up to hundreds of million transactions per minute (query/ judge/store/search). Can easily support over 100,000 hosts. 13

15 Performance With RRA(Round Robin Archive) mechanism, the one-year history data of 100+ metrics could be returned in just few seconds. stores 10+ years historical metrics. 14

16 High Availability No critical single point of failure Easy to operate and deploy 15

17 Flexibility Falcon-agent has already 400+ built-in server metrics. Users can collect their customized metrics by writing plugins or just simply run a script/program to relay metrics to falcon-agent. Extensive architecture. Customizable metrics. Abundant APIs. 16

18 Efficiency For easier management of alerting rules, Open-Falcon supports strategy, expression, template inheritance, and multiple alerting method, and callback for recovery. Auto discovery of endpoints and counters. API support. 17

19 User-Oriented Supports Grafana Datasource. Open-Falcon could present multi-dimension graph, including user-defined dashboard/screen. 18

20 Grafana with Open-Falcon Screenshot (1/3)

21 Grafana with Open-Falcon Screenshot (2/3)

22 Grafana with Open-Falcon Screenshot (3/3)

23 Architecture

24 Components COLLECT STORE PRESENT JUDGE NOTIFY 23

25 Components COLLECT STORE PRESENT JUDGE NOTIFY 23

26 Components COLLECT STORE PRESENT JUDGE NOTIFY 23

27 Components COLLECT Falcon-Agent Proxy-Gateway aggregator nodata STORE graph redis MySQL hbs JUDGE judge PRESENT Falcon-Dashboard Grafana Falcon-API NOTIFY alarm 23

28 Center Status 24

29 Before: Too many modules Alarm Dashboard Portal FE UIC Nodata Query 5 Alarm 1 HBS 5 Aggregator Graph 20 Judge 20 Sender 1 Tasks Transfer 60 Link 1 Agent1 Agent2 Agent3 Agent4 AgentN

30 After

31 After Alarm FE Dashboard Portal UIC

32 After Alarm Dashboard Portal FE Dashboard UIC Alarm Judge Graph API Gateway Transfer (Queue) HBS (Control)

33 After Alarm Dashboard Portal FE Dashboard UIC Alarm API Gateway Judge Graph Falcon-Plus Transfer (Queue) HBS (Control) Central Status

34 After Alarm Dashboard Portal FE Dashboard UIC Alarm API Gateway Judge Graph Falcon-Plus Transfer (Queue) HBS (Control) Central Status Agent1 Agent2 Agent3 Agent4 AgentN

35 After Grafana Alarm Dashboard Portal FE Dashboard UIC Alarm API Gateway Judge Graph InfluxDB Cassandra OpenTSDB ELK Falcon-Plus Transfer (Queue) HBS (Control) Central Status Agent1 Agent2 Agent3 Agent4 AgentN

36 Design Philosophy 27

37 Design Philosophy MICROSERVICES 10 major modules individual deployed 27

38 Design Philosophy MICROSERVICES 10 major modules individual deployed PUSH agent as a push proxy 27

39 Design Philosophy MICROSERVICES 10 major modules individual deployed PUSH agent as a push proxy RRDTOOL with consistent hash 27

40 Design Philosophy MICROSERVICES 10 major modules individual deployed PUSH agent as a push proxy RRDTOOL with consistent hash BINARY Go static binary 27

41 Design Philosophy MICROSERVICES 10 major modules individual deployed PUSH agent as a push proxy RRDTOOL with consistent hash BINARY Go static binary DEPLOYMENT mass deployment by ops-updater 27

42 Design Philosophy MICROSERVICES 10 major modules individual deployed PUSH agent as a push proxy RRDTOOL with consistent hash BINARY Go static binary DEPLOYMENT mass deployment by ops-updater GRAFANA open-falcon datasource 27

43 Comparison

44 Compared to Prometheus OPEN-FALCON PROMETHEUS Abundant APIs Push Model: Auto Discovery Easy to scale out simple alert management of own dashboard Faster query performance of RRA Simple shellscript as plugin Limited expression Metrics API Pull Model: Manual configuration Harder to scale out Alertmanager offers grouping, deduplication and silencing functionality Slower, Recording rules A bit learning curve to write exporter and collector PromQL 29

45 Compared to Prometheus OPEN-FALCON PROMETHEUS Abundant APIs Push Model: Auto Discovery Easy to scale out simple alert management of own dashboard Faster query performance of RRA Simple shellscript as plugin Limited expression Metrics API Pull Model: Manual configuration Harder to scale out Alertmanager offers grouping, deduplication and silencing functionality Slower, Recording rules A bit learning curve to write exporter and collector PromQL 29

46 Compared to Prometheus OPEN-FALCON PROMETHEUS Abundant APIs Push Model: Auto Discovery Easy to scale out simple alert management of own dashboard Faster query performance of RRA Simple shellscript as plugin Limited expression Metrics API Pull Model: Manual configuration Harder to scale out Alertmanager offers grouping, deduplication and silencing functionality Slower, Recording rules A bit learning curve to write exporter and collector PromQL 29

47 Compared to Prometheus OPEN-FALCON PROMETHEUS Abundant APIs Push Model: Auto Discovery Easy to scale out simple alert management of own dashboard Faster query performance of RRA Simple shellscript as plugin Limited expression Metrics API Pull Model: Manual configuration Harder to scale out Alertmanager offers grouping, deduplication and silencing functionality Slower, Recording rules A bit learning curve to write exporter and collector PromQL 29

48 Compared to Prometheus OPEN-FALCON PROMETHEUS Abundant APIs Push Model: Auto Discovery Easy to scale out simple alert management of own dashboard Faster query performance of RRA Simple shellscript as plugin Limited expression Metrics API Pull Model: Manual configuration Harder to scale out Alertmanager offers grouping, deduplication and silencing functionality Slower, Recording rules A bit learning curve to write exporter and collector PromQL 29

49 Compared to Prometheus OPEN-FALCON PROMETHEUS Abundant APIs Push Model: Auto Discovery Easy to scale out simple alert management of own dashboard Faster query performance of RRA Simple shellscript as plugin Limited expression Metrics API Pull Model: Manual configuration Harder to scale out Alertmanager offers grouping, deduplication and silencing functionality Slower, Recording rules A bit learning curve to write exporter and collector PromQL 29

50 Compared to Prometheus OPEN-FALCON PROMETHEUS Abundant APIs Push Model: Auto Discovery Easy to scale out simple alert management of own dashboard Faster query performance of RRA Simple shellscript as plugin Limited expression Metrics API Pull Model: Manual configuration Harder to scale out Alertmanager offers grouping, deduplication and silencing functionality Slower, Recording rules A bit learning curve to write exporter and collector PromQL 29

51 Compared to Prometheus OPEN-FALCON PROMETHEUS Abundant APIs Push Model: Auto Discovery Easy to scale out simple alert management of own dashboard Faster query performance of RRA Simple shellscript as plugin Limited expression Metrics API Pull Model: Manual configuration Harder to scale out Alertmanager offers grouping, deduplication and silencing functionality Slower, Recording rules A bit learning curve to write exporter and collector PromQL 29

52 Community

53 Ecosystem (1/2) Banking IaaS, SaaS CDN O2O Social Entertainment 31

54 Ecosystem (2/2) OS UI Switch Hadoop HBase Docker Redis Plugins MongoDB GPU RabbitMQ HAProxy Nginx JMX LVS Tomcat WebSphere IIS 32

55 Join us Github Homepage Contact us Wechat 33

56 Summary

57 100+ companies

58 40,000+ servers

59 400+ built-in metrics

60 5,000+ users

61 3 seconds

62 Q&A