@InfluxDB. David Norton 1 / 69

Transcription

1 @InfluxDB David Norton 1 / 69

2 Instrumenting a Data Center 2 / 69

3 3 / 69

4 4 / 69 The problem: Efficiently monitor hundreds or thousands of servers 5 / 69

5 The solution: Automate it! 6 / 69

6 Automate what? data collection, storage, analysis, & alerting 7 / 69

7 Easy! Write a few scripts & store the data in SQL. 8 / 69

8 1 server 100 measurements 8,640 per day (once every 10s) 365 days = 315 million records (points) per year 9 / servers

9 100 measurements per server 8,640 per day (once every 10s) 365 days = 3.2 billion records (points) per year 10 / 69 2,000 servers 200 measurements per server 17,280 per day (once every 5s) 365 days = 2.5 trillion records (points) per year

10 11 / 69 Time series database is ideal for this type of data & workload 12 / 69

11 "time series" 13 / 69

12 Values and time stamps? Value Time 14 / 69

13 Values and time stamps? Value time :00 PM :00 PM Time value 10,0 20,0 15 / 69

14 Step CPU usage every 10 seconds... Value Time 16 / 69 Cumulative steps taken...

15 Step Count Value Time 17 / 69 Happy InfluxDB users... InfluxDB Users Value Time

16 18 / 69 Time series database Value Time Step Count Value Time InfluxDB Users Value Time? Value Time 19 / 69

17 20 / 69

18 InfluxDB 21 / 69

19 InfluxDB time series database 22 / 69 InfluxDB time series database

20 no external dependencies 23 / 69 InfluxDB time series database no external dependencies distributed & scalable

21 24 / 69 InfluxDB time series database no external dependencies distributed & scalable easy to install, use, & maintain

22 25 / 69 InfluxDB time series database no external dependencies distributed & scalable easy to install, use, & maintain open source (MIT license) 26 / 69

23 InfluxDB time series database no external dependencies distributed & scalable easy to install, use, & maintain open source (MIT license) written in Go 27 / 69

24 Features SQL like query language HTTP(S) API for writes & queries Supports other protocols (collectd, graphite, opentsdb) Automated data retention policies Aggregate data on the fly 28 / 69

25 Data model / 69 influx d

26 influx d 30 / 69

27 influx d 31 / 69

28 32 / 69 influx d measurements DISK 33 / 69

29 influx d tags measurements country = 'DEU' region = 'neast' DISK 34 / 69

30 influx d series = measurement + unique tag set + country = 'DEU' region = 'west' 35 / 69

31 influx d series = measurement + unique tag set + country = 'DEU' region = 'west' OR + country = 'DEU' region = 'east' 36 / 69 influx d Points: values in a series

32 country = time :00 PM :01 PM value 10,0 20,0 + 'DEU' region = 'east' time :00 PM :01 PM value 14,0 38,0 + country = 'USA' region = 'east' 37 / 69 What's indexed?

33 38 / 69 What's indexed? Metadata: Measurement names

34 39 / 69 What's indexed? Metadata: Measurement names Tag keys & values 40 / 69

35 What's indexed? Metadata: Measurement names Tag keys & values Field keys 41 / 69

36 What's indexed? Metadata: Measurement names Tag keys & values Field keys Field values by time* WHERE time > now() 1m 42 / 69

37 Metadata index is held in memory at run time 43 / 69 What's not indexed?

38 44 / 69 What's not indexed? Field values by value WHERE value > 2.718

39 45 / 69 What's not indexed? Field values by value WHERE value > Metadata by time SHOW MEASUREMENTS WHERE time > now() - 1h

40 46 / 69 Data retention 47 / 69

41 48 / 69

42 InfluxDB has Retention Policies 49 / 69

43 How do they work? 50 / 69

44 All data is written to a retention policy 51 / 69 Each retention policy has a duration specified

45 (between 1 hour and infinite) 52 / 69 SELECT mean(value) INTO cpu_1h.cpu FROM raw.cpu WHERE time > now() 1w GROUP BY time(1h) raw (1 day) cpu_1h (1 week) shards shards

46 53 / 69 How to automate downsampling? 54 / 69

47 Continuous Queries Stored in the database and run periodically in the background 55 / 69

48 raw (1 day) CREATE CONTINUOUS QUERY mycq ON mydb BEGIN SELECT mean(value) INTO cpu_1h.cpu FROM raw.cpu GROUP BY time(1h) END cpu_1h (1 week) shards shards 56 / 69

49 count, distinct, mean, median, sum Replication is handled through Retention Policies 57 / 69 Functions Aggregations:

50 bottom, first, last, max, min, percentile, top 58 / 69 Functions Aggregations: count, distinct, mean, median, sum Selectors:

51 ceiling, derivative, difference, floor, histogram, 59 / 69 Functions Aggregations: count, distinct, mean, median, sum Selectors: bottom, first, last, max, min, percentile, top Transformations:

52 stddev 60 / 69 Data ingestion 61 / 69

53 InfluxDB supports: collectd opentsdb graphite 62 / 69

54 Client libraries Telegraf Open Source (MIT License) Also written in Go Plugin based (~30 currently and growing) Cross platform 63 / 69

55 Go Ruby Java Python etc. ( 64 / 69 HTTP(S) API Write data via HTTP using a simple text based protocol

56 cpu,host=servera value= / 69 Visualization

57 Grafana 66 / 69

58 67 / 69 Chronograf 68 / 69

59 Thank David Norton 69 / 69

60