opentsdb - Metrics for a distributed world Oliver Hankeln /

Transcription

1 opentsdb - Metrics for a distributed world Oliver Hankeln /

2 Who am I? Senior Engineer - Data and Infrastructure at gutefrage.net GmbH Was doing software development before DevOps advocate

3 Who is Gutefrage.net? Germany s biggest Q&A platform #1 German site (mobile) about 5M Unique Users #3 German site (desktop) about 17M Unique Users > 4 Mio PI/day Part of the Holtzbrinck group Running several platforms (Gutefrage.net, Helpster.de, Cosmiq, Comprano,...)

4 What you will get Why we chose opentsdb What is opentsdb? How does opentsdb store the data? Our experiences Some advice

5 Why we chose opentsdb

6 We were looking at some options Munin Graphite opentsdb Ganglia Scales well no sort of yes yes Keeps all data Creating metrics no no yes no easy easy easy easy

7 Bingo! We have a winner! Munin Graphite opentsdb Ganglia Scales well no sort of yes yes Keeps all data Creating metrics no no yes no easy easy easy easy

8 Separation of concerns $ unzip strip touch finger grep mount fsck more yes fsck fsck fsck umo unt sleep

9 The ecosystem App feeds metrics in via RabbitMQ We base Icinga checks on the metrics We evaluate etsy Skyline for anomaly detection We deploy sensors via chef

10 opentsdb Written at StumbleUpon but OpenSource Uses HBase (which is based on HDFS) as a storage Distributed system (multiple TSDs)

11 The big picture UI TSD This is really a cluster tcollector API TSD TSD TSD HBase

12 Putting data into opentsdb $ telnet tsd01.acme.com 4242 put proc.load.avg5min host=db01.acme.com

13 It gets even better tcollector is a python script that runs your collectors handles network connection, starts your collectors at set intervals does basic process management adds host tag, does deduplication

14 A simple tcollector script #!/usr/bin/php <?php#cast a die$die = rand(1,6);echo "roll.a.d6 ". tim

15 What was that HDFS again? HDFS is a distributed filesystem suitable for Petabytes of data on thousands of machines. Runs on commodity hardware Takes care of redundancy Used by e.g. Facebook, Spotify, ebay,...

16 Okay... and HBase? HBase is a NoSQL database / data store on top of HDFS Modeled after Google s BigTable Built for big tables (billions of rows, millions of columns) Automatic sharding by row key

17 How opentsdb stores the data

18 Keys are key! Data is sharded across regions based on their row key You query data based on the row key You can query row key ranges (say e.g. A...D) So: think about key design

19 Take 1 Row key format: timestamp, metric id

20 Take 1 Row key format: timestamp, metric id , 5 17 Server A Server B

21 Take 1 Row key format: timestamp, metric id , , 6 24 Server A Server B

22 Take 1 Row key format: timestamp, metric id , , , , Server A , , 8 99 Server B

23 Take 1 Row key format: timestamp, metric id , , , , Server A , , , 5 12 Server B , 6 42

24 Solution: Swap timestamp and metric id Row key format: metric id, timestamp 5, , , , Server A 6, , , Server B 6,

25 Solution: Swap timestamp and metric id Row key format: metric id, timestamp 5, , , , Server A 6, , , Server B 6,

26 Take 2 Metric ID first, then timestamp Searching through many rows is slower than searching through viewer rows. (Obviously) So: Put multiple data points into one row

27 Take 2 continued 5, ,

28 Take 2 continued Row key 5, ,

29 Take 2 continued Cell Name Row key 5, ,

30 Take 2 continued Cell Name Data point Row key 5, ,

31 Where are the tags stored? They are put at the end of the row key Both metric names and metric values are represented by IDs

32 The Row Key 3 Bytes - metric ID 4 Bytes - timestamp (rounded down to the hour) 3 Bytes tag ID 3 Bytes tag value ID Total: 7 Bytes + 6 Bytes * Number of tags

33 Let s look at some graphs

34 Our experiences

35 What works well We store about 200M data points in several thousand time series with no issues tcollector is decoupling measurement from storage Creating new metrics is really easy

36 Challenges The UI is seriously lacking no annotation support out of the box Only 1s time resolution (and only 1 value/s/time series)

37 salvation is coming OpenTSDB 2 is around the corner millisecond precision annotations and meta data improved API

38 Friendly advice Pick a naming scheme and stick to it Use tags wisely (not more than 6 or 7 tags per data point) Use tcollector wait for opentsdb 2 ;-)

39 Questions? Please contact I ll upload the slides and tweet about it