Elasticsearch & ATLAS Data Management. European Organization for Nuclear Research (CERN)

Transcription

1 Elasticsearch & ATAS Data Management European Organization for Nuclear Research (CERN)

2 ATAS Analytics Platform proposed eb. 2015; work in progress; correlate data from various system (data mining, machine learning) request for Elasticsearch cluster CERN already filed used among others by workflow management (PanDA) transfer management (TS) data management (Rucio) Elasticsearch & ATAS Data Management 2

3 MWT2 C-Analytics - ATAS Cluster Setup 6 bare metal nodes and 6 VMs 1 head node 11 data nodes (VMs added only recently) Elastic version ~ 5 to 6 KHz (460m per day) ~ 3 billion documents ~ 3 tb data over ~ 8.5k shards bare metal node resources 256 gb RAM (lots of cache) 4x1 tb hard drives (spinning) VM resources 48 gb RAM 2 tb disk (spinning) Elasticsearch & ATAS Data Management 3

4 ATAS Workload ~ 200k jobs in parallel data stored 650 million file copies 230 storage systems 190 pb data many software projects ~ 5 million lines of code 200+ developers around the globe Elasticsearch & ATAS Data Management 4

5 Rucio Deployment 63 puppet-managed virtual hosts 4 load balancer 2 authenticator 17 backend server (API) 26 daemon nodes 1 Redis node 1 ogstash node 12 miscellaneous (build-bot, redirector, analytics, nagios, ) totals to 251 cores, 502 gb RAM, 5 tb storage Elasticsearch & ATAS Data Management 5

6 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons B ogstash R edis E lasticsearch ogstash K ibana Backends Redis Elasticsearch & ATAS Data Management 6

7 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons B ogstash (primary copy) ~ 120 files per day ~ 30 gb raw log data Backends Redis Elasticsearch & ATAS Data Management 7

8 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons B ogstash used for asynchronous jobs e.g. providing static data dumps long-term storage Backends Redis Elasticsearch & ATAS Data Management 8

9 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons B Redis Elasticsearch () for realtime data monitoring Backends ogstash total index rate ~ 1.2 KHz Elasticsearch & ATAS Data Management 9

10 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons Redis ogstash luentd as shipper B Backends record transformer plugin to truncate field (32k limit) Elasticsearch & ATAS Data Management 10

11 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons B ogstash 3 indices m docs/day debug - 5 days ret!debug - 30 days ret api - 30 days ret Backends Redis Elasticsearch & ATAS Data Management 11

12 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons Redis ogstash 2 ogstash process B Backends each 5 to 10 workers per queue => up to 35 mbit/s Elasticsearch & ATAS Data Management 12

13 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons Redis ogstash templates are strict B Backends => data is uniformed in ogstash (GROK) Elasticsearch & ATAS Data Management 13

14 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons Redis ogstash one day per index B Backends => drop expired indices Elasticsearch & ATAS Data Management 14

15 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons B Redis analyzer almost always disabled Backends ogstash => many UUIDs in the logs Elasticsearch & ATAS Data Management 15

16 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons B ogstash 1. mitigate performance bottlenecks, e.g. at midnight several scripts are executed Backends Redis Elasticsearch & ATAS Data Management 16

17 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons B ogstash 2. no log persisting as risk of uncontrolled load is unacceptable (eg DDoS) Backends Redis Elasticsearch & ATAS Data Management 17

18 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons Redis ogstash buffer capacity ~ 18 hours B Backends v-host: 8gb ram/32gb swap = 50% disk space Elasticsearch & ATAS Data Management 18

19 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons B Redis queues monitored by Graphite Backends ogstash alarms are raised if to big Elasticsearch & ATAS Data Management 19

20 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons B ogstash Backends Redis Elasticsearch & ATAS Data Management 20

21 Rucio Centralized ogging - R-EK Stack Misc Auth User (various clients) Daemons B Redis if buffer is full, lunet caches also locally Backends ogstash => total ~ 48 hours capacity Elasticsearch & ATAS Data Management 21

22 ate top st r ue req Rucio API Usage rs use to tal s Elasticsearch & ATAS Data Management 22

23 scrip er us er ro rc od es me t na Rucio API Errors nt u co ac Elasticsearch & ATAS Data Management 23

24 Rucio ile Tracking operations need to know what happened with a file/dataset over time e.g. added to a dataset queued for transfer deleted from storage Track a file through the system Elasticsearch & ATAS Data Management 24

25 Rucio ile Tracking Elasticsearch & ATAS Data Management ch lin ing e lo at r e p s e e h m c i t ma over t n o m dae m ase t/fil nam e e g dat 25

26 Rucio ile Tracking sev erit y d n o m ae tim e Elasticsearch & ATAS Data Management 26

27 Many more Kibana dashboards Elasticsearch & ATAS Data Management 27

28 Rucio Daemon Activity Report sent daily via mail grouped by daemon and severity level delta to previous day HTM formatted for convenience (ie links) as irames are blocked by most mail clients Elasticsearch & ATAS Data Management 28

29 dae Rucio Daemon Activity Report m on sts ho n o i t c u d o r p e pr hosts Elasticsearch & ATAS Data Management 29

30 as a machine learning data source Metrics collection Where should we place data? Which sites are well-connected? Which data should we delete?... Batch Transfers Ad-hoc Transfers Data Transfers Data merging atency Workload Management Classification Clustering Prediction Data Placement Packet oss Data Popularity... Metrics from various sources are written to, and usually retrieved as aggregated time series Merged with additional metadata from Oracle and Machine learning output used by various ATAS Distributed Computing systems Metrics feedback loop! Elasticsearch & ATAS Data Management 30

31 essons earned Redis buffer mitigates network incidents i.e backlog on nodes less critical disabled analyzer for log files (plenty of UUIDs) - use ogstash (grok) for data parsing/manipulation IOPS are critical - failed to get sufficient performance on Cinder block storages Elasticsearch & ATAS Data Management 31

32 Conclusion Kibana is convenient but lacks support for basic arithmetic operations operational dependencies on Elasticsearch significantly increase reliability requirements => more experience is needed use cases and infrastructure need to be geared to each other => agile infrastructure may be a solution steep learning curve but provides great insights into your log data Elasticsearch & ATAS Data Management 32

33 Questions? Among others, special thanks to Ilija Vukotic and incoln Bryant for running the MWT2 cluster.