Kibana, Grafana and Zeppelin on Monitoring data

Transcription

1 Kibana, Grafana and Zeppelin on Monitoring data Internal group presentaion Ildar Nurgaliev OpenLab Summer student

2 Presentation structure About IT-CM-MM Section and myself Visualisation with Kibana 4 and Grafana Motivation and comparison Reporting and Plotting with Zeppelin Motivation and comparison Improvement by using Spark 11/08/2016 Ildar Nurgaliev CERN openlab 2

3 Experience Education Ildar Nurgaliev Start End University name Degree Faculty KFU - High School ISIT Bachelor Applied Informatics Innopolis university Bachelor Artificial Intelligence 2015 now Innopolis university Master Data Science Solution Developer at Fujitsu GDC (Global Delivery Center) - enterprise application desktop application thin web client (Ext-JS, Java, Spring-framework, Hibernate, Swing) Researcher - Cognitive Architecture NEUCOGAR. - Graph matching and percolation theory for huge graphs. (C++, math algorithms, Random graph models) 11/08/2016 Ildar Nurgaliev CERN openlab 3

4 Since /08/2016 Ildar Nurgaliev CERN openlab 4

5 IT-CM-MM Section (Monitoring) Evolution of Monitoring tools for the CERN Tier-0 & WLCG Monitoring of DC at CERN and Wigner meter, timber Experiment Dashboards data transfers, job information, WLCG reports 11/08/2016 Ildar Nurgaliev CERN openlab 5

6 Visualisation Comparison Upcoming upgrade of DC Monitoring (meter, timber) from Kibana 3 Data Centre Overview dashboard Kibana4 Grafana Investigation on Grafana Porting Host Metrics Porting FTS dashboard ACL and SSO Comparison of Kibana vs. Grafana (Table) 11/08/2016 Ildar Nurgaliev CERN openlab 6

7 Data Centre Overview Kibana 3 11/08/2016 Ildar Nurgaliev CERN openlab 7

8 Data Centre Overview Kibana 4 11/08/2016 Ildar Nurgaliev CERN openlab 8

9 Data Centre Overview Grafana 11/08/2016 Ildar Nurgaliev CERN openlab 9

10 HOST Metrics Kibana 3 11/08/2016 Ildar Nurgaliev CERN openlab 10

11 Kibana 3 Mem Util. Very similar plots HOST Metrics 11/08/2016 Ildar Nurgaliev CERN openlab 11

12 Grafana Templated repeat visualisation HOST Metrics 11/08/2016 Ildar Nurgaliev CERN openlab 12

13 Grafana HOST Metrics Auto visualisation generate: Mem Util 11/08/2016 Ildar Nurgaliev CERN openlab 13

14 FTS Monitoring Kibana 4 11/08/2016 Ildar Nurgaliev CERN openlab 14

15 FTS Monitoring Grafana Select VO = ATLAS;CMS 11/08/2016 Ildar Nurgaliev CERN openlab 15

16 Grafana Some endpoints selected FTS Monitoring 11/08/2016 Ildar Nurgaliev CERN openlab 16

17 FTS Monitoring Grafana Automated Menu (Grafana feature) 11/08/2016 Ildar Nurgaliev CERN openlab 17

18 Grafana Transfer Sites Dashboard FTS Monitoring 11/08/2016 Ildar Nurgaliev CERN openlab 18

19 Kibana 4 FTS Monitoring Ranking Country Dashboard (nominal axis) 11/08/2016 Ildar Nurgaliev CERN openlab 19

20 FTS Monitoring Grafana Ranking Country Dashboard (tables, pie, NO bar charts!) 11/08/2016 Ildar Nurgaliev CERN openlab 20

21 ACL in Grafana For testing purposes 3 organisations (groups) created: MONIT - default for newcomers (Viewer permission) [contains all the dashboards now] ATLAS CMS Tried to see whether users can access, read or write across those organisations Summary: Users could be assigned to different organizations Using this multiple subscription we could attach an LDAP as centralised ACL Grafana has good support for SSO SSO accounts automatically mapped Nested groups are not supported in Grafana 11/08/2016 Ildar Nurgaliev CERN openlab 21

22 Feature Aspect Search and exploring - Plots look refined without banners or edit button - Expandable visualizations - Lucene query for visualisation - By dashboard developer - Not very refined Comparison - basically it has 'data exploration' - show document structure - inbuild search highlights - Save search as object - By dashboard user Reusage objects - No visualisation reuse - Manual repeat search query Visualisation, Dashboards, Search are saved as objects (plugable into many dashboards) General plots All plots are Time-series based - Time series - Nominal axis - Basic Heat Map Export & Share Visualisation -> CSV, PNG, Render image url Visualisation -> CSV, JSON 11/08/2016 Ildar Nurgaliev CERN openlab 22

23 Comparison Feature Role-based access (RBA) - RBA/ACL supported by default as Organisations - no built-in RBA/ACL - Commercial plug-in Plotting derived fields Combined plots Plots from differe nt sources Yes Scripted fields Yes very flexible visualisation Overlapping plots Yes As many as you wish (ES, Graphite, Influx Db) Yes Scripted fields, TimeLion purposeful tool No No One ES source only Support for Templates Yes Automatic visualisations from limited set of values for a var, change datasource on the fly No 11/08/2016 Ildar Nurgaliev CERN openlab 23

24 Scrutiny Report Plots Scrutiny Report: Overview of usage of all WLCG resources (more than 50 pages) Motivation: Automate generation of plots for the ATLAS report with Zeppelin Datasets popularity plots Discovery of unused data Used to optimise improve data management policies (e.g. replicas and lifetime of data) 11/08/2016 Ildar Nurgaliev CERN openlab 24

25 Current Workflow - Input Data Cronjobs run PIG (hadoop) scripts that aggregate ATLAS datasets access events from Hadoop Summaries generated on a web server as CSV files CSV files manually downloaded and imported in Excel 11/08/2016 Ildar Nurgaliev CERN openlab 25

26 Current Workflow - Plots Cronjobs run PIG (hadoop) scripts that aggregate ATLAS datasets access events from Hadoop Summaries generated on a web server as CSV files CSV files manually downloaded and imported in Excel 11/08/2016 Ildar Nurgaliev CERN openlab 26

27 Why Zeppelin Web-based notebook that Enables interactive data analytics with powerful dynamic visualisations Supports several technologies out of the box (Python, SQL, Spark, Hadoop, etc) Provides interactive forms: (Text Input Forms, Select Forms, Checkbox forms, ) 11/08/2016 Ildar Nurgaliev CERN openlab 27

28 Datasets Popularity Plots Starting from the same CSV files Show the number of times ATLAS data were accessed to find datasets usage Volumes of data (V) vs. number of accesses (X) for last N months Number of times accessed in periods of 3, 6, 9, 12 months and infinity 11/08/2016 Ildar Nurgaliev CERN openlab 28

29 Sample Plots - View PyPlot for reportquality plots 0-access bins Old and new datasets Most accessed datasets Interactive built-in plots for discovery 11/08/2016 Ildar Nurgaliev CERN openlab 29

30 Developer s view Zeppelin Notebooks User s view 11/08/2016 Ildar Nurgaliev CERN openlab 30

31 Discovery of Unused Datasets Starting from the same CSV files Show Top-N unused data by project/datatype for last X months for every month it is clear which datasets/datatypes are unused (e.g. mc16_7tev) 11/08/2016 Ildar Nurgaliev CERN openlab 31

32 Unused Data by Type and Creation Time 11/08/2016 Ildar Nurgaliev CERN openlab 32

33 Aggregation with Spark - Ongoing Replace PIG scripts with Spark Improve execution speed Use Mesos/Chronos monitoring infrastructure for scheduling jobs 11/08/2016 Ildar Nurgaliev CERN openlab 33

34 Thank you CERN! List of studied technologies Elasticsearch Kibana dashboard Grafana dashboard Hadoop Apache PIG Apache Spark Zeppelin notebook 11/08/2016 Ildar Nurgaliev CERN openlab 34