MaDDash: Monitoring and Debugging Dashboard

Transcription

1 MaDDash: Monitoring and Debugging Dashboard john hess, CENIC PRP FIONA Workshop February 5-6, 2018 CENIC, La Mirada

2 MaDDash - Monitoring and Debugging Dashboard Orchestrates regular testing among several hosts (meshconfig) Visualizes test results original use-case: displaying measurements from perfsonar throughput (iperf) and latency / loss (owamp) python script for parsing GridFTP transfer logs to register disk-to-disk throughput can be extended to display other s of two-dimensional data includes community-contributed traceroute viewer (Dale Carder / U. Wisconsin) Gathers test results from Measurement Archive (MAs) MAs co-resident on ps toolkit hosts central, standalone MAs Presents high-level dashboard view of grids (meshes) of hosts (full-mesh, disjoint, ) Clicking on cells within grids leads to time-series graphs, host information, traceroute viewer Nagios checks may be configured to alarm based on thresholds for throughput, latency, packet loss,. 2

3 MaDDash - why regular testing is important Raises awareness of issues (read: symptoms) which may not have [yet, ever] been reported Provides a baseline performance reference and timeline from which to correlate changes Complements ad hoc testing 3

4 perfsonar MaDDash: Throughput and Packet Loss grids (disjoint) 4

5 MaDDash - dashboard grids, cells, check decoder Upper half of cell: reflects tests from Row node toward Column node Lower half of cell: reflects tests from Column node toward Row node image source: 5

6 perfsonar MaDDash: Throughput and Packet Loss time-series graphs 6

7 perfsonar MaDDash: Host Info and Traceroute Viewer 7

8 PRP K8s MaDDash and Traceroute visualization tools 8

9 MaDDash: develop a test plan Selection criteria for which hosts will participate campus border Science DMZ (topology close to DTN) regional How will the participating hosts be organized into grids host NIC uplink capacity separate grids for IPv4 vs IPv6 What is going to be measured throughput packet loss / latency traceroute Measurement frequency, duration, other parameters to consider throughput (10G: 4x / 30 seconds; 100G: 2x / 30 seconds) packet loss / latency traceroute ( 4x / day) other knobs - IPv4-only or IPv6-only; TCP Window Size; force bi-directional Measurement Archive (MA) distributed MAs: each ps toolkit host may (optionally) have an MA central MA: typically on same host / VM / container running MaDDash server 9

10 MaDDash: installation & configuration (during lab) Recipes for rpm bundle installation on CentOS: perfsonar-centralmanagement bundle will install a standalone MaDDash + esmond Measurement Archive (MA) Create and publish a mesh configuration file (json) consumed by MaDDash server for setting up dashboards of groups of nodes consumed by participating ps nodes for initiating regular tests Setup MA for receiving results from ps nodes and DTNs Configure aspects of the MaDDash webui Configure MaDDash server to display results from ps nodes and DTNs Set up ps nodes and DTNs to initiate tests and register results to the MA 10

11 MaDDash: packages maddash maddash-server The backend server that schedules checks and makes results available via a REST/JSON interface running on an embedded web server. This package has a dependency on java which will also be installed during the yum installation process. maddash-webui Container package that has dependencies on the maddash-server, maddash-webui, and perl-perfsonar_ps-nagios packages. The package itself does not install any additional software, it simply pulls in the aforementioned packages. The web pages that display the dashboard. It consists of a set of CGI scripts that run under Apache. The server contacts the REST server run by the maddash-server package and then presents the data on the web page. nagios-plugins-perfsonar Installs the perfsonar Nagios checks that can alarm based on throughput, loss and other data returned by perfsonar services. perfsonar-graphs Provides the performance graphs used by the maddash-webui package for perfsonar checks. source: 11

12 MaDDash: configuration files dashboard.conf has global defaults and four main sections organization, site, host test specifications (test_spec) groups (organizes hosts into groups: mesh, disjoint) tests (pulls together a test_spec and applies it to a group) mesh.json produced by tools processing dashboard.conf consumed by MaDDash GUI agent tools to create server YAML configuration consumed by participating ps nodes to initiate tests /etc/maddash/maddash-webui/config.json settings for dashboard title default dashboard color scheme for thresholds /etc/perfsonar/meshconfig-guiagent.conf settings for check interval, time range performance threshold values for acceptable, warning, critical /etc/maddash/maddash-server/maddash.yaml generated from mesh.json depending on use-case may require manual configuration (e.g. GridFTP) 12

13 CENIC-dashboard.conf -- global defaults Mesh Config file for CENIC top level / global defaults description CENIC <administrator> name John Hess jhess@cenic.org </administrator> <measurement_archive> perfsonarbuoy/bwctl read_url write_url </measurement_archive> <measurement_archive> traceroute read_url write_url </measurement_archive> <measurement_archive> perfsonarbuoy/owamp read_url write_url </measurement_archive> 13

14 CENIC-dashboard.conf -- organization, site, host SDSC <organization> description SDSC <site> description SDSC <host> description ps10g.sdsc.edu address ps10g.sdsc.edu address nate.sdsc.edu address address 2001:48d0:100:1::22 </host> </site> </organization> 14

15 CENIC-dashboard.conf -- host-specific MAs <host> description ps-40g-scidmz-0.tools.ucla.net address ps-40g-scidmz-0.tools.ucla.net address 2607:f010:3f9:8004::ea <measurement_archive> perfsonarbuoy/bwctl read_url write_url </measurement_archive> <measurement_archive> traceroute read_url write_url </measurement_archive> <measurement_archive> perfsonarbuoy/owamp read_url write_url </measurement_archive> </host> 15

16 CENIC-dashboard.conf -- no_agent hosts <host> description speedtest2.pnl.gov address speedtest2.pnl.gov no_agent 1 <measurement_archive> perfsonarbuoy/bwctl read_url write_url </measurement_archive> <measurement_archive> traceroute read_url write_url </measurement_archive> <measurement_archive> perfsonarbuoy/owamp read_url write_url </measurement_archive> </host> 16

17 CENIC-dashboard.conf -- test specifications (test_spec) <test_spec bwctl_8h_tcp_test_v6> Define a test spec for testing achievable bandwidth once every 4 hours perfsonarbuoy/bwctl Perform a bwctl test (i.e. achievable bandwidth) tool bwctl/iperf3 Use 'iperf' to do the bandwidh test protocol tcp Run a TCP bandwidth test interval (21600) Run the test every 6 hours ipv6_only 1 force ipv6 only duration 30 Perform a 20 second test force_bidirectional 1 do bidirectional test random_start_percentage 25 randomize start time omit_interval 5 ignore first few seconds of test window_size set 128MB TCP window </test_spec> <test_spec bwctl_8h_tcp_test> Define a test spec for testing achievable bandwidth once every 4 hours perfsonarbuoy/bwctl Perform a bwctl test (i.e. achievable bandwidth) tool bwctl/iperf3 Use 'iperf' to do the bandwidh test protocol tcp Run a TCP bandwidth test interval (21600) Run the test every 6 hours ipv4_only 1 force ipv4 only duration 30 Perform a 20 second test force_bidirectional 1 do bidirectional test random_start_percentage 25 randomize start time omit_interval 5 ignore first few seconds of test window_size set 128MB TCP window </test_spec> 17

18 CENIC-dashboard.conf -- groups: mesh and disjoint <group cenic_bwctl_100g_v4> mesh </group> perfsonar.nersc.gov fiona-ps.ucsc.edu ps-100g-hpr01.stanford.edu ps-100g-scidmz-0.tools.ucla.net ps-100g.sdsu.edu <group cenic_bwctl_40g_v6> mesh </group> bost-pt2-v6.es.net fiona-ps.net.berkeley.edu ps-40g-hpr01-v6.stanford.edu ps-antl-meter-40g-v6.nren.nasa.gov fiona.ucsc.edu ps-40g-scidmz-0.tools.ucla.net perf-main-40.ucr.edu ps-40g-prism.calit2.optiputer.net ps-40g-v6.sdsu.edu <group cenic_disjoint_throughput_10g_v4> disjoint a_ a_ </group> ps-svl-10g.cenic.net ps-lax-10g.cenic.net perf-scidmz-data.cac.washington.edu cc-bonsai-perfsonar.bonsai.uoregon.edu melange.noc.ucdavis.edu ps-border1-pt.lbl.gov ucsf-perfsonar1.ucsf.edu ps-arc-meter.nren.nasa.gov ps-10g-hpr01.stanford.edu dps10.ucsc.edu ucm-perfsonar00.ucmerced.edu ps-prp-10g.noc.ucsb.edu perfsonar.csusb.edu perfsonar.ultralight.org perfsonar.caltech.edu hpc-perfsonar.usc.edu ps-bw.ln.net ps10g-asm2.tools.ucla.net perf-main.ucr.edu fiona-ps.lp.ucinet.uci.edu speedtest.ucsd.edu ps10g.sdsc.edu ps-10g-prism.calit2.optiputer.net perfsonar.sdsu.edu 18

19 CENIC-dashboard.conf -- tests Tests <test> description IPv4 Throughput 1G-connected, Disjoint group cenic_disjoint_1g_v4 test_spec bwctl_1h_tcp_1g_v4 </test> <test> description group test_spec </test> <test> description group test_spec </test> <test> description group test_spec </test> IPv4 Packet Loss 1G-connected, Disjoint cenic_disjoint_1g_v4 owamp_test IPv4 Traceroute 1G-connected, Disjoint cenic_disjoint_1g_v4 traceroute_test IPv6 Throughput 1G-connected, Disjoint cenic_disjoint_1g_v6 bwctl_1h_tcp_1g_v6 19

20 ps-dashboard.cenic.net:/etc/maddash/maddash-webui/config.json } } "title":"cenic perfsonar Dashboard", "defaultdashboard": "CENIC 10G-connected", "enableadminui": true, "colors": { 0: "green", 1: "yellow", 2: "red", 3: "gray", 4: "black", 5: "orange" } } 20

21 ps-dashboard.cenic.net:/etc/perfsonar/meshconfig-guiagent.conf Use 'mesh' blocks to to specify each mesh that the agent should configure a display for <mesh> Use 'configuration_url' to specify the URL where the agent should obtain the mesh configuration from configuration_url To ensure that the configuration is trusted, you can set the 'validate_certificate' option to 1. This will validate that the certificate is valid, and matches the hostname. If the 'validate_certificate' option is set to 1, the 'ca_certificate_file' option must be set. validate_certificate 0 The 'ca_certificate_file' specifies which CAs to use to validate the certificates. ca_certificate_file /etc/pki/tls/bundle.crt the default RedHat CAs </mesh> You can define more meshes to configure against by adding more 'mesh' blocks. <mesh> configuration_url validate_certificate 0 ca_certificate_file /etc/pki/tls/bundle.crt </mesh> 21

22 ps-dashboard.cenic.net:/etc/perfsonar/meshconfig-guiagent.conf (continued) The default maddash test configurations. If the values aren't specified in the mesh, these will be used. <maddash_options> <perfsonarbuoy/owamp> check_command /usr/lib/nagios/plugins/check_owdelay.pl check_interval 1800 check_time_range 900 acceptable_loss_rate 0 critical_loss_rate 0.01 retry_attempts 1 </owamp> <perfsonarbuoy/bwctl> check_command /usr/lib/nagios/plugins/check_throughput.pl check_interval check_time_range acceptable_throughput 900 critical_throughput 500 </bwctl> </maddash_options> 22

23 ps-dashboard.cenic.net:/etc/perfsonar/meshconfig-guiagent.conf (continued) The default maddash test configurations. If the values aren't specified in the mesh, these will be used. <maddash_options> <perfsonarbuoy/owamp> check_command /usr/lib/nagios/plugins/check_owdelay.pl check_interval 1800 check_time_range 900 acceptable_loss_rate 0 critical_loss_rate 0.01 retry_attempts 1 </owamp> <perfsonarbuoy/bwctl> check_command /usr/lib64/nagios/plugins/check_throughput.pl check_interval check_time_range acceptable_throughput 7500 critical_throughput 5000 </bwctl> <perfsonarbuoy/bwctl> grid_name CENIC - IPv4 Throughput 100G-connected grid_name CENIC - IPv6 Throughput 100G-connected check_command /usr/lib64/nagios/plugins/check_throughput.pl check_interval check_time_range acceptable_throughput critical_throughput </bwctl> </maddash_options> 23

24 MaDDash: Measurement Archives esmond MA included in perfsonar-core and perfsonar-toolkit installation bundles packaged with MaDDash as a independent bundle, perfsonar-centralmanagement, which also includes the tools to produce the mesh configuration (json) 24

25 MaDDash: registering perfsonar results (during lab) Recipe: Measurement Archive (MA) generate credentials (username, and IP addresses or API key) for participating ps nodes MaDDash Server create dashboard / grids of participating ps nodes set check frequency, threshold values ps node configuration update /etc/perfsonar/meshconfig-agent.conf to include new mesh Verification ps node: tests added to /etc/perfsonar/meshconfig-agent.tasks ps node: tests are are scheduled pscheduler monitor ps node & MA: results registered MaDDash: checks are finding results and reflecting correctly on dashboard 25

26 MaDDash: registering GridFTP results (during lab) Recipe: DTNs - software requirements GridFTP server v6.0 or newer - python 2.7 esmond-client Python package (includes esmond-ps-load-gridftp script) Measurement Archive (MA) generate credentials (username and API key) to use when script on DTNs post results MaDDash Server create dashboard / grids of participating DTNs set check frequency, threshold values if the same GridFTP endpoints are also registering ps (event- throughput ) results to the same MA as GridFTP transfer results, update GridFTP-related checks in maddash.yaml to add the tool gridftp option to the corresponding Nagios command entries DTN configuration create cron jobs: to schedule transfers; and, parse GridFTP transfer log and upload results 26

27 GridFTP transfer data & metadata registered to esmond MA Esmond breaks information into metadata and data as described in perfsonar Client REST Interface. The metadata describes the parameters of the GridFTP transfer. This includes the following (metadata field names in parentheses): The source IP address (source) The destination IP address (destination) The fact that the tool used was gridftp (tool-name) The number of parallel streams (bw-parallel-streams) The TCP window size if set (tcp-window-size) If file striping is used, the number of stripes (bw-stripes) The GridFTP program used such as globus-gridftp-server(gridftp-program) The block size used by GridFTP in the transfer(gridftp-block-size) If you give the log scraper the -F option, the name of the file transferred (gridftp-file) If you give the log scraper the -N option, the name of the user that made the transfer (gridftp-user) If you give the log scraper the -V option, the name of the volume used in the transfer (gridftp-volume) source: 27