perfsonar: A Look Ahead Andrew Lake, ESnet Mark Feit, Internet2 October 16, 2017

Transcription

1 perfsonar: A Look Ahead Andrew Lake, ESnet Mark Feit, Internet2 October 16, 2017

2 Overview Introduction perfsonar 4.0 Recap perfsonar (November 2017) perfsonar 4.1 (Early 2018)

3 What is perfsonar? perfsonar is a tool to: Set (hopefully raise) network performance expectations Find network problems ( soft failures ) Help fix these problems All in multi-domain environments Over 2000 public hosts on many different networks These problems are all harder when multiple networks are involved Focus on Research and Education (R&E) Networking, 1Gbps links or higher perfsonar provides a standard way to publish active and passive monitoring data This data is interesting to network researchers as well as network operators 3

4 perfsonar Bundles perfsonar-tools Just the measurement tools: iperf, iperf3, nuttcp, pscheduler client, owamp perfsonar-testpoint Tools + pscheduler, mesh-config, Lookup Service registration perfsonar-core testpoint + esmond (for storing results) perfsonar-toolkit Perfsonar-core + Web UI, scripts to apply tuning and security settings Available as a full suite of tools for CentOS and Debian Available as CentOS 7 ISO October 16, , 4

5 perfsonar 4.0 Released April 17, 2017 Major release with many new features: pscheduler All-new system for scheduling and supervising measurements Graphs Cleaner display of several data types MaDDash 2.0 Added alerting features. Operating Systems Support for CentOS 7 and Debian 8 75%+ of registered systems on 4.0 or newer

6 perfsonar Released August 15, 2017 Minor release with primarily bug fixes and small changes: Full Ubuntu 16 and Debian 9 support Ability to use custom pscheduler port (default is 443) Improved pscheduler limits Feedback continues to be positive and constructive Keep it coming!

7 Important Dates October 17, 2017 perfsonar 3.5 end-of-life Web100 end-of-life November 2017 perfsonar final release Currently in beta Q perfsonar 4.1 released. CentOS 6 packages will not be released for perfsonar 4.1 Q (6 months after 4.1 release) perfsonar 4.0 end of life CentOS 6 versions of perfsonar no longer supported * All dates subject to change

8 New in 4.0.2

9 pscheduler: Streaming Archiving Per-run execution model led to high system load on highvolume systems. Archiver plugins now operate in streaming mode: Plugin components started once and fed multiple results Pool size managed by pscheduler based on utilization Significant load reduction seen when deployed.

10 pscheduler: SNMP Plugins Developed by University of Michigan ITS Collect device data in parallel with other tests Send results to management system (prefers traps as input) Tests snmp-get snmp-getbgm snmp-set One-shot fetch of SNMP MIB values Long-term fetch and delta detection Set SNMP values ( extras bundle) Tools pysnmp, net-snmp-set Archiver snmptrap Generate traps from result data

11 pscheduler: jq Support Tool for in-line manipulation of JSON Many built-in functions and constructs for transformation, decision making and output. Adds end-user programmability in configuration and task specification instead of installed code. Info and docs:

12 pscheduler: jq Support Archiving Transform result JSON on the way to any archiver for systems that require custom input. Limit System Make pass/fail decisions on proposed tasks Transform downloaded JSON CIDR lists to plain-text format without depending on another system. Library Collection of standard functions useful with pscheduler. Look for a workshop in early December.

13 pscheduler: Contexts Contexts allow modification of the execution contexts of tools before they are run. Think chroot for other things. Same process and execution context all the way through Currently one plugin: Linux Network Namespace. Others can be added by writing plugins.

14 pscheduler: Other Changes Tools for troubleshooting limit configuration files Better searching in the REST API Improves performance of Meshconfig, future applications The usual raft of minor enhancements and fixes

15 What s Planned for 4.1

16 pscheduler: Resource Management Configurable ranges of TCP and UDP ports Assigned and used during testing Greatly simplifies on- and off-host firewall configurations

17 pscheduler: Job Control Current implementation requires that measurements run to completion. Stop runs midstream Preempt pending runs (Details TBD) Useful for getting CLI tasks to run sooner

18 pscheduler: Remotely-Supervised Tasks Remotely-supervised tasks with local run archiving Node A supervises a task on B, retrieves and archives results. Solves reachability problems for some use cases Node A Fire wall Node B Node C Archive

19 pscheduler: Task Rewriting Make on-the-fly changes to tasks at ingest to do things the limit system can t Similar to Apache s URL rewriter Part of the limit system Work will be done with jq transforms

20 TWAMP Two-Way Active Measurement Protocol (RFC 5357) Similar to OWAMP Does a combined one-way and RTT measurement RTT measurements more accurate in the face of clock variation twampd running to process measurement requests twping to do a measurement from CLI Streaming measurements with something similar to powstream Devices from several manufacturers include TWAMP support Many more non-perfsonar targets for latency measurements

21 pscheduler: TWAMP Will be integrated with pscheduler pscheduler-tool-twping pscheduler-tool-twpstream (or pscheduler-tool-twostream) Able to produce results for latency and RTT Will be shown in the regular perfsonar graphs as the other tools Work made possible thanks to prior work by Rob Shearman at Brocade and Duncan Eastoe.

22 pscheduler: More New Plugins Network traffic capture HTTP response time Others We need your ideas. What do you want to measure? We need your participation. Plugins can be written by third parties.

23 pscheduler: BWCTL Retirement BWCTL daemon will no longer be run Backward-compatibility code will be removed from pscheduler Check your firewalls for 443 Some sites only have BWCTL ports open Systems were falling back

24 Docker Support Getting perfsonar running in containers is a popular topic on user list Currently have beta Docker images of tools and testpoint bundles Evaluating best way to bundle these since perfsonar is many services and Docker in general wants you to run one service per container Starting with Docker due to popularity, may look at other container technologies in future (and others are more than welcome to do it now)

25 psconfig: Re-imagining MeshConfig Major feature for perfsonar 4.1 will be refactoring of MeshConfig into psconfig The introduction of MeshConfig helped shape a lot of today s deployments perfsonar has changed a lot the past few years and MeshConfig hasn t kept up in every area. Just as pscheduler was transformative at the test scheduling layer in 4.0, we hope psconfig will be equally as transformative for the test configuration layer in 4.1

26 How people use meshconfig today MeshConfiggenerally used for: Medium to large internal deployments (e.g. ESnet) Medium to large collaborations (e.g. WLCG) Made it easier to deploy tests on lots of hosts and visualize it which has been good but interesting side-effects Over-testing not uncommon making it hard to see patterns Test design is hard and probably impossible to completely solve but can hopefully help Clearer syntax so take advantage of things like disjoint topologies and autodiscovery features. Easier to write tools to auto-generate tests based on topological knowledge, etc

27 Current MeshConfig Agent October 16, , 27

28 Current MeshConfig GUIAgent October 16, , 28

29 Code changes when we add plugin or new options October 16, , 29

30 psconfig MeshConfig renamed to psconfig Single JSON configuration format No need to change any code when adding new test or archiver, will validate against pscheduler or maddash check profiles Terminology such as task, test and archiver to be made consistent with how pscheduler uses them CLI for building meshes to avoid syntax errors and provide real-time guidance on options. Three packages: psconfig-publisher Tools to help edit and publish task JSON files psconfig-pscheduler Daemon to create pscheduler tasks from JSON files psconfig-maddash Daemon to configure maddashbased on JSON files October 16, , 30

31 New plugin = No code changes! October 16, , 31

32 New plugin = No code changes! October 16, , 32

33 Backward Compatibility psconfig packages will convert any config files to new format If running old meshconfig agent psconfig-publisher package will be able to generate old json Can use rewrite rules and user-agent header to serve-up old version If running new psconfig agent Option 1: Understand just the new (could ditch a lot of old code sooner) Option 2: Detect format and parse each MeshConfig GUI Generate both and serve-up correct based on user-agent header October 16, , 33

34 Getting Ready for 4.1 Dropping support for the following OSes CentOS 6 Debian 7 You need to upgrade if you are on one of the OSes and you want 4.1 Keep in mind 4.0 support dropped 6 months after 4.1 release BWCTL support will also be going away, thus backward with 3.X compatibility goes away

35 Conclusion coming soon with some new features 4.1 coming next year with some bigger features Would love feedback on planned feature set Upgrade to CentOS 7, Debian 9 or Ubuntu 16 if you have not already since support goes away middle of 2018 We are looking at retransmit behavior, would love help if you have any CentOS 7 boxes in the wild Thank you for your time!

36 Lists and Reference Materials

37 Mailing Lists Announcement Lists: ar-announce Users List (developers also monitor): ar-users 37

38 Useful URLs