perfsonar: The Road to 100k Nodes

Transcription

1 perfsonar: The Road to 100k Nodes Eric Boyd, Internet2, Luke Fowler, Indiana University, Brian Tierney, ESnet,

2 What is perfsonar? perfsonar is a tool to: Set (hopefully raise) network performance expectaions Find network problems ( soq failures ) Help fix these problems All in muli- domain environments These problems are all harder when muliple networks are involved perfsonar is provides a standard way to publish acive and passive monitoring data This data is interesing to network researchers as well as network operators 2

3 Current perfsonar components Measurement tools iperf3, bwctl, owamp, traceroute, etc. Measurement archive Central test mesh management tools Host management tools Configure tests, configure NTP, etc. Data analysis tools Plot data from the archive Dashboard tools Lookup Service October 15,

4 perfsonar History perfsonar can trace its origin to the Internet2 End 2 End performance IniIaIve from the year What has changed since 2000? The Good News: TCP is much less fragile; Cubic is the default CC alg, autotuning is and larger TCP buffers are everywhere Reliable parallel transfers via tools like Globus Online High- performance UDP- based commercial tools like Aspera more good news in latest Linux kernel, but it will take 3-4 years before this is widely deployed The Bad News: The wizard gap is sill large Under- buffered and switches and routers are sill common Under- powered/misconfigured firewalls are common SoQ failures sill go undetected for months User performance expectaions are sill too low 4

5 The perfsonar collaboraion The perfsonar collaboraion is a Open Source project lead by ESnet, Internet2, Indiana University, and GEANT. Each organizaion has commiced 1.5 FTE effort to the project Plus addiional help from many others in the community (OSG, RNP, SLAC, and more) The perfsonar Roadmap is influence by requests on the project issue tracker annual user surveys sent to everyone on the user list regular meeings with VO using perfsonar such as the WLCG and OSG discussions at various perfsonar related workshops Based on the above, every 6-12 months the perfsonar governance group meets to prioriize features based on: impact to the community level of effort required to implement and support availability of someone with the right skill set for the task 5

6 perfsonar Vision Grow perfsonar to be the ubiquitous, indispensible performance monitoring framework for R&E network customers that meets their needs in the long- term, supported in a self- sustaining way. October 15,

7 perfsonar Challenge Get to 100k nodes deployed worldwide An increase of 2 orders of magnitude Get to 100+ nodes per campus October 15,

8 perfsonar Challenge ImplicaIons Cut the price per node by an order of magnitude $4k (perfsonar 3.3) $2k (perfsonar 3.4) $100 (perfsonar 3.5) October 15,

9 perfsonar Hardware These days you can get a good 1U host capable of pushing 10Gbps TCP for around $500 (+10G NIC cost). See perfsonar user list And you can get a host capable of 1G for around $100! Intel Celeron- based (ARM is not fast enough) e.g.: hcp:// Item=N82E VMs are not recommended Tools work becer if can guarantee NIC isolaion 6/2/15 9

10 perfsonar Challenge ImplicaIons Scale the infrastructure to support 100k nodes Does the Lookup Service scale? Do the various UIs scale? Does the support model scale? Grow the number of operaing systems supported? Re- evaluate accessibility of nodes? October 15,

11 perfsonar Challenge ImplicaIons What is the total cost of ownership per node? O(n 2 )? O(n)? O(k)? How auto- configurable are the nodes? If you drop a node in a wiring closet, how much work is required to integrate it into a campus mesh? October 15,

12 perfsonar 3.5: What s new?

13 perfsonar 3.5 Themes Impact: Increase the number of nodes, reduce total cost of ownership, improve security Usability: Transformed user experience, reduced cost for NOC integraion Efficiency: Reduced development costs October 15,

14 perfsonar 3.5 ImplementaIon Modernize the GUIs Support for central host management and node auto- configuraion Support for Debian, VMs, and other installaion opions Support for low cost, 1Gbps nodes October 15,

15 Expanded perfsonar Use Cases Current Use Case perfsonar Toolkit Includes CentOS 6 and all perfsonar components New Use Cases perfsonar tools only Support for both RHEL- based and Debian- based hosts perfsonar hosts that are centrally managed perfsonar hosts that self- configure October 15,

16 Improved Support for Central Management Goals: Make it easy to incorporate perfsonar hosts into exising host management systems (puppet, chef, SaltStack, cfengine, etc.) Include sample puppet config files Make it easy to manage many perfsonar hosts at a single insituion New rpm and debian bundles to support this October 15,

17 perfsonar and Puppet Puppet is a configuraion management system that allows you to configure and maintain many nodes easily IniIal perfsonar puppet modules have been created Alpha/beta level Only CentOS6 supported We need your help to test/contribute improvements! Hoping to expand in perfsonar 3.6 October 15,

18 Puppet Support Features Node classes can be defined, i.e. endpoint or measurement archive Puppet will install soqware/dependencies, configure it, keep packages up- to- date Includes support for Internet2 repos, web100 kernel, Esmond, bwctl, owamp, rsyslog client or server, apache, meshconfig Most services have configurable port ranges Puppet also maintains iptables setngs for the configured services October 15,

19 New perfsonar InstallaIon opions In addiion to the tradiional Toolkit install, you now have the these addiional opions: perfsonar- Tools: iperf3, bwctl, owamp, nuccp, etc Install this on DTNs, etc to help with troubleshooing Does not support scheduled tesing CentOS and Debian support perfsonar- TestPoint: tools plus Lookup Service registraion and mesh agent For use in environments with a central measurement archive For use on low end/older hardware (e.g.: $100 nodes) Supports scheduled tesing CentOS and Debian support See: hcp://docs.perfsonar.net/install_opions.html October 15,

20 New perfsonar InstallaIon opions (cont.) perfsonar- Core: Includes everything except the web interface Use this in environments where your site sysadmins want to fully manage the host configuraion, but don t want to setup a central measurement archive CentOS only perfsonar- CentralManagement: Includes measurement archive, test mesh manager, dashboard Use this to manage a collecion of perfsonar hosts at your site/campus CentOS only October 15,

21 New perfsonar InstallaIon opions (cont.) perfsonar- Complete All perfsonar packages Use this environments where your sysadmins want to manage the install, but sill use the toolkit web interface, system setng, etc the toolkit install will override certain changes every update. CentOS only Other packages to note: Separate rpms/debs for iptables config, sysctl config, and ntp packages so you can add them on top of perfsonar- Core as desired. October 15,

22 perfsonar for Network Researchers AcIve measurement interesing for network researchers Traceroute data automaically collected along with bwctl/ owamp results Data easy to download for analysis esmond- ps- get- bulk Output CSV or JSON See: hcps://pypi.python.org/pypi/esmond_client AddiIonal InformaIon at: hcp://docs.perfsonar.net/client_apis.html October 15,

23 User Interface Refresh Easy to use web interface Targeted at users with variety of levels of technical experise. Our goals for personar 3.5: improve easy of use and visual appeal integraion of state of the art UI components to extend UI lifespan. Current Interface October 15,

24 New Host Management Interface 2014, hcp:// October 15,

25 perfsonar on Low Cost Hardware MoIvaIon: make perfsonar affordable enough to deploy on all subnets AssumpIons: 1Gbps test nodes Centralized measurement archive Centralized configuraion management Debian Linux October 15,

26 Auto- Updates Rock! October 15,

27 psui: the perfsonar web UI TechEX 2015

28 What is psui? Perfect tool for On demand measurements Accessing muliple archived measurements Access MP and MA from Global Lookup Service (~ 9000 services) Locally maintained list Calls bwctl through OPPD (calls 2 remote bwctl instances) Java applicaion running in tomcat Java 7 and Tomcat 7 CentOS and Debian packages To be deployed on separate host (VM is ok) Federated login with edugain credenials October 15,

29 2 3 1 October 15,

30 1 2 3 October 15,

31 Other resources Install your own psui for your NOC InstallaIon instrucions: hcp://docs.perfsonar.net/install_psui Usage instrucions: hcp://docs.perfsonar.net/using_psui Public demo psui instance (edugain enabled): hcps://psui.geant.net October 15,

32 Low Cost Node Project

33 perfsonar on Low Cost Hardware MoIvaIon: make perfsonar affordable enough to deploy on all subnets AssumpIons: 1Gbps test nodes Centralized measurement archive Centralized configuraion management Debian Linux? October 15,

34 Many Possible SoluIons Small, Low- cost Nodes Examples include the above; NUC, Gigabyte, Pi, cubox, August 10,

35 Started with ARM- based soluion Cheap ARM- based perfsonar nodes e.g.: Cubox ( perfcube ) $120 TCP max only 350Mbps owamp works well 35

36 Evaluated these Celeron- based devices ASUS CHROMEBOX- M004U ($160) GIGABYTE GB- BXBT ($115) Intel NUC Kit DCCP847DYE ($170) 10/15/15 36

37 LIVA by ECS $ USD Home: hcp:// Product/Product_LIVA.aspx? DetailID=1560&LanID=0#Overview hcp:// Product.aspx?Item=N82E $105 in Sept, , hcp:// 37

38 SpecificaIons 2015, hcp:// 38

39 Contents of box: Assembly Required Case top and bocom Motherboard with onboard memory, RAM, and 1GE network adapter Wireless adapter card Antennae for wireless card (1 long, black; 1 short, white) Power Supply (US) with adapter plate (EU included) 2015, hcp:// 39

40 Assembly Completed 2015, hcp:// 40

41 Strengths Complete package with limited assembly required 32Gig of flash onboard 2Gig of RAM True 1GE (tesing over clean paths M Only 15W power required POE PossibiliIes Small and lightweight ComparaIvely Low cost to other more robust hardware/server opions 2015, hcp:// 41

42 emmc Flash (Onboard) perfsonar s recommended CentOS 6 does not include driver support for the onboard storage Using instead as a recommended build Ubuntu Desktop Server doesn t have proper EMMC driver Debian (Ubuntu) perfsonar packages: hcp://docs.perfsonar.net/install_debian#configuraion 2015, hcp:// 42

43 Power Bucon / USB Issues (minor) Power bucon instability has been found on about 1 in 10 boxes. Easily overcome by muliple boot acempts or; Long- pressing the power bucon for a firm connecion or; Assume this is either a quirk of the manufacturing process and hope it is worked out on future models USB instability has been found on about 3 in 20 boxes; independent of the power bucon issue 2015, hcp:// 43

44 No fan in the box for cooling Passive Cooling No issues with this were found during tesing, but worth considering when placing in warmer spaces. 2015, hcp:// 44

45 TesIng Next GeneraIon Liva Liva X new run of hardware, expected to resolve minor issues. Nearly idenical hardware, very similar cost profile Arrives pre- assembled perfsonar project has acquired (arrived on Monday!), beginning tesing/validaion now. Stay tuned! hcp:// Item=N82E , hcp:// October 15,

46 More InformaIon hcps:// selecion/low- cost- hardware/ hcp://docs.perfsonar.net/low_cost_nodes.html hcp://github.com/perfsonar/project/wiki/perfsonar- Endpoint- Node- Project 2014, hcp:// October 15,

47 plans for perfsonar and 3.6

48 perfsonar Plans We are in the early stages of refining the perfsonar roadmap ExpecIng a minor release in the February Imeframe and a major release in late 2016 (Imeframe TBD) October 15,

49 perfsonar Community Planning a survey of the perfsonar user community in October, 2016 Planning to publish roadmap in November, 2016 October 15,

50 DRAFT: v3.5.1 (February 2016) New Test configuraion GUI New plots based on ESnet Tools team react- based plotng tools Node autoconfig minimize configuraion required to bring up a new centrally managed perfsonar node More automated tesing Bug fixes October 15,

51 DRAFT: v3.6 (Summer 2016) Theme for v3.6 will be Control and Scalability perfsonar is successful because of the default open model. BUT, as the number of perfsonar hosts worldwide grows, we need a way to control Who is running tests How oqen are they allowed to run tests October 15,

52 Roadmap for v3.6 (cont.) CentOS 7 / Debian 8 support Pre- packaged perfsonar VM images GUI to configure test meshes Maddash 2.0 See next slide October 15,

53 MaDDash 2.0 AlerIng Working in conjuncion with MaDAlert project to analyze dashboard pacerns and create useful alerts Scalability Dashboards keep growing and want to increase frequency and number of checks that can be run. Exploring alternaive back- ends and new modes for distribuing check execuion Usability Redesign of the web interface. Major goals include: Making dashboards easier to understand Making it easier to search for and find checks you care about

54 Test Node Auto- ConfiguraIon Goal: minimize configuraion required to bring up a new centrally managed perfsonar node Site will register in the lookup service: What hosts to run tests to Where to send the test results Eventual Goal is a zero config mode October 15,

55 Lists and Reference Materials

56 AcIve and Growing perfsonar Community AcIve lists and forums provide: Instant access to advice and experise from the community. Ability to share metrics, experience and findings with others to help debug issues on a global scale. Joining the community automaically increases the reach and power of perfsonar The more endpoints means exponenially more ways to test and discover issues, compare metrics 56

57 perfsonar Community The perfsonar collaboraion is working to build a strong user community to support the use and development of the soqware. perfsonar Mailing Lists Announcement Lists: hcps://mail.internet2.edu/wws/subrequest/perfsonar- announce Users List: hcps://mail.internet2.edu/wws/subrequest/perfsonar- users 57

58 Useful URLs hcp://docs.perfsonar.net/ hcps:// hcp://fasterdata.es.net/ hcp://fasterdata.es.net/performance- tesing/ network- troubleshooing- tools/ hcps://github.com/perfsonar hcps://github.com/perfsonar/project/wiki 58