The Software Journey: from networks to visualization

Transcription

1 Discovery, unconstrained by geography. The Software Journey: from networks to visualization Inder Monga Executive Director, ESnet Division Director, Scientific Networking Lawrence Berkeley National Laboratory NORDUnet conference September 20 th, 2016

2 An old debate SMART SMART Dumb Network? Smarter Network? 2

3 Is the network just a set of dumb pipes? Photo courtesy: Rod Wilson, Ciena 3

4 The age of Smart! 4

5 ESnet is a dedicated mission network engineered to accelerate a broad range of science outcomes. How to make the network smart? 5 imonga at es dot net

6 Software helps create intelligent Application to Network conversation Applications Science and Site Driven Network Software Opportunity! Network Hardware (and embedded sw) Not in direct control Partner, Specify, Identify and Select Research Prototyping and Testbed Production

7 Research and Software Journey 7

8 ESnet Research and Software Development Portfolio Research Prototyping Testbed Funded External Research Next- Generation Architectures Production Network Software Tools Development

9 ESnet Research and Software Development Portfolio Funded External Research Next- Generation Architectures Network Software Tools Development

10 perfsonar perfsonar can trace its origin to the community Internet2 group named End 2 End performance Initiative from the year After 16 years, re-invigorated collaboration with Internet2, Indiana University and GEANT is resulting in consistent and operational software package perfsonar 4.0 release full of great control and stability mechanisms, new GUI Full release planned for Fall public perfsonar hosts in 30 countries as adoption continues to rise Training videos released: New perfsonar.net website and user-manual both written from scratch 10

11 perfsonar timeline 2000: Formation of OGF group 2005: nonstandard GE performanc e testers 2007:perfSONAR schema published : Deployment at all 10G ESnet sites. perfsonar release every 6 months 2016: nodes, worldwide 2003: First OGF doc on network measurements 2006: 1 st release of software. perfsonar system 10GE b/w and GE OWAMP 2008: 1 st release of perfsonar- PS. 20 nodes deployed : Reenergized partnershi p with I2, IU, GEANT 11

12 Who is running perfsonar? Currently over 1700 deployments world-wide 12 ESnet Advanced Network Technologies -

13 OSCARS OSCARS contributions to networking technology and science continue to get recognized 2014 Secretary s Honor Award from US Department of Energy 2013 R&D 100 awards Current focus of OSCARS software development is to support LHCONE s multi-domain virtual circuit experimentation by implementing NSI building community consensus and helping with NSI deployments/ interoperability testing. Rewrite of current software to simplify and take advantage of many open-source advancements 13

14 OSCARS Historical Timeline 2003: Flow analysis and identification of large flows 2007: OSCARS production service due to LHC 2008: GLIF interops multiple implementations using FENIUS (ESnet driven) 2004: DOE funded research project 2008: OSCARS 0.5, SOAP API and Web services, IDC 1.0 implementation 2009: ARCHStone research project funded 2011: OSCARS 0.6 released. Standards based NSI protocol, successful interop with multiple independent implementations : Decrease in OSCARS development effort and team size 2016: OSCARS development restarted 2012: OSCARS used to manage Supercomputing floor network 2014: Joint opensource development with SURFnet on NSI Aggregator 14

15 iperf3 Simple tool to let you measure memory-to-memory performance access a network New implementation from scratch, with the goal of a smaller, simpler code base A library version of the functionality that can be used in other programs Maintained actively by ESnet, and helping answer community questions Good uptake by the community Docker build for iperf3 15

16 Lesson 1: Community open-source sometimes needs long-term care and feeding to realize the full potential (with caveats) 16

17 ESnet s SDN Investigations: Innovating with research software OSCARS deployed on ESnet production backbone DOE Secretarial Honor Award 2015 R&D100 Award 2013 RDMA over Ethernet using OpenFlow (in collaboration with NEC) Zero Configuration Circuits Network Virtualization One Switch (in collaboration with Ciena) Transport SDN (in collaboration with Infinera) Layer 3 Software Defined exchange (in collaboration with REANNZ, Corsa, and Google Research) Multi-Layer Transport SDN (in collaboration with Infinera and Brocade) Scalable BGP white-box router (in collaboration with Corsa) DANTE, Internet2, CANARIE, ESnet (DICE) Control Plane Collaborations Dynamic Layer 2-3 SDN router using NSI-CS protocol (in collaboration with WIX, GEANT, and SANRen) White-box router QoS testing (in collaboration with Corsa) ESnet Network Operating System (ENOS)

18 Isolated apps are less flexible than platforms. Single-purpose apps: Programmable platform: 18

19 An architecture we foresee: ESnet as platform for concurrent, domain-specific network apps. Requires network operating system for science. early-stage project multiple challenges in creating flexible, stable app execution environment We envision apps that will express high-level intentions: create and manage virtual networks enable programmatic resource allocation optimize link utilization Future apps could support: NDN for climate; data management for CMS, ATLAS, Belle-II; security overlay for KBase; replication for ESGF; detector / HPC coupling for light sources workflows we haven t imagined 19

20

21 SDN Software 21

22 Lesson 2: Software helps with rapid experimentation and demonstration of value 22

23 ESnet Research and Software Development Portfolio Funded External Research Next- Generation Architectures Network Software Tools Development

24 How do we make looking at networks intuitive to nonnetworking folks like scientists? MyESnet Portal: A data-driven website for displaying information and visualizations about ESnet, its sites and community. 24

25 Open Source Good news everyone, it s all open source! Motivation Develop a common toolkit for visualizing networks Lower barrier to entry Allow people to use what we ve built so they don t have to build their own Encourage others to share their visualizations Give back to the community 25

26 Time Series Charts 26

27 Time Series Charts 27

28 Network Diagrams 28

29 Network Diagrams 29

30 Pond: Time Series Operations We have lots of time series data: SNMP, netflow, log messages, etc Need a common abstraction and wire format for this data Provides common operations Animation showing how Pond generates summary rollups 30

31 Lesson 3: Leverage community open-source but respond by giving back to the community 31

32 esmond - ESnet Monitoring Daemon System for collecting, storing, visualizing and analyzing large sets of time-series data RESTful API which allows easy access to the data which is collected perfsonar project has begun using esmond to store time-series of network measurements Documentation Software 32

33 Lesson 4: Software needs ongoing care and feeding (aka. software resources), so close down projects for equivalent open-source or commercial products 33

34 Visualization of data is good, but not enough 34

35 Network Analytics Provide a unified way for analyzing network measurements. There are several ad-hoc approaches and each new analysis reinvents the wheel. Analysis of both in real time and of historical data Focus on the what, rather than the how. Provide a way to run code over some slice of data without having to fuss with the details of how to read all the data. Initial focus on SNMP data with flow following soon behind Use cases: Monthly traffic volume Capacity planning Statistical analysis Real time traffic views on MyESnet Traffic matrices Topology building Sector analysis (eg. R&E vs Commercial) Custom studies Machine learning Security analysis Uses Google Cloud DataFlow for scalability and minimizing the need to build internal infrastructure. Open source available as Apache Beam. 35

36 Network Analytics Architecture Monthly, yearly calculations Output Dashboard SNMP Router Data Minute-wise calculations 36

37 Lesson 5: Leverage rapid technology advancement: SaaS and Cloud 37

38 Software journey usually never ends 38

39 Though it is never as smooth and straightforward 39

40 We are still learning. 40

41 Future software efforts need to target Applications-Network interaction to enable automation, orchestration, optimization High-level network abstractions for applications-network to have an intent based conversation missing No abstractions for networks to feedback service state to the application App Workflow App Not possible as a network request today: I would like to retrieve a PB of climate data by tomorrow morning from these five ESGF sites failed congested delayed Network Network

42 Question worth exploring: How do the NRENs collaborate better on software? 42

43 Thank you! 43

44 Software Journey Strategy EL meeting, innovation software the missing layer, but what do we do Alignment with ESnet strategy History, investment Our investments Network: OSCARS (another rewrite), perfsonar, Visualization, tools Internal software ESDB Upcoming GNA, ENOS and SENSE Lessons Learnt, things to think about Technical debt, technology refresh, production software, service vs software, community software is hard open-source, 44