Embedded Network Systems. Internet2 Technology Exchange 2018 October, 2018 Eric Boyd Ed Colone

Transcription

1 Embedded Network Systems Internet2 Technology Exchange 2018 October, 2018 Eric Boyd Ed Colone

2 1. Background 2. Standards Principles Requirements 3. Emerging Technology a. Vision and End Goal b. Challenges 4. Manufacturing Collaboration 5. Advice for Next Generation Appliances 2

3 Background: Emerging Technology perfsonar: over 2,000 deployments worldwide (mostly Dell servers) Containers emerging as an offering on traditional network devices Vision: perfsonar embedded in major vendor hardware 3

4 Background: Problem of Scale The ability to benchmark and troubleshoot the last mile of large distributed networks. U-M current scope: Number of AL switches: 101 models, 3,825 devices Number of DL switches: 22 models, 239 devices 4

5 Background: Sample Soft Failure: Failing Optics Gb/s normal performance degrading performance repair one month Credit: Jason Zurawski, ESNet 5

6 Background: Challenge Legacy location issues: Space Power Climate control On-site support challenges: Too many locations (>1,000) Impossible to provide 24x7 operations Servers with management planes more expensive 6

7 Emerging Technology Vision Provision perfsonar on the network gear itself. Scalable Deploy Support Grow Fewer costs/ fewer machines Fewer dependencies Becomes a part of the in-building network replacement lifecycle 7

8 Standards, Principles, Requirements Embedded Solution vs. a stand-alone perfsonar node: Don t gouge: cost of the Embedded solution must not be greater than a discreet solution Don t suck: Network hardware must perform switching and routing duties unimpeded. Don t lie: performance measurements must be as accurate as discreet servers. 8

9 Embedded Platform Options Bare Metal port (Yocto / Wind River) High effort - Embedded Linux not a dev environment Unsupportable Poor performance Larger containers Better performance Standards for creation Some existing support (perfsonar on Docker) Start with base image/container (Docker, LXC, etc.) More complex provisioning process Security concerns may require perfsonar redesign / development KVM Docker, LXC Proprietary container process 9

10 Emerging Technology Architecture Current throughput constraints: Desired performance Max data port rate (10GB / 40GB / 100GB / etc.) MANAGEMENT PORT ROUTING ENGINE ~ 1GB OS Management port constraints CONTROL PLANE ~ 1 GB (often limited to copper interfaces) BUS < 1GB Bus performance Currently < 1 GB DATA PORTS PFE 1 / 10 / 40 / 100 GB DATA PLANE 10

11 Container Networking: Management Port Currently has most throughput performance (~1GBE) If Management port limited to copper, could be incompatible with the rest of the switch, bad for leaf nodes (can t plug a 1G copper port into a QSFP port) Least desirable from Net Admin perspective Precludes Out-Of-Band Management Cable required Outage risk from possible misidentification as a network loop Adds installation complexity, consumes additional interfaces CONTROL PLANE MANAGEMENT PORT ROUTING ENGINE OS BUS DATA PORTS PFE DATA PLANE 11

12 Container Networking: Single Data Port Limited throughput performance (<1GBE) Bus is shared with routing duties Unknown failure modes Cable required Outage risk from possible misidentification as a network loop Adds installation complexity CONTROL PLANE MANAGEMENT PORT ROUTING ENGINE OS BUS DATA PORTS PFE DATA PLANE 12

13 Container Networking: Any Port Any data port cableless Most desirable Still goes over internal bus Bus is shared with routing duties Unknown failure modes Ideally, ps container would be bridged to SVI zero touch deployment does not have to consume any ports on the front container can be deployed without additional cables CONTROL PLANE MANAGEMENT PORT ROUTING ENGINE OS BUS DATA PORTS PFE DATA PLANE 13

14 Obstacles: Net Admin Perspective Multiple proficiencies required to configure Complicated, not always well documented container creation Network Architecture Container requires public IP address + small subnet per building rebooting a switch to solve a container issue is problematic Switch First philosophy Security Architecture Non-network admins require administrative access to the switch How well is the container separated from control plane? What other security issues does this present? 14

15 Obstacles: Hardware Limitations High performance platforms, no budget network device solution yet Discreet solution currently has some performance advantages Historically switches didn t require a lot of storage: Storage quickly becomes an issue with the larger images required to run Typical flash might not have the space External USB storage a possibility Speed / bandwidth / performance concerns Shared bus = shared bandwidth 15

16 Obstacles: Vendor Support Readiness Early stage of vendor documentation Early and varied stages of collaboration by vendors Proof-Of-Concept (Company J) Pilot Stage (Company C) Alpha Stage (Company A) Container Networking Issues (Company A, J) High performance platforms, no budget solution yet 16

17 Current Lab Metrics ( Ideal Conditions - 1Gbps) Vendor Container Networking Latency Throughput A Docker Docker host Any Port ~1.3MS ~390 Mbps Management Port ~2.3MS ~943 Mbps Data Port ~1.3MS ~942 Mbps Management Port ~1.9MS ~943 Mbps C J Proprietary (Docker based) Proprietary (Docker or LXC based) 17

18 Company C Concurrent Throughput Testing Only tested in Management Port Config Tested Throughput from Dell to Dell while DC MACC TOR INTERNET Router was also Throughput testing Speeds / Throughput were not affected by either test, consistent results achieved. 18

19 U-M Vendor C Device perfsonar Active Does the Vendor C Device act like a good perfsonar node? U-M Vendor C Device & Dell Mbps Dell U - M Vendor C Device to Dell 330 ps node 19

20 perfsonar Active U-M Vendor C Device IU Vendor C Device Does the Vendor C Device act like a good perfsonar node? INTERNET U-M Vendor C Device & IU Vendor C Device Mbps Mbps U-M Vendor C Device to IU Vendor C Device 20

21 U-M Vendor C Device C Routing Does the Vendor C Device act like a good switch? Dell 330 & Dell 330 Dell Dell Mbps Mbps Dell 330 ps node to Dell 330 ps node 21

22 U-M Vendor C Device perfsonar Active and C Routing IU Vendor C Device Is ps working with switch while working? INTERNET Dell 330 & IU Vendor C Device Mbps Dell Dell 330 ps node to UI Vendor C Device 22

23 U-M Vendor C Device perfsonar Active and C Routing IU Vendor C Device Is ps working with switch while working? INTERNET Dell UM Vendor C Device & Dell 330 IU Vendor C Device & Dell Mbps Mbps Mbps Mbps Dell Dell 330 ps node to Dell 330 ps node + U-M Vendor C Device to UI Vendor C Device 23

24 U-M Vendor C Device perfsonar Active and C Routing IU Vendor C Device Is ps working with switch while working? INTERNET Dell 330 & U-M Dell 330 & IU Vendor C Vendor C Device Device Dell Mbps Mbps Mbps Mbps Dell Dell 330 ps node to U-M Vendor C Device and Dell 330 ps node to IU Vendor C Device 24

25 : POC Architecture C C C A J 25

26 Manufacturing Collaboration Vendor C Architecture 26

27 Manufacturing Collaboration Vendor C Observations Non-standard container Derived from Docker container Non-trivial container/switch config Container sometimes needs restarting Missing the option to bridge to SVI Currently all configs need extra cable Start of production pilot U-M 27

28 Manufacturing Collaboration Vendor J Architecture perfsonar demo at SC17 Ran on KVM Vendor J History Management Port Networking Non-EVO experiments (2018) Learning experience Container Networking Limitations prevented Successful pilot program 28

29 Manufacturing Collaboration Vendor J Observations Current Issues Complicated container process LXC with distrobuilder, chroot, no systemd, etc. Currently Ubuntu only Privileged vs unprivileged container security model Requires perfsonar REST interface re-development EVO on 5200 Experimental Release, Only in J s Lab 29

30 Manufacturing Collaboration Vendor A Architecture Container Architecture natively supported (Docker) Container not intended to be attached to external network (requires workaround) Disk space issues 30

31 Manufacturing Collaboration Vendor A Observations Container networking issues Forced to use Docker Host Networking Not feasible in production Feature Request for supported bridge container networking Standard Docker container support Disk / partition space issues Advanced Docker config required 31

32 Next Steps Finish deployment and testing with Company A and J Enhance documentation Automate container creation Explore container security architecture Further testing Add traffic, more hosts Inclusion in test meshes BGP Storm Stress Test DC MACC TOR INTERNET Company J idea 32

33 Lessons Learned: Staff Proficiency Necessary Breadth and Depth of Skills perfsonar Administrator Network Administration Network Administrator Switch configuration, basic container tshoot deploy / maintain perfsonar nodes & meshes Network Architecture System Administrator Operating System Expertise: UNIX Containers: Docker, LXC, proprietary, etc. NTP / Security / System Tuning System Administration 33

34 Advice for Next Generation Appliances Throughput speeds = max line speed Faster / parallel bus? Application specific ports? Implement container architecture Dedicated ASIC in Data Plane? Docker, LXC, etc. Simple & powerful container networking Reduce configuration complexity Route through any data port ( wireless ) 34

35 Next Steps Deploy perfsonar in Umich Lab EVO on 5200 Enhance documentation Automate container creation Explore container security architecture Further testing Add traffic, more hosts Inclusion in test meshes BGP Storm Torture Test DC MACC TOR INTERNET 35

36 University of Michigan Credits Eric Boyd, Phil Camp, Ed Colone, Dan Eklund, Brady Farver, Ryan Goniwiecha, Amy Liebowitz, John Simpkins, Katarina Thomas Internet2 Mark Feit Indiana University Dan Doyle, Michael Johnson Jisc Tim Chown, Raul Lopes Vendors Numerous engineers from multiple teams from multiple vendors 36