TeraPaths: Managing Flow-Based End-to- End QoS Paths Experience and Lessons Learned

Transcription

1 TeraPaths: Managing Flow-Based End-to- End QoS Paths Experience and Lessons Learned Presented by Dantong Yu Summerʼ08 ESCC/Internet 2 Joint Techs Workshop

2 Outline Background: the TeraPaths project Establishing flow-based end-to-end QoS paths Encountered issues and proposed solutions Scalability of Dynamic Layer 2 circuits Reliability of Dynamic Layer 2 circuits Project status and future work 2

3 Objective Provide QoS guarantees at the individual data flow level, all the way to the end hosts, transparently Data flows have varying priority/importance Video streams Critical data Long duration transfers Default best effort network behavior treats all data flows as equal Capacity is not unlimited Congestion causes bandwidth and latency variations Performance and service disruption problems, unpredictability Dynamic flow-based SLAs = schedule network utilization Regulate and classify (prioritize) traffic 3

4 TeraPaths: End-to-End Setup Site B host host b1 site host / border Site C site border x y2 regional provider host c1 ACLs: b1 a1 regional provider WAN domains ACLs: c1 a2 VLAN X x1 site border y1 VLAN Y virtual border ACLs: a1 b1 a2 c1 host host a1 Site A host a2 4

5 Establishing E2E QoS Paths Multiple administrative domains Cooperation, trust, but each maintains full control Heterogeneous environment Domain controller coordination through web services Coordination models Star Requires extensive information for all domains Daisy chain Requires common flexible protocol across all domains Hybrid (end-sites first) Independent protocols Direct end site negotiation 5

6 Path Setup End site subnets are configured by TeraPaths software instances (TeraPaths Domain Controllers or TDCs) TDCs configure end site LANs to prioritize and regulate authorized flows via the DiffServ framework at the network device level Source site polices/marks authorized flow packets Destination site admits/re-polices/re-marks packets End site LANs tx/rx marked packets to/from the WAN WAN provides MPLS tunnels or dynamic circuits Initiating TDC requests MPLS tunnel or dynamic circuit with matching bandwidth and lifetime, or TDC groups flows with common src/dst into MPLS tunnel or dynamic circuit with aggregate bandwidth and lifetime WAN preserves packet markings 6

7 Addressing L2-Specific Issues Limitations with VLANs Tag range (tentatively selected 50 VLANs 3550 to 3599) Each site may have its own range Tag conflicts Rely on WAN service Eliminate by synchronizing site databases VLAN renaming (if/when possible) Scalability issues Limited number of VLAN tags/circuits: Flow grouping / circuit consolidation Forward flows through same virtual WAN circuit Create circuit with new parameters / switch current flows / cancel old circuit Modify WAN reservations (if/when possible) PBR overhead Virtual border Sensitive/3 rd party network segments VLAN pass-thru 7

8 Flow Grouping/Circuit Consolidation Flows between same src and dst sites can share circuit, policing maintains bandwidth guarantee Multiple TeraPaths reservations associate with the same circuit reservation Easy when requirements are known in advance Modification of reservations required otherwise Selection/optimization to minimize resource waste Trade-off based on Δbw (bandwidth difference), Δtb, Δta (time period before and after a reservation) bandwidth current time 2 Δbw Δt 5 5 time 8

9 Flow Grouping/Circuit Consolidation (2) Similar approach to disk buffering (read ahead / write behind) Bring up ahead / teardown behind Reuse existing active circuits Reserve circuits with more bandwidth and longer duration depending on differences in start time, duration, bandwidth of reservations Delay teardown, modify circuit duration and/or bandwidth if possible bandwidth Δtb Δta bandwidth Δta current time current time time time 9

10 Limitation of Dynamic Circuits Two recent incidents in BNL LHCOPN (PIC LHCOPN, and PBR in CISCO implementation will only use the status of Interface to decide whether or not to forward packets. A network circuit is broken somewhere along the Path. But the interfaces on both ends show up. No probes and heartbeat monitoring tools exist to check the health of layer 2 circuits. Fail-over to the backup link does not work since primary interfaces are up even when such a problem exists. End site monitoring is the most effective way to detect such a problem. 10

11 Active Circuit Probing Each TeraPaths site instance periodically verifies well being of reservations: Selects active reservations initiated by site (site responsibility) Finds circuit/vlan associated with each reservation Performs a circuit check with a quick pinging of other siteʼs (private ip address space) Less than 100% success triggers a recheck with longer duration pings in both directions (to and from other site) Low success % triggers reservation cancellation reverting traffic to best effort network Optionally, the system adapts reservation data and attempts to setup a new end-to-end path (for given time period/number of attempts) 11

12 Prioritizing Traffic TeraPaths QoS test 1 (prioritize traffic) 1200 Bandwidth (Mbits/sec) QoS / circuit reservation active priority background total 0 competing traffic causes dramatic drop in bandwidth time (sec) 12

13 Recovering from Circuit Failure TeraPaths QoS test 2 (prioritize/fallback to best effort) 1200 Bandwidth (Mbits/sec) recovery to best effort priority background total 0 circuit interruption time (sec) 13

14 Status TeraPaths sites: BNL,, UMich, BU, all with 10Gbps connections, multiplem pass-thru configurations (BNL, UMich, NoX, Merit, MiLR) Utilization of L3 paths (MPLS tunnels, ESnet only), L2 paths (dynamic circuits, ESnet and Internet2) Multiple QoS reservations through same circuit (support for circuit consolidation) Multiple circuits per site subject to per-site VLAN availability (flow grouping/circuit consolidation) Active circuit probing for failures with fallback to best effort network/attempt to reconfigure e2e path (in testing phase) Dynamic bandwidth allocation within service classes (in testing phase) New command line client 14

15 Future Work Continue working on automatic flow grouping / circuit consolidation. Configurable reservation negotiation Grid-style AAA (GUMS/VOMS) Plug-ins: SRM (dcache), others Compatibility with Lambda Station Support for different hardware as needed ATLAS Production: Replicate ATLAS Physics data from BU and Umich with the existing ATLAS DDM stack, and with end-to-end QoS circuits. Tier 1 (BNL) and Tier 2 data replication. 15