Network Reliability. Artur Barczyk California Institute of Technology Internet2 Spring Member Meeting Arlington, April 25 th, 2011

Transcription

1 Network Reliability Artur Barczyk California Institute of Technology Internet2 Spring Member Meeting Arlington, April 25 th,

2 REAL-WORLD EXPERIENCE Reliability and Availability in Global R&E Environment Based on experience in US LHCNet 2

3 Networked Science More and more researchers rely on networks do to their science Reliability of network connectivity but also performance is becoming of primary importance Some (and increasing) scientific collaborations are truly global in nature World-Wide LHC Computing Grid (WLCG): 1 Tier-0 (CERN) 11 Tiers-1s; 3 continents 164 Tier-2s; 5 (6) continents Plus O(300) Tier-3s worldwide 3

4 MB/s per day Worldwide data distribution and analysis (F.Gianotti) Total throughput of ATLAS data through the Grid: 1 st January November. 6 GB/s ~2 GB/s (design) Peaks of 10 GB/s reached Grid-based analysis in Summer 2010: >1000 different users; >15M analysis jobs The excellent Grid performance has been crucial for fast release of physics results. E.g.: ICHEP: the full data sample taken until Monday was shown at the conference Friday 4

5 Transatlantic Operator s View High-Availability solutions require multiple links with carefully planned path redundancy TA links more complex than purely terrestrial ones Longer distance - more fibre, more equipment Typically constructed from segments from multiple owners Submarine segments: hostile environment, hard access Comparative Remarks on Outages Terrestrial spans: more frequent, shorter TTR Easier access for repairs; but also for diggers, copper thieves, Complex equipment from amplifiers to add/drop multiplexers more potential failure modes Submarine segments: less frequent, much longer TTR Difficult access longer repair time; Potential hazards: ship anchors, trawlers, geological events, sharks Repair speed depends on Time to Arrival of repair fleet, problem location, weather conditions What matters is Service Availability

6 US LHCNet Transatlantic Links Availability and Cuts % % % 96.19% % % % % % % % Cut in English Channel: 5.5 days Cut in Atlantic, 400 miles off US coast carrier rerouted to alternate path after initial period A single submarine cut can reduce availability significantly! Path Diversity is key to providing high-availability network service Two options to provide real robustness: Buy protected circuits Construct protected services from unprotected elements 6

7 High Service Availability in US LHCNet SIMULTANEOUS # of Failed TA links Effect on US LHC Tier1 services (primary and secondary) Effect on LHC Tier2 and other unprotected services Expected total duration within one year 1 Link No impact, service protected 16.8 Gbps operational per LHC Tier1 Degraded OR not operational 22 Days/year 2 Links Degraded, available bandwidth per LHC Tier1: Gbps Degraded OR not operational 6 Days/year 3 Links Degraded, at least 8.4 Gbps bandwidth available per LHC Tier1 Degraded OR not operational < 1 Hour/year (8 minutes/year observed) Small amount of protection capacity in US LHCNet is enough to protect its highest priority services against single link outages

8 Layer 1 Protection Robust, fast failover Transparent to most applications Linear/ring: 50ms Mesh: O(100ms) Mesh Protection scheme in US LHCNet Reduces capacity reserved for protection Flexible resource allocation Linear Protection Example Mesh Protection Example 8

9 Increasing Efficiency: Advanced Protocol Features Mesh Protection in Ciena s OSRP VCAT: Virtual Circuits Across Multiple Links Only a fraction of a virtual circuit is affected by an outage End-sites see only lower capacity in case of a link outage LCAS: Dynamic VC Capacity Adjustment VC capacity-adjustment provides smaller service impact during outages Combining VCAT/LCAS and Mesh Protection results in very flexible yet resilient network CERN-FNAL Primary Circuit CHI AMS NYC GVA

10 Provides hit-less capacity adjustment [Gbps] VCAT/LCAS in use Operational BW during simultaneous outage on two physical links (shared segment) lasting 11 hours Double cut, Virtual circuits automatically re-routed and resized Operator intervention, adjusting blue and red circuits capacity

11 Layer 3 Protection: US LHCNet + ESnet - Redundant Tier1 Paths Primary and secondary CERN- FNAL circuits on shortest path BGP failover to backup path for major incidents, e.g. facility failure Failover time: O(min) Same setup also for CERN-BNL circuits Backup CERN-FNAL path, completely path, device, and facility diverse between CERN and FNAL

12 Summary High service levels are increasingly important in order to support science goals To achieve high reliability Differentiate services Plan for outages Path diversity Device diversity Facility diversity Choose the right protection mechanism(s) for your environment US LHCNet is an example of high-performance transatlantic network ( % target availability) High availability targets met using Mesh Protection at Layer 1 VCAT and LCAS for increased availability 12

13 THANK YOU! 13