Resilient IP Backbones. Debanjan Saha Tellium, Inc.

Transcription

1 Resilient IP Backbones Debanjan Saha Tellium, Inc. 1

2 Outline Industry overview IP backbone alternatives IP-over-DWDM IP-over-OTN Traffic routing & planning Network case studies Research Issues 2

3 Tellium Overview Aurora optical switch Aurora 32 Aurora 128 Aurora 512 StarNet software suit Element management system Network operating system Planning and modeling tools Value proposition Dynamic provisioning Mesh restoration Aurora Optical Switch 512 x 512 configuration 3

4 Industry Overview 4 Good news - data traffic growing at a healthy rate 100% growth rate in 2001, on track for 85% growth rate in 2002 Data traffic surpassed voice traffic by volume. Bad news - it is not a profitable business Bandwidth prices falling 30% yearly Lot of unused capacity in the network Capital spending has fallen 40% from its peak in 2000 Need to increase network efficiency through consolidation and better utilization Common IP/MPLS backbone for all data traffic IP backbone need to be reliable Traffic growth Internet Traffic Growth +Source RHK consulting

5 Troubling Failure Statistics IP Networks are Fragile Failures are frequent 6 million outages recorded in three years More than 50% of routes have MTTF of 15 days More than 75% of routes have MTTF of 30 days Much lower than carrier class availability 30-35% routes have more than 99.99% availability 10% of the routes have availability below 95% Service can be affected for a long time Only 30% of the outages are repaired within an hour 40% of the failures last more than an hour to several days Routing convergence after a failure takes 15 minutes Experimental Study of Internet Stability and Wide-area Backbone Failures. Craig Labovitz, Abha Ahuja, and Farnam Jahanian 5

6 Internet Routing Architecture Impact of Failures Autonomous System# 2 Inter-domain route failures Faults in connectivity between providers Loss of provider s connectivity to customers BGP requires all routes from a peer withdrawn after loss of peering session Change in link state triggers OSPF flooding Area 1 IGP: OSPF/ISIS Area 2 Autonomous System# 1 BGP Area 1 Exchange Point Area 2 6

7 Router Interconnections UPSR/BLSR SONET Ring UPSR/BLSR SONET Ring Routers connected over protected SONET Ring Routers connected over unprotected WDM 7

8 Backbone POP Architecture Backbone routers are dual redundant Backbone routers are connected to other POPs over unprotected wavelengths Access routers are connected to both backbone routers Customer routers are connected to one or two (rare) access routers Customer routers OC3/OC12 OC3/OC12 Access routers OC48 OC48 OC48 Core routers OC192 OC192 OC192 To other PoPs OC192 8

9 Different Types of Failures Transport failures Caused by fiber cut and WDM failures May cause multiple links to fail at the same time and takes hours to fix Optical layer restoration can mask transport failures from IP layer Router failures Could be software or hardware failure Software failures can be fixed in minutes Network failures Typically due to congestion and misconfiguration, and malicious attacks Maintenance problems Primarily a customer network issue, backbone routers typically do not suffer from this problem 19% 26% 27% 28% Transport failure Router Failure Maintenance Network problem 9

10 Transport Failures Cause Multiple Failures at IP Layer 10 Wavelengths are often glass through at multiple PoPs Fibers between different PoP pairs sometime share conduits IP layer is agnostic of risk dependency at the transport layer Single failure at the transport layer may trigger multiple failures at the IP layer Protection at the transport layer can alleviate much of the problem PoP 1 PoP 5 Shares DWDM PoP 2 PoP 3 Shares conduit PoP 4 PoP 6

11 IP Layer Protection IP rerouting around the failure Current mode of operation in the Internet today Time tested mechanism; simple and robust Routing convergence can be slow; network is unstable during recovery Network run at very low utilization to absorb temporary overload due to failure End-to-end MPLS protection Potentially faster restoration Requires traffic engineering and primary and backup path pre-planning Backup and primary LSPs should be share disjoint More expensive than IP rerouting Routing stability and scalability issues 11 are still open A B D C Backup path F Primary path End-to-end MPLS Restoration E

12 MPLS Fast Reroute Detour to avoid AB Detour to avoid CD Detour to avoid link DE LSP shown traverses A, B, C, D, E, F B D F Each detour avoids Immediate downstream node & link towards it Except for last detour: only avoids link DE A C Detour to avoid BC Fast reroute example E Detour to avoid DE Merged detour reduces state maintained, reduces signaling overhead, and improves utilization A Detour to avoid AB Detour to avoid CD D E F B C Merged Detour to avoid AB and BC Fast reroute with merged detour example 12

13 Optical Mesh Restoration 13 Similar to protection at the SONET layer; but at a fraction of the cost Protects the backbone links between routers using shared mesh protection Fast, robust, and scalable mechanism for protecting transport layer Do not protect against router failures; complementary mechanisms necessary Optical bypass reduces the impact of router failures IP rerouting or MPLS restoration for handling router failures Transceiver Port C ount ( Thousoands) % 28% Ring 1+1 Dedicated Shared Mesh 75 cities with 93 fiber links, degree of connectivity of 2.48 Restoration Service

14 Optical Restoration Advantages Fast, Scalable, and Efficient Restores hundreds of wavelengths possibly containing thousands of MPLS LSPs in less than 200 ms Scalability of MPLS restoration is a big leap of faith. A fiber cut or DWDM failure may lead to thousands of LSPs to fail. Takes advantage of physical plant information for efficient diverse path routing; physical plant information is not available at IP/MPLS layer Physical plant information is very important for routing primary and backup paths in an SRLG disjoint fashion. Restoration Latencies (ms.) max. avg # of OC-48 lightpaths simultaneously failed 50 node network, 910 OC-48 lightpaths 14

15 Optical Restoration Advantages Improves Network Stability Primary route Optical restoration does not impact IP routing or MPLS label distribution after failure Every failed link is replaced by a backup link of same capacity - topology does not change In most cases the routers would not even detect the failures at the optical layer. Backup route Restoration at the IP Layer When optical restoration is used, IP layer utilization does not change after a failure and subsequent restoration Primary lightpath Connectivity between routers remains the same Avoids the need to revert thousands of LSPs from protection to working paths when failures are repaired 15 Backup lightpath Restoration at the Optical Layer

16 Routing with Express Bypass Links IP routed path BEFORE express bypass Lightpath established as express bypass IP routed path AFTER express bypass 16

17 Primary and Backup Path Routing MPLS and Optical 17 Pick the next demand d from the demand set Find the set of k-smallest cost paths S = {P 1,P 2,.., P n ) from source to destination For each primary path P i find the backup path as follows For each link in the link in the graph Set the cost to infinity if it is part of P i Set the cost of ε if it is shareable Set to original link cost otherwise Find the shortest backup path B i Find the primary and backup path pair such that P i + B i is the smallest Repeat until the demand set is exhausted Repeat the whole process N number of times ε α α α ε ε ε

18 Routing IP Flow over Optical Backbone Logical link Start with a fully connected logical graph consisting of physical links and logical lightpaths Pick the next demand d from the demand set Set the cost of a logical link as a function of Original cost of the physical link and the switch ports Remaining capacity of the link and the size of the demand Find the smallest cost path from the source to destination Repeat until the demand set is exhausted Repeat the whole process N number of times Physical link 18

19 IP Network Study Network, Traffic, and Pricing Assumptions Network Model 12 nodes, 17 links US backbone network. Average node degree is 2.8 Traffic Model Total of 66 bi-directional demands. Average demand between POPs is 3Gbps 75% year-over-year growth assumed Pricing model Street prices for routers used in the study. OC48 and OC-192 $45K and $125K per port, respectively OC48 and OC192 OXC ports are assumed to be $10K and $35K, respectively. OXC common equipment cost of $350K Different scenarios considered Network Topology 19

20 Router Port Comparison IP-over-WDM Link Utilization 50% IP-over-WDM: utilization kept below 50% to handle overload due to failure IP-over-OTN: utilization kept below 50% Large reduction in number of router ports From 2,096 to 1,581 in Y1 From 18,324 to 13,848 in Y5 Large reduction in number of routers From 64 to 52 in Y1 From 388 to 244 in Y5 Number of 2.5G Ports IP-over-WDM unprotected with 50% utilization IP-over-OTN mesh protected with 50% utilization IP-Over-WDM Router Ports IP-Over-OTN Router Ports IP-Over-OTN AOS Ports % reduction in router ports with IP-over-OTN Y1 Y2 Y3 Y4 Y5 Year

21 Network Cost Comparison IP-over-WDM Link Utilization 50% IP-over-WDM unprotected with 50% utilization IP-over-OTN mesh protected with 50% utilization Negligible cost increase Inter-quad router ports not accounted for Optical protection increases the robustness and stability of the network Extends the lifetime of routers by 2-3 years Protection capacity can be used by other applications Cost in Million IP-Over-WDM Total Cost Y1 Y2 Y3 Y4 Y5 265 Year IP-Over-OTN Total Cost $37M or 5% CAPEX increase with IP-over-OTN

22 Router Port Comparison IP-over-WDM Link Utilization 30% IP-over-WDM: utilization kept below 30% to handle overload due to failure IP-over-OTN: links are protected at the optical layer; utilization kept below 50% Large reduction in number of router ports From 2,706 to 1,581 in Y1 From 24,086 to 13,848 in Y5 Large reduction in number of routers From 88 to 52 in Y1 From 568 to 244 in Y5 IP-over-WDM unprotected with 30% max link utilization IP-over-OTN mesh protected with 50% max link utilization Number of 2.5G Ports IP-Over-WDM Router Ports IP-Over-OTN Router Ports IP-Over-OTN AOS Ports % reduction in router ports with IP-over-OTN Y1 Y2 Y3 Y4 Y5 Year

23 Network Cost Comparison IP-over-WDM Link Utilization 30% IP-over-WDM unprotected with 30% max link utilization IP-over-OTN mesh protected with 50% max link utilization Substantial cost savings Optical protection increases the robustness and stability of the network Extends the lifetime of routers by 2-3 years In IP-over-OTN protection capacity can be used by other applications Similar savings possible in ATM/FR networks Cost in Million IP-Over-WDM Total Cost IP-Over-OTN Total Cost Y1 Y2 Y3 Y4 Y5 Year $170M or 18% CAPEX savings with IP-over-OTN

24 MPLS and Optical Protection Cost Comparison Primary lightpath POP 2 Primary LSP POP 2 POP 1 POP 4 POP 1 POP 4 Backup LSP POP 3 Backup lightpath POP 3 MPLS Protection Burns 4 intermediate backbone router ports in POP2 and POP3 each Total cost $1.0M assuming 10G router ports at $125K each Optical Mesh Protection Burns 4 OXC ports in POP1 and POP4 each Burns 2 OXC ports in POP2 and POP3 each Total cost of $420K assuming 10G OXC ports at $35K each 24

25 IP Network Study Network, Traffic, and Pricing Assumptions Network Model 12 nodes, 17 links US backbone network. Average node degree is 2.8 Traffic Model Total of 66 bi-directional demands. Average demand is 1.5 Gbps 50% year-over-year growth assumed Pricing model Street prices for routers used in the study. OC48 and OC-192 $45K and $125K per port, respectively OC48 and OC192 OXC ports are assumed to be $10K and $35K, respectively. OXC common equipment cost of $350K Different scenarios considered Network Topology 25

26 Router Port Reduction MPLS vs. Optical Restoration IP-over-WDM: restoration using MPLS shared backup restoration IP-over-OTN: restoration using shared optical mesh restoration Only core routers considered Large reduction in number of router ports From 732 to 128 in Y1 From 3414 to 502 in Y5 Large reduction in number of routers From 42 to 24 in Y1 From 146 to 54 in Y5 Number of 2.5G Ports IP-Over-WDM Router Ports IP-Over-OTN Router Ports IP-Over-OTN AOS Ports Y1 Y2 Y3 Y4 Y5 Year % reduction in transit router ports Transit traffic at IP-over-WDM POPs Burns 2 router ports 50% of the time Burns 4 router ports 50% of the time

27 Network Cost Savings MPLS vs. Optical Restoration Substantial cost savings Average of 54% over 5 years Total of $133M over 5 years excluding DWDM Similar savings possible in ATM/FR networks Inter router-quad tie-ports not accounted for Cost in Million IP-Over-WDM Total Cost IP-Over-OTN Total Cost % CAPEX savings 245 Y1 Y2 Y3 Y4 Y5 Year Transit traffic at IP-over-WDM POPs Burns 2 router ports 50% of the time Burns 4 router ports 50% of the time

28 Switched Optical Backbone Advantages: Fast and Easy Provisioning Risk management IP traffic pattern can vary Long term variations: due to change in application mix, peering policy etc. Short term variations: due to user behavior, scheduled time-of-day activities etc. Traffic Projection is Difficult Fast reconfiguration reduces detrimental business impact due to inaccurate traffic projection New services Storage and backup applications, SANs Datacenter to NAP connectivity for special event hosting Remote peering with other ISPs 28

29 Research Issues IP Networks Multi-area traffic engineering Aggregation of traffic engineering information Traffic engineering across AS boundary Primary and backup path routing Shared Risk Group information Distributed routing avoiding same shared risk groups Constraint-based Shortest Path First routing algorithms Fast convergence of routing protocols Area 1 Area 2 Area 0 To other AS 29

30 Research Issues Optical Networks Dynamic provisioning Generalized MPLS (GMPLS) Neighbor discovery, topology discovery, routing, signaling Mesh restoration Shared mesh restoration Restoration protocols IP-optical convergence Optical UNI Event driven network reconfiguration UNI E-NNI Optical Network Optical subnet I-NNI Optical Network Optical subnet Optical subnet E-NNI E-NNI Optical Network Optical Network UNI 30