AMS-IX version 4. an MPLS/VPLS based internet exchange

Transcription

1 AMS-IX version 4 an MPLS/VPLS based internet exchange

2 Overview AMS-IX version 3 Short overview Bottlenecks and limitations AMS-IX version 4 The MPLS/VPLS platform AMS-IX v3 to v4 migration Operational Experience

3 June 2009 situation before start of migration AMS-IX version 3 GlobalSwitch pxc-nik-103 pxc-nik-104 NIKHEF pxc-nik-108 pxc-tel-105 pxc-tel-106 Telecity pxc-tel-110 pxc-sar-101 pxc-sar-102 pxc-sar-111 SARA pxc-sar-114 pxc-nik-112 pxc-tel-115 pxc-eun-113 eunetworks pxc-glo-107 pxc-glo-109 pxc-glo-117 pxc-nik-116 pxc-eqx-118 Equinix pxc-eqx-118 stub-nik-211 stub-nik-214 stub-eun-315 stub-eun-215 stub-glo-319 stub-glo-219 stub-eqx-318 stub-eqx-218 stub-sar-322 stub-sar-222 stub-nik-311 stub-nik-314 stub-tel-320 stub-tel-220 core-eun-301 core-glo-201 edge-sar-001 edge-tel-005 edge-nik-003 edge-eun-014 MLX8 edge-glo-007 edge-eqx-015 MLX8 edge-sar-001 edge-nik-003 edge-tel-005

4 AMS-IX version 3 Characterization E, FE and (N *) GE connections on BI-15k or RX8 switches (N * ) connections resilient connected on switching platform ( or ) via PXCs Brocade port security on customer interface to enforce one MAC per port rule for loop prevention

5 AMS-IX version 3 Characterization Two networks: one active at any moment in time Selection of active network by VSRP Inactive network switch blocks ports to prevent loops PSCD, photonic switch control daemon AMS-IX developed software to act on VSRP traps and manage PXCs

6 Customer router Customer router PXC pxc-sar-101 Access rood Access Blue Problem or Access rood Access Blue BLOCKED for traffic maintenance BLOCKED for traffic VSRP in red network VSRP Core Red MASTER VSRP HELLO Core Blue BACKUP VSRP Priority Core Red BACKUP VSRP HELLO Core Blue MASTER BLOCKED for traffic Red Master lower BLOCKED for traffic Edge than the Blue priority Edge Customer 1GE router Customer 1GE router AMS-IX Version 3 Platform Topology Failover

7 Traffic and Port Prognoses 9:8756:/"*+;"<6.;:/7.;"=>875?.6"<5678" #!!" 3"45678" '#!" '!!" &#!" &!!" %#!" %!!" $#!" $!!" #!"!" $()*+(,," $()*+(!!" &$(-./(!!" &$(-./(!$" &$(-./(!%" &$(-./(!&" &!(-./(!'" &!(-./(!#" &!(-./(!0" &!(-./(!1" %,(-./(!2" %,(-./(!," %,(-./($!" %,(-./($$" BA" $!BA" $!BA"C5+D"7.6?"<6.;:/E5+" '!F$!!BA"

8 AMS-IX version 3 Bottlenecks and Limitations Core switches (, 128 line rate) fully utilized Limits ISL upgrade Summer 2009 no substantial bigger switches on the market Platform failover introduces short link-flap on all customer ports. In few (but increasing) cases this leads to BGP flapping With more and more customer ports impact on overall platform stability becomes larger and larger Growth of number of 10G connections and customer LAG size requires larger switches Smaller switches => less local switching => larger ISL trunks

9 AMS-IX version 4

10 AMS-IX version 4 Requirements Scale the core to at least double amount of ports (Q2/3 2009) Keep resilience in platform and but reduce impact on failover. Increase amount of customer ports on switches More local switching Migrate to single architecture platform Reduce management overhead Use future proof (3 to 5 years) hardware that allows upscaling to high-density (2010) and 40/100GE (end 2010, early 2011)

11 Complete MPLS/VPLS topology AMS-IX version 4 GlobalSwitch pxc-sar-101 pxc-sar-102 SARA pxc-sar-111 pxc-tel-105 pxc-tel-106 Telecity pxc-tel-110 pxc-nik-103 pxc-nik-104 pxc-nik-108 NIKHEF pxc-nik-112 pxc-sar-114 pxc-tel-115 pxc-eun-113 eunetworks pxc-glo-107 pxc-glo-109 pxc-glo-117 pxc-nik-116 pxc-eqx-118 Equinix pxc-eqx-118 core-eun-302 core-eun-301 core-glo-202 core-glo-201 stub-nik-221 stub-nik-321 stub-eun-315 stub-eun-215 stub-glo-313 stub-glo-213 stub-eqx-318 stub-eqx-218 stub-tel-320 stub-tel-220 stub-sar-222 stub-sar-322 edge-sar-001 MLX8 edge-tel-005 MLX8 edge-nik-003 MLX8 edge-eun-008 MLX8 edge-glo-007 MLX8 edge-eqx-009 MLX8

12 AMS-IX version 4 Overview MPLS/VPLS-based peering platform Scaling of core switches by adding extra switches in parallel 4 LSPs between each pair of switches Load balancing of traffic over 4 LSPs between each pair of switches Retain switch resilience Keep customer connection on PXC No need for complete platform failover anymore Local impact only (single pair of switches on a site)

13 AMS-IX version 4 OSPF Characterization BFD for fast detection of link failures RSVP-TE signalled LSPs over predefined paths primary and secondary (backup) paths defined VPLS instance per VLAN Static defined VPLS peers (LDP signalled) Load balanced over parallel LSPs over all core routers Layer 2 ACLs instead of Port Security Manual adjustment for now

14 Physical Interconnection PXC PE-1-blue MLX-32 PE-1-red MLX-16 core-loctation-1 MLX-32 core-location-1 MLX-32 core-location-2 MLX-32 core-location-2 MLX-32 MPLS path over one core backup path over second PE-3 MLX-8 core on different site MPLS/VPLS setup

15 PE router failure LSP over LSP over Customers switched to primary Path PXC backup Path second switch PE-1-blue MLX-32 PE-1-red MLX-16 core-loctation-1 MLX-32 core-location-1 MLX-32 core-location-2 MLX-32 core-location-2 MLX-32 Primary Path PE-3 MLX-8 Backup Path MPLS/VPLS setup Resilience

16 How did we do the platform migration? AMS-IX v3 to v4 migration GlobalSwitch pxc-nik-103 pxc-nik-104 NIKHEF pxc-nik-108 pxc-tel-105 pxc-tel-106 Telecity pxc-tel-110 pxc-sar-101 pxc-sar-102 pxc-sar-111 SARA pxc-sar-114 pxc-nik-112 pxc-tel-115 pxc-eun-113 eunetworks pxc-glo-107 pxc-glo-109 pxc-glo-117 pxc-nik-116 pxc-eqx-118 Equinix pxc-eqx-118 stub-nik-211 stub-nik-214 stub-eun-315 stub-eun-215 stub-glo-319 stub-glo-219 stub-eqx-318 stub-eqx-218 stub-sar-322 stub-sar-222 stub-nik-311 stub-nik-314 stub-tel-320 stub-tel-220 core-eun-301 core-glo-201 edge-sar-001 edge-tel-005 edge-nik-003 edge-eun-014 MLX8 edge-glo-007 edge-eqx-015 MLX8 edge-sar-001 edge-nik-003 edge-tel-005 GlobalSwitch pxc-sar-101 pxc-sar-102 SARA pxc-sar-111 pxc-tel-105 pxc-tel-106 Telecity pxc-tel-110 pxc-nik-103 pxc-nik-104 pxc-nik-108 NIKHEF pxc-nik-112 pxc-sar-114 pxc-tel-115 pxc-eun-113 eunetworks pxc-glo-107 pxc-glo-109 pxc-glo-117 pxc-nik-116 pxc-eqx-118 Equinix pxc-eqx-118 core-eun-302 core-eun-301 core-glo-202 core-glo-201 stub-nik-221 stub-nik-321 stub-eun-315 stub-eun-215 stub-glo-313 stub-glo-213 stub-eqx-318 stub-eqx-218 stub-tel-320 stub-tel-220 stub-sar-222 stub-sar-322 edge-sar-001 MLX8 edge-tel-005 MLX8 edge-nik-003 MLX8 edge-eun-008 MLX8 edge-glo-007 MLX8 edge-eqx-009 MLX8?

17 Migration steps: Initial situation AMS-IX v3 to v4 migration GlobalSwitch pxc-nik-103 pxc-nik-104 NIKHEF pxc-nik-108 pxc-tel-105 pxc-tel-106 Telecity pxc-tel-110 pxc-sar-101 pxc-sar-102 pxc-sar-111 SARA pxc-sar-114 pxc-nik-112 pxc-tel-115 pxc-eun-113 eunetworks pxc-glo-107 pxc-glo-109 pxc-glo-117 pxc-nik-116 pxc-eqx-118 Equinix pxc-eqx-118 stub-nik-211 stub-nik-214 stub-eun-315 stub-eun-215 stub-glo-319 stub-glo-219 stub-eqx-318 stub-eqx-218 stub-sar-322 stub-sar-222 stub-nik-311 stub-nik-314 stub-tel-320 stub-tel-220 core-eun-301 core-glo-201 edge-sar-001 edge-tel-005 edge-nik-003 edge-eun-014 MLX8 edge-glo-007 edge-eqx-015 MLX8 edge-sar-001 edge-nik-003 edge-tel-005

18 Platform Migration Preparation Build new version of PSCD (Photonic Switch Control Deamon) No VSRP traps but LSP state in MPLS cloud Develop configuration automation Describe network in XML, generate configurations from this Move non MPLS capable switches behind MPLS routers and PXC as a customer connection Upgrade all non MPLS capable switches to Brocade MLX hardware Define migration scenario that would have no customer impact

19 2 Co-location sites only customer routers customer routers for simplicity < n*1 GE N * < n*1 GE N * Double L2 network PXC PXC VSRP for master slave selection and loop protection GE GE core core Colocation 1 Colocation 2 Migration steps: Initial situation simplified AMS-IX v3 to v4 migration

20 Not possible to connect customer routers customer routers GE switch to both MPLS/VPLS cloud and < n*1 GE N * < n*1 GE N * basic L2 network GE PXC GE PXC core core Colocation 1 Colocation 2 Migration steps: move GE behind PXC AMS-IX v3 to v4 migration

21 Production on L2 customer routers customer routers network (red) < n*1 GE N * < n*1 GE N * Migrate blue network to MPLS/VPLS GE PXC GE PXC Traffic between two PE routers load balanced over 2 LSPs, one over each P router PE router PE router Test functionality and connections using test traffic sent by Anritsu traffic generators P router P Router core Colocation 1 Colocation 2 Migration steps: Migrate one half to MPLS/VPLS AMS-IX v3 to v4 migration

22 Move production traffic customer routers customer routers to MPLS/VPLS cloud < n*1 GE N * < n*1 GE N * Use PXCs for failover New PSCD GE PXC GE PXC Run production on MPLS/ VPLS cloud for 6 weeks PE router PE router P router P Router core Colocation 1 Colocation 2 Migration steps: Production on MPLS/VPLS, L2 backup AMS-IX v3 to v4 migration

23 Migrate second half of the customer routers customer routers platform to MPLS/VPLS < n*1 GE N * < n*1 GE N * Test functionality and connections using test GE PXC GE PXC traffic sent by Anritsu traffic generators PE Router PE Router PE router PE router P Router P Router P Router P router Colocation 1 Colocation 2 Migration steps: Two MPLS/VPLS platforms AMS-IX v3 to v4 migration

24 Move production traffic customer routers customer routers to red MPLS/VPLS cloud using the newly developed < n*1 GE N * < n*1 GE N * version of PSCD to manage the PXCs GE PXC GE PXC Still two separate PE Router PE Router PE router PE router networks, both MPLS/ VPLS based P Router P Router P Router P router Colocation 1 Colocation 2 Migration steps: production on second MPLS/VPLS platform AMS-IX v3 to v4 migration

25 All PE routers connected customer routers customer routers to all P routers < n*1 GE N * < n*1 GE N * Between each pair of PE routers, 4 LSPs. One over each P router GE PXC GE PXC Traffic between each pair of PE routers load balanced over the 4 LSPs PE Router PE Router PE router PE router customer connections distributed over local PE routers Resilience in customer connection to local PE router by means of PXCs P Router P Router P Router Colocation 1 Colocation 2 P router Migration steps: integration to single MPLS/VPLS cloud AMS-IX v3 to v4 migration

26 Migration - Conclusion Traffic load balancing over multiple core switches solves scaling issues in the core Increased stability of the platform Backbone failures are handled in the MPLS cloud and not seen at the level. Access switch failures are handled by PXC for a single pair of switches only and not the whole platform Upscaling switches to Brocade allows for higher port density

27 Operational Experiences

28 Operational experience Issues BFD instability High LP CPU load caused BFD timeouts Resolved by increasing timers Bug: ghost tunnels Double Up event for LSP path Results in unequal load-balancing Scheduled to be fixed in next patch release

29 Operational experience ( 2 ) Issues Multicast replication Replication done on ingress PE, not on core Only uses 1st link of aggregate of 1 st LSP With PIM-SM snooping traffic is balanced over multiple links, but this has some serious bugs Bugfixes and load-sharing of multicast traffic over multiple LSPs scheduled for next major release

30 Operational experience ( 3 ) Issues

31 Operational experience ( 3 ) Issues Delay spikes in RIPE TTM graphs TTM datagrams have high interval (2 packets per minute), with ( changes some entropy (source port Brocade VPLS CAM: Entries programmed individually for each backbone port, age out after 60s For 24-port aggregates, traffic often passes port without programming => CPU learning => high delay Does not affect real-world traffic Much lower interval between frames Looking into changing/disabling CAM aging

32 Operational experience ( 4 ) Issues From : icmp_seq=1 Packet is claustrophobic Limited to single user (-; Suspecting problem caused by protocol-stack on client

33 Operational experience The good stuff Increased stability ( customers Backbone failures handled by MPLS (not seen by Access switch failures handled for a single pair of switches Phased relocation of traffic streams Looped traffic filtered by L2 ACL => No effect on linecard CPU

34 Operational experience ( 2 ) The good stuff Easier debugging of customer ports Simply swap to different, active switch using Glimmerglass PXC Config generation Absolute necessity due to size of MPLS/VPLS configuration Fairly simple because of single hardware platform

35 Operational experience ( 3 ) The good stuff ( options Scalability (future Bigger core devices Do not need to be MPLS-capable Load-sharing over > 4 cores Pending feature request Use of different cores for sets of PEs Multiple layers of P-routers

36 Conclusions Some issues found Nothing with impact on customer traffic Traffic load-sharing over multiple devices solves scaling issues in the core Increased stability of the platform Backbone failures not seen at the level Access switch failures trigger failover for corresponding Glimmerglass PXCs only Upscaling switches allows for higher port density Single hardware platform simplifies configuration generation

37 Questions?