Technical report. Simon MUYAL. France-IX General Meeting September 2018

Transcription

1 Technical report Simon MUYAL 1

2 Agenda Technical key numbers Paris & Marseille infrastructure upgrades Zoom on outages Securing route servers Automating connection process These achievements are the result of the work done by the technical team Alexandre Arnaud Anastasia Boulbaba Mikael Pierre Pierre-Malo Thierry Vittorio 2

3 Technical key numbers 3

4 New ports connected during last year Period: From July 2017 to June 2018 Global 1G G G +7 Global +93 4

5 Troubleshooting Tickets (NOC) Global Q Q Q Q Global 241 Resp. time: 4.7/5 Web portal: 4.2/5 Backbone infrastructure upgrade Enabling Route server filtering Other recurring requests MAC address change Portal credentials 5

6 Network Availability Network availability : from july 2017 to june 2018 Paris = % 4 outages (detailed later) Marseille = 100% no outage 6

7 Paris infrastructure upgrades Optical infrastructure Telehouse 2 Equinix PA6 7

8 Paris optical infrastructure upgrade 8 Coriant Groove G30 deployed Initial capacity deployed per link: 400G PoPs upgraded: Interxion PAR5 Telehouse 2 Interxion PAR2 Equinix PA6 SCHEMA avec liens 400G upgradé 400G link 8

9 Paris optical infrastructure upgrade Very easy to add 200G or 400G of capacity between 2 PoPs Feedback after 12 months No issue impacting members Some python scripts to collect monitoring information via APIs 9

10 Paris IP infrastructure upgrades: TH2 Starting with core PoPs: Interxion PAR5 : Done in July 2017 Telehouse2 : Done in November chassis with ~100 customers each one à 2 maintenances Half of our customers, half of our traffic 10

11 Paris IP infrastructure upgrades: TH2 From Juniper EX9200 to Extreme SLX Reminder 8 slots 36 x 100G LC 72 x 1G/10G LC 11

12 Paris IP infrastructure upgrades: TH2 Before migration: 2-3 months Intensive tests Scripts to adapt configurations from Juniper to Extreme, using our Information System Power increase per rack Interconnecting existing and new platforms to minimize the downtime 12

13 Paris IP infrastructure upgrades: TH2 During migration: 02:00 à 08:00am Moving customers and monitoring in parallel Minimizing downtime per member : 5-10minutes in average After migration: during some days Double checking using our tools: Observium: Global statistics, power/fan alerts, etc Icinga: Customer IPs monitoring BUM traffic sniffers: same amount of BUM traffic QoS probes: same delays BGP sessions with route servers: Always a good indicator 13

14 Paris IP infrastructure upgrades: TH2 200 customers moved from Juniper EX9200 to Extreme SLX

15 PA6 upgrade July 2018 Increase of 100G requests ~ 20 members/30 ports to migrate Baie PA6 après migration Extreme SLX deployed 15

16 Marseille infrastructure upgrades New PoP MRS2 Additional 100G LC Route servers upgraded 16

17 Interxion MRS2 May 2018 France-IX PoP in the new DC Interxion MRS2 Juniper EX9200 installed 1G/10G/100G ports available Baie PA6 après migration Interconnected to Interxion MRS1 with dual redundant path 17

18 Interxion MRS2 May 2018 Backbone capacity: 100G between Interxion MRS1 and MRS2, based on passive DWDM MUXes Easy to evolve to 200G capacity Baie MRS2 photo Baie PA6 après migration Possibility to install DWDM active equipment if traffic grows (400G) 18

19 Marseille infrastructure upgrades PAR2 TH2 MRS1 1 st 2 nd 3 rd Baie MRS2 photo among 12 PoPs In term of traffic, Interxion MRS1 is our 3 rd PoP LCs installed to address 100G customer MRS1 Physical servers hosting route servers have been MRS1 MRS01 19

20 Infrastructure upgrades: future works Solution for edge PoPs Interconnecting Paris and Marseille 20

21 Edge : finding a solution for dense 100G PoPs Reminder Juniper EX9214 deployed in some edge PoPs Not enough dense in terms of 100G ports and expensive Baie MRS2 photo Solution found in 2017 with Brocade (SLX-9850 and SLX-9540) But Extreme is not able to maintain pricing conditions obtained previously Oct-Nov 2018 : Tender to find a box to replace progressively EX9214 when 100G needs increase 21

22 Interconnecting Paris and Marseille Following a recent survey, members are asking for a connection between Paris and Marseille platforms Baie MRS2 photo On the technical side, we will compare and select operators providing 100G waves between Paris and Marseille We are currently working on how to provide this interconnection to our members Available beginning of

23 Zoom on outages 08/2018 proxy ARP 09/2018 SLX platform 23

24 Outage : Proxy ARP Proxy ARP configured on a customer port A member started replying to some ARP requests, giving its MAC address in the reply A similar issue occurred 3 years ago Our tools didn t detect immediately the issue because the member replies only when the MAC address was not in its cache Baie MRS2 photo 24

25 Outage : Proxy ARP: Detecting faster In the quarantine VLAN, scan all the France-IX LAN IP range, and not only few IPs Analyse BUM sniffers in real time Ongoing work Interesting not only for proxy ARP issue Baie MRS2 photo Proactive: DAI: Dynamic ARP inspection Analyse ARP traffic and filter ARP replies when they didn t match well known static entries (IP <-> MAC) Available on Extreme SLX platform Already tested in our lab, waiting for the vendor feedback 25

26 Outage : Extreme SLX platform instabilities Extreme SLX , platform stable since September 2017, 4 chassis deployed, connecting 250 customers Baie MRS2 photo 2 issues encountered over the last 12 months February 2018 : Reload of one chassis at TH2: 9 minutes Combination of several issues a lack of memory solved in the latest firmware installed Now: Able to monitor memory and prevent this type of issue September 2018 : Reload of 2 100G LC during a logs collection: 8 minutes Command identified and documented on our side Waiting feedback from vendor 26

27 Securing route servers Applying strict filtering based on ROA/RPKI and IRR 27

28 Securing route servers: Story France-IX started securing route servers 18months ago BGP communities used to tag routes IRR not found ROA invalid 51706:65012 = Prefix has ROA status: VALID 51706:65022 = Prefix has ROA status: INVALID 51706:65023 = Prefix has ROA status: UNKNOWN 51706:65011 = Prefix is present in an AS's announced AS/AS-SET 51706:65021 = Prefix is not present in an AS's announced AS/AS-SET Done using a combination of our DB, bird, some scripts and tools like bgpq3, NTT DB Every member was able to filter by himself but this is not enough to have a secure Internet Exchange... 28

29 Securing route servers: Story Sept 2017 : Following GM-2017, interesting discussion on tech-ml about applying strict filtering, based on IRR and ROA tags Oct 2017 : Survey launched to have a better feedback, and not only few people expressing their opinion Nov 2017: Survey results: 110 members expressed their opinion!! 73% in favour of a strict filtering based on ROA and IRR tags Calendar for applying strict filtering: Initially announced for January 2018 Applied in February

30 Securing route servers: issues Technically, very simple! IRR not found and ROA invalid BGP communities were already applied on routes present in the RS Just a new rule in bird to filter these routes by default... But... Baie MRS2 photo In nov 2017, 30% of members using RS had at least 1 ROA invalid or IRR not found route... Some announcements made on MLs 30

31 Securing route servers: issues Members having 100% of their routes tagged as invalid: Invalid AS-SET in peeringdb for example Members having some of their routes filtered: AS-SET not up-to-date Some route objects missing in the IRR Baie MRS2 photo It took some time to reach a situation where members updated their IRR objects 31

32 Securing route servers: Giving visibility A dedicated looking glass to check route validity Documentation describing how filtering is done how we consider a route is invalid à Helping members à Saving a lot of time for support and troubleshooting 32

33 Securing route servers: D Day No main issue encountered No traffic loss Baie MRS2 photo A couple of members reacted to previous s to update their route objects 33

34 Securing route servers: D Day Paris 7000 IPv4 routes filtered among IPv6 routes filtered in Paris Baie MRS2 photo 34

35 Automating connection process 35

36 Automating connection process: DONE Reminder: Our information system based on netbox Automating LOA patch panel integration: Very long to check with DCs the real status of a patch panel position... Automating welcome mails Baie MRS2 photo Automating config checks 36

37 Automating connection process: Ongoing 50% of new ports delivered are related to upgrades Working on an easy way for members to start the upgrade process and save time Baie MRS2 photo Rewriting tools.franceix.net to provide additional information Survey: One of the points to improve Technical information (IPs, ports, MACs, patchpannels, RS status with ROA/IRR filtering) Commercial view: services, billing,... Available beginning of

38 France-IX (0) THANK YOU FOR YOUR ATTENTION twitter.com/ixpfranceix youtube.com/user/thefranceix facebook.com/ixpfranceix 38