3ROX Spring Meeting. Thursday, 20 April 2017 Pittsburgh, PA

Transcription

1 3ROX Spring Meeting Thursday, 20 April 2017 Pittsburgh, PA

2 Agenda Welcome and introductions Updates December outage post-mortem Internet2 DDoS mitigation DNSSEC 3ROX routing (by request) eduroam Roundtable Adjourn 2

3 Updates 3

4 Commodity Have renewed contract with Cogent Will be talking with Level 3 No idea yet of impact, if any, from CenturyLink-Level 3 merger Telia now available in Pittsburgh Only at ACM (same as Cogent) Current mix has PoP, path diversity 4

5 Internet2 Nothing readily visible to end user Replaced all Brocade MLXe routers with Juniper MX series (480 or 960) Moving backbone from SDN/ OpenFlow to MPLS with SDN as overlay (part of short-term strategy) Using community input for requirements for upgrades 5

6 TR-CPS 3ROX Replaced Cisco CRS-1 router with Juniper MX80 Much more stable Internet2 Migrated onto common routers with R&E Adding capacity to many peers Tables now over 300,000 routes Some talk about keeping that in check 6

7 Peering and Caching Consolidated from CRS-1/MX-80 to EX4600/ EX4300 Simplified management More capacity Per their suggestion, will be requesting another Netflix appliance Wouldn t be surprised to hear the same from Akamai Always looking for more options 7

8 General Infrastructure 10 GbE member connections migrated from Cisco 6509s to Brocade MLXe s No idea yet of impact of Extreme s purchase of Brocade product line Further deployment of upgraded management network 8

9 Network Statistics Lost some cacti statistics because of version skew on previous server Now collecting statistics every minute to get better idea of shorter-term peaks Dumping five-minute static views of cacti graphs Realize there are some warts Will revisit opening up live graphs 9

10 Routing in 3ROX 10

11 Overview It s probably fairly well known that we use route servers, but why do we do this and how does it work? For reference, running Euro-IX fork of the open-source quagga routing package 11

12 Architecture Review 3ROX more-or-less layer-2 gigapop (Most) member connections land on switch ports AICUP WWAN aggregated on router External connections land on routers Which in turn land on switched core Each member connection on separate VLAN 12

13 Member VLAN Commodity Aggregation Router (x2) Member-Member Router Internet2 Router Route Server Member VLAN Route Server TR-CPS Router Member s Router CPC Router 13

14 Why a separate VLAN? Provides reinforcement of AUPs Traffic can t reach non-allowed router Spoofing filters can achieve same function Obtain better usage statistics In past, did not have capability to collect flow statistics on many platforms No longer a problem with more recent hardware 14

15 What is a route server? A BGP speaker that doesn t forward packets ebgp routing based on destination, next hop Next hop doesn t have to be the BGP speaker itself (third-party BGP) Only requires peering with route servers (O(n) vs O(n 2 )) Not to be confused with a route reflector (similar principle for ibgp within a routing domain) 15

16 Members, VLANs and Views Each member gets own VLAN Each member gets own tables ( view ) on route servers Only AUP-correct routers on VLAN Member s router(s) Commodity, peering/caching, Internet2, et cetera With all services: over 1.5 million routes 16

17 Receiving Routes Route server receives a route from a peer (member, provider, etc) All routes: apply one or more BGP community tags to identify the source eg, all CDN get 5050:5000; Akamai also adds 5050:5010 Member routes: add tags to indicate which routes to send eg, 65534:5000 means to send all CDN routes 17

18 Cooking Routes Routes learnable from multiple sources Don t rely on default BGP tie-breaking rules Set BGP local preference to implement our policy Member routes most preferred Commodity transit (Cogent/Level3) least preferred 18

19 Example From route server config route-map WVNET_EXPORT permit 2020 set local-preference set community 5050: : : : : :5000 additive Community tags mean This is a member route This is a WVNET route Send this route to R&E Send this route to local peering Send this route to commodity Send this route to peering/caching 19

20 Announcing Routes Routes announced only where they should be 65534:* communities stripped on announcements (internal use only) 5050:* communities (should be) retained 20

21 Issues/Questions/Concerns Do we still need a separate VLAN/view per member? Pros Clearly separates members configs and traffic Enables per-vlan statistics Cons Spoofing filters can handle separation Leave it up to the edge routers Very large configuration Flow statistics can gather traffic data Don t need VLANs just for statistics Requires beefy servers 21

22 Issues/Questions/Concerns Running fork of mainline quagga Can easily handle our large tables Mainline uses much more memory, time Others might not support multicast Supported by single developer in UK Git site down for several weeks Hasn t responded to in last week Raises concerns 22

23 Futures? Will probably keep route servers Have used for years Makes layer-2 much easier Move to single DMZ (rather than permember VLANs)? Use better supported software? Would welcome input from 3ROX community 23

24 eduroam 24

25 What is eduroam? Short for educational roaming Simplifies access to wifi when visiting other institutions (no worries about guest accounts) Use home institution credentials International federation of RADIUS servers Internet2 operates top-level US servers 25

26 How does it work? High-level view: 26

27 How does it work? Low-level view Maybe a bit more complex than the last slide would suggest Requires that wifi use WPA2-Enterprise (aka 802.1x) Which uses RADIUS servers Which must talk to upstream RADIUS servers 27

28 Why use eduroam? Enabling eduroam on your campus provides four main features: 1. It allows your campus to welcome eduroam enabled visitors in a strongly authenticated way (the strong authentication also provides a way to authorize users to different resources) 2. It allows your own users to travel to eduroam enabled locations around the world (some places only have eduroam as a guest Wi-Fi) 3. It saves provisioning time for your institution and for your visitors since eduroam authentication is automatic and access is immediate 4. It improves security since your visitors use a standard protocol (WPA2-enterprise, 802.1X) that encrypts traffic between their devices and the Wi-Fi infrastructure (shamelessly stolen from Internet2 FAQ on eduroam) 28

29 What is required to join? Not necessary to be Internet2 member to join $700 application fee Waived if you sign the agreement as-is Which isn t out yet But some/all of fee may be waived if changes are required because of state law 29

30 Annual Subscription Internet2 members get for free $0.10 per student (based on IPEDS data) for other eligible institutions Minimum of $400 per year Not a huge amount, but 30

31 Consortia Yes, they are allowed! Sum up students to get total fee Must have single (modulo redundancy) RADIUS server for consortium Must have single agreement/point of contact If there is interest and considerations can be worked out, 3ROX would be amenable to serving as aggregator 31

32 Roundtable 32

33 Next Meeting Plan for Thursday, 12 October 33