3ROX Spring Meeting Thursday, 20 April 2017 Pittsburgh, PA
Agenda Welcome and introductions Updates December outage post-mortem Internet2 DDoS mitigation DNSSEC 3ROX routing (by request) eduroam Roundtable Adjourn 2
Updates 3
Commodity Have renewed contract with Cogent Will be talking with Level 3 No idea yet of impact, if any, from CenturyLink-Level 3 merger Telia now available in Pittsburgh Only at ACM (same as Cogent) Current mix has PoP, path diversity 4
Internet2 Nothing readily visible to end user Replaced all Brocade MLXe routers with Juniper MX series (480 or 960) Moving backbone from SDN/ OpenFlow to MPLS with SDN as overlay (part of short-term strategy) Using community input for requirements for 2018+ upgrades 5
TR-CPS 3ROX Replaced Cisco CRS-1 router with Juniper MX80 Much more stable Internet2 Migrated onto common routers with R&E Adding capacity to many peers Tables now over 300,000 routes Some talk about keeping that in check 6
Peering and Caching Consolidated from CRS-1/MX-80 to EX4600/ EX4300 Simplified management More capacity Per their suggestion, will be requesting another Netflix appliance Wouldn t be surprised to hear the same from Akamai Always looking for more options 7
General Infrastructure 10 GbE member connections migrated from Cisco 6509s to Brocade MLXe s No idea yet of impact of Extreme s purchase of Brocade product line Further deployment of upgraded management network 8
Network Statistics Lost some cacti statistics because of version skew on previous server Now collecting statistics every minute to get better idea of shorter-term peaks Dumping five-minute static views of cacti graphs Realize there are some warts Will revisit opening up live graphs 9
Routing in 3ROX 10
Overview It s probably fairly well known that we use route servers, but why do we do this and how does it work? For reference, running Euro-IX fork of the open-source quagga routing package 11
Architecture Review 3ROX more-or-less layer-2 gigapop (Most) member connections land on switch ports AICUP WWAN aggregated on router External connections land on routers Which in turn land on switched core Each member connection on separate VLAN 12
Member VLAN Commodity Aggregation Router (x2) Member-Member Router Internet2 Router Route Server Member VLAN Route Server TR-CPS Router Member s Router CPC Router 13
Why a separate VLAN? Provides reinforcement of AUPs Traffic can t reach non-allowed router Spoofing filters can achieve same function Obtain better usage statistics In past, did not have capability to collect flow statistics on many platforms No longer a problem with more recent hardware 14
What is a route server? A BGP speaker that doesn t forward packets ebgp routing based on destination, next hop Next hop doesn t have to be the BGP speaker itself (third-party BGP) Only requires peering with route servers (O(n) vs O(n 2 )) Not to be confused with a route reflector (similar principle for ibgp within a routing domain) 15
Members, VLANs and Views Each member gets own VLAN Each member gets own tables ( view ) on route servers Only AUP-correct routers on VLAN Member s router(s) Commodity, peering/caching, Internet2, et cetera With all services: over 1.5 million routes 16
Receiving Routes Route server receives a route from a peer (member, provider, etc) All routes: apply one or more BGP community tags to identify the source eg, all CDN get 5050:5000; Akamai also adds 5050:5010 Member routes: add tags to indicate which routes to send eg, 65534:5000 means to send all CDN routes 17
Cooking Routes Routes learnable from multiple sources Don t rely on default BGP tie-breaking rules Set BGP local preference to implement our policy Member routes most preferred Commodity transit (Cogent/Level3) least preferred 18
Example From route server config route-map WVNET_EXPORT permit 2020 set local-preference 100000 set community 5050:1000 5050:1270 65534:2000 65534:3000 65534:4000 65534:5000 additive Community tags mean This is a member route This is a WVNET route Send this route to R&E Send this route to local peering Send this route to commodity Send this route to peering/caching 19
Announcing Routes Routes announced only where they should be 65534:* communities stripped on announcements (internal use only) 5050:* communities (should be) retained 20
Issues/Questions/Concerns Do we still need a separate VLAN/view per member? Pros Clearly separates members configs and traffic Enables per-vlan statistics Cons Spoofing filters can handle separation Leave it up to the edge routers Very large configuration Flow statistics can gather traffic data Don t need VLANs just for statistics Requires beefy servers 21
Issues/Questions/Concerns Running fork of mainline quagga Can easily handle our large tables Mainline uses much more memory, time Others might not support multicast Supported by single developer in UK Git site down for several weeks Hasn t responded to email in last week Raises concerns 22
Futures? Will probably keep route servers Have used for years Makes layer-2 much easier Move to single DMZ (rather than permember VLANs)? Use better supported software? Would welcome input from 3ROX community 23
eduroam 24
What is eduroam? Short for educational roaming Simplifies access to wifi when visiting other institutions (no worries about guest accounts) Use home institution credentials International federation of RADIUS servers Internet2 operates top-level US servers 25
How does it work? High-level view: 26
How does it work? Low-level view Maybe a bit more complex than the last slide would suggest Requires that wifi use WPA2-Enterprise (aka 802.1x) Which uses RADIUS servers Which must talk to upstream RADIUS servers 27
Why use eduroam? Enabling eduroam on your campus provides four main features: 1. It allows your campus to welcome eduroam enabled visitors in a strongly authenticated way (the strong authentication also provides a way to authorize users to different resources) 2. It allows your own users to travel to eduroam enabled locations around the world (some places only have eduroam as a guest Wi-Fi) 3. It saves provisioning time for your institution and for your visitors since eduroam authentication is automatic and access is immediate 4. It improves security since your visitors use a standard protocol (WPA2-enterprise, 802.1X) that encrypts traffic between their devices and the Wi-Fi infrastructure (shamelessly stolen from Internet2 FAQ on eduroam) 28
What is required to join? Not necessary to be Internet2 member to join $700 application fee Waived if you sign the agreement as-is Which isn t out yet But some/all of fee may be waived if changes are required because of state law 29
Annual Subscription Internet2 members get for free $0.10 per student (based on IPEDS data) for other eligible institutions Minimum of $400 per year Not a huge amount, but 30
Consortia Yes, they are allowed! Sum up students to get total fee Must have single (modulo redundancy) RADIUS server for consortium Must have single agreement/point of contact If there is interest and considerations can be worked out, 3ROX would be amenable to serving as aggregator 31
Roundtable 32
Next Meeting Plan for Thursday, 12 October 33