GRNET4 Design and Procurement of the new Layer2/3 network Andreas Polyrakis NOC Technical Manager apolyr@grnet.gr Terena Network Architects Workshop 21 22 November 2012, Copenhagen
What to expect 2 Design and Procurement of the new Layer2/3 network What to expect: Design principles and decisions Approach, methodologies, lessons learned What not to expect: Not a pure presentation about technologies Not a presentation about the procurement of our fiber network and DWDM equipment
GRNET DWDM and optical network 3 >9.000km of dark fiber (15-year IRUs) DWDM in most parts RED: ALU 1626LM LH, up to 21x10Gbps GREEN: ALU 1626LM Reg, up to 16x10Gbps YELLOW: ADVA FSP2000, up to 4x1Gbps BLACK: No DWDM, just Ethernet switches with1ge links The DWDM equipment and Fiber Network are also about to be re-procured
GRNET L2/L3 POPs 4 >40 PoPs all over Greece Large, Medium: IP router(s) Ethernet Switch(es) Small: Ethernet Switch(es) 2 (+½ ) DataCenters +1 planned
Why Upgrade? (1) 5 Lack of new features on the old hardware No reason to exist if we offer the same as commercial ISPs No funding unless we produce new services or get involved into GEANT/EU projects No fun for ourselves Also: EOL platforms Cost of 10GE ports / no 100G support Vendor/Platform mess Routers: Cisco 12000, Cisco 7200, Cisco 7300, Juniper T, Juniper MX Switches: Cisco 2970, Cisco 3750, Cisco 6500, Extreme X450/X450a/X350, Juniper 4200 Protocol interoperability issues No feature parity between platforms, eg RPKI, BGP flowspec Increased OPEX (daily ops, contracts & support, cases) Difficult or impossible to deploy new services (eg AutoBahn/BoD)
Why Upgrade? (2) 6 URGENT: Get rid of Ethernet as a WAN transport technology Spanning Tree problems Very poor implementations of MST, awful interoperability Common denominator, rapid spanning tree. Awful use of capacity, impossible to T.E. Hard to safeguard from loops within customers Hard to monitoring & troubleshooting Ethernet services Vlan propagation (VTP/GVRP/MRP) Scalability, exposure to customer MACs Downtime, increased OPEX
Approach 7 Set up a team (6 members + 3 reviewers) with the following duties: Prepare: 2-4Q2011 Design: Gather requirements Carry out the dimensioning Examine alternative technologies & topologies Choose the architecture, draft the solution, produce the requirements for the RFP Present to the administration, receive approval Discuss with vendors, finalize technical details design and requirements Draft the deployment/upgrade/replacement procedure Procure: Produce the RFP 1Q2012 Publish the RFP 3Q2012 Evaluate the proposals Award the contract 4Q2012 (estimated) Life Span of GRNET4: at least 5+years, estimated ~2020. Financial crisis in Greece may have an impact on this! The same team would examine the procurement of the optical network. At that time it was unclear if those two procurements would be published in parallel or if one of them (and which one) would proceed. That was also to be decided by the team.
Services 8 Internet connectivity IP feed (through BGP). IPv4/v6 unicast/multicast Various features (protocols/functionality) are necessary. Very big and heterogeneous list Layer 2 (Ethernet) VPN services connect sites of the same customer connect to other customers connect to GRNET datacenters (combined with our VPS service) Comments: No requests for L3 VPNs (service decommissioned) Sparse QoS requests However, good to be able to support both
Architecture 9 Service oriented approach Separate IP, Transport (Carrier and Access) and DataCenter equipment. Rules about roles. eg when a CPE switch needs to be upgraded to a Carrier Node.
Dimensioning - Questions 10 A Traffic Analysis and projections was conducted in order to dimension the network Customers capacity needs Number and locations of POPs of the IP network Capacities of the IP nodes, categorization and capacities of Carrier nodes and CPE switches Capacities of IP/Carrier backbone, access, uplink, IX links is 40G/100G necessary?
Dimensioning - Results 11 Capacity planning Customers: Majority@1Gbps, few @10Gbps. 2x-3x10Gbps might be seen in the future (2015+) Backbone for Carrier/IP: Minimum 10Gbps links (although this is over-engineering in many cases) Multiple of 10Gs on the core GEANT & IX projection (2015): 2x10G No 40G support. 100G may be used in the mid-term future. Location of IP POPs 93% of traffic is sourced from/destined to Athens (incl. IX & GEANT traffic) Max. one-way transmission delay from Athens to any POP: <7ms Makes sense to keep IP POPs in Athens only but Thessaloniki is kept as a future option (potential 2 nd exit point) Sizing of Carrier Nodes: 2 Large, 13 Medium, 26 Small 2 flavors of CPE switches: Large and Small
GRNET4 Topology 12
IP Network - Functionality 13 Collapsed IP network a small number of nodes with advanced functionality: protocols & features, esp. Routing Firewalling / (c.g.)nat / IPS & IDS Monitoring/netflow/IPFIX Decreased cost, easier upgrade of feature-set Carrier and DWDM networks deliver connectivity to IP network Backbone links, Access links, Datacenter connectivity, Upstream(s) and IX Design 2 POPs in Athens for redundancy; one IP router per POP Active/active setup, identical hardware, software, configuration Each resource (customer, IX, GEANT, datacenter) will be doublehomed (over Carrier or DWDM network)
14 Carrier Network Functionality Basic functionality: Transfer of Ethernet Frames between two points (E-LINE) or point-to-multipoint (E-LAN) E-Lines would be used to connect customers to our IP network E-Lines and E-LANs would be used to connect sites of customers (VPN functionality) (E-trees: a neat feature?) Traffic Engineering Fast Reroute Support for QoS / Shaping / Traffic Profiles OAM functions to monitor and troubleshoot Smooth handover and cooperation with Access Network (esp. in terms of management and OAM) MEF standards, terminology etc should be used
15 Carrier Network Technology candidates Technology candidates: MPLS-TP, PBB/T, EoMPLS Choice of the technology had to be done on an early stage* Great impact on procurement Had to be certain about the choice Evaluation/Validation Method: Participation in conferences and Fora Feedback from GEANT community: Ethernet Architecture Workshop with NORDUnet and Surfnet (Copenhagen, March 2012). Feedback from GEANT and other NRENs (eg PSNC). Endless discussions with vendors PBB/T evaluation on our own lab (Nortel equipment) Studying, studying, studying * A competitive dialog process could be very helpful the technology would not have to be chosen at an early stage. Unfortunately this was ruled out by the administration and legal department.
16 Carrier Network Technology choice And the winner is: EoMPLS Already familiar with this technology, no risks Easy deployment/upgrade path Small learning curve GEANT and many NRENs are on the same path MPLS-TP Too early, products seemed immature Promising but high risk Not so well suited to our environment PBB/T Poor vendor support, somehow abandoned Not convinced that it fits our needs
CPEs 17 (not so) Low-cost Ethernet switches Should cooperate smoothly with the Carrier Network i.e, OAM Two flavors, large and small Large: +2x10GE interfaces Ομάδα δικτύου ΕΔΕΤ 4 20/06/2011
NMS 18 One NMS to manage the entire network OS management Configuration management, templates Service provisioning, esp E-LINE/E-LAN services, T.E, OAM Web interface & Northbound API Customers view & requests Integration with provisioning tools, eg BoD Data Mining
Uniformity of equipment 19 Decision to go with a single vendor Interoperability of carrier/access OAM Better integration with (vendor) NMS Same platform for IP/Carrier Extra credits for any uniformity eg single OS, same SDK etc Attempt to minimize different types of optics (SFPs+/XFPs)
Bill of Material 20 Α/Α Type Quantity 1 CARRIER Small 38 2 CARRIER Medium 10 3 CARRIER Large 3 4 IP Router 2 5 CPE Small 50 6 CPE Large 30 7 DataCenter Switches 5 8 NMS 1
Procurement 21 Budget: 2.6M, all inclusive (licenses, features, etc) 5 years of support GPL price > 25M for Juniper/Cisco /$ rate was a big headache Status: RFP on air: August 1 st, 2012 Quite delayed, due to political and financial situation in Greece Closing date, Sept 25 th, 2012 About to finish the technical evaluation Rollout: 1-2Q 2013 Optical Network procurement & implementation is necessary to implement the final topology
Thank you 22 Questions?