Politecnico di Torino Network architecture and management Marcello Maggiora, Antonio Lantieri, Marco Ricca Outline Politecnico di Torino network: Overview Building blocks: Edge, Core, Distribution, Access network Network architecture Core network Distribution and Access network Wireless network Telephone service Edge network Fault Tolerance Datacenter and Business Continuity Network management Wired network management Wireless network management Unified management Marcello Maggiora, Antonio Lantieri, Marco Ricca 2 1
Politecnico di Torino network: Overview Marcello Maggiora, Antonio Lantieri, Marco Ricca 3 Campuses Internetworking Students: ~32,000 Staff: 2,000 Campus: ~300,000 mq Metropolitan campuses: 5 Regional campuses: 6 Marcello Maggiora, Antonio Lantieri, Marco Ricca 4 2
Campus Data Network Capacity Internet Connection (via GARR) = 10 Gbps 60 Gbps core bandwidth Datacenter 2 main datacenters in Business Continuity 3 switching rooms Many technical rooms Networking devices Access Points: ~450 Switches & routers: ~450 Networked devices Telephones: ~3,000 Network nodes: ~14,000 Thousands of mobile devices Marcello Maggiora, Antonio Lantieri, Marco Ricca 5 Core network at a glance Datacenters (DC) 2 main DC (labeled 1,5) 3 switching rooms (labeled 2,3,4) Many technical rooms 4 2 Core devices 2 Cisco 6500 switches in DC 1,5 6 HP 5500 switches in DC 2,3,4 Service modules Wireless LAN Controller Load Balancers 5 3 CORE 1 Marcello Maggiora, Antonio Lantieri, Marco Ricca 6 3
Politecnico di Torino Network Architecture Data center 5 EDGE Internet Data center 1 CORE Marcello Maggiora, Antonio Lantieri, Marco Ricca DISTRIBUTION ACCESS 7 Politecnico di Torino Network Architecture Data center 5 EDGE Internet Data center 1 CORE ACCESS DISTRIBUTION Marcello Maggiora, Antonio Lantieri, Marco Ricca 8 4
Three-layer hierarchical model Break a complex network into smaller, more manageable network Core network: optimized and reliable transport structure by forwarding traffic at very high speeds Distribution network: defines policy for the network such as Access Control List (ACL) and routing Access network: supplies traffic to the network and performs network entry control Layered models are useful because: Facilitate modularity Devices at each layer have similar and well-defined functions Scalable design Marcello Maggiora, Antonio Lantieri, Marco Ricca 9 Core network Marcello Maggiora, Antonio Lantieri, Marco Ricca 10 5
Core physical architecture 1 VIRTUAL SWITCH LINK 5 2 Cisco 6509 L2/L3 switches 2 3 4 6 HP 5500 L2 switches DISTRIBUTION and ACCESS Marcello Maggiora, Antonio Lantieri, Marco Ricca 11 Preventing switching loops Spanning Tree Protocol (STP) Standard approach Increase network management complexity Reduces capacity Convergence takes time Switch virtualization techniques Provide a loop free topology A smart way to build a logical topology Proprietary technologies like VSS/IRF clustering Marcello Maggiora, Antonio Lantieri, Marco Ricca 12 6
Core network: Switch virtualization Switch virtualization technologies: pools multiple switches into single virtual switch Virtual Switching System (VSS) Intelligent Resilient Framework (IRF) Benefits Simplifying the network increases operational efficiency Scales system bandwidth capacity Providing a loop-free Layer 2 topology: operate as a single logical virtual switch High availability Load balancing Marcello Maggiora, Antonio Lantieri, Marco Ricca 13 Core network: Virtual Switching System (VSS) Two physical chassis cluster into a single logical entity The cluster appears as a single logical switch to neighboring device Only one active control plane and the other as standby The switch fabric of both switch are in active state Aggregate switch fabric capacity = 1440 Gbps or 1.44TBps One supervisor engine act as central management point for the entire system VIRTUAL SWITCHControl Plane LINK 1 5 Forwarding Plane Control Plane Forwarding Plane Marcello Maggiora, Antonio Lantieri, Marco Ricca 14 7
Virtual Switching Link (VSL) Special signaling and control information must be exchanged between the two chassis Need a special link, VSL, to transfer both data and control traffic between the peer chassis The VSL is formed as a Cisco EtherChannel interface (1 to 8 ports) Control traffic gets highest priority across the VSL Virtual switch header is appended onto every frame sent across the VSL Marcello Maggiora, Antonio Lantieri, Marco Ricca 15 Core network: VSS Hardware Deployment Cisco 6509 switch Modules Supervisor Engine Connection modules Service modules Wireless LAN Controller WLC Application Control Engine ACE Marcello Maggiora, Antonio Lantieri, Marco Ricca 16 8
Core network: VSS Hardware Deployment Modules that support the formation of a VSL port channel 10 Gigabit Ethernet uplink ports on the Supervisor Engine 720-10G Cisco 8-Port 10 Gigabit Ethernet Switching Module Marcello Maggiora, Antonio Lantieri, Marco Ricca 17 Core network: VSS Hardware Deployment VSL formed out of Two Supervisor Engine Ports Fewer hardware components VSL module fails entire VSS fail VSL bandwidth not scalable Marcello Maggiora, Antonio Lantieri, Marco Ricca 18 9
Core network: VSS Hardware Deployment VSL formed out of Cisco 10 Gigabit Ethernet ports More hardware components VSS deployment is highly redundant Scalable VSL bandwidth Marcello Maggiora, Antonio Lantieri, Marco Ricca 19 Core network: VSS Hardware Deployment VSL across Supervisor Engine Ports and 10 Gigabit Ethernet ports PoliTo case: cost balanced scenario More hardware components Offers link and line-card redundancy Scalable VSL bandwidth Marcello Maggiora, Antonio Lantieri, Marco Ricca 20 10
Core network: VSS Logical topology after VSS configuration Single active control and management plane Combined switching fabric capacity 720 Engine switch #1 720 Engine switch #2 VSS 1440 Terabit switch 10Gbps 20Gbps Meshed network Loop-free network Marcello Maggiora, Antonio Lantieri, Marco Ricca 21 Switch virtualization benefits revisited Simplifying the network increases operational efficiency Only single logical switch Providing a loop-free Layer 2 topology No STP configuration needed Scales system bandwidth capacity Both chassis data plane is active No STP so all links are used fully Load balancing and redundancy High availability boosts nonstop forwarding/stateful switchover (NSF/SSO) Marcello Maggiora, Antonio Lantieri, Marco Ricca 22 11
Core network: VSS High Availability In an Stateful switchover (SSO) system, protocols and features synchronize events and state information from the active supervisor engine to the hot-standby supervisor engine In the event of a failover, the standby supervisor engine does not need to re-learn this information resulting in a minimal amount of outage time Marcello Maggiora, Antonio Lantieri, Marco Ricca 23 Core network: Intelligent Resilient Framework (IRF) Can we do the same for the 6 HP switches? 720 Engine switch #1 720 Engine switch #2 VSS 1440 Terabit switch Marcello Maggiora, Antonio Lantieri, Marco Ricca 24 12
Core network: Intelligent Resilient Framework (IRF) 6500-VSS M1-P1 (IRF1) M2-P2 (IRF2) M1-P1 (IRF1) M2-P2 (IRF2) M1-P1 (IRF1) M2-P2 (IRF2) M1-P1 (IRF1) M2-P2 (IRF2) M1-P1 (IRF1) M2-P2 (IRF2) M1-P1 (IRF1) M2-P2 (IRF2) Marcello Maggiora, Antonio Lantieri, Marco Ricca 25 Core network: Final logical architecture Importance Simplified management STP free High capacity High availability Protocols involved EtherChannel LACP Virtualization technologies VSS IRF Hardware 2 Cisco 6500 switches 6 HP 5500 switches VSSCORE DC1,5 Link aggregation control protocol (LACP) 60Gbps DC2,3,4 IRFCORE Marcello Maggiora, Antonio Lantieri, Marco Ricca 26 13
Politecnico di Torino network: Final logical architecture Data center 5 Data center 1 Bade system Enclosure #1 SWBLADE_1.2 SWBLADE_1.1 DC1,5 SWBLADE_2.2 SWBLADE_2.1 Bade system Enclosure #2 Blade Server CORE Blade Server DC2,3,4 Departments DISTRIBUTION ACCESS Marcello Maggiora, Antonio Lantieri, Marco Ricca 27 Access and Distribution network Marcello Maggiora, Antonio Lantieri, Marco Ricca 28 14
Access network Supply traffic to the network and performs network entry control Rapid STP configuration: Why? Use case scenarios Wireless network Telephone network Departments DC2,3,4 IRFCORE CORE DISTRIBUTION ACCESS Is it working? Marcello Maggiora, Antonio Lantieri, Marco Ricca 29 Polito WiFi In production since 2004 Protocols supported: 802.11a/g/n/ac 54Mbps, 600Mbps, 1.3Gbps SSID: polito, eduroam October November December January 802.11n (2.4GHz) 5200 4070 802.11ac 802.11n (5.0GHz) Time Associated Client Authenticated Client Marcello Maggiora, Antonio Lantieri, Marco Ricca 30 15
A Layer 3 Campus WLAN logical view ap_manager port CAPWAP tunnel (over IP) AP 1 AP 2 10.10.10.1 VLAN1 SSID: polito VLAN1 VLAN2 AP 3 10.10.10.2 10.10.10.254 10.10.20.254 PoliTo Network VLAN2 10.10.20.25 SSID: polito One CAPWAP tunnel for every AP-Centralized controller connection Users mobility management Alternative 1) disable roaming between VLAN 2) (try to) keep each user s session alive Same SSID, different VLANs VLAN deployment to define multiple broadcast domain Marcello Maggiora, Antonio Lantieri, Marco Ricca 31 Layer 2 Roaming CAPWAP tunnel (over IP) AP 1 10.10.10.1 SSID: polito VLAN1 10.10.10.1 VLAN1 AP1 polito 10.10.10.2 VLAN1 AP2 polito AP 2 10.10.10.2 VLAN 1 10.10.10.254 PoliTo Network VLAN, client IP address and router gateway IP address remain the same Centralized controller keeps the information about the user s session Roaming handled in less than 10 ms Marcello Maggiora, Antonio Lantieri, Marco Ricca 32 16
Layer 3 Roaming AP 1 10.10.10.1 SSID: polito AP 2 VLAN1 10.10.10.254 VLAN1 10.10.20.254 VLAN2 PoliTo Network Client IP Add. As. VLAN Cur. VLAN Cur. AP SSID Marcello Maggiora, Antonio Lantieri, Marco Ricca VLAN2 10.10.20.25 SSID: polito 10.10.10.1 VLAN1 VLAN1 AP1 polito Client IP Add. As. VLAN Cur. VLAN Cur. AP State SSID 10.10.20.25 VLAN2 VLAN2 AP8 polito 10.10.10.1 VLAN1 VLAN1 AP1 polito Design goal: Allow users mobility among the campus, w/out closing user s session Keep same IP address on a different VLAN Transparent operation from user s prospective Default gateway remains the same, wi-fi controller marks device as visitor on a different VLAN AP 8 10.10.20.25 VLAN2 VLAN1 AP2 Roamed polito 33 Access network: Wireless network Wi-Fi Controller Wired Network Marcello Maggiora, Antonio Lantieri, Marco Ricca 34 17
Wireless network: unique features Control And Provisioning of Wireless Access Points (CAPWAP) protocol Enables a central wireless LAN Controller (WLC) to manage Wireless Access points Provides configuration and device management Full Datagram Transport Layer Security (DTLS) tunnel Uses UDP ports 5246 (control channel) and 5247 (data channel) Mobility Layer 2 roaming: same VLAN, client IP address, router gateway IP address Layer 3 roaming: different VLAN, client labeled as visitor for the whole session Marcello Maggiora, Antonio Lantieri, Marco Ricca 35 Eduroam authentication procedure Hierarchal system of RADIUS servers 1. Client host institution RADIUS 2. Host institution NREN RADIUS 3. TERENA central RADIUS 4. Client home institution NREN RADIUS 5. Client home institution RADIUS 6. Auth. 'ack' travels back over the proxy-hierarchy to the host institution and the user is granted access. Marcello Maggiora, Antonio Lantieri, Marco Ricca 36 18
Edge network Marcello Maggiora, Antonio Lantieri, Marco Ricca 37 Edge network: Generalized schema Bastion Hosts (web, mail relay, other) Internal Hosts dmz Internet ISP Border Router outside inside Firewall dmz LAN switch Inside LAN DC Firewall LAN2LAN and Remote VPN OUTSIDE Network INSIDE Network Protected Server Marcello Maggiora, Antonio Lantieri, Marco Ricca 38 19
Edge network: Detail LAN2LAN and Remote VPN Border Router Firewall Core Switch Protected Server DC5 10Gbps Router DC5 outside dmz inside Firewall DC5 Core Switch DC5 Internal Hosts Firewall DC5 Only one BGP Process Inside LAN Others server 1Gbps outside inside Core Switch DC2, 3, 4 Firewall DC1 Router DC1 dmz Firewall DC1 Core Switch DC1 Protected Server DC1 LAN2LAN and Remote VPN Marcello Maggiora, Antonio Lantieri, Marco Ricca 39 Edge network: unique features VPN concentrator: advanced encryption and authentication techniques built specifically for creating a remote-access or site-to-site Firewalls: control the incoming and outgoing network traffic based on an applied rule set BGP: exterior gateway protocol designed to exchange routing and reachability information between autonomous systems on the Internet VSS implementation Connection to Internet remote places Marcello Maggiora, Antonio Lantieri, Marco Ricca 40 20
Datacenters and Business continuty Marcello Maggiora, Antonio Lantieri, Marco Ricca 41 Datacenters Layout Setup Marcello Maggiora, Antonio Lantieri, Marco Ricca 42 21
Business Continuity 600 m Marcello Maggiora, Antonio Lantieri, Marco Ricca 43 Business Continuity DC1 DC5 LUN A LUN B Sync Marcello Maggiora, Antonio Lantieri, Marco Ricca 44 22
Business Continuity DC1 DC5 LUN A LUN B Sync Marcello Maggiora, Antonio Lantieri, Marco Ricca 45 Application Control Engine (ACE) External DNS Server External Email Users External Email Server FW-POLITO INTERNAL POLITO LAN Local Email Users CLIENT SIDE VLAN CLIENT SIDE VLAN VIP address MX.POLITO.IT VIP address MAIL.POLITO.IT ACE CONTEXT Active ACE CONTEXT Stand-by ACE CONTEXT Active ACE CONTEXT Stand-by VIP address VIP address SERVER SIDE VLAN SERVER SIDE VLAN ANTISPAM ENGINE Marcello Maggiora, Antonio Lantieri, Marco Ricca MAIL BOXES 46 23
Network management Marcello Maggiora, Antonio Lantieri, Marco Ricca 47 Network management Tools mainly relying on SNMP protocols Wireless network management Wired network management Unified management Marcello Maggiora, Antonio Lantieri, Marco Ricca 48 24
Network management Wired network management Marcello Maggiora, Antonio Lantieri, Marco Ricca 49 Wireless management Marcello Maggiora, Antonio Lantieri, Marco Ricca 50 25
Unified management Marcello Maggiora, Antonio Lantieri, Marco Ricca 51 At a glance EDGE CORE ACCESS Marcello Maggiora, Antonio Lantieri, Marco Ricca 52 DISTRIBUTION 26
Marcello Maggiora (marcello.maggiora@polito.it) Antonio Lantieri (antonio.lantieri@polito.it) Marco Ricca (marco.ricca@polito.it) Marcello Maggiora, Antonio Lantieri, Marco Ricca 53 Fault Tolerance Marcello Maggiora, Antonio Lantieri, Marco Ricca 54 27
X 11/01/2016 VSS Failure scenarios (1) Active Supervisor Engine Failure Upon detecting the failure of the active supervisor, the hot-standby supervisor engine performs an SSO switchover and assumes the role of the active supervisor An online insertion and removal (OIR) removed event is simulated for all modules in the previous active chassis to remove those cards from the running chassis inventory If the failed active supervisor engine can reboot after being reset, it now becomes the new hot-standby supervisor engine Marcello Maggiora, Antonio Lantieri, Marco Ricca 55 VSS Failure scenarios (1) Active Supervisor Engine Failure The effect on the data path is that all the modules on the previous active virtual switch chassis are brought down, resulting in a slight traffic disruption for those traffic flows that were destined to the active virtual switch If the vast majority of interfaces in the Cisco Virtual Switching System are multichassis Cisco EtherChannel links, the remote endpoint of the link detects the failure of the active virtual switch ports and uses the links connecting to the standby virtual switch instead Availability is affected for approximately 50 to 200 ms for those traffic flows across the active virtual switch Marcello Maggiora, Antonio Lantieri, Marco Ricca 56 28
X 11/01/2016 VSS Failure scenarios (2) Hot-Standby Supervisor Engine Failure Upon detecting the failure of the host standby supervisor, the active supervisor engine performs an online insertion and removal (OIR) to remove those cards for all modules in the standby chassis The effect on the data path is that all line cards on the standby virtual switch are brought down. Only the flows being forwarded through the standby virtual switch are affected Availability is affected for approximately 50 to 200 ms for those traffic flows across the standby virtual switch Marcello Maggiora, Antonio Lantieri, Marco Ricca 57 VSS Failure scenarios (3) VSL Single-Link Failure The failure of a single VSL link is discovered by the active supervisor engine VSL link will be automatically updated to reflect the removal of a link from the VSL X Availability is not affected for those data flows that do not use the VSL Marcello Maggiora, Antonio Lantieri, Marco Ricca 58 29
VSS Failure scenarios (4) Complete VSL Failure (Dual Active) The active supervisor engine discovers the failure of the VSL. From the perspective of the active virtual switch chassis, the standby virtual switch is lost The standby virtual switch chassis also views the active virtual switch chassis as failed and transitions to active virtual switch state through an SSO switchover X Marcello Maggiora, Antonio Lantieri, Marco Ricca 59 VSS Failure scenarios (4) Complete VSL Failure (Dual Active) Each virtual switch assumes the role as the active virtual switch and each virtual switch controls only its local ports! At Layer 3, any virtual interfaces (for example, port channels, SVIs, loopbacks, etc.) are duplicated on both chassis, causing duplicate IP addresses on the network Any secure communications such as SSH and the cryptography feature set have the same set of keys on both chassis At Layer 2, the spanning tree has the same bridge ID in both switches, possibly causing conflict To avoid this disruptive scenario, we should configure the VSL as a multiple-link port channel and spread it across all the available supervisor engines and modules within the chassis. You should also run the individual members of the VSL across separate physical paths when possible! Marcello Maggiora, Antonio Lantieri, Marco Ricca 60 30