Designing High-Performance Campus Intranets with Multilayer Switching

Transcription

1 Designing High-Performance Campus Intranets with Multilayer Switching Author: Geoff Haviland WHITE PAPER Synopsis This paper briefly compares several approaches to designing campus intranets using multilayer switching. Then it describes the hierarchical approach called multilayer campus network design in greater detail. The multilayer design approach makes optimal use of multilayer switching to build a campus intranet that is scalable, fault tolerant, and manageable. Whether implemented with an Ethernet backbone or an Asynchronous Transfer Mode () backbone, the multilayer model has many advantages. A multilayer campus intranet is highly deterministic, which makes it easy to troubleshoot as it scales. The multilayer design is modular, so bandwidth scales as building blocks are added. Intelligent 3 services keep broadcasts off the backbone. Intelligent 3 routing protocols such as Open Shortest Path First (OSPF) and Enhanced Interior Gateway Routing Protocol (IGRP) handle load balancing and fast convergence across the backbone. The multilayer model makes migration easier, because it preserves existing addressing. Redundancy and fast convergence are provided by UplinkFast and Hot Standby Router Protocol (HSRP). Bandwidth scales from Fast Ethernet to, and from Gigabit Ethernet to Gigabit EtherChannel. The model supports all common campus protocols. The ideas expressed in this paper reflect experience with many large campus intranets. Detailed configuration examples are provided in the appendix to enable readers to implement the multilayer model with either a switched Ethernet backbone or an LAN Emulation (LANE) backbone. Contents Campus Network Design Considerations Flat Bridged Networks Routing and Scalability 2 Switching 3 Switching 4 Switching Virtual LANs and Emulated LANs Comparing Campus Network Design Models Hub and Router Model Campus-Wide VLAN Model Multiprotocol over The Multilayer Model The New 80/20 Rule Components of the Multilayer Model Redundancy and Load Balancing Scaling Bandwidth Policy in the Core Positioning s /LANE Backbone IP Multicast Scaling Considerations Migration Strategies Security Considerations Bridging in the Multilayer Model Benefits of the Multilayer Model Appendix A: Implementing the Multilayer Model Ethernet Backbone Farm LANE Backbone Page 1 of 33

2 Campus Network Design Considerations Flat Bridged Networks Originally campus networks consisted of a single local-area network (LAN) to which new users were added. This LAN was a logical or physical cable into which the network devices tapped. In the case of Ethernet, the half-duplex 10 Mbps available was shared by all the devices. The LAN can be considered a collision domain, because all packets are visible to all devices on the LAN and are therefore free to collide, given the carrier sense multiaccess with collision detection (CSMA/CD) scheme used by Ethernet. When the collision domain of the LAN became congested, a bridge was inserted. A LAN bridge is a store-and-forward packet switch. The bridge segments the LAN into several collision domains, and therefore increases the available network throughput per device. Bridges flood broadcasts, multicasts, and unknown unicasts to all segments. Therefore, all the bridged segments in the campus together form a single broadcast domain. The Spanning Tree Protocol (STP) was developed to prevent loops in the network and to route around failed elements. The following are characteristics of the STP broadcast domain: Redundant links are blocked and carry no data traffic. Suboptimal paths exist between different points. STP convergence typically takes 40 to 50 seconds. Broadcast traffic within the 2 domain interrupts every host. Broadcast storms within the 2 domain affect the whole domain. Isolating problems can be time consuming. Network security within the 2 domain is limited. In theory, the amount of broadcast traffic sets a practical limit to the size of the broadcast domain. In practice, managing and troubleshooting a bridged campus becomes increasingly difficult as the number of users increases. One misconfigured or malfunctioning workstation can disable an entire broadcast domain for an extended period of time. When designing a bridged campus, each bridged segment corresponds to a workgroup. The workgroup server is placed in the same segment as the clients, allowing most of the traffic to be contained. This design principle is referred to as the 80/20 rule and refers to the goal of keeping at least 80 percent of the traffic contained within the local segment. Routing and Scalability A router is a packet switch that is used to create an internetwork or internet, thereby providing connectivity between broadcast domains. Routers forward packets based on network addresses rather than Media Access Control (MAC) addresses. Internets are more scalable than flat bridged networks, because routers summarize reachability by network number. Routers use protocols such as OSPF and Enhanced IGRP to exchange network reachability information. Compared with STP, routing protocols have the following characteristics: Load balancing across many equal-cost paths (in the Cisco implementation) Optimal or lowest-cost paths between networks Fast convergence when changes occur Summarized (and therefore scalable) reachability information In addition to controlling broadcasts, Cisco routers provide a wide range of value-added features that improve the manageability and scalability of campus internets. These features are characteristics of the Cisco IOS software and are common to Cisco routers and multilayer switches. The IOS software has features specific to each protocol typically found in the campus, including the following: TCP/IP AppleTalk DECnet Novell IPX IBM Systems Network Architecture (SNA), data-link switching (DLSw), and Advanced Peer-to-Peer Networking (APPN) When routers are used in a campus, the number of router hops from edge to edge is called the diameter. It is considered good practice to design for a consistent diameter within a campus. This is achieved with a hierarchical design model. Figure 1 shows a typical hierarchical model that combines routers and hubs. The diameter is always two router hops from an end station in one building to an end station in another building. The distance from end station to a server on the backbone Fiber Distributed Data Interface (FDDI) is always one hop. Page 2 of 33

3 2 Switching 2 switching is hardware-based bridging. In particular, the frame forwarding is handled by specialized hardware, usually application-specific integrated circuits (ASICs). 2 switches are replacing hubs at the wiring closet in campus network designs. The performance advantage of a 2 switch compared with a shared hub is dramatic. Consider a workgroup of 100 users in a subnet sharing a single half-duplex Ethernet segment. The average available throughput per user is 10 Mbps divided by 100, or just 100 kbps. Replace the hub with a full-duplex Ethernet switch, and the average available throughput per user is 10 Mbps times two, or 20 Mbps. The amount of network capacity available to the switched workgroup is 200 times greater than to the shared workgroup. The limiting factor now becomes the workgroup server, which is a 10-Mbps bottleneck. The high performance of 2 switching has led to some network designs that increase the number of hosts per subnet. Increasing the hosts leads to a flatter design with fewer subnets or logical networks in the campus. However, for all its advantages, 2 switching has all the same characteristics and limitations as bridging. Broadcast domains built with 2 switches still experience the same scaling and performance issues as the large bridged networks of the past. The broadcast radiation increases with the number of hosts, and broadcasts interrupt all the end stations. The STP limitations of slow convergence and blocked links still apply. 3 Switching 3 switching is hardware-based routing. In particular, the packet forwarding is handled by specialized hardware, usually ASICs. Depending on the protocols, interfaces, and features supported, 3 switches can be used in place of routers in a campus design. 3 switches that support standards-based packet header rewrite and time-to-live (TTL) decrement are called packet-by-packet 3 switches. High-performance packet-by-packet 3 switching is achieved in different ways. The Cisco Gigabit Switch Router (GSR) achieves wire-speed 3 switching with a crossbar switch matrix. The Catalyst family of multilayer switches performs 3 switching with ASICs developed for the Supervisor Engine. Regardless of the underlying technology, Cisco s packet-by-packet 3 switching implementations are standards-compliant and operate as a fast router to external devices. Figure 1 Traditional Router and Hub Campus Workgroup Building A Building B Workgroup Building C Workgroup Access Hubs Hubs Hubs Core Token Ring Port FDDI Port FDDI Backbone Dual Homed FDDI Dual Ring s Page 3 of 33

4 Cisco s 3 switching implementation on the Catalyst family of switches combines the full multiprotocol routing support of the Cisco IOS software with hardware-based 3 switching. The Route Switch Module (RSM) is an IOS-based router with the same Reduced Instruction Set Computing (RISC) processor as the RSP2 engine in the high-end Cisco 7500 router family. The hardware-based 3 switching is achieved with ASICs on the NetFlow feature card. The NetFlow feature card is a daughter-card upgrade to the Supervisor Engine on a Catalyst 5000 family multilayer switch. 4 Switching 4 switching refers to hardware-based routing that considers the application. In Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) flows, the application is encoded as a port number in the packet header. Cisco routers have the ability to control traffic based on 4 information using extended access lists and to provide granular 4 accounting of flows using NetFlow switching. Multilayer switching on the Catalyst family of switches can optionally be configured to operate as a 3 switch or a 4 switch. When operating as a 3 switch, the NetFlow feature card caches flows based on destination IP address. When operating as a 4 switch, the card caches flows based on source address, destination address, source port, and destination port. Because the NetFlow feature card performs 3 or 4 switching in hardware, there is no performance difference between the two modes of operation. Choose 4 switching if your policy dictates granular control of traffic by application or if you require granular accounting of traffic by application. Virtual LANs and Emulated LANs One of the technologies developed to enable 2 switching across the campus is Virtual LANs (VLANs). A VLAN is a way to establish an extended logical network independent of the physical network layout. Each VLAN functions as a separate broadcast domain and has characteristics similar to an extended bridged network. STP normally operates between the switches in a VLAN. Figure 2 Virtual LAN (VLAN) Technologies Workgroup Pink VLAN Y Client Pink VLAN Client Green VLAN Z Workgroup Green VLAN Catalyst Switch X ISL Attached A B C Catalyst 5000 With LANE Card LANE Client (LEC) Pink, Purple, Green D LANE Client (LEC) Pink, Purple, Green Switch Workgroups: Pink Purple Green Page 4 of 33

5 Figure 2 shows three VLANs labeled pink, purple, and green. Each color corresponds to a workgroup, which is also a logical subnet: Pink = Purple = Green = One of the technologies developed to enable campus-wide VLANs is VLAN trunking. A VLAN trunk between two 2 switches allows traffic from several logical networks to be multiplexed. A VLAN trunk between a 2 switch and a router allows the router to connect to several logical networks over a single physical interface. In Figure 2, a VLAN trunk allows server X to talk to all the VLANs simultaneously. The yellow lines in Figure 1 are Inter-Switch Link (ISL) trunks that carry the pink, purple, and green VLANs. ISL, , and 802.1q are VLAN tagging protocols that were developed to allow VLAN trunking. The VLAN tag is an integer incorporated into the header of frames passing between two devices. The tag value allows the data from multiple VLANs to be multiplexed and demultiplexed. LANE permits multiple logical LANs to exist over a single switched infrastructure. Emulated LANs (ELANs) use a similar integer index, because ISL, and 802.1q, and are compatible with Ethernet VLANs from end to end. In Figure 2, LANE cards in Catalyst switches B and C act as LANE clients (LECs) that connect the Ethernet VLANs pink, purple, and green across the backbone. The server D is attached and has LECs for the pink, purple, and green ELANs. Thus server D can talk directly to hosts in the pink, purple, and green VLANs. LANE emulates the Ethernet broadcast protocol over connection-oriented. Not shown in Figure 2 are the LANE Configuration (LECS), LANE (LES), and Broadcast and Unknown (BUS) that are required to make work like Ethernet. The LECS and LES/BUS functions are supported by the Cisco IOS software and can reside in a Cisco LightStream 1010 switch, a Cisco Catalyst 5000 family switch with a LANE card, or a Cisco router with an interface. Ethernet-attached hosts and servers in one VLAN cannot talk to Ethernet-attached hosts and servers in a different VLAN. In Figure 2, client Z in the green VLAN cannot talk to server Y in the pink VLAN. That is because there is no router to connect pink to green. Comparing Campus Network Design Models The Hub and Router Model Figure 1 shows a campus with the traditional router and hub design. The access-layer devices are hubs that act as 1 repeaters. The distribution layer consists of routers. The core layer contains FDDI concentrators or other hubs that act as 1 repeaters. Routers in the distribution layer provide broadcast control and segmentation. Each wiring closet hub corresponds to a logical network or subnet and homes to a router port. Alternatively, several hubs can be cascaded or bridged together to form one logical subnet or network. The hub and router model is scalable because of the advantages of intelligent routing protocols such as OSPF and Enhanced IGRP. The distribution layer is the demarcation between networks in the access layer and networks in the core. -layer routers provide segmentation and terminate collision domains as well as broadcast domains. The model is consistent and deterministic, which simplifies troubleshooting and administration. This model also maps well to all the network protocols such as Novell IPX, AppleTalk, DECnet, and TCP/IP. The hub and router model is straightforward to configure and maintain because of its modularity. Each router within the distribution layer is programmed with the same features. Common configuration elements can be cut and pasted across the layer. Because each router is programmed the same way, its behavior is predictable, which makes troubleshooting easier. 3 packet switching load and middleware services are shared among all the routers in the distribution layer. The traditional hub and router campus model can be upgraded as performance demands increase. The shared media in the access layer and core can be upgraded to 2 switching, and the distribution layer can be upgraded to 3 switching with multilayer switching. Upgrading shared 1 media to switched 2 media does not change the network addressing, the logical design, or the programming of the routers. Page 5 of 33

6 The Campus-Wide VLAN Model Figure 3 shows a conventional campus-wide VLAN design. 2 switching is used in the access, distribution, and core layers. Four workgroups represented by the colors blue, red, purple, and green are distributed across several access-layer switches. Connectivity between workgroups is by Router X that connects to all four VLANs. 3 switching and services are concentrated at Router X. servers are shown behind the router on different logical networks indicated by the black lines. The various VLAN connections to Router X could be replaced by an ISL trunk. In either case, Router X is typically referred to as a router on a stick or a one-armed router. More routers can be used to distribute the load, and each router attaches to several or all VLANs. Traffic between workgroups must traverse the campus in the source VLAN to a port on the gateway router, then back out into the destination VLAN. Figure 3 Traditional Campus-Wide VLAN Design Building A Building B Building C Access X Core s Workgroup Green ISL Attached Four Workgroups Blue Pink Purple Green Page 6 of 33

7 Figure 4 shows an updated version of the campus-wide VLAN model that takes advantage of multilayer switching. The switch marked X is a Catalyst 5000 family multilayer switch. The one-armed router is replaced by a RSM and the hardware-based 3 switching of the NetFlow feature card. servers in the server farm may be attached by Fast Ethernet at 100 Mbps, or by Fast EtherChannel to increase the bandwidth to 200 Mbps FDX or 400 Mbps FDX. The campus-wide VLAN model is highly dependent upon the 80/20 rule. If 80 percent of the traffic is within a workgroup, then 80 percent of the packets are switched at 2 from client to server. However, if 90 percent of the traffic goes to the enterprise servers in the server farm, then 90 percent of the packets are switched by the one-armed router. The scalability and performance of the VLAN model are limited by the characteristics of STP. Each VLAN is equivalent to a flat bridged network. Figure 4 Campus-Wide VLANs with Multilayer Switching Building A Building B Building C Access Core FEC/ISL Workgroup Green Catalyst 5000 Multilayer Switch X Attached s Attached Four Workgroups Blue Pink Purple Green Page 7 of 33

8 The campus-wide VLAN model provides the flexibility to have statically configured end stations move to a different floor or building within the campus. Cisco s VLAN Membership Policy (VMPS) and the VLAN Trunking Protocol (VTP) make this possible. A mobile user plugs a laptop PC into a LAN port in another building. The local Catalyst switch sends a query to the VMPS to determine the access policy and VLAN membership for the user. Then the Catalyst switch adds the user s port to the appropriate VLAN. Multiprotocol over Multiprotocol over (MPOA) adds 3 cut-through switching to LANE. The infrastructure is the same as in LANE. The LECS and the LES/BUS for each ELAN are configured the usual way. Figure 5 shows the elements of a small MPOA campus design. Figure 5 X MPOA Campus Design MultiProtocol Routes First Packet of IP Unicast Flow s Client Green VLAN Attached Client Pink VLAN Access Switch MPC Access Switch MPC MPC With MPOA, the new elements are the multiprotocol client (MPC) hardware and software on the access switches as well as the multiprotocol server (MPS), which is implemented in software on Router X. When the client in the pink VLAN talks to an enterprise server in the server farm, the first packet goes from the MPC in the access switch to the MPS using LANE. The MPS forwards the packet to the destination MPC using LANE. Then the MPS tells the two MPCs to establish a direct switched virtual circuit (SVC) path between the green subnet and the server farm subnet. With MPOA, IP unicast packets take the cut-through SVC as indicated. Multicast packets, however, are sent to the BUS to be flooded in the originating ELAN. Then Router X copies the multicast to the BUS in every ELAN that needs to receive the packet as determined by multicast routing. In turn, each BUS floods the packet again within each destination ELAN. Packets of protocols other than IP always proceed LANE to router to LANE without establishing a direct cut-through SVC. MPOA design must consider the amount of broadcast, multicast, and non-ip traffic in relation to the performance of the router. MPOA should be considered for networks with predominately IP unicast traffic and trunks to the wiring closet switch. The Multilayer Model The New 80/20 Rule The conventional wisdom of the 80/20 rule underlies the traditional design models discussed in the preceding section. With the campus-wide VLAN model, the logical workgroup is dispersed across the campus, but still organized such that 80 percent of traffic is contained within the VLAN. The remaining 20 percent of traffic leaves the network or subnet through a router. The traditional 80/20 traffic model arose because each department or workgroup had a local server on the LAN. The local server was used as file server, logon server, and application server for the workgroup. The 80/20 traffic pattern has been changing rapidly with the rise of corporate intranets and applications that rely on distributed IP services. Page 8 of 33

9 Many new and existing applications are moving to distributed World Wide Web (WWW)-based data storage and retrieval. The traffic pattern is moving toward what is now referred to as the 20/80 model. In the 20/80 model, only 20 percent of traffic is local to the workgroup LAN and 80 percent of the traffic leaves. Components of the Multilayer Model The performance of multilayer switching matches the requirements of the new 20/80 traffic model. The two components of multilayer switching on the Catalyst 5000 family are the RSM and the NetFlow feature card. The RSM is a Cisco IOS-based multiprotocol router on a card. It has performance and features similar to a Cisco 7500 router. The NetFlow feature card is a daughter-card upgrade to the Supervisor Engine of the Catalyst 5000 family switches. It performs both 3 and 2 switching in hardware with specialized ASICs. It is important to note that there is no performance penalty associated with 3 switching versus 2 switching with the NetFlow feature card. Figure 6 illustrates a simple multilayer campus network design. The campus consists of three buildings, A, B, and C, connected by a backbone called the core. The distribution layer consists of Catalyst 5000 family multilayer switches. The multilayer design takes advantage of the 2 switching performance and features of the Catalyst family switches in the access layer and backbone and uses multilayer switching in the distribution layer. The multilayer model preserves the existing logical network design and addressing as in the traditional hub and router model. Access-layer subnets terminate at the distribution layer. From the other side, backbone subnets also terminate at the distribution layer. So the multilayer model does not consist of campus-wide VLANs, but does take advantage of VLAN trunking as we shall see. Figure 6 Multilayer Campus Design with Multilayer Switching Attached Workstation Building A Building B Building C Access Catalyst 5000 L2 Switch Catalyst 5000 Multilayer Switch Core Catalyst 5000 L2 Switch ISL Attached Building Attached Page 9 of 33

10 Because 3 switching is used in the distribution layer of the multilayer model, this is where many of the characteristic advantages of routing apply. The distribution layer forms a broadcast boundary so that broadcasts don t pass from a building to the backbone or vice-versa. Value-added features of the Cisco IOS software apply at the distribution layer. For example, the distribution-layer switches cache information about Novell servers and respond to Get Nearest queries from Novell clients in the building. Another example is forwarding Dynamic Host Configuration Protocol (DHCP) messages from mobile IP workstations to a DHCP server. Another Cisco IOS feature that is implemented at the multilayer switches in the distribution layer is called Local Area Mobility (LAM). LAM is valuable for campus intranets that have not deployed DHCP services and permits workstations with statically configured IP addresses and gateways to move throughout the campus. LAM works by propagating the address of the mobile hosts out into the 3 routing table. There are actually hundreds of valuable Cisco IOS features that improve the stability, scalability, and manageability of enterprise networks. These features apply to all the protocols found in the campus, including DECnet, AppleTalk, IBM SNA, Novell IPX, TCP/IP, and many others. One characteristic shared by most of these features is that they are out of the box. Out-of-the-box features apply to the functioning of the network as a whole. They are in contrast with inside-the-box features, such as port density or performance, that apply to a single box rather than to the network as a whole. Inside-the-box features have little to do with the stability, scalability, or manageability of enterprise networks. The greatest strengths of the multilayer model arise from its hierarchical and modular nature. It is hierarchical because the layers are clearly defined and specialized. It is modular because every part within a layer performs the same logical function. One key advantage of modular design is that different technologies can be deployed with no impact on the logical structure of the model. For example, Token Ring can be replaced by Ethernet. FDDI can be replaced by switched Fast Ethernet. Hubs can be replaced by 2 switches. Fast Ethernet can be substituted with LANE. LANE can be substituted with Gigabit Ethernet, and so on. So modularity makes both migration and integration of legacy technologies much easier. Another key advantage of modular design is that each device within a layer is programmed the same way and performs the same job, making configuration much easier. Troubleshooting is also easier, because the whole design is highly deterministic in terms of performance, path determination, and failure recovery. In the access layer a subnet corresponds to a VLAN. A VLAN may map to a single 2 switch, or it may appear at several switches. Conversely, one or more VLANs may appear at a given 2 switch. If Catalyst 5000 family switches are used in the access layer, VLAN trunking provides flexible allocation of networks and subnets across more than one switch. In our later examples we will show two VLANs per switch in order to illustrate how to use VLAN trunking to achieve load balancing and fast failure recovery between the access layer and the distribution layer. In its simplest form, the core layer is a single logical network or VLAN. In our examples, we show the core layer as a simple switched 2 infrastructure with no loops. It is advantageous to avoid spanning tree loops in the core. Instead we will take advantage of the load balancing and fast convergence of 3 routing protocols such as OSPF and Enhanced IGRP to handle path determination and failure recovery across the backbone. So all the path determination and failure recovery is handled at the distribution layer in the multilayer model. Redundancy and Load Balancing A distribution-layer switch in Figure 6 represents a point of failure at the building level. One thousand users in Building A could lose their connections to the backbone in the event of a power failure. If a link from a wiring closet switch to the distribution-layer switch is disconnected, 100 users on a floor could lose their connections to the backbone. Figure 7 shows a multilayer design that addresses these issues. Multilayer switches A and B provide redundant connectivity to domain North. Redundant links from each access-layer switch connect to distribution-layer switches A and B. Redundancy in the backbone is achieved by installing two or more Catalyst switches in the core. Redundant links from the distribution layer provide failover as well as load balancing over multiple paths across the backbone. Page 10 of 33

11 Redundant links connect access-layer switches to a pair of Catalyst multilayer switches in the distribution layer. Fast failover at 3 is achieved with Cisco s Hot Standby Router Protocol. The two distribution-layer switches cooperate to provide HSRP gateway routers for all the IP hosts in the building. Fast failover at 2 is achieved by Cisco s UplinkFast feature. UplinkFast is a convergence algorithm that achieves link failover from the forwarding link to the backup link in about three seconds. Load balancing across the core is achieved by intelligent 3 routing protocols implemented in the Cisco IOS software. In this picture there are four equal-cost paths between any two buildings. In Figure 7, the four paths from domain North to domain West are AXC, AXYD, BYD, and BYXC. These four 2 paths are considered equal by 3 routing protocols. Note that all paths from domains North, West, and South to the backbone are single, logical hops. The Cisco IOS software supports load balancing over up to six equal-cost paths for IP, and over many paths for other protocols. Figure 7 Redundant Multilayer Campus Design North West South Access A B C D Core X Y ISL Attached Building s s Attached Page 11 of 33

12 Figure 8 shows the redundant multilayer model with an enterprise server farm. The server farm is implemented as a modular building block using multilayer switching. The Gigabit Ethernet trunk labeled A carries the server-to-server traffic. The trunk labeled B carries the backbone traffic. All server-to-server traffic is kept off the backbone, which has both security and performance advantages. The enterprise servers have fast HSRP redundancy between the multilayer switches X and Y. Access policy to the server farm can be controlled by access lists on X and Y. In Figure 8, the 2 core switches V and W are shown separate from server distribution switches X and Y for clarity. In a network of this size, V and W would collapse into X and Y. Putting servers in a server farm also avoids problems associated with IP redirect and selecting the best gateway router when servers are directly attached to the backbone subnet as shown in Figure 7. In particular, HSRP would not be used for the enterprise servers in Figure 7; they would use proxy Address Resolution Protocol (ARP), Internet Router Discovery Protocol (IRDP), Gateway Discovery Protocol (GDP), or Routing Information Protocol (RIP) snooping to populate their routing tables. Figure 8 Multilayer Model with Farm North West South Access Core V B W ISL Attached Building s Gigabit Ethernet Gigabit Ethernet X A Y Attached s Attached Gigabit Ethernet/GEC Page 12 of 33

13 Figure 9 shows HSRP operating between two distribution-layer switches. Host systems connect at a switch port in the access layer. The even-numbered subnets map to even-numbered VLANs, and the odd-numbered subnets map to odd-numbered VLANs. The HSRP primary for the even-numbered subnets is distribution-layer Switch X, and the HSRP primary for the odd-numbered subnets is Switch Y. The HSRP backup for even-numbered subnets is Switch Y, and the HSRP backup for odd-numbered subnets is Switch X. The convention followed here is that every HSRP gateway router always has host address 100 so the HSRP gateway for subnet 15.0 is If gateway loses power or is disconnected, Switch X assumes the address as well as the HSRP MAC address within about two seconds as measured in the configuration shown in Appendix A. Figure 10 shows load balancing between the access layer and the distribution layer using Cisco s ISL VLAN trunking protocol. We have allocated VLANs 10 and 11 to access-layer Switch A, and VLANs 12 and 13 to Switch B. Each access-layer switch has two trunks to the distribution layer. The STP puts redundant links in blocking mode as shown. Load distribution is achieved by making one trunk the active forwarding path for even-numbered VLANs and the other trunk the active forwarding path for odd-numbered VLANs. Figure 10 VLANs 10, 11 VLAN Trunking for Load Balancing VLANs 12, 13 VLANs 14, 15 A B C D VLANs 16, 17 Figure 9 Host A 10.1 Even Subnet 10.0 Gateway Redundancy with HSRP Host B 11.1 Odd Subnet 11.0 Gateway Host C 15.1 Odd Subnet 15.0 Gateway Host D 17.1 Odd Subnet 17.0 Gateway F 10 B 11 F 11 B 10 F 12 B 13 F 13 B 12 F 14 B 15 F 15 B 14 F 16 B 17 F 17 B 16 ISL Trunks VLAN Multiplexing Fast Ethernet or Access STP Root Even VLANs 10, 12, 14, 16 X Y STP Root Odd VLANs 11, 13, 15, 17 ISL Trunks VLAN Multiplexing Fast Ethernet or HSRP Primary Even Subnets, Even VLANs, 10, 12, 14, 16 X Y HSRP Primary Odd Subnets, Odd VLANs, 11, 13, 15, 17 F Forwarding B Blocking On Switch A, the left-hand trunk is labeled F10, which means it s the forwarding path for VLAN 10. The right-hand trunk is labeled F11, which means it s the forwarding path for VLAN 11. The left-hand trunk is also labeled B11, which means it s the blocking path for VLAN 11, and the right-hand trunk is B10, which means blocking for VLAN Page 13 of 33

14 10. This is accomplished by making X the root for even VLANs and Y the root for odd VLANs. See Appendix A for the configuration commands required. Figure 11 shows Figure 10 after a link failure, which is indicated by the big X. UplinkFast changes the left-hand trunk on Switch A to be the active forwarding path for VLAN 11. Traffic is switched across trunk Z if required. Trunk Z is the 2 backup path for all VLANs in the domain, and also carries some of the return traffic that is load-balanced between Switch X and Switch Y. With conventional STP, convergence would take 40 to 50 seconds. With UplinkFast, failover takes about three seconds as measured in the configuration shown in Appendix A. Figure 11 VLAN Trunking with Uplink Fast Failover VLANs 10, 11 VLANs 12, 13 VLANs 14, 15 A B C D VLANs 16, 17 combines two or four Fast Ethernet links together into a single high-capacity trunk. Fast EtherChannel is supported by the Cisco 7500 family routers with IOS Release CA and above. It is supported on Catalyst 5000 switches with the line card or on Supervisor II or III. support has been announced by several partners, including Adaptec, Auspex, Compaq, Hewlett-Packard, Intel, Sun Microsystems, and Znyx. With trunking, a high-capacity server can be connected to the core backbone at 400 Mbps FDX for 800 Mbps total throughput. Figure 12 shows three ways to scale bandwidth between an access-layer switch and a distribution-layer switch. On the configuration labeled A - Best, all VLANs are combined over with ISL. In the middle configuration labeled B Good, a combination of segmentation and ISL trunking is used. On the configuration labeled C OK, simple segmentation is used. F 10 B 11 X F 12 B 13 F 13 B 12 F 14 B 15 F 15 B 14 F 16 B 17 F 17 B 16 ISL Trunks VLAN Multiplexing Fast Ethernet or Figure 12 Scaling Ethernet Trunk Bandwidth A Best VLANs 1, 2, 3, 4, 5, 6 B Good C OK VLANs VLANs 1, 2, 3, 4, 5, STP Root Even VLANs 10, 12, 14, 16 F Forwarding B Blocking X Z Y STP Root Odd VLANs 11, 13, 15, 17 FEC ISL VLANs 1, 2, 3, 4, 5, Mbps FDX Fast 1,2 3,4 5,6 Ethernet Fast Ethernet ISL Scaling Bandwidth Ethernet trunk capacity in the multilayer model can be scaled in several ways. Ethernet can be migrated to Fast Ethernet. Fast Ethernet can be migrated to or Gigabit Ethernet or Gigabit EtherChannel. Access-layer switches can be partitioned into multiple VLANs with multiple trunks. VLAN multiplexing with ISL can be used in combination with the different trunks. You should use model A if possible, because Fast EtherChannel provides more efficient bandwidth utilization by multiplexing traffic from multiple VLANs over one trunk. If a line card is not available, use model B if possible. If neither nor ISL trunking are possible, use model C. With simple segmentation, each VLAN uses one trunk, so one can be congested while another is unused. More ports will be required to get the same performance. Page 14 of 33

15 Scale bandwidth within backbones by adding more OC-3 or OC-12 trunks as required. The intelligent routing provided by Private Network-to-Network Interface (PNNI) handles load balancing and fast failover. Policy in the Core With 3 switching in the distribution layer, it is possible to implement the backbone as a single logical network or multiple logical networks as required. VLAN technology can be used to create separate logical networks that can be used for different purposes. One IP core VLAN could be created for management traffic and another for enterprise servers. A different policy could be implemented for each core VLAN. Policy is applied with access lists at the distribution layer. In this way, access to management traffic and management ports on network devices is carefully controlled. Another way to logically partition the core is by protocol. Create one VLAN for enterprise IP servers and another for enterprise IPX or DECnet servers. The logical partition can be extended to become complete physical separation on multiple core switches if dictated by security policies. Figure 13 shows the core separated physically into two switches. VLAN 100 on Switch V corresponds to IP subnet where the World Wide Web (WWW) server farm attaches. VLAN 200 on Switch W corresponds to IPX network BEEF0001 where the Novell server farm attaches. Of course the simpler the backbone topology, the better. A small number of VLANs or ELANs is preferred. A discussion of the scaling issues related to large numbers of 3 switches peered across many networks appears later in this paper in Scaling Considerations. Positioning s It is very common for an enterprise to centralize servers. In some cases, services are consolidated into a single server. In other cases, servers are grouped at a data center for physical security or easier administration. At the same time, it is increasingly common for workgroups or individuals to publish a Web page locally and make it accessible to the enterprise. With centralized servers directly attached to the backbone, all client/server traffic crosses one hop from a subnet in the access layer to a subnet in the core. Policy-based control of access to enterprise servers is implemented by access lists applied at the distribution layer. In Figure 14, server W is Fast Ethernet-attached to the core subnet. X is -attached to the core subnet. As mentioned, servers attached directly to the core must use proxy ARP, IRDP, GDP, or RIP snooping to populate their routing tables. HSRP would not be used within core subnets, because switches in the distribution layer all connect to different parts of the campus. Figure 13 Logical or Physical Partitioning of the Core Domain A Domain B Domain C Workgroup s Core VLAN 100 IP Subnet V W VLAN 200 IPX Network BEEF0001 X IP s World Wide Web Y Novell IPX File s Page 15 of 33

16 servers Y and Z are placed in a server farm by implementing multilayer switching in a server distribution building block. Y is Fast Ethernet-attached, and server Z is Fast EtherChannel-attached. Policy controlling access to these servers is implemented with access lists on the core switches. Another big advantage of the server distribution model is that HSRP can be used to provide redundancy with fast failover. The server distribution model also keeps all server-to-server traffic off the backbone. See Appendix A for a sample configuration that shows how to implement a server farm. M is within workgroup D, which corresponds to one VLAN. M is Fast Ethernet-attached at a port on an access-layer switch, because most of the traffic to the server is local to the workgroup. This follows the conventional 80/20 rule. M could be hidden from the enterprise with an access list at the distribution layer switch H if required. N attaches to a distribution layer at switch H. N is a building-level server that communicates with clients in VLANs A, B, C, and D. A direct 2 switched path between server N and clients in VLANs A, B, C, and D can be achieved in two ways. With four network interface cards (NICs), it can be directly attached to each VLAN. With an ISL NIC, server N can talk directly to all four VLANs over a VLAN trunk. N can be selectively hidden from the rest of the enterprise with an access list on distribution layer switch H if required. /LANE Backbone Figure 15 shows the multilayer campus model with LANE in the backbone. For customers that require guaranteed quality of service (QoS), is a good alternative. Real-time voice and video applications may mandate features like per-flow queuing, which provides granular control of delay and jitter. Figure 14 Attachment in the Multilayer Model VLANs A, B, C, D Access H N A, B, C, D M Workgroup D Core W X HSRP Fast EtherAttached s Y Z HSRP Attached Gigabit Ethernet Page 16 of 33

17 Each Catalyst 5000 multilayer switch in the distribution layer is equipped with a LANE card. The LANE card acts as LEC so that the distribution-layer switches can communicate across the backbone. The LANE card has a redundant OC-3 physical interface called dual-phy. In Figure 15, the solid lines represents the active link and the dotted lines represents the hot-standby link. Two LightStream 1010 switches form the core. Routers and servers with native interfaces attach directly to ports in the backbone. servers in the server farm attach to multilayer Catalyst 5000 switches X and Y. s may be Fast Ethernet- or Fast EtherChannel-attached. These Catalyst 5000 switches are also equipped with LANE cards and act as LECs that connect Ethernet-based enterprise servers to the ELAN in the core. The trunks between the two LightStream 1010 core switches can be OC-3 or OC-12 as required. The PNNI protocol handles load balancing and intelligent routing between the switches. Intelligent routing is increasingly important as the core scales up from two switches to many switches. STP is not used in the backbone. Intelligent 3 routing protocols such as OSPF and Enhanced IGRP manage path determination and load balancing between distribution-layer switches. Cisco has implemented the mple Redundancy Protocol (SSRP) to provide redundancy of the LECS and the LES/BUS. SSRP is available on Cisco 7500 routers, Catalyst 5000 family switches, and LightStream 1010 switches and is compatible with all LANE 1.0 standard LECs. Figure 15 Multilayer Model with LANE Core Domain A Building A Domain B Building B Domain C Building C OC-3 or OC-12 Uplinks Core LANE LightStream 1010 Switch LECS Backup B LightStream 1010 Switch LECS Backup Cisco 7500 LECS Primary Catalyst 5000 LES/BUS Primary Fast Ethernet-Attached s X Y Catalyst 5000 LES/BUS Backup Attached Page 17 of 33

18 The LANE card for the Catalyst 5000 family is an efficient BUS with broadcast performance of 120 kpps. This is enough capacity for the largest campus networks. In Figure 15 we place the primary LES/BUS on Switch X and the backup LES/BUS on Switch Y. For a small campus SSRP, LES/BUS failover takes only a few seconds. For a very large campus, LES/BUS failover can take several minutes. In large campus designs, dual ELAN backbones are frequently used to provide fast convergence in the event of a LES/BUS failure. As an example, two ELANs, Red and Blue, are created in the backbone. If the LES/BUS for ELAN Red is disconnected, traffic is quickly rerouted over ELAN Blue until ELAN Red recovers. After ELAN Red recovers, the multilayer switches in the distribution layer reestablish contact across ELAN Red and start load balancing between Red and Blue again. This process applies to routed protocols but not bridged protocols. The primary and backup LECS database is configured on the LightStream 1010 switches because of their central position. When the ELAN is operating in steady state, there is no overhead CPU utilization on the LECS. The LECS is only contacted when a new LEC joins an ELAN. For this reason, there are few performance considerations associated with placing the primary and backup LECS. A good choice for a primary LECS would be a Cisco 7500 router with direct attachment to the backbone, because it would not be affected by signaling traffic in the event of a LES/BUS failover. Figure 16 shows an alternative implementation of the LANE core using the Catalyst 5500 switch. Here the Catalyst 5500 operates as an switch with the addition of the Switch Processor (ASP) card. It is configured as a LEC with the addition of the OC-12 LANE/MPOA card. It is configured as an Ethernet frame switch with the addition of the appropriate Ethernet or Fast Ethernet line cards. The server farm is implemented with the addition of multilayer switching. The Catalyst 5500 combines the functionality of the LightStream 1010 and the Catalyst 5000 in a single chassis. See Appendix A for an example of configuring an backbone with Catalyst 5500 multilayer switches. Figure 16 LANE Core with Catalyst 5500 Switches Domain A Building A Domain B Building B Domain C Building C Core LANE Catalyst 5500 LES/BUS Primary Catalyst 5500 LES/BUS Backup OC-3 or OC-12 Uplinks s Attached Page 18 of 33

19 IP Multicast Applications based on IP multicast represent a small but rapidly growing component of corporate intranets. Applications such as IPTV, Microsoft NetShow, and NetMeeting are being tried and deployed. There are several aspects to handling multicasts effectively: Multicast routing, Protocol Independent Multicast (PIM) dense mode and sparse mode Clients and servers join multicast groups with Internet Group Management Protocol (IGMP) Pruning multicast trees with Cisco Group Multicast Protocol (CGMP) or IGMP snooping Switch and router multicast performance Multicast policy The preferred routing protocol for multicast is PIM. PIM sparse mode is described in RFC 2117, and PIM dense mode is on the standards track. PIM is being widely deployed in the Internet as well as in corporate intranets. As its name suggests, PIM works with various unicast routing protocols such as OSPF and Enhanced IGRP. PIM routers may also be required to interact with the Distance Vector Multicast Routing Protocol (DVMRP). DVMRP is a legacy multicast routing protocol deployed in the Internet multicast backbone (MBONE). Currently 50 percent of the MBONE has converted to PIM, and it is expected that PIM will replace DVMRP over time. PIM can operate in dense mode or in sparse mode. Dense-mode operation is used for an application like IPTV where there is a multicast server with many clients throughout the campus. Sparse-mode operation is used for workgroup applications like NetMeeting. In either case, PIM builds efficient multicast trees that minimize the amount of traffic on the network. This is particularly important for high-bandwidth applications such as real-time video. In most environments, PIM is configured as sparse-dense and automatically uses either sparse mode or dense mode as required. IGMP is used by multicast clients and servers to join or advertise multicast groups. The local gateway router makes a multicast available on subnets with active listeners, but blocks the traffic if no listeners are present. CGMP extends multicast pruning down to the Catalyst switch. A Cisco router sends out a CGMP message to advertise all the host MAC addresses that belong to a multicast group. Catalyst switches receive the CGMP message and forward multicast traffic only to ports with the specific MAC address in the forwarding table. This blocks multicast packets from all switch ports that don t have group members downstream. The Catalyst 5000 family of switches have an architecture that forwards multicast streams to one port, many ports, or all ports with no performance penalty. Catalyst switches will support one or many multicast groups operating at wire speed concurrently. One way to implement multicast policy is to place multicast servers in a server farm behind multilayer Catalyst Switch X as shown in Figure 17. Switch X acts as a multicast firewall that enforces rate limiting and controls access to multicast sessions. To further isolate multicast traffic, create a separate multicast VLAN/subnet in the core. The multicast VLAN in the core could be a logical partition of existing core switches or a dedicated switch if traffic is very high. Switch X is a logical place to implement the PIM rendezvous point. The rendezvous point is like the root of the multicast tree. Figure 17 Multicast Firewall and Backbone Clients for Multicast A Only A Only Multicast VLAN 100 IP Subnet Clients for Multicast B Only X B Only Gigabit Ethernet C Only Multicast Farm Clients for Multicast C Only Unicast VLAN 200 IP Subnet Unicast Farm Page 19 of 33

20 Scaling Considerations The multilayer design model is inherently scalable. 3 switching performance scales because it is distributed. Backbone performance scales as you add more links or more switches. The individual switch domains or buildings scale to over 1000 client devices with two distribution-layer switches in a typical redundant configuration. More building blocks or server blocks can be added to the campus without changing the design model. Because the multilayer design model is highly structured and deterministic, it is also scalable from a management and administration perspective. In all the multilayer designs discussed, we have avoided STP loops in the backbone. STP takes 40 to 50 seconds to converge and does not support load-balancing across multiple paths. Within Ethernet backbones, no loops are configured. For backbones, PNNI handles load balancing. In all cases, intelligent 3 routing protocols such as OSPF and Enhanced IGRP handle path determination and load balancing over multiple paths in the backbone. OSPF overhead in the backbone rises linearly as the number of distribution-layer switches rises. This is because OSPF elects one designated router and one backup designated router to peer with all the other 3 switches in the distribution layer. If two VLANs or ELANs are created in the backbone, a designated router and a backup are elected for each. So the OSPF routing traffic and CPU overhead increase as the number of backbone VLANs or ELANs increases. For this reason, it is recommended to keep the number of VLANs or ELANs in the backbone small. For large /LANE backbones, it is recommended to create two ELANs in the backbone as was discussed in the / LANE Backbone section earlier in this paper. Another important consideration for OSPF scalability is summarization. For a large campus, make each building an OSPF area and make the distribution-layer switches area border routers (ABRs). Pick all the subnets within the building from a contiguous block of addresses and summarize with a single summary advertisement at the ABRs. This reduces the amount of routing information throughout the campus and increases the stability of the routing table. Enhanced IGRP can be configured for summarization in the same way. Not all routing protocols are created equal, however. AppleTalk Routing Table Maintenance Protocol (RTMP), Novell Advertisement Protocol (SAP), and Novell Routing Information Protocol (RIP) are protocols with overhead that increases as the square of the number of peers. For example, say there are 12 distribution-layer switches attached to the backbone and running Novell SAP. If there are 100 SAP services being advertised throughout the campus, each distribution switch injects 100/7 = 15 SAP packets into the backbone every 60 seconds. All 12 distribution-layer switches receive and process 12 * 15 = 180 SAP packets every 60 seconds. The Cisco IOS software provides features such as SAP filtering to contain SAP advertisements from local servers where appropriate. The 180 packets is a reasonable number, but consider what happens with 100 distribution-layer switches advertising 1000 SAP services. Figure 18 shows a design for a large hierarchical, redundant campus backbone. The core designated B consists of eight LightStream 1010 switches with a partial mesh of OC-12 trunks. Domain C consists of three pairs of LightStream 1010 switches. Domain C can be configured with an prefix address that is summarized where it connects to the core B. On this scale, manual address summarization would have little benefit. The default summarization would have just 26 routing entries corresponding to the 26 switches in Figure 18. In domain A, pairs of distribution-layer switches attach to the fabric with OC-3 LANE. A server farm behind Catalyst switches X and Y attaches directly to the core with OC-12 LANE/MPOA cards. Page 20 of 33