How to Build a Data Center Network With QFabric Products Acting as a Layer 3 switch

Transcription

1 IMPLEMENTATION GUIDE Designing a Layer 3 Data Center Network with the QFabric Architecture How to Build a Data Center Network With QFabric Products Acting as a Layer 3 switch Although Juniper Networks has attempted to provide accurate information in this guide, Juniper Networks does not warrant or guarantee the accuracy of the information provided herein. Third party product descriptions and related technical details provided in this document are for information purposes only and such products are not supported by Juniper Networks. All information provided in this guide is provided as is, with all faults, and without warranty of any kind, either expressed or implied or statutory. Juniper Networks and its suppliers hereby disclaim all warranties related to this guide and the information contained herein, whether expressed or implied of statutory including, without limitation, those of merchantability, fitness for a particular purpose and noninfringement, or arising from a course of dealing, usage, or trade practice. Copyright 2011, Juniper Networks, Inc. 1

2 Table of Contents Introduction Scope Design Considerations QFabric Basics Node Groups QFabric Configuration Defining Node Groups Example 1: SNG configuration Example 2: RSNG configuration Example 3: NNG configuration Interface Naming Conventions for QFabric Architecture Interface Type Configuration Access Port Trunk Port Routed Interface Layer 3 LAG Configuration VLAN Configuration Trunk Port Design Use Cases Connecting Layer 3 Device to QFabric Architecture Route Lookup and Forwarding Decisions QFabric and VRRP Layer 3 Design Use Cases Use Case 1: Static Default Route Configuration Use Case 2: Putting QFabric Architecture into an OSPF Area Use Case 3: Putting QFabric Architecture into OSPF Stub Area Use Case 4: Connecting One-Armed SRX Series Device as Active/Active with QFabric Architecture Use Case 5: Connecting One-Armed SRX Series as Active/Backup with QFabric Architecture Use Case 6: Connecting One-Armed SRX Series Gateway to QFabric Architecture (VRF-Based Steering Mode) Use Case 7: QFabric Architecture Back-to-Back Extension with L3 LAG Summary About Juniper Networks Copyright 2011, Juniper Networks, Inc.

3 Table of Figures Figure 1: Juniper s data center solution with QFabric architecture, MX Series, SRX Series, vgw Virtual Gateway, and Junos Space Figure 2: QFabric logical and physical configuration Figure 3: LAG support between node groups Figure 4: Different types of redundancy for rack servers Figure 5: Different deployment scenarios with embedded blade switches in blade chassis Figure 6: Layer 3 devices can be located anywhere in the QFabric architecture Figure 7: NNG connecting to MX Series with LAG Figure 8: QFabric technology in OSPF area Figure 9: SRX Series one-armed deployment in a two-tier architecture Figure 10: One-armed SRX Series active/active deployment with QFabric technology Figure 11: One-armed SRX Series active/active deployment with QFabric architecture Figure 12: Applying security policy to inter-vrf routing on QFabric architecture Figure 13: Back-to-back extension with LAG Copyright 2011, Juniper Networks, Inc. 3

4 Introduction As people become more adept at employing virtualization technologies, and as applications become more efficient, the need for a high-performance and scalable data center infrastructure becomes increasingly critical. Today s data center network architecture has too many layers and is too rigid to meet those requirements. Juniper has developed a new technology called Juniper Networks QFabric architecture that addresses the inefficiencies of legacy data center networks. QFabric technology eliminates network complexity by reducing the number of switch layers and managed devices, while providing optimal network utilization and a pay-as-you-grow model that doesn t compromise overall network performance. Scope This document will discuss the design of a data center network where QFabric architecture acts as the Layer 3 switch. It will describe the overall network topology and provide relevant configuration templates for QFabric solutions. The target audiences for this document are architects, network engineers or operators, and individuals who require technical knowledge, although every effort has been made to make this document appeal to the widest possible audience. It is assumed that the reader is familiar with Juniper Networks Junos operating system and is knowledgeable about the QFabric family of products. Also, reading the Designing a Layer 2 Data Center Network with the QFabric Architecture implementation guide is highly recommended. Design Considerations One of the biggest challenges with today s data center is keeping the network simple while enabling it to grow without making uncomfortable trade-offs. Adding new switches is the typical response to network growth, but that means more devices to manage and, more importantly, a potentially negative impact on network performance due to switch locations. Juniper Networks has introduced QFabric technology to address these challenges. QFabric technology has the unique ability to reduce complexity by flattening the network to a single tier, providing any-to-any connectivity that ensures every device is no more than a single hop away from any other device. Increasing port counts with QFabric architecture does not increase complexity or add devices to manage, since all QFabric solution components are managed as a single device. 4 Copyright 2011, Juniper Networks, Inc.

5 Remote Data Center MX Series SRX5800 SRX Series vgw VMware vsphere Servers NAS FC Storage Figure 1: Juniper s data center solution with QFabric architecture, MX Series, SRX Series, vgw Virtual Gateway, and Junos Space. QFabric Basics Juniper Networks QFabric architecture is composed of three components: QFabric Director, QFabric Interconnect, and QFabric Node. Each component plays a vital role. The QFabric Director functions as a Routing Engine (RE) in a modular switch, where it is responsible for managing the overall QFabric system as well as distributing forwarding tables to the QFabric Nodes and QFabric Interconnects. The QFabric Interconnect is equivalent to a fabric, acting like the backplane of the switch and providing a simple, high-speed transport that interconnects all of the QFabric Nodes in a full-mesh topology to provide any-to-any port connectivity. The QFabric Node is equivalent to a line card, providing an intelligent edge that can perform routing and switching between connected devices. QFabric Interconnect CPE QFabric Director QFabric Node #1 QFabric Node #2 QFabric QFabric Node #3 Node #128 Figure 2: QFabric logical and physical configuration Copyright 2011, Juniper Networks, Inc. 5

6 Node Groups A node group is nothing more than an abstraction of a single or set of QFabric Nodes that are logically grouped with similar attributes. Node groups are not bound by physical location but by common traits. There are three different types of QFabric Nodes: server node group (SNG), redundant server node group (RSNG), and network node group (NNG). SNG is a single QFabric Node that is connected to servers, blade chassis, and storage devices (it may also be referred to as host-facing ports). Typically, host devices require a subset of protocols 1 such as Link Aggregation Control Protocol (LACP) and Link Layer Discovery Protocol (LLDP). Therefore, SNGs will only need to support host type protocols. Layer 2 or Layer 3 networking protocols 2 such as Spanning Tree Protocol (xstp) and OSPF are not supported and cannot be configured on SNG ports. RSNG is similar to SNG with a couple of differences. First, an RSNG requires two QFabric Nodes to be grouped. Second, it can support cross-member (node) link aggregation groups (LAGs), as shown in Figure 3. NNG is a set of QFabric Nodes connected to WAN routers, other networking devices, or service appliances such as firewalls or server load balancers. Because such devices will be connected to an NNG, all protocol stacks are available on these ports. The QFabric architecture requires at least one QFabric Node to be a member of an NNG (up to eight devices are allowed). While defined as an NNG, it does not limit connections to service appliances or networking devices; server and/or storage devices can also connect to an NNG. SNG RSNG NNG QFabric Node QFabric Node QFabric Node QFabric Node QFabric Node QFabric Node Figure 3: LAG support between node groups Table 1: Node Groups Support Matrix Node Groups Max. Number of Members per Node Group Max. Number of Node Groups Within the QFabric Architecture Same Member LAG Cross-Member LAG (Active/ Active) Support Host-Facing Protocols 3 Support Networking- Facing Protocols 4 Single node group (SNG) Redundant server node group (RSNG) Network node group (NNG) QFabric Configuration This document will not go over the deployment or bring-up of the system. It is assumed that the QFabric architecture has already been brought up by a certified specialist and is ready to be configured. This section will cover how to define node groups and how to configure port types (access or trunk), VLANs, LAGs, and VLAN membership. All management and configuration is done through the QFabric Director. There is no need to go into individual QFabric devices and configure them. The entire QFabric architecture can be managed from a single IP address that is shared by the QFabric Directors. 1 Host-facing protocols are LLDP, LACP, Address Resolution Protocol (ARP), Internet Group Management Protocol (IGMP) Snooping, Data Center Bridging (DCBx). 2 Network-facing protocols are xstp, OSPF, L3 unicast and multicast protocols, and IGMP. 3 Host-facing protocols are LLDP, LACP, ARP, IGMP Snooping, DCBx. 4 Network-facing protocols are xstp, L3 unicast and multicast protocols, and IGMP. 6 Copyright 2011, Juniper Networks, Inc.

7 Defining Node Groups Node groups are a new concept for the Junos operating system and are only relevant to QFabric technology. Therefore, a new stanza has been introduced to help manage QFabric Nodes and node groups. By default, all QFabric Nodes are identified by serial number. Serial numbers can be easily managed with a spreadsheet, and it is not humanly possible to manage without one. QFabric Nodes can be aliased with a more meaningful name, such as the physical location of the QFabric Node (row and rack), as shown with the example below. [edit fabric] netadmin@qfabric# set aliases node-device ABCD1230 row1-rack1 Just as in configuration mode, fabric has been introduced into the operational command to provide QFabric architecture-related administrative show commands. Below is an example of a serial number-to-alias assignment. The Connection and Configuration columns provide the current state of the QFabric Node. netadmin@qfabric> show fabric administration inventory node-devices Item Identifier Connection Configuration Node device row1-rack1 ABCD1230 Connected Configured row1-rack2 ABCD1231 Connected Configured row1-rack3 ABCD1232 Connected Configured row21-rack1 ABCD1233 Connected Configured QFabric Nodes even single devices need to be assigned to a node group. Any arbitrary name can be assigned to an xsng. NNG is the exception to this rule, as it already has a name (NW-NG-0) which cannot be changed. A QFabric Node can only be part of one node group type; it cannot be part of two different node groups. Typically members within node groups are close in proximity, but that is not a requirement. Members of a node group can be in different parts of the data center. Example 1: SNG configuration [edit fabric] netadmin@qfabric# set resources node-group SNG-1 node-device row1-rack1 Example 2: RSNG configuration [edit fabric] netadmin@qfabric# set resources node-group RSNG-1 node-device row1-rack2 netadmin@qfabric# set resources node-group RSNG-1 node-device row1-rack3 Note: Up to two QFabric Nodes can be part of an RSNG. Example 3: NNG configuration [edit fabric] netadmin@qfabric# set resources node-group NW-NG-0 network-domain netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack1 Note: Up to eight QFabric Nodes can be part of an NNG. Copyright 2011, Juniper Networks, Inc. 7

8 A corresponding show command, shown below, provides overall node group membership and status. show fabric administration inventory node-groups Item Identifier Connection Configuration Node group NW-NG-0 Connected Configured row21-rack1 ABCD1233 Connected Configured RSNG-1 Connected Configured row1-rack2 ABCD1231 Connected Configured row1-rack3 ABCD1232 Connected Configured SNG-1 Connected Configured row1-rack1 ABCD1230 Connected Configured Another helpful command, show fabric administration inventory, combines both node device and node groups. Interface Naming Conventions for QFabric Architecture The standard Junos OS port naming convention is a three-level identifier interface_name-fpc/pic/port_no. The fpc is the first level, and it provides slot location within the chassis. For QFabric architecture, the three-level identification poses a big challenge for management because QFabric technology can scale to include up to 128 QFabric Nodes, and there is no concept of a slot with QFabric Nodes. Therefore, the QFabric interface naming convention has been enhanced to include four levels, where a chassis-level identifier is added. The new interface name scheme is QFabric Node:interface_ name-fpc/pic/port. The QFabric Node can either be the serial number or the alias name that has been assigned. netadmin@qfabric> show interfaces row1-rack1:xe-0/0/10 Physical interface: row1-rack1:xe-0/0/10, Enabled, Physical link is Up Interface index: 49182, SNMP ifindex: Link-level type: Ethernet, MTU: 1514, Speed: 10Gbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled Interface flags: Internal: 0x0 CoS queues : 12 supported, 12 maximum usable queues Current address: 84:18:88:d5:b3:42, Hardware address: 84:18:88:d5:b3:42 Last flapped : :10:51 UTC (04:20:44 ago) Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps) Note: This interface naming convention only applies to physical interfaces. For logical interfaces such as LAGs, it is node-group:interface_name-fpc/pic/slot. Routed VLAN interfaces (RVIs) follow the standard naming convention used by Juniper Networks EX Series Ethernet Switches: vlan.x. Interface Type Configuration The next few sections will cover common configurations ports and VLANs. QFabric architecture follows the same configuration context as EX Series switches. Those who are familiar with configuring the EX Series will find the next few sections very familiar, with the only difference being the interface naming convention. There are three different interface types access, trunk, and routed interface. Just as with any other Junos OS platform, interface configurations are done under the interface stanza. The access and trunk ports can be configured on any node groups. Routed interfaces are limited to RVI or NNG ports. 8 Copyright 2011, Juniper Networks, Inc.

9 Access Port set row1-rack1:xe-0/0/0.0 family ethernet-switching port-mode access Note: Port mode access is optional. If port mode is not defined, the default port mode is access. The standard show interfaces command is available. Another helpful interface command for Layer 2 port is show ethernet-switching interfaces <QFabric Node:interface_name-fpc/pic/slot>. An example output is shown below: netadmin@qfabric> show ethernet-switching interfaces row1-rack1:xe-0/0/0 detail Interface: row1-rack1:xe-0/0/0.0, Index: 82, State: up, Port mode: Access Ether type for the interface: 0x8100 VLAN membership: default, untagged, unblocked Number of MACs learned on IFL: 0 Trunk Port netadmin@qfabric# set row1-rack1:xe-0/0/0.0 family ethernet-switching port-mode trunk Below is a sample show output command on a trunk interface: netadmin@qfabric> show ethernet-switching interfaces row1-rack1:xe-0/0/1 detail Interface: LC2:xe-0/0/1.0, Index: 89, State: down, Port mode: Trunk Ether type for the interface: 0x8100 Number of MACs learned on IFL: 0 Routed Interface As mentioned earlier, routed interfaces can either be RVI or Layer 3 ports on NNG. RVI provides routing between VLANs as well as between physical routed interfaces on the NNG. The following example shows physical Layer 3 interface configurations on both NNG and RVI. Example 1: L3 routed port on NNG netadmin@qfabric# set row21-rack1:xe-0/0/0.0 family inet address /24 Below is a sample show output command on a show interface for a Layer 3 route interface on an NNG: netadmin@qfabric> show interfaces row21-rack1:xe-0/0/0 Physical interface: row1-rack4:xe-0/0/0, Enabled, Physical link is Up Interface index: 131, SNMP ifindex: Link-level type: Ethernet, MTU: 1514, Speed: 10Gbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled Interface flags: Internal: 0x4000 CoS queues : 12 supported, 12 maximum usable queues Current address: 84:18:88:d5:e7:0c, Hardware address: 84:18:88:d5:e7:0c Last flapped : :53:59 UTC (00:21:30 ago) Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps) Logical interface row21-rack1:xe-0/0/0.0 (Index 86) (SNMP ifindex ) Copyright 2011, Juniper Networks, Inc. 9

10 Flags: 0x4000 Encapsulation: ENET2 Input packets : 0 Output packets: 1 Protocol inet, MTU: 1500 Destination: 1.1.1/24, Local: , Broadcast: Example 2: RVI Step 1. Configuring the RVI interface netadmin@qfabric# set vlan family inet address /24 Step 2. Binding the RVI interface to the VLAN netadmin@qfabric# set vlans v1250 l3-interface vlan.1250 Below is a sample show output command on a show interface for an RVI: root@qfabric> show interfaces vlan Physical interface: vlan, Enabled, Physical link is Up Interface index: 128, SNMP ifindex: Type: VLAN, Link-level type: VLAN, MTU: 1518, Speed: 1000mbps Link type : Full-Duplex Current address: 84:18:88:d5:ee:05, Hardware address: 00:1f:12:31:7c:00 Last flapped : Never Input packets : 0 Output packets: 0 Logical interface vlan.1250 (Index 88) (SNMP ifindex ) Flags: 0x4000 Encapsulation: ENET2 Input packets : 0 Output packets: 1 Protocol inet, MTU: 1500 Destination: /24, Local: , Broadcast: Layer 3 LAG Configuration Link aggregation provides link redundancy as well as increases bandwidth. QFabric architecture supports both static and dynamic LAGs, which can be configured on any QFabric Node. There are two typical LAG deployments same member and cross member. Same member LAGs are where all of the LAG child members are terminated on the same QFabric Node. Cross member LAGs are where child member LAGs are split between node group members. As discussed in the Defining Node Groups section, same member LAGs can be configured on any node group, while cross member LAGs are only supported on RSNGs and NNGs. Table 2: Node Groups LAG Support Matrix Node Groups Same Member LAG Cross-Member LAG (Active/ Active) SNG 3 RSNG 3 3 NNG Copyright 2011, Juniper Networks, Inc.

11 Example 1: Same member LAG configuration Step 1. Define number of supported LAGs per node group While the example below is for an SNG named SNG-1, the same configuration is applicable to RSNG or NNG the configuration will just need to reflect the correct node group name. All node groups support the same member LAG configuration. set chassis node-group SNG-1 aggregated-devices ethernet device-count 1 Step 2. Assign the interface to a LAG interface Note: The chassis identifier name is the QFabric Node. netadmin@qfabric# set row1-rack1:xe-0/0/46 ether-options 802.3ad ae0 netadmin@qfabric# set row1-rack1:xe-0/0/47 ether-options 802.3ad ae0 Step 3. Configure the LAG interface All common LAG parameters across child LAG members such as LACP, speed, duplex, and so on are centralized to the LAG interface itself. While the example below is for a Layer 2 interface, for Layer 3 the family needs to change from ethernet-switching to inet (L3 is only supported on NNG). For static LAGs, omit the LACP configuration. One thing to note is that the node identifier is the node group, not the QFabric Node. netadmin@qfabric# set SNG-1:ae0 aggregated-ether-options lacp active netadmin@qfabric# set SNG-1:ae0 unit 0 family ethernet-switching port-mode trunk Some relevant commands for LAG: show lacp ## applicable to dynamic LAG only ## show interface terse match node_group:interface_name ## example SNG-1:ae0 ## show interface node_group:interface_name Step 4. Assign IP address to LAG interface Example 2: Cross member LAG configuration Step 1. Define the number of supported LAGs per network node group netadmin@qfabric# set chassis node-group NW-NG-0 aggregated-devices ethernet device-count 10 Step 2. Assign the interface to a LAG interface Note: The interface name is the QFabric Node. netadmin@qfabric# set row1-rack2:xe-0/0/0 ether-options 802.3ad ae0 netadmin@qfabric# set row1-rack3:xe-0/0/0 ether-options 802.3ad ae0 Copyright 2011, Juniper Networks, Inc. 11

12 Step 3. Configure the LAG interface and assign it an IP address All common LAG parameters across child LAG members such as LACP, speed, duplex, and so on are centralized to the LAG interface itself. While the example below is for a Layer 2 interface, for Layer 3 the family needs to change from ethernet-switching to inet (L3 is only supported on NNG). For static LAGs, omit the LACP configuration. One thing to note is that the node identifier is the node group and not the QFabric Node. netadmin@qfabric# set NW-NG-0:ae0 aggregated-ether-options lacp active netadmin@qfabric# set NW-NG-0:ae0 unit 0 family ethernet-switching port-mode trunk Some relevant commands for LAG: show lacp ## applicable to dynamic LAG only ## show interface terse match node_group:interface_name ## example NW-NG-0:ae0 ## show interface node_group:interface_name Once the LAG interface is configured for Layer 2 link, change the family to inet and assign an IP address. netadmin@qfabric# set NW-NG-0:ae0.0 family inet address /24 VLAN Configuration VLANs allow users to control the size of a broadcast domain and, more importantly, group ports in a Layer 2 switched network into the same broadcast domain as if they were connected on the same switch, regardless of their physical location. QFabric architecture is no exception. VLANs can be contained to a single node group or spread across the same and/or different types of node groups. The steps below outline how to define VLANs and assign VLAN port membership. Step 1. Define the VLAN VLANs are defined under the VLAN stanza. Minimum configuration is VLAN name and vlan-id. [edit vlans] netadmin@qfabric# set default vlan-id 1 Below is an example of show vlan output. The asterisk denotes that the interface is up. netadmin@qfabric> show vlans Name Tag Interfaces default 1 row1-rack1:xe-0/0/0.0*, row1-rack1:xe-0/0/0.1*, row1- rack2:xe-0/0/3.0*, RSNG-1:ae0.0*, NW-NG-0:ae0.0* Step 2. VLAN port membership If VLAN membership is not explicitly configured on the access ports, then it reverts back to the default VLAN. For trunk ports, explicit configuration is required. There are two methods for assigning a port to a VLAN port centric and VLAN centric. Either method is valid, but if interface range or group profile isn t being used, then for ease of VLAN management, Juniper recommends that VLAN membership for the access port should be done under the VLAN method and under the port method for the trunk port. 12 Copyright 2011, Juniper Networks, Inc.

13 Method 1: VLAN centric [edit vlans] set default interface row1-rack1:xe-0/0/0.0 Method 2: Port centric Either the vlan-name or vlan-id (802.1Q) can be used. set row1-rack1:xe-0/0/0.0 family ethernet-switching vlan members 1 Trunk Port On trunk ports, VLAN ranges are supported for ease of configuration (i.e., 1-100). For nonsequential VLANs, enclose the membership with squared brackets and use a space for separation (i.e., ). netadmin@qfabric# set row1-rack1:xe-0/0/0.0 family ethernet-switching port-mode trunk vlan members [ ] In the above configuration, all VLANs are tagged on the interface. For hybrid trunks, untagged and tagged traffic use the native-vlan-id keyword for untagged. Below is an example trunk interface configured for VLAN 1 to be untagged and VLANs 2-25 to be tagged. Note that VLAN 1 is not part of the vlan members configuration. netadmin@qfabric# set row1-rack1:xe-0/0/0.0 family ethernet-switching port-mode trunk native-vlan-id 1 vlan members [2-25] Some helpful VLAN membership commands are: show vlans show vlans vlan-name detail show ethernet-switching interfaces brief show ethernet-switching interfaces node_identifier:interface_name-fpc/pic/port Below is an example of the media access control (MAC) address table for the QFabric: netadmin@qfabric> show ethernet-switching table Ethernet-switching table: 3 entries, 1 learned VLAN MAC address Type Age Interfaces default * Flood - NW-NG-0:All-members default 00:10:db:ff:a0:01 Learn 51 NW-NG-0:ae0.0 default 84:18:88:d5:ee:05 Static - NW-NG-0:Router Additional useful MAC address table commands include: show ethernet-switching table summary show ethernet-switching table interface node_identifier:interface_name-fpc/pic/port show Ethernet-switching table vlan Copyright 2011, Juniper Networks, Inc. 13

14 Design Use Cases This section will describe various Layer 3 design uses cases deploying QFabric technology. For cable deployment, there are few options top-of-rack (TOR), middle-of-row (MOR), or end-of-row (EOR) each of which has pros and cons. QFabric architecture offers benefits with all three types of deployments, including lower cabling costs, modularity and deployment flexibility, as well as fewer (one logical) devices to manage and a simplified STP-free Layer 2 topology. While QFabric architecture can be deployed as TOR, EOR, or MOR, for the following design use cases, the deployment of choice will be TOR. How the rack server or blade chassis is connected to the TOR depends on the high availability strategy, i.e., is it at the application, server/network interface card (NIC), or network level? For rack servers, there are three different types of connections and levels of redundancy, which are explained below. Single-attached: The server only has a single link connecting to the switch. In this model, there is either no redundancy, or the redundancy is built into the application. Dual-attached: The server has two links connecting to the same switch. NIC teaming is enabled on the servers, where it can be either active/standby or active/active. The second link provides the second level of redundancy. The more common deployment is active/active with a static LAG between the switch and rack server. Dual-homed: The server has two links that connect to two different switches/modules in either an active/standby or active/active mode. This is a third level of redundancy; in addition to link redundancy there is spatial redundancy. If one of the switches fails, then there is an alternate path. In order to provide an active/active deployment, the NIC needs to be in different subnets. If they are sharing the same IP/MAC, then some form of stacking or multichassis LAG technology needs to be supported on the switches so that a LAG can be configured between the switches and server. Single-attached Dual-attached Dual-homed (L) Active/Standby (R) Active/Active (L) Active/Standby (R) Active/Active Figure 4: Different types of redundancy for rack servers Depending on how the servers are connected and how NIC teaming is implemented, the QFabric Node should be configured with the appropriate node group. The table below shows the relationship between node group and server connections. Table 3: Node Group Selection Matrix for Rack Servers or Blade Switches with Pass-Through Modules Active/Passive Active/Active Single-attached SNG N/A Dual-attached SNG SNG Dual-homed RSNG RSNG 14 Copyright 2011, Juniper Networks, Inc.

15 Network redundancy is not specific to TOR deployment, as it also exists for MOR or EOR. The same deployment principles apply to TOR, EOR, and MOR, with minor exceptions for MOR or EOR where, in a dual-homed connection scenario using modular switches, the second link can be connected to either a different module or a different chassis, depending on cost and rack space. In the case where blade chassis are used instead of rack servers, physical connectivity may vary depending on the blade chassis intermediary connection, pass-through module, or blade switches. Juniper recommends the passthrough module as it provides a direct connection between the servers and the QFabric architecture. This direct connection eliminates any oversubscription and the additional switching layer that is seen with blade switches. The deployment options for pass-through are exactly the same as described for rack servers. As for blade switches, depending on the vendor, they all have one thing in common they represent another device to manage, which adds complexity to the overall switching topology. Figure 5 shows the common network deployment between blade switches and access switches. Blade Switch Blade Chassis Single-homed Dual-homed Active/Backup Dual-homed Active/Active Figure 5: Different deployment scenarios with embedded blade switches in blade chassis Single-homed: Each blade switch has a LAG connection into a single access switch. In this deployment, there are no Layer 2 loops to worry about or manage. Dual-homed (active/backup): In this deployment, each access switch is a standalone device. Since there are potential Layer 2 loops, the blade switch should support some sort of Layer 2 loop prevention STP or active/backuplike technology, which will effectively block any redundant link to break the Layer 2 loop. Dual-homed (active/active): This deployment provides the most optimized deployment, as all links between the blade and access switches are active and forwarding and provide network resiliency. The connection between the blade switch and access switch is a LAG, which means the external switches must support either multichassis LAG or some form of stacking technology. Since LAG is a single logical link between the blade and external switches, there are no Layer 2 loops to worry about or manage. Note: Figure 5 assumes that blade switches are separate entities and are not daisy-chained or logically grouped through a stacking technology. Since QFabric architecture is a distributed system that acts as a single logical switch, the two most likely deployments are single-homed or dual-homed (active/active). The QFabric Nodes will be configured as SNG for single-homed and RSNG for dual-homed (active/active). Table 4: Node Group Selection Matrix for Blade Chassis with Embedded Blade Switches Active/Passive Active/Active Single-homed SNG N/A Dual-homed (active/backup) SNG or RSNG SNG or RSNG Dual-homed (active/active) RSNG RSNG Copyright 2011, Juniper Networks, Inc. 15

16 In this document, the first hop router is the QFabric architecture. Use cases where a WAN edge router such as one of Juniper Networks MX Series 3D Universal Routers, a security device such as one of Juniper Networks SRX Series Services Gateways, or any other service layer devices (load balancer, WAN optimizer, service gateway) connect to the QFabric architecture as Layer 3 devices are discussed below. Connecting Layer 3 Device to QFabric Architecture When Layer 3 devices connect to the QFabric architecture, the network node group port must be used for the physical connection. Network node group members do not have to be deployed physically close together, they can span the data center. However, only eight QFabric Nodes can be in a network node group. It is impossible to have multiple network node groups per QFabric architecture configuration. MX Series SRX Series WX Series NNG Junos Pulse Gateway SNG SNG NNG: Network Node Group Load Balancer Figure 6: Layer 3 devices can be located anywhere in the QFabric architecture Route Lookup and Forwarding Decisions In the QFabric architecture, all of the data plane intelligence is distributed to each QFabric Node. In other words, if the packet comes into one of the QFabric Nodes and it requires Layer 3 lookup, the QFabric Node consults its routing table and decides on the destination QFabric Node. The ingress QFabric Node sends a packet to the 40 Gbps uplink. Once the egress QFabric Node receives the packet, it references its own Address Resolution Protocol (ARP) table to select an appropriate port. QFabric and VRRP In traditional data center architectures, Virtual Router Redundancy Protocol (VRRP) is typically required to secure the gateway redundancy for any Layer 3 devices. However, moving onto the QFabric architecture, VRRP is not necessary since a QFabric solution is a single logical switch, meaning that there is no need to have multiple devices running as gateways. Within a network node group, the high availability of a gateway has already been built in. For example, the SRX Series cluster in Figure 6 connects to two QFabric Nodes which are part of NNG. To the SRX Series cluster, it is the same as connecting to different ports on different line cards on a single switch. These line cards and ports are fully synchronized at the QFabric Director level. There is no need to run protocols to ensure the switchover between devices; therefore, users do not have to configure VRRP among network node groups. 16 Copyright 2011, Juniper Networks, Inc.

17 Layer 3 Design Use Cases Use Case 1: Static Default Route Configuration In the case where an MX Series device is present and provides most of the rich routing functionality, and the QFabric architecture just needs to provide basic routing, a static default route configuration will apply. Three QFabric Nodes in NNG will provide inter-vlan routing and upstream LAG access to redundant MX Series devices. On the MX Series side, there are two ways to provide one unique gateway IP to QFabric architecture one is Virtual Chassis technology on the MX Series and the other is VRRP between the two. MX Series NNG VLAN1100 VLAN1104 VLAN1101 VLAN1103 VLAN1102 Step 1. Define QFabric Node alias and NNG Figure 7: NNG connecting to MX Series with LAG [edit fabric] netadmin@qfabric# set aliases node-device ABCD1252 row21-rack1 netadmin@qfabric# set aliases node-device ABCD1253 row21-rack2 netadmin@qfabric# set aliases node-device ABCD1254 row21-rack3 netadmin@qfabric# set resources node-group NW-NG-0 network-domain netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack1 netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack2 netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack3 Step 2. Define Layer 2 configuration [edit vlans] netadmin@qfabric# set v1100 vlan-id 1100 netadmin@qfabric# set v1100 vlan-id 1101 netadmin@qfabric# set v1100 vlan-id 1102 netadmin@qfabric# set v1100 vlan-id 1103 netadmin@qfabric# set v1100 vlan-id 1104 Copyright 2011, Juniper Networks, Inc. 17

18 Step 3. Define LAG configuration NNG connecting to MX Series device [edit] set chassis node-group NW-NG-0 aggregated-devices ethernet device-count 24 set interface-range LAG-ae0 member row21-rack1:xe-0/0/[0-1] set interface-range LAG-ae0 member row21-rack2:xe-0/0/[0-1] set interface-range LAG-ae0 member row21-rack3:xe-0/0/[0-1] set interface-range LAG-ae0 ether-options 802.3ad ae0 set interface-range LAG-ae1 member row21-rack1:xe-0/0/[2-3] set interface-range LAG-ae1 member row21-rack2:xe-0/0/[2-3] set interface-range LAG-ae1 member row21-rack3:xe-0/0/[2-3] set interface-range LAG-ae1 ether-options 802.3ad ae0 set NW-NG-0:ae0 aggregated-ether-options lacp active set NW-NG-0:ae1 aggregated-ether-options lacp active Step 4. Assign IP address to LAG interfaces set NW-NG-0:ae0.0 family inet address /24 set NW-NG-0:ae1.0 family inet address /24 Step5: Configure RVI for five VLANs set vlan family inet address /24 set vlan family inet address /24 set vlan family inet address /24 set vlan family inet address /24 set vlan family inet address /24 Step 6. Bind the RVI interface to the VLAN set vlans v1100 l3-interface vlan.1100 set vlans v1101 l3-interface vlan.1101 set vlans v1102 l3-interface vlan.1102 set vlans v1103 l3-interface vlan.1103 set vlans v1104 l3-interface vlan.1104 Step 7. Configure default routes to the MX Series [Assumes that is the address of the MX Series Virtual Chassis configuration] [edit routing-option] set routing-options static route /0 next-hop Copyright 2011, Juniper Networks, Inc.

19 Step 8. Verify default route configuration show route terse inet.0: 16 destinations, 16 routes (16 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both A Destination P Prf Metric 1 Metric 2 Next hop AS path * /0 S * /24 D 0 NW-NG-0:vlan.1100 * /32 L 0 Local * /24 D 0 NW-NG-0:vlan.1101 * /32 L 0 Local * /24 D 0 NW-NG-0:vlan.1102 * /32 L 0 Local * /24 D 0 NW-NG-0:vlan.1103 * /32 L 0 Local * /24 D 0 NW-NG-0:vlan.1104 * /32 L 0 Local * /24 D 0 NW-NG-0:ae0.0 NW-NG-0:ae1.0 * /32 L 0 Local * /32 L 0 Local Note: The MX Series Virtual Chassis configuration will not be covered, since it is out of the scope of this document. Please visit for more information about Virtual Chassis technology. Use Case 2: Putting QFabric Architecture into an OSPF Area Another use case is to run OSPF on the QFabric architecture. This scenario is applicable where the user wants more granular control over advertised/advertising routes. In the following example, QFabric technology is deployed in OSPF Area0 and upstream MX Series devices will advertise the default route. MX Series NNG VLAN1100 VLAN1104 VLAN1101 VLAN1103 VLAN1102 OSPF Area0 Figure 8: QFabric technology in OSPF area0 Copyright 2011, Juniper Networks, Inc. 19

20 Step 1. Define QFabric Node alias and NNG [edit fabric] set aliases node-device ABCD1252 row21-rack1 set aliases node-device ABCD1253 row21-rack2 set aliases node-device ABCD1254 row21-rack3 set resources node-group NW-NG-0 network-domain set resources node-group NW-NG-0 node-device row21-rack1 set resources node-group NW-NG-0 node-device row21-rack2 set resources node-group NW-NG-0 node-device row21-rack3 Step 2. Define five VLANs [edit vlans] set v1100 vlan-id 1100 set v1100 vlan-id 1101 set v1100 vlan-id 1102 set v1100 vlan-id 1103 set v1100 vlan-id 1104 Step 3. LAG configuration NNG connecting to MX Series device [edit] set chassis node-group NW-NG-0 aggregated-devices ethernet device-count 24 set interface-range LAG-ae0 member row21-rack1:xe-0/0/[0-1] set interface-range LAG-ae0 member row21-rack2:xe-0/0/[0-1] set interface-range LAG-ae0 member row21-rack3:xe-0/0/[0-1] set interface-range LAG-ae0 ether-options 802.3ad ae0 set interface-range LAG-ae1 member row21-rack1:xe-0/0/[2-3] set interface-range LAG-ae1 member row21-rack2:xe-0/0/[2-3] set interface-range LAG-ae1 member row21-rack3:xe-0/0/[2-3] set interface-range LAG-ae1 ether-options 802.3ad ae0 set NW-NG-0:ae0 aggregated-ether-options lacp active set NW-NG-0:ae1 aggregated-ether-options lacp active Step 4. Assign IP address to LAG interfaces set NW-NG-0:ae0.0 family inet address /30 set NW-NG-0:ae1.0 family inet address /30 Step 5. Configure RVI for five VLANs set vlan family inet address /24 set vlan family inet address /24 set vlan family inet address /24 set vlan family inet address /24 set vlan family inet address /24 20 Copyright 2011, Juniper Networks, Inc.

21 Step 6. Bind the RVI interface to the VLAN set vlans v1100 l3-interface vlan.1100 set vlans v1101 l3-interface vlan.1101 set vlans v1102 l3-interface vlan.1102 set vlans v1103 l3-interface vlan.1103 set vlans v1104 l3-interface vlan.1104 Step 7. Enable OSPF and include LAG interface and RVI to area 0 [edit] netadmin@qfabric# set protocols ospf area interface NW-NG-0:ae0.0 netadmin@qfabric# set protocols ospf area interface NW-NG-0:ae1.0 netadmin@qfabric# set protocols ospf area interface vlan.1100 netadmin@qfabric# set protocols ospf area interface vlan.1101 netadmin@qfabric# set protocols ospf area interface vlan.1102 netadmin@qfabric# set protocols ospf area interface vlan.1103 netadmin@qfabric# set protocols ospf area interface vlan.1104 Step 8. Verify OSPF neighbor [edit] root@sv-poc-qf> show ospf neighbor Address Interface State ID Pri Dead NW-NG-0:ae0.0 Full NW-NG-0:ae1.0 Full Step 9. Verify routing table netadmin@qfabric> show route terse inet.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both A Destination P Prf Metric 1 Metric 2 Next hop AS path * /24 D 0 NW-NG-0:vlan.1100 * /32 L 0 Local * /24 D 0 NW-NG-0:vlan.1101 * /32 L 0 Local * /24 D 0 NW-NG-0:vlan.1102 * /32 L 0 Local * /24 D 0 NW-NG-0:vlan.1103 * /32 L 0 Local * /24 D 0 NW-NG-0:vlan.1104 * /32 L 0 Local * /24 D 0 NW-NG-0:ae0.0 NW-NG-0:ae1.0 * /32 L 0 Local * /32 L 0 Local * /0 O 10 1 > * /32 O 10 1 MultiRecv Copyright 2011, Juniper Networks, Inc. 21

22 Use Case 3: Putting QFabric Architecture into OSPF Stub Area Another use case is to run QFabric architecture in an OSPF stub area. This scenario is applicable where the user wants to minimize the routing table size. Most of the configurations are the same as those in Use Case 2. The only difference is to configure the OSPF area as stub at Step 7, then add RVI interfaces to the stub area. Note that it is possible not to advertise summary routes in the stub area by adding the no-summaries option. [edit] set protocols ospf area stub no-summaries set protocols ospf area interface NW-NG-0:ae0.0 set protocols ospf area interface NW-NG-0:ae1.0 set protocols ospf area interface vlan.1100 set protocols ospf area interface vlan.1101 set protocols ospf area interface vlan.1102 set protocols ospf area interface vlan.1103 set protocols ospf area interface vlan.1104 Use Case 4: Connecting One-Armed SRX Series Device as Active/Active with QFabric Architecture It is frequently required to connect firewalls to the core/aggregation device. The next two use cases will discuss how SRX Series Services Gateways can be deployed with QFabric solutions. The diagram below shows a typical deployment in which two Juniper Networks SRX5800 Services Gateway devices running in active/active mode connect to an EX Series/MX Series device in a one armed fashion. Core/Edge Tier SRX5800_A EX Series/MX Series SRX5800_B EX4200 Virtual Chassis EX4200 Virtual Chassis VLAN 500, 1001, 1003, 1005 VLAN 600, 1000, 1002, 1004 VLAN 1000 VLAN 1001 Figure 9: SRX Series one-armed deployment in a two-tier architecture 22 Copyright 2011, Juniper Networks, Inc.

23 When a customer migrates to the QFabric architecture, the one-armed deployment will appear as in Figure 10. There is no need to change the configuration on the SRX5800 side. The fundamental QFabric solution configuration is the same as on the EX Series/MX Series devices in Figure 9. SRX5800_A to WAN Edge SRX5800_B VLAN 500, 1001, 1003, 1005 VLAN 600, 1000, 1002, 1004 VLAN 1000 VLAN 1001 Figure 10: One-armed SRX Series active/active deployment with QFabric technology In this example, SRX5800_A and SRX5800_B connect to the QFabric solution as one-armed devices, deployed as an active/active cluster. The first VLAN trunk is handling VLANs 500, 1001, 1003, and 1005, while the second trunk handles VLANs 600, 1000, 1002, and This VLAN traffic will be distributed to the SRX5800 cluster SRX5800_A and SRX5800_B. A solid line denotes the primary link for the given VLAN, while a dotted line indicates the backup. With the virtual router functions of QFabric architecture, inter-vlan routing won t ensure that these two groups are totally isolated at the Layer 3 level. This is feasible in a multi-tenant environment. VLANs 500 and 600 will be used for uplink connections to the WAN edge router from the SRX Series under the set security zones security-zone uplink interface stanza. Here the first VLAN trunk is in virtual router instance 10 (VR10) while the second VLAN trunk is in VR20. In addition, RVI VLANs 500 and 600 will be in Core VR to provide uplink connection to the WAN edge routers. Servers just need to send packets to the VRRP address on the SRX Series gateway in each RVI VLAN (1000 through 1005). Note that SRX Series configuration details are not covered since they are out of scope for this document. The following configuration examples focus on network node group configuration. Please review previous use case or the L2 design guide for server node group configuration information. Step 1. Define QF/Node alias and NNG [edit fabric] netadmin@qfabric# set aliases node-device ABCD1252 row21-rack1 netadmin@qfabric# set aliases node-device ABCD1253 row21-rack2 netadmin@qfabric# set aliases node-device ABCD1254 row21-rack3 netadmin@qfabric# set aliases node-device ABCD1255 row21-rack4 netadmin@qfabric# set resources node-group NW-NG-0 network-domain netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack1 netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack2 netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack3 netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack4 Copyright 2011, Juniper Networks, Inc. 23