FlexFrame for SAP. Version 5.1A. Network Design and Configuration Guide. Edition March 2012 Document Version 1.1

Transcription

1 FlexFrame for SAP Version 5.1A Network Design and Configuration Guide Edition March 2012 Document Version 1.1

2 Fujitsu Limited Copyright Fujitsu Technology Solutions 2011 FlexFrame and PRIMERGY are trademarks or registered trademarks of Fujitsu Limited in Japan and other countries. SAP and NetWeaver are trademarks or registered trademarks of SAP AG in Germany and in several other countries Linux is a registered trademark of Linus Torvalds SUSE Linux is a registered trademark of Novell, Inc., in the United States and other countries Oracle and Java are trademarks of ORACLE Corporation and/or its affiliates Intel and PXE are registered trademarks of Intel Corporation in the United States and other countries MaxDB is a registered trademark of MySQL AB, Sweden MySQL is a registered trademark of MySQL AB, Sweden NetApp and the Network Appliance logo are registered trademarks and Network Appliance and Data ONTAP are trademarks of NetApp, Inc. in the U.S. and other countries. EMC, CLARiiON, Symmetrix and Celerra are trademarks or registered trademarks of EMC Corporation in the United States and other countries VMware, ESX, ESXi, VMware vcenter, VMware vsphere are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. Ethernet is a registered trademark of XEROX, Inc., Digital Equipment Corporation and Intel Corporation Windows and Word are registered trademarks of Microsoft Corporation All other hardware and software names used are trademarks of their respective companies. All rights, including rights of translation, reproduction by printing, copying or similar methods, in part or in whole, are reserved. Offenders will be liable for damages. All rights, including rights created by patent grant or registration of a utility model or design, are reserved. Delivery subject to availability. Right of technical modification reserved.

3 Contents 1 Introduction Purpose of this Document Notational Conventions... 1 Document History Related Documents Concept and Design Switch Groups Network Speed Network Connection Linux Bonding ESX NIC Teaming NetApps Interface Grouping EMCs virtual Devices BX Link State Propagation Virtual LAN Uplink Connection Client LAN Connection FlexFrame Network Versions Small Version Directly Connected Versions Enterprise Version Configuration Commands Linux Bonding Interface Linux VLAN Interface NetApp Filer Configuration EMC Celerra Configuration Network Switch Configuration Abbreviations Glossary Index Network Design and Configuration Guide

4

5 1 Introduction 1.1 Purpose of this Document This document describes the FlexFrame networking concept and design and possible networking topologies. The reader should be familiar with IP networking and needs basic knowledge about Ethernet-based networking with switches and virtual LANs. Knowledge of network switch configuration is recommended. The knowledge of the FlexFrame Installation Guide and the FlexFrame Administration and Operation Guide would be an advantage. 1.2 Notational Conventions The following conventions are used in this manual: Additional information that should be observed. Warning that must be observed. fixed font <fixed font> fixed font Names of paths, files, commands, and system output. Names of variables. User input in command examples (if applicable using <> with variables). 1.3 Document History Document Version Changes Date 1.0 First Edition Nexus 55xx, cat3750x Network Design and Configuration Guide 1

6 Introduction Related Documents 1.4 Related Documents FlexFrame for SAP Administration and Operation FlexFrame for SAP HW Characteristics Quickguides FlexFrame for SAP Installation ACC 7.3 FlexFrame for SAP Installation Guide for SAP Solutions FlexFrame for SAP Installation of a FlexFrame Environment FlexFrame for SAP Management Tool FlexFrame for SAP myamc.fa_agents Installation and Administration FlexFrame for SAP myamc.fa_messenger Installation and Administration FlexFrame for SAP myamc.fa_logagent Installation and Administration FlexFrame for SAP Network Design and Configuration Guide FlexFrame for SAP Security Guide FlexFrame for SAP Technical White Paper FlexFrame for SAP Upgrading FlexFrame 4.2B or 5.0A to 5.1A ServerView Documentation SUSE Linux Enterprise Server Documentation 2 Network Design and Configuration Guide

7 2 Concept and Design The design goals for the FlexFrame network concept: avoid single points of failure (SPOF) avoid unnecessary complexity are achieved by the following means: use redundant network use a lightweight design use standard interfaces and configurations keep interface configuration as identical as possible on all server systems keep design flexible enough to meet the requirements from small to medium up to enterprise sized installations reduce count of needed interfaces per system and results in the concept described above. 2.1 Switch Groups Application Control 1 Application Switch 1 Control 2 Application Blade Rack PORT a PORT b PORT c PORT a PORT b PORT c Switch 2 SwitchGroup NAS The very simplified drawing above may visualize the general concept (only data connections are considered). The core of the design is the switch group which provides redundant usable ports. If one member of the switch group fails the ports of the other members Network Design and Configuration Guide 3

8 Concept and Design are still working. Every end system and the uplink are using this redundancy as described later. In this way a failure of a single member of the switch group is tolerated and allows continuous operation. FlexFrame distinguishes the following switch group types: CAT3750-STACK Two up to nine switches of the Cisco Catalyst 3750G, 3750E or 3750X switch family are building a Cisco Catalyst switch stack. The switches are connected via Cisco StackWise cabling and behave like a single switch. For more than 4 switches with 10GbE ports the StackWise cabling may be a bottleneck. NEXUS5000-VPC Exactly two switches (nexus50xx) of the Cisco Nexus 5000 Switch family are building a Cisco Nexus vpc domain. For the vpc peer-link as a necessary special channel between the switches FlexFrame configures two ports on each switch. For the vpc peer-keepalive link as a necessary alternative Layer 3 connectivity between the switches FlexFrame uses the mgmt0 interface IP addresses as recommended from Cisco. NEXUS5500-VPC Exactly two switches (nexus55xx) of the Cisco Nexus 5000 Switch family are building a Cisco Nexus vpc domain. For the vpc peer-link as a necessary special channel between the switches FlexFrame configures two ports on each switch. For the vpc peer-keepalive link as a necessary alternative Layer 3 connectivity between the switches FlexFrame uses the mgmt0 interface IP addresses as recommended from Cisco. For further details about supported switches see the FlexFrame Support Matrix. For further details about the switches see the Cisco manuals. For further details about the purpose of the different switch group types see below. 2.2 Network Speed FlexFrame supports network connections for data communication with the following network speeds: 1Gbit/sec (1GbE) 10Gbit/sec (10GbE) Pay attention to the fact that 10GbE connected servers may also request a 10GbE connected NAS system and 10GbE uplinks between. For further details about supported end systems see the FlexFrame Support Matrix. 4 Network Design and Configuration Guide

9 Concept and Design 2.3 Network Connection Network connection is established by connecting ports of end systems to ports of network devices which itself are connected to other network devices. The characteristics of connected ports must be compatible and a proper connector must be used. Cat5 or Cat5e cable are used to connect RJ45 Service LAN ports of RX servers RJ45 management blade ports of BX cabinets to RJ45 10/100/1000 ethernet ports of switches of the Cisco Catalyst 3750 switch family. Cat5e cable are used to connect RJ45 1GbE LAN ports of RX servers RJ45 1GbE LAN ports of NAS systems RJ45 1GbE switch blade ports of BX cabinets to RJ45 10/100/1000 ethernet ports of switches of the Cisco Catalyst 3750 switch family. LC-LC fibre optic cable are used to connect 10GbE LC ports of RX servers 10GbE SFP+ modules plugged into RX servers 10GbE SFP+ modules plugged into switches of the Cisco Nexus 5000 switch family 10GbE SFP+ modules plugged into the Cisco Catalyst 3750x 10G module to 10 GbE SFP+ modules plugged into switches of the Cisco Nexus 5000 switch family or to 10 GbE SFP+ modules plugged into the Cisco Catalyst 3750x 10G module or to connect 1GbE SFP modules plugged into switches of the Cisco Catalyst 3750 switch family 1GbE SFP modules plugged into dual speed ports of switches of the Cisco Nexus 5000 switch family to 1GbE SFP modules plugged into switches of the Cisco Catalyst 3750 switch family. LC-SC fibre optic cable are used to connect 10GbE LC ports of RX servers 10GbE SFP+ modules plugged into RX servers 10GbE SFP+ modules plugged into switches of the Cisco Nexus 5000 switch family to 10 GbE ports of switches of the Cisco Catalyst 3750E switch family. 10GBASE-CU SFP+ cable (Twinax) are used to connect 10GbE ports of switches of the Cisco Nexus 5000 switch family used for peer link to switches of the Cisco Nexus 5000 switch family. Network Design and Configuration Guide 5

10 Concept and Design 2.4 Linux Bonding Linux provides a mechanism called bonding which can aggregate several physical interfaces to a virtual interface (bond). A bond interface can be used like a usual interface and is operational as long as at least one physical interface assigned to the bond is operational. Several bonding configurations are available. rack server or server blade eth0 eth1 active active Switch 1 bond0 Switch 2 When running Linux on a physical server (rack server or server blade) used as Application FlexFrame aggregates 2 physical interfaces using adaptive load balancing (mode 6) and failure detection relies solely on the link state provided by the adapter (MII monitoring). This is an active/active method and has no requirements on the link partners which can be switch blades or switch group switches. The requirement on the adapter to be able to change the MAC address is fulfilled from all supported servers. On the Control s bonding mode 1 (active-backup) is used. No requirements have to be fulfilled neither from the link partners nor from the adapter. Failure detection relies solely on the link state provided by the adapter (MII monitoring). 6 Network Design and Configuration Guide

11 Concept and Design 2.5 ESX NIC Teaming ESX provides a mechanism called NIC teaming which is used if multiple physical Ethernet adapters are assigned to a single virtual switch (vswitch). A vswitch is operational as long as at least one physical Ethernet adapter assigned to the vswitch is operational. Several NIC teaming policies are available and can be set on the port group level. rack server or server blade vmnic0 vmnic1 active Switch 1 vswitch active Switch 2 FlexFrame assigns 2 physical Ethernet adapters in ESX default configuration to the vswitch and puts all these adapters in the active list for the FlexFrame generated port groups. The adapters are used with load balancing based on the originating virtual Port ID and failure detection relies solely on the link state provided by the adapter This method has no requirements on the link partners which can be switch blades or switch group switches. Network Design and Configuration Guide 7

12 Concept and Design 2.6 NetApps Interface Grouping NetApps Data ONTAP provides a mechanism called interface grouping which can aggregate several physical interfaces to a virtual interface. An interface group can be used like a usual interface and is operational as long as at least one physical interface assigned to the group is operational. Several modes are available. active Switch 1 e0a NetApp Filer e0b active ifgrp Switch 2 SwitchGroup FlexFrame aggregates at least 2 physical interfaces into a dynamic multimode interface group which is compliant with IEEE 802.3ad. All physical interfaces are active. The corresponding ports on the switch group switches must build a channel also complaint with IEEE 802.3ad (LACP). 8 Network Design and Configuration Guide

13 Concept and Design 2.7 EMCs virtual Devices EMCs DART provides a mechanism called virtual device which can aggregate several physical devices to a virtual device. A virtual device can be used like a usual device and is operational as long as at least one physical device assigned to the virtual device is operational. Several modes are available. active Switch 1 cge0 cge1 EMC Celerra XBlade active trk Switch 2 SwitchGroup FlexFrame aggregates at least 2 physical devices into a trunk device which is compliant with IEEE 802.3ad. All physical devices are active. The corresponding ports on the switch group switches must build a channel also complaint with IEEE 802.3ad (LACP). 2.8 BX Link State Propagation BX switch blades provide a mechanism which allows associating several downstream ports with an upstream port or channel for link state propagation. If the link of the upstream port or channel goes down the link of all associated downstream ports are also forced to go down. In this way a bonding driver on a server blade will get aware of the upstream problem and switches the traffic of concerned interfaces. Network Design and Configuration Guide 9

14 Concept and Design active server blade eth0 Switchblade 1 Switch 1 eth1 bond0 active Switchblade 2 Switch 2 BX chassis SwitchGroup FlexFrame associates all downstream ports of a switch blade and a single upstream channel for link state propagation. The upstream channel is compliant with IEEE 802.3ad. The corresponding ports on the switch group switches must also build a channel complaint with IEEE 802.3ad (LACP). The channel consists of at least two ports with crossover cabling. More ports can be used for higher bandwidth. 2.9 Virtual LAN Virtual LAN (VLAN) is a mechanism which allows having multiple separated networks on the same physical network infrastructure. A widely used standard is tagged VLAN according IEEE 802.1q. The protocol specifies an extension of the Ethernet header by a VLAN identifier (tag). A packet belongs to the VLAN identified by the tag. Communication between VLANs is only possible by routing as is common for usual networks. For compatibility reason packets without a tag are also allowed. A switchport can be configured to assign received packets without tag to a specific VLAN and remove the tag when a packet of this VLAN should be send. The VLAN is then called native or untagged. End systems which access only one VLAN or cannot handle VLAN tags are connected this way. End systems used within FlexFrame like NAS systems or servers running Linux are able to handle VLAN traffic according IEEE 802.1q. FlexFrame can handle different pools with independent networks. The traffic between communication partner types (client server, server server and server storage) is also separated. Therefore VLANs are used as helpful mechanism to reduce the necessary physical infrastructure and increase the flexibility with respect to the pooling concept. According IEEE 802.1q the VLAN tag has a value range from 1 to 4094 but there are switch specific restrictions. 10 Network Design and Configuration Guide

15 Concept and Design Cisco Catalyst 3750 Switch Family 1005 VLANs are supported. Normal usable VLANs are in the range from 1 to VLAN numbers 1002 through 1005 are reserved for Token Ring and FDDI VLANs. VLAN numbers 1006 through 4094 are extended-range VLANs and usable if not already used internally for routed ports. Cisco Nexus 5000 Switch Family 507 VLANs are supported. Normal usable VLANs are in the range from 1 to VLAN numbers 1006 through 4094 are extended-range VLANs and usable with the following restrictions. The state is always active and VLANs 3968 to 4047 and 4094 are reserved for internal use. VLANs V4: Client V3: Control V2: Server V1: Storage Physical Network Layout & Virtual LANs per Pool Control s and Linux Application s (RX300S6), both Storage Types Logical View: VLANs Application n Control A (RX300S6) IPMI V1 V2 V4 Bond 0 eth 2 eth 0 eth 1 Onboard LAN Port 3 Onboard LAN Port 1 Onboard LAN Port 2 Control B (RX300S6) IPMI V1 V2 V4 Bond 0 eth 2 eth 0 eth 1 Onboard LAN Port 3 Onboard LAN Port 1 Onboard LAN Port 2 Control Station V1 cge0 cge1 V3 EMC Celerra Data Mover trunk NetApp Filer V1 V3 Bond: vif exa exb Application 1 (RX300S6) IPMI V2 V4 eth 2 eth 0 eth 1 Onboard LAN Port 3 Bond 0 Onboard Onboard LAN Port 1 LAN Port 2 Physical View: Redundant Connections Cisco Switch A Switchgroup Cisco Switch B The picture shows the physical and logical view of networking in FlexFrame with respect to a single pool. That is: An Application is assigned to a pool and connected to the pools client, server and storage VLAN. When running on ESX server for every connection an interface is created on the vswitch, when running native the interfaces for client and server LAN are created Network Design and Configuration Guide 11

16 Concept and Design as VLAN interface on top of the bond. The storage LAN is also used for PXE boot and assigned native because no tags can be handled in this early phase. Therefore the storage LAN is accessible via the bond directly. An ESX server is connected to all VLANs necessary for the Application s and to the control LAN which is assigned native. IPMI interfaces of rack servers and management blades of BX cabinets are connected native to the control LAN. A Control is connected to all VLANs, the control LAN is native. A NAS system is connected to all VLANs, all VLANs are tagged Uplink Connection Switch 1 Switch 2 SwitchGroup FlexFrame can assign uplinks to a switch group. An uplink consists of ports from different switch group members building a channel and carries the traffic of all used VLANs. An uplink is used to connect a switch group to another switch group or to the customer corporate LAN. The supported topologies stick to the following rules: A CAT3750-STACK has maximal one uplink and is connected to another CAT3750-STACK or to a NEXUS5x00-VPC or to the customer corporate LAN. A NEXUS5x00-VPC may have more than one uplink but maximal one of them is connected to another NEXUS5x00-VPC or to the customer corporate LAN. Each other uplink is connected to a CAT3750-STACK. 12 Network Design and Configuration Guide

17 Concept and Design Following the above rules ensures a topology without loop. Nevertheless, the Cisco per default enabled rapid per VLAN spanning tree (rapid-pvst) algorithm is used to avoid loops which may appear e.g. in early phase when booting a switch or by misconfiguration especially because FlexFrame does not check whether the rules are followed. Examples for supported topologies are given in chapter 3. When connected to customer corporate LAN the connected customer ports must also build a channel. The channel must be compliant with IEEE 802.3ad (LACP) and all used VLANs must be allowed tagged Client LAN Connection For accessing a FlexFrame infrastructure solution connection to the client LAN must be established. In general this means the client LAN must be available in the context of a router and appropriate routing must be configured. This is a manual task and the customer decides which network should have access to a special client LAN. To bring a client LAN in the context of a router different ways are available: a) If uplinks of switch groups are connected to the customer corporate LAN the client LAN is already available and no further arrangements are necessary. b) Special ports for client LAN connections are configured. These CLAN ports are thought being connected to a router and therefore configured as spanning tree edge ports. The ports can be distributed over several switch groups and it is possible to have particular ports for every client LAN. Only Ports of switches of a CAT3750-STACK are allowed to be used as CLAN ports. See below an example how CLAN ports configured for a single client LAN can be used. c) Special ports for a client LAN uplink channel are configured. This method is only available after request for special release and can be performed according a description paper. Network Design and Configuration Guide 13

18 Concept and Design outside FlexFrame Virtual Gateway Adress IP3 Layer 3 port IP1 Layer 3 port IP2 Switch 1 uplink channel Switch 1 Switch 2 Switch 2 SwitchGroup 1 SwitchGroup 2 The ports configured for client LAN connection are connected to Layer 3 ports of different Layer 3 switches. An IP address is assigned to every Layer 3 port and the Layer 3 switches are configured to build a logical router e.g. via HSRP. The logical router supports a virtual IP address (IP3) which is used as standard gateway for the special client LAN. 14 Network Design and Configuration Guide

19 3 FlexFrame Network Versions The FlexFrame network design is flexible enough to meet requirements from small to enterprise size networks. In the following some examples are given. 3.1 Small Version This version is based on one switch group as there is no need for splitting FlexFrame into different data centers providing more switch ports as one switch group can offer. client LAN Switch 1 Switch 2 CAT3750- STACK All devices are connected to a single CAT3750-STACK. The connection of the devices is shown simplified only with regard to the redundant connections for data communication. For client LAN connection two ports are configured on different members of the switch group. No uplink needs to be configured. 3.2 Directly Connected Versions Topologies according to the rules mentioned in 2.10 with more than one switch group and without any uplink connection to the customer corporate LAN are called directly connected versions. The following version may meet the requirements if FlexFrame has to be split into two data centers or one switch group might have not enough ports for all devices Network Design and Configuration Guide 15

20 FlexFrame Network Versions Example1: Two CAT3750-STACKs client LAN Switch 1 Uplink Channel Switch 1 Switch 2 Switch 2 CAT3750- STACK CAT3750- STACK All devices are connected to one of two CAT3750-STACKs. The connection of the devices is shown simplified only with regard to the redundant connections for data communication. For client LAN connection one port is configured on every switch group. The switch groups are connected via uplink channel a) directly without any other network Layer2 device between per copper or for greater distance per fiber optic cable. b) via network devices allowing a Layer2 connection e.g. DWDM for even greater distance. In this case the network devices between must be transparent to the switch groups, especially regarding link state. That means, if a port is shutdown on one switch group the corresponding port on the other switch group must see a link down event. Also if the connection between is broken both ports must see a link down event. 16 Network Design and Configuration Guide

21 FlexFrame Network Versions Example 2: single NEXUS5000-VPC with one CAT3750-STACK client LAN Switch 1 Uplink Channel Switch 1 Switch 2 CAT3750- STACK Switch 2 NEXUS5000- VPC The CAT3750-STACK is directly connected to the NEXUS5000-VPC without any other network Layer2 device between. The service LAN ports of rack servers and the control nodes and the management ports of the nexus switches are connected to the CAT3750- STACK. The end systems connected to NEXUS5000-VPC are 10GbE-connected. For client LAN connection two ports are configured on different members of the CAT3750- STACK. Network Design and Configuration Guide 17

22 FlexFrame Network Versions Example 3: two NEXUS5000-VPCs each with connection to one CAT3750-STACK client LAN Switch 1 uplink channel Switch 1 Switch 2 Switch 2 Switch 1 NEXUS5000- VPC NEXUS5000- VPC Switch 1 Switch 2 Switch 2 CAT3750- STACK CAT3750- STACK The service LAN ports of rack servers and the control nodes are connected to a CAT3750-STACK which is connected to the NEXUS5000-VPC. The end systems connected to NEXUS5000-VPC are 10GbE-connected. The NEXUS5000-VPC switch groups are connected via uplink channel directly without any other network Layer2 device between per fiber optic cable. via network devices allowing a Layer2 connection e.g. DWDM for greater distance. In this case the network devices between must be transparent to the switch groups, especially regarding link state. That means, if a port is shutdown on one NEXUS5000-VPC switch group the corresponding port on the other NEXUS5000-VPC switch group must see a link down event. Also if the connection between is broken both ports must see a link down event. 18 Network Design and Configuration Guide

23 FlexFrame Network Versions 3.3 Enterprise Version The possibilities of directly connected versions are restricted. Greater configurations need uplinks to customer corporate LAN e.g. the customer core switches. NEXUS5000- VPC NEXUS5000- VPC Uplink Channel Uplink Channel Uplink Channel CAT3750- STACK Uplink Channel CAT3750- STACK CAT3750- STACK Uplink Channel Customer own Core Switch Customer Corporate LAN Customer own Core Switch Each NEXUS5000-VPC switch group is connected with its uplink ports to core switch ports. CAT3750-STACK switch groups may be connected to core switches or to NEXUS5000-VPC switch groups. The switch groups may be connected to different core switches. Like the Uplink the corresponding ports on the core switch(es) must: build a channel carry all VLANs used within FlexFrame transfer traffic without blocking The core switches are not part of FlexFrame, neither of automated configuration nor support. Configuration changes at core switches have to be done manually. There are no messages or instructions from FlexFrame to support this. Network Design and Configuration Guide 19

24

25 4 Configuration Commands As described in Chapter 2 FlexFrame uses high availability configurations concerning network on many devices. This chapter describes how this is done in detail. 4.1 Linux Bonding Interface On the control nodes the bonding configuration is performed via configuration files /etc/sysconfig/network/ifcfg-bond0: STARTMODE=onboot BOOTPROTO=static IPADDR= NETMASK= NETWORK= BROADCAST= BONDING_MASTER=yes BONDING_SLAVE_0=eth0 BONDING_SLAVE_1=eth1 BONDING_MODULE_OPTS="miimon=100 mode=6" /etc/sysconfig/network/ifcfg-eth0 /etc/sysconfig/network/ifcfg-eth1 (both are the same): STARTMODE=auto On application nodes running native the bonding configuration is performed during boot via commands. The network parameters are obtained through DHCP and include the IP address for the storage LAN. Network Design and Configuration Guide 21

26 Configuration Commands 4.2 Linux VLAN Interface VLAN configuration is performed via configuration files /etc/sysconfig/network/ifcfg-vlan<vlan ID> e.g. for VLAN 10: /etc/sysconfig/network/ifcfg-vlan10: STARTMODE='onboot' BOOTPROTO='static' IPADDR= NETMASK= NETWORK= BROADCAST= ETHERDEVICE='bond0' WIRELESS='no' 4.3 NetApp Filer Configuration The configuration has to be done manually e.g. via serial cable (see chapter Sample Installation of ONTAP 7G on a FAS 3100 Series Filer in the manual Installation of a FlexFrame Environment ). 4.4 EMC Celerra Configuration The configuration has to be done manually e.g. via serial cable (see chapter Support of EMC NAS System in the manual Installation of a FlexFrame Environment ). 22 Network Design and Configuration Guide

27 4.5 Network Switch Configuration The network switch configuration is performed via commands. The following tables shows Cisco IOS and Cisco NX-OS configuration commands for different purposes. Cisco Catalyst 3750 Family Configuration (IOS commands) Basic configuration VLAN configuration for a system with a pool pool1 port configuration for a data port of a rack server used as application node service password encryption enable password secret clock timezone CET 2 clock summer-time cet recurring last Sun Mar 2:00 last Sun Oct 3:00 vtp mode transparent udld aggressive no ip http server snmp-server community public RO spanning-tree mode rapid-pvst spanning-tree loopguard default no spanning-tree optimize bpdu transmission spanning-tree extend system-id vlan 10 name client-pool1 vlan 11 name storage-pool1 vlan 12 name server-pool1 vlan 13 name control interface GigabitEthernet1/0/1 description rx300s6 datanic 1 switchport trunk encapsulation dot1q switchport trunk native vlan 11 switchport trunk allowed vlan switchport mode trunk no ip address speed auto no mdix auto udld port storm-control broadcast level 20 storm-control multicast level 10 no shutdown Network Design and Configuration Guide 23

28 Configuration Commands Cisco Catalyst 3750 Family Configuration (IOS commands) port configuration for a data port of a rack server used as esxserver port configuration for a IPMI port of a rack channel configuration e.g. for an uplink spanning-tree portfast trunk interface GigabitEthernet1/0/1 description rx300s6 switchport trunk encapsulation dot1q switchport trunk native vlan 13 switchport trunk allowed vlan 10-12,20-22 switchport mode trunk no ip address speed auto no mdix auto udld port storm-control broadcast level 20 storm-control multicast level 10 no shutdown spanning-tree portfast trunk interface GigabitEthernet1/0/1 description rx300s6 mgmtnic switchport trunk encapsulation dot1q switchport trunk native vlan 13 switchport trunk allowed vlan 13 switchport mode trunk no ip address speed auto no mdix auto udld port storm-control broadcast level 20 storm-control multicast level 10 no shutdown spanning-tree portfast trunk interface PortChannel 1 description ISL Uplink switchport trunk encapsulation dot1q switchport trunk allowed vlan 10-13,20-22 switchport mode trunk no shutdown interface GigabitEthernet1/0/1 description ISL Uplink switchport trunk encapsulation dot1q switchport trunk allowed vlan 10-13,20-22 switchport mode trunk no ip address 24 Network Design and Configuration Guide

29 Configuration Commands Cisco Catalyst 3750 Family Configuration (IOS commands) speed auto mdix auto udld port storm-control broadcast level 20 storm-control multicast level 10 cdp enable channel-group 1 mode active no shutdown interface GigabitEthernet2/0/1 description ISL Uplink switchport trunk encapsulation dot1q switchport trunk allowed vlan 10-13,20-22 switchport mode trunk no ip address speed auto mdix auto udld port storm-control broadcast level 20 storm-control multicast level 10 cdp enable channel-group 1 mode active no shutdown Save configuration copy running-config startup-config For details see the Catalyst 3750 Switch Software Configuration Guide at Cisco Nexus 5000 Family Configuration (NX-OS commands) Basic configuration no feature telnet no telnet server enable cfs eth distribute feature udld feature lacp feature vpc vpc domain 2 peer-keepalive destination vrf context management ssh key rsa 2048 udld aggressive spanning-tree mode rapid-pvst Network Design and Configuration Guide 25

30 Configuration Commands Cisco Nexus 5000 Family Configuration (NX-OS commands) spanning-tree loopguard default VLAN configuration for a system vlan 10 with a pool pool1 name client-pool1 vlan 11 name storage-pool1 vlan 12 name server-pool1 vlan 13 name control port configuration for a data port of a rack server used as application node port configuration for a data port of a rack server used as esxserver vpc configuration e.g. for an uplink (same vpc number on both nexus switches of the switch group, per convention equal to port-channel number) interface Ethernet1/1 description rx300s6 datanic 1 switchport trunk native vlan 11 switchport trunk allowed vlan switchport mode trunk spanning-tree port type edge trunk udld aggressive storm-control broadcast level 20 storm-control multicast level 10 no shutdown interface Ethernet1/1 description rx300s6 switchport trunk native vlan 13 switchport trunk allowed vlan 10-13,20-22 switchport mode trunk spanning-tree port type edge trunk udld aggressive storm-control broadcast level 20 storm-control multicast level 10 no shutdown interface port-channel2 description ISL Uplink switchport trunk allowed vlan 1,10-13,20-22 switchport mode trunk vpc 2 storm-control broadcast level 20 storm-control multicast level 10 no shutdown interface Ethernet1/1 description ISL Uplink switchport trunk allowed vlan 1,10-13,20-22 switchport mode trunk udld aggressive storm-control broadcast level 20 storm-control multicast level Network Design and Configuration Guide

31 Configuration Commands Cisco Nexus 5000 Family Configuration (NX-OS commands) channel-group 2 mode active no shutdown Save configuration copy running-config startup-config For details see the Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration Guide at Network Design and Configuration Guide 27

32

33 5 Abbreviations DART DHCP IP LAN MAC MII NAS NIC ONTAP PXE SPOF TFTP UDP VLAN VPC Data Access in Real Time Dynamic Host Configuration Protocol Internet Protocol Local Area Network Media Access Control Media Independent Interface Network Attached Storage Network Interface Card Open Network Technology for Appliance Products Preboot Execution Environment Single Point Of Failure Trivial File Transfer Protocol User Datagram Protocol Virtual Local Area Network Virtual Port Channel Network Design and Configuration Guide 29

34

35 6 Glossary Application A host for applications (e.g. SAP instances db, ci, agate, wgate, app etc.). This definition includes Application Servers as well as Database Servers. Blade A special form factor for computer nodes. Celerra NAS system of EMC. Client LAN Virtual network segment within FlexFrame, used for client-server traffic. Computing From the SAP ACI perspective: A host that is used for applications. Control LAN Virtual network segment within FlexFrame, used for system management traffic. Control A physical computer system, controlling and monitoring the entire FlexFrame landscape and running shared services in the rack (dhcp, tftp, ldap etc.). Control Station A Control in an SAP ACI environment. DART Operating system of Celerra data movers (Data Access in Real Time). Dynamic Host Configuration Protocol DHCP is a protocol for assigning dynamic IP addresses to devices on a network. Dynamic Host Configuration Protocol server A DHCP server provides configuration parameters specific to the DHCP client host, required by the host to participate on the Internet. EMC NAS Network attached storage for file systems of EMC. Ethernet A Local Area Network which supports data transfer rates of 10 megabits per second. Filer Network attached storage for file systems of NetApp. FlexFrame A joint project in which the main partners are SAP, Network Appliance, Intel and Fujitsu. Network Design and Configuration Guide 31

36 Glossary FlexFrame TM for SAP FlexFrame TM for SAP is a radically new architecture for SAP environments. It exploits the latest business-critical computing technology to deliver major cost savings for SAP customers. FlexFrame internal LAN Switch Cisco network switches which are integral part of the FlexFrame for SAP hardware configuration and which are automatically configured by the FlexFrame for SAP software. Gigabit Ethernet A Local Area Network which supports data transfer rates of 1 gigabit (1,000 megabits) per second. Host name The name of a node (assigned to an interface) that is resolved to a unique IP address. One node can have multiple host names (cf. node name). In SAP environments host names are currently limited to 13 alphanumeric characters including the hyphen ( - ). The first character must be a letter. In the SAP environment host names are case-sensitive. Internet Protocol Address A unique number used by computers to refer to each other when sending information through networks using the Internet Protocol. Local Area Network A computer network that spans a relatively small area. Most LANs are confined to a single building or group of buildings. However, one LAN can be connected to other LANs over any distance via telephone lines and radio waves. A system of LANs connected in this way is called a Wide Area Network (WAN). Local host name The name of the node (physical computer); it can be displayed and set using the command /bin/hostname. Media Access Control address An identifier for network devices, usually unique. The MAC address is stored physically on the device. NAS system Network Attached Storage of any vendor (in our context: EMC NAS or NetApp Filer). Network Attached Storage A data storage device that is connected via a network to one or multiple computers. Network Interface Card A hardware device that allows computer communication via networks. A physical computer system controlled by an OS. 32 Network Design and Configuration Guide

37 Glossary name The name of a physical node as returned by the command uname -n. Each node name within a FlexFrame environment must be unique. Open Network Technology for Appliance Products The operating system of Network Appliance Filers. Physical host Name of a physical computer system (node). Preboot Execution Environment An environment that allows a computer to boot from a network resource without having a local operating system installed. Server A physical host (hardware), same as node. Service A software program providing functions to clients. Storage LAN A virtual LAN segment within a FlexFrame environment, carrying the traffic to NAS systems. Trivial File Transfer Protocol A simple form of the File Transfer Protocol (FTP). TFTP uses the User Datagram Protocol (UDP) and provides no security features. It is often used by servers to boot diskless workstations, X-terminals, and routers. TFTP server A simple FTP implementation. Virtual host The name of the virtual host on which an application runs; it is assigned to a physical node when an application is started. Virtual Local Area Network A VLAN is a logically segmented network mapped over physical hardware according to the IEEE 802.1q standard. Network Design and Configuration Guide 33

38 7 Index C concept and design 3 E EMC Celerra configuration 22 F FlexFrame network versions 15 enterprise version 19 medium version 15 small version 15 L link aggregation 3, 6, 7, 8, 9, 12, 13 Linux bonding interface 21 VLAN interface 22 N NetApp Filer configuration 22 network speed 4 network switch configuration 23 node configuration 21 R related documents 2 V virtual LAN 10 Network Design and Configuration Guide 34