10 GbE Gateway (srb-20210g Module)

Transcription

1 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0

2 NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON INFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE., Inc. 350 Oakmead Parkway Suite 100 Sunnyvale, CA U.S.A. Tel: (408) Fax: (408) Ltd PO Box 586 Hermon Building Yokneam Israel Tel: Fax: Copyright All rights reserved. Mellanox, BridgeX, ConnectX, SwitchX, CORE-Direct, InfiniBridge, InfiniHost, InfiniScale, PhyX, Virtual Protocol Interconnect and Voltaire are registered trademarks of, Ltd. FabricIT and MLNX-OS are trademarks of, Ltd. All other trademarks are property of their respective owners. 2 Document Number: DOC A01

3 Contents Version 2.0 Contents Chapter 1. Introduction Overview Physical Description Plug-in Module LED Indicators Features Basic Features New in Version Functional Description High Performance InfiniBand-to-GbE Connectivity Comprehensive Management srb-20210g Architecture Supported Protocols srb-20210g Application Diagram Traffic Flow Management Overview Technical Specifications General Gigabit Ethernet (GbE) InfiniBand IETF Protocols Management Interfaces Physical Environmental Certifications Chapter 2. Initial Configuration Prerequisite Installation Tasks Configuring the srb-20210g Gateway in the ISR 2XXX Managing the Unit over Fast Ethernet (Out-of-Band) or InfiniBand (Inband) Handling Different Subnets Configuring an srb-20210g Module (10 GbE to InfiniBand Bridge) Setting Time and Date Default Settings Users and Passwords Verifying the Configuration Chapter 3. Mellanox Device Manager (VDM) Overview

4 Version 2.0 Contents 3.2 Device Manager Description Logging-in to the Device Manager The Switch Device Manager Main Window The Main Menu Device or Navigation Tree (Left Panel) srb-20210g Gateway Graphical Representation Right-Click Menus Left-Click Option Color-Coding Table Trap Table Resizing the Trap Table Rearranging Columns Viewing and Hiding Events Acknowledging Events Displaying Additional Data for an Event Chapter 4. Device Information Getting Started with the srb-20210g Device Manager Device Information via the CLI srb-20210g Port Information Viewing the System Time and Date (CLI only) Viewing srb-20210g Temperature (Switch CLI only) Setting the Uptime Checking the Device Software Version Device Version via the GUI Device Version via the CLI Chapter 5. Device Configuration IP Settings Overview Setting the I/O IP via the GUI Configuring the srb-20210g IP via the Switch CLI Resetting the srb-20120g Resetting the I/O via the GUI Resetting the I/O via the CLI User Settings (Setting Passwords) User Settings via the GUI User Settings via the CLI

5 Contents Version 2.0 Chapter 6. Advanced Configuration Updating the Software Identify the File Location for Software Update Updating the VDM (GUI) - Gateway Version Automatic Upgrade Remote Management Settings (SNMP) Setting Remote Management via the GUI Setting Remote Management via the CLI Remote Configuration Settings (FTP/SCP) Setting Remote Configuration via the GUI Setting Remote Configuration via the CLI Backing up the Configuration via the GUI and the CLI Setting Link Aggregation (LAG) Setting Link Aggregation via the GUI Setting Link Aggregation via the CLI Configuring I/O Interfaces Cluster Configuration Configuring a Cluster via the GUI Configuring a Cluster via the CLI Cluster Mode (srb-20210g High Availability) Setting srb-20210g Algorithms Loop Prevention Setting the Event Policy Filters Port MTU Configuration (Jumbo Frames) Setting the MTU via the GUI Setting the MTU via the CLI MIB Support Security Support Overview Enabling/Disabling Non-secure Protocols Chapter 7. Setting-up Networks Setting up a Network/Interface via the GUI Network Table Right-click Menu Defining (Adding) a New Network Viewing Network Information Removing Networks Updating the Default Gateway Removing a Default Gateway Refresh Table

6 Version 2.0 Contents 7.2 Setting up a Network/Interface via the CLI Viewing Networks Adding Networks Removing Networks Setting the IP and Broadcast Address Setting the Default Gateway Chapter 8. Traffic Flow Management Overview Setting TFM Setting TFM via the GUI Chapter 9. Static Route Configuration Overview Routing Configuration Setting Routing via the GUI Routing Configuration via the CLI Chapter 10. Logs and Statistics Using exportlog Viewing Port Counters Statistics Viewing Port Counters Statistics via the GUI Viewing Port Status and Statistics via the CLI Statistics Logs Viewing Traps/Events Viewing Events Viewing Traps Chapter 11. Multicast Settings Overview Definitions Multicast Settings MC-Table Session Chapter 12. Mellanox Fabric Manager (VFM) Overview Viewing I/O System Information via the GridVision Fabric Manager I/O System Overview Parameter Description Viewing an I/O via the CLI Connecting to an I/O Device via the CLI Viewing I/O Port Properties Viewing I/O Network (Partitioning) Information

7 Contents Version 2.0 Chapter 13. Partitioning Overview Brief Functional Description Mechanisms Used to Create and Enforce Partitions Quick Settings Switch Settings (GUI) srb-20210g Settings (GUI) Setting Partitioning via the GridVision Fabric Manager GUI Creating a Network Viewing Network Information Assigning Server Ports to a Network Deleting Server Ports from a Network Assigning Network Membership Mapping an InfiniBand Partition to an Ethernet VLAN via the 10 GbE Gateway Multiple Selection for Partitioning Configuration Setting Partitioning via the CLI Quick Steps to set up a Network and Perform Partitioning Creating a Network Deleting a Network Viewing Network Details Viewing the Network Error Log File Binding a Port GUID to a Network Chapter 14. Command Line Interface (CLI) Management Starting a CLI Management Session Mellanox CLI Model CLI Operating Conventions Keyboard Usage in CLI Operation CLI Command Structure Case-Sensitivity CLI Modes Exec Mode Commands Privileged Mode Commands Statistics Mode Configuration (Config) Commands Cluster Options Diagnostics Interface Options NTP Options

8 Version 2.0 Contents Port Options Remote Options Route Options Security Options SNMP Options Traffic Flow Manager (TFM) Safe Mode CLI Commands Software Upgrade Procedure in Safe Mode CLI Safe Mode

9 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 About this Manual This manual describes how to operate the srb-20210g module, also referred to as Mellanox s 10 GbE gateway. The manual describes the tasks commonly required to configure and use the device. Information regarding the srb-20210g hardware installation can be found in the Mellanox ISR 2004/2012 Grid Director Installation Manual. NOTE: Refer to the release notes for last minute updates and restrictions. Audience This manual is intended primarily for: System administrators who are familiar with the fundamentals of IP based networking, but might not be familiar with the specifics of Mellanox products or the routing protocols supported by Mellanox products System administrators who are responsible for configuring TCP/IP to InfiniBand based connectivity. Document Conventions NOTE: Text presented in this format indicates clarifying information, specific instructions, commentary, sidelights, or interesting points of information. IMPORTANT: Text presented in this format indicates important information regarding a specific feature. CAUTION: Text presented in this format indicates that failure to follow directions could result in damage to equipment or loss of information. Text presented in this manner indicates terminal/cli screen/command format. 9

10 Version 2.0 Introduction Chapter 1. Introduction This chapter introduces the Mellanox InfiniBand-to-10 GbE Gateway (srb-20210g). 1.1 Overview The srb-20210g module (see Figure 1-1 and Figure 1-2) is an InfiniBand-to-10 GbE gateway module, designed as a plug-in unit for installation in the Mellanox ISR 2004/2012 Grid Director. The srb-20210g module enables multiple virtual networks to span across InfiniBand and Ethernet-based networks providing network-agnostic enterprise-class connectivity. With this solution, InfiniBand can be used to accelerate applications without the need to replace Ethernet-based services and devices while allowing the provision of multiple virtual Ethernet and storage networks on from same director class switch. IMPORTANT: The srb-20210g module is used as a bridge in the system. It does not run on several subnets. However, it can be used to interface several subnets using partitions. For more information, refer to Chapter 13, Partitioning. 1.2 Physical Description Plug-in Module The Mellanox srb-20210g is a plug-in module, which consists of the following physical components: 1U-high line card for the Mellanox Grid Director 2012/2004 Twenty-two (22) CX4 20Gbps 4x InfiniBand ports Two (2) 10 GbE XFP cages supporting the following XFPs: 10GBASE-SR and -LR optic XFP MSA modules, LC connector 10GBASE-CX4 Copper XFP MSA modules, CX4 connector. Figure 1-1 shows a Mellanox srb-20210g with empty cages for installation in the ISR 2004/2012. The empty cages can accommodate optic or copper XFP modules. Figure 1-1. srb-20210g Empty Cages 10

11 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Figure 1-2 shows a Mellanox srb-20210g with optical XFP modules installed. Figure 1-2. srb-20210g With Pluggable Optic XFP Modules LED Indicators Three LEDs are located on the face of the module, indicating link status of the combined interfaces in the integrated Management port (mgmt) and overall system status. The LEDs of both the InfiniBand and 10 GbE ports have the same indication types, as follows: Table 1-1. LED Indicators LED Type Function Status Description Power [green] System Power ON, green OFF Internal power is OK Internal power failed Info [green] General purpose LED for management use Blinking OFF Sets the Info LED on a desired module. When you enable this LED through the GUI or CLI, you can go on site and immediately physically locate the module. System [green] System state (controlled by SW) In the GUI, note that the state is also shown by a tooltip when the mouse is over this LED. ON, green Blinking slow Blinking fast 10 GbE Gateway is active 10 GbE Gateway is initializing 10 GbE Gateway fault Software in safe mode Logical link indication (per port) [amber] Logical link (24 Log for the whole srb-20210g) ON Blinking, amber Subnet Manager has configured and activated the link Data traffic over the logical link. OFF The Subnet Manager has not yet configured the link. Physical link indication (per port) (green) Physical link (24 Phy for the whole ON, green Blinking Activated upon physical link Unstable physical link. 11

12 Version 2.0 Introduction LED Type Function Status Description srb-20210g) OFF The port is not connected or there is no physical link. 1.3 Features Basic Features The basic functional features of the srb-20210g module include: High performance InfiniBand-to-10 GbE connectivity using Mellanox-developed silicon Extremely low latency using cut-thru architecture (under two [2] microseconds) IBTA 1.1 compliant Standard IPoIB support VLAN support according to 802.1q (up to 64 IB partitions; see New in Version 2.0) IP Multicast support up to 1024 multicast groups Layer 2/3/4, traffic filtering, and classification Link aggregation, load sharing, and High Availability CLI and GUI management via the Mellanox Fabric Manager (VFM) and the Mellanox Device Manager (VDM) New in Version 2.0 Version 2.0 includes the following improvements and enhancements: Support for different MTU rates on the Ethernet and IB side. This allows the gateway to work in a Jumbo frame environment on the Ethernet side. It also reduces the burden of decreasing the default MTU of all IB hosts in the fabric from 2 Kbytes to 1.5 Kbytes. A loop prevention mechanism has been added for cases when multiple gateways are deployed in a specific application, so that the user would be able to configure them without creating loops on the Ethernet network, while preserving the high-availability requirements. Support for up to 64 partitions/vlans Support for up to 3,000 multicast groups Enhanced monitoring capabilities. 1.4 Functional Description High Performance InfiniBand-to-GbE Connectivity The srb-20210g core capabilities rely on the Mellanox in-house developed silicon used for forwarding traffic between InfiniBand and Ethernet infrastructures. This custom silicon enables the Mellanox srb-20210g to achieve wire-speed protocol processing with minimal latency, while transparently providing upper-layer traffic classification capabilities. In addition, multiple 10 GbE ports can be aggregated to scale connectivity and meet various application requirements. 12

13 10 GbE Gateway (srb-20210g Module) User Manual Version Comprehensive Management The rich functionality provided by the Mellanox srb-20210g is fully integrated into GridVision Grid Interconnect Management Software that is embedded in Mellanox InfiniBand switches and gateways. GridVision is an easy-to-use management suite that provides access through a Web-based GUI, a command line interface (CLI) and SNMP. Out-of-band management includes InfiniBand, Ethernet and EIA/TIA-232 connections. The Mellanox srb-20210g Gateway management is also integrated with leading third party grid and on-demand management platforms srb-20210g Architecture All data traffic is handled by the srb-20210g hardware (Silicon) to achieve the highest possible throughput, with the lowest possible latency. Management traffic is handled in software by the CPU. Software Ethernet Slow Path Silicon Fast Path Slow Path InfiniBand Figure 1-3. srb-20210g Conceptual Block Diagram Supported Protocols The srb-20210g supports the following protocols: 10 Gigabit Ethernet over fiber - IEEE 802.3ae - 10GBASE-SR and 10GBASE-LR. 10 Gigabit Ethernet over twin-axial cable IEEE 820.3ak 10GBASE-CX4 InfiniBand 1.1 compliant IETF RFC 4391 Transmission of IP over InfiniBand (IPoIB) IETF RFC 4392 IP over InfiniBand (IPoIB) Architecture SNMP v2c: IETF-RFC190x Java 1.6 for the GUI. IPoIB (IP-over-InfiniBand) is a protocol for encapsulating data requests and responses over InfiniBand. The srb-20210g uses IP over InfiniBand (IPoIB) protocols in order to provide IP link between Ethernet network and InfiniBand based network. The tunneling functionality is achieved by Layer II address translation of an InfiniBand address to an Ethernet address (MAC) and vice versa. 13

14 Version 2.0 Introduction srb-20210g Application Diagram Figure 1-4 shows an example of an IPoIB network. Ethernet NAS - NFS, Ethernet SAN - iscsi Clustered File System Router LAN/WAN Figure 1-4. IPoIB Network Response/Request Routing The srb-20210g provides transparent bridging of Ethernet traffic to servers on the InfiniBand fabric and vice versa, as if they were directly attached to the clients over a local network. This network behavior resembles a bridge methodology Traffic Flow Management Overview The Traffic Flow Management (TFM) feature allows a network administrator to determine rules for handling IP packets arriving at the srb-20210g, based on their source or destination IP address and ports. The administrator can define services and rules to control how the srb-20210g processes these packets. This processing includes: Forwarding the packet is forwarded to a destination host either on the Ethernet side or the InfiniBand side Drop the packet is discarded. The CLI add and remove CLI commands are used for managing the packet handling rules in the srb-20210g. The user can define rules or use the default rules. If no new rule was added or if there was no change to the default rules, the packets are handled by a default method. For further details, refer to the Traffic Flow Manager (TFM), Chapter Technical Specifications General Line board for Mellanox Grid Director ISR 2012 and ISR 2004 chassis Power and info LED indicators Full non-blocking connectivity Gigabit Ethernet (GbE) IEEE 802.3ab, and 802ad (link aggregation) compliant Two 10GbE interconnect ports, XFP interfaces IEEE P802.3ak and IEEE P802.3ae 10GBASE-SR, 10GBASE-LR, 10GBASE-CX4 14

15 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Port LED indicators: Physical connectivity and traffic indication Support for up to 64 partitions/vlan groups Support for Jumbo frames InfiniBand 22, auto-sensing 20/10 Gbps (4X) InfiniBand ports Connectors: 8-pair MicroGigaCN / LANELINK F CONN Support hot-pluggable media converters for optical InfiniBand connectivity. Port LED indicators: Physical connectivity and logical connectivity IETF Protocols TCP/IP: IETF RFC-793, IETF RFC-791, IETF RFC-768, RFC-926, RFC-1812, RFC-1027 IPoIB: IETF RFC 4391, 4392 VLAN support according IEEE 802.1q (up to 64 VLANS) IP multicast (IETF RFC 3171) and IGMPv2 (up to 4000 multicast groups) SNMP v2c: IETF RFC190x Management Interfaces Full management access is available via the Grid Director chassis management or the Mellanox GridVision management software Physical Mellanox Grid Director ISR 2012 and ISR 2004 Line Board compatible Supports any combination of slb-2024 and srb-20210g Line Boards Power and cooling through ISR 2012 or ISR 2004 chassis Environmental Operating Temperature: 32 to 104 F (0 to 45 C) Humidity: 15 to 80%, non-condensing Altitude: 0 to 9843 ft (3000m) Storage Temperature: -13 to 158 F (-25 to 70 C) Humidity: 5 to 90% non-condensing Altitude: 0 to 15,000 ft (4570m) Certifications Full safety and EMC as a part of the Mellanox Grid Director ISR 2012 and ISR

16 Version 2.0 Initial Configuration Chapter 2. Initial Configuration This chapter provides instructions on how to perform the basic initial configuration settings for the srb-20210g. These steps are typically performed when installing a new device. 2.1 Prerequisite Installation Tasks IMPORTANT: Before configuring the Mellanox srb-20210g for the first time, make sure you have completed the hardware installation according to the relevant Mellanox Installation Manual. The srb-20210g is managed through the Switch chassis management, via either inband (through the InfiniBand interface) or out-of-band (through the Ethernet interface) connection. For more information on Inband/out-of-band management, refer to the switch (ISR 2xxx) user manual. 2.2 Configuring the srb-20210g Gateway in the ISR 2XXX NOTES: Do not physically connect any of the srb-20210g GbE ports to the same physical network as the switch s Ethernet management port. In order to prevent broadcast packet loops, the switch s Ethernet management port must be physically-separated from the GbE network and InfiniBand fabric. The management interface of the srb-20120g (smb-mng) must be configured with a unique IP address on the same network with the Ethernet/InfiniBand management interface of the switch. Refer to the GridVision Integrated Documentation Suite (First-Time Configuration) for further details Managing the Unit over Fast Ethernet (Out-of-Band) or InfiniBand (Inband) When the system is configured to manage a chassis containing an srb-20210g, and the Fast Ethernet is the selected management interface in the ISR 2XXX Grid Director, the smb automatically enters into Proxy-ARP mode. This allows the connection between the manager that resides on the ETH network and the srb-20210g residing on the InfiniBand network. IMPORTANT: Make sure to configure the srb-20210g management (smb-mng) interface on the same subnet that was defined for the switch management (Fast Ethernet) interface. The management interface of the srb-20120g (smb-mng) must be configured with a unique IP address on the same network with the Ethernet/InfiniBand management interface of the switch. Refer to the GridVision Integrated Documentation Suite (First-Time Configuration) for further details. 16

17 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Configure the interfaces and srb-20210gs as follows: Step 1 Step 2 Step 3 Step 4 Configure the Switch smb Fast Ethernet interface (IP address, subnet mask), as described in the Switch User Manual (First-Time Configuration Section). Configure the Switch InfiniBand interface to the same values of the Fast Ethernet interface, the same IP address and same subnet mask, as described in the Switch User Manual (First-Time Configuration Section). Make sure that the IP address of each I/O Gateway module is set on the same subnet defined for the Fast Ethernet interface. Configure the gateways as detailed in the following procedure. NOTE: Run the following command to verify that the Switch default gateway is already set: ISR (config-if)# ip-address show ISR (config-if)# ip-address show port selection Ethernet ip mask broadcast DHCP client disabled default-gw Ip is: Link-speed auto-negotiation. To set the srb-20210g IP address and Default Gateway (Switch side): Use the following table to enter the appropriate commands: Commands Procedure From the Switch prompt: Step 1 guest a. Enter the guest mode. The default password for the guest user is voltaire. admin b. Enter the Privileged mode as follows: Type enable and enter the admin password (the default password for the admin user is ) OR Enter the privileged mode directly. The default password for the admin user is Note: When connecting to the srb-20210g via the switch using the io connect command, you are already in admin mode and do not need a password. Step 2 config Enter the Configuration mode from the Exec mode. Step 3 chassis Enter Chassis Configuration mode. 17

18 Version 2.0 Initial Configuration Commands Procedure Step 4 io show Use the io show CLI command to view the srb-20210g gateways and mark the position of the desired I/O to connect to it. Step 5 io ip-address set slot-number position-number ip-address netmask Sets the IP address of the I/O Gateway the ISR 2XXX Switch. switch(config-chassis)io ip-address set Step 6 io connect Connect to the selected interface (Enters as admin user to the srb-20210g Gateway). switch(config-chassis)io connect 4 1 You will now get the srb-20210g admin prompt. If the switch and the host are in the same subnet, the srb-20210g is now ready for operation Handling Different Subnets This section explains how to set the srb-2021g when the switch and the host are in different subnets. IMPORTANT: In case the switch and the host are in different subnets, you must set the default gateways for each I/O Module, as described in the following section. To set the srb-20210g IP address and Default Gateway on the srb-20210g side: From the srb-20210g prompt: Step 1 config Enter the srb-20210g Configuration mode from the srb-20210g Exec mode. Step 2 interface Enter the srb-20210g Interface mode to view which interfaces are available. Step 3 ip-address set [alias] [ip-address] [netmask] [optional broadcast] Set the IP address for the default interface (if0) srb-20210g-6044(config-if)# ip-address set ip-address set [alias] [ip-address] [netmask] [optional broadcast] 18

19 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Step 4 interface show all View all the interfaces in the srb-20210g Step 5 exit Exits the Interface menu. 10G (config-if)# interface show all entry alias ip mask broadcast vlan pkey(hex) trunk smb-mng no 0x7fff No port 2 if no 0x7fff P1-2 Step 6 route Enter the srb-20210g Configuration Route menu. Step 7 default-gw set [row [1..10]][Gateway][Iface] If you are setting the management through fast Eth (inband) then you must set the default gateway address to the smb-mng management interface. This command sets the default gateway as shown in the example below. This will add the default gateway to the Route Table. Set the default gateway of the management as shown in the following examples. This will add the default gateway to the Static Route Table. IMPORTANT: If you are setting the management through fast Ethernet interface (out-of-band), then you must set the default gateway address to the smb-mng management interface. If you are setting the management through InfiniBand (inband), then you must set the default gateway address to the if0 management interface. Example (Out-of-band Ethernet Interface): 10G-Card(config-route)# default-gw set smb-mng Expected Result: srb-20210g-6044(config-route)# static-route show entry destination netmask gateway interface smb-mng Example (Inband InfiniBand Interface): 10G-Card(config-route)# default-gw set if0 Expected Result: srb-20210g-6044(config-route)# static-route show entry destination netmask gateway interface if0 Step 8 end Exits to the Switch chassis mode. The srb-20210g is now ready; traffic is set to be transferred. 19

20 Version 2.0 Initial Configuration 2.3 Configuring an srb-20210g Module (10 GbE to InfiniBand Bridge) IMPORTANT: This section is only relevant if you intend to use multiple VLANs. You can separate traffic between different ports (no LAG) by adding bridges/vlans. The srb-20210g supports up to 16 bridges located on different partitions/vlans. To configure a GbE/InfiniBand Bridge: Commands Procedure From the Switch prompt: Step 1 guest a. Enter the guest mode. The default password for the guest user is voltaire. b. Enter the Privileged mode as follows: Type enable and enter the admin password (The default password for the admin user is ) OR admin Enter directly the privileged mode. The default password for the admin user is Note: When connecting to the srb-20210g via the switch using the io connect command, you are already in admin mode and do not need a password. Step 2 config Enter the config mode (from the privileged mode). Step 3 chassis Enter the chassis mode. Step 4 io show View (and note) the I/O (srb-20120g) position in the switch. Step 5 io connect {slot-number position-number} From the srb-20210g prompt: Connect to the desired I/O (srb-20120g). The srb-20210g prompt appears. Step 6 [password] At the srb-20210g prompt, enter the password. Default password is Step 7 config Enter the config mode (from privileged mode). Step 8 interface Enter the interface configuration mode. Note: Before going to the following step, you must first hand out this PKey to the SRB-20210G from the Switch. Refer to the GridVision User Documentation for further information. 20

21 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Commands Step 9 interface add [alias] [tag - no,[0-4094]] [P1,P2,P1-2][pKey 0x0001-0x7fff] [Ip] [Mask] Procedure Define an interface with the following parameters: Example 10G(config-if)# interface add V5 108 P1-2 0x Alias VLAN tag Trunk pkey IP mask A user-defined name for the interface, using any combination of letters and numbers. The VLAN tag assigned to the interface. Port aggregation in LAG mode (full, separate). Enter the port numbers, as required. [P1] or [P2] = separate [P1-2] = full Partition Key (equivalent to a VLAN on the InfiniBand side) Note: The user must first hand out this PKey to the srb-20210g from the Switch. IP address of the interface. Subnet mask of the interface. To configure the PKey, refer to Partitioning, in Chapter Setting Time and Date The time of the srb-20210g is synchronized with the switch. Setup is performed through the switch. Refer to the Switch User Documentation for details on how to set up the time and date. 2.5 Default Settings Table 2-1 provides a list that you can use to assist you in the installation process. Once you have completed the list, you are ready to continue with first time configuration of the srb-20210g. Table 2-1. srb-20210g InfiniBand Installation Check List System Parameter Guest user password. Provides access to the guest mode. Admin user password. Provides immediate access to the privileged mode. If logged in as a guest, you must type enable and enter the admin password to proceed to the privileged menus. Default voltaire management interface (If0) IP address

22 Version 2.0 Initial Configuration System Parameter Default Management interface subnet mask Users and Passwords There are two types of users: guest admin a. Enter the guest mode. The default password for the guest user is voltaire. b. Enter the Privileged mode as follows: Type enable and enter the admin password (The default password for the admin user is ) OR Enter directly the privileged mode. The default password for the admin user is Note: When connecting to the srb-20210g via the switch using the io connect command, you are already in admin mode and do not need a password. Passwords are changed via the switch CLI and applied to the srb-2021g. Guest and Admin user passwords are used to access the Ethernet management interface via the CLI. NOTE: The factory default passwords to enter the CLI are as follows: Guest User: Voltaire (limited access) Admin User: (unlimited access) enable mode (access to all modes from Guest Mode, for the admin user) To configure passwords: Commands Procedure Step 1 guest a. Enter the guest mode. The default password for the guest user is voltaire. b. Enter the Privileged mode as follows: Type enable and enter the admin password (The default password for the admin user is ) OR admin Enter directly the privileged mode. The default password for the admin user is Note: When connecting to the srb-20210g via the switch using the io connect command, you are already in admin mode and do not need a password. 22

23 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Step 2 password update [admin,enable,guest, root] Change the passwords for the following users. User Types admin Administrator. This user can access all the menus and configure the system. (Default password for Admin user is ) enable password: This is required if you are logging in as a Guest (in admin mode) but now need to proceed to Privileged menus as an administrator. guest Guest user. This user has only access to the Guest mode Default Password for Guest user is: voltaire. root for internal use only. Press Enter and type a new password (eight characters) and retype to confirm. srb-20210g-61f0# password update admin Insert new (up to 8 characters) Enter password :******* Please retype new Enter password :******* OK srb-20210g-61f0# 2.7 Verifying the Configuration Verify the system parameters using the steps below. Table 2-2. Verifying the Configuration Commands Procedure Step 1 guest a. Enter the guest mode. The default password for the guest user is voltaire. b. Enter the Privileged mode as follows: Type enable and enter the admin password (The default password for the admin user is ) OR admin Enter directly the privileged mode. The default password for the admin user is Note: When connecting to the srb-20210g via the switch using the io connect command, you are already in admin mode and do not need a password. Step 2 version show Use this command to verify the version of the current srb-20210g software. Step 3 remote show Use this command to display the remote client configuration. The remote show command shows the details of the server used (among other things) to save the configuration and perform an upgrade. Step 4 srb-20210g-xxxx# (config-if)# interface show Use this command to display interfaces IP addresses and configurations. 23

24 Version 2.0 Initial Configuration Commands Procedure Step 5 clock show Use this command to display the system clock. You can now start working with the basic functions by accessing the srb-20210g via the Switch VFM or via the CLI. Advanced srb-20210g features are detailed further in this manual. If you wish to add Ethernet VLAN mapping to InfiniBand Partitioning you may define more interfaces. For more information, refer to Partitioning in Chapter

25 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Chapter 3. Mellanox Device Manager (VDM) This chapter describes the operation of the Mellanox srb-20210g Device Manager (VDM). The VDM allows the user to configure and provision the gateway, and to view various device properties. 3.1 Overview The Mellanox Device Manager (VDM) is an embedded application with a web-based GUI that provides InfiniBand management functionality for the Mellanox srb-20210g. It enables users to view a graphical representation of the Mellanox srb-20210g, its LED indicators and ports, and to monitor and configure device parameters. The initial configuration of the GUI is performed through the Switch in which the srb-20210g is installed. Refer to the GridVision User Documentation (Part 1) for details on how to perform initial GUI installation and configuration. 3.2 Device Manager Description VDM is an embedded application with a Web-based GUI that provides InfiniBand management functionality for the Mellanox products. It enables users to view a graphical representation of the Switch as a standalone as well as the individual modules installed in the Switch. As such, you can access the srb-20210g device from the ISR 2004/2012. VDM enables you to monitor and configure the srb-20210g parameters. To start the VDM application: Step 1 To access the Grid Director in which the gateway is installed, open your Internet browser and enter the unit s IP address as the URL (for example: The GridVision Integrated VDM Window is displayed, as shown in the following figure. Figure 3-1. GridVision Integrated VDM Window The window displays the following icons: 25

26 Version 2.0 Mellanox Device Manager (VDM) Table 3-1. GridVision Integrated Icons Icon Description Launches the Mellanox Graphic Device Manager. Launches the GridVision Fabric Manager. Step 2 To start the GridVision Device Manager application, click Launch GridVision Graphic Device Manager. The VDM opens showing the Device Manager or the specific Switch. 3.3 Logging-in to the Device Manager The Login window is displayed when the GridVision Device Manager application starts, as shown in Figure 3-2. Figure 3-2. GridVision Login Dialog Box The Login dialog box fields are detailed in Table 3-2. Table 3-2. Login Dialog Box - Fields Description Field IP Address User type Password Description The IP address of the Switch is displayed in the read-only IP Address field. The GridVision Device Manager supports three types of users, each with a different authorization level. Admin user type. This user can perform all configuration and provisioning operations. Default password: Guest user type. This user can only view device configuration and status information. Default password: Voltaire The password for the specific type of user. 26

27 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 NOTE: You can change the SNMP settings by clearing the Use Default SNMP Community Strings check box (see Figure 3-3). Figure 3-3. Login Dialog Box Setting SNMP Community Strings The default Get and Set community string is private. To log in to the Device Manager: Step 1 Step 2 Select the User Type (admin, guest) Enter the Password and click OK. The GridVision Device Manager Main window opens. NOTE: The password is case-sensitive. 3.4 The Switch Device Manager Main Window The Main Window provides access to all functions of the GridVision Device Manager. It also provides a graphical representation of the connected Switch and components, with color-coded visual indicators that display alarms and important events occurring in the Switch. NOTE: The Main Window, and especially the device graphical representation, may differ depending on the specific Switch Device Manager representation. The figure below shows the front of the chassis in the GridVision Integrated Device Manager Main Window. The ISR 2004 was used for the example below. 27

28 Version 2.0 Mellanox Device Manager (VDM) Figure 3-4. Front of the Chassis The figure below shows the rear of the chassis in the GridVision Integrated Device Manager Main Window. The ISR 2004 was used for the example below. Figure 3-5. Rear of the Chassis containing an srb-20210g 28

29 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 In this example, a srb-20210g unit is installed in slot 2. The left panel (Subnet Tree) lists the srb-20210g under slb/srbd, and the right panel shows the srb-20210g information The Main Menu The main menu provides access to the Device Manager of the Switch only Device or Navigation Tree (Left Panel) The VDM Device tree displays the objects in the managed device in a hierarchical format. This includes sfb boards, line boards, smb, spsu, sfu, sctrl boards, srbds (I/O drawers), and gateways. Figure 3-6 shows the Device tree. The ISR 2004 Switch is used here as an example. Figure 3-6. Device Tree The navigation tree is used to select the Device Manager configuration and viewing mode. In order to select a type of view/configuration operation, simply click one of the items in the navigation tree. The information area of the Device Manager main window displays different information depending on your selection. You can expand/collapse the navigation tree to display the various components of the switch. You can show and hide the device tree by selecting View/View Tree from the Main Menu. Categories displayed in the navigation tree under srb-20210g are as follows: Table 3-3. Navigation Tree Categories Category srb-20210g Networks Description Clicking srb-20210g on the Subnet Tree shows the Gateway graphical representation as well as the detailed information. Displays the Networks table, which shows the interfaces and networks defined in the srb-20210g. This window enables you to define and delete networks as well as set the network default gateway. Note that if you need to make any changes to the network, you need to delete it and create it again. Refer to Chapter 7 for a description of this mode. 29

30 Version 2.0 Mellanox Device Manager (VDM) Category Traffic Flow Management Static Route Description Displays the TFM table, which displays the TFM rules defined in the gateway; from the TFM table you can also define, edit, and delete TFM rules. This table also allows you to set various services that can be used when defining the TFM rules. Refer to Chapter 8 for a description of this mode. Displays the Static Route table, which defines the next hop from a gateway to a particular destination. Using the static route mechanism, the user can set rules that will be used as gateways. See Section for a description of this mode srb-20210g Gateway Graphical Representation The following GUI graphical representation of the srb-20210g provides a view of the srb-20210g front panel with two 10 GbE ports and 22 InfiniBand ports. Figure 3-7 shows the graphical representation of the srb-20210g in the Device Manager. Figure 3-7. srb-20210g Graphical Representation The following figure shows the 10G gray area. Figure G Graphical Representation The 10G (gray area) includes two 10 GbE ports. 10 GbE port types can be any of the following: a. Copper b. Fiber short range c. Fiber long range d. Fiber extended range. LEDs LEDs are color coded to indicate the actual operating states. The LEDs correspond to the actual physical LEDs on the device: Green indicates an active physical link Orange indicates an active logical link Gray indicates a non-active link. For more information about LEDs, refer to Section

31 10 GbE Gateway (srb-20210g Module) User Manual Version Right-Click Menus Four right-click menus are available, depending on the area of the GUI where the right-click is performed (see Sections ) Right-Clicking an srb-20210g (Blue Area) When right-clicking the blue area of the srb-20210g at the rear of the chassis (see Figure 4-2) or on the srb-20210g graphical representation itself (see Figure 3-7), the following menu is displayed: Figure 3-9. srb-20210g Chassis Right-Click Menu The following details the features under the right-click menu. Reset slb Reset the 10GbE Gateway Reset All Disable All IB Ports Enable All IB Ports Enable/Disable Info Led Blink Zoom Out Allows you to reset the selected srb-20210g. Note that this srb-20210g is considered as a line board in the Switch and listed as such. In this particular example, it is Line Board Number 2, as shown on top of the srb-20210g. Choosing reset 10 GbE Gateway (gray area) allows you to reset the unit. Allows you to reset all Boards. Disables all 22 InfiniBand ports. You can only disable all IB ports if at least one IB port is enabled. Otherwise, this option will be grayed out on the menu. Enables all 22 InfiniBand ports. You can only enable all IB ports if at least one IB port is disabled. Otherwise, this option will be grayed out on the menu. Allows you to enable to disable the blinking of the unit s Info LED indicator. From the srb-20210g graphical representation only Returns to the rear view of the switch in which the srb-20210g is installed, as shown in Figure 4-2. Simply clicking the blue area of the srb-20210g graphical achieves the same result. 31

32 Version 2.0 Mellanox Device Manager (VDM) Right-Clicking InfiniBand Ports Figure srb-20210g InfiniBand Port Right-Click Menu Port Counters Enable Port or Disable Port InfiniBand Ports only - Right-clicking a port on an srb-20210g allows you to select the port counters feature providing port counters information and a graph of the port counters (see Section 10.2 for more information). Allows you to enable or disable an InfiniBand port. When the port is enabled, you can only see the disable option. When the port is disabled, you can only see the enable option Right-Clicking 10 GbE Ports (Gray Area) The two 10 GbE Ports are confined into a gray area onto the srb-20210g board. Right-clicking this area allows you to choose from the following options: Figure srb-20210g 10 GbE Port Right-Click Menu Remote Management Settings Remote Configuration Link Aggregation Configuration Cluster Configuration Policy Table Port Configuration About The Remote Management Settings window enables you to set the SNMP Manager path in the SNMP tab, and to provide general details for the Remote Management Setting via the General tab. (See Section 6.2 for further information). Allows you to access the Remote Configuration window in which you can set the FTP/SCP parameters. This procedure is identical to the one performed on the Switch (refer to Section 6.3 for further information). Opens the Link Aggregation Configuration window which enables you to set LAG modes (FULL 1-2 or no LAG); refer to Section 6.4 for further information). Opens the Cluster Configuration window (refer to Section 6.6 for further information). Opens the Policy Table dialog box. Opens the Port Configuration dialog box. Opens the Version Information dialog box (refer to Section 4.7 for further information). 32

33 10 GbE Gateway (srb-20210g Module) User Manual Version Right-Clicking the Trap Table Right clicking the Trap Table allows you to: View an event Acknowledge an event Acknowledge all events. More details are provided in Section Left-Click Option Click the srb-20210g at the rear switch Click a port on the srb-20210g at the rear switch Click the srb-20210g graphical representation From the rear of the switch only Clicking the blue area of the srb-20210g at the rear switch opens a window showing a larger view of the srb-20210g graphical representation and related detailed information, as shown in Figure 3-7. From the rear of the switch only Clicking a port on the srb-20210g at the rear switch shows the Information Area for the specific port (right panel of the main window), as shown in Section 4.3. From the srb-20210g graphical representation only Clicking the blue area of the srb-20210g graphical returns to the rear view of the switch in which the srb-20210g is installed, as shown in Figure Color-Coding Table The GUI Color coding is defined according to traps and events severity, as shown in the following table. The user can change the severity of these alarms (via the Switch Fabric Manager Policy table). Table 3-4. Event Severity and Color-Coding Table GUI Color-Coding Event Severity Description Examples Red Critical Critical means that the system or a system component fails to operate. Orange Minor Minor reflects a problem in the fabric but does not prevent its operation. Yellow Warning Warning/minor reflects a problem but does not prevent its operation. A warning is asserted when an event is exceeding a pre-defined threshold. Green Normal Information/Notification provided to the user of normal operating state or a normal system event. Temperature control fault on board number X Failed to set LID on PORT PortRcvSwitchRelayEr rors counter threshold was exceeded Entering SM Standby State. 33

34 Version 2.0 Mellanox Device Manager (VDM) It is done in switch VFM, when configuring the policy table via both the GUI and the CLI. Note that these settings are done via the switch only. The user may define events severity as well as filter the events via the GUI and CLI. The filtering policy determines for each event whether it is logged and whether a trap is generated. 3.6 Trap Table The Trap Table is displayed at the bottom of the main window. It provides a wide range of information on the events and alarms in the Switch system. Figure Trap Table (Log) For each event/alarm, the Trap Table displays the following information: Table 3-5. Trap Log Information Event Information Severity Date/Time Source Category Event ID Description Description Displays the severity of the event/alarm. Displays the date and time of the event/alarm occurrence. Displays the GUID and the module name of the source of the event/alarm. Displays the source of the event. Possible values are: SA for events that occurred in the Subnet Administration SM for events that occurred in the Subnet Manager Displays a system generated ID number that identifies the event. Displays a short description of the event/alarm Resizing the Trap Table The Trap Log can be resized as follows: position the cursor on the top border line of the Trap Log. The cursor changes into a double-headed arrow with two lines in between; drag the cursor up or down to increase or decrease the size of the Trap Log area Rearranging Columns You can rearrange columns positions within the Trap Log table, by dragging and dropping a selected column in the new desired position as well as resizing the column width Viewing and Hiding Events You can hide the Events Log from view by selecting View/View Events from the main menu. 34

35 10 GbE Gateway (srb-20210g Module) User Manual Version Acknowledging Events Clearing/acknowledging an event means that you have taken the steps to correct the problem and now wish to remove the event from the Trap Table. Acknowledging an event is essentially clearing it. The state of the node is refreshed accordingly. The user can also clear all events and alarms by using the Acknowledge All feature. To acknowledge an event: On the Trap Table, right-click a specific alarm and choose Acknowledge Event. To acknowledge all events: On the Trap Table, right-click a specific alarm and choose Acknowledge All Displaying Additional Data for an Event Events are displayed in the Trap Table at the bottom of the Fabric Manager window. To view more details pertaining to a specific event: Step 1 In the Trap Table, right-click the specific event for which you would like to get information. A menu is displayed, as shown below. Figure Events Log Right-Click Menu Step 2 From the right-click menu, select View Event (or double-click the event). The Event Data window is displayed. Figure Events Data Window 35

36 Version 2.0 Mellanox Device Manager (VDM) This window displays further data on the event, for both sides of the link. The Port Info pane (left-hand) lists data for the local port with which the event is associated; the Peer info pane (right-hand) lists data for the remote port of the link or node with which the event is associated. The table below details the fields of the Events Data window. Table 3-6. Events Data Fields Parameter Node GUID Platform Path Logical Port Port GUID IB Port LID Event Details Description Global unique identification number (GUID) of the device. Switch platform on which the port is installed. Module/board type on which the port is installed. Additional information, if applicable, regarding the location of the port. Global unique identification number (GUID) of the port. Displays the InfiniBand port number. Displays the port LID. Provides detailed information on the specific event. 36

37 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Chapter 4. Device Information This chapter describes the various srb-20210g information views provided by both the GUI and the CLI. 4.1 Getting Started with the srb-20210g Device Manager The srb-20210g is installed into the Switch. It is accessed and managed through the Grid Director Management GUI and CLI connections. This means that you must first get into the switch GUI and CLI and only then can you access the srb-20210g Device. This section provides instructions on how to access the srb-20210g and describes the Gateway s basic functions and operations Device Manager. All the features are available via both the Graphics User Interface and the CLI. To access a larger view of the Mellanox srb-20210g: Step 1 In the VDM Main Window, click Rear. The rear of the Switch is displayed, showing the srb-20210g installed in the chassis, as shown in Figure 3-5. Step 2 From the Switch Device navigation tree (left panel), under slb/srbds, click the desired srb-20210g. Alternatively, you can click the desired srb-20210g directly at the rear of the Switch graphical representation. Figure 4-1. Click the srb-20210g The srb-20210g Device Manager window opens, showing a larger view of the srb-20210g graphical representation as well as detailed port information, as shown in Figure

38 Version 2.0 Device Information Figure 4-2. srb-20210g Device Manager - 10 GbE Line Information Window NOTES: The status bar at the top of the window shows that this module is an srb-20210g line (listed under slb/srbd) installed in slot #3 of an ISR 2004 chassis. The columns can be sorted by clicking the column headings. The following parameters are displayed: Table 4-1. srb-20210g Device Information Serial Number Hardware Version Serial number of the module. The hardware version of the module. Table 4-2 describes the parameters displayed for each 10 GbE port. Table GbE Port Information Parameters Port Description Port 10 GbE port number (1 2) Type Status Speed Displays the port type: Optic, Copper or IB port. Shows whether the port is up (active) or down. Shows the link speed (10 Gbps for the srb-20210g) 38

39 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Port MTU Trunk Group VLAN List Admin State Description Shows the current Maximum Transmission Unit (MTU) settings of each port (1500 or 9198) Shows the aggregated ports in full (P1-2) or no Lag (separate) mode (P1), (P2). The VLAN tag assigned to the interface. Displays the Admin State Mode: Enabled/Disabled. The Admin state means that the port is down as configured by the administrator (user). Table 4-3 describes the parameters displayed for each InfiniBand port. Table 4-3. InfiniBand Port Information Parameters Parameter Description Port InfiniBand port number (1 22) State LID Width Speed MTU VLs Indicates the physical state of the port. Up Link is active Down Link is down or disconnected. Port Local ID Displays the width (data) of the link. Displays the supported link speed of the port (10Gbps for the 10 GbE port). Shows the current Maximum Transmission Unit (MTU) settings of each port (1500 or 2048) Displays the Virtual Lane (VL) range supported by the port. Current support is for VLs 0 to Device Information via the CLI To view the port status, run a port status show: From the srb-20210g CLI (config-port)# run a port ethernet show command, as shown below. 10G_1(config-port)# port ethernet show port type status mtu speed(gbps) enable/disable copper up enable 2 copper down enable This screen shows: 10 GbE port 1 is enabled, copper type, and is up (active). 39

40 Version 2.0 Device Information 10 GbE port 2 is enabled, copper type, and is down. To change the port status (enable/disable): 10G(config-port)# port ethernet set [1..2] [disable,enable] 10G(config-port)# port ethernet set 1 disable. 4.3 srb-20210g Port Information To view the srb-20210g Port Information Area: Step 1 Step 2 In the VDM Main Window, click Rear. The rear of the Switch is displayed, showing the srb-20210g installed in the chassis, as shown in Figure 3-5. Click a port on the srb-20210g. The port Information area (right panel) shows the specific port information. Clicking an InfiniBand port displays the InfiniBand port information. If you click the 10 GbE port, the right panel displays the 10 GbE port information. Figure 4-3. Port Information Area (Right Panel) The following table details the fields in the Port Information area. Table 4-4. Device Information Port Port Type Port Status Description Displays the port type: Optic, Copper, or IB port. Shows whether the port is Up (active) or Down. 40

41 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Port Port Speed Port MTU Trunk Group VLAN List Port Admin State Description Shows the link speed (Gbps). InfiniBand Port speed: 2.5 Gbps (SDR), 5 Gbps (DDR) 10 GbE port speed: 10Gbps Shows the present Maximum Transmission Unit (MTU) settings of each port (1500 or 9198). Shows the aggregated ports in full (P1-2) or no (separate) Lag mode (P1), (P2). The VLAN tag assigned to the interface. Displays the Admin State Mode: Enabled/Disabled. The Admin state means that the port is down as configured by the administrator (user). In the example above, the status bar of the Information Area shows the information of the 10 GbE gateway s port Viewing the System Time and Date (CLI only) To view srb-20210g system time, run the following command from the srb-20210g CLI admin or privileged mode: 10G> clock show Thu Mar 19 03:19:58 UTC 2009 To view srb-20210g system memory and CPU utilization, run the following command: 10G> Total memory : kbytes. Free memory : kbytes. Memory utilization : 30.42%. CPU idle : 97%. The following table details the System Utilization parameters: Table 4-5. System Utilization Parameters Parameter Total memory Free memory Description Total amount of memory (in kbytes). Total amount of memory available (in kbytes). Memory utilization Amount of memory used (in %). CPU idle Amount of CPU available (in %). 41

42 Version 2.0 Device Information 4.5 Viewing srb-20210g Temperature (Switch CLI only) To view srb-20210g temperature information: You can only view the srb-20210g temperature from the switch privileged mode by running a rear show command, as shown in the example below. ISR # rear show srb-20210g in slot #3: Temperature: 45C; HW Version: 004; Serial Number: CN slb-2024 in slot #4: Temperature: 42C; HW Version: A02; Serial Number: CN This example shows that the temperature of the srb-20210g is at 45C. The system polls the temperature sensors and notifies if threshold exceeded. srg-20210g temperature thresholds are as follows: Degrees Celsius Fahrenheit Normal Warning Alarm Setting the Uptime The Uptime is the date at which the node was activated or deactivated (up or down). Using the Uptime feature, you can view how long how long the system has been running, how many users are currently logged on, and the system load average. In addition, you can see the current time. To view the system uptime, run the following command from the srb-20210g admin or privileged mode: 10G> uptime 02:32:56 up 1 day, 3:27, load average: 0.37, 0.39, Checking the Device Software Version Device Version via the GUI To view the version information via the GUI: On the srb-20210g (zoom view) graphical representation, right click on the 10G gray area and select About, as shown in the figure below. 42

43 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Figure 4-4. Obtaining Version Information The following window is displayed Figure 4-5. Software Version Information This window showing the current version information: The Graphic Device Manager version of the 10 GbE Gateway The Graphic Device Manager version of the Switch The software version of the 10 GbE Gateway The software version of the switch. You can also view the srb-20210g hardware version information in the field under the srb-20210g graphical representation (Zoom view). Figure 4-6. Hardware Version Device Version via the CLI To view version information, run the following command from the srb-20210g admin or privileged mode: version show IPR software version: 1_0_0-47, Apr :53:51 IPR Board Version: 0x1. IPR Firmware Version: 0xE. IPR CPLD Version: 0x5. 43

44 Version 2.0 Device Configuration Chapter 5. Device Configuration This chapter describes the device configuration that can be performed using the Device Manager, which includes IP Settings, Remote Management Settings, and User Settings. Refer to Chapter 2 for information on Initial System Configuration for Management and detailed instructions how to set up your system for Management. 5.1 IP Settings Overview The GridVision Device Manager (VDM) allows you to configure the srb-20210g so that it can be managed by the switch, either through the Ethernet or through InfiniBand. This includes IP settings, remote management settings, user settings, setting remote configuration (SNMP) Setting the I/O IP via the GUI The I/O Panel window is used to configure the I/O IP on the switch side. It shows the type of I/O installed in the switch drawer slots. The slot number is the number on the Grid Director rear panel in which the srb-20210g is installed within the switch. It also shows the srb-20210g IP and mask. To change/set the IP of the srb-20210g: Step 1 Under the Main Menu, select Device I/O Panel. The I/O Panel dialog box is displayed. Figure 5-1. I/O Panel Dialog Box Step 2 In the I/O Panel table, right-click the desired gateway (row) and select Change IP Address. The following dialog box is displayed. 44

45 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Figure 5-2. Update I/O IP Dialog Box Step 3 Update the IP address and the Subnet Mask as required by editing the fields and click OK. The I/O Panel window provides the same functionality as the CLI Switch>config-chassis menu. You can view and set I/O via the config-chassis menu as detailed in the following sections Configuring the srb-20210g IP via the Switch CLI The srb-20210g IP address is configured via the CLI of the Switch by using the following command: ISRxxxx(config-chassis)#io ip-address set [slot-number] [position-number] [ip-address] [netmask]. To configure the I/O via the Switch CLI, perform the following steps: Step 1 Run a ISRxxxx(config-chassis)#io show command to see if the address is configured. This shows the Gateways installed inside the switch. This command is available from both the Admin menu as well as the config-chassis menu. Example 1: ISR fa(config-chassis)# io show slot #2, position #1, I/O Type 10 GbE Gateway, IP: Mask: GetCommunity:public SetCommunity:private In this example, a 10 GbE Gateway is installed in slot #2 the chassis rear panel. (Position 1 is the unique/default position) Step 2 Run an io ip-address set [slot-number] [position-number] [ip-address] [netmask] to set the io address and position in the chassis. Switch(config-chassis)# io ip-address set This command sets the IP address of the srb-20210g module. Step 3 Run io connect {slot-number position-number} to connect to the I/O This establishes a CLI session with an srb-20210g module installed in a switch. Switch> io connect 10 1 srb-20210g> 45

46 Version 2.0 Device Configuration NOTE: The IP address of the srb-20210g must be on the same subnet as the smb card with which the CLI session is established. 5.2 Resetting the srb-20120g Resetting the I/O via the GUI To reset the I/O: Step 1 Step 2 Right-click the srb-20210g graphical representation either from the switch or zoom view. From the menu, select Reset 10 GbE Gateway, the following window is displayed. Figure 5-3. Resetting the 10 GbE Gateway Step 3 Click Yes to reset the I/O. The I/O is reset Resetting the I/O via the CLI To reset the srb-20210g from the CLI, run the following command: srb-20210g-70b0# reload Reboot the system?[yes] The srb-20120g reboots. 10 GbE initiation after reboot: R:ISR (config-chassis)# io show slot #3, position #1, Type 10 GbE Gateway [Initiation mode] GetCommublic SetCommunity:private To set the srb-20210g unit to factory defaults: In privileged mode, run the following command: 10G-Card#( config-)#factory-default Factory default reboots the system. Choose Yes/No to perform factory default? [yes] After pressing Y, the unit is set to its factory defaults. 46

47 10 GbE Gateway (srb-20210g Module) User Manual Version User Settings (Setting Passwords) You can change your user password periodically or as necessary User Settings via the GUI The GridVision Device Manager allows you to configure three types of user authorization: The Admin level enables you to change user password configuration, to export or import configurations and more. The default password is The Guest level enables you to view configurations only. The default password is voltaire. To Configure Authorization Passwords: Step 1 From the Main Menu, select Device Users. The Users window opens, as displayed in Figure 5-4: Figure 5-4. Configuring User Authorization Step 2 Step 3 Step 4 Step 5 Select the user authorization to assign to the user (Admin, Guest) from the Authorization field. Type in old user password. Type in the new password and confirm it by typing it again in the New Password (retype) field. Click Apply and OK to save the new entry. 47

48 Version 2.0 Device Configuration User Settings via the CLI To configure passwords through the CLI, run the following commands: Command Description Step 1 guest a. Enter the guest mode. The default password for the guest user is voltaire. b. Enter the Privileged mode as follows: Type enable and enter the admin password (The default password for the admin user is ) OR Step 2 admin password update [admin,enable,guest,root] Enter directly the privileged mode. The default password for the admin user is Note: When connecting to the srb-20210g via the switch using the io connect command, you are already in admin mode and do not need a password. Set the new passwords. Step 3 exit Exit Privileged mode 10G# password update admin Insert new (up to 8 characters) Enter password :******* Please retype new Enter password :******* OK 10G# NOTE: In order to change the privileged mode (enable) password, you must change the password of the admin user. 48

49 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Chapter 6. Advanced Configuration This chapter provides instructions on how to perform advanced configuration settings for the srb-20210g using the Mellanox Command Line Interface (CLI). Refer to Chapter 14 of this manual for a comprehensive CLI commands reference. 6.1 Updating the Software The srb-20210g is designed to run on a continual basis without significant maintenance. However, from time to time, you may need to install updated software or modify device configuration. The srb-20210g stores a software image (along with configuration files, log files, and other information) on a local file system. This file system is stored on an internal flash memory. If you plan to use the CLI update software command to update the srb-20210g software from a remote FTP, be sure to set the FTP server name and IP address through the FTP server command and be sure the necessary file is located on the FTP server. To update srb-20210g software, perform the following steps: Step 1 Step 2 Make sure that the FTP server is active, and located in the same subnet as the srb-20210g management port subnet. Copy the new version files from the installation CD to the FTP root directory Identify the File Location for Software Update You must specify the location from which to retrieve the updated software. If the default download location is not appropriate, you can reset it. Use the steps in the following table to identify the default download location. Table 6-1. Identify Location Command Description Step 1 guest a. Enter the guest mode. The default password for the guest user is voltaire. b. Enter the Privileged mode as follows: Type enable and enter the admin password (The default password for the admin user is ) OR admin Enter directly the privileged mode. The default password for the admin user is Note: When connecting to the srb-20210g via the switch using the io connect command, you are already in admin mode and do not need a password. Step 2 version show List the software versions currently available for booting. Verify that the version of software required is not already available. 49

50 Version 2.0 Advanced Configuration If this location is not the one from which you would normally retrieve updated software, use the steps in the following table to set the appropriate remote location. Table 6-2. Setting Remote Location Command Description Step 1 guest a. Enter the guest mode. The default password for the guest user is voltaire. admin b. Enter the Privileged mode as follows: Type enable and enter the admin password (The default password for the admin user is ) OR Enter directly the privileged mode. The default password for the admin user is Note: When connecting to the srb-20210g via the switch using the io connect command, you are already in admin mode and do not need a password. Step 2 config Enter the config mode (from privileged mode). Step 3 remote Enter the remote mode (from config mode). Step 4 remote show In the remote mode, run a remote show command to check the Remote configuration. If the configuration is satisfactory, skip to step 6. Step 5 (config-remote)#ser ver <ip-address> Power up the srb-20210g (if needed) and configure the Remote server name and IP address. Step 6 Username <username> Configure the username for Remote server access. Step 7 Password <password> Set the user password to access the Remote server. Step 8 remote show After the update is complete, execute this command to verify the new settings. Step 9 exit Exit the remote menu back to the config mode. Step 10 exit Exit the config menu back to privileged mode. Step 11 update software [update-file-dir] In the privileged mode, run an update software command by specifying the directory location path [update-file-dir]. This starts the update software process. This can take several minutes. Note: if you do not specify the location, it is will updated at the default location. 10G# update software ***************************************************************** ********** Scp server ip : Scp download dir : Package file : srb-20210g-install.pkg Username : swget Interactive install : n 50

51 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Force burn : 0 ***************************************************************** ********** Do you want to continue the install procedure? [y/n] - Default value y y Getting Software image from remote server Updating the VDM (GUI) - Gateway Version Automatic Upgrade The Mellanox Graphic Device Manager (VDM) immediately recognizes a gateway with a higher version (for example, when installing a new gateway in the switch, or when performing a gateway upgrade). In such case, the VDM is automatically upgraded to the higher version. When the GUI is closed during the upgrade, the VDM is automatically updated and no action is required. When the GUI is open during the upgrade, the following warning message is displayed: Figure 6-1. Warning Message The user must then exit the GUI and reenter it in order for the change to take effect. 6.2 Remote Management Settings (SNMP) Setting Remote Management via the GUI The Remote Management Settings dialog box enables you to set SNMP parameters to interface with remote management systems, and to provide general details of the remote management system. To configure remote management settings: Step 1 On the srb-20210g zoom view, right-click the 10G gray area and select Remote Management Settings from the menu. The Remote Management Settings dialog box opens, as displayed in Figure 6-2: 51

52 Version 2.0 Advanced Configuration Figure 6-2. Remote Manager Settings - SNMP Tab Step 2 Step 3 Step 4 Click the SNMP tab. Enter the IP address. The relevant check box will be marked automatically. Change or reactivate the existing IP address: Click the relevant SNMP Manager check box and then type in the relevant IP address. Click OK. Click the General tab. Figure 6-3. Remote Management Settings - General Step 5 View (or edit) the information in the following fields: Device Name: SNMP Version: Name of the device. Note that you can also edit this field. Software version of the SNMP. Note: this field is read-only. 52

53 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Contact: Location: Name of contact person Physical location of the device (where it is placed) Step 6 Click Apply to save updates to this tab. Click OK to save updates to this tab and exit the dialog box Setting Remote Management via the CLI To define SNMP Communities, enter the following commands: Step Optional From the srb-20210g CLI (config-snmp) menu, run the following commands as required. Set the SNMP community get string using the following command: snmp community get [string] The default get community string is public default set community string is private. Updating the Switch community strings will update all the community strings in the device. This action requires a good familiarity with the SNMP community procedures. Make sure you update the GUI remote connections with these new changes. 10G(config-snmp)# snmp community get public Expected Result: 10G(config-snmp)# snmp show snmp functionality enable snmp get enable snmp set enable get community public set community private trap community trap location description 53

54 Version 2.0 Advanced Configuration Optional Set the SNMP community string using the following command: snmp community [string] The default set community string is private. Updating the Switch community strings will update all the community strings in the device. This action requires a good familiarity with the SNMP community procedures. Make sure you update the GUI remote connections with these new changes. 10G(config-snmp)# snmp community set private Expected Result: 10G(config-snmp)# snmp show snmp functionality enable snmp get enable snmp set enable get community public set community private trap community trap location description Optional Set the SNMP community trap string using the following command: snmp community trap [string] Updating the Switch community strings will update all the community strings in the device. 10G(config-snmp)# snmp community trap trap Expected Result: 10G(config-snmp)# snmp show snmp functionality enable snmp get enable snmp set enable get community public set community private trap community trap location description 54

55 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Optional Define the SNMP contacts string using the following command: snmp contacts set [contacts] Note that the contact is a string set by the user. 10G(config-snmp)# snmp contacts set Voltaire Expected results: 10G(config-snmp)# snmp contacts show Contact(s) is Voltaire Optional Set the SNMP description using the following command: snmp description set [description] Note that the description is a string set by the user. 10G(config-snmp)# snmp description set SNMP1 Expected results: 10G(config-snmp)# snmp description show description is SNMP1 Optional Set the SNMP location using the following command: snmp location set [location] Note that the location is a string set by the user. 10G(config-snmp)# snmp location set Lab1 Expected results: 10G(config-snmp)# snmp location show Location is Lab1 55

56 Version 2.0 Advanced Configuration Optional Disable the SNMP get functionality using the following command: snmp get [enable, disable] This disables the SNMP get commands and prevents connection to the GridVision GUI. The SNMP get commands are enabled by default. 10G(config-snmp)# snmp get disable Expected results: 10G(config-snmp)# snmp show snmp functionality enable snmp get enable snmp set enable get community public set community private trap community trap location description Optional Disable the SNMP set commands using the following command: SNMP set commands [enable, disable] This disables the SNMP set commands and prevents connection to the GridVision GUI. The SNMP set commands are enabled by default. 10G(config-snmp)# snmp set disable Expected results: 10G(config-snmp)# snmp show snmp functionality enable snmp get enable snmp set enable get community public set community private trap community trap location description 6.3 Remote Configuration Settings (FTP/SCP) The FTP configured for use with the srb-20210g is used for the following: Upgrading software versions Uploading log files to the remote server Uploading downloading the repository file. 56

57 10 GbE Gateway (srb-20210g Module) User Manual Version Setting Remote Configuration via the GUI To Configure Remote Configuration via the GUI: On the srb-20210g zoom view, right-click the 10G gray area and select Remote Configuration from the menu. The Remote Configuration dialog box opens, as displayed in Figure 6-4. Figure 6-4. Remote Configuration Dialog Box Setting Remote Configuration via the CLI To set Remote configuration parameters: Table 6-3. Setting Remote Server Parameters Steps GUI: Remote Configuration Window feature CLI Command (Privileged Mode) Description Setting the Remote Configuration parameters Step 1 Click the radio button to set the desired Protocol (FTP/SCP) 10G(config-remote )# protocol set [FTP SCP] Sets the protocol to access the remote server. Step 2 Set the Remote Server IP address or view the currently configured address. 10G(config-remote )# remote show Shows the remote client configuration. Step 3 In the User Name field, set (edit) the user name or view the existing user name. 10G(config-remote )# username [username] Sets the user name to access the remote server. 57

58 Version 2.0 Advanced Configuration Steps GUI: Remote Configuration Window feature CLI Command (Privileged Mode) Description Step 4 In the Remote Path field, enter the server remote path. Indicates the remote path in which the transferred file will be created. Step 5 In the Password field, set (edit) the password or use the existing password (hidden). 10G(config-remote )# password [password] Edit the password. Optional Step 6 Click the Refresh button to refresh the screen. It means that it reloads all of the screen s information from the server again. In addition, the filename is regenerated by the server, so a new filename might be displayed. Click Apply or OK to save the configuration. The Remote settings are now part of the system Backing up the Configuration via the GUI and the CLI Backing up the configuration Step 1 In the File Name field, set file repository file name or view the existing name. This field defines the file name in the repository to which the configuration will be copied. The file list under the Name header (not editable) shows the file names in the flash. (up to two names) 10G(config-remote) # flash show Shows the configuration files in flash. Optional Click copy to remote to upload the configuration to the remote server. 10G(config-remote )#copy running-config: remote: copy running-config remote: [remote-path] Uploads the configuration to the remote server using previously defined server name, user name, and password. The remote path can contain the imported file name. If the file name is not indicated, it is created using a default name. Any characteristic after the last slash in the path will be used as a user-defined file name. Optional Click Load from 10G(config-remote Downloads the configuration from the 58

59 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Backing up the configuration Remote to download the backup configuration from the remote server. )#copy remote: running-config copy remote: [remote-path] running-config remote server using previously defined server name, user name, and password. The remote path must contain the exported file name. The new values will be applied on reboot. Optional Click Copy to Flash to save a backup of the configuration in the local repository. 10G(config-remote )#copy running-config flash: copy running-config flash: [file-name] Backs up the configuration into file in the local repository. In case of empty name the file name is set by default (example of default format: repository_srb-20210g-38fa_6_47_ _IP_ new). You can back up to two files. Backing up additional files will prompt a message that you are over the limit. Optional Click Load from Flash to restore the backup configuration. 10G(config-remote )#copy flash: running-config copy flash: [file-name] running-config This restores the backup configuration. This command retrieves the repository configuration from a file stored in the Flash, restores it and saves it in the switch repository (database). The new values will be applied on reboot. Optional Click Delete from Flash to delete the specific repository file from flash. 10G(config-remote) #flash delete Deletes the specific repository file from flash. Step 2 Click the Refresh button ( ) to update this dialog box. This enables you to refresh the data and view the updated information without closing the window. Step 3 Click Apply and OK. 6.4 Setting Link Aggregation (LAG) This tool enables load balancing by logically aggregating ports as trunks to optimize network traffic performance. It also provides automatic redundancy. Should one of the multiple ports used in a link fail, network traffic is dynamically redirected to flow across the remaining good ports in the trunk. By default, the gateway is set to Full LAG Mode, meaning that two are aggregated into a single trunk. In the code, full link aggregation appears as P1-2. Separate LAG means no link aggregation. In the code, Separate LAG appears as P1, P2. The example in Figure 6-5 shows a 10 GbE Ports in full LAG mode (P1-2). 59

60 Version 2.0 Advanced Configuration Figure GbE Ports in Full LAG Full LAG is represented by the green line that connects the two ports. Note: the terms trunk and LAG are used interchangeably Setting Link Aggregation via the GUI To set Link Aggregation via the GUI: Step 1 On the srb-20210g zoom view, right-click the 10G gray area and select Link Aggregation Configuration from the menu. The Link Aggregation Configuration dialog box is displayed, as shown below. Figure 6-6. Link Aggregation Configuration Lag Mode Full Aggregation No Aggregation Description Both ports are aggregated into a single trunk. Separate LAG - No link aggregation. Step 2 Select the preferred LAG mode and click Apply and OK to save your settings. The Confirm LAG Change dialog box appears, with the following options: Figure 6-7. Confirm LAG Change 60

61 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Option Reboot Now Reboot Later Cancel Description Reboots the device immediately. The changes take effect once the reboot is completed. Allows you to reboot the device later manually. The changes will take effect only after you perform manual reboot. Cancels the Lag Mode change and returns to the existing LAG state Setting Link Aggregation via the CLI To set Link Aggregation via the CLI: From the 10G(config-if)# prompt, run the following command: lag-mode set [full,separate] 10G(config-if)# lag-mode set full NOTE: When setting to link aggregation, make sure that srb-20210g ports on the Ethernet Switch are aggregated as well. To view 10 GbE Link Aggregation Mode settings via the CLI: lag-mode show 10G(config-if)# lag-mode show Lag mode is: full lag To choose the Link Aggregation when setting up an interface: At the 10G(config-if)# prompt, type the following command, enter all the parameters and enter the desired LAG [P1,P2,P1-2]. interface add [alias] [no -[tag]] [P1,P2,P1-2][pKey 0x0001-0x7fff] [Ip] [Mask] 10G(config-if)# interface add V5 108 P1-2 0x

62 Version 2.0 Advanced Configuration 6.5 Configuring I/O Interfaces NOTE: smb-mng is the management interface via the switch. This management interface is always there except when you work in inband management. Inband is when the device is managed from the InfiniBand network. Out-of-band is when the device is managed from the switch Fast Ethernet network. if0 is the default interface, which cannot be deleted. To configure I/O interfaces, run the following commands from the 10G> config interface menu: Step 1 Run an interface show command to view interfaces and trunks (LAGs): 10G(config-if)# interface show entry alias ip mask broadcast vlan pkey(hex) trunk smb-mng no 0x7fff No port 2 if no 0x7fff P1-2 3 CP x112 P1-2 4 net x75 P1-2 Note that the Trunk (LAG mode) is set to P1-2. (This command is available from both, the 10G admin or config menus). Step 2 View the LAG mode status by running a lag-mode show command, as follows: 10G(config-if)# lag-mode show LAG mode is: full LAG. Step 3 From the 10G> config interface menu, set the LAG mode by running a lag-mode set command, as follows: 10G(config-if)# lag-mode set [full, separate] 10G(config-if)# lag-mode set separate Changing LAG mode will remove its interfaces, VLAN tags and associated TFM rules. It is necessary to reboot the system after this operation. Are you sure you want to continue [yes/no] - Default value yes Reboot is required after changing LAG mode. Expected Result: 10G(config-if)# interface show all entry alias ip mask broadcast vlan pkey(hex) trunk smb-mng no 0x7fff No port 2 if no 0x7fff P1 Note that the Trunk (LAG mode) was changed to P1, P2. (This command is available from both, the 10G admin or config menus). 62

63 10 GbE Gateway (srb-20210g Module) User Manual Version Cluster Configuration In a cluster where multiple 10 GbE Gateway ports are connected in parallel to each other (both on the Ethernet side and the InfiniBand side), these modules behave in High Availability and Load Balancing mode as described below. They automatically detect each other and are displayed in the Cluster Configuration window Active Node data according to priority. Note that if two 10 GbE Gateway ports have the same priority, they are sorted according to the master bit and then by the serial number. The 10 GbE gateway that has the highest priority becomes Master when there is no other 10 GbE gateway acting as master. If there is already a master in the system, it will stay the master even though its priority is lower. Multiple algorithms are available for distributing traffic between multiple srb-20210g Gateways in Cluster Mode. The traffic can be balanced between the srb-20210g Gateways according to the load balancing algorithm (options: active-active, master-standby). In active-active mode, the load balancing is performed according to the Hash Algorithm (options: XOR IB Eth, IB, Eth) Configuring a Cluster via the GUI To set Cluster Configuration via the GUI: Step 1 On the srb-20210g zoom view ( ), right-click the 10G gray area and select Cluster Configuration. The Cluster Configuration dialog box is displayed, as shown below. Figure 6-8. GUI (VDM) Cluster Configuration Dialog Box Step 2 Set the Cluster Configuration according to the Cluster Configuration Fields Table 7-2 below. Refresh Step 3 Step 4 Click Update next to the specific field to save the change performed for this field. Click the Refresh button to update this window. This updates all the data from the server. This is useful when there is a communication problem. 63

64 Version 2.0 Advanced Configuration The following table details the fields in the Cluster Configuration window. Table 6-4. Cluster Configuration Fields Parameter Cluster Mode Cluster ID Cluster Priority Hashtable Algorithm Loopback prevention Box ID (0x) Auto join cluster Load Balance Algorithm Description Disable Enable Identification number of this cluster Priority of this cluster The load balancing is performed according to the hash algorithm (XOR IB eth, IB, ETH): XOR-InfiniBand-Ethernet (Default) mathematical operation performed between IB and Eth IP addresses. InfiniBand IP address of the InfiniBand Host for any Ethernet address. Ethernet IP address of the Ethernet Host for any InfiniBand address. Prevents loops between gateways. The following prevention modes are available: None Should be used only on a standalone 10 GbE gateway. Full Blocks all traffic between 10 GbE gateways in the fabric. Automatic Blocks all 10 GbE gateways that are members of the same cluster. Useful when connecting two separate Ethernet subnets to one IB network or vice-versa. Manual Blocks all traffic between 10 GbE gateways that have the same cluster ID. Used for back-to-back configurations, where 10 GbE gateways are connected directly to each other over Ethernet links. Requires the configuration of a cluster ID on each gateway. A hexadecimal number that uniquely identifies a single 10GbE gateway (derives from its hardware address). It is used by one of the optional loop prevention methods. After being automatically detected a srb-20210g can be set to automatically join the cluster. This means that it is now able to transfer packets. Options: Yes/No Default: Yes The traffic can be balanced between the srb-20210g Gateways according to the load balancing algorithm. Options: active-active, master-standby Default: active-active active-active all srb-20210g Gateways in the cluster transfer packets. master-standby only one srb-20210g in the cluster transfers packets, the others are in Standby Mode. The srb-20210g that will transfer the traffic is the one with the highest priority. 64

65 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Parameter Active Nodes Table Description Shows the Active srb-20210g Gateways that are interconnected and sorted according to priority (note that if two srb-20210g Gateways have the same priority, they are sorted according to the master bit and the serial number). The srb-20210g that has the highest priority becomes master Configuring a Cluster via the CLI The srb-20210g can be configured as part of a cluster (also referred to as multiple srb-20210g gateways mode). When using multiple gateways in Cluster Mode, each ARP packet request is transparently handled by a single gateway selected according to the internal hash function calculation. NOTE: All srb-20210g Gateways in a cluster are configured in auto mode (default). It is highly recommended not to change the auto mode nor to use the disable feature [cluster-mode set disable, auto] Cluster Mode (srb-20210g High Availability) The 10G Cluster Mode shows how many srb-20210g Gateways are in the cluster and provides information on their status as well as on high availability. For example, active-active status shows that one srb-20210g backs up the other one. To attach a srb-20210g port to the Cluster: Step 1 From the 10G#config cluster menu, run as follows: 10G# (config-cluster)# cluster join cluster join [yes,no] Step 2 Step 3 Then run a join-now command, as follows: 10G# (config-cluster)# cluster join-now Set the cluster priority: (config-cluster)# cluster priority set [priority [0-255]] The srb-20210g with the highest priority value will be set as master as long there is no other master in the fabric. Each one of the other srb-20210g Gateways is set with a descending priority value in the event of the master failing. 65

66 Version 2.0 Advanced Configuration Example The following is an example showing a cluster with three interfaces: r10g(config-cluster)# cluster show cluster configuration mode : enable loop prevention : full priority : 100 join cluster automatically : yes load balancing algorithm : active-active hash table algorithm : xor-infiniband-eth multicast ip address : cluster id : 1 active node table pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff 0 0x x if0 srb-20210g x x x CP1 srb-20210g x x x net75 srb-20210g Setting srb-20210g Algorithms To set the 10 GbE algorithms: Step 1 At the 10 GbE>config cluster prompt, enter the following commands: Command cluster hash-table-algorithm set [xor-infiniband-eth,infin iband,ethernet] Description Sets the cluster hash table algorithm. In active-active mode, the load balancing is performed according to the hash algorithm (xor IB eth, IB, Eth): xor IB eth (Default) mathematical operation performed between InfiniBand and Ethernet IP addresses IB IP address of the InfiniBand Host for any Ethernet address Ethernet. IP address of the Ethernet Host for any InfiniBand address 10G(config-cluster)# cluster hash-table-algorithm set infiniband This change will take effect after reboot. Expected results: 10G(config-cluster)# cluster show cluster configuration state : enable loop prevention : full priority :

67 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 join cluster automatically : yes load balancing algorithm : active-active hash table algorithm : infiniband multicast ip address : cluster id : 1 active node table pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff 0 0x x if0 srb-20210g x x x CP1 srb-20210g x x x net75 srb-20210g Command cluster load-balance-algorithm set [active-active, master-standby] Description Sets the load balancing algorithm [active-active, master-standby]. The traffic can be balanced between the 10 GbEs according to the load balancing algorithm. Options: active-active, master-standby Default: active-active active-active means that all srb-20210g Gateways in the cluster transfer packets master-standby means that only the Master srb-20210g in the Cluster transfers packets; the others are in Standby mode 10G(config-cluster)# cluster load-balance-algorithm set active-active Expected results: 10G(config-cluster)# cluster show cluster configuration state : enable loop prevention : full priority : 100 join cluster automatically : yes load balancing algorithm : active-active hash table algorithm : infiniband multicast ip address : cluster id : 1 active node table pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff 0 0x x if0 srb-20210g x x x CP1 srb-20210g x x x net75 srb-20210g

68 Version 2.0 Advanced Configuration Loop Prevention When enabling clusters, the parallel links between InfiniBand and Ethernet create loops, which degrade performance. This happens because these loops allow broadcast and multicast storms that congest the whole network and may eventually crash the entire system. So in order to this, a loop-prevention mechanism is implemented in the 10GbE gateway, restricting the traffic between gateways. The loop-prevention mechanism has four different operation modes, applicable to different configurations. The loop-prevention modes are: None see Section Full see Section Automatic see Section Manual see Section None Mode None means that loop prevention is disabled. This mode is not recommended and should never be used when 10 GbE gateways are installed in any kind of cluster. It is applicable only when a single 10 GbE gateway connects an InfiniBand network to Ethernet Full mode Full is the default mode and is the most restrictive: no traffic is allowed between 10 GbE gateways. Each gateway discards all received packets sent by another gateway in the fabric. This mode is applicable to configurations with one InfiniBand subnet and one Ethernet subnet. X Etherne InfiniBand Figure 6-9. Loop Prevention in Full Mode NOTE: If the configuration includes more subnets (two separate Ethernet networks connected to one InfiniBand network and vice-versa), the 10 GbE gateways blocks the traffic between the two remote networks. In such a case, Automatic mode can be used when each two 10GbE are connected to a different Ethernet network. 68

69 10 GbE Gateway (srb-20210g Module) User Manual Version Automatic Mode Automatic is the most permissive mode: traffic is blocked between all 10 GbE gateways that are members of the same cluster. Each gateway discards all received packets sent by another gateway that belongs to the same cluster, but passes-on all packets sent by other gateways. It requires enabled cluster mode. Automatic is applicable for configurations that consist of two separate Ethernet networks (or more; up to four if using one IB switch) connected to a single InfiniBand network, and vice-versa. The 10 GbE gateways pass the traffic between the two remote networks, and at the same time preventing loops within each cluster. X Ethernet X InfiniBand Ethernet Figure Loop Prevention in Automatic Mode This mode automatically reacts and adapts to network topology changes, i.e. when a network splits or when two networks merge, the loop prevention mechanism is automatically updated by adding any new 10 GbE gateway that joined the cluster to the gateway blocking list, and by removing any gateway that left the cluster. Since such changes take time, this mode is less secure than the Full mode, and temporary loops may form when networks merge or split Manual Mode Manual mode is less permissive than Automatic: all traffic between 10 GbE gateways having the same cluster ID is blocked. There is no need for the gateways to be in a cluster. However, clusters must be enabled. This mode is applicable only for configurations where two InfiniBand networks are connected over Ethernet direct links between the 10 GbE gateways (without any other switch or router between them). The cluster operates differently in this mode: it does not apply load balancing to the traffic received from the Ethernet side (assuming it was balanced by the gateways in the other side). In addition, because of the VAAR packets, the gateways at each side can join the cluster although they are not directly connected at the Ethernet side. 69

70 Version 2.0 Advanced Configuration Ethernet Ethernet Ethernet InfiniBand X InfiniBand X Figure Loop Prevention in Manual Mode Manual mode requires the configuration of the cluster ID of each gateway in the system. Each InfiniBand subnet is connected to the other InfiniBand subnet using a cluster of gateways (green-circled in the figure above). Each cluster must have a unique ID (all gateways within one cluster must have the same ID), and the gateways linked to external Ethernet networks (top right and top left) must have different cluster IDs (different than the IDs of the clusters). However, the top-right and top-left gateways should be configured to work in automatic mode. It is highly recommended to set priorities on each of the 10 GbE gateways operating in manual mode, so that: Each gateway in a cluster has a unique priority. Each two gateways connected over the Ethernet direct links have the same priority (in the figure above, the two upper gateways may have their priority set to 10, and the two lower gateways may have their priority set to 20). This configuration ensures that: Traffic between two end stations follows the same route in both directions On each cluster only one 10 GbE gateway joins each Multicast group. Failing to set the priorities properly may not affect traffic directly, but may degrade the total throughput of the system, especially when dealing with heavy multicast flows. 6.7 Setting the Event Policy Filters The Policy Table dialog box allows you to view and control the event filters. You can decide whether: A certain event should be recorded in the log A trap should be sent for a certain event No action should be taken The event s severity would be considered as Normal, Minor, Warning, or Critical. 70

71 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 To set the event policy filters: Step 1 On the unit s srb-20210g graphic image, right-click the management module area; a pop-up menu appears. Step 2 Step 3 Step 4 From the pop-up menu, select Policy Table; the Policy Table dialog box appears. For each event type, make the required selections as explained below. Click Apply (apply the changes without closing the dialog box) or OK (apply the changes and close the dialog box). Figure Policy Table Dialog Box The following table summarizes the events for which you can determine the required settings: Table 6-5. Event Types Event SM LID Change IB Port LID Change SM Registered to Events Link ETH Port State Description The Subnet Manager s LID parameter has changed The InfiniBand port s LID parameter has changed (InfiniBand port of the 10 GbE gateway) The Subnet Manager has been registered to handle events An Ethernet link s port state has changed 71

72 Version 2.0 Advanced Configuration Event Link IB Port State Number of 10GbE Gateways Changed 10GbE Gateway is Going to Reboot 10GbE Gateway Finished Initialization Description An InfiniBand link s port state has changed The number of 10 GbE gateways in the logical cluster has changed The srb-20210g unit is about to reboot. The srb-20210g unit has finished its initialization (usually after reboot) You can control each event via the corresponding options of the following columns: Send to Log Select this check box (on the event s row) if you want the system to register each occurrence of this event in the system log. Send Trap Select this check box (on the event s row) if you want the system to send an appropriate trap whenever this event occurs. Severity From the corresponding list box (on the event s row), select and set the severity level of this event. The possible severity levels: Normal Minor Warning Critical. NOTE: The Set Alarm, Auto Correction, and Trap Consolidation columns are ISR switch-related and are not functional for the srb-20210g unit. 6.8 Port MTU Configuration (Jumbo Frames) The srb-20210g supports different MTU sizes for Ethernet and IB networks. The supported MTU values are 1.5 KB or 9 KB for the Ethernet networks, and 1.5 KB or 2 KB for IB networks. The srb-20210g intermediates between these networks using different MTU sizes and by implementing the Path MTU discovery (PMTU) mechanism. Upon receiving an IP packet that is too large to transfer to the other side, the srb-20210g issues an ICMP error toward the sender, notifying regarding the maximum packet size that can be delivered to the destination IP. A host that supports PMTU decreases the size of the IP packets accordingly. The following table shows the possible port MTU configurations: Table 6-6. Possible Port MTU Configurations ETH Port MTU IB Port MTU KB (default) 2 KB (default) KB 1.5 KB 72

73 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 ETH Port MTU IB Port MTU 3. 9 KB 2 KB 4. 9 KB 1.5 KB NOTE: The srb-20210g does not support fragmentation. Accordingly, when working in an environment of IB and Ethernet networks and with different MTU size, all hosts that communicate through the srb-20210g must support Path MTU Discovery. Otherwise, the network may perform in an unexpected manner Setting the MTU via the GUI You can set the MTU of each Ethernet and InfiniBand port individually, via the Port Configuration dialog box. To set the MTU for individual ports: Step 1 On the unit s srb-20210g graphic image, right-click the Ethernet port or management module area; a pop-up menu appears. Step 2 Step 3 Step 4 From the pop-up menu, select Port Configuration; the Port Configuration dialog box appears. Select the port you want to configure and then enter the configuration parameters as explained below. Click Apply (apply the changes without closing the dialog box) or OK (apply the changes and close the dialog box). Figure Port Configuration Dialog Box 73

74 Version 2.0 Advanced Configuration For each selected port, the following parameters are displayed: Port The selected port; read-only. Port Type Type of the selected port; read-only. Status Current status of the selected port; read-only. The status can be either: Active Disconnected. MTU A list box containing possible MTU rates. Select the required rate for the currently displayed port: 1500 or 9198 for Ethernet ports, and 1500 or 2048 for InfiniBand ports. Speed The maximum data rate of the currently selected port; read-only. Admin State The current admin state of the selected port; read-only. The state can be either: Enabled Disabled. NOTE: After performing changes and clicking OK or Apply, a reboot confirmation dialog box appears. The changes take effect only after the device reboot is complete Setting the MTU via the CLI To set the port MTU via the CLI: 1. Type port ethernet show to view the current Ethernet port details: srb-20210g-7074(config-port)# port ethernet show port type status mtu speed(gbps) enable/disable copper active enable 2 none disconnected enable 2. Type port ib show to view the current IB port details: srb-20210g-7074(config-port)# port ib show port status mtu speed(gbps) active Type port ethernet set mtu and then the Ethernet port number followed by the required MTU value (in bytes) in the following format: port ethernet set mtu [1, 2] [1500, 9198]. For example: srb-20210g-7074(config-port)# port ethernet set mtu Type port ib set mtu and then the IB port number followed by the required MTU value (in bytes): 74

75 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 srb-20210g-7074(config-port)# port ib set mtu 2048 NOTE: The system must be restarted for the CLI changes to take effect. 6.9 MIB Support The Mellanox Management Information Base (MIB) is a database used to manage the Mellanox devices (objects) via an external SNMP manager or MIB Viewer. This feature allows the user to access the Mellanox device configuration and modify it by performing the changes in the private MIB file supplied by Mellanox. The Mellanox private MIB provides the same features and functionality supported by the GridVision GUI (Device Manager and Fabric Manager). You can download the Mellanox private MIB from the Mellanox Support site. Once opened in a SNMP manager, the MIB file allows the user to manage all agents as is being done via the GUI. Supported Protocols and Standards Mellanox supports the following standards and protocols: SNMPv2c protocol Standard Mibs including: mib-2 system, mib-2 ifmib (IF-MIB,ifXTable), mib-2 interface (RFC1213-MIB,ifTable). NOTE: For more information about MIB configuration, refer to the Mellanox MIB Reference Guide Security Support Overview This section introduces the Grid Switch features including detailed practical guidelines showing how configure each security type. Mellanox provides security for its srb-20210g implementing secure protocols (SCP) and disabling non-secured protocols. You can disable ports port through the VDM. Set up your system according to your security needs by using the commands listed in this section. IMPORTANT: You cannot connect to the srb-20210g through the SSH protocol. However, you can use SSH with the Grid Director and use the following command to connect to the srb-20210g: switch(config-chassis)# io connect 75

76 Version 2.0 Advanced Configuration All security features are available and accessible through the Switch. In order to secure the srb-20210g, you must disable the srb-20210g security features (telnet/ftp). As a result, the srb-20210g will only be accessible through the Switch and will benefit from the Switch security features, such as RADIUS and srb-20210g Filtering. IMPORTANT: The default configuration is not secured (telnet and ftp are enabled by default) Enabling/Disabling Non-secure Protocols Disabling SNMP The switch SNMP configuration is set by default to public/private. Change you SNMP configuration to secure your switch. (Note that the Mellanox GUI uses the default public/private security setting). You will need to enter the new SNMP community string settings when logging into the GUI session. To disable SNMP: Use the following table to enter the desired commands. Command guest admin config snmp snmp set [disable] snmp get [disable] Description a. Enter the guest mode. The default password for the guest user is voltaire. b. Enter the Privileged mode as follows: Type enable and enter the admin password (The default password for the admin user is ) OR Enter directly the privileged mode. The default password for the admin user is Note: When connecting to the srb-20210g via the switch using the io connect command, you are already in admin mode and do not need a password. Enters the config mode from the privileged mode. Changes to SNMP configuration mode. Disables SNMP set commands Disables SNMP get commands CAUTION: While this above step provides enhanced security, performing this will not allow you to use the GUI. 76

77 10 GbE Gateway (srb-20210g Module) User Manual Version Disabling FTP Access ( FTPD ) Mellanox allows you to disable the FTP non-secure protocol on the remote server using the following commands to provide enhanced security access to your system. To set ftpd via the CLI: Step 1 Step 2 From the 10G CLI, go to config security mode and disable the ftpd status, as follows: 10G(config-security)# ftpd set disable View the ftpd status, as follows. 10G(config-security)# ftpd show ftpd disable Disabling Telnet Access The telnet daemon provides the server function for the telnet protocol. The telnet feature enables you to access the switch from a remote site. Telnet is less secure than SSH and in order to enhance security, it is recommended that you disable the telnet daemon as described below. To set telnetd via the CLI: Step 1 From the 10G CLI go to config security menu and set the telnetd status, as follows: 10G(config-security)# telnetd set disable Step 2 View the telnetd status, as follows. 10G(config-security)# telnetd show telnetd disable 77

78 Version 2.0 Setting-up Networks Chapter 7. Setting-up Networks This chapter provides instructions on how to set up networks using the Networks Table. 7.1 Setting up a Network/Interface via the GUI The Networks table displays a list of the interfaces defined in the gateway. To display the Networks table: Step To access the Networks table, click the Networks node in the Device Manager navigation tree. The Networks table is displayed in the main window. Figure 7-1. Networks Table The following parameters are displayed: Table 7-1. Networks Parameters Table Parameter Description # Index number that the Device Manager automatically attaches to each network. It is incremented by one for each new entry. Name Subnet VLAN PKey Trunk The user-assigned name given to the interface. The IP address, subnet mask, and broadcast, of the interface. The VLAN tag assigned to the interface. The partition key of the interface. Link Aggregation Mode of the port [Full (P1-2) or Separate LAG (P1, P2) Network Table Right-click Menu Right-clicking the Networks table allows you insert and remove networks, as well as set the network default gateway, change the broadcast, and refresh the table. 78

79 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Figure 7-2. Networks Table Right-click Menu Defining (Adding) a New Network To define a new Network: Step 1 Right-click inside the Networks Table and select New Network from the menu. The New Network dialog box is displayed, as shown in the Figure below. Figure 7-3. New Network Dialog Box Step 2 Enter/edit the parameters, as required. Table 7-2. New Network Parameters Parameter Name IP Address Net Mask Instructions Enter/edit the Name of the interface/network. Enter/edit the IP address of the interface. Enter/edit the IP address mask of the interface. 79

80 Version 2.0 Setting-up Networks Parameter Broadcast PKey VLAN Trunk Instructions Enter/edit the Broadcast address of the interface. Enter the Partition Key. Enter the VLAN tag assigned to the interface. From the drop-down menu, select the aggregated ports in full or no Link Aggregation mode. Step 3 Click Apply and OK. A new network is created Viewing Network Information To view Network information: Step 1 Right click inside the Networks Table and select View Network from the menu. The View Network dialog box is displayed. Figure 7-4. Update Network Dialog Box Step 2 Click Close to close the window Removing Networks To remove a network from the Network Table: Step Right-click a specific network inside the Networks Table and select Remove Network from the menu. The network is removed. Note that you cannot remove the If0 (default) or smb-mng (management) network. Note: smb-mng is the management interface. If0 is the default interface. Neither can be removed. 80

81 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 To remove all networks from the Network Table: Step Right-click inside the Networks Table and select Remove All Networks from the menu. All user-defined networks are removed Updating the Default Gateway To update a Default Gateway: Step 1 Right-click inside the Networks Table and select Default Gateway from the menu. The following dialog box is displayed. Figure 7-5. Update Default Gateway Dialog Box Step 2 Enter the following information: Table 7-3. Update Default Gateway Parameters Parameter Interface Name IP Address Description Enter the name of the interface the on which the default gateway resides. Enter the IP address of the default gateway. You can see the updates to the Default Gateway under the 10G Static Route menu. Step 3 Click OK to save your settings Removing a Default Gateway To remove a Default Gateway: Step Right-click a specific network inside the Networks Table and select Remove Default Gateway from the menu. The Gateway is removed. You can see the updated changes under the 10G Static Route table (accessible from the Device Tree). 81

82 Version 2.0 Setting-up Networks Refresh Table To refresh the Networks table: Step Right-click the Networks Table and select the Refresh Table option from the menu. The Networks Table is refreshed. 7.2 Setting up a Network/Interface via the CLI Viewing Networks To view the network or interface(s) via the CLI: Step At the 10G(config-if)# prompt, type interface show all. (Note: this command is also available from the admin menu). The Networks Table displays a list of interfaces defined in the gateway and the following information for each defined interface: User-assigned interface name (Alias) Interface IP address, subnet mask, and Broadcast IP VLAN assigned to the interface Pkey (Partition key, which is the VLAN on the InfiniBand side) and the interface Trunk (full LAG, separate) 10G(config-if)# interface show all entry alias ip mask broadcast vlan pkey(hex) trunk smb-mng no 0x7fff No port 2 if no 0x7fff P1-2 3 CP x112 P1-2 4 net x75 P1-2 Add interface show [interface] command displays specific interface info 10G(config-if)# interface show if0 entry alias ip mask broadcast vlan pkey(hex) trunk if no 0x7fff P Adding Networks To add a Network (or Interface) via the CLI: Step At the 10G(config-if)# prompt, type the following command: 82

83 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 interface add interface add [alias] [tag - [1-4094]] [P1,P2,P1-2][pKey 0x0001-0x7fff] [Ip] [Mask] 10G(config-if)# interface add V5 108 P1-2 0x This example adds an IP interface with a logical name (here called V5), on VLAN 108, trunk P1-2 (full LAG configuration), to destination InfiniBand Partition 0x108. The IP address of the interface is and the subnet mask is The PKey value should be given in hexadecimal format. The user must first configure the 10G port PKey in the Subnet Manager in order for the given PKey to be recognized. If the 10G PKey is not configured in the switch, you will get the following message: Error : This interface will not be joined until its PKey is added to the switch's PKey table Removing Networks To remove a desired Interface: Step 1 Step 2 From the CLI Networks table, select the interface to be removed. At the 10G(config-if)# prompt, type the following command: 10G(config-if)#interface remove [alias] 10G(config-if)# interface remove V Setting the IP and Broadcast Address To set the IP and Broadcast address: Step Run the IP address set command and set the desired IP and broadcast addresses, as shown below. #(config-if)# ip-address set? ip-address set ip-address set [alias] [ip-address] [netmask] [optional broadcast] 10G(config-if)#ip-address set if Setting the Default Gateway You can set the default gateway IP address for the defined interface by using the following command. 83

84 Version 2.0 Setting-up Networks To set the default gateway via the CLI: Step From the 10G(config-route)# menu, run the following command: 10G(config-route)# default-gw set [row [1..65]][Gateway][Iface] Examples: The following example shows how to configure the default gateway of the srb-20210g. 10G(config-route)# default-gw set 2 smb-mng NOTE: smb-mng is the srb-20210g managing interface though which the Switch accesses the srb-20210g. This is used to manage the srb-20210g from the Switch. The default gateway is defined on the srb-20210g smb-mng interface through which all packets are transferred to/from the managing interface of the Switch smb-mng. 84

85 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Chapter 8. Traffic Flow Management This chapter provides instructions on how to set Traffic Flow Management using the Traffic Flow Management table. 8.1 Overview When a packet (TCP/IP, UDP, ICMP, or other) arrives at the srb-20210g, its header is compared with the TFM Table. If there is a match between the header and a TFM rule, the packet is treated according to the rule. If no rule matches the packet, the packet is dropped. The TFM Table displays the TFM rules defined in the gateway; from the TFM Table you can also define, edit and delete TFM rules. 8.2 Setting TFM Setting TFM via the GUI To display the TFM rules: Step Click the Traffic Flow Management item in the main window's navigation tree (left panel) to display the TFM Table. Figure 8-1. TFM Table The following parameters are displayed: Table 8-1. TFM Table Parameters Parameter Description # Rule position in the TFM Table. The order of the rules is important because once a packet matches a specific rule, the search against all rules listed below stops. The rule is compared, starting from the TFM rule at the lower index (starting at 1) to the higher one. The system stops checking once it finds the first matching TFM rule. Source Destination The source IP address and the source IP mask (separated by a backslash character). When zero values are displayed, the rule is applied to all packets, regardless of the source IP address. The destination IP address and destination IP mask (separated by a backslash character). When zero values are displayed, the rule is applied to all packets, regardless of the destination IP address. 85

86 Version 2.0 Traffic Flow Management Parameter Interface Service Name Action Log Description The interface name on which the rule is to be applied. The service name applied to the matching rule. A value of any indicates that the rule is applied to all packets, regardless of the service. This is the action taken for packets that match a specified rule. Tunneling means that the packet will get passed via tunneling. Discard means the packet will be dropped. The log that reports hits on this rule. These logs are intended for debugging purposes; the logs are counters to which you associate a rule. Whenever a hit occurs on a rule, its log counter is increased TFM Table Icons The icons located at the top of the TFM Table can be used to manage the TFM rules defined in the gateway. The following table lists the icons and the actions that they trigger. Table 8-2. TFM Table Icons Icon Action Move up. Select the rule in the TFM Table and click this icon to advance the rule priority. A rule with a lower number is applied first. Move down. Select the rule in the TFM Table and click this icon to lower the rule priority. A rule with a lower number is applied first. Insert. Add a new rule to the TFM Table. See next page for a description of the procedure for adding rules. Append. Select the rule in the TFM Table and click this icon to edit the definitions of the rule. Remove. Select the rule in the TFM Table and click this icon to delete a rule. Services. Click this icon to display the Services Table, which displays the list of services defined in the gateway Right-click Menu Right-clicking the Traffic Rules Table allows you insert, update, and remove rules, as well as refreshing the Traffic Rules Table. 86

87 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Figure 8-2. TFM Table Right-click Menu Inserting a New Rule To add a new rule to the TFM table: Step 1 Click the Insert icon in the TFM Table or select Insert Rule from the right-click menu. The following dialog box is displayed. Figure 8-3. Inserting (Adding) a Rule The following tables detail the parameters listed in the Insert Rule Box: Table 8-3. New Rule Parameters Parameter Source Address Source Mask Destination Address Destination Mask Interface Name Description The source IP address. Enter a value of any when you wish to apply the rule to all packets, regardless of the source IP address. The source subnet mask. To specify a range of IP addresses, e.g. if we want to take into consideration all IP addresses in the * subnet, then we use as source-ip-mask. Enter a value of any when you wish to apply the rule to all packets regardless of the subnet mask value. The destination IP address. Enter a value of any when you wish to apply the rule to all packets, regardless of the destination IP address. The subnet mask of the destination. The interface to which the rule is applied. 87

88 Version 2.0 Traffic Flow Management Parameter Service Name Action Log Description Optionally, select the service name matching the rule to be applied. The default value is any (the rule is applied to all packets, regardless of the service). To define a new service, click the New Service button and refer to the instructions under the Creating a New Service Section, on page 89. Action taken for packets that match are applied with the specified rule. Tunneling means that the packet will be forwarded. Discard means the packet will be dropped. These logs are counters to which you associate a rule. Whenever a hit occurs on a rule, its log counter is increased. In the CLI, you can view the logs using the log show command in the Statistics sub-menu. Step 2 Configure as required and click OK to save the new entry. The new rule is added at the bottom of the TFM table. Use the TFM table icons to modify the position of the rule in the table Updating a Rule To update a Rule: Step 1 Inside the TFM Table, right-click the rule you wish to update and select Update from the menu. The Update Rule dialog box is displayed. Figure Update Rule Dialog Box 88

89 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Step 2 Step 3 Change the configuration as required. Click OK to update the rule Creating a New Service When adding a TFM rule, the service matching the rule must first be defined. The service refers to the protocol of the packet. Each TFM rule is applied to packets of a specific service type. Step 1 Step 2 Right-click the Traffic table and select Insert Rule; the Insert Rule dialog box opens. Click New Service. The New Service dialog box opens (this box is also accessible from the Services dialog box). Figure 8-5. Adding a New Service Enter the following information: Table 8-4. New Service Parameters Parameter Service Name Source Port Destination Port Protocol Protocol Type Description Enter the name (logical) of the service. (User-defined) Enter the service source port. (0-4096) [Enabled only for UDP and TCP] Enter the service destination port. (0-4096) [Enabled only for UDP and TCP] From the drop-down menu, select the protocol for the new service (TCP, ICMP, UDP, IGMP). Enter the protocol type. [Field enabled only for ICMP] Step 3 Step 4 First, choose the service type and relevant parameters. Note that not all parameters are valid for each service type. Click Apply and Close or click OK. The New Service box closes. 89

90 Version 2.0 Traffic Flow Management Parameter Description In the Insert Rule dialog box, you can view the new service that you created from the Service Name drop down menu (see previous section). You can also view the list of services by clicking the Services button on the toolbar, as described in the following section Viewing the List of Services To view the list of the services defined in the srb-20210g: Step 1 Click Services on the TFM table. The Services window is displayed, showing the list of services defined in the srb-20210g and their details, as shown below. Figure 8-6. Services Table The following parameters are displayed: Table 8-5. Services Table Parameters Parameter Description # Serial number of the service. It is incremented by one for each new entry. Service Name Source Port Destination Port User-assigned service name. The rule named any is system-defined and cannot be removed. Source port assigned to the service. Only packets sent from this port are considered. If you select a value of any, packets with any source port will be considered (all ports match this service). This field applies only to TCP/UDP services. Destination port assigned to the service. Only packets sent to this port match this service. If you select a value of any, packets with any destination port will be considered (this means that all ports match this service). This field applies only to TCP/UDP services. 90

91 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Parameter Protocol Protocol Type Description Protocol used by the service, e.g. ICMP, IGMP, TCP, UDP. Protocol type. Applies only to ICMP message types. The values accepted are: 0, 3-5, Refer to relevant ICMP documentation for ICMP message types and their matching values. NOTE: You can also create a new service by clicking the Create New button in the Services Table or by right-clicking the Service Table. Refer to Creating a New Service Section, on page 89. Step 2 Click Close to close the window Removing a Rule from the TFM Table To remove a rule from the TFM table: Click the Remove icon in the TFM Table or select Remove Rule from the right-click menu. The rule is removed. Refreshing the TFM Table To refresh (update) the TFM table: Right-click the TFM table and select Refresh from the menu. The TFM table is updated with the latest changes Setting TFM via the CLI When a TCP/IP packet arrives at the srb-20210g, its header is compared with and checked against the TFM Table. If there is a match between the header and the rule in the TFM Table the packet is treated according to the rule defined in the TFM Table. If there is no match, the packet is dropped. Rule Priority The order of appearance of the TFM rules is important. Each rule is associated with a row number, which dictates the rule s priority level. A row number at the top of the table means higher priority; a row number at the bottom of the table means lower priority. The result of each rule can be either one of the following: Discard - Drops all packets according the TFM rule settings Tunneling - The packet is tunneled (forwarded) to the InfiniBand or Ethernet network, according to the TFM rule settings In the example below, rule 1 has a higher priority than rule 2 and 3. 10G(config-tfm)# rules show --- Total number of entries

92 Version 2.0 Traffic Flow Management entry Source Source-mask Dest Dest-mask Interface Service Routing Statu s Log if0 UDP discard active if0 TCP discard active 2 3 any any any any if0 any tunneling active 1 10G(config-tfm)# If an arriving packet matches more than one TFM rule, the rule with the highest priority will be applied to that packet. The TFM default settings are as follows: 10G(config-tfm)#rules show entry Source Source-mask Dest Dest-mask Interface Service Routing Status Log any any any any if0 any tunneling active no These default settings mean that any data (all packets) is passed from Ethernet to InfiniBand and vice versa. Services When adding a TFM rule, the service to which the rule is applied, must be specified. The service refers to the protocol type of the packet and port numbers (in case of UDP, TCP packets) or service type (in case of ICMP packets.). Each TFM rule is applied to packets of a specific service only. Each rule should be associated with a specific protocol service, which must be defined by the user. Supported protocols include TCP, UDP, and IGMP. Services should be set up prior to prior to creating the tfm rules. The tfm rules are then implemented according to the defined services Setting TFM Rules and Services using the CLI Table 8-6 lists some of the basic commands in the TFM mode of the CLI. Refer to Section for a full list of TFM commands. Table 8-6. Service and TFM CLI Rules Commands Prompt Command name Description 10G(config)# service Switches to service menu. 10G(config-servic e)# service add ICMP Adds the service (ICMP is used as an example here). You can add the following protocols: ICMP, IGMP, TCP services. 10G(config-tfm)# rules show Displays the existing TFM rules. 10G(config-tfm)# rules add Defines a new rule using the supplied parameters. In this example, assume that an ICMP 92

93 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Prompt Command name Description if0 icmp tunneling 3 packet arrives at the srb-20210g from source IP to destination IP , on interface if0. The actions are: Route tunneling Increment counter 15 by 1 The rule will be treated according to the icmp service predefined by the user. If there are no other TFM rules with a higher priority (i.e., a lower index number), then this rule will be applied to the packet. Therefore, the packet will be either discarded or forwarded and log counter #3 will be incremented by 1. 10G(config-tfm)# rules remove [entry all] Removes a TFM rule entry or all rules. 10G(config-tfm)# rules show Displays all TFM rules defined on all interfaces. 10G# statistic Changes to statistic mode 10G(statistic)# Log show Displays all the statistics Adding a New Service via the CLI When adding a TFM rule, the service matching the new TFM rule must first be defined. The service refers to the packet s protocol type. Each TFM rule matches packets that match one service. Command name Description Step 1 guest a. Enter the guest mode. The default password for the guest user is voltaire. b. Enter the Privileged mode as follows: Type enable and enter the admin password (The default password for the admin user is ) OR admin Enter directly the privileged mode. The default password for the admin user is Note: When connecting to the srb-20210g via the switch using the io connect command, you are already in admin mode and do not need a password. Step 2 config Enter the Config mode (from Privileged mode). Step 3 service Enter the Service configuration mode. Step 4 service add Sets the service. You can add the following protocols: 93

94 Version 2.0 Traffic Flow Management Command name Description ICMP, IGMP, TCP services. In the example below, a service for ICMP packets, whose matching service type = 8 (ping requests), is added. service add ICMP 8 ping_req Add a new service matching ping request packets (ICMP packets of request type = 8). Step 5 Service show Shows services 10G(config-tfm)# service show --- Total number of entries 5--- entry name protocol Sport Dport type any any any any any 2 NNN TCP 33 any * 3 XXX TCP * 4 KLL TCP any any * 5 KLL2 TCP * Updating Services via the CLI NOTE: Note that you cannot changes the items under service any. To update the services: Command name Description Step 1 guest a. Enter the guest mode. The default password for the guest user is voltaire. b. Enter the Privileged mode as follows: Type enable and enter the admin password (The default password for the admin user is ) OR admin Enter directly the privileged mode. The default password for the admin user is Note: When connecting to the srb-20210g via the switch using the io connect command, you are already in admin mode and do not need a password. Step 2 config Enter the config mode (from the privileged mode). Step 3 service Enter the Service configuration mode. 94

95 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Step 4 Command name service show service update service update Dport [row][dport] Modifies the service destination port. service update Sport [row][sport] Modifies the service source port. Relevant only to TCP or UDP services. Description Shows the list of services available in the srb-20210g. 10G-ipr-20(config-tfm)# service show --- Total number of entries 7--- entry name protocol Sport Dport type any any any any any 2 icmp8 ICMP * * 8 3 icmp10 ICMP * * 10 4 tcp1 TCP any 5021 * 5 tcp2 TCP * 6 udp1 UDP any 5023 * 7 udp2 UDP * This command modifies the service and all the tfm rules that are using this service. 10G-ipr-20(config-tfm)# service update Dport G-ipr-20(config-tfm)# service show --- Total number of entries 7--- entry name protocol Sport Dport type any any any any any 2 icmp8 ICMP * * 8 3 icmp10 ICMP * * 10 4 tcp1 TCP any 5060 * 5 tcp2 TCP * 6 udp1 UDP any 5023 * 7 udp2 UDP * Optional service update name [row][name] Modifies the service name. 10G-ipr-20(config-tfm)# service update name 2 icmp9 10G-ipr-20(config-tfm)# service show --- Total number of entries 7--- entry name protocol Sport Dport type any any any any any 2 icmp9 ICMP * * 8 3 icmp10 ICMP * * 10 4 tcp1 TCP any 5060 * 95

96 Version 2.0 Traffic Flow Management Optional Command name service update type [protocol] Relevant only for ICMP services. Description 5 tcp2 TCP * 6 udp1 UDP any 5023 * 7 udp2 UDP * Modifies the service type. 10G-ipr-20(config-tfm)# service update type G-ipr-20(config-tfm)# service show --- Total number of entries 7--- entry name protocol Sport Dport type any any any any any 2 icmp9 ICMP * * 0 3 icmp10 ICMP * * 10 4 tcp1 TCP any 5060 * 5 tcp2 TCP * 6 udp1 UDP any 5023 * 7 udp2 UDP * Step 5 Service show Shows services 10G-ipr-20(config-tfm)# service show --- Total number of entries 7--- entry name protocol Sport Dport type any any any any any 2 icmp9 ICMP * * 0 3 icmp10 ICMP * * 10 4 tcp1 TCP any 5060 * 5 tcp2 TCP * 6 udp1 UDP any 5023 * 7 udp2 UDP * For the complete list of CLI Service Mode commands, refer to Section Adding TFM Rules via the CLI The following is a description of the CLI command for adding a TFM rule. The syntax for adding a TFM rule is: 10G(config-tfm)#rules add [row > 0][Source any][source-ip-mask any][dest any][dest-ip-mask any][interface][service any][tunneling,discard] [log [no,1-15]] 96

97 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 where: row Source Source-ip-mask Dest Dest-ip-mask Interface Service tunneling, discard log [1-15] Index number of the rule. Rules are applied according to priority: if several rules apply to arriving packets, the rules with the lowest index number will be applied. You can add up to 1,204 rules. Source IP address. Enter a value of any when you wish to apply the rule to all packets, regardless of the source IP address. Source subnet mask. Specifies a range of IP addresses. E.g. if we want to take into consideration all addresses in the * subnet, then we use as the source-ip-mask. Enter a value of any when you wish to apply the rule to all packets regardless of the subnet mask value. Destination IP address. Enter a value of any when you wish to apply the rule to all packets, regardless of the destination IP address. Note: The destination IP can be a subnet (for example ) and not just a specific IP address. Subnet mask of the destination. Interface name on which the rule is to be applied. Optionally, enter the service name for which the rule is to be applied. Enter a value of any when you wish to apply the rule to all packets, regardless of the service. Action taken for packets that match the specified rule. tunneling means that the packet will be passed via tunneling. discard means the packet will be dropped. These logs are counters to which you associate a rule. A single log counter can be shared among several TFM rules. Whenever a hit occurs on a rule, its log counter is incremented. A shared log counter is incremented whenever one of its associated rules is hit. You can view the log using the log show command in the Statistics menu. For further details on Logs, refer to Chapter 10 on Logs and Statistics. The log data include the following: counter tx total pack tx total byte rx total pack rx total byte Counter number. Note that a single counter can serve more than one rule at a time. Overall number of transmit packets (from Infiniband to Ethernet) counted by this counter with their associated rules. Overall sum of Transmit bytes that passed due to a specific counter (from Infiniband to Ethernet). Same explanation but for Rx (from Ethernet to Infiniband). Same explanation but for Rx (from Ethernet to Infiniband). 97

98 Version 2.0 Static Route Configuration Chapter 9. Static Route Configuration This chapter provides instructions on how to configure Static Route using the Static Route Table. 9.1 Overview Static routing provides a means of explicitly defining the next hop from a gateway for a particular destination on an interface. Using the static route mechanism the user can set rules that can create new default gateways in the srb-20210g. The Static Route table enables to define interface static routing rules and add and remove static route entries, per row. 9.2 Routing Configuration Setting Routing via the GUI To view the Static Route Table in the VDM: Step Under the Device, expand the srb-20210g and click Static Route. The right side of the window shows the Static Route table. Figure 9-1. Static Route Table The following table details the parameters in the Static Route Table: 98

99 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Table 9-1. Static Route Table Parameters Parameter Description # The row number matching this rule. Destination Netmask Gateway Interface destination subnet destination subnet Netmask IP or IB gateway through which the traffic is transferred Logical name (alias) of the interface that the user defined in the Interface table. Note that smb-mng is the management interface and if0 is the default interface. To add a Static Route entry to the Static Route Table: Step 1 Right-click the Static Route table and select add static route from the menu. The Add Static Route dialog box opens. The gateway must be on the same subnet as the interface. Figure 9-2. Adding a Static Route Dialog Box Step 2 Step 3 Step 4 Step 5 In the row field enter the row number (rule number) to set the priority in the table for the new rule (you can edit the field manually or use the arrows). Enter the destination subnet, the destination subnet IP matching this rule, and the gateway through which the traffic is transferred. From the drop-down menu, select the desired Interface for which you would like to set the static route (the interface is what you have previously defined in the system). Click Apply and then OK to save your changes. 99

100 Version 2.0 Static Route Configuration To remove Static Route from the Static Route Table: Step 1 Step 2 Right-Click a Static Route entry in the table and select Remove Static Route to remove the static route row the static-route table. Click Apply and then OK to save your changes. To refresh the changes in the Static Route Table: If the Static Route Table window remains open, you can manually refresh the Static Route table to show the current (updated) static route information for each rule by right clicking the Static Route table and selecting Refresh Static Route Table from the menu Routing Configuration via the CLI The srb-20210g forwards traffic back and forth between IP and InfiniBand. The gateway is defined when packets must be transferred outside the boundaries of the IP or InfiniBand subnets, using the Static Routing commands. Static routing provides a means of explicitly defining the next hop from a gateway for a particular destination. NOTE: Using the static route mechanism the user can set rules that will be used as default gateway. To set routing configuration, use the following table as a guide: Prompt Command name Description 10G(config)# config Switches to Config menu 10G(config)# interface Switches to the Interface menu. 10G-ipr-20(configif)# 10G-ipr-20(configif)# interface show interface show all exit Shows all the interfaces. For example: 10G-ipr-20(config-if)# interface show entry alias ip mask broadcast vlan pkey(hex) trunk smb-mng no 0x7fff No port 2 if no 0x7fff P1 3 Hipi x10 P1 Exits to previous menu. 100

101 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Prompt Command name Description 10G(config)# route Switches to Route mode 10G(config-route)# static-route set [row][destination] [netmask] [Gateway][Iface] Defines the static routing rules of interfaces in a table. row is the row number in the table chosen for the new rule. Destination is the destination subnet. netmask is the destination subnet Netmask. Gateway is the IP or IB gateway through which the traffic is transferred. Iface is the logical name of the interface that the user defined in the Interface table. 10G-ipr-20(config-route)# static-route show entry destination netmask gateway interface smb-mng 10G-ipr-20(config-route)# static-route set? static-route set static-route set [row [1..10]][subnetDestination][netmas k][gateway][iface] 10G-ipr-20(config-route)# static-route set Hipi 10G-ipr-20(config-route)# static-route show entry destination netmask gateway interface smb-mng Hipi 10G(config-route)# static-route show Shows the static route table with the list of interfaces defined by the user. 10G(config-route)# static-route show entry destination netmask gateway interface smb-mng if0 10G(config-route)# static-route remove Removes the static route (removes the row in the static-route table). 10G(config-route)# static-route remove 2 101

102 Version 2.0 Logs and Statistics Chapter 10. Logs and Statistics If you experience a problem with the srb-20210g, you may be asked to provide the Export Log files for debugging purposes. You can also use the Port Counters Statistics feature to collect specific statistics on each one of the 10 GbE ports Using exportlog If you experience a problem with the srb-20210g, you might be asked to provide the exportlogs file for debugging purposes. This log can be stored on a remote server using the following CLI command: exportlog Uploads the log files to the Remote server, using previously defined server name, user name, and password. exportlog [remote path] 10G(config-remote)# exportlog /pub/ 10.2 Viewing Port Counters Statistics Viewing Port Counters Statistics via the GUI This feature allows you to run and view statistics on each one of the selected 10 GbE ports. To view the 10 GbE Gateway port statistics according to category via the GUI, proceed as follows: Step 1 On the srb-20120g zoom view, right-click one of the 10 GbE Gateway Ethernet ports and select Port Counters Statistics from the menu. The Port Counters Statistics dialog box is displayed, as shown below. Figure GUI (VDM) Port Counters Statistics Dialog Box 102

103 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Step 2 Step 3 Optional Optional Step 4 Step 5 Select the counters on which you would like to run the statistics by checking the box next to the desired port. Set the Polling Interval to define the time interval in seconds between which the system polls the counters. The window is automatically updated according to those settings. Check the Accumulator box to allow the packets to accumulate. Note that the packets accumulate from the moment this box is not selected (default). You can also reset the counters per port by selecting the port you would like to reset from the drop-down menu (on the right side), and clicking the Reset button afterwards. Click the Refresh button to update the counters immediately (this bypasses the accumulator). Click Close to exit the window. The following tables list the port counter statistic parameters and their description: Table Port Counters Statistics Parameters IB-to-Ethernet (Tx) Path Packets on the IB-to-Ethernet Path - by Category This table contains statistics of transmit port counter information. Port Total VLAN Frag IP Options Unicast ARP/RARP ICMP/IGMP TCP UDP Pause Frame Index that identifies the port that transmits the packets. Total packets of transmitted messages per port. VLAN packets of transmitted messages per port. Fragmented packets of transmitted messages per port. IP Option packets of transmitted messages per port. Unicast packets of transmitted messages per port. ARP/RARP packets of transmitted messages per port. ICMP/IGMP packets of transmitted messages per port. TCP packets of transmitted messages per port. UDP packets of transmitted messages per port. Pause Frame packets of transmitted messages per port. Table Port Counters Statistics Parameters Ethernet-to-IB (Rx) Path Packets on the Ethernet-to-IB Path - by Category This table contains statistics of receive port counter information. Port Total VLAN Frag IP Options Unicast ARP Index that identifies the port that receives the packets. Total packets of received messages per port. VLAN packets of received messages per port. Fragmented packets of received messages per port. IP Option packets of received messages per port. Unicast packets of received messages per port. ARP/RARP packets of received messages per port. 103

104 Version 2.0 Logs and Statistics Packets on the Ethernet-to-IB Path - by Category ICMP/IGMP TCP UDP CPU Process ICMP/GCMP packets of received messages per port. TCP packets of received messages per port. UDP packets of received messages per port. CPU Process of received messages per port. NOTE: Software packets are not counted in these counters. The presented counters are per Ethernet port (source port on the ETH-to-IB path, and destination port on the IB-to-ETH path) Viewing Port Status and Statistics via the CLI Ports Statistics displays the receive (rx) and transmit (tx) statistics of the gateway s ports since the last port statistic reset command. gbe-rx refers to the direction from the IP to the InfiniBand side. Table Port Status and Statistics Prompt Command name Description 10G(config-port)# 10G(config-port)# 10G(config-port)# port ethernet set enable port statistic reset [port] port statistics show Make sure the Ethernet port is enabled. Resets the receive (rx) and transmit (tx) statistics stored for the gateway s ports; statistics are accumulated from zero values after entering this command. 10G(config-port)# port statistic reset 2 Displays the statistics per each Ethernet port, since the last port statistic reset. See the example following this table. 10G(config-port)# port ethernet show Shows the status of all Ethernet ports. 10G(config-port)# port ethernet set mtu Updates the MTU of an Ethernet port. 10G(config-port)# port ib show Shows the status of InfiniBand port. 10G(config-port)# port ib set mtu Updates the MTU of an InfiniBand port. 104

105 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Example of Port Statistics: 10G(config-port)# port statistics show Type: port1 port total tx: vlan tx: frag tx: 0 0 ip-option tx: unicast tx: arp/rarp tx: icmp/igmp tx: tcp tx: udp tx: total rx: vlan rx: frag rx: 0 0 ip-option rx: 0 0 unicast rx: arp/rarp rx: icmp/igmp rx: tcp rx: udp rx: The following details the parameters of the port statistics: Parameter type port total tx vlan tx frag tx ip-option tx unicast tx arp/rarp tx icmp/igmp tx tcp tx udp tx total rx vlan rx frag rx ip-option rx unicast rx arp/rarp rx icmp/igmp rx tcp rx Description Port number Statistic type Total packets transmitted Transmitted packets that are associated with a VLAN Number of transmitted IP fragments Number of transmitted packets with IP options Unicast transmitted packets ARP/RARP transmitted packets ICMP/IGMP transmitted packets TCP transmitted packets UDP transmitted packets Total packets received Packets received that are associated with a VLAN Number of received IP fragments Number of received packets with IP options. Unicast received packets ARP/RARP received packets ICMP/IGMP received packets TCP received packets 105

106 Version 2.0 Logs and Statistics Parameter udp rx Description UDP received packets Prompt Command name Description 10G(port)# port status show port ethernet show Shows: 1. The status of each of the gateway Ethernet ports (enable/disable) 2. The port type: a. Copper b. Fiber short range c. Fiber long range d. Fiber extended range. 3. If it the link is up or down. 10G3(config-port)# port status show port type status mtu speed(gbps) enable/disable copper up enable 2 copper down enable 10G(config-port)# port ethernet show port type status mtu speed(gbps) enable/disable copper active enable 2 none disconnected enable The following details the parameters for the status show port ethernet show command: Parameter port type status mtu speed Description Port number Port type: Copper, Fiber SR (Short Range), Fiber LR (Long Range), Fiber ER (Extended Range) Port status options: Disconnected (no link) Active (link is up) Shows the present Maximum Transmission Unit (MTU) settings of each port (1500/2048 for IB; 1500/9198 for ETH) Port speed in Mbits/sec Prompt Command name Description 10G(port)# port ib show Shows the status of IB port (enable/disable) 106

107 10 GbE Gateway (srb-20210g Module) User Manual Version G(config-port)# port ib show port status mtu speed(gbps) active The following details the parameters for the port ib show command: Parameter port mtu speed Description Port number Shows the present Maximum Transmission Unit (MTU) settings of each port (1500 or 2048) Port speed in Mbits/sec Prompt Command name Description 10G(port)# port ethernet set mtu Configure the MTU of a specific Ethernet port. Possible values: 1500, G(config-port)# port ethernet set mtu Prompt Command name Description IB(port)# port ib set mtu Configure the MTU of an IB port. 10G(config-port)# port ib set mtu 2048 Port ethernet set mtu 1500 Possible values: 1500, Statistics Logs Logs enable the user to see the number of hits of packets per TFM rule. The user can define up to 15 logs. Each log can be associated with a specific TFM rule. Table Statistics Logs Prompt Command name Description 10G(config)# 10G(config-tfm)# 10G(config-tfm)# rules add rules add Switches to Config menu Displays the existing TFM rules in the TFM table. Adds rules to the TFM table. In this example, assume that an ICMP 107

108 Version 2.0 Logs and Statistics Prompt Command name Description if0 icmp tunneling 3 packet on the if0 interface arrives at the srb-20210g with source IP of and destination IP of The rule will be treated according to the icmp service pre-defined by the user. If there are no other TFM rules with a higher priority (i.e., a lower index number), then this rule will be applied to the packet. Therefore, the packet will be tunneled using the if0 interface, and log counter #3 will be incremented. 10G(config-tfm)# rules update log [row][no,1-15] Updates the log counter matching this rule to a new value. 10G# statistic Changes to statistic mode 10G(statistic)# Log show Shows counters with statistics on each rule. 10G(statistic)# rx-reset [counter] Resets the rx (received packets) counters of the specific log. 10G(statistic)# tx-reset [counter] Resets the tx (transmitted packets) counters of the specific log. The following is an example of a Statistics Log: 10G(statistic)# log show --- Total number of entries counter tx total pack tx total byte rx total pack rx total byte l Viewing Traps/Events When an event occurs in the system, it can be sent as a trap to the trap table/snmp tool, or as a log to a log file as defined in the Policy Table or in the CLI config/policy menu where policy is set per event. Logs are events that are saved within the system and sent to the log file, traps are sent via the SNMP to the GUI (SNMP manager). The policy can be configured for each event defined in the system in the following manner: srb-20210g-6050# config srb-20210g-6050(config)# policy srb-20210g-6050(config-policy)# policy table show Event Id Event Name Severity 108

109 10 GbE Gateway (srb-20210g Module) User Manual Version Ethernet Link Change critical Infiniband Link Change critical SM Lid Change warning Port Lid Change warning SM Registered to Events warning Number of 10GbE Gateways Changed warning GbE Gateway is Going to Reboot critical GbE Gateway Finished to Reload normal srb-20210g-6050(config-policy)# srb-20210g-6050(config-policy)# policy event show 1002 Event Id = 1002 Event Name = Infiniband Link Change Event Type = IB port change Issue Alarm = log+trap Severity = critical srb-20210g-6050(config-policy)# srb-20210g-6050(config-policy)# policy event set issue-alarm? policy event set issue-alarm policy event set issue-alarm [event id ( )] [log, trap, log+trap, none] srb-20210g-6050(config-policy)# policy event set issue-alarm 1002 log srb-20210g-6050(config-policy)# srb-20210g-6050(config-policy)# policy event show 1002 Event Id = 1002 Event Name = Infiniband Link Change Event Type = IB port change Issue Alarm = log Severity = critical srb-20210g-6050(config-policy)# 109

110 Version 2.0 Logs and Statistics NOTE: The possible severities for each event are: Critical, Warning, Minor and Normal. The possible actions (issue-alarm) for each trap are: log (write to the log) trap (send as a trap) log + trap (send as a trap and write to the log) none (not send as a trap, nor write to the log) Viewing Events Events can be viewed via the trap table at the bottom of the Site Summary Window, by any other SNMP session and/or by the trap log file. By default, Events are pre-defined. They can be changed as required by using the Policy Table or in the CLI config/policy menu, as described above. The trap log file can be viewed after exporting the log files via the config/remote menu to a remote host Viewing Traps The Trap Table is displayed at the bottom of the VDM main window. It provides information on the traps and events in the system. The Trap Table is shown in the lower pane of the Site Topology Window. The Trap Table is available in both the Fabric Manager and from the Device Manager. Trap ranges are set according to severity. For each trap/event, the Trap Table displays information as described in table 4-5. NOTE: The Trap Table only displays the events that occur when the Fabric Manager session is current. Events are time-stamped with the date/time of the 10GbE Gateway clock, which could differ from the PC time. Events originating in the PC are time-stamped with the PC clock time. 110

111 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Chapter 11. Multicast Settings This chapter provides instructions on how to configure a multicast group to either send a packet onto the InfiniBand side or drop it Overview This section introduces the feature that supports multicast. This feature allows configuring the fabric behavior in case the multicast packet arrives from the Ethernet side but a multicast group for this packed does not exist on the InfiniBand side. In such case, the packet that arrived can either be dropped or it can be forwarded to a user-defined default multicast group Definitions multicast multicast address multicast router Multicast is an IP feature that refers to single packets copied by the network and sent to a specific subset of network addresses. These addresses are specified in the Destination Address Field. Compare with broadcast transmission and unicast transmission. A multicast address is a single address that refers to multiple network devices. Synonymous with group address. Compare with broadcast address and Unicast address. See also multicast. A multicast router is used to send IGMP query messages on attached local networks. Host members of a multicast group respond to a query by sending IGMP reports noting the multicast groups to which they belong. The multicast router takes responsibility for forwarding multicast datagram from one multicast group to all other networks that have members in the group. See also IGMP Multicast Settings To configure multicast: Step 1 Step 2 Step 3 Step 4 Run a default-multicast-group show command to view the group IP and the chosen action. 10G-Card(config-multicast)# default-multicast-group show default multicast group ip is default multicast group action is drop Set the desired action (drop or send). srb-20210g-6060(config-multicast)# Set the multicast group IP address: Run a default-multicast-group show command to view the results. srb-20210g-6060(config-multicast)# default multicast group ip is default multicast group action is send Note that the default multicast group ip did not change; changes will only be applied after reboot. 111

112 Version 2.0 Multicast Settings Step 5 Step 6 Reboot the srb-20210g. Run a default-multicast-group show command to view the results. srb-20210g-6060> enable password: ****** Type Ctrl-z to exit from enable mode to disable mode. srb-20210g-6060# config srb-20210g-6060(config)# multicast srb-20210g-6060(config-multicast)# default-multicast-group show default multicast group ip is default multicast group action is send srb-20210g-6060(config-multicast)# Note that now the new multicast group ip is updated MC-Table Session Displays all multicast groups opened by the user and handled by this specific srb-20210g. Assume that there are two srb-20210gs and three open multicast groups in the cluster, as follows: ; ; , the mc-table for each of the srb-20210g will report the following: srb-20210g 1 mc-table: srb-20210g 1: srb-20210g-6230# mc-table show IP address MLID PKEY STATE xC00C 0xFFFF Join In following example, the mc-table displays only the single group that it is handling: srb-20210g 2 mc-table: srb-20210g 2: srb-20210g-6060# mc-table show IP address MLID PKEY STATE xC00D 0xFFFF Join xC00A 0xFFFF Join In the following example, the mc-table displays the two groups that it is handling: IP address MLID PKEY STATE - multicast group - multicast lid assigned to this group by SM - on which partition this multicast group is opened - suppose to be "join", it means the 10GbE gateway takes care of forwarding traffic on this multicast group. NOTES: If a specific mc group is closed in the SM, it will disappear from the mc-table display. If any srb-20210g becomes not available, its mc-table will be displayed as empty. Other srb-20210g devices in the cluster will contain entries for those mc groups. When the disabled srb-20210g is restored, its groups will move back from other srb-20210gs and it will display them again. 112

113 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Chapter 12. Mellanox Fabric Manager (VFM) You can change and view I/O information both via the Switch Fabric Manager and via the CLI Overview The GridVision Fabric Manager allows you to view specific srb-20210g Information, including port information, as well as slot position in the Switch. VFM also allows you to easily setup VLANs and PKeys in the srb-20210g Viewing I/O System Information via the GridVision Fabric Manager To view I/O Information: Step 1 Step 2 On the subnet tree (left panel in the main window), expand the switch option to view the various switches. Expand more to see the srb-20210g. Click the desired I/O module under the selected switch. The system overview for the chosen I/O module is displayed. Figure 12-1 shows the detailed system overview of an srb-20210g. The fields in this window are detailed in Table Figure Viewing I/O Information - srb-20210g The I/O Information shows how the details of the selected srb-20210g board. The subnet Tree indicates the slot location in the Switch by displaying the slot number next to the module. 113

114 Version 2.0 Mellanox Fabric Manager (VFM) 12.3 I/O System Overview Parameter Description Table I/O System Overview Fields Parameter Unique ID (GUID) Node Name IP Address Type Device ID Contact Location Operational State Vendor Uptime Description Displays the Global Unique Identification number (GUID) of the device which is the parallel of the MAC in the Ethernet world. Displays the user-assigned name of the node for easy identification. Displays the IP address of the node. This is the local IP address; it is not related to the two 10 GbE ports which might have other IP addresses. Specifies the I/O node type (in our case srb-20210g). Displays the identification number of the device (known value for the srb-20210g). String for contact name. It can be edited to any string in the properties window. SNMP agent location. Displays the current node state. This window will always show the node Operational State as Active. You can view disconnected nodes via the Table View (which can be selected at the bottom of the window). The Node Properties Window also shows the Operational State as being down and the time and date the node was disconnected. More explanations on these windows can be found in the Switch User Documentation Suite listed in the Related Documentation Section, at the beginning of this document. Displays the name of the vendor that manufactured the device. Date at which the node was activated or deactivated (up or down). Table I/O System Overview Fields - System Information Parameter Description Module Number Position of the Module constant 1. Line Number System GUID Position of the srb-20210g in the chassis. Shows the GUID of the Switch in which the I/O Gateway is installed. I/O Port Information is available via the Port View, the Port Properties, and the Properties panel. You can also edit the srb-20210g properties by right-clicking the srb-20210g in the internal structure frame (upper right panel) and selecting properties. In the window that opens, you can see some properties that can be changed if you select the "manual name and IP". srb-20210g properties can also be viewed via the Table View Viewing an I/O via the CLI From the switch CLI, access the chassis menu by running the following command: 114

115 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 ISR fa(config-chassis)# To view to which I/O you are connected, run the following command: ISR fa(config-chassis)# io show slot #2, position #1, I/O Type 10 GbE Gateway, IP: Mask: GetCommunity:public SetCommunity:private This shows that a srb-20210g (10 GbE Gateway) is installed in slot 2 of the Switch Connecting to an I/O Device via the CLI From the switch CLI, access the chassis menu by running the following command: ISR fa(config-chassis)# To view to which I/O you are connected, run the following command: ISR fa(config-chassis)# io show slot #2, position #1, I/O Type 10 GbE Gateway, IP: Mask: GetCommunity:public SetCommunity:private This shows that a srb-20210g (10 GbE Gateway) is installed in slot 2 of the Switch. To connect to the I/O, run the command shown in the following example: ISR fa(config-chassis)# io connect 2 1 This shows that the connection is made in io slot 2, position 1 (default). Now, you can configure the 10 GbE gateway Viewing I/O Port Properties To view the srb-20210g I/O port information via the GUI: Step 1 Step 2 Step 3 Step 4 In the Fabric Manager Site Topology Window, under Internal Structure, select the desired srb-20210g. Scroll down until you reach to the 10G Card and then expand the tree. Right-click the desired 10G port. From the menu, and choose Port Info from the menu. The srb-20210g Port Info dialog box is displayed, as shown in the following figure. 115

116 Version 2.0 Mellanox Fabric Manager (VFM) Figure IPR I/O Port Info Dialog Box The following table details the fields in the I/O Port Info dialog box. Table I/O Port Info Window Field Description Field Description Possible Values Health State The most critical alarm severity that was reported for the port. Normal, Warning, Minor, Critical State I/O Port state: Active, Disconnected Port Type I/O port type 10G, 1G MTU Maximum Transmission Units 1500 Port # External (Physical) port number 1, 2 Speed Shows the link s data rate 10GbE: , 1G: Media The communication media type Copper Fiber-optic Trunk Unique ID (GUID/MAC) Auto-sense The port s trunk type Port GUID If selected, the GbE port s data rate is auto-sensed 116

117 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Field Description Possible Values Full duplex If selected, the GbE port works in full-duplex mode 12.7 Viewing I/O Network (Partitioning) Information You can view the I/O Network (Partitioning) Information per gateway. To view the port information via the GUI: Right-click an I/O port in the Site Topology Window and choose Port Info from the menu. The Partitioning window is displayed, enabling you to set partitions. This window enables you to create networks as well as attach a port to a network. For more information on partitioning, refer to Chapter

118 Version 2.0 Partitioning Chapter 13. Partitioning This chapter provides instructions on network partitioning to enforce isolation among the ports within the InfiniBand fabric and allow them to communicate within the predefined virtual networks Overview Network partitioning is a method of sub-dividing and enforcing isolation among the ports of systems/nodes/elements within the InfiniBand fabric into predefined virtual networks to allow inter-communication between the members of the specific network. A network can be expanded to an Ethernet VLAN (Virtual LAN) via a srb-20210g enabling the two networks to communicate internally as one single network. It is currently recommended to perform this via the GridVision Fabric Manager GUI. Each port of a node is a member of at least one partition and may be a member of multiple partitions. Some shared resources such as storage and management servers are typically part of multiple partitions. Partitioning is useful for security purposes and/or to create multiple virtual switched networks and virtual network interfaces in the same infrastructure. Partitioning is managed by the Subnet Manager (SM). The GridVision Fabric Manager facilitates a centralized network configuration. The fabric administrator can utilize the Fabric Manager GUI application to: Define networks Assign server ports to one or more networks Configure srb-20210g interfaces enabling it to map an InfiniBand partition to an Ethernet VLAN Brief Functional Description Networks must first be created. They can be defined by the GridVision Fabric Manager. Elements are assigned to networks via the GUI and the CLI. The administrator maps each channel adapter port or gateway port to a network. The ports are marked with a unique number tag called Partition key (PKey). Ports assigned to different partitions cannot communicate with one another. A single port can be associated with up to 16 different networks. The subnet manager supports configuration of fabric partitions. The partitioning feature is disabled by default and can be enabled via the CLI and configured via the GUI. Partitioning CLI resides under the Switch(config-sm-partition)# menu. The Pkey Table can be altered via the CLI. The administrator can set up a partition to have a limited or full membership within a partition. Limited members cannot communicate with other limited members but communication is allowed between every other combination of membership types. Full members may communicate with all hosts (members) within the partition; this way in a specific partition, 118

119 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 shared resources such as a storage nodes marked as full members can be seen by all clients marked as limited members without creating a security hole where the clients are seeing each other. Switches and gateways may optionally be used to enforce partitioning. Partitioning can be used to place a group of hosts, gateway ports, or devices within a single virtual switched network. Some resources can be part of multiple groups, as shown in the diagram below. Figure Partitioning Block Diagram Diagram Description In the above block diagram (Figure 13-1), Partition A (PKey A) is shared between InfiniBand Hosts 1, 2, and 3 and is connected to VLAN Y (Ethernet Hosts) via the srb-20210g. Partition B (PKey B) is shared between InfiniBand Hosts 4, 5, and 1. Note that InfiniBand Host 1 belongs to both Partition A and Partition B, meaning that it can communicate between all hosts in Partition A and B. Partition C (PKey A) is shared between InfiniBand Hosts 6 and 7 and is connected to VLAN X (Ethernet Hosts) via the srb-20210g. There is also support for server connection to the Fibre Channel Storage. All InfiniBand hosts are connected to the InfiniBand via the Switch. All the members in the fabric can communicate with the Switch management network, which is the default partition Mechanisms Used to Create and Enforce Partitions 1. Each HCA or gateway port has a partition table defining the partitions that can be accessed by that HCA port. Each port may be added to up to 16 partitions/networks. 119

120 Version 2.0 Partitioning 2. The SM initializes each HCA or gateway port partition table. NOTE: Make sure that your host network interface is configured. Refer to the User Manual of your host for specific instructions. If using an HCA, refer to the OFED User Manual. 3. The SA answers PKey path queries by looking for a common PKey between the two end nodes. 4. IPoIB is used to create multiple IP domains, one for each partition, providing name/address resolution. Each IPoIB instance uses a separate QP (Queue Pair) associated with a separate PKey partition. 5. Each Multicast group is associated with a partition. In the case of IPoIB, there is a well-known MGID - PKey mapping Quick Settings Switch Settings (GUI) 1. Go to the VFM Subnet Tree and select Networks. 2. Right-click the Networks Window and select Create Network. 3. Enter the name of the network, the IP, the PKey, and the VLAN Tag. 4. Go to the subnet tree and click Site Topology. 5. Select the host to which you want to add to the partition. 6. Right click the host and select Partitioning. 7. Select the network to which you want to add your host and add it to the list. NOTE: You can set the network membership to Partial or Full membership srb-20210g Settings (GUI) You can create a VLAN combining an InfiniBand Network and an Ethernet Network by attaching the 10G port as follows: 1. Go to the VFM Subnet Tree and double click the srb-20210g. 2. Verify that the srb-20210g has an IP address. If there is an IP address, proceed to the following step. 3. If there is no IP address (the IP is ), switch to the VDM to set the srb-20210g configuration then reopen the srb-20210g VFM (see note at the end of this section). 4. Create a new network with the desired tag as follows: go to the VFM Subnet Tree, select Networks, right-click the Networks Window, and select Create Network. Enter the required information. 5. Go to the Map View and select the switch in which the srb-20210g is installed. 120

121 10 GbE Gateway (srb-20210g Module) User Manual Version Go to the Internal Structure dynamic panel (right side of the window). 7. Expand the menu under the srb-20210g. Right-click the port to which you want to attach the network, and select Partitioning. 8. Select the network to which you want to attach your srb-20210g port and add the network to the list. NOTE: The displayed IP can be in following cases: a. New srb-20210g that has no configuration b. srb-20210g after factory-default. In such case, the user has to configure IP of the management interface. This can be done by running the following command: switch config chassis io ip-address set. Steps: 1. Enter VDM or switch CLI for IP configuration. 2. Then restart the VFM Setting Partitioning via the GridVision Fabric Manager GUI NOTE: It is currently recommended to perform partitioning via the GUI and not via the CLI. The GridVision Fabric Manager enables you to: Define (create) networks Attach/assign server ports to a network Configure srb-20210g interfaces. Each srb-20210g has two 10 GbE ports enabling you to assign/create up to sixteen InfiniBand partitions or networks (Ethernet VLANs) per srb-20210g. Each network can either be assigned to a single port or assigned to a group of ports Multiple Selections You can perform multiple selections for partitioning configuration from the following views: Port view you can select many server ports and assign them to a network. If other ports are selected (Switch, gateways), they are ignored in this operation Map view & table view selecting a server will apply partitioning to all its ports Rack selection (via Map View or Left Tree) - selecting a rack will apply partitioning to all its servers ports NOTE: The Multiple Selection feature is achieved by using shift + click (not control + click). 121

122 Version 2.0 Partitioning Creating a Network To create a network using partitioning via the GridVision Fabric Manager GUI: Step 1 Step 2 In the main window, under the Subnet Tree select Networks. Right-click the main window (Subnet: Networks) and select Create Network from the menu. Figure Creating a Network The Create Network dialog box is displayed. Figure Create Network Dialog Box Step 3 In the Create Network Window Name field, enter the name of the network (user-defined). Note 122

123 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 that if you want to configure a srb-20210g, this name will be used as the default name of its interface. NOTE: If, from the Switch CLI, you run a config chassis interface interface show command, you will see this name as the newly created srb-20210g interface. Refer to the Switch User Manual (GridVision Documentation Suite) for more information. Step 4 Step 5 Step 6 Step 7 Step 8 Enter the appropriate Subnet IP address, network mask, and Default Gateway according to the defined network. Enter the PKey of the network that you want to define (in Hexadecimal format). Select the default membership. Partial: Members with Limited membership cannot communicate with other members with limited membership but communication is allowed between every other combination of membership types. Full: Members with Full membership may communicate with all hosts (members) within the network/partition. Check VLAN Tagged if you wish to configure a VLAN to be part of the InfiniBand network. This is performed via the srb-20210g interface enabling it to attach an InfiniBand partition to an Ethernet VLAN. Click Apply and OK to save your changes. The new network is created and added to the Networks Table. In the example below, the new network is named Bob. Figure Network Table - A New Network Created Note that you can also create a network from the Port Info or Partitioning windows, which are accessible by right-clicking a port in any of the windows where ports are visible, such as: A Host (from the Topology Map, the Subnet Tree, or from the Table View) The Site Topology Port Table (Port tab) An I/O port under the Internal Structure in the right pane of the Site Topology Window A rack (Topology Map) Step 9 On the Partitioning tab, click New Network and proceed as described in the above procedure. 123

124 Version 2.0 Partitioning Figure Creating a Network using the Port Info Partitioning Tab Once your network is defined, you can connect srb-20210g ports to the VLAN if you wish to attach an InfiniBand partition to an Ethernet VLAN Viewing Network Information To view information for a specific network: Step 1 Step 2 Right-click the desired network. Select View Network. The View Network Information dialog box is displayed. Alternatively, you can double click the network to show the dialog box, which shows the same information per network as the information detailed in the Network Table window for all networks (see Figure 13-8). Figure View Network Information Dialog Box 124

125 10 GbE Gateway (srb-20210g Module) User Manual Version Assigning Server Ports to a Network To assign or add a Server/Host port to a network or partition: Step 1 Step 2 Step 3 Select the host you wish to attach to a network or partition (either from the Subnet Tree or from the Site Topology Map View). In the Map View, right-click the Host you wish to attach to a network/partition and select Partitioning. Alternatively, you can access the Partitioning window by right-clicking the selected host in the Table View, in the Internal Structure, a host within a Rack, or the Host port in the Ports View. Under Available Networks, select the network from the list (in our example Bob) and click Add to associate the port with the network. The network (Bob) is added to the lower section (Networks accessed by selected port/s). Note that when adding or deleting a network, the Add/Delete buttons are disabled once you added or deleted the selected network to the lower section of the table. Figure The Host is added to the Bob Network/Partition Step 4 Click Apply and OK to save your changes Deleting Server Ports from a Network You can delete server ports from a network or partition. To delete a Server/Host port from a network or partition: Step 1 Step 2 Select the Host you wish to delete from a network or partition (either from the Subnet Tree or from the Site Topology Map View). In the Port Info window (Figure 13-10), under the Networks accessed by selected port/s section, select the network port from the list (in our example Bob) and click Delete to remove the port from the network. The network (Bob) is removed from the lower section (Networks 125

126 Version 2.0 Partitioning accessed by selected port/s). Step 3 Click Apply and OK to save your changes Assigning Network Membership NOTE: This feature is only applicable to reassign the default membership of the management network. To change the Membership of a Network/Partition: Step 1 Step 2 Step 3 Step 4 In the Port Info window (Figure 13-10), under the Networks accessed by selected port/s section, select the network port from the list (in our example Bob). Under Membership (IB Only) set the membership for this network to Full or Partial, as desired. Click Update Members. Click Apply and OK to save your changes. The changes will only be effective if you click Apply. Alternatively, you can change the default membership from the Networks Table, as follows: Step 1 Right-click the Network Table and select Change Default Management. Figure Changing Default Membership for Management Network Step 2 A configuration message appears prompting for confirmation change from default membership to partial. Click Yes to confirm the change Mapping an InfiniBand Partition to an Ethernet VLAN via the 10 GbE Gateway Mapping an InfiniBand partition to an Ethernet VLAN is performed via the 10 GbE Gateway whose ports are assigned to the VLAN. The 10 GbE Gateway can provide up to 16 interfaces. 126

127 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 To connect 10 GbE Gateway Ports to the VLAN: Step 1 Step 2 Step 3 Step 4 Under the Site Topology Map View select a switch. Under the Internal Structure (dynamic right panel), select the desired srb-20210g and expand it. Scroll all the way down to 10 GbE Gateway and expand it. Right-click the desired 10 GbE10 GbE port that you want to attach to a partition. The 10 GbE Gateway ports can only be selected individually. Figure Selecting the Port of an 10 GbE Gateway NOTE: In Full LAG mode: If the 10 GbE Gateway is set to Full LAG mode, both 10 GbE Gateway ports are attached. In No LAG mode: If working in no LAG mode, you can only set the specific 10 GbE Gateway port (Port 1 or Port 2) to the VLAN tag. Step 5 From the right-click menu, select Partitioning. The Port Info window appears displaying on top the Partitioning tab (see Figure 13-7). 127

128 Version 2.0 Partitioning Figure VLAN attached to the InfiniBand Network Step 6 Step 7 Under Available Networks, select the network from the list (in the above example: Bob) and click Add to associate the srb-20210g port with the network. The network (Bob) is added to the lower section (Networks accessed by selected port/s). Click Apply. A Success message appears with the following message Add operation completed successfully. Click OK. NOTE: When removing a network, you get the following message: Remove operation completed successfully. Step 8 Click OK again in the Port Info Window to save your changes. The VLAN network is now connected to the InfiniBand fabric via the srb-20210g InfiniBand port that was previously selected Multiple Selection for Partitioning Configuration Under the Port view (Site Topology Ports tab), you can select multiple node ports and assign them to a network. If ports other than HCAs are selected (such as switches or I/O gateways) they are ignored in this operation and are not assigned to the network. Under the Site Topology Map view and Table view, when selecting a server and performing partitioning, partitioning is applied to all its ports. If you select a rack (via the Map view or under the Subnet Tree) and perform partitioning, partitioning is applied to all its server ports. Note that you can perform a multiple selection operation by using shift + click or simply by mouse selection. 128

129 10 GbE Gateway (srb-20210g Module) User Manual Version Setting Partitioning via the CLI This mode allows you to set up a network using the partition feature. The user has the option to work either with the port GUID or with the system (node/element) name and physical port number used as a combination. The CLI enables you to create a network but does not currently support network configuration, meaning that you cannot change the network parameters once a network has been created. The GridVision Fabric Manager GUI currently has more options to initially configure the network than has the CLI. Therefore, the GUI is currently the preferable option to set up a network. Once a network has been created, you cannot change its configuration. You can however delete a network and create a new network. IMPORTANT: When adding a srb-20210g to a Partition through the CLI, make sure you add both srb-20210g port GUIDs (you can view the srb-20210g GUIDs by running a netdiscover command from the Utilities Menu. Refer to the CLI config-sm-partition menu in the CLI Reference Guide for a full list of commands on partitioning and networks Quick Steps to set up a Network and Perform Partitioning To create a network for Hosts, perform the following steps: Step 1 Step 2 Add a network by running the network add command (see section below for more detail). Option 1: Bind the Port to a Network using the System Name and the number of the physical port by running the interface port add command (see following section for more detail). Option 2: In case you have the port GUID, bind the Port GUID to a Network by running the interface guid add command (see following section for more detail) Creating a Network To create a network, run the following command: Command: network add Adds a new network to the Network Table. IMPORTANT: 1. You must give a unique network name and a unique PKey value that do not exist in the Network Table. PKey values can range from 0x1 to 0x7ffe (0x7fff is reserved for management). 2. The management is a default network and appears by default in the table. network add [network] [pkey value (hex (0x1..0x7ffe))] 129

130 Version 2.0 Partitioning ISR (config-sm-partition)# network add John 0x1111 ISR (config-sm-partition)# Expected results: - Showing the newly created network information ISR (config-sm-partition)# network show network name pkey descr ip address subnet mask vlan gateway dhcp gvdhc membe trust QoS statu value tag ship level prior John 0x parti management 0x7fff full Deleting a Network To delete a network (remove a network from the Network Table), run the following command: Command: network remove Removes the network from the Network Table. network remove [network name (str)] ISR (config-sm-partition)# network remove John ISR (config-sm-partition)# Expected results: Removed the specific network (John is no longer in the table). ISR (config-sm-partition)# network show network name pkey descr ip address subnet mask vlan gateway dhcp gvdhc membe trust QoS statu value tag ship level prior management 0x7fff full Viewing Network Details To view the details of a specific network, run the following command: Command: network show Shows the details of the specific network. network show [network (str)] Switch(config-sm-partition)# network show 2323 network name: 2323 pkey value: 0x43 description: ip address: subnet mask: vlan tag: 34 gateway: dhcp: 0 130

131 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 gvdhcp: 0 membership: full trust level: 0 QoS priort: 1 status: Viewing the Network Error Log File Command: network-file error-log show Shows the network error file ((PartitionError.txt)). network-file error-log show Switch(config-sm-partition)# network-file error-log show In line 26: The port guid 0x0 doesn't have the "management" network in it's first network column. In line 26: Import failed. Fix the imported file first. Keeping the old tables Binding a Port GUID to a Network Adding a Port GUID to a Network To bind a port GUID and its specific membership to the network (adding the GUID to the Pkey Table), run the following command: Command: interface guid add Binds the GUID and its specific membership to the network. A GUID can be any string containing a hexadecimal number. Note that the system does not check whether the GUID exists in the system or is legal. If the GUID does not exist in the system or is not legal, the configuration will not be implemented. You can also use this command to update an existing network membership. For example, if a network membership is set up to partial membership (Part), you can set it to Full membership instead. interface guid add [guid (hex 0x)] [network (str)] [membership (full part)] Example - Option 1: Adding a new GUID (HCA, or gateway) to a network Switch(config-sm-partition)# interface guid add 0x8f br1 part Expected Results (Pkey Table for the specific GUID): Switch(config-sm-partition)# interface guid show 0x8f guid name network name membership x8f management full 0x8f br1 partial 131

132 Version 2.0 Partitioning Expected Results (Pkey Table): Switch(config-sm-partition)# interface show System name Port # Port Guid network name membership x0 management full HCABob 1 0x8f management full HCABob 1 0x8f br1 full HCABob 1 0x8f br3 full HCABob 1 0x8f br4 full HCABob 1 0x8f br2 full HCABob 1 0x8f Bob full HCAJohn 2 0x8f management full HCAJohn 2 0x8f br1 full HCAJohn 2 0x8f br3 full HCAJohn 2 0x8f br4 full HCAJohn 2 0x8f br2 full HCAJohn 2 0x8f Bob full NewHCA 0 0x8f management full NewHCA 0 0x8f br1 partial Example - Option 2: Adding an existing GUID to a new network Switch(config-sm-partition)# interface guid add 0x8f br5 full Expected Results (Pkey Table for the specific GUID): Switch(config-sm-partition)# interface guid show 0x8f guid name network name membership x8f management full 0x8f br1 full 0x8f br3 full 0x8f br4 full 0x8f br2 full 0x8f Bob full 0x8f br5 full Expected Results (Pkey Table): Switch(config-sm-partition)# interface show System name Port # Port Guid network name membership x0 management full HCABob 1 0x8f management full HCABob 1 0x8f br1 full HCABob 1 0x8f br3 full HCABob 1 0x8f br4 full HCABob 1 0x8f br2 full HCABob 1 0x8f Bob full HCABob 1 0x8f br5 full HCAJohn 2 0x8f management full HCAJohn 2 0x8f br1 full HCAJohn 2 0x8f br3 full 132

133 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 HCAJohn 2 0x8f br4 full HCAJohn 2 0x8f br2 full HCAJohn 2 0x8f Bob full NewHCA 0 0x8f management full NewHCA 0 0x8f br1 partial Setting GUID Membership To set GUID membership (full or part), run the following command: The membership of the GUID in the example below is set to part. Switch(config-sm-partition)# interface guid show 0x8f guid name network name membership x8f management full 0x8f br1 part Run the following command to change the membership to full : Switch(config-sm-partition)# interface guid add 0x8f br1 full Expected results: Switch(config-sm-partition)# interface guid show 0x8f guid name network name membership x8f management full 0x8f br1 full Removing a GUID from a Network To remove a GUID from a network (removing the GUID from the Pkey Table for that network), run the following command: Command: interface guid remove Removes the GUID from the network. interface guid remove [guid (hex 0x)] [network (str)] If the following GUID appears in the Pkey Table: Switch(config-sm-partition)# interface show System name Port # Port Guid network name membership HCABob 1 0x8f Bob full

134 Version 2.0 Partitioning Run the interface guid remove command to remove the port GUID attached to the network: Switch(config-sm-partition)# interface guid remove 0x8f Bob Expected results: Switch(config-sm-partition)# interface show System name Port # Port Guid network name membership x0 management full HCABob 1 0x8f management full HCABob 1 0x8f br1 full HCABob 1 0x8f br3 full HCABob 1 0x8f br4 full HCABob 1 0x8f br2 full HCABob 1 0x8f br5 full HCAJohn 2 0x8f management full HCAJohn 2 0x8f br1 full HCAJohn 2 0x8f br3 full HCAJohn 2 0x8f br4 full HCAJohn 2 0x8f br2 full HCAJohn 2 0x8f Bob full NewHCA 1 0x8f management full NewHCA 1 0x8f br1 partial Note that port GUID 0x8f was removed from the Bob network and was deleted from the Pkey Table. To remove a specific GUID from all networks ((removing the GUID from the Pkey Table for all networks), run the following command: Command: interface guid remove-all Removes the GUID from all the networks. interface guid remove-all [guid (hex 0x)] Switch(config-sm-partition)# interface guid remove-all 0x8f Expected results: Switch(config-sm-partition)# interface show System name Port # Port Guid network name membership x0 management full HCAJohn 2 0x8f management full HCAJohn 2 0x8f br1 full HCAJohn 2 0x8f br3 full HCAJohn 2 0x8f br4 full HCAJohn 2 0x8f br2 full HCAJohn 2 0x8f Bob full HCANew 1 0x8f management full 134

135 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 HCANew 1 0x8f br1 partial Viewing a GUID, Related Network, and Membership To view the GUID and related network as well as network membership, run the following command: Command: interface guid show Shows the port GUID belonging to the networks and memberships within the networks. This enables you to view to which network a specific GUID belongs. If the table is empty, it means that there are no networks assigned to the specific GUID (except for the management network, which is present by default). interface guid show [guid (hex 0x)] Switch(config-sm-partition)# interface guid show 0x8f guid name network name membership x8f management full 0x8f br1 full 0x8f br3 full 0x8f br4 full 0x8f br2 full 0x8f Bob full To view all the port GUIDs included in the network, run the following command: Command: Example 1: interface network show Shows all the port GUIDs included in the network. interface network show [network (str)] Switch(config-sm-partition)# interface network show management guid name membership x0 full 0x8f full 0x8f full 0x8f full This example shows the GUIDs that are associated with the management default network. Note that all GUIDs that are currently present in the PKey table should appear in this view since all GUIDs are associated with the management network. Example 2: Switch(config-sm-partition)# interface network show Bob guid name membership x8f full 0x8f full 135

136 Version 2.0 Partitioning CAUTION: Do not change the headers and general structure of the file. You may only change the parameters. RECOMMENDATION: Before editing the Partition.csv exported file, it is safe to back up the file in case you need to return to the previous configuration. Figure Partition.csv file This table gathers both the Network Table as well as the Pkey Table (Ports). The Network Table is the exact same table displayed after running the network show command. The Pkey Table is the exact same table displayed after running the interface show command. Table 13-1 details the Partition.csv fields. This table is divided in two sections: Networks and Ports. Table Partition.csv Field Description Table Networks Networks Network Name PKey Value Description IP address Subnet Mask Description Unique network name defined by the user (without spaces). Pkey number in hexadecimal format. Description of the network. Network subnet IP address. Network subnet mask. 136

137 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 VLAN Tag Gateway DHCP GV DHCP Membership Trust Level QoS Priority Status Integer defined by the user on the Ethernet side. Enables a VLAN to be part of the InfiniBand network. IP of the gateway. For future use. DHCP mode can be enabled for the smb Ethernet interface. This mode is operational only when the Ethernet is the selected management interface. By default, the system uses a static configuration for the IP address. For future use. GridVision DHCP. Network default membership. While binding the port GUID to this network, if you did not specify the port membership (Full/Partial), the system uses the default network membership (1=full, 0=partial). For future use. For future use. For future use. Table Partition.csv Field Description Table Ports Ports Port Subsystem Location Port# Port GUID Net Membership Description System/Node Name. Name of the module contained in the system (e.g., srbd, slb). Module physical location (for example slot number in which the module is installed in the Switch). Physical port number on the module for both external and internal ports. Port GUID. Network names with which the port GUID is associated. There can be up to 16 networks. Full or partial membership of the Port GUID in the network Importing the Partition.csv File To import a Partition.csv file, run the following command: Command: Network-File Import Downloads the Partition.csv file from the Remote server, using the previously defined server name, user name, and password. Note that the partition file includes both the Network Table and the Pkey Table. (Make sure your Remote server is defined as detailed in Section 6.3). This file overwrites your current Network Table and Pkey Table configuration. network-file import [remote path] 137

138 Version 2.0 Partitioning Switch(config-sm-partition)# network-file import pub Transferring file via FTP from remote site Transfer Ok Look in the PartitionError.txt file for encountered warnings. Expected results: You can view the list of partition errors by entering the path of the PartitionError.txt file in your browser, for example: List of errors reported in the PartitionError.txt file: In line 17: Failed to translate system name "" to guid. In line 20: Failed to translate system name "" to guid. 138

139 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Chapter 14. Command Line Interface (CLI) Management This chapter provides reference for the Command Line Interface (CLI), which can be used instead of the GUI for configuration and management of the gateway Starting a CLI Management Session To start a CLI management session via serial connection to the switch: Step 1 Step 2 Step 3 Step 4 Step 5 Connect the PC to the switch via its serial port, using the cable supplied by Mellanox. Start a HyperTerminal client (or compatible) on the computer. Configure the terminal emulation parameters Type the appropriate user name (admin or guest) and password at the logon prompt. Note that the password is represented by stars ********. guest admin a. Enter the guest mode. The default password for the guest user is voltaire. b. Enter the Privileged mode as follows: Type enable and enter the admin password (the default password for the admin user is ) OR Enter directly the privileged mode. The default password for the admin user is Issue the appropriate CLI commands to complete the required actions. NOTE: When the 10 GbE CLI syntax included parameters in brackets [ ], you must include at least one of the parameters in the command Mellanox CLI Model The Mellanox srb-20210g CLI model consists of a hierarchical structure of the menus and sub-menus, as shown in the diagram below. For further detail, refer to the srb-20210g CLI Reference, in Chapter

140 Version 2.0 Command Line Interface (CLI) Management Exec Mode Read-only Menu Login: admin Password: Privileged Mode Privileged mode: IPR>enable Password: voltaire Safe Mode Login: admin Password: config (Configuration Mode) Main Menu Sub-Menu Debug (for Voltaire use only) statistics config Sub-Menus - cluster - diagnostics - interface - multicast - ntp - port - remote - route - security - snmp - tfm Figure Mellanox srb-20210g CLI Model Safe Mode is provided for troubleshooting purposes. For more information on Safe Mode functionality, refer to Section CLI Operating Conventions Keyboard Usage in CLI Operation In addition to the standard commands, the CLI (command line interface) supports the use of the following special keys or key combinations for the functions: described in the following table. Table CLI Key Functions Key Tab Delete or Backspace Enter Left Arrow Right Arrow Function Completes a partial command name entry. When you enter a unique set of characters and press the Tab key, the system completes the command name. If you enter a set of characters that could indicate more than one command, the system beeps to indicate an error. Enter a question mark (?) immediately following the partial command (no space). The system provides a list of commands that begin with that string. Erases the character to the left of the cursor. At the command line, pressing the Enter key executes the user command. At the prompt on a terminal screen, pressing the Enter key scrolls down a line. Moves the cursor one character to the left. When you enter a command that extends beyond a single line, you can press the Left Arrow key repeatedly to scroll back toward the system prompt and verify the beginning of the command entry. Moves the cursor one character to the right. 140

141 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Key Up Arrow or Ctrl-P Down Arrow or Ctrl-N Ctrl-A Ctrl-B Ctrl-D Ctrl-E Ctrl-F Ctrl-K Ctrl-R Ctrl-T Ctrl-X Ctrl-W Ctrl-Y Esc B Esc C Esc D Esc F Esc L Function Recalls commands in the history buffer, beginning with the most recent command. Repeat the key sequence to successively recall older commands. Return to more recent commands in the history buffer after recalling commands with the Up Arrow or Ctrl-P. Repeat the key sequence to recall successively more recent commands. Moves the cursor to the beginning of the line. Moves the cursor back one character. Deletes the character at the cursor. Moves the cursor to the end of the command line. Moves the cursor forward one character. Deletes all characters from the cursor to the end of the command line. Searches for commands that were entered previously in the CLI session (command history). After you press Ctrl-R, enter the first letter of the command you wish to recall; the command is displayed at the CLI prompt. Transposes the character to the left of the cursor with the character located at the cursor. Moves the cursor to the beginning of the command line. Deletes the word to the left of the cursor. Recalls the most recent entry in the delete buffer. The delete buffer contains the last ten items you have deleted or cut. Moves the cursor from the middle of the word to the beginning. Capitalizes the word from the cursor position to the end of the word. Deletes from the cursor to the end of the word. Moves the cursor forward one word. Changes the word to lowercase at the cursor position to the end of the word CLI Command Structure All CLI commands follow a simple structure that includes the factory-default name of the module, a unique identifier, and the prompt, according to the correct level. The commands prompt for further information as the user types. There are two types of commands: the show (or view) commands, allowing you to view specific information, and the set commands, allowing you to configure your system according to your needs. The CLI provides the following basic commands: 141

142 Version 2.0 Command Line Interface (CLI) Management Table Basic Commands add Command Description Adds configuration to a previously created entity. remove set show Deletes selected entities that were created or added. Sets various parameters of a created entity. Displays configuration and operational data. Table Other Operational Commands exit Command Description Leaves current mode and returns to previous mode.? Displays help text for the CLI command. <command>? end Displays the help for the specific command. Ends the CLI. NOTE: When the srb-20210g CLI syntax defines parameters in brackets [], you must include at least one of the parameters in the command Case-Sensitivity CLI commands must be entered using the same case as shown in this manual. User-defined strings, such as names or descriptions, may appear in any case, including mixed case. Case information for user-defined strings is preserved in the configuration CLI Modes There are three CLI usage modes for the srb-20210g: Exec, Privileged, and Configuration. Some modes are password-protected and can be accessed from other mode levels. A Debug mode is included for Mellanox internal use only. Each mode is easily recognized by the command line prompt. The CLI command prompt includes the srb-20210g (10G) system name. An asterisk is added to the beginning of the prompt if the system configuration has been modified but not saved. NOTE: The default prompt for the srb-20210g in the CLI is srb-20210g-[boxid], where the four last bytes [BoxID] are unique. 142

143 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Table CLI Main Modes CLI mode Prompt Access Method Exit Method exec srb-20210g-[boxid]> Log in Use the end command to exit the CLI. privileged srb-20210g-[boxid]# From the Exec mode, use the command. To exit back to user Exec mode, use the disable or exit command. To enter global configuration mode, use the config command. Use the end command to exit the CLI. config srb-20210g-[boxid]( config)# From the Privileged mode, use the command. To exit back to Privileged mode, use the command. To enter the interface Configuration mode (interface sub-menu), use the command. Use the end command to exit the CLI. NOTE: Debug mode is for Mellanox internal use only Exec Mode Commands The commands available to the user at any given time depend on the current mode. Entering a question mark (?) at the system prompt allows the user to obtain a list of commands available for each command mode. When the user starts a session on the srb-20210g, the initial mode is User mode, often called Exec or Admin mode. Only a limited subset of the commands is available in Exec (Admin) mode. When a CLI session is started, the user enters the Exec mode that allows view-only access of Switch configuration parameters. In order to have access to all commands, the user must enter Privileged Mode. Normally, the user must enter a password to enter Privileged Mode. From the Privileged Mode, the user can enter any Exec command since they are also listed in that mode or enter the Configuration Mode. From this menu, the user can also go back to Exec Mode. The following are the commands available in the Exec Mode. NOTE: The > prompt indicates that you are in Exec Mode. 143

144 Version 2.0 Command Line Interface (CLI) Management 10G>? <command>? Displays the command usage string.? Displays the list of available commands. arp-table show Displays the arp cache of this computer. clock show Displays the system clock. cluster show Shows the cluster configuration. enable Changes to PRIVILEGED mode. end Ends the CLI. exit Exits to previous menu. interface show Shows the interface names. mc-table show Displays the multicast entries of this computer. node-port-info show Displays the guid and mac address of node and ports. ping Sends echo messages. remote show Shows the Remote client configuration. rules show Displays the TFM rules. service show Shows services. snmp show Shows the SNMP properties. static-route show Shows the static route. uptime currently version show Displays : the current time, how long the system has been running, how many users are logged on, and the system load averages. Displays hardware and software version information. The following are the commands available under the Exec Menu: <command>? Displays the command usage string. <command>?? Displays the list of available commands.? arp-table show The ARP Table is a list of all the hosts on both sides of the network, on the InfiniBand side and on the Ethernet side. The InfiniBand ARP Table is used to show the connection between the IP address (virtual), the LID (local ID), and the destination QP. The LID and the destination QP are the physical layer. This connection between the virtual and physical layer is much like the standard correlation between the IP address and the MAC address. 144

145 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 This table enables you to see the traffic transferred to a specific machine. arp-table show 10G-Card> arp-table show Arp Ethernet entries: ================================================================= ========== IP address MAC_ADDR VLAN PORT Device :1B:56:18:80:5F 0 0 if0 Arp Infiniband entries: ================================================================= ========= IP address GUID LID PKEY DQP Device :08:F1:04:03:96:0E:45 0xD 0xFFFF 0x404 if :08:F1:04:00:40:32:01 0x1 0xFFFF 0x2 if0 Note that the first two addresses are on the side of the InfiniBand Network and may not have the same LID for two different machines. The two last addresses are on the side of the external Ethernet Network. clock show Displays the system clock. clock show 10G> clock show Sat Mar 21 12:46:02 UTC 2008 cluster show 10G> cluster show Shows the cluster configuration. cluster show cluster configuration mode : enable loop back protection : full priority : 100 join cluster automatically : yes 145

146 Version 2.0 Command Line Interface (CLI) Management load balancing algorithm : active-active hash table algorithm : xor-infiniband-eth multi-cast ip address : time-to learn (seconds) : 30 active node table pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff if0 srb-20210g x CP1 srb-20210g x net75 srb-20210g Mode: enable (always) Means that the srb-20210g gateway (router) automatically connects to the cluster. Mode options are: auto or disabled. Loopback protection algorithm: Default is full Loop back protection prevents gateways from transferring packets in an infinite loop. This mechanism prevents loops between gateways. When they are in a cluster configuration, they can get into a loop. When used back-to-back, the loopback protection should be none. However, Mellanox does not recommend working in this back-to-back mode. Parameter State (Active-Active State) Priority (Active-Active Priority) Hash Table Algorithm Loopback protection Description srb-20210g Gateways detect each other automatically. Auto (default and only option) this field is grayed out and is automatically configured by the system. It cannot be changed. Once detected automatically the srb-20210g Gateways are sorted according to priority. The highest priority srb-20210g becomes master. Range (0-255) Default: 100 In Active-Active mode, the load balancing is performed according to the hash algorithm (xor IB eth, IB, Eth). XOR InfiniBand Ethernet (Default) mathematical operation performed between IB and Eth IP addresses. IB (InfiniBand) IP address of the InfiniBand Host for any Ethernet address Eth (Ethernet) - IP address of the Ethernet Host for any InfiniBand address Loop back protection prevents loops between gateways. Options: full/none Default: full Note that when used back-to-back, the loopback protection should be set to OFF. However, Mellanox recommends against working in this mode. 146

147 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Parameter Join cluster automatically (Auto join cluster) Description After being automatically detected a srb-20210g can be set to automatically join the cluster. This means that it is now able to transfer packets. Options: Yes/No Default: Yes load balancing algorithm lancing algorithm (Load Balance Algorithm) Active Node Table The traffic can be balanced between the srb-20210g Gateways according to the load balancing algorithm. Options: active-active, master-standby Default: active-active active-active means that all srb-20210g Gateways in the cluster transfer packets. master-standby means that only the master srb-20210g in the cluster transfers packets, the others are in Standby Mode. Shows the Active srb-20210g Gateways that are interconnected and sorted according to priority (note that if two srb-20210g Gateways have the same priority, they are sorted according to serial number). The srb-20210g that has the highest priority becomes master. enable Changes to Privileged mode from the Guest mode. See Section 14.5.for a list of commands available under this menu. enable <admin password> end Ends the CLI. end exit Exits to previous menu. exit interface show Returns information regarding the gateway s IP interfaces. The IP interfaces are set using the CLI config>interface mode, which is described in Section interface show all 147

148 Version 2.0 Command Line Interface (CLI) Management 10G> interface show all entry alias ip mask broadcast vlan pkey(hex) trunk smb-mng no 0x7fff No port 2 if no 0x7fff P x11 P1-2 mc-table show Displays this computer s multicast entries. mc-table show srb-20210g-6060# mc-table show IP address MLID PKEY STATE xC00D 0xFFFF Join xC00A 0xFFFF Join node-port-info show Displays the GUID and MAC addresses of node and ports. node-port-info show 10G> node-port-info show Node Guid : 00:08:f1:00:01:06:03:c0 IB Port Guid : fe80:0000:0000:0000:0008:f100:0106:03c1 ETH Port1 mac-address : 00:08:f1:06:03:c0 ETH Port2 mac-address : 00:08:f1:06:03:c1 Remote show Shows the name of the Remote server and the user name used by the srb-20210g to log in to the server. The password is not displayed for security reasons. remote show 10G> remote show remote configuration remote server: user: swget protocol: SCP 148

149 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 ping Sends echo message to a network host. ping <ip-address> rules show 10G> rules show Displays the rules of the Traffic Flow Management defined in the gateway. rules show --- Total number of entries 3--- entry Source Source-mask Dest Dest-mask Interface Service Routing Statu s Log if0 UDP discard active if0 TCP discard active 2 3 any any any any if0 any tunneling active 1 service show Shows the list of available services. service show 10G> service show entry name protocol Sport Dport type any any any any any 2 MYTCP TCP * 3 MyUDP UDP * 4 V1 ICMP * * any snmp community show Shows the SNMP properties. snmp show 10G> snmp show snmp functionality enable snmp get enable snmp set enable get community public 149

150 Version 2.0 Command Line Interface (CLI) Management set community trap community location description contacts private trap static-route show Shows the static route. static-route show 10G> static-route show entry destination netmask gateway interface smb-mng if0 uptime Displays the current time, how long the system has been running, how many users are currently logged on, and the system load average. uptime 10G> uptime 13:04:14 up 3 days, 13:24, load average: 0.40, 0.40, 0.36 version show Displays software version information. version show 10G> version show IPR software version: 1_0_0-47, Apr :53:51 IPR Board Version: 0x1. IPR FirmWare Version: 0xE. IPR CPLD Version: 0x Privileged Mode Commands The privileged command set includes many of the commands contained in Exec Mode, as well as various configuration commands which include commands that provide access to the 150

151 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 remaining modes. Privileged Mode also includes high-level testing commands that are not available to the user, such as the debug mode (for internal use only). If the system administrator has set a password, you are prompted to enter it before being allowed access to Privileged Mode. The password is not displayed on the screen and is case sensitive. The system administrator uses the admin password configuration command to set the password that restricts access to Privileged Mode. The following shows an example of how to access Privileged Mode: 10G> admin Password: G # 10G> admin Enter password: ******** Type Ctrl-z to exit from enable mode to disable mode. 10G# The following are the commands available in the Privileged Mode: NOTE: The # prompt shows that you are in Privileged Mode. 10G-Card#? <command>? Displays the command's usage string.? Displays the list of available commands. arp-table show Displays the ARP cache of this computer. clock show Displays the system clock. cluster show Shows cluster configuration. config Changes to configuration mode. debug Changes to debug mode. disable Changes to EXEC mode. end Ends the CLI. exit Exits to previous menu. factory-default Sets the IPR back to factory default, and reboots the system. mc-table show Displays the multicast entries of this computer. node-port-info show Displays the guid and mac address of node and ports. password update Changes the admin or enable password. ping Sends echo messages. reload Reboots the system. remote show Shows the remote client configuration. snmp show Shows the SNMP properties. static-route show Shows the static route. statistic Changes to Statistic mode. update software Updates the software version. Add remote server 151

152 Version 2.0 Command Line Interface (CLI) Management uptime currently version show information. name via 'ftp server [server-name].'. Displays : the current time, how long the system has been running, how many users are logged on, and the system load averages. Displays hardware and software version The following details the commands available in the Privileged Menu. <command>? Displays the command usage string. <command>?? Displays the list of available commands.? arp-table show The ARP Table is a list of all the hosts on both sides of the network, on the InfiniBand side and on the Ethernet side. The InfiniBand ARP Table is used to show the connection between the IP address (virtual), the LID (local ID), and the destination QP. The LID and the destination QP are the physical layer. This connection between the virtual and physical layer is much like the standard correlation between the IP address and the MAC address. This table enables you to see the traffic transferred to a specific machine. arp-table show 10G-Card# arp-table show Arp Ethernet entries: ================================================================= ========== IP address MAC_ADDR VLAN PORT Device :1B:56:18:80:5F 0 0 if0 Arp Infiniband entries: ================================================================= ========== IP address GUID LID PKEY DQP Device :08:F1:04:03:96:0E:45 0xD 0xFFFF 0x404 if :08:F1:04:00:40:32:01 0x1 0xFFFF 0x2 if0 152

153 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 clock show Displays the system clock. clock show srb-20210g-61f0# clock show Thu Apr 2 14:48:17 UTC 2009 cluster show Shows the cluster configuration. cluster show srb-20210g-61f0> cluster show cluster configuration mode : auto loop back protection : on priority : 100 join cluster automatically : yes load balancing algorithm : active-active multi-cast ip address : time-to learn (seconds) : 30 active node table pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff if0 srb-20210g Refer to Section 6.6 for details on cluster configuration and a description of the specific fields listed in the cluster menus. config Changes to Configuration mode. Refer to Section 14.6.for a list of commands available in this menu. config debug Changes to Debug mode, which is intended for the use of Mellanox personnel only. debug disable Changes to EXEC mode. Refer to Section 14.2.for a list of commands 153

154 Version 2.0 Command Line Interface (CLI) Management available in this menu. disable end Ends the CLI. end exit Exits to previous menu. exit mc-table show Displays the multicast entries of this computer. mc-table show srb-20210g-6060# mc-table show IP address MLID PKEY STATE xC00D 0xFFFF Join xC00A 0xFFFF Join node-port-info show Displays the node and ports GUID and MAC address. node-port-info show 10G-Card# node-port-info show Node Guid : 00:08:f1:00:01:06:03:c0 IB Port Guid : fe80:0000:0000:0000:0008:f100:0106:03c1 ETH Port1 mac-address : 00:08:f1:06:03:c0 ETH Port2 mac-address : 00:08:f1:06:03:c1 password update Changes the password. password update [admin, enable, guest, root] The following is a step-by-step example of how to update a password. 1. Type password update admin. The following message is displayed: Insert new (8 characters) password: 2. Type in the new password. 154

155 10 GbE Gateway (srb-20210g Module) User Manual Version When prompted, make sure you retype the password exactly as done in the previous step. NOTE: In order to change the privileged mode (enable) password, you must change the password of the admin user. ping Sends echo message to a network host. ping <ip-address> reload Reboots the srb-20210g. reload remote show Shows the name of the Remote server and the user name used by the srb-20210g to log on to the server. The password is not displayed for security reasons. remote show srb-20210g-61f0> remote show remote configuration remote server: ftp.mellanox.com user: release-products protocol: FTP snmp show Returns the SNMP community strings. snmp show srb-20210g-61f0# # snmp show snmp functionality enable snmp get enable snmp set enable get community public set community private trap community trap location lab 155

156 Version 2.0 Command Line Interface (CLI) Management description contacts desc QA static-route show Shows the static route. static-route show static-route show entry destination metmask gateway interface if if smb-mng statistic Changes to Statistic Mode. See Section of commands available in this menu. statistics update software Updates the software version, using the software located in the specified directory of the designated Remote server. NOTE: Remote settings must be configured prior to using the update software command. update software [update-file-dir] Where: update-file-dir is the directory of the image that is on the remote server. srb-20210g-61f0# update software /10 GbE Gateway-install/version-1.3 ***************************************************************** ** Ftp server ip : Ftp download dir : /10 GbE Gateway-install/version-1.3 Package file : 10 GbE Gateway-upgrade.pkg Username : swget Interactive install : n ***************************************************************** ** 156

157 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Do you want to continue the install procedure? [y/n] y Getting Software image from remote server... uptime Displays the current time, how long the system has been running, how many users are currently logged on, and the system load averages.. uptime 10G-Card# uptime 14:45:48 up 2 days, 15:10, load average: 0.61, 0.64, 0.58 version show Displays software version information. version show srb-20210g-61f0> version show IPR software version: 1_0_0-47, Apr :53:51 IPR Board Version: 0x1. IPR FirmWare Version: 0xE. IPR CPLD Version: 0x Statistics Mode The following are the commands available under the Statistic sub-menu. srb-20210g-61f0(statistic)#? <command>? Displays the command usage string.? Displays the list of available commands. end Ends the CLI. exit Exits to previous menu. log show Shows the statistic log. rx-reset rx reset. tx-reset tx reset. log show Parameters: Shows the list of counters that display statistics for each rule. The log data includes the following: counter tx total pack The counter number. Note that a single counter can serve more than one rule at a time. The number of transmit packets (from InfiniBand-to-Ethernet) associated with a TFM rule and has a log counter n. 157

158 Version 2.0 Command Line Interface (CLI) Management entry Source Source-mask Dest Dest-mask Interface Service Routing Status Log any any any any if0 any tunneling active any any if1 any tunneling active 1 Note that in this example, the log number is 1 (n=1) Expected result: srb-20210g-61f0(statistic)# log show counter tx total pack tx total byte rx total pack rx total byte This shows the number of packets sent or received between InfiniBand and Ethernet matching the two TFM rules with log counter #1. tx total byte The number of transmit bytes (from InfiniBand-to-Ethernet) associated with a TFM rule and has a log counter n. In the above example, this shows the number of bytes sent from Infiniband to Ethernet (one way) or sent between InfiniBand and Ethernet and matching the two TFM rules with log counter #1. rx total pack The number of receive packets (from Ethernet to InfiniBand) associated with a TFM rule and has a log counter n. rx total byte The number of receive bytes (from Ethernet to InfiniBand) associated with a TFM rule and has a log counter n. log show srb-20210g-61f0(statistic)# log show counter tx total pack tx total byte rx total pack rx total byte rx-reset Resets the rx counters of the specific log. rx-reset [counter] srb-20210g-61f0(statistic)# log show counter tx total pack tx total byte rx total pack rx total byte

159 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 tx-reset Resets the tx counters of the specific log. tx-reset [counter] srb-20210g-61f0(statistic)# log show counter tx total pack tx total byte rx total pack rx total byte Configuration (Config) Commands Configuration commands apply to features that affect the system as a whole, rather than just one protocol or interface. Config mode is entered through Privileged mode using the config command. In Config mode, you can access the following modes: Cluster, Interface, Route, Service, SNMP, and TFM, as described in Sections The following are the available commands in the Config mode: srb-20210g-61f0(config)#? <command>? Displays the command's usage string.? Displays the list of available commands. cluster Changes to Cluster configuration mode. diagnostics Changes to Diagnostics configuration mode. end Ends the CLI. exit Exits to previous menu. factory-default Sets the 10GbE Gateway back to factory default, and reboots the system. interface Changes to Interface configuration mode. ntp Changes to NTP configuration mode policy Changes to Policy mode. port Changes to Port configuration mode. remote Changes to Remote configuration mode route Changes to Route configuration mode. security Changes to Security configuration mode. snmp Changes to SNMP configuration mode. system-name set Sets the system name of this computer. system-name show Shows the system name of this computer. tfm Changes to TFM configuration mode. The following details the commands and sub-menus available under the Config menu: 159

160 Version 2.0 Command Line Interface (CLI) Management cluster Changes to Cluster Configuration mode. Refer to Section for a list of commands available in this menu. cluster Diagnostics Changes to Diagnostics Configuration mode. Refer to Section for a list of commands available in this menu. diagnostics factory-default Sets the 10 GbE gateway back to factory default, and reboots the system. Refer to Section for a list of commands available in this menu. factory-default interface Changes to Interface Configuration mode. Refer to Section for a list of commands available in this menu. interface ntp Changes to NTP Configuration mode. Refer to Section for a list of commands available in this menu. ntp policy Changes to Policy mode. policy port Changes to Port Configuration mode. Refer to Section for a list of commands available in this menu. port 160

161 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 remote Changes to Remote Configuration mode. Refer to Section for a list of commands available in this menu. remote route Changes to Route Configuration mode. Refer to Section for a list of commands available in this menu. route security Changes to Security Configuration mode. Refer to Section for a list of commands available under this menu. security snmp Changes to SNMP Configuration mode. Refer to Section for a list of commands available in this menu. snmp system-name set Sets the system name of this gateway. This name will be displayed as the CLI prompt. You can enter any combination of letters and digits, up to 20 characters in length. In order to register the name (and prompt) change, you must log out and log in again to the CLI. system-name set <new name> srb-20210g-61f0(config)# system-name set 10G Change will be effective next time CLI is activated Expected results: 10G(config)# system-name show system name is 10G system-name show Shows the system name of this gateway. system-name show 161

162 Version 2.0 Command Line Interface (CLI) Management 10G(config)# system-name show system name is 10G tfm Changes to TFM Configuration mode. Refer to Section for a list of commands available in this menu. tfm Cluster Options A cluster may include several gateways. The srb-20210g supports both High availability (load balancing in active-active fashion) and failover. Note that the active-active feature is currently available in limited functionality. The following are the commands available in the Cluster Mode. srb-20210g-61f0(config-cluster)#? <command>? Displays the command's usage string.? Displays the list of available commands. cluster hash-table-algorithm set Sets the hash table algorithm [xor-infinibandeth,infiniband,ethernet]. cluster join Allows the I/O Gateway to automatically join the cluster. cluster join-now Automatically joins the cluster. cluster load-balance-algorithm set Sets the load balancing algorithm [active-active,master-standby]. cluster loop-prevention set Sets the loop prevention mode. cluster mode set Enables/Disables cluster. cluster box-id set Set the box ID for Loop-Prevention None. cluster priority set Changes I/O Gateway priority. cluster show Shows cluster configuration. cluster id set Changes cluster ID number. end Ends the CLI. exit Exits to previous menu. Usage cluster hash-table-algorithm set cluster hash-table-algorithm set [xor-infinibandeth,infiniband,ethernet] cluster join cluster join [yes,no] cluster join-now cluster join-now cluster load-balance-algorithm set cluster load-balance-algorithm set [active-active,masterstandby] cluster loop-prevention set cluster loop-prevention set [none,full,automatic,manual] 162

163 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 cluster priority set cluster priority set [priority [0-255]] cluster id set cluster id set [ID [0-255]] cluster show cluster show cluster mode set cluster mode set [disable, enable] cluster box-id set cluster box-id set [box ID 0x0 0xffffff] The following details the commands available under the Cluster Menu: cluster hash-table-algorithm set Sets the cluster hash-table-algorithm set. In active-active mode, the load balancing is performed according to the Hash Algorithm (xor IB eth, IB, Eth). xor IB eth (Default) mathematical operation performed between InfiniBand and Ethernet IP addresses. IB IP address of the InfiniBand Host for any Ethernet address Ethernet IP address of the Ethernet Host for any InfiniBand address. cluster hash-table-algorithm set [xor-infiniband-eth,infiniband,ethernet] srb-20210g-61f0(config-cluster)# cluster hash-table-algorithm set infiniband This change will take effect after reboot. Expected results: srb-20210g-61f0(config-cluster)# cluster show cluster configuration mode : enable loop prevention : full priority : 100 join cluster automatically : yes load balancing algorithm : active-active hash table algorithm : infiniband multicast ip address : cluster id : 1 active node table pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff 0 0x x if0 srb-20210g-61f cluster join Allows the gateway to automatically join the cluster in active-active mode. In manual mode, the gateway will not join the cluster until the command is issued. 163

164 Version 2.0 Command Line Interface (CLI) Management cluster join [yes,no] srb-20210g-61f0(config-cluster)# cluster join yes Expected results: srb-20210g-61f0(config-cluster)# cluster show cluster configuration mode : enable loop prevention : full priority : 100 join cluster automatically : yes load balancing algorithm : active-active hash table algorithm : xor-infiniband-eth multicast ip address : cluster id : 1 active node table pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff 0 0x x if0 srb-20210g-61f x x x CP1 srb-20210g-61f x x x net75 srb-20210g-61f x x x CP2 srb-20210g-61f cluster join-now Allows the user to manually add the gateway to the cluster (if the gateway is in manual join mode). cluster join-now srb-20210g-61f0(config-cluster)# cluster join-now cluster load-balance-algorithm set Sets the load balancing algorithm [active-active, master-standby] The traffic can be balanced between the srb-20210g according to the load balancing algorithm. Options: active-active, master-standby Default: active-active active-active means that all srb-20210g in the cluster transfer packets master-standby means that only the master srb-20210g in the cluster transfers packets, the others are in Standby Mode. cluster load-balance-algorithm set [active-active, master-standby] srb-20210g-61f0(config-cluster)# cluster load-balance-algorithm set 164

165 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 active-active Expected results: srb-20210g-61f0(config-cluster)# cluster show cluster configuration mode : enable loop prevention : full priority : 100 join cluster automatically : yes load balancing algorithm : active-active hash table algorithm : infiniband multicast ip address : cluster id : 1 active node table pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff 0 0x x if0 srb-20210g-61f cluster loop-prevention set This feature prevents loops between multiple gateways in a cluster setup. NOTE: When gateways are connected in parallel, this feature should never be set to None. Default value is: Full cluster loop-prevention set [none, full, automatic, manual] srb-20210g-61f0(config-cluster)# cluster loop-prevention set automatic Expected results: srb-20210g-61f0(config-cluster)# cluster show cluster configuration mode : enable Loop prevention : automatic priority : 100 join cluster automatically : yes load balancing algorithm : active-active hash table algorithm : xor-infiniband-eth multicast ip address : cluster id : 1 165

166 Version 2.0 Command Line Interface (CLI) Management active node table pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff 0 0x x if0 srb-20210g-61f cluster mode set Enables/disables cluster. cluster mode set [disable, auto] srb-20210g-61f0(config-cluster)# cluster-mode set enable Expected results: srb-20210g-61f0(config-cluster)# cluster show cluster configuration mode : enable loop prevention : full priority : 100 join cluster automatically : yes load balancing algorithm : active-active hash table algorithm : xor-infiniband-eth multicast ip address : cluster id : 1 cluster priority set Changes the gateway active-active priority. cluster priority set [priority [0-255]] srb-20210g-61f0(config-cluster)# cluster priority set 1 Expected results: srb-20210g-61f0(config-cluster)# cluster show cluster configuration mode : enable Loop prevention : full priority : 1 join cluster automatically : yes load balancing algorithm : active-active hash table algorithm : xor-infiniband-eth multicast ip address : cluster id : 1 active node table 166

167 10 GbE Gateway (srb-20210g Module) User Manual Version pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff 0 0x x if0 srb-20210g-61f cluster id set Changes the gateway cluster id (For Loop Prevention Manual mode). cluster id set [priority [1-255]] srb-20210g-61f0(config-cluster)# cluster id set 1 Expected results: srb-20210g-61f0(config-cluster)# cluster show cluster configuration mode : enable Loop prevention : full priority : 1 join cluster automatically : yes load balancing algorithm : active-active hash table algorithm : xor-infiniband-eth multicast ip address : cluster id : 1 active node table pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff 0 0x x if0 srb-20210g-61f cluster show Shows the cluster configuration. cluster show 10G(config-cluster)# cluster show cluster configuration mode : enable loop prevention : automatic priority : 100 join cluster automatically : yes load balancing algorithm : active-active hash table algorithm : xor-infiniband-eth multicast ip address : cluster id : 1 167

168 Version 2.0 Command Line Interface (CLI) Management active node table pkey vlan master-s/n router-s/n router-prio if-name router name advertised ip 0x7fff 0 0x x if0 35_ipr x x x _ipr Diagnostics The commands available under the Diagnostics menu are listed below. srb-20210g-61f0(config-diagnostics)#? <command>? Displays the command usage string.? Displays the list of available commands. end Ends the CLI. exit Exits to previous menu. system-utilization show Displays the system memory and CPU utilization. system-utilization show Displays the system memory and CPU utilization. system-utilization show srb-20210g-61f0(config-diagnostics)# system-utilization show Total memory : kbytes. Free memory : kbytes. Memory utilization : 24.61%. CPU idle : 99% Interface Options The Interface menu enables you to configure interfaces and set the LAG Mode. A list of the commands available under the Interface Mode is listed below. srb-20210g-61f0(config)# interface srb-20210g-61f0(config-if)#? <command>? Displays the command usage string.? Display the list of available commands. end Ends the CLI. exit Exits to previous menu. interface add Interface add. interface remove Interface remove. interface show Displays the details of all interfaces, or an interface. interface vlan set Updates the VLAN of an interface. ip-address set Sets the interface IP address. lag-mode set Sets the LAG mode. lag-mode show Shows the LAG mode. 168

169 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Usage srb-20210g-61f0(config-if)# interface add interface add [alias] [tag - [1-4094]] [P1,P2,P1-2][pKey 0x0001-0x7fff] [Ip] [Mask] interface remove interface remove [alias all] interface show interface show [all,interface] interface vlan set interface vlan set [alias] [tag - no,[0-4094]] ip-address set ip-address set [alias] [ip-address] [netmask] [optional broadcast] lag-mode set lag-mode set [full,half,separate] lag-mode show lag-mode show The following details the commands available in Interface Mode: interface add Configure a gateway IP interface. The user can add up to two interfaces. The interface add command depends on the LAG Mode. LAG Mode is set using the LAG-mode set command. By default, LAG Mode is set to Full LAG: Full LAG mode: adds an interface with a trunk P1-2 Separate LAG mode: adds an interface with ports P1,P2 For more details on Link Aggregation, refer to Section 6.2. interface add [alias] [tag - [1-4094]] [P1,P2,P1-2][pKey 0x0001-0x7fff] [Ip] [Mask] srb-20210g-61f0(config-if)# interface add V5 108 P1-2 0x This example adds an IP interface, named V5, on VLAN 108, trunk P1-2, to destination InfiniBand Partition 0x8057. The IP address of the interface is and the subnet mask is Note that the default pkey is The pkey value should be given in hexadecimal format. The user must first hand out this pkey to the srb-20210g from the Switch. interface remove Removes one or all interfaces. interface remove [alias all] srb-20210g-61f0(config-if)# interface remove V5 169

170 Version 2.0 Command Line Interface (CLI) Management interface show Displays a single or a list of interfaces defined in the gateway and the following information on each interface: Alias User-assigned interface name IP Address mask broadcast VLAN pkey Interface IP address Interface subnet mask Interface broadcast IP VLAN assigned to the interface (on the Ethernet side) Partition key (on the InfiniBand side) trunk Ports trunk (full LAG [1-2], separate LAG [1], [2]) interface show [all,interface] srb-20210g-61f0(config-if)# interface show all entry alias ip mask broadcast vlan pkey(hex) trunk smb-mng no 0x7fff No port 2 if no 0x7fff P1-42 interface vlan set Updates the VLAN of an interface. interface vlan set [alias] [tag - no,[0-4094]] srb-20210g-61f0(config-if)# interface vlan set Net 101 Expected result: srb-20210g-61f0(config-if)# interface show all entry alias ip mask broadcast vlan pkey(hex) trunk smb-mng no 0x7fff No port 2 if no 0x7fff P1-2 3 Net x7fabc P1-2 ip-address set Sets the alias (name), IP address and subnet mask of the interface. ip-address set [alias] [ip-address] [netmask] [optional broadcast] srb-20210g-61f0(config-if)#ip-address set if

171 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 lag-mode set Changes the Link Aggregation Mode of the gateway. By default, the gateway is set to Full LAG (both ports are aggregated). In Separate Mode, there is no link aggregation. lag-mode set [full, separate] srb-20210g-61f0(config-if)# lag-mode set full lag-mode show Displays the Link Aggregation Mode settings of the gateway. lag-mode show srb-20210g-61f0(config-if)# lag-mode show LAG mode is: full LAG NTP Options The NTP menu allows you to configure NTP (Network Time Protocol). The NTP menu allows srb-20210g clock synchronization with a managing Switch. The following commands are available under the NTP menu. 10G-Card(config-ntp)#? <command>? Displays the command's usage string.? Displays the list of available commands. clock show Displays the system clock. end Ends the CLI. exit Exits to previous menu. ntp show Shows the NTP attributes time-zone list show Shows a list of time zones time-zone set Sets the switch time zone The NTP syntax usage is as follows: time-zone set [location number - refer to time zone list] The following details the commands available under the NTP menu. clock show Displays the srb-20210g system clock. clock show srb-20210g-61f0(config-ntp)# clock show Tue Mar 25 16:08:25 UTC

172 Version 2.0 Command Line Interface (CLI) Management ntp show Shows the NTP attributes. ntp show 10G-Card(config-ntp)# ntp show Status: enable Server IP Address: Server Port: 123 Client Polling Interval: 900 seconds Current Time Zone: UTC time-zone list show Shows a list of time zones. time-zone list show 10G-Card(config-ntp)# time-zone list show 1) Jamaica 2) Libya 3) MST7MDT 4) Chile/EasterIsland 5) Chile/Continental 6) Turkey 7) Portugal 8) Pacific/Guadalcanal 9) Pacific/Johnston 10) Pacific/Fakaofo 11) Pacific/Apia 12) Pacific/Tarawa 13) Pacific/Gambier 14) Pacific/Nauru 15) Pacific/Niue 16) Pacific/Honolulu 17) Pacific/Norfolk 18) Pacific/Marquesas 19) Pacific/Palau 20) Pacific/Efate 21) Pacific/Kwajalein 22) Pacific/Saipan 23) Pacific/Samoa 24) Pacific/Chatham 25) Pacific/Yap 26) Pacific/Tongatapu 27) Pacific/Kosrae 28) Pacific/Port_Moresby 29) Pacific/Ponape 30) Pacific/Pitcairn 31) Pacific/Wallis 172

173 10 GbE Gateway (srb-20210g Module) User Manual Version ) Pacific/Wake 33) Pacific/Galapagos 34) Pacific/Kiritimati 35) Pacific/Enderbury 36) Pacific/Pago_Pago 37) Pacific/Funafuti 38) Pacific/Fiji 39) Pacific/Midway 40) Pacific/Rarotonga 41) Pacific/Tahiti 42) Pacific/Easter 43) Pacific/Noumea 44) Pacific/Guam 45) Pacific/Auckland 46) Pacific/Majuro 47) Pacific/Truk 48) Canada/Eastern 49) Canada/Pacific 50) Canada/East-Saskatchewan 51) Canada/Mountain 52) Canada/Saskatchewan 53) Canada/Atlantic 54) Canada/Newfoundland 55) Canada/Central 56) Canada/Yukon 57) Asia/Seoul 58) Asia/Kuala_Lumpur time-zone set Sets the switch time zone time-zone set [location number - refer to time zone list] 10G-Card(config-ntp)# time-zone set 55 The change will be applied after next reboot Port Options The Port menu allows you to set the port values and view their status. It also enables you to view and reset port statistics. The commands available under the Port menu are listed below. srb-20210g-61f0(config)# port srb-20210g-61f0(config-port)#? <command>? Displays the command usage string.? Display the list of available commands. end Ends the CLI. exit Exits to previous menu. port statistics reset Resets port statistics. port statistics show Shows the statistics of all ports. 173

174 Version 2.0 Command Line Interface (CLI) Management Displays the list of available commands. end Closes current Cli session. exit Exits to previous menu. port ethernet set mtu Updates the mtu of an ethernet port. port ethernet set status Enables/disables the port. port ethernet show Shows the status of all ethernet ports. port ib set mtu Updates the mtu of an infiniband port. port ib show Shows the status of inifiniband port. port statistics reset Resets port statistics. port statistics show Shows the statistics of all the ethernet ports. Usage port ethernet set mtu port ethernet set mtu [1..2] [1500,9216] port ethernet set status port ethernet set status [1..2] [disable,enable] port ethernet show port ethernet show port ib set mtu port ib set mtu [1500,2048] port ib show port ib show port statistics reset port statistics reset [port [1..2]] port statistics show port statistics show port ethernet show port ethernet show The following details the commands available under the Ports menu: port set port ethernet set mtu Set the Ethernet port s MTU port ethernet set mtu [1..2] [disable,enable] [1500,9216] srb-20210g-61f0(config-port)# port ethernet set 1 disable mtu srb-20210g-61f0(config-port)# port ethernet set 1 mtu 2 enable 9198 Expected results: srb-20210g-61f0(config-port)# port ethernet show port type status mtu speed(gbps) enable/disable copper active enable 2 copper active enable port ethernet set status Set the Ethernet port s status port ethernet set status [1..2] [disable,enable] srb-20210g-61f0(config-port)# port ethernet set status 1 disable srb-20210g-61f0(config-port)# port ethernet set status 1 enable 174

175 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Expected results: srb-20210g-61f0(config-port)# port ethernet show port type status mtu speed(gbps) enable/disable copper active enable 2 copper disconnected enable port ethernet show port ethernet show srb-20210g-61f0(config-port)# port ethernet show port type status mtu speed(kbps) enable/disable copper disconnected enable 2 copper disconnected enable port ib set mtu port ib set mtu port ib set mtu [1500,2048] srb-20210g-61f0(config-port)# port ib set mtu 2048 Expected results: srb-20210g-61f0(config-port)# port ib show port status mtu speed(gbps) active port ib show port ib show srb-20210g-61f0(config-port)# port ib show port status mtu speed(gbps) active port statistics reset Resets the statistics of the specific port to zero. The whole column for this port is reset to zero. 175

176 Version 2.0 Command Line Interface (CLI) Management port statistics reset [port [1..2]] srb-20210g-61f0(config-port)# port statistics reset 1 port statistics show provides detailed statistics for each port. port statistics show The following table describes the port types listed by the command: Port Type Description total tx Total packets (bytes) sent to Ethernet port number n. vlan tx Total VLAN packets sent to Ethernet port n belonging to a VLAN partition that is not equal to zero. flrag tx Total fragmented packets sent to Ethernet port n. ip-option tx Total packets with an ip-option sent to Ethernet port n. unicast tx Total number of unicast packets sent on Ethernet port n. arp/rarp tx Total number of ARP/RARP packets sent to Ethernet port n. icmp/igmp Total number of ICMP/IGMP packets sent to Ethernet port n. tcp tx Total number of TCP packets sent to Ethernet port n. udp tx Total number of UDP packets sent to Ethernet port n. pause frame tx Total number of pause frames sent to Ethernet port n. total rx Number of packets received from Ethernet port n. vlan rx Total number of VLAN packets received from Ethernet port n. frag rx ip-option rx Total number of fragmented packets received from Ethernet port n. Total number of packets with an ip-option received from Ethernet port n. unicast rx Total number of unicast packets received from Ethernet port n. arp/rarp rx icmp/igmp rx Total number of ARP/RARP packets received from Ethernet port n. Total number of ICMP/IGMP packets received from Ethernet port n. tcp rx Total number of TCP packets received from Ethernet port n. udp rx Total number of UDP packets received from Ethernet port n. pause frame rx Total number of pause frames received from Ethernet port n. crc error rx total error rx cpu process rx Total number of packets received from Ethernet port n, that were dropped due to a CRC error. Total number of packets received from Ethernet port n, that were dropped due to an error. Total number of packets received from Ethernet port n, that were processed by the CPU. 176

177 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 srb-20210g-61f0(config-port)# port statistics show Type: port1 port total tx: vlan tx: frag tx: ip-option tx: unicast tx: arp/rarp tx: icmp/igmp tx: tcp tx: udp tx: total rx: vlan rx: frag rx: ip-option rx: unicast rx: arp/rarp rx: icmp/igmp rx: tcp rx: udp rx: Remote Options The Remote menu allows you to set FTP and SCP protocols to transfer files to and from remote hosts. You can also use this to backup/export the repository to the Remote Server or saves a back to flash. In addition, you can upload the logs for support purposes. IMPORTANT: When exporting a file, make sure to rename a previously exported file on the Remote server in order to avoid overwriting. Make sure you have writing permissions on the Remote server path. Configuration may vary according to the operating system (e.g., Linux, Windows) on which the Remote server and its setup are installed. The following are the commands available under the Remote menu. remote)#? <command>? Displays the command usage string.? Displays the list of available commands. copy flash: running-config Restores the configuration saved from a file in the local repository. The new values will be applied on reboot. copy remote: running-config Downloads the configuration from the remote server using previously defined server name, user name, and password. 177

178 Version 2.0 Command Line Interface (CLI) Management The remote path must contain the exported file name. The new values will be applied on reboot. copy running-config flash: Saves the configuration file in the local repository. In case of empty name the file name is evaluated by default. copy running-config remote: Uploads the configuration to the remote server using previously defined server name, user name, and password. The remote path can contain the imported file name. If the file name is not indicated, it will be created using a default name. Any characteristic after the last slash in the path will be used as your own defined file name. end Ends the CLI. exit Exits to previous menu. exportlog Uploads the Log to remote FTP server, using previously defined server name, user name, and password. flash delete Deletes specific file from flash. flash show Shows the configuration files in flash. password Sets the user password to access remote FTP server. protocol set Sets the protocol to access remote server. remote show Shows the remote client configuration. server Sets the remote FTP server name/ip-address. username Sets the user name to access remote FTP server. Usage Remote syntax usage is as follows: copy flash: running-config copy flash: [file-name] running-config. copy remote: running-config copy remote: [remote-path] running-config. copy running-config flash: copy running-config flash: [file-name]. copy running-config remote: copy running-config remote: [remote-path] exportlog exportlog [remote path]. flash delete flash delete [file-name]. flash show flash show. Password password protocol set protocol set [FTP SCP] remote show remote show server server [ip-address] username username [username] The following details the commands available under the Remote menu. 178

179 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 copy flash: running-config Restores the backup configuration. This command retrieves the repository configuration from a file stored in the Flash, restores it and saves it in the Switch repository (database). The new values will be applied on reboot. copy flash: [file-name] running-config This example shows how you restoring from the local disk a repository by the name JonhnDoe by copying it to a backup file and then using it for restoring the configuration. srb-20210g-61f0(config-remote)# copy flash: JohnDoe running-config New Configuration will take place after Reboot copy remote: running-config Downloads the configuration from the remote server using the previously defined server name, user name, and password. The remote path must contain the exported file name. The new values will be applied on reboot. copy remote: [file-name] running-config srb-20210g-61f0(config-remote)# copy remote: /home/avia/repository_srb-10g_15_34_ _ip_ n ew running-config New Configuration will take place after Reboot copy running-config flash: Example 1: Backs up the configuration into a file in the local repository. Where the name parameter is empty, the file name is set by default. Example of default format: repository_srb-20210g-70b0_6_47_ _ip_ new) You can back up to two files. Backing up additional files will prompt a message that you are over the limit. copy running-config flash: [file-name] In this case, we are running this command without specifying the backup file name. The system will be created using a default backup file name with the following format: repository_srb-20210g-70b0_6_47_ _ip_ new. srb-20210g-61f0(config-remote)# copy running-config flash: Successed to create the file: repository_srb-20210g-70b0_6_47_ _ip_ new. 179

180 Version 2.0 Command Line Interface (CLI) Management Example 2: In this case, we are running this command with the backup file name (JohnDoe). srb-20210g-61f0(config-remote)# copy running-config flash: JohnDoe Successed to create the file: JohnDoe. Example 3: If you have already two repository backups, you will get the following message if you want to create another one. srb-20210g-61f0(config-remote)# copy running-config flash: Failed to create the file: Cannot add another file (maximum: 2 files). copy running-config remote: Example 1: Uploads the configuration to the remote server using the previously defined server name, user name, and password. The remote path can contain the imported file name. If the file name is not indicated, it will be created using a default name. Any characteristic after the last slash in the path will be used as a user-defined file name. copy running-config remote: [remote-path] In this example, the system will upload the configuration to the remote server and will name the file user1. The file will be located under the folder called home. srb-20210g-61f0(config-remote)# copy running-config remote: /home/user1 Expected Result: srb-20210g-61f0(config-remote)# copy running-config remote: /home/user1 Writing the DB to a repository file Exporting the repository file to the remote destination... Succeeded to create export file: user1. Example 2: In this example, the system will upload the configuration to the remote server. It will get a default name and will be stored under the folder called /home/user1. srb-20210g-61f0(config-remote)# copy running-config remote: /home/user1/ Expected Result: srb-20210g-61f0(config-remote)# copy running-config remote: /home/user1/ Writing the DB to a repository file Exporting the repository file to the remote destination... Succeeded to create export file: repository_10 GbE 180

181 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 Gateway_12_11_ _IP_ new. exportlog Uploads relevant logs and information to the remote server, using the previously defined server name, user name, and password. exportlogs [remote path] srb-20210g-61f0(config-remote)# exportlogs /pub/ flash delete Deletes the specific file from flash. flash delete [file-name] srb-20210g-61f0(config-remote)# flash delete JohnDoe Expected result: srb-20210g-61f0(config-remote)# flash show Name Modification_date repository_10g_7_43_ _ip_ new Mon Jan 21 07:43: flash show Shows the configuration files in flash. flash show srb-20210g-61f0(config-remote)# flash show Name Modification_date JohnDoe Mon Jan 21 06:57: repository_10g_7_43_ _ip_ new Mon Jan 21 07:43: Note that the first file name is user defined. In this example, it is called JohnDoe. The file name in the second example is set by default by the system. password Sets the user password to access the remote server. password 181

182 Version 2.0 Command Line Interface (CLI) Management srb-20210g-61f0(config-remote)# password password: P <- will be displayed as ***** protocol set Sets the protocol to access the remote server. protocol set [FTP SCP] srb-20210g-61f0(config-remote)# protocol set SCP remote show Shows the protocol used to access the remote server. remote show srb-20210g-61f0(config-remote)# remote show remote configuration remote server: username: johnb protocol: FTP server Sets the remote server ip-address. server [ip-address] srb-20210g-61f0(config-remote)# server username Sets the user name to access the remote server (either through FTP or SCP defined by the user to access the remote server). username [username] srb-20210g-61f0(config-remote)# username Johnb 182

183 10 GbE Gateway (srb-20210g Module) User Manual Version Route Options The Route menu allows you to set the default gateway and the static route. A list of the commands available in the Route mode are listed below. srb-20210g-61f0(config-route)#? <command>? Displays the command usage string.? Display the list of available commands. default-gw set Sets a default-gw. end Ends the CLI. exit Exits to previous menu. static-route remove Removes an entry in the static route table. static-route set Sets an entry in the static route table. static-route show Shows the static route. Usage default-gw set default-gw set [row [1..10]][Gateway][Iface] static-route remove static-route remove [row [1..10]] static-route set static-route set [row [1..10]] [subnetdestination][netmask][gateway][iface] static-route show static-route show The following details the commands available in Route Mode: default-gw set Sets the default gateway IP address for the defined interface. default-gw set [row [1..10]][Gateway][Iface] Examples: The following example shows how to configure the default gateway of the srb-20210g. srb-20210g-61f0(config-route)# default-gw set 2 smb-mng NOTE: smb-mng is the srb-20210g managing interface though which the Switch accesses the srb-20210g. This is used to manage the srb-20210g from the Switch. The default gateway is defined on the srb-20210g smb-mng interface through which all packets are transferred to/from the managing interface of the Switch smb-mng. static-route remove Removes the static route. static-route remove [row [1..10]] srb-20210g-61f0(config-route)# static-route remove 2 183

184 Version 2.0 Command Line Interface (CLI) Management Expected results: srb-20210g-61f0(config-route)# static-route show entry Destination NetMask Gateway interface smb-mng smb-mng static-route set Sets the static route. static-route set [row [1..10]] [Destination][netmask] [Gateway][Iface] where Iface is the interface name. srb-20210g-61f0(config-route)# static-route set smb-mng static-route show Shows all static routes. static-route show srb-20210g-61f0(config-route)# static-route show entry Destination NetMask Gateway interface smb-mng smb-mng smb-mng smb-mng Security Options The commands available under the Security menu are listed below. srb-20210g-61f0(config-security)# srb-20210g-61f0(config-security)#? <command>? Displays the command usage string.? Display the list of available commands. console set Changes console enable/disable option. 184

185 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 console show end exit ftpd set ftpd show telnetd set telnetd show Usage ftpd set ftpd show telnetd set telnetd show Shows console enable/disable option. Ends the CLI. Exits to previous menu. Changes ftpd option. Shows ftpd option. Enables/disables the telnetd option. Shows the telnetd option status. ftpd set [disable, enable] ftpd show telnetd set [disable, enable] telnetd show The following details the commands available in Security mode: console set Changes the status of the console. console set [disable, enable] srb-20210g-61f0(config-security)# console set enable Change will be effective only after next reboot console show Shows the console enable/disable option. console show srb-20210g-61f0(config-security)# console show consoled enable ftpd set Sets the ftpd status. ftpd set [disable, enable] Switch(config-security)# ftpd set enable Change will be effective only after next reboot ftpd show Shows the ftpd status. ftpd show 185

186 Version 2.0 Command Line Interface (CLI) Management Switch(config-security)# ftpd show ftpd enable telnetd set Provides the option to disable Telnet for security purposes. telnetd set [disable, enable] srb-20210g-61f0(config-security)# telnetd set enable Change will be effective only after next reboot telnetd set Show the telnetd status telnetd show srb-20210g-61f0(config-security)# telnetd show telnetd enable SNMP Options On the SNMP menu, you configure the parameters of the SNMP interface. Using the SNMP interface, the gateway can be managed by third party managers. Refer to Section 5.1 for further information and for settings information via the GUI. A list of the commands available and their descriptions in the SNMP mode are listed below. srb-20210g-61f0(config-snmp)#? <command>? Displays the command usage string.? Display the list of available commands. end Ends the CLI. exit Exits to previous menu. snmp cluster-monitor set Sets the cluster monitor string. snmp cluster-monitor show Shows the cluster monitor string. snmp community get Sets the SNMP community get. snmp community set Sets the SNMP community set. snmp community trap Sets the SNMP community trap. snmp contacts set Sets the SNMP contacts. snmp contacts show Shows the SNMP agent contacts. snmp description set Sets the SNMP description. snmp description show Shows the SNMP agent description. snmp get Enables/disables SNMP get commands. snmp location set Sets the SNMP location. snmp location show Shows the location of the SNMP agent. snmp set Enables/disables SNMP set command. 186

187 10 GbE Gateway (srb-20210g Module) User Manual Version 2.0 snmp show snmp trap-port show snmp udp-port show Usage: snmp cluster-monitor set snmp cluster-monitor show snmp community get snmp community set snmp community trap snmp contacts set snmp contacts show snmp description set snmp description show snmp functionality set [enable,disable] snmp get snmp location set snmp location show snmp set snmp show snmp trap-port show snmp udp-port show Shows the SNMP properties. Shows the SNMP agent trap port. Shows the SNMP agent udp port. snmp cluster-monitor set [string] snmp cluster-monitor show snmp community get [string] snmp community set [string] snmp community trap [string] snmp contacts set [contacts] snmp contacts show snmp description set [description] snmp description show snmp functionality set snmp get [enable,disable] snmp location set [location] snmp location show snmp set [enable,disable] snmp show snmp trap-port show snmp udp-port show The following details the commands available in the SNMP menu. snmp cluster-monitor set Example in a browser: Sets the cluster monitor by entering the string of the cluster. snmp cluster-monitor set [string] Support+Cluster&h=$HOST&sh=1&hc=4 In the example below, the host is called hydra3.mellanox.com. 187

188 Version 2.0 Command Line Interface (CLI) Management Figure Ganglia (Cluster Monitor) snmp cluster-monitor show Shows the SNMP cluster monitor string. snmp cluster-monitor show srb-20210g-61f0(config-snmp)# snmp cluster-monitor show cluster monitor is snmp community get Sets the SNMP get community string. snmp community get [string] srb-20210g-61f0(config-snmp)# snmp community get A123 Expected results: srb-20210g-61f0(config-snmp)# snmp community show get community A123 set community private trap community trap 188