Grid Director 4000 Family

Transcription

1 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/

2 NOTE: MELLANOX TECHNOLOGIES, INC. AND ITS AFFILIATES ("MELLANOX") FURNISH THIS DOCUMENT "AS IS," WITHOUT WARRANTY OF ANY KIND. MELLANOX DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT AND THOSE ARISING FROM A COURSE OF PERFORMANCE, A COURSE OF DEALING, OR TRADE USAGE. MELLANOX SHALL NOT BE LIABLE FOR ANY ERROR, OMISSION, DEFECT, DEFICIENCY OR NONCONFORMITY IN THIS DOCUMENT AND DISCLAIMS ALL LIABILITY, INCLUDING LIABILITY FOR INFRINGEMENT OF ANY INTELLECTUAL PROPERTY RIGHTS RELATED TO THE INFORMATION CONTAINED IN THIS DOCUMENT. No license, expressed or implied, to any intellectual property rights is granted under this document. This document, as well as the software described in it, are furnished under a separate license and shall only be used or copied in accordance with the terms of the applicable license. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as any commitment by Mellanox. Except as permitted by the applicable license, no part of this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means without the express written consent of Mellanox. Names and logos identifying products of Mellanox in this document are registered trademarks or trademarks of Mellanox. Voltaire is a registered trademark of Mellanox Technologies, Ltd. All other trademarks mentioned in this document are the property of their respective owners. Copyright 2011 Mellanox Technologies, Inc. All rights reserved. Mellanox Technologies, Inc. 350 Oakmead Parkway Suite 100 Sunnyvale, CA U.S.A. Tel: (408) Fax: (408) Mellanox Technologies Ltd PO Box 586 Hermon Building Yokneam Israel Tel: Fax: Copyright Mellanox Technologies. All rights reserved. Mellanox, BridgeX, ConnectX, CORE-Direct, InfiniBridge, InfiniHost, InfiniScale, PhyX, Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies, Ltd. FabricIT, MLNX-OS and SwitchX are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners. 2 Document Number: DOC A04

3 Contents 3.6 Contents Preface Product Overview Grid Director 4000 Family Overview Common Features Grid Director 4700 Features Grid Director 4036E Features Grid Director 4036/2036 Features Grid Director Switch Management Out-of-Band Ethernet Management In-band InfiniBand Management (in 4036/4036E/2036 only) Management Interfaces Failover Mechanism and Redundancy/High Availability SM (Subnet Manager) Redundancy Management Board (smb) Redundancy (4700/4200) Ethernet Gateway (4036E) High Availability Redundant Power Supplies Redundant Fans Performance Management Congestion Control Management (CCM) Fabric Collective Acceleration (FCA) Command Line Interface (CLI) Setup CLI Connection Options Setting up a Local Terminal Connection Starting a CLI Session Telnet/SSH Client Settings CLI Command Prompt CLI Command Structure CLI Case-Sensitivity CLI Tips Keyboard Shortcuts First-time Switch Configuration Prerequisite Tasks Initial Switch Configuration for Management Configuring the Management Interface IP Address Time Settings

4 3.6 Contents Manually Configuring Time and Date on your Switch Configuring Time and Date via NTP (IPv4 Only) Setting the Time Zone Configuring Passwords Verifying Configuration Built-in Tests Built-in Test Overview Manually Running Built-in Tests Updating, Backing up, and Upgrading the Software and Firmware Overview Prerequisites Main Steps Defining the Remote Download Server Uploading the New Software to the Remote Server Software and Firmware Upgrade Procedure Viewing Firmware and Software Version per Module Basic Device Management via the CLI CLI Management Overview Setting the System Name Manually Obtaining Device Information Obtaining Detailed Information via the Device Manager Device Manager State and smb Redundancy Status (in 4700/4200 only) Resetting the Unit Reloading the Chassis Software Resetting the Whole Chassis (in 4700/4200 only) Factory Defaults Info LED Port Management Cable Management Fabric Management via CLI Subnet Manager Configuring the Subnet Manager Configuring Routing Algorithms Performance Manager Prerequisites Congestion Control Monitor (CCM) Fabric Collective Acceleration (FCA) Fabric Management LED States

5 Contents Utilities Remote Configuration Settings Overview Setting the Remote Configuration Parameters High Availability Settings smb Redundancy (4700/4200 Only) Viewing the smb Redundancy Status Installing a Passive smb Security Settings Grid Director 4000 Series Security Settings Overview SCP Secure Export and Import SCP Settings IP Filtering Enabling/Disabling Non-secure Protocols Disabling Telnet Access Web Server Settings Setting the Web Server Using the Web Server Using the Unified Fabric Manager (UFM) Agent Event and Error Logs Log Types Viewing the Event and Error Logs Error and Event Log Messages Error Log Event Log Viewing the CLI Log Viewing the UmServer Log Exporting and Viewing the Event and Error Log Exporting Logs and Debug Information Exporting Logs Viewing Exported Logs Remote Logging Utility Actions Performed on the Switch Side Actions Performed on the Linux Machine SNMP Management Settings SNMP Management Connections Enabling and Disabling SNMP

6 3.6 Contents 12.3 Defining SNMP Communities (SNMPv2c/1) Setting SNMP System Parameters Viewing the SNMP Information SNMP Managers List (SNMP v2c/1 only) CLI Usage Modes and Menus Using the CLI via the Active smb (in 4700/4200 only) Working in Guest Mode Working in Admin Mode Hardware Menu Health Monitor Menu Logs Menu Utilities Menu Working in Configuration (config) Mode Accessing config Mode CCM Configuration Menu Diagnostics Menu FCA Configuration Menu Interface Configuration Menu Names Configuration Menu NTP Configuration Menu PM Configuration Menu Policy Configuration Menu Ports Configuration Menu Remote Configuration Menu Security Configuration Menu SM Configuration Menu SNMP Configuration Menu Passive smb Configuration (in 4700/4200 only) Passive smb Overview Passive smb Guest Mode Command Reference Passive smb Admin Mode Command Reference Passive smb Configuration (config) Mode Command Reference List of CLI Commands CLI Command Modes Guest and Admin (Privileged) Commands <command>? ? bit

7 Contents bit-show cable-config show (4200/4700) clock show config (admin menu only) debug (admin menu only) disable (admin menu only) enable (guest menu only) end exit front show info-led show (guest menu only) hardware (admin menu only, 4200\4700 only) health monitor (admin menu only) logs (admin menu only) module-firmware show password update (admin menu only) ping rear show reload (admin menu only) remote show route default-gw show smb state show update firmware (admin menu only) update software (admin menu only) update software (Passive smb admin menu only) version show Hardware Commands (admin menu only, 4200\4700 only) Health Monitor Commands (admin menu only) Logs Commands (admin menu only) Utilities Commands (admin menu only) Configuration (config) Commands ccm diagnostics factory-default fca interface names

8 3.6 Contents ntp pm policy port remote security sm snmp CCM Commands Diagnostics Commands FCA Commands Interface Commands Names Commands NTP Commands Policy Commands Port Commands Remote Commands Security Commands SM Commands PM Commands SNMP Commands E Overview and Initial Configuration Ethernet Gateway (4036E) Overview E Grid Director Bridging E Grid Director 1G / 10G support Grid Director 4036E-LM (Low Memory Version) Traffic Flow Management Overview Getting Started Initial Configuration Handling Different Subnets E Device Information Ethernet Gateway (4036E) Device Manager Obtaining Device Information E Advanced Configuration Ethernet Gateway (4036E) Advanced Configuration Overview Setting Link Aggregation (LAG) Setting Link Aggregation Cluster Configuration

9 Contents Configuring a Cluster Cluster Mode (4036E High Availability) Setting 4036E Gateway Cluster Algorithms Loop Prevention Port MTU Configuration (Jumbo Frames) Setting the MTU MIB Support Setting up Networks using the 4036E Viewing Networks Adding Networks Removing Networks Setting the IP and Broadcast Address (IPv4) Setting the Interface VLAN E Traffic Flow Management Overview of TFM Using the TFM Rule Priority Setting TFM Rules and Services using the CLI Adding a New Service Updating Services Adding TFM Rules E Static Route Configuration Routing Overview Setting the Routing Configuration E Logs and Statistics Viewing Port Counters Statistics Viewing Port Status and Statistics Statistics Logs E Multicast Settings Multicast Overview Configuring Multicast Settings Multicast-Table Session E CLI Commands E Gateway CLI Model Admin Mode Commands <command>? ?

10 3.6 Contents arp-table show box-id show cluster show config debug end exit mc-table show node-port-info show ping static-route show statistic Configuration (Config) Commands cluster Commands interface Commands mc-filter Commands port Commands route Commands tfm Commands Appendix A: MIBs A.1 Overview of the Grid Director MIB A.2 Supported SNMP Private Parameters A.3 volproducts A.4 volremoteaccess A.5 volsystem A.6 sysmoduletable A.7 systrapcontroltable A.8 volibsmaobjects A.9 volibsmanotificationobjects A.10 volsnmptraps Appendix B: Port Counters Fields B.1 Description of PortCounters.csv Fields Appendix C: Supported Traps and Events C.1 About Traps C.2 Trap Structure C.3 Switch Trap Formats

11 Contents 3.6 List of Figures Figure 1: Telnet/SSH Client Settings Example (using Putty) Figure 2: PortCounters.csv File List of Tables Table 1: Management Interface Configuration Table 2: Terminal Emulation Configuration Table 3: CLI Command Prompts Table 4: Basic Commands Table 5: CLI Key Functions Table 6: CCM Parameters Table 7: Error and Event Log Messages Table 8: Additional Error and Event Log Messages Table 9: Common Utility Options Table 10: Device Information Table 11: Cluster Configuration Fields Table 12: Possible Port MTU Configurations Table 13: Service and TFM CLI Rules Commands Table 14: Port Status and Statistics Table 15: Statistics Logs Table 16: Multicast Definitions Table 17: Cluster show Parameter Table 18: Port Type by Command Table 19: volproducts OIDs Table 20: volremoteaccess OIDs Table 21: volsystem OIDs Table 22: sysmoduletable OIDs Table 23: systrapcontroltable OIDs Table 24: volibsmanotificationobjects Trap Headers Table 25: Port Counters Fields Description Table 26: InfiniBand Fabric Device Trap Format (relevant for all QDR platforms) Table 27: InfiniBand Fabric SM Trap Format (relevant only for 2036/4036/4036E)

12 3.6 Contents Preface Audience This manual is intended for system administrators who are familiar with the fundamentals of networking equipment and who are responsible for setting-up, configuring, and monitoring the Grid Directors and network infrastructure. We assume that the system administrators are familiar with the concepts and terminology of InfiniBand, Ethernet, and local area networking. Technical specifications for individual switches within the Grid Director 4000 family are in their respective Installation Manuals. Document Conventions The following lists conventions used in this document. NOTE: Identifies important information that contains helpful suggestions. CAUTION: Alerts you to the risk of personal injury, system damage, or loss of data. WARNING: Warns you that failure to take or avoid a specific action might result in personal injury or a malfunction of the hardware or software. Be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents before you work on any equipment. Related Documentation For additional information, refer to the following documents: Release Notes for the Grid Director TM 4000 Series Software Version 3.5 [DOC-00962] Grid Director 4036/2036/4036E/4700/4200 Getting Started Guide - Mellanox QDR Solution [LIT-00037] Grid Director 4700 Installation Manual [DOC-00726] Grid Director 4036E Installation Manual [DOC-00911] Grid Director 4200 Installation Manual [DOC-00912] Grid Director Installation Manual for 4036 QDR/2036 DDR [DOC-00467] 12

13 Contents 3.6 Unified Fabric Manager (UFM) User Manual for InfiniBand, [DOC-00600] Unified Fabric Manager (UFM) User Manual for Ethernet, [DOC-00947] Regulatory Compliance and Safety Information Reference Guide [DOC-00859] Fabric Collective Acceleration (FCA) User Manual [DOC-00983] Chapter 1 13

14 3.6 Product Overview 1 Product Overview 1.1 Grid Director 4000 Family Overview The Grid Director integrated family of switching hardware and network virtualization software delivers a high-performance, intelligent solution for deploying clusters and grids. The Grid Director 4000 Family includes the following switch models: 4700, 4200, 4036E, and 4036/2036.The Grid Director 4000 Series provides high performance and functionality, as detailed in the following feature lists. All the features described in this document are applicable to all the Grid Directors except where noted Common Features 19" rack mountable chassis Redundant power supplies and fan units Extreme application performance by removing I/O bottlenecks Unlimited scalability across application, database, and storage servers Ideal for scientific, commercial HPC, and enterprise applications Ultra-low latency: between 100 and 300 nanoseconds port-to-port Available bandwidth: up to 51.8 Tbps of non-blocking bandwidth Simple and fast device management Fully managed by Mellanox Unified Fabric Manager (UFM) Support for longer and more varied cable options Fast fabric bring-up and advanced routing algorithms Advanced congestion management Zero down time guaranteed with no single point of failure and real-time fault notifications QSFP ports supporting either copper or optical cables Virtual lanes: 8 data and 1 for management MTU: 4096 Bytes max Grid Director 4700 Features 324 QDR (40Gb/s) ports in a 19U switch 648-port option can be divided between two racks for weight distribution and greater ease of cabling Unique HyperScale architecture allows scaling to thousands of nodes with a single tier of switches (under 400 nanoseconds of latency) 14

15 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Grid Director 4036E Features 1U height, configurable chassis Aggregate data throughput: 2.72 Tb/s (QDR), 1360 Gbps (DDR) or 680Gbps (SDR) 34 QDR (40 Gbps) ports and two 1/10 GbE SFP+ ports in a 1U switch Ultra-low latency: Less than 100 nanoseconds between InfiniBand ports Less than one microsecond between InfiniBand and Ethernet Low latency, hardware-based (cut-thru) bridging between InfiniBand and Ethernet Provides high performance connectivity to Ethernet-based services and resources Consolidates network and I/O infrastructure Plug & Play, standards-based protocol bridging with zero configuration required on the Grid Director 4036E itself or on the servers Embedded subnet manager Transparent mapping between Ethernet VLANs and InfiniBand partitions to ensure continuity of security and service levels Accelerating IP multicast over InfiniBand using hardware-based multicast Flow control support over both InfiniBand and Ethernet links for efficient handling of congestion Layer 2/3/4-based packet filtering and classification Link aggregation (LAG) between Ethernet ports Aggregation of Ethernet ports from multiple gateways for scalability or redundancy purposes Built-in cable detection and optimization to achieve the longest available cable distance and highest performance Built-in high availability (HA) Grid Director 4036/2036 Features Thirty-Six auto-negotiating QSFP InfiniBand ports in a 1U chassis: The 4036 switch is equipped with QDR ports (40 Gbps per port) The 2036 switch is equipped with DDR ports (20 Gbps per port) 1Ux19"x15" non-blocking standalone switch using a single switch chip Ultra low latency under 100 ns (nanoseconds) Available aggregate bandwidth of up to 2.88 Tbps (QDR), 1440 Gbps (DDR), or 720 Gbps (SDR). 15

16 3.6 Product Overview 1.2 Grid Director Switch Management Grid Director switch management models are based on proprietary Grid Interconnect Management Software. This Device Management software simplifies the switch management and proactively maximizes the performance and availability of InfiniBand enabled servers, networks, and storage grid environments Out-of-Band Ethernet Management You can perform out-of-band management for the switch by using Ethernet or RS-232 (console) interfaces. When using out-of-band management, you can connect external user applications directly through a network, through the local Ethernet subnet to manage the switch In-band InfiniBand Management (in 4036/4036E/2036 only) You can manage the switch in-band using the IP over InfiniBand (IPoIB) interface Management Interfaces Grid Director switches have the following management interfaces: Ethernet interface (10/100/1000 Ethernet): provides an interface to the CLI via a Telnet or an SSH session, to a remote server to download new switch software versions and to upload backup files. Serial interface (RS-232): provides an out-of-band interface to the CLI. Table 1: Management Interface Configuration Interface Serial Ethernet Diagnostics/LOCAL Configuration The Serial interface (RS-232) provides an interface to the CLI. The Ethernet interface is part of the smb module. In redundant smb configurations, the IP address of the Ethernet interface is applied to the active smb card. In the event of failure on the active smb, the Management IP address of the Ethernet interface moves to the backup smb. The Local interface is an optional IP address that can be attached to the Ethernet interface of the smb module, in addition to the Ethernet interface IP address, and is intended for debug purposes. Unlike the Ethernet interface, the Local interface is configured per smb module, regardless of the redundancy configuration and regardless of the smb mode as active or standby. The Local interface is attached to a specific smb module. 1.3 Failover Mechanism and Redundancy/High Availability This section describes the failover mechanism and Redundancy/High Availability functionalities of the Grid Director 4000 series. 16

17 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ SM (Subnet Manager) Redundancy The main software running on the Grid Director 4000 series is the embedded Subnet Manager(SM) The embedded SM configures the InfiniBand fabric and enables traffic flow. The embedded SM implements the standard "SMInfo" protocol (InfiniBand Specification) running in an InfiniBand subnet. The "SMInfo" protocol defines failover between subnet managers in a given InfiniBand subnet to keep subnet configuration consistent when the SM may fail. SM priority between switches ranges between 1 and 15. The SM with the higher priority is designated as the Master and is the only SM that actively configures the fabric. You can change the SM priority, thereby changing the Master in the fabric. SM1 is defined with priority 8 and SM2 with priority 9. SM2 becomes the Master SM. You can change thesm priority using the CLI. Changing the priorities can result in failover, according to the priority policies. All other SMs are in standby mode, polling the activity of the Master SM. In case two switches in the same fabric have identical priorities, the switch with the lower GUID becomes Master SM. On Failover, the role of Master is handed to the next prioritized Standby SM by the priority-guid rule and it becomes the current Master. An SM software malfunction causes failover if there is another chassis in the fabric, according to the priority. In addition, in the 4700/4200 the Subnet Manager runs on the Line Boards, and therefore, the failover can be performed between lines Management Board (smb) Redundancy (4700/4200) The Grid Director 4700 and 4200 include two smbs, the Active smb (Left smb), and the Standby or Passive smb (right smb). There are two ways to connect to the switch: Via the Ethernet Management interface (residing on the Active smb): which has a floating IP address that roams between smbs). When connecting, you are directed to the Active smb for switch management. Via the local interface: where each smb has a different local IP address. When connecting, you are directed to the local smb with the specified IP. You can enter each local smb for diagnostic purposes. In case of boot failure of the Active smb, the Passive smb takes over Ethernet Gateway (4036E) High Availability In a cluster where multiple 10 GbE Ethernet Gateway ports are connected in parallel to each other (both on the Ethernet and InfiniBand sides), the modules behave in High Availability and Load Balancing mode. They automatically detect each other and appear in the Cluster Configuration window Active Node data according to priority. 17

18 3.6 Product Overview If two 10 GbE Ethernet Gateway ports have the same priority, they are sorted according to the master bit and then by the serial number. The 10 GbE Ethernet Gateway that has the highest priority becomes the master when there is no other 10 GbE gateway acting as master. If there is already a master in the system, it will stay the master even though its priority is lower. Multiple algorithms are available for distributing traffic between multiple 10 GbE Ethernet Gateways in Cluster Mode. The traffic can be balanced between the 10 GbE Ethernet Gateways according to the load balancing algorithm (options: active-active, master-standby). In active-active mode, the load balancing is performed according to the hash algorithm (options: XOR IB Eth, IB, Eth) Redundant Power Supplies The Grid Director series support redundant Power Supplies that are hardware-related mechanisms, detailed in the relevant Grid Director Installation Manual. Grid Director 4700 supports up to 6 redundant Power Supplies Grid Director 4200 supports up to 4 redundant Power Supplies Grid Director 4036/4036E/2036 supports up to two redundant Power Supplies Redundant Fans Grid Director fans are redundant. If one fails, the remaining fans shift to Turbo. If a fan fails and the system overheats, the system generates an error and event log and a thermal shutdown is performed automatically. Fan mechanisms are hardware-related and are detailed in the relevant Grid Director Installation Manuals. Refer to the Related Documentation section at the beginning of this document for the list of Installation Manuals Performance Management The Performance Monitor (PM) keeps track of counters reported by the InfiniBand ports and uses them to report ports state, derive bandwidth and packet count on each port, and trigger events and alarms according to the user preferences. The Fabric Manager enables fabric connectivity debugging using the built-in PM. The PM generates a periodic port counters report file (in CSV format) that can be loaded to Excel and further analyzed by the user. It also monitors port counters. The PM keeps track of counters reported by the IB ports and uses them to report ports state, derive bandwidth and packet count on each port Congestion Control Management (CCM) The Congestion Control Management class provides mechanisms to configure congestion control parameters in switches and channel adapters that support congestion control. It also provides mechanisms for retrieving information pertaining to congestion events from such devices. 18

19 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Fabric Collective Acceleration (FCA) Mellanox's Fabric Collective Accelerator (FCA) is a unique technology that scales out fabrics and improves performance from an application perspective (scoping beyond a server/network realm). FCA significantly reduces the runtime of collective operations on any fabric and is available as an add-on to Unified Fabric Manager (UFM ). In addition to UFM integration, FCA offloads collective operations from the Message Passing Interface (MPI) process into the CPUs of Mellanox's QDR InfiniBand switches. In the MPI environment, the FCA integrates with the MPI job scheduler, without having to install additional hardware. On the host side, the Mellanox Open MPI Accelerator (OMA) addresses intra-node collective computation. The FCA algorithm uses: Data from the job scheduler UFM to establish a topology map related to a specific job Processing power of Mellanox's switches to offload significant parts of the computation. Using the FCA algorithm with Mellanox switches and UFM ensures a single message per physical wire for any collective function (in contrast to traditional algorithms for collective function handling which potentially transmit hundreds or thousands of messages per wire). This non-blocking collective architecture allows InfiniBand to scale collective communication to thousands of nodes better than any current interconnect (standard or proprietary). FCA is an add-on module that needs to be ordered separately. Chapter 2 19

20 3.6 Command Line Interface (CLI) Setup 2 Command Line Interface (CLI) Setup 2.1 CLI Connection Options A CLI session can be established via a local serial RS-232 connection to the switch, or by establishing a Telnet or an SSH session to the management interface. 2.2 Setting up a Local Terminal Connection This section provides instructions on how to connect to a console and how to configure it. To connect to a local terminal: 1. Connect a PC with a terminal emulation program to the RS-232 console interface of the switch, as described in the relevant Installation Manual. 2. Make sure that the switch terminal emulation program is configured as shown in the following table. Table 2: Terminal Emulation Configuration Setting Terminal Mode Value VT-100 Baud Parity Stop Bits Flow Control No Parity 1 Stop Bit None Make sure that the Minicom Serial Device is configured as follows: /dev/ttys0 or ttys1, ttys2. ttys0 is the name of serial device on the host that is connected to the switch via a Minicom application. The last digit represents the physical position of the serial connector on the host. For example, if the host has only one serial connector, the device name should be /dev/ttys0. If the host has two serial connectors, the device name could either be /dev/ttys0 or /dev/ttys1, and so on. 2.3 Starting a CLI Session This section describes the steps required to start a CLI session. To start a CLI management session via serial connection to the switch: 1. Connect the PC to the switch via its serial port, using the cable supplied with the switch. 2. Start a HyperTerminal client (or compatible) on the computer. 3. Configure the terminal emulation parameters. 4. Change the factory default password for security reasons; first time users only in the following situations: 20

21 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ First log into the system (with default definitions: root, guest, admin) see step 5 below. First login after running the factory-default utility that sets default definitions of the system 5. Type the appropriate user name and password at the login prompt. root This is used to change the root password the first time you log in. The default root password is: br6000. Operations available in this mode may affect normal system operation and should only be performed with Mellanox Support approval. guest 1. Enter the guest mode. The default password for the guest user is voltaire. 2. Enter the admin (Privileged) mode as follows: Type enable and enter the admin password (The default password for the admin user is ) Type admin 3. Enter admin mode. The default password for the admin user is Issue the appropriate CLI commands to complete the required actions. 2.4 Telnet/SSH Client Settings When using a Telnet/SSH client to connect to, run, and configure Grid Director products, make sure to check the Telnet New Line settings to avoid CR/LF problems. Each client has its own location for those settings. 21

22 3.6 Command Line Interface (CLI) Setup For example, when using the Putty application, make sure that the Return key sends Telnet New Line instead of ^M Checkbox is disabled, as shown in the following figure. Figure 1: Telnet/SSH Client Settings Example (using Putty) 2.5 CLI Command Prompt The CLI is password-protected. There are several CLI operation levels, each corresponding to a different level of interface operations. Each command mode requires a different password. The CLI command prompt includes the switch name. Configuration changes are automatically saved as they are made. The > or # sign respectively show if you are in Exec (Admin) or Privileged configuration mode, as shown in the table below. The following are the main command modes: guest mode: for view-only access of switch configuration parameters admin mode: for configuring basic parameters such as date/time and reload (reset) switch, and for performing software and firmware updates The > or # symbol indicates whether you are in guest or admin (privileged) mode (which includes the config and other sub-menus), as shown in the following table. 22

23 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Table 3: CLI Command Prompts CLI Command Mode Command Prompt Access Method Exit Method guest Switch-xxxx> Log in Use the end command. admin Switch-xxxx# From the Guest mode, type enable. An admin user enters directly into this mode. To revert to Guest mode, type disable or exit. To enter config mode, type config. The CLI command prompt includes the switch name (number). The word switch in the prompts shown in this manual refers to the generic Grid Director name. The switch can be any of the following: 4700, 4200, 4036E, 4036, In the 4700 and 4200 switches, the prompt is preceded by the letters L or R. L means that the management connection is via the left-side Management Board (smb-cm) and R means that the connection is via the right-side smb-cm. 2.6 CLI Command Structure All CLI commands follow a simple structure, and are capable of prompting for further information as the user types. The following table lists the very basic CLI commands. Table 4: Basic Commands Command exit end Description Leaves current mode and returns to previous mode. Ends the CLI session.? Displays help text and usage for the Menu or CLI command. <command>? Displays help for the specific command. When typing "?" after a partial or complete command (use a space after the mark), the system provides a list of commands that begin with that string and relevant usage. To view the syntax, enter the whole command (by copying it) or simply by typing the first letter, space, and "?", as shown in the examples below: Example using the whole command: L:Switch# version show? version show version show The following shows the commands displayed by typing in the first word of the command: L:Switch-0006# update? update firmware update firmware [chassis line spine] [module slot number (not for chassis)][update-file-dir (if not present, takes local)]. update software update software [chassis line spine] [module slot number (not for chassis)] [update-file-dir]. The following shows the commands displayed by typing the first letter of the command: 23

24 3.6 Command Line Interface (CLI) Setup Command Description # r? rear show reload remote show route default-gw show rear show reload remote show default-gw show 2.7 CLI Case-Sensitivity All CLI commands are case-sensitive. All commands and keywords must be entered in lower case (without capital letters). User-defined strings can appear in any case (including mixed case). The case for user-defined strings is preserved in the configuration. 2.8 CLI Tips The following are tips for entering commands in the CLI: Automatic command completion: Use the Tab key to complete commands as follows: type in the first one or few letters of the command and then key in Tab; the CLI completes the current word, allowing you to continue entering the command. Use the Question Mark (?) key to list all of the options available at that point in the command line. Commands and keywords can be truncated at any point after they are unique. Configuration changes are automatically saved as they are made. 2.9 Keyboard Shortcuts The CLI supports the use of the following special keys for the described functions: Table 5: CLI Key Functions Key Function? Lists all options available at that point in the command line, along with a brief description of the command. If you enter a question mark (?) after a partial or complete command (press Enter after the mark), the system provides a list of commands that begin with that string and relevant usage. Tab Backspace Enter key Completes a partial command name entry. When you enter a set of characters that match the beginning of a command name and press the Tab key, the system completes the command name. If you enter a set of characters that could indicate more than one command, the system lists all available options. Erases the character to the left of the cursor. At the command line, pressing the Enter key performs the function of processing a command. At the prompt on a terminal screen, pressing the Enter key scrolls down a line. 24

25 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Key Left arrow Right arrow Up arrow or Ctrl-P Down Arrow or Ctrl-N Ctrl-A Ctrl-B Ctrl-D Ctrl-E Ctrl-F Ctrl-K Ctrl-T Ctrl-W Ctrl-Y Esc B Esc C Esc D Esc F Esc L Function Moves the cursor one character to the left. When you enter a command that extends beyond a single line, you can press the Left Arrow key repeatedly to scroll back toward the system prompt and verify the beginning of the command entry. Moves the cursor one character to the right. Recalls commands in the history buffer, beginning with the most recent command. Repeat the key sequence to recall successively older commands. Return to more recent commands in the history buffer after recalling commands with the Up Arrow or Ctrl-P. Repeat the key sequence to recall successively more recent commands. Moves the cursor to the beginning of the line Note: This button combo does not work when the session is established with Minicom. Moves the cursor back one character. Deletes the character at the cursor. Moves the cursor to the end of the command line. Moves the cursor forward one character. Deletes all characters from the cursor to the end of the command line. Transposes the character to the left of the cursor with the character located at the cursor. Deletes the word to the left of the cursor. Recalls the most recent entry in the delete buffer. The delete buffer contains the last ten items deleted or cut. Ctrl-Y can be used in conjunction with Esc Y. Moves the cursor from the middle of the word to the beginning. Capitalizes the character on which the cursor is pointing and moves it to the end of the word. Deletes from the cursor to the end of the word. Moves the cursor forward one word. Changes the word to lowercase at the cursor to the end of the word. Chapter 3 25

26 3.6 First-time Switch Configuration 3 First-time Switch Configuration 3.1 Prerequisite Tasks Before configuring system parameters, make sure you have completed the hardware installation according to the relevant Grid Director Installation Manual. Refer to the Related Documentation (on page 12) section at the beginning of this document for the list of Installation Manuals. 3.2 Initial Switch Configuration for Management The following sections describe the initial system configuration required to start the switch management Configuring the Management Interface IP Address You can manage the initial management system from the Ethernet or the InfiniBand interface. To allow remote management, you must configure the switch with an IP address. You can use IPv6 for chassis management. You can configure the management interface with an IPv6 link-local address. Additionally, you can add a global address to the interface. For 4700/4200 the floating IPv6 global address will be defined on the management interface (eth0). To configure the Management IP address (IPv4 or IPv6): 1. Type guest. The default password is: voltaire. Type enable and enter the admin password (the default password for the admin user is ) or Type admin The default password for the admin user is: Type config. Enter the config mode from the admin mode. 3. Type interface. Enter the interface configuration. 4. Enter the command interface ethernet set to set Ethernet as the Management interface. Enter the command interface infiniband set to set InfiniBand as the Management interface. Optional: Enter the command internal-subnet-prefix set 26

27 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ The internal subnet prefix is only relevant to: 4700, 4200, and 4036E. If your IP address is within the Internal Subnet, you must change the Internal Subnet Prefix. 5. If the management is set to Ethernet, disable DHCP (DHCP is enabled by default). Run: dhcp set disable. For IPv6, run dhcp6 set disable. In InfiniBand mode, there is no need to run this command. 6. Enter the command ip-address show View the IP address configuration. 7. Enter the command ip-address set [ip-address] [netmask] [optional broadcast] to set the management IP address, netmask and broadcast. Optional: Enter the command default-gw set to set the switch Default Gateway. The Default Gateway must be under the same subnet as the management interface. For IPv6, enter ip6-address set [ip6-address/mask]. 8. Type exit to exit the interface mode and return to config mode. 3.3 Time Settings Use the time settings commands to set the switch internal clock manually or to set the remote NTP server Manually Configuring Time and Date on your Switch Use the following sequence to configure the switch time and date parameters. The time and date are displayed on event reports that are time-stamped. To configure time and date on the switch: 1. Enter config mode. 2. Type ntp. Enter the Network Time Protocol (NTP) mode. 3. Type ntp show. Verify that NTP is in disable mode. 4. Enter the command clock set Enter time and date in military format, as shown in the syntax Sun Jan 6 22:00:11 UTC Sun Jan 6 22:00:11 UTC Enter the command clock show to view the system switch clock. 27

28 3.6 First-time Switch Configuration Configuring Time and Date via NTP (IPv4 Only) The Grid Director CLI NTP (Network Time Protocol) menu allows you to configure clock synchronization over the network. This allows networks to synchronize via an NTP server by referring the clocks of the chassis to the NTP server IP. To set NTP parameters Run the following: (config-ntp)# ntp server-ip-address set [ip-address] (config-ntp)# ntp status set enable To view NTP parameters Run the following: (config-ntp)# ntp show Setting the Time Zone You can configure the time zone for the switch according to location. Factory default: Coordinated Universal Time (UTC) To set the time zone: 1. From the CLI config menu, run the following command: time-zone list show 2. From the time zone list, note the desired location number. time-zone list show 3. Set the time zone as follows: time-zone set 3.4 Configuring Passwords Use the following procedure for configuring passwords for guest and admin mode access to the RS-232 console interface and to the management interface which is used for establishing a CLI session via Telnet or SSH. The factory-default passwords are: guest: voltaire admin: root: br6000 (When you first login, you are prompted to change the root password.) To configure passwords through the CLI: 1. Type guest 4. Enter guest mode. The default password for the guest user is voltaire. 5. Enter admin mode: Type enable and enter the admin password. The default password for the admin user is OR 28

29 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Type admin Enter the admin mode directly. The default password for the admin user is Type password update [admin, guest, root)to set the new passwords. 3. Type exit to exit the admin mode. 3.5 Verifying Configuration Verify the system parameters. Note that the ip-address show command is available under the config/interface menu. To verify the configuration: From the admin mode enter the following commands: version show Show the version of the current switch software. remote show Shows the remote server configuration. (config-if)ip-address show Show the IP interface address and configuration for both IPv4 and IPv6. (config) clock show Show the system clock. Chapter 4 29

30 3.6 Built-in Tests 4 Built-in Tests 4.1 Built-in Test Overview When you log in as a Guest user, the built-in test (BIT) feature automatically performs several tests before entering the CLI session. You can also run the tests manually from the admin mode, under the Health Monitor sub-menu. You can use the BIT to run one, several, or all scripts. At the end of each test, the BIT status appears: Success: indicated by the display of done. Failure: indicated by an alert message with relevant errors. When a Processes test fails, details of the specific process are shown in the umserver log. Troubleshooting option suggested for relevant cases. There is a 5-second delay before the BIT starts running. You can skip the automatic BIT by pressing the Enter key on the keyboard only if you logged into guest mode at startup. 4.2 Manually Running Built-in Tests When logging in Guest mode, the Built-in-test runs tests periodically in the background and also runs tests before entering the CLI session. From the Guest menu, you can choose the specific tests you want to run. This command provides the ability to run any script that was pre-defined by the CLI interface. You can run the Built-in-test manually from the Health Monitor sub menu. NOTE: This test runs automatically in the background and when logging into the CLI. When running a BIT, the BIT identifier is the number in the first column of the BIT show table (see following command) To execute all the tests, enter: bit all. To execute one or several BITs, type the test identifier separated by a comma (for example: 2, 5, 6). Note that you can run the BIT test in reverse order to, for example: bit 5, 1, 2 or in our example: bit 2, 1. To view the valid test list, type show The following table describes the available tests for BITs and all of the relevant platforms: Test name 4036/ E 4700/4200 Line/Spine Bring up test X X Firmware test X X X X Software test X X X Fan direction X X 30

31 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Test name 4036/ E 4700/4200 Line/Spine Processes state X X X X Tests may be run in the following 3 modes: Periodic Runs per predefined period On system startup - Only runs once On demand Runs by user demand The following lists the available mode for each specific test. Test name Periodic On bring up On demand Bring up test X X X Firmware test X Software test X X Fan direction X X Processes state X X X Periodic mode can be disabled/enabled. It is enabled by default. The default time period is 15 minutes, the minimum time period is 5 minutes. Both values can be modified via the CLI. SNMP: You can run tests to collect information that is gathered by SNMP. Located under: volsystem.volbittests The following table represents the bit tests in sysbitteststable. Index Integer Name- String State Integer Run- Boolean Time- Date Fail message - String To run a test Set the "Run" column value to true. While the test is running the "Run" field will be true. Once the test ends the "State" column will reflect test status (Fail/Pass). If a test failed the "Failed message" will show the reason. The "Time" field will show the last time the test was run. volbittestsperiodicmode Enable/Disable periodic testing volbittestsperiodiccycletime Sets the time period periodic-test cycle-time set Configure how long to wait between periodic tests periodic-test mode set Enable/Disable periodic bit testing periodic-test show Shows the periodic testsall BITs are automatically executed during startup. However, you can also run a BIT manually from the guest menu or from the health monitor menu. To execute one or more BITs (from the Guest Mode or the Health Monitor menu): 1. Run the following command to view a list of BITs with their identifier and description. 31

32 3.6 Built-in Tests (hm)# bit-show 2. To run a test, enter bit followed by the BIT Identifier or all for all tests. To execute one or several BITs, type bit followed by the test identifiers separated by a comma. Test identifiers may also be specified in reverse order. Examples: (hm)# bit 2 (hm)# bit all (hm)# bit 1, 2, 3 (hm # bit 3, 1, 2 Chapter 5 32

33 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Updating, Backing up, and Upgrading the Software and Firmware 5.1 Overview You can use the remote configuration FTP/SCP functionality to: Upgrade software versions Upload log files to the remote server Upload/download the repository file. All operations below can be performed via both FTP and SCP Prerequisites Before performing any switch maintenance tasks, make sure you have configured system parameters, as described in First-time Switch Configuration (on page 26). Note that DHCP is enabled by default. In addition, make sure that your remote server is set as described in the following procedure. Certain configuration tasks, such as identifying a location from which to download software, are optional and may not have been performed during initial configuration. You may perform these tasks at any time using the CLI. 5.2 Main Steps Software and firmware upgrade is performed in three main steps. To update, back up, and/or upgrade the software: 1. Define the remote configuration parameters for backup 2. Upload the new software/firmware to the remote server. 3. Perform the software and firmware upgrade. The following sections provide further instructions for each step. 5.3 Defining the Remote Download Server The new software is located on a remote server. This section shows how to define the remote server to allow the switch to download the new software version or upload the software. Perform the steps in the following table to set the appropriate download location. To configure the remoter server Enter the following commands: 1. admin 33

34 3.6 Updating, Backing up, and Upgrading the Software and Firmware Enter the admin mode. The default password for the admin user is config Enter the config mode (from admin mode). 3. remote Access the Remote Mode (from the config mode) 4. server [ip/ip6-address] Configure the IPv4 or IPv6 address of the Remote server. 5. username [username] Configure the user name for Remote server access and press enter. 6. password Set the user password to access the Remote server. Press Enter and then enter the password. 7. protocol set [FTP SCP] Set the protocol to access remote server. 8. remote show After the update is complete, execute this command to verify the new settings. 5.4 Uploading the New Software to the Remote Server The new software update is available to the user either in media format, for example, CD, or from the Mellanox Support website. Configure the switch IP and the default gateway prior to performing the following steps. If you did not configure a default gateway, the remote server must be located in the same subnet as the switch management port. To upload the new software version to the remote server: 1. Make sure that the remote server is active. 2. Copy the new version files to the Remote directory. 5.5 Software and Firmware Upgrade Procedure After uploading the new software/firmware to the pre-defined remote server, run an update software/firmware command (admin mode) to update the new switch software/firmware. Refer to the latest Grid Director Release Notes for the relevant software and firmware image revisions. To upgrade both the software and firmware 1. Enter the admin mode. 2. Run the following command to start the update software process. This can take several minutes to complete. 34

35 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ update software chassis [update-file-dir] The system reboots automatically. However, if there is a problem in the upgrade process, an error message appears and the system does not reboot. For the 4700/4200 switches, you can upgrade the whole chassis or the individual module. It is recommended to upgrade the whole chassis. The default IO capability in 4036E is 10G. To change the IO capability to 1G, follow the steps under the section: "To upgrade the 4036E to 1G". To upgrade the firmware only 1. Enter the admin mode. 2. Run the following command to start the Update Firmware process. This may take several minutes to complete. update firmware chassis [update-file-dir] The system reboots automatically. However, if there is a problem in the update process, an error message appears and the system does not reboot. For the 4700/4200 switches, you can update the firmware of the whole chassis or of each individual module. It is recommended to update the whole chassis. The system reports the software success or failure in the error log (for debugging purposes). Refer to Event and Error Logs (on page 52) for more information. To upgrade the 4036E to 1G 1. Enter the admin mode. 2. Run the following command to start updating the IO firmware. This may take several minutes to complete. update firmware io 1G [update-file-dir] The system will perform reboot automatically when the upgrade procedure is over. The system also reports the process success or failure in the Error Log (for debugging purposes). Refer to Event and Error Logs (on page 52) for more information. 5.6 Viewing Firmware and Software Version per Module To verify that the latest software and firmware versions are installed for each module: Enter the following command: module-firmware show Chapter 6 35

36 3.6 Basic Device Management via the CLI 6 Basic Device Management via the CLI 6.1 CLI Management Overview All Grid Director 4000 family switches have an embedded application that enable full device and InfiniBand management functionality. It also enables you to perform basic device functions, such as monitor and configure device and port parameters, chassis temperatures, and fan status via the CLI. This section details only the basic management functions. For more detailed information about the CLI management options, refer to the next chapters. 6.2 Setting the System Name Manually This section provides instructions for renaming the switch manually in configuration mode. To rename the switch manually Enter the following command: system-name set NewName The changes are applied after restarting the CLI. To preserve the switch name for future use Enter the following command: unique-default-name set disable 6.3 Obtaining Device Information This section provides information and examples about how to obtain device information in configuration mode. To view device name information Enter the following command: system-name show To view the software version Enter the following command: version show To view the software and firmware versions per module Enter the following command: module-firmware show 6.4 Obtaining Detailed Information via the Device Manager This device manager allows you to obtain detailed information, including temperature, hardware specifics, and device state. 36

37 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ To view detailed information on the modules installed at the front of the chassis Enter the following command: front show To view detailed information of the modules installed at the rear of the chassis (applicable to the 4700/4200 only) Enter the following command: rear show 6.5 Device Manager State and smb Redundancy Status (in 4700/4200 only) You can verify the status of the smb on which the device manager runs (Active, Passive smb or whether a single smb is installed in the chassis). To verify that the Device Manager is running on the Active smb Enter the following commands: smb-state show 6.6 Resetting the Unit Reloading the Chassis Software You can reload the Grid Director software using the CLI by using # reload command. The reload command is required after performing various software operations to confirm configuration changes. To reset the Grid Director software: 1. Enter the following command reload 2. When prompted, type yes to confirm Resetting the Whole Chassis (in 4700/4200 only) You can reset the 4700/4200 using the CLI by using 4700# reset all command to reboot the whole chassis. Important: Running a reset all command (hardware reset) may affect the traffic. To reset the Grid Director chassis 1. Enter the following command reset all 2. When prompted, type yes to confirm. 37

38 3.6 Basic Device Management via the CLI 6.7 Factory Defaults The Grid Director configuration can be reverted to the factory default parameter settings and reloaded to the system. Since the factory default command restores the switch default interfaces IP address, it is important to perform this action via a local terminal to maintain communication with the switch during this process. Running the factory default command erases all the non-default data. To apply factory defaults, run: 1. Enter the following command factory-default 2. When prompted, type yes to confirm. 6.8 Info LED The smb Info LED can be turned off/blink when needed, to help locating the specific physical device at the installation site. You can check the current Info LED status of the managed unit. To view the current Info LED status of the managed device Enter the following command: info-led show The required info-led show command syntax depends on the specific device you are using. Click the command link above for the device-specific syntax options. Refer to the relevant Grid Director Installation manual for the list of device-specific LEDs. To make the Info LED of the switch start blinking Enter the following command: info-led set 6.9 Port Management Port management allows you to set the administrative status of the Grid Director ports. You can disable ports if you want to block them for maintenance or if they are faulty (all ports are enabled by default). To disable or enable a port Enter the following command: port set On the Grid Director 4036E, the Ethernet Gateway ports can be configured as well. To view port configuration Enter the following commands: port state show The port is enabled by default. 38

39 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Cable Management The system is able to automatically detect the available cable information and display it as follows: Cable is not present no cable in installed in this port. Cable is present a cable is installed in this port but no cable information is available. Cable information a cable is installed in this port and the cable information is detailed. Cable error error reading the cable information. For the list of supported cables, refer to the 4036/2036 and 4036E Installation Manuals. To view the cable information, run: cable-config show Port 1: Not present... Port 17: Not present Port 18:Length 1m Vendor Name: WLGORE Code: QSFP Vendor PN: QSN Vendor Rev: A Vendor SN: Port 19:Length 1m Vendor Name: WLGORE Code: QSFP Vendor PN: SCN Vendor Rev: P1 Vendor SN: Port 20:Length 1m Vendor Name: WLGORE Code: QSFP Vendor PN: QSN Vendor Rev: A Vendor SN: Port 21:Length 1m Vendor Name: WLGORE Code: QSFP Vendor PN: QSN Vendor Rev: A Vendor SN: Port 22: Not present... Port 34: Not present Port eth1: Not present Port eth2:length 0m Vendor Name: Methode Elec. Code: Copper 100 Vendor PN: DM7041-R Vendor Rev: D Vendor SN: N/A For the 4700/4200 Only The cable-config show command should present the cable parameters for lines and double-density spines. Command: cable-config show [line,spine] [1..18] L:4700-A368# cable-config show spine 2 Port 1:Length 0m Vendor Name: MolexInc. Code: CXP Vendor PN: Vendor Rev: A Vendor SN: Port 2: Not present Port 3: Not present Port 4: Not present Port 5: Not present Port 6: Not present Port 7:Length 0m Vendor Name: MolexInc. Code: CXP Vendor PN: Vendor Rev: A Vendor SN: Port 8: Not present 39

40 3.6 Basic Device Management via the CLI Port 9: Not present Port 10: Not present Port 11: Not present Port 12: Not present L:4700-A368# cable-config show line 10 Port 1: Not present Port 2: Not present Port 3: Not present Port 4: Not present Port 5: Not present Port 6: Not present Port 7: Not present Port 8: Not present Port 9: Not present Port 10: Not present Port 11: Not present Port 12:Length 3m Vendor Name: WLGORE Code: QSFP Vendor PN: SCN Vendor Rev: P1 Vendor SN: Port 13: Not present Port 14: Not present Port 15: Not present Port 16: Not present Port 17:Length 3m Vendor Name: WLGORE Code: QSFP Vendor PN: SCN Vendor Rev: P1 Vendor SN: Port 18: Not present Chapter 7 40

41 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Fabric Management via CLI 7.1 Subnet Manager This section describes how to configure the Subnet Manager and routing algorithms Configuring the Subnet Manager The InfiniBand Subnet Manager (SM) is a centralized entity running in the switch. The DM discovers and configures all the InfiniBand fabric devices to enable traffic flow between those devices. The SM applies network traffic related configurations such as Quality of Service (QoS), routing, and partitioning of the fabric devices. You can view and configure the Subnet Parameters (SM) via the CLI (config-sm) menu. This section describes how to view and configure the SM parameters. NOTE: QoS can only be configured via the UFM. Refer to the UFM User Manual for further information. Partitioning can be configured by exporting or importing a partitioning configuration file. The embedded SM on the Grid Director can be used to manage fabrics up to 648 nodes. and by default is enabled on the 4036/2036/4036E, and by default disabled on the 4700/4200. For the 4700/4200, you can select active and standby Subnet Managers running on two different Line Boards. To view the Local SM Information Enter the following command: sm-info show Configuring Routing Algorithms This section describes how to configure the supported routing algorithms, including balanced and up/down routing schemes Supported Algorithms The Grid Director supports the following routing algorithms: Balanced-routing Up-down routing Note: Up-down routing is supported with automatic root switch recognition. Providing the root_guid file for the up-down algorithm is currently not supported by CLI. 41

42 3.6 Fabric Management via CLI Min Hop (Balanced Routing) Scheme The min hop routing scheme is the default routing scheme and is applicable to all Fabric configurations. Each InfiniBand switch maintains a simple forwarding database that defines, per destination LID, which physical port should be used for outgoing packets. The forwarding databases are configured by the Subnet Manager when the subnet is initially configured and when there are network changes. When multiple paths exist between a pair of nodes in the subnet, the Subnet Manager identifies multiple minimal paths among those end-ports, selects one path, and programs the fabric accordingly. However, to better utilize network bandwidth, the Subnet Manager balances its selections so that the traffic between all end-ports is balanced over all possible paths in the fabric Up/Down Routing Scheme Up-Down routing is designed for CLOS networks. The algorithm objective is to assure that given an N-stage CLOS network, each lid route between a couple of end-ports passes through the CLOS backbone level (for example, the spine level). The path from the source end-port up to the back-bone level is called up route, and the remaining path from the back-bone level till it gets to the destination end-port is called down route. To ensure optimal performance in 5-stages CLOS network, it is recommended to use the up-down routing The up-down algorithm eliminates the hazard of credit loops for a CLOS network. NOTE: When the route length between the two end ports is less than N, the route should not go through the backbone level. For example, in a 3-stage CLOS network, if both end-ports connect to the same switching chip on the first level of the CLOS, then the route should not traverse the CLOS backbone level Setting the Routing Algorithm To set the routing algorithm Enter the following command: sm-info routing_engine set updn/minhop The sm-info routing_engine set command sets the routing engine SM parameter. and sets the routing algorithm to be implemented by the Subnet Manager. 7.2 Performance Manager The PortCounters.csv file allows is diagnostic summary tool that allows you to check the performance of the Fabric Manager. It includes specific switch and port platform information as well as details on the peer on the other side of the port link. This information enables faster location of the faulty link and faster stabilization of the InfiniBand fabric. Using a browser, you can save a file in CSV format that lists a wide range of statistics for all active nodes and their active ports in the fabric. This includes information such as Node name, PM data, failure reports, and a detailed port counters summary. 42

43 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ The PortCounters.csv file also displays the port s peer node showing to which node and port it is connected. Each active node port information line is presented in a single line per port. The following is an example of an PortCounters.csv file. Figure 2: PortCounters.csv File The PortCounters.csv file can be saved from your browser and then opened as an Excel file Prerequisites Verify that the following values are set in order to enable the download of the portcounters.csv file: # (config-sm)# sm-info mode set enable # (config-pm)# pm mode set enable # (config-security)# web set http To access the file, type the following address in your browser: IP address>/voltaire/portcounters.csv where <switch IP address> is the IPv4 or IPV6 address of your switch. Refer to Security Settings (on page 48) for more information on web server and web server settings. Refer to Ports Counters Fields (on page 208) for a description of the CSV file and the port counters currently available. 7.3 Congestion Control Monitor (CCM) Congestion is a common occurrence within networks. The Grid Director Congestion Control Monitor (CCM) module enables you to monitor congestion. It is also possible to view CCM statistics via the web browser. To understand the congestion spreading problem, and the relationship between a root source and a victim of congestion, refer to Congestion Overview in "InfiniBandTM Architecture Specification Volume 1, Release 1.2.1". CCM enables the network administrator to configure a congestion probe window and to view a congestion statistics. Available parameters include: Table 6: CCM Parameters Parameter marking-rate Description A switch shall set the FECN bit in the BTH of packets that are candidates for marking determined by the marking rate. Marking-rate is the value that provides the mean number of packets, between marking eligible packets with a FECN. If 43

44 3.6 Fabric Management via CLI Parameter mode packet-sizes sampling-frequency sampling-window threshold victim-mask-policy Description marking-rate is set to zero, then all packets are eligible to be marked. Otherwise, if marking rate is "10", the average number of unmarked packets between 2 marked packets will be 10. Determines whether the CCM application is enabled or disabled. Any packet less than this size will not be marked with a FECN. This parameter s units are credits, a value greater than the maximum packet size will result in no packets being marked with a FECN. Sampling-frequency determines the rate at which logs will be taken from the InfiniScale IV chip. Because there is a very limited buffer for the log file that keeps all the congestion hits, as the value of the frequency increases, the statistic file in a highly congested fabric is more accurate. The hits that appear in the statistic file represent the number of hits in the fabric for each flow during a time interval called "window", The higher the value of "sampling-window", the bigger the time interval is. A value that indicates how aggressive congestion marking should be: 0x0: No packet marking 0x1: Loose 0x2:... 0xF: Very aggressive If enabled, the switch shall mark packets that encounter congestion with a FECN, whether they are the source or victim of congestion. 7.4 Fabric Collective Acceleration (FCA) Fabric Collective Acceleration (FCA) is an add-on module that needs to be ordered separately. For further and more detailed information, refer to the FCA User Manual. 7.5 Fabric Management LED States The FM LED shows the status of the Fabric Management. The name of this LED may vary according to the specific Grid Director, as follows: 4200/4700: FM (Fabric) LED 4036E: 4036/2036: SM LED Refer to the relevant Installation Manual for the list of LEDs. 7.6 Utilities The CLI provides a comprehensive list of utilities enabling you to view the all switches, routers, and/or hosts in the fabric and obtain detailed device and port information. The Grid Director utilities provide enhanced diagnostic functionality. Chapter 8 44

45 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Remote Configuration Settings 8.1 Overview You can use the remote configuration FTP/SCP functionality to: Upgrade software versions Upload log files to the remote server Upload/download the repository file. All operations can be performed via both FTP and SCP. 8.2 Setting the Remote Configuration Parameters This section shows how to configure the remote server to allow the switch to back up the configuration. To set Remote Configuration parameters in admin mode Enter the following commands: 1. config-remote)# protocol set [FTP SCP] Set the protocol to access remote server. 2. server [ip/ip6-address] Configure the IPv4 or IPv6 address of the Remote server. 3. username [username] Configure the user name for Remote server access and press enter. 4. password Set the user password to access the Remote server. 5. protocol set [FTP SCP] Set the protocol to access remote server. 6. remote show Use this command in admin mode to verify the new settings. Chapter 9 45

46 3.6 High Availability Settings 9 High Availability Settings 9.1 smb Redundancy (4700/4200 Only) The Grid Director 4700/4200 supports two Management Boards (smbs). One smb serves as the Active system manager while the other is a Passive management board. By default, the Active smb is the left board and the Passive smb is the right board. The Left smb has privilege over the Right smb. As long as the Left smb is installed in the chassis and is functional, it is considered Active (however the state is in Standalone). When you install two smbs in a chassis and the Active smb fails to boot, the Passive smb takes over the chassis fabric monitoring. It may take up to 3 minutes to detect a non-responsive the Active smb and additional 0.5 min to load all the switch modules. An smb boot failure event is reported in the Event Log - see Event and Error Logs (on page 52) for further information on event logs. smb IP Addresses Each smb carries the following IP addresses: LOCAL IP address for direct access Eth IP floating address to access the Active smb Duplicate IP: The Ethernet IP address is floating and is always attributed to the Active smb in the chassis. The management station continues to use the same IP address for the system management, regardless of any smb failover events that may have taken place in the chassis. 9.2 Viewing the smb Redundancy Status You can view the current redundancy status (active, passive or stand alone when there is only one smb in the chassis) via the CLI. To display the smb redundancy status, run: #smb-state show #smb-state show active 9.3 Installing a Passive smb Both smbs installed in the chassis must have the same software version and the same configuration. When installing an smb with a different software version or configuration than the one already installed in the chassis, run a update software chassis command or follow the procedure below. The switch will not work properly if you install a passive smb in the left slot when its software version is different than the one already installed in the chassis. For proper functionality, both 46

47 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ software versions must be identical. Also note that both smbs internal subnet prefixes must be identical. To install a passive smb Perform the following tasks on the active smb installed in the chassis: 1. Run the following command: (config-if)# ip-address show 2. Write down the internal subnet prefix (example: /16). 3. Run a version show command to view the software version. Perform the following tasks on the new passive smb 1. Physically install the new smb in the chassis. For more information, refer to the Installation Manual. 2. Connect to the new smb using the Local Interface (telnet or SSH) or serial interface (RS-232). 3. Run the following command: (config-if)# ip-address show 4. To configure the Local interface, run: (config-diagnostics-interface LOCAL)# ip-address-local set You can also configure an IPv6 address, by using: (config-diagnostics-interface LOCAL)# ip6-address-local set 5. Run a version show command to view the software version. Make sure that the software version is identical to the version of the other smb. If the software versions are different, run the following command from the admin mode): # update software smb Chapter 10 47

48 3.6 Security Settings 10 Security Settings 10.1 Grid Director 4000 Series Security Settings Overview The following security options and features are available in the Grid Director 4000 family: Secure protocols (SCP/SSH) Pre-defined IP hosts connection to the switch using IP filtering UFM Agent restrictions to prevent users from entering the system Disabling Telnet Web server NOTE: The Grid Director 4000 switches are delivered without predefined security settings. Set up your system according to your security requirements by using the commands detailed in the following sections SCP Secure Export and Import SCP (Remote Secure Copy Protocol) provides secure export and import of files and software upgrades between computers. SCP is based on the SSH protocol and provides enhanced security SCP Settings To set SCP in the switch, run: (config-remote)#protocol set SCP To view the Remote status, run. (config-remote)# remote show Refer to Remote Configuration Menu for more information IP Filtering The IP filtering on the Grid Director switch determines the IPs that can access your system via SSH, SCP, and SNMP. IP filtering is supported only for IPv4 addresses. To set IP filtering, run: 1. From the CLI config-security-ipfilt menu, set (enable) IP filtering as follows: (config-security-ipfilt)# ipfilt mode set enable 2. Add to the IP filtering table a new host that has permission to connect to the switch as follows. (config-security-ipfilt)# allowed-host add

49 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Remove from the IP filtering table an existing host for which you want to terminate permission to connect to the switch as follows: (config-security-ipfilt)# allowed-host delete Run the following command to view the list of hosts that are allowed to connect to your switch. They are listed in the Filtering table: (config-security-ipfilt)# allowed-host show Run the following command to view the IP filtering mode: (config-security-ipfilt)# ipfilt mode show Refer to Security Configuration (on page 48) for more information Enabling/Disabling Non-secure Protocols This section provides details on how to secure your system Disabling Telnet Access The Telnet daemon provides the server function for the telnet protocol. The telnet features enables you to access the switch from a remote site. Telnet is less secure than SSH. To enhance security, it is recommended that you disable the Telnet daemon. NOTE: Telnet is enabled by default. To set Telnet mode, run: (config-security)# telnetd set disable To view the Telnet status, run: (config-security)# telnetd show 10.5 Web Server Settings The Web server enables IT personnel to gain quick access to various logs and debug files. CAUTION: You must perform an Export log command before you can view the Logs. Refer to Exporting Logs. 49

50 3.6 Security Settings Setting the Web Server By default, the Web server is enabled. Use this feature to block access to this Web Server, if required. To configure the Web Server status via the CLI: 1. From the CLI config-security menu, view (show) the Web Server status, as follows. (config-security)# web show This shows whether the Web server is currently enabled. 2. Set the Web server status, as follows: (config-security)# web set [disable, http, https] Note: Any change to web server status will be effective only after the next reboot. To disable the Web Server: (config-security)# web set disable To enable the Web Server: (config-security)# web set http To enable the Web Server over https (IPv4 only): (config-security)# web set https 3. View (show) the Web Server status, as follows. (config-security)# web show After reboot, this shows the new Web Server status Using the Web Server This window is used to view Logs and Statistics from the Web Server. To use the Web server (browser-based): In the browser Address bar, enter the Server IP address. The Logs and Statistics Page opens showing all the log options. Refer to Event and Error Logs (on page 52) for further information Using the Unified Fabric Manager (UFM) Agent The Unified Fabric Manager (UFM) agent is an embedded tool that enables the UFM to discover the system and fabric management. The UFM agent provides the UFM server with information about the switch parameters; it uses the TCPIP protocol and listens to port The port number is configurable. You can disable the UFM agent if UFM functionality is not required. This section describes the commands that allow you to enable/disable the UFM agent software running on the switch, if you have UFM server installed on a host in the fabric. The UFM agent is enabled by default. Enabling or disabling the UFM agent take effect after rebooting the switch. The UFM Agent is supported only over IPv4. 50

51 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ To disable the UFM agent, run: (config-security)# ufmagent set disable To view the status of the UFM agent, run: (config-security)# ufmagent show To set a multicast IP address for UFM, run: ufmagent ip-address-multicast set To set a UFM multicast port, run: ufmagent port-multicast set Refer to the Security Configuration section in Chapter 13 for the list of UFM agent commands provided in the CLI config mode. Refer to the Unified Fabric Manager (UFM) User Manual for detailed information about the UFM and its usage options. Chapter 11 51

52 3.6 Event and Error Logs 11 Event and Error Logs 11.1 Log Types The following log types are available: CLI Log Error Log Event Log sm-log umserver In addition, you can view all logs and debug information (exportlog command). The remote logger feature allows you to send the logs on a remote server. The Web server is enabled by default. You can downloading the Logs via the web only when the Web server is enabled. To enable the Web Server run: (config-security)# web set Refer to the Web Server Settings (on page 49) section for more information Viewing the Event and Error Logs Use the Event Log to discover an event occurring in the fabric or on a specific device. Events are reported to the CLI Event Log file. Use the Error Log to discover an error or problem occurring in the fabric or on a specific device. Errors are reported to the CLI Error Log file. You can view error or event logs from the CLI Logs menu, by running the following commands: Error Log: (logs)# error-log show Event Log: (logs)# event-log show Refer to Chapter 15 for a list of log commands you can run through the CLI admin mode. Appendix C provides a detail of events and error types Error and Event Log Messages The following table provides details on error and event log messages. Table 7: Error and Event Log Messages Description Severity Error Log Event Log Remarks Example SNMP user from Ip %s Log in SNMP user from Ip Log in 52

53 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Description Severity Error Log Event Log Remarks Example SNMP user from Ip %s Log out Temperature is within the normal range Temperature exceeded the warning threshold Temperature exceeded the alarm threshold Temperature exceeded the alarm threshold (failed to get temperature) Could not perform fan air flow direction validation Fan air flow direction matches system type Fan air flow direction does not match the system type Fan air flow direction matches system type NORMAL Not in policy table WARNING Not in policy table CRITICAL Not in policy table CRITICAL Not in policy table critical Not in policy table normal Not in policy table critical Not in policy table normal Not in policy table < Dual Spine module> #<index> CXP port < port num 1..12>/ IB port <num2 1..3> is up Severity from policy table for SRV SYS_L INK_EVENT_ TRAP Num2 indicates the internal link inside the connector (1,2,3) sfb-4700-x2 #1 CXP port 1/ IB port 3 is up < Dual Spine module> #<index> CXP port < port num 1..12>/ IB port <num2 1..3> is down Severity from policy table for SRV SYS_L INK_EVENT_ TRAP sfb-4700-x2 #1 CXP port 1/ IB port 3 is down < spine/line module> Severity from policy table for slb-4018 #9 Port 5 is up 53

54 3.6 Event and Error Logs Description Severity Error Log Event Log Remarks Example #<index> Port < Physical port num > is up SRV SYS_L INK_EVENT_ TRAP < spine/line module> #<index> Port < Physical port num > is down Severity from policy table for SRV SYS_L INK_EVENT_ TRAP slb-4018 #9 Port 5 is down < dual spine/line module> #<index> port <Physical port num>: Cable is <in/out>. Severity from policy table for SRV SYS_C ABLE_EVEN T_TRAP sfb-4700-x2 #1 port 3: Cable is out slb-4018 #9 port 5: Cable is in <Power Supply module> #<num>: <message> Severity from policy table for SRV POWE R_DC_TRAP Message is one of the following: is now O.K fault spsu-s #2: was extracted was extracted power failure, DC fault power failure, AC fault state is unknown <Fan module>: <message>, fan rate in <turbo/normal> mode Severity from policy table for SRV FAN_T RAP Message is one of the following: is now O.K fault was extracted sfu-40h: is now O.K, fan rate in normal mode power failure, DC fault power failure, AC fault state is unknown <smb module> #<Position>: <message> Severity from policy table for SRV SYS_S PINE_LINE_E VENT_TRAP Position: L, R Message: is now O.K fault was extracted smb #L: is now O.K power failure, DC fault power failure, AC fault state is unknown 54

55 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ Description Severity Error Log Event Log Remarks Example <line/spine module> #<Position>: <message> Severity from policy table for SRV SYS_S PINE_LINE_E VENT_TRAP Position: L, R Message: is now O.K fault was extracted power failure, DC fault power failure, AC fault state is unknown 10G High availability event device number: %d interface name:%s Severity from policy table for SRV 10G_H IGH_AVAILA BILITY Table 8: Additional Error and Event Log Messages Description Severity Error Log Event Log Temperature is within the normal range NORMAL Temperature exceeded the warning threshold WARNING Temperature exceeded the alarm threshold CRITICAL Temperature exceeded the alarm threshold (failed to get temperature) CRITICAL Fan sfuh sub fan 2 fault WARNING Fan sfuh sub fan 2 is now O.K NORMAL All sub fans in the Fan Unit sfuh have failed. CRITICAL NOT all sub fans in sfuh are now in fault state. WARNING Some fans in the Fan Unit sfuh have failed. WARNING All fans in the Fan Unit sfuh are operational. NORMAL Fan sfuv sub ps 3 fault WARNING Fan sfuv sub ps 3 is now O.K NORMAL Fan Unit sfuv was extracted CRITICAL Fan Unit sfuv was inserted NORMAL Fan sfuv fault WARNING Fan sfuv is now O.K NORMAL Fan sfuv rate in turbo mode WARNING Fan sfuv rate in normal mode NORMAL 55

56 3.6 Event and Error Logs Description Severity Error Log Event Log PS #1: not present MINOR PS #1: is now present NORMAL PS #1: power failure, AC fault MINOR PS #1: power (AC) is now O.K NORMAL PS #1: power failure, DC fault CRITICAL PS #1: power (DC) is now O.K NORMAL Attempt to read PSU 1 VPD has failed CRITICAL Software upgrade started NORMAL Firmware upgrade started NORMAL Software upgrade finished successfully NORMAL Firmware upgrade finished successfully NORMAL Software upgrade terminated with error WARNING Firmware upgrade terminated with error WARNING High Availability Log Messages Failed to synchronize with the active smb CRITICAL smb Log Messages smb failover occurred INFO Traps are also included in the error and event log. Refer to Chapter 11 (on page 52) or Event and Error Logs (on page 52) for the list of log messages. Appendix C provides a list of additional errors, traps, and events Error Log The Error Log records InfiniBand-specific events (traps) and SM internal events in a text file format. These events are then reported to the user via the Browser window or the CLI Error Log file. 56

57 Grid Director 4000 Family User Manual for 4700, 4200, 4036E, & 4036/ You can use this Log to view event that occur in the system. The following is an example of an Error Log in text format, as reported through the Browser where you can see that there is a problem with the fan air flow direction. Use the Traps and Events table in Appendix C for a description of the errors and for troubleshooting. Viewing the Error Log via the CLI You can also view the Error logs from the Viewing the CLI Log (on page 58)s menu, by running the (logs)# error-log show command. Refer to the List of CLI Commands for a list of CLI logs mode and commands. You can view the Error Log by uploading the ExportLog to a remote server and viewing the log, as described below. To view the Error Log, run: (logs)# error-log show Event Log The Event Log is used to discover if there is a problem in the fabric or on a specific device. The Event Log provides information on the specific events. Event types are detailed in Appendix C. The Event Log records InfiniBand-specific events (traps) and SM internal events in a text file format. These events are then reported to the user via the Browser window or the CLI Event Log file. 57

58 3.6 Event and Error Logs The following is an example of an event log in text format, as reported through the browser: In this display you can see that the fan air flow direction is critical and that there is a Warning on Port 31. Use the Traps and Events table in Appendix C for information on these errors, and for troubleshooting if necessary. You can also view these logs from the Logs Menu, as explained in Chapter Viewing the Event Log via the CLI You can view the Event Log by uploading the ExportLog to a remote server and view the log, as described below. To view the Event Log via the CLI, run: (logs)# event-log show 11.3 Viewing the CLI Log The CLI log file contains a history of all the entered commands. To view the CLI log file, run: (logs) cli-log show 11.4 Viewing the UmServer Log The umserver log file is used for debug purposes. To view the umserver log file, run: (logs) umserver-log show 11.5 Exporting and Viewing the Event and Error Log The event and error logs are used for diagnostic purposes to view the errors that occur within the system. 58