User s Guide. Setting Up NPAR with QLogic Control Suite, QConvergeConsole, or Comprehensive Configuration Management

Transcription

1 User s Guide Setting Up NPAR with QLogic Control Suite, QConvergeConsole, or Comprehensive Configuration Management 3400 and 8400 Series Adapters Windows Operating Systems BC B

2 User s Guide Setting Up NPAR with QCC, QCS, or CCM 3400 and 8400 Series Adapters This document is provided for informational purposes only and may contain errors. QLogic reserves the right, without notice, to make changes to this document or in product design or specifications. QLogic disclaims any warranty of any kind, expressed or implied, and does not guarantee that any results or performance described in the document will be achieved by you. All statements regarding QLogic's future direction and intent are subject to change or withdrawal without notice and represent goals and objectives only. Rev A, March 30, 2015 Rev B, October 6, 2016 Document Revision History Changes Sections Affected Added QConvergeConsole to title. Updated disclaimer. Added Windows Server 2016 (with Nano Server support) to the list of Windows Server OSs. Added QConvergeConsole (QCC) to the list of management tools. Removed QCS GUI from the list of management tools (no longer supported). Added that QCC can be downloaded from the QLogic web site. Revised the paragraph preceding Figure 1-2 to remove QCS GUI and the CIM provider, and added QCC GUI, RPC agent, and QLASP NIC teaming driver. Updated Figure 1-2: removed the CIM provider package and added QCC GUI. Removed Figure 1-3 (was 32-bit example; 32-bit and 64-bit are now the same). Removed QCS GUI; added QCC GUI. Updated instructions to run both locally or remotely. In the bulleted list of 32-bit or 64-bit OSs supported for the remote control system, added Windows Server 2016 (with Nano Server support). In Step 8, changed the third paragraph to QCC GUI and QSC CLI can also control and view these settings. Removed instructions about setting up NPAR with QCS GUI. Added a section describing how to set up NPAR using QCC GUI. page i page ii Operating System on page 2 Management Tools on page 2 QLogic Adapter Software and Drivers on page 3 QCC GUI and QCS CLI Installation Considerations on page 4 Setting Up NPAR Using CCM on page 6 Setting Up NPAR Using QCC GUI on page 11 ii BC B

3 User s Guide Setting Up NPAR with QCC, QCS, or CCM 3400 and 8400 Series Adapters Added a new third paragraph about installing QLXNRemote. In the descriptions of the -u Administrator and -p PassWord123 parameters, expanded the second sentence to indicate that they are not applicable for local connections when the user is logged in as the administrator. In the descriptions of the -u Administrator and -p PassWord123 parameters, expanded the second sentence to indicate that they are not applicable for local connections when the user is logged in as the administrator. Removed references to CCM/QCC. Removed the instructions about setting the MTU size with QCS GUI. Added instructions about setting the MTU size with QCC GUI. Introduction on page 15 Identifying the PCI Bus and Device Numbering on page 16 Applying XML File Changes on page 19 Verifying NPAR Setup on Windows Network Connections on page 25 Setting the MTU Size in QCC GUI on page 44 iii BC B

4 User s Guide Setting Up NPAR with QCC, QCS, or CCM 3400 and 8400 Series Adapters iv BC B

5 Table of Contents Part I Preface Intended Audience What Is in This Guide Related Materials Documentation Conventions License Agreements Technical Support Downloading Updates Training Contact Information Knowledge Database Setting Up NPAR xi xi xii xiii xiv xiv xv xv xvi xvi 1 System Requirements QLogic Adapter Operating System Management Tools QLogic Adapter Software and Drivers QCC GUI and QCS CLI Installation Considerations Setting Up NPAR Using CCM 3 Setting Up NPAR Using QCC GUI Introduction Setting Up NPAR Setting Up NPAR Using QCS CLI Introduction QCS CLI Basics QCS CLI Command Format Identifying the PCI Bus and Device Numbering Saving a Copy of the Current NPAR Settings Setting NPAR Parameters in the XML File Applying XML File Changes v BC B

6 User s Guide Setting Up NPAR with QCC, QCS, or CCM 3400 and 8400 Series Adapters Viewing the Current NPAR Settings XML Sample Files Single Function Mode Dual-Port NPAR Mode Verifying NPAR Setup on Windows Network Connections Part II 6 Bandwidth Configuring NPAR Bandwidth Weight Maximum Bandwidth PCIe Bus Device Function Numbering 8 Setting MTU Sizes Introduction Setting the MTU Size in Windows Device Manager Setting the MTU Size in QCC GUI Part III Example Configurations 9 Non-DCB, 0 Weight, and 100 Maximum Bandwidth Over-subscription 10 Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription 11 Lossless FCoE over DCB, 100 Maximum Bandwidth, Over-subscription 12 Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription 13 Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription 14 Non-DCB, Weight, 100 Maximum Bandwidth, Over-subscription 15 Non-DCB, 0 Weight, 25 Maximum Bandwidth, Fixed Subscription 16 Non-DCB, 0 Weight, Maximum Bandwidth, Over-subscription 17 Lossless FCoE over DCB, Maximum Bandwidth, Over-subscription 18 Non-DCB, Weight, Maximum Bandwidth, Over-subscription vi BC B

7 User s Guide Setting Up NPAR with QCC, QCS, or CCM 3400 and 8400 Series Adapters Index vii BC B

8 User s Guide Setting Up NPAR with QCC, QCS, or CCM 3400 and 8400 Series Adapters List of Figures Figure Page 1-1 Windows Device Manager in SF Mode with No Offloads Enumerated QLogic QLE3400/8400 Driver Download Web Page Boot Screen CCM Device List Menu CCM Main Menu CCM Device Hardware Configuration Menu, Single Function Mode CCM Device Hardware Configuration Menu, NPAR Mode CCM NIC Partition Configuration Menu CCM PF #0 Menu CCM Exit Configuration Menu QCC GUI Configuring NPAR QCC GUI Enabling Mode QCC GUI NPAR Console Windows Device Manager NPAR Mode Adapter QCC GUI NPAR Mode Adapter QCS CLI list phyadapters Command Output QCS CLI Saving the NPAR File QCS CLI Restoring or Copying an Original or Modified npar.xml File QCS CLI CFG Multi-Function Output Network Connections NPAR Mode Network Connections Properties and Maximum Bandwidth Speed Non-DCB Mode, 100-Percent Sum, Relative Bandwidth Weight, Transmit Traffic Flow Non-DCB Mode, Zero Sum Relative Bandwidth Weight, Transmit Traffic Flow DCB Mode Transmit Traffic Flow with Lossless FCoE, Lossy iscsi, and Lossy Ethernet DCB Mode Transmit Traffic Flow with Lossless FCoE, Lossless iscsi, and Lossy Ethernet DCB Mode Transmit Traffic Flow with Lossless iscsi and Lossy Ethernet Single Port QLogic QLE3440-SR/CU and QLE8440-SR/CU Adapter Function Numbering Interleave Dual Port QLogic QLE3442-RJ/SR/CU and QLE8442-SR/CU Adapter Function Numbering Interleave Network Connections Properties Network Connections Properties Window Network Connections PCI Bus/Device/Function Number Windows Device Manager, Advanced Properties, Jumbo Packet QCC GUI Advanced Params Tab QCC GUI Advanced Params Tab, Jumbo Packet QCC GUI Configuration Tab, iscsi Management QCC GUI Configuration Tab, iscsi Management, MTU Size i Legend for Plot Drawings Non-DCB, 0 Weight, 100 Maximum Bandwidth, Over-subscription Plot viii BC B

9 User s Guide Setting Up NPAR with QCC, QCS, or CCM 3400 and 8400 Series Adapters 10-1 Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription Plot DCB FCoE 100 Maximum Bandwidth Over-subscription Plot Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription Plot Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription Plot Non-DCB, Weight, 100 Maximum Bandwidth, Over-subscription Plot Non-DCB, 0 Weight, 25 Maximum Bandwidth, Fixed Subscription Plot Non-DCB, 0 Weight, Maximum Bandwidth Over-subscription Plot Lossless FCoE over DCB, Maximum Bandwidth, Over-subscription Plot Non-DCB, Weight, Maximum Bandwidth, Over-subscription Plot ix BC B

10 User s Guide Setting Up NPAR with QCC, QCS, or CCM 3400 and 8400 Series Adapters List of Tables Table Page 9-1 Non-DCB, 0 Weight, and 100 Maximum Bandwidth Over-subscription Components Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription Components Lossless FCoE over DCB, 100 Maximum Bandwidth Over-subscription Components Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription Components Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription Components Non-DCB, Weight, 100 Maximum Bandwidth, Over-subscription Components Non-DCB, 0 Weight, 25 Maximum Bandwidth, Fixed Subscription Components Non-DCB, 0 Weight, Maximum Bandwidth Over-subscription Components Lossless FCoE over DCB, Maximum Bandwidth, Over-subscription Components Non-DCB, Weight, Maximum Bandwidth Over-subscription Components x BC B

11 Preface Intended Audience This guide is for users who want to set up QLogic switch independent NIC partitioning (NPAR) on their QLogic QLE3400/8400 Series Adapters in a Windows system using either the QConvergeConsole (QCC) GUI, QLogic Control Suite (QCS) CLI, or Comprehensive Configuration Management (CCM) tool. What Is in This Guide This guide contains the information you need to set up NPAR on your Windows OS using either the QCC GUI, QCS CLI, or CCM tool. This preface specifies the intended audience, summarizes the contents of this guide, lists related documents, describes the typographic conventions used in this guide, refers you to the QLogic license agreements, and provides technical support and contact information. The remainder of the user s guide is organized into the following parts and chapters: Part I Setting Up NPAR Chapter 1 System Requirements provides the system requirements for configuring NPAR on a QLogic QLE3400/8400 Series Adapter in a Windows server. Chapter 2 Setting Up NPAR Using CCM describes how to use the pre-boot CCM tool to set up NPAR. Chapter 3 Setting Up NPAR Using QCC GUI describes how to set up NPAR with QCC GUI. Chapter 4 Setting Up NPAR Using QCS CLI describes how to set up NPAR with QCS CLI. Chapter 5 Verifying NPAR Setup on Windows Network Connections shows you how to view your system s NPAR configuration using Windows Network Connections. xi BC B

12 Preface Related Materials Part II Configuring NPAR Chapter 6 Bandwidth provides detailed information about the NPAR bandwidth weight and Maximum Bandwidth settings. Chapter 7 PCIe Bus Device Function Numbering describes how PCIe bus device function numbering works. Chapter 8 Setting MTU Sizes describes how to set the MTU sizes using Window s Device Manager and QCC GUI. Part III Example Configurations The following sections provide examples of different NPAR configurations. Chapter 9 Non-DCB, 0 Weight, and 100 Maximum Bandwidth Over-subscription Chapter 10 Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription Chapter 11 Lossless FCoE over DCB, 100 Maximum Bandwidth, Over-subscription Chapter 12 Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription Chapter 13 Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription Chapter 14 Non-DCB, Weight, 100 Maximum Bandwidth, Over-subscription Chapter 15 Non-DCB, 0 Weight, 25 Maximum Bandwidth, Fixed Subscription Chapter 16 Non-DCB, 0 Weight, Maximum Bandwidth, Over-subscription Chapter 17 Lossless FCoE over DCB, Maximum Bandwidth, Over-subscription Chapter 18 Non-DCB, Weight, Maximum Bandwidth, Over-subscription Related Materials For information about downloading documentation from the QLogic Web site, see Downloading Updates on page xv. xii BC B

13 Preface Documentation Conventions Documentation Conventions This guide uses the following documentation conventions: The supported adapters are referred to as QLogic QLE3400/8400 Series Adapters, QLogic adapters, or simply adapters. NOTE provides additional information. Text in blue font indicates a hyperlink (jump) to a figure, table, or section in this guide, and links to Web sites are shown in underlined blue. For example: Table 9-2 lists problems related to the user interface and remote agent. See Installation Checklist on page 6. For more information, visit Text in bold font indicates user interface elements such as a menu items, buttons, check boxes, or column headings. For example: Click the Start button, point to Programs, point to Accessories, and then click Command Prompt. Under Notification Options, select the Warning Alarms check box. Text in Courier font indicates a file name, directory path, or command line text. For example: To return to the root directory from anywhere in the file structure: Type cd /root and press ENTER. Enter the following command: sh./install.bin Key names and key strokes are indicated with UPPERCASE: Press CTRL+P. Press the UP ARROW key. Text in italics indicates terms, emphasis, variables, or document titles. For example: For a complete listing of license agreements, refer to the QLogic Software End User License Agreement. What are shortcut keys? To enter the date type mm/dd/yyyy (where mm is the month, dd is the day, and yyyy is the year). Topic titles between quotation marks identify related topics within this manual. xiii BC B

14 Preface License Agreements QCS CLI command syntax conventions include the following: Plain text indicates items that you must type as shown. For example: qcscli list phyports < > (angle brackets) indicate a variable whose value you must specify. For example: <serial_number> [ ] (square brackets) indicate an optional parameter. For example: [<file_name>] means specify a file name, or omit it to select the default file name. (vertical bar) indicates mutually exclusive options; select one option only. For example: on off (ellipsis) indicates that the preceding item may be repeated. For example: x... means one or more instances of x. [x...] means zero or more instances of x. ( ) (parentheses) and { } (braces) are used to avoid logical ambiguity. For example: a b c is ambiguous {(a b) c} means a or b, followed by c {a (b c)} means either a, or b c License Agreements Refer to the QLogic Software End User License Agreement for a complete listing of all license agreements affecting this product. Technical Support Customers should contact their authorized maintenance provider for technical support of their QLogic products. QLogic-direct customers may contact QLogic Technical Support; others will be redirected to their authorized maintenance provider. Visit the QLogic support Web site listed in Contact Information for the latest firmware and software updates. For details about available service plans, or for information about renewing and extending your service, visit the Service Program Web page at xiv BC B

15 Preface Technical Support Downloading Updates The QLogic Web site provides periodic updates to product firmware, software, and documentation. To download firmware, software, and documentation: 1. Go to the QLogic Downloads and Documentation page: driverdownloads.qlogic.com. 2. Type the QLogic model name in the search box. 3. In the search results list, locate and select the firmware, software, or documentation for your product. 4. View the product details Web page to ensure that you have the correct firmware, software, or documentation. For additional information, click Read Me and Release Notes under Support Files. 5. Click Download Now. 6. Save the file to your computer. 7. If you have downloaded firmware, software, drivers, or boot code, follow the installation instructions in the Readme file. Instead of typing a model name in the search box, you can perform a guided search as follows: 1. Click the product type tab: Adapters, Switches, Routers, or ASICs. 2. Click the corresponding button to search by model or operating system. 3. Click an item in each selection column to define the search, and then click Go. 4. Locate the firmware, software, or document you need, and then click the item s name or icon to download or open the item. Training QLogic Global Training maintains a Web site at offering online and instructor-led training for all QLogic products. In addition, sales and technical professionals may obtain Associate and Specialist-level certifications to qualify for additional benefits from QLogic. xv BC B

16 Preface Technical Support Contact Information QLogic Technical Support for products under warranty is available during local standard working hours excluding QLogic Observed Holidays. For customers with extended service, consult your plan for available hours. For Support phone numbers, see the Contact Support link at support.qlogic.com. Support Headquarters QLogic Web Site Technical Support Web Site Technical Support Technical Training Knowledge Database QLogic Corporation Whitewater Drive, Suite 230 Minnetonka, MN USA support.qlogic.com The QLogic knowledge database is an extensive collection of QLogic product information that you can search for specific solutions. QLogic is constantly adding to the collection of information in the database to provide answers to your most urgent questions. Access the database from the QLogic Support Center: support.qlogic.com. xvi BC B

17 Part I Setting Up NPAR This section of the guide lists the system requirements for NPAR configuration, provides instructions on how to set up NPAR in QCC GUI, QCS CLI, and CCM; and shows you how to verify that NPAR is enabled. For details, see the following sections: System Requirements on page 2 Setting Up NPAR Using CCM on page 6 Setting Up NPAR Using QCC GUI on page 11 Setting Up NPAR Using QCS CLI on page 15 Verifying NPAR Setup on Windows Network Connections on page 25 1 BC B

18 1 System Requirements This section provides the system requirements for configuring NPAR on a QLogic QLE3400/8400 Series Adapter in a Windows server. QLogic Adapter You must have one of the following QLogic Adapters: QLE3442-RJ QLE3440-SR/CU QLE3442-SR/CU QLE8440-SR/CU QLE8442-SR/CU Operating System The QLogic QLE3400/8400 Series Adapter must be installed in a server using one of the following Windows Server OSs: 2008 R R (includes Nano Server support) Management Tools One of the following management tools must be available: Comprehensive Configuration Management (CCM) QLogic Control Suite (QCS) CLI QConvergeConsole (QCC) GUI CCM is embedded in the option ROM of the QLogic QLE3400/8400 Series Adapter. QCS (CLI) and QCC (GUI) can be downloaded from the QLogic web site. See QLogic Adapter Software and Drivers on page 3 for details. 2 BC B

19 1 System Requirements QLogic Adapter Software and Drivers QLogic Adapter Software and Drivers When the QLogic adapter is first installed on a server in single function (SF) mode, the iscsi and Fibre Channel over Ethernet (FCoE) offload devices may not enumerate. Figure 1-1 provides an example. Figure 1-1. Windows Device Manager in SF Mode with No Offloads Enumerated If the latest drivers are not present on the system, go to the QLogic driver download web site, located at: driverdownloads.qlogic.com 3 BC B

20 1 System Requirements QLogic Adapter Software and Drivers From this site, download and install the latest Windows drivers, QCC (GUI and RPC agent), QCS (CLI, RPC agent, and QLASP NIC teaming driver), and applicable firmware upgrade package for your specific system. See Downloading Updates on page xv. Figure 1-2 provides an example. Figure 1-2. QLogic QLE3400/8400 Driver Download Web Page QCC GUI and QCS CLI Installation Considerations QCC GUI and QCS CLI can be run locally on the target server or by connecting from a remote control system, as described in the following paragraphs. To run QCC GUI or QCS CLI software locally: 1. Install QCC GUI on the target server. 2. For QCS CLI, install a copy of the CLI folder on the target server. 4 BC B

21 1 System Requirements QLogic Adapter Software and Drivers To run QCC GUI or QCS CLI software remotely: 1. Install the RPC agent on the target server. 2. Install QCC GUI on the remote control system. 3. For QCS CLI, install a copy of the CLI folder on the remote control system. The remote control system can be either a host system or a virtual machine (VM) on any hypervisor. The remote control system must be one of the following 32-bit or 64-bit OSs: Windows Server 2008 R2 Windows Server 2012 Windows Server 2012 R2 Windows Server 2016 (includes Nano Server support) Windows 7 Windows 8 Windows BC B

22 2 Setting Up NPAR Using CCM This section describes how to use the pre-boot Comprehensive Configuration Management (CCM) tool to configure NPAR parameters. CCM is embedded in the option ROM of the QLogic QLE3400/8400 Series Adapter. To configure NPAR using CCM: 1. Start or reboot the server and wait for the line QLogic Ethernet Boot Agent to appear (see Figure 2-1). Figure 2-1. Boot Screen 2. Within 5 seconds, press Ctrl+S to open the CCM Device List menu (see Figure 2-2). Figure 2-2. CCM Device List Menu The Device List menu displays the ports from all QLogic QLE3400/8400 Adapters (including the other QLogic 57xx/5771x/578xx-based Adapters) detected in the system. 6 BC B

23 2 Setting Up NPAR Using CCM You can distinguish between multiple installed adapters by the MAC address. 3. Press the DOWN ARROW key to select the device port to configure, and then press ENTER to open that port s Main Menu (see Figure 2-3). Figure 2-3. CCM Main Menu 4. Select Device Hardware Configuration, and then press ENTER. The Device Hardware Configuration menu opens (see Figure 2-4). Figure 2-4. CCM Device Hardware Configuration Menu, Single Function Mode The Multi-Function Mode parameter is set to the default of single function (SF) mode. 7 BC B

24 2 Setting Up NPAR Using CCM 5. Change the value of the Multi-Function Mode parameter by pressing the RIGHT ARROW key to select NPAR mode (see Figure 2-5). Figure 2-5. CCM Device Hardware Configuration Menu, NPAR Mode When enabling NPAR on a port, it is also enabled on the other ports of the device. This menu also allows you to change the per-port SR-IOV mode. 6. Press ESC to return to the Main Menu (see Figure 2-3). 7. From the Main Menu, select NIC Partition Configuration, and then press ENTER. The NIC Partition Configuration menu opens (see Figure 2-6). This figure shows the first port of a single- or dual-port adapter. Figure 2-6. CCM NIC Partition Configuration Menu 8 BC B

25 2 Setting Up NPAR Using CCM From the NIC Partition Configuration menu, you can do the following: Configure the port s pause flow control with the Flow Control parameter. This parameter is used only in non-data center bridging (DCB) mode when DCB priority flow control (PFC) mode is not enabled. Reset the port s NPAR settings back to the default setting, Auto, with the Reset Configuration to Default parameter. This setting returns all bandwidth weights to 0, all maximum bandwidth settings to 100, and enables only the Ethernet protocol on all partitions. Configure each of the port s partition settings, as shown in Step Press the DOWN ARROW key to select the desired partition, and then press ENTER. The physical function (PF) #n page opens (see Figure 2-7). Figure 2-7. CCM PF #0 Menu From the PF#n menu, you can enable the protocols the PF will support and set its Bandwidth Weight and Maximum Bandwidth values (see Bandwidth on page 28). QCC GUI and QCS CLI can also control and view these settings. Each partition is independently configured. 9. Make the desired changes on the PF#n menu. Use the UP and DOWN ARROW keys to select the parameter, and then press the SPACEBAR to change the parameter s value. 9 BC B

26 2 Setting Up NPAR Using CCM 10. Press ESC twice. The Exit Configuration menu opens (see Figure 2-8). Figure 2-8. CCM Exit Configuration Menu 11. Press ENTER to Exit and Save Configurations. The Device List menu opens (see Figure 2-2). 12. Repeat Step 3 and Steps 7 through 11 to configure all of the desired ports of a single adapter. 13. Repeat Steps 1 through 12 to configure each separate single or dual port QLE3400/8400 Series Adapter on that host. 14. After configuring the last port, on the Main Menu, press CTRL+ALT+DEL to force a reboot. NOTE Do not press the ESC key to exit CCM s Main Menu, since that action will not apply your saved changes to the QLogic adapter s current running configuration. 10 BC B

27 3 Setting Up NPAR Using QCC GUI Introduction This section describes how to set up NPAR with QConvergeConsole (QCC) GUI. Before using QCC GUI to set up NPAR, ensure that the latest adapter drivers and QCC software have been installed. See Management Tools on page 2, QLogic Adapter Software and Drivers on page 3 and QCC GUI and QCS CLI Installation Considerations on page 4. QCC GUI requires the NX remote procedure call (RPC) agent (called QLNXRemote) to be installed on the target server (the server containing the QLE3400/8400 adapters). See the Installation Guide, QConvergeConsole GUI (part number SN ) for more details. Setting Up NPAR To configure NPAR with QCC GUI: 1. Launch QCC GUI. 2. Connect to the local host. 3. On the left pane, under the Host tab, expand the view for the adapter you want to configure by clicking the plus sign (+), and then select the adapter. 11 BC B

28 3 Setting Up NPAR Using QCC GUI Setting Up NPAR 4. On the right pane, click the Configuration tab (see Figure 3-1). Figure 3-1. QCC GUI Configuring NPAR 5. Click the NIC Partition tab, and then click NPAR (see Figure 3-2). Figure 3-2. QCC GUI Enabling Mode 6. Enable or disable the port s protocols by selecting the appropriate check box (Ethernet, iscsi, or FCoE). The Ethernet Ndis protocol can be enabled or disabled on any partition. Only two offload protocols (either one FCoE and one iscsi, or two iscsi) can be enabled per port, but not on the same partition. 12 BC B

29 3 Setting Up NPAR Using QCC GUI Setting Up NPAR Remote boot from SAN configurations always use the first partition of the designated boot port. Therefore, the FCoE protocol must be enabled on the first partition (on port 0 function 0 or port 1 function 1) if that specific port is used for FCoE remote boot from SAN. The same rule is true for iscsi-offload remote boot from SAN. However, if a software non-offloaded Ethernet pathway is enabled, then the iscsi-offload protocol need not be enabled on the first partition of that specific port (but the Ethernet protocol does need to be enabled). 7. Set the bandwidth by typing or selecting the desired values in the Bandwidth Weight (%) and Max Bandwidth boxes. 8. Click Apply, and then enter the password (the default password is config). The protocols are updated, and you are prompted to reboot the system (see Figure 3-3). Figure 3-3. QCC GUI NPAR Console 9. Reboot the system for Windows to discover the new physical functions and enumerate the selected devices. After the drivers are installed, the enabled protocols (Ethernet, FCoE, and iscsi) are visible in the Windows Device Manager and QCC GUI. 13 BC B

30 3 Setting Up NPAR Using QCC GUI Setting Up NPAR Figure 3-4 illustrates NPAR in Windows Device Manager. Figure 3-5 illustrates NPAR mode on an adapter in QCC GUI. The configuration contains a dual-port QLE8442 with eight system devices (QLogic Gigabit Ethernet Multi Function #95 #102) and eight Ethernet devices (QLogic Gigabit Ethernet Adapter Multi Function #95 #102). Figure 3-4. Windows Device Manager NPAR Mode Adapter Figure 3-5. QCC GUI NPAR Mode Adapter 14 BC B

31 4 Setting Up NPAR Using QCS CLI Introduction This section describes how to set up NPAR with QLogic Control Suite (QCS) CLI. Before using QCS CLI to set up NPAR, ensure that the latest adapter drivers, CLI folder, and CIM provider software have been installed. See Management Tools on page 2, QLogic Adapter Software and Drivers on page 3, and QCC GUI and QCS CLI Installation Considerations on page 4. See the QCSCLI_Readme.txt file (on the QLogic driver downloads Web site) for more details on using QCS CLI, for example, how to use the additional -scheme https option in remote connection commands when using an https connection. QCS CLI requires the NX remote procedure call (RPC) agent (called QLNXRemote) to be installed on the target server (the server containing the QLE3400/8400 adapters). See the Installation Guide, QConvergeConsole GUI (part number SN ) for more details. QCS CLI can configure NPAR mode using an XML file. All changes (except Maximum Bandwidth and Relative Bandwidth Weight) require a reboot to be applied. All QCS CLI commands can be used in a script. The following sections describe how to edit an XML file to set up NPAR and use QCS CLI commands to apply (restore) the file to a specified adapter port. QCS CLI Basics The following sections show you how to use QCS CLI commands to get the information you need to edit an XML file for NPAR. 15 BC B

32 4 Setting Up NPAR Using QCS CLI QCS CLI Basics QCS CLI Command Format The general format of the QCS CLI command is: QCScli [-t <target type>] [-f <target ID format>] [-i <target ID>] [-r <IP address>] [-u <username>] [-p <password>] [-n <port>] [-a digest basic] [-protocol <wmi cimxml wsman all>] [-scheme <http https>] [-persist] <command string> Identifying the PCI Bus and Device Numbering You need the QLogic adapter s function number for the QCS CLI commands that process the XML file. For NPAR mode commands, the QCS CLI operates in PhyAdapters mode, which does not use the function number. Instead, the PCI bus and device (B:D) numbers of an adapter on the host is used. To list the B:D numbers: 1. Open a DOS command prompt in the same location as the copied QCS CLI files. 2. Issue the following command: C:\QCScli>qcscli -r u Administrator -p PassWord123 "list phyadapters" Where: C:\QCScli = The DOS prompt. This prompt will differ, depending on the location of the QCS CLI files. A scripted command must have the path added. qcscli = The QCS CLI executable name -r = The IP address of the target system. This address is not applicable for local connections. -u Administrator = The case-sensitive user name to log into the remote system. This name is not applicable for local connections when the user is logged in as the administrator. -p PassWord123 = The case-sensitive password to log into the remote system. This password is not applicable for local connections when the user is logged in as the administrator. list phyadapters = The command to run. Be sure to use double quotes in the command, as shown. 16 BC B

33 4 Setting Up NPAR Using QCS CLI QCS CLI Basics Figure 4-1 provides an example. Figure 4-1. QCS CLI list phyadapters Command Output Saving a Copy of the Current NPAR Settings QLogic recommends saving a copy of the QLogic adapter s current settings to an XML file, and then editing the NPAR parameters. Alternately, you can edit the npar.xml file in the QCS CLI location. Both files can be edited to enable and configure NPAR mode or SF mode, set which partitions have which options enabled, and configure the partitions Maximum Bandwidth and Relative (minimum) Bandwidth Weight settings. Following is a command that saves a copy of the current NPAR settings: QCScli>qcscli -r u Administrator -p PassWord123 -t phyadapters -f BDF -i 01:00 "cfg multi-function -s filename.xml" Where: C:\QCScli = The DOS prompt. This prompt will differ, depending on the location of the QCS CLI files. A scripted command must have the path added. qcscli = The QCS CLI executable name -r = The IP address of the target system. This address is not applicable for local connections. -u Administrator = The case-sensitive user name to log into the remote system. This name is not applicable for local connections. -p PassWord123 = The case-sensitive password to log into the remote system. This password is not applicable for local connections. -t phyadapters = The type of command being using (in this example, phyadapters). -f BDF = The command that identifies the format of the -i parameter (in this example, Bus:Device:Function (BDF), although Function is not used in the phyadapter command mode). 17 BC B

34 4 Setting Up NPAR Using QCS CLI Setting NPAR Parameters in the XML File -i 01:00 = The desired QLogic adapter s B:D number, identified in Identifying the PCI Bus and Device Numbering on page 16. cfg = The command to run. Be sure to use double quotes in the multi-function command and command parameters, as shown. -s = The save option. filename.xml = The file name where the NPAR information will be saved. This file name can include the path. If no path is specified, the file is saved to the same location as the QCScli.exe file. Figure 4-2 provides an example. Figure 4-2. QCS CLI Saving the NPAR File Setting NPAR Parameters in the XML File NOTE You only need to configure one port to set all of the NPAR settings of that adapter s ports. This section provides details about enabling protocols and bandwidth in the XML file. For example configurations, see XML Sample Files on page 22. To allow the use of a specific protocol on a partition (or function), set the appropriate parameter to Enable. The EthernetNdis protocol can be enabled or disabled on any partition. Only two offload protocols (either one FCoE and one iscsi, or two iscsi) can be enabled per port, but not on the same partition. Remote boot from SAN configurations always use the first partition of the designated boot port. Therefore, the FCoE protocol must be enabled on the first partition (on port 0 function 0 or port 1 function 1) if that specific port is used for FCoE remote boot from SAN. The same rule is true for iscsi-offload remote boot from SAN. However, if a software non-offloaded Ethernet pathway is enabled, then the iscsi-offload protocol need not be enabled on the first partition of that specific port (but the Ethernet protocol does need to be enabled). 18 BC B

35 4 Setting Up NPAR Using QCS CLI Applying XML File Changes Each partition s Relative Bandwidth Weight (or minimum bandwidth) is used only in non-dcb mode and must be set in the range of 0 percent to 100 percent. A values of all zeros indicates that all traffic shares the available bandwidth equally. The sum total of all partitions settings must equal either 0 percent (where all partitions are set to 0) or 100 percent. QLogic recommends a minimum bandwidth setting of 10 percent on a single partition if nonzero values are used. Each partition s Maximum Bandwidth is used in all modes and can be set in the range of 1 percent to 100 percent. The sum total value range for a single port can be from 4 percent to 400 percent (on a single- or dual-port QLogic QLE3400/8400 Series Adapter); that is, its sum does not have to total 100 percent. This value is the percentage of the port s link speed that is the top-end maximum send (transmit) bandwidth that will be allowed on that specific partition at all times. For more information about bandwidth, see Bandwidth on page 28. Applying XML File Changes After you have modified a saved XML file, you can apply the changes by restoring it back into the host system with either the -c or -o QCS CLI cfg multi-function parameters. For example: C:\QCScli>qcscli -r u Administrator -p PassWord123 -t phyadapters -f BDF -i 01:00 "cfg multi-function -c filename.xml" Where: C:\QCScli = The DOS prompt. This prompt will differ, depending on the location of the QCS CLI files. A scripted command must have the path added. qcscli = The QCS CLI executable name -r = The IP address of the target system. This address is not applicable for local connections. -u Administrator = The case-sensitive user name to log into the remote system. This name is not applicable for local connections when the user is logged in as the administrator. -p PassWord123 = The case-sensitive password to log into the remote system. This password is not applicable for local connections when the user is logged in as the administrator. -t phyadapters = The type of command being using (in this example, phyadapters). 19 BC B

36 4 Setting Up NPAR Using QCS CLI Viewing the Current NPAR Settings -f BDF = The command that identifies the format of the -i parameter (in this example, Bus:Device:Function (BDF), although Function is not used in the phyadapter command mode). -i 01:00 = The desired QLogic adapter s B:D number, identified in Identifying the PCI Bus and Device Numbering on page 16. filename.xml = The file to restore back into the selected adapter. This file name can include the path. If no path is specified, QCS CLI looks for the file in the same location as the QCScli.exe file. cfg multi-function = The command to run. Be sure to use double quotes in the command and associated parameters, as shown. Figure 4-3 provides an example. -c = Writes back the referenced XML file to the port. -o (not shown in example) = Forces overriding the SR-IOV virtual function (VF) settings, which causes the applicable functions with offloads enabled to set the number of SR-IOV VFs in that function to 0. Figure 4-3. QCS CLI Restoring or Copying an Original or Modified npar.xml File Viewing the Current NPAR Settings Following is a sample command to view the current NPAR settings: C:\QCScli>qcscli -r u Administrator -p PassWord123 -t phyadapters -f BDF -i 01:00 cfg multi-function 20 BC B

37 4 Setting Up NPAR Using QCS CLI Viewing the Current NPAR Settings Figure 4-4 shows a sample command output. Figure 4-4. QCS CLI CFG Multi-Function Output 21 BC B

38 4 Setting Up NPAR Using QCS CLI XML Sample Files XML Sample Files Following are sample XML files for single function (SF) and NPAR modes. The QCS CLI version number may change in subsequent software releases, so be sure to check this value. At the time of publication, the QCS CLI version is 7. Single Function Mode Following is an XML file for SF mode. The lines are split up to make them easier to read. <?xml version="1.0" encoding="utf-8"?> <MultiFunctionConfiguration> <Version>7</Version> <MultiFunctionMode>SingleFunction</MultiFunctionMode> </MultiFunctionConfiguration> Dual-Port NPAR Mode Following is an XML file for dual-port NPAR mode. The lines are split up to make them easier to read. <?xml version="1.0" encoding="utf-8"?> <MultiFunctionConfiguration> <Version>7</Version> <MultiFunctionMode>Multi-Function</MultiFunctionMode> <PortConfig> <Port>0</Port> <FlowControl>Auto</FlowControl> <EEE>Maximum Performance</EEE> <FunctionConfig> <Function>0</Function> <EthernetNdis>Enable</EthernetNdis> <iscsi>disable</iscsi> <FCoE>Enable</FCoE> <RelativeBandwidth>20</RelativeBandwidth> <MaxBandwidth>88</MaxBandwidth> </FunctionConfig> <FunctionConfig> <Function>2</Function> <EthernetNdis>Enable</EthernetNdis> <iscsi>enable</iscsi> <FCoE>Disable</FCoE> 22 BC B

39 4 Setting Up NPAR Using QCS CLI XML Sample Files <RelativeBandwidth>25</RelativeBandwidth> <MaxBandwidth>95</MaxBandwidth> </FunctionConfig> <FunctionConfig> <Function>4</Function> <EthernetNdis>Enable</EthernetNdis> <iscsi>disable</iscsi> <FCoE>Disable</FCoE> <RelativeBandwidth>35</RelativeBandwidth> <MaxBandwidth>75</MaxBandwidth> </FunctionConfig> <FunctionConfig> <Function>6</Function> <EthernetNdis>Enable</EthernetNdis> <iscsi>disable</iscsi> <FCoE>Disable</FCoE> <RelativeBandwidth>20</RelativeBandwidth> <MaxBandwidth>40</MaxBandwidth> </FunctionConfig> </PortConfig> <PortConfig> <Port>1</Port> <FlowControl>Auto</FlowControl> <EEE>Optimal Power and Performance</EEE> <FunctionConfig> <Function>1</Function> <EthernetNdis>Enable</EthernetNdis> <iscsi>disable</iscsi> <FCoE>Disable</FCoE> <RelativeBandwidth>40</RelativeBandwidth> <MaxBandwidth>60</MaxBandwidth> </FunctionConfig> <FunctionConfig> <Function>3</Function> <EthernetNdis>Enable</EthernetNdis> <iscsi>disable</iscsi> <FCoE>Disable</FCoE> <RelativeBandwidth>20</RelativeBandwidth> 23 BC B

40 4 Setting Up NPAR Using QCS CLI XML Sample Files <MaxBandwidth>40</MaxBandwidth> </FunctionConfig> <FunctionConfig> <Function>5</Function> <EthernetNdis>Enable</EthernetNdis> <iscsi>disable</iscsi> <FCoE>Enable</FCoE> <RelativeBandwidth>25</RelativeBandwidth> <MaxBandwidth>55</MaxBandwidth> </FunctionConfig> <FunctionConfig> <Function>7</Function> <EthernetNdis>Enable</EthernetNdis> <iscsi>enable</iscsi> <FCoE>Disable</FCoE> <RelativeBandwidth>15</RelativeBandwidth> <MaxBandwidth>30</MaxBandwidth> </FunctionConfig> </PortConfig> </MultiFunctionConfiguration> 24 BC B

41 5 Verifying NPAR Setup on Windows Network Connections This section shows you how to view your system s NPAR configuration using Windows Network Connections. The QLogic adapter ports (devices) can be used by any application, as if they were a separate adapter port. They appear as separate Ethernet devices in the Windows Network Connections page, which you can access from the Windows Control Panel. Figure 5-1 shows eight enabled Ethernet protocol partitions as eight separate Ethernet network connections, as enumerated by Windows Server 2012 R2 with Consistent Device Naming enabled (the naming on your server may differ). Figure 5-1. Network Connections NPAR Mode 25 BC B

42 5 Verifying NPAR Setup on Windows Network Connections Each of these network connection devices can be accessed individually, as if they were separate adapters. The connection status shows the Maximum Bandwidth setting as the Speed of the connection (see Figure 5-2). Figure 5-2. Network Connections Properties and Maximum Bandwidth Speed 26 BC B

43 Part II Configuring NPAR This section of the guide provides details about some of the NPAR parameters and configurations. See the following sections: Bandwidth on page 28 PCIe Bus Device Function Numbering on page 36 Setting MTU Sizes on page BC B

44 6 Bandwidth This section provides detailed information about the NPAR Relative Bandwidth Weight and Maximum Bandwidth settings. Bandwidth Weight The Bandwidth Weight is the value the port gives to a single partition s send (transmit) or outgoing traffic with respect to other partitions outgoing traffic on that port when there is more outgoing traffic on the partitions than there is available bandwidth on that port. The bandwidth weight setting follows these rules: The individual configurable value range is from 0 to 100 (as a percentage). The sum of a single port s partition values must be either exactly 100 or exactly 0 (which means that all four of the partitions are set to 0). If one or more of a partition's weight is set to 0, but the sum is 100 (that is, not all of the partitions are set to 0), then that partition's Relative Bandwidth Weight is effectively 1 with respect to allocation calculations. QLogic does not recommend setting a single partition to a value lower than 10. Setting all of the partition s values to 0 gives every traffic flow on every partition equal access to the port s available bandwidth regardless of which partition they are on, unless they are restricted by the partition's Maximum Bandwidth settings. If the sum of the relative bandwidth weights is 100 and there is more than one type of traffic flow on a specific partition (iscsi and Ethernet, or FCoE and Ethernet), then the combined traffic on that specific partition shares the bandwidth being allocated to it as if there was only one traffic flow on that partition. The weight applies to all enabled protocols on that partition. The per-partition Relative Bandwidth Weight settings are not applicable in data center bridging (DCB) mode. In DCB mode, all traffic flows use the per traffic class DCB's enhanced transmission selection (ETS) values. 28 BC B

45 6 Bandwidth Bandwidth Weight The NPAR transmit direction traffic flow rates are affected by the three main modes (non-dcb sum of weights is 100, non-dcb sum of weights is 0, and DCB), as described in the following paragraphs. The traffic protocols can be on any partition, not just on the ones shown here. These examples specifically show one port of a dual-port QLogic QLE8442-SR/CU or a single-port QLE8440-SR/CU Adapter, which would have four partitions per port. In non-dcb mode where the sum of the partition s Relative Bandwidth Weights equal 100 percent, each partition s combined traffic flow is equally scheduled to transmit within the limitations of the partition s Relative Bandwidth Weight and Maximum Bandwidth settings, as well as the overall connection s link speed. Therefore, a specific partition's Relative Bandwidth Weight value restricts the traffic flows sharing that partition s bandwidth allocation, as if one combined traffic flow with respect to the other actively sending partitions. The partition s send flow rate is based on the ratio of that partition s individual weight verses the aggregated weights of all the other actively sending partitions. Furthermore, each partition's combined traffic flow is capped by that partition's maximum weight setting. See Example Configurations on page 47 for more details. The actual inter-partition ratio of the two sharing traffic flows is controlled by the host OS. For example, the dynamic weight ratio can be viewed as a variable sized funnel that could be further restricted by the maximum bandwidth fixed sized funnel with the OS determining how the sharing traffic types are pouring into the combined funnels. Figure 6-1 has two iscsi offloads and four Ethernet protocols enabled. The Scheduler uses the Relative Bandwidth Weight and Maximum Bandwidth settings to determine which partition s traffic is sent. Figure 6-1. Non-DCB Mode, 100-Percent Sum, Relative Bandwidth Weight, Transmit Traffic Flow 29 BC B

46 6 Bandwidth Bandwidth Weight In non-dcb mode, where the sum of the partition s Relative Bandwidth Weights equals 0 (each partition's Relative Bandwidth Weight is set to 0), each individual traffic flow (Ethernet, iscsi, or FCoE offload) is equally scheduled to transmit within the limitations of the partition s Maximum Bandwidth and the overall connection s link speed. Therefore, if the Maximum Bandwidth of a specific partition is set to less than 100 percent, then the traffic flows sharing that partition are further restricted to where their combined traffic flow bandwidth is capped by the per-partition maximum setting. If all four partitions individual Maximum Bandwidths are set to 100 percent (they are all unrestricted), each actively sending traffic flow (without regard to which partition they are on) equally shares the transmit direction s total bandwidth (the transmit link speed). Figure 6-2 has the same two iscsi offloads and four Ethernet protocols enabled as in Figure 6-1. The Scheduler uses the Maximum Bandwidth settings to determine which partition s traffic is sent. Figure 6-2. Non-DCB Mode, Zero Sum Relative Bandwidth Weight, Transmit Traffic Flow In DCB mode, all of the partition s Relative Bandwidth Weights are disregarded and the individual traffic flows transmit within the limitations of the per-priority group s ETS value (determined by its traffic type), each partition s Maximum Bandwidth setting, and the overall connection s link speed. 30 BC B

47 6 Bandwidth Bandwidth Weight In this first DCB example (Figure 6-3), the FCoE traffic type could be assigned to priority group 1 (PG1) and the other traffic types (iscsi and Ethernet) could be assigned to the default priority group (PG0). Each priority group has its own ETS value, which works similarly to the minimum bandwidth setting (as shown in Figure 6-3. As with the other two rate-controlling modes, the host OS determines the actual inter-partition traffic ratio when two traffic types share the same partition. The Scheduler uses the per-priority group DCB-ETS and per-partition Maximum Bandwidth settings to determine which partition s traffic is sent. Figure 6-3. DCB Mode Transmit Traffic Flow with Lossless FCoE, Lossy iscsi, and Lossy Ethernet 31 BC B

48 6 Bandwidth Bandwidth Weight In this second DCB example, the FCoE traffic type is assigned to priority group 1 (PG1). The iscsi offloaded traffic (lossless iscsi type-length-value (TLV) over DCB) is assigned to its own priority group (PG2). All of the remaining Ethernet traffic is assigned to the default priority group (PG0), as shown in Figure 6-4. The Scheduler uses the per-priority group DCB-ETS and per-partition Maximum Bandwidth settings to determine which partition s traffic is sent. Figure 6-4. DCB Mode Transmit Traffic Flow with Lossless FCoE, Lossless iscsi, and Lossy Ethernet 32 BC B

49 6 Bandwidth Maximum Bandwidth In this last DCB example, the FCoE traffic type is replaced by iscsi, so that there are two iscsi offloaded protocols (lossless iscsi-tlv over DCB). Both protocols are assigned to their own priority group (PG2), and all of the remaining Ethernet traffic is still assigned to the default priority group (PG0), as shown in Figure 6-5. Each priority group has its own ETS value. The Scheduler uses the per-priority group DCB-ETS and per-partition Maximum Bandwidth settings to determine which partition s traffic is sent. Figure 6-5. DCB Mode Transmit Traffic Flow with Lossless iscsi and Lossy Ethernet NOTE A traffic type s send rate is approximately the ratio of its individual partition s Relative Bandwidth Weight setting divided by the sum of the relative bandwidth weights of all the partitions actively sending on that port or that partition s Maximum Bandwidth setting, whichever is lower. When the Relative Bandwidth Weights are all zeros, each traffic type has an equal weight with respect to one another (see Example Configurations on page 47). Maximum Bandwidth Each partition s Maximum Bandwidth settings can be changed the same way as the Relative Bandwidth Weight settings. The Maximum Bandwidth setting has a range of 1 percent to 100 percent in increments of 1 percent of the port s current link speed. For example, 10Gbps is in approximately 100Mbps increments and 1Gbps is in approximately 10Mbps increments. 33 BC B

50 6 Bandwidth Maximum Bandwidth This setting limits the most send bandwidth this partition uses and appears as its approximate link speed in various places in the OS. The four partitions may advertise in the OS that their link speeds are 10Gbps each, but they are all still sharing the same single 10Gbps connection. The displayed values may be rounded off by various applications. The Maximum Bandwidth value is applicable in both DCB and non-dcb modes, but only in the send (transmit or outgoing) direction. NOTE A partition s send Maximum Bandwidth setting does not affect a partition s receive (or incoming) direction traffic bandwidth, so the link speed displayed for the partition is only for the outgoing traffic. All partitions receive direction Maximum Bandwidth is always the port s current link speed and is regulated by the attached switch port, just as it is in single function (SF) mode when multiple traffic protocol types (Ethernet and iscsi offload, and FCoE offload) are enabled. The Maximum Bandwidth settings can oversubscribe a port by setting the four partitions of a single port to a sum total Maximum Bandwidth setting greater than 100 percent (that is, 10000Mbps or 10Gbps). In this case, the various partitions try to take as much bandwidth as allowed (by their specific maximum and minimum settings) as their individual traffic flow needs change. In an over-subscription situation, the QLogic QLE3400/8400 Series Adapter rations out free bandwidth based on the weights (sum is 0 verses sum is 100), maximum settings, and the current operating mode (DCB verses non-dcb). For example, if the first port s four partitions had been set to = 288, that port would be 288 percent subscribed (28.8Gbps) or 188 percent oversubscribed, as calculated by: 28.8Gbps subscribed 10Gbps line rate available = 18.8Gbps oversubscribed The Maximum Bandwidth setting applies to all protocols enabled on that partition. NOTE When NPAR mode is first enabled or after a reset, all four partitions have a Relative Bandwidth Weight of 0 percent and Maximum Bandwidth setting of 100 percent. See Example Configurations on page 47 for more details on how the Relative Bandwidth Weight/DCB-ETS and Maximum Bandwidth settings affect traffic flow in DCB and non-dcb modes of operation. 34 BC B

51 6 Bandwidth Maximum Bandwidth Protocol selection follows these rules: A maximum of two offloads (two iscsi, or one FCoE and one iscsi) can be enabled over any two of the available partitions of a single port. The FCoE offload protocol requires that DCB also be enabled and active on that port (that is, the adapter port is connected to a DCB-compliant and enabled link partner). The iscsi offload protocol can function without DCB. However, if lossless iscsi offload TLV over DCB is required, then DCB must be enabled and active on that port. DCB is enabled per port and affects all partitions on that port. NPAR is enabled per adapter and affects all ports on that adapter. Only one offload protocol (either iscsi or FCoE) can be enabled per single partition in NPAR mode. Both offloads can be enabled on the single partition of a port in SF mode. For simplicity, QLogic recommends using the first partition of a port for FCoE offload protocol, because the FCoE port WWN will be the same for both SF and NPAR modes on the same port. This recommendation simplifies the Fibre Channel forwarder (FCF) switch configuration. For Windows OSs, you can enable the Ethernet protocol on all, some, or none of the four partitions on an individual port simultaneously with any enabled offload protocols. For iscsi remote boot from LUN (iscsi offload or software non-offloaded Ethernet pathway), always use the first partition of the desired port. When using iscsi offload, enable it on the first partition. For FCoE remote boot from LUN, always use and enable FCoE protocol on the first partition of the desired port. For Windows OSs, the Ethernet protocol need not be enabled for the iscsi or FCoE offload protocol to be enabled and used on a specific partition. 35 BC B

52 7 PCIe Bus Device Function Numbering This section describes how PCIe bus device function numbering works. The PCI interface bus and device position numbers are the same for both ports and all eight of the partitions on those ports. The only PCI position values that are different are the function numbers. In non-partitioned single function (SF) mode, the functions are as follows: Function 0 Single port QLogic QLE3440-SR/CU and QLE8440-SR/CU Adapter Functions 0 and 1 Dual port QLogic QLE3442-RJ/SR/CU and QLE8442-SR/CU Adapter In partitioned (switch independent NIC partitioning) mode, there are up to eight functions per device. For example: On the QLogic QLE3440-SR/CU and QLE8440-SR/CU Adapter, functions 0, 2, 4, and 6 exist on the single port. See Figure BC B

53 7 PCIe Bus Device Function Numbering The QLE8400 Series Adapters support iscsi offload and FCoE offload. Consequently, they have an iscsi offload/fcoe initialization protocol (FIP) MAC address, whereas the QLE3400 Series Adapters do not support those offloads but still have the same interleaving of MAC addresses. Figure 7-1. Single Port QLogic QLE3440-SR/CU and QLE8440-SR/CU Adapter Function Numbering Interleave 37 BC B

54 7 PCIe Bus Device Function Numbering On the dual port QLogic QLE3442-RJ/SR/CU and QLE8442-SR/CU Adapter, functions 0, 2, 4, and 6 exist on the first port; and functions 1, 3, 5, and 7 exist on the second port. See Figure 7-2. Figure 7-2. Dual Port QLogic QLE3442-RJ/SR/CU and QLE8442-SR/CU Adapter Function Numbering Interleave 38 BC B

55 7 PCIe Bus Device Function Numbering Windows assigns the position number of the adapter. This numbering is not related to the PCIe interface numbering; instead, it is related to the open position numbers available in the registry when the adapters are enumerated. Therefore, the first port s first function may not always occupy the first position in the Windows Device Manager s Network Adapters, Storage Controllers, or System Devices listings. The partition s MAC addresses (Ethernet, iscsi, and FIP) interleave along with the function numbers. One way to locate the PCI bus/device/function information is to open the individual Network Connection s Properties, as described in the following paragraphs. To view the PCI bus/device/function information using Network Connections Properties: 1. Open the Control Panel. 2. Click Network and Internet. 3. Click Network Connections. A list of devices opens (see Figure 7-3). Figure 7-3. Network Connections Properties 39 BC B

56 7 PCIe Bus Device Function Numbering 4. Right-click the adapter whose properties you want to see, and then click Properties. The Properties window opens (see Figure 7-4). Figure 7-4. Network Connections Properties Window 5. Click Configure. The NDIS client device property window opens. This window contains that connection s PCIe bus, device, and function information. Figure 7-5 provides an example. Figure 7-5. Network Connections PCI Bus/Device/Function Number You can see the same PCIe bus, device, and function information in Window s Device Manager, especially for iscsi storage devices. All of the enabled protocols on the same partition have identical PCI interface location information; only the function number varies. 40 BC B

57 8 Setting MTU Sizes Introduction This section describes how to set the MTU sizes using Window s Device Manager and QCC GUI. NOTE When setting MTU sizes, the switch port to which an NPAR port is connected must have the switch's MTU size set to the largest MTU size of all partitions of that port to support all four partitions MTU size settings. Additionally, the remaining network that the traffic flows though must also support the desired MTU sizes without being dropped, truncated, or fragmented by the network. The MTU size for each individual Ethernet protocol enabled partition can be independently set from standard (1,500 bytes) up to jumbo (9,600 bytes) in Windows Device Manager and QCC. Setting the MTU Size in Windows Device Manager One place to set the Ethernet protocol enabled partition s adapter MTU size is through Windows Device Manager. To set the MTU size through Windows Device Manager: 1. Click the Start button, Control Panel, and select Device Manager. 2. Expand the Network Adapters button to show all the adapters in the system. 3. Right-click the selected adapter, and then click Properties. 4. A window with tabs opens, providing access to various adapter properties. 41 BC B

58 8 Setting MTU Sizes Setting the MTU Size in Windows Device Manager 5. Click the Advanced tab (see Figure 8-1). Figure 8-1. Windows Device Manager, Advanced Properties, Jumbo Packet 6. In the Advanced tab, perform the following steps: a. In the Property list, select the property you want to change. b. In the Value list, select the property s value. c. Click OK. In Windows, each individual partition s Ethernet and iscsi protocol enabled adapter MTU size setting can be different. For example, on a QLogic QLE8400 Series Adapter: Port 0, partition 1 Ethernet can be set to 9,614 bytes. Port 0, partition 1 iscsi offload can be set to 2,500 bytes. Port 0, partition 2 Ethernet can be set to 1,514 bytes. Port 0, partition 2 iscsi offload can be set to 9,000 bytes. Port 0, partition 3 Ethernet can be set to 4,088 bytes. Port 0, partition 4 Ethernet can be set to 9,014 bytes. 42 BC B

59 8 Setting MTU Sizes Setting the MTU Size in Windows Device Manager To confirm the new configuration: In Windows, use the ping command with the -f option (to set the do not fragment (DF) flag) and the -l <size> option (lower case letter l followed by the frame size) to verify that jumbo frame support is configured throughout the desired network path. For example: ping -f -l 8972 A.B.C.D The unfragmentable ping packet size is the desired MTU size to be checked (9,000 bytes) minus the automatically added overhead (28 bytes), or 8,972 bytes. For example: C:\> ping f -l Pinging from with 8972 bytes of data: Reply from : bytes=8972 time<1ms TTL=64 Reply from : bytes=8972 time<1ms TTL=64 Reply from : bytes=8972 time<1ms TTL=64 Reply from : bytes=8972 time<1ms TTL=64 Ping statistics for : Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms If the command does not work, you will see output similar to the following. C:\> ping f -l Pinging from with 8972 bytes of data: Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Ping statistics for : Packets: Sent = 4, Received = 0, Lost = 4 (100% loss) If there is connectivity, try 1,472-bytes for standard frames to see if the non-jumbo frame size is passing through. 43 BC B

60 8 Setting MTU Sizes Setting the MTU Size in QCC GUI Setting the MTU Size in QCC GUI To configure Ethernet Networking MTU with QCC GUI: 1. Start QCC GUI. 2. On the left pane (Host View), expand the view for the adapter you want to configure by clicking the plus sign (+), and then select the desired adapter port. 3. Click that adapter port s plus sign (+) to expand its system device instances. 4. For the desired system devices, click the plus sign (+) to expand the enabled protocols (Ethernet adapter, iscsi adapter, and FCoE adapter). 5. Select that system device s Ethernet Adapter instance. On the right pane, select the Advanced Params tab. The device s Parameter can be set in the Value column (see Figure 8-2). Figure 8-2. QCC GUI Advanced Params Tab 44 BC B

61 8 Setting MTU Sizes Setting the MTU Size in QCC GUI 6. In the Value column, select the desired value from the Jumbo Packet box (see Figure 8-3). Figure 8-3. QCC GUI Advanced Params Tab, Jumbo Packet 7. Click the Apply button. You can also use QCC to set the MTU size for each individual iscsi offload protocol enabled partition from standard (1,500 bytes) to jumbo (9,600 bytes) in the QCC iscsi Management Configurations tab (see Figure 8-4). The MTU value set does not include any extra overhead bytes (like the Ethernet settings). To configure iscsi-offload MTU with QCC GUI: 1. Start QCC GUI. 2. On the left pane (Host View), expand the view for the adapter you want to configure by clicking the plus sign (+), and then select the desired adapter port. 3. Click that adapter port s plus sign (+) to expand its system device instances. 4. For the desired system devices, click the plus sign (+) to expand the enabled protocols (Ethernet adapter, iscsi adapter, and FCoE adapter). 5. Select that system device s iscsi adapter instance. In the right pane, select the Configuration tab. 45 BC B

62 8 Setting MTU Sizes Setting the MTU Size in QCC GUI 6. Select the MTU field in the iscsi Management section (see Figure 8-4). Figure 8-4. QCC GUI Configuration Tab, iscsi Management 7. In the Value column, enter the desired value in the MTU box. Valid values are in the range of 1,5000 to 9,6000 bytes (see Figure 8-5). Figure 8-5. QCC GUI Configuration Tab, iscsi Management, MTU Size 8. Click the Apply button. 46 BC B

63 Part III Example Configurations This part of the guide provides examples of different bandwidth configurations. See the following sections: Non-DCB, 0 Weight, and 100 Maximum Bandwidth Over-subscription on page 49 Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription on page 53 Lossless FCoE over DCB, 100 Maximum Bandwidth, Over-subscription on page 57 Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription on page 60 Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription on page 65 Non-DCB, Weight, 100 Maximum Bandwidth, Over-subscription on page 70 Non-DCB, 0 Weight, 25 Maximum Bandwidth, Fixed Subscription on page 74 Non-DCB, 0 Weight, Maximum Bandwidth, Over-subscription on page 77 Lossless FCoE over DCB, Maximum Bandwidth, Over-subscription on page 81 Non-DCB, Weight, Maximum Bandwidth, Over-subscription on page 85 NOTE All bandwidths given in these examples are approximations. Protocol overhead, application send-rate variances, and other system limitations may give different bandwidth values, but the ratio relationship between the send bandwidths of the four partitions on the same port should be similar to those in the following examples. 47 BC B

64 III Example Configurations Depending on OS requirements, the QLogic QLE8400 Series Adapter traffic types for each partition could be Ethernet or iscsi offload, or FCoE offload. Ethernet can be on any of the four partitions. Up to two iscsi offloads can be on any two of the partitions; or one FCoE offload can be on any one of the partitions, plus one iscsi offload can be on any one of the remaining partitions. The partitions data flows on Windows can be a combination of Ethernet traffic and/or iscsi or FCoE traffic. The examples in the following sections show one port of a single port QLogic QLE3440/8440 Adapter or dual port QLogic QLE3442/8442 Adapter. The QLE3400 Series Adapters do not have the iscsi offload or FCoE offload traffic types, but are otherwise the same. The plots illustrated throughout this part of the guide have the components shown in Figure i. Partition Partition number (1 4) protocol i = iscsi e = Ethernet f = FCoE time Figure i. Legend for Plot Drawings 48 BC B

65 9 Non-DCB, 0 Weight, and 100 Maximum Bandwidth Over-subscription Following is an example of over-subscribed bandwidth sharing in non-data center bridging (DCB) mode. All traffic types over the four partitions of the port have an equal weight (that is, they are all set to 0 percent) and can each use the Maximum Bandwidth of the connection (in this case, 10Gbps). In addition to the Ethernet protocols being enabled on all four partitions, the iscsi offload protocol is enabled on partitions 1 and 4. The iscsi offload protocol can be enabled in any two partitions. When all of the partitions have 0 percent Relative Bandwidth Weights, each traffic flow acts as its own separate partition, each taking an equal share of the available bandwidth up to that partition s Maximum Bandwidth (which is 100 percent in this example, so the traffic flows are not limited any further). FCoE offload is not available in non-dcb mode; therefore, two iscsi offload protocols are used in this example. 49 BC B

66 9 Non-DCB, 0 Weight, and 100 Maximum Bandwidth Over-subscription Table 9-1 lists the bandwidth percentages for each port and partition, and the associated protocol. Table 9-1. Non-DCB, 0 Weight, and 100 Maximum Bandwidth Over-subscription Components Port, Partition Relative Bandwidth Weight (%) Maximum Bandwidth (%) Protocol Plot Color a Port 0, Partition 1 (P1i) iscsi offload Port 0, Partition 1 (P1e) Ethernet Port 0, Partition 2 (P2e) Ethernet Port 0, Partition 3 (P3e) Ethernet Port 0, Partition 4 (P4e) Ethernet Port 0, Partition 4 (P4i) iscsi offload a As shown in Figure 9-1. For more information, see Figure i on page BC B

67 9 Non-DCB, 0 Weight, and 100 Maximum Bandwidth Over-subscription The plot in Figure 9-1 shows how all of the partitions share a port s available send (transmit) bandwidth. Traffic flows on the same partition (such as P1i and P1e) are expanded into their own bandwidth trace for ease of understanding. The send traffic flows are independent in each partition. The individual traffic type flow rate is balanced with each of the other traffic type flow rates when traffic demands exceed the available bandwidth. Figure 9-1. Non-DCB, 0 Weight, 100 Maximum Bandwidth, Over-subscription Plot 51 BC B

68 9 Non-DCB, 0 Weight, and 100 Maximum Bandwidth Over-subscription The following paragraphs describe what is happening during each time interval (tn) shown in Figure 9-1. Starting at t0, the first partition's iscsi offload traffic flow (P1i) initially takes approximately 100 percent of the available port's transmit bandwidth when an iscsi test application (such as IOMeter) is flooding that port by itself. When P1 s Ethernet (P1e) starts to send at t1, both P1i and P1e traffic flows stabilize to half of the bandwidth (approximately 5Gbps each). Even though they are in the same partition, they share the total available bandwidth. When P2e starts to send at t2, all three traffic flows (P1i, P1e, and P2e) stabilize to one-third of the bandwidth (approximately 3.3Gbps each). In other words, the partitions equally share the available bandwidth. When P3e starts to send at t3, all four traffic flows (P1i, P1e, P2e, and P3e) stabilize to one-fourth of the bandwidth (approximately 2.5Gbps each). In other words, the partitions equally share the available bandwidth. When P4e starts to send at t4, all five traffic flows (P1i, P1e, P2e, P3e, and P4e) stabilize to one-fifth of the bandwidth (approximately 2Gbps each). In other words, the partitions equally share the available bandwidth. When P4i starts to send at t5, all six traffic flows (P1i, P1e, P2e, P3e, P4e, and P4i) stabilize to one-sixth of the bandwidth (approximately 1.65Gbps each). In other words, the partitions equally share the available bandwidth. When P1i stops sending at t6, the five currently active traffic flows (P1e, P2e, P3e, P4e, and P4i) will readjust to one-fifth th of the bandwidth (approximately 2Gbps each), equally absorbing the bandwidth that has become available. As a previously sending traffic flow stops sending (t7, t8, t9, and t10), the remaining active flows readjust to equally fill any available bandwidth. Notice the symmetry of the bandwidth allocation. No matter which traffic type is currently running, each gets an equal share with respect to the other currently transmitting traffic type flows. This configuration assumes that the application creating the transmitted traffic type flow can fill the allocated amount of bandwidth it is given; if not, the other traffic flows equally absorb the unused bandwidth. Any one of the traffic flows takes 100 percent of the available bandwidth if it is the only sending traffic flow. 52 BC B

69 10 Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription Following is an example of over-subscribed bandwidth sharing with DCB enabled that is similar to the example Non-DCB, 0 Weight, and 100 Maximum Bandwidth Over-subscription on page 49. There are four Ethernet and two iscsi offload protocols enabled with the same two traffic types in six distinct flows, with the partitions similarly configured. The difference in this example is that the iscsi offloaded traffic type is assigned to DCB priority group (PG) 2 and is lossless (that is, iscsi-type-length-value (TLV)) with an enhanced transmission selection (ETS) setting of 50 percent. The Ethernet traffic type is still assigned to the default PG0 and is lossy with an ETS setting of 50 percent. If the iscsi offload protocol traffic flows had been assigned to the same PG as the Ethernet protocol traffic flows, then the traffic bandwidth would look very similar to the Non-DCB, 0 Weight, and 100 Maximum Bandwidth Over-subscription example, since ETS would never be activated for traffic flows belonging to the same PG. 53 BC B

70 10 Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription Table 10-1 lists the bandwidth percentages for each port and partitions, and the associated protocol. Table Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription Components Port, Partition Relative Bandwidth Weight (%) Maximum Bandwidth (%) Protocol Plot Color a Port 0, Partition 1 (P1i) N/A (PG2 50%) 100 iscsi offload Port 0, Partition 1 (P1e) N/A (PG0 50%) 100 Ethernet Port 0, Partition 2 (P2e) N/A (PG0 50%) 100 Ethernet Port 0, Partition 3 (P3e) N/A (PG0 50%) 100 Ethernet Port 0, Partition 4 (P4e) N/A (PG0 50%) 100 Ethernet Port 0, Partition 4 (P4i) N/A (PG2 50%) 100 iscsi offload a As shown in Figure For more information, see Figure i on page BC B

71 10 Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription The plot in Figure 10-1 shows how the two iscsi traffic streams in PG2 work compared to the Ethernet traffic streams in PG0. The traffic in the two PGs act almost independently of each other when their aggregated traffic bandwidth demands exceed the available bandwidth, each taking its half of the ETS-managed pie. Figure Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription Plot The following paragraphs describe what is happening during each time interval (tn) shown in Figure Starting at t0, only P1i (iscsi offload) is sending, so it takes approximately 100 percent (or all of the 10Gbps bandwidth), since its Maximum Bandwidth setting is wide open at 100 percent. When P1e (Ethernet) starts to send at t1, both flows stabilize to approximately 5Gbps each (P1i in PG2 takes its DCB ETS allocated 50 percent bandwidth and P1e in PG0 takes its DCB ETS allocated 50 percent bandwidth). 55 BC B

72 10 Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription When P2e starts to send at t2, the traffic in P1i is not affected; it remains at approximately 5Gbps because it is in a different priority group (PG). Both P1e and P2e stabilize to approximately 2.5Gbps each (P1e and P2e equally share PG0 s allocated portion of the bandwidth). When P3e starts to send at t3, P1i is still unaffected and remains at approximately 5Gbps. The three Ethernet traffic types split their 50 percent of PG0 s share between themselves, which is approximately 1.65Gbps (each takes one-third of approximately 5Gbps). When P4e starts to send at t4, the four Ethernet traffic flows take one-fourth of PG0 s bandwidth (approximately 1.25Gbps each), while P1i is still unaffected and remains at approximately 5Gbps. When P4i starts to send at t5, the four Ethernet traffic flows remain the same (approximately 1.25Gbps each). The two iscsi traffic flows split PG2 s allocated bandwidth (approximately 2.5Gbps each). When P1i stops sending at t6, the traffic flows in PG0 are unaffected. P4i s share increases to all of PG2 s allocated bandwidth of approximately 5Gbps. As each of the traffic flows stop sending in PG0, the traffic flows of the remaining members of PG0 equally increase their respective shares to automatically occupy all of the available bandwidth, as described in the following paragraphs. At t7, there are three active PG0 flows (P2e, P3e, and P4e), so each gets one-third of PG0 s 5Gbps (approximately 1.65Gbps each). At t8, there are two active PG0 flows (P3e and P4e), so each gets half of PG0 s 5Gbps (approximately 2.5Gbps each). At t9, there is only one active PG0 flow (P4e), so it gets all of PG0 s bandwidth (approximately 5Gbps). Through all of this operation, the lossless iscsi flow of P4i remains at 5Gbps, since it takes all of PG2 s portion of the overall bandwidth (ETS of 50 percent). At t10, there is only one active flow after P4e stops sending. At this point, P4i gets 100 percent of all the bandwidth (approximately 10Gbps). Any one of the traffic flows takes 100 percent of the available bandwidth if it is the only sending traffic flow. 56 BC B

73 11 Lossless FCoE over DCB, 100 Maximum Bandwidth, Over-subscription The following DCB example of over-subscribed bandwidth sharing replaces one of the iscsi offloads from the Lossless iscsi-tlv over DCB, 100 Maximum Bandwidth, Over-subscription example with an FCoE offload protocol. This configuration contains three distinct traffic types in six distinct flows, with similar partition settings. The other difference between this example and the Lossless FCoE over DCB, 100 Maximum Bandwidth, Over-subscription example is that the FCoE traffic type is assigned to DCB PG1 and is lossless with an ETS setting of 50 percent, while the Ethernet and iscsi offload traffic types are now in PG0 and are both lossy with an ETS setting of 50 percent. Table 11-1 lists the bandwidth percentages for each port and partition, and the associated protocol. Table Lossless FCoE over DCB, 100 Maximum Bandwidth Over-subscription Components Port, Partition Relative Bandwidth Weight (%) Maximum Bandwidth (%) Protocol Plot Color a Port 0, Partition 1 (P1f) N/A (PG1 50%) 100 FCoE offload Port 0, Partition 1 (P1e) N/A (PG0 50%) 100 Ethernet Port 0, Partition 2 (P2e) N/A (PG0 50%) 100 Ethernet Port 0, Partition 3 (P3e) N/A (PG0 50%) 100 Ethernet Port 0, Partition 4 (P4e) N/A (PG0 50%) 100 Ethernet Port 0, Partition 4 (P4i) N/A (PG0 50%) 100 iscsi offload a As shown in Figure For more information, see Figure i on page BC B

74 11 Lossless FCoE over DCB, 100 Maximum Bandwidth, Over-subscription The plot in Figure 11-1 shows how the first partition's FCoE traffic PG1 acts compared to the other traffic types in PG0. Figure DCB FCoE 100 Maximum Bandwidth Over-subscription Plot The following paragraphs describe what is happening during each time interval (tn) shown in Figure Starting at t0, only P1f (FCoE offload) is sending, so it takes approximately 100 percent (or all) of the 10Gbps bandwidth. When P1e (Ethernet) starts to send at t1, both flows stabilize to approximately 5Gbps of bandwidth each (P1f in PG1 takes 50 percent and P1e in PG0 takes 50 percent). 58 BC B

75 11 Lossless FCoE over DCB, 100 Maximum Bandwidth, Over-subscription When P2e starts to send at t2, the traffic in P1f is not affected it remains at approximately 5Gbps because it is in a different PG. Both P1e and P2e stabilize to approximately 2.5Gbps each. P1e and P2e equally share PG0 s portion of the bandwidth, so they get approximately 2.5Gbps each, which is computed by: ETS of 50 percent 10G bandwidth / 2 traffic flows When P3e starts to send at t3, P1f is still unaffected and remains at approximately 5Gbps. The three Ethernet traffic types take one-third of their 50 percent of PG0 s share between themselves, which is approximately 1.65Gbps (each takes one-third of 5Gbps). When P4e starts to send at t4, the four Ethernet traffic flows take one-fourth of PG0 s bandwidth (approximately 1.25Gbps each). P1f is still unaffected and remains at 5Gbps. When P4i starts to send at t5, the four Ethernet traffic flows plus the new iscsi traffic flow take one-fifth of PG0 s bandwidth (approximately 1Gbps each). P1f is still unaffected and remains at 5Gbps. When P1f stops sending at t6, the five traffic flows in PG0 take all of the ports bandwidth. Now their one-fifth of PG0 s bandwidth doubles to approximately 2Gbps each; the available bandwidth has increased from 5Gbps to 10Gbps. As each traffic flow stops sending in PG0, the remaining member traffic flows equally increase their respective shares to automatically occupy all of the available bandwidth, as described in the following paragraphs. At t7, there are four active PG0 flows, so each gets one-fourth of PG0 s 10Gbps (approximately 2.5Gbps). At t8, there are three active PG0 flows, so each gets one-third of PG0 s 10Gbps (approximately 3.3Gbps). At t9, there are two active PG0 flows, so each gets half of PG0 s 10Gbps (approximately 5Gbps). At t10, there is only one active PG0 flow (P4i), so it gets 100 percent of PG0 s 10Gbps (approximately 10Gbps). Any one of the traffic flows takes 100 percent of the available bandwidth if it is the only sending traffic flow. 59 BC B

76 12 Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription The following DCB example of over-subscribed bandwidth sharing is similar to Lossless FCoE over DCB, 100 Maximum Bandwidth, Over-subscription on page 57, except that the iscsi offload traffic type is lossless and in PG2 with an ETS setting of 20 percent. The FCoE offload traffic type is still lossless in PG1 with an ETS setting of 50 percent. The Ethernet traffic type is still lossy in default PG0, but with an ETS setting of 30 percent. Table 12-1 lists the bandwidth percentages for each port and partition, and the associated protocol. Table Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription Components Port, Partition Relative Bandwidth Weight (%) Maximum Bandwidth (%) Protocol Plot Color a Port 0, Partition 1 (P1f) N/A (PG1 50%) 100 FCoE offload Port 0, Partition 1 (P1e) N/A (PG0 30%) 100 Ethernet Port 0, Partition 2 (P2e) N/A (PG0 30%) 100 Ethernet Port 0, Partition 3 (P3e) N/A (PG0 30%) 100 Ethernet Port 0, Partition 4 (P4e) N/A (PG0 30%) 100 Ethernet Port 0, Partition 4 (P4i) N/A (PG2 20%) 100 iscsi offload a As shown in Figure For more information, see Figure i on page BC B

77 12 Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription The plot in Figure 12-1 shows how the first partition's FCoE traffic PG1 acts compared to Ethernet traffic in PG0 and iscsi traffic in PG2. Figure Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription Plot The following paragraphs describe what is happening during each time interval (tn) shown in Figure Starting at t0, only P1f (FCoE offload) is sending, so it takes approximately 100 percent (or all) of the 10Gbps bandwidth. When P1e (Ethernet) starts to send at t1: P1f in PG1 takes 62.5 percent (approximately 6.25Gbps) of the bandwidth, which is computed by: FCoE ETS of 50 % 10G BW / (FCoE ETS of 50 % + Ethernet ETS of 30%) 61 BC B

78 12 Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription P1e in PG0 takes 37.5 percent (approximately 3.75Gbps) of the bandwidth, which is computed by: Ethernet ETS of 30% 10G BW / (FCoE ETS of 50% + Ethernet ETS of 30%) When P2e starts to send at t2, the traffic in P1f is not affected; it remains at approximately 6.25Gbps since it is in a different PG. Both P1e and P2e stabilize to approximately 1.875Gbps each; that is, P1e and P2e equally share PG0 s portion of the bandwidth, so they each get half of the 3.75Gbps (approximately 1.875Gbps each), which is computed by: Ethernet ETS of 30% 10G BW / (2 traffic flows (FCoE ETS of 50% + Ethernet ETS of 30%)) When P3e starts to send at t3, P1f is still unaffected and remains at approximately 6.25Gbps. The three Ethernet traffic types split their 30 percent of PG0 s share between themselves, which is approximately 1.25Gbps (each takes one-third of 3.75Gbps), which is computed by: Ethernet ETS of 30% 10G BW / (3 traffic flows (FCoE ETS of 50% + Ethernet ETS of 30%)) When P4e starts to send at t4, the four Ethernet traffic flows take one-fourth of PG0 s bandwidth (approximately Gbps each). P1f is still unaffected and remains at approximately 6.25Gbps. The Ethernet share is computed by: Ethernet ETS of 30% 10G BW / (4 traffic flows (FCoE ETS of 50% + Ethernet ETS of 30%)) When P4i starts to send at t5: The four Ethernet traffic flows takes their share of PG0's bandwidth of one-fourth of approximately 3Gbps (approximately 0.75Gbps each), which is computed by: Ethernet ETS of 30% 10G BW / (4 traffic flows (FCoE ETS of 50% + Ethernet ETS of 30% + iscsi-tlv ETS of 20%)) The new lossless iscsi-tlv traffic flow takes its share of PG2 s bandwidth of approximately 2Gbps, which is computed by: iscsi-tlv ETS of 20% 10G BW / (FCoE ETS of 50% + Ethernet ETS of 30% + iscsi-tlv ETS of 20%) In addition, P1f drops down to its share of PG1 s bandwidth at 5Gbps, which is computed by: FCoE ETS of 50% 10G BW / (FCoE ETS of 50% + Ethernet ETS of 30% + iscsi-tlv ETS of 20%) 62 BC B

79 12 Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription When P1f stops sending at t6: The four Ethernet traffic flows in PG0 take their share of PG0 s bandwidth of one-fourth of approximately 6Gbps (or approximately 1.5Gbps each), which is computed by: Ethernet ETS of 30% 10G BW / (4 traffic flows (Ethernet ETS of 30% + iscsi-tlv ETS of 20%)) In addition, the lossless iscsi-tlv traffic flow takes it s share of PG2 s bandwidth of approximately 4Gbps, which is computed by: iscsi-tlv ETS of 20% 10G BW / (Ethernet ETS of 30% + iscsi-tlv ETS of 20%) As each traffic flow stops sending in PG0, the remaining traffic flows increase their respective shares to automatically occupy all of the available bandwidth, as described in the following paragraphs. At t7, there are three active Ethernet PG0 flows, which equates to approximately 2Gbps each, which is computed by: Ethernet ETS of 30% 10G BW / (3 (Ethernet ETS of 30% + iscsi-tlv ETS of 20%) In addition, the lossless iscsi-tlv PG2 flow of approximately 4Gbps stays the same, which is computed by: iscsi-tlv ETS of 20% 10G BW / (Ethernet ETS of 30% + iscsi-tlv ETS of 20%) At t8, there are two active Ethernet PG0 flows, which equates to approximately 3Gbps each, which is computed by: Ethernet ETS of 30% 10G BW / (2 (Ethernet ETS of 30% + iscsi-tlv ETS of 20%)) In addition, the lossless iscsi-tlv PG2 flow of approximately 4Gbps remains unaffected, which is computed by: iscsi-tlv ETS of 20% 10G BW / (Ethernet ETS of 30% + iscsi-tlv ETS of 20%) 63 BC B

80 12 Lossless iscsi-tlv and FCoE over DCB, 100 Maximum Bandwidth, Over-subscription At t9, there is one active Ethernet PG0 flow, which equates to approximately 6Gbps, which is computed by: Ethernet ETS of 30% 10G BW / (Ethernet ETS of 30% + iscsi-tlv ETS of 20%) In addition, the lossless iscsi-tlv PG2 flow of approximately 4Gbps remains the same, which is computed by: iscsi-tlv ETS of 20% 10G BW / (Ethernet ETS of 30% + iscsi-tlv ETS of 20%) At t10, there is only one active PG2 flow (P4i), so it gets 100 percent of the bandwidth (approximately 10Gbps). Any one of the traffic flows takes 100 percent of the available bandwidth if it is the only sending traffic flow. 64 BC B

81 13 Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription Following is an example of over-subscribed bandwidth sharing in non-dcb mode where all four partitions of the port have their weight set to 25 percent and can individually use the Maximum Bandwidth of the connection (that is, 10Gbps). In addition to Ethernet protocol being enabled on all four partitions, the iscsi offload protocol is enabled on partitions 1 and 4. By setting the partition s relative bandwidth weights to 25 percent, each partition s traffic flows are combined together: P1's iscsi (P1i) and Ethernet (P1e) are combined into one traffic flow on partition 1 P2's Ethernet (P2e) occupies all of partition 2 P3's Ethernet (P3e) occupies all of partition 3 P4's iscsi (P4i) and Ethernet (P4e) are combined into one traffic flow on partition 4 These flows are contained in their respective partition, while each partition takes an equal share of the available bandwidth. The traffic flows within that partition can only expand into that partition's allocated by weight portion. 65 BC B

82 13 Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription Table 13-1 lists the bandwidth percentages for each port and partition, and the associated protocol. Table Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription Components Port, Partition Relative Bandwidth Weight (%) Maximum Bandwidth (%) Protocol Plot Color a Port 0, Partition 1 (P1i) iscsi offload Port 0, Partition 1 (P1e) Ethernet Port 0, Partition 2 (P2e) Ethernet Port 0, Partition 3 (P3e) Ethernet Port 0, Partition 3 (P4e) Ethernet Port 0, Partition 4 (P4i) iscsi Offload a As shown in Figure For more information, see Figure i on page BC B

83 13 Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription The plot in Figure 13-1 shows how each traffic type flow must remain within a partition s share of a port s available send bandwidth; that is, if there are two different traffic type flows (such as P1i and P1e) in a single partition, they are combined, as if one flow, for determining the amount of bandwidth allocated to them. Figure Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription Plot The following paragraphs describe what is happening during each time interval (tn) shown in Figure Starting at t0, the first partition s iscsi offload traffic flow (P1i) initially takes approximately 100 percent of the available port's send (transmit) bandwidth when an iscsi test application is flooding that port by itself. When P1's Ethernet (P1e) starts to send at t1, both P1i and P1e stabilize to approximately 5Gbps each. Even though they are in the same partition, they share the total available bandwidth, because no other partition's traffic flow is active. 67 BC B

84 13 Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription When P2e starts to send at t2: The traffic flows in P1 (P1i and P1e) reduce to approximately 2.5Gbps each, which is computed by: P1's weight of 25% / (2 traffic flows (P1's weight of 25% + P2's weight of 25%)) At the same time, the P2 traffic flow (P2e) takes approximately 5Gbps, which is computed by: P2's weight of 25% / (1 traffic flow (P1's weight of 25% + P2's weight of 25%)) P2e s share of the traffic flow is greater than P1e s because the bandwidth is initially split by partition, and then traffic flows within each individual partition. In addition, P1 has two traffic flows and P2 has only one. When P3e starts to send at t3: The two traffic flows in P1 (P1i and P1e) are further reduced to approximately 1.7Gbps, which is computed by: P1's weight of 25% / (2 traffic flows (P1's weight of 25% + P2's weight of 25% + P3's weight of 25%)) At the same time, P2e and P3e have each stabilized at approximately 3.3Gbps. P2e s speed is computed by: P2's weight of 25% / (P1's weight of 25% + P2's weight of 25% + P3's weight of 25%) P3e s speed is computed by: P3's weight of 25% / (P1's weight of 25% + P2's weight of 25% + P3's weight of 25%) When P4e starts to send at t4, the three single partition traffic flows (P2e, P3e, and P4e) stabilize to one-fourth of 10Gbps (or approximately 2.5Gbps each). At the same time, the P1 partition shares its allocated bandwidth between its two users (P1i and P1e), so that each gets half of the allocated one-fourth of 10Gbps or one-eighth, which is approximately 1.25Gbps for each. When P4i starts to send at t5, the two single traffic flows (P2e and P3e) remain at approximately 2.5Gbps each. In addition, P1 partition s traffic flows (P1i and P1e) are still each getting approximately 1.25Gbps. P4 s allocated bandwidth is now split into two traffic flows (P4e and P4i), which means each flow gets approximately 1.25Gbps. 68 BC B

85 13 Non-DCB, 25 Weight, 100 Maximum Bandwidth, Over-subscription When P1i stops sending at t6, only partition P1 s traffic flow (P1e) readjusts upwards to approximately 2.5Gbps. All the other partitions traffic flows remain the same. When P1e stops sending at t7, the other traffic flows (P2e and P3e) readjust to approximately 3.3Gbps and approximately 1.65Gbps, respectively, for partition P4 s shared P4e and P4i traffic flows. When P2e stops sending at t8, partition P3 s single traffic flow (P3e) readjusts to approximately 5Gbps. Partition P4 s shared P4e and P4i traffic flows increase to half that (approximately 2.5Gbps each). When P3e stops sending at t9, the remaining partition (P4) now receives all of the available bandwidth. Its two traffic flows (P4e and P4i) equally share the bandwidth, approximately 5Gbps each. When P4e stops sending at t10, the remaining traffic flow (P4i) now receives all of the available bandwidth (approximately 10Gbps). If there is only one flow in a partition, it takes all of the bandwidth allocated for that partition. The difference between setting all four partition's relative bandwidth weights to 0 percent and setting them to all 25 percent is as follows: The 0 percent bandwidth setting causes the transit bandwidth to be shared between all active traffic flows. The logic works similarly to DCB mode, when all traffic types are in the same PG. The 25 percent bandwidth setting causes the transit bandwidth to be shared between the active sending partitions first, followed by the active sending traffic type flows, in a two-step process. If there is only one traffic flow in each partition, then the results are similar to setting each partition s Relative Bandwidth Weight to 0 percent, since the single traffic flow does not share a partition's bandwidth with another traffic type. 69 BC B

86 14 Non-DCB, Weight, 100 Maximum Bandwidth, Over-subscription Following is an example of weighted over-subscribed bandwidth sharing with different weights assigned to each partition in non-dcb mode. In this example, each partition takes the Maximum Bandwidth when no other partition is active. In addition, as each partition starts and stops sending, the amount of bandwidth is shared as an approximate ratio of the currently sending partitions weight values. Table 14-1 lists the bandwidth percentages for each port and partition, and the associated protocol. Table Non-DCB, Weight, 100 Maximum Bandwidth, Over-subscription Components Port, Partition Relative Bandwidth Weight (%) Maximum Bandwidth (%) Protocol Plot Color a Port 0, Partition 1 (P1i) iscsi offload Port 0, Partition 1 (P1e) Ethernet Port 0, Partition 2 (P2e) Ethernet Port 0, Partition 3 (P3e) Ethernet Port 0, Partition 4 (P4e) Ethernet Port 0, Partition 4 (P4i) iscsi offload a As shown in Figure For more information, see Figure i on page BC B

87 14 Non-DCB, Weight, 100 Maximum Bandwidth, Over-subscription Figure 14-1 illustrates this over-subscription example. Figure Non-DCB, Weight, 100 Maximum Bandwidth, Over-subscription Plot The following paragraphs describe what is happening during each time interval (tn) shown in Figure The first partition's traffic flow (P1i) initially takes approximately 100 percent of the available bandwidth at t0 when an iscsi test application is sending traffic out that port by itself. When P1e starts sending Ethernet traffic at t1, the two active traffic flows have equal weights with respect to each other, so they are allocated half of the total bandwidth available (approximately 10Gbps) to partition P1, which equates to approximately 5Gbps each for P1i and P1e. 71 BC B

88 14 Non-DCB, Weight, 100 Maximum Bandwidth, Over-subscription When P2e starts sending at t2, the partition's Relative Bandwidth Weights come into effect. Partition P1 has a weight of 10 percent, while P2 has twice as much at 20 percent. Therefore, P1 s two sending traffic flows are reduced to half of the partition's assigned one-third of the bandwidth, as computed by: P1's weight of 10% / (P1's weight of 10% + P2's weight of 20%) This equation is equal to approximately 1.65Gbps each for P1i and P1e. P2e starts at approximately 6.7Gbps (its relative weight is 20 percent to 30 percent of the total active weights). P2e is not decreased by half, since it is the only traffic flow on partition P2. When partition P3e starts sending Ethernet traffic at t3 with a relative weight of 30 percent, it takes approximately 5Gbps (30/60 of 10Gbps). P2e drops to approximately 3.3Gbps (20/( )). Partition P1's total drops to approximately 1.65Gbps (10/( )); therefore, P1i and P1e each get half of that (approximately 0.825Gbps each). When P4e starts (40 percent relative weight) at t4, it takes approximately 4Gbps (40/( )). The three other partitions send traffic drops. Partition P1 is reduced to approximately 1Gbps (10/( )), so P1i and P1e split that for approximately 0.5Gbps. Partition P2 drops to approximately 2Gbps (20/( )); since there is only one send traffic flow (P2e), it takes all of that assigned bandwidth. Partition P3 (with its single traffic flow P3e) drops to approximately 3Gbps (30/( )). When the second traffic flow on partition P4 (P4i) starts at t5, the two flows (P4e and P4i) on the same partition (P4) split the partition's assigned bandwidth of approximately 4Gbps, so that each gets approximately 2Gbps. The other transmit traffic on the other three partitions remains the same. When P1i stops at t6, the remaining traffic flow on partition P1 (P1e) absorbs that partition's share of the send bandwidth to approximately 1Gbps. The remaining traffic flows on the other three partitions are unaffected. When P1e stops at t7, the other partitions adjust slightly upwards to fill the newly available bandwidth. Partition P2 (and its traffic flow P2e) increases to approximately 2.2Gbps (20/( )). Partition P3 (and its traffic flow P3e) increases to approximately 3.3Gbps (30/( )). Partition P4 increases to approximately 4.5Gbps (40/( )), which means that P4e and P4i split that speed for approximately 2.25Gbps each. When P2e stops at t8, the other partitions adjust upwards again to fill the newly available bandwidth. Partition P3 (and its traffic flow P3e) increases to approximately 4.3Gbps (30/(30+40)). Partition P4 increases to approximately 5.7Gbps (40/(30+40)), which means that P4e and P4i split that speed for approximately 2.85Gbps each. 72 BC B

89 14 Non-DCB, Weight, 100 Maximum Bandwidth, Over-subscription When P3e stops at t9, partition P4 s share raises to approximately 100 percent of the bandwidth (40/40), so its two traffic flows (P4e and P4i) split this speed for approximately 5Gbps each. When P4e stops at t10, the only remaining traffic flow (P4i) takes all of the bandwidth at approximately 10Gbps. 73 BC B

90 15 Non-DCB, 0 Weight, 25 Maximum Bandwidth, Fixed Subscription This section provides an example of partition settings that have all of the Relative Bandwidth Weights set to 0 percent and the Maximum Bandwidths set to 2.5Gbps. Since the total Maximum Bandwidth values are set to exactly 100 percent (that is, an over-subscription situation cannot occur), the traffic flows in each partition share that partition's bandwidth allocation with respect to its overall Maximum Bandwidth ceiling, and the relative bandwidth weights are never used. Table 15-1 lists the bandwidth percentages for each port and partition, and the associated protocol. Table Non-DCB, 0 Weight, 25 Maximum Bandwidth, Fixed Subscription Components Port, Partition Relative Bandwidth Weight (%) Maximum Bandwidth (%) Protocol Plot Color a Port 0, Partition 1 (P1i) 0 25 iscsi offload Port 0, Partition 1 (P1e) 0 25 Ethernet Port 0, Partition 2 (P2e) 0 25 Ethernet Port 0, Partition 3 (P3e) 0 25 Ethernet Port 0, Partition 4 (P4e) 0 25 Ethernet Port 0, Partition 4 (P4i) 0 25 iscsi offload a As shown in Figure For more information, see Figure i on page BC B

91 15 Non-DCB, 0 Weight, 25 Maximum Bandwidth, Fixed Subscription The plot in Figure 15-1 shows how the four partition s outgoing (send) traffic flows are independent of each other. Unlike the examples in the previous sections, none of the partitions (or their associated traffic flows) take more than their designated bandwidth portion; the total bandwidth of all four partitions of the port is equal to or less than the total available bandwidth of the port. In this example, each partition takes only approximately 25 percent (or 2.5Gbps) of the total available bandwidth when their test application starts to transmit traffic. Furthermore, if a partition has more than one active traffic flow, these flows share that partition's allowed bandwidth. Unused port bandwidth is not re-allocated to any partition above its own Maximum Bandwidth setting. Figure Non-DCB, 0 Weight, 25 Maximum Bandwidth, Fixed Subscription Plot 75 BC B

92 15 Non-DCB, 0 Weight, 25 Maximum Bandwidth, Fixed Subscription The following paragraphs describe what is happening during each time interval (tn) shown in Figure When P1i starts to send traffic at t0, it only takes the subscribed 25 percent of the 10Gbps bandwidth available (approximately 2.5Gbps). When P1e starts to send at t1, it shares partition P1 s 25 percent bandwidth with P1i. Each partition is allocated approximately 1.25Gbps, and neither expands into the unused approximately 7.5Gbps of remaining bandwidth. When P2e starts to send at t2, it only takes its partition s subscribed approximately 2.5Gbps of bandwidth and does not affect either of partition P1 s sending traffic flows. When P3e starts to send at t3, it only takes its partition s subscribed approximately 2.5Gbps of bandwidth and does not affect P2e or either of partition P1 s transmitting traffic flows. When P4e starts to send at t4, it only takes its partition s subscribed approximately 2.5Gbps of bandwidth and does not affect any of the other partitions sending traffic flows. When P4i starts to send at t5, it shares partition P4 s 25 percent of bandwidth with P4e. Each partition is allocated approximately 1.25Gbps of bandwidth. The other partitions are not affected. When P1i stops sending at t6, it releases its 12.5 percent share of the bandwidth. The other remaining partition P1 traffic flow (P1e) increases to 2.5Gbps bandwidth. The other traffic flows are unaffected. When P1e stops sending at t7, there are only three partitions, but each is still assigned only 25 percent of the overall bandwidth. The other traffic flows (P2e, P3e, P4e, and P4i) do not change. When P2e stops sending at t8, there is no change to the other traffic flows. When P3e stops sending at t9, there is no change to the other traffic flows. When P4e stops sending at t10, the remaining traffic flow on P4 (P4i) absorbs the freed 12.5 percent of partition P4 s allocated bandwidth and increases its speed to 2.5Gbps. Each partition's flows on the same port are logically isolated from the others, as if they were on separate ports. Stopping or restarting a partitions send traffic flow does not affect its fellow partitions send traffic flows, except where the flows are on the same partition. In this case, the partitions take only the freed bandwidth for their own partition. 76 BC B

93 16 Non-DCB, 0 Weight, Maximum Bandwidth, Over-subscription This example shows partitions with all of the relative bandwidth weights set to 0, but with the partitions partially over-subscribing the available bandwidth in an equal fashion. Two of the partitions are set to use 10 percent (or 1Gbps) each of bandwidth and the other two (those with the offload protocols enabled) are set to use 80 percent (or 8Gbps) of the connection's bandwidth, thus oversubscribing the connection by 80 percent. Table 16-1 lists the bandwidth percentages for each port and partition, and the associated protocol. Table Non-DCB, 0 Weight, Maximum Bandwidth Over-subscription Components Port, Partition Relative Bandwidth Weight (%) Maximum Bandwidth (%) Protocol Plot Color a Port 0, Partition 1 (P1i) 0 80 iscsi offload Port 0, Partition 1 (P1e) 0 80 Ethernet Port 0, Partition 2 (P2e) 0 10 Ethernet Port 0, Partition 3 (P3e) 0 10 Ethernet Port 0, Partition 4 (P4e) 0 80 Ethernet Port 0, Partition 4 (P4i) 0 80 iscsi offload a As shown in Figure For more information, see Figure i on page BC B

94 16 Non-DCB, 0 Weight, Maximum Bandwidth, Over-subscription This is a combination example of a fixed subscription (three of the partitions weight totals 100 percent), but all partitions weights total 180 percent. When all four partitions (or at least the two larger partitions) are running traffic, they share the space with each other, up to their partition Maximum Bandwidth values; otherwise, they act as if they are independent connections. Figure 16-1 illustrates this example. Figure Non-DCB, 0 Weight, Maximum Bandwidth Over-subscription Plot 78 BC B

95 16 Non-DCB, 0 Weight, Maximum Bandwidth, Over-subscription The following paragraphs describe what is happening during each time interval (tn) shown in Figure The first partition's traffic flow (P1i) initially takes its Maximum Bandwidth designated (approximately 8Gbps) when the test application starts to transmit traffic at t0 to that port by itself, not expanding into the remaining unused approximately 2Gbps of bandwidth. When the second traffic flow on the first partition (P1e) starts to send at t1, the two active traffic flows on the same partition share its approximately 8Gbps bandwidth for approximately 4Gbps each. The extra 2Gbps remains unused. When the third traffic flow (P2e) starts sending at t2, it only takes its partition s Maximum Bandwidth allowed (approximately 1Gbps). Partition P1 s two traffic flows are unaffected. The extra 1Gbps is unused. When the fourth traffic flow (P3e) starts sending at t3, it only takes its partition s Maximum Bandwidth allowed (approximately 1Gbps). Partition P1 s two traffic flows and the traffic flow on partition P2 (P2e) are still unaffected. Now all 10Gbps of bandwidth are in use. When P4e starts to send traffic at t4, the condition is now oversubscribed, since P2e and P3e use approximately only 1Gbps of their allocated 2Gbps, which is computed by: 10Gbps /5 equally weighted traffic flows Therefore, approximately 8Gbps is free for the other three traffic flows. The remaining traffic flows (P1i, P1e, and P4e) are then allocated approximately 2.6Gbps each, which is computed by: 8Gbps / 3 equally weighted traffic flows When P4i starts to send traffic at t5, it shares the available bandwidth within its maximums with the other traffic flows. P2e and P3e are still using only approximately 1Gbps of their allocated 1.6Gbps, which is computed by: 10 Gbps / 6 equally weighted traffic flows Therefore, approximately 8Gbps is free for the other four traffic flows. These four partitions (P1i, P1e, P4e, and P4i) are allocated approximately 2Gbps each, which is computed by: 8 Gbps / 4 equally weighted traffic flows When P1i stops at t6, it releases its bandwidth to the available pool. Since P2e and P3e are capped by their Maximum Bandwidth value to 1Gbps, the other three traffic flows (P1e, P4e, and P4i) automatically take equal shares and increase their bandwidth to approximately 2.6Gbps each. 79 BC B

96 16 Non-DCB, 0 Weight, Maximum Bandwidth, Over-subscription When P1e stops sending at t7, P4e and P4i take the extra available bandwidth and increase their bandwidth to approximately 4Gbps each. Both P2e and P3e are unaffected and continue sending at approximately 1Gbps of bandwidth each. When P2e stops sending at t8, P4e and P4i cannot use the freed up bandwidth since they are both in partition P4, which has a Maximum Bandwidth of 8Gbps. Therefore, none of the traffic flows increase their sending rates, and this unused bandwidth is ignored. When P3e stops sending at t9, the same condition is still in effect. Therefore, none of the remaining active traffic flows increase their sending rates to use this extra bandwidth. Finally, P4e stops at t10, allowing its companion traffic flow (P4i) to increase to approximately 8Gbps of bandwidth, which is partition P4 s maximum. The remaining approximately 2Gbps of bandwidth is unassigned. 80 BC B

97 17 Lossless FCoE over DCB, Maximum Bandwidth, Over-subscription The following example is the same as the previous example ( Non-DCB, 0 Weight, Maximum Bandwidth, Over-subscription on page 77), but with FCoE in the first partition. Additionally, the FCoE traffic flow is lossless and in DCB PG1 with an ETS of 50 percent, and the other traffic flows (Ethernet and iscsi offload) are lossy and in PG0 with an ETS of 50 percent. Table 17-1 lists the bandwidth percentages for each port and partition, and the associated protocol. Table Lossless FCoE over DCB, Maximum Bandwidth, Over-subscription Components Port, Partition Relative Bandwidth Weight (%) Maximum Bandwidth (%) Protocol Plot Color a Port 0, Partition 1 (P1f) N/A (PG1 50%) 80 FCoE offload Port 0, Partition 1 (P1e) N/A (PG0 50%) 80 Ethernet Port 0, Partition 2 (P2e) N/A (PG0 50%) 10 Ethernet Port 0, Partition 3 (P3e) N/A (PG0 50%) 10 Ethernet Port 0, Partition 4 (P4e) N/A (PG0 50%) 80 Ethernet Port 0, Partition 4 (P4i) N/A (PG0 50%) 80 iscsi offload a As shown in Figure For more information, see Figure i on page BC B

98 17 Lossless FCoE over DCB, Maximum Bandwidth, Over-subscription This configuration is a combination example of a fixed subscription (three of the partitions total 100 percent), but all four partitions total 180 percent. When all four partitions (or at least the last two partitions) are running traffic, they share the space with each other, up to their partition Maximum Bandwidth values and their PG s ETS settings; otherwise, they act as if they are independent connections. Figure 17-1 illustrates this example. Figure Lossless FCoE over DCB, Maximum Bandwidth, Over-subscription Plot 82 BC B

99 17 Lossless FCoE over DCB, Maximum Bandwidth, Over-subscription The following paragraphs describe what is happening during each time interval (tn) shown in Figure The first partition's traffic flow (P1f) initially takes its Maximum Bandwidth (approximately 8Gbps) when the test application starts to transmit traffic at t0 to that port by itself, not expanding into the remaining unused approximately 2Gbps of bandwidth. When the second traffic flow on the first partition (P1e) starts to send at t1, the two active traffic flows on the same partition share its approximately 8Gbps of bandwidth (approximately 4Gbps each). ETS does not take effect, since the traffic in PG0 and PG1 are still less than the amount prescribed by their respective ETS values. When the third traffic flow (P2e) starts sending at t2, it only takes its partition s Maximum Bandwidth allowed (approximately 1Gbps). Partition P1 s two traffic flows are unaffected and the unassigned 1Gbps of bandwidth remains free. When the fourth traffic flow (P3e) starts sending at t3, it only takes its partition s Maximum Bandwidth allowed (approximately 1Gbps). The first partition's two traffic flows readjust so that PG0 does not get more than 50 percent of the overall bandwidth, that is: PG0's P1e + P2e + P3e = 40% + 10% +10% The sum of this equation is greater than 50 percent. The P1e traffic flow is reduced to 30 percent (approximately 3Gbps). The P1f traffic flow (in PG1) is adjusted upwards to 50 percent (approximately 5Gbps). But the total on P1 is still capped at 8Gbps of bandwidth. When P4e starts to send traffic at t4, it equally shares PG0 s ETS assigned bandwidth with P1e, P2e, and P3e. However, P2e and P3e use only approximately 1Gbps of their allocated 1.25Gbps of bandwidth, as computed by: 5 Gbps / 4 equally weighted traffic flows Therefore, approximately 3Gbps of bandwidth is free for the other two traffic flows (P1e and P4e), as computed by: 5 Gbps available 2 GBps assigned to P2e and P3e P1e and P4e are both allocated approximately 1.5Gbps of bandwidth each, as computed by: 3 Gbps / 2 equally weighted traffic flows P1f is in PG1 so it is unaffected and continues sending at approximately 5Gbps. 83 BC B

100 17 Lossless FCoE over DCB, Maximum Bandwidth, Over-subscription When P4i starts to send traffic at t5, it also equally shares PG0 s bandwidth, as computed by: 5 Gbps / 5 equally weighted traffic flows Therefore, P1e, P2e, P3e, P4e, and P4i all send at approximately 1Gbps of bandwidth. P1f in PG1 is still unaffected and keeps sending at approximately 5Gbps. When P1f stops at t6, it releases all of PG1 s bandwidth to the available pool. Since P2e and P3e are capped by their Maximum Bandwidth value to 1Gbps, the other three traffic flows (P1e, P4e, and P4i) automatically take equal shares of the 8Gbps and increase their bandwidth to approximately 2.6Gbps each. When P1e subsequently stops sending at t7, P4e and P4i take the extra available bandwidth and increase their sending rates to approximately 4Gbps each. P2e and P3e are unaffected and continue sending at approximately 1Gbps each. When P2e stops sending at t8, P4e and P4i cannot use the freed up bandwidth since they are both in partition P4, which has a Maximum Bandwidth of 8Gbps. Therefore, none of the traffic flows increase their sending rates, and the unused bandwidth is ignored. When P3e stops sending at t9, the same condition is still in effect. Therefore, none of the remaining active traffic flows increase their sending rates to use the extra bandwidth. P4e stops at t10, which allows its companion traffic flow (P4i) to increase to approximately 8Gbps of bandwidth, which is partition P4 s maximum. The remaining approximately 2Gbps of bandwidth is unassigned. 84 BC B

101 18 Non-DCB, Weight, Maximum Bandwidth, Over-subscription This example shows partitions over-subscribing the available bandwidth with different relative bandwidth weights and different Maximum Bandwidths. Two of the partitions are set to use 15 percent (or 1.5Gbps) each of bandwidth, and both of their weights are set to 10 percent. The other two partitions are set to use 80 percent (or 8Gbps) of the connection's bandwidth, and both of their Relative Bandwidth Weights are set to 40 percent. The total is oversubscribing the connection by 90 percent, which is computed by: = = 90 Table 18-1 lists the bandwidth percentages for each port and partition, and the associated protocol. Table Non-DCB, Weight, Maximum Bandwidth Over-subscription Components Port, Partition Relative Bandwidth Weight (%) Maximum Bandwidth (%) Protocol Plot Color a Port 0, Partition 1 (P1i) iscsi offload Port 0, Partition 1 (P1e) Ethernet Port 0, Partition 2 (P2e) Ethernet Port 0, Partition 3 (P3e) Ethernet Port 0, Partition 4 (P4e) Ethernet Port 0, Partition 4 (P4i) iscsi offload a As shown in Figure For more information, see Figure i on page BC B

102 18 Non-DCB, Weight, Maximum Bandwidth, Over-subscription This example is a combination of different weights and Maximum Bandwidths. Figure 18-1 illustrates this over-subscription example. Figure Non-DCB, Weight, Maximum Bandwidth, Over-subscription Plot The following paragraphs describe what is happening during each time interval (tn) shown in Figure The first partition's traffic flow (P1i) initially takes its maximum designated approximately 8Gbps of bandwidth when the test application starts to transmit traffic at t0 to that port by itself, not expanding into the remaining unused approximately 2Gbps of bandwidth. When the second traffic flow on the first partition (P1e) starts to send at t1, the two active traffic flows on the same partition share its Maximum Bandwidth (approximately 8Gbps) for approximately 4Gbps each. 86 BC B

103 18 Non-DCB, Weight, Maximum Bandwidth, Over-subscription When the third traffic flow (P2e) starts sending at t2, it could take up to 20 percent or 2Gbps of bandwidth (10/(40+10)). However, since its maximum is set to 15 percent, it takes only 1.5Gbps (the Maximum Bandwidth setting caps its bandwidth to that value). The P1 s two traffic flows remain at approximately 4Gbps of bandwidth each. The extra 0.5Gbps of bandwidth is unused. When the fourth traffic flow (P3e) starts sending at t3, it could take up to 16.7 percent of bandwidth, or approximately 1.7Gbps (10/( )). However, since its Maximum Bandwidth is set to 15 percent, it takes only 1.5Gbps. P2e continues to take only 1.5Gbps of bandwidth. P1 s two traffic flows are normally reduced to 66.7 percent (or approximately 6.7Gbps of bandwidth each: 40/( ), but since there is 7Gbps remaining (10 (2 1.5)), it takes its minimum plus the extra 0.3Gbps, so each traffic flow gets 3.5Gbps of bandwidth. When P4e starts to send traffic at t4, the traffic needs are oversubscribed, so the available bandwidth is redistributed based on each partition's individual weights and maximums settings. P2e and P3e use 10 percent each (10/( )), so their traffic flows are reduced to approximately 1Gbps (which is now below their Maximum Bandwidth setting). Therefore, approximately 8Gbps of bandwidth is still free for the other traffic flows. The two partitions on P1 share the traffic flow they are allocated, which is approximately 4Gbps of bandwidth (or 2Gbps each: 40/(2*( )). P4e also takes its allocated amount, which is approximately 4Gbps: (40/( )). When P4i starts to send traffic at t5, the bandwidth does not change on any of the other partitions. P4i takes half the bandwidth that is currently being used by P4e, so each get approximately 2Gbps of bandwidth each: (40/(2 ( ))). When P1i stops at t6, it releases its bandwidth to P1e, so P1e increases to approximately 4Gbps (40/( )) of bandwidth. All of the other partitions stay the same. When P1e stops sending at t7, the other traffic flows increase accordingly. Both P2e and P33 go to their maximum setting of approximately 1.5Gbps each. Without this cap they would go to approximately 1.7Gbps, which is derived from (10/( )). 87 BC B

104 18 Non-DCB, Weight, Maximum Bandwidth, Over-subscription The two traffic flows on P4 (P4e and P4i) also increase to fill the remaining bandwidth (7Gbps), each taking half (approximately 3.5Gbps each). Normally, the traffic on P4 would get only 66.7 percent (approximately 6.7Gbps of bandwidth each: 40/( )), but since both P2 and P3 are limited to their Maximum Bandwidth, the traffic on P4 increases to fill all of the bandwidth available. When P2e stops sending at t8, P3e does not increase its bandwidth because it is already at its maximum level of 1.5Gbps. The traffic on P4 (P4e and P4i) increases to that partition s maximum level of 8Gbps, each partition taking half (approximately 4Gbps each). The remaining 0.5Gbps of bandwidth is unallocated. When P3e stops sending at t9, the same maximum ceiling condition is still in effect on the traffic on P4. Therefore, neither P4e nor P4i increase their sending rates to use this extra bandwidth. Consequently, there is 2Gbps of unused bandwidth on the 10GbE link. P4e stops at t10, which allows its companion traffic flow (P4i) to increase to that partition s maximum setting of approximately 8Gbps of bandwidth. The remaining approximately 2Gbps of bandwidth is still unused. 88 BC B

105 Index Numerics 3400/8400 Series Adapter, See adapter A adapter function assignments in single function mode 36 function number 16 function numbering interleave examples model names 2 B Bandwidth See also Maximum Bandwidth, Relative Bandwidth Weight Weight (%) parameter 13 Weight rules 28 C -c 20 CCM, setting up NPAR with 6 cfg multi-function 18, 20 command format for QCS CLI 16 configuration examples fixed subscription 74 lossless FCoE over DCB 57, 81 lossless iscsi-tlv and FCoE over DCB 60 lossless iscsi-tlv over DCB 53 Maximum Bandwidth is , 53, 57, 60, 65, 70 configuration examples (continued) Maximum Bandwidth is Maximum Bandwidth varies 77, 81, 85 non-dcb mode 49, 65, 70, 74, 77, 85 over-subscription 49, 53, 57, 60, 65, 70, 77, 81, 85 Relative Bandwidth Weight is Relative Bandwidth Weight is zero 49, 74, 77, 81 Relative Bandwidth Weight varies 70, 85 D device number, identifying 16 documentation, downloading xv downloading software and documentation xv dual-port NPAR mode, XML file for 22 E Ethernet networking MTU, configuring with QCC GUI 44 EthernetNdis protocol 18 F -f 17, 20 Flow Control parameter 9 function numbering in QCS CLI 16 interleave examples viewing in Network Connections Properties 39 viewing in Windows Device Manager BC B

106 User s Guide Setting Up NPAR with QCC, QCS, or CCM 3400 and 8400 Series Adapters I -i 18, 20 J Jumbo Packet parameter 45 L list phyadapters 16 M Max Bandwidth parameter 13 Maximum Bandwidth changing 33 description of 19 oversubscribing 34 setting, DCB mode MTU configuring Ethernet networking MTU with QCC GUI 44 configuring iscsi-offload MTU with QCC GUI 45 parameter 46 size, setting in QCC GUI size, setting in Windows Device Manger 41 size, viewing 43 Multi-Function Mode parameter 7 N NPAR adapter requirements 2 CCM setup with 6 current settings, saving a copy of 17 drivers required 3 management tools required 2 operating systems supported 2 parameters, See parameters NPAR (continued) QCC GUI, setting up with 11 QCS CLI, setting up with 15 settings, viewing in QCS CLI 20 XML file, setting parameters in 18 O -o 20 operating systems needed for NPAR 2 oversubscribing Maximum Bandwidth settings 34 P -p 16, 17, 19 parameters See also QCS CLI parameters and commands Bandwidth Weight (%) 13 Flow Control 9 Jumbo Packet 45 Max Bandwidth 13 MTU 46 Multi-Function Mode 7 Reset Configuration to Default 9 PCI bus number, identifying 16 PCIe bus device function numbering description of 36 viewing in Network Connections Properties 39 viewing in Windows Device Manager 40 ping command 43 protocol selection rules 35 protocol, EthernetNdis 18 Q QCC GUI configuring iscsi-offload MTU with 45 installation considerations 4 MTU size, setting in BC B

107 User s Guide Setting Up NPAR with QCC, QCS, or CCM 3400 and 8400 Series Adapters QCC GUI (continued) setting up with NPAR 11 viewing enabled protocols in 13 QCS CLI command format 16 installation considerations 4 NPAR settings, viewing in 20 setting up NPAR with 15 XML file QCS CLI parameters and commands -c 20 cfg multi-function 18, 20 -f 17, 20 function number 16 -i 18, 20 list phyadapters 16 -O 20 -p 16, 17, 19 PCI bus number 16 qcscli 16, 17, 19 -r 16, 17, 19 -s 18 -t 17, 19 -u 16, 17, 19 XML file name 18, 20 qcscli 16, 17, 19 QLE3400/8400 Series Adapter, See adapter R -r 16, 17, 19 Relative Bandwidth Weight, description of 19 Relative Bandwidth Weight, Non-DCB mode Reset Configuration to Default parameter 9 S -s 18 single function mode, function assignments in 36 single function mode, XML file for 22 software, downloading xv SR-IOV VF settings, overriding 20 -t 17, 19 T U -u 16, 17, 19 W Windows Device Manager, setting MTU size in 41 Windows Device Manager, viewing enabled protocols in 13 X XML file for dual-port NPAR mode 22 for single function mode 22 name in QCS CLI 18, 20 restoring 19 setting NPAR parameters in BC B

108 Corporate Headquarters QLogic Corporation Aliso Viejo Parkway Aliso Viejo, CA International Offices UK Ireland Germany France India Japan China Hong Kong Singapore Taiwan Israel 2015, 2016 QLogic Corporation. QLogic Corporation is a wholly owned subsidiary of Cavium, Inc. All rights reserved worldwide. QLogic, the QLogic logo, and QConvergeConsole are registered trademarks of QLogic Corporation. PCIe is a registered trademark of PCI-SIG Corporation. Windows and Windows Server are registered trademarks of Microsoft Corporation. All other brand and product names are trademarks or registered trademarks of their respective owners. Information supplied by QLogic Corporation is believed to be accurate and reliable. QLogic Corporation assumes no responsibility for any errors in this brochure. QLogic Corporation reserves the right, without notice, to make changes in product design or specifications.