Deploying the Dell PowerConnect 8100 Series Switch with DCB Features in an iscsi Environment. Kili Land Manjesh Siddamurthy Kevin Locklear

Transcription

1 Deploying the Dell PowerConnect 8100 Series Switch with DCB Features in an iscsi Environment Kili Land Manjesh Siddamurthy Kevin Locklear

2 This document is for informational purposes only and may contain typographical errors and technical inaccuracies. The content is provided as is, without express or implied warranties of any kind Dell Inc. All rights reserved. Dell and its affiliates cannot be responsible for errors or omissions in typography or photography. Dell, the Dell logo, and PowerEdge are trademarks of Dell Inc. Intel and Xeon are registered trademarks of Intel Corporation in the U.S. and other countries. Microsoft, Windows, and Windows Server are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others. August 2012 Rev 1.0 ii

3 Contents Introduction... 5 Executive summary... 5 What is class of service?... 6 What is VLAN priority tagging?... 6 What is user priority?... 6 What is Dot1p?... 6 What is IP-DSCP?... 6 What is queue mapping?... 7 What is queue management?... 8 Tail Drop... 8 Weighted Random Early Detection (WRED)... 8 Packet Colouring... 8 What is scheduling?... 8 Weighting... 8 Round Robin... 9 Weighted Round Robin (WRR)... 9 Weighted Deficit Round Robin (WDRR)... 9 Strict... 9 What is Data Center Bridging? Isn t that the same as Data Center Bridging Exchange? What is Data Center Bridging Exchange? What is Priority Flow Control? What is Enhanced Transmission Selection? Configuring ETS DCBx version Dot1p mapping CoS queues iscsi enable iscsi CoS iscsi ports Traffic class groups Setting TCGs with weights Setting No Drop Setting TCG as strict Setting minimum bandwidth iii

4 Setting maximum bandwidth As a willing device Auto-up vs configuration source Auto-down Top three things to remember Confirmation and troubleshooting Is it connected? LLDP DCBx Viewing DCBx settings Non-willing Willing All in one Clearing the counters PFC settings CoS settings TCG settings Example topology iv

5 Introduction Executive summary The Dell PowerConnect 8100 series switches are the newest addition of 10 and 40 gigabit Ethernet multilayer switches with a feature set targeted at deployment in campus networks. The goal of this document is to deploy various settings to enable a Data Center Bridging (DCB) iscsi configuration. Figure 1.Dell PowerConnect 8100 series switch with installed expansion modules Figure 2. Expansion Modules that are used with the Dell PowerConnect 8100 series switch 5

6 What is class of service? Class of service is a way of helping to streamline network traffic by using VLAN priority tagging (VPT) or Internet Protocol-Differentiated Services Codepoint (IP-DSCP). By using VPTs/DSCPs and assigning them to various queues, traffic can be better managed to help avoid crippling congestion on oversubscribed network connections. What is VLAN priority tagging? VLAN priority tags are 32-bit fields placed between the MAC and the EtherType/Length fields in the Ethernet frame. Within this 32-bit field, the tag protocol identifier (TPID) and the tag control information (TCI) are present. The TCI consists of the priority code (from the IEEE 802.1p standard), the drop eligibility and the VLAN ID. The TPID simply makes sure that the equipment knows that the packet is a VLAN tagged frame. The priority code specifies the priority. This can range from 0 to 7 with the lowest being best effort and increasing in priority until level 7, which is considered the highest priority available. The drop eligibility is a 1-bit field that is used along with the priority code to determine how the packet is to be handled during congestion. The VLAN ID is simply the VLAN tag marking the packet for its specific VLAN. Figure 3. VLAN tag placement The specifics on VLAN priority tagging can be found in the 802.1Q IEEE standard. By using VLAN priority tagging, it is possible to assign VLANs to appropriate queues, which can then be further controlled by additional Class of Service settings and Data Center Bridging. What is user priority? User priority is also known as VLAN priority. The terms are used interchangeably in the Dell PowerConnect series of switches. What is Dot1p? Dot1p is the same as VLAN priority. The terms are used interchangeably. Dot1p refers to the standard (802.1p) for VLAN priority codes. What is IP-DSCP? IP DSCP is the Internet Protocol Differentiated Services Code Point. This is based off of RFC 2474 and This can be backward compliant with IP Type of Service. However, they are not interchangeable. There is a possibility of 64 forwarding behaviors (per hop behaviors PHBs) that provide a service level according to the network s policies. Within the Dell PowerConnect series of switches, the DSCP values are mapped to various Class of Service traffic classes. NOTE: This topic is not covered in this document. 6

7 What are Queues? Queues are addressed in the IEEE 802.1p standard. There are seven queues (shown as Traffic Class in the switch) available in the PowerConnect series of switches. They are numbered 0-6. Various user priorities (VPTs) can be placed within these queues as decided by the network administrator to help facilitate congestion control. Without congestion, these will schedule (see: What is Scheduling?) at a standard Round Robin (see: Round Robin). With congestion, depending on the platform, they will schedule at either Weighted Round Robin (see: Weighted Round Robin) or Weighted Deficit Round Robin (see: Weighted Deficit Round Robin). What is queue mapping? Queue mapping is taking the priority assigned to the packet either through VPT or IP-DSCP and mapping it to a specific Traffic Class (Queue). Figure 4. The default mapping for Dot1p (VPT) in the PowerConnect series of switches is:default Dot1p mapping Table 1. The recommended user priority to traffic class mapping from the standard. Priority Number of Available Traffic Classes (Default)

8 What is queue management? Queue management is a way of handling packets in a network to help control congestion within a queue. The types of queue management used in the Dell PowerConnect family of switches are: Tail Drop, Round Robin, and Weighted Random Early Detection. Tail Drop Tail Drop is the default congestion queue management implemented by Dell PowerConnect. This is a very basic congestion mechanism that drops packets when the queue (bucket) is full. No matter what packets are incoming, if the queue is full, the packet is dropped. No priority is given to any one packet. As an example; Imagine a plane that is overbooked and the VIPs and first class are made to stand in line with everyone else in no particular order. When the plane is full, whoever is left behind is out of luck. That is tail drop. Tail drop is the required queue management for no-drop (PFC) queues. Weighted Random Early Detection (WRED) The way WRED works is by assigning various thresholds within the particular queue that triggers certain traffic classes to be dropped or queued. As congestion increases, the packet loss begins to affect more user priorities, based on colouring assignment, until the entire queue is affected. At levels of extreme congestion, WRED no longer is effective and traffic is handled in a tail drop manner until congestion is reduced to manageable levels. It is because of the various thresholds of congestion handling within a WRED queue management mechanism that no-drop queues cannot be assigned to this type of queue management. If WRED is used for a PFC no-drop queue, as soon as the first level of congestion is reached for the PFC priority, the pause frame is sent. In the Dell PowerConnect series of switches, the congestion thresholds are all set to equitable distribution unless colouring is assigned to a particular user priority. Packet Colouring The Dell PowerConnect series of switches uses Single Rate mode of traffic policing. By default, all packets are coloured with the same conform-colour and are handled equitably. Note: Colouring information will not be covered in this document. What is scheduling? Earlier, it was mentioned the queue management options determined if the packet was dropped or queued. Scheduling is used to handle how the packets are going to be handled once they hit the queue. There are several modes of scheduling; and, there are two levels of scheduling. Scheduling is handled at both the Queue level (for CoS) and at the Traffic Group level (for ETS). Think of it as a series of Russian dolls that stacks one within the other. Weighting Weighting is used to assign the basic amount of bandwidth that is allocated to the assigned queue/tcg. As an example; if there were three Traffic Class Groups (TCGs) and TCG0 was assigned a weight of 10, a weight of 40 to TCG1 and a weight of 50 to TCG2. Essentially the bandwidth has been allocated as 10 percent bandwidth to TCG0, 40 percent to TCG1 and 50 percent to TCG2. This bandwidth allocation 8

9 fluctuates as demand for the bandwidth shifts. However, if all three TCGs are vying for their maximum allocation, then each is granted bandwidth based on their weight. Round Robin In the Dell PowerConnect series of switches, Round Robin is used when there is no congestion on the line. This type of scheduling sends packets across each queue in a sequential order that allocates equitable distribution for all traffic. Weighted Round Robin (WRR) As it can be imagined, Weighted Round Robin is built on top of the basic Round Robin. As congestion begins to happen on the line, the allocated weights are brought in to the equation. The higher the weight, the more packets are allowed for the queue before moving on to the next queue. WRR serves every non-empty queue. Weighted Deficit Round Robin (WDRR) Weighted Deficit Round Robin is another step in the evolution of Round Robin. It, as its name implies, bases its algorithm on the weights that have been assigned, just like WRR. However, unlike WRR, there is more that comes in to play. The packets at the head of every non-empty queue are checked against a deficit counter. If the deficit counter is greater than the packet size at the head of the queue, then the queue is served. If it is not greater, then the queue is skipped and it is given a credit. This credit is added each time through, which increases its deficit counter. Thus, eventually, allowing the queue to be served. Strict Strict queuing is exactly that. This takes priority over any other queue/tcg. Essentially, when assignment is done for strict scheduling type to a queue or TCG, the setting is enabling the switch to take up the maximum available bandwidth if it needs it. This means that other traffic can be choked out to allow this particular traffic through. 9

10 What is Data Center Bridging? Data Center Bridging (DCB) is an architecture type that brings together several different features from the IEEE802.1 standards that help to improve Ethernet in the Data Center environment. DCB is the overarching architecture that contains several protocols. Isn t that the same as Data Center Bridging Exchange? No. Data Center Bridging Exchange Protocol (DCBx) is a protocol featured within DCB. What is Data Center Bridging Exchange? DCBx allows configuration information to be automatically exchanged between DCB-compliant peers. It allows for peer discovery, configuration mismatch detection, and automatic configuration of willing peers. It is this protocol that is utilized to auto-populate the ETS and PFC configurations in a willing network. Willing Mode: A device mode in which the device actively takes the DCBX configuration from the upstream peer. Configuration Source: The upstream peer that will set the DCBx configuration for connected willing devices. Note: There are three versions of DCBx CEE, CIN and IEEE. These will not be covered in depth in this document. What is Priority Flow Control? Priority Flow Control (PFC) is another protocol in the DCB collection. It is used to pause traffic on a particular queue without having to pause traffic on all queues (as in classic Flow Control). By allocating a particular queue to be a PFC queue, it is ensured that the traffic is not dropped when congestion is present on the line. Instead, a pause frame is sent to the queue to halt the traffic until such a time as traffic can be reliably passed. While this causes a decrease in throughput rates, it increases data integrity and ensures there are no dropped packets. What is Enhanced Transmission Selection? Enhanced Transmission Selection (ETS) allows bandwidth allocation to be applied on groups of queues. This allows scheduling to be allocated and handled within each TCG across the queues, before moving on to the next TCG. As mentioned above in the weighting section (see Weighting), each TCG is handled according to its weight. If all the TCGs are weighted and scheduled the same, then the highest numbered TCG will be served first, down to the lowest numbered TCG. 10

11 Configuring ETS As of 2012, on the PC81xx series of PowerConnect switches have ETS capability. The PC8024/PC8204F and PCM8024-k have the ability to Proxy ETS (pass the commands to the downstream devices from the upstream device. They do not however perform ETS. DCBx version Generally, the default setting of Auto is fine for DCBx version. However, if equipment is being used that requires a specific, hard coded version be set, this can be done at the global or interface level with the following command at the config prompt: lldp dcbx version <option> The options available are: Auto, CEE (1.06), CIN (1.0) or IEEE (802.1Qaz) NOTE: It is strongly recommended that the global level DCBx version remain auto and only the ports connected to individual equipment that requires a hardset DCBx version be modified. The DCBx version can be checked with the show command: show lldp dcbx interface te1/0/23 status Figure 5. show lldp dcbx interface te1/0/23 status console#show lldp dcbx interface te1/0/23 status DCBX operational status:... Enabled Configured DCBX version:... Auto Peer DCBX version:... IEEE Peer MAC:... Peer Description:... Auto-configuration Port Role:... Manual Peer Is configuration Source:... False Error counters: ETS incompatible configuration... 0 PFC incompatible configuration... 0 Disappearing neighbor... 0 Multiple neighbors detected

12 Dot1p mapping CoS queues ETS sits on top of CoS. An example of this behavior is the Russian Nesting Dolls: Figure 6. Russian nesting dolls By default, all queues are put into TCG0. This, effectively, means there is no ETS being performed because every queue is in the same Traffic Class Group. Planning begins with determining which traffic gets priority? For the purpose of this paper, the traffic is going to be iscsi. The priorities must be mapped to the appropriate queues. The default mapping in the PowerConnect series of switches is: Figure 7. Default priority & traffic class mapping User Priority Traffic class The traffic that is being configured for this scenario is iscsi, this is generally assigned to priority 4. This typical configuration is used for this scenario. 12

13 Use this command to move the priority 4 traffic from traffic class (queue) 2 to 4 at the config prompt: classofservice dot1p-mapping 4 4 See that it has moved to four when looking at the dot1p mapping table: Figure 8. show classofservice dot1p-mapping reflecting changes console#show classofservice dot1p-mapping User Priority Traffic Class These settings can be done at both the global and interface level. When done at the interface level, the settings will preempt the global settings. Commands in this guide will be performed at the global level. Now that iscsi traffic is mapped to its appropriate queue, it must be ensured that the iscsi traffic is where it is supposed to be. iscsi enable By default, iscsi Enable is enabled. This is iscsi optimization. This does a few things. Most of which will not be covered this guide. The items of interest where things are concerned, in this guide, is that it will force the sending of the DCBx Application Priority TLV once all the settings are completed. The TLVs provide the attached devices with information that includes (but is not limited to) the iscsi protocol (3260) and priority (PFC). iscsi CoS iscsi CoS is not enabled by default and is only required to be enabled if there is not a configuration source (ToR) that is sending the iscsi application priority TLVs. Since it will be assumed that there is not a configuration source for this setup, it must be configured. By default the iscsi CoS has a default VPT assignment of 4. This is the desired setting, so changes are not necessary. Just enable the iscsi CoS. In order for the DCBx Application Priority TLV to be sent, it must have this enabled with no available configuration source. The command to enable iscsi CoS from the config prompt is: iscsi cos enable For changing the iscsi VPT from the default VPT of 4, use the following command at the config prompt: iscsi cos vpt <#> 13

14 The number can be anywhere in the range of 0-7. Since dot1p mapping has been configured with the assumption that iscsi CoS will be set to the default of 4, there are no changes to the VPT. The iscsi settings can be checked with the following command: show iscsi Figure 9. show iscsi displaying the iscsi settings. console#show iscsi iscsi enabled iscsi CoS disabled iscsi vpt is 4 Session aging time: 10 min Maximum number of sessions is iscsi Targets and TCP Ports: TCP Port Target IP Address Name iscsi Static Rule Table Index TCP Port IP Address IP Address Mask iscsi ports Notice that there may be iscsi ports listed. These can be changed, based on the network and what ports the iscsi traffic is traversing. This comes in useful when connecting classic Flow Control iscsi devices into the network. Since they do not have the ability to allocate PFCs, the ports which are sending and receiving the iscsi traffic can be added to this list and all traffic on those ports is put into the iscsi queue. Ports can be added with the following command at the config prompt: iscsi target port <TCP port> Ports can be removed with no in front of the command. 14

15 Figure 10. iscsi target port behavior console(config)#iscsi target port 908 console(config)#no iscsi target port 860 console(config)#show iscsi iscsi enabled iscsi CoS disabled iscsi vpt is 4 Session aging time: 10 min Maximum number of sessions is iscsi Targets and TCP Ports: TCP Port Target IP Address Name iscsi Static Rule Table Index TCP Port IP Address IP Address Mask Traffic class groups As configuration source, there are a few things that must be done to get ETS actually rolling. So far, everything that has been set up has been Class of Service. Those CoS settings are the foundation for the ETS configuration. At this point a strict priority CoS queue has not been configured. It is unnecessary, since the iscsi traffic belongs to an allocated Traffic Class Group (TCG). There are a couple of ways to configure the TCG, depending on the infrastructure network requirements. It can be configured allocating X% bandwidth using weights, or it can be set to give the iscsi traffic the right to take all the bandwidth it needs. If the device being configured is to be a willing device, skip the following section and go to As a willing device. Setting TCGs with weights For this example the first option will be used. The configuration starts with giving the TCGs a set of weights and let the traffic fluctuate as necessary. With this set up, at no time will the iscsi traffic be able to choke out the other network traffic. All the other queues will be left in TCG0 (the default TCG); and the iscsi traffic is moved from TCG0 to its own TCG which is TCG1. 15

16 This command issued at the config prompt will make these changes: classofservice traffic-class-group 4 1 The rest of the weights for the traffic class groups will be set so that our iscsi gets 80 percent of the traffic and the remaining network traffic gets 20 percent. The weights are only strictly applied when the TCG s require their fully allocated bandwidth. If the iscsi traffic was only using 60 percent of its allocated 80 percent and Ethernet needed 40 percent, through the ETS capabilities the Ethernet traffic would be granted this 40 percent. Then when the iscsi needed to run at the full 80 percent it would minimize the Ethernet traffic back to 20 percent. To configure the TCG weight use the following command at config prompt: traffic-class-group weight Using the following command the TCG settings can be verified: show interfaces traffic-class-group Figure 11. show interfaces traffic-class-group console#show interfaces traffic-class-group Global Configuration Traffic Class Group Min. Bandwidth Max. Bandwidth Weight Scheduler Type Weighted Round Robin Weighted Round Robin Weighted Round Robin Setting No Drop The TCGs are now configured; but, there is still the potential for dropped packets as soon as congestion occurs. The no-drop option of Priority Flow Control (PFC) comes in to the picture at this point. Priority Flow Control can only be set on the individual port level. This means that each port that needs the PFC set to no drop needs to be manually configured. This is easily done with a range command. 16

17 Figure 12. Configuring the no-drop queue console(config)#interface range te1/0/2-9,te1/0/48 console(config-if)#datacenter-bridging console(config-if-dcb)#priority-flow-control mode on console(config-if-dcb)#priority-flow-control priority 4 no-drop At this point the basic congestion control configuration has been completed. The default for the switch is that it is in port-mode manual which means that it will push this configuration to any connected, willing device. Setting TCG as strict However, if it is decided that the network would be better served by having the iscsi traffic able to take all available bandwidth, regardless of impact on the other network traffic, there is the option of strict scheduling. When configuring with strict scheduling, the weights look as if they have been configured for all the bandwidth to TCG0 because the setting is However, this is not the case. The strict TCG is serviced first. If there were three TCGs being configured: one as strict and two as not strict, then the following option at the config prompt could be used to allocate bandwidth: traffic-class-group weight And, still, the strict TCG (in this case TCG2) takes all the bandwidth and leaves TCG 0 and TCG 1 with absolutely no bandwidth at all. However, if TCG2 were not taking up all the bandwidth, then TCG0 and 1 would equally divide the remaining bandwidth between themselves. For our example, however, if there are only two TCGs. One will be strict (the iscsi). (config)#traffic-class-group strict 1 With this example the TCG that our iscsi was assigned to (TCG1) is configured strict. With the default weights at , there are no other changes that need to be made. However, if there had been some changes, default settings can be put back into place again with the following command at the config prompt: traffic-class-group weight Just like in the weighted settings no-drop PFCs may be needed, and will need to be implemented on a per interface configuration (see Setting No Drop). Now, there are some interesting tweaks that can be done in the Dell PowerConnect that are not common. 17

18 Setting minimum bandwidth By setting minimum bandwidth allocations it can make sure that our TCGs will always be guaranteed X amount of bandwidth, regardless of how congested the line becomes. Minimum bandwidth settings are bandwidth guarantees. The TCG will always have this bandwidth available, even if the strict queue wants to take the bandwidth, it will not be allowed. Do not set the minimum bandwidth totals to 100 percent. This hard sets the bandwidth. It is counter- productive and detrimental to a production environment. Always keep the minimum bandwidths below 100 percent, the preferred behavior is to configure at most 80 percent to allow for bursty traffic. The minimum bandwidth allocation can be set at the global or interface level. Note that as mentioned before the interface level settings preempt the global settings. (config)#traffic-class-group min-bandwidth In this example, the minimum bandwidth has been allocated to two TCGs. The total value of the minimums set for both equals 30 percent. This leaves more than enough room for bursty traffic. Remember. This is the minimum bandwidth. This is not the weight, or the maximum allowed. This is simply the amount of bandwidth that is guaranteed to our TCGs. Setting maximum bandwidth With the PowerConnect 8100 series switches there has also been a configuration setting implemented that caps the maximum allowed bandwidth. The following command can be run at the config prompt to make sure that the TCG never exceeds X amount of bandwidth by using the maximum bandwidth settings: traffic-class-group max-bandwidth In this example, the configuration has been set so that TCG0 traffic can never exceed 50% percent of the bandwidth, thus allowing TCG 1 and 2 to have the remaining 50 percent at all times. This is not a bandwidth guarantee for TCG0. It is the maximum it is allowed to use. If the TCG wants 61 percent, it cannot have it. Even if TCG1 and TCG2 are not transmitting traffic. As a willing device If the switch is going to be a downstream device set as willing and simply connected into the network, and there is already a configuration source upstream setup becomes much simpler. There are no ETS configurations that need to be set in the device, and there are no PFC allocations that must be configured. Instead, the configuration must be set for the ports connected to the upstream device, configuring to auto-up/configuration source. Then the downstream ports must be configured as auto-down. Auto-up vs configuration source The configuration source setting can only be applied to a single port. Therefore, if the configuration is a port-channel to the upstream device, the ports must all be configured to use auto-up on. 18

19 Figure 13. Upstream port-role configuration (config-if-te1/0/23)#lldp dcbx port-role configuration-source or (config-if-te1/0/23)#lldp dcbx port-role auto-up Port Modes can only be applied on the physical interface. When applying port modes on ports in a port-channel, be absolutely certain that all ports are set for the same mode. Auto-down Auto-down is used on the ports connecting to the downstream, willing devices. This mode takes the information obtained from the ports connected to the configuration source and pushes the configuration down to the willing devices. This command must be entered on all ports connected to the downstream, willing devices. Figure 14. Downstream port-role configuration (config-if-te1/0/23)#lldp dcbx port-role auto-down Top three things to remember Congestion control is only brought in to play when there is actually congestion on the line. If there is no congestion, then ETS and PFC have no part to play. Do not set minimum bandwidth allocations to a total of 100 percent as this cause an inability to adapt to bursts of traffic and causes problems on the network. Do not apply WRED queue management to the PFC queue as this causes PFCs to be sent well before the queue is truly congested. Confirmation and troubleshooting Here are some of the useful commands that will help to identify DCB & iscsi status of the switch. Is it connected? This seems like a no-brainer. Make sure the connection between the two ports is up and check if spanning tree is blocking these ports. Don t forget to keep an eye on link speed. LLDP DCBx The ETS and PFC information should match between the devices for things to work as expected, unless it s an asymmetrical network. Remember, DCBx is setting on the physical ports, not on the port channel. So, be certain to check each of the physical connections for their DCBx settings. Viewing DCBx settings Depending on if the system is Willing or Not-Willing, expect to see different results. 19

20 Non-willing Or Willing: False, what does this mean? This is showing that the particular port local or peer that is set most likely to be configuration source or set for manual settings so it will not take the negotiated values passed to it. The image below shows local configuration (all of the pertinent are circled) and the peer. If local device has PFC 4 enabled, but the peer device was thinking it to be PFC 3 then things will not be smooth for iscsi traffic. Even if peer is not a willing device (which it does not have to be), the settings need to match, at least where the PFCs are concerned. ETS is, by its nature, flexible and prone to need a bit of a change from device to device depending on network. However, the connected ports should match up (or at the very least not be signaling about being incompatible). This is where individual port configuration makes it possible to customize network for maximum efficiency. 20

21 Figure 15. LLDP DCBx Stats - Non-Willing console#show lldp dcbx interface te1/0/48 detail DCBX operational status:... Enabled Configured DCBX version:... Auto Peer DCBX version:... IEEE Peer MAC:... 00:1E:C9:DD:BB:09 Peer Description:... Auto-configuration Port Role:... Manual Peer Is configuration Source:... False Error counters: ETS incompatible configuration... 0 PFC incompatible configuration... 0 Disappearing neighbor... 0 Multiple neighbors detected... 0 Local configuration: PFC configuration (Tx enabled) Willing: False MBC: False Max PFC classes supported: 2 PFC enable vector: 0:0 1:0 2:0 3:0 4:1 5:0 6:0 7:0 ETS configuration (Tx enabled) Willing: False Credit shaper: False Number of TCs supported: 3 Priority assignment: 0:0 1:0 2:0 3:0 4:1 5:0 6:0 7:0 Traffic class bandwidth (%):0:100 1:0 2:0 3:0 4:0 5:0 6:0 7:0 Traffic selection algorithm:0:2 1:2 2:2 3:2 4:2 5:2 6:2 7:2 Application priority (Tx enabled) Type Application Priority Status TCP/UDP 0xcbc 4 Enabled Peer configuration: Willing: True MBC: False Max PFC classes supported: 2 PFC enable vector: 0:0 1:0 2:0 3:0 4:1 5:0 6:0 7:0 ETS configuration Willing: True Credit shaper: False Number of TCs supported: 3 Priority assignment: 0:0 1:0 2:0 3:0 4:1 5:0 6:0 7:0 Traffic class bandwidth (%):0:100 1:0 2:0 3:0 4:0 5:0 6:0 7:0 Traffic selection algorithm:0:2 1:2 2:2 3:2 4:2 5:2 6:2 7:2 Application priority (Tx enabled) Type Application Priority Status TCP/UDP 0xcbc 4 Enabled 21

22 Willing If configured as willing, then local information must match the peer s information and show the peer as configuration source. Figure 16. LLDP DCBx Stats - Willing console#show lldp dcbx interface te1/0/23 detail DCBX operational status:... Enabled Configured DCBX version:... Auto Peer DCBX version:... IEEE Peer MAC:... 00:1E:C9:DD:BD:0D Peer Description:... Auto-configuration Port Role:... Auto-up Peer Is configuration Source:... True Again, stats should match, just like in the willing stats in figure 17. However, the local settings should show willing and the peer should be non-willing. All in one To see if a configuration source is selected, and the DCBx stats for the ports, there is a cohesive overview available with the command: show lldp dcbx interface all The information returned on this command provides the status of the LLDP on the port, and what port role is assigned. It also provides what DCBx version has been allocated for the port (if it has been hardset) and then provides an overview of the DCBx TX and RX. Figure 17. LLDP DCBx statistics for all switch ports console#show lldp dcbx interface all Is configuration source selected... True Configuration source port... Te1/0/23 DCBX DCBX DCBX unknown Interface Status Role Version Tx Rx Errors TLV Te1/0/1 Enabled Auto-down Auto Te1/0/2 Enabled Auto-down Auto Te1/0/3 Enabled Auto-down Auto Te1/0/4 Enabled Auto-down Auto Te1/0/5 Enabled Auto-down Auto Te1/0/19 Enabled Auto-down Auto Te1/0/20 Enabled Auto-down Auto Te1/0/21 Enabled Auto-down Auto Te1/0/22 Enabled Auto-down Auto Te1/0/23 Enabled Auto-up Auto Te1/0/24 Enabled Auto-down Auto Te1/0/25 Enabled Auto-down Auto

23 Clearing the counters When troubleshooting, it is helpful to start with a clean slate and observe the changes that occur. To do this, use clear counters at either the global level or interface specific level. To clear all counters across the entire switch, the command is: clear counters To clear the counters on a specific interface, the command is: clear counters <interface> Again, it clears all the counters on the port, not just the LLDP/DCBx counters. PFC settings To confirm that the no drop queues were properly allocated use the following command: show interfaces priority-flow-control Figure 18. Viewing the allocated PFCs console#show interfaces priority-flow-control Port Drop No-Drop Operational Priorities Priorities Status Te1/0/1 0-7 Inactive Te1/0/2 0-3,5-7 4 Active Te1/0/3 0-3,5-7 4 Active Te1/0/4 0-3,5-7 4 Active Te1/0/5 0-3,5-7 4 Active Te1/0/6 0-3,5-7 4 Active Te1/0/7 0-3,5-7 4 Active Te1/0/8 0-3,5-7 4 Active Te1/0/9 0-7 Inactive... Te1/0/ Inactive Te1/0/ Inactive Te1/0/ Inactive /0/ i Results show as expected since only configured no-drop queue is PFC 4. CoS settings For CoS settings it is necessarily to once again check the user priority and traffic class mapping as seen earlier in the document. CoS settings can be confirmed using following commands: show classofservice dot1p-mapping The following command will show the queue management, scheduling and minimum bandwidth for each class of service queue: show interfaces cos-queue 23

24 Figure 19. Global CoS queue console#show interfaces cos-queue Global Configuration Interface Shaping Rate... 0 kbps WRED Decay Exponent... 9 Queue Id Min. Bandwidth Scheduler Type Queue Management Type Weighted Tail Drop 1 0 Weighted Tail Drop 2 0 Weighted Tail Drop 3 0 Weighted Tail Drop 4 0 Weighted Tail Drop 5 0 Weighted Tail Drop 6 0 Weighted Tail Drop If the individual ports have been set with CoS queue settings, then the command can be further detailed: show interfaces cos-queue te1/0/48 Figure 20. CoS Queue - interface level console#show interfaces cos-queue te1/0/48 Interface... Te1/0/48 Interface Shaping Rate... 0 kbps WRED Decay Exponent... 9 Queue Id Min. Bandwidth Scheduler Type Queue Management Type Weighted Tail Drop 1 0 Weighted Tail Drop 2 0 Weighted Tail Drop 3 0 Weighted Tail Drop 4 0 Weighted Tail Drop 5 0 Weighted Tail Drop 6 0 Weighted Tail Drop 24

25 The following command displays the mapping of the dot1p queue to the TCG: show classofservice traffic-class-group Figure 21. Traffic Class Group mapping console#show classofservice traffic-class-group Traffic Class Traffic Class Group TCG settings To view the traffic class group bandwidth allocations, weights and scheduling use the following command: show interfaces traffic-class-group Figure 22. show interfaces traffic-class-group console#show interfaces traffic-class-group Global Configuration Traffic Class Group Min. Bandwidth Max. Bandwidth Weight Scheduler Type Weighted Round Robin Strict Weighted Round Robin Just as the CoS queues can be set on an individual port level, so can the TCGs. To view the settings at the individual port level as well use the following command: show interfaces traffic-class-group te1/0/48 Figure 23. show interfaces traffic-class-group console#show interfaces traffic-class-group te1/0/48 Interface... Te1/0/48 Traffic Class Group Min. Bandwidth Max. Bandwidth Weight Scheduler Type Weighted Round Robin Strict Weighted Round Robin Round Robin 25

26 Example topology The topology provided here is just a quick overview of one type of environment in which ETS for iscsi would come in particularly useful. This is not a suggested topology, nor is it the only topology supported. It is solely for example purposes. Figure 24. Example topology 26