Cisco Nexus 9500 Series Switches Buffer and Queuing Architecture

Size: px

Start display at page:

Download "Cisco Nexus 9500 Series Switches Buffer and Queuing Architecture"

Lynne Thompson
6 years ago
Views:

1 White Paper Cisco Nexus 9500 Series Switches Buffer and Queuing Architecture White Paper December Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 74

2 Contents Cisco Nexus 9500 Architecture Overview... 3 Buffer on Network Forwarding Engine... 3 Buffer on Application Leaf Engine... 6 Queuing on Cisco Nexus 9500 Series Switches queue Mode and 8-queue Mode... 9 Class-Based Weighted Round Robin (WRR) and Priority Queuing Mechanism Egress Queue and Extended Output Queue Architecture Queuing Architecture on N9K-X9600 and N9K-X9400 Series Line Cards Egress Queuing Architecture on N9K-X9600 and N9K-X9400 Series Line Cards Buffer and Egress Queue Monitoring on N9K-X9600 and N9K-X9400 Series Line Cards Queuing Architecture on N9K-X9500 Series Line Cards Buffer Boost Function on N9K-X9500 Series Line Cards Extended Output Queue Architecture on N9K-X9500 Series Line Cards NFE Buffer and Egress Queue Monitoring on N9K-X9500 Series Line Cards ALE Buffer and Extended Output Queue Monitoring on N9K-X9500 Series Line Cards Queue Limit on Cisco Nexus 9500 Series Switches ALE Flow Prioritization Conclusion For More Information Appendix Appendix A: Buffer and Queue Monitoring on the N9K-X9600 Series Line Cards Appendix B: Buffer and Queue Monitoring on the N9K-X9400 Series Line Cards Buffer and Queue Monitoring on the N9K-X9432PQ Line Card Buffer and Queue Monitoring on N9K-X9464PX and N9K-X9464TX Line Cards Appendix C: Buffer and Queue Monitoring on N9K-X9500 Series Line Cards NFE Buffer and Queue Monitoring on N9K-X9564PX and N9K-X9564TX Line Cards ALE Buffer and Queue Monitoring on N9K-X9564PX and N9K-X9564TX Line Cards Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 74

3 Cisco Nexus 9500 Architecture Overview Cisco Nexus 9500 Series Switches are modular data center switch platforms. With the variety of line card types, Cisco Nexus 9500 Series Switches provide data center network solutions with high performance and costoptimized 1 GE/10 GE access/leaf layer and 40 GE/100 GE (future) aggregation/spine layer. This white paper discusses the buffer and queuing architecture of Cisco Nexus 9500 Series Switches, and assumes the prerequisite knowledge of the hardware architecture of Cisco Nexus 9500 Series Switches. For details on Cisco Nexus 9500 Series Switch architecture, refer to the following white paper: Cisco Nexus 9500 Series Switches are designed with shared on-chip buffer memory. Buffer memory space is provided by application-specific integrated circuit (ASIC) components on the fabric modules and line cards, including Network Forwarding Engine (NFE), Application Leaf Engine (ALE) and Application Spine Engine (ASE). Table 1 shows ASIC types on different line cards as well as the fabric module of Cisco Nexus 9500 Series Switches. Table 1. ASCI Types on Cisco Nexus 9500 Series Switch Fabric Module and Line Cards N9K-X9400 N9K-X9500 N9K-X9600 N9K-X9700 Fabric Module Line Cards Line Cards Line Cards Line Cards NFE only NFE+ALE NFE only ASE only NFE only This white paper discusses the buffer and queuing architecture of Cisco Nexus 9500 Series Switches running in Cisco NX-OS mode. N9K-X9700 Series line cards with ASE ASICs are outside the scope of this white paper as they only operate in Application Centric Infrastructure (ACI) mode. Buffer on Network Forwarding Engine Network Forwarding Engine (NFE) can be found on fabric modules and line cards of Cisco Nexus 9500 Series Switches. NFE has a 12-MB on-chip buffer that is dynamically shared by all active ports on NFE for traffic in both ingress and egress directions. A packet is buffered once, but can be accounted for more than once based on the quality of service (QoS) functions enabled on the switch. For instance, it can be accounted on the ingress for priority flow control (PFC) and accounted for again on the egress for egress queuing. The 12-MB NFE buffer is managed and utilized as a collection of 208-byte cells. This dynamically-shared buffer space is carved into multiple service pools to serve different traffic types. How the buffer is carved into which services pools depends on the module types. On modules that have only NFE, but not ALE, the NFE buffer is carved into two service pools (shown in Figure 1). Control traffic service pool Default service pool The control traffic service pool is dedicated to control plane traffic to help ensure sufficient buffer space is allocated for control traffic. All other traffic will share the default service pool. These types of modules include fabric modules, N9K-X9600 series line cards and N9K-X9400 series line cards Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 74

4 Figure 1. NFE Buffer Service Pool on NFE only Modules (Fabric Module, N9K-X9600 and N9K-X9400 Series Line Cards) Figures 2 and 3 display the output of the buffer monitoring command-line interface (CLI) command to demonstrate the different buffer service pools on NFE instances of fabric modules, N9K-X9600, and N9K-X9400 series line cards. Figure 2. NFE Buffer Service Pool Display on Fabric Module 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 74

5 Figure 3. NFE Buffer Service Pool Display on N9K-X9600 and N9K-X9400 Series Line Cards On modules that have both NFE and ALE ASICs (such as the N9K-X9500 series line card) the NFE buffer is carved into three service pools (shown in Figures 4 and 5): Control service pool Out-of-band flow control (OOBFC) unicast service pool Default service pool Control traffic is served with the dedicated buffer resource in the control service pool. The OOBFC unicast service pool serves unicast traffic that has extended output queues on the line card ALE. Figure 4. NFE Buffer Service Pools on N9K-X9500 Series Line Cards 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 74

6 Figure 5. NFE Buffer Information Display on N9K-X9500 Series Line Cards Buffer on Application Leaf Engine Application Leaf Engine (ALE) is built with a 40-MB on-chip buffer. This buffer is also managed and utilized as a collection of 208-byte cells. It is divided into three regions (shown in Figures 6, 7, and 8): Ingress straight traffic - 10 MB For traffic going to fabric modules through the internal links between ALE and fabric modules Ingress hairpin traffic - 10 MB For local traffic between two front-panel ports on the same line card NFE. This type of local traffic can be optionally redirected by the line card NFE to the connected ALE to take advantage of the additional 10-MB buffer. The feature that controls the local traffic redirection is a buffer boost that is a per-egress port configuration Egress straight traffic - 20 MB For traffic coming from fabric modules and going to line card front-panel ports through the internal links between line card ALE and NFE Each of the three buffer regions is dynamically shared by the ports that it serves in the corresponding direction Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 74

service pool --- for control traffic; 500 cells (approximately 104 KB) SPAN service pool --- for SPAN traffic; 256

7 Figure 6. Buffer on ALE All the three regions of buffer are carved into three service pools (shown in Figure 7): Control service pool --- for control traffic; 500 cells (approximately 104 KB) SPAN service pool --- for SPAN traffic; 256 cells (approximately 53 KB) Default traffic pool --- for all other data traffic Figure 7. Buffer Service Pools on ALE 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 74

8 Figure 8. ALE Buffer Information Display on N9K-X9500 Series Line Cards Queuing on Cisco Nexus 9500 Series Switches Cisco Nexus 9500 Series Switches have a unique internal architecture that resembles a two-tier spine/leaf fabric network comprised of fabric modules and line cards. This architecture allows Cisco Nexus 9500 switches to implement the simple yet effective class-based egress queuing mechanism throughout the switch system by taking full advantage of the shared buffer resources on fabric modules and line cards. Cisco Nexus 9500 Series Switches utilize the following types of traffic classes for queuing: Control traffic class Switched Port Analyzer (SPAN) traffic class User traffic classes Control traffic class and SPAN traffic class are systems internally defined and are transparent to users. Network control plane traffic, including traffic for network control protocols such as Open Shortest Path First (OSPF), Border Gateway Protocol (BGP), Network Transport Protocol (NTP), and others, is classified into the control class. SPAN traffic, including local SPAN and Encapsulated Remote Switched Port Analyzer (ERSPAN), goes into the SPAN class. Control traffic is treated with the highest priority and has reserved buffer resources. SPAN traffic is of the lowest priority on a port and uses the remaining bandwidth Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 74

9 The user traffic classes are system pre-defined objects, but the queuing policies for each user traffic class are user-definable. Users can also control what kind of traffic goes into which user classes through ingress classification by applying QoS policies on the ingress ports. On an ingress port, users can define and apply traffic classification policies to map traffic into one of the internal qos-groups. Qos-groups are system-internal identifications for the user traffic classes. Each qos-group is associated with an egress queuing class on egress ports. Once being classified into a qos-group on the ingress port, traffic will be subject to the egress queuing policies defined for this qos-group on the egress port. 4-queue Mode and 8-queue Mode Depending on the line card types installed in the switch chassis, a Cisco Nexus 9500 Series Switch running Cisco NX-OS Software can support 4-queue mode and 8-queue mode. (8-queue mode is introduced in Cisco NX-OS Software Release 6.1(2)I3(1).) In the 4-queue mode, a Cisco Nexus 9500 Series Switch has four internal qos-groups (qos-group 0 through 3) and four pre-defined egress queuing classes: c-out-q-default --- Egress default queue (qos-group 0) c-out-q1 --- Egress queue 1 (qos-group 1) c-out-q2 --- Egress queue 2 (qos-group 2) c-out-q3 --- Egress queue 3 (qos-group 3) In the 4-queue mode, each egress port has egress queues for user traffic based on the above four traffic classes. Figure 9 shows the ingress classification and egress queuing process in the 4-queue mode. Figure 9. Cisco Nexus 9500 Series Switch Ingress Classification and Egress Queuing Operation in 4-queue Mode 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 74

In the 8-queue mode, a Cisco Nexus 9500 Series Switch has eight internal qos-groups - qos-group 0 through 7 - and eight pre-defined egress queuing classes: c-out-8q-q-default --- Egress default queue

10 In the 8-queue mode, a Cisco Nexus 9500 Series Switch has eight internal qos-groups - qos-group 0 through 7 - and eight pre-defined egress queuing classes: c-out-8q-q-default --- Egress default queue (qos-group 0) c-out-8q-q1 --- Egress queue 1 (qos-group 1) c-out-8q-q2 --- Egress queue 2 (qos-group 2) c-out-8q-q3 --- Egress queue 3 (qos-group 3) c-out-8q-q4 --- Egress queue 4 (qos-group 4) c-out-8q-q5 --- Egress queue 5 (qos-group 5) c-out-8q-q6 --- Egress queue 6 (qos-group 6) c-out-8q-q7 --- Egress queue 7 (qos-group 7) In the 8-queue mode, each egress port has egress queues for user traffic based on the above eight traffic classes. Figure 10 describes the ingress traffic classification, qos-group mapping, and egress queuing process in the 8- queue mode. Figure 10. Cisco Nexus 9500 Series Switch Ingress Classification and Egress Queuing Operation in 8-queue Mode By default, the Cisco Nexus 9500 Series NX-OS Software system operates in the 4-queue mode. All line card types support the 4-queue mode. Additionally, NFE-only line cards, including N9K-X9600 and N9K-X9400 series line cards support the 8-queue mode as well. A Cisco Nexus 9500 Series Switch can run in the 8-queue mode if all of its line cards support the 8-queue mode. Table 2 summarizes queue mode support by different line card types Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 74

11 Table 2. Queue Mode Support by Different Line Card Types N9K-X9600 Series Line Cards N9K-X9500 Series Line Cards 4-Queue Mode Yes Yes Yes 8-Queue Mode Yes No Yes N9K-X9400 Series Line Cards Note that a Cisco Nexus 9500 Series Switch can run in 8-queue mode only if all of its line cards are capable of running 8-queue mode. If any line card does not support 8-queue mode, the entire switch is not able to run 8-queue mode. Below is an example of the error messages while trying to apply 8-queue policy to a Cisco Nexus 9500 Series Switch that does not support 8-queue mode: (config)# system qos (config-sys-qos)# service-policy type queuing output default-8q-out-policy ERROR: policy-map default-8q-out-policy can be activated only on 8q capable platforms Class-Based Weighted Round Robin (WRR) and Priority Queuing Mechanism Cisco Nexus 9500 Series Switches use WRR and priority queue mechanisms to manage the egress queues and extended egress queues on NFE and ALE. Following is the pre-defined default egress queuing policy for the four user traffic classes in the 4-queue mode. It serves as the default queuing policy for system QoS and all interfaces. If an interface has a user-defined queuing policy applied, it overwrites this default system queuing policy. policy-map type queuing default-out-policy class type queuing c-out-q3 priority level 1 class type queuing c-out-q2 bandwidth remaining percent 0 class type queuing c-out-q1 bandwidth remaining percent 0 class type queuing c-out-q-default bandwidth remaining percent 100 The following code shows the pre-defined default egress queuing policy for the 8-queue mode. It serves as the default queuing policy for system QoS and all interfaces when the switch is running in the 8-queue mode. If an interface has a user-defined queuing policy applied, it overwrites this default system queuing policy. policy-map type queuing default-8q-out-policy class type queuing c-out-8q-q7 priority level 1 class type queuing c-out-8q-q6 bandwidth remaining percent 0 class type queuing c-out-8q-q5 bandwidth remaining percent 0 class type queuing c-out-8q-q4 bandwidth remaining percent 0 class type queuing c-out-8q-q3 bandwidth remaining percent Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 74

12 class type queuing c-out-8q-q2 bandwidth remaining percent 0 class type queuing c-out-8q-q1 bandwidth remaining percent 0 class type queuing c-out-8q-q-default bandwidth remaining percent 100 In a WWR queuing policy, bandwidth can be defined in percentage of the link bandwidth, or percentage of the remaining bandwidth. When using priority queuing, you can define the bandwidth of the other non-priority queues (WRR queues) only in percentage of the remaining bandwidth. Cisco Nexus 9500 Series Switches support up to three priority queues. They must start with the class c-out-q3 in a 4-queue queuing policy-map configuration, and go to c-out-q2 and c- out-q1 in sequence, or start with the class c-out-8q-q7 in an 8-queue queuing policy-map configuration, and go to c-out-8q-q6 and c-out-q6 in sequence. Table 3 offers a sample configuration, including ingress classification and egress queuing. Table 3. Queuing Sample Configuration Ingress Classification Class map type qos for classification class-map type qos match-any class-gold match precedence 4-6 This example use IP precedence to classify traffic. IP DSCP, 802.1q CoS, access control list (ACL), etc., can also be used for classification. class-map type qos match-any class-brown match precedence 0-1 class-map type qos match-any class-silver match precedence 2-3 class-map type qos match-any class-plantinum match precedence 6-7 Policy-map type qos for classification policy policy-map type qos ingr-classify-policy class class-plantinum set qos-group 3 class class-gold set qos-group 2 class class-silver set qos-group 1 class class-brown set qos-group 0 Policy-map type qos is used to define the ingress classification policy. In this example, the qos policy-map ingr-classify-policy uses the above defined qos class-maps to map traffic, and set qos-group for each class. Apply service-policy qos on ingress interface Interface Ethernet1/1 service-policy type qos input ingr-classify-policy 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 74

Ingress Classification Egress queuing policy-map policy-map type queuing egr-queuing-policy class type queuing c-out-q3 priority level 1 class type queuing c-out-q2 priority level 2 class type

13 Ingress Classification Egress queuing policy-map policy-map type queuing egr-queuing-policy class type queuing c-out-q3 priority level 1 class type queuing c-out-q2 priority level 2 class type queuing c-out-q1 bandwidth remaining percent 60 class type queuing c-out-q-default bandwidth remaining percent 0 Policy-map type queuing is used to define the egress queuing policy. This example has two priority queues and two WRR queues (including the default queue). Apply the queuing service-policy to the egress interface interface Ethernet1/2 service-policy type queuing output egr-queuing-policy Egress Queue and Extended Output Queue Architecture In general, Cisco Nexus 9500 Series Switches running the Cisco NX-OS system are designed to use the egress queuing architecture. But depending on the line card capability, it could be just simple egress queuing on the line card NFE if the line card is equipped with only NFE, or extended egress queuing if the line card has both NFE and ALE. In the latter case, the ALEs provide additional buffer space. Table 4 summarizes the queuing capability of different line card types. Table 4. Cisco Nexus 9500 Series Switch Line Card Queuing Capability Line Card Types N9K-X9600 N9K-X9500 N9K-X9400 NFE or 1 ALE User traffic class 4 or or 8 Output queue on NFE Yes Yes Yes Extended output queues (EoQ) on ALE No Yes No Queuing Architecture on N9K-X9600 and N9K-X9400 Series Line Cards Egress Queuing Architecture on N9K-X9600 and N9K-X9400 Series Line Cards N9K-X9600 and N9K-X9400 series line cards for Cisco Nexus 9500 Series Switches have only NFE ASICs. Figure 11 shows the internal block diagrams for the N9K-X9600 and N9K-X9400 series line cards. Figure 11. Internal Architecture of N9K-X9600 and N9K-X9400 Series Line Cards 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 74

These line cards utilize the simple, class-based, egress queuing mechanism on NFE to handle link congestion. They support up to eight user traffic classes.

14 These line cards utilize the simple, class-based, egress queuing mechanism on NFE to handle link congestion. They support up to eight user traffic classes. Each class has a unicast queue and a multicast queue. Multicast traffic and unicast traffic of the same traffic class are counted together against the granted bandwidth for this class. Control traffic and SPAN traffic classes have their own separate queues. Among all the traffic classes, control traffic has the highest priority, SPAN traffic has the lowest priority, and the eight user traffic classes are subject to either priority queuing policy or WRR queuing policies. Figure 12 depicts the output queue structures of a frontpanel port on N9K-X9600 and N9K-X9400 series line cards. Figure 12. Egress Port Queue Structure on N9K-X9600 and N9K-X9400 Series Line Cards Buffer and Egress Queue Monitoring on N9K-X9600 and N9K-X9400 Series Line Cards The Cisco NX-OS system provides CLI commands to monitor buffer and queue statistics. The show hardware internal buffer info pkt-stats detail command shows dynamic buffer statistics for all ports on NFE on a per-class, per-queue basis. N9K-X9600 and N9K-X9400 series line cards can support up to eight traffic classes (class Q7 through Q0) and a CPU class and a SPAN class per port. The eight traffic classes and the SPAN class are served out of the default service pool while the CPU class uses the control service pool. Each class has a unicast queue and a multicast queue. Following is an example output of the show hardware internal buffer info pkt-stats detail command on an N9K- X9636PQ line card for its first NFE instance (Instance 0). Refer to Appendix A for the complete output for the following N9K-X9600 and N9K-X9400 series line cards: N9K-X9636PQ N9K-X9432PQ N9K-X9472PX/N9K-X9472TX 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 74

16 A variation of the above buffer and queue monitoring command shows the peak value of buffer utilization in each queue. Following is a sample output for the high-water-mark monitoring Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 74

17 Queuing Architecture on N9K-X9500 Series Line Cards Buffer Boost Function on N9K-X9500 Series Line Cards Given the 1/10 GE port density on the N9K-X9564PX or N9K-X9564TX line cards, they re very suitable for deployment in data center network access layer connectivity. One significant advantage of N9K-X9500 series line cards is the additional buffer space they have on the ALE ASICs. In addition to the 12 MB buffer on NFE, the N9K- X9500 series line cards have an additional 40 MB buffer on each ALE. The extra buffer space provides significant benefit in the event of link congestions. This happens more often in the access layer of a data center network where different port speeds co-exist, and the in-cast traffic pattern has more influence to the link bandwidth utilization. Within the 40 MB buffer on ALE, 10 MB is dedicated for the local traffic between two ports on the same NFE instance. Since NFE performs packet lookup and forwarding, the local traffic between two NFE front-panel ports doesn t need to go to ALE for forwarding process. However, in order to take advantage of the 10 MB of extra buffer on ALE, the NFE local traffic needs to be redirected onto ALE. Cisco introduces the Buffer Boost feature for this purpose. Figure 13 depicts this new function Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 74

18 Figure 13. Cisco Nexus 9000 Series Switch Buffer Boost Function When Buffer Boost is enabled on a NFE front-panel port, unicast traffic from another front-panel port on the same NFE instance will be redirected to the connected ALE instance to use the additional buffer for the local traffic. ALE will send the packets back to NFE so that NFE can forward them to the egress port. When Buffer Boost is disabled on a NFE front-panel port, NFE will not re-direct the traffic from another local port to this port to ALE. Instead, it forwards the traffic directly to this egress port. Buffer Boost is an egress port configuration property. It can be enabled or disabled on a per-port basis. It is enabled on all NFE 1/10 GE front-panel ports by default. Buffer Boost only applies to local unicast traffic. It won t change multicast traffic forwarding. Extended Output Queue Architecture on N9K-X9500 Series Line Cards N9K-X9500 series line cards have both NFE and ALE ASICs. They can take full advantage of both the 12-MB buffer on NFE and the 40-MB buffer on ALE for congestion management. Current line cards in the N9K-X9500 series include: N9K-X9536PQ GE Quad Small Form Factor Pluggable Plus (QSFP+) ports N9K-X9564PX GE SFP/10 GE SFP+ ports plus four 40 GE QSFP+ ports N9K-X9564TX /10 GE Base-T ports plus four 40 GE QSFP+ ports Figure 14 shows the ASIC block diagrams of N9K-X9500 series line cards. All cards have two instances of NFEs and two instances of ALEs Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 74

Figure 14. N9K-X9500 Series Line Card Architecture On N9K-X9500 series line cards, front-panel ports are connected to NFEs while ALEs provide internal connections between NFE and fabric modules.

19 Figure 14. N9K-X9500 Series Line Card Architecture On N9K-X9500 series line cards, front-panel ports are connected to NFEs while ALEs provide internal connections between NFE and fabric modules. NFE provides the direct egress queues for each front-panel port using its local 12 MB buffer space, and extends the egress unicast queues of its front-panel ports onto the directly connected ALE through the OOBFC signaling mechanisms. N9K-X9500 line cards support only 4-queue mode, so the queues on the line cards are structured based on the following classes: Four user traffic classes One control traffic class (system internal) One SPAN traffic class (system internal) Each front-panel port on NFE has a unicast queue and a multicast queue according to the above classes. Additionally, each front-panel port has an OOBFC unicast queue per user traffic class. They correspond to the unicast extended output queues on ALE. On ALE, there are four unicast extended output queues (EoQs) for each NFE front-panel egress port. Using the OOBFC signaling, NFE can extend its OOBFC unicast queues onto the connected ALE instance to take advantage of the additional buffer space on ALE. Figure 15 shows the NFE egress queues and ALE EoQs for an NFE front-panel port on a N9K-X9500 series line card Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 74

20 Figure 15. Extended Output Queues on N9K-X9500 Series Line Cards Based on the queue depth of the four OOBFC unicast queues per front-panel port, NFE uses the OOBFC signaling channel to notify ALE when to stop or when to resume sending traffic to NFE on a per-egress-port per-unicastclass basis. When the queue depth of a given OOBFC unicast queue exceeds the system-defined threshold, NFE will signal ALE to stop sending traffic into this queue. Upon receiving the stop signal, ALE begins to shift the packets into the corresponding EoQ using its local buffer. As a result, the egress unicast queues on NFE are extended onto ALE EoQs on a per-egress-port per-class basis. When the queue on NFE is drained, NFE will send a re-start signal to tell ALE to resume forwarding packets for this queue again. The unicast traffic that can take advantage of OOBFC-signaled EoQ on ALE includes the egress straight traffic coming from the fabric module and going to an NFE front-panel port, as well as the re-directed local traffic between two front-panel ports on the same NFE instance. The additional buffer space on ALE that can be utilized for OOBFC-signaled EoQ includes the 20-GB straight egress buffer and the 10-GB buffer for ingress traffic. NFE Buffer and Egress Queue Monitoring on N9K-X9500 Series Line Cards The show hardware internal buffer info pkt-stats detail command displays dynamic buffer and queue statistics for all ports on NFE on a per-traffic-class per-queue basis. Each port has six classes: Q3, Q2, Q1, Q0, CPU, and SPAN. Classes Q3 through Q0 are the user traffic classes, corresponding to the following queuing classes in the QoS configuration: Q3 -- c-out-q3 Q2 -- c-out-q2 Q1 -- c-out-q1 Q1 -- c-out-q-default Each user traffic class has an OOBFC unicast queue, a non-oobfc unicast queue, and a multicast queue. Classes CPU and SPAN each have a unicast queue and a multicast queue Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 74

21 Following is a sample output of the show hardware internal buffer info pkt-stats detail command for a N9K- X9564PX or N9K-X9564TX line card. It shows the buffer utilization statistics for all of the NFE instances on the line card and the queue statistics for all of the active ports on each NFE instance Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 74

24 A N9K-X9564PX line card has two NFE instances. NFE instance 0 hosts the 48 1/10 GE front-panel ports and 12 internal ports connecting to ALE instance 0. In the command output, ASIC ports 1-60 are mapped to: Ports the internal ports connecting to ALE instance 0 Ports the 1/10 GE front-panel ports NFE instance 1 has four 40 GE front-panel ports and six internal ports connecting to ALE instance 1. In the command output, ASIC ports 1-10 are mapped to: Ports the internal ports connecting to ALE instance 1 Ports the 40 GE front-panel ports A variation of the above buffer and queue monitoring command shows the peak value of buffer utilization in each queue. Following is some of the sample output for the high-water-mark monitoring for a N9K-X9564PX or N9K- X9564TX line card Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 24 of 74

25 ALE Buffer and Extended Output Queue Monitoring on N9K-X9500 Series Line Cards The Cisco NX-OS system provides CLI commands to monitor the dynamic buffer and queue statistics on ALE instances on N9K-X9500 series line cards. For each ALE instance, the buffer utilization and queuing statistics are displayed separately for the three buffer regions: Ingress straight traffic MB Ingress hairpin traffic MB Egress straight traffic MB Queuing statistics include the ASIC port statistics and EoQ statistics. EoQ information is listed for each EoQ port that is associated to a particular front panel port on NFE. The CLI command show hardware internal ns buffer info pkt-stats detail displays the buffer utilization and queuing statistics for all ALE instances on the line card and statistics for all the active ports on each instance. Below is a part of a sample output of this command for a N9K-X9564PX or N9K-X9564TX line card. For the complete command output, refer to Appendix C Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 25 of 74

29 Queue Limit on Cisco Nexus 9500 Series Switches Queue limit can be defined on a per-port, per-class basis on Cisco Nexus 9500 Series Switches. It provides a mechanism to prevent a given port or a given traffic class from utilizing too much buffer resource resulting in buffer starvation to other ports or traffic classes. Queue limit can also be used to allocate more buffer space to a given port or a given traffic class when it is needed. Cisco Nexus 9500 Series Switches support static queue limit as well as dynamic queue limit. A static queue limit specifies the exact number of bytes, kilobytes, or megabytes that a particular traffic class can have in the queue. It can also be specified in the duration of time in milliseconds for packets to be allowed to remain in the queue. Static queue limit is helpful if precise buffer and queue control is desired for a particular traffic class on some ports while dynamic queue limit provides a flexible and dynamic method to control per-port, per-class queue limit. By selecting a dynamic queue limit factor from the options shown below, a user can specify how much of the available buffer space a per-port per-class queue is allowed to consume at any given point in time. Dynamic Queue Limit Factor Queue Limit in % of Available Buffer Space 0-1/128 1% 1-1/64 2% 2-1/32 3% 3-1/16 6% 4-1/8 11% 5-1/4 20% 6-1/2 33% % % % % 1. Dynamic queue limit renders optimal utilization of the buffer space while maintaining the capability of preventing a queue from consuming too much buffer resource. The default queue-limit setting is option 8. Option 8 allows per class, per queue to use up to 67 percent of available buffer space. In a case where the traffic on a port or for a particular class is anticipated to be bursty (occurring at intervals in short sudden episodes or groups), a user can adjust the queue limit for it to option 9 or 10 to utilize up to 89 percent of the available bandwidth Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 29 of 74

30 ALE Flow Prioritization ALE has built-in intelligence that is capable of prioritizing flows based on their life spans. Given a mixture of long, live flows and short, bursty flows, ALE can recognize and prioritize the short flows. In the event of link congestion, where the switch has to drop some packets, ALE will first drop packets from the long flows while allowing short flows to go through without packet loss. Many data center applications use long flows for data transport while using short flows for state synchronization or making requests. These short flows are more sensitive to packet drops or latency. By prioritizing these short flows over the long-lasting data-transport flows, the ALE or ALE-2 flow prioritization feature can help improve data center application performance. Conclusion Cisco Nexus 9500 Series Switches are designed to provide high-performance, cost-effective network solutions for next-generation data centers. The 9500 Series industry-leading 1/10/40 GE/100GE (future) port density can enable organizations to migrate their data center networks from 1 GE to 10 GE for host access on the access layer, and from 10 GE to 40 GE or 100GE in the future on the aggregation layer. Our switches can also enable the non-blocking spine-leaf fabric design. The efficient egress queuing and extended output queuing architecture supports consistent, end-to-end congestion management behavior within the data center network, and provides more buffer space on the access layer to accommodate the co-existence of different port speeds and in-case traffic patterns. Augmented by the rejuvenated Cisco NX-OS infrastructure and rich programmability functions, Cisco Nexus 9500 Series Switches, with the dynamic shared buffer architecture and egress queuing mechanism, can enable new data center network designs with a focus on application performance and workflow automation. For More Information For more information, go to: Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 30 of 74

31 Appendix Appendix A: Buffer and Queue Monitoring on the N9K-X9600 Series Line Cards Note that the following outputs are captured when the Nexus 9508 switch is configured to run in the 8-queue mode. When it s running in the 4-queue mode, the same command outputs will display queues including Q3, Q2, Q1, Q0, CPU and SPAN. The rest parts of the command outputs remain unchanged Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 31 of 74

38 Appendix B: Buffer and Queue Monitoring on the N9K-X9400 Series Line Cards Note that the following outputs are captured when the Nexus 9508 switch is configured to run in the 8-queue mode. When it s running in the 4-queue mode, the same command outputs will display queues including Q3, Q2, Q1, Q0, CPU and SPAN. The rest parts of the command outputs remain unchanged. Buffer and Queue Monitoring on the N9K-X9432PQ Line Card 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 38 of 74

49 Appendix C: Buffer and Queue Monitoring on N9K-X9500 Series Line Cards NFE Buffer and Queue Monitoring on N9K-X9564PX and N9K-X9564TX Line Cards 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 49 of 74

Cisco Nexus 9508 Switch Power and Performance

White Paper Cisco Nexus 9508 Switch Power and Performance The Cisco Nexus 9508 brings together data center switching power efficiency and forwarding performance in a high-density 40 Gigabit Ethernet form