Hardware Telemetry. About Streaming Statistics Export (SSX) Packet Format. About Streaming Statistics Export (SSX), on page 1

Size: px

Start display at page:

Download "Hardware Telemetry. About Streaming Statistics Export (SSX) Packet Format. About Streaming Statistics Export (SSX), on page 1"

Eunice Alexander
5 years ago
Views:

About Streaming Statistics Export (SSX), on page 1 About Streaming Statistics Export (SSX) Packet Format The Streaming Statistics Export (SSX) module reads statistics from the ASIC and sends them to

1 About Streaming Statistics Export (SSX), on page 1 About Streaming Statistics Export (SSX) Packet Format The Streaming Statistics Export (SSX) module reads statistics from the ASIC and sends them to a remote server (collector) for analysis. SSX can read a register or (directly accessible) memory of the switch. The contents of registers and the packet format are specified by an instruction sequence. Statistics are assembled as a packet with a flexible MTU size and a user-defined header. These packets are sent through one of the front-panel ports of the switch. Headers are configured by software, and packets streamed are always UDP packets. Different headers can be configured and selected with an SSX instruction set. An ASIC can send the statistics (packets) to different collectors. Each header is followed by a metadata provided by the ASIC. The metadata contains information such as PTP timestamp and sequence number. A monotonically increasing packet sequence number is maintained by the ASIC for each header. This allows the collector to detect a lost packet by monitoring the sequence number. The packet payload (statistics) consists of TLV formatted data (T=type, L=length, V=values). Type specifies the contents of the TLV; length specifies the number of bytes contained in the TLV; and values is a list of statistic values. (Type also specifies which decoder required to extract the values in the TLV.) SSX architecture provides the flexibility for software to export statistics with programmable header formats. The headers are obtained from header memory. The config register CFG_header_base points to the start of the header memory. In this release, all the statistics are exported as a payload of a UDP packet. Each header entry in the memory corresponds to a collector. The header_idx field in the HDR instruction is used to pick collector to which to send the packet to. Having the flexibility to choose the collector allows for picking multiple tunnel end points for different kinds of data thus balancing the load across a set of collectors. For example, a given engine may receive stats only from a particular slice of the chip or receive all the statistics from only one chip in the system and so on. It is important to note that the IP_header length and header checksum are programmed correctly by the software. Software must ensure that TLVs that form the payload of this packet does not exceed the length specified in the header. If the TLV size is smaller than the packet length, ASIC will pad appropriate number of zeros. 1

Metadata Format Metadata Format An exported packet consists of one packet header (selected by the instruction HDR ), metadata and one or more TLV headers (selected by the instruction TLV ).

2 Metadata Format Metadata Format An exported packet consists of one packet header (selected by the instruction HDR ), metadata and one or more TLV headers (selected by the instruction TLV ). Each TLV header is followed by statistics values. The CRC for the packet are computed by the ASIC. Metadata fields precede the actual statistics data. It is inserted by the ASIC. typedef bitset metadata_t { big_endian 1; field ptp_tstp { bits 64; } field seq_num { bits 32; } field system { bits 16; } field board { bits 8; } field asic { bits 4; } field ver { bits 4; } } The metadata contains the following information: Field PTP timestamp Sequence number System ID ASIC Board Ver A 64-bit number that represents the time elapsed since epoch. The higher order 32 bits represent seconds and lower 32 bits represent nanoseconds. A monotonically increasing packet sequence number is maintained by the ASIC for each header. This allows the collector to detect any lost packet by keeping track of the sequence number. A configurable ID that is used to identify the source switch for each packet. Indicates the type of ASIC. Indicates the type of board. Indicates the software version of SSX. Note In order to notify the collectors of sequence of each UDP delivery, a monotonically increasing seq_num (32bits) is inserted by the ASIC into the packet s metadata record. This seq_num is unique per collector. Seqnum is maintained alongside each header entry, and is stored in the address immediately following the last address of the header entry. ASIC increments the seq_num (rolls over) in the header memory once a packet to the given collector is exported. Payload Each SSX packet contains a single TLV header. The packet payload consists of TLV formatted data (T=Type, L=Length, V=Values). Type specifies the contents of the TLV; Length specifies the number of bytes contained in the TLV; Values is a list of statistics values. 2

3 SSX Applications The Type field is divided into several sub-fields: bitset type { big_endian 1; field group { bits 8; } field block { bits 8; } field index { bits 32; } } The TLV header contains the following: Field Type Length Sub-Field Group Block Index Slice number from where the statistics are read. Indicates the block within the ASIC (such as BAX, PRX, MAC, etc.) The higher order 24 bits of the index indicate the counter index for each counter, which comes from the TLV file. When the SSX monitor is applied on the system, a TLV file is generated and saved as /mnt/pss/asic_tlv.cfg. Refer to the TLV file to determine the TLV encoding in the packets that are streamed by the switch for each counter type. The lower order 8 bits of the index indicates the packet index, which is needed when more than one packet is needed to fit the data for a single counter index. Indicates the length of the value in the payload. SSX Applications SSX supports many opcodes and general purpose registers to host a variety of applications. For standalone streaming statistics, certain user-specified statistics can be exported from ASIC to one or more external analyzers. Examples of these user-specified statistics include: Queue congestion Queue drops RMON counters Configuring SSX Configuring SSX on a switch requires: Enabling the SSX feature Configuring an SSX collector Configuring an SSX record Configuring an SSX monitor Applying an SSX monitor 3

4 Configuring SSX Examples of these are shown below. Enabling the SSX Feature switch# conf Enter configuration commands, one per line. End with CNTL/Z. switch(config)# feature hw_telemetry HW Telemetry enabled switch(config)# switch(config)# hardware-telemetry ssx switch(config-ssx)# Configuring an SSX Collector switch(config)# ssx collector collector1 switch(config-ssx-collector)#? destination Destination configuration dscp DSCP mtu MTU no Negate a command or set its defaults source Source configuration end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in switch(config-ssx-collector)# source ip-address switch(config-ssx-collector)# destination ip-address switch(config-ssx-collector)# source udp-port? < > UDP port to be configured, default switch(config-ssx-collector)# source udp-port switch(config-ssx-collector)# destination udp-port? < > UDP port to be configured, default switch(config-ssx-collector)# destination udp-port switch(config-ssx-collector)# mtu? < > MTU port to be configured switch(config-ssx-collector)# mtu 1500 switch(config-ssx-collector)# dscp? <0-63> DSCP to be configured switch(config-ssx-collector)# dscp 0 Configuring an SSX Record switch(config)# ssx record record1 switch(config-ssx-record)#? collect Stats type to be configured interval Time interval to be configured no Negate a command or set its defaults end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in switch(config-ssx-record)# collect? eg-q-depth eg-q-drops eg-q-microburst 4

5 SSX Counters eg-q-peak ing-q-depth ing-q-drops ing-q-peaks mac-mibs switch(config-ssx-record)# collect eg-q-depth switch(config-ssx-record)# interval? < > Time interval in increments of 100ms switch(config-ssx-record)# interval 100 Configuring an SSX Monitor switch(config)# ssx monitor monitor1 switch(config-ssx-monitor)#? collector Collector to be configured no Negate a command or set its defaults record Record to be monitored end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in switch(config-ssx-monitor)# collector collector1 switch(config-ssx-monitor)# record record1 Applying an SSX Monitor switch(config)# system ssx monitor monitor1 SSX Counters The following counters are supported by SSX. The information in the following table applies for NX-OS Releases 7.0(3)I7(4) and earlier: Counter eg-q-depth eg-q-drops eg-q-microburst eg-q-peak ing-q-depth ing-q-drops mac-mibs Meaning Instant buffer utilization per output queue Tail drops per output queue Microburst records Buffer peak counters per output queue Instant buffer utilization per input queue Overflow drops per input queue RMON counters per MAC channel The information in the following table applies for NX-OS Releases 7.0(3)I7(5) and later: 5

6 SSX show Commands Counter egress queue average-depth egress queue depth egress queue drops egress queue microburst egress queue peak egress pool-group depth egress buffer depth ingress queue depth ingress queue drops ethernet counters Meaning Average depth per output queue Note If you upgrade from NX-OS Release 7.0(3)I7(5) to NX-OS Release 9.2(1), remove feature hardware-telemetry before the upgrade and re-enable it after the upgrade if you have this ssx collect option configured. Instant buffer utilization per output queue Tail drops per output queue Microburst records Buffer peak counters per output queue Streams the buffer occupancy per pool group Streams the buffer occupancy per slice Instant buffer utilization per input queue Overflow drops per input queue RMON counters per MAC channel SSX show Commands The following are show commands that are associated with SSX and hardware telemetry. The information in the following table applies for NX-OS Releases 7.0(3)I7(4) and earlier: Command show running-config show hw_telemetry ssx details show hw_telemetry ssx monitor all show hw_telemetry ssx collector all show hw_telemetry ssx record all show system internal hw_telemetry details show system internal hw_telemetry event-history debug show system internal hw_telemetry event-history errors Displays information about current running-config. Displays details about SSX. Displays details about SSX monitor. Displays details about SSX collector. Displays details about SSX record. Displays details about system internals. Displays details about system internal debug history. Displays details about system internal errors. 6

7 Example: Read MAC MIBs Command show system internal hw_telemetry event-history msgs show system internal hw_telemetry mem-stats detail show system internal access-list resource hw-telemetry detail Displays details about system internal messages. Displays details about system internal memory statistics. Displays details about system internal access list details The information in the following table applies for NX-OS Releases 7.0(3)I7(5) and later: Command show running-config show ssx details show ssx monitor all show ssx exporter all show ssx record all show system internal hardware_telemetry details show system internal hardware_telemetry event-history debug show system internal hardware_telemetry event-history errors show system internal hardware_telemetry event-history msgs show system internal hardware_telemetry mem-stats detail show system internal access-list resource hardware-telemetry detail Displays information about current running-config. Displays details about SSX. Displays details about SSX monitor. Displays details about SSX exporter. Displays details about SSX record. Displays details about system internals. Displays details about system internal debug history. Displays details about system internal errors. Displays details about system internal messages. Displays details about system internal memory statistics. Displays details about system internal access list details Example: Read MAC MIBs The following is an example of reading MAC MIBs. Read MIBs from the 16 MACs in slice0 of a N9K-C9508-FM-2 and send it using a UDP header of size = 48B and payload of size 1518B (including 4B CRC). Each MAC contains 91 counters, each being 64 bits (or 8 Bytes) and the start address for each MAC MIBs is separated by 0x8000. After each data set is sent, the ASIC will generate an interrupt (ITR 5) to indicate that the packets have been sent. 7

8 Example: Read MAC MIBs The following table shows the packet that is generated. The TLV index indicates which MAC the statistic belong to. Bytes Contents UDP Header Metadata TLV MIBs counter 1 MIBs counter 2... MIBS counter 90 PAD CRC 8

Configuring sflow. Information About sflow. sflow Agent. This chapter contains the following sections:

Configuring sflow. Information About sflow. sflow Agent. This chapter contains the following sections: This chapter contains the following sections: Information About sflow, page 1 Licensing Requirements, page 2 Prerequisites, page 2 Guidelines and Limitations for sflow, page 2 Default Settings for sflow,