Traffic Monitoring and Engineering for UCS Steve McQuerry, CCIE# 6108 Technical Marketing Engineer
Agenda UCS Networking Overview Network Statistics in UCSM Understanding Collection Policies Hotspot Detection Engineering to Avoid Hotspots 3
UCS Networking Overview
System Components: High-level Overview FABRIC INTERCONNECT CHASSIS IO MODULE (FEX) 61XX 62XXUP 2104XP 2204XP 2208XP 2232PP Cisco (VIC1280, VIC1225) 3 rd party INTERFACE CARDS 5
UCS Networking Overview Fabric Interconnects (10GE ports) Chassis Up to 8 half width blades or 4 full width blades Fabric Extender (FEX or I/O Module) Host to uplink traffic engineering Up to 160Gb Flexible bandwidth allocation Adapters Virtualized adapter for single OS and hypervisor systems Dual connected Blade or Stand Alone Server 6
UCS Networking Overview 1 st Generation Hardware vpc SAN A SAN B Port-Channel Fabric A FEX A Port-Channel Fabric B FEX B Mezz Mezz Mezz n i c n i c h a h a b b n i c n i c h a h a b b n i c n i c h a h a b b 7
UCS 2104 (1st Gen FEX) Server to Fabric Pinning slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 1 link NIF Fabric Interconnect Server slots pinned to uplink Uplink: slots 1,2,3,4,5,6,7,8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 2 links NIF Fabric Interconnect Uplink 1: slots 1,3,5,7 Uplink 2: slots 2,4,6,8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 4 links NIF Fabric Interconnect Uplink 1: slots 1,5 Uplink 2: slots 2,6 Uplink 3: slots 3,7 Uplink 4: slots 4,8 8
UCS Networking Overview 1 st Generation Hardware Fabric A IOM A Fabric B IOM B Mezz n i c n i c h a h a b b 9
UCS Networking Overview 2 nd Generation Hardware vpc SAN A SAN B Port-Channel Fabric A IOM A Port-Channel Fabric B IOM B Mezz Mezz Mezz n i c n i c h a h a b b n i c n i c h a h a b b n i c n i c h a h a b b 10
UCS 2204 (2 nd Gen FEX) Sever to Fabric Pinning slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 1 link NIF Fabric Interconnect Server slots pinned to uplink Uplink: slots 1,2,3,4,5,6,7,8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 2 links NIF Fabric Interconnect Uplink 1: slots 1,3,5,7 Uplink 2: slots 2,4,6,8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 4 links NIF Fabric Interconnect Uplink 1: slots 1,5 Uplink 2: slots 2,6 Uplink 3: slots 3,7 Uplink 4: slots 4,8 11
UCS 2204 (2 nd Gen FEX) Sever to Fabric Pinning Server slots channeled across all uplinks slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 4 links NIF Fabric Interconnect Uplink 1: slots 1-8 Uplink 2: slots 1-8 Uplink 3: slots 1-8 Uplink 4: slots 1-8 12
UCS 2208 (2 nd Gen FEX) Server to Fabric Pinning slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 1 link NIF Fabric Interconnect Server slots pinned to uplink Uplink: slots 1,2,3,4,5,6,7,8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 2 links NIF Fabric Interconnect Uplink 1: slots 1,3,5,7 Uplink 2: slots 2,4,6,8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 4 links NIF Fabric Interconnect Uplink 1: slots 1,5 Uplink 2: slots 2,6 Uplink 3: slots 3,7 Uplink 4: slots 4,8 13
UCS 2208 (2 nd Gen FEX) Server to Fabric Pinning Server slots pinned to uplink slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X F E X 8 links NIF Fabric Interconnect Uplink 1: slot 1 Uplink 2: slot 2 Uplink 3: slot 3 Uplink 4: slot 4 Uplink 5: slot 5 Uplink 6: slot 6 Uplink 7: slot 7 Uplink 8: slot 8 Server slots channeled across all uplinks 8 links Uplink 1: slots 1-8 NIF Fabric Interconnect Uplink 2: slots 1-8 Uplink 3: slots 1-8 Uplink 4: slots 1-8 Uplink 5: slots 1-8 Uplink 6: slots 1-8 Uplink 7: slots 1-8 Uplink 8: slots 1-8 14
UCS Networking Overview 2 nd Generation Fabric Interconnect Fabric A IOM A Fabric B IOM B Mezz n i c n i c h a h a b b 15
Network Statistics in UCSM
UCSM Core IBM Tivoli HPNNM Events {SNMP Traps} HPOM Polls {SNMP GET} Misc. SNMP Events {xml} HTTP {xml} CIM-XML SNMP Syslog Smash XML API UCS CLI U C S M CIM Object Translation AG blade DME AG FI AG chassis AG n DB Polling/notification 17
UCSM Statistics Root topsys sys equipmentchassis sys/chassis-1 equipmentchassisstats equipmentchassisstatshist sys/chassis-1/stats/1 sys/chassis-1/stats Statistics are instances of statscurr Statistics are contained by monitored object 5 most recent stats history records are contained by the current stats instance. Stats history are subclasses of the stats. Statistics dn is often deducible.... equipmentchassisstatshist sys/chassis-1/stats/5 18
Stats Collection Policy A Systemwide Set of Stats Collection Policies (One per Application Domain) Allow Configuration of Collection and Reporting Intervals. Domains: adapter, chassis, host, port, server When a Collection Interval Is Changed, the reportinginterval Is Restarted collectioninterval: frequency that endpoints will send stats updates to DME reportinginterval: frequency that DME will report stats updates to external collectors and update stats history 19
Network Statistics in UCSM Network statistics are collected by UCSM from the NX-OS software in the Fabric Interconnects. These are counters that are available for networking components Because of NIV technology the Fabric has visibility to the Cloud (LAN/SAN uplinks), the IOM and the server NIC. 20
Network Statistics in UCSM Access Statistics through: LAN or SAN tab (port-group) Devices tab ( server ports, network uplink, storage ports, and mezz ports,) Server Tab (vnic) 21
Network Statistics in UCSM FI to LAN Network Uplink (Cloud) vpc SAN A Port-Channel Port-Channel Fabric A Fabric B 22
Network Statistics in UCSM Port Channel is the Aggregate of all interfaces in the channel Statistic for the channel are the sum of the statistics of the members Individual member statistics are also visible in the system Network usage is measured against TX Total bytes and RX Total bytes 23
Network Statistics in UCSM FI to Uplink Port Channel Statistics 24
Network Statistics in UCSM FI to Uplink Individual Port Statistics 25
Network Statistics in UCSM FI to SAN Port Statistics SAN A SAN B Fabric A Fabric B 26
Network Statistics in UCSM FI to SAN Uplink (Cloud) Port Statistics A FC Port Channel is the Aggregate of all interfaces in the channel Statistic for the channel are the sum of the statistics of the members Individual member statistics are also visible in the system FC usage is measured against Bytes RX and Bytes TX 27
Network Statistics in UCSM FI to SAN Port Statistics 28
Network Statistics in UCSM FI to IOM (Internal LAN) Fabric A Fabric B IOM A IOM B 29
Network Statistics in UCSM FI to IOM (Internal LAN) 30
UCS Networking Overview Server to IOM n i c n i Mezz c h a h a b b 31
Network Statistics in UCSM Server to IOM vnic Port Statistics 32
Using SNMP tools to Monitor Traffic The network statistics in UCSM are pulled from the NX-OS statistics for the networking components. It is possible to monitor the traffic for the networking components using an external SNMP collection tool such as MRTG. The advantage is long term analysis, however most tools have no alerting capabilities. 33
Understanding Collection Policies
Understanding Collection Policies A collection Policy consist of a collection interval and a reporting interval The collection policy is set under the admin tab in Stats Management -> Collection Policies -> Collection Policy name A unique policy can be set for, Adapters, Chassis, FEX, Port, Server, and Host* Not all policies involve network components *Host is an unused policy in UCSM at present 35
Understanding Collection Policies 36
Understanding Collection Policies Collection Interval The collection interval is how often the system will query a device for statistics. The default collection interval is 60 seconds The more frequent the interval the more granular the data. We will use 30 seconds. The timing of the collection interval is important because it will be used in BW calculations for hotspot detection 37
Understanding Collection Policies Reporting Interval The reporting interval is internal to UCSM and determines how often UCSM will store data from the collection interval. This data is stored in tables and the last 5 reporting intervals are available for inspection in the system Reporting interval data is used to calculate minimum, maximum and average values shown in the statics view. 38
Understanding Collection Policies Select the Policy you want to change. Make selections and press the save changes button 39
Hotspot Detection
Hotspot Areas There are three potential hotspot locations for UCS network connectivity. 1.) Send and Receive between Fabric Interconnect and LAN/SAN Fabric Interconnects to LAN/SAN FEX to FI FEX to Host SAN A Port-Channel Fabric A FEX A SAN B Port-Channel Fabric B FEX B 2.) Send and Receive between FEX and Fabric Interconnect 3.) Send and Receive between Host and FEX n i Mezz cn i ch ah a b b n i Mezz cn i ch ah a b b n i Mezz cn i ch ah a b b 41
Hotspot Detection Threshold Policies To identify hot spots we will use Threshold policies in conjunction with collection policies to alert as we pass thresholds. A threshold is calculated by measuring a statistic against a policy. The policy measures change against a user defined normal value and turns on the alert between a users set high/low threshold and turns off the alert below the user set low threshold. High - Up High - Up High - Up Low - Up Low - Up Low - Up Normal Normal Normal 42
Hotspot Detection Calculating BW Threshold Limits for an Element The statistic we use to calculate bandwidth is the delta in bytes. This should be measured for both TX and RX This delta is calculated in bytes changed over a period of time defined by the collection interval, for example 30 seconds 43
if standardizing on one size was a good thing, standardizing on several sizes would be even better. Radia Perlman Interconnections: Bridges and Routers 44
Kilo, Mega, etc IEEE Networking Standards for Prefix Values What is a Kilobit? Once upon a time Computer professionals noticed that 2 10 was very nearly equal to 1000 and started using the SI prefix kilo to mean 1024. The rest of the world uses the term Kilo in the SI format to mean 1000. The result is that no one seems to know what kilo and mega mean. The IEEE standards board decided that the IEEE standards will use the conventional, internationally adopted, definitions of the SI prefixes. Therefore Mega will mean 1,000,000 and Kilo will mean 1,000.* In Telecom and by extension networking Kilo means 1000 and NOT 1024 The SNMP reported speed for a 10Gig link (on any platform) will be reported by SNMP is 1,250,000,000 Bytes (MaxBytes[10.29.148.6_436207616]: 1250000000) *Sources: NIST and IEEE 45
Hotspot Detection Calculating BW Threshold Limits for an Element For Ethernet the BW of a single link is 10Gbps First we determine our desired threshold for example 8Gbps We need to calculate the expected change in bytes over the collection interval. To calculate divide the desired BW by 8 bits per byte and then multiply by the collection interval by the time to get the expected delta in bytes for our collection period. Example 8 Gbps over 30 seconds = 30,000,000,000 bytes 8Gbps / 8bits per byte = 1,000,000,000 bytes per second 1,000,000,000 bytes per second * 30 seconds = 30,000,000,000 bytes 46
Hotspot Detection Threshold Calculations Speed in Gbps Percentage of BW Conversion to Bytes Delta expected over 30 second collection interval 10 100% 1,250,000,000 37,500,000,000 9 90% 1,125,000,000 33,750,000,000 8.5 85% 1,062,500,000 31,875,000,000 8 80% 1,000,000,000 30,000,000,000 7.5 75% 937,500,000 28,125,000,000 7 70% 875,000,000 26,250,000,000 6.5 65% 812,500,000 24,375,000,000 6 60% 750,000,000 22,500,000,000 5 50% 625,000,000 18,750,000,000 4 40% 500,000,000 15,000,000,000 3 30% 375,000,000 11,250,000,000 47
Hotspot Detection Threshold Alerts 48
Hotspot Detection Threshold Policies Placement Threshold policies can be configured for the following: Internal LAN IOM to FI LAN Cloud FI to Upstream Ethernet switches SAN Cloud FI to Upstream SAN switches Server Between the server NIC and the IOM 49
Hotspot Detection Threshold Policies Internal LAN Navigate to admin->stats management and expand the fabric. Select thr-policy-default and create a threshold class Choose Ether Tx Stats from the stat class and click next. 50
Hotspot Detection Threshold Policies Internal LAN 51
Hotspot Detection Threshold Policies Internal LAN 52
Hotspot Detection Threshold Policies Click the add button and select Ether Tx Stats Total Bytes Delta as the property type enter 0.0 as the normal value Select the Alarm triggers you want to get and enter your values and click OK Click finish to be returned to the policy. Click the classes tab to see your policy. 53
Hotspot Detection Threshold Policies Internal LAN 54
Hotspot Detection Threshold Policies You will need to add another class for RX traffic. Click the + bottom to the right and repeat the steps from the previous policy choosing Eter Rx Stats as the stats class this time Click save changes once you have completed the steps 55
Hotspot Detection Threshold Policies 56
Hotspot Detection Threshold Policies For Uplinks you can repeat this process for the LAN cloud. For SAN use a single Stats class fcstats and create a definition for rx and tx stas under the same stats class 57
Hotspot Detection Threshold Policies 58
Hotspot Detection Threshold Policies For the vnic port you will need to create a threshold policy to be used with a service profile. Go the the appropriate organization level and select create threshold policy 59
Hotspot Detection Threshold Policies 60
Hotspot Detection Threshold Policies Give the policy an applicable name and description and press next Choose the vnic stats class and create a single threshold with the rx bytes delta and the tx bytes delta. 61
Hotspot Detection Threshold Policies 62
Hotspot Detection Threshold Policies Apply the threshold policy to the Service profile of the servers you want to monitor 63
Hotspot Detection Threshold Policies When a server reaches a threshold you will receive an alert on UCSM. This will exist while threshold is exceeded once it drops below the definition the alert will disappear Alerts show as system faults as defined by the threshold policy UCSM will send a TRAP when an alert is generated. The TRAP will be sent regardless of what level of alert has been set. 64
Hotspot Detection Threshold Policies 65
Hotspot Detection When a threshold policy has been crossed UCSM will send out an SNMP Trap alert This assumes that UCSM has been configured for SNMP operation and the Trap Receiver has been defined This feature was added in UCSM version 2.x.x 66
Hotspot Detection Long Term Monitoring vs. Point in Time UCSM has a limited amount of space to store statistics and can only provide the last 5 instances of collection intervals. It provides a good tool for monitoring thresholds, but does not provide a long term monitoring solution. Tools like MRTG are great for Long Term analysis of BW usage, but may not provide the required alerting and usually require a great deal of customization to be usable UCS Central version (TBD) will provide tools for both alerting as well as long term analysis of traffic within systems it manages. 67
Automating Threshold Policies GoUCS and CDN Creating threshold policies for the systems require time and effort. Isn t there some way to push standard configurations to the system? GoUCS is an automation tool available from the Cisco Developer Network (CDN) that allows users to build XML configuration scripts that can be sent to UCS. PowerTool can also be used to automate this process http://developer.cisco.com/ 69
Engineering to Avoid Hotspots
Traffic Engineering vnics can be pinned to a specific FI when created (with configurable failover to other switch) Depending on requirements, vnics could be pinned to one interconnect or distributed evenly vnic-1 vnic-2 FEX-1 2 Fabric Extenders in chassis, each with 1 link to the FI. vnic-3 VIC with 3 vnics Blade-2, VIC-1 vnic-1 vnic-2 Blade-1, VIC-1 FI-1 vnics in System Class C pinned to one interconnect 2 FI, both with 1 connection to each FEX vnic-3 FEX-2 Class-A Class-B Class-C FI-2 71
Controlling Pinning in Profile From the each server a fabric interconnect can be chosen to balance the traffic 72
Traffic Engineering vnics can be pinned to a specific FI when created (with configurable failover to other switch) Depending on requirements, vnics could be pinned to one interconnect or distributed evenly vnic-1 vnic-2 FEX-1 vnic-3 Blade-1, VIC-1 FI-1 vnics in System Class C distributed across interconnects Blade-2, VIC-1 vnic-1 vnic-2 vnic-3 FEX-2 Class-A Class-B Class-C FI-2 73
QoS Architecture SAN G G G A G Compute Chassis Fabric R Extender x8 M Adapter B LAN S Fabric Switch G I x8 C MGMT I x8 R x8 X X X X X X x86 Computer x86 Computer Compute Blade (Half slot) C S G Fabric Switch G P Adapter B G A G G P Adapter Compute Blade (Full slot) SAN Fabric Extender No packet drops within the array Largest buffers are on switch and host memory, so congestion pushed to edges Priority Flow Control (PFC) used to ensure packet drops are at vnic or Switch All traffic in a CA system belongs to 1 of 6 System Classes Four are user configurable while the other two are for FCoE and standard Ethernet QoS parameters can be configured at a per system class level, or a per vnic level. 74
System Buffering/Queuing 75
User Configuration Users configure QoS parameters at two levels Globally for each System Class COS value for packets in this class Drop/No-drop behavior Strict Priority Example Class Name FC Bronze COSValue 3 0 Drop/No-Drop No-Drop Drop Strict Priority No No Bandwidth/Weight 20% 30% Bandwidth/Weight 76
User Configuration Users configure QoS parameters at two levels Example: Logical Server A vnic1 vnic2 vnic3 Class FC FC Bronze Rate 4000 4000 5000 Burst 300 400 100 For each vnic (Egress properties) System Class for traffic from this vnic Rate limit (Mbps) Burst Size (Kbytes) 77
User Configuration Example Global System Class Definitions Class Name FC Gold Ethernet BE COS Value 3 1 0 Drop/No-Drop No-Drop Drop Drop Strict Priority No No No Bandwidth/Weight 1 (20%) 3 (60%) 1 (20%) FC Traffic High Priority Ethernet Best Effort Ethernet Logical Server A vnic1 vnic2 vnic3 Class FC FC Eth. BE Rate 4000 4000 5000 Burst 300 400 100 Logical Server B vnic1 vnic2 Class Gold Eth. BE Rate 600 4000 Burst 100 300 78
QoS Tools Transmit Queues One Two Three Four Five Six Seven Eight Priority Flow Control STOP Ethernet Link Receive Buffers One Two Three Four Five PAUSE Six Seven Eight Enables lossless Fabrics for each class of service PAUSE sent per virtual lane when buffers limit exceeded Eight Virtual Lanes 3G/s COS based Bandwidth Management Offered Traffic 3G/s 2G/s 3G/s 3G/s 3G/s 3G/s 4G/s 6G/s 10 GE Link Realized Traffic Utilization 3G/s 3G/s 3G/s HPC Traffic 3G/s Storage Traffic 3G/s LAN Traffic 4G/s t1 t2 t3 t1 t2 t3 Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth 802.1Qaz Enhanced Transmission 2G/s 3G/s 5G/s Among the tools used are aggregate shapers at the vnics (VIC Adapter), ETS, Policers at the switch for each vnic. 79
QoS Configuration in UCSM Enable QoS Classes in UCSM 80
QoS Configuration in UCSM Create a QoS Policy 81
Applying QoS to a Policy Apply Policy to Adapter 82
Summary UCSM is designed for optimized traffic flow Stats Management and Threshold policies allow you to monitor traffic levels QoS and Traffic engineering tools allow you to manage potential bottlenecks in UCS 83
Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Cisco Daily Challenge points for each session evaluation you complete. Complete your session evaluation online now through either the mobile app or internet kiosk stations. Maximize your Cisco Live experience with your free Cisco Live 365 account. Download session PDFs, view sessions on-demand and participate in live activities throughout the year. Click the Enter Cisco Live 365 button in your Cisco Live portal to log in. 84