Medianet 2.3 Media Monitoring

Transcription

1 Troubleshooting Guide Medianet 2.3 Media Monitoring Troubleshooting Guide Last updated: May 2012 Contents 1 Introduction Prerequisites Medianet Overview Performance Monitor Overview Mediatrace Overview IPSLA VO Overview Medianet Supported WAN Deployments and Configurations Medianet Quality of Service Overview Medianet SNMP Configuration and MIBs Performance Monitor Troubleshooting Performance Monitor Common Problems & Solutions Troubleshooting Performance Monitoring using IOS debugging Debug Commands for Performance Monitor Mediatrace Understanding Mediatrace Configuring Mediatrace Debugging Mediatrace Output IP SLA VO: IPSLA video operation cannot be seen in Perf-Mon stat s IP SLA VO fails to collect any usable data (show ip sla statistics) Some IP SLA Video Operations fail while others pass Troubleshooting procedures for IP SLA VO failures Medianet Media Monitoring Network Topology and Configuration Example Site#1 Router (3845-1) Site#2 Router (3845-2) Core Hub Site Router (3925-1) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 1 of 102

2 1 Introduction Cisco Medianet Media Monitoring capabilities provide network operators with a set of tools to monitor the packet performance of critical business applications. This document provides information explaining how to differentiate problems with the Media Monitoring capabilities, as it relates to network performance problems. Media Monitoring features within Medianet release 2.3 include support on the ISR-G2 for Performance Monitor IPv6 and IPSLA VO support on the Catalyst 4K switch and ISR-G2 router platforms. Refer to the product datasheets and configuration guides to determine which ISR platforms support IPSLA VO. Refer to the respective sections for troubleshooting guidance for any of the 2.2 and 2.3 Medianet media monitoring features. Please refer to the Medianet datasheet to review all support platforms for the media monitoring features. 2 Prerequisites Users should be familiar with the following topics: 1. Modular QoS Configuration Medianet Campus QoS Design 2. Flexible NetFlow Flexible NetFlow Configuration Guide 3. RSVP-Transport (RSVP-TP) RSVP Transport for Medianet: The RSVP Transport for Medianet feature extends RSVP to act as a transport mechanism for the clients. The following section provides information about this feature: Transport Mechanism Support in RSVP How to Configure RSVP The following commands were introduced or modified: ip rsvp transport, ip rsvp transport sender-host, show ip rsvp transport, show ip rsvp transport sender. Components Used The Cisco Medianet Media Monitoring features discussed in this document are restricted to the software and hardware versions listed in the Medianet Datasheet. 3 Medianet Overview Cisco Medianet is an end-to-end architecture that allows network operators to optimize the network platform to deliver rich media experiences over an IP network. Cisco Medianet addresses these business objectives: Accelerates application deployment Enables media traffic network performance visibility Scales network infrastructure for best quality of experience 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 2 of 102

3 Media Monitoring enhances visibility into the network to simplify, baseline, and accelerate troubleshooting of RTP and TCP multimedia and critical business applications. It allows operators to validate network performance and network quality of service configurations before deploying new applications or scheduled network media events. Media Monitoring is composed of three features: Performance Monitor, Mediatrace, and IP SLA Video Operation (IP SLA VO). These three features form a suite of tools to help enable you to perform media performance monitoring and troubleshooting. Performance Monitor allows you to analyze the performance of rich-media traffic across the network to provide a holistic view of the network service being delivered. Mediatrace discovers Layer 2 and Layer 3 nodes along a flow path. Two major Mediatrace profiles can provide a dynamic hop-by-hop analysis of media flows and system utilization metrics in real time to facilitate efficient and targeted diagnostics: performance-monitor and system profiles. IP SLA VO generates synthetic traffic streams that are very similar to real media traffic. It can be used in conjunction with Mediatrace to perform capacity planning analysis and troubleshooting even before applications are deployed. To learn more about these and other features please visit the Medianet Knowledge Base Portal where you will find high-level, as well as detailed documents, that can help you deploy, configure and use the Medianet features. 3.1 Performance Monitor Overview Performance Monitor is a Medianet Cisco IOS Software feature that measures the performance of Real Time Protocol (RTP), TCP, and IP Constant Bit Rate (CBR) traffic throughout a network. Performance Monitor analyzes RTP-based audio and video flows and reports on service-affecting metrics, like packet loss and network jitter. For TCP flows, Performance Monitor reports on round-trip time and packet loss occurrences. By measuring network performance on a hop-by-hop basis, the performance metrics collected along the network path allows granular fault isolation and simplifies troubleshooting of user traffic flows. Performance Monitor measures media flow performance and can generate alerts based on thresholds, and reports the collected data through CLI or a network management system. Performance Monitor maintains historical data about analyzed flows traversing routers and switches. The metrics collected by Performance Monitor can be exported to a network management tool through NetFlow Version 9 or collected by SNMP. Network management software can further analyze, summarize, and correlate this information to provide traffic profiling, baselining, and troubleshooting services for the application and network operator of the user network. Performance Monitor uses the Common Classification Engine (CCE), which is also used by the modular QoS command method to classify collected data. The existing Flexible NetFlow cache is leveraged by performance monitoring for traffic metering and reporting. Media Monitoring for Medianet 2.3 supports Performance Monitor for IPv6 media types, as well as IPv4. Flow records can be created to monitor either IPv4 or IPv6 as well as both IPv4 and IPv6. An example of this type of flow record is provided in section Please note that only one service policy can be applied to a given interface at a time. If you need to monitor both IPV4 and IPV6 media types on an interface please follow the configuration example provided in section which matches on both types of media Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 3 of 102

4 Figure 1 illustrates the configuration model for Performance Monitor on a Cisco IOS device. Note that this model applies to both IPv4 and IPv6 media types. Figure 1. Performance Monitor Configuration Model Performance Monitor Configuration Guide 3.2 Mediatrace Overview Mediatrace dynamically installs the performance monitoring configuration on each intermediate hop in the network, as a network performance inquiry message traverses across the network. Mediatrace allows network operators to dynamically collect traffic performance metrics from network devices along the media path. The source of the data (information provider) and the data collected are listed in Table 1. Table 1. Mediatrace Performance Metric Collection: Information Provider and Data Collected Information Provider RSVP-TP RSVP-TP SNMP MIBs SNMP MIBs SNMP MIBs Performance Monitoring Data Collected Interface Type Interface Name CPU Utilization Memory Utilization Interface Statistics Media Flow Statistics Cisco IOS Mediatrace helps network operators understand the status and health of a network path by collecting system information via hop by hop for specific IP traffic flows as it traverse the network. Mediatrace provides Layer 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 4 of 102

5 2 and Layer 3 node discovery using multiple profiles polling each node for information, such as flow statistics, incoming and outgoing interfaces, path/node discovery, and system utilization. This feature dynamically configures each performance monitoring policy as well as retrieves data from each Medianet capable node. Mediatrace can be configured to collect data periodically or use adhoc polling using Mediatrace polls. Mediatrace Configuration Guide 3.3 IPSLA VO Overview Cisco IP Service Level Agreements (IP SLA) is a mature Cisco technology that has found world-wide acceptance across varied industry verticals. IP SLA is deployed to: Measure end-to-end application and network performance Monitor health of a deployed network via thresholds and alerts Reduce time and cost of troubleshooting network issues in real time Provide pre-deployment assessment thereby accelerating application deployment IP SLA achieves these objectives by: Leveraging Cisco IOS to create synthetic traffic to simulate applications, referred to as application probes. These probes are created using Cisco routers and switches. Actively monitoring these probes and deriving end-to-end metrics like round-trip time, packet loss, jitter and one-way delay Help network administrators to identify baselines from different points in the network, and determine if these baselines are violated IP SLA Video Operation (VO) is a new type of IP SLA probe that generates video traffic with the intention of stressing the network with the same traffic characteristics as a real video endpoint/application would. IP SLA VO is a Cisco Medianet Media Monitoring technology that enables synthetic traffic generation and monitoring using Cisco routers and switches. IP SLA VO simulates the following video formats (dynamic-rtp video payload): TelePresence (6.6 Mbps) IP Video Surveillance Camera (2.2 Mbps) IP TV (2.6 Mbps) IP SLA VO provides the following statistics: Inter packet delay variation (jitter) One way delay Source to destination packet loss Tail drop Out of sequence packets 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 5 of 102

6 IPSLA White Papers: Cisco IOS IP Service Level Agreements white paper Cisco IP Service Level Agreement Video Operation white paper Medianet Media Monitoring 2.3 now features IPSLA VO capabilities on several platforms as shown in the Table 2. There are some limitations currently when using IPSLA VO from one platform to a different platform type and model. Please refer to the IP SLA Video Operation Across Platforms white paper to learn how each platform interoperability issues and platform dependancies required when using IPSLA VO across different platforms. Table 2 shows each platform which supports IP SLA VO and the IOS image required. Table 2. Supported Platforms for IP SLA VO Platform Cisco Catalyst 3K Series Sender Requirements No platform-specific requirement Responder Requirements No platform-specific requirement Cisco Catalyst 4K Series SUP-7E, SUP-7LE SUP-7E, SUP-7LE, SUP-6E Cisco ISR G2 Series PVDM3 available on 2900, 3900 platforms No DSP requirement; 1900, 2900, 3900, series Starting from Image 12.2(58)SE2 15.1(1)G 15.2(2)T License Requirements (Sender and Responder) IPBase/IPBase IPBase/IPBase UCk9/IPbase Table 3 shows some of the issues existing with the associated platform and Cisco IOS images noted. Please refer to the IPSLA Video Operation Across Platforms white paper to learn how each platform interoperates and the issues with respect to each platform required when using IPSLA VO across different platforms. Table 3. IP SLA VO Issues By Platform and Image Version Platform Image Version Cisco Catalyst 3K 12.2(58)SE Responder Cisco Catalyst 4K 15.1(1)SG Responder Cisco ISR G2 15.2(2)T Responder Cisco Catalyst 3K Series 12.2(58)SE2 Sender Jitter inaccurate (CSCts88988) Latency inaccurate (CSCtx45855) OK Cisco Catalyst 4K Series 15.1(1)SG Sender Latency inaccurate (CSCtx45855) Jitter inaccurate (CSCts88988) Latency inaccurate (CSCtx45855) Latency inaccurate (CSCtx45855) Cisco ISR G2 Series 15.2(2)T Sender Jitter inaccurate (CSCts88988) Bit Rate inaccrate (CSCtx35647) Host emulate fails (CSCtx55732) Latency inaccurate (CSCtx45855) Host emulate fails (CSCtx55732) OK 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 6 of 102

7 Each platform has some minor differences with configuration and CLI command syntax with the biggest change being the ISR-G2 platforms, which uses DSP s to generate and create video traffic profiles. Refer to the configuration guides for each platform to determine how best to configure each platform. Here are some basic differences to be aware of while using IPSLA VO: 1. IPSLA VO is supported on three Cisco platforms: Catalyst 3K series, Catalyst 4K series and the ISR G2 series. CLI configurations are very similar however the ISR-G2 uses DSP s to generate video, but can act as a responder without using DSP resources. 2. IPSLA VO feature on the Cat3K and Cat4K supports three different profiles; TelePresence, IPVSC and IPTV, however there are some slight differences between the implementation of these profiles so they are not exactly the same (Cat4K uses newer codec implementation). 3. The ISR-G2 supports building custom profiles using IOS CLI, however the Cat3K must use a Windows based application and a PCAP sample capture of the traffic type to build a profile template to implement the custom profile on the Cat3K. 4. The ISR-G2 platform has many Pre-packaged profiles available whereas the Cat3K and Cat4K only have three (TelePresence, IPVSC and IPTV). 5. The Cat4K does not support custom profiles. 6. The ISR-G2 supports a feature called emulated source which does not exist on the Cat3K or Cat4K. You can only use this feature when the Initiator and the Responder are an ISR-G2 platforms. The following configuration is an example of how to setup an IP SLA VO session: Cat3K and Cat4K Configuration example: ip sla 1 video source-ip source-port 6666 profile TELEPRESENCE dscp cs4 duration 60 freq 90 ip sla schedule 1 start-time now life 600 ageout 360 ISR-G2 Configuration example: Configure PVDM3 resources on ISR-G2 motherboard (voice-card 0). voice-card 0 voice-service dsp-reservation 0 //allows video/voice to share DSP s equally. ip sla 7 video source-ip source-port 3082 profile CTS-720P- Better duration 500 frequency 1800 ip sla schedule 7 recurring start-time now IP SLA VO Network Management Systems (NMS) IPSLA VO can be used via Cisco IOS CLI or with Cisco NMS solutions as well as other third party NMS vendors (see: Vendors and Partners). Cisco Prime Collaboration Manager (CM) provides a powerful web-based user 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 7 of 102

8 experience for managing and quickly troubleshooting end-to-end video collaboration over a borderless network. In conjunction with Cisco Medianet, Cisco Prime Collaboration Manager delivers richer end-to-end information that helps network operators easily identify and isolate video issues. Collaboration Manager uses Cisco Medianet features to provide enhanced path discovery, media-path packet statistics data collection, and synthetic traffic generation. CM provides the ability to use synthetic video traffic from network devices to assess video performance on a network. CM uses Medianet IP SLA VO to stress test a network to verify that the network can deliver quality video. Cisco Prime Collaboration Manager Website IPSLA SNMP-based network management MIBs: CISCO-VIDEO-TC CISCO-RTTMON-MIB CISCO-IPSLA-VIDEO-MIB CISCO-IPSLA-VIDEO-PROFILE-MIB CISCO-IPSLA-VIDEO-PROFILE-CAPABILITY 3.4 Medianet Supported WAN Deployments and Configurations It is recommended that Performance Monitor service policies be applied to WAN interfaces to provide performance data about media flows; in particular, for incoming traffic arriving from the WAN where media flows are the most at risk. When using DMVPN, IP Security (IPSec) over GRE, be sure to apply the Performance Monitor policy to the tunnel interface. If it is attached to the physical interface, Performance Monitor will not be able to compute statistics, because it receives encrypted data. Also note that GETVPN interfaces are not supported, because the interfaces are enabled for GETVPN encryption before the performance monitoring process receives the packet. Figure 1 provides an example of an DMVPN Tunnel Configuration. Figure 1. DMVPN Tunnel Configuration Site#1 - Router crypto isakmp policy 10 encr 3des authentication pre-share group 2 crypto isakmp key dmvpncae address crypto ipsec transform-set txform esp-3des esp-sha-hmac crypto ipsec profile caevpnprofile set transform-set txform interface Tunnel1 ip address Hub - Router crypto isakmp policy 10 encr 3des authentication pre-share group 2 crypto isakmp key dmvpncae address crypto ipsec transform-set txform esp-3des esp-sha-hmac crypto ipsec profile caevpnprofile set transform-set txform interface Tunnel0 bandwidth Site#2 - Router crypto isakmp policy 10 encr 3des authentication pre-share group 2 crypto isakmp key dmvpncae address crypto ipsec transform-set txform esp-3des esp-sha-hmac crypto ipsec profile caevpnprofile set transform-set txform interface Tunnel1 bandwidth Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 8 of 102

9 no ip redirects ip mtu 1400 ip nbar protocol-discovery ip flow ingress ip flow egress ip pim sparse-dense-mode ip nhrp authentication cisco ip nhrp map ip nhrp network-id 123 ip nhrp holdtime 360 ip nhrp nhs ip tcp adjust-mss 1360 ip ospf network point-to-multipoint ip ospf cost 10 delay 1000 tunnel source GigabitEthernet0/0 tunnel destination tunnel key tunnel protection ipsec profile caevpnprofile service-policy type performance-monitor input All-Media service-policy type performance-monitor output All-Media ip address no ip redirects ip mtu 1400 ip nbar protocol-discovery ip pim sparse-dense-mode ip flow ingress ip flow egress ip nhrp authentication cisco ip nhrp map multicast dynamic ip nhrp network-id 123 ip nhrp holdtime 360 ip tcp adjust-mss 1360 ip traffic-export apply gig0-cap size ip ospf network point-to-multipoint ip ospf cost 1 delay 1000 tunnel source GigabitEthernet0/0 tunnel mode gre multipoint tunnel key tunnel protection ipsec profile caevpnprofile service-policy type performance-monitor input All-Media service-policy type performance-monitor output All-Media ip address no ip redirects ip mtu 1400 ip nbar protocol-discovery ip flow ingress ip flow egress ip pim sparse-dense-mode ip nhrp authentication cisco ip nhrp map ip nhrp network-id 123 ip nhrp holdtime 360 ip nhrp nhs ip tcp adjust-mss 1360 ip ospf network point-to-multipoint ip ospf cost 10 delay 1000 tunnel source GigabitEthernet0/0 tunnel destination tunnel key tunnel protection ipsec profile caevpnprofile service-policy type performance-monitor input All-Media service-policy type performance-monitor output All-Media Cisco recommends applying the Performance Monitor policy to the tunnel interface and never applying the same Performance Monitor policy to both the physical interface at GigabitEthernet0/0 and the tunnel interface, which would be an unsupported configuration that would cause large amounts of packet loss at G0/0, thereby causing performance monitor packet loss reporting to no longer be accurate. Packet losses may show loss rates greater than 50%; however this is not the actual loss, but only a symptom of trying to measure statistics on an encrypted interface. Figure 2 provides the topology for the DMVPN tunnel configuration shown in Figure Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 9 of 102

10 Figure 2. Network Topology 3.5 Medianet Quality of Service Overview In many cases, Medianet will use QoS to classify media types within class maps and service policies. It is, therefore, important to know how QoS is used within a customer s network, and whether the media types have proper QoS associated prior to entering the router or switch using Performance Monitor. For example, if Performance Monitor policies are set up to monitor TelePresence video based on DSCP values and these media flows are not tagged properly, Performance Monitor will not monitor these flows. Medianet Quality of Service Design Guide for Enterprises Links Medianet Campus QoS Design 4.0 RTP Table 4. Common Application QoS; DSCP/TOS Hex and Decimal Values Application PHB DSCP Value (Decimal) DSCP Value (hex) TOS Value (Decimal) Voice (G.711) EF 46 0x2E 184 0xB8 Video Surveillance CS5 40 0x xA0 TelePresence CS4 32 0x x80 CUVA/CUPC Video AF x x88 TOS Value (hex) 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 10 of 102

11 Application PHB DSCP Value (Decimal) DSCP Value (hex) TOS Value (Decimal) Streaming Video (DMP) AF x1A 104 0x68 Webex AF x x48 Network-BGP CS6 48 0x xC0 Signaling (SCCP,SIP) CS3 24 0x x60 TOS Value (hex) 3.6 Medianet SNMP Configuration and MIBs A Performance Monitor enabled network node meters TCP and RTP metrics such as delay, loss, and jitter. It can forward those statistics using NetFlow records to any NetFlow Version 9 capable collector, and it also allows RTP metrics to be retrieved using SNMP. Performance Monitor collects the metrics shown in the Table 5. Table 5. Performance Monitor MIB Data Fields Data Fields for RTP Packets Expected Counter Packet Loss Packet Lost Rate Loss Event Count Interarrival Jitter (RFC3550) RTP Synchronization Source (SSRC) Application Packet Count Application Byte Rate Application Octet Count Measured Rate DSCP IP Flow destination address and port, source address and port, and protocol IP TTL IP TTL minimum Flow to Interface Mapping Data Fields for TCP Round Trip Time (RTT) Packet Loss Loss Event Count Application Packet Count Application Byte Rate Application Octet Count Measured Rate DSCP IP Flow destination address and port, source address and port, and protocol. IP TTL IP TTL minimum Flow to Interface Mapping Table 6. SNMP MIBs Introduced for Performance Monitor MIB CISCO-FLOW-MONITOR-TC-MIB CISCO-FLOW-MONITOR-MIB Description This MIB module defines textual conventions used to describe flow monitoring data. This MIB module defines objects that describe monitored flows carrying media streams, statistics relating to these flows, and a history of computed metrics for these flows. CISCO-RTP-METRICS-MIB This MIB module defines objects describing RTP metrics Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 11 of 102

12 MIB Description CISCO-IP-CBR-METRICS-MIB This MIB module defines objects describing IP CBR metrics. IOS SNMP configuration: snmp-server community public RW snmp-server community write RW snmp-server ifindex persist snmp ifmib ifindex persist snmp mib persist cbqos CISCO-RTP-METRICS-MIB CISCO-FLOW-MONITOR-MIB 4 Performance Monitor Troubleshooting 4.1 Performance Monitor Common Problems & Solutions Performance Monitor Cannot Find Media Flow on Interface To verify if a media type is matching the proper class-map and/or ACL, use the show policy-map performancemon interface command. If the packet counter and bit-rate seems appropriate for this media type, then the media flows are hitting the interface and we need to drill down further to troubleshoot this problem. Figure 3. Output Example 1: Performance Monitor Cannot Find Media Flow on Interface Verify all Performance Monitor configurations to confirm that a policy map of type Performance Monitor is applied to the appropriate interface where the media flow is present. Use Figure 1 as a configuration guide Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 12 of 102

13 Figure 4. Output Example 2: Performance Monitor Cannot Find Media Flow on Interface On the ISRs, use the IOS interface packet capture feature to verify that the media type matches the ACLs and class maps by analyzing the capture and confirming the media flow is using the correct class map and ACL. The capture will allow you to verify DSCP, IP addresses, port numbers, etc. The steps below show how to use this IOS capture feature on the ISRs. Step 1. Configure capture profile, ACL & apply to Interface ip traffic-export profile pcap mode capture bidirectional outgoing access-list 1 access-list 1 permit interface g0/0 ip traffic-export apply pcap size Step 2. Capture Media stream using CLI commands Router-1# traffic-export int g0/0 clear Router-1# traffic-export int g0/0 start [Wait for 8MB buffer to fill.] Router-1# traffic-export int g0/0 stop Step 3. Copy capture buffer to TFTP Server & verify media Router-1# traffic-export interface g0/0 copy tftp:// /router-1.pcap 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 13 of 102

14 On the Cat3K switches, you can use a monitor session to copy a packet coming to and from a source interface and send those packets to a destination interface where a host running a sniffer program can store a packet capture. Another way to verify if the media flow of interest is present on a given interface is to configure NetFlow on the same interface and use NetFlow to detect the media flow. If NetFlow can discover the media flow but Performance Monitor cannot and you have already confirmed the Performance Monitor configuration, then start debugging Performance Monitor as follows. Step 1. Configure a Netflow monitor flow monitor flow-mon-1 description Monitor-g0 record netflow ipv4 original-input Step 2. Apply Netflow monitor to interface interface gig0/0 ip flow monitor flow-mon-1 input Step 3. Verify using the show flow monitor cache command Router-1# show flow monitor flow-mon-1 cache format table Cache type: Normal Cache size: 4096 Current entries: 10 High Watermark: 22 Flows added: 368 Flows aged: Active timeout ( 1800 secs) 0 - Inactive timeout ( 15 secs) Event aged 0 - Watermark aged 0 - Emergency aged 0 IPV4 SRC ADDR IPV4 DST ADDR TRNS SRC PORT TRNS DST PORT INTF INPUT FLOW SAMPLER ID IP TOS IP PROT ip src as ip dst as ipv4 next hop addr ipv4 src mask ipv4 dst mask tcp flags intf output bytes pkts time first time last =============== =============== ============= ============= ==================== =============== ====== ======= ========= ========= ================== ============= ============= ========= ==================== ========== ========== ============ ============ 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 14 of 102

15 Gi0 0 0xA /28 /29 0x00 Vl :28: :28: Note: Source and Destination Addresses, Interface, IP protocol type = 17 (UDP) and IP TOS = 0xA0 which is equivalent to DSCP = CS5. This confirms the media flow is present on the input of Interface g0/ DMVPN Tunneling Configurations Performance Monitor Service Policy Should Only Be Applied on the Tunnel Interface (tun1) and Not the Source Interface (g0/0) Applying the Performance Monitor service policy on the source interface and not the tunnel interface will result in no Performance Monitor statistics being displayed. However, when you apply the Performance Monitor service policy on the source interface and the tunnel interface, Performance Monitor statistics could show large amounts of erroneous packet loss (IOS Release T1) or not display RTP statistics at all (IOS Release MT or higher) since this is an unsupported configuration (CSCtl95419). The applying of a Performance Monitor service policy to both the tunnel interface and the source interface is an unsupported configuration. Refer to the Medianet Design guide for supported configurations: Medianet Design Guide Performance Monitor Does Not Calculate Out-of-Order Packets Out-of-order packets will be counted as packet loss and if significant will increase the packet loss rate proportionally (CSCtj89640). If you are experiencing large amounts of packet loss from Performance Monitor statistics, and suspects something is wrong with Performance Monitor, you may need to take a packet capture on the interface and look for out-of-order or out-of-sequence packets. This action confirms that the potential rootcause of the large packet loss measured by Performance Monitor is likely due to a large number of out-ofsequence packets seen on the interface. An algorithm for calculating out-of-order packets will be available in a future release of Performance Monitor Performance Monitor Cannot Measure TCP retransmitted packets If TCP packets are dropped in the network, Performance Monitor will only count packets arriving or leaving an interface and will not measure the additional packets needed for retransmission of dropped packets. A future release of Performance Monitor will include TCP retransmission counters Performance Monitor Flow Rates Are Incorrect Performance Monitor show commands display the flow rate as measured in Byte per second (Bps). You can convert this metric by multiplying 8-times the flow rate value to obtain the more common bits per second (bps). Also, note that the DSCP value shown within the Performance Monitor statistics are in decimals. Use Table 1 to help identify the media type for the DSCP values shown by Performance Monitor Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 15 of 102

16 Figure 5. Output Example: Performance Monitor Flow Rates Are Incorrect Performance Monitor Does Not Collect or Report Statistics on Encrypted Interfaces This section provides solutions to common issues with IPSec interfaces and performance monitoring applied to encryption enabled interfaces. This issue also applies to any interface enabled for encryption with few exceptions. Figure 6. Network Diagram for Troubleshooting Scenario (Performance Monitor Does Not Collect or Report Statistics on Encrypted Interfaces) In Figure 6 there is bidirectional audio/video flows between TelePresence (TP-A) A( ) and TelePresence (TP-B) B( ) over a site-to-site IPSec configuration. Problem Diagnosis Performance Monitor is applied to the outbound interface E0/1 of Router R1 for the flows between TP-A to TP-B. First, issue show run interface e0/1 to confirm the policy configuration is attached to the crypto interface (that is, the interface with encryption enabled): R1#show running-config interface E0/1 interface Ethernet0/1 ip address Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 16 of 102

17 crypto map aesmap service-policy type performance-monitor input telepresence service-policy type performance-monitor output telepresence Next, execute show policy-map type performance to verify the policy attached to the interface: R1#show policy-map type performance interface E0/1 Ethernet0/1 Service-policy performance-monitor output: tp Class-map: telepresence (match-any) packets, bytes 30 second offered rate bps, drop rate 0 bps Match: access-group name telepresence packets, bytes 30 second rate bps media-monitoring: policy-map type performance-monitor tp flow monitor tp monitor parameters interval duration 30 timeout 10 history 10 flows 2000 monitor metric rtp min-sequential 3 max-dropout 5 max-reorder 5 clock-rate default ssrc maximum 50 react 1 rtp-jitter-average threshold value ge 5 alarm type discrete alarm severity info react 2 transport-packets-lost-rate threshold value ge 5.00 alarm type discrete alarm severity info Class-map: class-default (match-any) 18 packets, 1256 bytes 30 second offered rate 0 bps, drop rate 0 bps Match: any 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 17 of 102

18 The previous output clarifies any misconfiguration of the policy map or the class map and shows the TelePresence traffic matches against the class. R1# show class-map telepresence Class Map match-any telepresence (id 29) Match access-group name telepresence R1# show ip access-list telepresence Standard IP access list telepresence 10 permit ( matches) Next, execute the show performance monitor status policy policy-name class class-name command to display the statistics for the TelePresence call: R1# show performance monitor status policy tp class telepresence Codes: * - field is not configurable under flow record NA - field is not applicable for configured parameters Match: ipv4 source address = , ipv4 destination address = , transport source-port = 0, transport destination-port = 0, transport rtp ssrc = 0, ip protocol = 50, Policy: tp, Class: telepresence routing forwarding-status : Forward transport packets expected counter : NA transport packets lost counter : NA transport packets lost rate ( % ) : NA transport event packet-loss counter : NA transport rtp jitter mean (usec) : NA transport rtp jitter minimum (usec) : NA transport rtp jitter maximum (usec) : NA interface input : Vl220 interface output : Gi0/0 counter bytes : counter packets : counter bytes rate : application media bytes counter : 0 application media bytes rate : NA application media packets counter : 0 application media packets rate : NA application media event : Normal monitor event : false ip dscp : 0x Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 18 of 102

19 ip ttl : 255 In the above output, notice the source/destination address is and , instead of and This is because ESP (Encapsulated Security Payload) is using tunnel mode to transport encrypted packets. The whole packet, including the headers of the TelePresence stream, is encapsulated. In this situation, Performance Monitoring does not have the capability to read the RTP parameters and will not report RTP or transport statistics or application media details. Possible Fixes Solution 1 The solution to this issue is to apply performance monitoring on non-crypto interfaces on the same router. Figure 7. Network Diagram for Troubleshooting Scenario (Solution 1 for the Performance Monitor Does Not Collect or Report Statistics on Encrypted Interfaces Issue) In the previous topology, green arrows indicate the non-crypto interface. Performance Monitor collects statistics when applied to these interfaces. The following is output from the show performance monitor status policy command for a non-crypto interface. R1# show performance monitor status policy tp class telepresence Match: ipv4 source address = , ipv4 destination address = , transport source-port = 2344, transport destination-port = 49264, transport rtp ssrc = , ip protocol = 17, Policy: tp, Class: telepresence routing forwarding-status : Forward transport packets expected counter : 8450 transport packets lost counter : 0 transport packets lost rate ( % ) : 0.00 transport event packet-loss counter : 0 transport rtp jitter mean (usec) : 38 transport rtp jitter minimum (usec) : 0 transport rtp jitter maximum (usec) : 4616 interface input : E0/ Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 19 of 102

20 interface output : E0/1 counter bytes : counter packets : 8450 counter bytes rate : application media bytes counter : application media bytes rate : application media packets counter : 8450 application media packets rate : 140 application media event : Normal monitor event : false ip dscp : 0x23 ip ttl : 63 Solution 2 Another approach to solve this issue is to introduce a tunnel interface on routers R1 and R2. The following topology is used to explain this solution. Figure 8. Network Diagram for Troubleshooting Scenario (Solution 2 for the Performance Monitor Does Not Collect or Report Statistics on Encrypted Interfaces Issue) In this scenario, Performance Monitor policy is applied to the tunnel interfaces. The packets entering and leaving the GRE tunnel interfaces are not encrypted and are still readable by the Performance Monitor policy Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 20 of 102

21 The following is output from the show performance monitor status applied to the tunnel interface in this scenario. R1# show performance monitor status policy tp class telepresence Codes: * - field is not configurable under flow record NA - field is not applicable for configured parameters Match: ipv4 source address = , ipv4 destination address = , transport source-port = 2352, transport destination-port = 49270, transport rtp ssrc = , ip protocol = 17, Policy: tp, Class: telepresence routing forwarding-status : Forward transport packets expected counter : transport packets lost counter : 0 transport packets lost rate ( % ) : 0.00 transport event packet-loss counter : 0 transport rtp jitter mean (usec) : 116 transport rtp jitter minimum (usec) : 0 transport rtp jitter maximum (usec) : 7014 interface input : E0/0 interface output : Tu1 counter bytes : counter packets : counter bytes rate : application media bytes counter : application media bytes rate : application media packets counter : application media packets rate : 156 application media event : Normal monitor event : false ip dscp : 0x23 ip ttl : 63 Codes: * - field is not configurable under flow record NA - field is not applicable for configured parameters Match: ipv4 source address = , ipv4 destination address = , transport source-port = 2350, transport destination-port = 49268, transport rtp ssrc = , ip protocol = 17, Policy: tp, Class: telepresence routing forwarding-status : Forward transport packets expected counter : transport packets lost counter : 0 transport packets lost rate ( % ) : Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 21 of 102

22 transport event packet-loss counter : 0 transport rtp jitter mean (usec) : 115 transport rtp jitter minimum (usec) : 0 transport rtp jitter maximum (usec) : 1488 interface input : E0/0 interface output : Tu1 counter bytes : counter packets : counter bytes rate : application media bytes counter : application media bytes rate : 9000 application media packets counter : application media packets rate : 50 application media event : Normal monitor event : false ip dscp : 0x23 ip ttl : 63 Performance Monitor configuration and the relevant interface configuration for both routers are as follows. Router R1 crypto isakmp policy 10 encr aes 256 authentication pre-share lifetime 180 crypto isakmp key cisco123 address crypto ipsec transform-set aesset esp-aes 256 esp-sha-hmac mode transport crypto map aesmap 10 ipsec-isakmp set peer set transform-set aesset match address 147 interface Tunnel1 ip address tunnel source Ethernet0/1 tunnel destination service-policy type performance-monitor output tp end interface Ethernet0/1 mtu Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 22 of 102

23 ip address ip nbar protocol-discovery ip flow ingress ip flow egress load-interval 30 duplex auto speed auto crypto map aesmap access-list 147 permit gre any any Router R2 crypto isakmp policy 10 encr aes 256 authentication pre-share lifetime 180 crypto isakmp key cisco123 address crypto ipsec transform-set aesset esp-aes 256 esp-sha-hmac mode transport crypto map aesmap 10 ipsec-isakmp set peer set transform-set aesset match address 147 interface Tunnel1 ip address tunnel source Ethernet0/1 tunnel destination service-policy type performance-monitor output tp end interface Ethernet0/1 mtu 9216 ip address ip nbar protocol-discovery ip flow ingress ip flow egress load-interval 30 duplex auto 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 23 of 102

24 speed auto crypto map aesmap access-list 147 permit gre any any How Do You Monitor both IPV4 and IPV6 media types when an interface can only have a single servicepolicy applied at a time? If you need to monitor both IPV4 and IPV6 media types on an interface you must build a flow record with IPV4 and IPV6 header information. The configuration below shows an example of a flow record which matches on both IPV4 and IPV6 headers and collects many other new header elements to illustrate these new additions to Performance Monitor.The flow monitor inline flow record default-rtp does not monitor IPv6; it only monitors IPV4 traffic. You must built a new flow record with the appropriate IPV6 fields to monitor these media types. IPV4 and IPV6 Flow Record Config Example: flow record type performance-monitor RTP-IPV4-IPV6-ALL match ipv4 protocol match ipv4 source address match ipv4 destination address Match on IPV4 & IPV6 headers match ipv6 protocol match ipv6 source address match ipv6 destination address match transport source-port match transport destination-port match transport rtp ssrc collect interface input collect interface output collect flow direction collect flow sampler collect counter bytes collect counter bytes rate collect counter bytes rate per-flow collect counter bytes replicated collect counter packets collect counter packets rate collect counter packets rate per-flow collect counter packets replicated collect timestamp interval collect timestamp absolute first collect timestamp absolute last collect transport rtp flow count collect transport rtp payload-type collect transport rtp jitter mean collect transport rtp jitter minimum collect transport rtp jitter maximum Collect all transport RTP packet statistics collect transport packets lost rate collect transport packets lost counter collect transport packets expected counter collect transport packets out-of-order collect transport event packet-loss counter collect transport event packet-loss counter max collect transport event packet-loss counter min collect routing next-hop address ipv4 collect ipv4 length total 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 24 of 102

25 collect ipv4 length total minimum collect ipv4 length total maximum collect ipv4 tos collect ipv4 precedence collect ipv4 dscp collect ipv4 ttl collect ipv6 dscp collect ipv6 source mask collect ipv6 source prefix collect ipv6 destination mask collect ipv6 destination prefix collect ipv6 extension map collect ipv6 hop-limit collect ipv6 hop-limit max collect ipv6 hop-limit min collect ipv6 precedence collect ipv6 length header collect ipv6 length payload collect ipv6 length total collect ipv6 length total minimum collect ipv6 length total maximum collect ipv6 next-header collect ipv6 payload-length collect ipv6 version collect ipv6 traffic-class collect ipv6 flow-label collect ipv6 fragmentation flags collect ipv6 fragmentation id collect ipv6 fragmentation offset collect application media bytes counter collect application media bytes rate collect application media packets counter collect application media packets rate collect application media event collect application name collect monitor event collect routing forwarding-status reason collect routing next-hop address ipv6 collect routing is-multicast collect routing multicast replication-factor collect routing vrf input collect routing source as collect routing source as peer collect routing source traffic-index collect routing destination as collect routing destination as peer collect routing destination traffic-index collect datalink dot1q vlan input collect datalink dot1q vlan output collect datalink mac source address input collect datalink mac source address output collect datalink mac destination address input collect datalink mac destination address output flow monitor type performance-monitor RTP-IPV4-IPV6-ALL description Show all IPV4 and IPV6 RTP stats record RTP-IPV4-IPV6-ALL class-map match-all TelePresence match dscp cs4 class-map match-all class-all match any Collecting on IPV4 & IPV6 headers INFO will only be populated with valid data when the IP version matches. Build the appropriate class maps and ACLs to match traffic types. Build a new Flow Monitor and include the new Flow 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 25 of 102

26 class-map match-all Voice-EF match dscp ef policy-map type performance-monitor ALL-IPV4-IPV6-MEDIA class TelePresence flow monitor inline record RTP-IPV4-IPV6-ALL monitor parameters timeout 2 history 30 flows 16 monitor metric rtp ssrc maximum 50 react 5 transport-packets-lost-rate threshold value gt 3.00 alarm severity critical action syslog action snmp class class-all flow monitor inline record RTP-IPV4-IPV6-ALL monitor parameters timeout 2 history 30 flows 32 monitor metric rtp ssrc maximum 50 interface g0/0 service-policy type performance-monitor input ALL-IPV4-IPV6-MEDIA service-policy type performance-monitor output ALL-IPV4-IPV6-MEDIA Add a new policy-map which uses the new Flow Record. This class-map is a catch-all class which is only intended for lite traffic load testing. Limit the number flows this class can monitor by using flow 32 in parameters. Apply service policy to interface. Performance Monitor output for the soft-client Cisco Jabber Video session from a to using IPV4 Notice that due to the untrusted source of PC1 this media session was not tagged with the TelePresence DSCP CS4 and was found in the class-all class map. If this network had flows over 32 this session may not have been recorded by Performance Monitor. When troubleshooting why some media flows are not seen by Performance Monitor you can narrow the number of flows seen by a specific class map by creating a new class and inserting an ACL reference within the class map to monitor traffic flows from a specific host. The follow example shows the configuration on an IP Video Surveillance Camera for how to narrow the number of flows seen by a specific class map by creating a new class and inserting an ACL reference within the class map to monitor traffic flows from a specific host. class-map match-all IPVS-CS5 match ip dscp cs5 match access-group name IPVS-UDP-5000 ip access-list extended IPVS-UDP-5000 permit udp host eq 5000 host Save time by using the show performance monitor command with keyword filters to view specific media flows with known IP addresses and subnets. Adding filters will save time paging through media flows which are not of interest Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 26 of 102

27 The following example shows the audio and video media flow for a Cisco Jabber session. Note that since this is an IPv4 flow the IPv6 fields show zeros for Not Applicable. The IP version is displayed in the Performance Monitor statistics # show perfermance monitor status udp any Match: ipv4 source address = , ipv4 destination address = , ipv6 source address = ::, ipv6 destination address = ::, transport source-port = 21294, transport destination-port = 21374, transport rtp ssrc = , ip protocol = 17, Policy: ALL-IPV4-IPV6-MEDIA, Class: class-all datalink dot1q vlan input : 0 datalink dot1q vlan output : 0 datalink mac source address input : 588D.09C5.FD41 datalink mac source address output : C471.FEF datalink mac destination address input : C471.FEF3.15B8 datalink mac destination address output : 001A.30FC.4C00 routing source as : 0 routing destination as : 0 routing source as peer : 0 routing destination as peer : 0 routing source traffic-index : 0 routing destination traffic-index : 0 routing forwarding-status : Forward routing is-multicast : No routing multicast replication-factor : 0 routing vrf input : 0 (DEFAULT) routing next-hop address ipv4 : routing next-hop address ipv6 : A57:5D2E:: ipv6 flow-label : 0 ipv6 next-header : 0 ipv6 payload-length : 0 ipv6 extension map : 0x ipv6 source prefix : :: ipv6 source mask : /0 ipv6 destination prefix : :: ipv6 destination mask : /0 ipv6 fragmentation id : 0 transport packets expected counter : transport packets lost counter : 11 transport packets lost rate ( % ) : 0.01 transport event packet-loss counter : 5 transport rtp jitter mean (usec) : 443 transport rtp jitter minimum (usec) : 0 transport rtp jitter maximum (usec) : interface input : Gi1/0 interface output : Gi0/0 flow direction : Output flow sampler : 0 counter bytes : counter packets : counter bytes replicated : 0 counter packets replicated : 0 counter bytes rate : counter packets rate : 164 timestamp absolute first : 16:44: timestamp absolute last : 16:45: application media bytes counter : application media bytes rate : application media packets counter : application media packets rate : 164 application media event : Normal 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 27 of 102

28 monitor event : false counter bytes rate per-flow : 9033 counter packets rate per-flow : 7 transport event packet-loss counter min : 0 transport event packet-loss counter max : 4 transport rtp payload-type : 123 transport packets out-of-order : 0 ip tos : 0x00 ip dscp : 0x00 ip precedence : 0 ip ttl : 126 ip length total : 1231 ip length total minimum : 60 ip length total maximum : 1250 application id : cisco rtp ip version : 4 ip ttl minimum : 126 ip ttl maximum : 126 ip length header : 20 ip length payload : 1211 ip fragmentation offset : 0 ip fragmentation flags : 0x02 To determine whether this flow is Audio or Video look at; Payload-Type (PT=123), IP Length Payload = 1211, and Counter Packets Rate = 164 to determine this flow is Video. Codes: * - field is not configurable under flow record NA - field is not applicable for configured parameters Match: ipv4 source address = , ipv4 destination address = , ipv6 source address = ::, ipv6 destination address = ::, transport source-port = 21292, transport destination-port = 21372, transport rtp ssrc = , ip protocol = 17, Policy: ALL-IPV4-IPV6-MEDIA, Class: class-all datalink dot1q vlan input : 0 datalink dot1q vlan output : 0 datalink mac source address input : 588D.09C5.FD41 datalink mac source address output : C471.FEF datalink mac destination address input : C471.FEF3.15B8 datalink mac destination address output : 001A.30FC.4C00 routing source as : 0 routing destination as : 0 routing source as peer : 0 routing destination as peer : 0 routing source traffic-index : 0 routing destination traffic-index : 0 routing forwarding-status : Forward routing is-multicast : No routing multicast replication-factor : 0 routing vrf input : 0 (DEFAULT) routing next-hop address ipv4 : routing next-hop address ipv6 : A57:5D2E:: ipv6 flow-label : 0 ipv6 next-header : 0 ipv6 payload-length : 0 ipv6 extension map : 0x ipv6 source prefix : :: ipv6 source mask : /0 ipv6 destination prefix : :: ipv6 destination mask : /0 ipv6 fragmentation id : 0 transport packets expected counter : transport packets lost counter : 8 transport packets lost rate ( % ) : 0.01 transport event packet-loss counter : Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 28 of 102

29 transport rtp jitter mean (usec) : 513 transport rtp jitter minimum (usec) : 0 transport rtp jitter maximum (usec) : interface input : Gi1/0 interface output : Gi0/0 flow direction : Output flow sampler : 0 counter bytes : counter packets : counter bytes replicated : 0 counter packets replicated : 0 counter bytes rate : counter packets rate : 100 timestamp absolute first : 16:44: timestamp absolute last : 16:45: application media bytes counter : application media bytes rate : application media packets counter : application media packets rate : 100 application media event : Normal monitor event counter bytes rate per-flow : 606 counter packets rate per-flow : 4 transport event packet-loss counter min : 0 transport event packet-loss counter max : 2 transport rtp payload-type : 127 transport packets out-of-order : 0 : false ip tos : 0x00 ip dscp : 0x00 ip precedence : 0 ip ttl : 126 ip length total : 130 ip length total minimum : 130 ip length total maximum : 144 application id ip version : 4 ip ttl minimum : 126 ip ttl maximum : 126 ip length header : 20 ip length payload : 110 : cisco rtp ip fragmentation offset : 0 ip fragmentation flags : 0x02 To determine whether this flow is Audio or Video look at; Payload- Type (PT=127), IP Length Payload = 110, and Counter Packets Rate = 100 to determine this flow is Audio. The following example shows the output of our IPv4 & IPv6 flow record, however this time the Performance Monitor show command displays an IPv6 media flow. Notice now that the IPV4 field elements are zero and IPv6 elements are now populated with the IPv6 packet statistics. The media flow in this example is from a dual-stack (IPv4/IPv6) PC running VLC application from PC#1 to PC#2 with the VLC app sending MPEG-II RTP video stream at Payload Type = # show perf mon stat policy-map ALL-IPV4-IPV6-MEDIA class-map TelePresence Codes: * - field is not configurable under flow record NA - field is not applicable for configured parameters Match: ipv4 source address = , ipv4 destination address = , ipv6 source address = 2001:420:2CFF:3F:6::2, ipv6 destination address = 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 29 of 102

30 2001:420:2CFF:3F:4::11, transport source-port = 65350, transport destination-port = 5004, transport rtp ssrc = , ip protocol = 17, Policy: ALL-IPV4-IPV6-MEDIA, Class: TelePresence datalink dot1q vlan input : 0 datalink dot1q vlan output : 0 datalink mac source address input : 588D.09C5.FD41 datalink mac source address output : C471.FEF datalink mac destination address input : C471.FEF3.15B8 datalink mac destination address output : 001A.30FC.4C00 routing source as : 0 routing destination as : 0 routing source as peer : 0 routing destination as peer : 0 routing source traffic-index : 0 routing destination traffic-index : 0 routing forwarding-status : Forward routing is-multicast : No routing multicast replication-factor : 0 routing vrf input : 0 (DEFAULT) routing next-hop address ipv4 : routing next-hop address ipv6 : FE80::21A:30FF:FEFC:4C00 ipv6 flow-label : 0 ipv6 next-header : 17 ipv6 payload-length : 1336 ipv6 extension map : 0x ipv6 source prefix : 2001:420:2CFF:3F:6:: ipv6 source mask : /96 ipv6 destination prefix : 2001:420:2CFF:3F:4:: ipv6 destination mask : /96 ipv6 fragmentation id : 0 transport packets expected counter : transport packets lost counter : 48 transport packets lost rate ( % ) : 0.00 transport event packet-loss counter : 20 transport rtp jitter mean (usec) : 93 transport rtp jitter minimum (usec) : 0 transport rtp jitter maximum (usec) : interface input : Gi1/0 interface output : Gi0/0 flow direction : Output flow sampler : 0 counter bytes : counter packets : counter bytes replicated : 0 counter packets replicated : 0 counter bytes rate : counter packets rate : 687 timestamp absolute first : 16:53: timestamp absolute last : 16:53: application media bytes counter : application media bytes rate : application media packets counter : application media packets rate : 687 application media event : Normal monitor event : false counter bytes rate per-flow : counter packets rate per-flow : 22 transport event packet-loss counter min : 0 transport event packet-loss counter max : 4 transport rtp payload-type : 33 transport packets out-of-order : 0 Collecting IPV6 headers INFO will only be populated with valid data when the IP version matches Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 30 of 102

31 ip tos : 0x80 ip dscp : 0x20 ip precedence : 4 ip ttl : 126 ip length total : 1376 ip length total minimum : 1376 ip length total maximum : 1376 application id : cisco rtp ip version : 6 ip ttl minimum : 126 ip ttl maximum : 126 ip length header : 40 ip length payload : 1336 ip fragmentation offset : 0 ip fragmentation flags : 0x New Performance Monitor default flow record default-rtp has suppressed some IPV4 elements from the previous version. IOS 15.2(2)T includes Medianet Media Monitoring 2.3 features and has altered to default flow record. The new default-rtp flow record will contain the following IPV4 header fields for show performance monitor by default. To include other field elements not shown within the default flow record you must create a new flow record which include all the field elements of interest. New Default: Cisco Jabber Video media flow example from [IOS: 15.2(2)T]: # show perf mon status udp any Match: ipv4 source address = , ipv4 destination address = , transport source-port = 21166, transport destination-port = 21142, transport rtp ssrc = , ip protocol = 17, Policy: vid-mon, Class: class-all routing forwarding-status : Forward transport packets expected counter : transport packets lost counter : 2 transport packets lost rate ( % ) : 0.00 transport event packet-loss counter : 2 transport rtp jitter mean (usec) : 634 transport rtp jitter minimum (usec) : 2 transport rtp jitter maximum (usec) : interface input : Gi1/0 interface output : Gi0/0 counter bytes : counter packets : counter bytes rate : application media bytes counter : application media bytes rate : application media packets counter : application media packets rate : 143 application media event : Normal monitor event : false ip dscp : 0x00 ip ttl : Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 31 of 102

32 Old Default: Cisco TelePresence Video media flow example from [IOS: 15.1(3)M1]: # show perf mon sta int t1 Codes: * - field is not configurable under flow record NA - field is not applicable for configured parameters Match: ipv4 src addr = , ipv4 dst addr = , ipv4 prot = udp, trns src port = 21094, trns dst port = 16420, SSRC = Policy: All-Media, Class: TP-video-AF41, Interface: Tunnel1, Direction: output *counter flow : 7 counter bytes : counter bytes rate (Bps) : *counter bytes rate per flow (Bps) : *counter bytes rate per flow min (Bps) : *counter bytes rate per flow max (Bps) : counter packets : *counter packets rate per flow : 136 counter packets dropped : 0 routing forwarding-status reason : Unknown interface input : Gi0/1 interface output : Tu1 monitor event : false ipv4 dscp : 34 ipv4 ttl : 63 application media bytes counter : application media packets counter : application media bytes rate (Bps) : *application media bytes rate per flow (Bps) : *application media bytes rate per flow min (Bps) : *application media bytes rate per flow max (Bps) : application media packets rate (pps) : 136 application media event : Normal *transport rtp flow count : 7 transport rtp jitter mean (usec) : 276 transport rtp jitter minimum (usec) : 0 transport rtp jitter maximum (usec) : 4871 *transport rtp payload type : 112 transport event packet-loss counter : 0 *transport event packet-loss counter min : 0 *transport event packet-loss counter max : 0 transport packets expected counter : transport packets lost counter : 0 *transport packets lost counter minimum : 0 *transport packets lost counter maximum : 0 transport packets lost rate ( % ) : 0.00 *transport packets lost rate min ( % ) : 0.00 *transport packets lost rate max ( % ) : Cat3K and Cat4K Performance Monitor service-policy can only be applied to the INPUT of an interface The Cat3K and Cat4K platforms do not support having a Performance Monitor service policy on the output of any interface. An error message appears if someone attempts to apply a Performance Monitor service policy to the output of an interface as shown below. Cat3k(config-if)# service-policy type performance-monitor output All-Media %ERROR: output direction policy not supported 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 32 of 102

33 Cat3k(config-if)# 5. Troubleshooting Performance Monitoring using IOS debugging This section lists of a few examples of how to use IOS debugging information to resolve Medianet Performance Monitor problems. 5.1 Debug Commands for Performance Monitor Important Note: The debugs listed below do not have conditional filters and could over-run the console on a busy router or even effect system performance. On a production router or switch, you may want to collect debugs during non-peak hours to avoid any issues which may be the result of collecting a large amount of debugs. Table 7. Debug Commands Available for Performance Monitor Router-1#debug performance monitor? database dynamic event export flow-monitor metering provision sibling snmp tca timer Media Monitor flow database debugging Dynamic monitoring debugging Performance event debugging (react threshold events) Export debugging (Perf-Mon exporter activity) Flow monitor debugging Metering layer debugging Provisioning debugging Sibling mgt. debugging (for dynamic multiple classes) Snmp debugging TCA debugging (threshold crossing alarm) Timers debugging show performance monitor internal hidden IOS command The output of this show command displays Performance Monitor internal statistics. Process and database statistics are displayed first ( in white), and then each media flow discovered by Performance Monitor will be displayed with each interface and direction associated with the media flow. The example below shows a media flow of Class = IPVS-CS5, which arrives at the input of interface Tunnel1, flows out of Tun1 into Gig0/0 then out of Gig0/1. This command is useful to for verifying Performance Monitor database stats, as well as verifying that a class map is receiving packets Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 33 of 102

34 Figure 9. Output for show performance monitor internal show performance monitor cache policy-map all-media class-map IPVS-CS5 The output of this IOS command shows the metering engine cache for Flexible NetFlow (FNF) and Performance Monitor statistics. Performance Monitor uses the FNF cache for buffering the flow statistics. This command is useful for monitoring the health of the NetFlow buffers and showing the number of flows being monitored and when the flow exceeds the flow cache size. The flow spec s are displayed for the IPVS-CS5 class-map (Flow Timeout, Max Flows, RTP Max Dropout/Re-order, High Watermark). If the High Watermark exceeds the Cache size, then some flows for this class-map will not be monitored accurately. If this happens, you can increase the Max number of flows within the class-map to cover more flows, if necessary Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 34 of 102

35 Figure 10. Output for show performance monitor cache policy-map all-media class-map IPVS-CS #debug performance monitor database This debug command reports on the media monitoring database activity. The MM-database is responsible for aggregating packet statistics for each media flow. You can see from the example below, incoming and outgoing messages pertaining to each interface. This debug is useful to determine if a specific media flow s statistics are making to the MM-database. The fact that a specific media flow is identified within these debugs means the flow is hitting the database. You will need to know the IP addresses of the media flow to track it through the debugs. Figure 11. Output for debug performance monitor database 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 35 of 102

36 5.1.4 #debug performance monitor dynamic This debug command is useful for debugging internally generated Performance Monitor instances, such as when Mediatrace is executed. Mediatrace creates dynamic Performance Monitor instances to measure statistics for a given media flow. You can also use #show performance monitor clients detail all to debug dynamic Performance Monitor instances as well. More on debugging Mediatrace in section #debug performance monitor export This debug command is useful for troubleshooting problems with NetFlow collection. These debugs tell you which media flows are exporting NetFlow records to the configured NetFlow collector. The class-map and policy-map names are provided to allow you the see which media flows have been identified and are sending NetFlow records. Figure 12. Output for debug performance monitor export #debug performance monitor metering These debug commands are useful to detect when issues with the metering engine are suspect. The metering engine polls the cache for packet statistics every 30 seconds. If this debug does not see the metering engine polling the cache for packet statistics, this is a good indicator that something is wrong within Performance Monitor. Verify NetFlow and Flexible NetFlow is functioning as well. Performance Monitor uses the FNF cache to store packet stats until the data is pushed into the video monitoring database Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 36 of 102

37 Figure 13. Output for debug performance monitor metering #debug performance monitor SNMP This command is useful for debugging SNMP MIBs for Performance Monitor events such as threshold crossing alarms. Figure 14. Output for debug performance monitor snmp 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 37 of 102

38 6 Mediatrace 6.1 Understanding Mediatrace In a Cisco Medianet enabled network, routers and switches that are configured for Mediatrace use RSVP-TP messages as a transport mechanism. Mediatrace collects network performance statistics about single or multiple media streams on all network nodes along a routed data path. In addition, Mediatrace can also collect system utilization statistics. Note: It is important to differentiate that Mediatrace uses RSVP-TP and that it is not related to bandwidth reservation for network admission control. Even though the Cisco Mediatrace feature uses RSVP-TP to transport the collected media monitoring statistics, RSVP does not need to be exclusively configured on every network device s interface along the message s data path. When each node is configured as a Mediatrace responder or Mediatrace initiator, RSVP-TP is enabled by default. 6.2 Configuring Mediatrace Network Topology Figure 15 shows the reference network topology and the direction of the video flow. Figure 15. Mediatrace Network Topology Mediatrace Configuration The Mediatrace configuration syntax shown below differentiates the Mediatrace roles between R1 and R2. These commands configure the R1 as the initiator and R2 as the responder. R1# config t R1(config)# mediatrace initiator source ip R2# config t R2(config)# mediatrace responder 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 38 of 102

39 There are two configuration methods to invoke a Mediatrace session. Proper planning is involved when building a flexible mediatrace profile to intermittently monitor a specific media flow. Administrators are expected to provision the profile with specific parameters of the media flow Mediatrace Session The path specifer cam-2-pc-pathspec queries the network path for the video flow between the camera and PC. R2(config)# mediatrace path-specifier cam-2-pc-pathspec destination ip R2(config-mt-path)# source ip The custom flow specifier name cam-2-pc-flowspec instructs Mediatrace to monitor traffic between the camera and PC based on the 5- tuple information, minus the DSCP value. Cisco recommends using the actual source and destination IP addresses of the endpoints originating the media flow. By inserting the actual IP address of the endpoints with the path-specifier command, this allows Mediatrace to be congruent with the actual media flow path. R2(config)# mediatrace flow-specifier cam-2-pc-flowspec R2(config-mt-flowspec)# source-ip source-port 5000 R2(config-mt-flowspec)# dest-ip dest-port 1902 The Mediatrace profile cam-2-pc-profile defines the metrics that will be collected for the given video flow. There are two types of profiles: system and performance monitoring profiles. R2 is configured to use Mediatrace performance profiles to interact with the Performance Monitor features on each Medianet enabled network device. R2(config)# mediatrace profile perf-monitor cam-2-pc-profile R2(config-mt-prof-perf)# metric-list rtp R2(config-mt-prof-perf-rtp-params)# clock-rate R2(config-mt-prof-perf-rtp-params)# exit R2(config-mt-prof-perf)# admin-params R2(config-mt-prof-perf-params)# sampling-interval Mediatrace session-param The Media trace session parameter cam-2-pc-sessparm defines the Media trace session timeout and the historical value to retain the number of historical session records. R2(config)# mediatrace session-params cam-2-pc-sessparm R2(config-mt-sesparam)# response-timeout 3 R2(config-mt-sesparam)# history data-sets-kept Mediatrace session The Mediatrace session 1 logically groups the Mediatrace path-specifier, session-parameter, and Mediatrace profile. R2(config)# mediatrace 1 R2(config-mt-session)# description cam-2-pc R2(config-mt-session)# path-specifier cam-2-pc-pathspec R2(config-mt-session)# session-param cam-2-pc-sessparm R2(config-mt-session)# profile perf-mon cam-2-pc-profile flow-specifier cam-2-pcflowspec 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 39 of 102

40 Mediatrace Schedule The mediatrace schedule command specifies the scheduling parameters in which Mediatrace is invoked on the initiator. The following example shows how to immediately invoke Mediatrace profile 1: R2(config)# mediatrace schedule 1 start-time now On Demand Mediatrace The mediatrace poll command can be modified to discover next hop network devices and other specified protocol metrics that will be reported on the initiator console. R2# mediatrace poll path-specifier source destination perf-mon Started the data fetch operation. Waiting for data from hops. This may take several seconds to complete... Data received for hop 0 Data received for hop 1 Data received for hop 2 Data fetch complete. Results: Data Collection Summary: Request Timestamp: 16:30: EST Tue Nov Request Status: Completed Number of hops responded (includes success/error/no-record): 3 Number of hops with valid data report: 3 Number of hops with error report: 0 Number of hops with no data record: 0 Detailed Report of collected data: Number of Mediatrace hops in the path: 3 Mediatrace Hop Number: 0 (host=3845-2, ttl=255) Metrics Collection Status: Success Reachability Address: Ingress Interface: None Egress Interface: Tu1 Metrics Collected: Flow Sampling Start Timestamp: 16:30:26 Loss of measurement confidence: FALSE Media Stop Event Occurred: FALSE IP Packet Drop Count (pkts): 0 IP Byte Count (Bytes): 200 IP Packet Count (pkts): 5 IP Byte Rate (Bps): 6 Packet Drop Reason: 0 IP DSCP: 0 IP TTL: Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 40 of 102

41 IP Protocol: 6 Media Byte Rate Average (Bps): 0 Media Byte Count (Bytes): 0 Media Packet Count (pkts): 5 RTP Interarrival Jitter Average: NA RTP Packets Lost: NA RTP Packets Expected (pkts): 0 RTP Packet Lost Event Count: 0 RTP Loss Percent: NA Mediatrace Hop Number: 1 (host=3925-1, ttl=254) Metrics Collection Status: Success Reachability Address: Ingress Interface: Tu0 Egress Interface: Tu0 Metrics Collected: Flow Sampling Start Timestamp: 16:30:26 Loss of measurement confidence: FALSE Media Stop Event Occurred: FALSE IP Packet Drop Count (pkts): 0 IP Byte Count (Bytes): 200 IP Packet Count (pkts): 5 IP Byte Rate (Bps): 6 Packet Drop Reason: 0 IP DSCP: 0 IP TTL: 62 IP Protocol: 6 Media Byte Rate Average (Bps): 0 Media Byte Count (Bytes): 0 Media Packet Count (pkts): 5 RTP Interarrival Jitter Average: NA RTP Packets Lost: NA RTP Packets Expected (pkts): 0 RTP Packet Lost Event Count: 0 RTP Loss Percent: NA Mediatrace Hop Number: 2 (host=3845-1, ttl=253) Metrics Collection Status: Success Reachability Address: Ingress Interface: Tu1 Egress Interface: Gi0/1 Metrics Collected: Flow Sampling Start Timestamp: 16:30:26 Loss of measurement confidence: FALSE Media Stop Event Occurred: FALSE 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 41 of 102

42 IP Packet Drop Count (pkts): 0 IP Byte Count (Bytes): 200 IP Packet Count (pkts): 5 IP Byte Rate (Bps): 6 Packet Drop Reason: 0 IP DSCP: 0 IP TTL: 61 IP Protocol: 6 Media Byte Rate Average (Bps): 0 Media Byte Count (Bytes): 0 Media Packet Count (pkts): 5 RTP Interarrival Jitter Average: NA RTP Packets Lost: NA RTP Packets Expected (pkts): 0 RTP Packet Lost Event Count: 0 RTP Loss Percent: NA Troubleshooting why Mediatrace is not collecting statistics This section describes common procedures when troubleshooting why a Mediatrace does not report any statistics. Figure 16. Network Diagram for Mediatrace Troubleshooting Mediatrace Statistcs Here, 3750e-A is acting as the Mediatrace initiator and 3750e-B & C are acting as responders. Diagnose the problem The output of the show mediatrace session statistics command or the show mediatrace session data command on 3750e-A shows no metrics collected for the session. 3750e-A# show mediatrace session data 21 Session Index: 21 Global Session Id: Session Operation State: Active Bucket index: 8397 Data Collection Summary: Request Timestamp: 19:32: UTC Thu Nov Request Status: Completed Number of hops responded (includes success/error/no-record): 2 Number of hops with valid data report: 0 Number of hops with error report: 0 Number of hops with no data record: 2 Detailed Report of collected data: 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 42 of 102

43 Last Route Change Timestamp: Route Index: 0 Number of Mediatrace hops in the path: 2 Mediatrace Hop Number: 1 (host=3750e-b, ttl=254) Metrics Collection Status: Fail (19, No statistic data available for reporting) Ingress Interface: Gi1/0/30 Egress Interface: Gi1/0/33 Metrics Collected:==============================NO METRICS COLLECTED Mediatrace Hop Number: 2 (host=3750e-c, ttl=253) Metrics Collection Status: Fail (19, No statistic data available for reporting) Ingress Interface: Gi1/0/8 Egress Interface: NOT COLLECTED Metrics Collected:===========================NO METRICS COLLECTED Possible Fixes: Check that the Mediatrace responders in the path are up and reachable. According to the topology, the Mediatrace responders are 3750e-B and 3750e-C. Make sure you can ping from the initiator to all responders successfully. 3750e-A# ping Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds: Success rate is 100 percent (5/5), round-trip min/avg/max = 1/9/26 ms 3750e-A# ping Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds: Success rate is 100 percent (5/5), round-trip min/avg/max = 17/35/67 ms Check for the traffic flow between the source to the destination. Here video flow is from /38500 to / One could use ACL with the log option under the interface to check if traffic is passing though different devices on the path. Check that the outgoing interface counter is increasing. 3750e-A# sh int gi 1/0/11 GigabitEthernet1/0/11 is up, line protocol is up (connected) Hardware is Gigabit Ethernet, address is aca0.164e.8242 (bia aca0.164e.8242) Internet address is / Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 43 of 102

44 MTU 9000 bytes, BW Kbit/sec, DLY 10 usec, reliability 255/255, txload 2/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX input flow-control is off, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 00:16:01 Input queue: 0/75/1/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate bits/sec, 446 packets/sec 5 minute output rate bits/sec, 1084 packets/sec packets input, bytes, 0 no buffer Received 698 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 698 multicast, 0 pause input 0 input packets with dribble condition detected packets output, bytes, 0 underruns=====traffic send out through this interface 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out 0 output buffer failures, 0 output buffers swapped out 3750e-A#sh int gi 1/0/11 GigabitEthernet1/0/11 is up, line protocol is up (connected) Hardware is Gigabit Ethernet, address is aca0.164e.8242 (bia aca0.164e.8242) Internet address is /24 MTU 9000 bytes, BW Kbit/sec, DLY 10 usec, reliability 255/255, txload 2/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX input flow-control is off, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 00:20:37 Input queue: 0/75/1/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 44 of 102

45 5 minute input rate bits/sec, 429 packets/sec 5 minute output rate bits/sec, 1105 packets/sec packets input, bytes, 0 no buffer Received 900 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 900 multicast, 0 pause input 0 input packets with dribble condition detected packets output, bytes, 0 underruns======traffic out through this interface (counter increases) 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out Check that the Mediatrace responder is configured on all routers on the path Execute traceroute to the destination, and verify responder is configured on each hop. 3750e-B#sh run b mediatrace mediatrace responder 3750e-C#sh run b mediatrace mediatrace responder Check whether the Mediatrace initiator source ip address is configured correctly. Make sure a ping from the configured source ip address to the destination is successful. 3750e-A#sh run b mediatrace mediatrace responder max-sessions 300 mediatrace initiator source-ip max-sessions 300=====Here the sourceip is configured correctly mediatrace profile perf-monitor 500 metric-list tcp mediatrace profile perf-monitor tcp metric-list tcp Check whether the flow-specifier and path-specifier are configured with the same port numbers, which in turn should match with the video flow port numbers. Here, the video flow is from /38500 to / e-A#sh run b mediatrace mediatrace path-specifier Path21 destination ip port source ip port mediatrace flow-specifier testflow21 source-ip source-port dest-ip dest-port 39500==========Here the flow/path specifier ports are configured the same 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 45 of 102

46 mediatrace 21 path-specifier Path21 session-params Traffic-source1 profile perf-monitor tp1 flow-specifier testflow21 mediatrace schedule 21 life forever start-time now Check whether the collecting interface is a port-channel. If so, metrics will not be collected on a port-channel interface. In this example, the collecting interface is not a port-channel, but a physical interface; therefore, the statistics should be collected on the interface. The relevant router configurations for this example are as follows. 3750e-A Mediatrace mediatrace responder max-sessions 300 mediatrace initiator source-ip max-sessions 300 mediatrace profile perf-monitor tp1 metric-list rtp max-reorder 3 admin-params sampling-interval 10 mediatrace path-specifier Path21 destination ip port source ip port mediatrace flow-specifier testflow21 source-ip source-port dest-ip dest-port mediatrace 21 path-specifier Path21 session-params Traffic-source1 profile perf-monitor tp1 flow-specifier testflow21 mediatrace schedule 21 life forever start-time now 3750e-B Mediatrace responder mediatrace responder 3750e-C Mediatrace responder mediatrace responder 6.3 Debugging Mediatrace Output There may be instances where the Mediatrace data collection may not yield any results. Since we know that Mediatrace uses RSVP messages to collect flow statistics, we can enable specific debug commands on the router to monitor Mediatrace message exchanges to identify the layer in which the problem lies Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 46 of 102

47 The debug messages captured in the following sections are captured from successful Mediatrace attempts. The debug messages interpretation are provided to help identify what is actually occurring on the router, as it performs a Mediatrace inquiry for Performance Monitor stats debug mediatrace error initiator/responder This message indicates that an active Mediatrace session has been removed. The message may indicate that a Mediatrace schedule has expired or a Mediatrace session has timed out. %MEDIATRACE-6-R_SESS_TRANSIENT_REMOVE_SUCCESS: A transient session of (Session Admin Action Request) with global session id (0) successfully removed. This message shows the 5-tuple information for this Mediatrace session. The 5-tuple information displayed is the result of the path-specifier source and destination IP address configured on the Mediatrace initiator. %MEDIATRACE-5-R_SESS_DOWN: Responder received a SESSION_DOWN event for src_ip( ) src_port(0) dest_ip( ) dest_port(0) protocol(0) client_id(1) initiator_id( ) instance_id(8) This message indicates the DVMC status between Mediatrace and Performance Monitor. The DVMC is the API layer that exists between Mediatrace and Performance Monitor that allows Mediatrace to poll the router for performance monitoring statistics. %MEDIATRACE-6-R_SESS_REMOVE_ALL_DVMC_SESSIONS: No more active sessions at the moment. DVMC API was called to purge all Mediatrace sessions #debug mediatrace session-id The debug mediatrace session-id command output provides trace output that illustrates the status of a scheduled Mediatrace session. The trace messages show the Mediatrace function name that is seen when scheduler starts the specific session number. This output would be the same when an adhoc Mediatrace is triggered. IPSLA-INFRA_TRACE:OPER:66 slaschedulerstart: enter IPSLA-INFRA_TRACE:OPER:66 SLA_SCHED_START_TYPE_NOW IPSLA-INFRA_TRACE:OPER:66 Starting an operation MT-SESSION_TRACE:OPER:66 Enter sla_mt_data_collector_start_ios MT-SESSION_TRACE:OPER:66 Enter sla_mt_data_collector_activate_ios The flow spec source and destination IP address is used for collecting data from DVMC. DVMC is the layer between Performance Monitor and Mediatrace that allows interaction for statistics collection. MT-SESSION_TRACE:OPER:66 dvmc flow spec: MT-SESSION_TRACE:OPER:66 Added dvmc fspec to request, length = 20 The Flexible NetFlow template configured on the router specifies the specific metric that is to be collected for reporting. The information for each metric populated per Mediatrace request. The request includes DVMC information and configuration parameters for Performance Monitor. The message length varies as the configured template can vary as the specific fields can differ. MT-SESSION_TRACE:OPER:66 Using existing FNF template Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 47 of 102

48 MT-SESSION_TRACE:OPER:66 Building FNF template success. fnf_msg_len = 120 The trace messages show the start and end of the Mediatrace message body. The size of the payload is shown at the end of the trace and this reflects the size of the Flexible NetFlow template as configured on the IOS router. The information generated per request on the router populates the specified FNF template. The collected data used for reporting includes DVMC information and performance monitoring configuration parameters. The data collected are shown as hexadecimal values in the following output. MT-SESSION_TRACE:OPER:66 MT msg body start: len=389 MT-SESSION_TRACE:OPER:66 0x1 0x0 0x1 0x79 0x8 0x5D 0x4A 0x4A MT-SESSION_TRACE:OPER:66 0x0 0x0 0x0 0x0 0x4 0xA 0x57 0x5D MT-SESSION_TRACE:OPER:66 0x99 0x1 0x68 0x0 0x0 0x0 0x4 0xA MT-SESSION_TRACE:OPER:66 0x57 0x5D 0x9A 0xA 0x57 0x5D 0xA4 0x13 MT-SESSION_TRACE:OPER:66 0x88 0xC 0x96 0x0 0x11 0x0 0x0 0x0 MT-SESSION_TRACE:OPER:66 0xE4 0x3C 0x76 0x6D 0x3E 0x3C 0x6D 0x70 MT-SESSION_TRACE:OPER:66 0x6D 0x73 0x3E 0x3C 0x66 0x6C 0x73 0x3E MT-SESSION_TRACE:OPER:66 0x31 0x30 0x30 0x3C 0x2F 0x66 0x6C 0x73 MT-SESSION_TRACE:OPER:66 0x3E 0x3C 0x69 0x6E 0x74 0x3E 0x31 0x30 MT-SESSION_TRACE:OPER:66 0x3C 0x2F 0x69 0x6E 0x74 0x3E 0x3C 0x66 MT-SESSION_TRACE:OPER:66 0x74 0x6F 0x3E 0x32 0x3C 0x2F 0x66 0x74 MT-SESSION_TRACE:OPER:66 0x6F 0x3E 0x3C 0x68 0x69 0x73 0x3E 0x31 MT-SESSION_TRACE:OPER:66 0x3C 0x2F 0x68 0x69 0x73 0x3E 0x3C 0x2F MT-SESSION_TRACE:OPER:66 0x6D 0x70 0x6D 0x73 0x3E 0x3C 0x6D 0x72 MT-SESSION_TRACE:OPER:66 0x74 0x70 0x3E 0x3C 0x6D 0x73 0x71 0x3E MT-SESSION_TRACE:OPER:66 0x35 0x3C 0x2F 0x6D 0x73 0x71 0x3E 0x3C MT-SESSION_TRACE:OPER:66 0x6D 0x78 0x64 0x3E 0x35 0x3C 0x2F 0x6D MT-SESSION_TRACE:OPER:66 0x78 0x64 0x3E 0x3C 0x6D 0x78 0x72 0x3E MT-SESSION_TRACE:OPER:66 0x35 0x3C 0x2F 0x6D 0x78 0x72 0x3E 0x3C MT-SESSION_TRACE:OPER:66 0x63 0x6C 0x6B 0x72 0x3E 0x39 0x36 0x3A MT-SESSION_TRACE:OPER:66 0x33 0x30 0x30 0x30 0x30 0x3C 0x2F 0x63 MT-SESSION_TRACE:OPER:66 0x6C 0x6B 0x72 0x3E 0x3C 0x73 0x73 0x72 MT-SESSION_TRACE:OPER:66 0x63 0x6D 0x78 0x3E 0x32 0x30 0x3C 0x2F MT-SESSION_TRACE:OPER:66 0x73 0x73 0x72 0x63 0x6D 0x78 0x3E 0x3C MT-SESSION_TRACE:OPER:66 0x2F 0x6D 0x72 0x74 0x70 0x3E 0x3C 0x6D MT-SESSION_TRACE:OPER:66 0x63 0x62 0x72 0x3E 0x3C 0x63 0x70 0x72 MT-SESSION_TRACE:OPER:66 0x3E 0x32 0x30 0x3C 0x2F 0x63 0x70 0x72 MT-SESSION_TRACE:OPER:66 0x3E 0x3C 0x2F 0x6D 0x63 0x62 0x72 0x3E MT-SESSION_TRACE:OPER:66 0x3C 0x66 0x72 0x63 0x72 0x64 0x3E 0x3C MT-SESSION_TRACE:OPER:66 0x66 0x72 0x6D 0x6E 0x70 0x3E 0x6D 0x31 MT-SESSION_TRACE:OPER:66 0x6D 0x32 0x6D 0x33 0x6D 0x34 0x6D 0x35 MT-SESSION_TRACE:OPER:66 0x3C 0x2F 0x66 0x72 0x6D 0x6E 0x70 0x3E MT-SESSION_TRACE:OPER:66 0x3C 0x2F 0x66 0x72 0x63 0x72 0x64 0x3E MT-SESSION_TRACE:OPER:66 0x3C 0x2F 0x76 0x6D 0x3E 0x0 0x9 0x0 MT-SESSION_TRACE:OPER:66 0x1 0x24 0x3A 0xE9 0xF8 0x4E 0x7C 0xDB MT-SESSION_TRACE:OPER:66 0x42 0x0 0x0 0x0 0x1 0x0 0x0 0x Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 48 of 102

49 MT-SESSION_TRACE:OPER:66 0x0 0x0 0x0 0x0 0x64 0x1 0xE 0x0 MT-SESSION_TRACE:OPER:66 0x17 0x0 0x8 0x0 0x4 0x0 0xC 0x0 MT-SESSION_TRACE:OPER:66 0x4 0x0 0x7 0x0 0x2 0x0 0xB 0x0 MT-SESSION_TRACE:OPER:66 0x2 0x0 0x4 0x0 0x1 0x90 0x94 0x0 MT-SESSION_TRACE:OPER:66 0x1 0x90 0x93 0x0 0x1 0x0 0x59 0x0 MT-SESSION_TRACE:OPER:66 0x1 0x0 0xC3 0x0 0x1 0x0 0xC0 0x0 MT-SESSION_TRACE:OPER:66 0x1 0x90 0x8B 0x0 0x4 0x90 0x8E 0x0 MT-SESSION_TRACE:OPER:66 0x4 0x90 0x9F 0x0 0x4 0x90 0x9B 0x0 MT-SESSION_TRACE:OPER:66 0x4 0x90 0x96 0x0 0x4 0x90 0x99 0x0 MT-SESSION_TRACE:OPER:66 0x4 0x90 0x9D 0x0 0x4 0x90 0x95 0x0 MT-SESSION_TRACE:OPER:66 0x8 0x90 0x88 0x0 0x8 0x0 0x1 0x0 MT-SESSION_TRACE:OPER:66 0x8 0x0 0x2 0x0 0x8 0x90 0x8C 0x0 MT-SESSION_TRACE:OPER:66 MT msg body end: MT-SESSION_TRACE:OPER:66 request build success, payload size = 389 IAMP is a remote operations protocol that exists between RSVP and Mediatrace. When Mediatrace is invoked, the request is sent to the IAMP layer, which then passes the request to the RSVP layer so that a path message is sent into the network. The message shows that the IAMP session was initiated by the Mediatrace profile 66. MT-SESSION_TRACE:OPER:66 IAMP session initiated... Some of the debug messages display specific attributes of the Mediatrace request header. The Mediatrace header shows the Mediatrace software version 1. The payload type in this example indicates that a dynamic policy was invoked by Mediatrace to collect performance monitoring metrics as labeled with MT_DP_CFG_AND_GETDATA The alternative to the dynamic policy being listed as the payload type would be the static policy. If the device has been pre-configured with a static performance monitoring policy, payload type would be indicated with the label PT = MT_SP_CFG_AND_GETDATA. The payload length can be used to indicate if the Performance Monitor metrics were actually collected. A low value may indicate the initial request has not received responses from its query. The session ID attribute is unique for every Mediatrace session across the entire network. If two simultaneous Mediatrace sessions are initiated to the same Mediatrace responder, the unique session ID can be used to determine the specific sessions. The request ID value is used to help the Mediatrace responder to identify the sender that originated the Mediatrace request. MT-SESSION_TRACE:OPER:66 Request Sent = MT Header: version=1 payload_type=mt_dp_cfg_and_getdata payload_length=377 session_id= request_id=66 MT Header End 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 49 of 102

50 This message shows the Mediatrace response message from the responder. Note the session id and request id values from this output, as they are the same value in the Mediatrace request message. MT-SESSION_TRACE:OPER:66 response message received, header = MT Header: version=1 payload_type=mt_response payload_length=43 session_id= request_id=66 MT Header End These messages show the internal Mediatrace processes of the device. Specifically the message indicates an event where a response was received. MT-SESSION_TRACE:OPER:66 Exit sla_mt_data_collector_start_ios MT-SESSION_TRACE:OPER:66 sla_mt_data_collector_process_event_ios: event=[mt_data], state=[sla_mt_session_connecting] The Mediatrace initiator successfully decoded the response message for performance monitoring metrics. Please note that the initiator can decode only what was received in the response message. If data was not collected, error 19 would be displayed indicating that performance monitoring metrics were not collected as per the request. MT-SESSION_TRACE:OPER:66 decoding dvmc response MT-SESSION_TRACE:OPER:66 message decode success success The session time out value allows Mediatrace responders time to reply to an initiator s request. This message indicates the starting of the Mediatrace timeout timer. MT-SESSION_TRACE:OPER:66 starting timeout timer = 3 The following messages indicate the response timeout timer has expired. This can also be interpreted as during this time frame, all the Mediatrace responses along the media path have been consolidated for reporting. MT-SESSION_TRACE:OPER:66 state=[sla_mt_session_active] IPSLA-INFRA_TRACE:OPER:66 Updating result IPSLA-INFRA_TRACE:OPER:66 life left IPSLA-INFRA_TRACE:OPER:66 is it random? 0 MT-SESSION_TRACE:OPER:66 new state=[sla_mt_session_active] #debug mediatrace trace initiator This syslog message indicates a general Mediatrace event has occurred. An example of an event would be a route change event along the media path during a Mediatrace session. If a device is 10 hops away and its egress interfaces goes down, the route change event is sent back to the Mediatrace initiator for notification. MT-EP_TRACE: received iamp callback 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 50 of 102

51 6.3.4 #debug mediatrace trace responder The Mediatrace responder indicates that an IAMP event has been triggered. This event was triggered upon receiving a Mediatrace request. MT-EP_TRACE: received iamp callback The Mediatrace responder started a local session in response to the Mediatrace request to include the specific parameters to include in the Mediatrace payload. MT-RESP_TRACE: sla_mt_resp_create_new_session: added a new session for (VM Configured Data Polling) (gsid= ) (inactivity_timeout=360) (active_ MT-RESP_TRACE: sla_mt_resp_interface_change_check: current interface info: MT-RESP_TRACE: session_id = MT-RESP_TRACE: initiator_id = MT-RESP_TRACE: input_iftype = 1 MT-RESP_TRACE: input_ifnumber = 2 MT-RESP_TRACE: input_ifinfo = 0x6A255E2C MT-RESP_TRACE: input_if_name = Gi0/0 MT-RESP_TRACE: output_iftype = 1 MT-RESP_TRACE: output_ifnumber = 3 MT-RESP_TRACE: output_ifinfo = 0x6A255E34 MT-RESP_TRACE: output_if_name = Gi0/1 The Mediatrace responder trace shows the performance monitoring metrics that were requested by Mediatrace in XML format. MT-RESP_TRACE: sla_mt_decode_request_vm: servlet_vm->xml_block = (<vm><mpms><fls>100</fls><int>10</int><fto>2</fto><his>1</his></mpms><mrtp><msq>5 </msq><mxd>5</mxd><mxr>5</mxr><clkr>96:30000</clkr><ssrcmx>20</ssrcmx></mrtp><mcb r><cpr>20</cpr></mcbr><frcrd><frmnp>m1m2m3m4m5</frmnp></frcrd></vm>) (gsid= ) The Mediatrace responder locally registers itself as a client for DVMC services for this particular session, and provides a status of OK. DVMC is the middleware layer that provides performance monitoring metrics to Mediatrace. MT-RESP_TRACE: sla_mt_requested_action_vm: dvmc_client_register OK for (gsid= ), cid (1) The Performance Monitor object was successfully created for monitoring as shown by the OK status. If an error occurred, the monitoring object would not be created. The cause for this may likely be attributed to a bug. MT-RESP_TRACE: sla_mt_requested_action_vm: dvmc_monitor_object_build OK for (gsid= ), cid (1) MT-RESP_TRACE: sla_mt_requested_action_vm: dvmc_monitor_start OK for (gsid= ), cid (1) 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 51 of 102

52 The Mediatrace responder trace message sla_mt_build_response_vm fill size value helps determine if metrics were successfully collected from the DVMC. If the metrics were not collected from the DVMC, the fill size value would equal to 0. Please note, the first Mediatrace request event would indicate a fill size of 0 as the first Mediatrace response message signals a new Mediatrace session is being established. MT-RESP_TRACE: sla_mt_build_response_vm: filled_size is (146) node_info_len (28) (gsid= ) This message indicates the Mediatrace responder has sent a response back to the Mediatrace initiator s source IP address. MT-RESP_TRACE: sla_mt_deliver_response: target IP ( ) MT-RESP_TRACE: Response message delivered to target IP ( ) MT-RESP_TRACE: sla_mt_resp_remove_session: removing a session (VM Configured Data Polling) (ok_to_remove=false) (force=false) (gsid= ) #debug mediatrace error Error code 19 assumes that the transport is working. If the transport is working, the configuration must be checked that its querying flow actually exists. Also, the flow must be active in order for Mediatrace to poll stats. MT-SESSION_ERROR: OPER: 66 dvmc error = #debug ip rsvp api This identifies a particular media session. Note the instance ID value 19 in this example. This can be seen as a unique RSVP-TP session ID. The device role can be a Mediatrace initiator, responder, or both. The initiator ID is the ip address of the Mediatrace initiator source IP address. The following messages shows the path specifier 5-tuple information that is configured using the mediatrace pathspecifier command. Make sure the flow specifier for the 5-tuple configuration matches the actual media flow. ClientID: 1 InstanceID: 19 InitiatorID [rsvp_api_tp_get_info]: Contents of rsvp_tp_info RSVP-API: TP_INFO: 0xC12E _0-> _0[1_0xA575D99_19] event: Unknown, rx-msg: Unknown tx-type: Unknown, target-ip: forward: FALSE, is_initiator_flag: FALSE, output-if: GigabitEthernet0/0 tp-data: de The following messages show that a new RSVP session has started to trace the specific media flow. rsvp_tp_initiate_session: rsvp_tp_info->dest_ip = A575DA4 rsvp_tp_info->src_ip = A575D9A, rsvp_tp_info->ip_protocol = 0 rsvp_tp_info->dest_port = 0 rsvp_tp_info->src_port = 0 rsvp_tp_info->client_id = 1 rsvp_tp_info->initiator_id = A575D99 rsvp_tp_info->instance_id = 19 rsvp_tp_info->ctx_key.vrf_id = 0 rsvp_tp_info->rsvp_tp_data = 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 52 of 102

53 The RSVP path message shows the 5-tuple information of the video session. RSVP-API: _0-> _0[ ]: Processing PATH request [id=0x0]... The RSVP path and notify message show the direction of the flow based on IP address information. RSVP-API: _0-> _0[ ]: Created Path state [id=0x0 ClientID: 1 InstanceID: 19 Initiato RSVP-API: TP_INFO: 0xC12E _0-> _0[1_0xA575D99_19] event: Unknown, rx-msg: Unknown tx-type: RSVP_TP_MSG_TRANSMIT_NOTIFY_PHOP, target-ip: #debug ip rsvp messages The debug ip rsvp command should be used in the event that Mediatrace does not receive requests or responses. RSVP messages will tell us if the devices can communicate across the media path. RSVP Transport Notify messages are unique to RSVP-TP. In cases of RSVP bandwidth admission control, RSVP-TP messages will not be visible. The following messages show the actual RSVP Path and Notify messages. Note the flow direction of the messages. RSVP: session _0[ ]: Outgoing Path, I/F=Gi0/0, Layer=IP, NHOP= , Prerouted=Y IP HDR: > , TOS=0x00, Len=608, TTL=255, RA=Y RSVP HDR: RRC=N, TTL=255, Len=584, Cksum=0xD64C RSVP: session [TBD] Incoming Transport Notify, I/F=Gi0/0, Layer=IP IP HDR: > , TOS=0x00, Len=152, TTL=252, RA=N RSVP HDR: RRC=N, TTL=255, Len=132, Cksum=0xF14A RSVP: session _0[ ]: Outgoing Path, I/F=Gi0/0, Layer=IP, NHOP= , Prerouted=Y IP HDR: > , TOS=0x00, Len=168, TTL=255, RA=Y RSVP HDR: RRC=N, TTL=255, Len=144, Cksum=0x83F1 7 IP SLA VO: IP SLA Video Operation (VO) is a new type of IP SLA probe that generates video traffic with the intention of stressing the network with the same traffic characteristics as a real video endpoint/application would. IP SLA VO is a Cisco Medianet Media Monitoring technology that enables synthetic traffic generation and monitoring using Cisco routers and switches Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 53 of 102

54 IP SLA VO simulates the following video formats (dynamic-rtp video payload): TelePresence (6.6 Mbps) IP Video Surveillance Camera (2.2 Mbps) IP TV (2.6 Mbps) IP SLA VO provides the following statistics: Inter packet delay variation (jitter) One way delay Source to destination packet loss Tail drop Out of sequence packets Configuring IP SLA VO Feature Guide 7.1 IPSLA video operation cannot be seen in Perf-Mon stat s Verify that the IP SLA VO video duration and frequency is long enough for Performance Monitor to capture and display results. Typically, it is recommended to keep the video operations duration and frequency at a reasonable rate, so that the video operation does not conflict with live traffic. Be sure to note the additional bandwidth needed to run IP SLA VO for extended lengths of time. TelePresence video profile emulates 1080P video session at 6.6Mbps. Figure 17. IP SLA VO Duration and Frequency Verification Verify the proper QoS setting is applied to the IPSLA Video Operation. In many instances, Perf-Mon may classify a media flow by using the QoS DSCP value. Be careful when running video tests which demand high amounts of bandwidth. ip sla 1 video source-ip source-port 6000 profile TELEPRESENCE duration 60 Video session last for 60 seconds dscp cs4 frequency 90 ip sla schedule 1 life 600 start-time now Video session lasts 600 sec or 10 minutes 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 54 of 102

55 7.2 IP SLA VO fails to collect any usable data (show ip sla statistics) Note that IP SLA requires the following for proper operation: 1. An IP SLA Responder configured at the destination IP address 2. NTP synchronized for delay and jitter accuracy 3. IP SLA control channel port 1967 open across the network Router-1#show ip sla statistics 1 IPSLAs Latest Operation Statistics IPSLA operation id: 1 Type of operation: video Latest operation start time: 11:00:35 EST Fri Sep Latest operation return code: OK Packets: Sender Transmitted: Responder Received: Latency one-way time: Number of Latency one-way Samples: Source to Destination Latency one way Min/Avg/Max: 0/5/11 milliseconds NTP sync state: SYNC Inter Packet Delay Variation, RFC 5481 (IPDV): Number of SD IPDV Samples: Source to Destination IPDV Min/Avg/Max: 0/2/8 milliseconds Packet Loss Values: Loss Source to Destination: 0 Source to Destination Loss Periods Number: 0 Source to Destination Loss Period Length Min/Max: 0/0 Source to Destination Inter Loss Period Length Min/Max: 0/0 Out Of Sequence: 0 Tail Drop: 0 Number of successes: 4 Number of failures: 0 Operation time to live: Forever Note: In order to get "valid" stats from IP SLA VO, one should NTP sync the clocks for both the source and responder, so the timestamps can be accurate on both units Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 55 of 102

56 Latest operation return codes: OK Video session succeeded without error, and within all specified threshold and timeout parameters. No Connection Typically means the destination IP address is unreachable, or that the responder is not enabled on that destination device. Over Threshold In the case of video, since there are no RTT values, this code occurs if any time stamp delta (receive time send time) is greater than the provisioned threshold value. If there is no NTP sync, then this error code is meaningless. Timeout In the case of video, since there are no RTT values, this code occurs if any time stamp delta (receive time send time) is greater than the provisioned timeout value. If there is no NTP sync, then this error code is meaningless. Responder failure A video specific failure, normally caused by the video session on responder timing out, and clearing out all saved data before the sender transmits the Stat_Retrieval message, which will then be rejected because no active session remains that matches the request. The responder session timer is equal to the duration plus 15 seconds (3x retries of the Stat_Retrieval control message). Authentication failure This is caused by failure of the key-chain MD5 authentication protocol. Either the sender or responder is missing the correct key-chain from the other device. This is a common error for all SLA operations, not just for video. Format failure Control message incompatibility between sender and responder. For video, this means the destination device does not support video. Busy The sender is unable to start a new video session due to bandwidth limitations. Each device can only support a certain number of video sessions, depending on the platform, and bandwidth of the selected profile. Port in use For video, this means that the responder is out of available resources for another video session. This is similar to the error Busy, but a different code to distinguish it from the sender bandwidth issue. 7.3 Some IP SLA Video Operations fail while others pass The Cat3K has a maximum limit of 20 Mbps for high bandwidth video operations. Therefore only 3 TelePresence sessions or multiple combinations of video operations such as IPTV/IPVSC/TelePresence can run concurrently with a known limit of 20 Mbps. Generally the first 3- TelePresence sessions started will succeed and all other sessions which exceed the 20Mbps maximum will fail. The system will detect this and prevent the IPSLA process from starting. Typically the reason code for this condition is No Connection. Note the additional bandwidth needed to run IP SLA-VO for extended lengths of time will cause significant congestion within the network. TelePresence video profile emulates 1080P video session at 6.6Mbps, so 3 sessions would equate to about 20Mbps Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 56 of 102

57 7.4 Troubleshooting procedures for IP SLA VO failures Figure 18. Network Topology for Troubleshooting IP SLA VO Failures In the topology, CAT3750-A is the Initiator. CAT3750-B is the reflector or the IP SLA responder Diagnose the problem If you notice that the Number of failures field increases in the output of the show ip sla statistics session-id detail command there is likely some issue in the IP SLA operation between source to responder. The best way to identify the problem is by checking the return value of the Latest operation return code: field in the show ip sla statistics session-id command. Possible Fixes: Problem: ip sla statistics return code : unknown IP SLA is not scheduled currently and the operation is not starting. show ip sla statistics BA2-3750e#show ip sla statistics 1 IPSLAs Latest Operation Statistics IPSLA operation id: 1 Number of successes: Unknown Number of failures: Unknown Operation time to live: 0 show ip sla configuration BA2-3750e#show ip sla configuration 1 IP SLAs Infrastructure Engine-III Entry number: 1 Owner: Tag: Operation timeout (milliseconds): Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 57 of 102

58 Type of operation to perform: video Video profile name: IPVSC Video duration (seconds): 20 DSCP: cs5 Target address/source address: / Target port/source port: 39702/38802 Vrf Name: Control Packets: enabled Schedule: Operation frequency (seconds): 30 (not considered if randomly scheduled) Next Scheduled Start Time: Pending trigger Group Scheduled : FALSE Randomly Scheduled : FALSE Life (seconds): 3600 Entry Ageout (seconds): never Recurring (Starting Everyday): FALSE Status of entry (SNMP RowStatus): notinservice. Threshold (milliseconds): 1000 Distribution Statistics: Number of statistic hours kept: 2 Number of statistic distribution buckets kept: 1 Statistic distribution interval (milliseconds): 20 Enhanced History: Solution: Schedule the IP SLA VO session and verify if the Next Scheduled Start Time already passed field. ip sla operation-number start start-life duration BA2-3750e#show ip sla configuration 1 IP SLAs Infrastructure Engine-III Entry number: 1 Next Scheduled Start Time: Start Time already passed <output-omited> Problem: show ip sla statistics return code : no connection Since there is no end to end connection the statistics are not collected. BA2-3750e#show ip sla statistics 30 IPSLAs Latest Operation Statistics IPSLA operation id: 30 Type of operation: video Latest operation start time: 14:37: UTC Mon Nov Latest operations return code: No connection 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 58 of 102

59 Packets: Sender Transmitted: 0 Responder Received: 0 Latency one-way time: Number of Latency one-way Samples: 0 Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds NTP sync state: SYNC Inter Packet Delay Variation, RFC 5481 (IPDV): Number of SD IPDV Samples: 0 Source to Destination IPDV Min/Avg/Max: 0/0/0 milliseconds Packet Loss Values: Loss Source to Destination: 0 Source to Destination Loss Periods Number: 0 Source to Destination Loss Period Length Min/Max: 0/0 Source to Destination Inter Loss Period Length Min/Max: 0/0 Out Of Sequence: 0 Tail Drop: 0 Number of successes: 0 Solution: 7.4.3a Check if a routing problem exists as follows: 1. Determine if there are routing issues between the source and destination 2. Check the interface through which the IP SLA traffic is up by using the show ip interface command. 3. Identify where the connection is stopped using the traceroute command and make sure the end-to-end device is reachable. 4. Make sure the destination IP SLA responder is configured at the destination. (Use the ip sla responder command on the destination device.) 7.4.3b Check if Flapping links exist Determine if there are any network issues between the two end points c Check if High CPU use on Initiator and responder exist Use the show proc cpu and show proc cpu history commands to determine if a high CPU condition is causing issues d Check if the IP SLA video ports and control port (1967) are being denied at a firewall Problem: show ip sla statistics return code : busy Issue is the same port numbers used on multiple sessions for the same source and destination IP address pairs. Latest operation return code: Busy <output omitted> BA2-3750e#show ip sla configuration 50 IP SLAs Infrastructure Engine-III Entry number: Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 59 of 102

60 Owner: Tag: Operation timeout (milliseconds): 5000 Type of operation to perform: video Video profile name: IPTV Video duration (seconds): 20 DSCP: cs5 Target address/source address: / Target port/source port: 39704/38804 <output omitted> BA2-3750e#show ip sla configuration 60 IP SLAs Infrastructure Engine-III Entry number: 60 Owner: Tag: Operation timeout (milliseconds): 5000 Type of operation to perform: video Video profile name: IPTV Video duration (seconds): 20 DSCP: cs5 Target address/source address: / Target port/source port: 39704/38804 <output omitted> Solution: For each combination of source and destination IP address pairs, use a unique port number Problem: show ip sla statistics return code : overthreshold There may be a network congestion problem. IPSLAs Latest Operation Statistics IPSLA operation id: 40 Type of operation: video Latest operation start time:.17:28: UTC Fri Nov Latest operation return code: Overthreshold Packets: Sender Transmitted: 5574 Responder Received: 1022 Latency one-way time: Number of Latency one-way Samples: 0 Source to Destination Latency one way Min/Avg/Max: 100/254/1045 milliseconds NTP sync state: SYNC Inter Packet Delay Variation, RFC 5481 (IPDV): Number of SD IPDV Samples: Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 60 of 102

61 Source to Destination IPDV Min/Avg/Max: 0/23/282 milliseconds <output omitted> Number of successes: 24 Number of failures: 20 Operation time to live: Forever Solution: Check if the congestion occurs due to any of the below problems and address the issue with correct changes. Broadcast storm Failing switches/routers IP conflicts Excessive network-based applications Insufficient bandwidth DNS errors Problem: show ip sla statistics return code: format failure Format Failure means that the control message is not recognized by the responder. Therefore, no IP SLA VO is taking place. IPSLAs Latest Operation Statistics IPSLA operation id: 3 Type of operation: video Latest operation start time: 12:07:39 EST Tue Jul Latest operation return code: Format failure Packets: Sender Transmitted: Responder Received: 0 Latency one-way time: Number of Latency one-way Samples: 0 Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds NTP sync state: SYNC Inter Packet Delay Variation, RFC 5481 (IPDV): Number of SD IPDV Samples: 0 Source to Destination IPDV Min/Avg/Max: 0/0/0 milliseconds Packet Loss Values: Loss Source to Destination: 0 Source to Destination Loss Periods Number: 0 Source to Destination Loss Period Length Min/Max: 0/0 Source to Destination Inter Loss Period Length Min/Max: 0/ Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 61 of 102

62 Out Of Sequence: 0 Tail Drop: 0 Number of successes: 0 Number of failures: 1 Operation time to live: Forever Solution: For IP SLA VO, enhancements are made to the control message to transmit additional information. All current versions of video code should have this support now, so the reason for format failure is the sender or responder is running an older version of code. Changing the version of code in the device to a new version will solve the problem Problem: show ip sla statistics return code: video failure IP SLA VO operation is unsuccessful. This may be due to high or low system CPU. IPSLAs Latest Operation Statistics IPSLA operation id: 1000 Type of operation: video Latest operation start time: 14:18:26 EDT Mon Oct Latest operation return code: Video failed Packets: Sender Transmitted: 0 Responder Received: 0 Number of Received Bytes: 0 Average Received Bit Rate: 0 Latency one-way time: Number of Latency one-way Samples: 0 Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds NTP sync state: Unknown Inter Packet Delay Variation, RFC 5481 (IPDV): Number of SD IPDV Samples: 0 Source to Destination IPDV Min/Avg/Max: 0/0/0 milliseconds Packet Loss Values: Packet MIA: 0 Loss Source to Destination: 0 Out Of Sequence: 0 Tail Drop: 0 Duplicate Sequence Number Count: 0 Number of successes: 0 Number of failures: 1 Operation time to live: Forever 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 62 of 102

63 Solution: a) Check system cpu using the show processes cpu sorted command. Router# show processes cpu sorted CPU utilization for five seconds: 53%/50%; one minute: 50%; five minutes: 52% PID Runtime(ms) Invoked usecs 5Sec 1Min 5Min TTY Process % 0.63% 0.62% 0 RSVP % 0.39% 0.39% 0 Skinny Msg Serve % 0.30% 0.31% 0 IP Input <<<<<output omitted>>>>> b) Check for system memory usage with the show processes memory sorted command: Router# show processes memory sorted Processor Pool Total: Used: Free: I/O Pool Total: Used: Free: PID TTY Allocated Freed Holding Getbufs Retbufs Process *Init* *MallocLite* MMON PROCESS CCSIP_SPI_CONTRO PPP Events OER Border Route Note: In the above output, the router is using over 90% of memory. Processor pool memory is % should be around 664M. Now you have used up 706M which is above 90%. So CAC (Medianet Call Admission Control) kicked in and stops the video operation. The only solution is to increase the memory, or stop some memory intensive processes. c) Reserve DSP: High-Density Packet Voice Digital Signal Processor (DSP) Module (PVDM3) enables Cisco Integrated Services Routers to provide high-density voice connectivity, conferencing, and transcoding capabilities in Cisco IP Communications solutions. So, add PVDM module to the box and reserve the dsp for ip sla vo as follows. Router(config)#ip sla 10 Router(config-ip-sla-video)#reserve dsp Problem: show ip sla statistics return code: sequence error When several sessions are running, the responder is often unable to respond to the stat retrieval message within the timeout time, which causes the sender to reject the message when it does arrive. IPSLA operation id: 7 Type of operation: video Latest operation start time: 16:54:07 eastern Mon Feb Latest operation return code: Sequence error Packets: Sender Transmitted: Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 63 of 102

64 Responder Received: 0 Latency one-way time: Number of Latency one-way Samples: 0 Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds NTP sync state: SYNC Inter Packet Delay Variation, RFC 5481 (IPDV): Number of SD IPDV Samples: 0 Source to Destination IPDV Min/Avg/Max: 0/0/0 milliseconds Packet Loss Values: Loss Source to Destination: 0 Source to Destination Loss Periods Number: 0 Source to Destination Loss Period Length Min/Max: 0/0 Source to Destination Inter Loss Period Length Min/Max: 0/0 Out Of Sequence: 0 Tail Drop: 0 Number of successes: 0 Number of failures: 1 Operation time to live: Forever Solution: In Cisco IOS, the video statistics retrieval message timeout is changed to 10 seconds (from the default of 5) to allow the responder additional time to respond Problem: show ip sla statistics return code: Ok [with ntp unsynchronized] Even though the IP SLA statistics return code is okay, there is loss between the source to destination. It may be due to a NTP synchronization issue. Note the show ip sla statistics detail command displays the NTP sync state: NO_SYNC field, indicating the state of NTP synchronization. IPSLAs Latest Operation Statistics IPSLA operation id: 10 Type of operation: video Latest operation start time: 13:55:19 est Thu Oct Latest operation return code: OK Packets: Sender Transmitted: 6202 Responder Received: 5691 Latency one-way time: Number of Latency one-way Samples: 5991 Source to Destination Latency one way Min/Avg/Max: 4/6/21 milliseconds NTP sync state: NO_SYNC Inter Packet Delay Variation, RFC 5481 (IPDV): Number of SD IPDV Samples: 5862 Source to Destination IPDV Min/Avg/Max: 0/2/17 milliseconds 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 64 of 102

65 Packet Loss Values: Packet MIA: 0 Loss Source to Destination: 611 Out Of Sequence: 0 Duplicate Sequence Number Count: 0 Number of successes: 8 Number of failures: 2 Operation time to live: Forever NTP synchronization has not occurred, which may be due to NTP not being configured or loss of connection between the device to the NTP server. The output of the show ntp status command indicates that the NTP clock is not synchronized. CA2-FF1408# show ntp status Clock is unsynchronized, stratum 16, no reference clock nominal freq is Hz, actual freq is Hz, precision is 2**17 reference time is (20:00: est Thu Dec ) clock offset is msec, root delay is 0.00 msec root dispersion is 1.78 msec, peer dispersion is 0.00 msec loopfilter state is 'FSET' (Drift set from file), drift is s/s system poll interval is 64, never updated. Solution: a. Configure NTP server. CA2-FF1408(config)# ntp server ip-address b. Check for the reachability of the switch to NTP configured server. Once NTP is configured and synchronized the field Clock is synchronized will display in the output of the show ntp status command, as shown in the following output. CA2-FF1408# show ntp status Clock is synchronized, stratum 7, reference is nominal freq is Hz, actual freq is Hz, precision is 2**17 reference time is D241C533.B0249DC0 (16:14: est Thu Oct ) clock offset is msec, root delay is 2.66 msec root dispersion is msec, peer dispersion is 5.57 msec loopfilter state is 'CTRL' (Normal Controlled Loop), drift is s/s system poll interval is 64, last update was 471 sec ago. CA2-FF1408# Problem: show ip sla statistics return code : Video resources full IP SLA VO initiator node does not have the required resource for video operation. IPSLA operation id: 13 Type of operation: video 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 65 of 102

66 Latest operation start time: 23:36:53 EDT Sun Oct Latest operation return code: Video resources full Packets: Sender Transmitted: 0 Responder Received: 0 Latency one-way time: Number of Latency one-way Samples: 0 Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds NTP sync state: Unknown Inter Packet Delay Variation, RFC 5481 (IPDV): Number of SD IPDV Samples: 0 Source to Destination IPDV Min/Avg/Max: 0/0/0 milliseconds Packet Loss Values: Packet MIA: 0 Loss Source to Destination: 0 Out Of Sequence: 0 Tail Drop: 0 Duplicate Sequence Number Count: 0 Number of successes: 0 Number of failures: 1 Operation time to live: Forever Solution: More IP SLA VO bandwidth consuming initiator sessions are scheduled than what a router can allocate. Reduce the number of sessions, or schedule different IP SLA VO sessions at different time Problem: show ip sla statistics return code Video responder resources full IP SLA VO responder node does not have required resource for video operation. IPSLA operation id: 14 Type of operation: video Latest operation start time: 13:08:37 EDT Mon Oct Latest operation return code: Video responder resources full Packets: Sender Transmitted: 0 Responder Received: 0 Latency one-way time: Number of Latency one-way Samples: 0 Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds NTP sync state: Unknown Inter Packet Delay Variation, RFC 5481 (IPDV): Number of SD IPDV Samples: 0 Source to Destination IPDV Min/Avg/Max: 0/0/0 milliseconds Packet Loss Values: Packet MIA: 0 Loss Source to Destination: Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 66 of 102

67 Out Of Sequence: 0 Tail Drop: 0 Duplicate Sequence Number Count: 0 Number of successes: 0 Number of failures: 3 Operation time to live: Forever Solution: More IP SLA VO bandwidth consuming sessions are scheduled from a different initiator device to a single responder device. Reduce the number of sessions, or schedule different IP SLA VO sessions at different time IP SLA VO feature on Cat3K, Cat4K and ISR-G2 have interoperability issues This Table shows some of the issues existing with the associated platform and Cisco IOS images noted. Please refer to the IPSLA Video Operation Across Platforms white paper to learn how each platform interoperates and the issues with respect to each platform required when using IPSLA VO across different platforms. Table 8. IP SLA VO Interoperability Issues Platform Image Version Cisco Catalyst 3K 12.2(58)SE Responder Cisco Catalyst 4K 15.1(1)SG Responder Cisco ISR G2 15.2(2)T Responder Cisco Catalyst 3K Series 12.2(58)SE2 Sender Jitter inaccurate (CSCts88988) Latency inaccurate (CSCtx45855) OK Cisco Catalyst 4K Series 15.1(1)SG Sender Latency inaccurate (CSCtx45855) Jitter inaccurate (CSCts88988) Latency inaccurate (CSCtx45855) Latency inaccurate (CSCtx45855) Cisco ISR G2 Series 15.2(2)T Sender Jitter inaccurate (CSCts88988) Bit Rate inaccrate (CSCtx35647) Host emulate fails (CSCtx55732) Latency inaccurate (CSCtx45855) Host emulate fails (CSCtx55732) OK Table 9. Caveats CSCtt32852 CSCts88988 CSCtu34249 CSCtq40669 CSCtx33273 IP SLA VO Known Issues Description Cat3K calculate Jitter values for IP SLA VO TP is higher vs real TP traffic (this should apply to ISR G2 as well) IP SLA VO IPDV (RFC-5481) stats are not accurate (Resolved but not applied to cat3k CCO latest image) Cat4K does not generate SSRC s, thus Medianet Perf-Mon will not be able to measure performance metrics for IP SLA VO traffic coming from the Cat4K. IP SLA video: control message interoperability fix. IOS image SE1 causes SLA Interop issues fixed in SE2. IP SLA VO aggregated stats sum certain metrics which should not be summed, but averaged over all the tests. Avg Bit Rate, IPDV Min/Avg/Max 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 67 of 102

68 CSCtx45855 CSCtu36231 CSCtx55732 Cat4K IP SLA VO Stats does not measure Latency correctly. Cat4K ISR-G2 CPCM 1.1 to support different IP SLA profile for ISR G2 ISR-G2 ipsla-vo emulate feature fails interop w/ CAT3K and CAT4K IP SLA VO feature for each supported platform have different scalability requirements IP SLA VO on the ISR-G2 platform uses PVDM-3 DSPs and the Catalyst 4k platform uses FPGAs to perform video operations. On the Catalyst 3k, packet generation speeds are improved by directly writing packets to the hardware queues avoiding scheduling/queueing inefficiencies. The CPU utilization on the Responder side is the determining factor for IPSLA VO deployment, as the numbers supported on the responder side is lower than on the Sender side. Table 10. IP SLA VO Scalability Requirements Platform As a Sender As a Responder Cisco Catalyst 3K Series Cisco Catalyst 4K Series Cisco ISR G2 Series Maximum of 20 Mbps (Profiles can add up in any combination) Maximum of 128 sessions (Profiles can add up in any combination) Maximum of 20 Mbps (Profiles can add up in any combination) SUP specific SUP-7E can support a maximum of 15 TP sessions or 60 IPVSC sessions or 69 IPTV sessions Platform specific Platform specific 3945E can generate 175 CTS-1080P- 3945E can respond to 80 CTS-1080P- Best VO 20% CPU utilization Best 72% CPU utilization 8 Medianet Media Monitoring Network Topology and Configuration Example The following section provides Medianet Media Monitoring network topology and configuration example Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 68 of 102

69 Figure 19. Network Topology for 8.1 Site#1 Router (3845-1) # show run Last configuration change at 09:02:08 EST Thu Jun by cisco NVRAM config last updated at 09:02:49 EST Thu Jun by cisco version 15.1 service timestamps debug datetime localtime show-timezone service timestamps log datetime localtime show-timezone service password-encryption hostname boot-start-marker boot system flash:c3845-adventerprisek9-mz t.bin boot-end-marker card type t Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 69 of 102