Application Note NetComplete SAS Metro Ethernet Test and Troubleshooting Metro Ethernet has earned an industry-wide reputation as the most cost-effective, resilient, scalable, and reliable switching and transport technology that supports interoperability; the bandwidth demands of IPTV, VoIP, and digital data applications; and a rapidly expanding customer base. The migration toward Metro Ethernet, IP-based networks and services adds greater complexity to maintaining carrier-class quality at the node and network levels, where different technologies converge to deliver bundles of video, voice, and data services to multiple customers. Creating a bulletproof IP Ethernet network requires proactive monitoring, centralized testing, and data consolidation. Service providers need to see inside customer traffic; gather, capture, and access data from multiple points in the network; and isolate faults between the customer premises and the service provider as part of their service assurance process. An integrated operations support system (OSS) that promotes proactive live-traffic monitoring, simulated-traffic testing, and troubleshooting from a central location and performs in-depth analysis is a key component of the service assurance process, enabling service providers to: Turn up and verify end user services for fast revenue starts Detect, isolate, and fix faults between the core network, last mile, and customer premises Eliminate costly and unnecessary repeat dispatches for service turn-up and troubleshooting Web Client NetAnalyst Test OSS QT-EMS Test Creation Results Management QT Testhead Configuration QT-600 Ethernet NIU QT-600 QT-600 Testhead Test Execution Automatic Loopback, Swapping MAC SA/DA QT-600 FST-2802 T-BERD-8000 WEBSITE: www.jdsu.com
2 Metro Ethernet Architecture Overview The Metro Ethernet Forum has defined two Metro Ethernet connectivity services: Ethernet Line Service (ELS) for point to point services Ethernet LAN Service ( E-LAN) for multipoint to multipoint configurations Both ELS and E-LAN have associated service attributes and parameters for meeting stringent SLAs and ensuring interoperability between service providers and vendors implementing Metro Ethernet tripleplay services. An Ethernet-switched metropolitan area network can be segregated by VLANs, which separate voice, video, and data traffic and prioritize the delivery of delay-sensitive packets such as voice and video. Customers can be assigned their own set of VLANs to ensure that all Ethernet frames travel untouched through the service provider s network. This configuration, referred to as tag preservation or QinQ, adds a second layer that allows customers VLANs to overlap, reducing the number of VLANs a service provider needs to manage for the same number of customers. Customers internal VLAN identifiers, therefore, remain specific to the workings of their networks and are hidden from the service provider s network. Demarcation Point 6509 UNI EVC1 VLAN 10 VLAN 20 VLAN 30 Edge Access EVC2 Carrier Assigned VLANs MAC VLAN10 IP Port MAC IP Port MAC VLAN20 VLAN10 IP Port MAC VLAN10 IP Port Some of the questions that service providers must address when delivering and maintaining Ethernet services over distributed networks include: Is this single Ethernet segment working? Is there connectivity across my network? Is there connectivity across a multi-provider network? Is there connectivity site-to-site for the user? Is there multicast connectivity? What is the latency across the network for a given service/vlan? Is there any packet loss for a given service/vlan? What is the throughput for a given service/vlan?
3 A lack of standardized testing and monitoring tools and procedures, however, has forced many service providers to delay or limit the introduction of Metro Ethernet triple-play services. JDSU s NetComplete Service Assurance Solution combines NetAnalyst a powerful test OSS in the NOC with JDSU s QT-600 distributed IP Ethernet testheads, located at Ethernet aggregation switches or routers in the network as close to the edge as possible. This integrated solution provides the only testing and measuring system that provides in-depth testing and data analysis from the service level agreement perspective. NetComplete enables the information from the QT-600 s data capture and segmentation functions to be integrated with filter and alarm triggering features. Service providers ability to initiate data capture, data analysis, proactive monitoring, and the subsequent preventive measures and corrective actions from the NOC establishes operational and management efficiencies that result in significant reductions in MTTR, labor costs, and service interruptions. Integrated service assurance solution for Metro Ethernet turn-up and troubleshooting On-demand active tests are the key to service providers ability to provision services more quickly. With an integrated system, customized tests, easily created on the test OSS, are run on the IP Ethernet testheads, which access network resources and communicate results back to the test OSS. For example, the QT-600 conducts active tests, including Ping and TraceRoute, out to the customer premises, quickly segments the network, and identifies connectivity issues. Additionally, an integrated system provides a seamless approach to centralized turn-up testing and troubleshooting and enhances service providers service assurance process. As an example, service providers perform acceptance testing from the NOC before hand off to the customer without dispatching technicians to the field to verify service quality and network connectivity. Once the customer accepts the service, customized, automated tests can be created on the test OSS and run by the testheads in the network. Through automated data capturing, remote testing, and active and passive monitoring, service providers gain a coherent view of network performance, customer traffic, and usage patterns. Reports generated from analysis of the consolidated data show utilization trends, link statistics, and frame distributions. With topn analysis of applications, protocols, VLANs, VLAN priorities, IP/MAC pairs, listeners, and talkers along with supplemental reports generated via the test OSS, service providers can immediately identify offenders and troubleshoot degradations before service-quality becomes unacceptable.
4 Connectivity testing with Ping and TraceRoute Emulating an endpoint on the customer s VLAN, a QT-600 testhead runs Ping and TraceRoute tests out to the customer premises to verify service connectivity. PASS/FAIL results as well as detailed and summary results are sent to the server and stored for future reference and analysis. RFC-2544 testing Originally written for single device testing, it became apparent that the RFC-2544 specification could also be applied to testing networks. The RFC standard defines four different tests: Throughput Latency Frame loss Back to back With the exception of the latency test, which needs to be run in conjunction with the throughput test, important testing features include the flexibility to run each test separately and the ability to specify varying frame sizes.
5 Throughput test To determine the achievable maximum throughput out to the customer premises, the QT-600 sends a specified number of frames at configurable bandwidth rates, up to a maximum of 110 percent of the subscriber rate, to the device under test (DUT) and measures both the number of frames the DUT receives and the number of frames it transmits. This process is repeated using varying throughput durations and decreasing bandwidth until the bandwidth rate where the number of frames sent to the DUT and the number of frames the DUT sends back is equal. The testhead issues PASS/FAIL results that are reported in tabular and graphic formats. Latency test To determine the average round-trip delay of the frames being looped back by the far end device, a latency test is run in conjunction with the throughput test. The user can specify the test s duration for each frame size and set a target latency for PASS/FAIL criterion. Because smaller frames can cause network elements to work harder than larger frame sizes, the QT-600 generates both small and large frames. Each test runs once and the latency test results are reported in either a graphic or tabular format that shows the average delay values for each frame size.
6 Frame loss test The frame loss test determines the utilization rate where no frame loss is present on the customer s VLAN. The test begins at 100 percent of the subscriber s bandwidth rate and repeats in decreasing increments until no frame loss is measured. The user determines how much the bandwidth rate should be decreased between unsuccessful trials. The test passes only when no frame loss is detected. The results are presented in both tabular and graphical formats. Back to back test The back to back test determines the network s ability to sustain traffic bursts and is run at the physical line rate rather than the customer s subscribed line rate. The QT-600 sends multiple iterative bursts, calculates the average burst length based on the burst s frame size and duration, and reports the results. The user can specify a burst resolution granularity as a PASS/FAIL criterion. Burst results within that granularity are reported as PASSED.
7 The results are displayed in both graphical and tabular formats. Automatic loopback provisioning Because the RFC-2544 tests are intrusive, the QT-600 performs a layer 2 loopback at the far-end device to swap src/dest MAC addresses. NetComplete enables service providers to automate this process when performing RFC-2544 loopback testing with other test sets such as the JDSU FST-2802, another Ethernet testhead, or third-party devices such as Covaro and Canoga Perkins NIUs. Automatic switch configuration The QT-600 s ability to automatically configure a switch streamlines testing procedures. There are two categories of switch commands the QT-600 can issue, depending on whether it is performing an active test or passive monitoring. For active testing, the QT-600 emulates an endpoint on the customer s network and issues commands to join the customer s VLAN. Based on the customer s ability to support 1 or 2 VLAN tags, the QT-600 adds the appropriate number of VLAN tags to the frame. The traffic s VLAN priority can be changed and IP address/tos settings in the test frames easily specified, enabling the QT-600 to more accurately emulate the customer s traffic patterns. For passive monitoring, the QT-600 issues commands to the switch to mirror live traffic. The mirrored traffic can be specified at the port level, VLAN level, or a combination and sent to the QT-600's test port.
8 Troubleshooting and monitoring Passive testing with Netmon Short-term nonintrusive monitoring of live traffic, a key component of the troubleshooting process, detects problems and gives service providers a sense of how customers experience the service. Netmon is an interactive, customized test initiated from the test OSS that provides views of network utilization, data link statistics with frame distributions, and topn classifiers. The QT-600 runs Netmon either continuously or at scheduled times on specific segments of the network. At the NOC, the user can define the sample window update intervals (5-seconds minimum) as well as the number of topn users to display. The QT-600 can automatically define traffic of interest by issuing mirroring commands on the switch that specify the port numbers and/or multiple VLANs to monitor. Multiple filters based on several criteria such as MPLS tag, VLAN tag and priority, IP or MAC src/dst address, and sliding and/or fixed pattern matching can be applied. Once the test is launched, the user can change the filter criteria and topn classifier (VLAN, VLAN priorities, applications, etc.) on the fly. These changes take affect immediately with results displayed on the next sample window. The subsequent statistics and reports let service providers quickly determine if problems are caused by customers equipment or applications or if the fault lies in the network equipment or in the service layers, without dispatching a technician. The user can select the following views for display: Summary Provides a summary of the filter criteria, topn criteria, and port/vlans being mirrored Trending Displays trending graph for link and filtered data utilization Statistics Shows frame size distribution and type TopN Displays the topn statistics as determined by the selected test criteria. Trending view Total vs. filtered metrics are displayed graphically, providing 1 minute, 5 minute, and 15 minute rolling views.
9 Statistics The QT-600 captures and displays the cumulative breakdowns of multicast, unicast, and broadcast frames as well as the frame distribution. These, along with sample counts for both filtered and total traffic, are displayed with or without the sample data utilization trend graph. TopN analysis The filter criteria and topn classifiers can be changed on the fly even while monitoring is in progress, and the changes will be displayed on the next sample window. Capture test Independent of the Netmon test as well as during Netmon, capture tests can be set up to run continuously or at certain times. Multiple filters can be applied to define the specific traffic to capture. Capture tests use stop criteria that are based on a trigger definition or the percentage of buffer fullness. If the criteria are not met, testing will cease at the end of the test. The QT-600 can store up to 120 Mbytes of capture data and will automatically transfer the files to the NetAnalyst server for long-term storage. The files, saved in.pcap or.cap file formats, can be automatically displayed and the protocols decoded (full seven layer decode) for further post-analysis using Examine software. Running capture tests concurrently with analysis as well as capturing data on the fly are both tremendous troubleshooting assets. The functions allow technicians to identify and isolate intermittent problems as well as detect previously unexpected degradations in performance so that appropriate corrections are implemented.
10 JDSU s Metro Ethernet NetComplete Service Assurance Solution components NetAnalyst Test OSS and QT-EMS Both NetAnalyst and QT-EMS software run on a Sun server. The QT-EMS software is a basic element management system that manages testhead configuration information, performs automatic upgrades, and displays alarms. NetAnalyst centralizes crucial test creation and management functions. Service providers can create customized, predefined, repeatable tests and schedule them to run automatically, reducing userinduced errors and allowing specific trouble areas to be quickly detected and isolated. Results are automatically emailed to a list of users and stored on the server for use as baseline measurements, future reference, and performance comparison over time. Consistent and repeatable test campaigns created with NetAnalyst allow service providers to initiate proactive measures that prevent service degradation and network bottlenecks. A customer-to-qt-600 testhead mapping function makes testing more intuitive. Tests are created and run against customer services rather than against specific QT-600 testheads, making the testing process simpler and less prone to error. External OSS EMS Features QT Configuration and Provisioning QT S/W Upgrade User Profiles Equipment Alarms NetAnalyst NGT Sun Solaris QT-EMS Web Server Sys DB XML HTTP/ HTTPS Test OS XML Test OS Features Test Automation Test Queuing Test Profiles Result Archival/Reporting/ Export Capture File Management XML IP XML
11 QT-600 IP Ethernet testhead The QT-600 is a triple-play testhead. In addition to the Metro Ethernet test suite, the QT-600 s unique features are standard on each testhead: Built-in intelligence Linux server and XML I/F (more accurate; results processed in testhead in hardware at physical line rate) Open standards SNMP, XML, IP 100% web Automated TAM control enables direct communication with the switch to configure test port for active testing and passive monitoring Metro Ethernet testing and troubleshooting functions Ping, TraceRoute connectivity testing RFC-2544 (throughput, frame loss, latency, back to back) Automatic layer 2 loopback of third-party NIUs, QT-600, and JDSU FST-2802/T-BERD-8000 Data utilization trending and monitoring Frame statistics breakdown TopN analysis (conversations, pairs, listeners, talkers, VLANs, VLAN priorities) Other testing applications VoIP turn-up testing, troubleshooting, and monitoring Active call generation with capture, filters, and triggers QT-600 to QT-600 one way and two way QT-600 to HST-3000 QT-600 in answer mode with retransmit option Passive monitoring 6000+ streams Full QoS analysis for both active and passive streams (sample and cumulative) MOS and R-Factor (CQE, LQE, NPE) Jitter instantaneous and RFC, clock skew Packet statistics (loss, dropped, total) Stream duration, post-dial delay
12 IPTV testing and troubleshooting The QT-600 emulates the set-top box by generating and analyzing the IGMPv2 (RFC 2236) protocol for IPTV or RTSP (RFC 2326) protocol for VoD. Through in-depth analysis of video services, the QT-600 provides the following: Video service verification Video service ETSI TR 101 290 Measurements: Latency of the channel changing on IPTV Measurement of the items below for each PID present: Video bit rate (min, max & average) Audio bit rate (min, max & average) Data bit rate (min, max, & average) Measurement of the following IP transport stream statistics: Sync frame errors Errored packets (PDUs) Continuity counter errors Packet jitter Measurement of the MPEG2-TS PCR (program clock reference) jitter PCR jitter analysis to verify that the critical timing reference data is properly received Measurement of the MPEG2-TS packet loss, for continuity errors PAT, PID, PMT errors Additionally, the QT-600 can check the broadcast service name in the MPEG stream. All statements, technical information and recommendations related to the products herein are based upon information believed to be reliable or accurate. However, the accuracy or completeness thereof is not guaranteed, and no responsibility is assumed for any inaccuracies. The user assumes all risks and liability whatsoever in connection with the use of a product or its application. JDSU reserves the right to change at any time without notice the design, specifications, function, fit or form of its products described herein, including withdrawal at any time of a product offered for sale herein. JDSU makes no representations that the products herein are free from any intellectual property claims of others. Please contact JDSU for more information. JDSU and the JDSU logo are trademarks of JDS Uniphase Corporation. Other trademarks are the property of their respective holders. 2006 JDS Uniphase Corporation. All rights reserved. 30137421 001 0106 NETCOMPLETEMETROETH.AN.SAS.TM.AE Test & Measurement Regional Sales NORTH AMERICA TEL: 1 866 228 3762 FAX: +1 301 353 9216 LATIN AMERICA TEL: +55 11 5503 3800 FAX: +55 11 5505 1598 ASIA PACIFIC TEL: +852 2892 0990 FAX: +852 2892 0770 EMEA TEL: +49 7121 86 2222 FAX: +49 7121 86 1222 WEBSITE: www.jdsu.com