Testing Video over IP Product and Services

GIGANET S Y S T E M S Precision Performance Repeatability Testing Video over IP Product and Services Application Note

Introduction Video over IP has gone mainstream. Over the last few years, the number of Video over IP applications and uses has increased exponentially and now forms an integral part of our daily lives. Despite its seeming ubiquity however, Video over IP is still in its infancy. Cisco s annual Visual Networking Index estimates that IP Video from all sources - such as videoconferencing, Over-The-Top content including streamed movies/tv, Internet video clips, and others - will be the primary force in driving worldwide broadband data consumption to reach 1.3 zettabytes (1 billion terabytes) by 2016. Offering Video over IP services is one thing, delivering them effectively is yet another. It is a complex technology requiring the successful interoperation of many components spread across multiple networks and owned by different organizations - all while supporting a host of other voice/data services. These factors highlight the one key reality facing Video over IP services public IP networks are not optimized for them. Video is highly sensitive to latency and impairments and public IP networks typically provide best-effort environment with delay, dropped re-ordered packets, bandwidth issues and other impairments - the severity of which will vary with time and with how well the network is engineered. To successfully provide and support Video over IP, Network Equipment Manufacturers and Service Providers must thoroughly understand the technology, the services available, and how they are to be delivered. Only then can they design their products and networks to ensure interoperability and provide such key functions as error correction or recovery algorithms, among others. To realize the significant potential of Video over IP, both groups need to test properly and extensively. Emulate and Test In an ideal world, everything would be plug-and-play straight out of the box. Unfortunately, the risks inherent in introducing an untested or unproven product or service into a live network are just too high. There are many well publicized cases where a software upgrade or launch of a new product has had disastrous effects on the live network simply because not enough testing was done. Testing is a must. The seemingly obvious solution is to test using the actual live network where the product or service is to be used. Unfortunately, this is highly impractical for the reasons listed above, as well as for the fact that conditions in a live network are unpredictable. Interpreting results in this environment is challenging at best and, with no control over the conditions for any given test case, repeatable testing is impossible. What if you could build a test bed that would act like a real-world network but provide complete control and repeatability? You can, with a network emulator the right network emulator. A network emulator provides the test user with the ability to mimic the delays and other impairment conditions of a real-world network, and evaluate the performance, stability, or functionality of a product Copyright 2012 V1.0 www.giganetsystems.com P a g e 1

or service as it would function once deployed. It can be either software-based or use dedicated hardware, but note that these two will offer very different levels of precision and performance. Software-based emulators cannot guarantee repeatable testing Software or appliance-based network emulators will typically be on the lower end in terms of performance and precision for the simple fact that they lack the processing power to deal with higher volumes of traffic and multiple impairments simultaneously. In fact, when strained or loaded, they can actually ADD impairments (e.g. dropping packets when packet sizes are smaller) GigaNet Systems VirtualNet line of Ethernet Network Emulators are hardware-based network emulators (using dedicated FPGAs), delivering full line-rate performance for 1G and 10G Ethernet traffic at all frame sizes, AND regardless of the impairment settings applied by the user. With the ability to replicate real-world network conditions in a 100% controlled manner, users are guaranteed that no other variable has been introduced and results are precise and, most importantly, repeatable. With a VirtualNet Ethernet Network Emulator from GigaNet Systems you could confidently focus on any number of Video over IP specific test areas including: Validating system response to any/all of the standard network impairments Quantifying service quality or end user Quality of Experience (QoE) Checking network redundancy and hit-less switching functionality Accurately estimate SLAs needed for transporting video across a MAN and/or WAN Measure system response to MPEG2 TS layer errors such as: Sync header/continuity counter error Loss of program information Loss of Program Reference Clock (PCR) Loss of Presentation Time Stamp (PTS) Validate performance of error recovery mechanisms such as Frame Check Sequence (FCS) Evaluate audio / video quality degradation and error concealment algorithms Check robustness of video compression algorithms such as MPEG2 / H.262, MPEG4 / H.264 or JPEG 2000 Stress-test audio / video alignment buffers Test video clock recovery algorithms Stress-test de-jitter buffers in video transport equipment Stress-test Video FEC algorithms (SMPTE2022-5) WAN Evaluate end-to-end IP Video system performance or test the interworking of any of the network s components by emulating any of the links below Video Headend Core Router Edge Router MAN Mobile Router DSLAM HDTV VoD Servers VoD Servers Local Content Server Copyright 2012 V1.0 www.giganetsystems.com P a g e 2

Key Emulator Requirements for Video Over IP Testing To properly evaluate the areas above requires an emulator not only with a complete range of impairment capabilities, but also one that provides extreme granularity and functionality. VirtualNet Ethernet Network Emulators comprehensive list of impairments (and their importance to Video over IP testing) includes (among many others): Delay Typically considered one of the more basic impairment capabilities, its importance should not be underestimated. Simulating such things as the physical distances or network congestion to characterize video clock recovery requires being able to precisely control delay. VirtualNet does this well - providing up to 10s of delay in increments as little as 6.4µs. Jitter (Packet / Delay / PCR) Jitter can have a profound effect on the quality of video which, in turn, impacts what is considered to be the most important metric of any Video Over IP service, the end user s Quality of experience (QoE). Understanding the effects of jitter on video is critical for designing/optimizing de-jitter buffers and packet repair techniques to ensure service success. Building on the precision of the delay functionality, delay jitter will range between the Min. and Max. Delay as set by the user. Media Drop Media Drop allows users to target Ethernet packets carrying Video content using specific IP address/udp port numbers. Unique to VirtualNet however, is the ability to also target specific Packetized Elementary Stream (PES) by matching on specific program ID (PID) contained within MPEG2 TS packets. Dropping MPEG2 TS (such as Program Clock Reference (PCR), PES header or MPEG4-AVC I/P/B frames packets rather than the entire Ethernet frame provides the user with extreme precision and control for their testing purposes. FEC Drop Forward Error Correction (FEC) algorithms (as specified in SMPTE 2022-1 and 2022-5) are used in Video over IP Transmission networks to recover lost Media frames due to random bit errors or dropped frames resulting from network congestion. However, based upon the size and the type of FEC matrix utilized, there are conditions to when FEC will not work meaning many frame drop patterns are not recoverable. To simulate these scenarios, VirtualNet provides FEC Drop and allows users to stress their implementations, parameters, and recovery mechanisms. Frame Drop As its name implies, Frame Drop simply emulates frame/packet loss. Frame Drop controls allow the user to drop selected traffic (randomly or targeted) at a specified rate, duration, and distribution. Dropping packets can be used to simulate errors and evaluate everything from the system s response to MPEG2 TS layer errors (including Packetized Elementary Streams (PES) and ES), to error/clock recovery mechanisms, and audio/video/jitter buffers. Reorder Frame reorder impairment emulates situations where frames/packets arrive out of order at the destination from situations such as routing table updates, route flaps, or because of TCP retransmission. The receiving system must re-sequence the arriving packets to recover the data stream sent from the source. From a test perspective, users want to know what happens when re- Copyright 2012 V1.0 www.giganetsystems.com P a g e 3

ordered packets arrive too far out of order. How does the system handle it? Does it delete/drop it? Does it recover it in its original place? What is considered "too many" re-ordered packets? Modification / Corruption The Modification function changes protocol fields in the Ethernet Frames. Frame Corruption on the other hand, introduces bit errors into the user data traffic to emulate data corruption during transmission. Like Frame Drop, these two functions are used to simulate MPEG2 TS layer errors, stress error/clock recovery mechanisms (e.g. by modifying RTP sequence numbers to impact FEC), evaluate video compression algorithms such as MPEG2 / H.262, MPEG4 / H.264 or JPEG 2000, check audio/video/jitter buffers, and more. The corruption feature however adds another important capability from a Video over IP perspective. It is the ability to perform ASI corruption. Since there is no CRC in MPEG2 TS, ASI Links have no error detection ability. Bad header values will make it to the equipment and VirtualNet provides a way to test for this scenario. Checksum Correction Of course, both Modification and Corruption would actually be a hindrance in testing without Checksum Correction. Once any packet is modified or corrupted, the Checksum must be corrected all the way up to the UDP layer otherwise the packet will be dropped by the layer. This feature is one of the many areas where the differences between Software-based emulators and Hardware-based emulators like VirtualNet are obvious. Checksum Correction is a very intensive function and Software-based emulators would see significant performance degradation. For this reason and others, only VirtualNet offers Checksum Correction at both 1G and 10G speeds. Targeted Testing By themselves, each of the features described above provide some solid capabilities for evaluating Video over IP products and services. VirtualNet takes these capabilities once step further and offer users the unique ability to select a particular frame for specific impairments based on user-specified frame protocol field values right down to a specific TS packet (such as and I/B/P frame) within the Ethernet frame. The ability to perform Field Comparisons including =,,,, Within a Range, Out-of-Range Multiple comparisons when combined with AND / OR conditions provides the user with a powerful tool to do more than just randomized testing. White Box Testing In contrast to Random testing where users introduce impairments such as latency, drop, or re-order to observe the effects of these actions on their product or service, White Box testing takes a much more precise approach. Under this method, the user exercises greater control over the conditions of the test by targeting specific portions of the product or specific packets of the service. Using this approach to create targeted test cases, observed issues are simple to reproduce and design changes can be tested with confidence. There is no need to perform tedious stream analysis to deduce root cause (as the introduced anomaly is the root cause) and the results are not only easier to interpret, but they provide much greater insight into the actual functioning of the product or service. Used together, the White Box and Random Testing approaches enabled by VirtualNet provide users with a much more comprehensive and thorough test plan. Copyright 2012 V1.0 www.giganetsystems.com P a g e 4

Conclusion Video over IP is a complex application. Poorly executed solutions will result in commercial failure and can even cause irreparable damage to the vendor or service provider s reputation. Successful deployment requires the transparent interoperability of the multiple components that form an end-toend network. Not only does that mean testing, but it requires testing with the right product. GigaNet Systems portfolio of Ethernet Network Emulators are the right products. No longer do you have to perform testing in a vacuum which only provides best-case performance numbers. With VirtualNet Ethernet Network Emulators you can replicate any number of specific network conditions that your solution could face and truly validate that your product or service will perform as expected when deployed. These network emulators are in-line devices making them an easy addition to new or existing test setups and have a comprehensive feature GigaNet Systems network emulators allow you to perform real-world testing on Video over IP. set for performing detailed and dynamic real-world testing of Video over IP network products and systems. Whether pre- or post-deployment, the goal of any testing is to establish and maintain confidence in the product or service being delivered. VirtualNet Ethernet Network Emulators deliver the industry's greatest performance and precision to ensure that your tests, and your results, are repeatable and reliable. Better Testing Better Reliability Better Products Copyright 2012 V1.0 www.giganetsystems.com P a g e 5