Key findings and conclusions: Lab Testing Summary Report July 2008 Report 080711 Vendor Tested: Product Tested: WinEyeQ Product Category: VoIP Analysis and Management Tools WinEyeQ was rated Best in Test and Most Accurate of VoIP analysis tools tested by Miercom WinEyeQ won overall first place for VoIP Monitor and Analysis Test Tools Reviewed in March 2008 by Miercom for Network World Magazine Designed from the ground up for VoIP network analysis and troubleshooting Found to provide detailed, real-time information managing 1,000 simultaneous calls in lab tests T ouchstone's WinEyeQ was evaluated in early 2008 as part of Miercom s 2008 VoIP Analysis Tool Review, an industry-wide round-up. It was named Best-in-Test as well as Most Accurate Analysis Product. Subsequently, Touchstone engaged Miercom to conduct a more extensive battery of tests on WinEyeQ to further demonstrate the product's VoIP monitoring and analysis capabilities. As discussed in this report, the recent tests bolstered Miercom's initial opinion about WinEyeQ's performance. WinEyeQ consistently proved itself highly sensitive to even the most hidden network anomalies. The product always presented extremely detailed and accurate information, delivering comprehensive and precise statistics about voice quality and network operations. Additionally, WinEyeQ's clean and intuitive interface makes it one of the most easy-to-use VoIP analysis products we've encountered. Most Accurate VoIP Analysis Product 2008 WinEyeQ's well organized screens allow you to easily drill down to network details.
Supplemental Test Bed Diagram Supplemental Testing Shortly after we reviewed WinEyeQ as part of our 2008 VoIP Analysis Tool Review, Touchstone engaged us to perform some intensive supplemental tests of the product. The testing took place in Miercom's lab. WinEyeQ was subjected to a series of tests; during the first batch of ten tests, ten concurrent calls were run with varying degrees of jitter and packet loss. The average of the results for all calls in the group were calculated from the results collected in the aggregator s database. In the second batch of tests, the same parameters were run for a single call to determine how the results matched the findings of the group tests. The purpose of these tests was to determine whether WinEyeQ s Conversational R Factor score accurately reflected degradation of quality in an IP network. To accomplish this, we employed the PacketStorm network impairment generator. The PacketStorm is a pass-through device that allows the user to simulate a wide variety of impairments in a particular path. The WinSIP Initiator placed the calls through the PacketStorm, whose output was directed into a non-switched hub into which a WinEyeQ probe and aggregator were connected. A WinSIP placed in auto answer mode responded to the initiated calls. The signaling and media of the WinSIP initiator was subjected to varying levels of impairments both in jitter and packet loss individually so that we could observe the impact of changes in each irrespective of the other. The impairment chain was set up in the following order: 1: Jitter 2: Latency 3: Packet Loss Copyright 2008 Miercom VoIP Analysis and Management Tools Page 2
The first tests of each group (1 and 11) were baseline tests for the LAN characteristics. All of the control chain elements of the PacketStorm were set to zero. Tests 2 through 4 and 12 through 14 introduced increasing levels of jitter without any packet loss. Tests 5 through 10 and 15 through 20 introduced increasing amounts of packet loss without any artificial jitter. WinEyeQ was using an adaptive jitter buffer similar in configuration to most endpoints. The use of an adaptive jitter buffer is tantamount to emulating a true user experience and eliminating false negative triggers (i.e. the Chicken Little effect ). The results of these tests clearly show that WinEyeQ is a highly accurate tool for assessing voice call quality and that it is very sensitive to even small changes or variations in individual impairments. In our prior analysis of WinEyeQ we found it provided extremely precise voice-quality and the most accurate network-operations statistics of the products we tested. WinEyeQ came closest when determining actual MOS, R-factor and jitter conditions. It was consistently within 0.1 of the "actual" MOS rating for voice quality without the need for the extensive tuning required by other similar products. This chart shows the results of the tests. Values in yellow indicate impairment levels that affect perceptual voice quality. Those in blue are within acceptable ranges when a standard jitter buffer is applied. Copyright 2008 Miercom VoIP Analysis and Management Tools Page 3
Usability While accuracy is the main measure of a VoIP analysis tool's worth, precise data is useless if it is hard to find and use. In that regard, Touchstone WinEyeQ did not disappoint. The product's user interface design is clean and functional, offering network administrators and troubleshooters all they can ask for in terms of easy-to-access information. WinEyeQ can dig deep for packet-level analysis for VoIP and video transmissions. This ability is crucial when attempting to find, confirm and fix VoIP issues. We were able to routinely locate the sources of VoIP problems and quickly make corrections that repaired and maintained call quality. The interface offers call-by-call report cards and stream-quality indexes that not only allowed us to rapidly spot low-quality calls but to also determine why they were happening. During our testing we placed a hundred calls and WinEyeQ enabled us to effortlessly isolate one that was of poor quality. On the other hand, WinEyeQ proved able to deal with massive amounts of data. It analyzed and provided detailed reports on up to 1,000 simultaneous calls. WinEyeQ's alarms were reliably accurate and nearly instantaneous. In fact, its alarm system was the fastest of the VoIP analysis products we tested and it did not compromise quality for speed. This ability to instantly alert administrators to VoIP network issues allows for faster responses, a crucial feature since good voice communication is the backbone of many companies. For our testing, WinEyeQ was installed on a span port on a managed switch. This would be a common installation for a small company. We reviewed the WinEyeQ Professional version, designed to operate either as a stand-alone analysis tool or as a distributed probe within a broad VoIP management system. Bigger businesses should install WinEyeQ into a mirrored port or use a line tap. These methods will provide a higher level of accuracy. When WinEyeQ is deployed in a distributed fashion, as a probe within Touchstone's Net Observer distributed system, its Web-based user interface plays a vital roll. Through the GUI you can poll the remote probes and push to them software updates as needed. WinEyeQ provides a very exhaustive list of metrics that detail just about every measureable VoIP factor. It can measure and report on about 650 metrics per call. The product provides analysis of most standards-based voice and video protocols, including SIP, H.323, Media Gateway Control Protocol (MGCP) and Megaco. It also analyzes Real-time Transport Protocol (RTP) audio and video streams, Real-time Transport Control (RTCP) and RTCP XR calls. We were impressed by WinEyeQ's ability to provide us with many different types of system reports. These reports included call, error, watch, session, alert, alarm and endpoint summaries. Unlike some other VoIP analysis tools, WinEyeQ will not have network administrators scrambling in vain due to false alarms. Versions and Other Features WinEyeQ's excellent performance likely can be attributed to the fact that it was designed, from the ground up, for VoIP analysis. Touchstone sells a number of different configurations for WinEyeQ including carriergrade, customer-premises distributed systems and stand-alone analysis tools. Copyright 2008 Miercom VoIP Analysis and Management Tools Page 3
Miercom Rated Best in Test and Most Accurate Analysis Based on Miercom s detailed testing of the Touchstone WinEyeQ VoIP analysis product and validation of its capabilities, operation and features, Miercom attests the product is Rated Best. The award is in accordance with the Rated Best Testing Program of Miercom, effective for one year from test certification or until the next product release. The Rated Best program recognizes the best products in a series of competitive reviews and the award is given for specific category excellence. Best VoIP Analysis Product 2008 Touchstone Technologies 228 North York Road Suites C & D Hatboro, PA 19040 215.672.6550 215.672.6551 (fax) About Miercom s Product Testing Services With hundreds of its product-comparison analyses published over the years in such leading network trade periodicals as Business Communications Review, Network World and VoIP Magazine, Miercom is a leading, independent product test center. Founded in 1988, the company has pioneered the comparative assessment of networking hardware and software, having developed methodologies for testing products from enterprise class VoIP gateways and IP PBX s to carrier grade switching equipment and gateway products. Miercom s private test services include competitive product analyses, as well as individual product evaluations. Miercom features comprehensive certification and test programs. They include Certified Reliable, Certified Secure and Certified Green. Products may also be evaluated under the NetWORKS As Advertised program, in which networking-related products must pass a comprehensive, independent assessment of the products usability and performance as compared to advertised specifications. 609-490-0200 fax 609-490-0610 www.miercom.com Report 080711 Copyright 2008 Miercom VoIP Analysis and Management Tools Page 4