ENTERPRISE WIRELESS LAN COMPETITIVE TESTING Performance Results of Industry s Top Seven g Wireless LAN Systems for Mid-Sized Enterprises

Transcription

1 ENTERPRISE WIRELESS LAN COMPETITIVE TESTING Performance Results of Industry s Top Seven g Wireless LAN Systems for Mid-Sized Enterprises Executive Summary Jones-Petrick and Associates (JPA) recently conducted a technical evaluation for Wireless of the latest generation of enterprise wireless LAN (WLAN) controllers and access points in a real world office environment. While the industry has converged on a centralized WLAN model to deploy, manage and secure Wi-Fi networks within large corporations, innovations have mostly focused on the security and manageability aspects of the WLAN residing on intelligent WLAN controllers. Little has been done to solve the fundamental issue of sustaining wireless connectivity and performance in the face of interference, increasing distance and obstructions. As Wi-Fi becomes more prevalent and WLAN controller technologies mature, the focus is now on fixing the neglected RF component. The biggest complaints about WLANs today are no longer about management or security rather, they center around RF problems that manifest weak Wi-Fi coverage, unstable signals and performance fluctuations. Without the ability to maintain reliable wireless connections and predictable performance, even the best WLAN security and management system is moot. We found the test results to be impressive. The testing revealed that WLAN systems that focused only on security and management and lacking RF optimizations were unable to provide consistent performance levels across all locations. Conversely, systems that implement comprehensive RF optimizations demonstrated markedly better overall performance across all the test cases. The RF-optimized WLAN systems were also better able to adapt to real-time changes in the wireless environment such as interference, and were able to deliver ubiquitous Wi-Fi coverage with consistent performance mbps 1.66 TEST HIGHLIGHTS The ZoneFlex system continually demonstrated higher TCP throughput at the farthest and most challenging locations At the most challenging location, the ZoneFlex delivered 1.66 mbps of aggregate TCP throughput to 20 clients, over 2X that of the nearest competitor (see Chart 1 below) In single g client testing, the ZoneFlex exhibited the highest average TCP downstream throughput over all competitive products (with and without interference) In single g client testing with interference, ZoneFlex performed 2.x faster than the competitive average In voice over Wi-Fi tests, the ZoneFlex system demonstrated the highest quality MOS (4.33) and R-Value (90.47) scores across all locations over all competitive systems CHART 1 DATA THROUGHPUT AT DISTANCE g Centrino clients, aggregate performance as measured by Most Difficult (10 feet), F January,2008

2 In all cases, as the distance between the Wi-Fi client and AP increased, performance decreased, some drastically. This was exacerbated by interference. Only the Wireless ZoneFlex Smart WLAN system was able to provide reasonably consistent performance across all locations, even under interference for voice, data and mixed traffic. attributes this to technical advances with their smart antenna technology embedded in its ZoneFlex WLAN system. The ZoneFlex system ranked at or near the top in all the JPA tests. While the ZoneFlex performance was on par with all other systems in short range tests, at the hardest-to-reach locations, the ZoneFlex system delivered five times the throughput over the worst performing system and twice the throughput of its nearest competitor. It also delivered the highest voice quality measured by Mean Opinion Score (MOS) and exhibited the lowest average jitter across all test locations. In the most difficult test, the ZoneFlex system delivered an aggregate TCP downstream throughput of 1.7 mbps to 20 Centrino clients at a distance of over 10 feet, through five walls. The nearest competitive vendor delivered 7 mbps (see Chart 1) in the same test. While most systems fared well at short distances, consistent performance across the coverage area and the ability to adapt to the environment variables in real time are what interest enterprise users the most. Note that to remain completely independent, JPA worked with all vendors being tested to ensure technical accuracy. Testing Objectives The WLAN testing measured performance over a large number of dissimilar locations among the industry s top seven enterprise wireless LAN systems in a typical office environment, using the latest commercially available hardware and software from each vendor. TCP throughput with a single client and multiple clients in disperse locations, with and without interference, were measured. In addition, JPA conducted a VoIP-over-WiFi test measuring the average mean opinion scores (MOS) and R-Values over nineteen voice connections terminated at different locations. The Testers The competitive wireless LAN (WLAN) testing was performed by Al Petrick, principal at Jones-Petrick and Associates (JPA). With over 2 years of experience and widely considered an authority on Wi-Fi, Mr. Petrick is a frequent writer on WLAN technology, is the vice chairman of the IEEE working group and co-author of the popular IEEE Handbook, A Designer s Companion. Testing Performed 1) TCP downlink performance on a single client in six different locations with four client rotations with and without interference introduced. 2) Aggregate TCP downlink performance for multiple clients (20) at varying distances with and without interference introduced. 3) Voice quality of traffic to 20 clients with background data traffic. The three tests selected (see Test Configuration, Tools and Methodology section), focused on some of the most common enterprise concerns such as: Data and voice applications Dense client population in disperse locations Interference in the environment

3 TEST CONFIGURATION, TOOLS AND METHODOLOGY The testing was conducted in a real-world office space in Fremont, California using the latest vendor equipment and standard test tools such as Ixia s and Iperf. The single story office complex occupied 18,726 square feet with a layout typical of a medium to large business. The floor plan included private offices, laboratories and warehouse space along with open cubicals. This unoccupied space was selected because it reflected a common enterprise environment and was ideal for objective testing as there were no Wi-Fi networks or other RF interference within the 2.4GHz band. The construction of the interior office and laboratory walls provided a challenge for RF signal penetration. The 10 high interior walls were constructed of metal studs and single layer /8 dry wall material. Parts of the office and laboratory walls were made of ½ lightly tinted plate glass. The ceiling was a standard metal track acoustical ceiling. The 10 office doors were made of solid wood and the laboratory doors were constructed of solid plate steel. The WLAN testing included three basic tests focusing on TCP throughput and voice quality across a number of dissimilar locations over standard g. The tests performed were: 1) TCP downlink performance to a single client, in six different locations, with four client rotations per location, with and without interference introduced. 2) Aggregate TCP downlink performance to multiple dispersed clients (20), at varying AP locations. 3) Voice quality on 19 VoIP-over-WiFi clients and background data traffic, at varying AP locations. Each vendor was provided with the test methodology and was given the opportunity to alter the configuration of their system to accommodate any performance irregularities caused by the testbed. The placement of the APs, client notebooks, and interference source are shown in the test Figures 2, 4 and. The interference source, simulated using a single Linksys AP and two Centrino clients running Iperf, was positioned at the rear entrance of the building. It simulated the existence of neighboring g networks. Testing Environment 18,726 square feet Fremont, California No external Wi-Fi networks Conventional office space Metal, steel and glass walls Test Tools and Details Ixia s and Iperf All testing on Channel 1 TX power set to +20dBm (100mW) A total of 20 notebook clients were installed in various locations around the premises (see Figure 4). The notebooks were equipped with a mix of client adaptors: g, a/g, and n (Draft 2.0) based on radio chipsets from Atheros, Broadcom and Intel. All client adaptors were configured for 2.4GHz g only with security disabled. Each was assigned a static IP address ( ).

4 The controller and AP configurations were basic with advanced management and security features disabled. All devices-under-test (DUTs) were upgraded with the latest version of drivers provided by their respective maker. The APs were constrained to operate on channel 1 (2.412MHz) and their transmit power was limited to 100mW. Within the 2.4 GHz frequency, channel 1 is the most challenging channel in terms of guaranteed performance because of RF requirements for meeting the out-of band emission regulations at the edges within this band. Ixia s release.0 and its TCP data and VoIP test scripts were used as the test platformon the client notebooks. is an industry standard for benchmarking performance of WLAN systems. Each vendor, with one exception, provided input on how to configure for mixed-mode voice and data TCP downstream data tests. To ensure consistent environmental conditions across the test, the 2.4GHz spectrum was scanned for RF interference and other Wi-Fi radio sources using WiSpy/Wireshark (a PC based spectrum analyzer) version and NetStumber v0.4.0 prior to each test. JPA reviewed the details of the test plan, test platform, configurations and the results with each competitive vendor. Only one vendor did not respond to the requests. TEST RESULTS Test One Single Client TCP Downstream Throughput Results FIGURE 1: Test One Testbed TCP downlink to single client with and without interference Console CHART 2 8 Vendor Controllers Interference simulated with Iperf g 2.4 GHz AP Client endpoint DATA THROUGHPUT WITH INTERFERENCE Single g Centrino client, performance as measured by mbps The first test quantified TCP data throughput versus distance using the platform configuration on a single g Intel Centrino client as shown in Figure 1. The physical topology is shown in Figure 2 (next page). For this test, the APs were positioned in location A and the client was placed in the farthest and most difficult locations: D, F and G (see Figure 2, next page). Typical Wi-Fi devices exhibit different signal strengths and throughputs when their orientation is changed. To protect against these anomolies, JPA rotated the client four times (0, 90,180,270 ) at each test location. The average TCP downstream throughput was reported by for each client rotation. The test was run for 30 seconds per rotation and the throughputs were averaged D, 97 feet

5 Test results demonstrated that at a moderate range without obstruction, (location D at 97 feet and through two walls), was able to deliver 20.2 mbps and the ZoneFlex delivered 17.7 mbps. FIGURE 2: Test One Topology TCP downlink to single client with and without interference When the client was moved further away from the AP, to locations F and G, (10 and 17 feet through two and four walls, respectively), only the ZoneFlex was able to sustain its performance, at above 1 mbps, while the others declined by 0% or more relative to their throughputs at location D (see Chart 3). G AP location Client location(s) Interference The same test was run with interference introduced into the testbed. The interference source was 148 feet away from the client at location D and the AP remained at location A (97 feet away). This time the ZoneFlex system delivered 6.8 mbps while the nearest competitive system,, delivered 4.11 mbps and the worst performing system barely delivered 1 Mbps (see Chart 2, previous page). Interference was generated by two Dell notebooks with Centrino g clients running Iperf through a Linksys WRT4G v8 AP on channel 1. The RF interference seen by the client was -8 dbm and the RF signal from the ZoneFlex AP in location A was -66 dbm. JPA verified the levels using WiSPY/Wireshark and NetStumber. A CHART 3 2 mbps 17 feet 97 feet 10 feet TCP THROUGHPUT VS. DISTANCE Single g Centrino client, performance measured by, no interference F D D 97 feet F 10 feet G 17 feet

6 Test Two Multiple Clients TCP Throughput Results FIGURE 4: Test Two Topology TCP downlink to 20 clients The second test benchmarked aggregate TCP downstream throughput versus distance on 20 clients dispersed at various locations. Twenty client notebooks were equipped with a mix of g client adaptors running the test platform as shown below in Figure 3. Figure 4 shows the physical topology. The clients placements were fixed throughout this test. AP locations Static client location(s) The APs were positioned in locations A, then C and F. TCP downstream packet length was set to 100 bytes. All tests ran for one minute. The results at location A (in close proximity to 12 of the 20 clients) showed delivering the best performance at mbps, while the ZoneFlex delivered mbps. F Again JPA found that even at the most challenging location (location F), the ZoneFlex was able to deliver 1.66 mbps aggregate throughput while performance on all the other DUTs declined drastically, delivering less than half the throughput (see Chart 1, first page). A C This test conclusively showed that the ZoneFlex system exhibited minimal degradation in throughput across all locations, with multiple clients, as compared to other products tested (see Chart 4). FIGURE 3: Test Two Testbed TCP downlink to 20 clients CHART 4 TCP THROUGHPUT VS. DISTANCE g Centrino clients, performance measured by 2 mbps A C F Console Vendor Controllers g 2.4 GHz APs Client End Points

7 Test Three VoIP and Data Mixed-Mode Throughput Results FIGURE : Test Three Topology VoIP over Wi-Fi performance testing The third test benchmarked mixed voice and data performance versus distance using 20 notebooks (19 for voice plus one for data) dispersed at different locations (see Figures and 6). AP locations Static client location(s) JPA used the standard G.711 CODEC voice scripts and configured the notebooks to simulate 19 VoIP clients, with one notebook streaming background TCP traffic. APs were positioned in locations A, then B, C, E, and F. All of the APs tested had quality of service (QoS) priority queues enabled. The APs came with different levels of QoS configuration complexity. JPA found that the APs worked best when configured with two SSIDs, one for voice and the other for data. E F The ZoneFlex system required no QoS configuration as it uses advanced heuristics to characterize voice and video traffic. The ZoneFlex AP supports four traffic queues on a per client basis. A B C The scripts on the clients ran for three minutes for each AP test location. With some of the WLAN products, there was difficulty getting all the clients to respond at location F. The results for this test benchmarked the signal quality for a large number of voice calls. Mean Opinion Score (MOS) and R-Value are two metrics commonly used to determine voice quality, both of which are reported calculations from. FIGURE 6: Test Three Testbed Mixed mode voice and data throughput vs. distance without inteference JPA found that the ZoneFlex system consistently delivered the highest average MOS of 4.33 across all locations, and a corresponding average R-Value of (see Chart, next page). Even at the most challenging AP location (location F), the average MOS and R-Value did not degrade from location to location. The average jitter at location F was 1.32 msec, and the end-to-end delay was 0 msec. All other DUTs reported lower and fluctuating MOSs and R-Values across the five AP locations (See Chart on next page). As an industry standard, a MOS above 4.0 and an R-Value above 8 are considered excellent voice quality. JPA was extremely impressed with the voice quality performance that the ZoneFlex system exhibited especially at the furthest locations. Console Vendor Controllers g 2.4 GHz APs Client End Points 7

8 Wireless Outperformed Others. Why? The ZoneFlex system utilizes a unique antenna subsystem it calls BeamFlex. This software controlled smart antenna array is used to focus RF energy toward each client and steer signals around interference in real time. The ZoneFlex AP employs sophisticated software algorithms to select the best antenna configurations for a given client on a per packet basis. The AP has over 4000 unique antenna configurations at its disposal. CHART B VOIP OVER WI-FI PERFORMANCE MOS Values, VoIP traffic to 19 clients with background data Mean Opinion Score (MOS) A C E F This ensures high reliability, predictable performance and extended range. Consequently, while the ZoneFlex system performs on par with other systems at short range, at farther and more challenging locations, the ZoneFlex system excels. further notes that the self optimizing smart antenna array eliminates AP placement related variability and minimizes adjacent co-channel interference while the ZoneDirector controls dynamic channel assignment and power management. No RF experts required Smart antenna array eliminates AP placement issues and minimizes adjacent AP interference while ZoneDirector controls dynamic channel assignment and power management. Breakthrough user performance and density Advanced antenna array and precision QoS software delivers 20 concurrent voice call, 0 simultaneous data users or 20 mbps of sustained video quality throughput per access point. Extended coverage means fewer APs Directional, high-gain antennas dynamically combined to give hotspot operators and enterprises up to two to four times the coverage over typical consumer and enterprise APs. Optimized for voice over Wi-Fi Specialized antenna design, dynamic signal path selection and precision QoS software optimize delay-sensitive, high-density Vo-Fi applications. Power Save (UAPSD) maximizes handset battery life.

9 In Summary Wireless LAN systems must evolve to meet the changing needs of a more demanding user base equipped with more sophisticated devices and Wi-Fi applications. While centralized security and management are important, providing reliable signal coverage everywhere, all the time, is the key issue facing middle market enterprises today. At short distances, there was little performance differentiation between many of the competitive WLAN systems. However, the JPA testing found the ZoneFlex Smart WLAN solution to perform better than all leading alternatives tested in providing extended range, consistent performance across locations, support for multimedia and the ability to automatically adapt to environmental changes caused by interference and other obstacles. Jones Petrick & Associates Treetop Ct Orlando, Florida info@jpasoc.com The information and analysis contained in this document are believed to be correct. Jones-Petrick and Associates assumes no liability for any inaccuracies that may be contained in this document. Jones-Petrick and Associates reserves the right to make revisions to this document with out notice at any time Jones-Petrick and Associates, LLC. All Rights Reserved. Doc Number: