TOLLY. No August 2002

Transcription

1 No August 2002 Check Point Software Technologies Ltd. VPN-1 Pro vs. Cisco Systems PIX-535 and NetScreen Technologies NetScreen-500 Competitive Evaluation of Enterprise-Class Internet Security Solutions Premise: When considering the purchase of firewall and VPN devices, network architects and managers need to validate performance characteristics of available security products. With the growth in high-speed Internet access, network managers must contend with the potential performance degradation frequently associated with providing a secure data path. Check Point Software Technologies Ltd. commissioned The Tolly Group to evaluate its VPN-1 Pro, an enterprise-class security solution that integrates firewall and VPN capabilities against two other enterprise products: a Cisco Systems Inc. PIX- 535 and a Net-Screen Technologies Inc. NetScreen-500. T H E TOLLY G R O U P Test Highlights Test Summary outperforms Cisco PIX-535 and NetScreen-500 by 14% and 100% respectively in a singlerule firewall configuration using 64-byte packets outperforms both PIX-535 and NetScreen by 400% each in an IPSec 3DES firewall/vpn configuration using 64-byte packets Check Point s AES outperforms the fastest competitive VPN 3DES configurations by 277% and 294% respectively delivers 57% more throughput while securing "real-world" traffic than NetScreen-500 in a firewall configuration Gigabit Ethernet Firewall Throughput Zero-Loss ( 0.01%) Bidirectional Performance with UDP Traffic Streams Measured by SmartFlow The security devices were subjected to a battery of performance tests that were grouped into three sections: UDP firewall benchmarks, UDP VPN performance, and a real-world mix of Layer 7 application traffic. Tests were conducted in July and August Results Throughput in Mbit/s Single-rule Firewall Bidirectional Throughput Engineers measured the bidirectional packet throughput of each device Devices under test (DUT) Source: The Tolly Group, August 2002 Figure The Tolly Group Page 1

2 configured as a single-rule firewall with two Gigabit Ethernet interfaces. In tests of 64-byte UDP packets traversing a Gigabit Ethernet link, Check Point s VPN-1 Pro achieved 160 Mbit/s of throughput, or 14% greater throughput than the PIX-535 and 100% greater throughput than the NetScreen- 500, which delivered 140 Mbit/s and 80 Mbit/s of throughput, respectively (see Figure 1). In tests of 1,518-byte UDP packets traversing a Gigabit Ethernet link, Check Point s VPN-1 Pro achieved 1.92 Gbit/s of bidirectional throughput, which represents a 9% lead over the PIX-535 and more than three times that offered by the NetScreen-500, which delivered 1.76 Gbit/s and 600 Mbit/s of throughput, respectively. See Throughput in Mbit/s 3DES VPN Throughput Zero-Loss ( 0.01%) Bidirectional Performance with UDP Traffic Streams Measured by SmartFlow 1 Devices under test (DUT) 1 DUTs were peered with a Check Point device. The Check Point device may have limited NetScreen's throughput on the 1,450-byte frame size test. Source: The Tolly Group, August 2002 Figure 2 Check Point Software Technologies leads all vendors in dual-port firewall tests, but its VPN-1 Pro also posted generous firewall throughput results in four-port tests. Check Point Excels in Four-Port Test achieved 240 Mbit/s or 50% more throughput than offered in a dual-port configuration. With 1,518-byte packets, the VPN-1 Pro forwarded 2.84 Gbit/s of traffic. The Tolly Group tested the firewall functionality of the VPN-1 Pro in a four-port configuration with both 64- byte and 1,518-byte packets across Gigabit Ethernet interfaces. With 64-byte packets, the VPN-1 Pro These results show that if the need occurs, Check Point can outfit a device with four Gigabit Ethernet ports that can deliver roughly 50% more traffic than with the dual-port configuration. VPN-1 Pro Four-port Firewall Performance Throughput (Mbit/s) 3,000 2,000 1, byte packets 2,840 1,518-byte packets 2002 The Tolly Group Page 2

3 sidebar on four-port test results, page 2. IPSec Firewall-VPN Tunnel Throughput The Tolly Group tested VPN tunnel throughput using IPSec 3DES and IPSec Advanced Encryption Standard (AES). Engineers benchmarked the bidirectional packet throughput of VPN tunnels with each device paired with the Check Point VPN-1 Pro solution (including a test of matching Check Point devices). Engineers utilized a UDP packet stream for this test. This is a rigorous test of firewall/vpn functionality because the traffic is being filtered rather than just being allowed to pass through unexamined. Engineers created a single tunnel between the device peers and streams of packets were generated from a Spirent SmartBits, including 64-byte packets and 1,450-byte packets. The packet sizes selected represented 64-byte packets, the smallest for Gigabit Ethernet, as well as 1,450-byte packets that approximate the largest packet size that can be sent across the VPN tunnel without fragmentation. Check Point Software Technologies Ltd. VPN-1 Pro Competitive Evaluation VPN Throughput: IPSec 3DES Engineers measured the throughput of a VPN tunnel supporting the 3DES encryption algorithm. MD5 was utilized as the hashing algorithm. In tests of 64-byte Check Point Software Technologies Ltd. VPN-1 Pro Product Specifications* Performance 1,500,000 concurrent firewall connections 40,000 concurrent VPN tunnels 3.1 Gbit/s firewall throughput 800 Mbit/s AES VPN throughput 400 Mbit/s 3DES VPN throughput (with accelerator card) Firewall ActiveX and Java blocking SmartDefense attack blocking Stateful inspection of 150+ applications Security servers Online attack defense updates VPN IPSec/IKE, AES 256/128, DES/3DES, SHA1/MD5 Integrated certificate authority OneClick VPNs Data compression VPN Client Integrated client firewall Security configuration verification Windows domain login Authentication via RADIUS, SecureID, LDAP, TACACS/TACACS+, S/Key, OS/internal password, digital certificates NAT traversal Management One-Click Management Automated client/gateway software distribution Up to 100 virtual systems and VLANs Management server high availability Visual policy management Redundant logging Automated log archival Real-time performance and status monitoring High Availability/Load Sharing Active session fail-over (including VPN) Integrated load sharing QoS Quality of Service (QoS) Bandwidth weights, limits, guarantees DiffServ For more information contact: Check Point Software Technologies Ltd. Phone: (800) URL: *Vendor-supplied information not verified by The Tolly Group 2002 The Tolly Group Page 3

4 packets, the Check Point VPN-1 Pro achieved 100 Mbit/s of throughput, versus 20 Mbit/s of throughput each for the Cisco PIX-535 and the NetScreen-500. AES VPN Throughput Zero-Loss ( 0.01%) Bidirectional Performance with UDP Traffic Streams Measured by SmartFlow 1 For 1,450-byte packets, Check Point s VPN-1 Pro achieved 180 Mbit/s, the NetScreen-500 forwarded the same amount of traffic, and the PIX-535 supported 100 Mbit/s of traffic (see Figure 2). VPN Throughput: IPSec AES Next, The Tolly Group used AES as the encryption algorithm. For 64-byte packets, the achieved 100 Mbit/s of throughput while the NetScreen-500 forwarded just 5 Mbit/s of traffic. NetScreen passed (0.25%) traffic, but not enough to reach the 1% resolution of the data point for the test tool. For this purpose, we are showing that NetScreen did pass some traffic. Cisco does not support AES in the release of the PIX-535 available for testing. For 1,450-byte packets, Check Point VPN-1 Pro supported 500 Mbit/s and the NetScreen-500 passed 20 Mbit/s of traffic. In this test, Check Point proved it is ahead of other rivals in deployment of AES, the National Institute of Standards and Technologies (NIST) FIPS-approved standard for next-generation security by U.S. government organizations. See Figure 3. Throughput in Mbit/s Real-world application performance The final benchmark involved generating real-world traffic by simulating application layer traffic flows with the use of a NetIQ Chariot. The theory behind doing a realworld test is that using application traffic is a much more realistic type of traffic than sending one frame size (64- byte packets) continuously. Therefore, The Tolly Group attempted to find the throughput value for a mixture of packet sizes that approximates the distribution observed for actual Internet traffic. This test encompassed both firewall and VPN test configurations. Devices under test (DUT) 1 DUTs were peered with a Check Point device. Cisco did not participate because the PIX-535 does not support AES. Source: The Tolly Group, August 2002 Figure 3 In the firewall portion of the test, Check Point s VPN-1 Pro forwarded 758 Mbit/s of application traffic, versus 726 Mbit/s for the PIX-535 and 478 Mbit/s for the NetScreen-500. In the tests using 3DES encryption, Check Point s VPN-1 Pro supported 106 Mbit/s versus 69 Mbit/s for the NetScreen- 500 and 66 Mbit/s for the PIX-535. In the AES tests, the Check Point VPN-1 Pro supported traffic at 260 Mbit/s and the NetScreen-500 supported traffic at 10 Mbit/s (see Figure 4). Cisco s PIX-535 does not support AES, thus did not participate in that portion of the test The Tolly Group Page 4

5 Firewall and VPN Throughput Real-World Traffic Mix Throughput in Mbit/s Devices under test (DUT) 1 Cisco did not participate because the PIX-535 does not support AES. Source: The Tolly Group, August 2002 Figure 4 Analysis The consistently demonstrated higher performance than competitive products in both firewall and VPN tests over Gigabit Ethernet connections. On the firewall side, the VPN-1 Pro achieved 15% greater throughput than its nearest competitor in 64-byte tests. In VPN tests using 3DES with firewall functionality enabled, the VPN- 1 Pro demonstrated five times more throughput than both rival products tested. And in VPN tests utilizing the AES algorithm with firewall functionality enabled, the VPN-1 Pro delivered 25X greater throughput than the NetScreen In tests measuring the throughput of application traffic flowing across a VPN connection with an active firewall, the VPN-1 Pro delivered an average 57% more traffic. From a price/performance perspective, the VPN-1 Pro offered a cost-per-megabit that is three times less costly than its nearest rival, the NetScreen- 500 and more than four times less costly than the PIX-535. The cost-per-megabit metric divides the real-world firewall Mbit/s throughput results into the total cost of the device under test to obtain a relative cost-per-megabit of throughput. The configuration as tested, including software, PC and NICs, cost $16,200, or 116% less than the NetScreen-500's configuration price of $34,995 (hardware, NICs and software included) and more than 400% less costly than the Cisco PIX-535 configuration price of $66,000 ($59,000 for hardware, $6,000 for two Gigabit Ethernet fiber NICs and $1,000 for a 3DES license). When those prices are divided by their firewall Mbit/s performance results, the cost per megabit of throughput is $21.37 for the VPN-1 Pro, $73.21 for the NetScreen-500 and $90.91 for the PIX-535. This shows that the VPN-1 Pro holds a 2002 The Tolly Group Page 5

6 performance and a capital investment advantage over the rival products tested. Related Tests The Tolly Group conducted a similar set of firewall/vpn performance benchmarks for NetScreen Technologies, Ltd., which was published in July 2001 (Document ). It is important to note that the performance results of the NetScreen- 500 in this report do not match the performance of the device in the earlier report. In the July 2001 tests, NetScreen tested Version Beta 3 of its software, versus Version 3.1 for the current tests. While the tests may have yielded different results, it is important to note they do not share identical test parameters or configurations. Test Configuration and Methodology IPSec Tunnel Test Bed Configuration For IPSec tunnel performance tests, engineers connected a pair of Check Point Software VPN-1 Pro devices, SecurePlatform NG FP2 with Performance Pack to each other via full-duplex Gigabit Ethernet using SC fiber-optic cable. This software was run on a 2.4-GHz Xeon processor with 1 Gigabyte of RAM and served as a VPN peer with all of the devices under test. The other Check Point Software VPN-1 Pro configured as an IPSec gateway was running on a 2.2-GHz processor with 1 Gigabyte of RAM. For competitive IPSec tunnel performance tests, engineers replaced one of the Check Point devices configured as an IPSec gateway with either a Cisco Systems, Inc. PIX-535, Version 6.1(2) equipped with a PIX VPN Accelerator card or a NetScreen Technologies NetScreen-500, ScreenOS version 3.1. The test devices were connected via Gigabit Ethernet to a Spirent Communications SMB Advanced Multiport Performance/Tester/Analyzer. The SmartBits console ran Spirent SmartFlow 1.5. Each device under test also was connected to an Acterna Domino DA-360 hardware-based network analyzer running DominoNAS software version 3.1 configured for 1000BASE- SX. This device verified packet sizes, utilization and encryption. In all tests, DUTs were tested in a dual-port configuration, except for firewall tests where engineers also tested the VPN-1 Pro in a four-port configuration. Firewall Test Bed Configuration and Methodology For firewall throughput, engineers removed one of the Check Point Software VPN-1 Pro devices; the remaining test bed was the same as the IPSec tunnel test bed. Engineers configured the DUT as a singlerule, allow-all firewall. The Tolly Group engineers measured the percent of traffic the device forwarded when offered 100% of the theoretical maximum load and adjusted the offered load in 1% decrements until zero-loss ( 0.01%) was achieved. SmartBits generated 64-byte and 1,518-byte UDP packets. SmartBits measured the percent of traffic forwarded by the firewall under test. Engineers ran three iterations of each packet size for 60 seconds and results were averaged. IPSec Tunnel Test Methodology IPSec performance tests were conducted with each DUT operating as a VPN/IPSec gateway. All traffic was encrypted using 3DES/MD5 for one series of tests and using AES/MD5 for a second series of tests. Tolly Group engineers tested the DUTs for zero-loss ( 0.01%) throughput. For steady-state, zero-loss, bidirectional packet-per-second throughput tests, engineers measured the packet loss of the DUT when offering 100% of the theoretical maximum load and adjusted the offered load in 1% decrements until zero-loss was achieved. SmartBits generated 64-byte and 1,450-byte UDP packets (1,450-byte packets were used to avoid packet fragmentation). Smart- Bits measured the percent of zero-loss line-rate traffic that traversed the device under test. Engineers ran three iterations of each packet size for 60 seconds and results were averaged. Note: Typically, The Tolly Group tests VPN tunnel throughput with matching pairs of devices on each end of the VPN tunnel. For budget reasons, Check Point requested the use of mismatched units with one 2002 The Tolly Group Page 6

7 of the Check Point devices serving as one end of the VPN tunnel. In the IPSec 3DES test at 1,450 bytes, the use of the Check Point device as a VPN tunnel pair for the NetScreen- 500 may have been a gating factor in overall VPN throughput performance. Layer 7 Application VPN/firewall throughput test For Layer 7 application throughput across a VPN/firewall, engineers connected a pair of Check Point Software VPN-1 Pro devices to each other via full-duplex Gigabit Ethernet using SC fiber-optic cable. Engineers then replaced one of the Check Point devices with either a Cisco Systems PIX-535 or a NetScreen Technologies NetScreen-500. For this series of tests, engineers configured the DUT as a singlerule, allow-all firewall with NAT disabled, and as an IPSec gateway. All traffic was encrypted using 3DES, MD5 for one series of tests and using AES, MD5 for a second series of tests. The test devices connected to an Extreme Networks Alpine This switch supported links to eight PCs configured as NetIQ Chariot endpoints and a connection to a Chariot server console. The following devices ran Chariot Endpoint software: an 800-MHz Intel Pentium 3 with 512 Mbytes of RAM; two 800-MHz Intel Pentium 3 processors with 512 Mbytes of RAM; three 2.4-MHz dual-processor Xeon Pentium 4 with 512 Mbytes of RAM; and two 500-MHz Intel Pentium 3 processors with 128 Mbytes of RAM. All endpoints ran Chariot 4.1 software and Windows 2000, Service Pack 2. A 1-GHz Pentium III processor with 256 Mbytes of RAM served as the Chariot console; the Domino DA-360 console ran Spirent SmartFlow 1.5. The real-world traffic test consisted of massive numbers of concurrent variable sized TCP (HTTP) transactions. The transaction size distribution used in the mix was based on the well known SPECWeb99 benchmark and imitated the HTTP transaction mix recorded on busy real-life enterprise Web servers. The test-bed consisted of multiple pairs of Chariot endpoints, simulating HTTP clients and servers. Each HTTP client performed multiple, variable-sized HTTP- GET transactions with HTTP servers. Engineers ran the test script for duration of two minutes, for three iterations and averaged results. For all performance tests, a Domino DA-360 device verified packet sizes and utilization but was removed prior to testing. Equipment Acquisition and Support The Cisco Systems PIX-535 and the NetScreen Technologies NetScreen-500 were acquired through normal product distribution channels. The Tolly Group contacted executives at the vendor companies and invited them to provide a higher level of support than available through normal channels. Executives from Cisco did not respond; NetScreen accepted the offer. The Tolly Group verified product release levels and shared test configurations with executives from NetScreen in order to give them an opportunity to optimize their device for the tests. Results were shared with NetScreen executives who neither acknowledged nor disputed their accuracy The Tolly Group Page 7

8 VPN Test Bed 1 1 The test bed for the firewall test replicated this scenario except that it used a single firewall instead of two VPN endpoints. Source: The Tolly Group, August 2002 Figure 5 The Tolly Group gratefully acknowledges the providers of test equipment used in this project. Vendor Product Web address Acterna Corp. Domino Internetwork Analyzer Cybex Slimline Commander Extreme Networks Alpine NetIQ Chariot Console Spirent Communications SmartBits SmartFlow Tolly Group Services With more than a decade of testing experience of leadingedge network technologies, The Tolly Group employs time-proven test methodologies and fair testing principles to benchmark products and services with the highest degree of accuracy. Plus, unlike narrowly focused testing shops, The Tolly Group combines its vast technology knowledge with focused marketing services to help clients better position product benchmarks for maximum exposure. The company offers an unparalleled array of reports and services including: Test Summaries, Tolly Verifieds, performance certification programs, educational Webcasts, white paper production, proof-of-concept testing, network planning, industry studies, end-user services, strategic consulting and integrated marketing services. Learn more about The Tolly Group services by calling (732) , or send to info@tolly.com. For info on the Fair Testing Charter, visit: Project Profile Sponsor: Check Point Software Technologies Ltd. Document number: Product class: Enterprise-class Internet security system Products under test: Check Point Software Technologies Ltd. VPN-1 Pro, SecurePlatform NG FP2 Cisco Systems, Inc. PIX 535 Firewall, Version 6.1(2) NetScreen Technologies, Inc. NetScreen-500, ScreenOS, version 3.1 Testing window: July through August 2002 Software status: Generally available For more information on this document, visit our Web site at send to info@tolly.com, or call (732) Internetworking technology is an area of rapid growth and constant change. The Tolly Group conducts engineering-caliber testing in an effort to provide the internetworking industry with valuable information on current products and technology. While great care is taken to assure utmost accuracy, mistakes can occur. In no event shall The Tolly Group be liable for damages of any kind including direct, indirect, special, incidental, and consequential damages which may result from the use of information contained in this document. All trademarks are the property of their respective owners. The Tolly Group doc rev. clk 24 Oct The Tolly Group Page 8