Fortinet Firewall Validation FortiGate-3810D

Transcription

1 Spirent Professional Services Final Report Spirent Communications Validation

2 Spirent Professional Services Final Report Spirent Communications 1325 Borregas Avenue Sunnyvale, CA USA Web: Americas SPIRENT Europe and the Middle East +44 (0) Asia and the Pacific Spirent. All Rights Reserved. All of the company names and/or brand names and/or product names referred to in this document, in particular, the name Spirent and its logo device, are either registered trademarks or trademarks of Spirent plc and its subsidiaries, pending registration in accordance with relevant national laws. All other registered trademarks or trademarks are the property of their respective owners. The information contained in this document is subject to change without notice and does not represent a commitment on the part of Spirent. The information in this document is believed to be accurate and reliable; however, Spirent assumes no responsibility or liability for any errors or inaccuracies that may appear in the document. About Fortinet Fortinet (NASDAQ: FTNT) protects the most valuable assets of some of the largest enterprise, service provider and government organizations across the globe. The company s fast, secure and global cyber security solutions provide broad, high-performance protection against dynamic security threats while simplifying the IT infrastructure. They are strengthened by the industry s highest level of threat research, intelligence and analytics. Unlike pure-play network security providers, Fortinet can solve organizations most important security challenges, whether in networked, application or mobile environments -- be it virtualized/cloud or physical. More than 200,000 customers worldwide, including some of the largest and most complex organizations, trust Fortinet to protect their brands. Learn more at the Fortinet Blog or FortiGuard Labs. Copyright 2015 Fortinet, Inc. All rights reserved. The symbols and denote respectively federally registered trademarks and unregistered trademarks of Fortinet, Inc., its subsidiaries and affiliates. Fortinet s trademarks include, but are not limited to, the following: Fortinet, FortiGate, FortiGuard, FortiManager, FortiMail, FortiClient, FortiCare, FortiAnalyzer, FortiReporter, FortiOS, FortiASIC, FortiWiFi, FortiSwitch, FortiVoIP, FortiBIOS, FortiLog, FortiResponse, FortiCarrier, FortiScan, FortiAP, FortiDB, FortiVoice and FortiWeb. Other trademarks belong to their respective owners. 2 spirent.com

3 Table of Contents Executive Summary Engagement goals Test Results Highlights Test bed topologies Test case scenarios Test main results General Comments Detailed results Max IPv4 Connection Rate Max IPv4 Concurrent Sessions Max IPv4 HTTP Throughput Max IPv6 Connection Rate Max IPv6 Concurrent Sessions Max IPv6 Throughput Max UDP Throughput without packet loss for IPv Max UDP Throughput without packet loss for IPv Spirent equipment used in this test spirent.com 3

4 Spirent Professional Services Final Report Executive Summary Fortinet s is a High Performance Datacenter firewall, providing six 100Gbps Ethernet interfaces (6x100GbE) in a 3U appliance. It is the first 100G appliance form factor to appear in the firewall market. Fortinet asked Spirent to test & measure the performance of the appliance using the Avalanche and Spirent Test Center load generators, both at the layer 3 and layer 7 of the OSI model. One of the key metrics of the test session was to measure the impact if any of IPv6 on the performances when compared to IPv4. The version 6 of Fortinet s Network Processor (NP), which introduces the acceleration of IPv6 traffic, has performed well with no performance differences between IPv4 and IPv6. Such requirement is common in today s Datacenters. The 100GbE interfaces were tested both directly (load generator directly connected) and indirectly (through an interconnection switch) and behaved as expected. In order to perform the test of the, we applied RFC 3511 and RFC 2544 test methodologies. Engagement goals The engagement goals were threefold: Push the device to its maximum performances Validate the behavior under high load Validate that IPv6 test traffic caused no degradation of performance when compared to the IPv4 test traffic. Test Results Highlights Test IPV4 IPV6 HTTP CPS HTTP BW 290 Gbps 290 Gbps UDP BW Gbps Gbps Device is stable under sustained high load. No degradation of performance in IPv6 vs IPv4. 4 spirent.com

5 POWER RESET TEMP FAN LINK STATUS POWER 1 2 EXTERNAL TIME REFERENCE SERIAL DCE 1PPS 10MHZ MASTER STATUS SPT SYNC BITS IN OUT IN ETHERNET 10/100 Mbps LINK/ACT FDX 1000 Mbps CONSOLE SERIAL DCE 1 2 Test bed topologies Fortigate 3810D 6x100GbE 4x100GbE SPT-3U Figure 1: Layer 2-3 Test topology spirent.com 5

6 Communications Mgmt Port DO NOT REMOVE Spirent Professional Services Final Report Fortigate 3810D 6x100GbE 4-60x 10GbE Eth8 Eth9 Eth0 Eth1 Eth2 Eth3 Eth4 Eth5 Eth6 Eth7 Avalanche/ 3100 Figure 2: Layer 4-7 Test topology Avalanche C100 GT x8 6 spirent.com

7 Test case scenarios All of the test cases below were completed using realistic protocol behavior: no Piggyback, Delayed ACK & Congestion Control enabled and Application Layer (HTTP) payload for every test. 1. Max Connection Setup Rate IPv4: CPS (Connections per Second). Maximum of new TCP (Layer 4) connections the device can handle. 2. Max Concurrent Connections IPv4: CCS (concurrent sessions). Maximum concurrent TCP connections the device can handle. 3. Max HTTP Transfer Rate IPv4: BW (bandwidth). Maximum Layer 7 bandwidth the device can handle. 4. Max Connection Setup Rate IPv6: CPS (Connections per Second). Maximum of new TCP (Layer 4) connections the device can handle using IPv6. 5. Max Concurrent Connections IPv6: CCS (concurrent sessions). Maximum concurrent TCP connections the device can handle using IPv6. 6. Max HTTP Transfer Rate IPv6: BW (bandwidth). Maximum Layer 7 bandwidth the device can handle using IPv6. 7. Max UDP Throughput Test for IPv4: Maximum UDP bandwidth with no loss for different frame sizes using IPv4 8. Max UDP Throughput Test for IPv6: Maximum UDP bandwidth with no loss for different frame sizes using IPv6. spirent.com 7

8 Spirent Professional Services Final Report Test main results RFC 3511 IPV4 IPV6 HTTP CPS HTTP CCS HTTP BW 290 Gbps 290 Gbps RFC 2544 IPV4 IPV6 UDP BW (64B) Gbps Gbps (86B) UDP BW IPv4 (128B) Gbps Gbps UDP BW IPv4 (256B) Gbps Gbps UDP BW IPv4 (512B) Gbps Gbps UDP BW IPv4 (1518B) Gbps Gbps UDP BW IPv4 (9216B) Gbps Gbps 8 spirent.com

9 General Comments All the L2-3 test cases used Spirent Test Center version 4.45 on an 11U blade chassis. The test modules used were the MX-100G-F2 and the MX-100G-P2 modules. The test ports of the modules were directly connected to the FortiGate- 3810D Firewall. All the L4-7 test cases used Spirent Avalanche version We used up to eight C100 Appliances with 10GbE interfaces. A high-performance, low-latency switch was used in order to convert the 10GbE traffic to 100GbE. The Device Under Test (DUT) performed well. The Spirent test equipment was also configured in order to load it beyond documented safe parameters for Concurrent Sessions the DUT entered in what Fortinet calls this the conserve mode and it worked as designed to prevent the DUT from crashing. Even under very high loads (99% CPU or Memory usage) the traffic was switched between client and servers in a satisfactory fashion: no transport- or application-layer errors. A special note can be made for the device s excellent TCP timings, notably a tiny difference between the Average Time to SYN/ACK and the Average Round Trip Time. This is explained by the fact that each SYN packet is not accelerated (it must be forwarded by the Network Processor to the CPU, hence not accelerated since it follows a slow path ) while the rest of the TCP segment is accelerated. The difference between the two values is therefore the average two-way latency value of the Network Processors (but also includes the Average Time To SYN/ACK so the actual value is a bit lower). spirent.com 9

10 Spirent Professional Services Final Report Detailed results Max IPv4 Connection Rate Identification ID Description Runtime Max IPv4 Connection Rate To determine the maximum TCP connection establishment rate through or with the DUT/SUT. This test is intended to find the maximum rate the DUT/SUT can update its connection table. The RFC 3511 specifies that HTTP 1.1 or higher MUST be used, and both the servers and clients. MUST have the same HTTP version level. The RFC does not, however, specify how many HTTP GET messages each simulated user must retrieve, or how many HTTP GET can be sent within one TCP connection. In order to maximize the performances of the Avalanche, 10 HTTP level 1 GETs will be retrieved by each SimUser and each TCP connection will accept one HTTP Transaction maximum. The effect of this will be that each SimUser will generate 64 TCP connections sequentially. 5 minutes Execution Parameters IP version Transport/Application HTTP object size Traffic setup Key measurements Connectivity DUT configuration IPv4 TCP/HTTP 1 byte 762 client IP addresses (three /24 subnets 24 servers IP addresses TCP response time, application response time, TCP connections (per second, concurrent) 6x10GbE interfaces (Spirent Avalanche), 6x100GbE (Fortigate), SFP+ Multimode L4 Firewalling, Virtual Domain, 6x IPv4 Virtual Router Points of failure Response time TCP/HTTP High value response time is a point of failure. Any unsuccessful TCP Connection or HTTP Transaction error is a point of failure Execution Report Date Testers (initials) AC, VJ Test Run Result OK 10 spirent.com

11 Results Figure 3: TCP Connection Rate Figure 4: TCP Time to First Byte, TCP Time to SYN/ACK spirent.com 11

12 Spirent Professional Services Final Report Figure 5: HTTP Response Time Other Metrics Total Attempted HTTP Transactions 172,264,576 Total Successful HTTP Transactions 172,264,576 Total Unsuccessful HTTP Transactions 0 Total Attempted TCP Connections 172,264,576 Total Successful TCP Connections 172,264,576 Total Unsuccessful TCP Connections 0 Average Time to TCP SYN/ACK ms Average Round Trip Time ms 12 spirent.com

13 Max IPv4 Concurrent Sessions Identification ID Description Max IPv4 Concurrent Sessions The objective is to determine the maximum number of concurrent TCP connections supported through or with the DUT/SUT using IPv4 traffic. This test is intended to find the maximum number of entries the DUT/SUT can store in its connection table. For this test, HTTP 1.1 MUST be used, on both the client and server side. Runtime The goal is to have Simulated Users open several concurrent connections to different HTTP servers and maintain them open for the duration of the test. Once the maximum load is reached and maintained for 60 seconds, the connections will begin to be torn down. 5 minutes Execution Parameters IP version Transport/Application HTTP object size Traffic setup Key measurements Connectivity DUT configuration IPv4 TCP/HTTP 64 bytes 1524 clients IP addresses (six /24 subnets) 24 servers TCP response time, application response time, TCP connections (per second, concurrent) 12x10GbE interfaces (Spirent Avalanche), 6x100GbE (Fortigate), SFP+ Multimode L4 Firewalling, Virtual Domain, 12x IPv4 Virtual Router Points of failure Response time Transactions High value response time (value to be determined with the Customer) is a point of failure Any unsuccessful TCP Connection or HTTP Transaction error is a point of failure Execution Report Date Testers (initials) AC, VJ Test Run Result OK spirent.com 13

14 Spirent Professional Services Final Report Results Figure 6: Established TCP Connections Figure 7: TCP Time to First Byte, TCP Time to SYN/ACK 14 spirent.com

15 Comments Figure 8: TCP Connection Rate The TCP Establishment Rate (CPS) was purposely high in this test. Not only because it takes a lot of time to reach 100 million established connections, but also to illustrate that the device could reach its maximum concurrent sessions while also under heavy new connection rate. On the Figure 2 graph we can see two small spikes: these happened when the load generator started establishing and tearing down connections, respectively. The spikes look big because of the large scale and do not exceed 4.5 milliseconds. Other Metrics Total Attempted HTTP Transactions 187,127 Total Successful HTTP Transactions 187,127 Total Unsuccessful HTTP Transactions 0 Total Attempted TCP Connections 100,502,136 Total Successful TCP Connections 100,502,136 Total Unsuccessful TCP Connections 0 Average Time to TCP SYN/ACK ms Average Round Trip Time ms spirent.com 15

16 Spirent Professional Services Final Report Max IPv4 HTTP Throughput Identification ID Description Runtime Max IPv4 HTTP Throughput The goal here is to measure the goodput a device can handle, using layer 7 data. In that case we use HTTP for several reasons: the flexibility of the protocol, the ability to very easily change the page size and the high performance Avalanche can provide. Note that the given bandwidth value is measured at the layer 2 and therefore represents the overall throughput. This is done intentionally as it represents the actual load on the network, including all protocol headers. 5 minutes Execution Parameters IP version Transport/Application HTTP object size Traffic setup Key measurements Connectivity DUT configuration IPv4 TCP/HTTP 100,000 bytes 6096 clients IP addresses (twenty four /24 subnets) 48 servers TCP response time, application response time, TCP connections (per second, concurrent) 48x10GbE interfaces (Spirent Avalanche), 6x100GbE (Fortigate), SFP+ Multimode L4 Firewalling, Virtual Domain, 48x IPv4 Virtual Router Points of failure Response time Transactions High value response time (value to be determined with the Customer) is a point of failure Any unsuccessful TCP Connection or HTTP Transaction error is a point of failure Execution Report Date Testers (initials) AC, VJ Test Run Result OK 16 spirent.com

17 Results Figure 9: HTTP Throughput Figure 10: TCP Time to First Byte, TCP Time to SYN/ACK spirent.com 17

18 Spirent Professional Services Final Report Comments Figure 11: HTTP Response Time The device reached a maximum of 292 Gbps of throughput but even under these 100% load conditions, both the TCP and HTTP response time were below a few milliseconds. Particularly impressive was the 0.5 ms of Time to SYN/ACK. Other Metrics Total Attempted HTTP Transactions 57,066,260 Total Successful HTTP Transactions 57,066,260 Total Unsuccessful HTTP Transactions 0 Total Attempted TCP Connections 11,413,252 Total Successful TCP Connections 11,413,252 Total Unsuccessful TCP Connections 0 Average Time to TCP SYN/ACK Average Round Trip Time spirent.com

19 Max IPv6 Connection Rate Identification ID Description Runtime Max IPv6 Connection Rate The same methodology as the one described in section 5.1 was used here. The only difference was the IP version changed from IPv4 to IPv6 and the amount of Avalanche ports. We needed to include more Avalanche ports in order to reach the performances of the DUT. 5 minutes Execution Parameters IP version Transport/Application HTTP object size Traffic setup Key measurements Connectivity DUT configuration IPv6 TCP/HTTP 1 byte 1016 client IP addresses (four /24 subnets) 32 servers IP addresses TCP response time, application response time, TCP connections (per second, concurrent) 8x10GbE interfaces (Spirent Avalanche), 6x100GbE (Fortigate), SFP+ Multimode L4 Firewalling, Virtual Domain, 6x IPv4 Virtual Router Points of failure Response time TCP/HTTP High value response time is a point of failure. Any unsuccessful TCP Connection or HTTP Transaction error is a point of failure Execution Report Date Testers (initials) AC, VJ Test Run Result OK spirent.com 19

20 Spirent Professional Services Final Report Results Figure 12: TCP Connection Rate Figure 13: TCP Time to First Byte, TCP Time to SYN/ACK 20 spirent.com

21 Figure 14: HTTP Response Time Other Metrics Total Attempted HTTP Transactions 189,870,560 Total Successful HTTP Transactions 189,870,560 Total Unsuccessful HTTP Transactions 0 Total Attempted TCP Connections 189,870,560 Total Successful TCP Connections 189,870,560 Total Unsuccessful TCP Connections 0 Average Time to TCP SYN/ACK Average Round Trip Time spirent.com 21

22 Spirent Professional Services Final Report Max IPv6 Concurrent Sessions Identification ID Description Runtime Max IPv6 Concurrent Sessions The same methodology as the one described in section 5.2 was used here. The only difference was the IP version changed from IPv4 to IPv6 and the amount of Avalanche ports. We needed to include more Avalanche ports in order to reach the performances of the DUT. 5 minutes Execution Parameters IP version Transport/Application HTTP object size Traffic setup Key measurements Connectivity DUT configuration IPv6 TCP/HTTP 64 bytes 1016 client IP addresses (four /24 subnets) 32 servers IP addresses TCP response time, application response time, TCP connections (per second, concurrent) 8x10GbE interfaces (Spirent Avalanche), 6x100GbE (Fortigate), SFP+ Multimode L4 Firewalling, Virtual Domain, 6x IPv4 Virtual Router Points of failure Response time TCP/HTTP High value response time is a point of failure. Any unsuccessful TCP Connection or HTTP Transaction error is a point of failure Execution Report Date Testers (initials) AC, VJ Test Run Result OK 22 spirent.com

23 Results Figure 15: Established TCP Connections Figure 16: TCP Time to First Byte, TCP Time to SYN/ACK spirent.com 23

24 Spirent Professional Services Final Report Figure 17: TCP Connection Rate Other Metrics Total Attempted HTTP Transactions 100,517,400 Total Successful HTTP Transactions 100,517,400 Total Unsuccessful HTTP Transactions 0 Total Attempted TCP Connections 100,517,400 Total Successful TCP Connections 100,517,400 Total Unsuccessful TCP Connections 0 Average Time to TCP SYN/ACK Average Round Trip Time spirent.com

25 Max IPv6 Throughput Identification ID Description Runtime Max IPv6 Throughput The same methodology as the one described in section 5.3 was used here. The only difference was the IP version changed from IPv4 to IPv6 and the amount of Avalanche ports. We needed to include more Avalanche ports in order to reach the performances of the DUT. 5 minutes Execution Parameters IP version Transport/Application HTTP object size Traffic setup Key measurements Connectivity DUT configuration IPv6 TCP/HTTP 100,000 bytes 7620 client IP addresses (thirty /24 subnets) 240 servers IP addresses TCP response time, application response time, TCP connections (per second, concurrent) 60x10GbE interfaces (Spirent Avalanche), 6x100GbE (Fortigate), SFP+ Multimode L4 Firewalling, Virtual Domain, 60x IPv6 Virtual Router Points of failure Response time TCP/HTTP High value response time is a point of failure. Any unsuccessful TCP Connection or HTTP Transaction error is a point of failure Execution Report Date Testers (initials) AC, VJ Test Run Result OK spirent.com 25

26 Spirent Professional Services Final Report Results Figure 18: HTTP Throughput Figure 19: TCP Time to First Byte, TCP Time to SYN/ACK 26 spirent.com

27 Figure 20: HTTP Response Time Other Metrics Total Attempted HTTP Transactions 62,230,200 Total Successful HTTP Transactions 62,230,200 Total Unsuccessful HTTP Transactions 0 Total Attempted TCP Connections 6,223,020 Total Successful TCP Connections 6,223,020 Total Unsuccessful TCP Connections 0 Average Time to TCP SYN/ACK Average Round Trip Time spirent.com 27

28 Spirent Professional Services Final Report Max UDP Throughput without packet loss for IPv4 Identification ID Description Max UDP Throughput without packet loss for IPv4 The goal is to measure the maximum throughput without any packet loss the device can handle by using UDP traffic with different frame sizes with IPv4 traffic. In that test case we use the RFC2544 methodology to generate IPv4/UDP packets with incremental fix frame size from 64 bytes up to 9216 bytes. Runtime Execution Parameters IP version Transport/Application Frame size Traffic setup Key measurements Connectivity The maximum bandwidth without packet loss is automatically defined through a dichotomy mechanism. 60 seconds per trial Number of trails defined by the dichotomy mechanism IPv4 UDP 64,128, 256, 512, 1024, 1280, 1518 and 9216 bytes Many to one model with a total of 1024 client IP addresses 512 Client IP to 1 server IP address per 100GEth port pair 2 pairs in total Bi-directional traffic (many to one & one to many) Throughput, Loss, Latency and Jitter 4x100GbE interfaces (Spirent TestCenter) DUT configuration 1024 sessions, MTU 2916 Points of failure Packet loss As soon as a packet is lost, the dichotomy mechanism try to defined the maximum bandwidth reachable on the DUT without any loss Execution Report Date Testers (initials) MH, VJ Test Run Result OK 28 spirent.com

29 Results Figure 21: UDP max Throughput Figure 22: Latency and Jitter by frame size at Throughput Comments Figure 23: Latency and Jitter by frame size at Throughput With the minimum frame size (64 bytes) the throughput was 155 Gbps With a small frame size (128 bytes) the throughput was 274 Gbps. With any frame size larger than 256 bytes the achieved throughput is Gbps. In all test runs, the impact of the DUT on the latency was minimal (4.23μs on 64 bytes). In order to see if the MTU size configured in the DUT could have an impact on the Latency results, the tests has been done with both 1518 and 9216 values. The difference between both configurations is negligible. spirent.com 29

30 Spirent Professional Services Final Report Max UDP Throughput without packet loss for IPv6 Identification ID Description Runtime Execution Parameters IP version Transport/Application Frame size Traffic setup Key measurements Connectivity Max UDP Throughput without packet loss for IPv6 The goal is to measure the maximum throughput without any packet loss a device can handle by using UDP traffic with different frame sizes with IPv6 traffic. In that test case we use the RFC2544 methodology to generate IPv6/UDP packets with incremental fix frame size from 86 bytes up to 9216 bytes. The maximum bandwidth without packet loss is automatically defined through a dichotomy mechanism. 60 seconds per trial Number of trials defined by the dichotomy mechanism IPv4 UDP 64, 128, 256, 512, 1024, 1280, 1518 and 9216 bytes Many to one model with a total of 1024 client IP addresses 512 Client IP to 1 server IP address per 100GEth port pair 2 pairs in total Bi-directional traffic (many to one & one to many) Throughput, Loss, Latency and Jitter 4x100GbE interfaces (Spirent TestCenter) DUT configuration 1024 sessions, MTU 2916 Points of failure Packet loss As soon as a packet is lost, the dichotomy mechanism try to defined the maximum bandwidth reachable on the DUT without any loss Execution Report Date Testers (initials) MH, VJ Test Run Result OK 30 spirent.com

31 Results Figure 24: UDP max Throughput Figure 25: Latency and Jitter by frame size at Throughput Comments Figure 26: Latency and Jitter by frame size at Throughput The IPv6 Max Throughput values are identical to those for IPv4 which proves that the IPv6 traffic has no impact on the DUT compared to the IPv4. spirent.com 31

32 Spirent Professional Services Final Report Spirent equipment used in this test Spirent SPT-N11U Chassis N11U_Mainframe_Chassis_Datasheet.PDF Spirent TestCenter Spirent Avalanche Security_Testing_and_Application_Performance_Datasheet.pdf Spirent C100 STC_C100.pdf Mx100G-F2 STC_HyperMetrics_mX_40-100G_Module_Datasheet.pdf Mx100G-P2 Spirent_mX_100G_CFP2_Datasheet.pdf IPv6 IPv6_Testing.pdf spirent.com AMERICAS SPIRENT EUROPE AND THE MIDDLE EAST +44 (0) ASIA AND THE PACIFIC Spirent. All Rights Reserved. All of the company names and/or brand names and/or product names referred to in this document, in particular, the name Spirent and its logo device, are either registered trademarks or trademarks of Spirent plc and its subsidiaries, pending registration in accordance with relevant national laws. All other registered trademarks or trademarks are the property of their respective owners. The information contained in this document is subject to change without notice and does not represent a commitment on the part of Spirent. The information in this document is believed to be accurate and reliable; however, Spirent assumes no responsibility or liability for any errors or inaccuracies that may appear in the document. Rev B. 03/15