Wide-Area InfiniBand RDMA: Experimental Evaluation

Transcription

1 Wide-Area InfiniBand RDMA: Experimental Evaluation Nagi Rao, Steve Poole, Paul Newman, Susan Hicks Oak Ridge National Laboratory High-Performance Interconnects Workshop August 31, 2009, New Orleans, LA Research Sponsored by Department of Defense

2 Outline Motivation Test Environment Wide-Area InfiniBand Longbow Measurements ABEx Measurements USN and Anue Connections Voltaire and Mellanox HCAs TCP over 10GigE Conclusions

3 ORNL and NLR Collaboration On-Going Activities: Long distance network testing and measurement analysis since Supercomputing in Dallas Future Planned Activities: 40/100/1000 Gbps Wide-Area Network for Large Datasets IB and other technologies over - SONET, 10GigE, L2PT, IP and other connections Experimental Design and Measurements Performance analysis and comparison NLR:

4 Wide-Area High-Performance Transport Main Question: Best ways to support high throughput data transport between remote high-performance computing, storage and analysis systems Technical Topic Addressed: Wide-area data transfers at 10Gbps rates at thousands of miles: Robust end-to-end solutions: physical- through transport-layers Significant Technical Challenges: Two different options A. Performance of conventional TCP/IP solutions is unclear: 1. Many TCP variants: complex to implement, analyze and test 2. Enormous in-situ efforts are needed per connection basis to reach multiple Gbps rates B. InfiniBand solutions seem promising but need to be studied: enterprise centric 1. Very limited wide-area results are available 2. Objective side-by-side comparisons with TCP/IP are not available 3. Require capable test environments network centric

5 Approach and Results Specific Topics Discussed : 1. High-performance testbed: Collection of high-end hosts, storage and edge systems Flexible 10Gbps, 8600 mile network core 2. Throughput performance of IB over wide-area connections (SONET, 10GigE) 3. Throughput performance of 10GigE + TCP high-performance variants Overall Summary: Scalability and plug-in capability of IB 4x SDR demonstrated over 8600 miles two brands of IB WAN devices Longbow XR and ABEx 2020 two brands of HCAs Voltaire and Mellanox two types of 10Gbps connection s - Anue Emulated and USN testbed Technical Results Summary: In a nutshell, over 8600 mile connection throughput decrease: IB Over SONET/10GigE: 0.02Mbps/mile 10GigE-HTCP: 1.3Mbps/mile

6 Conventional I/0 Architecture for WAN Wide-Area Transport Traditionally Relies on TCP/IP : Requires conversion from enterprise I/O to TCP/IP data format and protocol Efficient TCP/IP transport CPU CPU system memory Host bus system controller PCI storage storage controller NIC wide-area TCP/IP Issues: Storage-to-WAN conversion needs careful impedance matching TCP/IP wide-area tuning is still complex and connection-specific

7 InfinBand Architecture Provides Efficient High-Performance I/O: 4x SDR offers 8Gbps throughput out-of-box Primarily designed for system/enterprise-level I/O - few millisecond latencies Question: Are these bandwidths achievable in widearea? Benefits: Extend natively highperformance connectivity between super-computers, storage systems without TCP-transition

8 IB Over Wide-Area IB can be supported over wide-area: Over SONET OC192 or 10GigE LAN-PHY or WAN-PHY infrastructure system memory CPU Host bus system controller HCA CPU Capability: Storage across WAN will simply appear as local IB WAN extension device: Frame conversion Terminates/re-starts IB flow control storage IB switch IB WAN wide-area SONET 10GigE

9 Longbow XR extends IB to WAN Performs link-level buffer credit extension to span up to 150,000km WAN distances Full bandwidth of the 4X SDR link (1Gbyte/second full-duplex) Protocol transparent; complete interoperability

10 ABEx 2020 IB WAN Extension Devices ABEx 2020 platform supports InfiniBand over OC192, 10GigE and L2TP (Layer-2 Tunneling Protocol)

11 UltraScience Net: Experimental network research testbed for advanced networking and associated technologies for high-performance Since 2005 Features End-to-end guaranteed bandwidth channels Dynamic, in-advance, reservation and provisioning of fractional/full lambdas Secure control-plane for signaling Peering with ESnet, National Science Foundation CHEETAH, and other networks Department of Defense

12 USN Architecture: Separate Data-Plane and Control-Planes No data plane continuity: can be partitioned into islands - necessitated out-of band control plane Secure control-plane with: Encryption, authentication and authorization On-demand and advanced provisioning GMPLS in IP is not secure enough: Dual OC192 backbone: SONET-switched in the backbone Ethernet-SONET conversion

13 USN data-plane: Node configuration In the core: Two OC192 switched by Ciena CDCIs At the edge: 10/1 GigE provisioning using Force10 E300s Node Configuration e300 Linux host 10 GigE GigE 10 GigE WAN PHY CDCI OC192 to Seattle Connections to CalTech and ESnet Data plane user connections: Direct connections to Core switches SONET and 1 GigE Edge switches Ethernet channels Utilize UltraScience Net hosts

14 Infiniband Over SONET: Obsidian Longbows RDMA throughput measurements over USN quad-core dual socket Linux host Linux host IB 4x ORNL longbow IB/S longbow IB/S ORNL CDCI ORNL loop -0.2 mile: 7.48Gbps OC miles Chicago CDCI ORNL-Chicago loop 1400 miles: 7.47Gbps ORNL- Chicago - Seattle loop 6600 miles: 7.37Gbps 3300 miles 4300 miles Seattle CDCI Sunnyvale CDCI IB 4x: 8Gbps (full speed) Host-to-host local switch:7.5gbps Hosts: dual-socket quad-core 2GHz AMD Opteron, 4GB memory 8-lane PCI-Express slot Dual-port Voltaire 4x SDR HCA. ORNL Chicago Seattle - Sunnyvale loop 8600 miles: 7.34Gbps Supercomputing 2008

15 Performance Profiles IB RDMA Throughputs Throughput Distance Profile Plot throughput as a function connection length (d) and message size (s) B=SONET, WAN-PHY T (, ) B d s Throughput Stability Profile Plot throughput as function of connection length and repetition number for fixed message size T (, ) B d s T ( d, s) B Average throughput over 10 iterations with 8M message size Tˆ ( d ) B Throughput Decrease Per Mile (DPM): at connection length ˆ ˆ ˆ T ( 0) ( ) ( ) B d TB di DB di d d i 0 d i

16 Distance and Stability Profiles of IB over SONET Measurements using ib_rdma-bw c It uses IB CM for connection setup and management distance profile distance profile stability profile 8M message size T ( ) B d T (, ) B d s Connection length (miles) d i Throughput (Gbps) 8M msg Std-dev (Mbps) D ( d ) DPM (Mbps) B i

17 Wide-Area Connections: SONET OC192 and 10GigE WAN/LAN-PHY SONET: Widely deployed over long-haul optical backbone networks OC192 over DWDM: 9.6Gbps over single wavelength - robust wellunderstood technology Utilizes Time-Division Multiplexing to provide well separated sublambdas 4*OC48 or 64*OC3 10Gig Ethernet: Relatively recent technology: high-potential for wide deployment WAN-PHY Ethernet frames are packed into SONET OC192c payload: peak 9.6 Gbps LAN-PHY Natively transports Ethernet packets: full 10Gbps A Comparison: SONET is widely deployed but needs more expensive infrastructure Sub-channel separation is more robust in SONET than 10GigE 10GigE more naturally transits between LAN to WAN environments

18 IB over 10GigE LAN-PHY and WAN-PHY host quad-core dual socket host longbow IB/S longbow IB/S ORNL loop -0.2 mile ORNL-Chicago loop 1400 miles ORNL 700 miles ORNL E300 switched in software ORNL CDCI ORNL- Chicago - Seattle loop 6600 miles Chicago CDCI Chicago E miles Seattle CDCI Optional WAN-PHY switching IB 4x OC GigE WAN-PHY 10 GigE LAN-PHY 1GigE 4300 miles Sunnyvale CDCI Sunnyvale E300 ORNL Chicago Seattle - Sunnyvale loop 8600 miles

19 Performance Profiles of IB Over 10GigE WAN-PHY distance profile peak distance profile average distance profile T (, ) B d s Results are almost the same as in SONET case Connection length (miles) d i Throughput (Gbps) 8M msg Std-dev (Mbps) Dˆ ( d ) DPM (Mbps) B i

20 Cross-Traffic Generation quad-core dual socket host 1 host 2 ORNL longbow IB/S longbow IB/S USN host1 ORNL E300 USN host2 ORNL loop -0.2 mile ORNL-Chicago loop 1400 miles dual-core single socket Netrion10GigE ORNL CDCI ORNL- Chicago - Seattle loop 6600 miles 700 miles Chicago CDCI Chicago E miles Seattle CDCI IB 4x OC GigE WAN-PHY 10 GigE LAN-PHY 1GigE 4300 miles Sunnyvale CDCI Sunnyvale E300 ORNL Chicago Seattle - Sunnyvale loop 8600 miles

21 Cross-Traffic Effect of IB over 10GigE WANPHY Average throughput for 8M miles cross-traffic G G G G G Competing traffic: UDP streams on WAN at 1,2,3,4 Gbps Distance profiles are unaffected for cross-traffic levels of up to 1Gbps IB throughput was drastically effected at cross-traffic level of 4 Gbps Effect of cross-traffic is more on large message sizes Below 1Gbps

22 ABEx Measurements Mellanox HCA LAN PHY WAN PHY L2TP Measurements: Quite similar to Longbow

23 Anue WAN Emulator Replicates the delays, bandwidth of OC192 and 10GigE wide-area connections Various connection lengths can be emulated some not realizable on USN Objective: Compare IB measurements on emulated and USN connections With potential interpolation and extrapolation to other connection lengths test device Hosts, Longbows, ABEx, KGs, etc Emulated emulated OC192 OC192 wan 10GigE ANUE test device Hosts, Longbows, ABEx, KGs, etc

24 Anue Emulated OC192 and 10GigE Connections host quad-core dual socket host longbow IB/S longbow IB/S ORNL Fujitsu 10GigE switched in software ORNL E300 Anue 10GigE host quad-core dual socket host longbow IB/S longbow IB/S ORNL Anue OC192 OC192 IB 4x OC GigE WAN-PHY 10 GigE LAN-PHY 1GigE

25 Throughput MB/s Throughput MB/s Throughput MB/s USN and Anue Emulated Connections For average throughput: SONET: USN higher IB throughput for long connections LANPHY: Anue has higher IB throughput for long connections WANPHY both throughputs are same at all lengths Peak IB throughputs are same for all lengths Longbow Mellanox Tests (WANPHY) ib_rdma_bw -n s i 2 -p Longbow Mellanox Tests (POS) ib_rdma_bw -n s i 2 -p Latency (ms) 170 ANUE (A) USN (A) ANUE (P) USN (P) Longbow Mellanox Tests (LANPHY) ib_rdma_bw -n s i 2 -p ANUE (A) USN (A) ANUE (P) USN (P) ANUE (A) USN (A) ANUE (P) USN (P) Latency (ms) Latency (ms)

26 Anue-USN Measurements - ABEX LAN PHY WAN PHY L2TP Performance differences are fairly small

27 LAN PHY Anue-USN Measurements Longbow Mellanox HCA SONET Anue results are slightly more optimistic WAN PHY

28 Mellanox and Voltaire - ABEX LAN PHY WAN PHY Performance differences are fairly small

29 Longbow and ABEx Mellanox HCAs LAN PHY WAN PHY Performance differences are fairly small - More detailed experimentation and analysis are needed

30 10GigE Connections Quad-core Dual socket host ORNL 700 miles 3300 miles 4300 miles host ORNL E300 ORNL CDCI Chicago CDCI Seattle CDCI Sunnyvale CDCI Myrinet 10 GigE NICS Over PCI-Express Chicago E300 Sunnyvale E300 ORNL loop -0.2 mile ORNL-Chicago loop 1400 miles ORNL- Chicago - Seattle loop 6600 miles 10 GigE WAN-PHY 10 GigE LAN-PHY OC192 ORNL Chicago Seattle - Sunnyvale loop 8600 miles

31 Challenges of Assessing TCP Transport TCP or similar transport method is needed to ensure reliable data delivery over 10GigE provisioned connections Numerous TCP variants for High-Performance: 1. Standard slow-start and Additive-Increase and Multiplicative Decrease (AIMD) are not effective for high performance operation 2. A variety of TCP variations have been proposed for high performance operation difficult to implement, analyze and test 3. All of them require multiple streams and significant amount of tuning to provide multi-gbps throughputs Testing TCP variants: 1. Congestion control dynamics are very complex and not amenable to simple analytical treatment need complementary experiments 2. Dynamically loadable congestion control modules in linux kernels: auto-tuning is effective in buffer management BIC, CUBIC, Hamilton TCP (HTCP), Scalable TCP, Highspeed TCP, TCP Vegas Max buffer sizes are set to highest allowed on hosts through sysctrl 3. Measurements show high variance robust measurement and analysis methods are needed

32 Performance Profiles TCP Throughputs BIC and Hamilton TCP pluggable Linux modules Throughput Distance Profile Plot throughput as a function connection length (d) and number of streams (n) A=BIC,HTCP T (, ) A d n Throughput Stability Profile Plot throughput as function of connection length and repetition number of streams Average throughput over repetitions and range of number of streams Tˆ ( d ) B Throughput Decrease Per Mile Tˆ ( d ) Tˆ ( d ) ˆ ( ) A 0 A i A di di d0 D

33 Performance of TCP over 10GigE BIC with Linux auto-tuning Connection length (miles) d i Throughput (Gbps) 8M msg Std-dev (Mbps) Dˆ ( d ) DPM (Mbps) B i high deviations Better than IB for local connections

34 Performance of TCP over 10GigE Hamilton TCP with Linux auto-tuning Connection length (miles) d i Throughput (Gbps) 8M msg Std-dev (Mbps) Dˆ ( d ) DPM (Mbps) B i

35 Comparative Performance of BIC and Hamilton TCP 1400 miles 8600 miles BIC HTCP Multiple streams are needed to get high throughputs Highspeed TCP We also tested Highspeed TCP Scalable-TCP TCP-Vegas not as good as BIC and HCTP

36 Decrease Per Mile Empirical Results Throughput Decrease Per Mile (DPM): at connection length ˆ ˆ ˆ T ( 0) ( ) ( ) B d TB di DB di d d Average Over Connection Lengths: k ˆ 1 ˆ C DB ( di ) k i 1 i 0 d i IB Over SONET/10GigE: 0.02Mbps/mile 10GigE-HTCP: 1.3Mbps/mile DPM estimate is of derivative of random quantity: need a systematic way to assess its robustness

37 Decrease Per Mile Estimation Measurements involve random components For a fixed connection, measurements depend on the properties of (i) connection, (ii) edge devices, (iii) host configurations, and (iv) measurement software: T (, ) C d a random fixed Connection lengths are decided by which ones are possible for the test-bed d1, d2,, dn Composite Distribution: Complex and hard to characterize due to interaction of multiple factors distributed according to distributed according to P P P T, d T d d P Td P d DPM is a derivative of random quantity, and robustness of its estimator must be explicitly established C D C ( d, c ) dp d D C TC ( d, c) d T ( d, a) E T ( d, c) C C

38 Preliminary Robustness Result of DPM Estimator How good ˆ C is as an estimator of ( d d ) P kp T d T d 0 ˆ 0 ˆ( ) ( ) k C C k k 2k 2 2 k ( d ( 0 ) T d0 ) T ( dk ) k d B 2k 2 a 2k Ae C T ( d ) T ( d ) k This result is direct application of Vapnik-Chervenenkis equation for monotonic regression functions guaranteed irrespective of the underlying Roughly speaking, ˆ C estimator is probabilistically close to gets better if well-separated connection lengths - finer results appear to be possible C

39 10Gbps Crypto Devices: Test Configuration quad-core dual socket ORNL host 3 Fujitsu 10GigE host 4 10 Gbps 10 Gbps ORNL E300 USN ORNL CDCI 700 miles Chicago CDCI Chicago E miles Seattle CDCI 4300 miles Sunnyvale CDCI Sunnyvale E300 ORNL loop -0.2 mile ORNL-Chicago loop 1400 miles OC GigE WAN-PHY 10 GigE LAN-PHY ORNL- Chicago - Seattle loop 6600 miles ORNL Chicago Seattle - Sunnyvale loop 8600 miles

40 host1-host2 Connections host3-host4 Connections through 10Gbps Devices ORNL host 1 host 3 host 4 VLAN VLAN Fujitsu 10GigE VLAN 10 Gbps 10 Gbps VLAN ORNL E300 VLAN USN ORNL CDCI host 2 VLAN OC GigE WAN-PHY 10 GigE LAN-PHY

41 TCP Profiles: Before and after MTU Alignment host3-4 Encrypted Connection: File transfer Fiber loop between 10Gbps devices : 9 Gbps TCP throughput When connected to E300: 9Gbps throughput locally MTU size is modified on E300 IP segment/datagram size set to byte MTU 1400 byte MTU 8950 byte MTU

42 TCP Profiles Comparison: Plain and Encrypted Connections Fiber loop between 10Gbps devices : 9 Gbps TCP throughput Chicago loop: host3-4 connection achieved 8Gbps Sunnyvale loop: host3-4 connection 1.5 time higher throughput Observations: Compared to plain connections, for encrypted connections: High throughput is achieved with less number of streams Higher throughput is achieved at longer distances

43 plain encrypted UDP File Transfer Profiles: host 3-4 Connection Iperf udp modules exchanged measurement information after MTU Alignment Server results were received at the client These are higher than UDT4 throughput possibly due to congestion control

44 UDT4 vs. Modified UDT4: host3-4 Encrypted Connection MTU is set to 8950 bytes and congestion control is partially commented out Achieved throughputs more than doubled Approximately same variation under measurements File transfer throughput doubled: Variance under repetition Did not depend on distance

45 plain encrypted Modified UDT4 Partial Results: Plain and Encrypted Connections Plain throughputs are 1Gbps better than encrypted throughtputs

46 UDT4: CWND and RTT estimates Host3-4 uses essentially same RTT for all connections up to Chicago much larger than ping values Host1-2 uses different RTTs for different connections plain encrypted Host3-4 encrypted connection larger windows for 8600 mile essentially uses same window size for all other connection Host1-2 uses different window sizes for different connections plain encrypted

47 Conclusions and Results We conducted structured experiments to asses: 1. Throughput performance of IB RDMA over wide-area connections SONET and 10GigE two brands of IB WAN devices Longbow XR and ABEx 2020 two brands of HCAs Voltaire and Mellanox two types of 10Gbps connection s - Anue Emulated and USN testbed 2. Throughput performance of 10GigE + TCP high performance versions Experimental Results: 1. Scalability and plug-in capability of IB ~7.3 Gbps over 8600 miles 2. Performance testing of recent high-performance TCP over Ethernet: best performance is ~2Gbps at 8600 miles, but above 9Gbps locally 3. Side-by-side comparison over 8600 mile connection: IB Over SONET/10GigE: 0.02Mbps/mile 10GigE-HTCP: 1.3Mbps/mile 4. Cross-traffic effects: IB performance is robust with upto1gbps cross-traffic either on WAN connection or using their own 1GigE ports. TCP is less prone to such effects but more testing is needed. 5. Infrastructure: SONET or 10GigE can support both solutions IB WAN devices are significantly more expensive needed for IB

48 Acknowledgements This work was supported by the United States Department of Defense & used resources of the Extreme Scale Systems Center at Oak Ridge National Laboratory.

49 Thank you