High Performance Highly Available z Systems RoCE Networks

Transcription

1 High Performance Highly Available z Systems RoCE Networks Stephen Guendert, Ph.D. z Systems Technology CTO Brocade Communications Insert Custom Session QR if Desired.

2 Abstract RDMA over Converged Ethernet (RoCE), especially with z13, is a great technology for improving application performance, reducing CPU usage, and saving money. This session will discuss how using Ethernet Fabric technology based switching for your RoCE connectivity will enhance these even further.

3 Agenda Introduction: Why RoCE on zsystems? RDMA, RoCE and SMC-R basics SMC-R with RoCE on z Systems Ethernet Fabrics Technology benefits for SMC-R Examples and Performance Summary

4 Why SMC-R (RoCE ) on z Systems? CPU Savings and application response time improvement Significant response time improvements and CPU savings for CICS transactions using Distributed Program Link (DPL) when using SMC-R vs. standard TCP/IP. Up to 48% reduction in response time and 10% CPU savings Significant overall transaction response time improvement for Websphere Application Server (WAS) accessing DB2 in another system. 40% reduction vs standard TCP/IP Brocade VCS Fabric technology further enhances this performance

5 Part 1: RDMA, RoCE, SMC-R and z Systems

6 Remote Direct Memory Access (RDMA) Allows a host to write or read memory from a remote host without involving the remote host s CPU and operating system (OS). Bypasses OS layers and many communications protocol layers that are otherwise required for communication between applications. Reduces software overhead, providing for high throughput, low latency networking.

7 RoCE - RDMA over Converged Ethernet RDMA-based technology has been available in the industry for many years primarily based on Infiniband (IB) RDMA technology provides the capability to allow hosts to logically share memory Infiniband requires a completely unique network ecosystem (unique hardware such as host adapters, switches, host application software, system management software/firmware, security controls, etc.) IB is common in the HPC market RDMA technology is now available on Ethernet RDMA over Converged Ethernet (RoCE)

8 RoCE RDMA protocol over Ethernet Uses RoCE Network Interface Adapter (RNIC) and Layer-2 switches with IEEE Converged Enhanced Ethernet (CEE) capability. Provides low latency, high bandwidth, high throughput, low processor utilization data transfer between hosts Direct RoCE port to RoCE port (no switch) is possible, but not recommended by IBM. Switches must support global pause frame (IEEE 802.3x)

9 Shared Memory Communications Remote Direct Memory Access (SMC-R) Definition Shared Memory Communications Remote Direct Memory Access (SMC-R) is a new communication protocol aimed at providing transparent acceleration for socketsbased TCP/IP applications and middleware Remote Direct Memory Access (RDMA) technology provides low latency, high bandwidth, high throughput, low processor utilization attachment between hosts SMC-R utilizes RDMA over Converged Ethernet (RoCE) as the physical transport layer SMC-R is built on the following concepts: RDMA enablement of the communications fabric Partitioning a part of OS host real memory into buffers and using RDMA technology to access this memory Establishing an out of band connection over which data is passed to the partner peer using RMDA writes and signaling

10 SMC-R Shared Memory Communication over RDMA Sockets over RDMA communication protocol that allows existing TCP applications to transparently benefit from RoCE. Requires no application changes. Provides host-to-host direct memory access without the traditional TCP/IP processing overhead. z/os V2R1 includes SMC-R support SMC-R is only used over 10GbE RoCE Express features to a partner z/os V2R1 system.

11 SMC-R Additional Benefits Load balancing The first application data is sent over one RoCE pair between the two hosts The second application data would be sent over the second RoCE pair between the two hosts High Availability If there was a failure of the connection between the first RoCE pair, all the sessions using the first RoCE pair would transparently move to the second RoCE pair.

12 SMC-R Additional Benefits (2) Provides high availability and load balancing when redundant network hardware paths are available. Introduces minimal administrative and operational changes. Provides dynamic discovery of partner RDMA capabilities and dynamic setup of RDMA connections over RoCE.

13 The Network A single Layer 2 (no IP router) 10GbE network is required. RoCE provides low latency, high bandwidth, high throughput, low processor utilization data transfer between hosts by avoiding TCP/IP capabilities such as IP routing. Both partners must be in the same IP subnet (no IP router between them)

14 OSA channels (Open Systems Adapter) SMC-R does require OSA channels Still requires the OSA TCP connect SMC-R uses the TCP connection to determine eligibility for RoCE and to build the point-to-point SMC-R link. Then the actual data traffic is sent over the SMC-R link. The TCP session is also used for Keepalive and to terminate both the RDMA and TCP connections when the session ends. OSA channels can be connected to the same switches as the RNICs.

15 Recommended configuration Minimum of 2 RoCE features and minimum 2 switches Once a session has been switched to SMC-R it cannot fall back to the TCP/IP OSA path. If a failure occurs with the RoCE connection and an alternate RoCE path exists, the active sessions on the original path will transparently move to the alternate path. If a failure occurs with the RoCE connection and an alternate RoCE path is not available, then all active sessions will fail. All new sessions will not switch to SMC-R but will flow over the TCP/IP OSA path instead.

16 SMC-R Link Group Two SMR-R links using different RNICs, between two peers are logically grouped into an SMC-R Link Group. This provides redundancy and load balancing. If one link fails, all active connections are automatically and dynamically moved to the other link without interruption. After recovery of the failed link, no connections are moved back. All new connections will be setup over the recovered link and load balancing recovered.

17 Application use cases for SMC-R Application servers such as the z/os WebSphere Application Server communicating (via TCP based communications) with CICS, IMS or DB2 particularly when the application is network intensive and transaction oriented Transactional workloads that exchange larger messages (e.g. web services such as WAS to DB2 or CICS) will see benefit. Applications that use z/os to z/os TCP based communications using Sysplex Distributor

18 z13 SMC-R advantages Allows concurrent sharing of a RoCE Express feature by multiple virtual servers (OS instances) Up to 31 virtual servers (OS instances, LPARs or 2nd level guests under zvm) can share a single feature Support for up to 16 RoCE Express features per zcpc Enables concurrent use of both RoCE Express ports by z/os (SMC-R) For High Availability each OS instance requires access to two unique (physical) features

19 IFP IFP z13: 10GbE RoCE Express Sample Configuration z13 On z13 each 10GbE RoCE FC 0411 Can support up to 31 logical partitions (Two or more features for each server recommended) z13 LPAR A z/os V2.1 LPAR B LPAR C z/os V2.1 RoCE RoCE Brocade VDX RoCE RoCE LPAR 1 z/vm V6.3 + z/os V2.1 LPAR 2 z/os V2.1 LPAR 3 z/os V2.1 LPAR D LPAR E OSA/OSD OSA/OSD Brocade VDX OSA/OSD OSA/OSD LPAR 4 LPAR 5 This configuration allows redundant SMC-R connectivity among LPAR A, LPAR C, LPAR 1, LPAR 2, and LPAR 3 LPAR to LPAR OSD connections are required to establish the SMC-R communications 1 GbE OSD connections can be used instead of 10 GbE OSD connections can flow through the same 10 GbE switches or different switches z13 exclusive: Simultaneous use of both 10 GbE ports on 10 GbE RoCE Express features

20 Measuring CPU usage TCP/IP address space CP usage in the time interval shown in: RMF Report Class for TCP/IP SMF Record Type 30

21 CPU Usage by Active Jobs

22 Performance benchmarks of SMC-R at distance Performance summary Technology viable even at 100km distances with DWDM At 10km: Retain significant latency reduction and increased throughput At 100km: Large savings in latency and significant throughput benefits for larger payloads, modest savings in latency for smaller payloads CPU benefits of SMC-R for larger payloads consistent across all distances Use cases for SMC-R at distance TCP Workloads deployed on Parallel Sysplex spanning sites Software based replication (i.e. TCP based) across sites (Disaster Recovery) e.g. InfoSphere Data Replication suite for z/os File transfers across z/os systems in different site FTP, Connect:Direct, SFTP, etc. Opportunity: Lower CPU cost for sending/receiving data while boosting throughput and lowering latency

23 Part 2: Ethernet Fabrics Technology and SMC-R

24 Ethernet Fabrics Evolve Data Centers Continual evolution with Ethernet fabrics Efficient Automated Cloud-Optimized All links fully active Multipathing at all layers: Layers 1, 2, and 3 IP storage-aware Automatic provisioning Zero-touch VM discovery, Layer 2/Layer 3 configuration, and mobility Self-forming trunks Manage many switches as a single logical device Multitenancy at scale with Virtual Fabric feature Scale out non-disruptively Orchestrated via OpenStack DevOps support Greater network utilization Lower OpEx Faster time to application deployment

25 Access Aggregation Core Leaf / Spine Core Ethernet Fabrics vs. Legacy Networks Classic Hierarchical Architecture Ethernet Fabric Architecture Scale-out Rigid architecture, north-south optimized Inefficient link utilization Individually managed switches VM-ignorant No network virtualization Automated provisioning All links active, L1/2/3 multipathing Fabric managed as one logical switch VM-aware Native and overlay network virtualization

26 Key VDX Capabilities for SMC-R Operational automation and efficiency Provides an automated RoCE Fabric Single Logical Chassis for entire fabric Scale-out fabric technology Automatic trunk formation Per-frame load balancing between switches Deep on-chip buffering Simplified automation and visibility

27 Automated RoCE fabric with Brocade VDX RDMA over Converged Ethernet the automatic lossless fabric RoCE enabled automatically on all fabric ports VDX VDX Automatic, end-to-end lossless Ethernet throughout the Ethernet fabric. VDX VDX VDX VDX RoCE hosts Configure host's RoCE network adapter to enable DCB Priority Flow Control (DCB PFC) and set it for PFC class 3 (priority 3). On the VDX s host-connected interfaces: -Ensure the setting for the CEE map is set to its default with global command: cee-map default -Disable LLDP with interface command: lldp disable -Enable CEE map with interface command: cee default

28 Advanced Flexibility Address business-critical SLAs with a resilient, high-performance fabric Deliver predictable performance and unmatched resiliency for business-critical applications Provision network capacity with minimal intervention and virtually no learning curve Configure and manage multiple switches in the fabric as a single logical element Provide storage-class resiliency with nondisruptive failover after a path or link failure NAS iscsi Flash

29 Auto-Configuration with Logical Chassis Configuring LAG (for 2 members) Execute the following commands on one switch: configure terminal interface port-channel 1 switchport VCS switchport mode trunk switchport trunk allowed vlan all qos flowcontrol tx on rx on mtu 9208 no shutdown interface tengigabitethernet 1/Ø/5 VCS channel-group 1 mode active VCS type standard no shutdown interface tengigabitethernet 1/Ø/6 channel-group 1 mode active type standard no shutdown exit Repeat same commands on other end switch. Total commands: 30 Others Simplifies Brocade VCS fabric deployment, scalability, and management of the network Enables VCS fabric capabilities on each switch (on by default) Connects the switches Configuring ISL Trunking (for up to 8 members) Absolutely no configuration required. Total commands: 0 Brocade Fabric automatically forms Common configuration across all switches, Inter-Switch Link (ISL) trunks auto-form Managed as a single logical chassis

30 Eliminate Protocol Exotica You don t have to be an IP expert Traditional Shared Network BGP MP-BGP RSVP-TE IPsec ISIS-TE Dedicated IP Storage Network OSPF PIM-SM IS-IS LLDP MSTP RSTP RIP VPLS EIGRP MSDP IGMP TACACS+ AAA X.509 DRR DVMRP RED WRED VRRP CIDR RADIUS IRDP MS-CHAP LISP DSCP LDP MPLS PWE3 IKE WFQ L2TP STP WRR RSVP HMAC SNTP IPv6 SHA MLD sflow NTP GRE DWFQ Layer 2+3 IP Connectivity vs. Step 1: Configure addressing and VLANs Step 2: Connect switches and automatically form fabric Step 3: Connect hosts and storage Layer 2 Ethernet Fabric Connectivity

31 Ethernet Fabrics Logical Chassis Architecture Simple and efficient v Virtual IP Management v Configuration Management Centralized Software Upgrade Downgrade and Auto Provisioning Centralized Monitoring and Troubleshooting

32 Ethernet Fabrics and Trunking Frame-based, high throughput ISL trunking using Brocade ASICs Brocade ISL Trunking provides high link utilization and ease-of-use All 10GE ports are not alike Frame-level, hardware-based trunking at Layer 1 Near 100% link utilization versus 802.3ad LAG groups ~60 70% link utilization Spill and Fill across links in trunk group Single flows can be split across all links Built into Brocade ASIC ISL Trunks automatically form Once both switches are in VCS mode, multiple ISLs automatically form a trunk No CLI entries necessary Brocade ISL Trunking (8 links active) 80 Gbps Frame-based trunking at Layer ad Link Aggregation (8 links active) ~50 Gbps Flow-based trunking at Layer 2 = 10GE link, width represents utilization

33 LEAF SPINE Brocade VDX Switches Complete breadth of portfolio Brocade VDX 8770 Family Brocade VDX 6740 Family Brocade VDX 6940 Family Brocade VDX Q Brocade VDX Brocade VDX Brocade VDX 6740 Brocade VDX 6740T Brocade VDX S Brocade VDX 6740T-1G 1 GbE/10 GbE/10 GBASE-T optimized with 40 GbE uplinks 10 GbE/40 GbE optimized with 100 GbE uplinks SCALABILITY 10 GbE/10 GBASE-T/40 GbE/100 GbE optimized modular system

34 Brocade VDX 6740 Switches Product details Simplicity and Automation Brocade VCS Fabric technology Automatic Migration of Port Profiles (AMPP) VCS Logical Chassis Auto-fabric provisioning Advanced Capabilities Full IP storage support with DCB capabilities Auto QoS prioritizes storage traffic VCS Virtual Fabric feature supports multitenancy VCS Gateway for NSX unifies virtual and physical networks SDN-capable (OpenFlow support) IPv6 hardware-ready Leading Scalability and Performance Fixed 48 1/10 GbE SFP+ (6740) /48 1/10 GBASE-T ports (Brocade VDX 6740-T) and 4 40GbE QSFP+ GbE; option up to 64 ports 10 GbE Fixed 48 1 GbE with 10 GbE software upgrade option 4 40 GbE QSFP+ ports; each 40 GbE can optionally be configured as 4 10 GbE in break-out mode High-performance Layer 2/Layer 3 switching Industry-leading deep buffers with dynamic buffering Increases MAC, VLANS, port profiles; delivers increased scalability Single ASIC, non-blocking with cut-through architecture Up to 160 GbE Brocade ISL Trunking improves switch capacity Efficient multilayer, multipathing for reliability and elasticity Brocade VDX 6740

35 BNA for Storage Stakeholders Extending Operational Ownership into IP Storage Building on the combined dashboard customizable today Ongoing Strategy: Combined usability focus via a Policy Configuration Center across both FOS & NOS to include: Extending MAPS to include NOS Leverage dynamic group capabilities from dashboards across MAPS and Configuration Policies Consistent Usability Focus Common Troubleshooting navigation Unified Storage Dashboard MAPS Common Configuration Policy UI Concepts Configuration Policy

36 VDX Mainframe Deployment example VDX 6740 IBM z13 FICON SAN Use the VDX for: RoCE/SMC-R IBM DB2AA GDPS AA Qrep TS7700 Grid IBM z13 DASD DB2AA Virtual tape Brocade Network Advisor SAN + IP Management FICON SAN 10GbE OSA channels RoCE 10GbE

37 Performance impact of SMC-R on real z/os workloads 40% reduction in overall transaction response time for WebSphere Application Server v8.5 Liberty profile TradeLite workload accessing z/os DB2 in another system measured in internal benchmarks * Linux on x Workload Client Simulator (JIBE) WebSphere to DB2 communications using SMC-R TCP/IP HTTP/REST z/os SYSA WAS Liberty TradeLite JDBC/DRDA SMC-R RoCE z/os SYSB DB2 File Transfers (FTP) using SMC-R Up to 50% CPU savings for FTP binary file transfers across z/os systems when using SMC-R versus standard TCP/IP. ** z/os SYSA FTP Client SMC-R FTP z/os SYSB FTP Server * Based on projections and measurements completed in a controlled environment. Results may vary by customer based on individual workload, configuration and software levels. ** Based on internal IBM benchmarks in a controlled environment using z/os V2R1 Communications Server FTP client and FTP server, transferring a 1.2GB binary file using SMC-R (10GbE RoCE Express feature) vs standard TCP/IP (10GbE OSA Express4 feature). The actual CPU savings any user will experience may vary. RoCE

38 Performance impact of SMC-R on real z/os workloads (cont) Up to 48% reduction in response time and up to 10% CPU savings for CICS transactions using DPL (Distributed Program Link) to invoke programs in remote CICS regions in another z/os system via CICS IP interconnectivity (IPIC) when using SMC-R vs standard TCP/IP * CICS to CICS IP Intercommunications (IPIC) using SMC-R z/os SYSA CICS A DPL calls SMC-R IPIC RoCE z/os SYSB CICS B Program X WebSphere MQ for z/os using SMC-R z/os SYSA WebSphere MQ SMC-R MQ messages RoCE z/os SYSB WebSphere MQ WebSphere MQ for z/os realizes up to 200% increase in messages per second it can deliver across z/os systems when using SMC-R vs standard TCP/IP **** * Based on internal IBM benchmarks using a modeled CICS workload driving a CICS transaction that performs 5 DPL (Distributed Program Link) calls to a CICS region on a remote z/os system via CICS IP interconnectivity (IPIC), using 32K input/output containers. Response times and CPU savings measured on z/os system initiating the DPL calls. The actual response times and CPU savings any user will experience will vary. ** Based on internal IBM benchmarks using a modeled WebSphere MQ for z/os workload driving non-persistent messages across z/os systems in a request/response pattern. The benchmarks included various data sizes and number of channel pairs The actual throughput and CPU savings users will experience may vary based on the user workload and configuration.

39 Network performance comparison Network latency reduced up to 80% for z/os TCP/IP multi- tier OLTP workloads such as web-based claims and payment systems * Transaction without SMC-R TCP Network Transaction with SMC-R SMC-R Network Reduced latency, CPU consumption and improved wall clock time TIME * Based on internal IBM benchmarks of modeled z/os TCP sockets-based workloads with request/response traffic patterns using SMC-R vs TCP/IP. The actual throughput that any user will experience will vary.

40 SMC-R and Ethernet Fabrics Optimized Network Performance (leverageing RDMA technology) Transparent to (TCP socket based) application software More efficient, highly available SMC-R fabric Simpler to manage and configure Preserves existing network security model Resiliency (dynamic failover to redundant hardware) Transparent to load balancers Preserves existing IP topology, network administrative and operational models.

41 CPU Savings (cost reduction) With response time improvements SMC-R can lead to substantial CPU savings (lower costs) SMC-R also simultaneously improves response times Lower costs with improved performance

42 Thank You for Attending! Please remember to complete your evaluation of this session in the SHARE mobile app. High Performance Highly Available z Systems RoCE Networks