10GbE RoCE Express and Shared Memory Communications RDMA (SMC-R) Frequently Asked Questions

Transcription

1 10GbE RoCE Express and Shared Memory Communications RDMA (SMC-R) Frequently Asked Questions 10GbE RoCE Express and Shared Memory Communications RDMA (SMC-R) Frequently Asked Questions What are RDMA and RoCE? What is RDMA? What is a communication fabric? What is RoCE? I ve heard of RoCE before. Is the one in the market today the same? What is a communication protocol? What is the 10GbE RoCE Express feature? What is SR-IOV and how does 10 GbE RoCE Express feature use it? How many ports are on the 10GbE RoCE Express feature? What is an RDMA-capable Network Interface Card? I ve heard of RDMA-capable Network Interface Cards (RNICs) or host adapters that are currently available in the industry. Is this existing technology similar to what you are announcing for z Systems? What is SMC-R What is Shared Memory Communications over RDMA (SMC-R)? Does the SMC-R exploitation of RoCE require special Ethernet (RoCE-capable) switches? How far apart can different servers be to communicate with each other using SMC- R/RoCE? Will I still need OSA once I exploit SMC-R using the 10 GbE RoCE Express feature or 10 GbE RoCE Express2 feature? Is SMC-R used for anything other than together with RoCE? Benefit of SMC-R and RoCE What is the value I will get from SMC-R and the 10GbE RoCE Express feature or 10GbE RoCE Express2 feature? I have a single CPC within my data center. Does it make sense for me to exploit SMC-R? How will SMC-R and the 10GbE RoCE Express feature or 10GbE RoCE Express2 feature be positioned with HiperSockets? I have multiple CPCs in my data center but I don't know if I have the type of or enough of the workloads with the network traffic patterns that could benefit from SMC-R and RoCE Express. I need help with building the justification and business plan to add RoCE Express to my datacenter. How can I determine if SMC-R applies to my environment and what level of benefit I might expect? The term Shared Memory Communications suggests that special memory might be required in the operating system. Are there any new or special memory requirements or memory considerations for exploiting SMC-R? Does Flow Control as we know it for TCP/IP still exist with the SMC-R protocol? How is Flow Control handled when the data is flowing over the RoCE fabric? Will SMC-R and RoCE support SNA? Will SMC-R and RoCE support SNA with Enterprise Extender? Configuration

2 4.1. What is the feature code for 10GbE RoCE Express? Can the 10GbE RoCE Express feature be shared between multiple LPARs? How is the 10GbE RoCE Express feature defined to the CPC? How many 10GbE RoCE Express features will I need to have on my system? What are the design considerations for implementing the required OSA Express feature on behalf of SMC-R communications? What operating systems support this new technology? How are these two types of adapter ports associated with each other in the HCD and software configurations? When deploying SMC-R are there any VLAN Ethernet switch configuration considerations? If the OSA to be associated with the RoCE feature is being used for other traffic, do I need to stop that traffic to add the PNET ID to the OSA? Are the two physical ports on the RoCE feature and the two ports on an OSA CHPID similar? What MTU is used by the RoCE feature? What are the possible FID or PFID values? Application Questions What application traffic is supported using SMC-R with RoCE? What are the SMC-R protocol connection eligibility requirements for two peer hosts to connect using SMC-R? Can two hosts that are in unique IP security zones that must traverse a network firewall exploit SMC-R? What are the steps required to use SMC-R with RoCE between two partner z/os servers? What changes do I need to make to the applications to take advantage of RoCE and SMC-R? Can I enable SMC-R but exclude some applications from using it? Can I communicate memory to memory between my distributed system and my z System such as a SAP application server and the z/os DB2 database? How does Sysplex Distributor support SMC-R with RoCE? I have an external Load Balancer deployed between z/os application tiers. Will those z/os TCP connections also be eligible for SMC-R? RAS and Security What happens if the 10GbE RoCE Express has a problem? What security is there for the 10GbE RoCE Express feature? How do I know that data traveling across the RDMA fabric is protected? How do I diagnose problems with SMC-R and RoCE? Positioning of RoCE with SMC-R versus OSA-Express and HiperSockets What differences exist between an OSA-Express OSD (QDIO) network fabric and a RoCE communication fabric? What differences exist between a HiperSockets network fabric and a RoCE communication fabric? Non-z/OS Support, Announcement Letters, and Statements of Direction What are the Announcement Letters referring to the 10GbE RoCE Express feature? What are the statement of directions from the July 2013 announcement concerning 10GbE RoCE Express? What operating systems support the 10GbE RoCE Express feature? Pricing

3 9.1. What is the price of SMC-R? GbE RoCE Express and Shared Memory Communications RDMA (SMC-R) Frequently Asked Questions What are RDMA and RoCE? What is RDMA? What is a communication fabric? What is RoCE? I ve heard of RoCE before. Is the one in the market today the same? What is a communication protocol? What is the 10GbE RoCE Express feature? Any new news for the 10 GbE RoCE Express feature in the February 9, 2016 announcement? How many ports are on the 10GbE RoCE Express feature? What is an RDMA-capable Network Interface Card? I ve heard of RDMA-capable Network Interface Cards (RNICs) or host adapters that are currently available in the industry. Is this existing technology similar to what you are announcing for z Systems? What is SMC-R What is Shared Memory Communications over RDMA (SMC-R)? Does the SMC-R exploitation of RoCE require special Ethernet (RoCE-capable) switches? How far apart can different servers be to communicate with each other using SMC- R/RoCE? Will I still need OSA once I exploit SMC-R using the 10 GbE RoCE Express feature? Is SMC-R used for anything other than together with RoCE? Benefit of SMC-R and RoCE What is the value I will get from SMC-R and the 10GbE RoCE Express feature? I have a single CPC within my data center. Does it make sense for me to exploit SMC-R? How will SMC-R and the 10GbE RoCE Express feature be positioned with HiperSockets? I have multiple CPCs in my data center but I don't know if I have the type of or enough of the workloads with the network traffic patterns that could benefit from SMC-R and RoCE Express. I need help with building the justification and business plan to add RoCE Express to my datacenter. How can I determine if SMC-R applies to my environment and what level of benefit I might expect? The term Shared Memory Communications suggests that special memory might be required in the operating system. Are there any new or special memory requirements or memory considerations for exploiting SMC-R? Does Flow Control as we know it for TCP/IP still exist with the SMC-R protocol? How is Flow Control handled when the data is flowing over the RoCE fabric? Will SMC-R and RoCE support SNA? Will SMC-R and RoCE support SNA with Enterprise Extender? Configuration What is the feature code for 10GbE RoCE Express? Can the 10GbE RoCE Express feature be shared between multiple LPARs? How is the 10GbE RoCE Express feature defined to the CPC? How many 10GbE RoCE Express features will I need to have on my system? What are the design considerations for implementing the required OSA Express feature on behalf of SMC-R communications? What operating systems support this new technology? How are these two types of adapter ports associated with each other in the HCD and software 3 Formatted: Default Paragraph Font, Check spelling and grammar Formatted... [1] Formatted... [2] Formatted... [3] Formatted... [4] Formatted... [5] Formatted... [6] Formatted... [7] Formatted... [8] Formatted... [9] Formatted... [10] Formatted... [11] Formatted... [12] Formatted... [13] Formatted... [14] Formatted... [15] Formatted... [16] Formatted... [17] Formatted... [18] Formatted... [19] Formatted... [20] Formatted... [21] Formatted... [22] Formatted... [23] Formatted... [24] Formatted... [25] Formatted... [26] Formatted... [27] Formatted... [28] Formatted... [29] Formatted... [30] Formatted... [31] Formatted... [32] Formatted... [33]

4 configurations? When deploying SMC-R are there any VLAN Ethernet switch configuration considerations? If the OSA to be associated with the RoCE feature is being used for other traffic, do I need to stop that traffic to add the PNET ID to the OSA? Are the two physical ports on the RoCE feature and the two ports on an OSA CHPID similar? What MTU is used by the RoCE feature? What are the possible FID or PFID values? Application Questions What application traffic is supported using SMC-R with RoCE? What are the SMC-R protocol connection eligibility requirements for two peer hosts to connect using SMC-R? Can two hosts that are in unique IP security zones that must traverse a network firewall exploit SMC-R? What are the steps required to use SMC-R with RoCE between two partner z/os servers? What changes do I need to make to the applications to take advantage of RoCE and SMC-R? Can I enable SMC-R but exclude some applications from using it? Can I communicate memory to memory between my distributed system and my z System such as a SAP application server and the z/os DB2 database? How does Sysplex Distributor support SMC-R with RoCE? I have an external Load Balancer deployed between z/os application tiers. Will those z/os TCP connections also be eligible for SMC-R? RAS and Security What happens if the 10GbE RoCE Express has a problem? What security is there for the 10GbE RoCE Express feature? How do I know that data traveling across the RDMA fabric is protected? How do I diagnose problems with SMC-R and RoCE? Positioning of RoCE with SMC-R versus OSA-Express and HiperSockets What differences exist between an OSA-Express OSD (QDIO) network fabric and a RoCE communication fabric? What differences exist between a HiperSockets network fabric and a RoCE communication fabric? Non-z/OS Support, Announcement Letters, and Statements of Direction What are the Announcement Letters referring to the 10GbE RoCE Express feature? What are the statement of directions from the July 2013 announcement concerning 10GbE RoCE Express? What operating systems support the 10GbE RoCE Express feature? Pricing What is the price of SMC-R? Formatted... [34] Formatted... [35] Formatted... [36] Formatted... [37] Formatted... [38] Formatted... [39] Formatted... [40] Formatted... [41] Formatted... [42] Formatted... [43] Formatted... [44] Formatted... [45] Formatted... [46] Formatted... [47] Formatted... [48] Formatted... [49] Formatted... [50] Formatted... [51] Formatted... [52] Formatted... [53] Formatted... [54] Formatted... [55] Formatted... [56] Formatted... [57] Formatted... [58] Formatted... [59] Formatted... [60] Formatted... [61] 1. What are RDMA and RoCE? 1.1. What is RDMA? 4

5 Remote Direct Memory Access (RDMA) is a protocol defined by the Infiniband Trade Association (IBTA). RDMA allows a host to write or read memory from a remote host without involving the remote host s CPU and Operating System (OS). It requires a specialized network adapter to transfer data as it bypasses Operating System layers and many communications protocol layers that are otherwise required for communication between applications. RDMA thus reduces software overhead, providing for highthroughput, low-latency networking. The IBTA states that RDMA has been shown to deliver value propositions that are not available through any other communications standards, including low latency, improved resource utilization, flexible resource allocation, scalability and unified fabric. 1 RDMA protocols can flow over an Infiniband (IB) transport or communication fabric which requires an investment in special network equipment. More recently, RDMA protocols can be used over the more commonly available Ethernet transport or fabric What is a communication fabric? A communication fabric comprises the transmission media required to create a communication link between two computing nodes in a network. The media may include physical components, like a cable, switch and network adapters. Examples are the physical Ethernet cables, Ethernet NIC adapter ports, and Ethernet switches that are used to interconnect nodes in a TCP/IP network. Another example is Infiniband (IB) technology where IB requires different cable, adapter, and switch types. Alternatively the communication fabric may include virtual components, like the firmware links and adapters of a communication fabric like HiperSockets on the z Systems What is RoCE? RDMA over Converged Ethernet (RoCE) is a standard that defines a new RDMA protocol over Ethernet. With advances in data center convergence over reliable Ethernet and Data Center Bridging (DCB), RoCE uses the efficient RDMA mechanism to provide lower CPU overhead and increase mainstream data center application performance at 10GigE and 40GigE link speeds and beyond. 2 The Ethernet fabric is comprised of Ethernet cables, special RoCE features or RDMA network Interface Cards (RNICs), and optionally Layer-2 Ethernet switches with or without IEEE Converged Enhanced Ethernet (CEE) capability. RoCE provides low latency, high bandwidth, high throughput, low processor utilization data transfer between hosts while taking advantage of existing Ethernet networks. InfiniBand Trade Association (IBTA) standardized RoCE in April SMC-R attempts to 1 Quoted from Page 1 of RDMA over Converged Ethernet (RoCE) Fact Sheet at 2 Quoted from Page 1 of RDMA over Converged Ethernet (RoCE) Fact Sheet at 5

6 provide faster time to value by allowing applications to reap the benefits of RDMA without requiring the applications themselves to change I ve heard of RoCE before. Is the one in the market today the same? The 10GbE RoCE Express and 10GbE RoCE Express2 features and theirits ports are equivalent to the RDMA Network Interface Cards (RNICs) discussed in the literature about RDMA. However the only current z Systems exploitation of the RoCE feature is by z/os V2R1+ using the new SMC-R communication protocol What is a communication protocol? A communication protocol is a set of rules for exchanging messages between communication partners. The protocol may have rules defining how to signal the initiation and termination of a message exchange, how to locate or address a partner for communication, how to detect errors in the communication and recover from them, how to adjust the communication rate to accommodate the receptiveness of the communication partner or the network. Examples of communication protocols are TCP/IP, X.25, SNA, APPN, APPN/HPR, SMC-R and so on What is the 10GbE RoCE Express feature? The 10GbE RoCE Express and 10GbE RoCE Express2 features areis a new PCIe based network adapters on z Systems that will provide access to RDMA technology over an Ethernet fabric that provides low latency and high network bandwidth with very efficient host CPU utilization. It was first available on the zec12 and zbc What is SR-IOV and how does 10 GbE RoCE Express feature use it? Any new news for the 10 GbE RoCE Express feature in the February 9, 2016 announcement? The good news is that, similar to the z13, a 10 GbE RoCE Express feature and 10 GbE RoCE Express2 feature can be shared across LPARs in the new z13s. Single Root I/O Virtualization (SR-IOV) for 10 GbE RoCE allows a PCIe device to appear to be multiple Formatted: Answer 6

7 separate physical PCIe devices. The SR-IOV specification is an industry standard specification to help promote interoperability. This provides the ability for that PCIe device to present multiple instances of itself up to the Operating System instance or hypervisor. RoCE continues to support SMC-R for z/os to z/os communication (z/os LPAR or 2 nd level guest running under z/vm). The IBM SMC-R Linux source code is pending implementationevaluation by the Linux open source community. The 10GbE RoCE Express has two ports. On z14, z14 ZR1, z13, and z13s the feature supports up to 31 VFs using both ports. On zec12 and zbc12, the feature supports only one port and no virtualization. The 10GbE RoCE Express2 has two ports and can support up to 63 VFs per port (126 per feature) on z14 and 31 VFs per port (62 per feature) on z14 ZR1. Formatted: Answer 1.8. How many ports are on the 10GbE RoCE Express feature? 1. Two 10 GbE SFP+ ports are provided on each 10GbE RoCE Express feature. The ports are numbered 1 and 2. The z/os exploitation of the physical RoCE ports is based on the z Systems RoCE support summarized as follows: 1. The zec12 (zbc12) provides "dedicated RoCE" support. With the dedicated support only a single OS instance (LPAR or 2 nd level guest running under z/vm) can use the card (no sharing). In the dedicated environment the z/os SMC-R exploitation is limited to using a single port (one of the two ports, whichever port is configured first in the z/os PROFILE.TCPIP file). 2. The IBM z13 and IBM z13s (z13 and z13s)later IBM Z systems provide "shared RoCE" support. With the shared (RoCE virtualization) support up to 31 OS instances (LPARs or 2 nd level guests) can use (share) the card. In the shared environment (RoCE virtualization) the z/os SMC-R exploitation (z13 with PTF) concurrently use both ports. The RoCE port number is configured in the z/os PROFILE.TCPIP file What is an RDMA-capable Network Interface Card? I ve heard of RDMA-capable Network Interface Cards (RNICs) or host adapters that are currently available in the industry. Is this existing technology similar to what you are announcing for z Systems? The IBM System z 10 GbE RoCE Express feature is an RDMA Network Interface Card. The RDMA Network Interface Card is sometimes referred to as an RNIC. 3 The exploitation of the 10GbE RoCE Express feature by z/os uses the SMC-R communication protocol. The IBM System z 10 GbE RoCE Express feature requires SMC-R for usage and requires a partner using the IBM System z 10 GbE RoCE Express feature. 3 See Controllers/BCM5709S for a reference to this acronym 7

8 2. What is SMC-R 2.1. What is Shared Memory Communications over RDMA (SMC-R)? Shared Memory Communication over RDMA (SMC-R) is a new sockets over RDMA communication protocol that allows existing sockets applications that exploit TCP to transparently benefit from RDMA for exchanging data over a RoCE network. The exploitation of SMC-R is transparent to sockets applications. This means that no application change is required. SMC-R provides host-to-host direct memory access without the traditional TCP/IP processing overhead. In addition to transparent application exploitation of RDMA, the SMC-R protocol allows customers to benefit from InfiniBand technology by leveraging their existing 10GbE Ethernet infrastructure. SMC-R also provides for critical qualities of services that are required by the Enterprise Data Center environment such as: When redundant network hardware paths are available, SMC-R offers both high availability and load balancing SMC-R offers preservation of existing IP topology and IP security SMC-R introduces minimal administrative and operational changes It provides dynamic discovery of partner RDMA capabilities and dynamic setup of RDMA connections over RoCE fabrics. SMC-R support is included as part of z/os V2R1 and later. SMC-R exploitation in z/os V2R1 and later requires the 10GbE RoCE Express feature Does the SMC-R exploitation of RoCE require special Ethernet (RoCE-capable) switches? No, SMC-R does not require a RoCE-capable switch. SMC-R requires a standard 10GbE switch 4. Auto negotiate to 1GbE is not supported. The Global pause frame (a standard Ethernet switch feature for Ethernet flow control described in the IEEE 802.3x standard) should be enabled on the switch. 5 4 SMC-R can also be exploited using direct RoCE Express to RoCE Express connectivity (without any switch). However, this type of direct physical connectivity forms a single physical point to point connection disallowing any other connectivity with other LPARs (e.g. any other SMC-R peers). While this is a viable option for test scenarios it is not practical (not recommended) for production deployment. 5 The 802.3x Pause feature is implemented differently with different switch vendors. Some implement the feature on a per-port basis; others on a global basis. Most switches have Global Pause enabled but you will want to verify with the switch vendor that this is the case on the switch that is part of the RoCE fabric. 8

9 2.3. How far apart can different servers be to communicate with each other using SMC-R/RoCE? The distance from the 10GbE RoCE Express or Express2 port to the 10GbE switch is limited to 300 meters with OM3 fiber cable. Minimizing the number of switches and distance between the servers will help maximize the latency reduction benefits of RDMA. This makes the technology especially well-suited for deployment within a datacenter. However, SMC-R and RoCE Express can be deployed across longer distances and still deliver significant latency, throughput and CPU benefits. In support of multiple site (ie. disaster recovery) configurations IBM has conducted SMC-R performance testing with 10GbE RoCE Express feature at longer distances (10km and 100km). In each case the support for the extended distances was provided by the switch optics (10km) and DWDM technology (100km). For additional information on these performance benchmarks refer to the SMC-R Reference Information SMC-R / RoCE at Distance : The IBM z Systems Wavelength Division Multiplexer (WDM) vendor product qualification process for Mulit-site Sysplex and GDPS solutions has been enhanced to include RoCE Express and SMC-R testing as part of the process. For more details on qualified WDM vendor products for RoCE Express and SMC-R refer to Resource Link: After you login select Library on the left side of the page. Under the heading Hardware products for servers in the middle of the page select System z Qualified Wavelength Division Multiplexer (WDM) products for GDPS solutions 2.4. Will I still need OSA once I exploit SMC-R using the 10 GbE RoCE Express feature or 10 GbE RoCE Express2 feature? Yes, the SMC-R protocol leverages your existing IP topology and the TCP/IP connection to control your SMC-R connectivity. There are numerous advantages to this architecture. Therefore you will still need to establish a standard TCP/IP connection over a QDIO OSA port and an Ethernet network to the peer. SMC-R uses this TCP connection to determine eligibility to exploit the RoCE fabric and then to build a point-to-point SMC-R link for the data flow over the RDMA connection path. The SMC-R link is built for the first eligible connection between two RoCE-enabled partners and the same link can be reused for subsequent TCP connections between the same peer hosts. TCP/IP is used not only to establish the TCP/IP and RDMA connections, but also for Keepalive functions and to terminate the TCP and the associated RDMA connections. The standard TCP/IP path over the QDIO OSA port need not be dedicated to RDMA usage; it can be used simultaneously for other, non-rdma traffic. The QDIO OSA port can be attached to either a 10 Gigabit Ethernet or a 1 Gigabit Ethernet network Is SMC-R used for anything other than together with RoCE? 9

10 SMC-R is only used together with the RoCE feature. Shared Memory Communications Direct (SMC-D) is supported for same CPC partners only and does not require a RoCE feature. Please see the SMC-D FAQ document at 10

11 3. Benefit of SMC-R and RoCE 3.1. What is the value I will get from SMC-R and the 10GbE RoCE Express feature or 10GbE RoCE Express2 feature? The value of RDMA technology is about providing significantly improved network performance for CPC to CPC communications. Network improvement attributes are typically described as latency, throughput, CPU cost and scalability. Improvements in network performance can potentially improve (increase) application workload transaction rates while reducing your CPU cost. The combination of the 10GbE RoCE Express feature or 10GbE RoCE Express2 feature and the SMC-R protocol provide for significant improvements in network performance. The network latency characteristics provided by this combined solution are compelling: network latency is typically expressed as network round trip time. This latency attribute can translate to an improved overall application transaction rate for z/os to z/os workloads. Workloads that are network intensive and transaction oriented (sometimes described as request/response workloads -- that require multiple and even hundreds of network ( client/server ) flows to complete a single transaction will realize the most benefit. An example of such a workload is a WebSphere application server communicating with a remote database server such as DB2. Another is CICS to CICS communications with IPIC. IBM internal benchmarking has shown up to 48% reduction in response time and up to 10% CPU savings for CICS transactions using DPL (Distributed Program Link) to invoke programs in remote CICS regions in another z/os system via CICS IP interconnectivity (IPIC) when using SMC-R vs standard TCP/IP. 6 Message sizes also impact the performance characteristics of a workload. Transactional workloads that exchange larger messages (e.g. web services such as WAS to DB2 or CICS) will see even greater benefit. IBM has seen a 40% reduction in overall transaction response time for WebSphere Application Server v8.5 Liberty profile TradeLite workload accessing z/os DB2 in another system measured in internal benchmarks. 7 Another benchmark we have done shows WebSphere MQ for z/os realizes up to 3X increase in messages per second it can deliver across z/os systems when using SMC-R vs. standard TCP/IP. 8 Finally, streaming (or bulk) application workloads (e.g. FTP) will also experience reduction in CPU consumption. Our internal measurements show up to 50% CPU savings for FTP binary 6 Based on internal IBM benchmarks using a modeled CICS workload driving a CICS transaction that performs 5 DPL calls to a CICS region on a remote z/os system, using 32K input/output containers. Response times and CPU savings measured on z/os system initiating the DPL calls. The actual response times and CPU savings any user will experience will vary. 7 Based on projections and measurements completed in a controlled environment. Results may vary by customer based on individual workload, configuration and software levels. 8 Based on internal IBM benchmarks using a modeled WebSphere MQ for z/os workload driving non-persistent messages across z/os systems in a request/response pattern. The benchmarks included various data sizes and number of channel pairs. The actual throughput and CPU savings users will experience may vary based on the user workload and configuration. 11

12 file transfers across z/os systems when using SMC-R vs standard TCP/IP. 9 Ultimately the actual performance benefits that can be achieved by each user are very unique to each application workload and other unique system environmental factors (CPU and memory utilization, network bandwidth, network path and congestion, etc.) For additional performance information refer to the SMC-R Performance White Paper found along with other SMC-R reference information at: Formatted: Font: (Default) Arial I have a single CPC within my data center. Does it make sense for me to exploit SMC-R? How will SMC-R and the 10GbE RoCE Express feature or 10GbE RoCE Express2 feature be positioned with HiperSockets? SMC-R is designed to provide high-performance networking over Ethernet (RoCE). RDMA also has distance implications. SMC-R is designed to provide HiperSockets-like performance across Ethernet (RoCE) for multiple CPCs. SMC-R is supported for any LPAR-to-LPAR configuration (same or different CPCs). Within a single CPC HiperSockets SMC-D would be your best performance option. Also note that on zec12 multiple LPARs cannot share a single 10GbE RoCE Express feature. If your CPC has Ethernet LAN access to another CPC then you are a candidate to benefit from SMC-R. Also refer to the next Question and Answer for information about how to evaluate your workloads for applicability to SMC-R I have multiple CPCs in my data center but I don't know if I have the type of or enough of the workloads with the network traffic patterns that could benefit from SMC-R and RoCE Express. I need help with building the justification and business plan to add RoCE Express to my datacenter. How can I determine if SMC-R applies to my environment and what level of benefit I might expect? IBM has provided a new tool called the SMC Applicability Tool (SMCAT). The SMCAT has been provided for z/os V1R13 and V2R1 (via PTFs). The tool does not have any dependency on RoCE or SMC-R. Instead SMCAT will monitor your TCP/IP traffic for a specified IP address, group of IP addresses or IP subnets and then produce a summary report describing the network traffic associated with the monitored addresses. The summary report will provide information about how much traffic (percentage) to/from those IP addresses is eligible for and well suited for SMC-R. SMCAT PTFs: 9 Based on internal IBM benchmarks in a controlled environment using z/os V2R1 Communications Server FTP client and FTP server, transferring a 1.2GB binary file using SMC-R (10GbE RoCE Express feature) vs standard TCP/IP (10GbE OSA Express4 feature). The actual CPU savings any user will experience may vary. 12

13 V1R13: APAR PI27252 PTF UI24872 V2R1: APAR PI29165 PTFs UI24762 and UI24763 V2R2 and later do not require any PTFs. SMCAT Overview document is available on the SMC-R reference material web site: The term Shared Memory Communications suggests that special memory might be required in the operating system. Are there any new or special memory requirements or memory considerations for exploiting SMC-R? No, there are no new or special memory hardware or operating system memory requirements for the exploitation of SMC-R. The memory used for SMC-R to exchange application data is managed by z/os Communications Server. The vast majority of the memory is managed by and owned by the TCP/IP stack and is allocated within TCP (ASID) 64-bit private, on behalf of socket application middleware. This storage is not CSMmanaged. Application middleware memory management is not affected or changed. Exploiting RDMA technology does require using fixed memory. The majority of this memory is a variable amount managed by the TCP/IP stack. A smaller (static) portion is managed by the device driver. The amount of fixed memory used for SMC-R can be planned for, monitored and controlled by the administrator (limited by the FIXEDMEMORY definition on IP Global Config statement). Refer to the SMC-R memory planning worksheet in the IP Configuration Guide (SC ) for planning the amount of fixed memory for your environment Does Flow Control as we know it for TCP/IP still exist with the SMC-R protocol? The SMC-R protocol defines its own version of flow control. The TCP receive and send buffers as defined in the TCP/IP stack indirectly influence flow control with SMC-R communications. The TCP receive buffer for an application is used to allocate a fixed size for the SMC-R receive buffer space which is called the RDMA Buffer Element (RMBE). At this point flow control mechanisms for SMC-R are used to avoid congestion in the data flows How is Flow Control handled when the data is flowing over the RoCE fabric? 13

14 Flow control over the RoCE fabric has two levels of flow control: 1. SMC-R (software) transport layer: Flow control mechanisms similar to the window sizing and congestion controls of TCP connections exist with the SMC-R protocol. The information for determining congestion is maintained in an RMBE control block that is housed in the memory assigned to SMC-R processing in the GLOBALCONFIG statement. RDMA write operations notify the peer with Consumer Cursor Updates to modify window sizes when necessary. The memory consumed for these Remote Memory Buffer Elements (RMBEs) can be monitored with a NETSTAT command. Consult the RFC for SMC-R in the IETF database for more information (RFC 7609). 2. RoCE Express and RoCE Express2 (hardware) Ethernet pause frames: The IEEE 802.3x pause frame (a standard Ethernet switch feature for Ethernet flow control described in the IEEE 802.3x standard) when enabled on the switch also provides an important level of flow control on the LAN. It is strongly recommended that Global Pause is enabled on your switch Will SMC-R and RoCE support SNA? No. SMC-R requires TCP Will SMC-R and RoCE support SNA with Enterprise Extender? No. SMC-R requires TCP, and Enterprise Extender relies on UDP. 14

15 4. Configuration 4.1. What is the feature code for 10GbE RoCE Express? The 10GbE RoCE Express is FC# GbE SR and the 10GbE RoCE Express2 is FC# Can the 10GbE RoCE Express feature be shared between multiple LPARs? The ability to share the RoCE Express feature is based on the level of your z System. 1. zec12 (zbc12): No. Each 10GbE RoCE Express feature can only be used by a single LPAR (or 2 nd level guest). Note that each z/os does support sharing among multiple TCP/IP stacks within the same z/os instance. 2. z13 and z13s: Yes, z13 and z13s provide RoCE Express virtualization using SR-IOV allowing up to 31 OS instances (LPARs or 2 nd level guests) to share the same RoCE Express feature (PCHID). Each concurrent user (OS instance) will configure a unique PFID which consumes a unique Virtual Function (VF). Note that for z/os, each stack that shares a RoCE Express feature will consume a unique Virtual Function. 3. z14 and z14 ZR1: Yes, z14 and z14 ZR1 provide RoCE Express virtualization using SR-IOV allowing up to 63 VFs per port on z14 and 31 VFs per port on z14 ZR1. Each concurrent user (OS instance) will configure a unique PFID which consumes a unique Virtual Function. Note that for z/os, each stack that shares a RoCE Express feature will consume a unique Virtual Function. Formatted: Font: (Default) Arial, Font color: Black 4.3. How is the 10GbE RoCE Express feature defined to the CPC? The 10GbE RoCE Express feature and 10GbE RoCE Express2 feature are is not defined as a CHPID and does not consume a CHPID number. Instead the 10GbE RoCE Express feature is defined with PCIe-based definitions called PCIe Function IDs or PFIDs. PFIDs are defined as part of an I/O configuration using the Hardware Configuration Definition (HCD) program. Each physical device including the RoCE feature has a PCHID (Physical Channel ID) which represents the physical location of the PCIe device in the PCIe drawer. The LPAR access and candidate lists are defined in HCD. With zec12 (zbc12) a RoCE Express feature is active to a single LPAR. With z13 and z13s, a RoCE Express feature can be active to (or shared among) multiple (up to 31) OS instances (LPARs or 2 nd level guests). Each OS instance is 15

16 configured with a unique FID (associated with the same PCHID). Each FID consumes a RoCE Virtual Function. Virtual Function IDs are also configured in HCD. RoCE supports up to 31 Virtual Functions per RoCE feature (PCHID). With z14 and z14 ZR1, a RoCE Express feature can be active to (or shared among) multiple (up to 63 per port on z14 and up to 31 per port on z14 ZR1) OS instances (LPARs or 2 nd level guests). Each OS instance is configured with a unique FID (associated with the same PCHID). Each FID consumes a RoCE Virtual Function. Virtual Function IDs are also configured in HCD How many 10GbE RoCE Express features will I need to have on my system? The answer depends on the level of your IBM z System. 1. zec12 (zbc12) supports RoCE Express in dedicated mode. In dedicated mode the RoCE feature cannot be shared. For redundancy each OS instance (LPAR or 2 nd level guest) should be provisioned with 2 physically unique RoCE Express features (unique PCHIDs). Up to 16 features are supported on zec12 or zbc z13 and z13s support RoCE Express in "shared mode". With "shared mode" (RoCE virtualization) each RoCE Express feature can be shared by multiple (up to 31) OS instances (LPARs or 2 nd level guests). Both 10GbE RoCE physical ports are available to all OS instances (i.e. providing up to 20GbE of bandwidth per feature). Up to 16 physical features are supported on the z13 and z13s. Access to each RoCE port requires a unique FID to be configured z14 and z14 ZR1 support RoCE Express in "shared mode". With "shared mode" (RoCE virtualization) each RoCE Express feature can be shared by multiple (up to 63 per port on z14 and up to 31 per port on z14 ZR1) OS instances (LPARs or 2 nd level guests). Both 10GbE RoCE physical ports are available to all OS instances (i.e. providing up to 20GbE of bandwidth per feature). Up to 8 physical features are supported on the z14 and up to 4 physical features are supported on the z14 ZR1. Access to each RoCE port requires a unique FID to be configured. Similar to sharing OSA-Express, an assessment will be necessary to determine the appropriate level of resource sharing for RoCE Express within your environment. For the systems (z/os images) that require access to RoCE there are two areas of considerations: a. System types: Systems with like usage characteristics can influence your ability to share physical resources. Factors such as security requirements/levels, type of systems (production vs. test), lines of business (sales vs. manufacturing), etc. b. Capacity planning: Aspects such as service level agreements, bandwidth requirements (i.e. bandwidth requirements for workloads within each OS instance (LPAR or 2 nd level guest)), periods of peak traffic loads, etc. Once you determine the level of RoCE Express sharing that is appropriate for 16

17 your environment (i.e. which OS instances could or should share features) then you can determine the number of required physical features. Note: This note is applicable to both the dedicated and shared RoCE environment. When provisioning you RoCE PCHIDs to each OS instance (LPAR or 2 nd level guest) it is recommended that you provide for full hardware redundancy (ie. provision unique PCHIDs). SMC-R will group the provisioned PFIDs into a logical group called an SMC-R Link Group. When more than 2 PFIDs have been provisioned to z/os, the SMC stack attempts to select 2 PFIDs that provide the best level of redundancy. The level of redundancy for given link group can be displayed with z/os TCPIP Netstat. To achieve full hardware redundancy for RoCE Express on z Systems the following recommendations are made, on a per unique physical network (PNet ID) basis: 1. Each OS instance (LPAR or 2 nd level guest) should be provisioned with 2 physically unique RoCE Express features (unique PCHIDs). 2. Each RoCE feature should be associated with a unique z PCIe internal path (be in a different Resource Group). Refer to I/O (HCD) configuration for internal path (I/O drawer) recommendations What are the design considerations for implementing the required OSA Express feature on behalf of SMC-R communications? An OSA port connection will still be required when implementing SMC-R communication. The OSA connection is used for the TCP/IP and SMC-R connection setup processing. The OSA connection can be a 1GbE or 10GbE QDIO connection. The OSA ports can be connected on the same 10GbE switch as the 10GbE RoCE Express feature but it doesn t have to be. The OSA and RoCE ports must be associated with (connected to) the same physical network (determined by your defined PNET IDs configured in HCD). The OSA connection must be a Layer-2 connection. This means that the physical interface endpoints of the path must reside in a single subnet. However the TCP connection endpoints may be Virtual IP Addresses (VIPAs) in a different subnet. The OSA does not have to be dedicated to the OS instance (LPAR or 2 nd level guest); the OSA may be shared with other OS instances. The other OS instances may even include the other [SMC-R/RoCE] partner if the two partners are on the same z System platform (CPC) What operating systems support this new technology? z/os V2R1 and later, contains support for the SMC-R protocol and exploitation of the 10GbE RoCE Express feature on the zec12 or zbc12 (or later). z/os V2.1 requires a PTF for z13 RoCE shared mode. 17

18 z/vm V6R3, contains guest exploitation of the 10GbE RoCE Express feature. z/os guests running under z/vm are able to use the RoCE features. Other z/vm guests are not able to exploit the RoCE features because they do not have SMC-R support. For z13 z/vm V6R3 requires APAR VM The IBM SMC-R Linux source code is pending evaluation implementation by the Linux open source community. I understand that two adapter types are required to exploit RoCE: The QDIO OSA port to support the SMC-R protocol and The RoCE adapter port (the RNIC) to support the data traffic How are these two types of adapter ports associated with each other in the HCD and software configurations? The RoCE feature is assigned a PCHID in the HCD together with a PFID (PCIe Function ID). A QDIO OSA adapter is assigned a PCHID together with a CHPID type of OSD. An additional definition in HCD is required to implement RoCE, the assignment of a Physical Network ID (PNet ID). A unique PNet ID is assigned through HCD to the RoCE feature ports and OSA card ports that are to cooperate with each other in establishing a RoCE connection to a peer. The operating systems dynamically associate the eligible groups of RNICs and OSA ports into groups assigned the same PNet ID. When the adapter types are activated in the operating system, any VLAN IDs defined in the operating system or TCP/IP are inherited by the associated RoCE features. Also see question 4.8 in this document When deploying SMC-R are there any VLAN Ethernet switch configuration considerations? Yes, when considering your VLAN Ethernet configuration options for SMC-R / RoCE the following should be noted: Trunk Mode: When your OSA switch ports are configured in trunk mode, then your RoCE switch ports must also be configured in trunk mode and enabled for all associated OSA VLANs. Access Mode: When your OSA switch ports are configured in access mode, then your RoCE switch ports must also be configured in access mode within a single VLAN. For additional information refer to SMC-R Reference Information SMC-R / RoCE VLAN Considerations : If the OSA to be associated with the RoCE feature is being used for other traffic, do I 18

19 need to stop that traffic to add the PNET ID to the OSA? Dynamic I/O Reconfiguration may be used to define PNET ID on an OSA that is in use. In order to pick up the change the OSA will require a CHPID OFF and then CHPID ON (from every LPAR sharing it). If there is normal dynamic fail over, rerouting, or retransmission, then the current connections should not be disrupted Are the two physical ports on the RoCE feature and the two ports on an OSA CHPID similar?... They may look similar but the OSA ports are labeled and defined (in TCP/IP) as port 0 and port 1. The RoCE feature ports are labeled port D1 and port D2. The RoCE feature physical port labeled D1 is defined (in TCP/IP) as port 1, not port 0. The RoCE feature physical port labeled D2 is defined (in TCP/IP) as port 2. Also see question What MTU is used by the RoCE feature? RoCE feature MTU may be specified in the z/os PROFILE.TCPIP file on the GLOBALCONFIG SMCR statement. z/os only supports MTU of 1024 (1K default) or 2048 (2K). The 1024 value will be used even if the 2048 value is defined on both sides unless the switch ports have Jumbo Frames enabled. Only modest benefit has been seen with the larger size so far, therefore the default setting is not seen as a draw back What are the possible FID or PFID values? The supported FID values are based on the level of your z System. 1. zec12 (zbc12): In HCD the Function FID value may be 000 to 0FF. In the z/os PROFILE.TCPIP on the GLOBALCONFIG SMCR statement the PFID value may be 0000 to 00FF. The two definitions must match (ignoring the leading zeros). 2. z13 and z13s and later: In HCD the Function FID value may be 000 to FFF. In the z/os PROFILE.TCPIP on the GLOBALCONFIG SMCR statement the PFID value may be 0000 to 0FFF. The two definitions must match (ignoring the leading zeros). 19

20 5. Application Questions 5.1. What application traffic is supported using SMC-R with RoCE? With the exception of IPsec traffic, all other TCP/IP traffic that could flow between QDIO OSA adapter ports on two z/os V2R1 or later partners and that is found to be RDMAeligible during the initial TCP setup, can exploit the RDMA connection. Connection level security such as SSL or AT-SSL can be used with SMC-R. Examples include but are not limited to -- WAS type 4 connections, Cognos (z/os) to DB2 connectivity that goes cross- LPAR, CICS to CICS via IPIC, WebSphere MQ, etc. Sysplex Distributor traffic is also supported. If the connection is usable it will be used and there is no required change needed for applications What are the SMC-R protocol connection eligibility requirements for two peer hosts to connect using SMC-R? In order to be eligible to connect using SMC-R the following must be true about the two peer hosts network connectivity between two peer hosts. Each host must have: Direct network access to the same physical Layer-2 network (same physical LAN) Direct network interface configured with the same IP Subnet (for IPv4 or IP mask for IPv6) and VLAN ID (if VLAN is applicable) Both hosts must also have access to an RNIC and be enabled for SMC-R. Also see question 4.8 in this document Can two hosts that are in unique IP security zones that must traverse a network firewall exploit SMC-R? No, RDMA is not IP routable. Both peer hosts must be within the same Layer-2 IP network What are the steps required to use SMC-R with RoCE between two partner z/os servers? Prerequisites for the establishment of an SMC-R/RoCE connection: The communication partners must be at z/os V2R1 or later on an eligible z Systems hardware platform. 20

21 The TCP route selected at both ends of the connection must lead over an interface for a QDIO OSA port that has been enabled for SMC-R. The QDIO OSA adapters and the RoCE features at an end of the connection must have been associated with each other using the same Physical Network IDs (PNet IDs). SMC-R must be enabled on the GLOBALCONFIG statement with the same PFID and on the OSA OSD Interface statement(s) in the PROFILE.TCPIP file on both partners. The partnering OSA adapter ports and RoCE ports at both ends of the RoCE connection must be using the same IP subnet and VLAN ID if one has been assigned. (Note: VLAN IDs are only necessary if more than one VLAN ID is required across the SMC-R link.) Subnet mask must be configured on the OSA interface that is partnering with the RoCE ports. Also see question 4.8 in this document What changes do I need to make to the applications to take advantage of RoCE and SMC-R? There are no changes required. The application is not involved in the decision to use SMC- R or not. This is all handled inside z/os Communications Server code Can I enable SMC-R but exclude some applications from using it? Yes, for applications that act as the TCP server you can configure NOSMCR on the port definition Can I communicate memory to memory between my distributed system and my z System such as a SAP application server and the z/os DB2 database? No. The only operating system that has announced support for the SMC-R protocol is z/os. So the offering is only for z/os to z/os communication How does Sysplex Distributor support SMC-R with RoCE? If a z/os Sysplex Distributor (SD) client has a RoCE connection to the selected z/os Sysplex Distributor target, and VIPARoute has been enabled (to send the traffic over the OSAs), then the connection is eligible to use SMC-R. In this case both outbound and inbound traffic will flow directly to and from the target and client over the RoCE connection. Thus even the inbound flow, which normally traverses the SD node, bypasses the SD route to provide better throughput directly to the target. 21