DB2 purescale: High Performance with High-Speed Fabrics. Author: Steve Rees Date: April 5, 2011

Transcription

1 DB2 purescale: High Performance with High-Speed Fabrics Author: Steve Rees Date: April 5, IBM 2011 Copyright 1

2 Agenda Quick DB2 purescale recap DB2 purescale comes to Linux DB2 purescale and RoCE Multi-HCA for increased capacity Some futures Challenges 2

3 Disclaimer: Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

4 Introducing DB2 purescale Virtually Unlimited Capacity Buy only what you need, add capacity as your needs grow Application Transparency Avoid the risk and cost of application changes Continuous Availability Deliver uninterrupted access to your data with consistent performance 4

5 DB2 purescale : Technology Overview Leverage System z Sysplex Experience and Know-How CS Secondary Clients Single Database View Member Member Member Member CS CS CS CS IB Cluster Interconnect Log Log Log Log Shared Storage Access Database CS Primary Clients connect anywhere and see a single database Clients connect into any member Automatic load balancing and client reroute may change underlying physical member to which client is connected DB2 engine runs on several host machines Co-operate with each other to provide coherent access to the database from any member Low latency, high speed interconnect Special optimizations provide significant advantages on RDMA-capable interconnects (eg. Infiniband. RoCE) Cluster Caching Facility (CF) from STG Efficient global locking and buffer management Synchronous duplexing to secondary ensures availability Data sharing architecture Shared access to database Members write to their own logs Logs accessible from another host (used during recovery) Integrated cluster services Failure detection, recovery automation (TSA / RSCT) Cluster file system (GPFS) 5

6 Sidebar: Send/Receive vs. RDMA Send / Recv (TCPIP socket) Send / Recv (Reliable Datagram Socket - RDS) send msg send msg App Krnl Copy App NIC DMAs from kernel buffer NIC DMAs from kernel buffer NIC DMAs into kernel buffer & raises interrupt NIC DMAs app buffer & raises interrupt Krnl App Copy App recv msg Kernel int handler schedules receiving proc/thread recv msg Kernel int handler schedules receiving proc/thread RDMA RDMA write App RDMA NIC DMAs directly from app memory RDMA NIC DMAs directly into app memory App recv msg 6 Receiver spins looking for msg 6

7 purescale scales with RDMA & udapl RDMA exploitation via udapl over low latency fabric Enables round-trip response time ~10-15 microseconds Silent page invalidation Informs members of page updates Requires no CPU cycles on those members No interrupt or other message processing required Increasingly important as cluster grows Hot pages available to members from GBP memory without disk I/O RDMA and dedicated threads enable read page operations in 10s of microseconds Lock Mgr Lock Mgr Lock Mgr Lock Mgr Buffer Mgr Can I have this lock? Yup, here you are. New page image Read Page worker threads GBP GLM SCA 7

8 DB2 purescale on Linux Cluster IB Interconnect Log Log Log Log Shared Storage Access Database Customer demand to broaden platform base from initial release on Power AIX Respond to Linux 'sweet spots' for deployment Address customer skill focus areas Easy 'port' in use for years internally for development of purescale Differences in OFED delivery by distro created some challenges Introduced 2010 IBM SystemX systems (x3650, x3850, x3690) Mellanox ConnectX-2 QDR IB SLES 10.3 / RHEL 5.5 8

9 purescale & Linux & QDR IB Cluster IB Interconnect Log Log Log Log Shared Storage Access Database Big data movement happens, but small message latency is king Throughput boost in going to QDR a big win for latency? For this workload Message response time gets a nice DDR, QDR Overall workload TPS improvement well damped by other factors Normalized CF Message Response time 100 Normalized Average Application throughput 110 percent of SDR SDR DDR QDR percent of QDR average TPS 100 SDR DDR QDR 0 90 Read Write Lock 9

10 purescale & Infiniband perfect, right? Yes well, almost IB is mature IB has obvious technical strengths IB is a great fit for a high-performance clustered database But Some customers are hesitant to deploy a new network type, however wonderful it is 10

11 purescale & RoCE IB Ethernet is really mature Ethernet is ubiquitous IB RDMAoE is a great fit for high-performance clustered database too RoCE support added in DB2 purescale Mellanox ConnectX-2 10Gb EN + PFC switch OFED Initially SLES 10 11

12 purescale & RoCE Performance Q: How visible is the difference in nominal bandwidth between 40Gb QDR IB vs 10Gb EN for an average application? A: Not very comparable performance to IB makes even 10Gb Ethernet a viable option for many customers Noticeable at message level Average tps at application level within 5-10% Normalized Median CF Message Response time Normalized Average Application throughput percent of QDR Message rsp time Read Write Lock QDR EN percent of QDR average TPS tps QDR EN

13 purescale & Ethernet futures Strong customer interest encouraging wider support in future purescale releases Cards, vendors, distros, platforms etc. Looking forward to common availability of 40 Gb EN to close gap with QDR IB Larger workloads shipping very large data volumes benefit from the greater throughput And what about iwarp? 13

14 Multiple CF HCAs Low latency to the CF ensures high performance for purescale Duplexed primary & secondary CFs already avoid SPoF Very heavy workloads and/or very large clusters could overload the IB / RoCE HCA at the CF Member HCA Member HCA HCA Primary CF IB Switch Member HCA HCA Sec'dary CF Member HCA Multiple CF HCAs in beta fall/

15 Example purescale + banking app Transactional Banking Application 20 HS22 blades Application Server Application Server 2 x3850 X5 CFs - (64c, 256 GB) 4 x3850 X5 members - (64c, 256 GB) 1 IB HCA per member 4 IB HCAs per CF DB2 purescale Prim CF DB2 purescale Member DB2 purescale Member DB2 purescale Member DB2 purescale Member DB2 purescale Sec CF DB2 v98 purescale cluster Cluster Storage DS

16 Near Linear 1-4 members Millions of items processed per hour as function of number of members

17 purescale futures 'stretched' clusters Splitting the purescale cluster over two sites offers some disaster resistance Fire, power or communication outage, etc. N km M1 M3 CFP CFS M2 M4 Site A Site B Must be able to stretch RDMA over long distances Currently testing with Obsidian Longbow IB extenders Obvious implications from finite speed of light 17

18 Stretching the purescale cluster Infiniband Infiniband switch Single site cluster Mbr 1 Mbr 3 CF pri CF sec Mbr 2 Mbr 4 Log Log Log Log GPFS Shared Storage (logical view) Ethernet Fiber Channel Database Physical Storage Infiniband + IPoIB Dark Fiber or WAN Obsidian 'Longbow' IB range extender Potential 'stretched' cluster config Mbr 1 Mbr 3 CF pri N km CF sec GPFS Shared Storage (logical view) Log Log Log Log Mbr 2 Mbr 4 Fiber Channel Database Physical Storage site 'A' GPFS replication between sites Physical Storage site 'B'

19 Challenges / observations re: RDMA fabrics Inconsistent OFED implementations / packaging across platforms / distros Impediment to porting & commercial DC adoption RDMA transports can be challenging to manage Integration with management stacks & basic utilities needed OpenView, Tivoli, even netstat Improving with Ethernet-based implementations Still rough edges around OS & stack integration outside of HPC deployments High demand for well-supported virtualization on Linux SR-IOV, KVM, VMware Moving in that direction, but not there yet 19

20 Challenges / observations re: RDMA fabrics Growth of transport bandwidth Gb/s is goodness, but small message latency is what really counts in many cases Adapter bonding required for greater reliability & capacity 20