Accelerating SAP Sybase IQ Performance with Bay Storage High-performance and Low-cost Data Access Technology

Transcription

1 Accelerating SAP Sybase IQ Performance with Bay Storage High-performance and Low-cost Data Access Technology September 2013

2 Table of Contents Summary...3 Solution Overview...3 SAP Sybase IQ Bay Storage Technology - BayStor...5 Comparison of Storage system Technologies...5 HDD Array...5 SSD Array...5 PCIe Flash Card... 6 BayStor RAMdisk... 6 Storage Performance Summary... 6 Test Scenarios... 6 RLV Merge Test... 6 Analytic Query Test...7 Results Observed... 8 RLV Load/Merge... 8 MPX Analytic Queries... 9 Differences between results in the Two Test cases...10 Conclusion...10 ResourceS...10 Acknowledgements...10 Appendix System Configuration...11 Appendix Analysis of RLV Merge Test Results...12 Appendix Analysis of Analytic Query Test Results

3 SUMMARY Fast time to results for Big Data requires advances in server technology such as large memory configurations, and multi-core processors for computational parallelism. New generation, high-performance data access technologies can also radically improve complex query and high rate transactional processing performance in both scale up and scale out database deployments. SAP and Bay Storage Technology recently conducted a joint evaluation of the benefits of high-performance, yet costeffective data access. Although there are many available solutions that provide high IOPS (I/O operations per second), Bay Storage s BayStor technology provides scalable, network attached access to block data with under 10 microsecond latency for a 4KB block. This is 500X faster than enterprise disk arrays, 15X faster than the best external Flash arrays, and 5X faster than the best internal, non-network attached Flash cards. In addition, BayStor provides high performance network access for any standard disk, SSD, Flash or SAN technology, with the ability to provide a caching tier to improve data access down to less than 10 microseconds. BayStor is offered as a software product and runs on standard server systems. This white paper describes the results of a joint effort between SAP and Bay Storage Technology to evaluate the performance of various data access technologies with SAP Sybase IQ 16. Several technologies were selected for evaluation with various bandwidth, IOPS, latency and cost characteristics. Flash-based technologies provide high IOPS, and DRAM (dynamic random access memory) solutions additionally provide the lowest latency and best random access performance. Two different use cases were evaluated using arrays of hard drives, arrays of solid state drives, a PCIe Flash card and DRAM cache. Various deployment alternatives were evaluated, including locally connecting the storage devices within the server hosting SAP Sybase IQ itself, and network attaching the storage devices with the BayStor technology. We found that the highest performance was achieved for both use cases with either local arrays of solid state drives, or networked BayStor technology to provide arrays of solid state drives or DRAM. In a write intensive use case with BayStor, performance increased by a factor of 6.7X, and for a read intensive use case, performance increased by a factor of 1.8X over the locally attached hard drive array. Lowering I/O latency also greatly improves CPU utilization, and removes processing bottlenecks. SOLUTION OVERVIEW SAP Sybase IQ 16 SAP Sybase IQ is a high performance, analytic database engine. It was the first commercial column storage engine; the first to put a major emphasis on data compression; and one of the earliest to place a focus on complex queries and analytics, rather than on online transaction processing. It has a patented column indexing subsystem, and is particularly adept at executing ad hoc queries. SAP Sybase IQ is an open platform with standard ODBC and JDBC interfaces, and offers a rich SQL language. SAP Sybase IQ runs on commodity hardware and a variety of operating systems. SAP Sybase IQ keeps its database on disk, and has large caches for retaining data in memory as much as possible. The SAP Sybase IQ database is structured as a set of stores: the catalog stores holds metadata, the main store holds user data, and the temporary stores hold intermediate results. Each store is implemented as a set of logical data containers called DBSpaces. SAP Sybase IQ may operate in a single server configuration called a Simplex, and for scale-out processing it has a grid deployment option called a Multiplex. SAP Sybase IQ Multiplex has a massively parallel processing shared everything architecture. User queries can distribute across all the nodes in a Multiplex cluster, and the IQ server dynamically rebalances CPU and memory use as concurrent queries enter and exit the system. In a Multiplex, one node is the coordinator and manages transaction IDs for writeable transactions. The other nodes can be either readers may execute queries only, or reader/writers may execute queries or update transactions. Nodes may be grouped into elastic sets called logical servers to support multi-tenancy requirements, where workloads need to be segregated from each other within the same server environment. SAP Sybase IQ s shared everything architecture is augmented with a data affinity feature. With data affinity, the query engine partitions work intelligently, and assigns distributed query processing work to nodes that already have the pertinent data in cache. Caches stay warm, and load on the shared store is reduced: 3

4 IQ Server IQ Server IQ Server IQ Server IQ Server Scale Out Data Affinity Data Affinity Data Affinity Data Affinity Data Affinity Scale Out Elastic Logical Server Elastic Logical Server High Speed Interconnect to Shared Store Figure 1: SAP Sybase IQ Multiplex Architecture SAP Sybase IQ 16 also incorporates a write optimized, in memory delta store, called the RLV (row level versioned) store. The RLV store supports high velocity data loading to the database with the following benefits: Continuous analytics over operational data High velocity, concurrent data modifications Ability to exploit large memory and core footprints The RLV store is fully recoverable with its own transaction log. Data can be loaded at high speed from multiple concurrent workstreams into the RLV store in memory, and then the data migrates asynchronously into the main store on disk. The query engine interacts transparently with the two stores through a consistent view that wraps them as a unified data repository: IQ DML Engine Load/Insert/ Update/Delete Hybrid Table Storage Write Ahead/ Commit In Memory RLV Store Lock Manager IQ Query Engine Consistent Row ID View RLV Store Rows Main Store Rows Periodic Merge RLV Store Transaction Log Transaction Manager Row ID View Manager Version Manager On Disk Main Store Figure 2: SAP Sybase IQ RLV (Row Level Version) Store 4

5 Bay Storage Technology BayStor BayStor is an innovative, software-based technology for deploying network attached DRAM and tiered storage on x86 commodity servers. DRAM can be used directly as an ultra-high performance store (RAMdisk mode), or as an acceleration cache for standard storage devices (Cache mode). BayStor can be used for network attachment of any storage devices with or without the DRAM caching. Since BayStor is seen by Linux systems as a standard block device, I/O intensive applications are accelerated without modification. The following diagram shows the overall BayStor architecture, including the capability to scale capacity along with bandwidth and IOPS. Scale Capacity, Bandwidth and IOPS Block Device Interface BayStor Applicance Customer Application Server(s) Fabric InfiniBand Ethernet DRAM BayStor Applicance Software 2.5 Drive Bays SSD HDD SAN Gateway Performance Persistence/Capacity Application <10 usec Figure 3: BayStor Appliance Architecture BayStor technology was selected for this joint investigation because of its ability to deliver ultra-low access latencies along with high IOPS and data transfer bandwidth for random data access. For this evaluation, each BayStor appliance was network attached via InfiniBand 56Gbps FDR links through an InfiniBand switch. COMPARISON OF STORAGE SYSTEM TECHNOLOGIES This section describes the storage technologies that were evaluated, and highlights the particular benefits of BayStor. HDD Array The local hard disk drive arrays (HDD Array) were used for baseline measurements. As expected, access latencies are dominated by the physical spin of the magnetic platters and the movement of the read/write arms. Access latency ranged from 2 to 15 milliseconds. Although sequential bandwidth was relatively good for the large array, random access bandwidth suffers greatly from the dependency on physical movement. For the baseline evaluation, locally attached arrays of hard drives were used, configured as a single Linux md device. Drives were Western Digital 750GB 7200 RPM SSD Array The solid state drive array (SSD Array) with current generation Flash technology performs dramatically better than the HDD Array in all respects, consistent with its emergence as a very important data center storage technology. The array of SSDs was selected for this evaluation as it presents a very cost effective way to deploy Flash technology. The Linux md facility was used to consolidate the drive arrays. Drives were 240GB Samsung SM843 SSDs, overprovisioned to 200GB to enhance endurance. 5

6 PCle Flash Card A PCIe Flash Card was evaluated for one of the use cases since it represents a popular way to accelerate performance. Such cards provide relatively low latency. However, they are more expensive than an array of solid state drives. A FusionIO iodrive2 750GB MLC card was used for this evaluation. BayStor RAMdisk For any random access measures, BayStor s unique RAMdisk technology provides substantial advantage. It offers less than 10 microsecond latencies for 4KB blocks, and provides superior bandwidth and IOPS. Linux md was used to consolidate multiple RAMdisks. Storage Performance Summary Here are the performance details for each storage technology: 4KB Random 1 MB Sequential Latency (µsec) Bandwidth (MB/sec) IOPS Bandwidth (MB/sec) Read Write Read Write Read Write Read Write Local HDD Array (2) 15,100 4, Local HDD Array (8) 12,500 2, , ,300 1,200 Local SSD Array (6) ,100 1, , ,000 3,200 2,800 BayStor SSD Array (8) ,000 95,000 3,400 3,600 BayStor RAMdisk (4) 9 8 3,700 3, , ,000 2,900 3,400 BayStor RAMdisk (1) 9 8 2,400 3, , ,000 2,500 5,000 BayStor PCIe Flash Card (1) , ,000 1, An important point to note is that BayStor provides network attached storage with essentially the same application performance as locally attached storage. And BayStor s network attached model offers significant operational benefits over locally attached storage for virtualized environments, where resource sharing and management flexibility to move and reallocate storage resources are essential. TEST SCENARIOS Two test scenarios were evaluated. The first use case involved merging high velocity, low latency transactional updates from SAP Sybase IQ s in-memory delta (RLV) store into its disk-based store. The second use case focused on shared data store access by high-throughput analytic queries distributed across an SAP Sybase IQ Multiplex grid. For each test scenario, baseline tests were run with various SAP Sybase IQ 16 DBSpaces deployed on conventional hard disk arrays, and then rerun with these DBSpaces deployed on BayStor low-latency appliances and other technologies. No SAP Sybase IQ related performance tuning was done for any of the tested technologies. The system configuration for each test is described in the appendix, System Configuration. RLV Merge Test The RLV Merge feature was selected as one of two test scenarios to investigate the benefits of low-latency data access. A TPC-C transaction workload was used to drive this test scenario: TPC-C Load testing 1. Bulk load database with 150M records, except for order_line 2. Bulk load database with 1.5G order_line records 3. RLV load additional 150M order_line records 4. Perform RLV Merge 6

7 Steps 3 and 4 were driven by 24 TPC-C threads. The SAP Sybase IQ 16 and the TPC-C workload generator ran on the same server. Several I/O configurations were evaluated as shown in the diagram below. The SAP Sybase IQ DB Catalog and RLV DBSpace were deployed on a local HDD array in all cases, because the amount of I/O to them was relatively low. SAP Sybase IQ s main and local temporary DBSpaces were evaluated on several technologies as show below. SAP Sybase IQ parameters were kept the same, and no device specific tuning was done. Performance was reported by the TPC-C workload generator. IQ + RLV DB Catalog RLV Log IQ Main Database Temp DB Space Local HDD Array Local HDD Array (8) Local SSD Array (6) Network BayStor-RAMdisk (4) Network BayStor SSD Array (6) Network BayStor PCIe Flash Card Figure 4: Set up of RLV Test Analytic Query Test To evaluate the benefits of low-latency data access, a TPC-H query workload was run against a scale 700 (700 GB) database. This was run using three servers as shown in the diagram below. One of the servers was the SAP Sybase IQ 16 Multiplex (MPX) coordinator, and the other two servers were SAP Sybase IQ 16 MPX readers. The three servers shared an InfiniBand network attached SAN comprised of an array of 12 solid state disks providing storage for the IQ Main Database and Shared Temp DBSpaces. Each server had its own SAP Sybase IQ local temporary DBSpace. For the SAP Sybase IQ MPX coordinator, this was provided by a local direct attached pair of hard drives. For the SAP Sybase IQ MPX readers, we evaluated several local temporary DBSpace configurations as shown in the diagram below. A total of six TPC-H query streams were processed on the first server. Three query streams came from a workload generator on the first SAP Sybase IQ MPX reader, and the other three query streams came from a workload generator on the second SAP Sybase IQ MPX reader. Performance was reported by the TPCH workload generator on the first server. 7

8 IQ Main Database SharedTemp Space InfiniBand Attached SAN with 12 SSDs IQ MPX Coordinator IQ MPX Reader 3 3 IQ MPX Reader TPC-H Query Streams LocalTemp Space LocalTemp Space LocalTemp Space Local HDD Local HDD Array (2) Local SSD Array (6) Network BayStor SSD Array (6) Network BayStor PCIe Flash Card (1) Network BayStor RAMdisk (1) Local HDD Array (2) Figure 5: Set up of MPX Test RESULTS OBSERVED Results were collected via a combination of internal IQ reporting, TPC workload generator reporting, and the Linux commands time, vmstat and iostat. Linux blktrace/blkparse utility commands collected the data access traces. RLV Load/Merge The performance metric used for this evaluation was the overall processor time for RLV to load and merge 150M order_ line records into the pre-populated database. The overall results are shown below lower is better: Figure 6: RLV Test Results with Local HDD Array included 8

9 The performance of both the local and network BayStor attached SSD arrays, and the network BayStor RAMdisk were essentially the same. Each was 6.7X faster than the local HDD array. The HDD array is removed from the following diagram to more easily see the differences among the remaining technologies. The network attached PCIe Flash card was about 25% slower than any of the other technologies shown below: Figure 7: RLV Test Results with Local HDD Array not included Analysis of these results is described in the appendix, Analysis of RLV Merge Test Results. MPX Analytic Queries The results for the TPC-H query workloads running the MPX readers are shown below lower is better. Any of the Flash and DRAM technologies performed 1.8X better than the local hard drive array. Figure 8: MPX Test Results Analysis of these results is described in the appendix, Analysis of Analytic Query Test Results. 9

10 Differences Between Results in the Two Test Cases In the RLV merge use case, performance improved by a factor of 6.7X, and for the analytic query workload, performance improved by a factor of 1.8X over the locally attached hard drive array. The difference between the performance improvements observed in the two cases is attributed to the following: RLV/tpcC has relatively more writes than reads as compared to MPX/tpcH. If the overall performance is sensitive to write commit latency, then RLV would see a larger gain. MPX/tpcH is a read intensive workload (compared to RLV/tpcC). It may be easier for SAP Sybase IQ to hide read latency for this throughput oriented workload, therefore there would be less sensitivity to latency. For both test scenarios, significant performance improvements were observed with BayStor. CONCLUSION In addition to memory capacity and multi-core servers, I/O data access performance is critical for the performance of large scale data processing environments. This targeted benchmark proves that BayStor technology can dramatically improve performance at a favorable cost factor, across a variety of workloads running on SAP Sybase IQ s scalable analytics platform. High performance levels were achieved without requiring any software modifications or tuning, just by using cost-effective locally attached arrays of solid state drives, or network attached solid state drive arrays via Bay Storage Technology s technology. Using BayStor instead of a locally attached hard drive array, the following performance improvements were observed: write-intensive RLV merge use case: performance improved by a factor of 6.7X read-intensive analytic query workload case: performance improved by a factor of 1.8X RESOURCES SAP Sybase IQ: SAP web site: SAP Sybase IQ: Documents and Whitepapers for SAP Sybase IQ: SAP-Sybase Makes Running Business Critical Databases on vsphere Hassle Free: alliances/2012/06/vmware_sybase_sap.html Bay Storage: Bay Storage Technology web site: ACKNOWLEDGEMENTS The technical participants in this joint technology evaluation were Abhay Kale and Dhimant Chokshi from SAP; Lloyd Dickman and Jeff Rubin from Bay Storage Technology. 10

11 APPENDIX SYSTEM CONFIGURATION SAP Sybase IQ 16 SP1 (support package 1) was run on these servers: RHEL core x86-64 servers Dual Intel E processors, 2.6 GHz with turbo enabled, no hyper-threading 384GB DDR DRAM PCIe Gen 3 The RLV merge scenario used an SAP Sybase IQ single server configuration. Following best current practices, the SAP Sybase IQ configuration was as follows: Main Cache 50GB (gigabyte) Temp Cache 50GB RLV memory 45GB Main, Temp and RLV DBSpaces consumed 1 raw device each, for a capacity of 1TB (terabyte) All other configuration parameters were not modified The analytic query testing scenario involved an SAP Sybase IQ Multiplex with 1 coordinator node and 2 reader nodes. All servers had an identical configuration. Main and Shared Temp DBSpaces were on shared storage, as required for Multiplex. Local Temp DBSpaces (3 in all) were private storage devices, 1 for each node. The configuration for all three nodes was as follows: Main Cache 80GB Temp Cache 80GB Main, Temp and Shared Temp DBSpaces consumed 1 raw device each, for a capacity of 1TB In all, 5 raw devices were used All other server configuration parameters were not modified from their default values The SAP Sybase IQ server configuration file for the RLV Merge Test was set as follows: -n tpcc -x tcpip{port=5788} -gc 20 -gd all -gl all -c 1g -gp gm 100 -iqmc iqtc iqrlvmem The SAP Sybase IQ server configuration file for the Analytical Query Test was set as follows: -n tpch_15_x_abhay_r_b3 -x tcpip{port=5788} -c 500M -gd all -gm 25 -gc gr gp tl 0 -iqmc iqtc

12 APPENDIX ANALYSIS OF RLV MERGE TEST RESULTS Looking beyond the bottom line results, we see from the vmstat graphs below that using the HDD Array, iowait time (green line) is about 40% for half of this run. IQ processor utilization (blue line) is mostly below 20% for half of the time and very high for the remainder of the time. Each period of high processor utilization is preceded by a large spike in write activity. We conclude that SAP Sybase IQ performance is severely throttled by the performance of the HDD Array. Figure 9: Processor Utilization and I/O Wait Times for HDD Array in RLV Test The vmstat graph below for the BayStor RAMdisk technology shows I/O wait (green line) largely eliminated. Processor utilization (blue line) is in the range % for the majority of the run. Note that the BayStor technology can absorb the write spikes much better than the HDD array. IOPS peaks of 450K are seen: Figure 10: Processor Utilization and I/O Wait Times for BayStor SSD Array in RLV Test 12

13 The following graph of SAP Sybase IQ s storage access patterns may help shed light on the underlying reasons for the performance differences. The graph is the result of tracing all IQ storage accesses during the RLV Load/Merge when run on the HDD array shown above. The trace is then analyzed for runs of sequential read and writes accesses. The horizontal axis shows the sequential run length in bytes, while the vertical axis shows the total number of megabytes transferred for all runs of that length. The total number of megabytes read and written is shown at the lower left corner of the graph. We see that there were 3.6X as many bytes written as read. Most bytes were read in sequences of from 32KB to 64KB. Much of the writing was in sequences of from 32KB to 128KB, with some write sequences as long as 512MB. Therefore, technologies with strong sequential bandwidth, especially for writes are expected to perform better. Low latency random accesses, a strong point of the BayStor RAMdisk technology, were not needed to accelerate this workload: Figure 11: Storage Access Patterns in RLV Test 13

14 APPENDIX ANALYSIS OF ANALYTIC QUERY TEST RESULTS Looking beyond the bottom line results, we see from the vmstat graphs below that using the local HDD Array, iowait time (green line) is about 80% for a substantial portion of this run, with SAP Sybase IQ processor utilization correspondingly low during these periods. High I/O wait appears to be related to the lower ongoing levels of write activity, rather than spikes of intense read activity: Figure 12: Processor Utilization and I/O Wait Times for HDD Array in MPX Test BayStor RAMdisk reduces iowait to near zero, eliminating most dead time and increasing overall processor utilization as show in the follow graph: Figure 13: Processor Utilization and I/O Wait Times for BayStor SSD Array in MPX Test 14

15 Below is another graph of the data access patterns of the local temporary DBSpace for this work. We see that there were 1.6X as many bytes written as read. Most bytes were read in sequences of from 64KB to 1MB. Much of the writing was in sequences of from 64KB to 16MB. Therefore, technologies with strong sequential bandwidth, especially for writes are again expected to perform better. Low latency random accesses, a strong point of the BayStor RAMdisk technology, were not needed to accelerate this workload: Figure 14: Storage Access Patterns in MPX Test 15

16 SAP AG or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP AG and its affiliated companies ( SAP Group ) for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Please see epx#trademark for additional trademark information and notices.