An Oracle White Paper May Dramatically Accelerate Oracle Database Performance

Transcription

1 An Oracle White Paper May 2010 Dramatically Accelerate Oracle Database Performance

2 Introduction... 2 A Solution for Optimum Database Performance... 4 Best-in-Class Platforms for Enterprise Deployment... 5 Processing Performance... 5 Large Memory Support... 5 High I/O Throughput Capabilities... 5 Fine Grain High Availability Architecture... 5 Creating a Hybrid Storage Architecture... 6 Sun Storage 6000 Array Family... 6 Storage Manageability... 6 Introduction to Oracle s Sun FlashFire Technology... 6 Sun FlashFire Module... 8 Sun Storage F5100 Flash Array... 9 Deployment Considerations Understanding and Addressing I/O Performance Issues Common I/O Bottlenecks Evaluating Current I/O Performance Accelerating Database Application Performance Using Flash to Extend the System Global Area Solving I/O Bottlenecks Summary For More Information... 21

3 Introduction Maximizing the performance of enterprise applications can be critical to gaining the types of operational efficiencies that help organizations remain competitive in today s fast-paced markets. Optimizing performance at every layer of the IT infrastructure is important. Servers, storage devices, operating systems, and database engines all contribute to the effectiveness of a solution. Oracle s Sun SPARC Enterprise M-Series servers running the Oracle Solaris operating system are ideal platforms for meeting the demands of enterprise application workloads. Built for top performance with SPARC64 VII processors, large memory capacity, and massive I/O expansion capabilities, Sun SPARC Enterprise M-Series servers excel at executing highthroughput transaction processing workloads while offering mission critical reliability and fine grain high availability architecture. Furthermore, the Oracle Solaris multithreaded execution model, memory placement optimization features, and multipathing I/O capabilities help garner the best possible performance from these systems. For applications that rely on large databases, fast I/O throughput often becomes a top priority within the solution architecture. For many application types including enterprise resource planning (ERP), supply chain management (SCM), customer relationship management (CRM), and custom online transaction processing (OLTP) systems optimizing database response times and throughput plays an important role. Oracle s Sun Storage 6780 array is a modular, seventh generation storage system, offering reliable storage of up to 896 TB of data and high-performance data transfer capabilities. While the Sun Storage 6780 array is ideal for storing massive datasets and offers fast throughput, adding Oracle s Sun FlashFire technology to the solution can help further enhance I/O performance. Devices with Sun FlashFire technology such as the Sun Storage F5100 Flash Array from Oracle offer exceptional I/O rates that can help drive database transaction rates well past typical performance expectations. 2

4 Strategic use of the Sun Storage F5100 Flash Array along with Sun SPARC Enterprise M-Series servers, Oracle s Sun Storage arrays, and Oracle Solaris 10 can result in radical performance improvements for Oracle database workloads. For example, using a Sun Storage F5100 Flash Array to extend the System Global Area (SGA) in an Oracle database software instance resulted in a nearly 500% improvement in transaction throughput 1. Adding a Sun Storage F5100 Flash Array to an implementation of Oracle s PeopleSoft Enterprise Payroll for North America software helped alleviate hot tablespaces minimizing response times and decreasing the time the Oracle database software spent executing single block physical reads by 39%. As evidenced by these examples, adding the Sun Storage F5100 Flash Array to a database solution architecture can help organizations raise enterprise application performance to an entirely new level. 1 Extending the SGA requires use of the Oracle Flash Cache feature available in Oracle Database 11g Release 2 software. 3

5 A Solution for Optimum Database Performance An architecture built with Sun SPARC Enterprise M-Series servers, Sun Storage 6000 arrays, Oracle Solaris, and Sun Storage F5100 Flash Arrays (Figure 1) can deliver best-in-class performance and reliability when used to deploy the Oracle 11gR2 database. Oracle s Sun SPARC Enterprise M-Series servers offer a high-performance, scalable architecture that supports advanced processor technology, large memory expansion, and massive I/O bandwidth along with the reliability of mainframe-class servers and the flexibility of open systems. Oracle Solaris supports thousands of commercial applications and is optimized to drive top performance from Sun SPARC Enterprise M-Series servers, with support for innovative new features such as flash technology and Oracle Solaris Zettabyte File System (ZFS). Sun Storage 6000 arrays from Oracle support fast data transfer, high-availability, and extensive reliability for enterprise projects that require storage of large volumes of data. With regard to performance, the advanced architecture of the Sun Storage 6780 array supports multidimensional scalability in terms of IOPS, bandwidth, and capacity. In addition, these arrays offer high levels of data reliability and availability through redundant components, automated I/O path failover, and extensive online reconfiguration and maintenance capabilities. Oracle s Sun Storage F5100 Flash Array dramatically improves the solution by contributing exceptional I/O performance capabilities. Built with Sun FlashFire technology, the Sun Storage F5100 Flash Array can be used strategically within a solution architecture to address specific I/O bottlenecks. Figure 1. A high-performance architecture for demanding database applications 4

6 Best-in-Class Platforms for Enterprise Deployment The Sun SPARC Enterprise M-Series server product family offers a scalable range of compute power, from Oracle s rack mount Sun SPARC Enterprise M4000 and M5000 servers to Oracle s datacenter Sun SPARC Enterprise M8000 and M9000 servers (Figure 2). And when combined with Oracle s Solaris high performance, fault tolerant operating system Processing Performance Sun SPARC Enterprise M-Series servers speed processing with a high-performance architecture that utilizes the latest SPARC64 VII processors and Jupiter system interconnect. Within the family of Sun SPARC Enterprise M-Series servers, system sizes range from a single processor quad-core system to a system that supports up to 64 processors, 256 cores, and 512 virtual processors. Equipped with four cores, the SPARC64 VII processor implements a chip multiprocessing (CMP) architecture and utilizes simultaneous multithreading (SMT) to support parallel execution of a total of eight threads (two threads per core). Multiple system controllers and crossbar units provide point-to-point connections between CPU, memory, and I/O subsystems. Offering more than one bus route between components enhances performance and reliability. In fact, the system interconnect used in these servers can deliver as much as 737 GB/sec. of peak interconnect bandwidth. Large Memory Support The memory subsystem of Sun SPARC Enterprise M-Series servers with Oracle s Solaris increases the scalability and throughput of these systems. For example, at the high-end of this server family, the Sun SPARC Enterprise M9000 server accommodates up to 4 TB of memory (to support in-memory database operations). High I/O Throughput Capabilities Sun SPARC Enterprise M-Series servers and Oracle s Solaris use a PCI Express bus to provide highspeed data transfer within the I/O subsystem. PCI Express technology doubles the peak data transfer rates of original PCI technology and reaches 2.5 Gb/sec. of throughput. PCI Express was developed to accommodate high-speed interconnects such as Fibre Channel, InfiniBand, and Gigabit Ethernet. Sun SPARC Enterprise M4000, M5000, M8000, and M9000 servers also support PCI-X expansion cards for fast access to external devices. PCI-X is backward compatible with existing PCI cards while increasing bandwidth to support data transfer rates of up to 1 GB/sec. for 64-bit devices. Fine Grain High Availability Architecture The combination of Sun SPARC Enterprise M-Series servers and Solaris operating system provides redundant and hot-swappable system components, fault resilient power options, diagnostic and error recovery features throughout the design, and built-in remote management features. The advanced architecture of these reliable servers fosters high levels of application availability and rapid recovery from many types of hardware faults. 5

7 Sun SPARC Enterprise M-Series servers and Solaris also support a variety of tools to create a virtualized environment and simplify management of application and resource isolation, including the following: Dynamic Domains partition Sun SPARC Enterprise M-Series servers along physical boundaries, providing complete resource isolation between applications. Dynamic Reconfiguration helps support continuous uptime and increases flexibility by allowing the addition and removal of CPU/memory and I/O boards and transfer of these resources from one Dynamic Domain to another without interrupting application processing. Oracle Solaris Containers technology isolates software applications and services using flexible, software-defined boundaries. With this technology, many private execution environments can be created within a single instance of Oracle Solaris. Creating a Hybrid Storage Architecture Since exclusive use of flash devices is not economical for sizeable projects, enterprise storage arrays still must handle the bulk of the data storage requirements. Creating a hybrid storage architecture that utilizes Sun Storage 6000 arrays for mass data storage and Sun Storage F5100 Flash Arrays to relieve specific I/O bottlenecks can help maximize database performance without breaking budgets. Sun Storage 6000 Array Family The Sun Storage 6000 array family includes Oracle s Sun Storage 6180 arrays, Oracle s Sun Storage 6580 arrays, and Oracle s Sun Storage 6780 arrays (Figure 3). These arrays offer enterprise-class storage capabilities in the areas of performance, manageability, and data protection. At the high-end of this family of storage arrays, the Sun Storage 6780 array scales to support up to 896 TB of data storage capacity, 16 GB dedicated cache, and sixteen 8 Gb/sec direct or switched attached FC ports. The controller for Sun Storage 6000 arrays is the culmination of more than 20 years of design knowledge focused entirely on open systems storage. This end-to-end, 8 Gb/sec. Fibre Channel RAID controller, helps Sun Storage 6000 arrays offer exceptional throughput and bandwidth. Storage Manageability Helping to reduce complexity, Sun Storage Common Array Manager software from Oracle is utilized across the entire storage and Flash product families. Sun Storage Common Array Manager software makes it easy to manage numerous arrays, and storage profiles enable administrators to quickly and predictably provision storage to precise application demands. In addition, the built-in performance tracking utility helps optimize application performance and minimize downtime through predictive, preemptive maintenance and automated diagnostics for proactive health monitoring. Introduction to Oracle s Sun FlashFire Technology Sun FlashFire technology takes advantage of the capabilities of standard flash technology and adds advanced performance and reliability features. 6

8 Oracle White Paper Dramatically Accelerate Oracle Database Performance Sun FlashFire technology takes advantage of flash technology to bridge the gap between the capabilities of the compute and storage subsystem. Using storage devices with Sun FlashFire technology offers a number of advantages over hard disk drives: Sun FlashFire technology can offer orders of magnitude improvements in IOPS rates and services times, along with substantial increases in data rates over hard disk drives. By eliminating the need for moving parts, devices with Sun FlashFire technology also offer inherently better reliability and environmental characteristics no bearings to wear out, heads to crash, or rotational vibration sensitivity, and greater mechanical shock tolerance. Devices with Sun FlashFire technology (such as the Sun Storage F5100 Flash Array) also exhibit much lower power consumption, especially when compared to hard disk drives on a Watt per performance level for IOPS, service times, and data rates. Pictured in Figure 4, Sun s portfolio of products with FlashFire technology include the following: Oracle s Sun FlashFire Modules are the building blocks for all other devices in the family of Sun products with FlashFire technology. Sun FlashFire Modules deliver ultra high I/O performance with many enterprise features in an ultra compact SODIMM sized form factor. Oracle s Sun Flash Accelerator F20 PCIe Card delivers the IOPS performance of over 300 disk drives and 96 GB capacity, making it ideal for reducing storage latencies and eliminating bottlenecks in I/O intensive applications. The Sun Storage F5100 Flash Array utilizes Sun FlashFire technology to deliver over 1M IOPS in 1 rack unit (1U), improving database application response times up to 200%, eliminating bottlenecks, and slashing energy consumption by up to 80% as compared to spinning disks. Figure 4. Sun products with FlashFire technology include the Sun FlashFire Module, Sun F20 Flash Accelerator PCIe Card, and Sun Storage F5100 Flash Array 7

9 Sun FlashFire Module Each device within Sun s portfolio of storage products with FlashFire technology relies on the Sun FlashFire Module to provide stable and persistent storage media for the storage subsystems. There are some similarities between the Sun FlashFire Module and enterprise-grade solid state disks (SSDs). For example, the Sun FlashFire Module and SSDs both utilize Single Level Cell (SLC) NAND Flash memory and SATA-II technology and offer low power consumption rates of around 2 to 2.5 Watts. SSDs are normally engineered for general purpose I/O, whereas Sun FlashFire Modules are specifically designed to accelerate database performance while maximizing reliability. As shown in Table 3, Sun FlashFire Modules provide fast, balanced read and write performance in contrast to the asymmetric performance of SSDs. TABLE 3. COMPARISON OF SUN FLASHFIRE MODULE AND SSD PERFORMANCE CHARACTERISTICS SUN FLASHFIRE MODULE SOLID STATE DRIVE (SSD) READ PERFORMANCE 30k random 4KB read IOPS 20k to 30k random 4KB read IOPS WRITE PERFORMANCE 21k random 4KB write IOPS 3k to 5k random 4KB write IOPS To maximize reliability, Sun FlashFire Modules offer persistence algorithms that mark writes as complete only when they are written to flash and not just DRAM. In addition, Oracle worked in partnership with flash chip vendors to ensure that Sun FlashFire Modules offered enhanced write durability. As a result, the Sun FlashFire Module can support greater than 1 million writes per page and provide Advanced Wear Leveling Algorithms that deliver three year 100% write and read endurance. 8

10 Sun Storage F5100 Flash Array The Sun Storage F5100 Flash Array accelerates databases with over 1 million IOPS performance and up to 100 times less power and space than traditional disk-based solutions for the same performance level. Exceeding the IOPS performance of over 3,000 disk drives, and more than four times the performance of other flash technology-based systems, the Sun Storage F5100 Flash Array provides a cost-effective building block that can deliver breakthrough value for high-performance database applications. A single Sun Storage F5100 Flash Array is designed as four separate SAS domains that can be attached to a number of servers. The Sun Storage F5100 Flash Array supports a variety of configurations, so storage architects can design flexible, cost-effective solutions that complement the existing storage infrastructure and meet performance, capacity, and availability goals. In this way the array helps to deliver substantial ROI for high-performance business-critical database applications. Table 4 provides a brief summary of Sun Storage F5100 Flash Array features. TABLE 4. SUN STORAGE F5100 FLASH ARRAY CHARACTERISTICS FEATURE STORAGE DENSITY PERFORMANCE SUN STORAGE F5100 FLASH ARRAY 1.92 TB maximum per 1U array Random reads: 1.6 million IOPS (4 KB block size, 32 threads) Random writes: 1.2 million IOPS (4 KB block size, 32 threads) Sequential reads: 12.8 GB/sec (1 MB block size, 32 threads) Sequential writes: 9.7 GB/sec (1 MB block size, 32 threads) Read latency: 405 microseconds (4 KB block size, single thread) Write latency: 282 microseconds (4 KB block size, single thread) CONNECTIVITY RELIABILITY Sixteen x4 mini-sas ports (four x4 3Gb/s mini-sas ports per domain) Redundant power supplies and fan modules Supercapacitor-backed DRAM Storage Common Array Manager for system management POWER CONSUMPTION 2.1 Watts per Sun Flash Module; 300 Watts per fully populated array 9

11 Deployment Considerations The performance characteristics of the Sun Storage F5100 Flash Array are well suited to the database type I/O workloads. Access patterns and transfer sizes. The SLC NAND utilized by Sun FlashFire Modules is read, written, and erased in pages. Since Sun FlashFire Modules implement 4096 byte (4 KB) pages, logical block access (LBA) patterns that are on exact 4 KB boundaries, and read and write transfer sizes that are multiples of 4 KB are set to achieve best performance. This characteristic meshes well with the needs of database workloads. In fact, the default block size for Oracle databases is 8 KB, which is 4 KB aligned and an integral multiple of 4 KB length. Block Size. Most generic device drivers, volume managers, and file systems assume a 512 byte block size for raw storage. Tools that issue I/O requests to arbitrary 512 byte boundaries or use odd transfer sizes cannot take maximum advantage of the I/O performance advantages of Sun FlashFire Modules. However, a move to larger block sizes is underway. As more systems, interfaces, and best practices optimize for the 4,096 byte block size being implemented in many of today s storage and I/O systems, application performance is likely to increase. In terms of application access patterns and latency sensitivity, the ideal fit and optimal benefit can be found with smaller datasets, particularly those with higher I/O access densities. Utilizing flash technology can help optimize the performance of workloads with an access density greater than 1 IOP/GB as well as latency bound applications that do not benefit from caching. 10

12 Understanding and Addressing I/O Performance Issues Gaining an understanding of typical I/O bottlenecks in the storage subsystem and methods to detect these issues is a necessary part of the performance tuning process. With this knowledge, architects are best able to maximize the benefits of deploying FlashFire technology to improve the performance of new or existing implementations. Common I/O Bottlenecks Although I/O patterns and performance can be a complex subject, the following three scenarios cover the majority of I/O performance issues single threaded latency, streaming data rate, and multithreaded asynchronous IOPS. While these I/O factors remain in play regardless of the storage technology in use, the choice of hard disk drives or devices with flash technology can dramatically alter the performance results. Single Threaded Latency Single threaded latency measures the total time for a committed write operation or completed read operation and is commonly referred to as service time. Usually measured in milliseconds for hard disk drives and microseconds for flash or memory-based storage, service time is the length of time from command initiation until I/O operation completion. Measurements that determine the service time offered by a particular device are generally taken while no other concurrent demands are placed on the storage subsystem and for operations on a relatively small or single block of data a few kilobytes or even 512 bytes. Since most block-based storage systems today utilize hard disk drive technology, service times for small blocks are limited by mechanical behavior moving actuator arms over the correct track on a rotating disk (seek latency) or waiting for the correct block or sector of data to be under the head to be written or read (rotational latency). Today, even the best hard disk drives create total latencies of a few milliseconds for random block accesses and commodity and consumer grade hard drives can take over 10 milliseconds. In contrast, Sun FlashFire technology can deliver average 8 KB read and write service times of about 400 microseconds, representing a 10x reduction in service time over hard disk drives. From an application perspective, the impact of service time is often greatest for synchronous I/O operations that lack the benefit of caching. For instance, performance can stall in cases where the application, database, or operating system process thread is blocked or waiting for I/O completion response before allowing other processing to continue. Good examples of scenarios that suffer the performance impact of long service times include database index lookups and writes to redo logs within online transaction type workloads. In many cases, every microsecond of service time for synchronous I/O s results in dead wait time for the database and its application. In other words, during the elapsed service time there is nothing else the application or database can operate on until the row or column index is returned or the redo log is committed to stable storage to insure database integrity. There are other types of structures that are limited by service times aside from databases. For example, metadata updates for standard file systems and journals and intent logs are generally synchronous, and 11

13 even application lock files or metadata lock files can block application or data distribution while locks are being arbitrated. Maximum Streaming Data Rate Often measured in megabytes per second, the maximum streaming data rate is usually optimized by employing very large data block sizes for the read or write operations. These block sizes can be hundreds of kilobytes or even a few megabytes in length. In many cases more than one concurrent I/O thread or queue is needed to keep the storage subsystem busy enough to achieve its best performance for this data movement metric. However, only a few threads are usually needed to maximize application performance for a given storage subsystem. For each thread, the reads or writes are to sequential logical block addresses within the storage subsystem. This activity is also known as streaming, and allows the firmware in the storage device to perform prefetch, read-ahead and writebehind optimizations in the controller or hard disk drive firmware. The hard disk drive data rate is limited by the rotational speed of the drive and the bit recording density of the magnetic disk. In some cases, the channel bandwidth of the disk controllers and storage interconnects are the performance limiters. In most cases, working around this common I/O storage bottleneck requires increasing various aspects of parallelism. Any application that processes huge amounts of data as part of its implementation can be impacted by the maximum stream rate. For databases, stream performance bottlenecks can often be reached quickly in table scans in online analytical processing (OLAP), decision support systems (DSSs), and business intelligence data warehousing (BIDW) workloads. Outside of databases, data rate limits of the storage subsystem can dramatically affect bulk file transfer, replication, and high performance computing workloads. Multithreaded, Asynchronous IOPS There are cases where the saturation I/O rate of random accesses to the storage subsystem or hard disk drive limits database and application performance or scaling. Most databases use the Stripe And Mirror Everything (SAME) strategy to spread this type of I/O workload over as many hard disk drives as possible and as uniformly as possible. This technique seeks to reduce hot spots in the storage subsystem and can keep response time across the entire storage subsystem within asynchronous processing limits. Modern enterprise databases often parallelize queries and update tablespaces using asynchronous I/O s. Database consistency and integrity is maintained with less frequent synchronous checkpoints, while journals and logs are usually applied as part of application commitment requests. Even modest transactions or queries can generate tens or even hundreds of physical storage subsystem I/Os. Since they are often asynchronous, the database and application can often continue processing in parallel with I/O completion. Note that the response time of a storage subsystem as a function of demanded IOPS rate is nonlinear and usually increases asymptotically. However, for the types of workloads that issue hundreds or thousands of concurrent queued I/O s, as long as the average service time is under a few tens of milliseconds, overall system performance does not degrade. In other words, reducing the response time 12

14 for these types of I/O operations typically does not improve overall system performance in any significant way. Moreover, to get higher concurrent I/O rates from these storage devices, internal storage subsystem processing usually completes the queued I/O requests out of order. This is one reason why the average response time increases while offering higher asynchronous IOPS throughout. Beyond databases, many messaging and Web applications are also limited by overall IOPS bottlenecks. In fact, mail message storage is a classic example of trading off higher response times with no impact on service delivery to get higher scale in number of concurrent users (IOPS). Ultimately, the maximum IOPS limit is hit because of the number of hard disk drives that can be aggregated and computationally managed in the storage subsystem. This performance limit is imposed in part by each hard disk drive which can mechanically can process only a couple hundred random I/O s per second. The overall storage controller also contributes to this performance issue since it only has enough memory to manage a modest number of hard disk drives and a limited amount of CPU power to process a relatively low number of concurrent I/O requests. Interestingly, service delivery workloads such as low bit rate content distribution that may demand high aggregate data rates over thousands of streams usually do not hit the data rate bottleneck of a storage system. Rather, these workloads reach a random IOPS limit. This limit or bottleneck occurs because with so many streams reading so many blocks, the access pattern appears to be random and hits the I/O processing limit of the system first. Evaluating Current I/O Performance Using I/O monitoring utilities that are part of the operating system or system management environment can help determine if an I/O bottleneck exists, and whether using a flash array can help alleviate the situation. In environments running Oracle Solaris, the iostat command can be useful. Oracle also provides downloadable utilities that can assist with this analysis, such as the Sun Flash Analyzer available at Checking Service Times for Database Index LUNs A good first step for a database tuning effort involves identifying the logical unit numbers (LUNs) and storage locations for the database index and determining if service or wait times are 10 milliseconds or higher for those devices. If so, a flash array might help system performance. If these total service times are short approximately 1 millisecond or less then the indexes are being cached in the storage subsystem and are likely already optimized. In databases that are inhibited by slow service times, a subset of the database typically exhibits the I/O bottleneck, such as deep trees in the indexes. A good rule of thumb is to place 25% of the database (indexes) on the Sun Storage F5100 Flash Array. Each array can support up to 960 GB of protected storage (1.92 TB raw). Use this ratio to determine the number of Sun Flash Modules per array, or number of arrays, needed overall. Identifying Wait Events Long response times for database file reads and writes can lead to degraded application performance. Databases often have internal reporting tools that can help identify system bottlenecks. For example, 13

15 administrators can use the Oracle STATSPACK or Oracle Automated Workload Repository (AWR) reporting tools in existing Oracle Standard Edition (SE) and Oracle Enterprise Edition (EE) database deployments. When using a tool such as STATSPACK, find the Top 5 Wait Events and check to see if db file sequential read is one of the important wait events. A sample of the Top 5 Wait Events portion of the STATSPACK output is shown here: Top 5 Timed Events ~~~~~~~~~~~~~~~~~ Event Waits Time (s) %Total ELA Time CPU time 3, db file sequential read 861, latch free 171, enqueue db file scattered read 295, The db file sequential read statistic refers to a process sequence that is waiting for data (reads) from the storage subsystem. It is usually the case that the db file sequential read wait time correlates to the storage subsystem service time for that wait event (along with the operating system and database I/O processing and queueing). When db file sequential read is in the Top 5 Wait Events list, a storage I/O bottleneck exists. If db file sequential read is the top event, addressing this I/O issue is likely very worthwhile toward improving performance. Moreover, if the percentage of total wait time is significant, then this is an upper bound for the amount of unused system performance that can be recovered. The Sun Storage F5100 Flash Array can reduce db file sequential read events to 1 to 3 ms for affected datasets, which can help dramatically accelerate database performance. 14

16 Accelerating Database Application Performance Using the Sun Storage F5100 Flash Array to reduce the latencies associated with the reading and writing of specific datasets can significantly improve database application performance. Indeed, use cases and reference architectures have shown substantial gains in performance when moving database indexes and hot tablespaces from high-performance traditional RAID systems to the Sun Storage F5100 Flash Array. Two detailed examples of utilizing the Sun Storage F5100 Flash Array to accelerate database performance are presented within this chapter. Using Flash to Extend the System Global Area In Oracle database software, the System Global Area (SGA) is a group of shared memory areas that are dedicated to an Oracle database instance. Oracle database processes use the SGA to store incoming data (data and index buffers) and internal control information that is needed by the database. As illustrated by the following example, utilizing the Sun Storage F5100 Flash Array to extend the Oracle Shared SGA can have a positive impact on performance levels. Testing Overview Traditionally, the size of the SGA is limited by the size of the available physical memory. The following tests measured the performance of a feature found first in Oracle Database 11g Release 2 known as Oracle Flash Cache. This capability allows extension of the SGA size and caching beyond physical memory to a large flash memory storage device such as the Sun Storage F5100 Flash Array. The benchmark workload consisted of a high volume of SQL select transactions accessing a very large table in a typical business oriented OLTP database. The database consisted of various tables, such as products, customers, orders, and warehouse inventory data. The warehouse inventory table alone was three times the size of the db_buffer_size decreasing the probability of completing the majority of data retrievals from the database in-memory cache. To obtain a baseline, throughput and response times were measured by applying the workload against a traditional storage configuration constrained by disk I/O demand. The workload was then executed with an added Sun Storage F5100 Flash Array that was configured to contain an Extended SGA of incremental size. During each test, the in-memory SGA was limited to 25 GB. The Extended SGA was allocated on a raw Oracle Solaris Volume created with the Oracle Solaris Volume Manager on a set of Sun FlashFire Modules residing on the Sun Storage F5100 Flash Array. The benchmark test configuration is listed in Table 5. TABLE 5. EXTENDED SGA CACHING ON A SUN STORAGE F5100 FLASH ARRAY BENCHMARK CONFIGURATION SERVER CONFIGURATION Oracle s Sun SPARC Enterprise M5000 Server Eight SPARC64 VII 15

17 2.4 GHz Quad Core96 GB memory STORAGE CONFIGURATION Eight Sun Storage J4200 Arrays from Oracle 2 12x 146 GB 15K RPM disks each (96 disks total) One Sun Storage F5100 Flash Array SOFTWARE CONFIGURATION Oracle Database 11g Release 2 Oracle Solaris 10 In order to verify the performance improvement brought by an Extended SGA, the test needed to include a sizeable database and a workload that induced significant disk I/O activity. For that purpose, the size of the database needed to be a multiple of the physical memory size, avoiding the case in which the accessed data could be entirely or almost entirely cached in physical memory. This test scenario represents a typical use case in which the extending the SGA using the Sun Storage F5100 Flash Array is able to show remarkable performance advantages. Test Results The tests results graphed in Figure 6 show the performance impact of increasing the Flash Cache size utilized for the Extended SGA. As flash cache size increased, throughput scaled steadily and transaction response time decreased. In fact, using the Oracle Extended SGA feature with a Sun Storage F5100 Flash Array to reach SGA sizes in the hundred gigabyte range demonstrated an almost 500% performance improvement. As compared to traditional RAM-based solutions, this implementation does a better job of balancing cost and performance by creating a configuration that is achievable at a more reasonable cost than equivalently sized RAM, and with much faster access times than disk I/O. 2 While a Sun Storage J4200 array provided a sufficient amount of mass storage capacity for this example, the Sun Storage 6780 array can replace the role of the Sun Storage J4200 array for workloads with larger data set size requirements 16

18 Figure 6. Utilizing the Sun Storage F5100 Flash Array to store an Oracle Extended SGA can significantly improve database performance. Solving I/O Bottlenecks Sun FlashFire technology can also help improve performance by solving specific I/O bottlenecks. To help illustrate this point, the following benchmark tested the ability of adding a Sun Storage F5100 Flash Array to alleviate hot tablespaces in an implementation of Oracle s PeopleSoft Enterprise Payroll for North America software. Oracle s PeopleSoft Enterprise Payroll for North America software provides the tools to calculate earnings, taxes, and deductions efficiently, maintain balances, and report payroll data. The payroll system also enables companies to calculate gross-to-net or net-to-gross pay, leave accruals, and retroactive pay, as well as automatically calculate imputed income for group-term life insurance and process unlimited direct deposits. Sun and Oracle PeopleSoft engineers tested and tuned a configuration for Oracle s PeopleSoft Enterprise Payroll for North America software consisting of the components listed in Table 6. 17

19 TABLE 6. ORACLE S PEOPLESOFT ENTERPRISE PAYROLL FOR NORTH AMERICA SOFTWARE BENCHMARK CONFIGURATION SERVER CONFIGURATION Oracle s Sun SPARC Enterprise M4000 server Four SPARC64 -VII processors at 2.53 GHz 32 GB memory Four dual-port serial-attached SCSI (SAS) HBA cards STORAGE CONFIGURATION One Sun Storage J4200 array 3 Twelve 450 GB SAS 15K RPM disk drives One Sun Storage F5100 Flash Array, forty 24 GB Flash modules Four domains with 10 FMods per domain SOFTWARE CONFIGURATION Oracle PeopleSoft HRMS and Campus Solutions 9.0 Oracle PeopleSoft Enterprise (PeopleTools) bit Oracle Database 11g Enterprise Edition Release Oracle Solaris 10 The payroll application is sensitive to I/O latency. In other words, the performance of this workload hinges on reducing the response time of physical I/O operations, primarily single-block reads. This susceptibility can be easily seen from the Top 5 Timed Foreground Events section of the Oracle AWR Report, shown in Table 6. The results in Table 7 represent a test run of the Pay Confirmation step where the database was configured on a Fibre Channel (FC) cached storage array. The event db file sequential read is a counter of single block reads executed by Oracle database foreground processes. The 3 millisecond (ms) average response time is respectable for a disk-based I/O subsystem but the sheer number of read requests (waits) makes it account for 68.8% of the processing time in the Oracle database. To improve the performance of this workload the number and/or the response time of physical reads must be reduced. This can be managed by increasing the size of the Oracle database buffer cache in the SGA to eliminate reads. However, unless there is enough memory to cache the entire database, the application must still perform physical reads from the storage subsystem, and this is where the sub-millisecond performance of the Sun Storage F5100 Flash Array can help. 3 While a Sun Storage J4200 array provided a sufficient amount of mass storage capacity for this example, the Sun Storage 6780 array can replace the role of the Sun Storage J4200 array for workloads with larger data set size requirements. 18

20 TABLE 7. BASELINE FOR TOP 5 TIMED FOREGROUND EVENTS USING TRADITIONAL STORAGE ARRAYS EVENT WAITS TIMES(S) AVG. WAIT(MS) % DB TIME WAIT CLASS db file sequential read 6,183,399 18, User I/O DB CPU 7, enq: TX omdex contention Concurrency read by other session 230, User I/O Cursor: pin S wait on X 2, Concurrency Microbenchmark tests with a profile of 50% reads and 50% writes have shown that each Sun Flash Module in a Sun Storage F5100 Flash Array can achieve 10,600 IOPS. At half-capacity (40 Sun Flash Modules), 426,000 IOPS can be achieved. I/O response time is impacted as I/Os/second increase. Oracle s PeopleSoft Enterprise Payroll for North America software application averages 3,100 reads/sec and 2,400 writes/sec to data files containing tables and indexes. This workload easily fits within the Sun Storage F5100 Flash Array s capabilities to deliver the fastest I/O response times. The database was redeployed on a Sun Storage F5100 Flash Array configured at half of its capacity. Oracle Solaris Volume Manager software was used to consolidate the Sun FlashFire Modules from Oracle into a 32-way metadevice and an 8-way metadevice. The 32-way metadevice was used for the tablespaces containing tables and indexes. The 8-way metadevice was used to contain the undo tablespace. The Oracle Redo Log files were assigned to a 6-way metadevice on the Sun Storage J4200 array, again created using Oracle Solaris Volume Manager software. This layout resulted in less than 1 ms I/O response times when accessing tables and indexes. Undo tablespaces I/O response times averaged 2.5 ms and Redo Log I/O response times on the Sun Storage J4200 array averaged 4 ms. Table 8 shows the Top 5 Timed Foreground Events section of the Oracle AWR Report after reconfiguration of the database on the Sun Storage F5100 Flash Array. TABLE 8. TOP 5 TIMED FOREGROUND EVENTS RESULTS WITH SUN STORAGE F5100 FLASH ARRAY EVENT WAITS TIMES(S) AVG. WAIT(MS) % DB TIME WAIT CLASS DB CPU 7, db file sequential read 5,784,794 4, User I/O SQL *Net message to client 38,581, Network read by other session 135, User I/O enq: TX index contention Concurrency 19

21 By utilizing the Sun Storage F5100 Flash Array, the time the Oracle database software spent executing single block physical reads decreased by 39%. It should be noted that a few simple guidelines were followed in laying out the database on the Sun Storage F5100 Flash Array: Only allocate files that perform I/Os on a 4 KB boundary alignment on the Sun Storage F5100 Flash Array. For the Oracle database this means Redo Logs should not be allocated on the Sun Storage F5100 Flash Array. In this case Redo logs were allocated on the attached Sun Storage J4200 array from Oracle. Follow Stripe and Mirror Everywhere (SAME) guidelines. Segregate 4 KB aligned write-intensive files. In this case the Undo tablespace was moved to its own set of Flash Modules. Following these basic rules resulted in a 14% processing time improvement for an implementation of Oracle s PeopleSoft Enterprise Payroll for North America software as compared to a highly optimized configuration of the database on Fibre Channel cache-enabled storage arrays. For more information about this test, please see the full report at Summary Combining the power of Sun SPARC Enterprise M-Series servers, Oracle Solaris, Sun Storage 6000 Arrays, and Sun Storage F5100 Flash Arrays can help organizations create solutions that deliver exceptional database application performance. Oracle s Sun SPARC Enterprise M-Series servers feature scalable, high-performance processing capabilities, large memory capacity, and massive I/O bandwidth while Oracle Solaris works to optimize overall performance and reliability. A hybrid storage architecture combines the strengths of two storage approaches. High-performance enterprise-grade Sun Storage 6000 arrays from Oracle offer the capacity and data management features required for large scale database deployments. To move past typical I/O performance barriers, Sun Storage F5100 Flash Array adds extremely fast data storage and retrieval rates to the solution. Utilizing Oracle s Sun Storage F5100 Flash Array to host performance-critical aspects of database operation such as the SGA, indexes, and hot tablespaces can alleviate I/O bottlenecks and dramatically accelerate overall performance. 20

22 For More Information To learn more about Sun products and the benefits of Oracle s Sun SPARC Enterprise M-Series servers, Oracle s Sun Storage 6000 arrays, and Oracle s Sun Storage F5100 Flash Arrays, please contact a Sun sales representative, or consult the related Web sites listed in Table 9. TABLE 9. WEB SITES FOR MORE INFORMATION DESCRIPTION Sun SPARC Enterprise M-Series Servers Sun Storage F5100 Flash Array Sun Storage 6780 Array Sun Storage F5100 Flash Array, Storage Considerations for Accelerating Databases Accelerating Databases with the Sun Storage F5100 Flash Array Deploying Hybrid Storage Pools with Sun Flash Technology and the Solaris ZFS File System Sun Flash Analyzer Sun Storage Common Array Management Software URL

23 Dramatically Accelerate Oracle Database Performance April 2010 Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA U.S.A. Worldwide Inquiries: Phone: Fax: oracle.com Copyright 2010, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd. 0310