EMC Backup and Recovery for Oracle Database 11g Enabled by EMC Celerra NS-120 using DNFS Abstract This white paper examines the performance considerations of placing Oracle Databases on Enterprise Flash Drives (EFDs) versus FC disks. Also, it validates a solution for Oracle Database 11g backup and recovery on the EMC Celerra NS-120 unified storage platform and by using EMC SnapSure over Oracle Direct NFS (DNFS). May 2010
Copyright 2010 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com All other trademarks used herein are the property of their respective owners. Part number: H7044 2
Table of Contents Executive summary... 5 Business case... 5 Introduction... 5 Purpose... 5 Scope... 5 Audience... 6 Terminology... 6 Technology overview... 8 Introduction... 8 EMC Celerra unified storage platform... 8 Disk drives... 8 Enterprise Flash Drives (EFDs)... 9 EMC SnapSure... 9 Oracle software stack... 9 Oracle DNFS... 9 Configuration... 11 Overview... 11 The site... 11 Environment profile... 11 Physical environment... 11 Storage layout... 12 Disk layout... 13 Network architecture... 13 Hardware resources... 14 Software resources... 14 Test and validation... 16 Introduction... 16 Store solution... 16 Basic Backup solution... 17 Advanced Backup solution... 19 Conclusion... 22 Introduction... 22 Reduced total cost of ownership... 22 Improved performance... 22 Ease of use... 23 Business continuity... 23 Robust performance and scaling... 23 Next steps... 23 References... 24 3
Reference document... 24 4
Executive summary Business case The EMC Celerra NS-120 multi-protocol array provides a low-cost, performing, and scalable solution for midsize Oracle customers. This array provides access over either Network File System (NFS) on IP networks or Automatic Storage Management (ASM) on Fibre Channel Protocol (FCP) networks. This white paper summarizes the backup, recovery, and performance best practices that were discovered and validated for Oracle Database 11g using the EMC Celerra NS-120 storage platform and EMC SnapSure over Oracle Direct NFS (DNFS). Introduction Purpose The purposes of this solution are to: Demonstrate the functional, performance, resiliency, and scalability capabilities of an Oracle software stack. The Oracle software stack is physically booted and uses the DNFS protocol to access the storage elements for the Oracle Database. EMC has a strong partnership with Oracle in the area of DNFS, and has published numerous solutions using this protocol, in the process taking a leadership role among Oracle s partners in support for DNFS. This solution continues this momentum. Demonstrate the use of EMC SnapSure to enable the logical storage backup and recovery of an Oracle 11g production database while offloading all performance impacts of the backup operation from the production server. This demonstrates significant performance and manageability benefits in comparison to normal Oracle Recovery Manager (RMAN) backup and recovery. Demonstrate the significant performance, manageability, and efficiency benefits of DNFS over Kernel NFS (KNFS). Demonstrate the significant increase in performance and other advantages by using Enterprise Flash Drives (EFDs), when compared to Fibre Channel (FC) drives. Demonstrate the simplicity advantages of running Oracle Database 11g with a single DNFS file system for all database objects, using Celerra Automatic Volume Management (AVM) with system-defined pools. The solution shows that storage for Oracle Databases 11g can be easily, quickly and simply configured and managed. Scope The scope of the white paper covers the following topics: User scalability by applying a scale-up OLTP TPC-C like workload using Quest Benchmark Factory for Databases, scaling user numbers to the point where query performance falls below the TPC-C industry-accepted standard. Multiple architectural mixes should be considered and performance charted including the following examples: 5
Simplified file layouts that minimize costs, including single file system layouts or utilization of vault drives and alternative RAID protection. Database storage on EFDs. This is particularly pertinent for small databases with high performance needs. The price point of EFDs is also falling quickly, so the solution tests the performance of EFD database layouts with low disk numbers. Basic backup and recovery by using RMAN, the built-in backup and recovery tool provided by Oracle. Advanced backup and recovery by using EMC value-added software or hardware. EMC SnapSure is used to provide advanced backup functionality. Audience The intended audience for the white paper is: Internal EMC personnel EMC partners Customers Terminology The terms in this white paper are defined in the following table: Term Direct NFS (DNFS) Enterprise Flash Drive (EFD) Kernel NFS (KNFS) Serial Advanced Technology Attachment (SATA) drive Basic Backup and Recovery Advanced Backup and Recovery Definition A network storage protocol in which the NFS client is embedded in the Oracle 11g database kernel. A drive that stores data using Flash memory and contains nomoving parts. A network storage protocol in which the NFS client is embedded in the operating system kernel. SATA is a newer standard for connecting hard drives into computer systems. SATA is based on serial-signaling technology while Integrated Drive Electronics (IDE) hard drives use parallel-signaling technology. A solution component that provides backup and recovery functionality through the operating system and database server software stack. In the case of Oracle, this is RMAN. Basic Backup and Recovery uses the database server s CPUs for all I/O and processing of backup and recovery operations. A solution component that provides backup and recovery functionality through the storage layer using specialized hardware or software. Advanced Backup and Recovery makes the following benefits: 6
Offload the database server s CPUs from the I/O and processing requirements of the backup and recovery operations. Superior Mean Time to Recovery (MTTR) through the use of logical storage layer replication (commonly referred to as snapshots). Scale-up OLTP Using an industry-standard OLTP benchmark against a single database instance, comprehensive performance testing is performed to validate the maximum achievable performance using the solution stack of hardware and software. Two cases of scale-up OLTP are used: One for FC disk and the other for EFDs. 7
Technology overview Introduction This section provides an overview of the technologies that are used in this solution: EMC Celerra unified storage platform Disk drives EFDs EMC SnapSure Oracle software stack Oracle DNFS EMC Celerra unified storage platform Celerra unified storage platform products offer a flexible architecture and multiprotocol connectivity. This enables connectivity over IP/Ethernet, iscsi, and Fibre Channel SAN environments. EMC Celerra NS-120 is an affordable unified storage system that scales to 120 drives. With Celerra NS-120, you can connect to multiple storage networks via NAS, iscsi, Fibre Channel SAN, and Celerra Multi-Path File System (MPFS). The key features provided by the Celerra are described in the following table: Feature NAS iscsi storage SAN storage Provided by Network File System (NFS) and Common Internet File System (CIFS) protocols Celerra s Data Movers FCP through the back-end EMC CLARiiON storage array Disk drives The general recommendations for disk drives are as follow: Drives with higher revolutions per minute (rpm) provide higher overall randomaccess throughput and shorter response times than drives with slower rpm. For optimum performance, higher-rpm drives are recommended for datafiles and tempfiles as well as online redo logfiles. Because of significantly better performance, FC drives are always recommended for storing datafiles, tempfiles, and online redo logfiles. SATA II drives have slower response and rotational speed, and moderate performance with random I/O. However, they are less expensive than the FC drives for the same or similar capacity. SATA II drives are usually the best option for storing archived redo logs and the fast recovery area. In the event of high performance requirements for backup 8
and recovery, FC drives can also be used for this purpose. Enterprise Flash Drives (EFDs) EFDs can be used to dramatically improve the cost, performance, efficiency, power, space, and cooling requirements of Oracle databases stored on EMC Celerra. These drives are especially well suited for latency-sensitive applications that require consistently low read/write response times. This is an extremely common condition in Oracle databases. If this is the case, move this set of Oracle datafiles to EFDs. EMC SnapSure SnapSure creates a logical point-in-time image (checkpoint) of a production file system (PFS) that reflects the state of the PFS at the point in time when the checkpoint is created. SnapSure can maintain a maximum of 96 read-only checkpoints and 16 writeable checkpoints per PFS while allowing PFS applications continued access to realtime data. How SnapSure works SnapSure uses a copy on first modify principle. A PFS consists of blocks. When a block within the PFS is modified, a copy containing the block s original contents is saved to a separate volume called the SavVol. Subsequent changes made to the same block in the PFS are not copied into the SavVol. The original blocks from the PFS in the SavVol and the unchanged PFS blocks remaining in the PFS are read by SnapSure according to a bitmap and blockmap data-tracking structure. These blocks combine to provide a complete point-in-time image called a checkpoint. Oracle software stack The Oracle software stack covered by this solution consists of: Oracle Database 11g Oracle DNFS Oracle DNFS DNFS is a new feature introduced in Oracle RAC 11g. It integrates the NFS client directly inside the database kernel instead of the operating system kernel. DNFS provides significant performance, manageability, and efficiency benefits over KNFS. Better performance Transactions per second and user load are both higher with DNFS than with KNFS, and this enables organizations to gain more output from the same infrastructure. CPU costs on both the database server and the file server are lower. In addition, port scaling with DNFS is much better, enabling higher bandwidth and higher scaling. High availability Load balancing and high availability (HA) are managed internally within the DNFS client. Concurrent I/O The DNFS client performs concurrent I/O by bypassing the operating system. The benefits include: 9
Better performance because of the reduction of memory consumption and CPU utilization Consistent NFS performance is observed across all operating systems Optimized for database workloads DNFS is optimized for database workloads and supports asynchronous I/O, which is suitable for most databases. It delivers optimized performance by automatically load balancing across the available paths. Load balancing in DFNS is almost invariably superior to the conventional KNFS. 10
Configuration Overview All database objects are stored on an NFS mount. Datafiles, tempfiles, control files, online redo logfiles, and archive log files are accessed using the DNFS protocol. The site The site consists of: A physically booted single instance Oracle Database 11g server. A Celerra NS-120 connected to the Oracle Database 11g server through the storage network, with EMC SnapSure used to provide an advanced backup solution. The Oracle Database 11g server connected to the client and storage networks. Environment profile The white paper was validated with the following environment profile: Profile characteristic Value Database Size 350 GB Database Profile Workload Profile User Scaling Storage Network Connectivity OLTP TPC-C industry-standard benchmark To maximum stable load 1 GbE Physical environment The following diagram illustrates the overall physical architecture of the environment. 11
Storage layout The following table lists the Oracle file system allocation in the usage case while using FC drives: What File-system File-system type Oracle datafiles Oracle tempfiles Oracle online redo logfiles Oracle controlfiles /data_fs_fc RAID-protected NFS file system on FC disk using AVM with system-defined pools The following table lists the Oracle file-system allocation in the usage case while using EFDs: What File-system File-system type Oracle datafiles Oracle tempfiles Oracle online redo logfiles Oracle controlfiles /data_fs_efd RAID-protected NFS file system on EFDs using AVM with system-defined pools In both usage cases, all of these files are accessed using DNFS. RAID-protected NFS file systems are designed to meet the I/O demands of particular database objects. All datafiles, online redo logfiles, controlfiles, and tempfiles are contained in a single NFS file system stored on RAID 5. For backup and recovery usage case, the following objects are stored on SATA disk: What File-system File-system type Backup target Archived log files /arch_fs RAID-protected NFS file system on SATA disk To facilitate the hot backup process and to allow media recovery, archivelog mode is enabled during the validation of the backup and recovery usage case. However, for optimal performance, it is not enabled during the FC and EFD performance usage cases. Note: These objects are accessed using KNFS. As stated in the tables above, AVM with system-defined pools was used for all NFS file systems in this configuration. The impact of AVM with system-defined pools is a significant improvement in simplicity in configuring the storage. The configuration steps are: 12
1. Create RAID groups on the underlying CLARiiON storage with desired characteristics in terms of RAID type and disk type. 2. Provision the NFS file systems on the system-defined pools. There is no need for further configuration. The system-defined pools are simply created automatically by the Celerra. All of the disks in this configuration were provisioned in this way. Disk layout The following diagram illustrates the disk layout of the environment. Network architecture The network architecture implements the following physical connections: TCP/IP provides network connectivity. DNFS provides file system semantics for Oracle Database 11g. 13
The client and storage networks consist of dedicated network switches and virtual local area networks (VLANs). The storage networks consist of a single IP connection. Jumbo frames are enabled on this network. Hardware resources The hardware used to validate the solution is listed in the following table: Equipment Quantity Configuration Celerra NS-120 (including an EMC CLARiiON CX4-120 back-end storage array) Database server (Oracle 11g server) 1 1 2 Data Movers 1 GbE network connections per Data Mover 2 FC shelves (30 FC 300 GB 15k rpm disks) 1 EFD shelf (5 400 GB 4 GB Fibre Channel EFDs) 1 SATA shelf (15 SATA 1 TB 7200 rpm disks) 1 Control Station 2 storage processors DART version 5.6.47-11 2 2.66 GHz Intel Pentium 4 quad-core processors 24 GB of RAM 146 GB 15k internal SCSI disks 2 onboard GbE Ethernet NICs 2 additional Intel PRO/1000 PT quad-port GbE Ethernet NICs Software resources The software used to validate the solution is listed in the following table: Software Version Red Hat Enterprise Linux 5.4 Microsoft Windows Server 2003 R2 Enterprise Edition 2003 R2 Oracle Database Enterprise Edition 11g (11.2.0.1.0) Quest Benchmark Factory for Databases 5.8.0 EMC Celerra Manager Advanced Edition 5.6 14
EMC DART 5.6.47-11 EMC Navisphere Management 6.29.0.6.34 EMC FLARE 04.29.000.5.003 15
Test and validation Introduction This white paper provides a summary and characterization of the tests that were performed to validate the solution. The testing goal was to characterize the end-toend solution and the component subsystem response under a reasonable load, representing the market for Oracle 11g on Red Hat Enterprise Linux 5 with Celerra NS-120 over DNFS. Store solution The Store solution component was designed as a set of performance measurements to determine the bounding point of the solution stack in terms of performance. A reasonable amount of fine tuning was performed to ensure that the performance measurements achieved were consistent with real-world and best-of-breed performance. Test procedures The following procedures were used to validate the Store solution component: Step Action 1 Close all the Benchmark Factory agents that are running. 2 Restart all the client machines. 3 Stop database instance. 4 Initiate the Benchmark Factory console and agents on the client machines. 5 Start the Benchmark Factory job. 6 Monitor the progress of the test. 7 Capture the results after the test is completed. Test results The summary of the test results for the physically booted Oracle 11g single instance database DNFS configuration is as shown below. The memory on the database server was 24 GB. FC disks performance: Users: 3900; TPS: 192.51; Response Time: 1.392 seconds 16
TPS/Response Time TPS 250 200 150 100 50 0 1000 2000 3000 4000 5000 6000 User Count 14 12 10 8 6 4 2 0 Response Time (in sec) TPS Response Time EFDs performance: Users: 6600; TPS: 311; Response Time: 1.986 seconds TPS/Response Time TPS 350 300 250 200 150 100 50 14 12 10 8 6 4 2 Response Time (in sec) TPS Response Time 0 0 1000 2000 3000 4000 5000 6000 7000 8000 User Count Basic Backup solution The Basic Backup solution component demonstrates that the Oracle 11g configuration is compatible with RMAN disk-to-disk backup. The backup tests are performance tests, where the performance of each node level was observed and RMAN backup/restore was performed on one node. The restore is a functionality test, but the amount of time required to perform the RMAN restore was tuned and measured. The transactions restored and recovered are measured to ensure that there is no data loss. Test procedures The following procedure were used to validate the Basic Backup Solution component: 17
Step Action 1 Close all the Benchmark Factory agents that are running. 2 Close the Benchmark Factory console. 3 Restart the Benchmark Factory console and agents. 4 Stop the database on each node and restart all the nodes. 5 Start listener and database instances on all nodes. 6 Start the Benchmark Factory test with the user load ranging from 1000 to 6000 with an interval of 100. 7 When the user load reaches the 1500 th iteration, initiate RMAN backup on the first node and monitor the performance impact on the production database. 8 Verify that the RMAN backup is completed successfully and allow the test to complete. Test results The RMAN backup operation was performed while the Benchmark Factory load was running. The RMAN backup started at user load 1500 and ended at user load 3900. When RMAN was initiated at user load 1500, there was a moderate increase in the response time and moderate decrease in transaction throughput as shown in the following chart. TPS/Response Time TPS 200 150 100 50 RMAN backup start RMAN backup 0 end 1000 2000 3000 4000 5000 6000 User Count 16 14 12 10 8 6 4 2 0 Response Time (in sec) TPS Response Time Backup and recovery summary The following table provides a summary of the backup and restore test: Test run duration 6 hours and 10 minutes User load range 1000-6000 with interval of 100 Profile Mteoradb51 18
Driver Oracle Start time of test Tue Apr 06 2:59:24 GMT 2010 End time of test Tue Apr 06 9:09:00 GMT 2010 RMAN backup start time 03:44:36 GMT 2010 at user load 1500 RMAN backup end time 06:38:16 CST 2010 at user load 3900 Total time for RMAN backup Total time for RMAN restore Total time for RMAN recovery 2 hours and 54 minutes 3 hours and 42 minutes 1 hour and 36 minutes Basic backup and recovery conclusion RMAN provided a reliable high-performance backup solution for Oracle 11g. However, the time required to restore the database was significant. Advanced Backup solution The purpose of the Advanced Backup solution component was to demonstrate the use of the Celerra s unique storage capabilities with Celerra SnapSure checkpoint. It also demonstrated that the Oracle 11g configuration is compatible with Celerra SnapSure. The backup test run was a performance test. We tested the performance impact while creating a SnapSure checkpoint. The restore was a functionality test, but the amount of time required to perform the SnapSure restore was tuned and measured. The transactions restored and recovered were measured to ensure no data loss. Test procedures The following procedures were used to validate the Advanced Backup solution component: Step Action 1 Close all the Benchmark Factory agents that are running. 2 Close the Benchmark Factory console. 3 Restart the Benchmark Factory console and agents. 4 Stop and restart the database instances. 5 Create a SnapSure checkpoint for the data file system. 6 Start the Benchmark Factory test with the user load ranging from 1000 to 6000. 7 When the user load is at the 1500 th iteration, place the database in hot backup mode, and monitor the performance impact on the production database. 8 After the database is in hot backup mode, refresh the data file system checkpoint. 19
9 Take the database out of hot backup mode and allow the test to complete. 10 After the test is completed, shut down the database and restore the database. 11 Capture the time taken to restore the database. Note: Per metalink note 604683.1, hot backup mode may no longer be needed for creating a recoverable Oracle database backup. For the purposes of this white paper, EMC has continued to use the hot backup approach, but in the future the method described in this metalink note may be explored as well. Test results The advanced backup operation using SnapSure was performed while the OLTP load was running on the database. When the database was taken into the hot backup mode at the 1500 th iteration to refresh the checkpoint, there was a significant increase in response time and a significant decrease in transaction throughput. Apart from that, the TPS to user load was linear and there was minimal impact to the production database as shown in the following chart. TPS/Response Time 200 150 TPS 100 50 Hot backup mode start Hot backup mode end 0 1000 2000 3000 4000 5000 6000 User Count 16 Response Time (in sec) 14 12 TPS 10 Response Time 8 6 4 2 0 Advanced backup and restore summary The following table provides a summary of the advanced backup and restore test: Test run duration 6 hours and 9 minutes User load range 1000-6000 with interval 100 Profile Driver Mteoradb51 Oracle Start time of test Tue Apr 06 23:55:00 GMT 2010 20
End time of test Wed Apr 07 06:04:00 GMT 2010 Hot backup mode start time Wed Apr 07 00:40:30 2010 at user load 1500 Checkpoint refresh start time Wed Apr 07 00:52:16 GMT 2010 Checkpoint refresh end time Wed Apr 07 00:52:16 GMT 2010 Hot backup mode end time Total time for restore Total time for recovery Wed Apr 07 00:52:17 GMT 2010 at user load 1600 16 seconds 2 hour and 4 minutes Advanced backup and recovery conclusion The Celerra SnapSure feature works with Oracle 11g in the configuration. A modest performance hit is observed while taking the database in hot backup mode to refresh the checkpoint. However, this is temporary as the performance recovered to the expected levels after that point for the entire test run. The restore from SnapSure checkpoint is obviously much faster than RMAN disk-to-disk restore. 21
Conclusion Introduction The EMC Celerra unified storage platform s high-availability features combined with EMC s proven storage technologies provide a very attractive storage system for the Oracle RAC 11g over DNFS. Reduced total cost of ownership In any reasonable configuration, the database server's CPU is the most important component of the entire architecture. Therefore, the over-arching principle of EMC's Oracle Database 11g solutions is to free up the database server's CPU (as well as memory and I/O channels) from utility operations such as backup and recovery. The highest and best use of the database server s CPUs is to parse and execute the SQL statements that are required by the application user. CPU usage This solution reduces the load on the database server CPU by using: EMC SnapSure to carry out a logical storage backup of an Oracle 11g production database while offloading all performance impacts of the backup operation from the production server. Oracle DNFS to achieve better performance due to the reduction of memory consumption and CPU utilization. Time saving The Celerra checkpoint command (fs_ckpt) allows the database administrator to capture an image of the entire file system as of a point in time, which is referred to as a logical storage backup. This image takes up very little space and can be created rapidly. In this solution, the checkpoint creation time was 5 seconds and the refresh time was 1 second, while RMAN took 2 hours and 54 minutes for the backup. Also, the process of restoring from a logical storage backup can be completed quickly. In this solution, it was 16 seconds, while RMAN took 3 hours and 42 minutes for the restore. Note: Taking logical storage backups is not enough to protect the database from all risks. Physical storage backups are also required to protect the database against double disk failures and other hardware failures at the storage layer. Improved performance The performance can be improved by the following two parts: EFDs Incorporation of EFDs into EMC Celerra provides a new Tier 0-storage layer that can deliver very high I/O performance at a very low latency, which can dramatically improve OLTP throughput and maintain very low response time. Traditional magnetic disk drive technology no longer defines the performance boundaries for mission-critical storage environments. The costly approach of spreading workloads, over dozens or hundreds of underutilized disk drives, is no longer necessary. Based on the Oracle 11g environment, EFDs can lead to increased performance and savings in power, cooling, and data center floor space requirements. 22
EMC Celerra now combines the performance and power efficiency of EFDs with traditional disk drive technology in a single array managed with a single set of software tools, to deliver advanced functionality, ultra-performance, and extended storage tiering options. DNFS The DNFS client performs concurrent I/O by bypassing the operating system. The benefits include: Consistent NFS performance is observed across all operating systems. DNFS is optimized for database workloads and supports asynchronous I/O, which is suitable for most databases. It delivers optimized performance by automatically load balancing across the available paths. Load balancing in DNFS is frequently superior to the conventional KNFS. Ease of use DNFS The use of DNFS simplifies network setup and management by eliminating administration tasks such as: Setting up network subnets LACP bonding Tuning of Linux NFS parameters Load balancing and high availability (HA) are managed internally within the DNFS client. AVM system-defined storage pool AVM system-defined storage pool provides system administrators with a simple volume and file system management tool. It creates and expands file systems without manually creating and managing stripe volumes, slice volumes, or metavolumes. It is an easy way to create file systems and is adequate for most high availability and performance considerations. Business continuity Advanced backup and recovery with EMC SnapSure dramatically improves the mean time to recovery (MTTR) by reducing the time required for the restore operation. Further, as the backup operation has minimal impact on the database server performance, the backup can be run more often. This means that the recovery operation is also optimized since fewer archived logs must be applied. Robust performance and scaling The performance testing carried out by EMC utilizes an industry-standard OLTP benchmark, but does so without exotic tunings that are not compliant with best practices. In addition, real-world configurations are used. This enables the configuration you choose to run on your application will be predictable and reliable. Next steps EMC can help accelerate assessment, design, implementation, and management 23
while lowering the implementation risks and costs of an end-to- end solution for an Oracle Database 11g environment. To learn more about this and other solutions, contact an EMC representative or visit: http://www.emc.com/solutions/application-environment/oracle/solutions-for-oracledatabase.htm References Reference document For additional information, see the following references: Celerra Forum Celerra Solutions for Oracle Environments Proven Solution Guide EMC Business Continuity for Oracle Database 11g/10g Enabled by EMC Celerra using DNFS and NFS White Papers EMC Solution for Oracle 10g/11g for Midsize Enterprises EMC Celerra Unified Storage Platform Best Practices Planning Leveraging EMC CLARiiON CX4 with Enterprise Flash Drives for Oracle Database Deployments Oracle Database 11g Direct NFS Client 24