EMC VSPEX FOR VIRTUALIZED ORACLE DATABASE 12c OLTP

Transcription

1 IMPLEMENTATION GUIDE EMC VSPEX FOR VIRTUALIZED ORACLE DATABASE 12c OLTP VMware vsphere 5.5 Red Hat Enterprise Linux 6.4 EMC VSPEX Abstract This describes the high-level steps and best practices required to implement the EMC VSPEX Proven Infrastructure for Virtualized Oracle Database 12c on a VSPEX Private Cloud with VMware vsphere enabled by EMC VNXe and EMC Data Protection. Usage guidelines and major customer benefits are also included. February 2015

2 Copyright 2015 EMC Corporation. All rights reserved. Published in the USA. Published February 2015 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC 2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Part Number H EMC VSPEX for Virtualized Oracle Database 12c OLTP

3 Contents Contents Chapter 1 Introduction 7 Purpose of this guide... 8 Business value... 8 Scope... 9 Audience... 9 Terminology Chapter 2 Before You Start 11 Overview Pre-deployment tasks Deployment workflow Deployment prerequisites Planning and sizing the Oracle Database 12c environment Overview Storage pools Example: Small-sized Oracle Database 12c server Essential Reading VSPEX Proven Infrastructure EMC Data Protection for VSPEX Guide VSPEX Design and s Chapter 3 Solution Overview 19 Overview EMC VSPEX Proven Infrastructure Solution architecture Summary of key components Chapter 4 Solution Implementation 25 Overview Preparing the physical components Implementing the network Implementing the storage array Overview EMC VSPEX for Virtualized Oracle Database 12c OLTP 3

4 Tables Set up initial VNXe configuration Configure storage layout for Oracle example Provision storage for NFS example Configure FAST VP on VNXe Configure FAST Cache on VNXe Storage layout example Implementing ESXi and vcenter Preparing the Oracle Database 12c virtual machine template Complete Oracle install prerequisites Install Oracle Database 12c binaries and patches Deploying the Oracle Database 12c virtual machine Overview Configure database file layout for both ASM and dnfs Configure initialization parameters Enable the HugePages setting Create the database using DBCA Perform database post creation activities Implement EMC Data Protection Chapter 5 Solution Verification 41 Verifying hardware Verify ESXi functionality Verify solution component redundancy Use health monitoring tools Verifying Oracle 12c OLTP database performance Overview Verify baseline performance Verify FAST Cache performance acceleration Chapter 6 Reference Documentation 47 Reference documentation EMC Oracle VMware Links Appendix A Configuration Worksheet 51 Configuration worksheet for VSPEX EMC VSPEX for Virtualized Oracle Database 12c OLTP

5 Contents Figures Figure 1. VSPEX Proven Infrastructure Figure 2. Architecture of the validated infrastructure Figure 3. Provisioning storage for VMFS example Figure 4. Provisioning storage for NFS example Figure 5. FAST VP-enabled storage pool example Figure 6. Storage Pool properties: FAST Cache option Figure 7. Example of virtualized Oracle storage layout for the VNXe series 31 Figure 8. Oracle Database Configuration Assistant setup screen Figure 9. Performance comparison before and after FAST Cache enabled. 46 Tables Table 1. Terminology Table 2. Tasks for pre-deployment Table 3. VSPEX for virtualized Oracle deployment process Table 4. Deployment prerequisites checklist Table 5. Storage pools Table 6. Example of customer evaluation: Large-size Oracle Database 12c server Table 7. Example of required resources: Large-sized user database Table 8. Example of additional storage layout recommendations: Largesized Oracle Database 12c server Table 9. Solution components Table 10. Setup tasks for physical components Table 11. Tasks for switch and network configuration Table 12. Tasks for storage configuration Table 13. Storage layout example on VNXe Table 14. Steps for installing and configuring ESXi, vcenter, and infrastructure servers Table 15. Example of a virtual machine template specification Table 16. Oracle install prerequisites Table 17. Installing the Oracle Database binaries and patches Table 18. Location of Oracle files for ASM Table 19. Location of Oracle files for NFS EMC VSPEX for Virtualized Oracle Database 12c OLTP 5

6 Tables Table 20. Tasks to verify the VSPEX installation Table 21. Tools for monitoring the solution Table 22. Example of key metrics for an Oracle Database 12c deployment 43 Table 23. Oracle Database 12c virtual machine profile Table 24. Oracle Database 12c storage pool design example Table 25. Key performance metrics for Oracle Database 12c without FAST Cache Table 26. Key metrics for Oracle Database 12c with FAST Cache Table 27. Comparison before and after FAST Cache enabled Table 28. Common server information Table 29. ESXi server information Table 30. Array information Table 31. Network infrastructure information Table 32. VLAN information Table 33. Service accounts EMC VSPEX for Virtualized Oracle Database 12c OLTP

7 Chapter 1: Introduction Chapter 1 Introduction This chapter presents the following topics: Purpose of this guide... 8 Business value... 8 Scope... 9 Audience... 9 Terminology EMC VSPEX for Virtualized Oracle Database 12c OLTP 7

8 Chapter 1: Introduction Purpose of this guide Business value EMC VSPEX Proven Infrastructures are optimized for virtualizing businesscritical applications. VSPEX provides modular solutions built with technologies that enable faster deployment, more simplicity, greater choice, higher efficiency, and lower risk. VSPEX provides partners with the ability to design and implement the virtual assets required to support a fully integrated virtualized solution for Oracle relational database management systems on a VSPEX private cloud infrastructure. The VSPEX for virtualized Oracle Database 12c infrastructure provides customers with a system capable of hosting a virtualized database solution that is scalable and delivers a constant performance level. This solution uses VMware vsphere 5.5 to provide virtualization supported by the EMC VNXe storage array. The compute and network components, while vendor-definable, are designed to provide redundancy and sufficient power to handle the processing and data needs of the virtual machine environment. This describes how to best implement the VSPEX Proven Infrastructure for virtualized Oracle Database 12c on a VSPEX Private Cloud with VMware vsphere. This VSPEX Proven Infrastructure is focused on helping EMC partners understand the value that the VNXe series and EMC backup and recovery systems bring to customers with Oracle Database 12c deployments who often have growing, isolated IT environments running server-centric applications and who face increasing Oracle backup and recovery issues. This VSPEX solution is designed to meet your Oracle database challenges while enabling customers to increase performance, scalability, reliability, and automation. By consolidating their database applications on VNXe, they can consolidate onto a single centralized storage platform that enables them to more effectively manage the exploding growth in data that is challenging businesses today. This solution has been sized and proven to do the following: Deploy your systems faster, saving time and effort with EMC Proven Solutions Increase performance and scalability out of the box Minimize storage requirements and reduce costs 8 EMC VSPEX for Virtualized Oracle Database 12c OLTP

9 Chapter 1: Introduction Scope This describes the high-level steps required to deploy the VSPEX solution for Oracle Database 12c on a VSPEX Private Cloud for VMware vsphere with an EMC VNXe3200 storage array. This Implementation Guide assumes that a VSPEX Proven Infrastructure already exists in the customer s environment. Audience This guide is intended for internal EMC personnel and qualified EMC VSPEX partners. This guide assumes that the VSPEX partners who intend to deploy this solution are: Qualified by EMC to sell, install, and configure the EMC VNX family of storage systems Qualified to sell, install, and configure the network and server products required for VSPEX Proven Infrastructures Certified for selling VSPEX Proven Infrastructures Partners implementing this solution should also have the necessary technical training and background to install and configure: VMware vsphere virtualization platforms 5.5 Red Hat Enterprise Linux 6.4 Oracle Database 12c This document provides external references where applicable. EMC recommends that partners implementing this solution are familiar with these documents. For details, refer to Essential reading and Chapter 6: Reference documentation. EMC VSPEX for Virtualized Oracle Database 12c OLTP 9

10 Chapter 1: Introduction Terminology Table 1 lists the terminology used in this guide. Table 1. Term AWR dnfs OLTP SGA PGA VMDK VMFS Terminology Definition Automatic Workload Repository is a monitoring utility bundled with Oracle Database 10g and later releases Direct Network File System client can be used to provide access to Network File System (NFS) storage devices Online transaction processing is a class of systems that manage applications for data entry and retrieval transaction processing System global area is a group of shared memory structures that contain data and control information from one oracle database instance Program Global Area is memory specific to an operating process or thread that is not shared by other processes or threads on the system VMware Virtual Machine Disk is an open file format that is a container for virtual hard disk drives for virtual machines VMware Virtual Machine File System is a cluster file system that uses storage virtualization for multiple installations of VMware ESX Server 10 EMC VSPEX for Virtualized Oracle Database 12c OLTP

11 Chapter 2: Before You Start Chapter 2 Before You Start This chapter presents the following topics: Overview Pre-deployment tasks Deployment workflow Deployment prerequisites Planning and sizing the Oracle Database 12c environment Essential Reading EMC VSPEX for Virtualized Oracle Database 12c OLTP 11

12 Chapter 2: Before You Start Overview Pre-deployment tasks This chapter provides an overview of important information you need to be aware of, documents you need to be familiar with, and tasks you need to perform before you start implementing your VSPEX for virtualized Oracle Database 12c solution. The EMC VSPEX for Virtualized Oracle Database 12c OLTP Design Guide for this solution describes how to size and design your solution and how to select the right VSPEX Proven Infrastructure on which to layer Oracle Server. The deployment examples in this are based on the recommendations and examples in the Design Guide. EMC recommends that you complete the pre-deployment tasks as described in the following section before you deploy Oracle Server on a VSPEX Proven Infrastructure. Pre-deployment tasks include those not directly related to the environment installation and configuration, but are prerequisites for installation. These tasks include collecting hostnames, IP addresses, license keys, and installation media. Perform the tasks in Table 2 to decrease the time required at the customer site. Table 2. Tasks for pre-deployment Task Description Reference Gather documents Gather tools Gather data Get the related documents listed in Essential reading. These are used throughout this document to provide details on setup procedures and deployment best practices for the various components of the solution. Get the required and optional tools for the deployment. Use Table 4 to confirm that all equipment, software, and appropriate licenses are available before the deployment process. Get the customer-specific configuration data for networking, naming, and required accounts. Complete the Customer Oracle Database 12c configuration worksheet for reference during the deployment process. Essential reading Deployment prerequisites Appendix A Configuration Worksheet 12 EMC VSPEX for Virtualized Oracle Database 12c OLTP

13 Chapter 2: Before You Start Deployment workflow EMC recommends that you follow the process flow in Table 3 to design and implement your VSPEX Proven Infrastructure for virtualized Oracle Database 12c OLTP solution. Table 3. VSPEX for virtualized Oracle deployment process Step Action 1 Use the VSPEX qualification worksheet to collect user requirements. The qualification worksheet is in Appendix A of the companion EMC VSPEX For Virtualized Oracle Database 12c OLTP Design Guide. 2 Use the VSPEX Sizing Tool to determine the recommended VSPEX Proven Infrastructure for a virtualized Oracle Database 12c OLTP solution. For more information, refer to the VSPEX Sizing Tool on the EMC Business Value Portal. Note: You must register the first time you access the tool. If the VSPEX Sizing Tool is not available, you can manually size the application by using the sizing guidelines in Appendix B of the companion Design Guide. 3 Refer to the companion Design Guide to determine the final design for the VSPEX Proven Infrastructure for virtualized Oracle Database 12c OLTP. Note: Consider all application requirements, not just Oracle application requirements. 4 Refer to the applicable VSPEX Proven Infrastructure document in Essential reading to select and order the right solution. 5 To deploy and test your VSPEX solution, use this. EMC VSPEX for Virtualized Oracle Database 12c OLTP 13

14 Chapter 2: Before You Start Deployment prerequisites Table 4 itemizes the hardware and software requirements to configure the solution. For additional information, refer to the hardware and software tables in the EMC VSPEX for Virtualized Oracle Database 12c OLTP Design Guide. Table 4. Deployment prerequisites checklist Component Description Version Reference notes Hardware Physical servers: Sufficient physical server capacity to host the required number of virtual machines, as recommended by the companion Design Guide and VSPEX Sizing Tool. N/A EMC VSPEX Private Cloud: VMware vsphere 5.5 for up to 200 Virtual Machines Virtual servers: VMware vsphere servers that host the virtual infrastructure servers. This requirement may be covered in the existing infrastructure. vsphere 5.5 Networking: Switch port capacity and capabilities as required by the virtual server infrastructure. N/A EMC VNXe array: Multiprotocol storage array with the required disk layout. VNXe3200 Note: The arrays must provide sufficient additional storage for applications. Software EMC VNXe Operating Environment ( OE) EMC Online Support EMC Unisphere for VNXe EMC vstorage APIs for Array Integration Plug-in Release for vsphere 5.5 VMware ESXi VMware vcenter Server 5.5 Oracle Database Server Red Hat Enterprise Linux Licenses VMware vsphere license keys Oracle Database Server license key 12c Release 1 Standard or Enterprise Edition 14 EMC VSPEX for Virtualized Oracle Database 12c OLTP

15 Chapter 2: Before You Start Planning and sizing the Oracle Database 12c environment Overview Storage pools Follow the recommendations and VSPEX Sizing Tool proposals introduced in the Design Guide to plan and size your Oracle Database 12c environment on the VSPEX Proven Infrastructure. We used the storage pool configuration as detailed in Table 5 in this VSPEX for virtualized Oracle Database 12c solution. Table 5. Storage pools Pool name VSPEX private cloud pool Oracle Database data pool Oracle Database redo pool Oracle Database FRA pool (optional) Purpose Store the virtual machine OS files and database binary files Store database datafiles Store database redo log files Store database backup files Note: Fast Recovery Area (FRA) is a feature Oracle recommends that you enable to store all backup files in one place. If the FRA feature is enabled, the sizing logic for Oracle archived log and backup files is different from that for Oracle datafiles and redo log files. For detailed support, consult your EMC representative. Example: Smallsized Oracle Database 12c server Complete the evaluation as shown in Table 6. A similar example is described in the companion Design Guide. Table 6. Question Example of customer evaluation: Large-size Oracle Database 12c server Do you have an existing Oracle Database 12c server in the environment for which you would like to size? Answer example Yes How many databases do you want to deploy? 1 What is the size of the user database (GB)? 1,000 What is the annual growth rate? 10% Do you intend to use FAST Cache? Do you intend to use FAST VP? What is the maximum number of IOPS (read/write)? No No 2,900/1,900 EMC VSPEX for Virtualized Oracle Database 12c OLTP 15

16 Chapter 2: Before You Start Question (Optional) What is the expected number of concurrent users at peak loads? Answer example 4,500 You will see results similar to those shown in the following tables after you have received a completed qualification worksheet from the customer and entered the answers into the VSPEX Sizing Tool: Table 7 lists the number of virtual machines and their characteristics. Table 8 lists the additional storage hardware that is required to run the OLTP workload. This is in addition to the VSPEX private cloud pool. Table 7 and Table 8 are examples based on the customer information provided in Table 6. The suggested storage layout is in addition to the VSPEX private cloud pool. Table 7. Example of required resources: Large-sized user database vcpu Memory OS volume capacity OS volume IOPS GB 100 GB No. of virtual machines Table 8. Example of additional storage layout recommendations: Large-sized Oracle Database 12c server Pool name RAID type Disk type Disk capacity VSPEX private cloud pool RAID 5 SAS disks 10,000 rpm 600 GB 5 Oracle database data file pool RAID 5 SAS disks 10,000 rpm 600 GB 30 Oracle database log file pool RAID 1/0 SAS disks 10,000 rpm 600 GB 44 No. of disks Note: To plan and size your Oracle Database 12c deployment on the VSPEX infrastructure, follow the recommendations in the companion Design Guide. 16 EMC VSPEX for Virtualized Oracle Database 12c OLTP

17 Chapter 2: Before You Start Essential reading Before implementing the solution described in this document, EMC recommends that you read the following documents, available from the VSPEX space on the EMC Community Network or from the VSPEX Proven Infrastructure Pages on EMC.com. If you do not have access to a document, contact your EMC representative. VSPEX Proven Infrastructure EMC Data Protection for VSPEX Guide VSPEX Design and Implementation Guides Refer to the following EMC VSPEX Proven Infrastructure Guide: EMC VSPEX Private Cloud VMware vsphere 5.5 for up to 200 virtual machines Enabled by Microsoft Windows Server 2012 R2, EMC VNXe3200, and EMC Powered Backup Refer to the following Design and for EMC Data Protection for VSPEX: EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11gR2 Design and Refer to the following VSPEX documents: EMC VSPEX for Virtualized Oracle Database 12c OLTP Design Guide VSPEX Infrastructure as a Service with VMware vcloud Suite Design and EMC VSPEX for Virtualized Oracle Database 12c OLTP 17

18 Chapter 2: Before You Start 18 EMC VSPEX for Virtualized Oracle Database 12c OLTP

19 Chapter 3: Solution Overview Chapter 3 Solution Overview This chapter presents the following topics: Overview EMC VSPEX Proven Infrastructure Solution architecture Summary of key components EMC VSPEX for Virtualized Oracle Database 12c OLTP 19

20 Chapter 3: Solution Overview Overview This chapter provides an overview of the VSPEX Proven Infrastructure for Oracle Database 12c and the key technologies used in this solution. The solution described in this includes servers, storage, network components, and Oracle Database 12c components. The solution enables customers to quickly and consistently deploy a virtualized Oracle Database 12c in the VSPEX Proven Infrastructure. EMC VSPEX Proven Infrastructure A VSPEX Proven Infrastructure, as shown in Figure 1, is a modular virtualized infrastructure validated by EMC and delivered by EMC VSPEX partners. VSPEX includes virtualization, server, and network layers, and EMC storage and backup, designed by EMC to deliver reliable and predictable performance. Figure 1. VSPEX Proven Infrastructure 20 EMC VSPEX for Virtualized Oracle Database 12c OLTP

21 Solution architecture Chapter 3: Solution Overview VSPEX provides the flexibility to choose network, server, and virtualization technologies to create a complete virtualization solution. VSPEX delivers faster deployment for EMC partner customers, with greater simplicity, efficiency, and choice, and lower business risk. You can deploy application-based solutions such as Oracle Server on VSPEX Proven Infrastructures. We validated the VSPEX Proven Infrastructure for virtualized Oracle Server solution using VNXe and a VMware virtualized Linux Server platform to provide storage and server hardware consolidation. You can centrally manage the virtualized infrastructure, which enables you to efficiently deploy and manage a scalable number of virtual machines and associated shared storage. Figure 2 shows the architecture that characterizes the infrastructure validated for an Oracle Database 12c overlay on a VSPEX infrastructure. We used the VSPEX Sizing Tool for Oracle Database 12c to determine the amount of compute resources for each Oracle database. Figure 2 shows the validated architecture for the VSPEX Proven Infrastructure for Oracle 12c Server. We used the VSPEX Sizing Tool to determine the recommended storage layout for Oracle Database 12c and the virtual infrastructure pool in the VNXe3200 storage array. The sizing tool is located on the EMC Business Value Portal. Note: The minimum Oracle version for this solution is We refer to this as 12c throughout this document. EMC VSPEX for Virtualized Oracle Database 12c OLTP 21

22 Chapter 3: Solution Overview Figure 2. Architecture of the validated infrastructure 22 EMC VSPEX for Virtualized Oracle Database 12c OLTP

23 Chapter 3: Solution Overview Summary of key components Table 9 summarizes the key components used in this solution. The companion Design Guide provides more details about the individual components. Table 9. Solution components Layer Application Components Oracle Database 12c Virtualization VMware vsphere 5.5 Compute Network Storage Minimum amount of compute layer resources required, as defined by VSPEX Note: You can implement any server hardware that meets these requirements. Minimum number of network ports required, as defined by VSPEX Note: VSPEX defines the minimum number of network ports required and provides general guidance on network architecture, and enables you to implement any network hardware that meets these requirements. EMC VNXe3200 OS Red Hat Enterprise Linux 6.4 Backup EMC Data Protection solutions EMC VSPEX for Virtualized Oracle Database 12c OLTP 23

24 Chapter 3: Solution Overview 24 EMC VSPEX for Virtualized Oracle Database 12c OLTP

25 Chapter 4: Solution Implementation Chapter 4 Solution Implementation This chapter presents the following topics: Overview Preparing the physical components Implementing the network Implementing the storage array Implementing ESXi and vcenter Preparing the Oracle Database 12c virtual machine template Deploying the Oracle Database 12c virtual machine EMC VSPEX for Virtualized Oracle Database 12c OLTP 25

26 Chapter 4: Solution Implementation Overview This chapter describes the steps required to install, configure, and implement an Oracle Database 12c solution on the VSPEX Proven Infrastructure environment enabled by VMware vsphere 5.5 and VNXe. Note: If you already have a VSPEX Proven Infrastructure environment, you can disregard any implementation steps that you have already completed. Preparing the physical components This section lists the tasks needed to prepare the solution s physical components. After you complete the tasks described in Table 10, the new hardware components will be racked, cabled, powered on, and ready for network connections. Table 10. Setup tasks for physical components Task Description Reference Preparing network switches Preparing servers Preparing VNXe Install the switches in the rack and connect them to power. Install the servers in the rack and connect them to power. Install the VNXe in the rack and connect it to power. Vendor documentation Vendor documentation EMC VNXe3200 Installation Guide Implementing the network This section describes requirements for the network infrastructure that is needed to support the solution architecture. Table 11 provides a summary of the tasks for switch and network configuration along with references for further information. Table 11. Tasks for switch and network configuration Task Description Reference Configuring network Completing network cabling Configure storage network and host infrastructure networking as specified in the solution reference architecture. Connect the following: Switch interconnect ports VNXe ports ESXi NIC/FC ports VSPEX Proven Infrastructure Guide, listed in Essential reading N/A 26 EMC VSPEX for Virtualized Oracle Database 12c OLTP

27 Chapter 4: Solution Implementation Task Description Reference Configuring VLAN Configuring VMware vsphere networking Configure private and public VLANs as required. Create virtual switches for separation of application, management, and storage networks. Vendor documentation VMware vsphere Networking Guide Refer to the companion Design Guide for details on the network implementation. Implementing the storage array Overview This section describes how to configure the VNXe storage array. Table 12 provides a summary of the tasks for storage configuration and references for further information. Table 12. Tasks for storage configuration Task Description Reference Setting up the initial VNXe configuration Provisioning storage Configure the IP address information and other key parameters on the VNXe array. Provision the storage for VMFS datastore for Oracle Server data or create Network File System (NFS) that will be mounted on VMware hosts EMC VNXe3200 Installation Guide VNXe Series Configuration Worksheet Set up initial VNXe configuration Configure storage layout for Oracle example Ensure that network interfaces, IP address information, and other key parameters, such as DNS and NTP, are configured on the VNX before provisioning the storage. Refer to the relevant VSPEX Proven Infrastructure Guide in Essential reading for more information on how to configure VNXe. Follow the recommendations and VSPEX Sizing Tool proposals introduced in the Design Guide before you provision the storage. Table 13 shows an example of a storage layout for Oracle on VNXe. Table 13. Storage layout example on VNXe Storage pool name RAID type Disk type No. of disks Oracle Database data FAST Cache enabled pool RAID 5 (4+1) 10,000 rpm SAS disks 30 RAID 1 (1+1) FAST Cache SSD 2 EMC VSPEX for Virtualized Oracle Database 12c OLTP 27

28 Chapter 4: Solution Implementation Storage pool name RAID type Disk type No. of disks Oracle Database redo pool RAID 10 (2+2) 10,000 rpm SAS disks 4 Creating a pool with the appropriate number of disks To create a pool: 1. In Unisphere, select Storage > Storage Configuration > Storage Pools. 2. Click Create to start the Storage Pool wizard. Manually create a new pool using SAS disks: Use RAID 5 (4+1) for a storage pool with 30 drives for the database datafile pool. This is an example of the large-sized Oracle database. In this example, for database redo log files, use RAID 10 (2+2) for a storage pool with 4 drives. Note: Create your hot spare disks at this point. For more information, refer to the EMC VNXe3200 Installation Guide. Follow the recommendations and VSPEX Sizing Tool proposals in the Design Guide to provision the storage for the VMFS datastore. Figure 3. Provisioning storage for VMFS example 28 EMC VSPEX for Virtualized Oracle Database 12c OLTP

29 Chapter 4: Solution Implementation Figure 3 shows that Oracle databases use different VMDKs to contain the OS, Oracle binaries, Oracle datafiles, and redo log files. Oracle databases also use separate datastores that are mapped and dedicated to their Oracle ASM disk groups for datafiles, redo log files, and other database files. Note: Figure 3 is only an example and actual configurations for customer deployments can differ. Refer to the Design Guide for more information about VMFS recommendations and design. Provision storage for NFS example Oracle Database 12c includes a feature for storing Oracle datafiles on a NAS device by using Oracle Direct NFS (dnfs). With dnfs over IP, all database objects are accessible through an NFS mount. Datafiles, online redo log files, and other database files are accessed with dnfs, as shown in Figure 4. Refer to the Design Guide for more information about the storage layout recommendations and design. Figure 4. Provisioning storage for NFS example Configure FAST VP on VNXe FAST VP increases disk resource efficiency and achieves better performance at a lower cost. FAST VP can automatically adapt to changing business needs by adding additional FAST VP SSDs to the Oracle Database datafile pool. In VNXe, FAST VP enables moving Oracle Server data automatically between FAST VP SSDs and SAS tiers with a slice granularity of 256 MB. This helps to EMC VSPEX for Virtualized Oracle Database 12c OLTP 29

30 Chapter 4: Solution Implementation reduce the response time dramatically and improve the Oracle Database OLTP performance. Enabling FAST VP on VNXe is a transparent operation for Oracle Database Server. No reconfiguration or downtime is necessary. EMC recommends that all the LUNs in the pool have the same tiering policy and that you set the FAST policy for the participating pool LUNs to Start High and Auto-Tier for the Oracle database datafile pool. Refer to the relevant VSPEX Proven Infrastructure Guide in Essential reading for detailed steps about how to configure FAST VP at the LUN level. Figure 5 shows an example of the storage pools created for the VNXe system in this solution. The storage pools use 30 SAS disks and FAST VP to store Oracle datafiles. Figure 5. FAST VP-enabled storage pool example Configure FAST Cache on VNXe By using SSDs in the array as an extended read-and-write cache, FAST Cache increases IOPS and dramatically decreases database response times compared to the high cost, all SAS configuration. Enabling FAST Cache is a transparent operation to Oracle Database Server and no reconfiguration or downtime is necessary. Refer to the Design Guide for more details. Perform the following steps to create and configure FAST Cache: 1. Create FAST Cache on the VNXe storage array. Refer to the relevant VSPEX Proven Infrastructure Guide listed in Essential reading for detailed steps. 2. In Unisphere, click Storage and select Storage Configuration. 30 EMC VSPEX for Virtualized Oracle Database 12c OLTP

31 Chapter 4: Solution Implementation 3. Select Storage Pools, select the Oracle data pool, and then click Details. 4. Click Settings. 5. Click Allow this pool to use the FAST Cache to enable FAST Cache, as shown in Figure 6. Figure 6. Storage Pool properties: FAST Cache option Storage layout example Figure 7 shows the details of storage layout for VNXe. This is an example only. The number of disks used in the VSPEX private cloud pool and Oracle database datafile or redo log file pool may vary according to the customer s requirements. Refer to the Design Guide and the relevant VSPEX Proven Infrastructure Guide listed in Essential reading for more information about the storage layout recommendations and design. Figure 7. Example of virtualized Oracle storage layout for the VNXe series EMC VSPEX for Virtualized Oracle Database 12c OLTP 31

32 Chapter 4: Solution Implementation Implementing ESXi and vcenter This section describes the installation and configuration requirements to support the architecture, including ESXi hosts, vcenter, and infrastructure servers. Table 14 describes the tasks to complete. Table 14. Steps for installing and configuring ESXi, vcenter, and infrastructure servers Task Description Reference Installing ESXi Configuring and deploying vcenter Server Configuring ESXi networking Creating a virtual data center Applying vsphere license keys Install the ESXi 5.5 hypervisor on the physical servers that are being deployed for the solution. Install VMware vcenter Server 5.5. Configure ESXi networking including: NIC trunking, VMkernel port, virtual machine port groups, and jumbo frames. Create a virtual data center. Type the vsphere license keys in the vcenter licensing menu. vsphere Installation and Setup vcenter Server and Host Management Installing vcenter Server 5.5 best practices vsphere Networking vcenter Server and Host Management vsphere Installation and Setup Adding ESXi hosts Connect vcenter to ESXi hosts. vcenter Server and Host Management Configuring vsphere clustering (optional) Performing array ESXi host discovery Create a vsphere cluster. Perform ESXi host discovery in the Unisphere console. vsphere Resource Management vsphere Resource Management Refer to the relevant VSPEX Proven Infrastructure Guide in Essential reading for details on ESXi and vcenter. Preparing the Oracle Database 12c virtual machine template VMware provides the ability to create a template that you can use as a master copy to quickly create and provision virtual machines. By using a template, you can install a guest OS and apply it to a virtual machine with application users and software configured and ready for use with minimal administrative intervention. This minimizes deployment time and avoids repetitive installation and configuration tasks for each virtual machine that is required. Customization specifications, maintained in vcenter, further simplify the rollout of virtual machines. A deployment wizard, automation tool, or script 32 EMC VSPEX for Virtualized Oracle Database 12c OLTP

33 Chapter 4: Solution Implementation can use these templates to automatically create or amend server settings (such as server name, time zone, and network configuration) prior to building the new virtual machine. This section documents the preparation of a VMware virtual machine template to enable rapid deployment of new Oracle Database 12c environments. Complete the following steps to configure the template with the requirements and prerequisites for the Oracle software install: 1. Create a virtual machine with the following resources: 2 vcpus 8 GB of vram 100 GB virtual disk 2. Install the operating system and RPM packages. 3. Configure the system and kernel parameters. 4. Create the OS groups and users. 5. Install and detach the Oracle binaries that are ready for cloning. 6. Convert the virtual machine to a VMware template. Table 15 provides an example of a virtual machine template specification. Table 15. Component CPU Memory Virtual disk Example of a virtual machine template specification Description 2 vcpus 8 GB 100 GB with: 8 GB for the swap partition 92 GB for the bootable root partition Operating system Red Hat Enterprise Linux Server release 6.4 Kernel Network interfaces OS users OS groups Eth0: public/management IP network Eth1: Storage Network Username: oracle Group names: oinstall and dba Database binaries Oracle EMC VSPEX for Virtualized Oracle Database 12c OLTP 33

34 Chapter 4: Solution Implementation Complete Oracle install prerequisites Table 16 provides the steps to install and configure the operating system and Oracle prerequisites in the virtual machine before converting it to a template. Table 16. Oracle install prerequisites Task Installing the guest OS Installing VMware tools Installing the required RPM packages Setting the kernel parameters Setting the resource limits for the Oracle user Creating the required software directories Configuring HugePages Description Install the Red Hat Enterprise Linux bit operating system and configure the network interfaces. Install VMware tools using Red Hat Package Manager (RPM) Install or update required RPM packages Configure the kernel with Oracle-specific settings for semaphores, virtual and shared memory, network, and asynchronous I/O. Check and adjust the resource limits for the Oracle software installation users Create the required directories for the Oracle software Configure HugePages to reduce the number of memory pages and virtual memory management overhead. Reference (on the Oracle website) Red Hat Enterprise Linux 6 Installation Guide Installing VMware Tools in a Linux virtual machine using Red Hat Package Manager (RPM) (VMware Knowledge Base Article ID ) Oracle Database Installation Guide 12c Release 1 (12.1) for Linux Latest version of My Oracle Support Note ID : USE_LARGE_PAGES To Enable HugePages Install Oracle Database 12c binaries and patches Table 17 includes the steps required to install the Oracle Database 12c binaries and patches. Table 17. Task Installing the Oracle database binaries Installing the Oracle Database binaries and patches Description Run the Oracle Universal installer and select the following installation option, which installs only the database binaries: Install database software only Reference (on the Oracle website) Oracle Database Installation Guide 12c Release 1 (12.1) for Linux Installing the latest PSU and CPU Install the latest database Patch Set Update (PSU). A user ID, password, and a support contract are required. Latest available version of Critical Patch Updates, Security Alerts, and Third Party Bulletins 34 EMC VSPEX for Virtualized Oracle Database 12c OLTP

35 Chapter 4: Solution Implementation Task Enabling dnfs Description Enable Oracle dnfs (direct network file system) if Oracle Database is not shipped with Direct NFS enabled by default. Reference (on the Oracle website) Oracle Database Installation Guide 12c Release 1 (12.1) for Linux EMC VSPEX for Virtualized Oracle Database 12c OLTP 35

36 Chapter 4: Solution Implementation Deploying the Oracle Database 12c virtual machine Overview The following methods are available for deploying a virtual machine from a template with custom specifications: Choosing Deploy Virtual Machine from this Template in vcenter Choosing Customize using an existing customization specification in vcenter Use the VSPEX for Virtualized Oracle Qualification Worksheet in the Design Guide to size the virtualization infrastructure and the overall NFS file systems used to store the Oracle database. That Design Guide also provides the Oracle initialization parameters required by the database. Determine the size of the individual datafiles based on the schema design and table space placement of tables and indexes. Refer to Configure database file layout for both ASM and dnfs for more details. Refer to the Design Guide and Configure initialization parameters for guidance on configuring the initialization parameters. Configure database file layout for both ASM and dnfs In this solution the Oracle 12c Database was built with two configurations to test the unified storage platform solution with different protocols. An ASM implementation with a RAID-protected ASM disk group and a dnfs implementation with a RAID-protected NFS file system were used to build the database. The back-end storage layout, excluding the file system type, is almost the same for both configurations. Database file layout for ASM Table 18 lists datafiles, online redo log files, CRS files, and other database files that reside on Oracle ASM-managed storage. The database was built with several distinct ASM disk groups: +DATA, +REDO, and +CRS. Table 18. Location of Oracle files for ASM Content Database binary files Datafiles, control files Online redo log files OCR and voting disk files Location Database server s local disk, or the file system dedicated to virtual servers +DATA +REDO +CRS Oracle ASM is the default file system in this solution, but Red Hat 6.4 supports multiple file systems, such as VFAT, ext2,ext3, ext4, and ReiserFS. Oracle 36 EMC VSPEX for Virtualized Oracle Database 12c OLTP

37 Chapter 4: Solution Implementation generally does not certify file systems, but Linux is a specific case. The current support includes ext3, ext4, and OCFS2. Database file layout for dnfs Oracle Database components, such as datafiles and online redo log files, are stored by using Oracle Direct NFS (dnfs). These file systems use the appropriate RAID level and number of disks for each type of file. Table 19 lists the datafiles, online redo log files, and other database files and indicates where they reside. Table 19. Location of Oracle files for NFS Content Database binary files Datafiles, control files Online redo log files Location Database server local disk or file system dedicated for virtualized servers Spread across file systems dedicated to Oracle datafiles Multiplexed across two different file systems dedicated to redo log files EMC recommends using the Oracle dnfs client, which offers improved performance over the operating system-supplied for the NFS client. The Oracle dnfs client provides enhanced resiliency and performance by providing automatic failover on Gb Ethernet fabric, and performing concurrent I/O that bypasses any OS caches or write-order locks. dnfs also performs asynchronous I/O that allows processing to continue while the I/O request is submitted and processed. Note: Oracle recommends that you set the NFS buffer size parameters to 32,768 in the system configuration file /etc/fstab. However, you can use the file oranfstab to specify additional Oracle Database specific options for Direct NFS. For more details, refer to Database Installation Guide 12c Release 1 (12.1) for Linux on the Oracle website. Configure initialization parameters Use one of the following methods to configure the initialization parameters: Automatic Shared Memory Management Implement Automatic Shared Memory Management (ASMM) to automate the management of shared memory structures. The following shared memory structures are automatically managed: DB_CACHE_SIZE SHARED_POOL_SIZE LARGE_POOL_SIZE JAVA_POOL_SIZE STREAMS_POOL_SIZE EMC VSPEX for Virtualized Oracle Database 12c OLTP 37

38 Chapter 4: Solution Implementation ASMM enables the Oracle database to distribute memory between these components based on workload requirements. These components are considered auto tuned parameters. The Memory Manager (MMAN) background process coordinates the sizing of these components and moves memory to where it is needed most. Set the following initialization parameters to implement this feature: SGA_TARGET set to a nonzero value STATISTICS_LEVEL set to TYPICAL (or ALL) Automatic Memory Management Do not use Oracle Automatic Memory Management (AMM) because AMM is incompatible with HugePages. If you want to use HugePages make sure that both MEMORY_TARGET and MEMORY_MAX_TARGET initialization parameters are not set. For more information, refer to My Oracle Support Note ID on the Oracle website. I/O operations for file system files Set FILESYSTEMIO_OPTIONS = SETALL to enable direct I/O and asynchronous I/O for file system files. With asynchronous I/O, normal processing continues while the I/O request is submitted and processed. This enables asynchronous I/O to bypass some of the performance bottlenecks associated with I/O operations. Direct I/O is available in the file system and delivers data directly to the application without caching in the file system buffer cache. Direct NFS does not depend on the value of FILESYSTEMIO_OPTIONS. Direct NFS always issues asynchronous and direct I/O, as it does not depend on OS support. Set FILESYSTEMIO_OPTIONS = SETALL if the OS supports it to back up the OS NFS client in case of a misconfiguration. Enable the HugePages setting HugePages was enabled in the Linux Kernel as part of the template creation process, as described in Table 16. HugePages is crucial for faster Oracle database performance on Linux if you have a large amount of RAM and SGA size. You need to configure HugePages if your combined database SGAs are more than 8 GB. Refer to the latest version of My Oracle Support Note ID : HugePages on Oracle Linux 64-bit on the Oracle website for more details. Create the database using DBCA The Database Configuration Assistant (DBCA) is the preferred way to create a database because it is a more automated approach and the database is ready 38 EMC VSPEX for Virtualized Oracle Database 12c OLTP

39 Chapter 4: Solution Implementation to use when DBCA completes. Start DBCA as a stand-alone tool to create an Oracle Database, as shown in Figure 8. Type the following command in a UNIX or Linux shell or Windows command prompt and press Enter to start DBCA: dbca Note: The dbca utility is typically located in the ORACLE_HOME/bin directory. Figure 8. Oracle Database Configuration Assistant setup screen Oracle DBCA runs in interactive mode. You can either modify one of the preconfigured database types or customize the database by using the screens in Oracle DBCA. General Purpose/Transaction Processing is a preconfigured database template that you can create and customize during the installation. This involves amending the following: Initialization parameters Size and placement of the database files to use the NFS mount points Perform database post creation activities After you create a database as described in Create the database using DBCA, the instance is open and available for normal database use. However, the Oracle database does not automatically start or stop with the virtual machine. EMC VSPEX for Virtualized Oracle Database 12c OLTP 39

40 Chapter 4: Solution Implementation Refer to My Oracle Support Note ID : How to Automate Startup/Shutdown of Oracle Database on Linux on the Oracle website to automate the Oracle Database 12c startup and shutdown processes. Implement EMC Data Protection All VSPEX solutions are sized and tested with EMC Data Protection products, including EMC Avamar and EMC Data Domain. If your solution includes EMC Data Protection components, refer to EMC Backup and Recovery Options for VSPEX for Virtualized Oracle Database 11gR2 Design and Implementation Guide for detailed information on implementing these options into your VSPEX solution. 40 EMC VSPEX for Virtualized Oracle Database 12c OLTP

41 Chapter 6: Solution Verification Chapter 5 Solution Verification This chapter presents the following topics: Verifying hardware Verifying Oracle 12c OLTP database performance EMC VSPEX for Virtualized Oracle Database 12c OLTP 41

42 Chapter 6: Solution Verification Verifying hardware This section provides a list to review after configuring this VSPEX solution. Use the information in this section to verify the configuration and functionality of specific aspects of the solution, and to ensure that the configuration supports core availability requirements. Table 20 describes the required verification tasks. Table 20. Tasks to verify the VSPEX installation Task Verify the basic ESXi functionality of the solution with a post-installation checklist. Reference Using EMC VNX storage with VMware vsphere TechBook Verify the redundancy of the following solution components: Storage ESXi host Network switch Use the monitoring tool to monitor the health of the VSPEX Proven Infrastructure for virtualized Oracle databases Vendor documentation VSPEX documentation Verify ESXi functionality Verify solution component redundancy EMC recommends that you verify the ESXi configurations before deploying the system to production on each ESXi server. Refer to the relevant VSPEX Proven Infrastructure Guide in Essential reading for more detailed information. You can test specific scenarios related to maintenance or hardware failure to ensure that the various components of the solution maintain availability requirements. EMC recommends that you verify redundancy of the solution components, including storage, ESXi hosts, and network switches. Refer to the relevant VSPEX Proven Infrastructure Guide in Essential reading for details. Use health monitoring tools Health monitoring is a simplified measurement that reflects the reliability, stability, and performance of the entire VSPEX Proven Infrastructure for a virtualized Oracle database. Table 21 lists some tools for you to monitor and troubleshoot your VSPEX solution. 42 EMC VSPEX for Virtualized Oracle Database 12c OLTP

43 Chapter 6: Solution Verification Table 21. Tool Esxtop Tools for monitoring the solution Description The esxtop tool provides a real-time view (updated every five seconds, by default) of ESXi hypervisor performance metrics, such as CPU utilization. These performance metrics enable you to measure and troubleshoot Oracle database performance-related issues. Automatic Workload Repository (AWR) The Automatic Workload Repository (AWR) collects, processes, and maintains performance statistics for problem detection and selftuning. This data is stored both in memory and in the database. The gathered data can be displayed in both reports and views. EMC recommends that you use these tools to monitor the performance of the entire Oracle Database environment. Refer to the companion Design Guide for details about performance verification and test methodology. Verifying Oracle 12c OLTP database performance Overview Verify baseline performance This section provides an example performance test to verify Oracle 12c OLTP on a VNXe3200. A test enables you to ensure this VSPEX solution meets your business requirements. Refer to the Design Guide to better understand test methodology before you verify your solution. Two storage pools without FAST Cache enabled (one for Oracle datafiles and the other for redo log files) were built on the SAS 10K RPM disk to verify the baseline performance. Key metrics After you identify whether or not to use FAST Cache and the business requirements are satisfied, you must choose which Oracle 12c testing metrics to capture and the thresholds for each metric when running the Oracle Database 12c verification tests. Refer to the VSPEX Sizing Tool to identify the Oracle Database 12c key performance metrics that are shown in Table 22. Table 22. Example of key metrics for an Oracle Database 12c deployment Metric Threshold Average CPU utilization (%) Less than 80% Average disk latency Less than 20 ms The Oracle Database 12c virtual machine profile in Table 23 was used to verify that the compute and storage infrastructure satisfies the performance requirements. EMC VSPEX for Virtualized Oracle Database 12c OLTP 43