Consolidate with Oracle Exadata

Consolidate with Oracle Exadata Manage Resources and Availability - CON7770 Sue K. Lee, Director of Development René Kundersma, Consulting Member of Tech. Staff Oracle Server Technologies Oct 1, 2014

Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.

Program Agenda 1 2 3 4 5 Introduction to Consolidation Oracle Exadata - Optimized for Consolidation and DBaaS HA Tiers and Consolidation Planning Best Practices Operational Best Practices in a consolidated environment Managing Resources in Consolidated Environments

Why Consolidate? Efficient server and storage utilization Each generation of servers and storage is more powerful Typical database workload may not fully utilize hardware Database workloads are often bursty, with long periods of low utilization Lots of test, development, and (non-)critical databases reduce hw footprint Fewer systems to administer Reduce maintenance efforts

Consolidation Challenges Bridge not redundant = Single Point Of Failure More tenants than resources

Consolidation Challenges HA challenges Conflicting HA requirements Planned maintenance difficult Database users apprehensive about consolidation Users want security, isolation and performance guarantees Workload surges from one application can affect others Excessive CPU, PGA, or I/O usage Surges can originate from heavy application usage or a runaway query DBAs want to control resource usage Get what you pay for and get consistent performance

Consolidation Methodologies Schema Consolidation Multiple applications share a database Server Consolidation Multiple databases share a server Multi-Tenancy -Multiple Pluggable Databases share a Container Sales PDB No right approach! Each approach has its pros and cons! Multitenant Database Assets Billing PDB PDB Parts PDB Single Physical Container Database Single O/S, No VMs Needed

No Performance Bottlenecks for Consolidation and DBaaS Query offload in storage Data intensive query operations offloaded to storage CPUs 100 GB/sec SQL data throughput Storage Index data skipping Database storage compression Hybrid Columnar for 10x DB size reduction and faster analytics Database optimized PCI Flash Smart caching of database data 2.66 Million Database IOs/sec Smart Flash log speeds transactions Database optimized and comprehensive resource management End-to-end prioritization from application to DB to network to storage Prioritization of CPU and IO by multitenant pluggable database (12c) Database optimized messaging SQL optimized InfiniBand protocol for high throughput low latency SQL End-to-End prioritization of critical database messages (11.2.0.4), including log writes and RAC

Consolidation Test: Exadata vs. Non-Exadata End User End User Application Servers OLTP/DW Load Drivers X4-2 / 250Gb memory Application Servers OLTP/DW Load Drivers X4-2 / 250Gb memory Exadata X4-2 Full Rack Database Servers 12

Exadata for OLTP: 4X Consolidation Density Challenge: How many more databases can be consolidated on Exadata, compared to a fast x86 system? Approach: Compare the same Exadata system, with and without Exadata features* enabled, increasing databases until saturation. Results: 160 databases (CPU-bound) on Exadata, compared to 40 databases (I/O-bound) on equivalent x86 hardware. Exadata consolidates 4X as many databases with faster transaction response times. *Includes Exadata Smart Flash Logging, Smart Flash Cache, Smart Scan, Smart Flash Cache compression, Storage Indexes, Network RM, IORM

Exadata for Mixed Workloads: 15X Faster Challenge: How much faster can Exadata execute mixed workloads, compared to a fast x86 system? Approach: Compare the same Exadata system, with and without Exadata features* enabled, when Exadata is undersubscribed. The workload is OLTP plus a reporting Data Warehouse. Results: Exadata responds 15X faster with over 6X more transactions while still consolidating twice as many databases. *Includes Exadata Smart Flash Logging, Smart Flash Cache, Smart Scan, Smart Flash Cache compression, Storage Indexes, Network RM, IORM

Consolidation Principles Standardize Consolidate Monitor and Manage Set up HA reference architectures and map databases to tiers Setup hardware pools. Migrate one database at a time. Start Smart Manage Resources from the start

Exadata MAA Portfolio Products & Features Real Application Clusters Automatic Storage Mgmt Data Guard Active Data Guard GoldenGate Application Continuity Flashback Global Data Services Oracle Secure Backup Redundant database servers, storage servers, network, power Active-active DB clusters. Mirrored storage and flash. Log-based data replication with data consistency validation http://oracle.com/maa LAN Local Standby Active standby systems, local HA and/or remote DR WAN Remote Standby Online patching, reconfiguration and rolling upgrades

What do I really need? Where to Start? Step1 RTO RPO Requirement Maintenance window DR Time What solutions to use? Define key HA requirements: RTO, RPO and DR time first!

Planning for Exadata Consolidation Step 2 High Availability Tiers for Unplanned and Planned Outages Platinum Silver Bronze Unplanned Outages (local site) Zero outage for platinum ready applications HA with automatic failover Single instance with auto restart for recoverable instance and server failures Planned Maintenance Zero application outage Gold Comprehensive HA/DR All rolling or online Some rolling, some online, and some offline Some online, most offline Data Protection Comprehensive runtime validation combined with manual checks Comprehensive runtime validation combined with manual checks Basic runtime validation combined with manual checks Basic runtime validation combined with manual checks Unrecoverable local outages and DR Zero application outage for platinum-ready applications, in-flight transactions are preserved, zero data loss Real-time failover, zero or near-zero data loss Restore from backup, potential to lose data generated since last backup Restore from backup, potential to lose data generated since last backup

Oracle MAA Availability Tiers Reference Architectures PLATINUM GOLD App Cont. / GDS Clusters and Replication Clusters Replication (GG/DG) SILVER BRONZE Clusters Backups Single Instance Backups

Planning for Exadata Consolidation Step 3 Assign Databases to Availability Tier Apps IT I need a database for the new HR system I require HA with automatic failover PLATINUM GOLD Appl. Cont. Clusters and Replication Clusters Replication I need to copy a production database for testing SILVER Clusters Backups Apps QA The cheapest configuration is fine. This is just for testing BRONZE Single Instance Backups

Planning for Exadata Consolidation Step 4 Assign tiers to their hardware pool Hardware Pool: A machine or group of machines used as target consolidation platform Guidelines Do not assign different tiers to the same hardware pool Minimum: Exadata X4-2 Half Rack Critical smaller Hardware Pools should use standby When more capacity is required than provided by a single hardware pool, divide the target databases in that category into two separate groups

Consolidation and Large Environments Best practice for consolidated environments Consolidation GI version >= DB version Required to 4th digit, recommended to 5th digit Large Environments Mixed Exadata version supported Recommended only during rolling upgrade Advance diskgroup compatible.rdbms with care Once advanced cannot be reset Share database homes whenever possible Partially extend /u01 size (leave snapshot room) Patch Out-of-Place Move databases individually to new home Split updates across multiple maintenance window In general upgrade down the stack 1. Grid Infrastructure / Database 2. Exadata Database Servers 3. Exadata Storage Servers 4. InfiniBand switches

Role Separation and ASM/DB scoped security Segregating databases from each other in a platform consolidation environment To separate Grid/ASM administration from database administration, request a job role-separated environment for the initial Exadata installation Especially effective to manage multiple administrators Privileges on griddisk level Restrict griddisks to certain clusters and/or certain database(s). Exadata ASM v.s. database-scoped security CellCLI> create key 4bb27f70c17f88232c936ed1fc437049 CellCLI> ASSIGN KEY FOR +ASM= 4bb27f70c17f88232c936ed1fc437049 CellCLI> ALTER GRIDDISK ALL availableto=\'+asm\ Place key on the database server.

Maintain Stability of the environment Apply consolidation best practices when adding databases: Settings to review: Huge-pages Minimize process count Operations 1. Exachk 2. Monitor / Calculate 3. Test / Validate 4. Add / Validate Set ASM process count Adjust SEMMSL and SEMMNS Instance Cage the DWH workload Use iorm objective=auto 4. Change Manage resources (CPU, Disk and Flash)

Reduce Risk and Downtime with Data Guard Best data protection for consolidated environments Gold and Platinum Reduce Planned Maintenance Risk and Downtime with Data Guard 1. Update standby system then test 2. Switchover then update new standby Standby First Apply to BP, PSU, CPU and PSE Max 1 year in between Example: 11203BP5=>11203BP9 Transient Logical Standby / GoldenGate Major Database Patch sets Example: 11.2.0.4 => 12.1.0.2 (DBMS_ROLLING starting 12.1.0.2)

Performance Tiers Private Jet PLATINUM GOLD SILVER BRONZE Lowest consolidation density Highest performance isolation Low consolidation High performance isolation High consolidation Medium performance isolation Highest consolidation Lowest performance isolation 1 st Class Economy Class

Resource Contention Databases consolidated on common hardware contend for CPU Memory Flash Space and Bandwidth Disk Bandwidth How can we guarantee resources for each database and achieve good consolidation density?

Managing CPU

The Need for Instance Caging Your database starts with enough CPU, resulting in good performance! And then another database hogs the CPU, leaving you with not enough CPU and bad performance 100 90 80 70 60 50 40 Marketing Support 30 20 10 0

Manage CPU with Instance Caging 100 Cage a database by limiting the amount of CPU it can use. 90 80 70 60 50 40 MARKETING SUPPORT 30 20 10 0 Step 1: Set cpu_count to the max CPU threads the instance can use at any time alter system set cpu_count = 8; Step 2: Set resource_manager_plan to enable CPU Resource Manager alter system set resource_manager_plan = default_plan ;

Partition Approach for Gold Databases Partition CPUs among the database instances sum(cpu_counts) <= # cpu threads Partitioning provides maximum isolation No CPU contention between instances Best for performance-critical databases 32 28 24 20 16 12 8 Instance D: 4 CPUs Instance C: 4 CPUs Instance B: 8 CPUs This server has a total of 24 CPU threads Instance B s cpu_count But if Instance A is idle, its CPU allocation is unused 4 0 Instance A: 8 CPUs

Over-Subscribe Approach for Bronze Databases Over-subscribe the CPUs among the database instances sum(cpu_counts) <= 3 x # cpu threads Over-subscribing provides efficient CPU utilization Some contention for CPU if databases are sufficiently loaded Best for non-critical databases 32 28 24 20 16 12 8 4 This server has a total of 24 CPU threads Instance D: 8 CPUs Instance C: 8 CPUs Instance B: 8 CPUs Instance A: 8 CPUs Instance B s cpu_count 0

Over-Subscribe Approach for Bronze Databases If Instance A is idle, there is no CPU contention! The databases can fully utilize the server. 32 28 24 20 16 Instance D: 8 CPUs This server has a total of 24 CPU threads 12 Instance C: 8 CPUs 8 4 Instance B: 8 CPUs 0

Over-Subscribe Approach for Bronze Databases If all databases are active, there is some contention. This is your maximum excess load. Without Instance Caging, there is no limit on your maximum load! 32 28 24 20 This server has a total of 24 CPU threads Instance D: 8 CPUs Instance C: 8 CPUs 16 12 Instance B: 8 CPUs 8 4 Instance A: 8 CPUs 0

Instance Caging Best Practices Keep the strategy simple Initial cpu_count settings can be a guess Monitor actual CPU usage and then tweak cpu_count can be adjusted dynamically no database bounce needed! If over-subscribing, monitor the server to make sure it s not over-loaded Avoid large changes to cpu_count cpu_count corresponds to CPU threads, not cores! Beware of over-subscribing on SPARC T-Series processors Don t set cpu_count to 1 Makes database instance susceptible to hangs or poor performance

Managing Memory

How Is Physical Memory Used? 512 GB of RAM Kernel Memory OS Page Tables SGA DB #1 SGA DB #2 SGA DB #3 PGA DB #1 PGA DB #2 PGA DB #3 Your physical memory should look like this!

How Is Physical Memory Used? 512 GB of RAM Kernel Memory OS Page Tables SGA DB #1 SGA DB #2 SGA DB #3 PGA DB #1 PGA DB #2 PGA DB #3 Excessive memory usage causes paging, leading to performance problems, leading to RAC evictions. If you see WARNING: Heavy swapping observed on system in the alert log, take action! (11.2.0.4+)

Memory Configuration Huge Pages 512 GB of RAM Kernel Memory OS Page Tables SGA DB #1 SGA DB #2 SGA DB #3 PGA DB #1 PGA DB #2 PGA DB #3 With huge pages, shrink OS page tables from 8 GB to 16 MB! And boost performance! Use Huge Pages! See MOS notes #361468.1 and #401749.1.

Memory Configuration SGA 512 GB of RAM Kernel Memory OS Page Tables SGA DB #1 SGA DB #2 SGA DB #3 PGA DB #1 PGA DB #2 PGA DB #3 Configure SGA with SGA_TARGET and huge pages. (MEMORY_TARGET doesn t support huge pages.) Coordinate SGA_TARGET and huge page configuration. PGA cannot use huge pages, so excessive huge pages waste memory.

Memory Configuration PGA 512 GB of RAM Actual PGA usage. Can be 3x bigger than PGA_AGGREGATE_TARGET! Kernel Memory OS Page Tables SGA DB #1 SGA DB #2 SGA DB #3 PGA DB #1 PGA DB #2 PGA DB #3 PGA_AGGREGATE_TARGET. Who obeys PGA_AGGREGATE_TARGET? Good citizens: hash joins, sorts ( tunable operations that can spill to temp space) Bad citizens: parallel queries with high DOPs, badly behaved PL/SQL Monitor v$pgastat maximum PGA allocated historical max total PGA allocated current usage

Memory Configuration PGA 512 GB of RAM PGA_AGGREGATE_LIMIT is a hard limit Kernel Memory OS Page Tables SGA DB #1 SGA DB #2 SGA DB #3 PGA DB #1 PGA DB #2 PGA DB #3 PGA_AGGREGATE_LIMIT New in 12c PGA usage will not exceed this hard limit! Biggest untunable memory users are throttled and eventually killed Default value based on available memory and pga_aggregate_target PGA_AGGREGATE_TARGET.

Managing Exadata Flash

Exadata Smart Flash Cache Exadata Flash provides impressive capacity and bandwidth* 45 TB space 100 GBps of scan throughput ~2M IOPS Use Exadata Flash to Cache frequently-used OLTP data Accelerate scans, using excess flash resources Accelerate log writes * For a full X4-2/8 rack

Exadata Smart Flash Cache Why Exadata? Other market offerings All-flash storage: expensive Disks with flash to accelerate writes and cache recently-used data: doesn t support mixed workloads Only Exadata guarantees that flash stores frequently-accessed OLTP data Scans must not evict OLTP data from the cache Only Exadata can identify OLTP from scan IOs! All-flash storage guarantees this too, but at many times the price!

Exadata Smart Flash Cache Flash Bandwidth How can you capitalize on 100 GBps of flash bandwidth? Use up to 20% for OLTP workloads Maximum sustainable rate by OLTP Use the rest for scans! Pre 11.2.2.3, mark tables with KEEP 11.2.2.3+, scans automatically use flash if there s space Scans use both flash and disks for maximum throughput Using flash increases scan throughput by up to 5x! OLTP s peak load only consumes 20% of flash throughput Scans OLTP Flash Bandwidth Use excess throughput for scans!

Exadata Smart Flash Cache Space Contention Flash Logs use just 0.02% of total space. Don t disable! Inter-Database IORM Plan Use Flash Log? Use Flash Cache? Disabling flash cache may cause a heavier disk I/O load DB #1 DB #2 DB #3 For most workloads, Exadata already uses flash space optimally. To customize, use the Inter-Database IORM Plan

Managing Disk

Use Case for IORM: Scan Workloads 100 90 80 70 60 Your disk utilization is around 50% with 1 database. Should you consolidate and add another database? 50 Sales 40 30 20 10 0 Disk Utilization by Database Disk Utilization by Database

Use Case for IORM: Scan Workloads Non-Exadata Storage 100 90 80 70 The new database hogs the disk bandwidth. Without Exadata IORM, databases issuing the most load get the most bandwidth. 60 50 40 30 20 Marketing Sales The original database sees a sudden drop in throughput. 10 0 Disk Utilization by Database

Use Case for IORM: Scan Workloads Exadata Storage 100 90 80 One database can use excess disk bandwidth without affecting the other! 70 60 50 40 Marketing Sales 30 20 10 0 Disk Utilization by Database IORM ensures predictable throughput for each database.

Exadata IORM To enable IORM Set IORM objective from basic to auto, using CellCLI or EM Cloud Control 12c CellCLI> alter iormplan objective = auto Configure a resource plan By default, IORM runs in basic mode, which ensures reasonable latencies for OLTP I/Os. Shares specify the relative importance of each database. A database with 2x shares can consume 2x the disk bandwidth. Database Shares Utilization Limit Inter-Database IORM Plan Guaranteed Disk Utilization Maximum Disk Utilization Sales 2 50% 100% Support 1 25% 100% Marketing 1 50% 25% 50% Default 1 25%

Exadata IORM To enable IORM Set IORM objective from basic to auto, using CellCLI or EM Cloud Control 12c CellCLI> alter iormplan objective = auto Configure a resource plan By default, IORM runs in basic mode, which ensures reasonable latencies for OLTP I/Os. You can also place a hard limit on the disk I/O utilization for a database. Primarily for cloud deployments for pay for performance. Database Shares Utilization Limit Inter-Database IORM Plan Guaranteed Disk Utilization Maximum Disk Utilization Sales 2 50% 100% Support 1 25% 100% Marketing 1 50% 25% 50% Default 1 25%

Use Case for IORM: High Disk Latencies for OLTP % of Waits ----------------------------------------------- Total Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s -------------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- cell single block physical 213.7 90.1 3.3.8.1.1 2.0 3.6 log file parallel write 951.5 93.9 2.6 1.3 1.1 1.0.0.0 On Exadata, most OLTP I/Os are serviced from flash, except Flash cache misses Log writes when flash is busy During a smart scan, OLTP disk I/Os can be very slow Enable IORM to avoid outliers. Use the objective knob to enforce low disk latencies.

Use Case for IORM: High Disk Latencies for OLTP Non-Exadata Storage OLTP A smart scan can issue a flood of IOs, dominating the front of the queue. Parallel Execution Table Scan O/S Disk I/O Queue 200 ms wait time Long waits for OLTP IOs mean long latencies. I/Os are processed in the order received. The O/S usually lets small OLTP I/Os cut towards the front of the queue. But waiting behind the I/Os in front can still take 100s of ms.

Use Case for IORM: High Disk Latencies for OLTP Exadata Storage OLTP IORM tags every IO: Database, PDB, Consumer Group id Background or foreground? Buffer cache or smart scan? Exadata IORM can put a new IO anywhere in the queue! Parallel Execution Smart Scan Exadata IORM IO Queue Exadata Disk Exadata IORM controls the order of the queue, based on the Resource Plan.

Managing Standby Databases

Resource Manager for Standbys Primary Database Standby Database Database X 1) Primary DB Workload can slowdown 2) Redo and Apply 3) Read-Only Queries can slowdown 4) DB X s Workload Issue #1: If synchronous redo transport is enabled, the Primary DB will suffer from slow redo writes on the Standby Issue #2: Heavy read-only queries and other database workloads can slow the Standby s redo and apply

Resource Manager for Standbys Prioritize Redo and Apply over Read-Only Queries Enable CPU and I/O resource management Set a database resource plan on Standby Use the primary s resource plan or DEFAULT_PLAN (best practice - use the same resource plan in case a switch-over occurs!) Enable I/O Resource Manager (alter iormplan set objective = auto) On physical standbys, all resource manager objects are replicated on the standby New resource plans cannot be configured on the standby The primary and standby can use different resource plans See MOS note 1930540.1 for known issues for consumer group mappings On logical standbys, all resource manager objects must be recreated

Resource Manager for Standbys Guarantee Resources for the Standby Enable Instance Caging for all database instances on the Standby s server CPU_COUNT must be sufficiently high to handle the load in the event of a switch-over Enable inter-database IORM for all databases on the Standby s storage cells Create the allocation using standby s DB_UNIQUE_NAME See MOS note 1930540.1

References / Additional Resources MAA Best Practices for Database Consolidation and Oracle Multitenant with Oracle 12c Best Practices For Database Consolidation On Oracle Exadata Database Machine Oracle MAA Reference Architectures - The Foundation for Database as a Service Oracle Exadata Database Machine Consolidation: Segregating Databases and Roles Application Continuity with Oracle Database 12c Oracle.com/maa http://tinyurl.com/kruf7pe http://tinyurl.com/meh3wku http://tinyurl.com/lwm9994 http://tinyurl.com/oxlonht http://tinyurl.com/o46hvuq Use Oplan and out-of-place patching Utilize standby-first patching Best Practices for Corr. Detection, Prevention and Autom. Repair in a Data Guard Config Exadata Database Machine Software and Hardware Maintenance Planning Guide Master Note for Oracle Database Resource Manager MOS 1306814.1 MOS 1265700.1 MOS 1302539.1 MOS 1461240.1 MOS 1339769.1 Support.oracle.com 64