What is and What is not Exadata Engineered System

Transcription

1 What is and What is not Exadata Engineered System Radosław Cisz

2 Introduction to Database Machine Fully integrated platform for Oracle Databases High performance and High availability solution Consolidation platform Answers all workload types Fast implementation Balanced configuration Based on Oracle Storage Servers technology

3 Introduction to Database Machine 14 Storage Servers (High Capacity / High Performance) 8 Database Nodes 3 Infiniband Switches (40Gb/s) Cisco Ethernet Switch 2 PDUs

4 Storage Servers technology Architecture depend on Exadata version, X bit Intel Sun Fire servers preinstalled with Oracle Linux/Solaris for x86 Drivers and Utilities Exadata Storage Server Software Part of Infiniband Network Smart Storage Operations Capable HP, HC configurations Equipped with Smart Flash Cache

5 Storage Server Hardware Specification Processors 12 Intel CPU Cores System memory 64 GB DDR4 Flash Drives 1.6 TB Disk Controller Host Bus Adapter, 512 MB Battery Backed Write Cache Infiband Network Dual-Port QDR 40 Gb/s HBA Remote management ILOM Ethernet Port Power Supplies 2 x Hot-Swappable

6 Storage Server Parameters Specification Parameter HP (High Performance) HC (High Capacity) Raw Capacity 7.2 TB 36 TB Data Capacity 3.25 TB 16 TB Raw Disk Throughput 1.8 GB/s 1.3 GB/s Flash Throughput 7.25 GB/s 6.75 GB/s Up to 14 Storage Servers in Full Rack configuration All High Capacity or all High Performance servers

7 Database Nodes X3-2 Hardware Specification Processors 16 Intel CPU Cores, System memory 256 GB DDR4 Disk Controller Host Bus Adapter, 512 MB Battery Backup Write Cache Network Interfaces Dual-Port QDR 40 Gb/s HBA Infiniband 4 x 1/10Gb Ethernet (copper) 2 x 10Gb Ethernet (optical) Remote management ILOM Ethernet Port Power Supplies 2 x Hot-Swappable

8 Infiniband Network Overview 40 Gb/s speed in each direction Zero-loss Zero-copy Datagram Protocol (ZDP) Zero copy implementation of RDSv3 There is no difference for tools like ping Used for RAC interconnect, storage networking, external connectivity RDS is used for RAC interconnect RDS is used for DbNodes Storage Servers communication With Infiniband you can connect up to 18 Exadata Racks

9 Infiniband Network Overview

10 Infiniband Network Overview

11 Exadata Network Overview

12 Smart Flash Cache Intelligently caches database objects (not data blocks) Replaces slow mechanical I/O operations Smart Flash Logging Supports "log file sync/log file parallel write" Parallel write to Disk Controller Cache and Flash memory For X3-2 Full Rack Maximum Flash bandwith 100 GB/s Maximum Flash read IOPS 1,500,000 Maximu Flash write IOPS 1,000,000

13 Smart Flash Cache Exadata uses Flash PCI cards, not flash disks It is not disk replacement Exadata Storage Software decides how and when to use flash storage ESS Can simultaneously scan from flash and disks Traditional Storage Arrays Controllers are bottleneck for storage subsystem Adding Flash Layer makes bottleneck more visible Does not have knowledge on type of workload cached

14 Smart Flash Cache Each PCI Flash Card provides 1,33 GB/s throughput Exadata scales linearly across 4 cards DBAs and Developers can influence how DB objects are keeped on Flash Cache (CACHE,NOCACHE,EVICT, DEFAULT,KEEP,NONE) In practice it is common to see > 90% 'flash cache hit' Can be combined with HCC Exadata Software automatically detects and bypass poorly performing or crippled flash IO Resource Manager can manage I/O on flash and disks between consolidated databases

15 Smart Flash Cache Can work in two configurations Write Through Write Back (write intensive applications "free buffer wait") Cached type of workload Frequently accesed data and index blocks Controlfile reads and writes File header reads and writes KEEP objects Not - Cached type of workload Backup related I/O Data Pump I/O Datafile formatting Table scans, which are suppose to monopolize Flash I/Os mirrors copies are managed intelligently

16 Smart Flash Cache Write Back vs Write Through Generally Session in not waiting for dirty buffers writes (DBWR), the only wait is log file sync on commit There are cases when session must wait because of too slow DBWR When there are too many dirty blocks and no free space in buffer cache Session wants to read another lock, and has to first place it in cache Session waits till DBWR will flush dirty blocks to make room for new blocks When we can see inefficient DBWR? When there is workload with huge amount of updates When writes goes to the same file/block ("free buffer wait" connected with "buffer busy wait") Imbalance in Read vs Write operations (write through flash cache, accelerates read factor by up to 10 times!)

17 Smart Flash Cache

18 Smart Flash Cache

19 Database Machine Software Architecture Diskmon checks cell liveness, DBRM plans propagation Diskmon.bin master process, dskm slave process for each d instance Provides data fencing to protect data integrity

20 Exadata Horizontal Scalablility

21

22 Exadata Storage Expansion Rack Full Rack 18 Exadata Storage Servers 3 Infiniband Switches Half Rack 9 Exadata Storage Servers 3 Infiniband Switches Quarter Rack 4 Exadata Storage Servers 2 Infiniband Switches

23 Database Machine Benefits for Warehousing Parallel storage grid providing huge throughput Very efficient Infiniband storage network Most of DW like query processing moved to storage servers Exadata Hybrid Columnar Compression less I/O In-memory parallel query

24 Exadata Hybrid Columnar Compression Saved Space (cost saving) Performance Improvement (reduced I/O operations) Average compression ration 10x 15x, depending on compression level Query Low / Query High Archive Low / Archive High Best compression ratios for Direct Load Operations Decompression offloaded to storage servers DBMS_COMPRESSION predicting compression ratio

25 Database Machine Benefits for OLTP Parallel storage grid providing large I/O throughput Low-latency Infiniband Network PING qrceladm01.oracle.com.pl ( ) 56(84) bytes of data. 64 bytes from qrceladm01.oracle.com.pl ( ): icmp_seq=1 ttl=64 time=0.101 ms 64 bytes from qrceladm01.oracle.com.pl ( ): icmp_seq=2 ttl=64 time=0.122 ms 64 bytes from qrceladm01.oracle.com.pl ( ): icmp_seq=3 ttl=64 time=0.125 ms 64 bytes from qrceladm01.oracle.com.pl ( ): icmp_seq=4 ttl=64 time=0.131 ms 64 bytes from qrceladm01.oracle.com.pl ( ): icmp_seq=5 ttl=64 time=0.121 ms High-End database servers, with a lot of computing memory Exadata Smart Flash Cache

26 Database Machine Software Architecture For Different Database versions

27 Classic Database I/O vs Exadata SQL Processing Model

28 Classic Database I/O vs Exadata SQL Processing Model

29 Infiniband Storage Network and Smart Scan Scale-Out Each cell can deliver 1.8 GB/s 14 cells have potential scanning power 25.3 GB/s Select /* full(t1) */ count(*) from huge_4800gb where id < 10; Without smart-scan - Extents are identified - Table is evenly distributed across all servers - All blocks of table huge_4800gb are send to Database - Each cell can scan with speed GB/s - In full rack there are 14 cells - Infiniband connection to DB nodes is 40Gb/s / 14 what gives us 2.85Gb/s = 0.357GB/s - In this case each cell scan speed is 0.357Gb/s - With table size 4800 GB it gives ~16 minutes! With smart-scan - Database (shadow process) asks to send only matching rows - Returning rows are only 2MB in size - In this case Interconnect speed is not an issue - Each cell can scan with a speed 1.8 G/s - Operation will last 3 minutes and 10 seconds!!!!

30 Exadata Smart Scan Predicate Filtering Only filtered rows are returned to database nodes Column Projection Storage Server Software, applies 'where' predicates on storage layer. Only columns referenced in SQL query, and necessary to fulfill join processing and sorting columns are returned Join Processing Star joins are executed on storage layer

31 Exadata Smart Scan Scans on encrypted data Scans on compressed data Data Mining Scoring Model Subset of functions is offloaded (V$SQLFN_METADATA) Formatting datafile blocks RMAN datafile restoration One block granurality for incremental backups using Block Change Tracking File

32 Exadata Smart Scan Virtual Columns computation Balancing CPU usage

33 Smart Scan Prerequisites There must be full scan of object FTS Full Table Scan FFIS Fast Full Index Scan Materialized View Full Scan Scan must use Direct-Path Read mechanism Higher 'direct path read' Waits in 11g when Compared to 10g (Doc ID ) _serial_direct_read, _small_table_threshold - default 2% of buffer cache blocks This decision is based on the size of the table, buffer cache size, and various other statistics Any table having more than 5 times the number of blocks in "_small_table_threshold" will automatically use direct path reads for serial full table scans (FTS) Object must be stored on Exadata Storage

34 Exadata Storage Indexes SQL> Select * from TABLE where B < 2

35 Multiplied Benefits

36 IORM I/O Resource Manager

37 IORM I/O Resource Manager I/O redo writes and controlfile write always prioritized before user writes Database writer writes are scheduled at the same priority level as user I/O Background queues correspond to low,medium,high priority

38 IORM I/O Resource Manager

39 IORM I/O Resource Manager CellCLI> alter iormplan dbplan=(name=db1,level=1,allocation=70), (name=db2,level=1,allocation=30), (name=db3,level=2,allocation=50), (name=db4,level=2,allocation=50), (name=other,level=3,allocation=100), catplan=''; DbName Level1 db1 70 db2 30 Level2 db3 50 db4 50 other Level3 100

40 IORM I/O Resource Manager CellCLI> alter iormplan dbplan=(name=db1,level=1,allocation=70, role=primary,flashcache=on,flashlog=on), (name=db2,level=1,allocation=20, role=standby,flashcache=off,flashlog=off), (name=db3,level=2,allocation=20, role=standby,flashcache=off,flashlog=off), (name=db4,level=2,allocation=80, role=primary,flashcache=off,flashlog=on), (name=other,level=3,allocation=100), catplan=''; CellCLI> alter iormplan dbplan=(name=db1,share=7), (name=db2,share=18), (name=db3,share=15), (name=db4,share=5), (name=default,share=3), Catplan=''; Max Value for share=32 Max number of directives=1024

41 Interleaved Grid Disks and ASM Intelligent Data Placement CellCLI> CREATE CELLDISK ALL HARDDISK; CellCLI> CREATE GRIDDISK ALL PREFIX=DGDATA, SIZE=500G CellCLI> CREATE GRIDDISK ALL PREFIX=DGRECO, SIZE=300G CellCLI> CREATE CELLDISK ALL HARDDISK INTERLEAVING='normal_redundancy'; or CellCLI> CREATE CELLDISK ALL HARDDISK INTERLEAVING='high_redundancy'; CellCLI> CREATE GRIDDISK ALL PREFIX=DGDATA, SIZE=500G CellCLI> CREATE GRIDDISK ALL PREFIX=DGRECO, SIZE=300G

42 Interleaved Grid Disks and ASM Intelligent Data Placement Interleaved GridDisks works with ASM IDP (Intelligent Data Placement) Primary Extents are placed on Hot Portion of disk Mirror Extent are placed on Cold Portion of disk No special commands necessary

43 Exadata Specific System Statistics SQL> exec dbms_stats.gather_system_stats('exadata') New method for monitoring IOFTRSPEED Captures accurate MBRC for Exadata Asses correctly for DB machine CPUSPEED Requires at least BP18

44 Network Resource Management Prorititize critical database network traffic Ensures fast response time for latency-critical operations (QoS) Gives priority to redo writes and RAC Cache Fusion messages

45 Automatic Hard Disk Scrubbing and Repair Corruption on seldom accessed data can be unnoticed Reads are made from primary extent copy, so secondary extents could be long time corrupted Proactive SCRUB I/O is issued when disk is idle Scrubbing periodically scans each disk every two weeks by default In write-back, if data is dirty in FC, it will be overwirtten when cached copy will be flushed to disk If database block is clean in flash, it is repaired from flash, if not it is repaired from mirror copy CELLCLI> ALTER CELL harddiskscrubinterval = daily weekly biweekly none

46 Active-Active IB Connectivity Prior to Storage Sotfware only activepassive IB bonding was used In >= it is possible to configure Active-Active Do not configure Active-Active bonding on hardware before X4. Exadata X4 is equipped with IB cards that supports PCIe 3.0, allowing the combined bandwith of both ports can be exploited

47 Faster Disk Rebalance Operation (appliance mode) New ASM attribute Exadata specific 'appliance.mode ' improves disk rebalance times 'compatible.asm' => 'cell.smart_scan_capable' = 'true' Improves disk rebalance times, by taking advantage of the specific configuration settings applied to ASM groups on Exadata

48 I/O Latency Capping (Read/Write) Because of Internal maintenance operations, or precursor to failure, disk and flash can exhibit occasionally unusual high latencies Storage Software detects such situations Read operation ESS messages database that initiated the I/O Database redirects read to mirror copy Write operation Requires Flash in write-back mode Redirects slow write operation to another flash device on the same cell

49 Exadata X5-2 Extreme Flash Storage Server Processors 16 Intel CPU Cores, 2 x Eight-Core Intel Xeon E v3 2.4 GHz System memory 64 GB DDR4 Flash Drives 12.8 TB 8 x 1.6 TB Sun Flash Accelerator F160 PCI Infiband Network Dual-Port QDR 40 Gb/s HBA Remote management ILOM Ethernet Port Power Supplies 2 x Hot-Swappable

50 Exadata X5-2 Elastic Configuration Start with Quarter Rack 2 database servers 3 storage servers Add maximum of 22 servers Mixture of HC and HP (Extreme Flash) servers is supported

51 Oracle Capacity-On-Demand for Database Servers # resourcecontrol -capacity_on_demand display disable enable -cores numofcores Limits the number of active cores Available commencing with version Set during Initial Configuration with Deployment Assistant Number of active cores cannot be decreased Minimum number of active CPU cores 50% for X4 40% for X5 Number of active processor cores must be equal among db nodes sockets

52 Oracle Exadata Virtual Machines Available since Configured OEDA VM machines are considered as trusted partitions Multiple clusters on the same sets of physical servers All features of Exadata available Requires Oracle Linux

53 I/O Resource Management For Flash Flash Cache usage limits by database In multitenant configuration, PDBs limits can be set as percentage of the container database limit CellCLI> alter iormplan dbplan=(name=db1,share=7,flashcachemin=8192), (name=db2,share=18, flashcachemin=1g,flashcachelimit=10g), (name=db3,share=15,limit=40), catplan=''; Based on shares

54 Exadata Storage Server Performance Statistics in AWR reports Exadata specific configuration and performance statistics are collected in AWR reports Configuration Hardware model Software versions Storage Configuration Health reports Offline disks Open alerts Performance statistics OS Storage servers Smart scan Database statistics

55 Database Machine Automated Support Ecosystem Automatic Service Request Available as a feature of Oracle hardware Warranty and Oracle Premier Support for Systems Automatically opens SRs with Oracle Suppport Compromises hardware faults on CPUs, disk controllers,disks, flash cards,flash modules,infiniband cards,memory modules,system boards,power supplies and funs

56 Database Machine Automated Support Ecosystem

57 Metalink Note Exadata Database Machine and Exadata Storage Server Supported Versions Applies to Oracle Exadata Storage Server Software Oracle Exadata Hardware Oracle Database - Enterprise Edition - Version to [Release 11.2 to 12.1] Oracle Platinum Services - Version N/A to N/A Enterprise Manager for Oracle Database - Version to [Release 10.1] Oracle Solaris on x86-64 (64-bit) Linux x86-64 Oracle Exadata Database Machine

58 Follow-up Courses In order of importance: D92887GC10 - Exadata Database Machine: 12c Administration Workshop Ed 1 NEW 5 days D78850GC20 - Oracle Database 12c: Backup and Recovery Workshop Ed 2-5 days D81250GC10 - Oracle Database 12c: RAC Administration Ed days D81242GC10 - Oracle Database 12c: ASM Administration - 2 days D79232GC10 - Oracle Database 12c: Data Guard Administration - 4 days D79236GC10 - Oracle Database 12c: Performance Management and Tuning - 5 days D79128GC10 - Oracle Database 12c: Managing Multitenant Architecture - 2 days D85572GC10 - Oracle Database 12c: Implement Partitioning Ed 1-2 days D70064GC20 - Oracle Database 11g: Administer a Data Warehouse - 4 days D55650GC10 - Oracle Database 11g: Managing Oracle on Linux for DBAs - 2 days