<Insert Picture Here> Controlling resources in an Exadata environment

Transcription

1

2 <Insert Picture Here> Controlling resources in an Exadata environment

3 Agenda Exadata Security Flash Cache and Log Storage Indexes Parallel Execution

4 Agenda Exadata Security Flash Cache and Log Storage Indexes Parallel Execution

5 EXADATA SECURITY

6 Exadata Security Model Exadata Storage Interconnect uses IP and InfiniBand Limit the storage connection to specific adapters Defined when creating the cell This is physical security, robust and easy to maintain If systems (DB nodes) are connected to InfiniBand See Exadata cells See / use (empty) grid disks unless Exadata Security is setup Per-database access-control is possible Open Security, ASM-Scoped, Database-Scoped Security

7 Security setup on Exadata Database nodes are customer responsibility No firewall setup Password limitations in place 90 days, case restrictions, sizes, 5 attempts etc Exadata storage cells are secured Password restrictions 90 days, case restrictions, sizes, 5 attempts etc Firewall setup Only SSH access through Mgmt and IB network OS activity monitored Automatic log-off of SSH sessions

8 Open Security (default) No access restrictions Useful for environments without security requirements. Default security mode after creating new storage cell No security setup required Works out of the box. No ACLs or access-limitations Any database can access all grid-disks Normal ASM protection is in place through headers Mistakes (force addition) could happen

9 ASM-Scoped Security Security per ASM cluster ASM cluster identified by unique key in local file on DB node Alias is defined in Exadata cell for each unique key Griddisks are enabled for specific alias(es) Only one key needs to be set up per ASM cluster Same key on all nodes in of the same ASM cluster Every Griddisk needs to be enabled Aliases in properties of Griddisk (field AvailableTo)

10 Database-Scoped Security Each database needs different key Each database, not each instance! Make sure all DB in a cluster have same key setup If no key has been setup, fallback to ASM security If wrong key has been setup, access is denied Access per GridDisk (availableto) field Each cell gets alias for unique key Access by setting alias(es) in the availableto field Multiple entries are allowed in the availableto field ASM cluster used must have access too! Both Database and ASM must have access to GridDisk

11 Creating a Security Key A new key can be generated on any cell: CellCLI> CREATE KEY 66e12adb bf f5050 Use result on all cells en DB nodes Do not generate a key on each cell, one key is enough Do not generate a key for each instance, just each database User is responsible for copying the key to the right locations

12 Setting up ASM-Scoped Security 1. Start with Open Security, tested and working 2. Shutdown all databases and ASM 3. Create a new key on any cell 4. Copy key file cellkey.ora to all hosts in cluster 5. Copy key to all cells in the storage pool 6. Set up ACLs to allow ASM on all cells 7. Restart all databases and ASM All databases using this ASM can access Griddisks. No other user can access any of the Exadata storage.

13 ASM-Scoped Security How to setup? On any cell CellCLI> CREATE KEY 66e12adb bf f5050 Lookup the name of the ASM cluster SQL> SHOW PARAMETER db_unique_name Create cellkey.ora file on ASM node Located in /etc/oracle/cell/network-config/cellkey.ora Permissions 640 #cellkey.ora file for ASM key=<unique key> (66e12adb bf f5050) asm=<db_unique_name> (+asm) realm=<any name, optional> (my_realm)

14 ASM-Scoped Security How to setup? (continued..) On all of the the cells (not just one) ASSIGN KEY FOR +asm='66e12adb bf f5050' Create new Grid disks with ACL by default CREATE GRIDDISK ALL PREFIX=sales, availableto='+asm' Or change ACL for existing cells ALTER GRIDDISK ALL availableto='+asm' ALTER GRIDDISK sales_cd_1_cell01, sales_cd_2_cell01, sales_cd_3_cell01, sales_cd_4_cell01, sales_cd_5_cell01, sales_cd_6_cell01 availableto='+asm'

15 Setting Up Database-Scoped Security 1. Start with ASM-Scoped Security, tested and working. 2. Shutdown all databases and ASM 3. Create a key for each database (not each instance) 4. Copy key file cellkey.ora to each database home 5. Copy key to all cells in the storage pool 6. Set up ACLs to allow each database, as needed. 7. Restart all databases and ASM Each database can access its own grid-disks. If a key is defined, no other user can access any of the Exadata storage.

16 Database-Scoped Security How to setup? Create new key on any cell CellCLI> CREATE KEY 51a826646ebe1f29e33c6ed7c4965c9a Create cellkey.ora file on Database node Located in $ORACLE_HOME/admin/<db_name>/pfile//cellkey.ora Permissions 640 #cellkey.ora file for Database key=<unique key> (51a826646ebe1f29e33c6ed7c4965c9a) asm=<db_unique_name_for_the_asm> (+asm) realm=<any name, optional> (my_realm)

17 Database-Scoped Security How to setup? (continued..) On all of the cells (not just one!) ASSIGN KEY FOR PROD='51a826646ebe1f29e33c6ed7c4965c9a' Use DB_UNIQUE_KEY for key alias! Create new Grid disks with ACL by default CREATE GRIDDISK ALL PREFIX=sales, availableto='+asm,prod' Or change ACL for existing cells ALTER GRIDDISK ALL availableto='+asm,prod' ALTER GRIDDISK sales_cd_1_cell01, sales_cd_2_cell01, sales_cd_3_cell01, sales_cd_4_cell01, sales_cd_5_cell01, sales_cd_6_cell01 availableto='+asm,prod Make sure you repeat existing values in the availableto! Alter GRIDDISK will overwrite every value previous set

18 Adding/Removing Databases Keep cellkey.ora in each database directory It doesn t hurt, makes the environment more secure Retrieve list of defined keys from the cells CellCLI> LIST KEY +asm 66e12adb bf f5050 PROD b67d5587fe728118af47c57ab8da650a Issue ALTER GRIDDISK to change the availableto field Remember that ALTER GRIDDISK overwrites the contents of the availableto field!

19 Database-Scoped to ASM-Scoped 1. Shutdown all databases and ASM 2. Remove all database keys, from cells CellCLI> ASSIGN KEY FOR PROD='' 3. Delete cellkey.ora from database directory(s) 4. Remove all databases from Grid-Disk availableto CellCLI> ALTER GRIDDISK ALL availableto='+asm' 5. Restart ASM and databases.

20 ASM-Scoped to Open Security 1. Shutdown all databases and ASM 2. Remove ASM keys, from cells CellCLI> ASSIGN KEY FOR +asm='' 3. Delete cluster cellkey.ora from hosts rm /etc/oracle/cell/network-config/cellkey.ora 4. Remove ASM from Grid-Disk availableto CellCLI> ALTER GRIDDISK ALL availableto='' 5. Restart ASM and databases.

21 Exadata Security Best Practices When configuring security, the flow is always Open security to ASM-scoped security to Database-scoped security. Make sure disk security setup is consistent All disks in an ASM group and all disks for a database When using RAC, cellkey.ora and permissions are vital Make sure they are the same on any node in the cluster. Don t mix ASM and DB scoped security in a system Use DCLI to automate security setup

22 Exadata Security Summary Prevent access to Grid-Disks using keys Key on DB nodes level for ASM scoped Security Key on DB Home directory for DB scoped Security All nodes in ASM cluster or all DB Instances use the same key Create alias for keys on Storage cell level Setup is per Griddisk on Storage Cell level Availableto: field needs both ASM and DB level access Shutdown all instances before making changes One wrong command can cause issues in the database

23 Q U E S T I O N S A N S W E R S On Exadata Security

24 Agenda Exadata Security Flash Cache and Log Storage Indexes Parallel Execution

25 FLASH CACHE

26 World s First Flash Database Machine 5TB of Flash Memory Smart Flash Cache Million I/Os per Second Sub-Millisecond I/O time 50GB/sec Flash Scan 3x to 50x Compression on Flash

27 Semiconductor Cache Hierarchy Massive throughput and IOs through innovative Cache Hierarchy Database DRAM Cache 400GB raw capacity Up to 4TB compressed user data 100 GB/sec Exadata Smart Flash Cache 5TB raw capacity Up to 50TB compressed user data 50 GB/sec raw scan 1 million IO/sec Exadata disks 100TB or 336TB raw Up to 500TB compressed user data 21 GB/sec scan 50K IO/sec

28 Sun Flash Accelerator F20 PCIe Card 96GB Storage Capacity 4 x 24GB Flash modules/doms 6GB reserved for failures x8 PCIe card Super Capacitor backup Optimized for Database caching Fully integrated into the Exadata V2 database machine. Measured end-to-end performance 3.6GB/sec/cell or 50GB/sec per DBM 75,000 IOPs/cell or 1 million IOPS per DBM

29 How can we use Flash Storage Exadata Smart Flash Cache Store frequently used blocks (data and index) on Flash storage Exadata Smart Flash Logging Use Flash as a buffer for your redolog writes Exadata Flash-Based Storage Uses flash as permanent high speed storage Or ALL of the above As we have 4 cards with 96GB of Flash memory We can split cards into separate sections with different functionality

30 Flash storage as Flash Cache What is Cached? Frequently accessed data and index blocks Control file reads and writes File header reads and writes DBA can also influence caching priorities What is Not Cached? I/O to mirror copies Backup related I/O Data Pump I/O Data file formatting Table scans don t monopolize the cache

31 Flash Storage as Flash Logging Redologs are sequential writes Disk DRAM cache usually as fast (or faster) than Flash writes Provided Disk cache gets flushed fast enough Can be bottleneck on high write workloads Redolog will write to Flash Cache and DRAM cache At the same time First one to finish will signal write complete to the DB Starting Cell Patchset / BP11 DB version Can be turned on or off via the IO Resource Manager

32 Flash logging example

33 Flash Storage as Datafile storage Using Flash as DB storage reduces Flash space 2x when using Normal Redundancy 3x when using High Redundancy Only use for heave scattered reads and writes Sequential writes are usually faster using Lots of disk and disk cache Dedicated disks so disk-arm does not have to move Can be used in a partitioning or ILM scheme Most (almost all) customers do not use this feature

34 Flash Best Practices Use Flash only for production Unless you have enough (speed and space) for everything Use Flash logging for redo-log bottlenecks Use Flash Cache and Logging instead of File storage Use all available Flash space Most performance benefit will be from reading and redologs Only change after AWR reports issues with small writes Use as much for Cache and Logging as possible If you need file storage, only use the part of Flash you need Keep the other parts for Cache and Logging

35 Exadata Smart Flash Cache Commands Creating Flash Cache by default CREATE CELL flashcache Creating Flash Cache explicitly CREATE FLASHCACHE ALL size=100g Users can prioritize caching of tables ALTER TABLE table_name STORAGE (CELL_FLASH_CACHE KEEP) Without the setting, individual tables and indexes are cached Display Flash Cache Configuration LIST FLASHCACHE DETAIL

36 Control access to flash cache Prevent databases to access flash cache Low priority databases Test databases Etc New attribute introduced for IORM: FlashCache Cellcli> ALTER IORMPLAN dbplan=((name=prod, flashcache=on), (name=dev, flashcache=on), (name=test, flashcache=off), (name=other, flashcache=off))

37 Exadata Flash Summary 5+ Terabytes of Flash available in Exadata Machine About 400Mb per Exadata Storage Cell No controller bottlenecks; cards instead of disks 3.6GB/sec/cell or 50GB/sec per DBM 75,000 IOPs/cell or 1 million IOPS per DBM Use it as.. Exadata Flash Cache Exadata Flash Logging Exadata Flash file storage Needs Normal of High Redundancy Only use it if AWR report shows issues with scattered writes

38 Q U E S T I O N S A N S W E R S On Exadata Flash storage

39 Agenda Exadata Security Flash Cache and Log Storage Indexes Parallel Execution

40 STORAGE INDEXES

41 Biggest performance Bottleneck Disks are a big bottleneck for a database Its all about getting to data as fast as possible Flash disks are much faster than normal disks Normal databases need to read data To determine if the where clause applies to the data More data is read than needed Indexes help a lot But not-reading unneeded data would be faster!

42 Introducing Exadata Storage Indexes Memory based index of contents of blocks Index automatically build using memory of the Exadata cells Information about a block only stored in one place For each column in frequently accessed blocks Minimum and Maximum value is stored Occurrence of NULL values is stored Values automatically stored after first read of the block Values in RAM Take only small amount of memory Destroyed when power is removed (reboot, crash etc)

43 How do Storage Indexes work An example.. Block 1 Block 2 Block 3 A B C D A B C D A B C D Min B = 1 Max B = Min B = 3 Max B = Min B = 6 Max B = 9 select * from table where B < 2 Only block 1 matches Other blocks do not need to be retrieved Saves huge amounts of IO per second Only works with Smart Scans / Direct Reads

44 Storage Index with Partitions Example Orders Table Order# Order_Date Ship_Date Item Partitioning Column Queries on Ship_Date do not benefit from Order_Date partitioning However Ship_date and Order# are highly correlated with Order_Date e.g. Ship dates are usually near Order_Dates and are never less Storage index provides partition pruning like performance for queries on Ship_Date and Order# Takes advantage of ordering created by partitioning or sorted loading

45 Measuring performance Use V$SYSSTAT for information cell physical IO bytes saved by storage index

46 Storage Indexes - Summary In memory storage of block characteristics Highest and Lowest value of each column in a block Build when the block is accessed the first time Stays in memory Destroyed when cellsrv is down (power down, reboot etc) No communication between servers Low memory footprint Cannot turn it on or off No extra license fee

47 Q U E S T I O N S A N S W E R S On Storage Indexes

48 Agenda Exadata Security Flash Cache and Log Storage Indexes Parallel Execution

49 PARALLEL EXECUTION

50 Automated Degree of Parallelism An 11gR2 generic feature In the past, tuning parallelism was a manual process one degree of parallelism does not fit all queries too much parallelism could flood the entire system Automated Degree of Parallelism automatically decides If a statement will execute in parallel or not (serial execution would take longer than a set threshold 30 secs) What degree of parallelism the statement will use Optimizer derives the DoP from the statement based on resource requirements Uses the cost of all scan operations Balanced against a max limit of parallelism

51 Automated Degree of Parallelism How it works SQL statement Statement is hard parsed And optimizer determines the execution plan If estimated time greater than threshold Optimizer determines ideal DOP If estimated time less than threshold PARALLEL_MIN_TIME_THRESHOLD Actual DOP = MIN(default DOP, ideal DOP) Statement executes serially Statement executes in parallel

52 Parallel Statement Queuing Automatic Degree of Parallelism means More statements will run in parallel Potential system thrashing due to too many processes Parallel Statement Queuing automatically decides if a statement can execute immediately or not When a parallel statement starts it checks if enough parallel servers (aka Parallel Query Slaves) are available If not, queue the statement Once enough parallel servers become available, statement is de-queued and executed

53 Parallel Statement Queuing How it works SQL statements Statement is parsed and Oracle automatically determines DOP If not enough parallel servers available queue FIFO Queue If enough parallel servers available execute immediately When the required number of parallel servers become available the first stmt on the queue is dequeued and executed 8 128

54 Parallel Statement Queuing Monitoring in Enterprise Manager Click on the SQL ID for more info Clock symbol indicated a queued statement Awaiting screen shot from EM

55 Parallel Statement Queuing Monitoring in Enterprise Manager Wait event indicates stmt is at the head of the queue

56 In-Memory Parallel Execution Data warehouse environments can have large amounts of memory that is not always used An algorithm places fragments of an object (partitions) in memory on different nodes Compression gets more data in memory Parallel servers (aka PQ Slaves) are then executed on the corresponding nodes Requires Automatic Degree of Parallelism Real Application Clusters

57 In-Memory Parallel Execution How it works SQL statement Determine the size of the table being looked at Table is a good candidate for In-Memory Parallel Execution Fragments of Table are read into each node s buffer cache Table is extremely small Table is extremely Large Read into the buffer cache on any node Always use direct read from disk Only parallel server on the same RAC node will access each fragment

58 Source: Transaction Processing Council, as of 9/14/2009: Oracle on HP Bladesystem c-class 128P RAC, 1,166,976 available 12/1/09. Exasol on PRIMERGY RX300 S4, 1,018,321 available 08/01/08. ParAccel on SunFire X ,842 $4.57 available 10/29/07. In-Memory Parallel Execution QphH: 1 TB TPC-H 1,166,976 A single database machine has over 400GB of memory usable for caching 1,018, ,842 ParAccel Exasol Oracle & HP Exadata Faster than specialized in-memory warehouse databases Database release 11.2 introduces parallel query processing on memory cached data Harnesses memory capacity of entire database cluster for queries Foundation for world record 1TB TPC-H Exadata Hybrid Columnar Compression enables multi-terabyte tables or partitions to be cached in memory Memory has 100x more bandwidth than Disk

59 Paralellism - Summary Parallelism is important Speed up queries, uses all resources available No need for Parallelism tuning Needs init.ora parameter Optimizer decides on parallelism based on statistics No need for Parallelism limitation Statements will wait for free slaves Prevents overload on the system Queries are still faster than single threaded execution Parallelism can cause in-memory behaviour Qualified tables can be stored in memory automatically

60 Q U E S T I O N S A N S W E R S On Storage Indexes

61 Q U E S T I O N S A N S W E R S

62 Now you do it!

63