12.1 Multitenancy in real life

Transcription

1 12.1 Multitenancy in real life 2017 HOUG szakmai nap Jozsef Horvath Budapest,

2 Disclaimer This presentation: Does not intend to judge Oracle Multitenancy Does not intent to judge Oracle Corporation or Oracle products Is strictly based on documented features and on-site experiences Is not about showing Multitenancy basics and command syntaxes

3 Agenda Multitenancy overview Current environment Cloning Backup-Recovery Transparent Data Encryption Miscellaneous Future plans

4 Multitenancy overview Consolidation pre-12c Too much overhead per instances 1 SGA per 1 database instance Multiple background processes per database instance Operational environment multiplied by database instance Complicated patching, upgrade process

5 Multitenancy overview 12c (12.1) Multitenant Architecture Multitenant licence - Not required with one PDB Container database (CDB) - CDB$ROOT Pluggable Databases (PDB) - PDB$SEED (created internally) - Additional user-created PDB s Maximum number of PDB s: 252 Transparent for end-users PDB s are transferable among CDB s Create (pluggable) database by cloning

6 Current Environment 2 x Exadata X5-2 1/4 Rack - 36CPU core, 256GB RAM per compute node - 36x8TB 7200RPM disks TB (raw) Flash Upgraded from 1/8 to 1/4 in Q12017 Virtualized configuration Oracle Linux 6.8 on compute nodes, VM s Oracle RDBMS, GI: (with latest Bundle Patch) Rack1: - 2 VM s: UAT/Test/Pre-Production RAC database - 2 VM s: DR RAC Database for Production environment Rack2: 2 VM s: Production RAC database

7 Current Environment Contd. UAT, Test, Pre-Production - 1 container DB, 158 client PDB + 3 release seed PDB - 1 PDB per client - Multiple customer with more than 1 PDB for different testing purposes - Number of PDB s constantly changing Production - 1 container DB, 109 live PDB + 3 release seed PDB - PDB s for 109 different customer - New PDB only for new customer DR (disaster recovery) site - Physical standby database for production - DR Invocation tests: Every six months

8 Current Environment Contd. SaaS vs On-Premise - Software-as-a-Service (SaaS - Cloud) solution for the customers - SaaS started in Q22015: 30 Customer customer on SaaS by the end of 2017, - About 150+ customer are on-premise (They host the infrastructure) - By the end of 2019: No on-premise customer, SaaS: 300+ SaaS goals - Optimal Total Cost of Ownership (TCO) - Scaling environment - Cloud-style services for the customers 1 Pluggable Database - 1 version of in-house developed application - Created from release seed PDB - 4 schema, 5 tablespaces (besides Oracle core) Client Application - Oracle Fusion Middleware 11g, 12c - Oracle Forms, Reports - Oracle BI Discoverer 11g - HTML, JavaScript, PL/SQL (mod_plsql)-based Portal lines of PL/SQL code

9 Cloning Goals, requirements, considerations Why do we clone? - Create new customer environment (PDB) Not really a challenge - Refresh PDB s from other PDB s Which environment to clone? - Usually Production to UAT/Test - Go-live from Pre-production to Production Downtime allowed? Out of vs during business hours? Current state or PITR required? How do we do it?

10 Cloning Contd. No downtime allowed How do we do it? - DataPump Export/Import? - RMAN (active) duplicate - Other? In a non-multitenant environment: - Database clone with RMAN (active) duplicate does the job (most of the times) In a Multitenant environment (12R1): - Cloning one or more PDB s requires an auxiliary container database instance DataPump is out of the question - Too slow, too complex, too fragile

11 Cloning Contd. Cloning PDB with no downtime using RMAN duplicate 1. Prepare and start auxiliary database instance Parameter file(init.ora/spfile), TNS Listener, password file, ASM diskgroup/filesystem, etc 2. Do the duplication with RMAN rman target auxiliary RMAN> duplicate target database to <auxcdb> 2> pluggable database <sourcepdb> from active database; 3. Unplug the PDB from auxiliary CDB sqlplus as sysdba SQL> alter pluggable database <sourcepdb> close immediate; SQL> alter pluggable database <sourcepdb> unplug into <directory>/sourcepdb.xml'; 4. Plug the clone PDB in the target CDB sqlplus as sysdba SQL> alter pluggable database <sourcepdb> close immediate instances=all; SQL> drop pluggable database <sourcepdb> including datafiles; SQL> alter pluggable database <sourcepdb> unplug into <directory>/sourcepdb.xml ; SQL> alter pluggable database <sourcepdb> open instances=all;

12 Cloning Contd. Cloning PDB with minimal downtime 1. Restart source PDB in read only mode sqlplus as sysdba SQL> alter pluggable database <sourcepdb> close immediate instances=all; SQL> alter pluggable database <sourcepdb> open read only; 2. Create local clone PDB SQL> create pluggable database <source_clone> from <sourcepdb>; 3. Restart source PDB in read write mode SQL> alter pluggable database <sourcepdb> close immediate; SQL> alter pluggable database <sourcepdb> open instances=all; 4. Do the clone to the target container sqlplus as sysdba SQL> alter pluggable database pdb1 close immediate instances=all; SQL> drop pluggable database pdb1 including datafiles; SQL> CREATE DATABASE LINK LINK_PDB CONNECT TO <DBA_USER> IDENTIFIED BY <PASSWORD> 2> USING 'SOURCE_CLONE_TNS ; SQL> create pluggable database pdb1 from pdb1@link_pdb1; SQL> alter pluggable database pdb1 open instances=all;

13 Cloning Contd. Analysis No Downtime Minimal Downtime Scriptable? Easy to automate? Yes, fairly easy to script and automate Yes, fairly easy to script and automate Reasonable resource requirements? Is it scaling? Is it fast enough Every cloning process requires an auxiliary CDB to be created as well as the PDB to be cloned doubles the space usage temporarily. Auxiliary CDB causes significant memory and CPU overhead (SGA and background) Multiple cloning at the same time is problematic due to large footprint of one or more auxiliary CDB Experience in our environment: One cloning session for one PDB takes minutes (overall) due to large number of PDB s in the source container Provisionally doubling the space used by local clone for each cloning session (negligible). Other than that resource requirements are acceptable Minimal footprint, scaling fairly well without significant contention Experience in our environment: One cloning session for one PDB takes minutes (overall) Acceptable by customers? No Yes (with restrictions) Conclusion: Minimal downtime cloning procedure is implemented, automated, mostly during out of business hours or in accordance with customer schedule

14 Backup/Recovery Backup/Recovery strategy - CDB+PDB full backup: Once a week - Incremental cumulative CDB+PDB backup: 6 times a week - Archivelog backup is part of each backup days retention: 30 days on disk, 70 days on tape Our cloning procedure impact the Backup/Recovery strategy - Dropping, re-creating PDB s may invalidate previous backups - Clone PDB is not the same as the pre-clone PDB Point-in-time Recovery (PITR) of PDB s frequently invoked - After failed application patch installation - User errors on large volume of data - After clone, to restore pre-clone state - Backup/Recovery testing - RMAN duplicate with auxiliary instance is used (backup based cloning) - User errors on small volume: Flashback Query, Transaction, - FLASHBACK DATABASE is not used PDB level operation is not possible

15 Backup/Recovery Contd. PDB Point-in-time Recovery For PDB that was not dropped earlier - PDB PITR with in-place restore, recovery - Only CDB$ROOT files restored to auxiliary place For PDB that was dropped earlier - Example: Cloning - PDB PITR using auxiliary instance PDB PITR with auxiliary instance issues - How to Restore - Dropped Pluggable database (PDB) in Multitenant (Doc ID ) - Bug RMAN Recover of Dropped PDB in Multitenant may fail with errors like RMAN-6813 (Doc ID ) - Bug : BACKUP BASED DUPLICATE FAILS(WITH CATALOG) - RMAN (WITH DROPPED PDB NAME) - RMAN During Cloning Pluggable Database in 12c CDB (Doc ID )

16 Disaster Recovery Oracle Data Guard, Physical Standby Database Maximum Availability protection mode No licence for Active Data Guard Snapshot Standby conversion for DR Invocation Tests No Fast-start Failover configured - DR Invocation is always manually triggered Data Guard in Multitenant environment (12R1): - Role transition at the CDB level only - Creating PDB s in primary could cause problems at standby side - standbys=none clause is important in most scenarios

17 Disaster Recovery Contd. Data Guard in Multitenant environment (12R1) Creating PDB from PDB$SEED - Datafiles get transferred over to standby database - Redo Apply process can copy the files to the standby site - Managed Recovery Process (MRP) continues to apply the redo information Creating PDB as a clone of non-pdb$seed may break MRP: - If Active Data Guard is not used and standbys=none clause omitted from create pluggable database statement - standbys=none will disable the managed recovery for the cloned PDB - Managed recovery must be enabled manually at standby side Creating PDB as a clone of non-pdb$seed works properly when Active Data Guard is used - Local clones: no manual intervention is needed - Remote clones via db link: Parameter standby_pdb_source_file_dblink is recommended to set Plugin operations - Parameter standby_pdb_source_file_directory is recommended to set

18 Disaster Recovery Contd. Disaster Recovery Invocation Tests Invoked every six month Snapshot Standby database is used - Fully updatable standby database - Flashback Database/Guaranteed restore point technology is used here - MRP is stopped - Physical Standby Database is converted to Snapshot Standby - Only a few PDB is used for DR testing by a few customers Middleware servers have their own DR pair servers Application production FORMS servers - Tnsnames.ora contains DR host references as well - They are not used for DR invocation Tests Application DR URL s are separated from Production for the test - They re removed from Load Balancer for the DR Tests - Full featured replica of production servers

19 Disaster Recovery Contd. Disaster Recovery Invocation Tests - Steps Converting Physical Standby Database to Snapshot Standby sqlplus sys/login@<drcdb> as sysdba SQL> alter database recover managed standby database cancel; SQL> alter database convert to snapshot standby; SQL> alter database open; *: Followed by opening remaining RAC instances Start required PBD s only SQL> alter pluggable database ALL close immediate; *: due to save state SQL> alter pluggable database PDB1 open; SQL> alter pluggable database PDB2 open; SQL> alter pluggable. Perform application tests Convert Snapshot Standby to Physical Standby sqlplus sys/login@<drcdb> as sysdba SQL> shutdown immediate; *: ALL RAC instance must be shut down SQL> startup mount; *: only one RAC instance should be started SQL> alter database convert to physical standby; SQL> alter database recover managed standby database disconnect;

20 Disaster Recovery Contd. Converting back to Physical Standby Database may fail SQL> alter database convert to physical standby; alter database convert to physical standby * ERROR at line 1: ORA-38769: FLASHBACK DATABASE failed after modifying data. ORA-00600: internal error code, arguments: [kcvvra_pdbinc], [], [], [], [], [],[], [], [], [], [], [] General: Bugs, useful Notes - ORA [KCVVRA_PDBINC] WHILE CONVERTING SNAPSHOT STANDBY TO PHYSICAL STANDBY (Doc ID ) - Data Guard Impact on Oracle Multitenant Environments (Doc ID ) - standby_pdb_source_file_dblink, standby_pdb_source_file_directory (Doc ID ) - Making Use Deferred PDB Recovery and the STANDBYS=NONE Feature with Oracle Multitenant (Doc ID )

21 Transparent Data Encryption (TDE) Used to protect data files (Where data is stored) TDE Tablespace Encryption used Two-tiered, key-based architecture Password-based vs auto-login software keystore consideration Hardware keystore is not considered

22 Transparent Data Encryption (TDE) Contd. Password-based vs Auto-login keystore Software keystore (wallet) - Stores TDE master encryption key - Location is for CDB level, not PDB - sqlnet.ora configuration - ENCRYPTION_WALLET_LOCATION - RAC: shared location ASM or other cluster filesystem Auto-login keystore - Convenient when CDB and/or PDB s are restarted - No manual intervention needed from operations team - Works only with static set of keys - Does not support new key imports (i.e.: During PDB cloning) Password-based keystore - Requires intervention from operations team during CDB/PDB restart - Possible ORA-28365: wallet is not open errors - CDB/PDB level startup trigger used to open keystore wallet - Master key export/import required when PDB s being cloned

23 Transparent Data Encryption (TDE) Contd. Restarting CDB/PDB s with password-based keystore/wallet Create keystore/wallet CDB level only sqlplus sys/login@<cdb> as sysdba SQL> administer key management CREATE KEYSTORE <DIR> * identified by <Wallet_Password>"; *: DIR should be the same as ENCRYPTION_WALLET_LOCATION in sqlnet.ora Open keystore/wallet CDB level sqlplus sys/login@<cdb> as sysdba SQL> administer key management SET KEYSTORE OPEN identified by <Wallet_Password> CONTAINER=ALL; Open keystore/wallet PDB level sqlplus sys/login@<pdb> as sysdba SQL> administer key management SET KEYSTORE OPEN identified by <Wallet_Password> ; AFTER STARTUP trigger at CDB/PDB level - To avoid manual operations during CDB/PDB startup - ADMINISTER KEY command used in WRAP d procedure

24 Transparent Data Encryption (TDE) Contd. Cloning PDB with TDE password-based keystore/wallet Perform PDB clone (as per discussed earlier) - PDB_PLUG_IN_VIOLATIONS view will show PDB needs to import keys from source - PDB is only open in restricted mode until TDE issue is not solved Export master key in source PDB sqlplus sys/login@<sourcepdb> as sysdba SQL> administer key management set keystore open identified by <Wallet_Password>"; --open the wallet in the PDB if not open SQL> administer key management EXPORT encryption keys with secret <encryption_key_pwd>" TO <DIR>/key.exp' identified by <export_password>"; --export master key for source pdb from TDE wallet Transfer export file (when remote clone is performed) Import master key in target PDB sqlplus sys/login@<targetpdb> as sysdba SQL> administer key management set keystore open identified by <Wallet_Password>"; --open the wallet in the PDB if not open SQL> administer key management IMPORT encryption keys with secret <encryption_key_pwd>" FROM <DIR>/key.exp' identified by <export_password> WITH BACKUP; --import master key into TDE wallet Restart clone PDB (targetpdb) in target CDB - PDB_PLUG_IN_VIOLATIONS view should be cleared for TDE errors - PDB should be open in read-write mode

25 Miscellaneous Init parameters - init.ora/spfile is common for all PDB s, CDB level only - PDB: ispdb_modifiable in v$system_parameter - Experience: Set job_queue_processes parameter at CDB level only - Experience: Set nls_* parameters at CDB level only - db_files parameter CDB level Manage pluggable databases (PDB) - Create/Drop only at CDB level - PDB level: startup/shutdown as usual - CDB level: alter pluggable database close/open - alter pluggable database save state Multitenancy in RAC environment - instances=all clause cluster level - (g)v$pdbs view

26 Miscellaneous Contd. Scheduler jobs and job_queue_processes init parameter - Default value: Problematic with scheduler jobs per PDB (i.e.: 60x160=9600) - Setting at PDB level impacts CDB and other PDB s as well - job_queue_processes in multitenant database (Doc ID ) - Experience: job_queue_processes=36 (Number of CPU cores) Patching CDB and PDB - Usually in rolling fashion in RAC environment - Datapatch utility (as of 12.1) is used for dictionary patching - Datapatch could experience problems with high number of PDB s - Bug datapatch takes significant time with many pdb s - Bug datapatch gets deadlock messages on 'insert into metaview$ Patching Production and Test - With time difference - Could impact cloning due to different patch level - Datapatch must be run explicitly during each cloning session

27 Future plans 12c Release 2 (12.2) upgrade - PDB cloning: Online cloning, PDB Refresh - Disaster Recovery: Subset Standby (Active Data Guard needed) - Transparent Data Encryption: TDE Tablespace Live Conversion - Resource Manager: Memory Resource Management, PDB Performance Profiles - Backup/Recovery: Flashback PDB, - Plenty of known bugs are fixed - Maximum number of PDB s: 4096 Split PDB s into multiple CDB s - Separate software home for each - Reduce single point of failure on 1 CDB configuration - Allows to provide different service levels for customers

28 Thank you for your attention! Author: Jozsef Horvath