Data Replication With Oracle GoldenGate Looking Behind The Scenes Robert Bialek Principal Consultant Partner

Transcription

1 Data Replication With Oracle GoldenGate Looking Behind The Scenes Robert Bialek Principal Consultant Partner BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

2 Who Am I Principal Consultant, Partner and Trainer at Trivadis GmbH in Munich robert.bialek@trivadis.com Focus: Oracle Database High Availability Database Architecture/Internals Backup/Recovery Troubleshooting/Performance Tuning Linux Administration Trainer for the following Trivadis courses Oracle Grid Infrastructure (O-GRINF) Oracle Real Application Cluster (O-RAC) Oracle Data Guard (O-DG) Data Replication With Oracle GoldenGate - Looking Behind The Scenes

3 Our company. Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: O P E R A T I O N Trivadis Services takes over the interactive operation of your IT systems Data Replication With Oracle GoldenGate - Looking Behind The Scenes

4 With over 600 specialists and IT experts in your region. COPENHAGEN 14 Trivadis branches and more than 600 employees HAMBURG 200 Service Level Agreements Over 4,000 training participants Research and development budget: CHF 5.0 / EUR 4 million DÜSSELDORF FRANKFURT Financially self-supporting and sustainably profitable Experience from more than 1,900 projects per year at over 800 customers BASEL STUTTGART FREIBURG BRUGG ZURICH MUNICH VIENNA BERN LAUSANNE GENEVA Data Replication With Oracle GoldenGate - Looking Behind The Scenes

5 Technology on its own won't help you. You need to know how to use it properly Data Replication With Oracle GoldenGate - Looking Behind The Scenes

6 Agenda 1. Introduction 2. Oracle GoldenGate Architecture 3. Oracle GoldenGate Setup 4. Core Messages Data Replication With Oracle GoldenGate - Looking Behind The Scenes

7 Introduction Data Replication With Oracle GoldenGate - Looking Behind The Scenes

8 What is Oracle GoldenGate? Oracle strategic Middleware software product used for data extraction and replication Use-cases: data/service high-availabillity, zero-downtime upgrades/migrations, easy fallback (minimize risk) Supports asynchronous data transfer (DML, DDL) heterogeneous sources and targets (databases, operating systems) different extraction, propagation and apply methods mappings, transformations, filtering, error handling, Topologies Active/passive, active/standby, active/active Broadcasting, cascading, consolidation Source as well as the target environment requires a license Includes (EE): Oracle Active Data Guard and Oracle Global Data Services ( ) Data Replication With Oracle GoldenGate - Looking Behind The Scenes

9 Why Oracle GoldenGate and not Streams? Oracle Streams is deprecated in the Oracle 12c version (but still supported!) According to the Oracle GoldenGate Statement of Direction Oracle Streams continues to be supported but will not be enhanced to support any new database features in future releases. This means that Streams will continue to support exactly what it supports today (11.2) in future releases. It also means that any new database features, data types, or capabilities will not be supported by Streams. Beginning with the version , the non-cdb architecture is also deprecated As of now, Streams is not supported with the new CDB architecture Data Replication With Oracle GoldenGate - Looking Behind The Scenes

10 Oracle GoldenGate Architecture Data Replication With Oracle GoldenGate - Looking Behind The Scenes

11 Optional Oracle GoldenGate Architecture: Big Picture Oracle GoldenGate Instance Oracle GoldenGate Instance Source DB Optional Primary (Passive) Extract Manager (VIP) MONITOR Manager (VIP) MONITOR Optional Primary (Alias) Extract Target DB Primary (Regular) Extract Collector Replicat Local Trail Secondary Extract Data Pump Remote Trail Static Configuration Runtime Configuration Runtime Configuration Static Configuration Data Replication With Oracle GoldenGate - Looking Behind The Scenes

12 Primary Extract Group: Capture Source DB Primary Extract can run in either Classic Capture mode (all database versions) Integrated Capture mode (full support: ) Primary (Regular) Extract Primary (Passive) Extract Extracts and optionally filters/transforms the transactions from the source database Only for configured objects Many Extracts per source database can operate at the same time Only committed transactions (CSN) will be propagated to the target system (in contrast to Oracle Streams ) Optionally written to a local trail (recommended) Without a local trail, the primary Extract send the data to the target system Not committed transactions will be discarded from memory or dirtmp directory Data Replication With Oracle GoldenGate - Looking Behind The Scenes

13 Primary Extract Group: Classic Capture Classic Capture mode Traditional mode, reads directly Oracle database redo logs Able to mine redo logs in ASM (two methods) Redo Logs All database versions supported Source DB With some restrictions. For example no support for: BASIC, OLTP, Exadata Hybrid Columnar compression PDML on RAC database CDB architecture! DDL, TDE support no out-of-the box support Primary Extract OS System Calls Local Trail Secondary Extract Data Pump Not transparent to RAC configuration changes (#threads) Data Replication With Oracle GoldenGate - Looking Behind The Scenes

14 Primary Extract Group: Integrated Capture Integrated Capture mode Interacts with the database logmining server Required to capture data from a CDB architecture Fully transparent with RAC, TDE, ASM Full support for databases Source database does not require supplementary objects for DDL support Supports more complex data types than classic capture Integrated capture deployments Local or a downstream deployment The best choice for most deployments Logmaining Server LCR LCR Primary Extract Local Trail Redo Logs Secondary Extract Data Pump Source DB Data Replication With Oracle GoldenGate - Looking Behind The Scenes

15 Replicat Process: Apply The Replicat process can run in either Classic Replicat mode (nonintegrated) - all database versions Coordinated Replicat mode (nonintegrated) - all database versions, GoldenGate 12c required Integrated Replicat mode database versions Target DB Replicat The Replicat process reads the remote trail, reconstructs the SQL and applies them to the target database Optionally filters/transforms the data Remote Trail The application of the data can be optionally delayed (DEFERAPPLYINTERVAL parameter) Data Replication With Oracle GoldenGate - Looking Behind The Scenes

16 Replicat Group: Classic/Coordinated Mode Classic Replicat Constructs SQL statements from the trails and applies the SQL through OCI Supported for all database versions Transactions are applied sequentially (single-threaded) Replicat SQL OCI Remote Trail Target DB Coordinated Replicat Transactions are applied in parallel (multi-threaded) There is 1 coordinator plus many apply threads Supported for all database versions Remote Trail Coordinator Thread 1 SQL OCI Target DB Thread 2 Thread n Data Replication With Oracle GoldenGate - Looking Behind The Scenes

17 Replicat Group: Integrated Mode Integrated Replicat Supported for target database versions Replicat LCR Database Inbound Server Target DB Constructs LCRs and transmits DMLs to the database inbound server Remote Trail SQL OCI Some changes (e.g. DDL) are not supported directly by the inbound server In those cases the Replicat process disconnects from the inbound server and executes the transaction in direct apply mode through OCI With many transactions applied in direct apply mode it might be necessary to switch to nonintegrated Replicat Data Replication With Oracle GoldenGate - Looking Behind The Scenes

18 Capture/Apply: CDB & Pluggable Databases Extract must operate in integrated mode Extract must connect to the CDB$ROOT container as a common user (c##) One Extract can capture from one or many PDBs to a single trail common_user_prefix ( ) Replicat can connect and apply only to one PDB (one Apply/PDB, connect using service) Source DB CDB CDB$ROOT PDB1 PDB2 Target DB CDB CDB$ROOT PDB1 PDB2 Primary Extracts INTEGRATED Data Pumps Collectors Replicats Data Replication With Oracle GoldenGate - Looking Behind The Scenes

19 Oracle GoldenGate Setup Data Replication With Oracle GoldenGate - Looking Behind The Scenes

20 Oracle GoldenGate: Setup Flow Architecture Software Installation/ Design Patches Database Preparation 6 5 Primary Extract, Data 4 Register/Start Primary Pump, Replicat Extract, Data Pump Configuration Files Manager Configuration Target Database Register/Start Replicat Instantiation Cluster Integration / INIT Scripts n Tuning/Customizing/ Maintenance/ Data Replication With Oracle GoldenGate - Looking Behind The Scenes

21 Supplemental Logging: Source 3 Database Preparation Activate minimal database level supplemental logging on the source database To add row chaining information to the redo stream SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; Add schema or table level supplemental logging on the source database Schema level logging is required for DDL support For all schema tables ADD SCHEMATRANDATA <schema> [ALLCOLS] Activates unconditional supplemental logging for PK and conditional for unique/foreign keys (scheduling columns) all existing an future tables For a table Depending on the use-case, different parameters are necessary ADD TRANDATA <schema>.<table> [COLS ( ) ALLCOLS] Data Replication With Oracle GoldenGate - Looking Behind The Scenes

22 Manager Configuration: Target/Source 4 Manager Configuration Manager configuration (tune the values according to your needs) <GG_HOME>/dirprm/mgr.prm PORT 7809 unreserved and unrestricted DYNAMICPORTLIST AUTOSTART ER * AUTORESTART ER *, RETRIES 4, WAITMINUTES 4, RESETMINUTES 60 PURGEOLDEXTRACTS <...>/dirdat/*ext*/*t*, USECHECKPOINTS, & MINKEEPHOURS 4 LAGREPORTMINUTES 30 LAGINFOMINUTES 5 LAGCRITICALMINUTES Data Replication With Oracle GoldenGate - Looking Behind The Scenes

23 Primary Extract Configuration 5 Primary Extract, Data Pump, Replicat Configuration Files Create the configuration file for the primary Extract (source) EXTRACT app1ext1 SETENV (ORACLE_HOME = "/u00/app/oracle/product/ ") USERIDALIAS ggadm_app1_11204 You need GoldeGate LOGALLSUPCOLS Credential Store UPDATERECORDFORMAT COMPACT DDL INCLUDE MAPPED EXCLUDE OBJNAME APP1.T4 & EXCLUDE OBJNAME APP1.T5 ENCRYPTTRAIL AES256 EXTTRAIL <...>/dirdat/app1ext1/lt SEQUENCE APP1.*; TABLE APP1.*; A Master-Key Wallet is necessary Data Replication With Oracle GoldenGate - Looking Behind The Scenes

24 Secondary Extract/Replicat Configuration Files 5 Primary Extract, Data Pump, Replicat Configuration Files Create the configuration file for the Data Pump Extract (source) Not necessary if the trail is already EXTRACT app1pmp1 encrypted USERIDALIAS ggadm_app1_11204 RMTHOST ggmgr-cl121.trivadis.com, MGRPORT 7809,ENCRYPT AES256,& COMPRESS, COMPRESSTHRESHOLD 1200 RMTTRAIL <...>/dirdat/app1rep1/rt SEQUENCE APP1.*; TABLE APP1.*; Create the configuration file for the Replicat group (target) REPLICAT app1rep1 SETENV (ORACLE_HOME = "/u00/app/oracle/product/ ") DBOPTIONS INTEGRATEDPARAMS(parallelism 6) USERIDALIAS ggadm_app1_12102 DDLOPTIONS REPORT PDB ASSUMETARGETDEFS Mapping: non-cdb CDB MAP APP1.*, TARGET APP1.APP1.*; Data Replication With Oracle GoldenGate - Looking Behind The Scenes

25 Primary/Secondary Extract Group 6 Register/Start Primary Extract, Data Pump Register the primary Extract with the database (integrated) GoldeGate Credential DBLOGIN USERIDALIAS gg_app1_11204 Store REGISTER EXTRACT app1ext1 DATABASE [CONTAINER (APP1)] Extract APP1EXT1 successfully registered with database at SCN Create the primary Extract group (integrated) ADD EXTRACT app1ext1, INTEGRATED TRANLOG, BEGIN NOW Add a local trail an associate it with the primary Extract ADD EXTTRAIL <...>/dirdat/app1ext1/lt, EXTRACT app1ext1 Add the Data Pump Extract ADD EXTRACT app1pmp1, EXTTRAILSOURCE <...>/dirdat/app1ext1/lt Data Replication With Oracle GoldenGate - Looking Behind The Scenes

26 Secondary Extract/Replicat Group 8 Register/Start Replicat Add a remote trail and associate it with the secondary Extract ADD RMTTRAIL <...>/dirdat/app1rep1/rt, EXTRACT app1pmp1 Add the Replicat group (integrated) ADD REPLICAT app1rep1, INTEGRATED, EXTTRAIL <...>/dirdat/app1rep1/rt Start the Replicat on the target with the correct CSN GGSCI> START REPLICAT app1rep1, AFTERCSN Data Replication With Oracle GoldenGate - Looking Behind The Scenes

27 Oracle GoldenGate: Demo Environment Source DB: 2-nodes RAC Schema: APP1 GoldenGate version: Active/Passive data replication Target DB: CDB 2-nodes RAC PDB: APP1, Schema: APP1 GoldenGate version: Active/Passive data replication CDB$ROOT APP1 APP2 APP3 Schema: APP1 Primary Extract INTEGRATED Local Trail RemoteTrail Replicat INTEGRATED Secondary Extract Data Pump Collector Data Replication With Oracle GoldenGate - Looking Behind The Scenes

28 Core Messages Data Replication With Oracle GoldenGate - Looking Behind The Scenes

29 Core Messages GoldenGate is a very powerful replication software framework For most deployments the integrated Extract/Replicat is the best choice Supplemental logging generates more undo and redo In some cases you need even to supplementally log all columns! Script all the configuration steps Very useful if you need to re-setup the whole environment You don t need to replace your current Streams installation ASAP by Oracle GoldenGate, but Streams is deprecated in the version 12c and will not be enhanced in future releases There is a framework to ease the migration from Streams to GoldenGate (streams2ogg) As of now Streams is not supported with CDB architecture For new projects it might be a better choice (licensing)! Data Replication With Oracle GoldenGate - Looking Behind The Scenes

30 Questions and answers Robert Bialek Principal Consultant Partner Tel Data Replication With Oracle GoldenGate - Looking Behind The Scenes