Dbvisit Software. The 3 fundamental principles of Oracle replication. Mike Donovan CTO Dbvisit Software Dbvisit Software dbvisit.

Transcription

1 Dbvisit Software The 3 fundamental principles of Oracle replication Mike Donovan CTO Dbvisit Software 2017 Dbvisit Software dbvisit.com

2 Mike Donovan Chief Technology Officer, Dbvisit Software Multi-platform DBA, (Oracle, MSSQL) Conference speaker: OOW, RMOUG, dbtech Showcase, Collaborate NZOUG member Technical Writer and Editor Oracle ACE Tech enthusiast PROFESSIONAL NOT-KNOWER OF THINGS

3 Result: 1,300+ customers in 6 continents

4 Dbvisit Software Real-time Oracle Database Streaming software solutions In the Cloud Hybrid On-Premises New Zealand-based, US office, Asia Sales office, EU office (Prague) Unique offering: disaster recovery solutions for Oracle Standard Edition Logical replication for moving data where ever and whenever you wish Flexible licensing, cost effective pricing models available Exceptional growth, customers Peerless customer support

5 The World we live in The Situation: The enterprise is increasingly powered by data OLTP transactional data essential The use of real-time data for competitive advantage is disrupting most industries Real-time continuous streaming is changing the dataflow in organizations The Objective: Understanding the fundamental principles and considerations of Oracle logical replication 2017 Dbvisit Software dbvisit.com

6 The Thrills of Matchmaking and Cloud Integration Hello world! No, potatoes Dbvisit Software dbvisit.com

7 The way of the (new) world

8 Agenda The difference between physical and logical replication How does logical replication work 3 replication principles and how they apply Considerations with logical based replication 2017 Dbvisit Software dbvisit.com

9 Two Oracle Replication Types Physical Replication One to one copy of the primary database in recovery state Use database block level (redo) to keep up to date 100% binary copy, structure and data Referred to as a standby database Best suited for DR

10 Two Oracle Replication Types Logical Replication Independent 2 nd database kept in sync by replication mechanism Uses SQL statements to keep database up to date Subset of data is replicated Cross version, cross platform, cross database Separate physical database structure Best suited for information sharing, migrations, real-time reporting etc

11 Logical Replication Options 1. Selects Periodic retrieval of data from the source Identify deltas Simple, but doesn t scale and misses changes 2. Trigger based Changes to source database required Performance impact A lot more maintenance 3. Redo log mining (preferred Oracle solution) No changes to source database Lower impact to source environment Less maintenance DDL changes can be replicated

12 What is the most fundamental structure in the entire Oracle database? The Online Redo Log Files The redo log stores a continuous chain of chronological order of every change vector applied to the database. This will be the bare minimum of information required to reconstruct, or redo, all the work that has been done. If a datafile (or the whole database) is damaged or destroyed, these change vectors can be applied to datafile backups to redo the work, bringing them forward in time until the moment that the damage occurred. OCA EXAM GUIDE (page 89)

13 Redo log - Redo gets it first Changes written to database in order: 1. Written to redo log buffer 2. Written from redo log buffer to redo log (at commit, 3 seconds, 1MB, 1/3 full or when DBWn writes) 3. Written to database files Redo will include: 1. Insert, Update, Delete, Merge 2. Select for Update 3. DDL 4. No Select, no nologging operations

14 Oracle Redo logs Reading the Oracle redo logs is not easy. Oracle doesn t really publish the API. Because of this not many companies have developed the capability of replicating of Oracle data.

15 How does redo log mining work? CHANGE #3 TYP:0 CLS:1 AFN:4 DBA:0x b OBJ:27521 SCN:0x ab0ab9 SEQ:2 OP:11.2 ENC:0 RBL:0 KTB Redo op: 0x01 ver: 0x01 compat bit: 4 (post-11) padding: 1 op: F xid: 0x c uba: 0x00c31f4a.043f.15 KDO Op code: IRP row dependencies Disabled xtype: XA flags: 0x bdba: 0x b hdba: 0x a itli: 1 ispac: 0 maxfr: 4858 tabn: 0 slot: 0(0x0) size/delt: 24 fb: --H-FL-- lb: 0x1 cc: 6 null: col 0: [ 2] c1 07 col 1: [ 5] col 2: [ 2] c1 16 col 3: [ 2] c1 02 col 4: [ 2] col 5: [ 2] c1 07 insert into HR.EMPLOYEES values (6,'Perry',21,1,'IT,6);

16 Typical logical replication flow MINE Mines the redo logs and converts into a parsed log PLOG Parsed logs binary files specific to solution Platform independent APPLY Converts parsed log data into target DB native SQL

17 3 principles of Oracle redo log mining replication 1. Conversion process All set based SQL operations are converted to row-byrow SQL changes 2. Identification For the data in the table to be replicated, each row in the table must be uniquely identified 3. Conflicts (collisions) Conflicts warn of data divergence and lost updates

18 Principle 1 - Conversion process Set based SQL operations on the source database are converted to row-by-row SQL changes on the target database. Why? - This is the way that Oracle writes to the Redo - Redo does not contain SQL, it has to be rebuilt from LCRs Observations: - Each SQL on target only affects 1 row - SQL is not the same on source as on target - True for all logical replication based solutions

19 1. Logical replication principle Source update EMP set SAL = 1.05*SAL where DEPTNO = 10; PK has been added to the WHERE clause Source SQL updates 3 rows, then 3 individual updates statements are produced Salary formula has been replaced by hardcoded value Target update EMP set SAL = where (1=1) and SAL = 2450 and EMPNO = 7782; update EMP set SAL = 5250 where (1=1) and SAL = 5000 and EMPNO = 7839; update EMP set SAL = 1365 where (1=1) and SAL = 1300 and EMPNO = 7934;

20 1. Logical replication principle select * from REPOE.COLOUR; COLOUR green green DESCRIPTION Fresh green Fresh green delete from REPOE.COLOUR where description like '%green%';

21 1. Logical replication principle SEQ# = / FRAG# = 0 (last frag) - RBA = cf SCN = TIME = :22:41 - TIME(THL) = :32:20 - SCHEMA = REPOE - TRANSACTION = SQL(0) = delete from "REPOE"."COLOUR" where (1=1) and "COLOUR" = 'green' and "DESCRIPTION" = 'Fresh green' - ACTION = DELETE - SCHEMA = REPOE - TABLE = COLOUR - ROW# = 0 - COLOUR(:1) = green - DESCRIPTION(:2) = Fresh green SEQ# = / FRAG# = 0 (last frag) - RBA = d SCN = TIME = :22:41 - TIME(THL) = :32:20 - SCHEMA = REPOE - TRANSACTION = SQL(0) = delete from "REPOE"."COLOUR" where (1=1) and "COLOUR" = 'green' and "DESCRIPTION" = 'Fresh green' - ACTION = DELETE - SCHEMA = REPOE - TABLE = COLOUR - ROW# = 0 - COLOUR(:1) = green - DESCRIPTION(:2) = Fresh green

22 Principle 2 - Identification For the data in the table to be successfully replicated, each row in the source table must be able to be uniquely identified To uniquely identify each row, ONE of the following must be true: 1. Primary key 2. Unique key 3. All data in columns in row must produce a unique result If none are true, then table replication may cause conflicts

23 2. Logical replication principle Example CREATE TABLE SALES (SALE_ID NUMBER PRIMARY KEY, PROD_ID NUMBER NOT NULL, CUST_ID NUMBER NOT NULL, QUANTITY_SOLD NUMBER(10,2) NOT NULL, AMOUNT_SOLD NUMBER(10,2) NOT NULL, AMOUNT_RECEIVED NUMBER(10,2), SALES_STATUS VARCHAR2(10) NOT NULL ); ALTER TABLE SALES ADD CONSTRAINT SALES_UK UNIQUE (PROD_ID, CUST_ID);

24 2. Logical replication principle Source statement: update SALES set AMOUNT_RECEIVED = 120 where SALES_STATUS = 'OVERDUE' (old value AMOUNT_RECEIVED is NULL) Target statement: Primary key (sale_id) update SALES set AMOUNT_RECEIVED = 120 where SALE_ID = 101 and AMOUNT_RECEIVED is NULL PK in where clause Old value of updated column in where clause

25 2. Logical replication principle Unique key (prod_id, cust_id) update SALES set AMOUNT_RECEIVED = 120 where PROD_ID = 101 and CUST_ID = 201 and AMOUNT_RECEIVED is NULL UK in where clause Old value of updated column in where clause

26 2. Logical replication principle No primary or unique keys update SALES set AMOUNT_RECEIVED = 120 where SALES_STATUS = 'OVERDUE' and PROD_ID = 101 and CUST_ID = 201 and QUANTITY_SOLD = 233 and AMOUNT_SOLD = 1299 and AMOUNT_RECEIVED is NULL All columns in where clause Old value of updated column in where clause If all columns in the where clause do not uniquely identify the row, then the table cannot be guaranteed to be replicated Why? Because of principle #3

27 Principle 3 - Conflicts Conflict = potential for the data between source and target to be out of sync data divergence (not enough, too much, subsidiary effects) Conflict occurs when the SQL statement run on the target affects: update statement updates 0 rows Zero rows delete statement deletes 0 rows More than one row update statement updates 2 or more rows Oracle errors delete statement deletes 2 or more rows Why are these conflicts? Because otherwise it breaks principle 1 Unique Key or Foreign key violations any other oracle error

28 3. Logical replication principle Update statement replicated from source update SALES set AMOUNT_RECEIVED = 120 where AMOUNT_RECEIVED is NULL and SALES_ID = 101 Meantime someone on target has updated row update SALES set AMOUNT_RECEIVED = 60 where SALES_ID = 101 Result: Conflict 0 rows updated Potential to lose update (amount_received = 60 instead of 120)

29 Review - 3 principles of Oracle logical replication 1. Conversion process Row-by-row SQL: - because of the redo log structure - to ensure row level conflict detection 2. Identification - each row uniquely identified - old value included in WHERE 3. Conflicts Conflicts warn of data divergence and lost updates (More info:

30 Demo 1. Force update on target first 2. Make change to duplicate rows (no PK/UK/not distinct) on source

31 Considerations with logical based replication 1. Effectively manage conflicts 2. Data instantiation 3. Triggers and cascading constraints 4. Sequences 5. Commit strategies

32 Considerations with logical based replication 1. Effectively manage conflicts a. Conflict detection when there is a possibility of data divergence, a notification or alert is triggered. b. Conflict resolution when a conflict occurs, enough information about the conflict is known so that the conflict can be resolved (ignore, rewrite ) c. Conflict handling set predefined rules to automatically determine what to do when conflict occurs (includes setting PL/SQL business rules) Be careful! The goal isn t to get rid of the conflicts without understanding the cause.

33 Considerations with logical based replication 2. Data instantiation Before replication can start, data needs to be in sync between source and target. Ensure: - All data is captured including in-flight data - All required indexes - PL/SQL Use 1. Inbuilt options of tool 2. Datapump (export/import) 3. Rman duplicate 4. Standby database (resetlogs)

34 Considerations with logical based replication 3. Triggers and cascading constraints Beware of side effects These are recursive SQL/changes and they also get replicated at row level. So if you have such trigger/fk, you end up applying the same change twice (and create conflicts). Turn off or disable triggers at the target for data that is replicated. Some products can deal with this.

35 Considerations with logical based replication 4. Sequences - Replication does not update target Oracle sequences - Value inserted by sequence is replicated End result - Primary key values will be the same on both source and target - Sequences will not be the same Ensure strategy in place for Active-active or migrations

36 Considerations with logical based replication 5. Commit strategies 1. Optimistic commit (do not wait for commit before mining and applying) 2. Wait for commit before applying (do not wait for commit before mining) 3. Wait for commit before mining and applying Examples of products with above strategies: 1. Dbvisit Replicate 2. Streams 3. Golden Gate

37 CDC (Change Data Capture) Is a type of logical replication - Is used to record the history of changes - Source and target are not in sync - Every change is an insert into the target table, even deletes are an insert - Used in loading BI and streaming - Think of getting a full history of shopping behavior before people commit to purchase

38 Thank you. Come and see Dbvisit at Booth 2017 Dbvisit Software dbvisit.com