Application Development Best Practice for Q Replication Performance

Transcription

1 Ya Liu, InfoSphere Data Replication Technical Enablement, CDL, IBM Application Development Best Practice for Q Replication Performance Information Management

2 Agenda Q Replication product overview How application changes are replicated? Application design requirements for Q Replication to function correctly Application design best practice for Q Replication to work with high performance

3 Q Replication IBM s strategic data replication technology for the DB2 Family Has a large, well-established customer base Is a key technology of IBM s Active-Active strategy Is developed in close cooperation with DB2 development teams Supports new DB2 features at DB2 GA time Available in multiple products Supports both mainframe and distributed platforms Technical characteristics Asynchronous replication not limited by geographical distance Application oriented can replicate a subset of tables Log-based change data capture lowest impact to source system Only changed data is delivered minimum data processing Transferring and staging data in IBM WebSphere MQ queues excellent data recovery capability Target data always transactional consistent target data available at any time Parallel data applying high performance Run as a service no batch window 3

4 Q Replication Architecture Administration & Monitoring Tools Replication Center Dashboard Alert Monitor GUI tools to configure and monitor Q Replication Script tool to deploy Q Replication for plenty of tables Utilities to compare table contents Source SOURCE1 SOURCE2 Target Capture Apply TARGET2 TARGET1 DB Log WebSphere MQ WebSphere MQ A Capture program reads changed data from the database recovery log, and puts data directly into WebSphere MQ queues WebSphere MQ delivers data to a target system where an Apply program runs The Apply program pulls data from queues and applies to target tables 4

5 Usage Scenarios Log-based Capture Real-time Capture Asynchronous Changed Data Only Transactional Source DB 1) Multiple systems with a center or not for: - Information Integration - Synchronization - Distribution and Consolidation 2) One target system for: - Offload Query/Reporting - Real-time BI (Dynamic Warehouse) - Audit 3) Two data centers for: - High/Continuous Availability - Active/Active 4) Two systems for system maintenance such as: - Migration - Upgrade 5) Changed data pushed into message queues for Event-Driven Business 5

6 What To Buy: Products and Licenses Q Replication is available in multiple IBM products No additional license is needed to run Q Replication if deployed via InfoSphere Data Replication or InfoSphere Replication Server products Additional licenses may be needed to run Q Replication if deployed via DB2 Database or InfoSphere Warehouse products IBM Product Platform Additional License Needed InfoSphere Data Replication InfoSphere Data Replication z/os LUW DB2 Database Enterprise Server Edition LUW Homogeneous Replication Feature DB2 Database Advanced Enterprise Server Edition LUW Free when replicating with another DB2 LUW server (v97) Free when replicating with other two DB2 LUW servers (v10) InfoSphere Warehouse LUW Free when replicating with another DB2 LUW server (v97) Free when replicating with other two DB2 LUW servers (v10) 6

7 Customers > 1000 Worldwide Customers Across Banking, Insurance, Retail, Public, Energy, Manufacturing, Education, Health Industries 7

8 Agenda Q Replication product overview How application changes are replicated? Application design requirements for Q Replication to function correctly Application design best practice for Q Replication to work with high performance

9 How Application Changes Are Replicated? Data changes made by user applications are saved in the recovery log of source database as transactions. Each row change is described by a single log record. Q Capture scans source recovery log, and extracts the committed transactions. These transactions are packaged as WebSphere MQ messages and published to MQ. Q Capture only cares DML changes (Insert/Update/Delete) and some special DDL changes. Log records for other changes are ignored. The MQ messages are transferred by MQ channels to target site. Q Apply unpacks source database transactions from MQ messages, generates a SQL statement for each row DML change, and commits the generated transaction SQL to target database in a parallel way. Q Apply is also a database application. The generated SQL may be totally different than the source SQL used in applications. Internal transaction data MQ messages Internal transaction data Source SOURCE1 SOURCE2 Target Capture Apply TARGET2 TARGET1 DB Log WebSphere MQ WebSphere MQ LOG records / UOWs MQ messages MQ messages SQL statements generated by Q Apply

10 A Single Log Stream For All Source Applications Database transaction recovery log is a complete history of all changes to the database Write Ahead Log (WAL) : Changes are written to log before they are saved to the database. No log record is ever deleted. The log records of different transactions are usually interleaved. With full logging enabled (DATA CAPTURE CHANGES), entire row is logged. App 1 App 2 App 3 App 4 Tx1 Tx2: Tx3: Tx4: INSERT T1,R3 UPDATE T2,R2 ROLLBACK UPDATE T2,R1 DELETE T1,R1 COMMIT INSERT T1,R4 Server T1 T2 DB2 Database recovery log:... Tx1: insert T1, R3 Tx2: update T2, R1 Tx3: delete T1,R1 Tx1: update T2,R2 Tx3: commit Tx1: undo insert T1 Tx4: insert T1, R4 Tx1: undo update T2 Tx1: rollback... 10

11 Application Changes Are Captured As Transactions Application SQL: TX1 TX2 INSERT INTO T1 UPDATE T2 WHERE DELETE FROM T4 WHERE COMMIT INSERT INTO T2 Q CAPTURE In-Memory Transactions Send Queue TX3 DELETE FROM T1 WHERE DB2 LOG TX1: INSERT T1 TX2: INSERT T2 LOGRD Thread TX3: DELETE T1 TX3: ROLLBACK TX1: INSERT T1 TX1: UPDATE T2 TX1: TX1: COMMIT TX2: INSERT T2 PUBLISHER Thread TX1: MQ Put when database COMMIT record is found TX3: DELETE T1 TX1: UPDATE T2 TX1: TX3: ROLLBACK TX1: COMMIT TX3: Zapped at database ABORT Never makes it to send queue TX2: Transaction is still in-flight Nothing inserted yet. One application SQL statement may result in multiple row changes or log records. Only following types of data changes or operations will be published by Q Capture: DML: INSERT/UPDATE/DELETE DDL: ALTER TABLE ADD COLUMN, ALTER TABLE ALTER COLUMN SET DATA TYPE LOAD, REORG DISCARD (only the operation type is published) 11

12 Application Changes Are Transferred As MQ Messages Source Server Target Server DB2 (log) MQ Server Queue Manager MQ Server Queue Manager DB2 Q C A P T U R E SENDQ ADMINQ XMITQ Send Channel Send Channel TX2: msg 3 NETWORK TX1: msg 2 TX1: msg 1 Receive Channel Receive Channel XMITQ RECVQ ADMINQ Q A P P L Y RESTARTQ U D I D D U U I First In First Out (FIFO) SPILLQ 12 Usually 1 source database transaction is packaged as 1 MQ message, containing all row changes in the transaction. Too big source transactions are segmented into multiple MQ messages, with boundaries at rows. Special data types, such as large object (LOB), may also be sent in separated MQ messages. Q Capture also sends control messages to Q Apply, via the same MQ channels as data messages. Q Apply sends control messages to Q Capture via the MQ channels for ADMINQ.

13 Application Changes Are Replayed Using Row Operation SQL Q Apply re-constructs database transactions in memory Constructed transactions are assigned to agent threads for parallel applying Agent threads generate target SQL statements for each assigned transaction based on row-level changes Generated SQL statements may be totally different than SQL statements issued by source applications Q Apply control table (DONEMSG) is also changed for each applied target transaction MQ messages and DONEMSG records are asynchronously cleaned later by pruning thread Target SQL (by Q Apply): TX1 TX2 INSERT INTO T1 UPDATE T2 WHERE UPDATE T2 WHERE DELETE FROM T4 WHERE DELETE FROM T4 WHERE INSERT INTO DONEMSG (msg1, msg2) COMMIT INSERT INTO T2 DELETE FROM T3 WHERE UPDATE T2 WHERE INSERT INTO DONEMSG (msg3) COMMIT Receive Queue TX2: msg 3 TX1: msg TX1: msg 1 BROWSER Thread PRUNING Thread Q APPLY TX1: INSERT T1 TX1: UPDATE T2 TX1: TX1: COMMIT TX2: INSERT T2 TX2: TX2: COMMIT AGENT AGENT AGENT AGENT DB2 connections DB2 TARGET2 TARGET1 DONEMSG U D I D D U U I In-memory Transactions First In First Out (FIFO)

14 Agenda Q Replication product overview How application changes are replicated? Application design requirements for Q Replication to function correctly Non-logged data changes Auto-generated column values Multi-row update statements with temporary duplicates Duplicated rows Application design best practice for Q Replication to work with high performance

15 Not All Data Changes Can Be Replicated Q Replication is unable to replicate all data changes by applications This is a native limit of Q Replication and cannot be fixed by changing configuration Application developers must be aware of the requirements from Q Replication, and avoid to create such data changes in the applications Some data changes that Q Replication is unable to replicate Non-logged data changes Auto-generated column values Multi-row SQL update statements Duplicated rows

16 Non-logged Data Changes Q Replication obtains source database changes by reading recovery log Non-logged data changes (or operations) are not replicated Examples of non-logged data changes Non-logged LOB columns Non-logged LOAD operations Non-logged utilities RUNSTATS REORG

17 Auto-generated Column Values Q Replication replays source database changes by executing SQL statements against target database For target table columns whose values are automatically generated, Q Replication cannot replicate the values Such columns are excluded from replication definition (subscriptions) The actual values of these columns depends on the value generation rules in target database For these columns, the values may diverge between source and target Examples of auto-generated columns IDENTITY GENERATED ALWAYS GENERATED ALWAYS

18 Multi-row Update Statements With Temporary Duplicates One SQL update statement may update multiple rows, and may cause temporary duplicates during SQL statement execution DB2 can execute such a statement because it postpones uniqueness constraints until the whole statement execution completes Q Replication will translate the changes by such a statement into multiple SQL update statement, and the execution will fail due to uniqueness constraint Such SQL statements are bad practice and should be avoided Example Table definition create table T1 (ID int not null primary key, NAME char(20)) Table data before SQL execution ((1, Tom ), (2, Katty ), (3, David )) Source SQL statement update T1 set ID = ID + 1

19 Duplicated Rows There may be duplicated rows if there is no uniqueness constraints in target table at all primary key, unique constraint, or unique index Q Apply will use all subscribed target columns as a replication key, and generates target SQL statements based on this replication key If the replicated operation is DELETE or UPDATE, and operating on a duplicated row, all rows in target table with the same content will be deleted or updated, and cause data mismatch Example Table definition create table T1 (C1 int, C2 char(20)) Table data before SQL execution ((1, Tom ), (1, Tom ), (3, Katty ), (4, David )) Source change Delete the 2 nd row (2, Tom ) Target SQL generated by Q Apply Delete from T1 where C1=1 and C2= Tom This will delete 2 rows in target table

20 Agenda Q Replication product overview How application changes are replicated? Application design requirements for Q Replication to function correctly Application design best practice for Q Replication to work with high performance LOB data No unique constraint Hot rows Big transactions Long running transactions Target table space lock level

21 LOB Data Non-logged LOB Q Capture fetches LOB column value from table when publishing row changes The performance is very bad Logged LOB In-lined LOB Supported since DB2 for z/os v10 and DB2 for LUW v9.7 Q Capture fetches LOB column value directly from DB2 recovery log records Users should define inline LOB in DB2 whenever possible. This requires the LOB data is not too big in application design. Not in-lined LOB Q Capture fetches LOB column value from table when publishing row changes The performance is very bad Since DB2 for LUW v10.1, Q Capture fetches LOB column values directly from DB2 recovery log records, no matter they are in-lined or not Best practice Avoid using big LOB data in application design Define LOB data as logged if LOB must be used Use in-lined LOBs as much as possible 21

22 No Unique Constraint Target table is created with no unique index Q Replication is unable to find a unique index/constraint definition in DB2 database to use as replication key, which makes Q Apply capable to uniquely identify a row in the target table when applying row changes Q Replication will use all columns as replication key This is not always functionally correct when replicating DELETE/UPDATE operations This has performance issues and may result in poor DB2 execution plan, such as table scan since no DB2 index is available to optimize SQL execution performance Best practice For each target table with lots of changes, if there is no unique index, create one If there is no uniqueness in target table, alter target table to add a new column for uniqueness (such as, identity), and define unique key on it

23 Hot Rows Hot row impacts Q Apply performance Example: A common table stores a row which contains variables used and updated by each transaction in a batch job Each transaction contains row changes on this hot row Q Apply calculates dependency between transactions by checks the uniqueness of replication key and secondary unique indexes, and RI constraints Hot row will cause lots of dependencies on uniqueness of replication key, and impacts Q Apply parallelism and throughput Best practice Avoid to use hot row in application design. If there is, revise application design to make it not so hot. 23

24 Big Transactions Big transactions (called monster transactions in Q Replication) impact both Q Capture and Apply performance If they are bigger than Q Capture MEMORY_LIMIT, Q Capture will spill them to disk before publishing them, resulting in more I/O operations If they are bigger than Q Apply MEMORY_LIMIT, Q Apply will work in serial mode Best practice Avoid very big transactions as much as possible, such as, forcefully commit after some (such as, 1000) row changes Use the Q Capture warntxsz parameter to detect the existence of very large transactions, and tune MEMORY_LIMIT parameters Some customers chose to exclude some very large transactions (e.g., a purge job that deletes GBs of data in a single DB2 transaction) from replication by using the IBMQREP_IGNTRAN table, and manually execute the job at each site 24

25 Long Running Transactions Although a long running transaction may be not big for example containing only several row changes, it still impacts Q replication performance If Q Capture stops before a long running transaction terminates, when Q Capture restarts, it must start reading source database log from a position no later than the beginning of this long running transaction for example, reading log from 2 hours ago. This will impact Q Capture restart performance. The long running transaction occupies Q Capture memory until it s finally committed or rolled back, wasting Q Capture resource. Best practice Avoid long running transactions as much as possible Some users monitor if there re long running transactions (such as, not terminated after started for 5 minutes), and send out alerts to notify application developers to change application design

26 Target Table Space Lock Level Target database tablespace lock level sometimes is important to Q Apply performance For example, if target tablespace is page locking, and many Q Apply agents are changing tables in this tablespace, there may be severe lock contention. Best practice For a target tablespace where number of row changes per second is high, recommend to use row-level locking, to reduce lock contention and improve Q Apply performance.

27 References Information Center: Database constructs and workloads that require special consideration for Q Replication onstructs.html Information Center: Q Replication and Event Publishing Performance Tuning Guide over.html Red Paper: InfoSphere Data Replication for DB2 for z/os and WebSphere Message Queue for z/os Q Replication Information Roadmaps (IBM developerworks) Q+SQL Replication Forum (IBM developerworks) nityuuid=a82063fa-aecf a4-8d33425a