Oracle-to-Oracle Data Transfer Options. Sudhanshu Kumar 1 st Feb 2010

Transcription

1 Oracle-to-Oracle Data Transfer Options Sudhanshu Kumar 1 st Feb 2010

2 Data Extraction & Transfer Options Non Real Time Real/Near-Real Time Transportable Tablespaces Pulling Tablespaces Data Pump Oracle Replication Change Data Capture Direct Access using dblink (SQL*Net) File Extracts

3 Transportable Tablespaces Pros The quickest way to transport an entire tablespace to a target system is to simply transfer the tablespace's underlying files. For a tablespace to be transported to a target system, it must be selfcontained. The source database tablespace needs to be made read-only until the copying of tablespace ends. Cross-platform issues - Endian-ness of operating system ASM (will need RMAN to pull out tablespaces into file structures)

4 Data Pump Data Pump, introduced in 10g, replaces the older Export-Import tool. Data Pump can be used in below modes Transportable tablespace Just a Datapump approach of TTS discussed in previous slide Table/Schema dump Discussed below External Tables Discussed below Network Link (Dblink) Discussed in next slide Slower than Transportable tablespaces. Tests Using EXPDP & IMPDP (Parallel=4) Uses Table dump mode Rows - 65 million Export - 00:02:30 FTP - Depends on n/w speed 10 GB file Can use shared volume/disk Import - 00:01:44 ODI LKM Oracle to Oracle (Data pump) - Uses External Tables mode 1 million records 00:00:17 5 million records 00:01:45 10 million records 00:04:14 20 million records 00:07:23 40 million records 00:13:53 65 million records 00:26:34

5 Data pump (Network Link Mode) Uses Dblink between source & target i.e. data transfer happens through SQL*Net Pros In comparison to copy data directly using dblink method which shows exponential increase in time over data growth, this method shows consistent linear increase. Slower than Transportable tablespaces Slower than Data Pump (dump file) Slower than Data Pump (External Tables) Tests Using Network Link (Parallel=4) 2 million records 00:01:22 4 million records 00:02:27 8 million records 00:04:45 16 million records 00:09:00 32 million records 00:17:09

6 Pulling Tablespaces New tool in Oracle Database 10g that simplifies the transportable tablespace approach so that it involves only the execution of a single packaged procedure. In this approach, the user "pulls" the tablespaces from the source system, using a supplied package, DBMS_STREAMS_TABLESPACE_ADM. This package uses Data Pump to transport tablespaces and transfers the data files to the target system's format. It also performs any required endian conversion automatically. Limitation in terms of flexibility because it s a packaged approach.

7 Change Data Capture Synchronous using Triggers Asynchronous Redo log files HotLog Distributed HotLog AutoLog Pros Completeness Performance minimal impact on source systems Easy Interface Minimal overhead cost Additional structures needs to be created on Source systems e.g. triggers for synchronous. Source system needs to operate in archivelog mode. Needs additional maintenance and monitoring compared to other methods

8 Direct Access using dblink Pros Simplest data transfer mechanism. There should be some way to identify new records i.e. timestamps Not the fastest way for moving large data across Scalability Shows exponential rise in timings when copying large data set Tests Test performed on moving data in single thread: 0.3 million records - 00:00: million records - 00:00: million records - 00:02: million records - 00:06:35 Linear progression from 0.3 till 4.75 million records, but it becomes exponential from 4.75 to 9.5 million records!!! Running in 5 threads: 9.5 million records - 00:00:50 20 million records - 00:02:46 Running in 10 threads: 80 million records - 00:09:28

9 File Extracts Source systems with batch feeds will not be able to provide a near real time feed. Frequent extracts (hourly) can be taken, but this may be an overhead on source systems. Usually source systems only provide a daily extract. Two methods SQL Loader External Tables One of the slowest methods for data transfer.

10 Oracle Replication using MViews There can be different ways to perform replication. Actually it comes under CDC. We will focus here on replication using Materialized Views only. Near real time Additional structure needs to be created on source database side i.e. MV logs.

11 Comparison of Methods Real Time (RT)/ Non-Real Time (NRT) File Extracts Database Link Data Pump External Tables Data Pump - Network Link Transportable Tablespaces NRT NRT NRT NRT NRT RT CDC Performance impact on source system Significant Significant Significant Significant Minimal Minimal the impact is caused only by additional logging and log transfer Changes in source systems for implementing the method Performance of data extraction and transfer No No No No Yes Yes Not Tested 9.5 million in 6 mins 35 secs 20 million in 7 mins 23 secs 16 million in 9 mins Not Tested Not Tested Reliability of change data identification Timestamp Based Timestamp Based if timestamps not updated, changes are not identified 100% - all the changes are identified via the redo log files Complexity of the method Medium Simple one step only defined by the SQL Select statement Medium Simple Medium Complex CDC config, change tables on different instance, cannot use complex joins