Golden Gate. Zero downtime migrations DOAG 2014

Transcription

1 Golden Gate Zero downtime migrations DOAG 2014

2 Who am I? Klaas-Jan Jongsma Living in the Netherlands Working for VX Comany Working with Oracle since mid-ninties Started with Oracle 7 Working with Exadata since V2 Oracle ACE Associate kjongsma@vxcompany.com

3 Some facts about Golden Gate Not an internal developed Oracle product Acquired by Oracle in overview/statement-of-direction-gg pdf Is set to replace Oracle Streams in the future It is expensive with $ per core, compared to $ for Active Dataguard An Active Dataguard license comes free with ever Golden Gate License Works in hetrogenous environments like Oracle, Enscribe & SQL/MX, MySQL, SQL Server, DB/2 and Terradata

4 How does it work? Manager Manager Redolog Extract Datapump Server Collector Replicat Trail files Trail files Source Target

5 Fallback Trail files Trail files Replicat Server Collector Datapump Extract Extract Server Collector Datapump Replicat Trail files Trail files Source Target

6 Integrated mode New in Golden Gate 11g Database must be at least For capturing DDL, database must be or higher Limited support for TDE, Securefile Lobs and XML columns Integrated with RAC and ASM, no additional setup is needed For and higher additional setup for DDL is not needed anymore Is the only option if run Golden Gate in a 12c CDB

7 How does it work? integrated mode Manager Manager Redolog Logminer Datapump Server Collector Logical Change Records (LCR s) Extract Replicat Trail files Trail files Source Target

8 Integrated mode Database proceses AS SYSDBA> select sid, username, module, event, action from v$session where username like 'OGG%'; SID USERNAME MODULE EVENT ACTION OGG_OWNER_T OGG-IPUMP-GLOPEN_DATA_SOURCE SQL*Net message from client 37 OGG_OWNER_T OGG-IPUMP-IXAsyncTrans SQL*Net message from client 784 OGG_OWNER_T GoldenGate LogMiner client: transaction OGG$CAP_IPUMP - Capture 794 OGG_OWNER_T OGG-IPUMP-REDO_ORA_XO SQL*Net message from client 797 OGG_OWNER_T OGG-IPUMP-REDO_ORA_XO SQL*Net message from client 799 OGG_OWNER_T OGG-IPUMP-OCI_META_THREAD SQL*Net message from client 6 rows selected. SYS@t121 AS SYSDBA> select s.username, s.module, s.event, l.role from v$session s, v$logmnr_process l where s.sid=l.sid; USERNAME MODULE EVENT ROLE SYS GoldenGate LogMiner reader: redo (idle) reader SYS GoldenGate LogMiner builder: idle builder SYS GoldenGate LogMiner preparer: idle preparer SYS GoldenGate LogMiner preparer: idle preparer

9 Managing credentials Create an encrypted password GGSCI (dm02db01.db.gen.local) 1> encrypt password ogg_owner No key specified, using default key... Encrypted password: AACAAAAAAAAAAAJAYALCJAZEYFXGSDNJKCZBGGREUBKBPIBC Use the encrypted password, e.g. pumpfile userid password AACAAAAAAAAAAAJAYALCJAZEYFXGSDNJKCZBGGREUBKBPIBC, encryptkey default

10 Managing credentials Golden Gate 12c gives us a credential store: GGSCI (oel65.local.domain) 1> add credentialstore Credential store created in./dircrd/. GGSCI (oel65.local.domain) 10> alter credentialstore add user ogg_owener_t@t121 alias ogg Password: Credential store in./dircrd/ altered. GGSCI (oel65.local.domain) 3> info credentialstore Reading from./dircrd/: Domain: OracleGoldenGate Alias: ogg Userid: ogg_owener_t@t121

11 Managing credentials Golden Gate 12c gives us a credential store: [oracle@oel65 [t121] dircrd]$ orapki wallet display -wallet./cwallet.sso Oracle PKI Tool : Version Copyright (c) 2004, 2012, Oracle and/or its affiliates. All rights reserved. Requested Certificates: User Certificates: Oracle Secret Store entries: newcredstore@#3#@ogg OracleGoldenGate@#3#@5c1192bd-oggt OracleGoldenGate@#3#@88ed03f0-ogg OracleGoldenGate@#3#@fe30bed3-oggs Trusted Certificates: [oracle@oel65 [t121] dircrd]$

12 Basic steps Determine what schema s to sync Investigate schema for unsupported datatypes, tables without PK or unique indexes Determine wetter or not you need add PK s or unique indexes Get redolog rates Determine duration for initial load Calculate size needed to store trailfiles during load configure manager, extract, datapump and replicat Start manager, extract & datapump Run initial load Start replicat Switch application (initiate fallback)

13 What to sync Determine what you want synchronize (tables, schemas) DDL synchronization on Oracle Reversed Schemas (SYS, SYSTEM, OLAPSYS, OUTLN etc.) is not supported Check if the objects are usable by Golden Gate Not supported datatypes are: orddicom, anydata, anydataset, anytype, bfile, mlslabel, timezone_abbr, timezone_region, uritype, urowid Table compression, compress for OLTP and EHCC is not supported (is supported in integrated mode) Invisible columns (is supported in integrated mode) Distributed transactions (is supported in integrated mode)

14 What to sync Check for unsupported datatypes e.g. SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE FROM all_tab_columns WHERE OWNER = KJJ AND (data_type in ('ORDDICOM', 'BFILE', 'TIMEZONE_REGION', 'BINARY_INTEGER', 'PLS_INTEGER', 'UROWID', 'URITYPE', 'MLSLABEL', 'TIMEZONE_ABBR', 'ANYDATA', 'ANYDATASET', 'ANYTYPE') or data_type like INTERVAL% ); ANYDATA datatype is common, used in AQ queue tables Normally solved at the switchover by stopping the queue

15 How to configure trail files Initial load trail files start extract start replicat

16 How to configure If one big initial load is not possible split it up, for example use a special load en separate extracts/replicats for that one big table. all other tables trail files Initial load trail files start extract start replicat big table extract/replicat trail files trail files start extract Initial load start replicat stop replicat

17 How to configure Initial load howto When using Golden Gate between the same platforms use database native load methods if possible (datapump, backup/restore) (Make sure no DDL is going on when doing the initial load) When going cross platform use Golden Gate to do the initial load Golden Gate extract can create sqlldr files (add formatascii, sqlloader to extract, genloadfiles to replicate) add an initial load extract and replicate e.g.: add extract ext-load, sourceistable add replicat rep-load, specialrun

18 Redolog rates & sizes Frequency of Log Switches by hour and day: select subsrt(to_char(first_time, 'MM-DD-YY HH:MI:SS'),1,5) DAY, TO_CHAR(SUM(DECODE(SUBSTR(TO_CHAR(FIRST_TIME, 'MM-DD-YY HH:MI:SS'),10,2),'00',1,0)),'99') "00", TO_CHAR(SUM(DECODE(SUBSTR(TO_CHAR(FIRST_TIME, 'MM-DD-YY HH:MI:SS'),10,2),'01',1,0)),'99') "01", TO_CHAR(SUM(DECODE(SUBSTR(TO_CHAR(FIRST_TIME, 'MM-DD-YY HH:MI:SS'),10,2),'22',1,0)),'99') "22", TO_CHAR(SUM(DECODE(SUBSTR(TO_CHAR(FIRST_TIME, 'MM-DD-YY HH:MI:SS'),10,2),'23',1,0)),'99') "23" V$LOG_HISTORY from where first_time > sysdate - 30 group by substr(to_char(first_time, 'MM-DD-YY HH:MI:SS ),1,5); Average log size: select round( (sum (BLOCKS) * max(block_size)/ count(*))/1024/1024) "Avg Log Size (MB)" from gv $ARCHIVED_LOG; Trailfiles are about 20% smaler then logfiles Determine duration of initial load Make decision on buffer time during the replicat run

19 How to install? Go to edelivery.oracle.com Select Oracle Fusion Middleware product pack Select the correct OS version Download the version for your database platform(s) Upload it to your server...and unzip it in a directory of your choice. but then there was release 12 and a OUI

20 How to install release 12c? Go to OTN or edelivery.oracle.com Download the correct version for your platform Linux x64 is now 340MB in size Upload it to your server Unzip it in a directory of your choice Go to./fbo_ggs_linux_x64_shiphome/disk1 Start X or VNC Start./runInstaller

21 How to configure Create Golden Gate owner autoextend on next 100M maxsize unlimited extent management local segment space management auto; create user ogg_owner identified by very_complex_password default tablespace ogg_data temporary tablespace temp; grant connect, resource to ogg_owner; grant select any dictionary to ogg_owner; grant select any table to ogg_owner; grant flashback any table to ogg_owner; grant execute on dbms_flashback to ogg_owner; grant execute on utl_file to ogg_owner; grant unlimited tablespace to ogg_owner; grant alter any table to ogg_owner; grant insert any table to ogg_owner; grant delete any table to ogg_owner; grant update any table to ogg_owner; grant create table to ogg_owner; exec dbms_goldengate_auth.grant_admin_privilege('ogg_owner'); alter database add supplemental log data; alter system switch logfile;

22 How to configure Add supplemental logging to tables GGSCI (oel65.local.domain) 1> dblogin userid password ogg_owner Successfully logged into database PDB1. GGSCI (oel65.local.domain) 3> add schematrandata user :46:58 INFO OGG SCHEMATRANDATA has been added on schema user :46:58 INFO OGG SCHEMATRANDATA for scheduling columns has been added on schema user1. GGSCI (oel65.local.domain) 133> add trandata tb303.* :55:44 WARNING OGG No unique key is defined for table ACID. All viable columns will be used to represent the key, but may not guarantee uniqueness. KEYCOLS may be used to define the key. Logging of supplemental redo data enabled for table PDB1.TB303.ACID. TRANDATA for scheduling columns has been added on table 'PDB1.TB303.ACID'. GGSCI (oel65.local.domain) 134>

23 How to configure Why use supplemental logging > desc thanksforthefish Name Null? Type CHARACTER NOT NULL VARCHAR2(50) ALIEN VARCHAR2(1) ROBOT VARCHAR2(1) > select * from thanksforthefish; CHARACTER ALIEN ROBOT FordPrefect n n ArthurDent n n TheGuide y n Trillian n n ZaphodBeeblebrox y n MarvintheParanoidAndroid y y Slartibartfast y n 7 rows selected.

24 How to configure Why use supplemental logging select * from dba_log_groups where table_name='thanksforthefish'; no rows selected KJJ@t121> update thanksforthefish set alien='y' where character='fordprefect'; 1 row updated. KJJ@t121> commit; Commit complete.

25 How to configure Why use supplemental logging alter session set events = '1348 trace name context forever, level 1032'; alter session set events = '1354 trace name context forever, level 32768'; SYS@t121 AS SYSDBA>!oerr ora , 00000, "Supplemental log data must be added to run this command" // *Cause: An attempt was made to perform an operation that required // that supplemental log data be enabled. // *Action: Execute a command such as // ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; // and then reissue the command that failed with this error. SYS@t121 AS SYSDBA>!oerr ora , 00000, "LogMiner testing event" // *Cause: // *Action:

26 How to configure Why use supplemental logging KDO Op code: URP row dependencies Disabled xtype: XA flags: 0x bdba: 0x hdba: 0x itli: 1 ispac: 0 maxfr: 4858 tabn: 0 slot: 5(0x5) flag: 0x2c lock: 0 ckix: 0 ncol: 3 nnew: 1 size: 0 col 2: [ 1] 79 LOGMINER DATA 10i: opcode: UPDATE Number of columns supplementally logged: 0 Flag: SE kdogspare1: 0x1000 [ NSRCI] Objv#: 1 segcol# in Undo starting from 3 segcol# in Redo starting from 3 SYS@t121 AS SYSDBA> select utl_raw.cast_to_varchar2(hextoraw('79')) ASCII from dual; ASCII y

27 How to configure Why use supplemental logging GGSCI (oel65.local.domain) 19> add trandata kjj.* Logging of supplemental redo data enabled for table KJJ.THANKSFORTHEFISH. TRANDATA for scheduling columns has been added on table 'KJJ.THANKSFORTHEFISH'. GGSCI (oel65.local.domain) 20> > select log_group_name, table_name from dba_log_groups where table_name='thanksforthefish'; LOG_GROUP_NAME TABLE_NAME GGS_20324 THANKSFORTHEFISH SYS_C THANKSFORTHEFISH SYS_C THANKSFORTHEFISH SYS_C THANKSFORTHEFISH 4 rows selected. KJJ@t121 > update thanksforthefish set robot='n' where character='marvintheparanoidandroid'; 1 row updated.

28 How to configure Why use supplemental logging? itli: 2 ispac: 0 maxfr: 4858 tabn: 0 slot: 5(0x5) flag: 0x2c lock: 0 ckix: 0 ncol: 3 nnew: 1 size: 0 col 2: [ 2] 79 LOGMINER DATA 10i: opcode: UPDATE Number of columns supplementally logged: 2 Flag: SE kdogspare1: 0x1000 [ NSRCI] Objv#: 1 segcol# in Undo starting from 3 segcol# in Redo starting from 3 Supplemental logging: col 0: segcol#: 1 len [24] 4d e e 6f e f col 2: segcol#: 2 len [ 1] 79 SYS@t121 AS SYSDBA> select utl_raw.cast_to_varchar2(hextoraw( 4d e e64726f6964 )) ASCII from dual; ASCII MarvintheParanoidAndroid SYS@t121 AS SYSDBA>

29 How to configure Create checkpoint table (on target), only if non integrated replicated is being used [oracdb] ~]$ echo "CHECKPOINTTABLE OGG_OWNER_S.CHECKPOINT >>GLOBALS GGSCI (oel65.local.domain) 1> dblogin userid password very_complex_password Successfully logged into database PDB2. GGSCI (oel65.local.domain) 2> add checkpointtable ogg_owner.checkpoint;

30 How to configure Create manager, extract, datapump and replicat processes GGSCI (dm02db01.db.gen.local) 1> edit params mgr GGSCI (dm02db01.db.gen.local) 2> start mgr Manager started. GGSCI (dm02db01.db.gen.local) 3> info all Program Status Group Lag Time Since Chkpt MANAGER RUNNING GGSCI (dm02db01.db.gen.local) 4>

31 How to configure Start the intial load Check for any (long) running transactions, e.g.: SQL> SELECT s.sid, s.serial# FROM v$session s 2 WHERE s.saddr in (SELECT t.ses_addr FROM V$transaction t, 3 dba_rollback_segs r WHERE t.xidusn=r.segment_id); no rows selected Find out what the current SCN is: SQL> select current_scn from v$database; CURRENT_SCN Start the export [oracle@dm02db01 ogg]$ expdp directory=gg_dumpdir \ parallel=4 dumpfile=ggd1832a_%u.dmp flashback_scn= \ schemas=xauser,jmsuser,aia,uddiuser,orawsm,oraesb,orabpel

32 How to configure Start the initial load impdp directory=gg_dumpdir parallel=4 dumpfile=ggd1832a_%u.dmp \ schemas=xauser,jmsuser,aia,uddiuser,orawsm,oraesb,orabpel Disable triggers on target side.: spool disable_triggers.sql select 'alter trigger "' owner '"."' trigger_name '" disable;' from dba_triggers where owner in ('XAUSER','JMSUSER','AIA','UDDIUSER','ORAWSM','ORAESB','ORABPEL') and status = 'ENABLED' order by owner / spool off

33 How to configure Start the replicat process (initial load SCN) GGSCI (dm04db07.db.gen.local) 16> start replicat rd1832a, aftercsn Sending START request to MANAGER... REPLICAT RD1832A starting GGSCI (dm04db07.db.gen.local) 17> info all Program Status Group Lag Time Since Chkpt MANAGER RUNNING REPLICAT RUNNING RD1832A 00:00:00 00:00:07 GGSCI (dm04db07.db.gen.local) 20>

34 Putting it all together port 7809 dynamicportlist purgeoldextracts./dirdat/x1*, minkeepdays 3, usecheckpoints add extract pdb1, exttrailsource./dirdat/x1 Manager add rmttrail./dirdat/x1, extract pdb1, megabytes 500 Manager add extract xdb1, tranlog, begin now, threads 2 add replicat rdb2, exttrail./dirdat/x1 add exttrail./dirdat/x1, extract xdb1, megabytes 500 extract xdb1 setenv (ORACLE_SID=DB1) useridalias oggt tranlogoptions asmuser sys@mys-can:1521/+asm, asmpassword welcome Extract Datapumpdiscardfile./dirrpt/xd1832a.dsc, Server purge Replicat tranlogoptions excludeuser Collector ogg_owner extract pdb1 reportcount every 30 minutes, rate passthru Redolog exttrail./dirdat/x1 rmthost , mgrport 7809 sequence XAUSER.* rmttrail./dirdat/x1 replicat rdb2 sequence JMSUSER.* sequence XAUSER.* setenv (ORACLE_SID=DB2) sequence AIA.* sequence JMSUSER.* userid ogg_owner, password sequence ogg_owner UDDIUSER.* sequence AIA.* discardfile./dirrpt/rd1832a.dsc, sequence ORAWSM.* purge sequence UDDIUSER.* assumetargetdefs sequence ORAESB.* sequence ORAWSM.* reportcount every 30 minutes, sequence rate ORABPEL.* sequence ORAESB.* mapexclude XAUSER.aqtable_example tableexclude XAUSER.aqtable_example sequence ORABPEL.* map XAUSER.*, target table XAUSER.* XAUSER.*; ; tableexclude XAUSER.aqtable_examplemap JMSUSER.* Trail files, target table JMSUSER.* JMSUSER.*; ; Trail files table XAUSER.*; map AIA.*, target table AIA.* AIA.*; ; table JMSUSER.*; map UDDIUSER.*, target Source table UDDIUSER.* UDDIUSER.* ; Target table AIA.*; map ORAWSM.*, target table ORAWSM.* ORAWSM.*; ; table UDDIUSER.*; map ORAESB.*, target table ORAESB.* ORAESB.*; ; table ORAWSM.*; map ORABPEL.*, target table ORABPEL.* ORABPEL.*; ; table ORAESB.*; table ORABPEL.*;

35 Integrated capture mode Additional rights AS SYSDBA> exec dbms_goldengate_auth.grant_admin_privilege('ogg_owner_t'); Create paramter file GGSCI (oel65.local.domain) 61> edit params ipump extract ipump useridalias oggt logallsupcols exttrail /u01/app/oracle/product/12.1.2/oggcore_1/dirdat/ip table kjj.*; Add extract and trail files GGSCI (oel65.local.domain) 22> add extract ipump, integrated tranlog, begin now EXTRACT added. GGSCI (oel65.local.domain) 26> add exttrail /u01/app/oracle/product/12.1.2/ oggcore_1/dirdat/ip, extract ipump EXTTRAIL added.

36 Integrated capture mode Register extract with the database GGSCI (oel65.local.domain) 60> register extract ipump database Extract IPUMP successfully registered with database at SCN Start the extract GGSCI (oel65.local.domain) 63> start extract ipump Sending START request to MANAGER... EXTRACT IPUMP starting Verify GGSCI (oel65.local.domain) 70> info extract ipump EXTRACT IPUMP Last Started :04 Status RUNNING Checkpoint Lag 00:00:05 (updated 00:00:08 ago) Process ID Log Read Checkpoint Oracle Integrated Redo Logs :56:44 SCN (756472)

37 Peeking into the trail files Logdump utility Logdump 2 >open./ip Current LogTrail is /u01/app/oracle/product/12.1.2/oggcore_1/dirdat/ip Logdump 3 >show env Version : Linux, x64, 64bit (optimized) on Sep :02:47 Current Directory : /u01/app/oracle/product/12.1.2/oggcore_1/dirdat LogTrail : /u01/app/oracle/product/12.1.2/oggcore_1/dirdat/ip Trail Format : New End of File : 1480 Current Position : 0 Forward Next Position : 0 Last Modtime : 2014/02/26 02:04: Display RecLen : 140 Logtrail Filter : On Trans History : 0 Transactions, Records 100, Bytes LargeBlock I/O : On, Blocksize Local System : LittleEndian Metadata Byte Order : BigEndian Logtrail Data : BigEndian/ASCII Logtrail Headers : ASCII Dump : ASCII Savefile comments : Off Timeoffset : LOCAL Scan Notify Interval: records, Scrolling On

38 Peeking into the trail files Logdump utility Logdump 5 >n 2014/02/26 02:04: FileHeader Len 1472 RBA 0 Name: *FileHeader* d0a 544c 0a0d C0...GG..TL f dc x= a 6f65 6c36 353a 6c6f!%4..G.Euri:oel65:lo c3a 646f 6d61 696e 3a3a a a cal:domain::u01:app: 6f c65 3a70 726f a31 322e 312e oracle:product: a 6f f72 655f 313a d :oggcore_1:IPUMP6.. 3a00 382f f f 6f c65 2f70 :.8/u01/app/oracle/p Logdump 6 >n 2014/02/26 02:14: Delete Len 34 RBA 1480 Name: KJJ.THANKSFORTHEFISH Before Image: Partition 4 G s c TheGuide e...y...n Logdump 7 >

39 Some caveats Oracle Golden Gate is not Dataguard (or streams... yet) Golden Gate will not reduce the amount of work Know your application, know your schema Be prepared to customize - A default Golden Gate configuration does not exists Think about naming conventions Take your time to learn Golden Gate, it is very modular

40

41

42 I knew i needed this slide again! Manager Manager Redolog Extract Datapump Server Collector Replicat Trail files Trail files Source Target

43