Hubble Technical Mirroring Overview

Transcription

1 Hubble Technical Mirroring Overview Source Database Platforms Mirroring data is supported on the following platforms that host the ERP Database, Library or Schema: Oracle 10g and 11g, 12c* SQL Server 2005, 2008, 2012 and 2014 DB2 for IBM iseries * If the Oracle 12c database is multitenant, we can only replicate from the PDBs and not the CDB. Database-specific Preparation Data Source: Oracle Prerequisites The mirroring is log-based, which means that only changes are read. This reduces the impact on the source database. The mirrored data is kept synchronized with the ERP data using a Change Data Capture (CDC) process. The CDC process captures changes from the online and archive redo logs using one of the following: LogMiner Oracle Binary Reader API (BFILE) Access to logs via directory objects Direct access to logs* *In order for the CDC process to read the online and archive redo logs by directly accessing the logs, we will need to set up a file share using the network file system protocol (NFS) between the Hubble Accelerator Server and the folder location on the server that contains the online and archive logs. This way the logs can be accessed directly from the Hubble Accelerator using the share. The Oracle BFILE data type does not support accessing binary files larger than 4 GB. Therefore, when the redo logs are larger than 4 GB and are not stored on ASM or directly accessible, use LogMiner to capture the changes. The following are the prerequisites to enable data mirroring from the source Oracle database: Oracle ARCHIVELOG mode must be enabled. Supplemental logging: both the database and the individual tables must be enabled for supplemental logging (see page 2). If mirroring from a standby database, the CDC process can only read from a physical standby configuration and not a logical standby database. Oracle account access must have certain privileges granted. (A script can be provided if requested.) Page 1

2 Supported Oracle Compression Methods Oracle Version OLTP Basic Others Oracle 10 N/A No No Oracle 11 and above - BFile Yes Yes No Oracle 11 and above - Logminer Yes Yes Yes * * Any compression method supported by Logminer Supplemental Logging Configuration Both the database and the individual tables must be enabled for logging. Supplemental Logging Set at the Database Level To check that supplemental logging is enabled for the database: 1. Run the following query: SELECT name, value, description FROM v$parameter WHERE name = 'compatible'; The returned result should be from GE to Run the following query: SELECT supplemental_log_data_min FROM v$database; The returned result should be YES or IMPLICIT. To enable supplemental logging, execute the following query: ALTER DATABASE ADD SUPPLEMENTAL LOG DATA If your Oracle database instance is on Amazon RDS, a different command needs to be executed. Supplemental Logging Set at the Table Level You can use either of the below two methods to add supplemental logging for individual tables: 1. Use Attunity to add the supplemental logging. (The default behavior is to have Attunity automatically configure supplemental logging on the tables marked for mirroring.) To do this, the Hubble connection user created in the script must be given permission to automatically add the required level of supplemental logging. This is done using the following: GRANT ALTER ON <SCHEMA>.<TABLE> TO <DB_USER_NAME>; 2. Add the supplemental logging as part of the script. Supplemental logging will be added for each mirrored table. The script alters the tables as follows: ALTER TABLE <SCHEMA>.<TABLE> ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; Page 2

3 Access to Online and Archive Redo Logs Attunity supports two methods for accessing the online and archive redo logs: 1. Using BFILE. Attunity uses directory objects to access the online and archive logs. The default behavior is to have Attunity create and manage the directory objects. If the user does not have CREATE DIRECTORY privileges, grant the following privilege: CREATE ANY DIRECTORY Optionally, if you want to manually create and manage the directory objects that point to the redo and archive log locations, then the following permission must be granted after creating the directory objects: GRANT READ on ANY DIRECTORY The archive directory object will need to be updated if Oracle changes the location, for example if each day has a separate directory. 2. Using LogMiner. If using LogMiner, grant the following privileges: EXECUTE on DBMS_LOGMNR SELECT on V_$LOGMNR_LOGS SELECT on V_$LOGMNR_CONTENTS If you use ASM (Automatic Storage Management), you must use option 1, BFILE, to read the logs. Using BFILE, Attunity can connect to the ASM server in order to read the logs. Additional Database Access Required The additional database access required is as follows: Reading the source data for the initial load SELECT on <any-replicated-table> Lookup log sequence and thread info SELECT on V_$ARCHIVED_LOG SELECT on V_$LOG SELECT on V_$LOGFILE SELECT on V_$THREAD Check for logging level, time zone and character set SELECT on V_$DATABASE SELECT on V_$PARAMETER SELECT on V_$NLS_PARAMETERS SELECT on V_$TIMEZONE_NAMES SELECT on ALL_LOG_GROUPS SELECT on SYS.DBA_REGISTRY Page 3

4 Lookup active transactions when starting CDC SELECT ANY TRANSACTION SELECT on V_$TRANSACTION Lookup metadata for source tables SELECT on ALL_INDEXES SELECT on ALL_OBJECTS SELECT on DBA_OBJECTS SELECT on ALL_TABLES SELECT on ALL_USERS SELECT on ALL_CATALOG SELECT on ALL_CONSTRAINTS SELECT on ALL_CONS_COLUMNS SELECT on ALL_TAB_COLS SELECT on ALL_IND_COLUMNS Check Partition information SELECT on ALL_TAB_PARTITIONS Check details on the tables for capturing SELECT on SYS.OBJ$ Checking for Encryption SELECT on DBA_TABLESPACES SELECT on ALL_ENCRYPTED_COLUMNS The following permission maybe required to be granted for logon: CREATE SESSION If the database is an Oracle standby database, grant the following: SELECT on V_$STANDBY_LOG If reading logs stored in ASM, grant the following: SELECT on V$TRANSPORTABLE_PLATFORM Non-supported Oracle Data Types Columns with following data types cannot be replicated: BFILE ROWID REF RROWID Nested Table User-defined data types Replicated columns with this data types will show as null: XMLTYPE Page 4

5 Required ASM Privileges From Oracle 11g Release 2 ( ), the replication user account must be granted the SYSASM privilege in order to access the ASM account. For older supported versions, granting the Replication user account the SYSDBA privilege should be sufficient. When connecting to ASM, the mirroring software will first try to log in as SYSDBA and, if unsuccessful, will try to log in as SYSASM. Sample Scripts Sample scripts are available by request. Data Source: SQL Server Supported Versions of SQL Server using MS-Replication Versions 2005, 2008, 2012 and 2014 in the following editions: Enterprise Edition Standard Edition Workgroup Edition Developer Edition Supported Windows Platforms for a SQL Server Database The supported Windows platforms are the following: Windows Server 2003, 2008 and 2012 Windows XP Windows Vista Windows 7 The mirroring is log-based and consumes changes captured from the database transaction LOG (TLOG). The TLOG is maintained by SQL Server for recovery purposes. All changes made to a database are written to the TLOG. The mirroring task automatically sets up MS-REPLICATION for each of the source tables configured in a mirroring task. For optimal performance, the mirroring process will try to capture all unread changes from the active transaction log (TLOG). However, sometimes due to truncation, the active TLOG may not contain all of the unread changes. When this occurs, the mirroring process accesses the backup log to capture the missing changes. To minimize the need to access the backup log, the mirroring process prevents TLOG truncation by mimicking a transaction in the database. As long as such a transaction is open, changes that appear after the transaction started will not be truncated. The mirroring vendor recommends that while you are performing any database maintenance tasks (Index Rebuilds, index ReOrganize, etc) that due to the increased activity on the TLog at this time, that you stop the replications tasks and then reload/resume the tasks after the maintenance task has completed. If you do not stop and restart the replication tasks around your maintenance tasks, you will see the size of your TLOG increase while the replication processes the increased volume of changes in the TLOG. This can eventually lead to the database server running out of space and completely stopping the database running. Page 5

6 The prerequisites to enable data mirroring from the source SQL Server database are as follows: MS-REPLICATION for the SQL Server database must be enabled Recovery model must be bulk-logged or full mode A full database backup must exist The database account used for mirroring must have the sysadmin fixed server role granted Limitations Limitations are as follows: If you are using a SQL Server source database in a mirroring task, the SQL Server Replication Publisher definitions for the database that was used in the task are not removed when you remove a task. A SQL Server system administrator must delete these definitions from SQL Server. Non-supported SQL Tables with more than 8000 bytes of information in a SQL Server source, including header and mapping information, are not processed correctly due to limitations in buffer size. The SQL Server source endpoint does not support the use of sparse tables. Changes to computed fields in a SQL Server source will not be mirrored. SQL Server partition switching is not supported. When using the WRITETEXT and UPDATETEXT utilities, mirroring does not capture events applied on the source database. The following DML pattern is not supported: select <*> into <new_table> from <existing_table> Column-level encryption is not supported. Microsoft SQL Server backup to multiple disks is not supported. Backup from non-microsoft SQL Server programs is not supported, e.g. 3rd party server backup programs. Due to a known issue with the Microsoft SQL Server ODBC driver on Linux, source columns with a non-latin code page may be truncated on the target. Due to a known issue with Microsoft SQL Server 2008, mirroring does not support server level audits on Microsoft SQL Server Change Data Capture operations are not supported on memory optimized tables. This limitation applies to Microsoft SQL Server 2014 (when the feature was first introduced) and above. When inserting a value into SQL Server spatial data types (GEOGRAPHY and GEOMETRY), one can either ignore the SRID (Spatial Reference System Identifier) property - in which case the default SRID will be used (0 for GEOMETRY and 4326 for GEOGRAPHY) - or specify a different number. When replicating tables with spatial data types, the mirroring process replaces the SRID that was inserted by user with the default SRID. Backup files that are not in Microsoft SQL Server format cannot be read by the mirroring software. The Mirroring software cannot connect directly to a Microsoft SQL Server backup file. The mirroring software cannot replicate from a Microsoft SQL Server contained database. Page 6

7 Supported Microsoft SQL Server Compression Methods Microsoft SQL Server Version Row/Page Compression (at Partition Level) Vardecimal Storage Format Vardecimal Storage Format Sparse Columns Vardecimal Storage Format Sparse Columns Columnar Structure Compression 2005 No No No No 2008 Yes No No No 2012 Yes No No No Microsoft SQL Server AlwaysOn Availability Groups The Microsoft SQL Server AlwaysOn Availability Groups feature is a high-availability and disaster-recovery solution that provides an enterprise-level alternative to database mirroring. To use AlwaysOn Availability Groups as a source: Notes: 1. Enable Distribution on all Microsoft SQL Server instances in your Availability Replicas. 2. The Server Name and IP address provided for replication should be the details configured for the Availability Group Listener. When the mirroring task is run for the first time, it may take longer than usual to start as the creation of the table articles is being duplicated by the Availability Groups Server. After you failover, and the replication task will restart. This is normal as the connection is being forcibly closed by the failover. As soon as the new primary server is available, processing will resume. Non-supported SQL Server Data Types Columns with the following data types cannot be replicated: CURSOR SQL_VARIANT TABLE Setting up Microsoft SQL Server for Replication By default, the mirroring application uses MS-REPLICATION. To enable your Microsoft SQL Server database for MSREPLICATION, go to Microsoft SQL Server Management Studio. Follow the instructions provided by the Configure Distribution wizard to set up replication or see the Microsoft SQL Server documentation. To open the wizard from Microsoft SQL Server: 1. In the Microsoft SQL Server Management Studio, right-click on the Replication folder. 2. Select Configure Distribution to open the Configure Distribution Wizard. 3. Make the following selections in the wizard: a. In the Distributor step, select <Microsoft SQL Server Name> will act as its own distributor. b. Microsoft SQL Server will create a distribution database and log. Page 7

8 Mirroring Tables that do not have a Primary Key MS-CDC functionality is supported only for SQL Server Enterprise edition and is not supported on SQL Server By default, the mirroring application automatically sets up MS-REPLICATION for each of the source tables configured for mirroring. However, MS-REPLICATION requires each of the source tables to have a primary key, which may not always be the case. Therefore, if you need to replicate tables that do not have a primary key, you can configure the MS SQL Server database to use MS-CDC. Setting up MS-CDC To set up MS-CDC, you first need to enable MS-CDC for the database by running the following command: use [DBname] EXEC sys.sp_cdc_enable_db Then you need to enable MS-CDC for each of the source tables by running the following command: EXECUTE = = = NULL; Data Source: IBM DB2 AIS Server Data is mirrored from DB2 via an AIS server; this is additional software that needs to be installed on the server that hosts the DB2 data. This creates subsystem jobs for Attunity on the AS400. Please reference the Attunity AIS Installation Guide for AS400 for steps to install the AIS Server. Attunity Studio, a GUI tool for managing the life cycle activities of AIS installations, must also be installed. See the Attunity Studio Installation Guide for Attunity Studio requirements and installation details. AS/400 Hardware and Software Requirements The requirements are the following: Machine: Any IBM AS/400 machine Disk space: 60 MB Tables required for mirroring must be journaled and the journal image set to *BOTH. Installation Requirements: User with QSECOFR Account Ensure it has special authorities: *JOBCTL *ALLOBJ *SERVICE *SPLCTL Page 8

9 Runtime: QSECOFR can be revoked Special authorities have to stay (*JOBCTL, *ALLOBJ, *SERVICE, *SPLCTL). *ALLOBJ, might be removed if all access to necessary journal, journal receivers and tables is granted Possible updates/patches: QSECOFR will be required during the update. Setup Tasks Pre-installation Tasks The pre-installation tasks include the following: Access to AS/400 system with QSECOFR account. For the provided files, create SAVF files on the AS/400 and FTP in binary mode to QGPL library. Gather required information required for each table: File File Library Journal Journal Library Create a spreadsheet with the relationship between individual library, table and journal such as in this example: Installation Tasks The installation tasks include: Create a library where all components will be placed. Install Subsystem. Test installation. Page 9

10 Hubble Technical Mirroring Overview Notices Create PGM to auto-start mirroring components. Step by step installation can be provided on demand. Post Installation Tasks The post installation tasks include: Revoke QSECOFR rights from user and make sure it has the following special authorities: *JOBCTL *ALLOBJ (Might be removed if all access to necessary journal, journal receivers and tables is granted) *SERVICE *SPLCTL To ensure the Mirroring system will be started after the system goes offline for backup, etc., the DBA would have to add subsystem to the startup process. Limitations On large tables, issuing a DELETE FROM table statement without a WHERE clause can cause the database to optimize it as a clear operation. A clear operation is not written to the journal as a DELETE. This causes the agent to ignore the operation, so the target results will not be deleted. Notices Copyright Hubble is a brand name of the insightsoftware.com Group. insightsoftware.com is a registered trademark of insightsoftware.com Limited. Hubble is a registered trademark of insightsoftware.com International Unlimited. Other product and company names mentioned herein may be the trademarks of their respective owners. The insightsoftware.com Group is the owner or licensee of all intellectual property rights in this document, which are protected by copyright laws around the world. All such rights are reserved. The information contained in this document represents the current view of insightsoftware.com on the issues discussed as of the date of publication. This document is for informational purposes only. insightsoftware.com makes no representation, guarantee or warranty, expressed or implied, that the content of this document is accurate, complete or up to date. Disclaimer This guide is designed to help you to use the Hubble applications effectively and efficiently. All data shown in graphics are provided as examples only. The example companies and calculations herein are fictitious. No association with any real company or organization is intended or should be inferred. Page 10

11 Version History Hubble Technical Mirroring Overview Version History Date Revision Software Version Comments 5th Jan and above 23rd Jan and above Updated based on Attunity version th Feb and above Added information on Oracle BFILE data type and Microsoft SQL Server for Replication 24th Nov and above Comments incorporated. Customer Support For more information regarding our products, please contact us at For product support including Training, Documentation and Customer Support, visit our Hubble Community at Page 11