Replication to SAP HANA Database with Oracle GoldenGate 12c

Transcription

1 Replica on to SAP HANA Database with Oracle GoldenGate 12c Replication to SAP HANA Database with Oracle GoldenGate 12c Partner with Bristlecone to op ize your supply chain investment and implement quick -to-value solu ons. To set up a discussion with our team, please send your inquiry to sales@bcone.com San Jose 10 Almaden Blvd, Suite 600 San Jose, CA T F Bristlecone Ltd. Bristlecone Managed Services is part of Bristlecone s services. Bristlecone believes the informa on in this publica on is accurate as of its publica on date. The informa on is subject to change without no ce. Bristlecone acknowledges the proprietary rights of the trademarks and product names of all companies men oned in this document.

2 Table of Contents Abstract... 4 Introduction... 4 Prerequisites... 4 Additional OGG Considerations... 5 Overview... 5 Solution... 6 OGG Application Adapter... 6 Core Product Specific... 7 Custom Adapter (custom application s Java API)... 7 Architecture... 7 Implementation Process... 8 Custom Adapter code... 9 Custom Adapter compilation... 9 Replication from Oracle to SAP HANA Database... 9 Setup of OGG on Oracle DB (Source System)... 9 Source Setup... 9 Create OGG user (oragg)... 9 Grant oragg user permissions Enable Supplemental logging at the schema level Create a credential store and Add user credentials Creating the Wallet and Adding a Master Key Create CMDSEC file in the OGG Install Directory MGR Parameter File Extract Parameter File Extract Pump Parameter File Generate Source Definitions Creating OGG Processes and Trail Files on Source system Setup of OGG Application Adapter on SAP HANA DB (Target System) Installation of OGG Java Adapter Define Java Virtual Machine Library Location P a g e

3 Copy the Wallet from the Source System to the Target Adapter Properties File Troubleshooting Tips Summary Where to Go for More Information P a g e

4 Abstract The purpose of this paper is to detail step by step instructions to configure GoldenGate Data Replication engine with SAP HANA Database as target using generic Application Adapter Interface. Whitepaper will help Application developers and System Integrators (Sis) establish connectivity with SAP HANA Database with ease, following step by step procedure as detailed in white paper, reducing integration complexity and time spent, building such an integration otherwise. Introduction Oracle GoldenGate (OGG) includes an optional set of adapters that adds functionality to replicate transactional data to targets other than conventional Relational databases, e.g. No-SQL Databases, In-Memory databases, through generic JDBC interface. This white paper presents basic configuration options and discusses best practices to configure Oracle GoldenGate 12c ( ) with SAP HANA In-Memory databse. It would be done through custom Java Adapter. Similar Java Adapter has been provided by Oracle for Big Data, No-SQL targets like Hive, HBase, and HDFS. The source database is used in this white paper is Oracle RDBMS version 11g Enterprise Edition Release bit. Installation and configuration of SAP HANA database, Oracle RDBMS and OGG database capture is outside the scope of this white paper. However, to understand the concepts presented readers should have an understanding of these products and their use. Also how to write the custom adapter in Java is outside scope of this white paper however the Java code for custom HANA adapter would made available for user s reference through this link. Prerequisites If you plan to execute the instructions in this white paper, make sure both the source and target database (Oracle and SAP HANA) is already installed. For the midtier, the full database binaries must be installed as well as the OGG software. The reader should be familiar with basic OGG architecture and functionality. For Oracle RDBMS and above, the init.ora parameter, enable_goldengate_replication must be set to TRUE in both the source and target databases. User must be having primary development knowledge of Java programming language. As it would be require to write the custom adapter for HANA. JVM version requires [jdk1.6.0_24] The following table describes items that are referred to throughout the document. You will need to identify your installation-specific values and substitute them as you go. Item Reference Description Unix Programs /ggs Directory of Unix GoldenGate installation. Unix Parameter /ggs/dirprm Directory for GoldenGate parameter files. Files Unix Report Files /ggs/dirrpt Directory for output from GoldenGate programs. Unix Definitions Files GGS temporary storage /ggs/dirdef /ggs/dirdat Directory for generated Oracle DDL and definition files. Directory to hold temporary Extract trails 4 P a g e

5 Oracle Logon userid, password User ID and password for the source or target database. When implementing Integrated Extract or Replicate, this user must be granted admin privileges with the DBMS_GOLDENGATE_AUTH procedure on the both source and target databases. Additional OGG Considerations Password Security Process Management Monitoring Overview Passwords used by GoldenGate for Database access encrypted using AES GoldenGate can be configured to auto start / restart processes upon any failure including network failure All GoldenGate deployments can be monitored by Oracle Enterprise Manager (EM) with optional Monitor Agent which are not covered in this document This document will address how to replicate data from Oracle to SAP HANA Database using Oracle GoldenGate Application Adapter. The Oracle GoldenGate Application Adapters extend the capabilities of Oracle GoldenGate to provide mechanisms for real-time integration with SAP HANA Database through a custom Java Plug-in for GoldenGate Application Adapter. The custom Java plug-in should be compiled/customized by user as per specific needs of customer. Plug-in code provides flexibility to users to write Java code on each operation event based on specific integration/replication needs. SAP HANA supports JDBC interface to perform DML operations on the objects created in HANA database. The custom Java adapter is leveraging JDBC interface to push the data into HANA database. For the purpose of illustration in this whitepaper, we have used Oracle RDBMS as source. GoldenGate replication engine generates Trail Files on fly as soon as changes made to objects in Oracle databases, Application Adapter read these Trial Files and write data to HANA target objects using custom Java Plugin. Scenario in this document covers only real-time replication use case, Batch processing for existing data in Oracle database would need a different approach. Oracle GoldenGate Application Adapters can be used to replicate change data to targets other than a relational database: for example, ETL, JMS messaging, or custom Java APIs. There are a variety of options for integration with Oracle GoldenGate using these Adapters: 5 P a g e

6 Once Extract and Replicate are running, changes are replicated perpetually. Notes on Command Syntax: Commands throughout the document make specific references to directories, file names, checkpoint group names, begin times, etc. Unless otherwise noted, these items do not have to correspond exactly in your environment; they are used to illustrated concrete examples. For exact syntax, consult the Oracle GoldenGate Reference Guide. Solution As mentioned earlier Oracle GoldenGate (OGG) includes an optional set of adapters that adds functionality to apply transactional data to targets other than relational databases. The implementation details of OGG Application Adapter is as follows: OGG Application Adapter OGG Application Adapter (aka OGG Java Adapter) provides the ability to execute code written in Java from the OGG Extract program. OGG for Java consists of two components: A dynamically linked or shared library, implemented in C/C++, integrating as a user exit (UE) with the OGG Extract process through a C API. A set of Java libraries (jars), which comprise the OGG Java API. This Java framework communicates with the user exit through the Java Native Interface (JNI). Through the OGG Java API, transactional data captured by OGG can be delivered to targets such as JDBC, Java Message Service (JMS), disk files, or a custom application s Java API. 6 P a g e

7 Core Product Specific The OGG Application Adapter is core product specific; meaning the version 11.2 adapter is designed to function only with the OGG 11.2 core product. This is due to changes in the OGG Trail format and other core product improvements which prevents the UE (User Exits) from being linked with other versions of the Extract program. Custom Adapter (custom application s Java API) Custom Adapter (aka Custom event handlers, Custom java handlers) can be written in Java to process transaction, operation and metadata changes captured by Oracle GoldenGate on the source system. These custom Java handlers can apply these changes to a third-party Java API exposed by the target system. In our case, SAP HANA has exposed JDBC interface, which we will use to apply OGG changes to SAP HANA database. The custom adapter extends the Abstract class provided by OGG Application Adapter, which has all the event handlers. For example, Begin Transaction, Commit Transaction, Insert Operation, Update Operation and Delete Operation etc. These events are captured inside the custom code and appropriate operation is performed on objects in HANA database using JDBC interface. The OGG Application Adapter will read source trail files and call the appropriate events in custom adapter. These event handlers then format the DML data into SAP HANA specific events. In this custom adapter we have been accumulating the whole transaction in memory and on the event of Commit Transaction, we are flushing out the complete transaction into SAP HANA database through JDBC interface. The Custom Adapter was compiled using MAVEN tool. Please see detailed information in following section. Architecture Referring to the architecture diagram below, custom handler integration with SAP HANA Database is developed using Oracle GoldenGate's Java API and SAP HANA JDBC API. The custom adapter is deployed as an integral part of the Oracle GoldenGate Pump process (OGG Application Adapter). The Pump and the custom adapter are configured through Pump parameter file and custom adapter's properties file. We will discuss the adapter properties for integrating with SAP HANA in the implementation section below. 7 P a g e

8 The Pump process executes the custom adapter in its address space. The Pump reads the trail file created by the Oracle GoldenGate Extract process (Source System) and passes the transactions to the custom adapter. Based on the configuration, the custom adapter will write the transactions to SAP HANA Database. Please refer to the Oracle GoldenGate Developing Custom Java Adapter Handbook for details about the architecture and developing a custom adapter. Please see section where to go for More Information. Implementation Process The process for implementing OGG delivery to SAP HANA includes: 1. Download and install the OGG Java Adapter package 2. Modify the existing Primary Change Data Capture (Extract) to get before images for updates 3. Generate definitions for the source data 4. Setup an Extract Data Pump to move data from the database server to the target implementation server 5. Configure the SAP HANA Database with target tables. 8 P a g e

9 6. Configure the target integration server OGG Manager 7. Configure the OGG Java Adapter properties 8. Create the OGG Java Extract Data Pump (Java Data Pump) Custom Adapter code The reference code for Custom Adapter can be downloaded from Bristlecone s E-Delivery site following the link. Custom Adapter compilation The compilation of custom adapter was done through MAVEN tool. The Oracle GoldenGate Application Adapter build comes with out of the box examples for HDFS, Hive, HBase & Flume. The examples are available in following path../$installation_dir/adapterexamples/bigdata/hdfs./$installation_dir/adapterexamples/bigdata/hive./$installation_dir/adapterexamples/bigdata/hbase This path also contains PDF document for how to compile the custom adapter for Big Data platforms. We have used these files and documents as reference to build our Custom Adapter for HANA. We referred section 2.2 Building with Maven. We also changed pom.xml file to remove the dependencies of Big Data platforms. Replication from Oracle to SAP HANA Database Setup of OGG on Oracle DB (Source System) Source Setup See 12c Installation Guide 1. Install OGG Software 2. Set Library paths, Oracle Environmental Variables 3. Create subdirectories for OGG GGSCI> create subdirs 4. Enable minimal supplemental logging in the database SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; 5. Increase UNDO to allowed maximum (recommend 24 hours) 6. Turn on FORCE LOGGING in the tablespaces where tables are created with NOLOGGING Create OGG user (oragg) See 12c Installation Guide SQL> create user oragg identified by xxxxxx; 9 P a g e

10 SQL> alter user oragg default tablespace <OGG Tablespace Name> temporary tablespace TEMP quota unlimited on <OGG Tablespace Name>; Grant oragg user permissions See 12c Installation Guide SQL> grant resource, dba to oragg; SQL> exec dbms_goldengate_auth.grant_admin_privilege('oragg' ); Enable Supplemental logging at the schema level See 12c Installation Guide Source System GGSCI> DBLOGIN USERID oragg, PASSWORD xxxx GGSCI> ADD SCHEMATRANDATA APPS Create a credential store and Add user credentials See 12c Installation Guide 1. (Optional) To store the credential store in a location other than the dircrd subdirectory of the Oracle GoldenGate installation directory, specify the desired location with the CREDENTIALSTORELOCATION parameter in the GLOBALS file. 2. Create the credential store. GGSCI> ADD CREDENTIALSTORE 3. Add each set of credentials to the credential store. GGSCI> ALTER CREDENTIALSTORE ADD USER oragg, PASSWORD xxxx ALIAS oggalias Creating the Wallet and Adding a Master Key See 12c Installation Guide 1. (Optional) To store the wallet in a location other than the dirwlt subdirectory of the Oracle GoldenGate installation directory, specify the desired location with the WALLETLOCATION parameter in the GLOBALS file. WALLETLOCATION directory_path 2. Create a master-key wallet GGSCI> CREATE WALLET 3. Add a master key to the wallet with the GGSCI> ADD MASTERKEY 4. Confirm that the key you added is the current version. In a new installation, the version should be 1. GGSCI> INFO MASTERKEY 5. Copy the wallet to all of the other Oracle GoldenGate systems once they are setup 10 P a g e

11 Create CMDSEC file in the OGG Install Directory CMDSEC blocks all commands to all Operating System users except the OGG User. Lock access to this file (chmod go-r CMDSEC). --Command Object Group User Access Allowed? * * oinstall oragg YES * * * * NO MGR Parameter File PORT 7801 DYNAMICPORTLIST PURGEOLDEXTRACTS./et*, USECHECKPOINTS, MINKEEPHOURS 72 AUTORESTART ER *, RETRIES 3, WAITMINUTES 10, RESETMINUTES 60 DOWNREPORTMINUTES 15 DOWNCRITICAL LAGCRITICALSECONDS 10 LAGINFOMINUTES 0 LAGREPORTMINUTES 15 Extract Parameter File The primary Extract which reads data from the Oracle database transaction logs must be set to capture update before images. This is accomplished by adding the option GETUPDATEBEFORES to the parameter file. Stop and restart the Extract to activate the configuration change. EXTRACT EXTSRC SETENV (ORACLE_SID = <oracle_sid> ) SETENV (NLS_LANG = AMERICAN_AMERICA.AL32UTF8 ) --- Use Alias to login USERIDALIAS oggalias DISCARDFILE./dirrpt/extsrc.dsc, APPEND DISCARDROLLOVER AT 01:00 ON SUNDAY --- Use Wallet to encrypt local trail ENCRYPTTRAIL AES128 EXTTRAIL./dirdat/et STATOPTIONS REPORTFETCH REPORTCOUNT every 10 minutes, RATE REPORTROLLOVER AT 01:00 ON SUNDAY --- DDL Parameters DDL INCLUDE MAPPED DDLOPTIONS REPORT --- Before image params GETUPDATEBEFORES NOCOMPRESSUPDATES TABLE APPS.* ; Extract Pump Parameter File EXTRACT PEXTSRC 11 P a g e

12 RMTHOST <Mid-tier ip address>, MGRPORT 7801 PASSTHRU RMTTRAIL /ogg/gg_adp_home/dirdat/mt TABLE APPS.* ; Generate Source Definitions The OGG DEFGEN utility reads the data structure of all defined source database tables and creates a representation of the structure in a text file. This text file is used to identify and parse the data within the Extract Trail. Run DEFGEN from a Linux shell and transfer the output text file to the OGG dirdef directory on the target integration server. DEFSFILE./dirdef/source.def, purge SETENV (ORACLE_SID = <oracle_sid> ) SETENV (NLS_LANG = AMERICAN_AMERICA.AL32UTF8 ) --- Use Alias to login USERIDALIAS oggalias TABLE APPS.* ; Creating OGG Processes and Trail Files on Source system GGSCI> DBLOGIN USERID oragg, PASSWORD xxxx GGSCI> REGISTER EXTRACT EXTSRC, DATABASE GGSCI> ADD EXTRACT EXTSRC, INTEGRATED TRANLOG, BEGIN NOW GGSCI> ADD EXTTRAIL./dirdat/et, EXTRACT EXTSRC, MEGABYTES 500 GGSCI> ADD EXTRACT PEXTSRC, EXTTRAILSOURCE./dirdat/et GGSCI> ADD RMTTRAIL /ogg/gg_adp_home/dirdat/mt, EXTRACT PEXTSRC, MEGABYTES 500 GGSCI> SH./defgen paramfile./dirdef/defgen.prm Setup of OGG Application Adapter on SAP HANA DB (Target System) On the target integration server, configure and create the OGG Manager and Java Data Pump. A Java Data Pump functions the same as a normal Extract Data Pump except it uses the special Java User Exit shared object. The Java User Exit contains instructions for converting OGG Trail data for submission to the SAP HANA JDBC interface. Installation of OGG Java Adapter Since SAP HANA database is on AWS cloud system, we can install OGG Java adapter on any of machine, and either it can be installed on source system or on to a completely different system. We chose to install it on same source system. The adapter package consists of two parts, the OGG Java Adapter and Oracle GoldenGate Generic Adapter. OGG Java Adapter is comprised of the user exit, java libraries, and sample configuration files. Oracle GoldenGate Generic Adapter bare application shell with no built-in support for target database/application. To install the package: 1. On the source system you must be having OGG for Oracle setup 2. Create a separate directory on the source system other than OGG for Oracle to install OGG Java Adapter. We don t want to mix these two different OGG systems. One is meant for Source and other is for Target 3. Unzip OGG Generic into this newly created location 12 P a g e

13 4. Unzip OGG Java Adapter into this location 5. From a Linux shell, start the GGSCI utility and execute the command CREATE SUBDIRS Define Java Virtual Machine Library Location The OGG Java Adapter needs a system environment variable that defines the Java Virtual Machine (JVM) Library. This environment variable is Operating System specific: Windows: set PATH to find jvm.dll Linux/Solaris/etc: set LD_LIBRARY_PATH to find libjvm.so, AIX, z/os: set LIBPATH HP-UX: set SHLIB_PATH The test environment is running Oracle Enterprise Linux 64-bit; therefore, the environment variable setting is: LD_LIBRARY_PATH=/usr/lib/jvm/java openjdk x86_64/jre/lib/amd64/server:/u01/app/oracle/product/11.2.0/dbhome_1/lib:/u01/app/oracle/M iddleware/wlserver_12.1/server/lib If there are multiple entries to the library path environment variable, like shown above, the JVM Library must be specified first. Copy the Wallet from the Source System to the Target See 12c Installation Guide Create CMDSEC file in the OGG Install Directory CMDSEC blocks all commands to all Operating System users except the OGG User. Lock access to this file (chmod go-r CMDSEC). --Command Object Group User Access Allowed? * * oinstall oragg YES * * * * NO MGR Parameter File As with any OGG implementation, a Manager process must be configured and running. A sample Manager Parameter file is shown below. Best Practice: Use the Manager parameter DYNAMICPORTLIST to define valid OGG communication ports. Best Practice: Use the Manager parameter PURGEOLDEXTRACTS to define OGG Trail retention policy. PORT 7809 DYNAMICPORTLIST PURGEOLDEXTRACTS./et*, USECHECKPOINTS, MINKEEPHOURS 72 AUTORESTART ER *, RETRIES 3, WAITMINUTES 10, RESETMINUTES 60 DOWNREPORTMINUTES P a g e

14 DOWNCRITICAL LAGCRITICALSECONDS 10 LAGINFOMINUTES 0 LAGREPORTMINUTES 15 If a firewall exists between the Source Database Server and the Target Integration Server, the OGG ports must be open for two-way communication. Java Data Pump Parameter File Create the parameter file for the Java Data Pump, and set it to read from the OGG Trail produced by the Extract Data Pump on the source database server EXTRACT PHANA SETENV ( GGS_USEREXIT_CONF = "dirprm/phana.props") CUSEREXIT libggjava_ue.so CUSEREXIT PASSTHRU INCLUDEUPDATEBEFORES GETUPDATEBEFORES NOCOMPRESSUPDATES SOURCEDEFS./dirdef/source.def TABLE APPS.*; When the PHANA Java Data Pump is started, the Java User Exit will check for a file named phana.properties in the OGGdirprm directory. This {group}.properties file must exist before the Java Data Pump can be started. Adapter Properties File The {group}.properties file contains information required by the SAP HANA Handler portion of the Java User Exit to: Establish communication with the SAP HANA through JDBC interface. Format the data for delivery. Using our example for the PHANA Java Data Pump, we create a text file phana.properties in the OGGdirprm directory that contains the runtime options shown below. gg.handlerlist=hjdbc gg.handler.hjdbc.type=com.goldengate.delivery.handler.jdbc.jdbchandler gg.handler.hjdbc.connectionurl=jdbc:sap://xx.xx.xx.177:30015 gg.handler.hjdbc.schema=hanauser gg.handler.hjdbc.driverclass=com.sap.db.jdbc.driver gg.handler.hjdbc.username=hanauser gg.handler.hjdbc.password=xxxxx gg.handler.hjdbc.mode=tx ### Native library config ### goldengate.userexit.nochkpt=true goldengate.userexit.timestamp=utc goldengate.log.logname=cuserexit goldengate.log.level=info goldengate.log.tofile=true goldengate.userexit.writers=javawriter javawriter.stats.full=true 14 P a g e

15 javawriter.stats.display=true gg.classpath=ggjava/jdbc/$hana-jdbc.jar: javawriter.stats.full=true javawriter.stats.display=true javawriter.bootoptions=-xmx32m -Xms32m -Djava.class.path=ggjava/ggjava.jar Lines denoted by the character # are comments. The Oracle GoldenGate Adapters Administration Guide for Java, available for download from the Oracle Technology Network, contains details about all of the shown properties. We will discuss several to provide greater insight into the configuration requirements. OGG Application adapter would need SAP HANA JDBC driver class (JAR files) to open communication channel with SAP HANA database through JDCB interface. gg.handler.hjdbc.type gg.handler.hjdbc.connectionurl gg.handler.hjdbc.schema gg.handler.hjdbc.driverclass gg.handler.hjdbc.username gg.handler.hjdbc.password gg.classpath=ggjava/jdbc/$hana-jdbc.jar: Handler Java class which handles source DML events from Application adapters and pushing data into SAP HANA Database. JDBC Connection URL to SAP HANA cloud instance SAP HANA database schema name SAP HANA JDBC driver class name SAP HANA database user name SAP HANA database user password Path to SAP HANA JDBC Jar file. Creating OGG Processes on Target system GGSCI> ADD EXTRACT PHANA, EXTTRAILSOURCE /ogg/gg_adp_home/dirdat/mt Troubleshooting Tips When troubleshooting Java Data Pump issues, sometimes the root cause of the failure is not written to the logs or OGG Report file. Best practice is to run the Java Data Pump from a terminal shell. When run from a terminal shell, all output is written to stdout. For example, to run the P_OGGQ Java Data Pump from a terminal shell: GGSCI> sh./extract pf./dirprm/phana.prm Summary Oracle GoldenGate can be used to replicate data to the SAP HANA Database via OGG Application Adapter. This white paper presented information that would allow users to quickly install and configure the HANA database connectivity with Oracle GoldenGate. 15 P a g e

16 Where to Go for More Information Hopefully, this white paper sufficiently cover all important technical aspects to replicate data from Oracle database to SAP HANA In-Memory database. Should you need to customize the Java Adapter, source code for adapter is provided to developers to enable customizations for specific implementations. Refer Oracle Database 12.1 Documentation for additional information on the Oracle 12.1 RDBMS. Refer Oracle GoldenGate 12c Reference Guide and the Oracle GoldenGate 12c Administration Guide for additional information on: Extract Parameters for Wi Replicate Replicate Management Considerations Refer Installation guide of Oracle GoldenGate Application Adapters 12C, for detailed information on GoldenGate Application Adapter setup and configuration. 16 P a g e