How to Install and Configure Big Data Edition for Hortonworks

Transcription

1 How to Install and Configure Big Data Edition for Hortonworks Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. All other company and product names may be trade names or trademarks of their respective owners and/or copyrighted materials of such owners.

2 Abstract Install and configure Big Data Edition to run mappings on a Hadoop cluster on Hortonworks HDP. After you install Big Data Edition, you must enable mappings to run on Hortonworks HDP. You must also configure the Big Data Edition Client files to communicate with the Hadoop cluster. Supported Versions Big Data Edition Hotfix 2 Update 1 Big Data Edition Hotfix 3 Update 2 Table of Contents Overview Before You Begin Install and Configure PowerCenter Install and Configure PowerExchange Adapters Install and Configure Data Replication Pre-Installation Tasks for a Single Node Environment Pre-Installation Tasks for a Cluster Environment Informatica Big Data Edition Installation Installing in a Single Node Environment Installing in a Cluster Environment Installing in a Single Node Environment Installing in a Cluster Environment from the Primary NameNode Using SCP Protocol Installing in a Cluster Environment from the Primary NameNode Using FTP, HTTP, or NFS Protocol Installing in a Cluster Environment from any Machine Reference Data Requirements Configure the Hadoop Cluster Update Hadoop Cluster Configuration Parameters on the Hadoop Cluster Add hbase_protocol.jar to the Hadoop classpath Copy Teradata JDBC.jar Files to Hadoop Nodes Configure the Informatica Domain Configure Hadoop Pushdown Properties for the Data Integration Service Update Hadoop Cluster Configuration Parameters on the Informatica Domain Library Path and Path Variables for Mappings in a Hive Environment Hadoop Environment Properties File Hive Variables for Mappings in a Hive Environment Configure Hadoop Cluster Properties for Hortonworks HDP Copy Teradata JDBC.jar Files to the Data Integration Service Machine Update the Repository Plug-in Configure the Client Machine

3 Informatica Developer Files and Variables Copy Teradata JDBC.jar Files to the Client Machine Enable Tez Configure High Availability Configuring a Highly Available Hortonworks Cluster Connections HDFS Connection Properties HBase Connection Properties Hive Connection Properties Creating a Connection Informatica Big Data Edition Uninstallation Uninstalling Big Data Edition Overview The Informatica Big Data Edition installation is distributed as a Red Hat Package Manager (RPM) installation package. After you install Big Data Edition, you must enable mappings to run on a Hadoop cluster on Hortonworks HDP. You must also configure the Big Data Edition Client files to communicate with the Hadoop cluster. Informatica supports Hortonworks HDP clusters that are deployed on-premise, on Amazon EC2, or on Microsoft Azure. For Hortonworks HDP, Informatica supports MapReduce v2 and CapacityScheduler. After you install Big Data Edition, you must enable Informatica mappings to run on a Hadoop cluster on a Hadoop distribution. After you enable Informatica mappings to run on a Hadoop cluster, you must configure the Big Data Edition Client files to communicate with a Hadoop cluster on Hortonworks HDP. Before You Begin Before you begin the installation, install the Informatica components and PowerExchange adapters, and perform the pre-installation tasks. Install and Configure PowerCenter Before you install Big Data Edition, install and configure Informatica PowerCenter. You can install the following PowerCenter editions: PowerCenter Advanced Edition PowerCenter Standard Edition PowerCenter Real Time Edition You must install the Informatica services and clients. Run the Informatica services installation to configure the PowerCenter domain and create the Informatica services. Run the Informatica client installation to create the PowerCenter Client. 3

4 Install and Configure PowerExchange Adapters Based on your business needs, install and configure PowerExchange adapters. Use Big Data Edition with PowerCenter and Informatica adapters for access to sources and targets. To run Informatica mappings in a Hive environment you must install and configure PowerExchange for Hive. For more information, see the Informatica PowerExchange for Hive User Guide. PowerCenter Adapters Use PowerCenter adapters, such as PowerExchange for Hadoop, to define sources and targets in PowerCenter mappings. For more information about installing and configuring PowerCenter adapters, see the PowerExchange adapter documentation. Informatica Adapters You can use the following Informatica adapters as part of PowerCenter Big Data Edition: PowerExchange for DataSift PowerExchange for Facebook PowerExchange for HBase PowerExchange for HDFS PowerExchange for Hive PowerExchange for LinkedIn PowerExchange for Teradata Parallel Transporter API PowerExchange for Twitter PowerExchange for Web Content-Kapow Katalyst For more information, see the PowerExchange adapter documentation. Install and Configure Data Replication To migrate data with minimal downtime and perform auditing and operational reporting functions, install and configure Data Replication. For information, see the Informatica Data Replication User Guide. Pre-Installation Tasks for a Single Node Environment Before you begin the Big Data Edition installation in a single node environment, perform the pre-installation tasks. Verify that Hadoop is installed with Hadoop File System (HDFS) and MapReduce. The Hadoop installation should include a Hive data warehouse that is configured to use a non-embedded database as the MetaStore. For more information, see the Apache website here: To perform both read and write operations in native mode, install the required third-party client software. For example, install the Oracle client to connect to the Oracle database. Verify that the Big Data Edition administrator user can run sudo commands or have user root privileges. Verify that the temporary folder on the local node has at least 700 MB of disk space. Download the following file to the temporary folder: InformaticaHadoop- <InformaticaForHadoopVersion>.tar.gz Extract the following file to the local node where you want to run the Big Data Edition installation: InformaticaHadoop-<InformaticaForHadoopVersion>.tar.gz 4

5 Pre-Installation Tasks for a Cluster Environment Before you begin the Big Data Edition installation in a cluster environment, perform the following tasks: Install third-party software. Verify the distribution method. Verify system requirements. Verify connection requirements. Download the RPM. Install Third-Party Software Verify that the following third-party software is installed: Hadoop with Hadoop Distributed File System (HDFS) and MapReduce Hadoop must be installed on every node within the cluster. The Hadoop installation must include a Hive data warehouse that is configured to use a MySQL database as the MetaStore. You can configure Hive to use a local or remote MetaStore server. For more information, see the Apache website here: Note: Informatica does not support embedded MetaStore server setups. Database client software to perform read and write operations in native mode Install the client software for the database. Informatica requires the client software to run MapReduce jobs. For example, install the Oracle client to connect to the Oracle database. Install the database client software on all the nodes within the Hadoop cluster. Verify the Distribution Method You can distribute the RPM package with one of the following protocols: File Transfer Protocol (FTP) Hypertext Transfer Protocol (HTTP) Network File System (NFS) protocol Secure Copy (SCP) protocol To verify that you can distribute the RPM package with one of the protocols, perform the following tasks: 1. Ensure that the server or service for your distribution method is running. 2. In the config file on the machine where you want to run the Big Data installation, set the DISTRIBUTOR_NODE parameter to the following setting: FTP: Set DISTRIBUTOR_NODE=ftp://<Distributor Node IP Address>/pub HTTP: Set DISTRIBUTOR_NODE= Node IP Address> NFS: Set DISTRIBUTOR_NODE=<Shared file location on the node.> The file location must be accessible to all nodes in the cluster. Verify System Requirements Verify the following system requirements: The Big Data Edition administrator can run sudo commands or has root user privileges. 5

6 The temporary folder in each of the nodes on which Big Data Edition will be installed has at least 700 MB of disk space. Verify Connection Requirements Verify the connection to the Hadoop cluster nodes. Big Data Edition requires a Secure Shell (SSH) connection without a password between the machine where you want to run the Big Data Edition installation and all the nodes in the Hadoop cluster. Download the RPM Download the following file to a temporary folder: InformaticaHadoop-<InformaticaForHadoopVersion>.tar.gz Extract the file to the machine from where you want to distribute the RPM package and run the Big Data Edition installation. Copy the following package to a shared directory based on the transfer protocol you are using: InformaticaHadoop- <InformaticaForHadoopVersion>.rpm. For example, HTTP: /var/www/html FTP: /var/ftp/pub NFS: <Shared location on the node> The file location must be accessible by all the nodes in the cluster. Note: The RPM package must be stored on a local disk and not on HDFS. Informatica Big Data Edition Installation You can install Big Data Edition in a single node environment. You can also install Big Data Edition in a cluster environment from the primary NameNode or from any machine. Install Big Data Edition in a single node environment or cluster environment: Install Big Data Edition in a single node environment. Install Big Data Edition in a cluster environment from the primary NameNode using SCP protocol. Install Big Data Edition in a cluster environment from the primary NameNode using FTP, HTTP, or NFS protocol. Install Big Data Edition in a cluster environment from any machine. Install Big Data Edition from a shell command line. Installing in a Single Node Environment You can install Big Data Edition in a single node environment. 1. Extract the Big Data Edition tar.gz file to the machine. 2. Install Big Data Edition by running the installation shell script in a Linux environment. 6

7 Installing in a Cluster Environment You can install Big Data Edition in a cluster environment. 1. Extract the Big Data Edition tar.gz file to a machine. 2. Distribute the RPM package to all of the nodes within the Hadoop cluster. You can distribute the RPM package using any of the following protocols: File Transfer Protocol (FTP), Hypertext Transfer Protocol (HTTP), Network File System (NFS), or Secure Copy Protocol (SCP). 3. Install Big Data Edition by running the installation shell script in a Linux environment. You can install Big Data Edition from the primary NameNode or from any machine using the HadoopDataNodes file. Install from the primary NameNode. You can install Big Data Edition using FTP, HTTP, NFS or SCP protocol. During the installation, the installer shell script picks up all of the DataNodes from the following file: $HADOOP_HOME/conf/slaves. Then, it copies the Big Data Edition binary files to the following directory on each of the DataNodes: /<BigDataEditionInstallationDirectory>/Informatica. You can perform this step only if you are deploying Hadoop from the primary NameNode. Install from any machine. Add the IP addresses or machine host names, one for each line, for each of the nodes in the Hadoop cluster in the HadoopDataNodes file. During the Big Data Edition installation, the installation shell script picks up all of the nodes from the HadoopDataNodes file and copies the Big Data Edition binary files to the /<BigDataEditionInstallationDirectory>/Informatica directory on each of the nodes. Installing in a Single Node Environment You can install Big Data Edition in a single node environment. 1. Log in to the machine. 2. Run the following command from the Big Data Edition root directory to start the installation in console mode: bash InformaticaHadoopInstall.sh 3. Press y to accept the Big Data Edition terms of agreement. 4. Press Enter. 5. Press 1 to install Big Data Edition in a single node environment. 6. Press Enter. 7. Type the absolute path for the Big Data Edition installation directory and press Enter. Start the path with a slash. The directory names in the path must not contain spaces or the following special characters: { # $ % ^ & * ( ) : ; ' ` < >,? + [ ] \ If you type a directory path that does not exist, the installer creates the entire directory path on each of the nodes during the installation. Default is /opt. 8. Press Enter. The installer creates the /<BigDataEditionInstallationDirectory>/Informatica directory and populates all of the file systems with the contents of the RPM package. To get more information about the tasks performed by the installer, you can view the informatica-hadoopinstall.<datetimestamp>.log installation log file. 7

8 Installing in a Cluster Environment from the Primary NameNode Using SCP Protocol You can install Big Data Edition in a cluster environment from the primary NameNode using SCP protocol. 1. Log in to the primary NameNode. 2. Run the following command to start the Big Data Edition installation in console mode: bash InformaticaHadoopInstall.sh 3. Press y to accept the Big Data Edition terms of agreement. 4. Press Enter. 5. Press 2 to install Big Data Edition in a cluster environment. 6. Press Enter. 7. Type the absolute path for the Big Data Edition installation directory. Start the path with a slash. The directory names in the path must not contain spaces or the following special characters: { # $ % ^ & * ( ) : ; ' ` < >,? + [ ] \ If you type a directory path that does not exist, the installer creates the entire directory path on each of the nodes during the installation. Default is /opt. 8. Press Enter. 9. Press 1 to install Big Data Edition from the primary NameNode. 10. Press Enter. 11. Type the absolute path for the Hadoop installation directory. Start the path with a slash. 12. Press Enter. 13. Type y. 14. Press Enter. The installer retrieves a list of DataNodes from the $HADOOP_HOME/conf/slaves file. On each of the DataNodes, the installer creates the Informatica directory and populates all of the file systems with the contents of the RPM package. The Informatica directory is located here: / <BigDataEditionInstallationDirectory>/Informatica You can view the informatica-hadoop-install.<datetimestamp>.log installation log file to get more information about the tasks performed by the installer. Installing in a Cluster Environment from the Primary NameNode Using FTP, HTTP, or NFS Protocol You can install Big Data Edition in a cluster environment from the primary NameNode using FTP, HTTP, or NFS protocol. 1. Log in to the primary NameNode. 2. Run the following command to start the Big Data Edition installation in console mode: bash InformaticaHadoopInstall.sh 3. Press y to accept the Big Data Edition terms of agreement. 4. Press Enter. 5. Press 2 to install Big Data Edition in a cluster environment. 6. Press Enter. 7. Type the absolute path for the Big Data Edition installation directory. 8

9 Start the path with a slash. The directory names in the path must not contain spaces or the following special characters: { # $ % ^ & * ( ) : ; ' ` < >,? + [ ] \ If you type a directory path that does not exist, the installer creates the entire directory path on each of the nodes during the installation. Default is /opt. 8. Press Enter. 9. Press 1 to install Big Data Edition from the primary NameNode. 10. Press Enter. 11. Type the absolute path for the Hadoop installation directory. Start the path with a slash. 12. Press Enter. 13. Type n. 14. Press Enter. 15. Type y. 16. Press Enter. The installer retrieves a list of DataNodes from the $HADOOP_HOME/conf/slaves file. On each of the DataNodes, the installer creates the /<BigDataEditionInstallationDirectory>/Informatica directory and populates all of the file systems with the contents of the RPM package. You can view the informatica-hadoop-install.<datetimestamp>.log installation log file to get more information about the tasks performed by the installer. Installing in a Cluster Environment from any Machine You can install Big Data Edition in a cluster environment from any machine. 1. Verify that the Big Data Edition administrator has user root privileges on the node that will be running the Big Data Edition installation. 2. Log in to the machine as the root user. 3. In the HadoopDataNodes file, add the IP addresses or machine host names of the nodes in the Hadoop cluster on which you want to install Big Data Edition. The HadoopDataNodes file is located on the node from where you want to launch the Big Data Edition installation. You must add one IP addresses or machine host names of the nodes in the Hadoop cluster for each line in the file. 4. Run the following command to start the Big Data Edition installation in console mode: bash InformaticaHadoopInstall.sh 5. Press y to accept the Big Data Edition terms of agreement. 6. Press Enter. 7. Press 2 to install Big Data Edition in a cluster environment. 8. Press Enter. 9. Type the absolute path for the Big Data Edition installation directory and press Enter. Start the path with a slash. Default is /opt. 10. Press Enter. 11. Press 2 to install Big Data Edition using the HadoopDataNodes file. 12. Press Enter. The installer creates the /<BigDataEditionInstallationDirectory>/Informatica directory and populates all of the file systems with the contents of the RPM package on the first node that appears in the HadoopDataNodes file. The installer repeats the process for each node in the HadoopDataNodes file. 9

10 Reference Data Requirements If you have a Data Quality product license, you can push a mapping that contains data quality transformations to a Hadoop cluster. Data quality transformations can use reference data to verify that data values are accurate and correctly formatted. When you apply a pushdown operation to a mapping that contains data quality transformations, the operation can copy the reference data that the mapping uses. The pushdown operation copies reference table data, content set data, and identity population data to the Hadoop cluster. After the mapping runs, the cluster deletes the reference data that the pushdown operation copied with the mapping. Note: The pushdown operation does not copy address validation reference data. If you push a mapping that performs address validation, you must install the address validation reference data files on each DataNode that runs the mapping. The cluster does not delete the address validation reference data files after the address validation mapping runs. Address validation mappings validate and enhance the accuracy of postal address records. You can buy address reference data files from Informatica on a subscription basis. You can download the current address reference data files from Informatica at any time during the subscription period. Installing the Address Reference Data Files To install the address reference data files on each DataNode in the cluster, create an automation script. 1. Browse to the address reference data files that you downloaded from Informatica. 2. Extract the compressed address reference data files. 3. Stage the files to the NameNode machine or to another machine that can write to the DataNodes. 4. Create an automation script to copy the files to each DataNode. The default directory for the address reference data files in the Hadoop environment is /reference_data. If you staged the files on the NameNode, use the slaves file for the Hadoop cluster to identify the DataNodes. If you staged the files on another machine, use the Hadoop_Nodes.txt file to identify the DataNodes. You find this file in the Big Data Edition installation package. 5. Run the script. The script copies the address reference data files to the DataNodes. Configure the Hadoop Cluster Configure the Hadoop cluster to run mappings on Hortonworks. To enable mappings to run on the cluster, complete the following tasks: 1. Update Hadoop cluster configuration parameters. 2. Add hbase_protocol.jar to the Hadoop classpath. 3. Copy teradata JDBC.jar Files to Hadoop nodes. 10

11 Update Hadoop Cluster Configuration Parameters on the Hadoop Cluster Hadoop cluster configuration parameters that set Java library path in mapred-site.xml can override the paths set in hadoopenv.properties. Update the mapred-site.xml cluster configuration file on all the cluster nodes to remove Java options that set the Java library path. The following cluster configuration parameters in mapred-site.xml can override the Java library path set in hadoopenv.properties: mapreduce.admin.map.child.java.opts mapreduce.admin.reduce.child.java.opts If the Data Integration Service cannot access the native libraries set in hadoopenv.properties, the mappings can fail to run in a Hive environment. After you install, update the cluster configuration file mapred-site.xml to remove the Java option -Djava.library.path from the property configuration. Example to Update mapred-site.xml on Cluster Nodes If the mapred-site.xml file sets the following configuration for mapreduce.admin.map.child.java.opts parameter: <property> <name>mapreduce.admin.map.child.java.opts</name> <value>-server -XX:NewRatio=8 -Djava.library.path=/usr/lib/hadoop/lib/native/:/mylib/ -Djava.net.preferIPv4Stack=true</value> <final>true</final> </property> The path to Hadoop libraries in mapreduce.admin.map.child.java.opts overrides the following path set in the hadoopenv.properties file: infapdo.java.opts=-xmx512m -XX:GCTimeRatio=34 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:NewRatio=2 -Djava.library.path=$HADOOP_NODE_INFA_HOME/services/ shared/bin:$hadoop_node_hadoop_dist/lib/*:$hadoop_node_hadoop_dist/lib/native/linux-amd Djava.security.egd=file:/dev/./urandom Add hbase_protocol.jar to the Hadoop classpath Add hbase-protocol.jar to the Hadoop classpath on every node on the Hadoop cluster. Then, restart the Node Manager for each node in the Hadoop cluster. hbase-protocol.jar is located in the HBase installation directory on the Hadoop cluster. For more information, refer to the following link: Copy Teradata JDBC.jar Files to Hadoop Nodes To use Lookup transformations with a Teradata data object in Hive pushdown mode, you must copy the Teradata JDBC drivers to the Informatica installation directory. You can download the Teradata JDBC drivers from Teradata. For more information about the drivers, see the following Teradata website: The software available for download at the referenced links belongs to a third party or third parties, not Informatica Corporation. The download links are subject to the possibility of errors, omissions or change. Informatica assumes no responsibility for such links and/or such software, disclaims all warranties, either express or implied, including but not limited to, implied warranties of merchantability, fitness for a particular purpose, title and non-infringement, and disclaims all liability relating thereto. Copy tdgssconfig.jar and terajdbc4.jar from the Teradata JDBC drivers to the following directory on every node in the Hadoop cluster: <Informatica installation directory>/externaljdbcjars 11

12 Configure the Informatica Domain Configure the Informatica domain to run mappings in a Hive environment. To configure the Informatica domain to run mappings in a Hive environment, complete the following tasks: 1. Configure the Hadoop pushdown properties for the Data Integration Service. 2. Update Hadoop cluster configuration parameters on the Informatica domain. 3. Configure the library path and path variables for mappings in a Hive environment. 4. Configure the Hadoop environment variable properties. 5. Configure the Hive environment variables. 6. Configure Hadoop cluster properties for Hortonworks HDP. 7. Copy Teradata JDBC.jar files to the Data Integration Service machine. 8. Update the repository plug-in. Configure Hadoop Pushdown Properties for the Data Integration Service Configure Hadoop pushdown properties for the Data Integration Service to run mappings in a Hive environment. You can configure Hadoop pushdown properties for the Data Integration Service in the Administrator tool. The following table describes the Hadoop pushdown properties for the Data Integration Service: Property Informatica Home Directory on Hadoop Hadoop Distribution Directory Data Integration Service Hadoop Distribution Directory Description The Big Data Edition home directory on every data node created by the Hadoop RPM install. Type /<BigDataEditionInstallationDirectory>/Informatica. The directory containing a collection of Hive and Hadoop JARS on the cluster from the RPM Install locations. The directory contains the minimum set of JARS required to process Informatica mappings in a Hadoop environment. Type /<BigDataEditionInstallationDirectory>/ Informatica/services/shared/hadoop/[Hadoop_distribution_name]. The Hadoop distribution directory on the Data Integration Service node. The contents of the Data Integration Service Hadoop distribution directory must be identical to Hadoop distribution directory on the data nodes. Hadoop Distribution Directory You can modify the Hadoop distribution directory on the data nodes. When you modify the Hadoop distribution directory, you must copy the minimum set of Hive and Hadoop JARS, and the Snappy libraries required to process Informatica mappings in a Hive environment from your Hadoop install location. The actual Hive and Hadoop JARS can vary depending on the Hadoop distribution and version. The Hadoop RPM installs the Hadoop distribution directories in the following path: <BigDataEditionInstallationDirectory>/Informatica/services/shared/hadoop. 12

13 Update Hadoop Cluster Configuration Parameters on the Informatica Domain Hadoop cluster configuration parameters that set Java library path in mapred-site.xml can override the paths set in hadoopenv.properties. If the Data Integration Service cannot access the native libraries set in hadoopenv.properties, the mappings can fail to run in a Hive environment. After you install Big Data Edition, edit hadoopenv.properties to include the user Hadoop libraries in the Java Library path. Note: Before you perform this task, update mapred-site.xml on all the cluster nodes to remove Java options that set the Java library path. For more information, see Update Hadoop Cluster Configuration Parameters on the Hadoop Cluster on page 11. To run mappings in a Hive environment, change hadoopenv.properties to include the Hadoop libraries in the path /usr/lib/hadoop/lib/native and /mylib/ with the following syntax: infapdo.java.opts=-xmx512m -XX:GCTimeRatio=34 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:NewRatio=2 -Djava.library.path=$HADOOP_NODE_INFA_HOME/services/ shared/bin:$hadoop_node_hadoop_dist/lib/*:$hadoop_node_hadoop_dist/lib/native/linuxamd64-64:/usr/lib/hadoop/lib/native/:/mylib/ -Djava.security.egd=file:/dev/./urandom Library Path and Path Variables for Mappings in a Hive Environment To run mappings in a Hive environment configure the library path and path environment variables in hadoopenv.properties. Configure following library path and path environment variables: If the Data Integration Service runs on a machine that uses SUSE, verify that the following entries are set to a valid value that is not POSIX: - infapdo.env.entry.lc_all=lc_all - infapdo.env.entry.lang=lang For example, you can use US.UTF-8. When you run mappings in a Hive environment, configure the ODBC library path before the Teradata library path. For example, infapdo.env.entry.ld_library_path=ld_library_path=$hadoop_node_infa_home/ services/shared/bin:$hadoop_node_infa_home/odbc7.0/lib/:/opt/teradata/client/13.10/tbuild/ lib64:/opt/teradata/client/13.10/odbc_64/lib:/databases/oracle11.2.0_64bit/lib:/databases/ db2v9.5_64bit/lib64/:$hadoop_node_infa_home/datatransformation/bin: $HADOOP_NODE_HADOOP_DIST/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH. Hadoop Environment Properties File To add environment variables or to extend existing ones, use the Hadoop environment properties file, hadoopenv.properties. You can optionally add third-party environment variables or extend the existing PATH environment variable in hadoopenv.properties. 1. Go to the following location: <InformaticaInstallationDir>/services/shared/hadoop/ <Hadoop_distribution_name>/infaConf 2. Find the file named hadoopenv.properties. 3. Back up the file before you modify it. 4. Use a text editor to open the file and modify the properties. 13

14 5. Save the properties file with the name hadoopenv.properties. Hive Variables for Mappings in a Hive Environment To run mappings in a Hive environment, configure Hive environment variables.. You can configure Hive environment variables in the file /<BigDataEditionInstallationDirectory>/Informatica/ services/shared/hadoop/<hadoop_distribution_name>/conf/hive-site.xml. Configure the following Hive environment variables: hive.exec.dynamic.partition=true and hive.exec.dynamic.partition.mode=nonstrict. Configure if you want to use Hive dynamic partitioned tables. hive.optimize.ppd = false. Disable predicate pushdown optimization to get accurate results for mappings with Hive version You cannot use predicate pushdown optimization for a Hive query that uses multiple insert statements. The default Hadoop RPM installation sets hive.optimize.ppd to false. Configure Hadoop Cluster Properties for Hortonworks HDP Configure Hadoop cluster properties in the yarn-site.xml file and mapred-site.xml file that the Data Integration Service uses when it runs mappings on a Hortonworks HDP cluster Configure yarn-site.xml for the Data Integration Service You need to configure the Hortonworks cluster properties in the yarn-site.xml file that the Data Integration Service uses when it runs mappings in a Hadoop cluster. If you use the Big Data Edition Configuration Utility to configure Big Data Edition, yarn-site.xml is automatically configured. Open the yarn-site.xml file in the following directory on the node on which the Data Integration Service runs: <Informatica installation directory>/services/shared/hadoop/conf/ Configure the following property in the yarn-site.xml file: yarn.resourcemanager.scheduler.address Scheduler interface address. Use the value in the following file: /etc/hadoop/conf/yarn-site.xml The following sample text shows the property you can set in yarn-site.xml: <property> <name>yarn.resourcemanager.scheduler.address</name> <value>hostname:port</value> <description>the address of the scheduler interface</description> </property> Configure mapred-site.xml for the Data Integration Service You need to configure the Hortonworks cluster properties in the mapred-site.xml file that the Data Integration Service uses when it runs mappings in a Hadoop cluster. Open the mapred-site.xml file in the following directory on the node on which the Data Integration Service runs: <Informatica installation directory>/services/shared/hadoop/hortonworks_<version>/conf/ Configure the following properties in the mapred-site.xml file: mapreduce.jobhistory.intermediate-done-dir Directory where the MapReduce jobs write history files. Use the value in the following file: /etc/hadoop/conf/mapred-site.xml 14

15 mapreduce.jobhistory.done-dir Directory where the MapReduce JobHistory server manages history files. Use the value in the following file: /etc/hadoop/conf/mapred-site.xml The following sample text shows the properties you must set in the mapred-site.xml file: <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/mr-history/tmp</value> <description>directory where MapReduce jobs write history files.</description> </property> <property> <name>mapreduce.jobhistory.done-dir</name> <value>/mr-history/done</value> <description>directory where the MapReduce JobHistory server manages history files.</ description> </property> If you use the Big Data Edition Configuration Utility to configure Big Data Edition, the following properties are automatically configured in mapred-site.xml. If you do not use the utility, configure the following properties in mapredsite.xml: mapreduce.jobhistory.address Location of the MapReduce JobHistory Server. Use the value in the following file:/etc/hadoop/conf/mapred-site.xml mapreduce.jobhistory.webapp.address Web address of the MapReduce JobHistory Server. Use the value in the following file: /etc/hadoop/conf/mapred-site.xml The following sample text shows the properties you can set in the mapred-site.xml file: <property> <name>mapreduce.jobhistory.address</name> <value>hostname:port</value> <description>mapreduce JobHistory Server IPC host:port</description> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>hostname:port</value> <description>mapreduce JobHistory Server Web UI host:port</description> </property> Configure Rolling Upgrades for Hortonworks HDP To enable support for rolling upgrades for Hortonworks HDP, you must configure the following properties in mapredsite.xml on the machine where the Data Integration Service runs: mapreduce.application.classpath Classpaths for MapReduce applications. Use the following value: $PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/ mapreduce/lib/*:$pwd/mr-framework/hadoop/share/hadoop/common/*:$pwd/mr-framework/hadoop/ share/hadoop/common/lib/*:$pwd/mr-framework/hadoop/share/hadoop/yarn/*:$pwd/mrframework/hadoop/share/hadoop/yarn/lib/*:$pwd/mr-framework/hadoop/share/hadoop/hdfs/*: $PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/<hadoop_version>/hadoop/lib/ hadoop-lzo jar:/etc/hadoop/conf/secure Replace <hadoop_version> with your Hortonworks HDP version. For example, use for a Hortonworks HDP 2.2 cluster. 15

16 mapreduce.application.framework.path Path for the MapReduce framework archive. Use the following value: /hdp/apps/<hadoop_version>/mapreduce/mapreduce.tar.gz#mr-framework Replace <hadoop_version> with your Hortonworks HDP version. For example, use for a Hortonworks HDP 2.2 cluster. The following sample text shows the properties you can set in the mapred-site.xml file: <property> <name>mapreduce.application.classpath</name> <value>$pwd/mr-framework/hadoop/share/hadoop/mapreduce/*:$pwd/mr-framework/hadoop/share/hadoop/ mapreduce/lib/*:$pwd/mr-framework/hadoop/share/hadoop/common/*:$pwd/mr-framework/hadoop/share/ hadoop/common/lib/*:$pwd/mr-framework/hadoop/share/hadoop/yarn/*:$pwd/mr-framework/hadoop/share/ hadoop/yarn/lib/*:$pwd/mr-framework/hadoop/share/hadoop/hdfs/*:$pwd/mr-framework/hadoop/share/ hadoop/hdfs/lib/*:/usr/hdp/<hadoop_version>/hadoop/lib/hadoop-lzo jar:/etc/ hadoop/conf/secure </value> <description>classpaths for MapReduce applications. Replace <hadoop_version> with your Hortonworks HDP version. For example, use for a Hortonworks HDP 2.2 cluster. </description> </property> <property> <name>mapreduce.application.framework.path</name> <value>/hdp/apps/<hadoop_version>/mapreduce/mapreduce.tar.gz#mr-framework</value> <description> Path for the MapReduce framework archive. Replace <hadoop_version> with your Hortonworks HDP version. For example, use for a Hortonworks HDP 2.2 cluster. </description> </property> Copy Teradata JDBC.jar Files to the Data Integration Service Machine To use Lookup transformations with a Teradata data object in Hive pushdown mode, you must copy the Teradata JDBC drivers to the Informatica installation directory. You can download the Teradata JDBC drivers from Teradata. For more information about the drivers, see the following Teradata website: The software available for download at the referenced links belongs to a third party or third parties, not Informatica Corporation. The download links are subject to the possibility of errors, omissions or change. Informatica assumes no responsibility for such links and/or such software, disclaims all warranties, either express or implied, including but not limited to, implied warranties of merchantability, fitness for a particular purpose, title and non-infringement, and disclaims all liability relating thereto. Copy tdgssconfig.jar and terajdbc4.jar from the Teradata JDBC drivers to the following directory on the machine where the Data Integration runs: <Informatica installation directory>/externaljdbcjars Update the Repository Plug-in If you upgraded an existing repository, you must update the repository plug-in to enable PowerExchange for HDFS to run on the Hadoop distribution. If you created a new repository, skip this task. 1. Ensure that the Repository service is running in exclusive mode. 2. On the server machine, open the command console. 3. Run cd <Informatica installation directory>/server/bin 4. Run./pmrep connect -r <repo_name> -d <domain_name> -n <username> -x <password> 5. Run./pmrep registerplugin -i native/pmhdfs.xml -e -N true 16

17 6. Set the Repository service to normal mode. 7. Open the PowerCenter Workflow manager on the client machine. The distribution appears in the Connection Object menu. Configure the Client Machine Configure the Big Data Edition Client files to communicate with the Hadoop cluster. To configure these files to communicate with the Hadoop cluster, complete the following tasks: 1. Configure the Big Data Edition client files. 2. Copy Teradata JDBC.jar files to the client machine. Informatica Developer Files and Variables Edit developercore.ini to enable the Developer tool to communicate with the Hadoop cluster on a particular Hadoop distribution. After you edit the file, you must click run.bat to launch the Developer tool client again. developercore.ini is located in the following directory: <InformaticaClientInstallationDirectory>\<version> \clients\developerclient Add the following property to developercore.ini: -DINFA_HADOOP_DIST_DIR=hadoop\<HadoopDistributionName> Copy Teradata JDBC.jar Files to the Client Machine To use Lookup transformations with a Teradata data object in Hive pushdown mode, you must copy the Teradata JDBC drivers to the Informatica installation directory. You can download the Teradata JDBC drivers from Teradata. For more information about the drivers, see the following Teradata website: The software available for download at the referenced links belongs to a third party or third parties, not Informatica Corporation. The download links are subject to the possibility of errors, omissions or change. Informatica assumes no responsibility for such links and/or such software, disclaims all warranties, either express or implied, including but not limited to, implied warranties of merchantability, fitness for a particular purpose, title and non-infringement, and disclaims all liability relating thereto. Copy tdgssconfig.jar and terajdbc4.jar to the following directory on the machine where the Developer tool runs: <Informatica installation directory>\clients\externaljdbcjars. Enable Tez To use Tez to push mapping logic to the Hadoop cluster, enable Tez for the Data Integration Service or for a Hive connection. When you enable Tez for the Data Integration Service, Tez becomes the default execution engine to push mapping logic to the Hadoop cluster. When you enable Tez for a Hive connection, Tez takes precedence over the execution engine set for the Data Integration Service. Enable Tez for the Data Integration Service To use Tez to push mapping logic to the Hadoop cluster, enable Tez for the Data Integration Service. Open hive-site.xml in the following directory on the node on which the Data Integration Service runs: <Informatica installation directory>/services/shared/hadoop/hortonworks_<version>/conf/ 17

18 Configure the following property: hive.execution.engine Chooses the execution engine. You can use "mr" for MapReduce or "tez", which requires Hadoop 2. The following sample text shows the property you can set in hive-site.xml: <property> <name>hive.execution.engine</name> <value>tez</value> <description>chooses execution engine. Options are: mr (MapReduce, default) or tez (Hadoop 2 only)</description> </property> To use MapReduce as the default execution engine to push mapping logic to the Hadoop cluster, use "mr" as the value for the hive.execution.engine property. Enable Tez for a Hive Connection When you enable Tez for a Hive connection, the Data Integration Service uses Tez to push mapping logic to the Hadoop cluster regardless of what is set for the Data Integration Service. 1. Open the Developer tool. 2. Click Window > Preferences. 3. Select Informatica > Connections. 4. Expand the domain. 5. Expand the Databases and select the Hive connection. 6. Edit the Hive connection and configure the Environment SQL property on the Database Connection tab. Use the following value: set hive.execution.engine=tez; If you enable Tez for the Data Integration Service but want to use MapReduce, you can use the following value for the Environment SQL property: set hive.execution.engine=mr;. Configure Tez After you enable Tez, you must specify the location of tez.tar.gz in tez-site.xml. You can find tez-site.xml in the following directory on the machine where the Data Integration Service runs: <Informatica installation directory>/services/shared/hadoop/hortonworks_<version>/conf. Configure the following property: tez.lib.uris Specifies the location of tez.tar.gz on the Hadoop cluster. Use the value specified in tez-site.xml on the cluster. You can find tez-site.xml in the following directory on any node in the cluster: /etc/tez/conf. Use the following syntax when you configure the tez.lib.uris property: <property> <name>tez.lib.uris</name> <value><file system default name>:<directory of tez.tar.gz></value> </property> For example, if ez.tar.gz is in the /apps/tez/lib directory on HDFS, enter the following property in tez-site.xml: <property> <name>tez.lib.uris</name> <value>hdfs://ivlhdp41/apps/tez/lib/tez.tar.gz</value> </property> 18

19 Configure High Availability You can configure the Data Integration Service and the Developer tool to read from and write to a highly available Hadoop cluster. A highly available Hadoop cluster can provide uninterrupted access to the JobTracker, NameNode, and ResourceManager in the cluster. The JobTracker is the service within Hadoop that assigns MapReduce jobs on the cluster. The NameNode tracks file data across the cluster. The ResourceManager tracks resources and schedules applications in the cluster. Configuring a Highly Available Hortonworks Cluster You can enable Data Integration Service and the Developer tool to read from and write to a highly available Hortonworks cluster. The Hortonworks cluster provides a highly available NameNode and ResourceManager. Perform the following steps: 1. Go to the following directory on the NameNode of the cluster: /etc/hadoop/conf 2. Locate the following files: hdfs-site.xml yarn-site.xml 3. Note: If you use the Big Data Edition Configuration Utility to configure Big Data Edition, skip this step. Copy the files to the following directory on the machine where the Data Integration Service: <Informatica installation directory>/services/shared/hadoop/hortonworks_<version>/conf 4. Copy the files to the following directory on the machine where the Developer tool runs <Informatica installation directory>/clients/developerclient/hadoop/hortonworks_<version>/ conf 5. Open the Developer tool. 6. Click Window > Preferences. 7. Select Informatica > Connections. 8. Expand the domain. 9. Expand Databases and select the Hive connection. 10. Edit the Hive connection and configure the following properties in the Properties to Run Mappings in Hadoop Cluster tab: Default FS URI Use the value from the dfs.nameservices property in hdfs-site.xml. Job tracker/yarn Resource Manager URI Enter any value in the following format: <string>:<port>. For example, enter dummy: Expand File Systems and select the HDFS connection. 12. Edit the HDFS connection and configure the following property in the Details tab: NameNode URI Use the value from the dfs.nameservices property in hdfs-site.xml. 19

20 Connections Define the connections you want to use to access data in Hive or HDFS. You can create the following types of connections: HDFS connection. Create an HDFS connection to read data from or write data to the Hadoop cluster. HBase connection. Create an HBase connection to access HBase. The HBase connection is a NoSQL connection Hive connection. Create a Hive connection to access Hive data or run Informatica mappings in the Hadoop cluster. Create a Hive connection in the following connection modes: - Use the Hive connection to access Hive as a source or target. If you want to use Hive as a target, you need to have the same connection or another Hive connection that is enabled to run mappings in the Hadoop cluster. You can access Hive as a source if the mapping is enabled for the native or Hive environment. You can access Hive as a target only if the mapping is run in the Hadoop cluster. - Use the Hive connection to validate or run an Informatica mapping in the Hadoop cluster. Before you run mappings in the Hadoop cluster, review the information in this guide about rules and guidelines for mappings that you can run in the Hadoop cluster. You can create the connections using the Developer tool, Administrator tool, and infacmd. Note: For information about creating connections to other sources or targets such as social media web sites or Teradata, see the respective PowerExchange adapter user guide for information. HDFS Connection Properties Use a Hadoop File System (HDFS) connection to access data in the Hadoop cluster. The HDFS connection is a file system type connection. You can create and manage an HDFS connection in the Administrator tool, Analyst tool, or the Developer tool. HDFS connection properties are case sensitive unless otherwise noted. Note: The order of the connection properties might vary depending on the tool where you view them. The following table describes HDFS connection properties: Property Name Description Name of the connection. The name is not case sensitive and must be unique within the domain. The name cannot exceed 128 characters, contain spaces, or contain the following special characters: ~ `! $ % ^ & * ( ) - + = { [ } ] \ : ; " ' <, >.? / ID Description Location Type String that the Data Integration Service uses to identify the connection. The ID is not case sensitive. It must be 255 characters or less and must be unique in the domain. You cannot change this property after you create the connection. Default value is the connection name. The description of the connection. The description cannot exceed 765 characters. The domain where you want to create the connection. Not valid for the Analyst tool. The connection type. Default is Hadoop File System. 20

21 Property User Name NameNode URI Description User name to access HDFS. The URI to access HDFS. Use the following format to specify the NameNode URI in Hortonworks: hdfs://<namenode>:<port> Where - <namenode> is the host name or IP address of the NameNode. - <port> is the port that the NameNode listens for remote procedure calls (RPC). HBase Connection Properties Use an HBase connection to access HBase. The HBase connection is a NoSQL connection. You can create and manage an HBase connection in the Administrator tool or the Developer tool. Hbase connection properties are case sensitive unless otherwise noted. The following table describes HBase connection properties: Property Name ID Description The name of the connection. The name is not case sensitive and must be unique within the domain. You can change this property after you create the connection. The name cannot exceed 128 characters, contain spaces, or contain the following special characters: ~ `! $ % ^ & * ( ) - + = { [ } ] \ : ; " ' <, >.? / String that the Data Integration Service uses to identify the connection. The ID is not case sensitive. It must be 255 characters or less and must be unique in the domain. You cannot change this property after you create the connection. Default value is the connection name. Description The description of the connection. The description cannot exceed 4,000 characters. Location Type ZooKeeper Host(s) ZooKeeper Port Enable Kerberos Connection The domain where you want to create the connection. The connection type. Select HBase. Name of the machine that hosts the ZooKeeper server. The name is case sensitive. When the ZooKeeper runs in the replicated mode, specify a comma-separated list of servers in the ZooKeeper quorum servers. If the TCP connection to the server breaks, the client connects to a different server in the quorum. Port number of the machine that hosts the ZooKeeper server. Enables the Informatica domain to communicate with the HBase master server or region server that uses Kerberos authentication. 21

22 Property HBase Master Principal Description Service Principal Name (SPN) of the HBase master server. Enables the ZooKeeper server to communicate with an HBase master server that uses Kerberos authentication. Enter a string in the following format: hbase/<domain.name>@<your-realm> Where: - domain.name is the domain name of the machine that hosts the HBase master server. - YOUR-REALM is the Kerberos realm. HBase Region Server Principal Service Principal Name (SPN) of the HBase region server. Enables the ZooKeeper server to communicate with an HBase region server that uses Kerberos authentication. Enter a string in the following format: hbase_rs/<domain.name>@<your-realm> Where: - domain.name is the domain name of the machine that hosts the HBase master server. - YOUR-REALM is the Kerberos realm. Hive Connection Properties Use the Hive connection to access Hive data. A Hive connection is a database type connection. You can create and manage a Hive connection in the Administrator tool, Analyst tool, or the Developer tool. Hive connection properties are case sensitive unless otherwise noted. Note: The order of the connection properties might vary depending on the tool where you view them. The following table describes Hive connection properties: Property Name ID Description Location Type Description The name of the connection. The name is not case sensitive and must be unique within the domain. You can change this property after you create the connection. The name cannot exceed 128 characters, contain spaces, or contain the following special characters: ~ `! $ % ^ & * ( ) - + = { [ } ] \ : ; " ' <, >.? / String that the Data Integration Service uses to identify the connection. The ID is not case sensitive. It must be 255 characters or less and must be unique in the domain. You cannot change this property after you create the connection. Default value is the connection name. The description of the connection. The description cannot exceed 4000 characters. The domain where you want to create the connection. Not valid for the Analyst tool. The connection type. Select Hive. 22