How to Configure Informatica HotFix 2 for Cloudera CDH 5.3

Transcription

1 How to Configure Informatica HotFix 2 for Cloudera CDH Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. All other company and product names may be trade names or trademarks of their respective owners and/or copyrighted materials of such owners.

2 Abstract Enable Big Data Edition to run mappings on a Hadoop cluster on Cloudera CDH 5.3. Supported Versions Informatica Big Data Edition HotFix 2 Table of Contents Overview Configure Hadoop Cluster Properties for Cloudera CDH 5.3 in yarn-site.xml Create a Staging Directory on HDFS Configure Virtual Memory Limits Add hbase_protocol.jar to the Hadoop classpath Configure the Hadoop Environment Variables Configure hive-site.xml Configure Hadoop Pushdown Properties for the Data Integration Service Known Limitations Overview You can enable Informatica mappings to run on a Hadoop cluster on Cloudera CDH 5.3. To enable Informatica mappings to run on a Cloudera CDH 5.3 cluster, complete the following tasks: 1. Configure Hadoop cluster properties in yarn-site.xml on the machine on which the Data Integration Service runs. 2. Configure virtual memory limits. 3. Add a staging directory to mapred-site.xml. 4. Add hbase_protocol.jar to the Hadoop classpath. 5. Configure the Hadoop environment variables. 6. Configure the Hadoop Pushdown Properties for the Data Integration Service. Configure Hadoop Cluster Properties for Cloudera CDH 5.3 in yarn-site.xml Configure Hadoop cluster properties in the yarn-site.xml file that the Data Integration Service uses when it runs mappings on a Cloudera CDH 5.3 cluster. yarn-site.xml is located in the following directory: <BigDataEditionInstallationDirectory>/services/shared/hadoop/cloudera_cdh5u2/conf In yarn-site.xml, configure the following properties: mapreduce.jobhistory.address Location of the MapReduce JobHistory Server. 2

3 Use the value in the following file:/etc/hadoop/conf/mapred-site.xml mapreduce.jobhistory.webapp.address Web address of the MapReduce JobHistory Server. Use the value in the following file: /etc/hadoop/conf/mapred-site.xml yarn.resourcemanager.scheduler.address Scheduler interface address. Use the value in the following file: /etc/hadoop/conf/yarn-site.xml The following sample code describes the properties you can set in yarn-site.xml: <name>mapreduce.jobhistory.address</name> <value>hostname:port</value> <description>mapreduce JobHistory Server IPC host:port</description> <name>mapreduce.jobhistory.webapp.address</name> <value>hostname:port</value> <description>mapreduce JobHistory Server Web UI host:port</description> <name>yarn.resourcemanager.scheduler.address</name> <value>hostname:port</value> <description>the address of the scheduler interface</description> Create a Staging Directory on HDFS If the Cloudera cluster uses HiveServer 2, you must grant the anonymous user the Execute permission on the staging directory or you must create another staging directory on HDFS. By default, a staging directory already exists on HDFS. You must grant the anonymous user the Execute permission on the staging directory. If you cannot grant the anonymous user the Execute permission on this directory, you must enter a valid user name for the user in the Hive connection. If you want to create another staging directory to store mapreduce jobs, you must create another staging directory on HDFS. After you create the staging directory, you must add it to mapred-site.xml. To create a staging directory on HDFS, run the following commands from the command line of the machine that runs the Hadoop cluster: hadoop fs mkdir /staging hadoop fs chmod R 0777 /staging Add the staging directory to mapred-site.xml. mapred-site.xml is located in the following directory: /etc/hadoop/conf/mapred-site.xml In mapred-site.xml, add the following entry: <name>yarn.app.mapreduce.am.staging-dir</name> <value>/staging</value> Configure Virtual Memory Limits Configure the virtual memory limits in yarn-site.xml for every node in the Hadoop cluster. After you configure virtual memory limits you must restart the Hadoop cluster. 3

4 yarn-site.xml is located in the following directory on every node in the Hadoop cluster: /etc/hadoop/conf/yarnsite.xml In yarn-site.xml, configure the following property: yarn.nodemanager.vmem-check-enabled Determines virtual memory limits. The following example describes the property you can configure in yarn-site.xml: <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> <description>enforces virtual memory limits for containers.</description> Add hbase_protocol.jar to the Hadoop classpath Add hbase-protocol.jar to the Hadoop classpath on every node on the Hadoop cluster. Then, restart the Node Manager for each node in the Hadoop cluster. hbase-protocol.jar is located in the HBase installation directory on the Hadoop cluster. For more information, refer to the following link: Configure the Hadoop Environment Variables Configure the Hadoop environment variables on the Informatica server to point to the native libraries on the Cloudera CDH cluster. Verify the Native Directory for the Cluster Before you configure the Hadoop environment variables, ask the Cloudera CDH cluster administrator for the native directory of the cluster. Typically, the native directory has the following path: <Cloudera installation directory>/lib/hadoop/lib/native. For example, if the cluster was installed and configured manually, the native directory is /usr/lib/hadoop/lib/ native. If the cluster was installed and configured with Cloudera Manager, the native directory is /opt/cloudera/ parcels/cdh/lib/hadoop/lib/native. Verify that the native directory contains the following files: libhadoop.so lipsnappy.so libsnappy.so.1 libsnappy.so libsnappy.so Configure the hadoopenv.properties File You must configure the hadoopenv.properties file on the machine that runs the Data Integration Service and all the nodes that are used for Hadoop pushdown To configure the Hadoop environment variables, perform the following steps: 1. Go to the following directory: <Informatica installation directory>/services/shared/hadoop/<hadoop_distribution_name>/ infaconf 4

5 2. Back up the hadoopenv.properties file. 3. Edit the hadoopenv.properties file. 4. Search for the infapdo.env.entry.ld_library_path variable, and find the following path: $HADOOP_NODE_HADOOP_DIST/lib/native. The following image shows the $HADOOP_NODE_HADOOP_DIST/lib/native path: 5. Replace the path with the path for the Cloudera CDH native directory: <Cloudera installation directory>/lib/hadoop/lib/native For example, the following image shows the replaced path for a cluster that is configured with Cloudera Manager: 6. Search for the Djava.library.path variable, and find the following path: $HADOOP_NODE_HADOOP_DIST/lib/ native. The following image shows the $HADOOP_NODE_HADOOP_DIST/lib/native path: 7. Replace the path with the path for the Cloudera CDH native directory: <Cloudera installation directory>/lib/hadoop/lib/native For example, the following image shows the replaced path for a cluster that is configured with Cloudera Manager: 8. Save and close the file. Configure hive-site.xml Configure hive-site.xml on the machine on which the Data Integration Service runs. 1. Go to the following directory on the machine on which the Data Integration Service runs: <Informatica installation directory>/services/shared/hadoop/<hadoop_distribution_name>/conf 2. Open hive-site.xml. 3. Configure the following property: hive.metastore.uris The metastore URI used to access metadata in a remote metastore setup. For a remote metastore, you must specify the Thrift server details. Use the following connection URI: thrift://<hostname>:<port> Hostname is the name or IP address of the Thrift metastore server. Port is the port on which the Thrift server is listening. The default port is The following sample code shows the property you can configure in hive-site.xml: <name>hive.metastore.uris</name> <value>thrift://hostname:port</value> <description>the metastore URI used to access meta data in a remote metastore setup.</ description> 5

6 Configure Hadoop Pushdown Properties for the Data Integration Service Configure Hadoop pushdown properties for the Data Integration Service to run mappings in a Hive environment. You can configure Hadoop pushdown properties for the Data Integration Service from the Administrator tool. The following table describes the Hadoop pushdown properties for the Data Integration Service: Property Informatica Home Directory on Hadoop Hadoop Distribution Directory Data Integration Service Hadoop Distribution Directory Description The Big Data Edition home directory on every data node created by the Hadoop RPM install. Type / <BigDataEditionInstallationDirectory>/ Informatica. The directory containing a collection of Hive and Hadoop JARS on the cluster from the RPM Install locations. The directory contains the minimum set of JARS required to process Informatica mappings in a Hadoop environment. Type / <BigDataEditionInstallationDirectory>/ Informatica/services/shared/hadoop/ cloudera_cdh5u2. The Hadoop distribution directory on the Data Integration Service node. The contents of the Data Integration Service Hadoop distribution directory must be identical to Hadoop distribution directory on the data nodes. Type / <BigDataEditionInstallationDirectory>/ Informatica/services/shared/hadoop/ cloudera_cdh5u2. Known Limitations The following table describes known limitations: CR Description When a data domain discovery profile runs as a part of enterprise discovery on Cloudera CDH 5.3, the profile fails with the following error: Ambiguous column reference a2 This error is caused by a limitation in Hive For more information, see the following issue: Workaround: Run the profile in the Native run-time environment Author Big Data Edition Team 6