Informatica Intelligent Data Lake (Version 10.1) Installation and Configuration Guide

Transcription

1 Informatica Intelligent Data Lake (Version 10.1) Installation and Configuration Guide

2 Informatica Intelligent Data Lake Installation and Configuration Guide Version 10.1 August 2016 Copyright Informatica LLC 2015, 2016 This software and documentation contain proprietary information of Informatica LLC and are provided under a license agreement containing restrictions on use and disclosure and are also protected by copyright law. Reverse engineering of the software is prohibited. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica LLC. This Software may be protected by U.S. and/or international Patents and other Patents Pending. Use, duplication, or disclosure of the Software by the U.S. Government is subject to the restrictions set forth in the applicable software license agreement and as provided in DFARS (a) and (a) (1995), DFARS (1)(ii) (OCT 1988), FAR (a) (1995), FAR , or FAR (ALT III), as applicable. The information in this product or documentation is subject to change without notice. If you find any problems in this product or documentation, please report them to us in writing. Informatica, Informatica Platform, Informatica Data Services, PowerCenter, PowerCenterRT, PowerCenter Connect, PowerCenter Data Analyzer, PowerExchange, PowerMart, Metadata Manager, Informatica Data Quality, Informatica Data Explorer, Informatica B2B Data Transformation, Informatica B2B Data Exchange Informatica On Demand, Informatica Identity Resolution, Informatica Application Information Lifecycle Management, Informatica Complex Event Processing, Ultra Messaging, Informatica Master Data Management, and Live Data Map are trademarks or registered trademarks of Informatica LLC in the United States and in jurisdictions throughout the world. All other company and product names may be trade names or trademarks of their respective owners. Portions of this software and/or documentation are subject to copyright held by third parties, including without limitation: Copyright DataDirect Technologies. All rights reserved. Copyright Sun Microsystems. All rights reserved. Copyright RSA Security Inc. All Rights Reserved. Copyright Ordinal Technology Corp. All rights reserved. Copyright Aandacht c.v. All rights reserved. Copyright Genivia, Inc. All rights reserved. Copyright Isomorphic Software. All rights reserved. Copyright Meta Integration Technology, Inc. All rights reserved. Copyright Intalio. All rights reserved. Copyright Oracle. All rights reserved. Copyright Adobe Systems Incorporated. All rights reserved. Copyright DataArt, Inc. All rights reserved. Copyright ComponentSource. All rights reserved. Copyright Microsoft Corporation. All rights reserved. Copyright Rogue Wave Software, Inc. All rights reserved. Copyright Teradata Corporation. All rights reserved. Copyright Yahoo! Inc. All rights reserved. Copyright Glyph & Cog, LLC. All rights reserved. Copyright Thinkmap, Inc. All rights reserved. Copyright Clearpace Software Limited. All rights reserved. Copyright Information Builders, Inc. All rights reserved. Copyright OSS Nokalva, Inc. All rights reserved. Copyright Edifecs, Inc. All rights reserved. Copyright Cleo Communications, Inc. All rights reserved. Copyright International Organization for Standardization All rights reserved. Copyright ejtechnologies GmbH. All rights reserved. Copyright Jaspersoft Corporation. All rights reserved. Copyright International Business Machines Corporation. All rights reserved. Copyright yworks GmbH. All rights reserved. Copyright Lucent Technologies. All rights reserved. Copyright (c) University of Toronto. All rights reserved. Copyright Daniel Veillard. All rights reserved. Copyright Unicode, Inc. Copyright IBM Corp. All rights reserved. Copyright MicroQuill Software Publishing, Inc. All rights reserved. Copyright PassMark Software Pty Ltd. All rights reserved. Copyright LogiXML, Inc. All rights reserved. Copyright Lorenzi Davide, All rights reserved. Copyright Red Hat, Inc. All rights reserved. Copyright The Board of Trustees of the Leland Stanford Junior University. All rights reserved. Copyright EMC Corporation. All rights reserved. Copyright Flexera Software. All rights reserved. Copyright Jinfonet Software. All rights reserved. Copyright Apple Inc. All rights reserved. Copyright Telerik Inc. All rights reserved. Copyright BEA Systems. All rights reserved. Copyright PDFlib GmbH. All rights reserved. Copyright Orientation in Objects GmbH. All rights reserved. Copyright Tanuki Software, Ltd. All rights reserved. Copyright Ricebridge. All rights reserved. Copyright Sencha, Inc. All rights reserved. Copyright Scalable Systems, Inc. All rights reserved. Copyright jqwidgets. All rights reserved. Copyright Tableau Software, Inc. All rights reserved. Copyright MaxMind, Inc. All Rights Reserved. Copyright TMate Software s.r.o. All rights reserved. Copyright MapR Technologies Inc. All rights reserved. Copyright Amazon Corporate LLC. All rights reserved. Copyright Highsoft. All rights reserved. Copyright Python Software Foundation. All rights reserved. Copyright BeOpen.com. All rights reserved. Copyright CNRI. All rights reserved. This product includes software developed by the Apache Software Foundation ( and/or other software which is licensed under various versions of the Apache License (the "License"). You may obtain a copy of these Licenses at Unless required by applicable law or agreed to in writing, software distributed under these Licenses is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the Licenses for the specific language governing permissions and limitations under the Licenses. This product includes software which was developed by Mozilla ( software copyright The JBoss Group, LLC, all rights reserved; software copyright by Bruno Lowagie and Paulo Soares and other software which is licensed under various versions of the GNU Lesser General Public License Agreement, which may be found at The materials are provided free of charge by Informatica, "as-is", without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. The product includes ACE(TM) and TAO(TM) software copyrighted by Douglas C. Schmidt and his research group at Washington University, University of California, Irvine, and Vanderbilt University, Copyright ( ) , all rights reserved. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (copyright The OpenSSL Project. All Rights Reserved) and redistribution of this software is subject to terms available at and This product includes Curl software which is Copyright , Daniel Stenberg, <daniel@haxx.se>. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. The product includes software copyright ( ) MetaStuff, Ltd. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at license.html. The product includes software copyright , The Dojo Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at This product includes ICU software which is copyright International Business Machines Corporation and others. All rights reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software copyright Per Bothner. All rights reserved. Your right to use such materials is set forth in the license which may be found at kawa/software-license.html. This product includes OSSP UUID software which is Copyright 2002 Ralf S. Engelschall, Copyright 2002 The OSSP Project Copyright 2002 Cable & Wireless Deutschland. Permissions and limitations regarding this software are subject to terms available at This product includes software developed by Boost ( or under the Boost software license. Permissions and limitations regarding this software are subject to terms available at / This product includes software copyright University of Cambridge. Permissions and limitations regarding this software are subject to terms available at This product includes software copyright 2007 The Eclipse Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at and at

3 This product includes software licensed under the terms at license.html, httpunit.sourceforge.net/doc/ license.html, license.html, license-agreement; /copyright-software ; forge.ow2.org/projects/javaservice/, license.html; protobuf.googlecode.com/svn/trunk/src/google/protobuf/descriptor.proto; current/doc/mitk5license.html; blob/master/license; page=documents&file=license; blueprints/blob/master/license.txt; twbs/bootstrap/blob/master/license; master/license, and This product includes software licensed under the Academic Free License ( the Common Development and Distribution License ( the Common Public License ( the Sun Binary Code License Agreement Supplemental License Terms, the BSD License ( the new BSD License ( licenses/bsd-3-clause), the MIT License ( the Artistic License ( and the Initial Developer s Public License Version 1.0 ( This product includes software copyright Joe WaInes, XStream Committers. All rights reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software developed by the Indiana University Extreme! Lab. For further information please visit This product includes software Copyright (c) 2013 Frank Balluffi and Markus Moeller. All rights reserved. Permissions and limitations regarding this software are subject to terms of the MIT license. See patents at DISCLAIMER: Informatica LLC provides this documentation "as is" without warranty of any kind, either express or implied, including, but not limited to, the implied warranties of noninfringement, merchantability, or use for a particular purpose. Informatica LLC does not warrant that this software or documentation is error free. The information provided in this software or documentation may include technical inaccuracies or typographical errors. The information in this software and documentation is subject to change at any time without notice. NOTICES This Informatica product (the "Software") includes certain drivers (the "DataDirect Drivers") from DataDirect Technologies, an operating company of Progress Software Corporation ("DataDirect") which are subject to the following terms and conditions: 1. THE DATADIRECT DRIVERS ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. 2. IN NO EVENT WILL DATADIRECT OR ITS THIRD PARTY SUPPLIERS BE LIABLE TO THE END-USER CUSTOMER FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR OTHER DAMAGES ARISING OUT OF THE USE OF THE ODBC DRIVERS, WHETHER OR NOT INFORMED OF THE POSSIBILITIES OF DAMAGES IN ADVANCE. THESE LIMITATIONS APPLY TO ALL CAUSES OF ACTION, INCLUDING, WITHOUT LIMITATION, BREACH OF CONTRACT, BREACH OF WARRANTY, NEGLIGENCE, STRICT LIABILITY, MISREPRESENTATION AND OTHER TORTS. Part Number: IDL-INS

4 Table of Contents Preface Informatica Resources Informatica Network Informatica Knowledge Base Informatica Documentation Informatica Product Availability Matrixes Informatica Velocity Informatica Marketplace Informatica Global Customer Support Chapter 1: Introduction to Intelligent Data Lake Installation Intelligent Data Lake Installation Overview Live Data Map Installation Intelligent Data Lake Installation Installation Process Chapter 2: Before You Install Before You Install Overview Read the Release Notes Verify the License Key Install and Configure Live Data Map Before Installation During Installation After Installation Install Big Data Management Packages Create HDFS and Hive Connections for the Data Lake Verify System Requirements Verify Temporary Disk Space Requirements Verify Database Requirements Verify Services Installation Requirements Verify Hardware Requirements Setup the Database for Data Preparation Service Setup Keystore and Truststore Files Stop the Services and Domain Chapter 3: Intelligent Data Lake Installation Overview of the Intelligent Data Lake Installation Intelligent Data Lake Installation in Console Mode Intelligent Data Lake Installation in Silent Mode Configuring the Silent Install Properties File Table of Contents

5 Running the Silent Installer Secure the Passwords in the Properties File Chapter 4: After You Install Intelligent Data Lake After You install Overview Create the Application Services Install Python Enable Logging of User Activity Events Index Table of Contents 5

6 Preface The Intelligent Data Lake Installation and Configuration Guide contains information about the installation and setup of Intelligent Data Lake. It includes information about the Informatica domain requirements, system requirements, and the installation and configuration process. This guide assumes you have knowledge of databases and Hadoop clusters. This guide also assumes that you are familiar with your enterprise systems and network. Informatica Resources Informatica Network Informatica Network hosts Informatica Global Customer Support, the Informatica Knowledge Base, and other product resources. To access Informatica Network, visit As a member, you can: Access all of your Informatica resources in one place. Search the Knowledge Base for product resources, including documentation, FAQs, and best practices. View product availability information. Review your support cases. Find your local Informatica User Group Network and collaborate with your peers. As a member, you can: Access all of your Informatica resources in one place. Search the Knowledge Base for product resources, including documentation, FAQs, and best practices. View product availability information. Find your local Informatica User Group Network and collaborate with your peers. Informatica Knowledge Base Use the Informatica Knowledge Base to search Informatica Network for product resources such as documentation, how-to articles, best practices, and PAMs. To access the Knowledge Base, visit If you have questions, comments, or ideas about the Knowledge Base, contact the Informatica Knowledge Base team at KB_Feedback@informatica.com. 6

7 Informatica Documentation To get the latest documentation for your product, browse the Informatica Knowledge Base at If you have questions, comments, or ideas about this documentation, contact the Informatica Documentation team through at Informatica Product Availability Matrixes Product Availability Matrixes (PAMs) indicate the versions of operating systems, databases, and other types of data sources and targets that a product release supports. If you are an Informatica Network member, you can access PAMs at Informatica Velocity Informatica Velocity is a collection of tips and best practices developed by Informatica Professional Services. Developed from the real-world experience of hundreds of data management projects, Informatica Velocity represents the collective knowledge of our consultants who have worked with organizations from around the world to plan, develop, deploy, and maintain successful data management solutions. If you are an Informatica Network member, you can access Informatica Velocity resources at If you have questions, comments, or ideas about Informatica Velocity, contact Informatica Professional Services at ips@informatica.com. Informatica Marketplace The Informatica Marketplace is a forum where you can find solutions that augment, extend, or enhance your Informatica implementations. By leveraging any of the hundreds of solutions from Informatica developers and partners, you can improve your productivity and speed up time to implementation on your projects. You can access Informatica Marketplace at Informatica Global Customer Support You can contact a Global Support Center by telephone or through Online Support on Informatica Network. To find your local Informatica Global Customer Support telephone number, visit the Informatica website at the following link: If you are an Informatica Network member, you can use Online Support at About the Intelligent Data Lake Installation and Configuration Guide 7

8 C H A P T E R 1 Introduction to Intelligent Data Lake Installation This chapter includes the following topics: Intelligent Data Lake Installation Overview, 8 Live Data Map Installation, 9 Intelligent Data Lake Installation, 9 Installation Process, 10 Intelligent Data Lake Installation Overview Intelligent Data Lake is a collaborative self-service Big Data discovery and preparation solution for data analysts and data scientists. It enables analysts to rapidly discover and turn raw data into insight and allows IT to ensure quality, visibility, and governance. With Intelligent Data Lake, analysts to spend more time on analysis and less time on finding and preparing data. Intelligent Data Lake provides the following benefits: Data analysts can quickly and easily find and explore trusted data assets within the data lake and outside the data lake using semantic search and smart recommendations. Data analysts can transform, cleanse, and enrich data in the data lake using an Excel-like spreadsheet interface in a self-service manner without the need for coding skills. Data analysts can publish data and share knowledge with the rest of the community and analyze the data using their choice of BI or analytic tools. IT and governance staff can monitor user activity related to data usage in the lake. IT can track data lineage to verify that data is coming from the right sources and going to the right targets. IT can enforce appropriate security and governance on the data lake IT can operationalize the work done by data analysts into a data delivery process that can be repeated and scheduled. Intelligent Data Lake requires an Informatica domain and Live Data Map. Intelligent Data Lake uses Live Data Map to search for data and discover data lineage and relationships. You must install Live Data Map and create and configure the Live Data Map services before you install Intelligent Data Lake. For more information, see the Live Data Map Installation and Configuration Guide. 8

9 Complete the pre-installation tasks to prepare for the installation. You can install the Intelligent Data Lake services on a machine with Red Hat Enterprise Linux version 6 or higher. You can run the installer in console or silent mode. Live Data Map Installation Informatica provides an installer that installs Informatica services and Live Data Map. You must install Live Data Map 2.0 on an external cluster and configure the services before you install Intelligent Data Lake. Note: You must select the external cluster option during Informatica Live Data Map installation. Intelligent Data Lake cannot use an internal Live Data Map Hadoop cluster. The installer installs Live Data Map with the option to create the following services: 1. Data Integration Service 2. Model Repository Service 3. Catalog Service 4. Content Management Service must be configured if you want to use the data domain discovery feature. For more information about Live Data Map installation, see the Live Data Map Installation and Configuration Guide. Intelligent Data Lake Installation Before you install Intelligent Data Lake, you must have an Informatica domain with Live Data Map and the following services: Data Integration Service Model Repository Service Catalog Service Use the Intelligent Data Lake installer to install the application. You can install Intelligent Data Lake in the following modes: Console Mode Silent Mode You can choose to create the following services during installation: Data Preparation Service Intelligent Data Lake Service If you do not want to create the services during installation, you can create these services after installation using the Administrator tool. Note: During Intelligent Data Lake installation, the files are copied to the Live Data Map installation directory. To uninstall Intelligent Data Lake, you must uninstall Live Data Map. For more information, see the Live Data Map Installation and Configuration Guide. Live Data Map Installation 9

10 Installation Process The installation process for Intelligent Data Lake consists of the following phases: 1. Install and configure Live Data Map 2. Create HDFS and Hive connections for the data lake 3. Verify system requirements 4. Setup the database for Data Preparation Service 5. Setup Keystore and Truststore files 6. Install Intelligent Data Lake 7. Complete any required post-install configuration 10 Chapter 1: Introduction to Intelligent Data Lake Installation

11 C H A P T E R 2 Before You Install This chapter includes the following topics: Before You Install Overview, 11 Read the Release Notes, 11 Verify the License Key, 12 Install and Configure Live Data Map, 12 Install Big Data Management Packages, 14 Create HDFS and Hive Connections for the Data Lake, 14 Verify System Requirements, 14 Setup the Database for Data Preparation Service, 16 Setup Keystore and Truststore Files, 16 Stop the Services and Domain, 16 Before You Install Overview You can install Intelligent Data Lake with Informatica services on Red Hat Enterprise Linux 6 or higher machine. Before you start the installation on a Red Hat Enterprise Linux 6 or higher machine, set up the machine to meet the requirements to install and run Intelligent Data Lake. If the machine where you install Intelligent Data Lake is not configured correctly, the installation can fail. Read the Release Notes Read the Informatica Release Notes for updates to the installation process. You can also find information about known and fixed limitations for the release. 11

12 Verify the License Key Before you install the software, verify that you have the license key available. You can get the license key in the following ways: Installation DVD. If you receive the Informatica installation files in a DVD, the license key file is included in the Informatica License Key CD. FTP download. If you download the Informatica installation files from the Informatica Electronic Software Download (ESD) site, the license key is in an message from Informatica. Copy the license key file to a directory accessible to the user account that installs the product. You can provide the license key for Intelligent Data Lake when you install Live Data Map. You cannot specify the license during Intelligent Data Lake installation. Ensure that you have the following license options enabled: Hive option for the Data Integration Service Live Data Map option for the Catalog Service Data Lake option for the Data Preparation Service and the Intelligent Data Lake Service Contact Informatica Global Customer Support if you do not have a license key or if you have an incremental license key and you want to install Intelligent Data Lake. If you use an incremental license for Intelligent Data Lake, the serial number of the incremental license must match the serial number for an existing license object in the domain. If the serial numbers do not match, the AddLicense command fails. You can get more information about the contents of the license key file used for installation, including serial number, version, expiration date, operating systems, and connectivity options in the installation debug log. Install and Configure Live Data Map Intelligent Data Lake uses Live Data Map to search for data and discover data lineage and relationships. You must install Live Data Map and create and configure the Live Data Map services before you install Intelligent Data Lake. Before Installation Verify the system requirements and prerequisites for Live Data Map installation. The license details for Intelligent Data Lake can be provided when you install Live Data Map. You cannot specify the license during Intelligent Data Lake installation. Ensure that you have the following license options enabled for Intelligent Data Lake: Hive option for the Data Integration Service Live Data Map option for the Catalog Service Data Lake option for the Data Preparation Service and the Intelligent Data Lake Service 12 Chapter 2: Before You Install

13 During Installation Install Live Data Map 2.0 with external cluster option. You must select the external cluster option during Informatica Live Data Map installation. Internal cluster is not supported for Intelligent Data Lake. For more information, see the Live Data Map Installation and Configuration Guide. You can choose to create the following services during the Live Data Map installation or you can also create them after installation in the following order using the Administrator tool. For more information, see the Live Data Map Administrator Guide. Data Integration Service. Note: If you plan to use the operating system profiles option for the Data Integration Service, ensure that you create and associate a different Data Integration Service for Live Data Map and Intelligent Data Lake. Live Data Map does not support operating system profiles. For more information, see the Intelligent Data Lake Administrator Guide. Model Repository Service Catalog Service Note: Service notifications for Catalog Service is not supported in Intelligent Data Lake. You must disable Service notifications for Catalog Service before you install Intelligent Data Lake. Content Management Service must be configured if you want to use the data domain discovery feature. After Installation After you install Live Data Map, complete the following tasks: Create a Hive resource for the data lake. For more information about creating a Hive resource for Intelligent Data Lake, see the Live Data Map Administrator Guide. You must create the Hive resource with the following specific settings for Intelligent Data Lake: - In the General > Connection Properties > Url field: Ensure that you use the Fully Qualified Domain Name (FQDN) of the Hive server in the JDBC connection URL used to access the Hive server. - In the Metadata Load Settings > Additional Properties > Schema field: By default, all the schemas available in the Hadoop cluster will be selected. You must select only the schemas required for Intelligent Data Lake. - If you are using operating system profiles, the Hive user name you specify in the Hive resource must be an HDFS superuser. For more information about operating system profiles, see the Intelligent Data Lake Administrator Guide. To extract metadata from the Hive sources, you need to import the relevant connectors, modify the scannerdeployer.xml file, and then restart the Catalog Service. For more information, see How To Configure Scanner Deployer for Hive in Live Data Map. If you are using operating system profiles, create a new Data Integration Service for Intelligent Data Lake using the Administrator tool. Install and Configure Live Data Map 13

14 Install Big Data Management Packages To install Big Data Management on Cloudera or HortonWorks, the tar.gz file includes the Big Data Management packages and binary files that you need. Intelligent Data Lake uses the pushdown mode to run mappings for data upload and publish. For running the mappings in pushdown mode, you must install Big Data Management packages on all nodes of the Hadoop cluster. For more information, see the Big Data Management Installation and Configuration Guide. Create HDFS and Hive Connections for the Data Lake If you are installing Intelligent Data Lake in console mode, you must use the Big Data Management Configuration Utility to create the HDFS and Hive connections for Intelligent Data Lake. This step can be skipped if you are installing Intelligent Data Lake in silent mode. If you are configuring for high availability, run the Big Data Management Utility on each node where you install the Data Preparation Service and the Intelligent Data Lake Service. The Big Data Management Configuration Utility will update the Hadoop configuration files. For more information, see the Big Data Management Installation and Configuration Guide.You must create the HDFS and Hive connections with the following specific requirements for Intelligent Data Lake: For the Hadoop distribution version, select either Cloudera CDH or Hortonworks HDP. - If you select Cloudera CDH, select Cloudera Manager to access files on the Hadoop cluster. - If you select Hortonworks HDP, select Apache Ambari to access files on the Hadoop cluster. Select the Hive on MapReduce option for running the mappings. In the Cluster Configuration Connection type selection panel: select No to use the default Hive command Line Interface to run mappings. Hive connection: - Connection Details Panel: for the Metastore Execution Mode, select remote. HDFS connection: - Connection Details Panel: in the HDFS username field, enter the user name with superuser privileges. The user must have access to all the Intelligent Data Lake schemas. Verify System Requirements Verify that your planned setup meets the minimum system requirements for the Intelligent Data Lake installation process, temporary disk space, databases, and application service hardware. For more information about product requirements and supported platforms, see the Product Availability Matrix on the Informatica Customer Portal: 14 Chapter 2: Before You Install

15 Verify Temporary Disk Space Requirements The installer writes temporary files to the hard disk. Verify that you have enough available disk space on the machine to support the installation. When the installation completes, the installer deletes the temporary files and releases the disk space. The installer requires 2 GB of temporary disk space. Verify Database Requirements Verify that the database server has adequate disk space for the databases required by the Data Preparation Service. The following table describes the database requirements for the data preparation repository database: Database Data Preparation Repository Database Requirements Create the Data Preparation Repository database on a MySQL database. Allow 5 GB of disk space for the database. Allocate more space based on the amount of metadata you want to store. Verify Services Installation Requirements Verify that your machine meets the minimum system requirements to install the Intelligent Data Lake services. The following table lists the processor, minimum memory and disk space required to install the Intelligent Data Lake services. Services RAM Disk Space Disk space for local storage Data Preparation Service Approximately 512 MB per user and 4 GB for the server. Allocate more space as required. 60 GB 10 GB Verify Hardware Requirements Verify that your machine meets the minimum hardware requirements to install the Intelligent Data Lake services. The following table lists the hardware required to install the Intelligent Data Lake services. Services Data Preparation Service Processor 2 CPUs with a minimum of 4 cores Recommended: 2 CPUs with 8 cores Verify System Requirements 15

16 Setup the Database for Data Preparation Service Set up the MySQL server database that the Data Preparation Service connects to. The MySQL database must meet the following requirements: You can use MySQL version or higher. - For MySQL version and higher, set lower_case_table_names=1. - For MySQL version 5.7 and higher, set explicit_defaults_for_timestamp=1. The database must have the following permissions: - Create tables and views. - Drop tables and views. - Insert, update and delete data. Setup Keystore and Truststore Files When you install Live Data Map, you can configure secure communication for the domain and specify the location of the keystore files for the security certificates. The domain must have keystore and truststore files named infa_keystore and infa_truststore in PEM and JKS format. For more information see the Live Data Map Installation and Configuration Guide. If the domain is secure, you must secure the services that you create in Intelligent Data Lake. The following services in the domain and the YARN application must share the same common truststore file: - Data Integration Service - Model Repository Service - Catalog Service - Data Preparation Service - Intelligent Data Lake Service The Data Preparation Service and Intelligent Data Lake Service must also share the same keystore file. You can use different keystore files for the Data Integration Service, Model Repository Service, and Catalog Service. If you use different keystore files, you must add certificates corresponding to each of the keystores into a common truststore file. If you have configured Intelligent Data Lake with one primary node and one or more backup nodes, you must copy the truststore files to a common directory and specify the same directory path for all nodes assigned to Intelligent Data Lake. Stop the Services and Domain Stop the following services before you install Intelligent Data Lake: 1. Data Integration Service 16 Chapter 2: Before You Install

17 2. Model Repository Service 3. Catalog Service 4. Content Management Service, if you have configured it for the Profiling feature 5. Data Preparation Service After you have stopped the services, stop the domain and then proceed to install Intelligent Data Lake. Stop the Services and Domain 17

18 C H A P T E R 3 Intelligent Data Lake Installation This chapter includes the following topics: Overview of the Intelligent Data Lake Installation, 18 Intelligent Data Lake Installation in Console Mode, 18 Intelligent Data Lake Installation in Silent Mode, 25 Overview of the Intelligent Data Lake Installation You can install the Intelligent Data Lake application on a Red Hat Enterprise Linux 6 or higher machine in console or silent mode. Complete the pre-installation tasks to prepare for the installation. You can install the Intelligent Data Lake services on multiple machines. You must configure and enable the Data Preparation Service before you create the Intelligent Data Lake Service. Note: It is recommended to use the silent mode installation for Intelligent Data Lake. Intelligent Data Lake Installation in Console Mode You can install the Intelligent Data Lake services in console mode on Linux. When you run the installer in console mode, the words Quit and Back are reserved words. Do not use them as input text. 1. Log in to the machine with a system user account. 2. Close all other applications. 3. On a shell command line, run the install.sh file from the root directory. The installer displays the message for documentation and copyright information. 4. Press Enter to continue. The installer displays the message about the prerequisites and pre-installation tasks. 5. If the prerequisites or pre-installation tasks are not completed, press n to exit the installer and set them as required. If the prerequisites and pre-installation tasks are completed, press y to continue. 6. Press Enter to continue. 7. Type the path for the Live Data Map installation directory. 18

19 The directory names in the path must not contain spaces or the following special * $ #! % ( ) { } [ ], ; '. Intelligent Data Lake and Live Data Map must be installed in the same directory. 8. Review the pre-installation summary, and press Enter to continue. The installer copies the Intelligent Data Lake files to the installation directory. 9. To install Intelligent Data Lake services on the master gateway node, press 1. To install Informatica Intelligent Data Lake services on any other node, press Specify the domain and node details. The following describes the domain and node parameters you must set: Property Domain Name Node Name Domain User Name Domain User Password Name of the domain created during Live Data Map installation. Name of the node where you want to install Intelligent Data Lake. If you are installing Intelligent Data Lake for the first time, you must install it on the master gateway node. The subsequent installs can be on any node. User name for the domain administrator. Password for the domain administrator. The password must be more than 2 characters and must not exceed 16 characters. 11. Choose whether to create services during installation or after installation from the Administrator tool. If the machine on which you are installing Intelligent Data Lake has 16 GB of RAM or less, Informatica recommends that you create the Data Preparation Service and the Intelligent Data Lake Service on different nodes. If you create the services on different nodes, you must create the Data Preparation Service on the master gateway node. You can create the Intelligent Data Lake Service on any other node. Option 1->Data Preparation Service 2->Intelligent Data Lake Service To create the Data Preparation Service, press 1. To create the Intelligent Data Lake Service, press 2. Ensure that the Data Preparation Service is already created and enabled because you must associate a Data Preparation Service with the Intelligent Data Lake Service. 3->Both To create the Data Preparation Service and the Intelligent Data Lake Service, press 3. 4->None If you do not want to create the Data Preparation Service and Intelligent Data Lake Service during installation, press 4. You can create the services from the Administrator tool. 12. To configure Hadoop configuration files, Hive and HDFS connections for the Data Integration Service or Data Preparation Service on this node, press 2. If you create the Hive connection from the installer, you must replace the _HOST parameter in the Metadata Connection String and Data Access Connection String fields for the Hive connection with the actual host name of the server on which HiveServer2 is running. If you have already configured the HDFS and Hive connections for Intelligent Data Lake and do not want to create the connections during installation, press 1. Intelligent Data Lake Installation in Console Mode 19

20 13. Select the Hadoop distribution of the Hadoop cluster for data preparation storage. To select Cloudera, press 1 or to select HortonWorks, press Specify the details for the services associated with Intelligent Data Lake. The following table describes the service parameters you must set: Property Catalog Service Name Model Repository Service Name Data Integration Service Name Name of the Catalog Service associated with the Intelligent Data Lake Service. Name of the Model Repository Service associated with the Intelligent Data Lake Service. Name of the Data Integration Service associated with the Intelligent Data Lake Service. 15. To enable the Content Management Service, press 2. Specify the Content Management Service name and press Enter. 16. Specify the Hadoop distribution details for Cloudera or Hortonworks based on your selection in step 13. Note: If you have not configured the Hadoop cluster using Cloudera Manager or Ambari server, you must run the Big Data Management Configuration Utility from the following location: <Informatica installation directory>/tools/bdeutil, on all nodes where the Data Preparation Service and Intelligent Data Lake Service are configured. The following table describes the options for the Ambari server: Property Ambari Host Name Ambari User Name Ambari User Password Ambari Port Number Host where the Ambari server runs. User name for the Ambari server. Password for the Ambari server user. Port number for the Ambari server. The following table describes the options for the Cloudera Manager: Property Cloudera Host Name Cloudera User Name Cloudera User Password Cloudera Port Number Host where the Cloudera Manager runs. User name for the Cloudera Manager. Password for the Cloudera Manager user. Port for the Cloudera Manager. 17. Specify the data lake options. 20 Chapter 3: Intelligent Data Lake Installation

21 The following table describes the data lake options you must set: Property Hive Connection Impersonation User Name HDFS Connection HDFS User Name Hive connection for the data lake. User name to use in Hadoop impersonation as set in core-site.xml. HDFS connection for the data lake. Name of the Hadoop Impersonation user used by Intelligent Data Lake. This user must exist in the Hadoop Cluster. 18. To update the Data Integration Service properties, press 1. The Data Integration Service must be present in the Informatica domain. If the Hadoop cluster uses Kerberos authentication, then copy the krb5.conf file to the {INFA_HOME}/services/shared/security folder on the machine where the Data Integration Service is configured. The Data Integration Service will restart after you update the properties. The following table describes the Data Integration Service parameters you must set: Property Data Integration Service Name Informatica Home Directory on Hadoop Hadoop Kerberos Service Principal Name Hadoop Kerberos Keytab Location Name of the Data Integration Service associated with the Intelligent Data Lake Service. The Informatica Hadoop home directory on every data node created by the Hadoop RPM installation. The default is /opt/informatica. This field will be displayed if the Hadoop cluster uses Kerberos authentication. Service Principal Name (SPN) of the Data Integration Service to connect to a Hadoop cluster that uses Kerberos authentication. This field will be displayed if the Hadoop cluster uses Kerberos authentication. Path and file name of the Kerberos keytab file on the machine on which the Data Integration Service runs. If you do not want to update the Data Integration Service properties, press If you selected to create the Data Preparation Service or both the Data Preparation Service and the Intelligent Data Lake Service, follow step 19 to step 29. The data preparation repository requires a MySQL database. Enter the details for Data Preparation repository database: Property Database Host Name Database User Name Database User Password Host name of the machine that hosts the Data Preparation repository database. Database user account to use to connect to the Data Preparation repository. Password for the Data Preparation repository database user account. Intelligent Data Lake Installation in Console Mode 21

22 Property Database Port Number Database name Port number for the database. Schema or database name of the Data Preparation repository database. 20. Specify the Data Preparation Service details: Property Data Preparation Service Name Node Name License Name of the Data Preparation Service. The name is not case sensitive and must be unique within the domain. It cannot exceed 128 characters or begin It also cannot contain spaces or the following special characters: ` ~ % ^ * + = { } \ ; : ' " /?., < >! ( ) ] [ Name of the node on which the Data Preparation Service runs. License object with the data lake option that allows the use of the Data Preparation Service 21. To enable secure communication for the Data Preparation Service, press 1 and specify the HTTPS Port number. To disable secure communication, press 2 and skip to step Select the type of SSL certificates to use. You must use the same keystore and truststore files for the Data Preparation Service and the Intelligent Data Lake Service. You must import the certificate file to the truststore location for the Data Preparation Service and Intelligent Data Lake Service. You must also use the same truststore files for the following services: Data Integration Service Model Repository Service Catalog Service 23. Select the Hadoop distribution for the data lake. To select Cloudera, press 1 or to select Hortonworks, press Select the Hadoop authentication mode. To select NonSecure, press 1 and skip to step 25. To select Kerberos, press 2 and specify the following details: Property HDFS Principal Name Hadoop Impersonation User Name Kerberos Keytab File Service Principal Name (SPN) for the data preparation Hadoop cluster. Specify the service principal name in the following format: user/_host@realm. User name to use in Hadoop impersonation as set in core-site.xml. If the Hadoop cluster uses Kerberos authentication, the Hadoop impersonation user must have read, write, and execute permissions on the HDFS storage location folder. Path and file name of the SPN keytab file for the user account to impersonate when connecting to the Hadoop cluster. The keytab file must be in a directory on the machine where the Data Preparation Service runs. 22 Chapter 3: Intelligent Data Lake Installation

23 25. Specify the HDFS storage location, HDFS connection, local storage location, and Solr port number details: Property HDFS Storage Location HDFS Connection Local Storage Location Solr port HDFS location for data preparation file storage. If the Hadoop cluster uses Kerberos authentication, the Hadoop impersonation user must have read, write, and execute permissions on the HDFS storage location folder. HDFS connection for data preparation file storage. Directory for data preparation file storage on the node on which the Data Preparation Service runs. If the connection to the local storage fails, the Data Preparation Service recovers data preparation files from the HDFS storage location. Solr port number for the Apache Solr server used to provide data preparation recommendations. 26. To enable the Data Preparation Service immediately after you create the service, press 2. To enable the service at a later time from the Administrator tool, press If you selected to create the Intelligent Data Lake Service or both the Data Preparation Service and the Intelligent Data Lake Service, follow step 27 to step 34. Specify the Intelligent Data Lake Service details: Property Intelligent Data Lake Service Name Data Preparation Service Name Node Name License Name of the Intelligent Data Lake Service. Name of the Data Preparation Service associated with the Intelligent Data Lake Service. Node on which the Intelligent Data Lake Service runs. License object with the data lake option that allows the use of the Intelligent Data Lake Service. 28. To enable secure communication for the Intelligent Data Lake Service, press 2 and specify the HTTPS Port number. To disable secure communication for the Intelligent Data Lake Service, press Select the type of SSL certificates to use. You must use the same keystore and truststore files for the Data Preparation Service and the Intelligent Data Lake Service. You must import the certificate file to the truststore location for the Data Preparation Service and Intelligent Data Lake Service. You must also use the same truststore files for the following services: Data Integration Service Model Repository Service Catalog Service 30. Type the data lake resource name which you have created in Live Data Map and press Enter. 31. Specify the data lake connection options. Intelligent Data Lake Installation in Console Mode 23