New Features and Enhancements in Big Data Management 10.2

Transcription

1 New Features and Enhancements in Big Data Management 10.2 Copyright Informatica LLC Informatica, the Informatica logo, Big Data Management, and PowerCenter are trademarks or registered trademarks of Informatica LLC in the United States and many jurisdictions throughout the world. A current list of Informatica trademarks is available on the web at trademarks.html

2 Abstract This article is written for all Informatica Big Data Management users. The following sections describe the new features and enhancements in Informatica Big Data Management for Table of Contents Overview Hadoop Ecosystem Cloud Platform Ease of Use Overview Version 10.2 improves the simplicity and stability of Big Data Management. Informatica focuses on four major feature categories for Big Data Management 10.2: Hadoop Ecosystem Cloud Platform Ease of Use Informatica has enhanced support for all Hadoop distributions. Informatica is making Big Data Management a key player in the Amazon and Azure ecosystems. Overall connectivity is improved in version 10.2 with support for Amazon S3, RedShift, Azure DW, Blob, and other ecosystems. Informatica always aims to improve the stability and performance of the core Big Data Management platform. With improved mapping concurrency, support for hierarchical data and stateful computing, and enhanced transformation support on the Blaze and Spark engines, the overall quality and stability of Big Data Management has significantly improved in version A central objective for version 10.2 was to improve the efficiency and simplicity of Big Data Management. To address this issue, Informatica has introduced an automatic installation feature with one-step Hadoop integration. Hadoop Ecosystem This section describes updates to the Hadoop ecosystem in version Distribution Supported Version Amazon EMR 5.4 Azure HDInsight Cloudera CDH Hortonworks HDP 3.6.x 5.11.x 2.6.x 2

3 Distribution IBM BigInsights Supported Version 4.2.x MapR MEP 2.x To see a list of the latest supported versions, see the Product Availability Matrix on the Informatica Customer Portal: Cloud This section describes enhanced support for the Amazon and Azure econsystems in version Support for Amazon Redshift and Amazon S3 You can read data from or write data to the Amazon S3 buckets in the following regions: - Asia Pacific (Mumbai) - Asia Pacific (Seoul) - Canada (Central) - China (Beijing) - EU (London) - US East (Ohio) You can run Amazon Redshift and Amazon S3 mappings on the Spark engine. When you run the mapping, the Data Integration Service pushes the mapping to a Hadoop cluster and processes the mapping on the Spark engine, which significantly increases the performance. You can use AWS Identity and Access Management (IAM) authentication to securely control access to Amazon S3 resources and to run a mapping on the EMR cluster. Support for Microsoft Azure Blob Storage You can read data from and write data to a subdirectory in Microsoft Azure Blob Storage. Support for Microsoft Azure SQL Data Warehouse You can run Microsoft Azure SQL Data Warehouse mappings in a Hadoop environment on Kerberos enabled clusters. Support for Azure Data Lake Store You can read data from and write data to Azure Data Lake Store on the Spark engine. Platform This section describes new big data platform features and enhancements in version PowerCenter Reuse Report Enhancement of PowerCenter Reuse Report The PowerCenter Reuse Report estimates how many PowerCenter mappings can be reused in the Model repository for a native or Hadoop environment. The stability of the report has been improved. The report also contains more details about mapping errors and failures. For large repositories, you can control the number of mappings evaluated concurrently in the report. 3

4 Enhancement of PowerCenter Import You can now import sessions, multiple pipelines, workflows, and worklets from PowerCenter into the Model repository. Sessions within a workflow are imported as Mapping tasks in the Model repository. Worklets within a workflow are imported as objects within the workflow in the Model repository. Multiple pipelines within a mapping are imported as separate mappings in the Model repository based on the target load order. Data Integration Service Concurrency and Queuing If you deploy multiple mapping jobs or workflow mapping tasks at the same time, the Data Integration Service queues the jobs in a persisted queue and runs the jobs when resources are available. In a grid environment, each node has its own queue. By queuing job requests, the Data Integration Service can handle thousands of requests in parallel based on resource availability. Informatica also implemented concurrency enhancements for the Blaze, Spark, and Hive engines. You can now run more jobs in parallel with the same amount of hardware resources. Engine Adoption Informatica has introduced a new command that allows you to set the validation and execution environments for an entire project. You can also make changes to deployed mappings without having to redeploy them. These enhancements allow you to quickly adopt the Blaze and Spark engines. Blaze Engine Effective in version 10.2, mappings that run on the Blaze engine can read from and write to bucketed and sorted sources and targets. Spark Engine The following enhancements were made to the Spark engine in version 10.2: Hierarchical Data You can use complex data types, such as array, struct, and map, in mappings that run on the Spark engine. With complex data types, the Spark engine directly reads, processes, and writes hierarchical data in Avro, JSON, and Parquet complex files. Develop mappings with complex ports, operators, and functions to perform the following tasks: Generate and modify hierarchical data. Transform relational data to hierarchical data. Transform hierarchical data to relational data. Convert data from one complex file format to another. Stateful Computation You can perform stateful computations on the Spark engine with the new windowing functionality. You can use window functions in an Expression transformation to perform stateful calculations on the Spark engine. Window functions operate on a group of rows and calculate a single return value for every input row. You can use window functions to perform the following tasks: Retrieve data from previous or subsequent rows. Calculate a cumulative sum based on a group of rows. Calculate a cumulative average based on a group of rows. 4

5 Transformations The Spark engine now supports the following transformations in version 10.2: Lookup Normalizer Rank Update Strategy for Hive Ease of Use This section describes new features affecting big data usability in version Zero-Footprint Hadoop Installation With zero-footprint installation, you no longer need to install the Big Data Management binaries on the Hadoop data nodes to integrate the Informatica domain with the Hadoop cluster. This process leverages the industry-standard YARN distributed cache to transfer the files through the Informatica Hadoop staging directory on HDFS. When you apply emergency bug fixes and hotfixes on the Informatica server and restart the Data Integration Service, the next job that you run accounts for all the changes that you made to your environment. Cluster Configuration Object Informatica has released a new object called the cluster configuration. The cluster configuration simplifies the integration between the Informatica environment and the Hadoop cluster. The object represents the configuration of the Hadoop cluster, and it enables the Data Integration Service to push mapping logic to the Hadoop cluster. When you run the Cluster Configuration wizard, you can choose to create Hadoop, HBase, Hive, and HDFS connections. The wizard creates the connections based on information from the cluster. It also associates the HBase, Hive, and HDFS connections with the cluster configuration You can refresh the cluster configuration in one step when the Hadoop cluster changes, so that the Informatica environment is in-sync with the configuration of the Hadoop environment. The import process imports values from *-site.xml files into configuration sets based on the individual *-site.xml files. You no longer need to maintain *-site.xml files to enable the Informatica domain to communicate with the cluster. Authors Elizabeth Snyder Technical Writer 5