Talend Open Studio for Big Data. Release Notes 5.4.1

Transcription

1 Talend Open Studio for Big Data Release Notes 5.4.1

2 Talend Open Studio for Big Data Publication date December 12, 2013 Copyleft This documentation is provided under the terms of the Creative Commons Public License (CCPL). For more information about what you can and cannot do with this documentation in accordance with the CCPL, please read: Notices All brands, product names, company names, trademarks and service marks are the properties of their respective owners.

3 Table of Contents System Requirements... 1 Big Data: New Features Kerberos security Upgraded support for Hadoop distributions Hadoop file formats File management in HDFS NoSQL databases In-memory technology Cloud technology Demo project Other features... 3 Big Data: Bug Fixes / Change Log Bug Fixes... 4 Big Data: Known Issues Studio multi-instance starting issue Note for the developers of custom components... 5 Big Data: Hints and Notes Installing required third-party licences... 7 Documentation Talend Help Center Revised documents Known issues Open issues... 8

4 System Requirements System Requirements Users should refer to the Installation and Upgrade Guides on the Talend Help Center ( for more information on Installation and System Requirements. 1

5 Big Data: New Features Big Data: New Features 1. Kerberos security 1. The Kerberos kinit authentication mode has been enabled for all the Big Data components, including the Hive components. 2. Except to the HBase ones, the Kerberos keytab authentication mode has been added to all the Big Data components. 2. Upgraded support for Hadoop distributions 1. New versions of the following Hadoop distributions are supported: Hortonworks Data Platform 1.3 and 2.0 Cloudera 4.3 and 4.4 MapR and EMC Pivotal is now available. 3. Hadoop file formats Support for Sequencefile, RC, ORC and Avro has been added to several components: 1. The thivecreatetable and the thiveload components are created. They support not only a wide range of commonly used file formats such as Sequencefile, RC, ORC and Avro, but also the formats that are not officially supported by Talend. 2. In addition to their existing functions, tpigload and tpigstoreresult can now process a Sequencefile, RC or Avro file. 4. File management in HDFS 1. The tsqoopmerge component has been created for merging two datasets with newer records overwirting the older ones.. 2. Upgrade of HDFS components The thdfscopy component can now merge the part files generated at the end of a MapReduce computation. The input and the output components are enabled to handle header rows. The thdfsinput component can read sub-directories of a specified directory. 2

6 Big Data: New Features 5. NoSQL databases 1. The following components have been created to enable transactions with their related NoSQL databases: tcassandrabulkload, tcassandraoutputbulk, tcassandrabulkexec and tcassandraoutputbulkexec tmongodbbulkload The Riak components 2. The 2.4 and the 2.5 versions of MongoDB are now supported by its related components. 6. In-memory technology 1. The newly added SAP Hana components help users easily configure the connection to a SAP Hana system and process transactions with this in-memory computing platform. 7. Cloud technology 1. With the addition of support for Amazon S3 (Simple Storage Service), users can use dedicated components to perform transactions with this data storage service. 2. GS (Google Storage) components are now available for users to perform interactions with Google Storage and prepare their data before transferring the data to Google BigQuery. 8. Demo project 1. A Big Data demo project is provided with the Studio. The project includes a number of easy-to-use sample Jobs to help familiarize users with the various features and functions of Talend Studio with Big Data. 9. Other features 1. Support for OAuth2 security has been added to the Salesforce components. 2. With the addition of support for Amazon S3 (Simple Storage Service), users can use dedicated components to perform transactions with this data storage service. 3. The Vertica components now officially support Vertica 5.1 and Vertica

7 Big Data: Bug Fixes / Change Log Big Data: Bug Fixes / Change Log 1. Bug Fixes In addition to the above new features a number of minor improvements within the entire product and significant bug fixes have been made. See the corresponding Change Log on our bug tracking system for more details on the individual issues: 4

8 Big Data: Known Issues Big Data: Known Issues We encourage you to consult the JIRA bug tracking tool for a full list of open issues: Note that this list shows issues from both Talend's Community and Subscription products. 1. Studio multi-instance starting issue If you are using the open source version of the Studio and have tried to launch it twice or even more at the same time, the Studio might not be able to restart any more after you close all of its instances. 2. Note for the developers of custom components A new finally component template such as tfileoutputdelimited_finally.javajet has been created for processing the finally block. This change might provoke code compilation errors of a custom component when this component has been migrated to and is used there to process multiple outputs. Issue diagnostic: A custom components subject to this issue is typically developed with either of the following practices: 1. This custom component is written to open a try block in the begin part and close it in the end part. 2. This custom component is based on a duplicate of any of the following components released between (exclusive) and (exclusive). tfileoutputdelimited tsapoutput tbigqueryoutputbulk, tcassandraoutput, thbaseoutput, tmongodboutput, tmongodbwriteconf, tneo4joutput, tneo4joutputrelationship, tneo4jrow, triakoutput taccessoutputbulk, tbonitainstantiateprocess, tgreenplumoutputbulk, tinformixoutputbulk, tingresoutputbulk, tmssqloutputbulk, tmomoutput, tmysqloutputbulk, toraclebulkexec, toracleoutputbulk, tparacceloutputbulk, tpivottocolumnsdelimited, tpostgresplusoutputbulk, tpostgresqloutputbulk, tsalesforceoutputbulk, tsybaseoutputbulk, tverticaoutputbulk tgenkeyhadoopin, tgenkeyhadoopout, tmatchgrouphadoopin, tmatchgrouphadoopout tcollector, tdepartitioner, tpartitioner, trecollector Recommended solution: 1. Remove any try, catch or finally blocks from your begin and end parts. 2. Put any resources that you will need to use in your finally code in the new resourcemap variable. For example, resourcemap.put("resources_tfileoutputdelimited_1",object); 5

9 Big Data: Known Issues 3. Create a finally code template which will then be able to use objects from the resourcemap variable and close connections. The following links present a complete example for implementing this solution: 1. Modification of begin.javajet: 2. Modification of end.javajet: 3. Addition of a finally part: tsapoutput/tsapoutput_finally.javajet?rev=

10 Big Data: Hints and Notes Big Data: Hints and Notes 1. Installing required third-party licences Users must install certain required third-party libraries for all Talend products to work correctly. These libraries can be installed via the Modules View. 7

11 Documentation Documentation 1. Talend Help Center Find out more about how to get the most out of your Talend products on the Talend Help Center: help.talend.com. New articles for this release include: A Knowledge Base article providing a full list of the different Map/Reduce components: pages/viewpage.action?pageid= Revised documents In addition to updates to the content across the documentation set, the following specific documentation changes have been made. Talend Open Studio for MDM User Guide now includes parts describing how to work with the Integration and Profiling perspectives, as well as the MDM perspective. This guide merges the information contained in the Talend Open Studio for Data Integration User Guide and the Talend Open Studio for Data Quality User Guide with the previous standalone Talend Open Studio for MDM User Guide. Talend Big Data Studio Getting Started Guide has been renamed to Talend Big Data Getting Started Guide. A new chapter "Getting started with Talend Big Data using the demo project" has been added to the Talend Big Data Studio Getting Started Guide. This chapter provides short descriptions about the sample Jobs included in the demo project and introduces the necessary preparations to run the sample Jobs on a Hadoop platform. Talend Open Studio for ESB Mediation Components Reference Guide and Talend ESB Mediation Components Reference Guide have been merged into one guide, Talend ESB Mediation Components Reference Guide. In the ESB Getting Started Guide, the chapter "Downloading and installing Talend ESB software" is now called "Getting started with Talend ESB", and the demo chapters are now split into two categories ("Basic deployment and runtime use cases" and "Advanced deployment and runtime use cases with SOA Governance"). In the ESB Infrastructure Services Configuration Guide and the STS User Guide, some conceptual information has been added that was previously found in the ESB Getting Started Guide. 3. Known issues In the Talend ESB Mediation Components Reference Guide, the documentation for the cmap component does not specify that this component is only available with Talend Platform products. 4. Open issues We encourage you to consult the JIRA bug tracking tool for a full list of open issues: 8

12 Documentation 9