Creating an Avro to Relational Data Processor Transformation

Transcription

1 Creating an Avro to Relational Data Processor Transformation 2014 Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. All other company and product names may be trade names or trademarks of their respective owners and/or copyrighted materials of such owners.

2 Abstract A Data Processor transformation can convert data from Avro into relational format. After you create a Data Processor transformation for Avro input, you add it to a mapping with a complex file reader. The complex file reader passes Avro input to the transformation. This article describes how to create the Data Processor transformation, add it to a mapping with a complex file reader, and then configure the complex file reader to pass Avro input to the transformation. Supported Versions Data Transformation HotFix 1 Table of Contents Overview Creating an Avro to Relational Data Processor Transformation Overview Prerequisites Step 1. Create a Data Processor Transformation Step 2. Edit the Avro to Relational Mapping Step 3. Add the Transformation to a Mapping Step 4. Configure the Complex File Reader Step 5. Run the Mapping Overview You can create a Data Processor transformation and select the input and output formats through the Data Processor transformation wizard. Use the wizard to create a transformation with Avro input and relational output. When you create a Data Processor transformation to transform the Avro format, you select an Avro schema or example file that defines the expected structure of the Avro data. Apache Avro is a data serialization system in binary or other data formats. Avro data is in a format that might not be directly human-readable. For more information about Avro, see Note: Use binary encoded Avro to create a transformation with Avro input. Avro input in other formats is not processed. The wizard creates components that transform Avro format to other formats. These components can include a relational to hierarchical mapping, a hierarchical to relational mapping, and an XMap. After the wizard creates the transformation, you can further configure the transformation to determine the mapping logic. The Data Processor transformation creates a transformation solution according to the formats selected and the specification file, the example file, or the copybook. The transformation might not be complete, but it contains components that you connect and customize to complete the transformation definition. When you create a transformation that transforms Avro to relational format, or relational format to Avro, the wizard creates a relational mapping. The Ports panel in the Overview view shows the Avro hierarchical schema nodes and the relational ports. Use the Ports panel to link the hierarchical elements to the relational ports and groups. After you create a Data Processor transformation for Avro input, you add it to a mapping with a complex file reader. The complex file reader passes Avro input to the transformation. 2

3 Creating an Avro to Relational Data Processor Transformation Overview You can generate a Data Processor transformation with the transformation wizard. Configure the Avro to relational transformation, then add it to a mapping. Add a complex file reader to the mapping and configure the complex file reader. 1. In the Developer, create a Data Processor transformation using the transformation wizard. 2. If needed, edit the auto-generated Avro to relational transformation in the Ports panel of the Overview view. 3. Add the transformation to a mapping. 4. Add a complex file reader to the mapping. 5. Configure the complex file reader. 6. Test and run the mapping. Prerequisites You can read compressed Avro data with the complex file reader. If you use the Snappy codec for Avro data compression, you must update the Snappy codec.jar file before you test or run the transformation. To use the Snappy codec, replace the default Snappy.jar file in the Informatica server installation and in the Hadoop environment with the updated version. The updated snappy-java jar file is available at the following link: Step 1. Create a Data Processor Transformation Create the Data Processor transformation in the Model repository. 1. In the Developer tool, click File > New > Transformation. 2. Select the Data Processor transformation and click Next. 3. Enter a name for the transformation and browse for a Model repository location to put the transformation. 4. Select Create a data processor using a wizard and click Next. 5. Select the Avro input format and click Next. 6. To add a schema file or sample files that defines the Avro input hierarchy, browse to and select the Avro schema file or sample Avro file. Click Next. The Developer tool adds a schema representing the Avro hierarchy to the Model repository. If you select a sample file, Developer creates a test file from the first record in the sample file. You can use this file to test the transformation. To find the file, in the Ports panel of the Overview view, check the path file listed in the Input Location field. 7. Select relational output format and click Next. 8. Click Finish. The Developer tool creates the transformation in the repository. The Overview view appears in the Developer tool. 3

4 Step 2. Edit the Avro to Relational Mapping After you create the Data Processor transformation, you can edit the Avro to relational transformation as needed. When the wizard auto-generates the transformation, the wizard might add extra groups and keys based on the structure of the schema or sample file. To avoid unexpected behavior, delete any unnecessary groups and keys. 1. To view the mapping, in the Overview view Ports area, select Output Mapping. 2. Expand the trees in the Ports grid. To the left, the Transformation input panel shows the expected hierarchical input, and to the right, the Transformation Transformation output panel shows the relational output. 3. To add an input group or port to the Transformation output area, use one of the following methods: Drag a simple or complex element in the Transformation input area to an empty column in the Transformation output area. If the node is a group node, the Developer tool adds a relational group without ports. To add a relational group, select a row and right-click to select New > Group. To add a relational port, right-click to select New > Field. 4. To clear the Avro hierarchical node settings for locations of ports, use one of the following methods: Select one or more nodes in the Transformation input area, right-click and select Clear. Select one or more lines that connect the relational output ports to the hierarchical nodes in the Transformation input area, right-click and select Delete. Step 3. Add the Transformation to a Mapping Add the Data Processor transformation to a mapping. 1. In the Object Explorer view, create a mapping, or select a mapping in the Explorer and select Open Mapping. 2. Drag the Data Processor transformation that you created into the editor. Step 4. Configure the Complex File Reader Add a complex file reader to the mapping. Configure the complex file reader to process Avro input. 1. In the Mapping editor, create a complex file reader object. 2. To configure the complex file reader, perform the following steps: a. In the Advanced tab of the Properties view, select the File Format property, then choose Input Format. b. In the Advanced tab, select the Input Format property, then type com.informatica.avro.avrotoxml. 3. Add the Data Processor transformation to the mapping. The transformation input port should remain set to binary input, the default setting for Avro input. 4. Link the complex file reader output port to the Data Processor transformation input port. The complex file reader output port should remain set to binary output. Step 5. Run the Mapping To test that the mapping works, run the mapping. u To test the mapping, right-click in the editor, and select Run Mapping. Review the target flat file to see the mapping results. 4

5 Author Rachel Bell Technical Writer 5