Converting Relational Input into Hierarchical Output using Google BigQuery Connector

Transcription

1 Converting Relational Input into Hierarchical Output using Google BigQuery Connector Copyright Informatica LLC Informatica, the Informatica logo, and Informatica Cloud are trademarks or registered trademarks of Informatica LLC in the United States and many jurisdictions throughout the world. A current list of Informatica trademarks is available on the web at

2 Abstract When you use Google BigQuery Connector to read data from relational sources and write data to Google BigQuery, you can use the Hierarchical Builder transformation to convert relational input into hierarchical output. The transformation processes a relational input from the upstream transformation and provides JSON output to the downstream transformation. This article describes how to use the Hierarchical Builder transformation with Google BigQuery Connector to convert relational input into hierarchical output. Supported Versions Informatica Cloud Spring 2017 Table of Contents Overview Configuration Tasks Before You Begin Step 1. Configure a JSON Schema Definition File on the Secure Agent or Hosted Agent Step 2. Add a Source Transformation Add an Expression Transformation Step 3. Add a Hierarchical Builder Transformation to the Mapping Step 4. Associate the JSON Schema Definition File with the Hierarchical Builder Transformation Output Settings Step 5. Configure the Field Mapping to Select Schema Elements Selecting a Primary or Foreign Key Selecting Schema Elements Step 6. Add a Target Transformation Step 7. Configure a Mapping Configuration Task Overview When you use Google BigQuery Connector to read data from relational sources and write data to Google BigQuery, you can use the Hierarchical Builder transformation to convert relational input into hierarchical output. If you configure the Google BigQuery connection to use complex connection mode, the Secure Agent generates a JSON schema definition file in the directory that you specify. The schema definition file represents the schema of the Google BigQuery target object. Create and configure the Hierarchical Builder transformation in Informatica Cloud and add it to a mapping. Then, associate the schema definition file with the Hierarchical Builder transformation. If the source contains date, time, and datetime fields, you must convert the date/time data to string data by adding an Expression transformation before the Hierarchical Builder transformation in the mapping. Configuration Tasks To read data from relational sources and write data to Google BigQuery by using a Hierarchical Builder transformation in a Google BigQuery mapping, perform the following steps: 1. Configure a JSON schema definition file on the Secure Agent or Hosted Agent. 2

3 2. Add a source transformation. 3. If the source contains date, time, and datetime fields, add an Expression transformation before the Hierarchical Builder transformation to convert the data into Google BigQuery compatible format. 4. Add a Hierarchical Builder transformation to the mapping. 5. Associate the JSON schema definition file with the Hierarchical Builder transformation. 6. Configure the field mapping and link the relational fields to schema elements that provide the hierarchical output. 7. Add a target transformation. 8. Run the mapping or the Mapping Configuration task. Before You Begin Before you read data from relational sources and write data to Google BigQuery by using a Hierarchical Builder transformation, perform the following prerequisite tasks: 1. Create a Google account to access Google BigQuery. To create a Google service account, click the following URL: 2. On the Credentials page of the Google API Console, navigate to the APIs and auth section, and create a service account. 3. After you create the service account, download the JSON file. 4. Create the relational source connections. 5. Configure a Google BigQuery target connection to use complex connection mode. 6. Ensure that you have the permission to use the following license and packages: DT XML Processing (License) Data Transformation UTDforHierarchy Step 1. Configure a JSON Schema Definition File on the Secure Agent or Hosted Agent You must associate a JSON schema definition file with the Hierarchical Builder transformation to define the schema of the Google BigQuery table. The Hierarchy Builder transformation converts hierarchical input based on the JSON schema definition file. If you use the Secure Agent as the runtime environment, the Secure Agent creates a JSON schema definition file on the Secure Agent machine. In the Schema Definition File Path field of the Google BigQuery connection, you can define the directory where the Secure Agent must create the file. 3

4 The following image shows the View Connection dialog box that displays the schema definition file path: Select the JSON schema definition file and associate it with the Hierarchical Builder transformation. If you use the Hosted Agent as the runtime environment, you must specify a storage path in Google Cloud Storage in the Schema Definition File Path field under the connection properties. You can then download the sample schema definition file for the Google BigQuery table from the specified storage path in Google Cloud Storage to a local machine. Step 2. Add a Source Transformation Add one or more relational source objects and configure the source properties. 1. Click Design > Mappings on the Informatica Cloud home page. 2. Click New Mapping. The New Mapping page appears. 3. In the New Mapping page, enter a name for the mapping and optionally enter a description. 4. Drag a source transformation from the Transformation palette into the editor. Click the Source tab under the Properties panel. 5. Select an active relational connection and select a relational source object. 6. Repeat the steps to add more relational sources to the mapping, if required. 4

5 Add an Expression Transformation If the source contains fields of the date, time, and datetime data type, place an Expression transformation before the Hierarchical Builder transformation to convert the data into Google BigQuery compatible format. 1. Drag the Expression transformation from the Transformation palette into the editor. 2. On the General tab, enter a name and description. 3. Draw a link to connect the Source transformation and the Expression transformation. 4. On the Incoming Fields tab, you can configure the field rules that define the data that enters the transformation. 5. On the Expression tab, create a new field. Select the Field Type as Output Field and Type as string. The New Field dialog box appears. The following image shows the New Field dialog box: 6. Click OK. 7. Repeat the steps to create output fields for time and datetime fields, if required. 8. Click Configure to configure the field expression. The Field Expression dialog box appears. 9. Select the Expression as Not Parameterized. 5

6 10. Configure the Expression field as follows: a. Add the following expression field to convert the date data in the relational source into string data in the Google BigQuery target: SUBSTR(TO_CHAR(COL_DATE,'YYYY-MM-DD'),1,10) The following image shows the Field Expression dialog box: b. Add the following expression to convert the time data in the relational source to string data in the Google BigQuery target: SUBSTR(COL_TIME,12) The following image shows the Field Expression dialog box: 6

7 c. Add the following expression to convert the datetime data in the relational source to string data in the Google BigQuery target: SUBSTR(TO_CHAR(COL_DATETIME,'YYYY-MM-DD HH24:MI:SS'),1,19)) The following image shows the Field Expression dialog box: 11. Click Validate to validate the expression. 12. Click OK. 13. Draw a link to connect the Expression transformation to the Hierarchical Builder transformation. Step 3. Add a Hierarchical Builder Transformation to the Mapping You must add a Hierarchical Builder transformation to the mapping to convert the relational input into hierarchical output. The transformation processes the relational input from the upstream transformation and provides JSON output to the downstream transformation. The Hierarchy Builder transformation produces hierarchical output based on the JSON schema definition file that you associate with the transformation. 1. Drag the Hierarchical Builder transformation from the Transformation palette into the editor. Click the Properties panel to configure the properties. 2. Establish a connection between the source and the Hierarchical Builder transformation. Step 4. Associate the JSON Schema Definition File with the Hierarchical Builder Transformation To associate a JSON schema definition file with the Hierarchical Builder transformation, configure the output settings properties. 7

8 Output Settings Select the sample schema on the Output Settings tab of the Properties panel. 1. Select the Hierarchical Builder transformation and click the Output Settings tab under the Properties panel. The following image shows the Output Settings tab: 2. Click Create new from sample. The Create New Hierarchical Schema dialog box appears. 3. Click Choose File to browse to the directory specified in the Schema Definition File Path option in the Google BigQuery connection properties and select the JSON sample schema definition file. 4. Select the schema file and click Validate to validate the schema. 5. To search for an existing sample schema, click Select. The Select Schema dialog box appears. 6. In the View list, select a sample schema. 7. To search for a sample schema, enter the search characters in the Search field, and click Search. 8. To fix errors, edit the displayed sample. 9. To save the schema and include it in the Hierarchical Builder transformation, click Ok. The selected hierarchical schema appears in the Properties panel. Step 5. Configure the Field Mapping to Select Schema Elements Configure the field mapping in the Hierarchy Builder transformation to define which relational elements link to schema elements to provide the hierarchical output. If you have more than one group, use the field mapping editor to define primary and foreign keys. You also use the field mapping editor to link relational fields to schema elements. The field 8

9 mapping editor displays the transformation input elements and output fields. To the left, the editor shows the schema elements. To the right, the editor shows the relational output fields. u In the Properties panel of the Hierarchical Builder transformation, click the Field Mapping tab. The following image shows the field mapping editor: To the left, the field mapping editor shows the relational fields. To the right, the editor shows the schema elements. In this example, the transformation has two relational groups so you must designate primary and foreign keys. In this example, the DATE_COL field is a primary key in the NewExpression group, and the DOB_OUT field is the foreign key that links to the NewExpression1 group. A primary key is signified by the primary key icon in the Key column. A foreign key is signified by the foreign key icon. To link a relational field to schema element, drag the relational element to the schema element. The Mapped Field column shows the relational field to which the schema element is mapped. If the input relational fields constitute just one group, the data will be treated as denormalized input and you do not need to define primary or foreign keys. Selecting a Primary or Foreign Key Select a relational field to designate as a primary or foreign key. 1. In the Properties panel of the Hierarchy Builder transformation, click the Field Mapping tab. The field mapping editor displays the transformation input elements and output fields. To the left, the editor shows the relational fields. To the right, the editor shows the schema elements. 2. To search for a relational field, enter the characters to search for in the Search field, and click Search. 3. To designate a relational field as a primary or foreign key, click the Key column of the field. A key selector window appears. 4. Select whether the key is a primary key or foreign key. 5. If the key is foreign key, select the group to which the field relates as a foreign key. 6. To remove a primary or foreign key, click the Key column of the field and select Not a key. 7. To clear all the keys, click the action menu at the top of the Hierarchy Fields panel, and then select Clear Keys. Selecting Schema Elements Link relational fields to schema elements to define how to convert the relational input fields to hierarchical output elements. 1. In the Properties panel of the Hierarchy Builder transformation, click the Field Mapping tab. The field mapping editor displays the transformation input elements and output fields. To the right, the editor shows the schema elements. To the left, the editor shows the relational input fields. 2. To search for a relational field, enter the characters to search for in the Search field, and click Search. 9

10 3. To view child schema elements, expand a field. 4. To search for an element, enter the characters to search for in the Search field, and click Search. 5. To link a relational input field to a schema element, drag the relational field to the schema element. 6. To automatically link a relational input field to a schema element, select a schema element, click the action menu at the top of the Hierarchy Fields panel, and then select Map Selected. 7. To automatically unlink a relational input field to a schema element, select one of the following options: Select a schema element. In the Action column, click the delete icon. Select a schema element. Click the action menu at the top of the Hierarchy Fields panel and select Unmap Selected. 8. To clear all the links, click the action menu at the top of the Hierarchy Fields panel, and then select Clear Mapping. Step 6. Add a Target Transformation Use the Target transformation to define the Google BigQuery target in the mapping. 1. Drag a target object from the Transformation palette into the editor. Configure the target properties under the Properties panel. 2. Click the Target tab under the Properties panel. 3. Select an active Google BigQuery connection that uses complex connection mode and select a Google BigQuery target object. 4. Establish a connection between the Hierarchical Builder transformation and the target. 5. Click the Field Mapping tab under the Properties panel and map the Output field to the STRING_DATA field. The following image shows the Field Mapping tab: 6. Click Save > Save and Close. Step 7. Configure a Mapping Configuration Task You can save and run the mapping directly or configure a Mapping Configuration task, add the configured mapping, and run the mapping. 1. To create a Mapping Configuration task, click Task Wizards > Mapping Configuration, and then click New. 2. In the Mapping Configuration Task Details area, provide a task name, and provide the configuration details. 3. Configure a schedule and advanced options, if required. 4. Select the configured mapping, and click OK. 10

11 The following image shows the configured mapping: 5. Click Save. You can run the mapping manually or on a schedule. Author Anu Chandrasekharan Lead Technical Writer Acknowledgements The author would like to acknowledge Nagashree Upadhya, Lead QA Engineer, for her technical assistance. 11