CloverETL User's Guide

Transcription

1 IBM Initiate Workbench CloerETL User's Guide Version9Release7 SC

2

3 IBM Initiate Workbench CloerETL User's Guide Version9Release7 SC

4 Note Before using this information and the product that it supports, read the information in Notices and trademarks on page 87. Copyright IBM Corporation 1995, US Goernment Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

5 Contents Introduction to CloerETL in IBM Initiate Workbench Where to go for more information on CloerETL.. System requirements for CloerETL How to get help i Technical Support Contacts i On the web i By telephone i Chapter 1. CloerETL features and benefits Sample graphs in IBM Initiate Workbench About Master Data Extract Using the CloerETL interface Switching to the Cloer perspectie The CloerETL layout Callout handler registration for CloerETL graphs.. 3 Chapter 2. CloerETL graph components 5 Oeriew of CloerETL components CloerETL Readers CloerETL Transformers CloerETL Joiners CloerETL Writers CloerETL Edges Other CloerETL components Deprecated CloerETL components Working with CloerETL graphs Preparing the CloerETL extract file best practices Using a CloerETL sample graph Building a CloerETL graph Adding and configuring a CloerETL Reader.. 8 Adding and configuring a CloerETL Transformer 9 Using CloerETL Edges to connect components 10 Adding and configuring a CloerETL Writer.. 12 Connections in CloerETL Creating a database connection for CloerETL graph execution Creating an Initiate connection for CloerETL graph execution Creating a JMS Connection for CloerETL graph execution Executing a CloerETL graph Executing CloerETL graphs with the madconfig launch_etl utility Using madconfig launch_etl to execute a CloerETL graph Recording responses to the madconfig utility.. 16 Viewing logs and error messages about CloerETL graph execution The CloerETL JMS Reader component How the CloerETL JMS Reader component and Edges process messages Using the CloerETL JMS Reader components's timeouts to manage message flow Limiting the CloerETL JMS Reader component max message count for debugging Using the CloerETL JMS Reader max message counts and timeouts together Managing the CloerETL Edge's message buffer for immediate message processing The CloerETL Clean and Deduplicate Sample Graph Linking The CloerETL Clean and Deduplicate sample Reader to a source file Adding metadata for the CloerETL Clean and Deduplicate sample graph Editing the Ext Filter component in the CloerETL Clean and Deduplicate sample graph. 21 Specifying a sort key in the Ext Sort component of the CloerETL Clean and Deduplicate sample graph Specifying a dedup key in the CloerETL Clean and Deduplicate the sample graph Writers in the CloerETL Clean and Deduplicate sample graph The IBM Initiate Member Model Transform Graph 24 Input metadata in the Member Model Transform graph Linking to source data files and output directories in the Member Model Transform graph Configuring the IBM Initiate Member Model Transform Graph The CloerETL Extract Household Graph Creating a project for the CloerETL Extract Household graph Customizing the extract_household graph Chapter 3. IBM Initiate custom components Using Key fields in CloerETL MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to join records IBM Initiate custom Reader components Comparing the IBM Initiate custom Reader components The IBM Initiate MemGet component The IBM Initiate MemSearch component The IBM Initiate Handler Reader component.. 40 The IBM Initiate TskGet component The IBM Initiate TskSearch component IBM Initiate custom Writer components The IBM Initiate MemPut component The IBM Initiate MemDelete component The IBM Initiate Handler Writer component.. 56 The IBM Initiate TskPut Component Copyright IBM Corp. 1995, 2011 iii

6 IBM Initiate custom Transformer components for CloerETL The IBM Initiate MemScore component The IBM Initiate Attribute Reformat component 67 The IBM Initiate MemSeqNoIncrementer component IBM Initiate custom address-standardization components for CloerETL Attribute requirements for address standardization in CloerETL Deciding between batch and transactional standardization in CloerETL The CloerETL IBM Initiate Address Verification Interface component Configuring the CloerETL standardization and alidation graph IBM Initiate Address Verification Interface transliteration in CloerETL Legal Statement Notices and trademarks Index Contacting IBM i CloerETL User's Guide

7 Introduction to CloerETL in IBM Initiate Workbench The CloerETL application in IBM Initiate Workbench enables you to ensure data quality easily by executing extract, transform and load (ETL) operations in a graphical format that is similar to a flow chart. ETL operations are a part of Master Data Management that enable you to extract data from a source or sources, such as the IBM Initiate Master Data Serice or flat files transform the data format, table structure, or other characteristics into a data model that fits your data target load the transformed data into the target, for example a data warehouse, or een back into the IBM Initiate Master Data Serice Within the CloerETL graph editor, you can create the flow chart structure, which is called a graph or template, by dragging and dropping graphical components. The configuration behind the graph is stored as an editable XML document; howeer, many users find that the graphical format is much easier to work with. CloerETL takes adantage of the IBM Initiate Master Data Serice APIs by proiding many components that perform interactions such as MemPut, MemSearch, and MemGet. Where to go for more information on CloerETL The IBM Initiate Workbench Cloer User's Guide is not intended to proide detailed information about all CloerETL functionality; rather, it discusses specific components and procedures that are specific to the IBM Initiate suite of products. For more information on the CloerETL application, refer to the following documents and resources: The CloerETL user documentation, aailable ia the IBM Initiate Workbench Help menu, or at The CloerETL wiki at For related information on IBM Initiate products, refer to the following documents: IBM Initiate Workbench User's Guide IBM Initiate Workbench Installation Guide For additional information on address standardization solutions, refer to supplemental documentation as follows: IBM Initiate Address Verification Interface: See the IBM Initiate Address Verification Interface proprietary documentation that was proided in PDF format with your IBM Initiate Address Verification Interface software System requirements for CloerETL This document assumes that IBM Initiate Workbench has been installed according to the instructions published in the IBM Initiate Workbench Installation Guide. Refer to IBM Initiate Master Data Serice System Requirements for detailed information. Copyright IBM Corp. 1995, 2011

8 How to get help You can get additional help for all products by contacting your organization's Technical Support Contact. You can also find additional information on the web, or get help by telephone. Technical Support Contacts On the web Each organization should designate two (2) or more indiiduals to sere as Technical Support Contacts to interface with IBM Software Support. These indiiduals should form an internal core software support team who become experts in the usage and maintenance of the IBM Initiate software If you hae questions or concerns about the software, and if the information in this guide does not answer your questions, contact your internal Technical Support Contacts. The Technical Support Contacts can engage the IBM Initiate Master Data Serice Support team to resole your issue. To find help on the web, go to This site contains the IBM Initiate Master Data Serice library, news, links to web resources, and the ability to open a Serice Request. By telephone If you are in North America, call IBM-SERV ( ). If you are outside of North America, check the web page planetwide/ for contact information for your area. i CloerETL User's Guide

9 Chapter 1. CloerETL features and benefits CloerETL can benefit your business operation in seeral ways, including the following scenarios: Batch inbound processing for a bulk load of data You want to load a large set of data into the Hub (IBM Initiate Master Data Serice); howeer, certain data-cleansing processes must be executed before you can load the data. Running CloerETL can ensure that consistently formatted, clean data is inserted into the Hub, particularly at the initial load time in an implementation or when bringing on a new source system. Batch outbound processing for a bulk extract of data You want to extract a large set of data from the Hub into some other system, for example, a reporting database, or you want to synchronize systems. CloerETL extracts the data from the Hub, runs it through a set of data-cleansing processes, and ensures that it is properly formatted for use in the target system. Real-time data load or update In this ongoing, long running serice, one record at a time is run through a data-cleansing process and then inserted into the Hub. The single transaction typically uses a graph similar to one that is used for cleansing the data at initial load. Real-time data extraction CloerETL accepts a record from the Hub during this ongoing, long running serice, and processes it through a set of data-cleansing processes to prepare it for consumption by another system. In each scenario, CloerETL processing can include an Initiate custom component that calls the IBM Initiate Address Verification Interface address standardization feature. As a result, each record's address information is properly formatted for your business needs, either for temporary use and storage or database update. Sample graphs in IBM Initiate Workbench Sample graphs are included with the IBM Initiate Workbench component. You can customize these graphs to use in your enironment. The following sample graphs are supported in the base IBM Initiate Master Data Serice product: Clean and Deduplicate Initiate Member Model Transform (as a wizard) Extract Full All Extract Household Copyright IBM Corp. 1995,

10 About Master Data Extract Master Data Extract is an ETL product which includes wizards that create the customized extraction graphs: Initiate Member Model MMCA (Member Most Current Attributes) graph Initiate Member Model EMCA (Entity Most Current Attributes) graph These graphs extract both incremental and initial MMCA and EMCA data from the Hub. Using the CloerETL interface The following sections describe how to use the CloerETL interface. Switching to the Cloer perspectie To switch to the Cloer perspectie in IBM Initiate Workbench, follow these steps: Procedure 1. Click the Open Perspectie icon. 2. Choose Other to open the Other Perspectie window group. 3. In the Open Perspectie window, choose CloerETL. 4. Click OK. The CloerETL layout The CloerETL layout utilizes the most common features of the Workbench deelopment tool, including perspecties, iews and editors. Detailed information about these features can be found in the IBM Initiate Workbench User's Guide. When using the CloerETL function, you will work in the following areas. The CloerETL Naigator iew The Naigator iew proides a tree structure for browsing Workbench artifacts, including Workbench projects and graphs. The CloerETL Outline iew The Outline iew gies an oeriew of the components used in a graph, and shows graph-related information such as database connections, parameters, metadata, and sequences. Within the Outline iew, you can toggle between a hierarchical iew and a graphical iew of the graph by clicking the iew buttons: The CloerETL Graph editor The Graph editor is the main workspace for creating, editing, and iewing graphs. Graphs are created by dragging graph components from the Palette and dropping them into the Graph editor. The CloerETL Palette The Palette is where you select graph components Readers, Writers, Transformers, Joiners, and other graph components as well as Edges for joining components, and editing tools. Click the headings within the palette to expand or collapse the list of components. 2 CloerETL User's Guide

11 Note: To deactiate an actie component or tool selected in the palette, press Esc. The CloerETL Tabs iew The lower-right portion of the screen shows a tabbed list of iews that proide additional information and help with graph execution and troubleshooting. The Properties tab enables you to iew and edit properties of the graph components, such as XML attributes, filtering, expressions, and metadata information. The Console tab shows progress messages and errors. The Problems tab proides a list of configuration and alidation problems The Cloer-Regex Tester tab is an editor that assists with testing of regular expressions. The Cloer-Log tab displays logging details for the graph execution. Callout handler registration for CloerETL graphs Some CloerETL graphs require certain handler class information and handler arguments. For information about packaging a CloerETL graph as a callout, refer to the Callout handlers information in the IBM Initiate Workbench User's Guide. Chapter 1. CloerETL features and benefits 3

12 4 CloerETL User's Guide

13 Chapter 2. CloerETL graph components CloerETL graph components proide a isual representation of the flow of the extract, transform, and load process. Components are joined by Edges; data flows through Edges, which are structured by metadata. Graph components are selected from the Palette list. Oeriew of CloerETL components IBMInitiateIBM Initiate Workbench uses seeral CloerETL components in the extract, transform and load processes. Most components require some configuration to work correctly within a graph. Configuration is done through the component's editor, which displays all the component's properties. Missing property alues that are required are flagged with a yellow warning icon. Default property alues are displayed in gray italics. CloerETL Readers Readers can be thought of as the starting point for the ETL process, in that they read the input data that is to be processed. Readers access data from specified data sources, such as external files, LDAP repositories, or databases. Each graph must include at least one Reader. CloerETL Transformers Transformers perform operations on the data, such as filtering, sorting, de-duplicating, and merging. Transformers receie data ia input ports and export it to output ports. You can concatenate a series of Transformers to perform multiple operations against your data. When using multiple Transformers, care should be taken with the sequence in which they are executed. For example, it is necessary to sort your data before doing a deduplication operation. CloerETL Joiners Joiners can put together records with different metadata, including those with different numbers of fields, according to a specified key and transformation. These components perform some of the same transformations that the Transformer components perform. Joiners receie data ia multiple input ports and export it to a single output port. CloerETL Writers Writers output data to specified output sources, such as external files, database tables, or LDAP repositories. They are often the terminal components in graphs.. Copyright IBM Corp. 1995,

14 Note: The Trash component is a quick way to terminate a graph for testing. CloerETL Edges Edges appear as the connecting lines between Readers, Transformers, and Writers. Eery Edge carries metadata information for structuring the data. This includes information such as field names, field types, and delimiters. Metadata can be deried from header rows in the data, entered manually, or applied ia a format file (*.fmt). In addition, metadata can be propagated from one Edge to another. Edges are always bound to component ports. Components read data from input ports and write data to export ports. Some components proide specific numbers of ports, and others proide an infinite number of ports. Filter components, for example, hae two output ports: one (port 0) for writing alid records, and another (port 1) for writing inalid records. Other CloerETL components CloerETL also supports components that execute graphs and some system, Jaa, and database commands. In addition, there are components that perform lookup and foreign key definition functions. Deprecated CloerETL components Some CloerETL components hae been deprecated but are still aailable in the product. These components hae been replaced by components that function more efficiently; howeer, their behaior might not be exactly the same as the components that they replace. A deprecated component will continue to function, but in a future release might be remoed. Note: The Console log will include the following warning message when you use a deprecated component: WARN [main] - [COMPONENT_NAME] - Component is of type COMPONENT_TYPE, which is deprecated For a more thorough discussion of CloerETL components, refer to the Cloer documentation Working with CloerETL graphs Workbench includes seeral sample graphs that you can customize for your own use; these graphs can also proide a resource for you to learn how graphs are built. Preparing the CloerETL extract file best practices You should obsere the following best practices when you prepare your data for transformation in CloerETL: Sort your data on an index or key. This speeds up the processing time and can eliminate the need for a Sort step in your graph. 6 CloerETL User's Guide

15 Include header rows where possible. This facilitates the creation of metadata required by the graph. Howeer, when configuring some Reader components, remember to set the properties related to skipping lines in source data appropriately. Using a CloerETL sample graph To iew and use a sample graph you must import it into a project. For information about working with projects, refer to the IBM Initiate Workbench User's Guide. Importing a CloerETL graph into a project You can import a graph anywhere in the project folder. Creating a folder just for storage of graphs helps with efficient file management. About this task Procedure 1. In the Naigator pane, right-click within the project folder (or on the folder in which you want to store graphs), and choose Import. 2. In the Import - Select dialog box, select Import graphs ersion conersion. 3. Click Next. 4. In the Import Cloer.ETL Graphs dialog, click the Browse buton beside the From directory field. 5. Naigate to and select the <ROOTDIR>\<Workbench (ersion)>\samples\ graphs directory (where ROOTDIR is your the program files installation directory, and <Workbench (ersion)> is the directory where Workbench is installed). 6. Click OK. 7. The Import Cloer.ETL Graphs window is now populated with all aailable sample graphs in that directory. In the right pane, select each item that you want to import, or select graphs in the left pane to select all the items. 8. Click Finish to complete the import procedure. Building a CloerETL graph The section discusses the basic steps for creating and executing a CloerETL graph. About this task Depending on your business needs, the graphs that you build will ary in design and complexity; howeer, the basic steps include: selecting graph components connecting them with Edges defining their properties selecting sources for data that will be transformed selecting targets for the transformed data identifying metadata and assigning it to the Edges running the graph Chapter 2. CloerETL graph components 7

16 For more detailed information, refer to the CloerETL Designer User's Guide, and to The CloerETL Clean and Deduplicate Sample Graph on page 20. Before you can design and execute a graph, you must create a graph placeholder in Naigator within your project. Procedure 1. Right-click the folder in which you want to create the graph, and select New > ETL Graph to open the New graph wizard. 2. In the CloerETL node, click ETL Graph. 3. In the Create new graph dialog box: a. Type a name for the graph. b. (Optional) Type a description of what the graph accomplishes. c. Clear the Allow inclusion of parameters from external file? check box. 4. Click Next. 5. In the Output dialog box, erify the parent folder (the parent project and folder) for the graph. (Optional) You can change the name of the graph by updating the File name field. Note: As a best practice, create the graph under the Initiate project folder that is associated with your Hub and with the data you will use this graph to transform. 6. Click Finish. The new graph name is displayed in the Naigator tree pane within the folder in which you created it. Adding and configuring a CloerETL Reader Adding and configuring a reader enables your graph to read in data from an external source such as a text file, database, or LDAP repository. About this task Readers access data from external sources. Procedure 1. In the Palette, click the Readers header to expand the list of Readers. 2. Drag the type of Reader you want to use into the Graph editor. 3. Double-click the Reader to open the Edit component dialog. Note: You can also access the properties of the component ia the Properties tab at bottom of the Workbench. When using the Properties tab, ensure that you hae selected the correct component in the Graph editor. In the Properties tab, port information is listed under Input Port and Output Port in on the Properties tab. 4. Enter a File URL linking to your source data: a. Click in the Value field for File URL (on the Properties tab). An ellipsis icon is displayed. b. Click the ellipsis icon to open a file browser. c. Naigate to and select your source data file, and click Open. 5. Edit the other Reader properties as needed. Aailable properties ary depending on the type of Reader you hae chosen; detailed information about 8 CloerETL User's Guide

17 Reader-specific properties is aailable in the CloerETL Designer User's Guide. In most cases you can accept the default properties. 6. Click OK. Results Note: A yellow warning icon is displayed at the upper right of the Reader when no data source is selected. Adding and configuring a CloerETL Transformer Transformers perform operations on the data, such as filtering, sorting, de-duplicating, and merging. You can concatenate multiple Transformers to perform complex operations on your data. About this task To add a Transformer component to your graph and configure it, follow these steps. Procedure 1. In the Palette, click the Transformers header to expand the list of Transformers. 2. Drag the type of Transformer you want to use into the Graph editor. 3. Double-click the Transformer to open the Edit component window. Note: You can also access the Properties of the component ia the Properties tab. Note, howeer, that when using the Properties tab, you must be sure the correct component is selected in the Graph editor. In the Properties tab, port information is listed under Input Port and Output Port. 4. Edit the Transformer properties as needed. Aailable properties will ary depending on which type of Transformer you hae selected. Required properties with missing alues are marked with a yellow warning icon. For example, a sort key is required for the Ext Sort Transformer, and a filter expression is required for the Ext Filter Transformer. Detailed information about Transformer-specific properties is aailable in the CloerETL Designer User's Guide. Note: Some properties, such as filter expressions, cannot be alidated until metadata has been defined for the data being transformed. See Adding CloerETL metadata on page 10 for more information. Before you can use the Dedup component for deduplication operations, data must be sorted. If your source data is already sorted, you can direct data directly from the Reader to the Dedup component. If your source data has not been sorted, you must direct your data through an Ext Sort component before directing it to the Dedup component. 5. When you hae you entered the Transformer's properties, click OK. Using regular expressions with the CloerETL filterexpression editor The Ext Filter Transformer can use regular expressions and built-in functions for filtering data. Chapter 2. CloerETL graph components 9

18 Each input record is processed by the filter expression. If the result of the expression is true, the record is passed to port 0 (accepted); otherwise it is passed to port 1 (rejected). You can find additional information about constructing regular expressions in the CloerETL Designer User's Guide, and on the Web at sites such as and essential/regex/. CloerETL Expression tester CloerETL includes an editor that can assist you with testing of regular expressions. The expression tester is accessible from the Tabs pane. More information about the expression tester is aailable from the CloerGUI Help, aailable ia IBM Initiate Workbench's help menu. Using CloerETL Edges to connect components Graph components are linked with Edges. Edges represent the data flow between components. Edges require metadata to define the data that flows between components. Adding Edges to the graph, then, inoles seeral steps: Connecting components with an Edge Adding metadata information Applying metadata to an Edge Propagating metadata from one Edge to adjacent Edges These steps are described in detail below. Connecting CloerETL components with an Edge To pass data from one component to the next in your graph, follow these steps to connect the components with Edges. Procedure 1. In the Palette, click to select the Edge tool. The mouse icon changes to show a plug, indicating that the Edge tool is actie. 2. Click the output port of the component you want to connect from. Output ports appear as notches along the right edge of a component. Note: For components with multiple ports, be sure to connect to the correct port. You can mouse oer a component's port to see a brief description of the port's output. 3. Click the input port of the component you want to connect to. Input ports appear as notches along the left edge of a component. Note: When the Edge is displayed as a dotted line, this indicates that metadata is missing. A solid-line Edge indicates that all required information is present. Adding CloerETL metadata Metadata defines the data that flows between components ia Edges. 10 CloerETL User's Guide

19 About this task While CloerETL offers multiple ways to add metadata, this section describes the recommended way for use with the IBM Initiate sample graphs. Refer to the CloerETL Designer User's Guide for information on alternatie ways of adding metadata to an Edge. Procedure 1. In the Outline pane, right-click Metadata and choose New Metadata. Note: You can also access the New Metadata menu by right-clicking on an Edge in the graph. 2. From the New Metadata menu choose Extract from flat file. Note: For detailed information on other methods of adding metadata, refer to the CloerETL Designer User's Guide. 3. In the Flat file dialog, click Browse. 4. Browse to and select your source data file. Click Open. 5. Click Next. 6. If your file is delimited, choose a delimiter from the Delimiter drop-down list. If your delimiter is not listed in the Delimiter drop-down list, you can add it by following these steps: a. From the Delimiter drop-down list, choose Other <specify>. b. In the field to the right of the Delimiter list, enter the delimiter character (such as a pipe). c. Click the << icon to add the delimiter to the drop-down list. 7. If your source data file includes header records, check the Extract names box to pull field names from the source file. Note: If your source data file does not include header records, you will be able to add field names on the next tab. 8. Click Next. The metadata is displayed. 9. If you want to gie this metadata a specific name rather than accepting the default name, click the Record:default name field (in the Name column) and enter the new name. 10. If your source file does not include header records, you can edit the default field names (listed in the Name column) as needed. 11. If you want to preiew the data before continuing, click the Preiew tab. Note: The Preiew tab is aailable only when adding new metadata. If you open an existing metadata record from the Outline pane, you will not see the Preiew tab. 12. Click Finish. Applying CloerETL metadata to an Edge About this task Once you hae created metadata for a graph, you can associate the same metadata to other Edges. Chapter 2. CloerETL graph components 11

20 Procedure 1. Using the Palette's Select tool, right-click the Edge to which you want to apply metadata. Note: You can apply metadata to multiple Edges at once by using Ctrl-click to select multiple Edges. 2. From the drop-down list, select Select metadata. A list of saed metadata for this graph appears. Choose the metadata you want to apply to this Edge. Note: You can also apply metadata to an Edge by editing the Metadata property on the Edge's Properties tab. Propagating CloerETL metadata from one Edge to another Metadata can be propagated from one Edge to adjacent Edges recursiely, proiding a quick and conenient alternatie to assigning metadata manually to adjacent Edges. About this task To propagate metadata to an adjacent Edge, right-click on the Edge with metadata already assigned and choose Propagate metadata. Note: Edges with metadata assigned appear as solid lines. Edges missing metadata appear as dotted lines. When you hae propagated the metadata to adjacent Edges, those Edges will appear as solid lines. Debugging a CloerETL graph Edge CloerETL offers a debug feature, to help you troubleshoot problems in your graphs. About this task To debug a graph Edge, right-click on the Edge and choose Debug > Enable Debug. A green bug icon is displayed on edges with debugging enabled. Debug information is captured when the graph is run. You can iew debug data after the graph is run by right-clicking the edge and choosing Debug > View Data. Adding and configuring a CloerETL Writer Writers write transformed data to specified output files. About this task To use a Writer in your graph, you must add to the graph, configure it, and connect it to the other component(s) that will proide the data to write. Procedure 1. In the Palette, click the Writers header to expand the list of Writers. 2. Drag the type of Writer you want to use into the Graph editor. 3. Double-click the Writer to open the Edit component dialog. 12 CloerETL User's Guide

21 Connections in CloerETL Note: You can also access the Properties of the component ia the Properties tab. Note howeer that when using the Properties tab, you must be sure the correct component is selected in the Graph editor. 4. Edit the Writer properties as needed. Aailable properties will ary depending on which type of Writer you hae selected. For example, the DB2 Data Writer requires database connection information, and the LDAP Writer requires an LDAP URL. Required properties are marked with a yellow warning icon. For example, database connection information is required for the DB2 Data Writer. Detailed information about Writer-specific properties is aailable in the CloerETL Designer User's Guide. Note: You can easily discard records by using the Trash Writer. 5. When you hae you entered the Writer's properties, click OK. 6. Connect the Writer to the appropriate Transformer ia an Edge. Refer to Using CloerETL Edges to connect components on page 10 for more information. Connections in CloerETL let your graph communicate with external data resources. To execute some graphs, you must establish connections to resources such as the IBM Initiate Master Data Serice or to databases from which data will be extracted. Creating a database connection for CloerETL graph execution A database connection lets your CloerETL graph components connect to a database. About this task After creating a database connection, you can use any of the database readers in graphs that you build. Procedure 1. Open a graph. 2. In the Outline pane, right-click Connections and choose Connections > Create internal. This opens the database connections window. Note: You must open a graph in the Graph editor to see the nodes, including the Connection node, in the Outline pane. 3. Click to select a database drier from the aailable driers window. Note: It is recommended that you use one of the supplied driers. 4. Enter a Name for the connection. 5. Enter the User and Password for connecting to the database. 6. Click the Validate button to alidate the database connection. 7. Click Finish. Chapter 2. CloerETL graph components 13

22 Creating an Initiate connection for CloerETL graph execution To use IBM Initiate custom components you must create an IBM Initiate connection, which is a connection to the IBM Initiate Master Data Serice. 14 CloerETL User's Guide About this task To create your IBM Initiate connection, you will need to specify the hostname, port, user name and password for connecting to your Master Data Engine (Hub) instance. Procedure 1. Open a graph. 2. In the Outline pane, right-click Connections and choose Connections > IBM Initiate connection. This opens the Initiate connection window. 3. Select internal. 4. Enter a Connection name for the connection. 5. Enter the Host and Port for your Hub. 6. Enter the User and Password for connecting to your Hub. 7. To configure reconnection attempts after a network, engine, or database outage: a. Set a number of attempts in the Retry count field. b. Set the interal between attempts in the Retry delay field. 8. Click Finish. Creating a JMS Connection for CloerETL graph execution Use a JMS connection to connect your graph to a message queue source. About this task After creating a JMS connection you can use the JMS Reader and JMS Writer components in graphs that you build. Procedure 1. Open a graph. 2. In the Outline pane, right-click Connections and choose Connections > JMS internal connection. This opens the JMS connection window. Note: You must open a graph in the Graph editor to see the nodes, including the Connection node, in the Outline pane. 3. Enter a Name for the connection. 4. Click the green plus symbol to browse to and select the Libraries for the connection. These reference any custom jar files that may need to be appended to the classpath. The prefix for your libraries should be file: For example, a library entry for Apache ActieMQ would take a format such as file:c:/apache-actiemq-5.2.0/actiemq-all jar 5. Enter an Initial ctx factory class. This is the fully-qualified class name implementing the JNDI API and allowing lookup of the connection factory. This item will detect class name automatically when a jar file with the class is specified. Otherwise, the class name may be specified manually. 6. Choose a URL from the drop-down list. The URL specifies the connection's protocol.

23 Executing a CloerETL graph 7. Choose a Connection factory JNDI name from the drop-down list. This is the JNDI name of the factory creating JMS connections. Values in the drop-down list will ary depending on the JMS proider. 8. Enter a Destination JNDI for the connection. This is a JMS proider-specific address that identifies queue/topic of messages. 9. Enter the User and Password for connecting to the JMS. 10. Check the Encrypt password box if you want to encrypt the password. Note: Click Validate connection to alidate the connection information. If the connection does not alidate, make sure that the Master Data Engine serice is running and that you hae entered alid connection parameters. 11. Click Finish. To execute a graph, ensure that necessary connections are established, and launch the graph using the Run icon. About this task To execute a graph, click the Run icon in the toolbar, or right-click in the graph editor and choose Run As > Cloer.ETL graph. When a graph has finished, the number of records processed along each Edge is displayed. Note: The CloerETL GUI is intended for graph deelopment and debugging. Production leel graphs with large data sets should be run by using the command-line option (refer to Executing CloerETL graphs with the madconfig launch_etl utility ) or on the same serer where the data is located (refer to the IBM Initiate Workbench User's Guide). Running graphs by using the command line or remote serer enables improed performance due to the aailability of increased memory and 64-bit processing. For detailed information about graph runtime options, refer to the CloerETL Designer User's Guide. Executing CloerETL graphs with the madconfig launch_etl utility The madconfig launch_etl utility proides a means of running a CloerETL graph automatically, for example as part of a scheduled job. About this task After you hae configured the sample templates and created CloerETL graphs, you can execute them automatically by using the madconfig utility as part of a scheduled job. To update parameters, such as audit record number range, when you run the scheduled job you can create an external properties file that the scheduled madconfig utility can reference. You can also use the madconfig launch_etl utility to execute a graph independently of a job. Chapter 2. CloerETL graph components 15

24 Using the madconfig utility to create a properties file for a scheduled job Parameters for running a scheduled job are stored in a properties file. You can create a properties file for a scheduled graph execution using the madconfig utility. About this task Incremental extracts typically select data that is based on a range of audit record numbers that change each time the graph is run. Although you may manually set the range of record numbers to extract manually in the graph, it may be more practical to generate a properties file automatically ia a scheduled job. The properties file then supplies the graph with the appropriate alues for the record number range. This section describes how to use the madconfig utility to launch a graph using a designated, external properties file. You can set up a scheduled job to launch the madconfig utility on a regular basis. Note: It is outside the scope of this document to describe how to set up a scheduled job that generates the properties file. You can use a standard utility such as the Windows Task Scheduler or a Unix chron utility (or other methods) to set up a scheduled job. Using madconfig launch_etl to execute a CloerETL graph The madconfig launch_etl operation can automatically launch a CloerETL graph. It can optionally use a properties file to proide input to the madconfig utility. About this task The madconfig launch_etl operation can make use of a properties file that contains auditor record number files. Procedure 1. From a command prompt, run madconfig launch_etl Note: This utility is run from the ROOTDIR\Engine x.x.x\scripts directory 2. At the prompt, enter the path to the graph (*.grf file) you want to run. 3. (Optional) At the prompt, enter the path to your configuration file (that is, the file containing the properties for your graph's audit record number parameters). 4. At the prompt, enter a heap memory size setting or accept the default (256 MB). 5. (Optional) At the prompt, enter a statement that includes any additional parameters needed to execute the graph. Note: For complete documentation of the madconfig utilities, refer to the IBM Initiate Master Data Serice Engine Installation Guide. Recording responses to the madconfig utility If you want to launch a graph by using madconfig on a scheduled basis, you can record a set of responses to the madconfig utility's prompts. 16 CloerETL User's Guide

25 About this task To record a set of responses to the madconfig launch_etl function, run madconfig -recordfile myfile.properties launch_etl where myfile.properties is the name of the file that will store your responses. Note: In addition to recording your responses, this command also executes the graph. To run madconfig using the recorded responses, run madconfig propertyfile myfile.properties launch_etl, where myfile.properties is the name of the file where your responses are stored. Viewing logs and error messages about CloerETL graph execution Warning and error messages, processing information, and graph status are captured on the Console, Problems, Cloer Graph tracking, and Cloer Log tabs. About this task Refer to the CloerETL Designer User's Guide for detailed information about the contents of these tabs. The CloerETL JMS Reader component The JMS Reader component processes messages in accordance with Jaa's standard JMS API. Messages can be routed directly to a Writer component, or can be transformed before writing, using any of CloerETL's transformer components. The table below lists JMS Reader properties which are typically configured. For a full description of all JMS Reader component properties, refer to the CloerETL User's Guide or the CloerETL online help in Workbench. Table 1. JMS Reader component properties Property Required? Default Description JMS connection yes The JMS connection ID. JMS connections are created in the Connections node of the Outline. For more information on creating a JMS connection, see Creating a JMS Connection for CloerETL graph execution on page 14. You can also refer to the CloerETL User's Guide or the CloerETL online help in Workbench. Processor code no Inline Jaa code defining the processor class. Processor code is applied only if the Processor class is not specified Processor URL no URL to file which contains Jaa source of the processor class. This is applied only if the Processor class and Processor code are not specified Processor class no org.jetel. component. jms.jmsmsg2 DataRecord Properties Name of the Processor class. The default alue ( org.jetel.component.jms.jmsmsg2 DataRecordProperties ) is applied only if the Processor code and Processor URL attributes are not specified. Chapter 2. CloerETL graph components 17

26 Table 1. JMS Reader component properties (continued) Property Required? Default Description JMS message selector Processor source charset Message charset no no ISO (check DataParser. DEFAULT_ CHARST_ DECODER in default Properties) ISO (check DataParser. DEFAULT_ CHARST_ DECODER in default Properties) JMS selector specifying messages to be processed. The charset of the Processor code, if it is specified by the processorurl attribute. The charset of the Message Max msg count no The maximum number of messages to process before exiting the graph. Setting this to 0 means to read foreer (that is, to enforce no maximum). When debugging a graph, it can be useful to set the max message count to a manageable number of messages for reiew. Timeout no The maximum amount of time (in milliseconds) to wait for the next message before exiting the graph. Setting to 0 means to wait foreer (that is, to enforce no timeout). Message body field no Maps the message body to a field on the output port's Edge metadata. Once your metadata is defined, the message body field can be selected from the drop-down. How the CloerETL JMS Reader component and Edges process messages When configuring the JMS Reader component, it is important to understand how messages are processed, in order to aoid suboptimal performance. Once connected to the message queue source, the JMS Reader component reads messages as they arrie, and passes them ia the component's output port to the Edge, which in turn routes the messages on to the next component in the graph. The JMS Reader's default behaior is to read and process each message as it arries, regardless of the number of messages, or the rate at which they arrie. The JMS Reader, by default, does not time out or impose any kind of maximum count on the messages it processes. The Edge that carries messages from the JMS Reader component to the next component by default buffers the messages before routing them to the next component. This buffering behaior is designed to improe performance; howeer, it is important to realize that in a low-traffic scenario, it may take time for the 18 CloerETL User's Guide

27 Edge's buffer to fill up and trigger the sending of messages to the next component. This may result in delays which are unacceptible from a performance standpoint. The sections below describe how to use timeout alues, max message counts, and Edge properties to determine the manner and rate of message processing. Using the CloerETL JMS Reader components's timeouts to manage message flow The CloerETL JMS Reader component's timeout properties help you manage message flow for optimum performance. With message queues, unlike (for example) text or database files, there is often no clear end point for the message stream. A message queue can contain anything from no messages at all to a irtually endless stream of messages. The JMS Reader component's default behaior is to process messages as they arrie, and to continue to process messages indefinitely. If you hae a scenario in which you want the JMS Reader to process a set of messages and then, when that set is completely processed, to stop and close the graph, you can use the timeout property. A typical use case for using a timeout is when you hae stopped the Master Data Engine for a time, and know that during this downtime you hae built up a queue of messages to process; in this scenario you want to drain the existing queue, and when all the messages are processed, to hae your graph stop automatically. To do so, set the timeout alue in the JMS Reader properties to a number other than zero. The timeout property is set in milliseconds. If you set the timeout alue to, for example, 10,000 milliseconds, the JMS Reader component will wait no more than ten seconds for the next message to come. If ten seconds elapse with no further messages, the graph will cease processing and close. The timeout property's default setting of 0 means to wait foreer (that is, to enforce no timeout). Limiting the CloerETL JMS Reader component max message count for debugging The CloerETL JMS Reader component's maxmsgcount property determines the maximum number of messages to process before ceasing processing and closing the graph. Typically this is set to 0, to allow the graph to process all messages, without maximum. In a debugging situation, howeer, you may want to set the maxmsgcount to a manageable number, for reiewing data and troubleshooting any graph problems. Using the CloerETL JMS Reader max message counts and timeouts together This table shows how the maxmsgcount and timeout alues can be used together for different kinds of processing behaior: Chapter 2. CloerETL graph components 19

28 Table 2. JMS Reader max message count and timeout behaior Max msg count timeout Behaior 0 0 The JMS Reader keeps waiting for new messages. The phase in which this node is embedded neer stops. greater than 0 0 The JMS Reader reads new messages until its count reaches maxmsgcount, regardless of how long it takes. 0 greater than 0 The JMS Reader waits for new messages, as long as the last one arries within the specified number of milliseconds, regardless of how many messages it reads. greater than 0 greater than 0 The JMS Reader stops when the count of read messages reaches the maxmsgcount alue, or when the timeout occurs, whicheer comes first. Managing the CloerETL Edge's message buffer for immediate message processing Use the CloerETL Edge's message buffer to ensure quick processing of messages. About this task The default behaior for Edges connected to a JMS Reader component is to buffer messages, waiting to pass the messages downstream to the next graph component until the buffer is full. In situations where the message traffic is low, this can mean long waits for messages to be processed. To ensure quick processing, you can set a "direct fast propagate" property on the graph's Edges, to direct the graph to output messages as they are processed, regardless of buffer status. This setting must be applied to the Edge of the JMS Reader component's output port and all downstream Edges. Any time you are propagating messages in real time, it is recommended that you use direct fast propagate. These instructions assume the graph has been created, and Edges hae been populated with metadata. Procedure 1. Right-click the Edge connected to the JMS Reader component's output port. 2. From the context menu, choose Edge Type. 3. Set the Edge Type to Direct fast propagate. Note: Repeat these steps for all downstream Edges in the graph. The CloerETL Clean and Deduplicate Sample Graph A sample data cleaning and deduplication graph (clean_and_dedup.grf) is proided with CloerETL to demonstrate erification of a load file. The sections below describe the sample graph components and how to use them. 20 CloerETL User's Guide

29 You must import the sample graph into IBM Initiate Workbench in order to access it in CloerETL. Refer to Importing a CloerETL graph into a project on page 7. Linking The CloerETL Clean and Deduplicate sample Reader to a source file About this task The sample graph includes a Uniersal Data Reader to read sample data from an extract file. In order to use this component, you must link the Reader to your source file. Procedure 1. Double-click the Reader to open the Edit component dialog. 2. Enter a File URL linking to your source data: a. On the Properties tab, click in the Value field for File URL. An ellipsis icon is displayed. b. Click the ellipsis icon to open a file browser. c. Naigate to and select your source data file, and click Open. 3. Click OK to close the Edit component dialog. 4. Check the data for a byte order mark (BOM) at the beginning of the file: a. Right-click on the Reader and choose View Data. b. Choose the number of records to iew (you can accept the default alue of records 1 through 10) and whether to display them in a plain text or grid format. c. Click OK. d. Reiew the data in the View Data dialog. If you see problematic characters at the beginning of the file, you can use a text editor such as Notepad to open and correct the file. Adding metadata for the CloerETL Clean and Deduplicate sample graph The sample graph requires metadata defining your source data. About this task Detailed instructions for adding and applying metadata can be found in Adding CloerETL metadata on page 10 and Applying CloerETL metadata to an Edge on page 11. Note: After you hae added metadata, be sure to propagate it to the rest of the Edges in the graph. Right-click on the Edge containing metadata and choose Propagate metadata. Editing the Ext Filter component in the CloerETL Clean and Deduplicate sample graph The first Transformer in the sample graph is an Ext Filter component that filters records containing problematic characters from the data. A regular expression has been defined for the sample graph's Ext Filter component to perform this filtering. Chapter 2. CloerETL graph components 21

30 About this task Records that meet the criteria (that is, records that contain only alphanumeric characters) are written to the Ext Filter's output port 0, and records that do not meet the criteria are written to output port 1. Before using the graph, you must edit the filter expression in this component to (put the fields in place of placeholder fields, also to add fields you need.) Procedure 1. Double-click on the Ext Filter component to open the Edit component dialog. 2. On the Properties tab, click in the Value field for Filter expression. An ellipsis is displayed in the Value field. 3. Click the ellipsis to open the Filter expression dialog. The sample filter expression is shown in the lower pane on the dialog. Note: Fields from the metadata are displayed in the upper left area of this dialog. If you hae not created and applied metadata, you will not see a list of fields in this area. 4. The first line of the filter expression proides an example of how to check for specific alues in a field: ($Field0 ~= "[AB]") and // Can be A or B This example erifies that the alue in Field0 is A or B. You can edit this line as needed to check for specific alues, or comment it out by adding two forward slash (/) characters to the beginning of the line. Note: To replace Field0 with a field from the metadata, double-click Field0 to highlight it, then double-click the alue in the list of metadata fields to insert it in the filter expression in place of Field0. 5. The second line in the filter expression checks that a Field1 contains only alphanumeric characters: ($Field1 ~= "[0-9A-Za-z]+") and // Can be alphanumeric characters You can copy and edit this field as needed to filter for alphanumeric characters. Substitute the field name in your data for Field1 as needed. 6. The third line of the filter expression checks that Field2 contains only alphanumeric or empty characters: (isnull($field2) or ($Field2 ~= "[0-9A-Za-z]+")) and // Can be Null (empty) or alphanumeric characters You can copy and edit this field as needed to filter for alphanumeric and empty characters. Substitute the field name in your data for Field2 as needed. 7. The fourth line of the filter expression checks that Field3 contains only alphanumeric and designated other characters (double quotes, forward slash, single quote, dash, period, comma, and space): (isnull($field3) or ($Field3 ~= "[\\"/ -., 0-9A-Za-z]+")) // Can be Null (empty) or characters listed You can copy and edit this field as needed to filter for alphanumeric and special characters. Substitute the field name in your data for Field3 as needed. 8. If you want to alidate the filter expression, click the Validate button. Any errors encountered during alidation are displayed at the bottom of the dialog. 9. When you hae finished editing the filter expression, click OK to sae the changes and close the Filter expression dialog. 22 CloerETL User's Guide

31 10. Click OK to close the Edit component dialog. Specifying a sort key in the Ext Sort component of the CloerETL Clean and Deduplicate sample graph The Ext Sort component in the sample graph sorts the records from the Ext Filter's output port 0 (that is, records that met the Ext Filter component's filter criteria) and sorts them by a specified key; in the sample graph, the key is RecordID. About this task You must edit this component to specify one or more sort keys for your data. Procedure 1. Double-click the Ext Sort component to open the Edit component dialog. 2. On the Properties tab, click in the Value field for Sort key (in the Basic node). An ellipsis is displayed in the Value field. 3. Click the ellipsis to open the Edit key dialog. 4. Select the field(s) you want to sort on from the Fields list on the left. Use the arrows to moe them to the Key parts list on the right. You can sort on more than one key. The fields' positions in the Key parts list determines the order in which sorting is performed. 5. Click OK to sae the changes and close the Edit key dialog. 6. To choose whether to sort in ascending or descending order, click in the Value field for Sort order (on the Properties tab in the Basic node) and choose Ascending or Descending. Note: If a sort order is not specified, the component will default to sorting in ascending order. 7. Click OK to close the Edit component dialog. Specifying a dedup key in the CloerETL Clean and Deduplicate the sample graph The Dedup component remoes duplicate records from the sample data, based on specified keys; in the sample graph, the key is RecordID.. About this task Unique records are written to output port 0, and duplicates are written to output port 1. Note: In order for the Dedup component to work, you must sort the data based on a dedup key first so that records with duplicate key fields are in sequence. You must edit this component to specify one or more dedup keys for your data. Procedure 1. Double-click the Dedup component to open the Edit component dialog. 2. On the Properties tab, click in the Value field for Dedup key (in the Basic node). An ellipsis is displayed in the Value field. 3. Click the ellipsis to open the Edit key dialog. 4. Select the field(s) on which you want to deduplicate from the Fields list on the left. Use the arrows to moe them to the Key parts list on the right. You can Chapter 2. CloerETL graph components 23

32 deduplicate on more than one key. The fields' positions in the Key parts list determines the order in which deduplication is performed. 5. Click OK to sae the changes and close the Edit key dialog. 6. To specify whether to keep the first or last copy of a duplicate record, click in the Value field for Keep (on the Properties tab in the Basic node) and choose First or Last. 7. Click OK to close the Edit component dialog. Writers in the CloerETL Clean and Deduplicate sample graph The Writers in the sample graph write output from the filer and deduplication transformations to a series of output files. The Ext Filter component writes dirty records (that is, records that do not meet the filter criteria) to a Uniersal Data Writer labeled Write dirtyextract.txt The Dedup component writes unique records to a Uniersal Data Writer labeled Write cleanextract.txt, and duplicate records to a Uniersal Data Writer labeled Write dupextract.txt To use these Writers, you must specify an external file to which the component writes the data. Note: For data tracking and organization, it is useful to assign each Writer a name that matches or references the name of the external file data is written to. Information on how to rename a component is gien in the steps below. Specifying an external file for data writing in the CloerETL Clean and Deduplicate sample graph Procedure 1. Double-click the Writer to open the Edit component dialog. 2. Enter a File URL linking to the external file. You can either type in a file name, or browse to link to an existing file. To browse to an existing file: a. Click in the Value field for File URL (on the Properties tab). An ellipsis icon is displayed. b. Click the ellipsis icon to open a file browser. c. Naigate to and select the external file, and click Open. Note: If you place the extract file in the folder that contains the project, you can open and iew the output files from the Naigator pane. For example, if the project folder is C:\Program Files\IBM\Initiate\Workbench970\workspace\ SampleProject, put the output files under the..\sampleproject folder to make them appear in the Naigator under SampleProject. You may hae to refresh the Naigator in order for changes to appear. 3. If you want to rename the component to reference the name of the external file, update the Component name alue (in the Visual properties node on the Properties tab). 4. Click OK to close the Edit component dialog. The IBM Initiate Member Model Transform Graph You create this graph by executing a wizard. The graph prepares source data into the database unload format (*.unl files) required for data deriation and for bulk load into the Hub. As such, it proides a graph-based alternatie to the member model transformation part of the mpxdata function. 24 CloerETL User's Guide

33 Note: After Member Model Transform Graph transforms the data to *.unl files, data deriation must be run. You can run mpxfsdd as a command-line function or as a job in IBM Initiate Workbench to perform data deriation. For information about the mpxfsdd job, refer to the IBM Initiate Workbench User's Guide. The IBM Initiate Member Model Transform Graph requires that certain source data is located within your project: Input metadata, in the form of a *.fmt file Prepared source data, such as the output files of the clean_and_dedup.grf sample graph A target directory for the *.unl files produced by the IBM Initiate Member Model Transform graph Input metadata in the Member Model Transform graph There are multiple ways to create an input metadata (*.fmt) for the graph: Export metadata from an existing graph outline Import IBM Initiate Hub metadata (Import > Cloer ETL> IBM Initiate metadata) Import metadata from DDL (Import > Cloer ETL > Import metadata transform from DDL) New Metadata from Database (New > Cloer ETL > Metadata (Database)) New Metadata defined by hand (New > Cloer ETL > Metadata (Define by Hand)) New Metadata from flat file (New > Cloer ETL > Metadata (Flat File)) This document discusses the first two in detail below. For more information on the other ways to create input metadata, refer to the CloerETL Designer User's Guide. Creating a metadata file by exporting metadata About this task You can create an input metadata file by exporting metadata from an existing graph outline. If you hae not defined any metadata at all, see Adding CloerETL metadata on page 10 for information on how to create it. Procedure 1. In the Outline pane, right-click on the metadata you want to create the file from, and choose Export metadata. 2. Select the project folder for the metadata file, and enter a Name for it. 3. Click OK. A.fmt file with the Name you entered is displayed in the Project folder you specified. Chapter 2. CloerETL graph components 25

34 Importing IBM Initiate Hub metadata to the Member Model Transform graph About this task You can create metadata format files directly from data in your IBM Initiate Hub with the Import IBM Initiate metadata feature. This feature enables you to select attributes from which to create metadata. Procedure 1. From the File menu, choose Import. 2. In the Select wizard, expand the Cloer ETL node. 3. Select IBM Initiate metadata, and click Next. 4. In the Import dialog, enter the connection properties for connecting to your Initiate Hub. a. Enter Host, Port, User, and Password alues for connecting to your IBM Initiate Hub. b. If you are using SSL security, check the Use SSL box and select the SSL Version from the list. If you want IBM Initiate Workbench to erify the SSL certificate returned from the Hub, click Verify SSL Certificate. Note: You can click Validate to alidate the connection information. If the connection does not alidate, make sure that the Master Data Engine serice is running, and that you entered correct connection parameters. Note that the Validate option alidates only the host and port, not the user name and password. 5. Click Next. IBM Initiate Workbench connects to the IBM Initiate Hub and retriees attributes. 6. In the Import dialog, use Ctrl-click to select the attributes for which you want to import metadata. You can use the Select All button to select all the listed attributes. 7. Specify a Destination folder for the metadata files. It is good practice to specify a project folder where the Member Model Transform Graph will also reside. 8. Click Finish. The metadata files appear in the Naigation pane. Linking to source data files and output directories in the Member Model Transform graph About this task 26 CloerETL User's Guide Source data for the IBM Initiate Member Model Transform Graph must be in a directory accessible through a Project folder in the Naigator pane. You can link to a file or folder anywhere in your file system in order to make it accessible from the Naigator. While it is not required that the output directory be linked in a Naigator Project folder, you may find it useful to link to the output file in the same manner you link to your source data, so that related files are accessible in the same area. Procedure 1. In the Naigator pane, right-click on Project folder you want to add files or directories to and choose New > Folder (or New > File). 2. In the New Folder or New File dialog, click Adanced. 3. Check the Link to folder (or file) in file system box.

35 4. Click Browse to browse to the file or folder you want to add. 5. Click Finish. Configuring the IBM Initiate Member Model Transform Graph About this task After the metadata, source files, and output directory are ready, you can launch the IBM Initiate Member Model Transform Graph wizard to configure the graph for the data. Procedure 1. In the Naigator pane, right-click on a project and choose New > Other. 2. In the New wizard, choose IBM Initiate Member Model Transform Graph (in the CloerETL node) and click Next. 3. Enter a Name and optional Description for the graph. 4. Click Next. 5. Select the parent folder (that is, the parent project) for the graph, and click Next. 6. Enter configuration information for the graph: Input metadata: Click Browse to browse to and select the metadata format (*.fmt) file for this graph. Note: Refer to Input metadata in the Member Model Transform graph on page 25 for information on how to create input metadata files. Input file: Click Browse to browse to and select the source data to input into this graph. Output directory: Click Browse to browse to and select the directory where the output (*.unl) files will go. Initiate connection properties: Host: Enter the host name for your IBM Initiate Hub. Port: Enter the port for your IBM Initiate Hub. User: Enter the user name for connecting to your Initiate Hub. Password: Enter the password for connecting to your Initiate Hub. Load from file: if you hae created and saed an external Initiate connection, you can browse to it here. Use SSL: If you are using SSL security, check the Use SSL box and select the SSL Version from the list. If you want IBM Initiate Workbench to erify the SSL certificate returned from the Hub, click Verify SSL Certificate. Note: You can click Validate to alidate the connection information. If the connection does not alidate, make sure that the Master Data Engine serice is running and that you entered correct connection parameters. 7. Click Next. 8. The Attributes dialog lists the attributes in your Hub. Select the attributes you want to use (using shift-click or control-click). 9. Click Next. 10. The Model Mapping dialog displays tabs for each of the attributes you selected. Drag input fields from the Input fields column to the Attribute fields (in the Model mapping area) you want to map them to. Chapter 2. CloerETL graph components 27

36 Note: Text alues entered in any of the attribute fields must be enclosed in double quotes. 11. Click Next. A summary screen is displayed. 12. If the summary information on the summary screen is OK, click Finish. Note: When you create an IBM Initiate Member Model Graph, CloerETL generates a lookup table, as a flat file, named SRCHEAD that is used to perform lookups for MemIdent and Memhead sources. You must edit the SRCHEAD lookup table with the sources that are applicable to your project. 13. The graph is displayed in the Graph editor. Click the Run icon to execute the graph. The CloerETL Extract Household Graph The extract_household graph extracts data that represents the association of physical locations (households) and indiiduals liing in them as they hae been grouped by the Hub. The output of the graph is four delimited text files that you can use as source data for other tasks. Creating a project for the CloerETL Extract Household graph About this task To use the extract_household graph, you must create an Initiate project that uses the Householding template. Procedure 1. From the Initiate menu, select New Initiate Project. More detailed information on creating projects is located in the IBM Initiate Workbench User's Guide. 2. In the New Initiate Project - New Initiate Project dialog box, type a descriptie name for the project in the Project Name field, and click Next. 3. In the New Initiate Project - Registered Hubs dialog box, select a hub to associate with the project, and click Next.. 4. In the New Initiate Project - Templates dialog, select the Initiate_Demo_Householding template from the Aailable Templates list, and click Finish. The graph is now aailable in the Naigator pane when you create the project. 5. In the Naigator pane, double-click the extract_household graph to load the graph in the Editor. Customizing the extract_household graph Most specifications that you need to run the graph are pre-defined; howeer, there are configuration settings that you may want to change. The sections below describe how to customize the graph. Editing the Database Reader components for the extract_household graph About this task The extract_household graph uses DBInputTable Reader components that include SQL statements for querying the database. For more information about DBInputTable Readers, refer to the CloerGUI Help. 28 CloerETL User's Guide

37 A database connection is defined in the DBInputTable Readers. To execute the graph you will need to edit the database connection. Procedure 1. In the Outline pane, expand Connections. 2. Right-click the database connection placeholder, and click Edit. 3. In the Database connection dialog box, edit the connection properties and credentials. a. Enter a Name for the database connection. b. Enter the User and Password for connecting to the database. c. Click to select a database drier from the aailable driers window. Note: It is recommended that you use one of the supplied IBM Initiate driers. d. Edit the URL as needed to substitute your connection information. 4. Click the Validate button to alidate the database connection. 5. Click OK. Editing the query for the extract_household graph About this task Follow this procedure if you need to edit the SQL in the Reader components: Procedure 1. Double-click the Reader to open the Edit component dialog box. 2. Click in the Value column of the SQL query property, and click the ellipsis that is displayed. 3. In the SQL query editor, edit the Query text (in the lower pane) as needed. 4. Click OK. Editing the parameters for the extract_household graph About this task The SQL queries written into the database Reader components use parameters, or named constants, to represent attributes in the database and the project location. Howeer, these parameters might not coincide with those in your enironment. Procedure 1. In the Outline pane, expand Parameters. 2. Right-click any one of the parameters, and select Edit from the context menu. 3. In the Graph parameters dialog box, click in the Name and Value columns to make changes as needed. 4. Click Finish to sae the changes. Specifying a location for the extract_household graph's output files About this task The Writer components (relationship_links, delimited_name, delimited_phone, delimited_addr) write output from the deduplication transformations to a series of text files in the output folder; howeer, you can specify an external location for each output file. Chapter 2. CloerETL graph components 29

38 Note: The default File URL uses the {WORKSPACE} parameter to define a relatie path to your workspace. Procedure 1. Double-click a Writer component to open the Edit component dialog box. 2. To edit the output file, click in the Value column of the File URL property, and click the ellipsis that is displayed. 3. In the Path field, you can type a file location, or click the control and browse to a location. 4. Click OK to close the URL File Dialog box. 5. In the Edit component dialog box, complete other customization as needed. For more information about the Uniersal Data Reader component, refer to the CloerGUI Help 30 CloerETL User's Guide

39 Chapter 3. IBM Initiate custom components The CloerETL application in IBM Initiate Workbench includes seeral components that are aailable only to IBM Initiate users. These components behae like IBM Initiate interactions, taking full adantage of the IBM Initiate APIs. The Initiate custom components are categorized as follows: Readers IBM Initiate MemGet IBM Initiate MemSearch IBM Initiate Handler Reader IBM Initiate TskSearch IBM Initiate TskGet Writers IBM Initiate MemPut IBM Initiate MemDelete IBM Initiate Handler Writer IBM Initiate TskPut Transformers IBM Initiate MemScore IBM Initiate Attribute Reformat IBM Initiate MemSeqNo Incrementer IBM Initiate Address Verification Interface Using Key fields in CloerETL MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to join records About this task The IBM Initiate MemPut, IBM Initiate MemSearch, IBM Initiate TskPut, and IBM Initiate Handler Writer components enable you to use Key fields to join records from different input ports. Note: The Master key always comes from data on input port 0, and slae keys come from data on input ports 1-n. Procedure 1. Double-click the component to open the Edit component dialog. 2. Click in the Value field next to the Key fields parameter. An ellipsis is displayed. 3. Click the ellipsis to open the Join key dialog. Note that there is one tab for the Master key, and Slae key tabs for as many other input ports are being used. 4. On the Master key tab, click to highlight a master key field from the Fields list, and click the arrow icon to moe it to the Master key column. 5. On each Slae key tab, chose the field you want to map to the Master key field from the Fields list and drag it to the Slae key field. 6. Click OK on the Join key dialog to sae the updates and close the dialog box. Copyright IBM Corp. 1995,

40 7. Click OK on the Edit component dialog to sae the updates and close the dialog. IBM Initiate custom Reader components This section describes the IBM Initiate Reader components: IBM Initiate MemGet, IBM Initiate MemSearch, IBM Initiate Handler Reader, IBM Initiate IBM Initiate TskSearch, and IBM Initiate TskGet. Comparing the IBM Initiate custom Reader components When deciding which of these custom Initiate Readers to use in your graph, consider the following: The IBM Initiate MemGet component retriees data by using a member key alue, such as an EntRecno, SrcCode and MemIdNum, or MemRecno. The IBM Initiate MemSearch component returns members that, in comparison with the input data, hae a score higher than the minimum score defined. The IBM Initiate Handler Reader component retriees data from the dictionary according to the key inputmember. The IBM Initiate TskGet component retriees task-related data using a known member alue such as an Member ID number (memidnum). or Member Record number (memrecno) in the same manner as the TskGet API interaction The IBM Initiate TskSearch component takes search criteria input (for example, task type, task owner, or task status) and returns a list of members with task issues The IBM Initiate MemGet component The MemGet interaction is a Reader that retriees indiidual members and their attributes, using a known member alue such as an Enterprise ID (EID), or Source Code (srccode) and Member ID number (memidnum). The IBM Initiate MemGet component retriees data in the same manner as the MemGet API interaction. Arguments are supplied to the IBM Initiate MemGet component in the form of component parameters. Note: For additional information on MemGet, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. The IBMInitiate MemGet component requires input from a CloerETL Reader. The Reader supplies a connection to the external source of the data, and can also apply filtering or other criteria to determine which data is passed on to the IBM Initiate MemGet component. CloerETL proides a number of Readers that can read data from sources such as databases, text files, and LDAP repositories. The IBM Initiate MemGet component has one input port and an unlimited number of output ports. Output port 0 is used for successful results of the MemGet interaction. This output port can be linked to a Writer in a CloerETL graph, to write the results of the MemGet interaction to the target destination (such as database table, text file) of your choice. Additional ports 1-n can be used for mapping MemGet output. Different combinations of attributes can be mapped to each output port. For example, first name and last name can be mapped to port 0, while street, city, state are mapped 32 CloerETL User's Guide

41 to port 1. Segment and attribute alues are cached when the Mapping dialog box is opened, eliminating multiple trips to the serer when configuring the component. About IBM Initiate MemGet metadata CloerETL graphs using an IBM Initiate MemGet component use two distinct sets of metadata: the input port metadata defines which data will be retrieed from the IBM Initiate Hub, and the output port metadata defines the data being written to the target destination. In many cases these two sets of metadata will not be identical, and must be defined and applied independently. Creating a graph with an IBM Initiate MemGet component About this task The basic steps required for creating a graph with an IBM Initiate MemGet component are described below. For detailed information about working with Readers and metadata, see Working with CloerETL graphs on page 6. Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page Add a Reader and connect it to the external data source, using the steps outlined in Adding and configuring a CloerETL Reader on page From the Readers section of the Palette, select and drag an IBM Initiate MemGet component onto the graph. 4. Add an Edge to connect the Reader to the IBM Initiate MemGet component. Note: You can also add Transformers between the Reader and the IBM Initiate MemGet component to filter, sort, or otherwise transform the data before passing it to the IBM Initiate MemGet component. 5. Create and apply metadata to the Edge, using the steps outlined in Using CloerETL Edges to connect components on page 10. Note that the metadata you define here describes the data being retrieed from the IBM Initiate Hub. 6. From the Writers section of the Palette, select and drag a Writer onto the graph. This Writer component will process the information retrieed by the IBM Initiate MemGet component. 7. Add an Edge to connect the IBM Initiate MemGet component to the Writer. 8. Create and apply metadata to the Edge, using the steps outlined in Using CloerETL Edges to connect components on page 10. Note that the metadata you define here describes the data being written to the target file. 9. Double-click the IBM Initiate MemGet component to open the Edit Component dialog. 10. Specify a connection to the IBM Initiate Hub by clicking in the Value field for the IBM Initiate connection property and choosing a connection from the drop-down list. Note: If there are no connections in the drop-down, you can create one using the steps outlined in Connections in CloerETL on page Map the IBM Initiate Hub attributes to the fields defined in the output metadata: a. Click in the Value field for the Mapping property, and click the ellipsis control to open the Mapping dialog box. Chapter 3. IBM Initiate custom components 33

42 Note: You must hae an Initiate connection and metadata for the Edges leading into and out of the IBM Initiate MemGet component defined before you can map attributes. b. On the Attributes tab, select the attributes to map using Ctrl-click. Click Next. c. On the Output port tab, drag the metadata Input fields (attributes) to the output fields you want to map them to. d. By clicking the Auto mapping button you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. You can also type in names of attribute alues if the attribute alues you want to use are not in the Input list. Note: The Output fields listed on the Output port tab correspond to the fields in the Output Edge's metadata. If you hae not added a Writer, joined it to the IBM Initiate MemGet component with an Edge, and added metadata to the Edge, as described in the steps aboe, you will not see Output fields on the Output port tab. e. Click Finish. 12. Enter additional properties in the Edit Component dialog, as needed to configure the IBM Initiate MemGet component. Required properties are marked with a yellow warning icon. Property definitions are gien in the MemGet Properties table below. 13. Sae the graph. IBM Initiate MemGet properties The table below lists the properties you can define for the IBM Initiate MemGet component. Note: For additional information on MemGet, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Table 3. MemGet properties Property Initiate connection Get type Key type Member type Entity type Value The MemGet component requires a connection to an IBM Initiate Hub in order to retriee attributes. Detailed instructions on how to create a connection and specify it for this parameter are gien in Connections in CloerETL on page 13. Get type is used to determine the types of data that is retrieed: As member: Returns only members that match the search criteria As entity: Returns members that match the search criteria and any members that are linked to those matching members Key type defines the key that is used for retrieing data: MemIdnum: SrcCode and MemIdnum is the retrieal key MemRecno: MemRecno is the retrieal key EntRecno: EntRecno is the retrieal key Specifies the type of Members to get Specifies the type of Entities to get 34 CloerETL User's Guide

43 Table 3. MemGet properties (continued) Property Value Mapping Use the Mapping parameter to map metadata fields to your IBM Initiate Hub attributes. See Creating a graph with an IBM Initiate MemGet component on page 33 for detailed instructions on mapping. Segment code filter Filters by segment code. You can enter multiple alues in this field, separated by commas. The list of segment codes can be found in the segcode field of the Identity Hub mpi_seghead table. In order to retriee any member segments, at least the MEMHEAD segcode must always be specified. Segment attribute filter Record status filter Source code filter Member status filter Composite iew name Client args Use error port Skip rows In addition to specific segment codes, you can use the following alues as noted: ALL: all segments MEMALL: all member segments DICALL: all dictionary segments AUDALL: all audit segments MEMATTRALL: all member attribute segments Filters results by segment attribute. You can enter multiple alues in this field, separated by commas. Filters results by record status. Although optional, no attributes (only memhead records) will be returned if this is not set. Typically you would want to get at least 'A'ctie attributes. You can select multiple status codes in the dialog box. A: Actie I: Inactie D: Deleted S: Shadow Filters results by source code. You can enter multiple alues in this field, separated by commas. Filters results by member status. You can enter multiple alues in this field, separated by commas. Valid member status alues are: A: Actie O: Oerlay M: Merged D: Deleted Sets the Composite View Name. Composite iews enable Initiate Hub administrators to create and name a iew that controls the data that is returned for the interaction. Use "DEFAULT" to get the default composite iew. Client args are used for external pre- and post-inocation handlers. For more information, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Specifies how the last output port is used: True: port is used an error port. The tab for this port will display ERR_CODE and ERR_MESSAGE attribute fields that you can map to the applicable output fields. False: port is used as a standard port Specifies how many rows should be skipped from the source port. This is useful for handling files where first rows are a header rather than real data. Chapter 3. IBM Initiate custom components 35

44 Table 3. MemGet properties (continued) Property Number of records Data policy Value Specifies how many records should be read from the Initiate Hub source. Specifies how to handle misformatted or incorrect data: Strict: aborts processing Controlled: logs the entire record while processing continues Lenient: attempts to set incorrect data to default alues while processing continues The IBM Initiate MemSearch component The MemSearch interaction uses selected criteria (such as minimum score, name, address, date of birth, or ID number) to search and return a list of records matching the criteria. The interaction creates a irtual member based on the attribute criteria submitted and compares those attributes with other records in the database. Note: For additional information on MemSearch, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. The IBM Initiate MemSearch component requires input from a CloerETL Reader component. The Reader supplies a connection to the external source of the data, and can also apply filtering or other criteria to determine which data is passed on to the Initiate MemSearch component. CloerETL proides a number of Readers that can read data from sources such as databases, text files, and LDAP repositories. Data can be routed to the IBM Initiate MemSearch component ia a Reader in the same way that it can be routed to an Initiate MemPut component, as shown in the illustration in the section The IBM Initiate MemPut component on page 50. Creating a graph with an IBM Initiate MemSearch component About this task The basic steps required for creating a graph with an IBM Initiate MemSearch component are described below. For detailed information about working with Readers and metadata, see Working with CloerETL graphs on page 6. Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page Add a Reader and connect it to your external data source, using the steps outlined in Adding and configuring a CloerETL Reader on page From the Readers section of the Palette, select and drag an IBM Initiate MemSearch component onto the graph. 4. Add an Edge to connect the Reader to the IBM Initiate MemSearch component. Note: You can also add Transformers between the Reader and the Initiate MemSearch component to filter, sort, or otherwise transform the data before passing it to the Initiate MemSearch component. 36 CloerETL User's Guide

45 5. Create and apply metadata to the Edge, using the steps outlined in Using CloerETL Edges to connect components on page 10. Note that the metadata you define here describes the data being retrieed from the Initiate Hub. 6. From the Writers section of the Palette, select and drag a Writer onto the graph. 7. Add an Edge to connect the IBM Initiate MemSearch component to the Writer. 8. Create and apply metadata to the Edges, using the steps outlined in Using CloerETL Edges to connect components on page 10. Note that the metadata you define here describes the data being written to the target file. 9. For multiple outputs, add other components as needed. 10. Double-click the IBM Initiate MemSearch component to open the Edit Component dialog box. 11. Specify a connection to the IBM Initiate Hub by clicking in the Value field for the IBM Initiate connection property and choosing a connection from the drop-down list. Note: If there are no connections in the drop-down, create one by following the steps outlined in Connections in CloerETL on page Map your Initiate Hub attributes to the fields defined in the output metadata: Note: You must hae an IBM Initiate connection and metadata for the Edges leading into and out of the IBM Initiate MemSearch component defined before you can map attributes. a. Click in the Value field for the Input Mapping property. An ellipsis is displayed. b. Click the ellipsis to open the MemSearch input mapping dialog. c. Select the attributes to map by using Ctrl-click. Click Next. d. The MemSearch input mapping dialog displays tabs for each of the attributes you selected. Drag the metadata (input) fields to the attribute fields you want to map them to. Click each tab to expose the input and attribute fields for the selected attributes. e. By clicking the Auto mapping button you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. f. Click Finish. g. In the Edit Component dialog box, click in the Value field for the Output Mapping property. An ellipsis is displayed. h. Click the ellipsis to open the MemSearch output mapping dialog. i. Select the attributes to map using Ctrl-click. Click Next. Note: You can select attributes for output mapping that were not selected for input mapping, and then map the additional attributes to additional output ports. This capability enables you to pass additional data through from the Hub to the output when the search for the selected attributes is successful. j. On the Output port tab, drag the metadata input (attribute) fields to the output fields you want to map them to. Chapter 3. IBM Initiate custom components 37

46 You can also type in names of attribute alues if the attribute alues you wish to use are not present in the Input list. Note: You can define an output port for error codes and error messages. You must define the Value for the property Use error port as True in the Edit component dialog box to enable this functionality. k. Click Finish. 13. Enter additional properties in the Edit Component dialog, as needed to configure the IBM Initiate MemSearch component. Required properties are marked with a yellow warning icon. Property definitions are gien in the MemSearch Properties table below. 14. Sae the graph. IBM Initiate MemSearch properties The table below lists the properties you can define for the IBM Initiate MemSearch component. Note: For additional information on MemSearch, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Table 4. MemSearch properties Property Initiate connection Get type Search type Member type Entity type Input Mapping Output Mapping Key fields Description The MemSearch component requires a connection to an IBM Initiate Hub in order to retriee attributes. Detailed instructions on how to create a connection and specify it for this parameter are gien in Connections in CloerETL on page 13. Get type specifies the type of member-leel aggregation to perform during retrieal operations: AsMember: Returns only members that match the search criteria AsEntity: Returns members that match the search criteria and any members that are linked to those matching members SearchType is used by the interaction classes to specify the type of member-leel aggregation to perform during a search operations. AsMember: Tells the search to compare the input on a member by member basis Unknown: Used as a placeholder when a Search type is not set Member type specifies the Member type to search for. Entity type specifies the Entity type to search for. Use the Input Mapping parameter to map metadata fields to your IBM Initiate Hub attributes. See Creating a graph with an IBM Initiate MemSearch component on page 36 for detailed instructions on mapping. Use the Output Mapping parameter to map metadata fields to attributes for the output data. See Creating a graph with an IBM Initiate MemSearch component on page 36 for detailed instructions on mapping. The Key fields parameter specifies how records from different input ports are joined. See Using Key fields in CloerETL MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to join records on page 31 for detailed instructions on using Key fields. 38 CloerETL User's Guide

47 Table 4. MemSearch properties (continued) Property Description Segment code filter Filters by segment code. You can enter multiple alues in this field, separated by commas. The list of segment codes can be found in the segcode field of the Identity Hub mpi_seghead table. In order to retriee any member segments, at least the MEMHEAD segcode must always be specified. Segment attribute filter Record status filter Source code filter Member status filter Composite iew name Client args Min score Max rows Max candidate In addition to specific segment codes, you can use the following alues as noted: ALL: all segments MEMALL: all member segments DICALL: all dictionary segments AUDALL: all audit segments MEMATTRALL: all member attribute segments Filters results by segment attribute. You can enter multiple alues in this field, separated by commas. Filters results by record status. Although optional, no attributes (only memhead records) will be returned if this is not set. Typically you would want to get at least 'A'ctie attributes. You can select multiple status codes in the dialog box. A: Actie I: Inactie D: Deleted S: Shadow Filters results by source code. You can enter multiple alues in this field, separated by commas. Filters results by member status. You can enter multiple alues in this field, separated by commas. Valid member status alues are: A: Actie O: Oerlay M: Merged D: Deleted Sets the Composite View Name. Composite iews enable Hub administrators to create and name a iew that controls the data that is returned for the interaction. Use "DEFAULT" to get the default composite iew. Client args are used for external pre- and post-inocation handlers. For more information, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Sets the minimum score that a search result must hae in order to be returned in the output. Use zero to indicate that any score is acceptable. Sets the maximum number of members to return in the output. Entities returned are created from the member record results. Use zero to indicate that any number of members or entities may be returned. Sets the maximum number of candidates that will be examined during a search. A setting of zero indicates that any number of candidates will be examined. Chapter 3. IBM Initiate custom components 39

48 Table 4. MemSearch properties (continued) Property Use error port Skip rows Number of records Data policy Description Specifies how the last output port is used: True: port is used an error port. The tab for this port will display ERR_CODE and ERR_MESSAGE attribute fields that you can map to the applicable output fields. False: port is used as a standard port Specifies how many rows should be skipped from the source port. This is useful for handling files where first rows are header rows rather than real data. Specifies how many records should be read from the source. Specifies how to handle misformatted or incorrect data: Strict: aborts processing. Controlled: logs the entire record while processing continues Lenient: attempts to set incorrect data to default alues while processing continues The IBM Initiate Handler Reader component The IBM Initiate Handler Reader component retriees data from the dictionary. It passes attributes from a member row list into a graph where they can be reformatted for temporary display and use, or passed back to the dictionary or to a permanent file by a Writer component. The IBM Initiate Handler Reader component has no input ports; the data is passed in from the dictionary through an Initiate connection. There are multiple output ports aailable for connecting to components such as Transformers and Joiners. Creating a graph with an IBM Initiate Handler Reader component About this task The basic steps required for creating a graph with an IBM Initiate Handler Reader component are described below. For detailed information about working with Readers and metadata, see Working with CloerETL graphs on page 6. Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page From the Readers section of the Palette, select and drag an IBM Initiate Handler Reader component onto the graph. 3. Add the appropriate components, such as Readers and Transformers, needed to complete the graph. 4. Create and apply metadata to the Edges, using the steps outlined in Using CloerETL Edges to connect components on page 10. Note that the metadata you define here describes the data being retrieed from the IBM Initiate Hub. 5. Double-click the IBM Initiate Handler Reader component to open the Edit Component dialog. 6. Specify a connection to the IBM Initiate Hub by clicking in the Value field for the IBM Initiate connection property and choosing a connection from the drop-down list. 40 CloerETL User's Guide

49 Note: If there are no connections in the drop-down, you can create one using the steps outlined in Connections in CloerETL on page Click Apply. 8. Map the IBM Initiate Hub attributes to the fields defined in the output metadata: a. Click in the Value field for the Mapping property, and click the ellipsis control to open the Mapping dialog box. b. Select the attributes to map using Ctrl-click. c. Click Next. d. On the Output port tab, drag the metadata input (attribute) fields to the output fields you want to map them to. Note: The Output fields listed on the Output port tab correspond to the fields in the Output Edge's metadata. If you hae not added a Writer, joined it to the IBM Initiate Handler Reader component with an Edge, and added metadata to the Edge, as described in the steps aboe, you will not see Output fields on the Output port tab. e. By clicking the Auto mapping button you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. 9. Click Finish. 10. Enter additional properties in the Edit Component dialog as needed to configure the IBM Initiate Handler Reader component. Required properties are marked with a yellow warning icon. Property definitions are gien in the Handler Reader Properties table below. 11. Sae the graph. IBM Initiate Handler Reader properties The table below lists the properties you can define for the IBM Initiate Handler Reader component. Table 5. Handler Reader properties Property Initiate connection Dictionary key Mapping Component name Description ID Component type Specification Value The IBM Initiate Handler Reader component requires a connection to an IBM Initiate Hub in order to retriee attributes. Detailed instructions on how to create a connection and specify it for this property are gien in Connections in CloerETL on page 13. Specifies the key that the component uses to find member rows in the dictionary. The default alue is inputmember. Use the Mapping property to map metadata fields to your IBM Initiate Hub attributes. Default label for the component. You can change this label to make it more specific to your business uses. Optional user-defined explanation of the component's use. Internal identification. Internal identification. Description of the component's use. Chapter 3. IBM Initiate custom components 41

50 Table 5. Handler Reader properties (continued) Property Value Phase For graph executions that include multiple graphs running sequentially, indicates the location in the sequence for the graph containing this component. Phase identifiers begin with 0 (zero). Enabled Status of the component within the graph. The options are: enabled the component is actie in the graph. It will parse data and pass it to the next component. disabled the component is inactie in the graph. A disabled component will neither parse data nor pass data to the next component. passthrough the component is inactie in the graph. It will not parse data, but will pass data to the next component Input port If the component's status is passthrough, enables you to specify which input port should receie the data records and which Output port output port should pass them to the next component. The IBM Initiate TskGet component The IBM Initiate TskGet component retriees task-related data using a known member alue such as an Member ID number (memidnum). or Member Record number (memrecno) in the same manner as the TskGet API interaction. The TskGet performs a cross-match using a member or entity task record as input to the cross match, and returns all members in the same entity as the input member, as well as any members that match aboe the Clerical Reiew threshold. Arguments are supplied to the IBM Initiate TskGet component in the form of component parameters. Note: For additional information on TskGet, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. The Initiate TskGet component has one input port and an unlimited number of output ports. Output port 0 is used for successful results of the TskGet interaction. This output port can be linked to a Writer in a CloerETL graph to write the results of the TskGet interaction to the target destination (such as database table, text file) of your choice. Additional ports 1-n can be used for mapping TskGet output. Different combinations of attributes can be mapped to each output port. About IBM Initiate TskGet metadata CloerETL graphs using an IBM Initiate TskGet component use two distinct sets of metadata: the input port metadata defines which data will be retrieed from the IBM Initiate Hub, and the output port metadata defines the data being written to the target destination. In many cases these two sets of metadata will not be identical, and must be defined and applied independently. Creating a graph with an IBM Initiate TskGet component About this task The basic steps required for creating a graph with an IBM Initiate TskGet component are described below. For detailed information about working with Readers and metadata, see Working with CloerETL graphs on page CloerETL User's Guide

51 Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page Add a Reader and connect it to the external data source, using the steps outlined in Adding and configuring a CloerETL Reader on page From the Readers section of the Palette, select and drag an IBM Initiate TSKGET component onto the graph. 4. Add an Edge to connect the Reader to the IBM Initiate TSKGET component. Note: You can also add Transformers between the Reader and the IBM Initiate TskGet component to filter, sort, or otherwise transform the data before passing it to the IBM Initiate TskGet component. 5. Create and apply metadata to the Edge, using the steps outlined in Using CloerETL Edges to connect components on page 10. Note that the metadata you define here describes the data being retrieed from the IBM Initiate Hub. 6. From the Writers section of the Palette, select and drag one or more Writers onto the graph. The Writer components will process the information retrieed by the IBM Initiate TSKGET component. 7. Add one or more Edges to connect the IBM Initiate TSKGET component to the Writer(s). 8. Create and apply metadata to the Edge(s), using the steps out lined in Using CloerETL Edges to connect components on page 10. Note that the metadata you define here describes the data being written to the target file. 9. Double-click the IBM Initiate TSKGET component to open the Edit Component dialog. 10. Specify a connection to the Initiate Hub by clicking in the Value field for the IBM Initiate connection property and choosing a connection from the drop-down list. Note: If there are no connections in the drop-down, you can create one using the steps outlined in Connections in CloerETL on page Map the IBM Initiate Hub attributes to the fields defined in the output metadata: a. Click in the Value field for the Mapping property, and click the ellipsis control to open the Mapping dialog box. Note: You must hae an Initiate connection and metadata for the Edges leading into and out of the IBM Initiate TSKGET component defined before you can map attributes. b. Select the attributes to map using Ctrl-click. Click Next.. Note: If you map non-segment attributes for output, such as entxtsk, memxtsk, or idtxtsk, you must also select MEMHEAD or no records will be returned. c. On the Output port tab(s), drag the metadata input (attribute) fields to the output fields you want to map them to. d. By clicking the Auto mapping button you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. Chapter 3. IBM Initiate custom components 43

52 You can also type in names of attribute alues if the attribute alues you want to use are not in the Input list. Note: The Output fields listed on the Output port tab correspond to the fields in the Output Edge's metadata. If you hae not added a Writer, joined it to the IBM Initiate TskGet component with an Edge, and added metadata to the Edge, as described in the steps aboe, you will not see Output fields on the Output port tab. e. Click Finish. 12. Enter additional properties in the Edit Component dialog, as needed to configure the IBM Initiate TSKGET component. Required properties are marked with a yellow warning icon. Property definitions are gien in the TSKGET Properties table below. 13. Sae the graph. IBM Initiate TSKGET properties The table below lists the properties you can define for the IBM Initiate TSKGET component. Note: For additional information on TSKGET, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Table 6. TSKGET properties Property IBM Initiate connection Get type Key type Member type Entity type Mapping Value The TSKGET component requires a connection to an Initiate Hub in order to retriee attributes. Detailed instructions on how to create a connection and specify it for this parameter are gien in Connections in CloerETL on page 13. Get type is used to determine the types of data that is retrieed: As member: Returns only members that match the search criteria As entity: Returns members that match the search criteria and any members that are linked to those matching members Key type defines the key that is used for retrieing data: TskRecno: TskRecno is the retrieal key MemIdnum: SrcCode and MemIdnum is the retrieal key MemRecno: MemRecno is the retrieal key EntRecno: EntRecno is the retrieal key Specifies the type of Members to get Specifies the type of Entities to get Use the Mapping parameter to map metadata fields to your IBM Initiate Hub attributes. See Creating a graph with an IBM Initiate TskGet component on page 42 for detailed instructions on mapping. 44 CloerETL User's Guide

53 Table 6. TSKGET properties (continued) Property Value Segment code filter Filters by segment code. You can enter multiple alues in this field, separated by commas. The list of segment codes can be found in the segcode field of the Identity Hub mpi_seghead table. In order to retriee any member segments, at least the MEMHEAD segcode must always be specified. Segment attribute filter Record status filter Source code filter Member status filter Client args Use error port Skip rows Number of records Data policy In addition to specific segment codes, you can use the following alues as noted: ALL: all segments MEMALL: all member segments DICALL: all dictionary segments AUDALL: all audit segments MEMATTRALL: all member attribute segments Filters results by segment attribute. You can enter multiple alues in this field, separated by commas. Filters results by record status. Although optional, no attributes (only memhead records) will be returned if this is not set. Typically you would want to get at least 'A'ctie attributes. You can select multiple status codes in the dialog box. A: Actie I: Inactie D: Deleted S: Shadow Filters results by source code. You can enter multiple alues in this field, separated by commas. Filters results by member status. You can enter multiple alues in this field, separated by commas. Valid member status alues are: A: Actie O: Oerlay M: Merged D: Deleted Client args are used for external pre- and post-inocation handlers. For more information, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Specifies how the last output port is used: True: port is used an error port. The tab for this port will display ERR_CODE and ERR_MESSAGE attribute fields that you can map to the applicable output fields. False: port is used as a standard port Specifies how many rows should be skipped from the source port. This is useful for handling files where first rows are a header rather than real data. Specifies how many records should be read from the IBM Initiate Hub source. Specifies how to handle misformatted or incorrect data: Strict: aborts processing. Controlled: logs the entire record while processing continues Lenient: attempts to set incorrect data to default alues while processing continues Chapter 3. IBM Initiate custom components 45

54 Table 6. TSKGET properties (continued) Property Adanced > IxnTskGet ersion Value When using the Adanced Issue Management feature's implementation-defined tasks, it is recommended that you use the IxnTskGet2 ersion. IxnTskGet2 is similar to the traditional IxnTskGet get interaction, but IxnTskGet2 populates the new MEMRECNOs array in all task types. The traditional IxnTskGet interaction leaes this field null. In addition, IxnTskGet2 "rolls up" idtxtsk records with the same TSKRECNO into a set so that only one implementation-defined task object (idtxtsk) is returned for a taskset. This differs from the traditional IxnTskGet, which will preset all mpi_idtxtsk records. The IBM Initiate TskSearch component The TskSearch interaction takes search criteria input (for example, task type, task owner, or task status) and returns a list of members with task issues. Note: For additional information on TskSearch, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. The IBM Initiate TskSearch component requires input from a CloerETL Reader component. The Reader supplies a connection to the external source of the data, and can also apply filtering or other criteria to determine which data is passed on to the Initiate TskSearch component. CloerETL proides a number of Readers that can read data from sources such as databases, text files, and LDAP repositories. Data can be routed to the IBM Initiate TskSearch component ia a Reader in the same way that it can be routed to an IBM Initiate MemPut component, as shown in the illustration in the section The IBM Initiate MemPut component on page 50. Creating a graph with an IBM Initiate TskSearch component About this task The basic steps required for creating a graph with an IBM Initiate TskSearch component are described below. For detailed information about working with Readers and metadata, see Working with CloerETL graphs on page 6. Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page Add a Reader and connect it to your external data source, using the steps outlined in Adding and configuring a CloerETL Reader on page From the Readers section of the Palette, select and drag an IBM Initiate TskSearch component onto the graph. 4. Add an Edge to connect the Reader to the IBM Initiate TskSearch component. Note: You can also add Transformers between the Reader and the IBM Initiate TskSearch component to filter, sort, or otherwise transform the data before passing it to the IBM Initiate TskSearch component. 46 CloerETL User's Guide

55 5. Create and apply metadata to the Edge, using the steps outlined in Using CloerETL Edges to connect components on page 10. Note that the metadata you define here describes the data being retrieed from the IBM Initiate Hub. 6. From the Writers section of the Palette, select and drag one or more Writers onto the graph. 7. Add one or more Edge(s) to connect the IBM Initiate TskSearch component to the Writer(s). 8. Create and apply metadata to the Edges, using the steps outlined in Using CloerETL Edges to connect components on page 10. Note that the metadata you define here describes the data being written to the target file. 9. Double-click the IBM Initiate TskSearch component to open the Edit Component dialog box. 10. Specify a connection to the Initiate Hub by clicking in the Value field for the IBM Initiate connection property and choosing a connection from the drop-down list. Note: If there are no connections in the drop-down, create one by following the steps outlined in Connections in CloerETL on page Map your IBM Initiate Hub attributes to the fields defined in the output metadata: Note: You must hae an IBM Initiate connection and metadata for the Edges leading into and out of the IBM Initiate TskSearch component defined before you can map attributes. a. Click in the Value field for the Input Mapping property. An ellipsis is displayed. b. Click the ellipsis to open the TskSearch input mapping dialog. c. Select the attributes to map by using Ctrl-click. Click Next. d. The TskSearch input mapping dialog displays tabs for each of the attributes you selected. Drag the metadata (input) fields to the attribute fields you want to map them to. Click each tab to expose the input and attribute fields for the selected attributes. e. By clicking the Auto mapping button you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. f. Click Finish. g. In the Edit Component dialog box, click in the Value field for the Output Mapping property. An ellipsis is displayed. h. Click the ellipsis to open the TskSearch output mapping dialog. i. Select the attributes to map using Ctrl-click. Click Next. Note: You can select attributes for output mapping that were not selected for input mapping, and then map the additional attributes to additional output ports. This capability enables you to pass additional data through from the Hub to the output when the search for the selected attributes is successful. j. On the Output port tabs, drag the metadata input (attribute) fields to the output fields you want to map them to. Chapter 3. IBM Initiate custom components 47

56 You can also type in names of attribute alues if the attribute alues you wish to use are not present in the Input list. Note: In the example aboe, the last output port is being used for error codes and error messages. You must define True as the Value for the property Use error port in the Edit component dialog box to enable this functionality. k. Click Finish. 12. Enter additional properties in the Edit Component dialog, as needed to configure the IBM Initiate TskSearch component. Required properties are marked with a yellow warning icon. Property definitions are gien in the TskSearch Properties table below. 13. Sae the graph. IBM Initiate TskSearch properties The table below lists the properties you can define for the IBM Initiate TskSearch component. Note: For additional information on TskSearch, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Table 7. TskSearch properties Property IBM Initiate connection Member type Entity type Input Mapping Output Mapping Key fields MemRecnos key fields Description The TskSearch component requires a connection to an IBM Initiate Hub in order to retriee attributes. Detailed instructions on how to create a connection and specify it for this parameter are gien in Connections in CloerETL on page 13. Member type specifies the Member type to search for. Entity type specifies the Entity type to search for. Use the Input Mapping parameter to map metadata fields to your IBM Initiate Hub attributes. See Creating a graph with an IBM Initiate TskSearch component on page 46 for detailed instructions on mapping. Use the Output Mapping parameter to map metadata fields to attributes for the output data. See Creating a graph with an IBM Initiate TskSearch component on page 46 for detailed instructions on mapping. The Key fields parameter specifies how records from different input ports are joined. See Using Key fields in CloerETL MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to join records on page 31 for detailed instructions on using Key fields. Key fields of input metadata which group multiple memrecnos to IdtXtsk rows. Use this option if you want to find multiple members associated with a single custom task instance; the custom task instance is associated with a primary member record in one input file, and the additional member records are mapped to the primary member record. 48 CloerETL User's Guide

57 Table 7. TskSearch properties (continued) Property Description Segment code filter Filters by segment code. You can enter multiple alues in this field, separated by commas. The list of segment codes can be found in the segcode field of the Identity Hub mpi_seghead table. In order to retriee any member segments, at least the MEMHEAD segcode must always be specified. Segment attribute filter Record status filter Source code filter Member status filter Client args Max rows Max candidate Use error port Skip rows In addition to specific segment codes, you can use the following alues as noted: ALL: all segments MEMALL: all member segments DICALL: all dictionary segments AUDALL: all audit segments MEMATTRALL: all member attribute segments Filters results by segment attribute. You can enter multiple alues in this field, separated by commas. Filters results by record status. Although optional, no attributes (only memhead records) will be returned if this is not set. Typically you would want to get at least 'A'ctie attributes. You can select multiple status codes in the dialog box. A: Actie I: Inactie D: Deleted S: Shadow Filters results by source code. You can enter multiple alues in this field, separated by commas. Filters results by member status. You can enter multiple alues in this field, separated by commas. Valid member status alues are: A: Actie O: Oerlay M: Merged D: Deleted Client args are used for external pre- and post-inocation handlers. For more information, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Sets the maximum number of members to return in the output. Entities returned are created from the member record results. Use zero to indicate that any number of members or entities may be returned. Sets the maximum number of candidates that will be examined during a search. A setting of zero indicates that any number of candidates will be examined. Specifies how the last output port is used: True: port is used an error port. The tab for this port will display ERR_CODE and ERR_MESSAGE attribute fields that you can map to the applicable output fields. False: port is used as a standard port Specifies how many rows should be skipped from the source port. This is useful for handling files where first rows are header rows rather than real data. Chapter 3. IBM Initiate custom components 49

58 Table 7. TskSearch properties (continued) Property Description Number of records Specifies how many records should be read from the source. Data policy Specifies how to handle misformatted or incorrect data: Strict: aborts processing. Controlled: logs the entire record while processing continues Lenient: attempts to set incorrect data to default alues while processing continues IBM Initiate custom Writer components This section describes the IBM Initiate Writer components: IBM Initiate MemPut, IBM Initiate MemDelete, and IBM Initiate Handler Writer. The IBM Initiate MemPut component A MemPut interaction inserts or updates member data in the Hub database. The IBM Initiate MemPut component processes and inserts or updates data in the same manner as the MemPut API interaction. Arguments are supplied to the IBM Initiate MemPut component in the form of component parameters. Note: For additional information on MemPut, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. The IBM Initiate MemPut component requires input from a CloerETL Reader. The Reader supplies a connection to the external source of the data, and can also apply filtering or other criteria to determine which data is passed on to the MemPut component. CloerETL proides a number of Readers that can read data from sources such as databases, text files, and LDAP repositories. The diagram below illustrates some of the ways you can route data to the MemPut component ia a Reader. 50 CloerETL User's Guide

59 Creating a graph with an IBM Initiate MemPut component About this task The basic steps required for creating a graph with an IBM Initiate MemPut component are described below. For detailed information about working with Readers and metadata, see Working with CloerETL graphs on page 6. Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page Add a Reader and connect it to the external data source, using the steps outlined in Adding and configuring a CloerETL Reader on page From the Writers section of the Palette, select and drag an IBM Initiate MemPut component onto the graph. 4. Add an Edge to connect the Reader to the IBM Initiate MemPut component Note: You can also add Transformers between the Reader and the IBM Initiate MemPut component to filter, sort, or otherwise transform the data before passing it to the IBM Initiate MemPut component. 5. Create and apply metadata to the Edge, using the steps outlined in Using CloerETL Edges to connect components on page Double-click the IBM Initiate MemPut component to open the Edit Component dialog box. 7. Map IBM Initiate Hub attributes to the fields defined in the metadata: a. Ensure that you hae defined an IBM Initiate connection. b. Click in the Value field for the Mapping property, and then click the ellipsis control to open the MemPut Mapping dialog. c. Select the attributes to map using Ctrl-click. d. Click Next. e. The MemPut Mapping dialog displays tabs for each of the attributes you selected. Drag the metadata (input) fields to the attribute fields you want to map them to. f. By clicking the Auto mapping button you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. g. You can specify a row indicator setting for an indiidual attribute that oerrides a global row indicator setting in the Edit component dialog box. 1) Select the attribute. 2) Beside the RowInd/Filter table, click the green plus (+) button. CloerETL inserts a row indicator alue of Insert into the table. 3) To change the alue, click the row in the RowInd column to display the drop-down list, and then select the applicable alue. 4) To add a filter expression to the row indicator alue, click the row in the Filter column to display the Filter dialog box. Build the expression and click OK. h. Click Finish. 8. If you want to map failure messages to an output file: a. Add a Writer to capture the failure message. Chapter 3. IBM Initiate custom components 51

60 b. Connect the Writer to the MemPut component output port 1, and add Metadata to the Edge. c. Open the MemPut component and click Output mapping, to map your output as needed. When mapping failure messages, keep in mind the following: Failure output should include the memrecno. Is it recommended that failure output include ERR_CODE and ERR_MESSAGE. 9. Enter additional properties as needed to configure the IBM Initiate MemPut component. Required properties are marked with a yellow warning icon. Property definitions are gien in Initiate MemPut properties. 10. Sae the graph. Initiate MemPut properties The table below lists the properties you can define for the Initiate MemPut component. Note: For additional information on MemPut, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Table 8. MemPut properties Property Initiate connection Description Specify a connection to the Initiate Hub by clicking the Value field and selecting from the drop-down list. The MemPut component requires a connection to an Initiate Hub in order to retriee attributes. If no connections are listed, you can create one using the steps outlined in Connections in CloerETL on page 13. Put type PutType limits the types of changes that can occur for a gien interaction. For example, you might want to update a Task. By using the UpdateOnly setting, the Task will only be updated if it already exists. By using the InsertOnly setting, the Task will only be inserted if it does not already exist. InsertUpdate: If the member data does not exist, create it; or if the member data already exists, update it InsertOnly: If the member data does not exist, create it; or if the member data already exists, abort the interaction UpdateOnly: If the member data already exists, update it; or if the member data does not exist, abort the interaction 52 CloerETL User's Guide

61 Table 8. MemPut properties (continued) Property Description Member mode Member mode specifies the type of input data present in the member data presented to the interaction: Partial: The input member attributes do not contain a complete picture of all the members attributes. The attributes will be inserted and no oerlay tasks will be created. AttrComp: For each gien attribute, all alues for that attribute are present in the input member, so the Engine can make decisions regarding the Actie/Inactie status of historical attributes. During put processing, this flag indicates that attributes in the database that share the same attrrecno, but are not included in the input should hae their recstat set to Inactie. Complete: Complete member is present in the input record. This is the only setting that supports the creation of Oerlay tasks. The Engine can compare this complete input member to the image in the database to make decisions regarding the creation of an oerlay task. Explicit: The input member is used as a map for what to insert/update/delete. No oerlay tasks will be created Match mode Match mode specifies the type of matching to perform after the put interaction: Immediate: Cross match the member immediately following the update of the member data. This option has no additional effect in asynchronous mode Deferred: Member matches when Entity Manager gets to this spot in the queue DoNothing: Update the member data, but do not cross match the updated data against other members in the Hub database. The interaction returns control to the caller as soon as the update is complete. This blocks this member from being matched when it is put, but another put that has buckets in common could cause this member to be matched. Mapping Use the Mapping parameter to map metadata fields to your Hub database attributes. Refer to Creating a graph with an IBM Initiate MemPut component on page 51 for more information on mapping. Output Mapping If you want to map failure messages to a file, use the output mapping option to map failure messages to port 1. Key fields The Key fields parameter specifies how records from different input ports are joined. See Using Key fields in CloerETL MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to join records on page 31 for detailed instructions on using Key fields. Entity type Specifies the type of Entity to process. Client args Client args are used for external pre- and post-inocation handlers. For more information, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Chapter 3. IBM Initiate custom components 53

62 Table 8. MemPut properties (continued) Property RowInd Description Specifies a global action that the Master Data Engine should perform on all rows that are passed in. The alues are: (I)gnore (I)nsert (U)pdate (D)elete The RowInd property is also aailable for indiidual attributes. The indiidual attribute setting oerrides the global setting. Number of skipped records Number of written records Specifies how many records/rows should be skipped from the source port. This is useful for handling files where first rows are a header not a real data. Specifies how many members should be written to the Initiate Hub. If no number is specified, the number of records written is not limited. The IBM Initiate MemDelete component A MemDelete component inactiates member data in the Hub database in the same manner as the MemDelete API interaction. Arguments are supplied to the IBM Initiate MemDelete component in the form of component parameters. Note: For additional information on MemDelete, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. The IBM Initiate MemDelete component requires input from a CloerETL Reader component. The Reader supplies a connection to the external source of the data, and can also apply filtering or other criteria to determine which data is passed on to the IBM Initiate MemDelete component. Creating a graph with the IBM Initiate MemDelete component About this task The basic steps required for creating a graph with an IBM Initiate MemDelete component are described below. For detailed information about working with Readers and metadata, see Working with CloerETL graphs on page 6. Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page Add a Reader and connect it to your external data source, using the steps outlined in Adding and configuring a CloerETL Reader on page From the Writers section of the Palette, select and drag an IBM Initiate MemDelete component onto the graph. 4. Add an Edge to connect the Reader to the IBM Initiate MemDelete component. Note: To filter, sort, or otherwise transform the data before passing it to the IBM Initiate MemDelete component, add Transformers between the Reader and the IBM Initiate MemDelete component. 5. Press [ESC] or click Select in the Palette to turn off the Edge tool. 54 CloerETL User's Guide

63 6. Create and apply metadata to the Edge, using the steps outlined in Using CloerETL Edges to connect components on page Double-click the IBM Initiate MemDelete component to open the Edit Component dialog. 8. Specify a connection to the IBM Initiate Hub by clicking in the Value field for the IBM Initiate connection property and choosing a connection from the drop-down list. If no connections exist in the drop-down list, refer to Connections in CloerETL on page 13 to create a connection. 9. Click the Value field for the Key type property to select the key to be used for retrieing data. 10. Specify the mapping of metadata fields to the IBM Initiate Hub attributes by clicking the Value field of the Mapping property to open the Mapping dialog box. Refer to Creating a graph with an IBM Initiate MemSearch component on page 36 for more information about mapping. 11. If you want to map failure messages to an output file: a. Add a Writer to capture the failure message. b. Connect the Writer to the MemDelete component output port 1, and add Metadata to the Edge. c. Open the MemDelete component and click Output mapping, to map your output as needed. When mapping failure messages, keep in mind the following: Failure output should include the memrecno. Is it recommended that failure output include ERR_CODE and ERR_MESSAGE. 12. Specify other properties as needed, and then click OK to sae the graph. IBM Initiate MemDelete properties This table lists the properties you can define for the IBM Initiate MemDelete component. Note: For additional information on MemDelete, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Table 9. MemDelete properties Property IBM Initiate connection Key type Values The MemDelete component requires a connection to an IBM Initiate Hub in order to retriee attributes. Detailed instructions on how to create a connection and specify it for this parameter are gien in Connections in CloerETL on page 13. Key type defines the key that is used for retrieing data: MemIdnum: SrcCode and MemIdnum combination is the retrieal key. MemRecno: MemRecno is the retrieal key. Key types are mutually exclusie. Eery input record is mapped to a single MemDelete execution. Mapping Output mapping Use the Mapping parameter to map metadata fields to your Hub database attributes. If you want to map failure messages to a file, use the output mapping option to map failure messages to port 1. Chapter 3. IBM Initiate custom components 55

64 The IBM Initiate Handler Writer component The IBM Initiate Handler Writer component writes data to a shared resource or temporary memory. It is often used to display reformatted data for temporary use rather than to produce a permanent file or update a storage system. The IBM Initiate Handler Writer component has no input ports; the data is passed in from the dictionary through an IBM Initiate connection. There are multiple output ports aailable for connecting to components such as Transformers and Joiners. Creating a graph with an IBM Initiate Handler Writer component About this task The basic steps required for creating a graph with an IBM Initiate Handler Reader component are described below. For detailed information about working with Readers and metadata, see Working with CloerETL graphs on page 6. Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page From the Readers section of the Palette, select and drag a Reader component onto the graph. 3. From the Writers section of the Palette, select and drag an IBM Initiate Handler Writer component onto the graph. 4. Add intermediate components needed to complete the graph. 5. Create and apply metadata to the Edges, using the steps outlined in Using CloerETL Edges to connect components on page Double-click each component to open its Edit Component dialog and edit the component's properties. 7. In the IBM Initiate Handler Writer Edit component dialog, specify a connection to the IBM Initiate Hub by clicking in the Value field for the IBM Initiate connection property and choosing a connection from the drop-down list. Click Apply. Note: If there are no connections in the drop-down, you can create one using the steps outlined in Connections in CloerETL on page Map the IBM Initiate Hub attributes to the fields defined in the output metadata: a. Click in the Value field for the Mapping property, and click the ellipsis control to open the Mapping dialog box. b. Select the attributes to map by using Ctrl-click. c. Click Next. d. Drag the metadata input (attribute) fields to the output fields you want to map them to. e. By clicking the Auto mapping button you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. f. Click Finish. 9. Enter additional properties in the Edit Component dialog, as needed to configure the IBM Initiate Handler Writer component. Required properties are 56 CloerETL User's Guide

65 marked with a yellow warning icon. Property definitions are gien in IBM Initiate Handler Writer properties. 10. Sae the graph. IBM Initiate Handler Writer properties The table below lists the properties you can define for the IBM Initiate Handler Writer component. Table 10. Handler Writer properties Property IBM Initiate connection Dictionary key Mapping Key fields Component name Description ID Component type Specification Phase Enabled Input port Output port Value The Handler Writer component requires a connection to an IBM Initiate Hub in order to display attributes. Detailed instructions on how to create a connection and specify it for this property are gien in Connections in CloerETL on page 13. Specifies the key that the component uses to store member rows in the dictionary. The default alue is outputmember. Use the Mapping property to map metadata fields to your IBM Initiate Hub attributes. The Key fields property specifies how records from different input ports are joined. See Using Key fields in CloerETL MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to join records on page 31 for detailed instructions on using Key fields. Default label for the component. You can change this label to make it more specific to your business uses. Optional user-defined explanation of the component's use. Internal identification. Internal identification. Description of the component's use. For graph executions that include multiple graphs running sequentially, indicates the location in the sequence for the graph containing this component. Phase identifiers begin with 0 (zero). Status of the component within the graph. The options are: enabled the component is actie in the graph. It will parse data and pass it to the next component. disabled the component is inactie in the graph. A disabled component will neither parse data nor pass data to the next component. passthrough the component is inactie in the graph. It will not parse data, but will pass data to the next component If the component's status is passthrough, enables you to specify which input port should receie the data records and which output port should pass them to the next component. The IBM Initiate TskPut Component The IBM Initiate TskPut component is designed to let you insert custom task data into the Hub and associate it with existing member records. This functionality is part of the Adanced Issue Management feature, which allows users to create implementation-defined or custom task types, to supplement Initiate's predefined task types. Chapter 3. IBM Initiate custom components 57

66 For an oeriew of the Adanced Issue Management feature, including information about creating custom task types in IBM Initiate Workbench, determining which data to include as part of your custom tasks, and resoling custom tasks in IBM Initiate Inspector, refer to the IBM Initiate Master Data Serice Hub Oeriew. You should be familiar with the information in the Adanced Issue Management section of the IBM Initiate Master Data Serice Hub Oeriew before creating an IBM Initiate TskPut graph. The IBM Initiate TskPut component takes custom task data from an external source, inserts it into the Hub, and associates it with existing member records which are already present in the Hub. In a CloerETL TskPut graph, custom task data is read into the graph ia a CloerETL Reader component. The Reader supplies a connection to the external source of the data, and can also apply filtering or other criteria to determine which data is passed on to the TskPut component. When planning your IBM Initiate TskPut graph, it is helpful to keep the following in mind: The IBM Initiate TskPut component can be configured to use either IxnTskPut or IxnTskPut2 as the ixntskput ersion. When you are putting custom task data which references more than one member record, always use IxnTskPut2. If you will assign a single custom task instance to multiple member records using IxnTskPut2, you should plan to proide separate input for the custom task data and for the member record data. You can map custom task data to member record data in the TSKPUT component ia key fields. The sample graph in the following section illustrates how this is done. When mapping your ouput, you should include both the taskrecno and the memrecno; otherwise you risk outputting incomplete information. Hints let you tell the IBM Initiate Inspector application to highlight certain fields in the task resolution screen, to help IBM Initiate Inspector users find and resole data issues, and descriptions let you add descriptie text to the custom task data, to further refine the leel of detail for your custom tasks. If you plan to use hints or descriptions, be sure to map these as part of your TSKPUT output. Hints are mapped from the idtxtsk table (idtxtsk.hint); descriptie text is also mapped from the idtxtsk table, but the field name in the TSKPUT component is text (idtxtsk.text). Creating a graph with an IBM Initiate TskPut component About this task Note: Before creating an IBM Initiate TskPut graph, you should reiew the oeriew of the Adanced Issue Management feature in the IBM Initiate Master Data Serice Hub Oeriew. The basic steps required for creating a graph with an IBM Initiate TskPut component are described below. For detailed information about working with Readers and metadata, see Working with CloerETL graphs on page 6. The example here shows how to take custom task data from multiple external sources (in this case, a set of custom task data, and a list of member records the custom data will be applied to) and map them together. This use case is a typical one if you want to apply a gien custom task instance to more than one member record. If you hae a one-to-one relationship between your custom tasks instances 58 CloerETL User's Guide

67 and member records (that is, a gien custom task is applied only to a single member record), you do not need to add a second external data source. When mapping custom task instances to multiple member records, you must include a single member record number (memrecno) in the custom task data (which will sere as your primary member record), then use keys to map additional member records from the other data source(s) to the primary member record in the custom task data. Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page Add a Reader for your custom task data and connect it to the external data source, using the steps outlined in Adding and configuring a CloerETL Reader on page Optional. Add a second Reader for your additional member record data, and connect it to the external data source. 4. From the Writers section of the Palette, select and drag an IBM Initiate TskPut component onto the graph. 5. Add Edges to connect the Readers to the IBM Initiate TskPut component. Note: You can also add Transformers between the Reader and the IBM Initiate TskPut component to filter, sort, or otherwise transform the data before passing it to the IBM Initiate TskPut component. 6. Create and apply metadata to the Edges, using the steps outlined in Using CloerETL Edges to connect components on page 10. When creating metadata for the custom task and member record Edges, keep in mind the following: If you are using hints or descriptions, be sure that your source data and Edge metadata include the hint and text fields. In this example, which maps additional member records to the custom task data, the member record source data and Edge metadata should include keys to map the member records in the member record source data to the primary member record listed in the custom task data. The following figures show sample metadata for a custom task data set (idxtsk) and a second set of data for additional member records (memrecnos). Chapter 3. IBM Initiate custom components 59

68 Figure 1. Sample metadata for a custom task set (idxtk) Figure 2. Sample metadata for additional task records in the custom task set (idxtsk) 60 CloerETL User's Guide 7. Double-click the IBM Initiate TskPut component to open the Edit Component dialog box. 8. Map IBM Initiate Hub attributes to the fields defined in the metadata: a. Ensure that you hae defined an IBM Initiate connection. b. Click in the Value field for the Mapping property, and then click the ellipsis control to open the TskPut Mapping dialog. c. Select the any attributes or non-attribute segments to map, using Ctrl-click.

69 Note: You must include at minimum the IdtXtsk non-attribute segment. d. Click Next. e. The TskPut Mapping dialog displays tabs for each of the attributes you selected. Drag the metadata (input) fields to the attribute fields you want to map them to. By clicking the Auto mapping button you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. f. You can specify a row indicator setting for an indiidual attribute that oerrides a global row indicator setting in the Edit component dialog box. 1) Select the attribute. 2) Beside the RowInd/Filter table, click the green plus (+) button. CloerETL inserts a row indicator alue of Insert into the table. 3) To change the alue, click the row in the RowInd column to display the drop-down list, and then select the applicable alue. 4) To add a filter expression to the row indicator alue, click the row in the Filter column to display the Filter dialog box. Build the expression and click OK. g. Click Finish. 9. If you want to map success and/or failure messages to output files: a. Add a Writer for each kind of output you want to capture (success and/or failure). b. Connect the Writers to the TskPut component's output ports, and add Metadata to the Edges. Port 0 is for success messages and port 1 is for failures. c. Open the TskPut component and click Output mapping, to map your output as needed. When mapping success and failure messages, keep in mind the following: Both success and failure output should include the memrecno. Is it recommended that failure output include ERR_CODE and ERR_MESSAGE. If you used keys to join data from multiple input files, include these keys in your output as well. 10. Enter additional properties as needed to configure the IBM Initiate TskPut component. Required properties are marked with a yellow warning icon. Property definitions are gien in Initiate TskPut properties. 11. Sae the graph. Initiate TskPut properties The table below lists the properties you can define for the Initiate TskPut component. Note: For additional information on TskPut, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Chapter 3. IBM Initiate custom components 61

70 Table 11. TskPut properties Property Description Initiate connection Specify a connection to the Initiate Hub by clicking the Value field and selecting from the drop-down list. The TskPut component requires a connection to an Initiate Hub in order to retriee attributes. Put type Member mode Match mode If no connections are listed, you can create one using the steps outlined in Connections in CloerETL on page 13. PutType limits the types of changes that can occur for a gien interaction. For example, you might want to update a Task. By using the UpdateOnly setting, the Task will only be updated if it already exists. By using the InsertOnly setting, the Task will only be inserted if it does not already exist. InsertUpdate: If the member data does not exist, create it; or if the member data already exists, update it InsertOnly: If the member data does not exist, create it; or if the member data already exists, abort the interaction UpdateOnly: If the member data already exists, update it; or if the member data does not exist, abort the interaction Member mode specifies the type of input data present in the member data presented to the interaction: Partial: The input member attributes do not contain a complete picture of all the members attributes. The attributes will be inserted and no oerlay tasks will be created. AttrComp: For each gien attribute, all alues for that attribute are present in the input member, so the Engine can make decisions regarding the Actie/Inactie status of historical attributes. During put processing, this flag indicates that attributes in the database that share the same attrrecno, but are not included in the input should hae their recstat set to Inactie. Complete: Complete member is present in the input record. This is the only setting that supports the creation of Oerlay tasks. The Engine can compare this complete input member to the image in the database to make decisions regarding the creation of an oerlay task. Explicit: The input member is used as a map for what to insert/update/delete. No oerlay tasks will be created Match mode specifies the type of matching to perform after the put interaction: Immediate: Cross match the member immediately following the update of the member data. This option has no additional effect in asynchronous mode Deferred: Member matches when Entity Manager gets to this spot in the queue DoNothing: Update the member data, but do not cross match the updated data against other members in the Hub database. The interaction returns control to the caller as soon as the update is complete. This blocks this member from being matched when it is put, but another put that has buckets in common could cause this member to be matched. 62 CloerETL User's Guide

71 Table 11. TskPut properties (continued) Property Description Mapping Use the Mapping parameter to map metadata fields to your Hub database attributes. Refer to Creating a graph with an IBM Initiate TskPut component on page 58 for more information on mapping. Output Mapping If you want to map success and/or failure messages to a file, use the output mapping option to map success messages to port 0 and failure messages to port 1. Key fields The Key fields parameter specifies how records from different input ports are joined. See Using Key fields in CloerETL MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to join records on page 31 for detailed instructions on using Key fields. MemRecnos key fields Key fields of input metadata which group multiple memrecnos to IdtXtsk rows. Use this option if you want to associate a single custom task instance with more than one member; the custom task instance is associated with a primary member record in one input file, and the additional member records, in a different input file, are mapped to the primary member record. Entity type Specifies the type of Entity to process. Client args Client args are used for external pre- and post-inocation handlers. For more information, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. RowInd Specifies a global action that the Master Data Engine should perform on all rows that are passed in. The alues are: (I)gnore (I)nsert (U)pdate (D)elete Number of skipped records Number of written records Adanced > IxnTskPut ersion The RowInd property is also aailable for indiidual attributes. The indiidual attribute setting oerrides the global setting. Specifies how many records/rows should be skipped from the source port. This is useful for handling files where first rows are a header not a real data. Specifies how many members should be written to the Initiate Hub. If no number is specified, the number of records written is not limited. Specifies which ersion of of the IxnTskPut interaction to use: IxnTskPut, or IxnTskPut2. When working with custom task data which references more than one member record, always use IxnTskPut2. IBM Initiate custom Transformer components for CloerETL This section describes the IBM Initiate Transformer components: IBM Initiate MemScore, IBM Initiate Atrribute Reformat, and IBM Initiate MemSeqNo Incrementer. IBM Initiate custom components for address standardization are described in IBM Initiate custom address-standardization components for CloerETL on page 71. Chapter 3. IBM Initiate custom components 63

72 The IBM Initiate MemScore component The IBM Initiate MemScore component runs the scoring algorithm of the IBM Initiate Hub on 2 or more records that are not stored in the IBM Initiate Hub; the first record is the target against which the others are scored. The IBM Initiate MemScore component requires input from a CloerETL Reader. The Reader supplies a connection to the external source of the data, for both the first record (that is, the target) and other records that will be scored against it. CloerETL proides a number of Readers which can read data from sources such as databases, text files, and LDAP repositories. The IBM Initiate MemScore component allows multiple input ports for both the target record and the records scored against it. The target record data is supplied on the Master port(s), and the comparison records are supplied on the Slae port(s). If you will use multiple Master ports (for example, to supply a name and address from one data source, and a phone number from a second data source), the data must be joined ia a Master key field. Likewise, comparison data supplied ia multiple Slae ports must be joined by a Slae key field. Note: For additional information on IBM Initiate MemScore, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Creating a graph with an IBM Initiate MemScore component About this task The basic steps required for creating a graph with an IBM Initiate MemScore component are described below. For detailed information about working with Readers and metadata, see Working with CloerETL graphs on page 6. Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page Add a Reader and connect it to your external data source, using the steps outlined in Adding and configuring a CloerETL Reader on page 8. If you will use multiple data sources for either the Master or Slae inputs, add and connect a Reader for each. 3. From the Transformers section of the Palette, select and drag an IBM Initiate MemScore component onto the graph. 4. Add an Edge to connect each Reader to the IBM Initiate MemScore component. Note that you can connect many Readers to a single IBM Initiate MemScore component. 5. Create and apply metadata to the Edge(s), using the steps outlined in Using CloerETL Edges to connect components on page Add a Writer component to write the output of the IBM Initiate MemScore component: a. From the Writers section of the Palette, select and drag Writer component onto the graph. b. Add an Edge to connect the IBM Initiate MemScore component to the Writer. c. Create and apply metadata to the Edge, using the steps outlined in Using CloerETL Edges to connect components on page Double-click the IBM Initiate MemScore component to open the Edit Component dialog. 64 CloerETL User's Guide

73 8. Specify a connection to the IBM Initiate Hub by clicking in the Value field for the IBM Initiate connection property and choosing a connection from the drop-down list. Note: If there are no connections in the drop-down, you can create one using the steps outlined in Connections in CloerETL on page Specify the Count of Master ports. 10. Specify the Count of Slae Members. This is the number of slae records that will be scored against the master record. 11. Map the Input Master attributes to the fields defined in the metadata: a. Click in the Value field for the Input Master mapping property. An ellipsis is displayed. b. Click the ellipsis to open the MemScore master mapping dialog. Note: You must hae an IBM Initiate connection defined before you can map attributes. c. Select the attributes to map using Ctrl-click. Click Next. d. The MemScore master mapping dialog displays tabs for each of the attributes you selected. Drag the metadata (input) fields to the attribute fields you want to map them to. Note: The Count specified for the Master ports in the step aboe determines which ports are read for the Master mapping. For example, if the Master port count is set to one, the Master mapping dialog will display input fields for one port (Port 0); if the Master port count is set to two, the Master mapping dialog will display input fields for two ports (Ports 0 and 1). e. By clicking the Auto mapping button you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. f. Click Finish. 12. Map the Input Slae attributes to the fields defined in the metadata: a. Click in the Value field for the Input Slae mapping property. An ellipsis is displayed. b. Click the ellipsis to open the MemScore slae mapping dialog. Note: You must hae an IBM Initiate connection defined before you can map attributes. c. Select the attributes to map using Ctrl-click. Click Next. d. The MemScore slae mapping dialog displays tabs for each of the attributes you selected. Drag the metadata (input) fields to the attribute fields you want to map them to. Note: The Count specified for the Slae ports in the step aboe determines how many ports are displayed in the input fields area of this dialog. e. By clicking the Auto mapping button, you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. f. Click Finish. 13. Map the Output attributes to the fields in the metadata. Chapter 3. IBM Initiate custom components 65

74 a. Click in the Value field for the Output mapping property. An ellipsis is displayed. Note: If you hae not added a Writer to the graph and connected it to the MemScore component ia an Edge, you will not be able to open the MemScore output mapping dialog. b. Click the ellipsis to open the MemScore output mapping dialog. Note: You must hae an IBM Initiate connection defined before you can map attributes. c. Select the attributes to map using Ctrl-click. d. Drag the metadata input (attribute) fields to the output fields you want to map them to. e. Click OK. 14. If you want to map failure messages to an output file: a. Add a Writer to capture the failure message. b. Connect the Writer to the MemScore component output port 1, and add Metadata to the Edge. c. Open the MemScore component and click Output mapping, to map your output as needed. When mapping failure messages, keep in mind the following: Failure output should include the memrecno. Is it recommended that failure output include ERR_CODE and ERR_MESSAGE. 15. Sae the graph. IBM Initiate MemScore properties The table below lists the properties you can define for your IBM Initiate MemScore component. Note: For additional information on MemScore, refer to the SDK interaction examples and the IBM Initiate Jaa SDK Jaadoc Information. Table 12. MemScore Properties Property IBM Initiate connection Member Type Entity type Input Master mapping Input Slae mapping Values The MemScore component requires a connection to an IBM Initiate Hub in order to retriee attributes. Detailed instructions on how to create a connection and specify it for this parameter are gien in Connections in CloerETL on page 13. Member type specifies the Member type to score. Specifies the type of Entity to process. Use the Input Master mapping parameter to map master metadata fields to your IBM Initiate Hub attributes. Creating a graph with an IBM Initiate MemScore component on page 64 for detailed instructions on mapping. Use the Input Slae mapping parameter to map slae metadata fields to your IBM Initiate Hub attributes. Creating a graph with an IBM Initiate MemScore component on page 64 for detailed instructions on mapping. 66 CloerETL User's Guide

75 Table 12. MemScore Properties (continued) Property Output mapping Count of Master Ports Values Use the Output mapping parameter to map metadata fields to attributes for the output data. See Creating a graph with an IBM Initiate MemScore component on page 64 for detailed instructions on mapping. If you want to map failure messages to a file, use the output mapping option to map failure messages to port 1. The number of input ports that are in the Master set (for example, if Count of Master Ports is 3, then the first 3 ports are in the Master set, and the remaining ports are in the Slae set). These ports are used to read the Master member, against which the other members (read from the Slae port set) are scored. If you will use more than one Master port to combine data from multiple sources, you must use the Master key field to join records from these multiple sources. Count of Slae Members Master Key fields The number of Slae members read from the Slae set of ports, which are used in one MemScore interaction. This parameter specifies how records from different input ports are joined. The key is used to construct the Master member from the data coming from the master ports. This attribute is mandatory if there is more than 1 Master port. See Using Key fields in CloerETL MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to join records on page 31 section for detailed instructions on using Key fields. Slae Key fields This parameter specifies how records from different input ports are joined. The key is used to construct the Slae members from the data coming from the Slae ports. This attribute is mandatory if there is more than 1 Slae port. See Using Key fields in CloerETL MemPut, MemSearch, TskSearch, TskPut, and Handler Writer to join records on page 31 for detailed instructions on using Key fields. The IBM Initiate Attribute Reformat component The custom IBM Initiate Attribute Reformat component, similar to the standard CloerETL Reformat component, receies data from a Reader through a single connected input port, transforms it in a user-specified way, and sends the reformatted outgoing data records to the connected output port(s). Howeer, the IBM Initiate Attribute Reformat component includes the additional Attribute code property, in which you can identify the attribute record number of an attribute from the IBM Initiate Master Data Serice. Transformations that this component can execute include: changing the number of fields in a record changing case for alpha characters conerting data types (from one type to another) concatenating or reordering data fields changing metadata Chapter 3. IBM Initiate custom components 67

76 Creating a graph with the IBM Initiate Attribute Reformat component About this task The basic steps required for creating a graph with an IBM Initiate Attribute Reformat component are described below. For detailed information about working with Readers and metadata, see Working with CloerETL graphs on page 6. Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page Add a Reader and connect it to your external data source, using the steps outlined in Adding and configuring a CloerETL Reader on page From the Writers section of the Palette, select and drag an IBM Initiate Attribute Reformat component onto the graph. 4. Add an Edge to connect the Reader to the IBM Initiate Attribute Reformat component. Note: To filter, sort, or otherwise transform the data before passing it to the IBM Initiate Attribute Reformat component, add Transformers between the Reader and the Initiate Attribute Reformat component. 5. Press [ESC] or click Select in the Palette to turn off the Edge tool. 6. Create and apply metadata to the Edge, using the steps outlined in Using CloerETL Edges to connect components on page Double-click the IBM Initiate Attribute Reformat component to open the Edit Component dialog. In the edit dialog box, you must specify how the data is to be transformed by editing one of the Transform properties. In the Transform class property, you can specify the path and file name of a class, jar, or zip file. In the Transform property, you define the transform within the graph by using Jaa or Cloer transformation language (CTL). In the Transform URL property, you can specify the path and file name of a Jaa or CTL file. 8. Click the Transform property that you want to use, and then click the ellipsis that is displayed. For detailed information about the Transform properties, refer to the CloerGUI Help. For the Transform class property, the Open Type wizard is displayed. Type a few characters in the text field that identify the file to populate the wizard with file names that match. Select the appropriate file, and then click OK. For the Transform property, the Transform editor is displayed. You can define the transformation in seeral ways: dragging and dropping input fields in the Transformations tab using CTL in the Source tab clicking Open Tab to open a window in the graph editor and using CTL For the Transform URL property, the URL File dialog is displayed. Browse to the local or remote file, select it, and then click OK. 68 CloerETL User's Guide

77 9. To specify that the transformation is applicable to an indiidual attribute, click the Attribute code property and the resulting ellipsis. Note: This feature requires an IBM Initiate connection. If you hae not already done so, you can define an IBM Initiate connection by following the steps outlined in Connections in CloerETL on page In the Select attribute dialog box, select an IBM Initiate connection and click Connect. The dialog box is populated with the attribute codes from the source data. 11. Select the applicale attribute code, and then click OK. The Attribute code field is populated with the AttrRecNo for the selected attribute. 12. Click OK to sae the transformation properties and close the dialog box. The IBM Initiate MemSeqNoIncrementer component The IBM Initiate Member Sequence Number Incrementer (MemSeqNoIncrementer) component is used to assign memseqno alues to data, in a batch mode. The memseqno is a system-generated alue that is used to proide a unique primary key per attribute within a member. For example, a record may include two phone attributes one for home phone number (homephone) and another for mobile phone number (mobilephone); each are of the type memphone, but each would be assigned a unique memseqno. The MemSeqNoIncrementer component is used primarily with the IBM Initiate Member Model Transform Graph. It is created automatically as part of the graph produced by the Member Model Transform Graph wizard. For more information on creating a Member Model Transform Graph, see The IBM Initiate Member Model Transform Graph on page 24. Howeer, you can also use the IBM Initiate MemSeqNoIncrementer component in other graphs, to assign memseqno alues to specified fields, one by one. Records with changed memseqno alues can be reformatted according to a formula specified in the IBM Initiate MemSeqNoIncrementer component's Field Mapping attribute. IBM Initiate MemSeqNoIncrementer attributes This section describes the attributes of the IBM Initiate MemSeqNoIncrementer component. You can use this information to undertand how IBM Initiate MemSeqNoIncrementer components are generated as part of an IBM Initiate Member Model Transform Graph, or to build your own graphs utilizing the IBM Initiate MemSeqNoIncrementer component. IBM Initiate MemSeqNoIncrementer components should be concatenated to assign memseqno alues to more than one kind of attribute. To edit an IBM Initiate MemSeqNoIncrementer component's attributes, double-click the component to open the Edit component window. Note: You do not need to manually edit the attributes of a IBM Initiate MemSeqNoIncrementer component that has been created by the IBM Initiate Member Model Transform Graph; the attributes are populated automatically based on alues you entered in the IBM Initiate Member Model Transform Graph wizard. Chapter 3. IBM Initiate custom components 69

78 There are three required fields in the IBM Initiate MemSeqNoIncrementer component: Not null fields The Not null fields are fields that make up the attributes. These are essentially the fields to which you want to assign memseqno alues, in the eent that one of the fields is not null. To select Not null fields, click on the ellipsis to open the Edit key dialog. The dialog lists aailable attributes, based on the metadata from the Edge connecting the Reader to the IBM Initiate MemSeqNoIncrementer component. Select fields from the Fields column and use the arrow buttons to moe them to the Key parts column. MemSeqNo field The MemSeqNo field is the field to increment as memseqno alues are assigned. To select a MemSeqNo field, click on the ellipsis to open the Edit key dialog. The dialog lists aailable attributes, based on the metadata from the Edge connecting the Reader to the IBM Initiate MemSeqNoIncrementer component. Select fields from the Fields column and use the arrow buttons to moe them to the Key parts column. Field mapping The Field mapping dialog maps fields from the component's input file to its output file(s). To map input fields to output fields, click on the ellipsis to open the Mapping dialog.. Note: You can not open the Field mapping dialog unless you hae defined metadata for the Edge connecting the IBM Initiate MemSeqNoIncrementer to the Writer. If no metadata has been applied to this Edge, the ellipsis for opening the Field mapping dialog does not appear. The Input Fields column lists attributes based on the metadata from the Edge connecting the Reader to the IBM Initiate MemSeqNoIncrementer component. The Field Mapping table's Output fields are fields defined in the metadata for the Edge connecting the IBM Initiate MemSeqNoIncrementer component to the Writer on Port 0. Drag fields from the Input Fields column to the Output fields you want to map them to. IBM Initiate MemSeqNoIncrementer component Input and Output ports The IBM Initiate MemSeqNoIncrementer component has two output ports. Port 0 is required. It is used for reformatted records for which the memseqno was changed. When at least one of the key fields is not null, the alue of the field set as memseqnofield is unique for this attribute, and the record is reformated according to the formula gien in the Mapping attribute and sent to port 0. Port 1 is optional. It is used for records with the same structure as on input, but with modified memseqno alues. It outputs a copy of the input record, but the field set as memseqnofield can be incremented by one (if at least one of the key fields is not null). If you wish to concatenate multiple IBM Initiate MemSeqNoIncrementer components, use Port 1 to route data from one IBM Initiate MemSeqNoIncrementer component to the next. 70 CloerETL User's Guide

79 IBM Initiate custom address-standardization components for CloerETL Address standardization is the process of formatting and alidating address data according to standards from postal agencies world-wide. Address standardization also inoles deliering correct address data to applications and business processes that need it, and maintaining standardized address data as it changes oer time. CloerETL proides address standardization functionality that integrates with seeral address-erification endors through the use of Transformer components. If you hae a license for one of these endors, you can use the applicable CloerETL component to connect with the endor while you are working in IBM Initiate Workbench. IBM Initiate Workbench includes an IBM Initiate Address Verification Interface custom component for building graphs for address standardization. Attribute requirements for address standardization in CloerETL Before you can perform address standardization and store the results in the Hub, you must structure the data model to accommodate the standardized address data. At a minimum you would define an attribute for the standardized street address, but the type of standardization and erification that you perform dictates your modification of the data model. Requirements for the attributes will depend on the type of address data you will be handling. When you set up a CloerETL graph to perform standardization, you will map the standardized data returned by the callout to the attribute(s) you hae defined for the standardized address data. Deciding between batch and transactional standardization in CloerETL Standardization supports both batch and transactional scenarios. As you determine the best standardization approach for your organization, you will need to balance performance requirements and the currency of the erification platform s data (that is, the data used to standardize addresses). Retrieal speed can be faster if standardized data is stored in the Hub database, but because the postal reference databases are updated regularly, the retrieal results are likely to be more up to date if standardization is done on the fly as data is requested. File-to-file batch standardization in CloerETL File-to-file batch standardization can be accomplished by a simple graph that includes three components: a Reader to retriee the source data records, a Transformer component to standardize and erify the data, and a Writer to create useful output. For example, you might retriee records from a list of international customers in order to create mailing labels or to display in IBM Initiate Inspector. Transactional processing by using a callout in CloerETL For transactional address standardization, you might consider registering a callout that includes the address-standardization graph. With this model, to standardization process flows as follows: a request for address data is made the database extracts the requested data the Master Data Engine inokes the callout, which contains an address-standardization graph Chapter 3. IBM Initiate custom components 71

80 the Transformer component sends the data to the endor the endor returns the standardized, erified address data the standardized, erified data is displayed to the requester Complete the following steps in order to set up this type of transactional processing: 1. Create and register a callout handler in IBM Initiate Workbench. Refer to the Callout handlers chapter in the IBM Initiate Workbench User's Guide. 2. Create a standardized address Attribute in the data model. Refer to the Configuration editor chapter in the IBM Initiate Workbench User's Guide. 3. Create the address-standardization graph in CloerETL. The CloerETL IBM Initiate Address Verification Interface component IBM Initiate Address Verification Interface is a subscription-based product that gies you access to postal reference data that is standardized by national postal serices from around the world. The postal reference data is updated on a quarterly basis, enabling the IBM Initiate Address Verification Interface component to access the most up-to-date address data for any gien country. Updates are made aailable the first week of each calendar quarter (January 1st, April 1st, July 1st, and October 1st) and are only enabled for a period of 4 months. Important: Because each update to the postal reference data is only enabled for 4 months, you should schedule regular updates to be applied, in order maintain functionality. The IBM Initiate Address Verification Interface component is a Transformer component that you can use together with a Reader component (for example, Initiate Handler Reader) and a Writer component to perform the address-standardization callout. For details about changes between the IBM Initiate Address Verification Interface 4 (offered in ersions 8.7 and earlier of IBM Initiate Workbench) and ersion 5 (which replaces ersion 4 with the December 2009 patch), see Changes from IBM Initiate Address Verification Interface ersion 4 to IBM Initiate Address Verification Interface ersion 5 in CloerETL on page 73. About postal reference data The postal reference data to which IBM Initiate Address Verification Interface gies you access is proided to you at the time of purchase and is updated regularly. This data must be stored in a directory that is accessible to the IBM Initiate Master Data Serice serer. Refer to the IBM Initiate Address Verification Interface documentation (proided to you at the time of purchase) for more details about working with postal reference data, including information on how to update the data. The IBM Initiate Address Verification Interface is supported by technology and postal reference data proided by AddressDoctor. AddressDoctor has compiled the postal reference database used by the IBM Initiate Address Verification Interface using data obtained from data suppliers. In collating this data, AddressDoctor has proceeded diligently and made checks for correctness, completeness and alidity. 72 CloerETL User's Guide

81 Neertheless, it may be possible that indiidual data may be incorrect, incomplete or inalid. Neither AddressDoctor nor IBM proides any warranty in this regard. The software libraries used by AddressDoctor for the erification of postal addresses use fault-tolerant methods and fuzzy matching algorithms. They can lead to wrong corrections or suggestions een if the input data is correct. It may be possible that an address is corrected incorrectly (false positie) or that an address that would be correctable is not corrected (false negatie). Changes from IBM Initiate Address Verification Interface ersion 4 to IBM Initiate Address Verification Interface ersion 5 in CloerETL In release 9.2 of IBM Initiate Workbench, the ersion of the IBM Initiate Address Verification Interface component has been upgraded from ersion 4 to ersion 5. If you are upgrading from a preious ersion of IBM Initiate Workbench, you must recreate any existing IBM Initiate Address Verification Interface graphs using the new IBM Initiate Address Verification Interface 5 component in the current ersion of IBM Initiate Workbench. For complete details on new features and changes to existing features, refer to the IBM Initiate Address Verification Interface proprietary documentation that was proided to you at time of purchase. Some key changes to note include the following: The addressformat.cfg file, which in ersion 4 had to be present in the directory which held your postal database(s), is no longer used. You can now specifiy multiple unlock codes for your IBM Initiate Address Verification Interface component. Refer to the IBM Initiate Address Verification Interface proprietary documentation for information on how multiple codes work together. IBM Initiate Address Verification Interface component can be configured ia a Cloer dialog, or by referencing a configuration XML file that conforms to the SetConfig.DTD proided in the IBM Initiate Address Verification Interface proprietary documentation. IBM Initiate Address Verification Interface component parameters can be set ia a Cloer dialog, or by referencing a parameters xml file that conforms to the parameters.dtd proided in the IBM Initiate Address Verification Interface proprietary documentation. Input and output mapping has changed; additional options are now aailable. Refer to the documentation below, and to the IBM Initiate Address Verification Interface proprietary documentation, for details. The Preloading option (for preloading postal databases into memory for improed performance) is now set with the database (in the Parameters dialog on the Database tab) The Validation parameter in ersion 4 has been replaced with a Mode option in ersion 5. To set the alidation mode, use the Mode setting on the Process tab of the Parameters dialog. Refer to the IBM Initiate Address Verification Interface proprietary documentation for additional details on IBM Initiate Address Verification Interface 5 features and functions. Note: Any address standardization graphs you hae created using an earlier ersion of IBM Initiate Workbench and ersion 4 of IBM Initiate Address Chapter 3. IBM Initiate custom components 73

82 Verification Interface must be recreated using the new IBM Initiate Address Verification Interface5 component in the updated ersion. Country and CASS certification in CloerETL Some countries offer a certification process for address alidation software. In the United States, the postal serice offers the CASS system (Coding Accuracy Support System), which improes the accuracy of carrier routes, ZIP codes, and deliery point codes that appear on mail, and is required for sending bulk mail at discounted rates. To use CASS certification with IBM Initiate Address Verification Interface, additional databases that contain CASS related information (such as Carrier Route codes, EWS, ZIPMOVE, LACSLink, DPV, DFS2, SuiteLink etc.) are required. This information is contained in files named USA5C1.MD to USA5C21.MD. Please refer the IBM Initiate Address Verification Interface proprietary documenation for more detail. (At the time of this writing, CASS information is included in section Special Remarks for USA CASS Certified Mode in the IBM Initiate Address Verification Interface documentation.) For more information about country certification, refer to the postal authority for the applicable country. Enabling the IBM Initiate Address Verification Interface component The IBM Initiate Address Verification Interface must be enabled before you can use an address erification component in your graphs. This section explains how. About this task When you purchase address database(s), you are proided with an enablement file, IAVI.txt, which contains an encrypted key that gies you full access to the address standardization features within IBM Initiate Workbench for those database(s). This file must be placed in the same directory as your address database(s), and a path to the database(s) and enablement file must be set in each IBM Initiate Address Verification Interface component you use in a graph. Procedure 1. Place a copy of the IAVI.txt file in the same directory as your address database(s). Note that if you use multiple directories to store arious address database(s), a copy of IAVI.txt must be placed in each one. 2. In each graph that uses an IBM Initiate Address Verification Interface component, follow these steps to set the path to the directory containing the database and the IAVI.txt file. a. Double-click the IBM Initiate Address Verification Interface component to open the Edit Component dialog b. Click in the alue field for Configuration to open the Configuration dialog. c. On the DataBase tab, click the green plus sign to add a row for your database. d. Modify the CountryISO3, Type, and Preloading type as needed. e. In the Path field, enter the path to the directory where the database and IAVI.txt files are located. When entering paths, note the following: Paths can be direct (for example, C:\Mydata\) or parameterized (for example, ${my_db_dir}) 74 CloerETL User's Guide

83 UNIX paths must use forward slashes (/); Windows paths can use forward slashes (/) or use backslashes (\) Network paths (UNIX or Windows) must use forward slashes (/) f. Optional. If you want to erify that your component is enabled, click the UnlockCode tab. This tab lists the alidation status for any databases referenced on the DataBase tab ("alid" for correctly enabled, or "inalid" if the IAVI.txt file is missing or not functional). g. Click OK to sae your changes and return to the Configuration dialog. h. Click Apply to apply your change. Using a manual unlock code to enable the IBM Initiate Address Verification Interface: In some cases, such as demonstrations or debugging scenarios, you may need to manually enter an unlock code, as an alternatie to using the IAVI.txt enablement file. This section explains how. About this task When you manually enter an unlock code for the IBM Initiate Address Verification Interface, you must enter the code for eery IBM Initiate Address Verification Interface component in your graphs. Procedure 1. In the graph that contains the IBM Initiate Address Verification Interface component, double-click the IBM Initiate Address Verification Interface component to open the Edit Component dialog 2. Click in the alue field for Configuration to open the Configuration dialog. 3. On the UnlockCode tab of the configuration dialog, click the green plus sign to add a row for your unlock code. 4. Type or copy your unlock code oer the Enter an unlock code text in the new row. You can enter multiple unlock codes in this manner. Refer to the IBM Initiate Address Verification Interface proprietary documentation for information about how multiple unlock codes work together. 5. Click OK to return to the Configuration dialog. 6. Click Apply to apply your change. Expiration of the unlock code: The IBM Initiate Address Verification Interface unlock code expires on a periodic basis. Although the unlock code does not expire as frequently as the postal reference data, it is good practice to update your unlock code each time you update your postal reference data, to ensure continued operation. About this task To update your unlock code, you must obtain an updated IAVI.txt file from IBM. Updates to this file are proided by IBM along with postal reference data. It is good practice to update your IAVI.txt file any time you update your postal reference data. Failure to install this new key will result in your address standardization process discontinuing. The follow error will be reported in your log files, but will not be displayed on the user interface: Chapter 3. IBM Initiate custom components 75

84 ERROR com.addressdoctor.ad_exception: [-100] (CRITICAL) Expired Creating a graph with an IBM Initiate Address Verification Interface component About this task The basic steps required for creating an address standardization graph with an IBM Initiate Address Verification Interface component are described below. For detailed information about working with Transformers, see Working with CloerETL graphs on page 6. In an address standardization graph, data to be standardized can be read from any data Reader, or from Initiate dictionary data if used in a callout. For callouts, the graph will use a special Reader component, the Initiate Handler Reader, which is specially designed to read data from the dictionary area. Likewise, a custom Writer component, the Initiate Handler Writer is used to return or write standardized data. Note: An address standardization graph requires that both input and output data are mapped appropriately. This mapping cannot be completed until the Reader, Transformer (IBM Initiate Address Verification Interface), and Writer components are placed in the graph and connected with Edges containing metadata. For this reason, Initiate recommends that you add and connect all the elements of the graph before beginning to configure each element. Procedure 1. Create and open a new graph using the steps outlined in Building a CloerETL graph on page From the Readers section of the Palette, select and drag an Initiate Handler Reader component onto the graph, and connect it to your external data source, using the steps outlined in Adding and configuring a CloerETL Reader on page Add the IBM Initiate Address Verification Interface (AddressDoctor) component, and connect it to the Initiate Handler Reader with an Edge: a. From the Transformers section of the Palette, select and drag an IBM Initiate Address Verification Interface (AddressDoctor) component onto the graph. b. Add an Edge to connect the Initiate Handler Reader to the IBM Initiate Address Verification Interface (AddressDoctor) component. c. Create and apply metadata to the Edge, using the steps outlined in Using CloerETL Edges to connect components on page 10. The metadata should include both the attributes you want to standardize, and any attributes you will pass through to the requesting application. 4. Add the Initiate Handler Writer component, and connect it to the IBM Initiate Address Verification Interface (AddressDoctor) component with an Edge: a. From the Writers section of the Palette, select and drag an Initiate Handler Writer component onto the graph. b. Add an Edge to connect the Initiate Handler Writer to the IBM Initiate Address Verification Interface(AddressDoctor) component. c. Create and apply metadata to the Edge, using the steps outlined in Using CloerETL Edges to connect components on page CloerETL User's Guide

85 The metadata should include attributes for both your source data and the standardized ersion of it, as well as any other attributes you will pass through to the requesting application. 5. Map the Initiate Handler Reader attributes that will be passed to the IBM Initiate Address Verification Interface component. Note: Remember that the metadata for the Edge connecting the Initiate Handler Reader to the IBM Initiate Address Verification Interface component must be defined and applied before mapping can be done. a. Double-click the Initiate Handler Reader to open the Edit Component dialog. b. Click the ellipsis in the Value field for the Mapping property to open the Attributes and Non-attribute segments dialog. c. Select the attributes and non-attribute segments to map. On the Attributes tab, HOMEADDR and LEGALNAME are typically mapped. Remember that you need to map eery attribute you want to pass, een if not all these attributes represent data to be standardized. For example, if customer's name is included in the data that will be passed on to the requesting application, you should include it here een though it will not be standardized On the Non-attribute segments tab, MEMHEAD is typically mapped, because the memrecno is needed to identify the record. d. Map records from the selected attributes to the releant output fields. e. By clicking the Auto mapping button you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. f. Click Finish. 6. Double-click the IBM Initiate Address Verification Interface(AddressDoctor) component to open the Edit Component dialog. 7. Click Configuration to enter the following required configuration properties in the IBM Initiate Address Verification Interface component: a. On the Unlock tab, enter your unlock code(s). Unlock codes are proided at the time of purchase. b. On the Database tab, enter information about your postal reference databases. The postal reference data was proided to you at the time of purchase. 1) Choose a Country ISO3 for the database. 2) Choose the Type for the database. The type must match the filetype of the database; for example, choose FASTCOMPLETION as the type for a database with a filetype of FastCompletion. 3) Enter the Path to the postal reference database. 4) Choose a Preloading mode. PARTIAL loads the metadata and indexing structures of the selected databases. FULL loads the complete selected databases into memory. NONE disables loading of the CASS certification database, and should not be selected for use with the database of any other country. 5) Click OK to exit from the Configuration dialog. Chapter 3. IBM Initiate custom components 77

86 8. Click Parameters and enter any additional parameters for your graph. Refer to the IBM Initiate Address Verification Interface proprietary documentation for detailed information about parameter options. Press OK when finished. Note: The input and result encoding alues are set during run-time based on the operating system. 9. Click Input Mapping to map the IBM Initiate Address Verification Interface input address properties to your input metadata fields. The AddressDoctor address properties are the fields in the IBM Initiate Address Verification Interface postal reference data that map to the Initiate attributes you want to standardize. The following table summarizes some IBM Initiate Address Verification Interface address properties and the common address elements to which you would map them. Table 13. Mapping IBM Initiate Address Verification Interface (AddressDoctor) properties to address elements AddressDoctor address property Example user-friendly address input field Country Country name Proince County (UK), State (United States), Proince (Canada), Kanton (Switzerland), Prefecture (Japan) Postal code Diision within a city (post code, ZIP code) Locality City name, municipality name Street Street name (1000 Harard DR) House number Building address number (1000 Harard DR) Building Recognizable building name (Emirates Palace Hotel, Sears Tower) Subbuilding Diision within a building (Apartment 3B, Seenth floor, Suite 120) The Subbuilding property is contained within the Building property. From the Properties list, select a Building property. In the Mapping dialog box, select the Building property. The Property Configuration: Building type drop-down list box is displayed. From the drop-down list, select the Subbuilding property. Drag the applicable address input field to the Building property. a. On the Address properties dialog, use Ctrl-click to choose the properties you want to be aailable for mapping. Click Next. b. Map the input fields to the selected IBM Initiate Address Verification Interface address properties. c. Click Finish, then click OK to close the IBM Initiate Address Verification Interface component. 10. Click Output Mapping to map the IBM Initiate Address Verification Interface output address properties to your output metadata fields. In this step, you are mapping standardized data from the IBM Initiate Address Verification Interface postal reference data to fields in your output metadata, that will be passed to the requesting application. a. On the Address properties dialog, use Ctrl-click to select address properties for mapping, and click Next. 78 CloerETL User's Guide

87 b. On the Address output mapping dialog, map the IBM Initiate Address Verification Interface address properties to the output metadata. c. Click Finish. 11. Double-click the Initiate Handler Writer component to open the Edit component dialog. 12. Specify an Initiate connection for the Initiate Handler Writer. 13. Map the output attributes: a. Click the ellipsis in the Value field of the Mapping property to open the Handler Writer mapping dialog. b. Select the attributes to map, and click Next. c. Map your selected attributes to the appropriate Input fields. d. By clicking the Auto mapping button, you can enable CloerETL to perform the mapping by making a best guess based on the attribute codes and attribute names. e. Click Finish. 14. If you want to select key fields to designate master keys in your data, use the Key fields property to select your master key(s). 15. Sae the graph. Configuring the CloerETL standardization and alidation graph About this task The address standardization process takes an existing address record from an external file or from the IBM Initiate Hub, erifies it against a standardization or erification platform, and creates output with a standardized ersion of the address. The output can reside in temporary memory or as a record to be written to a file or database, depending on the design of the graph. The standardized output does not replace the original record, but supplements it with the standardized data. To illustrate, here is some sample, nonstandard input: Example Company Inc 200 W. Madison Suite 2300 Chicago Ill US Example Company Inc 5001 Plaza on the Lake Suite 111 Austin Texas USA Example Company Inc 2550 Union Hills Drie Suite 201 Phoenix Ariz USA In the example, postal codes are missing and the names of streets, state, and country are inconsistent. To standardize and alidate the input data, set up a graph with the following settings: Input to IBM Initiate Address Verification Interface component: deliery address line 1 = street number/name deliery address line 2 = suite number Output from IBM Initiate Address Verification Interface component: deliery address line 2 = street number/name The following data results from executing the graph: Example Company Inc 200 W Madison St Ste 2300 Chicago IL United States Chapter 3. IBM Initiate custom components 79

88 Example Company Inc 5001 Plaza On the Lk Ste 111 Austin TX United States Example Company Inc 2550 W Union Hills Dr Ste 201 Phoenix AZ United States In this example output, the addresses are in a standard and consistent format, the correct postal codes hae been added, and there are three additional fields of data. Let's look at how the data and graph are configured to produce this result. Fielding source data for best results in the CloerETL standardization and alidation graph How source data is structured and associated with IBM Initiate Address Verification Interface fields can affect the reliability of alidation. Fully fielded data is, in general, broken down into indiidual columns, each of which contains only one address element that is assigned to one IBM Initiate Address Verification Interface field. This structure yields the most accurate and reliable results. In our example, the data is considered fully fielded. Note: In fully fielded data, HouseNumber and Street alues can reside in the same column and be assigned to the Street field. With partially fielded data, some address elements are grouped into the same column and assigned to the more generic Address Line' fields. The grouping of similar but different data makes alidation less reliable. Unfielded data has no explicit structure and is usually not alidated. The Formatted Address fields are used with unfielded data, which usually is formatted according to line feeds or other delimiters, to produce usable output. More detailed information about data structure and IBM Initiate Address Verification Interface fields is aailable in the IBM Initiate Address Verification Interface documentation. Edge metadata for the CloerETL standardization and alidation graph On the IBM Initiate Address Verification Interface input Edge, we are including all the columns of the source file. Howeer, on the output Edge, we are passing additional metadata to complete the addresses and display information that is helpful for understanding how the data was standardized and alidated. 80 CloerETL User's Guide

89 We hae added the following items to the output metadata. ZIP: to include the missing postal codes. ElementResultStatus: an indicator of whether and how each address element was modified. Refer to the IBM Initiate Address Verification Interface proprietary documentation for the significance of ElementResultStatus codes. Mapping the CloerETL IBM Initiate Address Verification Interface component This section discusses mapping for the IBM Initiate Address Verification Interface component. About IBM Initiate Address Verification Interface address properties Mapping inoles an understanding of the IBM Initiate Address Verification Interface properties. There are three logical groups: Address element: Examples are Organization, Street, Proince (State), PostalCode, and Locality (City). These properties set and retriee alues for indiidual address elements. FormattedAddress: This set of properties presents alternaties to the address element properties. You can use them to map unstructured (unfielded) data that cannot be used with the more structured address element properties. Control and status: These properties, for example the DefaultCountry property, can determine how IBM Initiate Address Verification Interface works. They also can be used to set and retriee information about operations; the ElementResultStatus is an example. For more detailed information about IBM Initiate Address Verification Interface properties, refer to the IBM Initiate Address Verification Interface documentation. Input mapping in the example Our sample exercise included the following input mapping. Chapter 3. IBM Initiate custom components 81

90 These input fields are the same as the input Edge metadata. They are mapped to the corresponding IBM Initiate Address Verification Interface address properties. For input mapping, the Deliery address line 1 property accesses the Street1 data field; Deliery address line 2 accesses the Street2 data field, which, in our example, contains the Suite number. Output mapping in the example The following configuration is mapped for the output data. 82 CloerETL User's Guide In the output mapping, the fields for output data are the same as the output Edge metadata. They are mapped to the corresponding IBM Initiate Address Verification Interface properties.