Contents. Introducing Collibra Connect 1. About Collibra Connect 2. Collibra Connect deployment 2. Installing Collibra Connect 5

Transcription

1 1.4.1

2 Contents Introducing Collibra Connect 1 About Collibra Connect 2 Collibra Connect deployment 2 Installing Collibra Connect 5 Integrations with Collibra Connect: from development to production 7 Installing a development environment 8 System requirements - Anypoint Studio 8 Hardware requirements 8 Java Runtime Environment 8 Operating Systems 8 Install Anypoint Studio 9 Install the Collibra DGC Connector in Anypoint Studio 9 Prerequisites 9 Steps 9 Import the Collibra domain in Anypoint Studio 10 Prerequisites 10 Steps 10 Result 10

3 Installing a test and production environment 11 System requirements - Mule ESB runtime 11 Hardware requirements 11 Java Runtime Environments 11 Operating Systems 11 Install Mule ESB Server runtime 12 Prerequisites 12 Steps 12 Run Collibra Connect in Mule ESB 12 Prerequisites 13 Steps 13 Collibra domain deployments in Mule ESB Standalone server 13 Install the Mule Management Console 13 Prerequisites 14 Steps 14 Controlling a Mule ESB Server runtime instance 14 Linux 14 Windows 14 About the Collibra Connect components 17 Collibra domain application 18 Collibra domain purpose 18 Collibra domain configuration 19 About the gateway application 19 Gateway as single point of entry 19

4 Gateway for authentication 20 Gateway integration with workflow processes 20 Collibra DGC Connector 21 About upserting assets 21 About upserting assets by ID 21 Upserting assets by external ID 22 About upserting assets by name 23 Upserting assets by name 23 Configure the Collibra DGC Connector connection 24 Configure the parameters for upsert by external ID 25 Configure the parameters for upsert by name 26 About the payload of an upsert operation 27 UUIDs 27 Related assets 27 About assigning responsibilities 28 About the upsert parameters 29 General properties 29 DataSense Explorer 30 Fixed properties 31 Dynamic properties 33 Configuring Collibra Connect to connect to a proxied Collibra DGC 33 Enable debug logging 34 Override HTTP PUT and DELETE methods with HTTP POST 35

5 Running integrations in Collibra Connect 37 Import an integration template 39 Prerequisites 39 Steps 39 Run integrations with Collibra Connect in Anypoint Studio 39 Steps 40 Run integrations with Collibra Connect in Mule ESB Standalone server 41 Triggering an integration 42 Manually start an integration using the Collibra domain and gateway 42 Manually start an integration: options 43 Starting an integration without the Collibra domain and gateway 43 Setting the runtime environment of integration templates44 Check the Mule ESB runtime version 45 Change the embedded Mule runtime 46 Install a new Mule runtime version 46 Steps 46 Result 47 Developing a custom integration 49 University development course 51 Designing an integration template 51 Overview use case 51 Overview possible scenarios 52

6 Configuring integration templates 57 Importing CSV data 57 Upserting assets 58 Enriching assets with mapping 59 Generate natural key 60 Enrich with mapping 60 Converting data to CSV 62 Importing CSV data in Collibra DGC 63 Table view configuration 64 Learn about the Collibra Connect components 66 Use gateway with a custom integration template 66 Prerequisites 66 Steps 66 Result 69 About the gateway entry and end point 69 Entry point 69 End point 69 Connecting to the gateway from a workflow 70 Create a connection to the gateway 70 Connect to the gateway from a workflow 70 Sending information to the workflow 71 Processing a message in a workflow 72 Scheduling the trigger of an integration 73 Configuring the trigger interval 73

7 Configuring the integration template to start 74 Configuring the payload 75 Starting the integration template 75 Edit Collibra integration templates in Anypoint Studio 75 Collibra Marketplace 77 Frequently Asked Questions 78 Do I need a license for Collibra Connect? 78 What are the current versions of Collibra Connect? 78 Are there trainings provided? 78 Is Collibra Connect Installed on the Same Server of Collibra DGC? 78 What is the application server Collibra Connect runs on? 78 How does Collibra Connect communicate with SaaS Systems such as Workday, ServiceNow and Salesforce.com...? 79 How does Collibra Connect communicate with onpremise Systems such as SAP and Oracle Financial etc.? 79 Is there any Intrusion Detection Mechanism in place? 79 How is patching performed? Is there Auto Update? 79 What is the rollback process? 80 How about authentication, authorization, access control and data security? 80 What do I need to do when one of the applications I'm integrating with requires a secure (SSL) connection to self-signed certificate? 81

8 How do I check which version of the Collibra DGC Connector is used? 82 What if I need to go through a proxy server in order to connect to Collibra DGC from Collibra Connect? 83 How can I know which REST calls are made by the Collibra DGC Connector during execution of given integration flow? 83 How can I set verbose exception stacktrace? 84 How can I see the actual HTTP call message data? 84 Glossary 86 Index 91

9

10 CHAPTER 1 Introducing Collibra Connect Collibra Connect offers you a way to integrate your own application with Collibra Data Governance Center to enable active data governance. Tip Collibra DGC Connector supports Collibra DGC 5.1 or newer. If you are running any older version of Collibra DGC, use Collibra DGC Connector About Collibra Connect 2 Collibra Connect deployment 2-1 -

11 Chapter 1 About Collibra Connect Collibra Connect is an integration platform that enables integrations between Collibra Data Governance Center and other third-party products, such as Informatica, Salesforce.com and JIRA. Collibra Connect includes Anypoint Studio, which is an Eclipse-based development environment for developing and enhancing existing integrations. Collibra Connect also includes the Mule ESB Standalone server, which is the server application where the integration applications are deployed and run after they are developed in Anypoint Studio. You can use Collibra Connect to: leverage your existing infrastructure by reading metadata from your current systems. automate compliance. provision changes resulting from an issue resolution or data modification. implement active data governance. Collibra Connect deployment In the following schema, you see a typical deployment of Collibra Connect, Collibra Data Governance Center and any external system. Collibra Connect consists of two major components: Integration flow, often referred to as (integration) templates, defining the mapping between Collibra DGC and the external system. Connector which communicates with the Collibra DGC REST API

12 Introducing Collibra Connect In the schema, you can distinguish the following elements: Element Description 1 Collibra Data Governance Center 2 Collibra Connect. In the production environment, it is recommended to deploy the Mule ESB (Collibra Connect) server on a separate machine (physical or virtual) ensure that Collibra DGC and Collibra Connect do not share memory resources or CPU processing power. Additionally, the port on the Collibra Connect server to which Collibra Connect has to listen, must be open and reachable by the Collibra DGC server, to allow the triggering of integration flows. You can decide and configure which port that is, for example, 443 for standard HTTPS. 3 Collibra Connect integration templates communicate with Collibra DGC, using the Collibra DGC Connector. The Collibra DGC Connector encapsulates the REST API calls to Collibra DGC to provide session management support and streamline operations. It is also possible for Collibra DGC workflows to communicate (start integration flows or receive BPMN Message Events) with Collibra Connect, see Connecting to the gateway from a workflow. 4 Integration flows. Each integration flow performs integration logic (migration, broadcast...) between Collibra DGC and one or more external systems. You can find the documentation and download integration templates on the delivery site. 5 External systems that are to be integrated with Collibra DGC. The communication with the external system can happen through: A custom connector to the external system; for example IBM BG Connector. A REST API of the external system. A Relational Database connector to connect to a relation database. An other file-based connector, such as FTP, to read a file from a remote location. Any other method that is allowed by the system. In general, you can integrate any system with accessible APIs with Collibra DGC through Collibra Connect

13 Chapter 1 Element Description 6 Starting the integration. There are numerous ways of starting the integration automatically: An integration starts periodically with a configurable frequency. The integration starts automatically after a change in Collibra DGC or in the external system: A workflow in Collibra DGC is started automatically after a workflow-starting event, for example when an asset is added or modified. In turn, the started workflow triggers the required integration flow. With a built-in mechanism, an integration flow periodically checks the external system. If it detects a change, it starts the actual integration flow. An external system itself triggers the call to start the required integration flow when necessary. This is only possible if the external system can execute a REST call or register an event callback. A user starts a workflow in Collibra DGC that in turn calls the integration flow in Collibra Connect. A workflow in Collibra DGC calls the integration flow in a recurring interval. A user triggers the integration flow manually when the most upto-date view of the data is required. The user can do this by sending the trigger request (REST call) to the Gateway, see About the collibra-domain application. This request can be made from any computer, using tools like curl or POSTMAN

14 CHAPTER 2 Installing Collibra Connect The installation of Collibra Connect consists of adding the Collibra DGC Connector to Anypoint Studio or Mule ESB Standalone server. Note Anypoint Studio is used for development and configuration purposes, Mule ESB Standalone server for production purposes. Integrations with Collibra Connect: from development to production 7 Installing a development environment 8 System requirements - Anypoint Studio 8 Install Anypoint Studio 9 Install the Collibra DGC Connector in Anypoint Studio 9 Import the Collibra domain in Anypoint Studio 10 Installing a test and production environment 11 System requirements - Mule ESB runtime 11 Install Mule ESB Server runtime 12 Run Collibra Connect in Mule ESB 12 Collibra domain deployments in Mule ESB Standalone server

15 Chapter 2 Install the Mule Management Console 13 Controlling a Mule ESB Server runtime instance

16 Integrations with Collibra Connect: from development to production Installing Collibra Connect Similar to development processes, it is recommended to create integrations with Collibra Connect in multiple phases. The diagram depicts a sample deployment environment for Collibra Connect with three tiers: a development, a testing and a production environment. Each environment has its own purpose: Environment Purpose Development The development environment is used to develop the integration with Collibra Connect in Anypoint Studio. In this environment, you can use a test environment of Collibra Data Governance Center. A version control system is used to track the development history and to support collaborative development. UAT The UAT environment is used to test the integration. In the testing phase, you no longer run the integration in Anypoint Studio but on a non-production Mule ESB Standalone server. Production The production environment is used to run the integration on a production Mule ESB Standalone server. You can use MMC (Mule Management Console) to monitor Mule ESB Standalone server instances. It provides a centralized, convenient, and intuitive web-based interface to monitor, manage, and administer the run-time aspects of Mule ESB

17 Chapter 2 Installing a development environment In a Collibra Connect development environment, you develop integrations between Collibra Data Governance Center and third-party applications. The development environment consists of Anypoint Studio and the Collibra DGC Connector. System requirements - Anypoint Studio 8 Install Anypoint Studio 9 Install the Collibra DGC Connector in Anypoint Studio 9 Import the Collibra domain in Anypoint Studio 10 System requirements - Anypoint Studio It is highly recommended to use Anypoint Studio 6.x or newer. Hardware requirements 4 GB RAM 2 GHz CPU 10 GB free hard disk space Java Runtime Environment One of the following Java runtime environments: Warning On OS X or macos, you first have to install JRE 1.6 before installing one of the following JRE versions. Oracle Java SE JDK 7 or 8 (recommended). See docs.oracle.com/javase/8/docs/technotes/guides/install/install_overview.html. IBM JVM 1.7 Open JDK 8 Operating Systems Windows 7, Windows 8, Windows 10 (32-bit and 64-bit) OS X / macos or newer - 8 -

18 Installing Collibra Connect RHEL 7.0 Ubuntu Server Install Anypoint Studio If your computer meets all the system requirements, follow the steps in the Mulesoft documentation to download and install Anypoint Studio. Install the Collibra DGC Connector in Anypoint Studio Prerequisites You have downloaded the latest version of the Collibra DGC Connector (CollibraDGCConnector141UpdateSite.zip in the DGC Connector section) from the Collibra downloads page. Your Anypoint Studio setup is fully operational. Consult the Mulesoft documentation. You have set up the correct Mule ESB server runtime version in Anypoint Studio. We recommend version or newer. Steps To install the Collibra DGC Connector in Anypoint Studio, follow these steps: 1. On the Anypoint Studio menu bar, click Help Install New Software. 2. In the Install dialog box, click Add. 3. In the Add Repository dialog box, type a name for your repository, for example DGC Connector, then click Archive. 4. Locate the connector archive file (CollibraDGCConnector141UpdateSite.zip) and click Open. 5. Click OK. You return to the Install dialog box. 6. Expand Community and select CollibraDGC Connector (Mule ). 7. Click Next twice. 8. Accept the terms of the license agreement

19 Chapter 2 9. Click Finish. If you see a warning about unsigned content, click OK to proceed. 10. Click Yes to restart Anypoint Studio. Import the Collibra domain in Anypoint Studio The collibra-domain project enables sharing resources between different applications. It also enables you to deploy all integration templates with one click in Anypoint Studio. See Shared Resources to get a better understanding of the domain concept. Prerequisites You have downloaded the Collibra domain deployable archive (collibradomain.zip) from the delivery site. You have correctly installed Anypoint Studio with Collibra DGC Connector. See Install the Collibra DGC Connector in Anypoint Studio. Steps To import the Collibra domain in Anypoint Studio, follow these steps: 1. On the menu bar, click File Import. 2. In the Import dialog box, browse to Anypoint Studio and select Anypoint Studio generated Deployable Archive (.zip). 3. Click Next. 4. Locate the file collibra-domain.zip and click Open. 5. Click Finish. 6. Select the server runtime that you want to use and click OK. For the correct Result runtime version, consult the compatibility matrix of the archive on the delivery page. In the Package Explorer pane, you see collibra-domain in the list. If you imported version 1.0.0, then you see also gateway in the list

20 Installing Collibra Connect Installing a test and production environment A test or a production environment runs integration templates in a Mule ESB Standalone server instance instead of Anypoint Studio. System requirements - Mule ESB runtime 11 Install Mule ESB Server runtime 12 Run Collibra Connect in Mule ESB 12 Collibra domain deployments in Mule ESB Standalone server 13 Install the Mule Management Console 13 Controlling a Mule ESB Server runtime instance 14 System requirements - Mule ESB runtime It is highly recommended to use Mule ESB or newer. Hardware requirements 2 GHz, dual-core CPU, or 2 virtual CPUs in virtualized environments (Intel Xeon or equivalent) 2 GB RAM 4 GB of storage Java Runtime Environments Oracle JRE 7 or 8 (recommended) IBM version 1.7 Open JDK 8 Operating Systems Windows (32-bit and 64-bit) Server R2 8.1 OS X / macos or newer

21 Chapter 2 RHEL (64-bit) 5.8, 5.11, 6.6, 7 Ubuntu Server Oracle Solaris 11 IBM AIX 7.1 The list of supported operating systems is non-exhaustive. Mule ESB Standalone server should also be compatible with any newer operating system than the mentioned ones. It should also be supported by any operating system that supports the given Java Runtime Environments. Install Mule ESB Server runtime Prerequisites You have a Mule ESB Server license file. For more information, contact your Collibra Customer Success agent. You have downloaded the Mule ESB Server archive from the Collibra Community downloads page. Collibra recommends version or newer. Steps To install Mule ESB Server runtime on your server, follow these steps: 1. Extract the Mule ESB archive file to the location of your preference. 2. Install the license file, obtained via your Collibra Customer Success agent. See the MuleSoft documentation. To control the Mule ESB Server runtime instance, use the Mule Management Console (see Install the Mule Management Console) or see Controlling a Mule ESB Server runtime instance. Run Collibra Connect in Mule ESB Running Collibra Connect in Mule ESB is basically deploying the Collibra domain and the Collibra Connect integration template on this instance

22 Installing Collibra Connect Prerequisites Steps You have installed a Mule ESB Standalone server instance. You have downloaded the Collibra domain application (collibradomain.zip) from the Collibra downloads page. You have downloaded CollibraDGCConnector141UpdateSite.zip from the Collibra downloads page. To run the Collibra Connect in the Mule ESB Standalone server, follow these steps: 1. Save the collibra-domain.zip in the $MULE_HOME/domains folder. 2. Save the CollibraDGCConnector141UpdateSite.zip in the $MULE_ HOME/apps folder. 3. Start Collibra Connect. See Run integrations with Collibra Connect in Mule ESB Standalone server. Tip For more information about running applications in Mule ESB Standalone server, consult the Mule documentation. Collibra domain deployments in Mule ESB Standalone server To deploy the Collibra domain in Mule ESB Standalone server, see Run Collibra Connect in Mule ESB. To redeploy the Collibra domain, drop the new archive file in the domains folder. It is not necessary to first remove the old domain, a redeploy automatically overwrites the old domain. To remove the Collibra domain, delete the anchor file and directory of the Collibra domain in the domains directory. Install the Mule Management Console To monitor and manage your Mule ESB Server runtime instances, you can use the Mule Management Console (MMC)

23 Chapter 2 Prerequisites You have downloaded the MMC WAR file from the Collibra Community downloads page. Collibra recommends version or newer. You have downloaded Apache Tomcat. Steps To install the Mule Management Console on your system, follow these steps: 1. Rename the downloaded MMC WAR archive file to mmc.war. 2. Extract the Tomcat archive file in the location of your preference. 3. Copy mmc.war to <tomcat install directory>/webapps. 4. Start Tomcat. 5. Open a browser and go to It is possible that you have to use a different port. 6. Sign in with the initial credentials (admin/admin). To register a Mule ESB Server runtime instance, consult "Registering the Server" in the MMC documentation. Controlling a Mule ESB Server runtime instance Linux To control the Mule ESB Server runtime instance on Linux, go to the bin directory of the installation directory. Action Command Start instance mule start Stop instance mule stop Windows To control the Mule ESB Server runtime instance on Windows, go to the bin directory of the installation directory. Then convert the Mule ESB Server to a Windows server: mule install Action Start instance Command mule start

24 Installing Collibra Connect Action Command Start instance using.net utility net start mule Stop instance mule stop Stop instance using.net utility net stop mule

25 Chapter 2

26 CHAPTER 3 About the Collibra Connect components Collibra Connect is a combination of tools and applications that enable you to connect Collibra Data Governance Center with a third-party application. In this section, you get more details about the Collibra Connect components, such as the Collibra domain and the Collibra DGC Connector. Collibra domain application 18 Collibra domain purpose 18 Collibra domain configuration 19 About the gateway application 19 Gateway as single point of entry 19 Gateway for authentication 20 Gateway integration with workflow processes 20 Collibra DGC Connector 21 About upserting assets 21 Configuring Collibra Connect to connect to a proxied Collibra DGC 33 Enable debug logging 34 Override HTTP PUT and DELETE methods with HTTP POST

27 Chapter 3 Collibra domain application The collibra-domain application enables sharing resources between different applications. It also enables you to deploy all integration templates with one click in Anypoint Studio, see Run integrations with Collibra Connect in Anypoint Studio. See Shared Resources to get a better understanding of the domains concept. This section describes the purpose of the Collibra domain (collibra-domain) and how to edit and deploy it on a server. Collibra domain purpose 18 Collibra domain configuration 19 Collibra domain purpose The Mule ESB Standalone server runs on a single Java Virtual Machine (JVM). Its architecture allows for multiple applications to be deployed on a single server efficiently. However, in order to allow for resources, for example TCP ports, to be shared across different applications, deployed onto that same server, the configuration of the connector for the resources must be shared as well. Therefore, the configuration needs to be defined in a domain, rather than with each individual application, and the applications sharing the same domain can then share the same resource, for example TCP port 80. Basically the Collibra domain is an application which serves as a shared configuration for other applications

28 About the Collibra Connect components Collibra domain configuration When you deploy the Collibra domain for the first time, there are some default settings which most likely don't apply to your own environment, for example: TCP ports: For example, development and Quality Assurance environments usually run on other ports for HTTP/S while a production environment typically runs on the standard TCP ports 80 and 443. Certificate for HTTPS / SSL. Connectors for other shared services. To edit the domain project, follow these steps: 1. Import collibra-domain.zip as a mule deployable archive file (zip) into Anypoint Studio. See Import the Collibra domain in Anypoint Studio. 2. Change the necessary parameters to reflect your environment. 3. Export again as deployable archive file. See the MuleSoft documentation for more information. About the gateway application The Collibra gateway is an application that has the following features: A single entry point for all integration processes. An authentication mechanism. An integration with Collibra DGC workflow processes. Gateway as single point of entry 19 Gateway for authentication 20 Gateway integration with workflow processes 20 Gateway as single point of entry The gateway as single point of entry, listens for HTTP POST requests under collibra.connect.gateway on port The URL and port values can be changed by modifying the gateway.xml file in the gateway project (under src/main/app). When the gateway receives a message, it starts the integration process defined by the flowid parameter of the call

29 Chapter 3 See Triggering an integration for an example of starting an integration by using the gateway. Gateway for authentication Gateway authenticates and authorizes all incoming requests. Default credentials are: Username: connect_user Password: connect_password The credentials can be changed in the gateway.properties under src/- main/app. Example of specifying Basic Authentication credentials in Postman REST client:. Gateway integration with workflow processes Gateway provides an easy way for integration with Collibra DGC workflows. If the integration process was started from a Collibra DGC workflow and the dgcworkflowprocessinstanceid and dgcworkflowmessageeventname parameters were provided in the POST request (x-www-form-urlencoded), then the gateway sends the whole message payload back to message event identified by the above parameters. See Connecting to the gateway from a workflow for more details

30 About the Collibra Connect components Collibra DGC Connector The Collibra DGC Connector encapsulates the REST API calls to Collibra Data Governance Center to provide session management support and streamline operations. This section describes the features of the Collibra DGC Connector. Tip Collibra DGC Connector supports Collibra DGC 5.1 or newer. If you are running any older version of Collibra DGC, use Collibra DGC Connector About upserting assets 21 Configuring Collibra Connect to connect to a proxied Collibra DGC 33 Enable debug logging 34 Override HTTP PUT and DELETE methods with HTTP POST 35 About upserting assets Upserting assets is the process of synchronizing entities that are retrieved from an external system with assets in Collibra Data Governance Center. The integration logic creates new assets in Collibra DGC for entities that are synchronized for the first time and updates assets in Collibra DGC for entities that have been synchronized before. The outcome of the operation is the same: the state of the asset in Collibra DGC is updated to the current state of the external entity. This method is DataSense documentation enabled, for easier integration development. You can configure the upsert operations to either create a new domain and/or community for the assets that need synchronizing or to upsert to a default domain. About upserting assets by ID This operation uses the integration framework to upsert assets to Collibra DGC. Because of that, you can make multiple calls to the REST API of Collibra DGC

31 Chapter 3 during the execution of that function. Upsert assets by external entity id splits input assets into one collection with assets that have been previously synchronized using upsert and another collection with assets that are synchronized for the first time. After the split, two imports are done to create or update assets accordingly. Because of that, the function is not atomic. Additional calls can be made to retrieve required information, for example, checking if upserted assets were previously synchronized or finding IDs of related assets. The number of API calls does not depend on the number of assets that are upserted. The operation is idempotent, so performing multiple upserts with the same data will yield the same result. Each upserted asset has to have the same structure (same 'keys' defined) because of the import operation being used. By using DataWeave transform to produce the data as input for the upsert assets operation, the structure is guaranteed to be the same for all assets. Upserting assets by external ID Upserting assets by external ID enables you to create or update assets by using the entity ID that is used in the external system. The combination external entity ID and external system ID serves to identify assets. You need the following attributes to enable creating assets in Collibra DGC: The name or ID of the default domain where the asset has to be upserted. Optionally, the name or ID of the community in which the specified domain is located. If there is no asset in Collibra DGC that corresponds with the asset name, then the asset is created. If there is no domain in Collibra DGC that corresponds with the domain name, the domain is created, provided that the following attributes are specified: domain type id and/or domain type name domain name community id and/or community name

32 About the Collibra Connect components If not all of the attributes are specified, the upsert operation fails. If Collibra DGC has no community that corresponds with the upserted community name, the community is created if the community name is provisioned in the upsert operation. If the community name is not provisioned, the upsert operation fails. If you upsert assets by external ID, note that you can move assets to any domain in Collibra DGC afterward because of the external ID which ensures that the asset can be found and updated. About upserting assets by name A few additional calls can be invoked to enrich the given input with all the necessary data, for instance, for the given domain ID, the method searches for the domain name that corresponds to this ID. Such API calls are invoked once per whole set of assets, therefore they do not depend on the size of input data. Each upserted asset has to have the same structure (same 'keys' defined) because of the import operation being used. By using DataWeave transform to produce the data as input for the upsert assets operation, the structure is guaranteed to be the same for all assets. Upserting assets by name Upserting assets by name enables you to either create or update assets by using the name of an asset and a domain. You can use this type of upsert if the assets do not have an external ID. You need the following attributes to enable creating assets in Collibra Data Governance Center: The name or ID of the domain where the asset has to be upserted. Optionally, if you only provide the domain name, the name or ID of the community in which the specified domain is located. If there is no asset in Collibra DGC that corresponds with the asset name, then the asset is created. If there is no domain in Collibra DGC that corresponds with the domain name, then the domain is created provided that the following attributes are specified:

33 Chapter 3 domain type ID and/or domain type name domain name community ID and/or community name If Collibra DGC has no community that corresponds with the upserted community name, the community is created if the community name is provisioned in the upsert operation. If the community name is not provisioned, the upsert operation fails. If you upsert assets by name, note that if an asset name has changed, either in Collibra DGC or in the external system, the asset is not going to be updated but a new asset will be created instead. Configure the Collibra DGC Connector connection To configure a Collibra DGC Connector connection, follow these steps: 1. In Anypoint Studio and open your project. 2. Select the Collibra DGC Connector and click Edit

34 About the Collibra Connect components 3. Fill in the proper values of the following fields: Username Password Base Application Url: URL of your Collibra DGC instance, which must be running. 4. Select the Enable DataSense option. 5. Click Test Connection to test if all parameters are correct. 6. In the Test connection dialog box, click OK. 7. Click OK to close the Global Element Properties dialog box. Configure the parameters for upsert by external ID To configure the parameters for upserting assets by external ID, follow these steps: 1. In Anypoint Studio, open your project. 2. Select the Collibra DGC Connector. 3. On the General tab, go to the Basic Settings section and select Upsert assets by external entity ID in the Operation field. 4. Fill in the parameters on the same tab in the General section: Default Domain Id External System Id 5. Click next to the Asset Type Id field to refresh the list. The refresh retrieves all available asset types of the selected Collibra DGC instance. This action may take a couple of seconds. 6. From the new Asset Type Id list, select the type of asset that you want to upsert to Collibra DGC. A progress bar appears to show the progress of retrieving all the properties that an asset of the selected type may have. When this process is completed, the DataSense Explorer tab is populated with the retrieved attributes and relations. If there are new attributes or relations assigned to the selected asset type, they remain invisible in

35 Chapter 3 DataSense Explorer until you refresh the metadata. 7. Save the changes. Configure the parameters for upsert by name To configure the parameters for upserting assets by name, follow these steps: 1. In Anypoint Studio, open your project. 2. Select the Collibra DGC Connector. 3. On the General tab, go to the Basic Settings section and select Upsert assets by name in the Operation field. 4. In the General section, click next to the Asset Type Id field to refresh the list. The refresh retrieves all available asset types of the selected Collibra DGC instance. This action may take a couple of seconds. Click the refresh button again if new asset types do not appear in the list. 5. From the new Asset Type Id list, select the type of asset that you want to upsert to Collibra DGC A progress bar appears to show the progress of retrieving all the properties that an asset of the selected type may have. When this process is completed, the DataSense Explorer tab is populated with the retrieved attributes and relations. If there are new attributes or relations assigned to the selected asset type, they remain invisible in

36 About the Collibra Connect components DataSense Explorer until you refresh the metadata. 6. Save the changes. About the payload of an upsert operation UUIDs There are no attribute or relation type names in the textual representation of the transformation. Names of attribute types are translated to attribute type UUIDs with the DataSense feature. Relation types are translated to relation type UUIDs and the kind of relation, which specifies if the given asset is the head or tail of a given relation. Using UUIDs of properties instead of names during runtime allows a Collibra DGC user to freely modify the names of characteristics in Collibra DGC without the need of modifying the integration flow logic. If keys of a given attribute or relation type have not been defined in the payload, then any corresponding attributes and relations are not modified by the upsert assets operations. This means that you can manually modify other characteristics in Collibra DGC without worrying that those changes are removed by the integration process. Related assets You can define the relations by using id or externalid of related assets and by specifying the name of the related asset and its context, namely the domain and community. An example of defining relations using names is shown on the following image:

37 Chapter 3 About assigning responsibilities You can assign responsibilities to users on upserted assets. You can do this by using user ID, user name, group ID or group name. In Anypoint Studio, link the proper fields by dragging and dropping the input to the output. See Configure the payload of the upsert operation for more information. To assign responsibility for: a user by user name, use the property user.username. a user by user ID, use the property user.id. a group by group name, use the property group.groupname. a user by user name, use the property group.id. An example of defining responsibilities is shown in the following image: In the given example:

38 About the Collibra Connect components A role of Administrator ( ) is assigned to: A user named john.smith A user group with UUID e01987e3-ec95-4ee0-a7c d9fa A role Steward is assigned to a user with user name derived form the StewardUserName field in the input payload. Responsibilities on each asset defined in the mapping are replaced. This means that existing users and groups are overwritten with the users and groups defined in the mapping script. About the upsert parameters When you configure the upsert settings, there are some parameters which may require some more details. General properties Parameter Default Domain Id Description This is the ID of the domain in Collibra DGC where all the new assets have to be created. This parameter is used only when an asset is created, not when it is updated. This enables you to move the imported assets to different domains in Collibra DGC and still get them properly updated by the integration flow, without them being moved back to the original domain. This field is not displayed if you have selected the Upsert by name operation. You can define the default domain in the payload for

39 Chapter 3 Parameter External System Id Description each asset separately. If it is defined in the payload, the Default Domain Id field is ignored This is the ID of the external system. It can be any string that uniquely identifies any external system that is integrated with Collibra DGC. Please check here to get more details. This field is not displayed if you have selected the Upsert by name operation. DataSense Explorer In the DataSense Explorer pane, you can see a set of properties that you can assign to the assets that are going to be synchronized. The list is composed of two sets: Fixed properties that exist for each asset type (green section in the following image). Dynamic properties that depend on the selected asset type (gray section in the following image)

40 About the Collibra Connect components Fixed properties Parameter defaultdomain Description Functions the same as Default Domain Id but can be

41 Chapter 3 Parameter externalid name status lastsyncdata Description defined per asset. In this way, you can upsert assets into multiple domains during one execution of the operation. You can define the default domain by either: specifying the domain's id; or the domain's name and id; or the name of the parent community. This property is optional. If you do not define it, then the Default Domain Id is used. If you do define it, it takes precedence over the Default Domain Id. The unique identifier of an entity in the given external system. This property is required for each entity that has to be synchronized. External IDs have to be represented by immutable properties, so good candidates are UUIDs or database primary keys. They are used to find the related asset in Collibra DGC if a given entity has already been synchronized before. Please refer to Integration Framework for a detailed explanation. The name of the asset. This property is required for each entity that has to be synchronized. It is often just the name of the entity in the external system. This name must be unique in the domain in which the corresponding asset is going to be created. The status of the asset. This property is optional. The date of the last synchronization of the given entity. This property is optional. It can be used to perform incremental updates. These are updates that only affect assets that have been changed before

42 About the Collibra Connect components Dynamic properties Parameter Example Description attribute types relation types Description: List<Description> Groups: List<Business Asset> The attribute types are represented by a list of objects with only one property value. The values provided in the list are synchronized as values of the attribute with the corresponding name in the synchronized asset in Collibra DGC. The relation types are represented by a list of objects with two property values, namely id and externalid. You have to make sure that at least one of those two property values is defined. The properties are used to identify the related asset in Collibra DGC. id: represents the Collibra DGC ID of the asset. If this is defined, then the relation of type Groups, for example, is created for the asset with the given ID. externalid: represents the entity in the external system. If this is defined, then it can be used to create relations to assets that have already been synchronized with the upsert assets operation or with the integration framework. Configuring Collibra Connect to connect to a proxied Collibra DGC If you are using a proxy server to connect to Collibra Data Governance Center, you have to define this in the gateway configuration of the Collibra DGC Connector. Proceed as follows: 1. Open the gateway configuration (Package Explorer pane gateway src/main/app gateway.xml)

43 Chapter 3 2. In the Connections Explorer pane, expand gateway and double-click CollibraDGC to open the properties. 3. On the Advanced Settings tab, you can define the proxy settings: You have to provide proxy settings for every integration template in the Collibra domain that needs the proxy. This is due to the fact that the configuration of the Collibra DGC Connector is not shared across all templates in the domain, but set separately for each project. Tip To simplify setting and changing the proxy in the future, you can use property placeholders in each configuration. You can provide values for the property placeholders similarly to how it is done for dgc.config.baseapplicationurl or dgc.config.user in running integrations (see Running integrations in Collibra Connect), or you can follow the steps described in Mule's Configuring properties. Enable debug logging To enable debug logging of an integration template, follow these steps:

44 About the Collibra Connect components 1. In Anypoint Studio, click Run Run Configurations. 2. In the Run Configurations dialog box, click Mule Applications collibra-domain. 3. Click the Arguments tab. 4. Update the VM arguments as follows: 5. Click Apply to save the changes. 6. Click Run to start the domain in debug mode. Alternatively, you can follow the instructions for working with system properties on the Mulesoft website. Full information about calls and parameters that are made, along with the responses sent by the Collibra DGC instance, is available in the console log. Override HTTP PUT and DELETE methods with HTTP POST You can configure the Collibra DGC Connector to use the HTTP POST method instead of the HTTP PUT and HTTP DELETE methods. Open the advanced settings of the gateway to enable this feature, see Configuring Collibra Connect to connect to a proxied Collibra DGC

45 Chapter 3 You have to provide proxy settings for every integration template in the Collibra domain that has to replace the PUT and DELETE methods with POST. This is due to the fact that the configuration of the Collibra DGC Connector is not shared across all templates in the domain, but set separately for each project

46 CHAPTER 4 Running integrations in Collibra Connect In this section you learn how to run integrations in Collibra Connect using either Anypoint Studio or Mule ESB Standalone server. Import an integration template 39 Prerequisites 39 Steps 39 Run integrations with Collibra Connect in Anypoint Studio 39 Run integrations with Collibra Connect in Mule ESB Standalone server 41 Triggering an integration 42 Manually start an integration using the Collibra domain and gateway42 Manually start an integration: options 43 Starting an integration without the Collibra domain and gateway 43 Setting the runtime environment of integration templates 44 Check the Mule ESB runtime version 45 Change the embedded Mule runtime 46 Install a new Mule runtime version

47 Chapter 4 Steps

48 Running integrations in Collibra Connect Import an integration template To link Collibra DGC with an external system, you have to import the correct integration template or develop your own integration template. See Developing a custom integration. Prerequisites You have downloaded the integration template of your choice from the Collibra Marketplace. You have imported the Collibra domain. See Import the Collibra domain in Anypoint Studio Steps The import of an integration template is identical to the import of the Collibra domain, the only difference is that you have to select the integration template archive instead of the collibra-domain.zip archive. 1. On the menu bar, click File Import. 2. In the Import dialog box, browse to Anypoint Studio and select Anypoint Studio generated Deployable Archive (.zip). 3. Click Next. 4. Locate your integration template archive and click Open. 5. Click Finish. 6. Select the server runtime that you want to use and click OK. For the correct runtime version, consult the compatibility matrix of the archive on the downloads page. In the Package Explorer pane, you see the imported integration template appear. Run integrations with Collibra Connect in Anypoint Studio Running integrations with Collibra Connect in Anypoint Studio starts the Collibra DGC Connector, deploys the selected integration templates and establishes a connection to your Collibra DGC environment

49 Chapter 4 Tip Only run Collibra Connect in Anypoint Studio for development purposes. Use Mule ESB Standalone server for production purposes. See Run integrations with Collibra Connect in Mule ESB Standalone server. Steps To run integrations with Collibra Connect in Anypoint Studio, follow these steps: 1. On the menu bar, click Run Run Configurations. 2. In the Run Configurations dialog box, right-click Mule Applications, then click New. 3. Define the new configuration: In the Name field, type a name for your configuration. In the General tab, select the check box in front of collibra-domain. Optionally select the correct runtime version in the Target Server Runtime field. 4. In the Environment tab, configure the connection settings to Collibra Data Governance Center To create a variable, click New, fill in the Name and Value and click OK to save the variable. Add the proper variables. See Collibra DGC environment variables. 5. Click Apply. 6. Click Run. If the integration templates have been successfully deployed, you can see the following in the Console tab:

50 Running integrations in Collibra Connect Run integrations with Collibra Connect in Mule ESB Standalone server Running integrations with Collibra Connect consists of deploying the Collibra domain and one or more integration templates in the Mule installation directories and defining a connection to a Collibra DGC environment. To run an integration with Collibra Connect in Mule ESB Standalone server, follow these steps: 1. Deploy the required integration templates and collibra-domain on the server. Save the collibra-domain.zip in the $MULE_HOME/domains folder. Save the required integration templates in the $MULE_HOME/apps folder. 2. Define the variables of the connection to Collibra DGC as additional JVM parameters during the startup of the server or add to the $MULE_HOME/- conf/wrapper.conf file. See Collibra DGC environment variables. Example of configuration in wrapper.conf:

51 Chapter 4 wrapper.java.additional.8=\ -Ddgc.config.user=username wrapper.java.additional.9=\ -Ddgc.config.password=password wrapper.java.additional.10=\ -Ddgc.- config.baseapplicationurl= 3. Start the Mule ESB runtime server from $MULE_HOME: MAC or Linux:./mule Windows: mule.bat Read the Mule deployment section for a description of all deployment possibilities. You can deploy and manage applications on Mule ESB Standalone server with the Mule Management Console. See Mule Management Console for the user guide and reference materials. Triggering an integration With Collibra Connect, you can start integrations automatically, for example by using Collibra DGC workflows. However, when you use the Collibra Connect gateway, you can also manually start an integration by performing an HTTP POST call. Manually start an integration using the Collibra domain and gateway 42 Manually start an integration: options 43 Starting an integration without the Collibra domain and gateway 43 Manually start an integration using the Collibra domain and gateway To manually start an integration, follow these steps: 1. Open your HTTP client, for example Postman. 2. Send a POST request to with the correct options. See Manually start an integration: options

52 Running integrations in Collibra Connect As a result the integration with the given flowid starts and the payload parameter specifies a message that is passed to and interpreted by the integration process. The following image displays an example of starting the integration process from the Postman REST client. Tip Instead of manually starting an integration, you can also schedule the start of an integration. See Scheduling the trigger of an integration. Manually start an integration: options Option Description flowid Identifier of the integration. payload The payload that is sent to the process. Username Collibra Connect username. Password Collibra Connect password. Starting an integration without the Collibra domain and gateway Instead of starting an integration by using the Collibra domain and gateway, you can also start an integration by using Anypoint endpoints. Examples:

53 Chapter 4 Anypoint Studio endpoint HTTP inbound endpoint File inbound endpoint Database endpoint JMS endpoint Description Starts an integration if it receives an HTTP request (PUT, GET, DELETE, POST). Starts an integration if it receives an incoming file. Starts an integration if it receives a database request, for example, to poll content from the database. Starts an integration if it receives a new messages in a configured queue. Tip Instead of manually starting an integration, you can also schedule the start of an integration. See Scheduling the trigger of an integration. Setting the runtime environment of integration templates A Collibra Connect application package often contains multiple properties files, one per environment. The file name of those properties files contains the environment name in the middle, as shown in the following example. In this situation, the placeholder configuration contains an environment variable as shown in the following image:

54 Running integrations in Collibra Connect For development environments, you can specify the environment variables in the mule-project.xml file: For deployment to a server, environment variables can either be appended to the server's wrapper.conf file (located in the conf directory), or be added to the operating system. In wrapper.conf, the environment variables are part of the additional parameters: wrapper.java.additional.15=-dmule.env=dev wrapper.java.additional.16=-dapp.key=testkey Note The number must be sequential and can vary between servers, depending on the other parameters that are being used. Check the Mule ESB runtime version Before you install the integration templates and the Collibra DGC Connector, you have to check if you have the correct embedded Mule runtime version in Anypoint Studio. To check the version of the embedded Mule runtime in Anypoint Studio, follow these steps: 1. Open Anypoint Studio. 2. In the Package Explorer pane, click the arrow next to any project. 3. In the files, look for the text Mule Server and check its version. For example:

55 Chapter 4 Change the embedded Mule runtime It is possible that you have installed more than one version of the Mule ESB server runtime. To change the runtime of a project, follow these steps: 1. In the Package Explorer pane, expand the project of your choice and search for mule-project.xml. 2. Double-click it to open the package details. 3. In the Server Runtime field, select the runtime that you want to use for the package. 4. Close the details tab. Install a new Mule runtime version Steps To install an extra Mule runtime version, follow these steps: 1. On the Anypoint Studio menu bar, click Help Install New Software. 2. In the Work with field, select All Available Sites. 3. In the filter field, type Mule ESB Server Runtime to limit the number of results. 4. Scroll through the results and select the check box(es) of the runtime instances that you want to install. 5. Click Next twice consecutively. 6. Accept the terms of the license agreement and click Finish

56 Running integrations in Collibra Connect Result The new Mule ESB Server runtime versions are installed and can be selected per integration template

57 Chapter 4

58 CHAPTER 5 Developing a custom integration In this section, you learn how to develop custom integrations. It provides more information on how to integrate your own integration template with Collibra Connect. University development course 51 Designing an integration template 51 Overview use case 51 Overview possible scenarios 52 Configuring integration templates 57 Learn about the Collibra Connect components 66 Use gateway with a custom integration template 66 Prerequisites 66 Steps 66 About the gateway entry and end point 69 Connecting to the gateway from a workflow 70 Create a connection to the gateway 70 Connect to the gateway from a workflow

59 Chapter 5 Sending information to the workflow 71 Processing a message in a workflow 72 Scheduling the trigger of an integration 73 Configuring the trigger interval 73 Configuring the integration template to start 74 Configuring the payload 75 Starting the integration template 75 Edit Collibra integration templates in Anypoint Studio

60 Developing a custom integration University development course Collibra University has a development course that can help you to create your own integration templates. This course covers the following topics: Install Anypoint and Create a Basic Flow Install Mulesoft and MMC on a Server Walkthrough of the HANA, Tableau and Hadoop Flows Traceability Customization You can access this course on the Collibra University pages. Designing an integration template When you start creating a new integration template, you first have to decide which metadata you are going to import with Collibra Connect and how you are going to update it. Overview use case 51 Overview possible scenarios 52 Configuring integration templates 57 Overview use case The following figure is a high-level overview of the integration scenario for importing data in Collibra DGC

61 Chapter 5 Step Description 0 Starting the integration flow. This can be done in three different ways, by: A user who starts a workflow in Collibra DGC that calls the integration flow in Collibra Connect. A workflow in Collibra DGC that calls the integration flow in a recurring interval. Calling the integration flow directly. 1 The integration flow has started. 2 The integration flow retrieves data from an external system. This can be done by using: A custom connector to the external system. For example: IBM BG Connector. A Relational Database connector to connect to a relation database. An other file-based connector, such as FTP to read a file from a remote location. 3 The integration flow transforms the data from the external system to a format that the Collibra DGC Connector can understand. To do this: i. Define the CSV data model of the data to be imported in Collibra DGC. ii. Use the Mulesoft Mapper to convert the data from the external system into the data model required by Collibra DGC. 4 Import the data into Collibra DGC by using the CSV Import function from the Collibra DGC Connector. Overview possible scenarios When you are defining an integration flow to import data from a different application in Collibra DGC, there are a few scenarios that the integration has to

62 Developing a custom integration support. Below is a list of the most common and important scenarios. Scenario Description Solution Only import the delta Move asset in Collibra DGC You do not typically want to import the entire external system every time you run the integration. You only want to import the assets from the external system that are new or have changed since the last time the integration was running. This is mainly for performance reasons. When you have imported an external entity into Collibra DGC with the integration flow, you usually want to move that asset to a different community or domain. Other systems typically don't have the same concept of communities or domains, as they are not governance solutions. When you have moved an asset in Collibra DGC to a different community or domain, you do not want the asset to be created again the next time you run the integration flow. The integration flow has to be smart enough to recognize that the asset already exists in Collibra DGC, but that it's in a different location. To accomplish this, you have to: 1. Uniquely identify the entity in the external system. 2. Map the external entity to the Collibra DGC asset. 3. Keep track of the last sync date. 4. Include the last sync date in the query to get the external system entities. To accomplish this, you have to: 1. Uniquely identify the entity in the external system. 2. Map the external entity to the Collibra DGC asset. 3. Update the (moved) asset

63 Chapter 5 Scenario Description Solution Update asset in external sys- the asset in Collibra DGC. You do not want to tem Recreate instead of creating a new one when importing. When an entity is updated in the external sys- To accomplish this, tem, you want the changes to be reflected in create a completely new asset in Collibra DGC, because you might have added other attributes or relations to the asset that you do not want to lose. asset in Col- deleted by accident, you typically want it to libra DGC When an imported asset in Collibra DGC is be re-created the next time the integration flow is run. you have to: 1. Uniquely identify the entity in the external system. 2. Find the corresponding asset in Collibra DGC through the mapping. If the corresponding asset exists in Collibra DGC, update it instead of creating a new one. To accomplish this, you have to: 1. Uniquely identify the entity in the external system. 2. Find the corresponding

64 Developing a custom integration Scenario Description Solution Delete asset When entity is deleted in the external system in external system you usually also want to delete the corresponding asset in Collibra DGC. Other approaches could be to change the status of the asset in Collibra DGC to Deleted or Deprecated for example. asset in Collibra DGC through the mapping. 3. Re-create the corresponding asset if it no longer exists in Collibra DGC. To accomplish this, you have multiple options: Sometimes external systems provide easy ways to retrieve entities that were removed since given point in time. If this is the case you can: 1. Retrieve unique identifiers of the entities that were removed in the external system. 2. Find corresponding assets in Collibra DGC through the mapping. 3. Remove or

65 Chapter 5 Scenario Description Solution change the status of the assets that were found. If external system doesn't provide information about entities that were removed then you can: 1. Review all the Collibra DGC assets that are mapped to an entities in an external system during the integration process. 2. Check if the entity still exists in the external system for each mapped asset in Collibra DGC (ideally in batch instead of one by one). 3. Delete the asset in Collibra DGC or change its status if the

66 Developing a custom integration Scenario Description Solution asset does no longer exist in the external system. When using the upsert assets operation, you do not have to look manually for corresponding assets in Collibra DGC through mapping because it is done automatically. You only have to provide the unique identifier of the entity in the external system (here you can find more information about that). The operation also takes care of recreating assets that have been removed from Collibra DGC and of checking if an asset already exists and creating or updating it accordingly. This way, you are free to move assets to different domains in Collibra DGC, as described in the Move asset in Collibra DGC scenario. Configuring integration templates Typically, the high-level integration flow logic is composed of several steps but the configuration that you have to perform is slightly different for each method. Importing CSV data To configure the import of CSV data, you have to configure the following steps: Integration Step Get External Data Description In this step, the external system is queried to retrieve the data. This can be done through an SQL connector, a custom connector or something similar. Sometimes it is useful to create a Java Class (POJO), so that the retrieved data is instantiated as Java Objects. That makes it

67 Chapter 5 Integration Step Enrich with Mapping Convert to CSV Import in Collibra DGC Description easier to test and map the data to the Collibra DGC format in future steps. It also improves the robustness of the solution. This is the most complex step. In this step: 1. The unique ID of the external asset is identified. 2. The mapping between the external asset and the Collibra DGC asset is created or updated. 3. The asset is created. 4. The two Delete cases are handled, either on the Collibra DGC side or on the external system side. For more information about enriching assets with mapping, see Enriching assets with mapping In this step, the external data is transformed to the format that Collibra DGC understands. Usually a CSV format is used, because this is the most performing way to update the assets in Collibra DGC. The import only updates the existing assets in Collibra DGC and does not create new assets. This is necessary to cope with the following scenarios: Move assets in Collibra DGC and Update asset in the external system. For more information about converting data to CSV format, see Converting data to CSV. In this step, the Collibra DGC Connector is used to import the CSV data from the previous step in Collibra DGC. For more information about importing CSV data in Collibra DGC, see Importing CSV data in Collibra DGC. Upserting assets

68 Developing a custom integration Integration Step Get External Data Description In this step, the external system is queried to retrieve the data. This can be done through an SQL connector, a custom connector or something similar. Sometimes it is useful to create a Java Class (POJO), so that the retrieved data is instantiated as Java Objects. That makes it easier to test and map the data to the Collibra DGC format in future steps. It also improves the robustness of the solution. Transform using DataWeave Upsert to Collibra DGC This step is identical as in the method where import to CSV is used. This step describes the core business logic of the integration. Here you define how to transform the entity retrieved from the external system to an asset of chosen type in Collibra DGC. In this step, given assets are upserted into Collibra DGC. It is equal to the last three steps described in the import CSV section (Importing CSV data). Here each asset is enriched with mapping to check if it was previously synchronized into Collibra DGC. Conversion into CSV and import are also made automatically. For more information about upserting assets, see About upserting assets. Enriching assets with mapping Enriching assets with mapping iterates over every external asset, generates its unique ID, and enriches the asset with the mapping. You can accomplish this with a subflow as shown in the following image:

69 Chapter 5 Generate natural key How you generate the natural key of the external system asset completely depends on the external system. Ideally, the external system already has a unique ID that can be used to uniquely identify the asset in a consistent way. If this is not the case, it is a good practice to create a sub-flow that generates the natural key in a consistent way, as shown in the following image: Enrich with mapping To enrich each asset with the right mapping information, a subflow is created that typically looks like the next image: It can consist of the following steps: Step Find the mapping Description Use the Collibra DGC Connector to find a mapping to a Collibra DGC asset based on the external asset ID (using its unique ID) and the external system ID. Example:

70 Developing a custom integration Step Description Delete mapping When a mapping is found, but the mapping has no Collibra DGC asset ID, it means that the asset has been deleted in Collibra DGC. You need to clean up the existing mapping by deleting it. Example: Check if mapping exists Depending on whether a mapping already exists between the external asset and a Collibra DGC asset, you need different behaviors. If the mapping already exists, it means that the asset in Collibra DGC also already exists and you only have to update its last sync date. If not, you need to create the asset in Collibra DGC, as well as the mapping. Example:

71 Chapter 5 Step Description Create asset and mapping If the mapping does not yet exist, you have to create the asset in Collibra DGC, as well as the mapping. Example: Converting data to CSV The first step to converting the external data to the Collibra DGC format, is deciding what you want to import in Collibra DGC. The unique identifier of the asset in Collibra DGC is always used. That is important to support the following scenarios: If an asset has been renamed in Collibra DGC, the correct asset is updated and is also renamed to the external system asset name. If an asset has been moved in Collibra DGC, the correct asset is updated instead of a new asset being created. You get the asset ID from the mapping information you have gathered in the previous step

72 Developing a custom integration The next image displays an example of a mapping from a POJO to a simple CSV structure: Importing CSV data in Collibra DGC You can use the Collibra DGC Connector to import the CSV that resulted from converting the external data. The important part here, is the TableViewConfig that specifies how Collibra DGC has to interpret the CSV data and map it to Collibra DGC concepts. The following image is an example import step as well as the TableViewConfig

73 Chapter 5 Table view configuration You have to configure the TableViewConfig as follows: Asset (Term) ID should be the unique identifier of the Collibra DGC assets. You can get that ID from the mapping information. The default operation has to be UPDATE, to cope with the scenarios described earlier. You already created the asset, so you do not have to CREATE anything anymore. Next you find an example TableViewConfig as used in the previous examples. { "TableViewConfig": { "Columns": [ { "Column": { "fieldname": "Id", "index": 0 } }, { "Column": { "fieldname": "Signifier", "index": 1 } }, { "Column": { "fieldname": "ParentId", "index": 2 } }, { "Column": { "fieldname": "Name", "index": 3 } }, { "Column": { "fieldname": "Description", "index": 4 } }, { "Column": {

74 Developing a custom integration "fieldname": "DataType", "index": 5 } } ], "Default": { "Operations": { "Term": [ "UPDATE" ] } }, "Resources": { "Term": { "Id": { "name": "Id" }, "Signifier": { "name": "Signifier" }, "StringAttribute": [ { "LongExpression": { "name": "Description" }, "labelid": "${dgc.- metamodel.attribute.descriptionid}" }, { "LongExpression": { "name": "Name" }, "labelid": "${dgc.- metamodel.attribute.nameid}" }, { "LongExpression": { "name": "DataType" }, "labelid": "${dgc.- metamodel.attribute.datatypeid}" } ], "Relation": { "typeid": " ", "type": "SOURCE", "KeyContent": [

75 Chapter 5 } } } } "ParentId" ], "Target": { "Id": { "name": "ParentId" } } }, "name": "Asset" Learn about the Collibra Connect components Before you can actually start developing an integration template, ensure that you completely understand all the Collibra Connect components. Collibra domain application About the gateway application Collibra DGC Connector Use gateway with a custom integration template When you develop a custom integration template, you may have to integrate with Collibra DGC workflows or configure the security of those templates. The gateway template provides an example and a quick start for adding those features to the integration templates. The only thing you need to do to reuse gateway features, is "registering" the integration template that the gateway has to use. Prerequisites You have installed collibra-domain. See Import the Collibra domain in Anypoint Studio. Steps To use the gateway with a custom integration template, follow these steps:

76 Developing a custom integration 1. In your new integration template, find the VM connector in the palette: 2. Drag the connector to create the message source of your main integration template: Note The name of the connector (The entry point) is by default VM. 3. Double-click the connector to change its properties: i. On the General tab, update the Display Name to a meaningful name. ii. From the Connector Configuration list, select shared_vm_connector. iii. Define the Queue Path. It specifies the name by which the gateway recognizes the integration. It corresponds to the flowid parameter that is used when triggering integrations with the use of the gateway. See Triggering an integration

77 Chapter 5 4. Drag and drop a VM connector at the end of your integration flow. Note The added VM connector is also added as the last element of the exception handling strategy. When an exception occurs during the integration process, it is also useful to notify the workflow about it and close the Collibra DGC session. 5. Double-click the connector to change its properties: i. On the General tab, update the Display Name to a meaningful name. ii. Select shared_vm_connector from the Connector Configuration list. iii. In the Queue Path field, type gateway_end_process. 6. Define the payload, which must be sent to the gateway, by adding an expression in front of the end point. Example of setting a response to "Success" using the Expression com

78 Developing a custom integration ponent: Result Your integration script is fully integrated with the Collibra gateway template. For more examples using the gateway, see About integration templates. About the gateway entry and end point Entry point Gateway communicates with other applications using the VM Connector. Adding an entry point to the application allows the gateway to start the required integration process. End point If the integration process is triggered by a Collibra DGC workflow, you may want to send the result back to the workflow instance or close the Collibra DGC session. This would prevent the session from being left open after the integration process has ended. Each of those steps may be performed manually in any part of the flow, see Sending information to the workflow, but the gateway handles it automatically. The gateway sends all text properties that are contained in the "payload" to the Collibra DGC workflow instance and closes the session. Those properties can be accessed later as variables of a given workflow process instance. The gateway sends the response back to the workflow only if the dgcworkflowprocessinstanceid and dgcworkflowmessageeventname properties were sent during the start of the integration process. See Connecting to the gateway from a workflow for more details

79 Chapter 5 Connecting to the gateway from a workflow In this section you learn how to integrate the gateway in a Collibra DGC workflow, trigger the gateway and send messages back to the workflow. Create a connection to the gateway 70 Connect to the gateway from a workflow 70 Sending information to the workflow 71 Processing a message in a workflow 72 Create a connection to the gateway You can start Collibra Connect gateway by performing an HTTP POST call to a predefined URL, see About the collibra-domain application. In order to be able to call the gateway from Collibra DGC, you have to provide the following connection properties in the Collibra DGC configuration.xml file: Property Value base-url URL where the Collibra Connect gateway is waiting for the requests. username Name of the user that is allowed to start the gateway process. password Password of the given user. An example with default values: Connect to the gateway from a workflow To call the gateway from a Collibra DGC workflow, you have to use the following delegate: com.- collibra.dgc.core.workflow.activiti.delegate.startcollibraconnectflow

80 Developing a custom integration The integration is started when the workflow reaches the state defined by the delegate. An example of configuring the delegate to start the workflow (flowid parameter defines which integration has to be started): Sending information to the workflow To send information from the integration template to the Collibra DGC workflow, you can use a message event mechanism. Collibra DGC Connector contains a method for sending message events:

81 Chapter 5 By using the displayed configuration, a message event "dgcworkflowmessageeventname" is sent to the workflow process instance, identified by "dgcworkflowprocessinstanceid". Additionally, all text properties contained in the message payload are accessible in the workflow as standard variables, for example ${status}. Processing a message in a workflow To receive the BPMN Message Events in the workflow, you have to use a MessageCatchingEvent. You can add the message event to the workflow: Additionally, the name of the message event and the process instance ID of the workflow have to be sent to the gateway using the delegate. The "messageeventname1" corresponds to the configuration of MessageCatchingEvent

82 Scheduling the trigger of an integration Developing a custom integration The following sections provide an example and describe how to trigger an integration periodically without using Collibra DGC workflows or any other external mechanism. To start an integration process periodically from Mule, you can just send a trigger event (with the required frequency) to an endpoint of the chosen flow. You can find the example application on the Collibra delivery site. The following screenshot is the visual representation of the flow. Configuring the trigger interval 73 Configuring the integration template to start 74 Configuring the payload 75 Starting the integration template 75 Configuring the trigger interval A Poll Scope is used to produce events with the required frequency. To configure the frequency, go to the properties of the Poll component and fill in the required value in the Frequency field. Example:

83 Chapter 5 The trigger event is produced every 30 seconds if the configuration as shown above is used. Also set the Start delay property to 30 seconds, to allow the required integration flow to start before the first trigger event is sent. Configuring the integration template to start The required integration is started by sending an event to a queue of the VM endpoint exposed by the integration. The following image shows that an event is sent to the integration, which is started by events sent to the dgc2ibmbg queue. To find the name to put in the Queue Path field, you can go to the section that describes triggering the integration process in the documentation of the given template and look for the value of the flowid parameter that is used to start the integration. If you have already imported the templates into Anypoint Studio, then you can go to the file called endpoints.xml and check the 'Queue Path' of the required VM connector as shown in the following image

84 Developing a custom integration Configuring the payload To configure the payload of the event that has to be sent to the required integration, edit the value of the Set Payload component, as shown in the following image: Starting the integration template This example integration template is attached to the collibra-domain, see About the collibra-domain application,so it is started along with other applications in the domain. The trigger events are sent periodically, as configured in the section configuring the frequency, see Configuring the trigger interval. Edit Collibra integration templates in Anypoint Studio Collibra Connect integration templates are shipped with generic development endpoints, settings and credentials. Therefore, you have to modify them before

85 Chapter 5 you can deploy them in your specific environment. You can modify a template with Anypoint Studio, which can be download from the Collibra Community downloads page. Import the template in Anypoint Studio as a Mule project for editing. Tip Templates can be mavenized or not mavenized. A template is mavenized if there is a pom.xml file in the template ZIP file. To edit an existing Collibra integration template, follow these steps: 1. In Anypoint Studio, click File Import. 2. In the Import dialog box: Action Import mavenized template using Maven: Import mavenized template without using Maven: Import template as deployable archive: Description If you use Maven as your build automation framework: i. Extract the template archive file. ii. Expand Anypoint Studio and select Maven-based Mule Project from pom.xml. If you do not use Maven: i. Extract the template archive file. ii. Expand Anypoint Studio and select Anypoint Studio Project from External Location. If the template is packaged as deployable archive, the archive does not contain a folder src/main/apps, expand Anypoint Studio and select Anypoint Studio generated Deployable Archive (.zip). 3. Apply the necessary changes to the template. 4. Click File Export. 5. Expand Mule and click Anypoint Studio Project to Mule Deployable Archive (includes Studio metadata). Note If you use Maven to create a package: i. Open a terminal session. ii. Go to the directory that contains the template's pom.xml file. iii. Execute the command: mvn clean package

86 CHAPTER 6 Collibra Marketplace The Collibra Marketplace contains a variety of integration templates. The templates help you to accelerate Collibra implementations and to adopt a data culture by data citizens in your organization. You can reach the Collibra Marketplace with marketplace.collibra.com, where you can download the templates and their documentation

87 Chapter 7 Frequently Asked Questions Do I need a license for Collibra Connect? Yes. In order to use Collibra Connect, your license for the Collibra DGC has to support Collibra Connect. Please reach out to your Collibra Customer Success representative or contact us at contact@collibra.com to get more information on pricing and features on Collibra Connect license. What are the current versions of Collibra Connect? The current version is 1.3.x for the Collibra DGC Connector running on Anypoint Mule ESB with Java 1.8 ( Java 1.7 is also supported). Collibra recommends Mule ESB or newer. Are there trainings provided? Please contact your account executive and customer success manager for Collibra Connect specific training courses. For general Anypoint development, free and paid courses can be found here. Is Collibra Connect Installed on the Same Server of Collibra DGC? For non-production usage, assuming the server has enough storage (100 GB+) and memory (4 GB+) Collibra Connect can run on the same server as Collibra DGC. For production the recommendation is always to use separate servers. What is the application server Collibra Connect runs on? Collibra Connect runs on the Anypoint ESB Server, version 3.7.x or newer. This is a java application server and can run on Linux, Unix, macos and Windows based servers respectively

88 Frequently Asked Questions How does Collibra Connect communicate with SaaS Systems such as Workday, ServiceNow and Salesforce.com...? Collibra Connect can leverage existing Anypoint Connectors for most common systems such as Workday, ServiceNow and Salesforce.com. For systems without existing connector, direct REST or SOAP API calls can also be made using out of the box web services consumer. Under the hood, the connector is using HTTPS to communicate to the system. Note that data encryption during transmission depends on the endpoint's API capability. Generally, message is encrypted and transmitted via HTTPS/SSL. How does Collibra Connect communicate with on-premise Systems such as SAP and Oracle Financial etc.? Collibra Connect can leverage existing Anypoint Connectors for most common systems such as Workday, ServiceNow and Salesforce.com. For systems without existing connector, direct REST or SOAP API calls can also be made using out of the box web services consumer. Under the hood, the connector is using HTTPS to communicate to the system. Java libraries based communication, JMS, FTP and JDBC etc. are also supported. Note that data encryption during transmission depends on the endpoint's API capability. Generally, message is encrypted and transmitted via HTTPS/SSL. Is there any Intrusion Detection Mechanism in place? Collibra Connect runs at the application server layer level, therefore, there is no intrusion detection mechanism out of the box since the detection is generally at the operating system and network level. It will not interfere with any network or operating system level mechanism. How is patching performed? Is there Auto Update? There is no auto update for Collibra Connect. Patches are delivered as jar or zip files, and can simply be dropped into directory

89 Chapter 7 What is the rollback process? Collibra Connect applications can be un-deployed at any given time. For the integration process itself, if rollback capability is required, it must be designed and the built in. How about authentication, authorization, access control and data security? Collibra Connect is an integration platform, on top of which actual use-case specific integration applications can be deployed and run. Therefore, out of the box the platform simply comes with capabilities that support most common authentication protocols in use today, such as LDAP, OAuth, SAML and Webservices Security. However, no specific mechanism is turned on. Specific authentication and control will depend on the actual integration application running on Collibra Connect. Note that the integration is at system to system level, which may require a different mechanism than that for end user authentication, as dictated by the end system being connected to. For example, one does not "log into" Collibra Connect, but will build a Collibra Connect app to log into Workday, using a Workday API user which can be stored in encrypted format. As a best practice, integration applications do not store data in transit, although data can still be stored, either on file or in database, by the integration application if the requirements call for persistence. For sensitive information, such as username and password, these can be stored as encrypted using the Anypoint Credential Vault that comes with Collibra Connect. Basic Auth backed by LDAP One of the most common use cases, especially for the gateway, is to have Basic Auth backed by LDAP (for example for user control via Active Directory). To create a security provider for LDAP to work with Basic Auth, please follow the documentation on the MuleSoft website. Note the documentation is for Anypoint ESB 3.7 but it applies to newer versions as well

90 Frequently Asked Questions What do I need to do when one of the applications I'm integrating with requires a secure (SSL) connection to selfsigned certificate? When connecting to a server that has a self-signed (not CA-verified) certificate, Collibra Connect may throw an exception with the message: "unable to find valid certification path to requested target". For Mule runtime and higher, simply check the "insecure" attribute in TLS configuration for the HTTP requester connector: For Mule runtime version 3.7.x, the insecure certificates should be imported into a custom truststore and the configuration should include the truststore information. Alternatively, adding the certificate to the default Java truststore used by Collibra Connect can work as well but note that this change will be at the operating system level. 1. Get the certificate from the server. 2. Import the certificate to the truststore of the Java that is used by the Collibra Connect (example below): keytool -keystore "c:\program Files\Java\jdk1.7.0_75\jre\lib\security\cacerts" \ -importcert -alias specifythealiashere -file thecertificate.cer

91 Chapter 7 3. Verify that the certificate was installed: i. keytool -keystore \ "c:\program Files\Java\jdk1.7.0_ 75\jre\lib\security\cacerts" -list ii. Look for the alias that was specified for the certificate If necessary, it is also possible to specify a different truststore using the standard Java mechanism. This can be done by providing the path to the given truststore with the "javax.net.ssl.truststore" startup parameter. Example: -Djavax.net.ssl.trustStore="d:\Java 7 JDK\jre\lib\security\cacerts" How do I check which version of the Collibra DGC Connector is used? When working in Anypoint Studio, go to Help Installation Details and look for CollibraDGC Connector entry. You can also see the version of used connector in the log file. It is logged when the server is started. You should see a message similar to the one provided below: DevKit Extensions (1) used in this application CollibraDGC (DevKit Build UNNAMED.2363.ef5c8a7)

92 Frequently Asked Questions What if I need to go through a proxy server in order to connect to Collibra DGC from Collibra Connect? Open the configuration of given connector and go to Advanced Settings tab. Fill the fields with necessary proxy server information. For more details, see Configuring Collibra Connect to connect to a proxied Collibra DGC. This feature is available since Collibra DGC Connector version How can I know which REST calls are made by the Collibra DGC Connector during execution of given integration flow? You can define a java system property com.collibra.dgc.rest.client.debug=true and restart the server. All calls made with the use of Collibra DGC Connector will be logged. You can find this property described also in the documentation of the connector or you can see detailed description on this page. Please remember that this setting should not be used in production environment. To add this setting, either use runtime argument for the JVM:

93 Chapter 7 Or as environment variables for the application: This feature is available since Collibra DGC Connectorversion How can I set verbose exception stacktrace? In your runtime argument for the Java VM, add this: - Dmule.verbose.exceptions=true How can I see the actual HTTP call message data? Collibra Connect logging is built on the open-source Apache Log4j2 framework. More details about the logging capability can be found in standard logging documentation. For HTTP calls, in order to see the actual messages sent and