IBM. IBM i2 Analyze Data Acquisition Guide. Version 4 Release 0

Transcription

1 IBM IBM i2 Analyze Data Acquisition Guide Version 4 Release 0

2 Note Before using this information and the product it supports, read the information in Notices on page 29. This edition applies to version 4, release 0, modification 0 of IBM i2 Analyze and to all subsequent releases and modifications until otherwise indicated in new editions. Copyright IBM Corporation 2012, US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

3 Contents Chapter 1. Data acquisition in i2 Analyze 1 Chapter 2. Approaches to data acquisition Data load direct diagram Data load direct diagram Data load ELP stage Data load ELP stage diagram Data access on-demand Solution types External data subset creation External data subset exploration External data retrieval Chapter 3. Data conversion Conversion of external data to XML XML schema definitions XSD reference Conversion of XML to i2 Analyze items Modification of i2 Analyze items Chapter 4. Development process for data access on-demand Supported operations Data source queries Data transformation Creation and deployment Notices Programming interface information Trademarks Copyright IBM Corp. 2012, 2015 iii

4 iv Data Acquisition Guide

5 Chapter 1. Data acquisition in i2 Analyze Data acquisition extends the functionality of i2 Analyze by enabling users to visualize and investigate data from third-party providers. Users have the choice of leaving the data in place or adding it to the Analysis Repository, in full or in part. Data acquisition solutions enable users to manipulate the data in external sources with the tools in the Intelligence Portal and Analyst's Notebook Premium. However, different sources and different data imply different approaches to writing data acquisition projects: v If your organization has an existing store of analysis-ready data, you can use data acquisition to reproduce that data in the Analysis Repository. For most purposes, loading the data into the repository is a one-off process, although later manipulation is possible in some circumstances. v For a data store that changes relatively slowly, and that you want to retain as a separate store, there is a different approach. You can take a snapshot of the store, and use data acquisition to load its data into a parallel of the Analysis Repository that is read-only for users. v Another possibility is a data store that contains live data that you want to query on demand. (A data store that generates short-lived result subsets in response to external queries has a similar profile.) In this case, a data acquisition solution can obtain results, present items to the user, and enable analysis as though those items were repository data. v In all of the preceding approaches, the data that the user sees might be out-of-date, compared with the original data store. The final approach to data acquisition takes an item that was retrieved through another approach, and retrieves the latest version from its source. (With some data stores, you are also able to retrieve items that are related to the specified item.) For some data providers, it is appropriate to use more than one approach to data acquisition. In particular, it can be useful to combine the ability to analyze data with the ability to retrieve the most up-to-date information. Copyright IBM Corp. 2012,

6 2 Data Acquisition Guide

7 Chapter 2. Approaches to data acquisition In IBM i2 Analyze, data acquisition is about connecting to, importing, and analyzing information from data providers other than the Analysis Repository. Data acquisition projects can involve different technologies, depending on the nature of the data that is being acquired. Data acquisition is a term that i2 Analyze uses to describe a number of different approaches to accessing external data. Data load direct Data acquisition projects that use data load direct enable users to import external data into the Analysis Repository. Clients can interact with the imported data in the same way as they interact with any other data in the repository. As its name suggests, this approach is the only one that causes the addition of third-party data to the Analysis Repository without further user intervention. Data load ELP (entity, link, property) stage In data acquisition, an ELP stage behaves like an Analysis Repository that is read-only for i2 Analyze users. An administrator can use data acquisition to load data into an ELP stage, allowing clients to use that data as they would use the repository. The data in an ELP stage is not mixed with repository data. However, clients can choose to add items from an ELP stage to the Analysis Repository. Data access on-demand Data acquisition projects that use data access on-demand enable users of the Intelligence Portal to search and retrieve data from third-party providers. These facilities closely match the standard Analysis Repository operations, and the retrieved data is displayed in the Intelligence Portal. A data access on-demand solution can present functionality to users through the standard Intelligence Portal user interface, or through a custom user interface that is integrated into the Intelligence Portal. Data that clients retrieve through data access on-demand is not automatically added to the Analysis Repository. As with an ELP stage, however, users can add it manually when required. External data retrieval A data acquisition project that uses external data retrieval enables a live link with third-party data. The approach allows simple retrieval of data (or related data) that was already identified through one of the other approaches. It is legitimate to implement external data retrieval alongside one of the other approaches on the same data provider. External data retrieval presents users with the latest information from the original provider of a piece of data. The Analysis Repository is not updated with new information unless the user opts to add it. Copyright IBM Corp. 2012,

8 Data load direct diagram In a data load direct solution, calls to the createitems and updateitems methods of the bulk data loader add external data to the Analysis Repository. Calls to deleteitems and purgeitems have the same effect on the repository as their namesakes on the command and purge services. The diagram illustrates how a data load direct solution can change the contents of the Analysis Repository: 1. If appropriate, the Analysis Repository loader interacts with the XML converter to create the items to be added or updated. 2. The Analysis Repository loader uses the command generator to create commands that drive the i2 Analyze command service. 3. The loader passes the generated commands to the Java client, which is a standard i2 Analyze extension point. 4 Data Acquisition Guide

9 4. The Java client calls the standard command or purge service to create, update, delete, or purge items in the Analysis Repository. Data load direct diagram In a data load direct solution, calls to the createitems and updateitems methods of the bulk data loader add external data to the Analysis Repository. Calls to deleteitems and purgeitems have the same effect on the repository as their namesakes on the command and purge services. The diagram illustrates how a data load direct solution can change the contents of the Analysis Repository: 1. If appropriate, the Analysis Repository loader interacts with the XML converter to create the items to be added or updated. 2. The Analysis Repository loader uses the command generator to create commands that drive the write side of the Intelligence Analysis Platform. Chapter 2. Approaches to data acquisition 5

10 Data load ELP stage 3. The loader passes the generated commands to the Java client, which is a standard Intelligence Analysis Platform extension point. 4. The Java client calls the standard command or purge service to create, update, delete, or purge items in the Analysis Repository. Data load ELP stage is one of the approaches for making data from a provider outside the Analysis Repository available in the Intelligence Portal. The approach involves loading data from the external provider into a semi-permanent store that has the same structure as the Analysis Repository. In a deployment of i2 Analyze, an ELP (entity, link, property) stage is a data store that is read-only for users. The data in an ELP stage supports the same searching and filtering operations as the data in the Analysis Repository. The purpose of an ELP stage is to present a large volume of relatively static information for investigation, without adding it to the Analysis Repository. A data load ELP stage solution uses the same mechanism for converting and editing external data that the other approaches to data acquisition use. You must provide data to the loader as appropriately formatted XML. The loader converts the XML into i2 Analyze items, editing them to add extra information if necessary. After the data loader creates items, it uses an event generator to turn the items into events, and places the events onto a message queue. The ELP stage retrieves the events from the queue, and reconstitutes the items that it stores from the information that the events contain. Data load ELP stage should not be added to new deployments. The following information should be used for existing data load ELP stage deployments. 6 Data Acquisition Guide

11 Data load ELP stage diagram In a data load ELP stage solution, calls to the loader result in the addition of events to the message queue. An ELP stage receives the events, and then creates or modifies the items that it stores in accordance with the methods that were called. The diagram illustrates how a data load ELP stage solution manages changes to its own data store: 1. If appropriate, the ELP stage loader interacts with the XML converter to create the items to be added or updated in the ELP stage. 2. The ELP stage loader uses the event generator to create the events that drive an ELP stage in an i2 Analyze deployment. 3. The loader places the generated events in the standard i2 Analyze message queue. 4. The ELP stage receives the events from the queue, and uses their contents to populate its data store by creating, updating, deleting, or purging items. Chapter 2. Approaches to data acquisition 7

12 Data access on-demand Data access on-demand is one of the approaches for making data from a provider outside the Analysis Repository available in i2 Analyze. The approach usually involves selecting and then analyzing the data in a subset of a larger body of information. Data access on-demand solutions are appropriate in situations where the external provider has its own, reasonably rich, search support. A typical solution uses the external provider to create the subset, and then enables analysis on the subset contents. Usually, the query to the external provider takes place when the user requests the data (rather than preemptively, for example). A data access-on demand solution has its own set of services. The services can be deployed on the same server as the Analysis Repository, or a separate server. The precise structure of the solution depends on the relationship between the data provider and the subsets that i2 Analyze works with. In general, however, there are two phases: subset creation, and subset exploration. Solution types Conceptually, a data access on-demand solution for external data involves a subset creation phase and a subset exploration phase. In practice, you might be able to meet the needs of your client or the data provider better by partially or completely merging those phases. 8 Data Acquisition Guide

13 The following diagram illustrates the mechanisms for three subset creation options: using a direct query (A), exported files (B), or a creation service (C). v The "A" interactions involve a subset creation service (SCS) and a subset exploration service (SES). In this scenario, however, the SCS does not actually create a subset in response to a query. Instead, the SCS returns information about the query back to the client (A1), which forwards it to the SES. The SES then uses a locator to query the data provider directly (A2), or through an adapter (A3). Note: To integrate a data acquisition solution with the standard Intelligence Portal user interface for searching, you must implement an SES and an SCS. v The "B" interactions involve an SES, but not an SCS. In this scenario, the data acquisition solution includes a custom user interface component that queries an external data provider, resulting in the creation of an exported subset. The client passes information about the subset to the SES, and the SES tells a locator where to find that subset. v The "C" interactions show a request from the Intelligence Portal to an SCS to query the external data provider. The SCS creates the subset and returns a token that describes the location of the subset to the client. The interaction between the client and the SES then proceeds in the same way as the "B" interactions. When the client calls the SES for the first time, the SES populates the item cache that it uses to respond to future requests. The key to this operation is the information in the token that the client passes to the service, which the service then passes to the locator. Chapter 2. Approaches to data acquisition 9

14 The locator uses the information in the token to access the appropriate subset, or query the external data provider. After that, the following sequence takes place: 1. The locator retrieves data from the subset (or provider), and returns an XML representation of the data to the connector. 2. The connector uses the converter and the editor to turn the data from the subset into i2 Analyze items. 3. The connector returns the items to the SES, which creates an indexed cache of those items. 4. Depending on which method the client called, the SES returns a modified identifier, item data from the cache, or both to the client. If the SES receives further requests that specify the same subset, it uses the item cache to respond to them. If the client makes a new query to the data provider, the result is a new subset and a second item cache. By default, all item caches remain available for 30 minutes from the time when they were last accessed by a client. External data subset creation The subset creation phase of a data access on-demand solution ends in the existence of some data that a subset exploration service can consume. How the subset creation phase begins, and what it involves, depend on how the external provider delivers its data. In data acquisition, an external data subset is an XML representation of some information that was drawn from an external data provider. In a data access on-demand project, there are three main ways in which a subset can be created: v A subset exploration service can query an external data provider directly. In this case, the subset is transient, and the exploration service is responsible for all parts of the process. v An external data provider can export XML data to a known location, in response to a user request. The subset exploration service is then instructed to look for subsets in that location. v A subset creation service can retrieve data from an external provider, and store an XML representation of the data. The caller of the creation service receives a token that contains the subset location, which is later passed to the exploration service. Manipulating the XML data from an external provider into i2 Analyze items is also a two-phase process. The data is first transformed to a compatible XML format, and then converted into items. In some projects, the XML in a subset is not in a compatible format, and the exploration service has more work to do as a result. The implementation of the exploration service is therefore tightly bound to the nature of the creation phase. 10 Data Acquisition Guide

15 Query results as data subsets In some data access on-demand projects, there is never a physical subset of data from the external provider. Instead, the subset exploration service queries the provider directly, and uses the results as the "subset" that is presented to users. Note: It is relatively unusual for an external provider to have built-in support for the rich querying that a data access on-demand solution can request. For example, it is rare to find an external provider that supports "expand" or "network search" operations in a form that the platform expects. As an alternative, consider one of the other data access on-demand strategies, and then implement an external data retrieval service against the same provider. Depending on the intended user interface, a solution in which the exploration service queries the provider directly might or might not include a subset creation service. By implementing a creation service in these circumstances, you allow users to run searches from the Intelligence Portal. The actual query against the provider still takes place in the exploration service. Exported files as data subsets One way to create the subset for a data access on-demand project is to have the external provider export data directly to a file. An exploration service then reads the file and processes the data that it contains so that users can run further queries against it. A typical model for a project that uses exported file subsets is one where the external provider has an existing user interface for running queries. Each query that a user runs creates a separate subset, and the exploration service makes the results of all the queries available in i2 Analyze. In some projects, the external provider can generate the exported file in the format that is required for conversion to i2 Analyze items. In other projects, the subset exploration service is responsible for transforming and converting the exported data. Chapter 2. Approaches to data acquisition 11

16 Data subset creation service In data acquisition, a subset creation service can allow users to run initial searches against an external provider through the same interface as Intelligence Portal searches. On completion, the creation service returns a token that the portal can use in subsequent calls to a subset exploration service. A successful call to a subset creation service can result in the creation (or the replacement) of an external data subset. However, the detail of the interaction between the creation service and the external data provider is highly dependent on the nature of the provider. For example, the Intelligence Portal might use a subset creation service to query for data that matches a particular text string. In response, the portal receives a token that it can use to identify the created subset in future operations. Later, the Intelligence Portal can use a subset exploration service to retrieve data from the subset in the form of i2 Analyze items. From the perspective of the Intelligence Portal, a subset creation service has a web service endpoint that provides typical i2 Analyze search methods. The portal can use a subset creation service to call Search, DumbbellSearch, NetworkSearch, FullTextSearch, and Expand against an external data source. For a developer who is implementing an external data subset creation service to perform subset creation, client requests are calls to the IExternalDataSubsetCreationService interface. The requests must be mapped to queries against the external data provider. 12 Data Acquisition Guide

17 External data subset exploration In a data acquisition solution that uses data access on-demand, the external data subset exploration service is equivalent to the standard search service. The subset exploration service enables clients to browse into, query, and expand against data that is retrieved from an external data provider. From the perspective of the client, the subset exploration service has a web service endpoint that provides typical i2 Analyze search methods. However, these methods also allow the client to identify the subset against which the searches take place. Clients use the subset exploration service to call Search, DumbbellSearch, NetworkSearch, Expand, and Retrieve against an external data subset. For a developer who is implementing an external data subset exploration service, client requests are calls to the IExternalDataSubsetExplorationService interface. The actions of the exploration service when it receives the first client call depend on the presence and nature of any preceding subset creation phase. 1. Receive a client request to search within a subset of data from an external provider, which is identified through a token. 2. Use the specified token to "locate" the subset. Depending on the nature of the data access on-demand project, this action takes one of three forms: v Execute a query against an external data provider. v Load files that were earlier exported from an external data provider. v Access a subset that was created by a subset creation service. 3. Optional: Convert the contents of the located subset into an XML representation of the data. (The subset might already be in XML format.) 4. Optional: Transform the XML contents of the subset to a format that is compatible with i2 Analyze. (The XML might already be in platform-compatible format.) 5. Convert the platform-compatible XML into i2 Analyze items. 6. Populate an in-memory cache with the converted items, which now represent the query results. 7. Execute the search that the client originally requested against the items in the cache, and return results to the client. The support that IBM provides for data acquisition includes a number of tools that make creating data access on-demand projects easier. For example, the in-memory cache is supplied by default, and there are implementations of the code for location and transformation that work in several common situations. Chapter 2. Approaches to data acquisition 13

18 Subset exploration service diagram The first successful call to a subset exploration service eventually results in the initialization of an item cache from an external data subset. For a particular project, the detail of initialization lies in the interaction between a data connector, a subset locator, and an XML converter. Note: If a client does not initialize the item cache explicitly before it requests data, then initialization takes place implicitly during the first request. 14 Data Acquisition Guide

19 The diagram reflects the variety of choices available in a subset exploration service, and especially in the behavior of the subset locator. v In some deployments, a call to the locator must trigger the creation of an external data subset. In other deployments, the subset exists when the locator is called. The locator can use the identifier that it receives to determine its behavior. v Some subsets contain data that was not retrieved in XML format from the external provider. In these cases, the locator is responsible for converting the data to XML so that other parts of the service can interact with it. v Some subsets contain XML that the locator must transform to be compatible with the exploration service code for creating i2 Analyze items. In other subsets, the data is already in this form. Regardless of the process detail, the XML data that the locator returns to the connector must be in platform-compatible format. From that point on, the behavior of the exploration service is common to all projects. External data connector The role of the external data connector in a data access on-demand project is to coordinate two processes. The connector manages retrieving data through an external data subset locator, and converting that data into items for i2 Analyze. IBM supplies an implementation of an external data connector that is called by the supplied subset exploration service. The connector has a single getsubset method that receives an external data subset identifier and then performs the following tasks: 1. The connector passes the subset identifier to the locator. The locator uses information in the identifier to find or create the subset, and then processes the subset data to return appropriately formatted XML. 2. The connector sends the returned XML to a converter that turns the XML into i2 Analyze items. In most circumstances, the supplied implementation of the IExternalDataConnector interface delegates its tasks in such a way that replacing it is an unusual requirement. External data subset locator In a data access on-demand project, an external data subset locator has two roles. It must find or create an external data subset, and then retrieve well-formed XML data from it. Like an external data connector, an external data subset locator has a single method named getsubset that takes a subset identifier argument. The identifier contains all the information that the locator needs to retrieve the appropriate XML data. The behavior of the locator depends on the identifier that it receives. The contents of a subset identifier have meaning only in the context of a complete data access on-demand solution. For example, the identifier might contain one of the following artifacts: v A query to execute against a data source, possibly through a data adapter v The location of one or more files in the file system v The name of a method to call on a "subset" object Chapter 2. Approaches to data acquisition 15

20 The nature of the data that an external data subset locator retrieves depends on the provider. The locator might need to convert the data to XML, or to transform XML into the format that the converter requires. If the data starts in XML format, then the transformation is relatively straightforward, and IBM supplies tools that assist with it. If the data does not start in XML format, then the locator is responsible for this part of the processing. IBM supplies a simple implementation of an external data subset locator that delegates all its work to a data adapter. That locator implementation assumes that it can return the XML that it receives straight to the connector; it does no extra processing for itself. External data adapter In a data access on-demand project that queries the external data provider directly, a data adapter can act as the interface that other components call. In such projects, writing a data adapter is often the easiest way to enable the retrieval of information from an external provider. An external data adapter implements the IExternalDataAdapter interface, which has methods with the same names as IExternalDataSubsetCreationService: search, dumbbellsearch, networksearch, fulltextsearch, and expand. In a particular project, the adapter might be called by either the locator or a creation service, depending on where responsibility for querying lies. Note: There is no supplied implementation of a data adapter because there is no standard external data provider. When you create a data adapter, you must call the API of the data provider appropriately from the methods that you implement. Supplied implementation IBM supplies an implementation of the IExternalDataSubsetExplorationService interface in i2 Analyze Developer Essentials. This implementation ultimately relies on your implementation of a subset locator to interpret the token that it receives. The supplied implementation has the following features: v An implementation of initializesubset that uses a data connector to convert data from a subset to i2 Analyze format, and then caches that data. v Implementations of getlatestitems, search, dumbbellsearch, networksearch, and expand that ensure initializesubset was called, and then operate on the contents of the item cache. v Implementations of retrieve and networkretrieve that support further filtering and sorting on search (or similar) results, and then return items to the caller. The external data connector that the supplied implementation of IExternalDataSubsetExplorationService uses has its own dependencies, in particular the subset locator and the XML converter. 16 Data Acquisition Guide

21 External data retrieval External data retrieval is one of the approaches for making data from a provider outside the Analysis Repository available to i2 Analyze. The approach involves querying the provider for the latest version of data that was already retrieved through one of the other approaches. An external data retrieval solution requires a single new service, and a provider that supports a basic querying function. In some circumstances, external data retrieval is the only way to access a provider directly from i2 Analyze. When you retrieve data with one of the other approaches to data acquisition, the cards on each item in the data contain provenance information. This information connects the retrieved data with its source in the external data provider. Sometimes, the information can also connect the retrieved data to its origin in another data source. In the diagram, the subset exploration service (from a data access on-demand solution) retrieves data from a data warehouse. The data warehouse was originally populated from another pair of data sources. The cards on the items that the exploration service retrieves contain provenance information about the original source of their data. A client can use the provenance information to query the original source for the latest version of the data, through an external data retrieval service. Like all data acquisition solutions, an external data retrieval service converts the data from its format in the provider to i2 Analyze format. To enable an external data retrieval service, the provider must support retrieval of an individual "item", given some identifying information. The service is more successful if the provider can also retrieve "items" that are related to the identifying information. Chapter 2. Approaches to data acquisition 17

22 18 Data Acquisition Guide

23 Chapter 3. Data conversion One of the problems that any data acquisition project must solve is that of data conversion. Regardless of the provider, the project must be able to convert external data into a format that i2 Analyze can understand. To help with the conversion of external data, projects use a pair of standard components: an XML converter, and an item editor. To benefit most from the support available in data acquisition, projects can convert from the native format of the external data in three steps: 1. Arrange for data that is entering the conversion process to be in XML format. 2. Convert the external data in XML format to i2 Analyze items. 3. Add information to the items that the platform requires, but which the provider did not contain. Conversion takes place at different times in the different data acquisition approaches. In data load direct (and ELP stage), the input must be in XML format, and the loader starts by converting the XML into items. In data access on-demand and external data retrieval, all of the conversion happens between the query to the provider and the creation of the cache. Copyright IBM Corp. 2012,

24 Conversion of external data to XML In any data acquisition solution, the XML converter requires the XML input that it receives to be in a specific format. The XML input must validate against an XML schema that is derived from the i2 Analyze schema of the target deployment. It is your responsibility to ensure or arrange that the data provider can supply data in XML format. It is also your responsibility to transform the XML from the provider to the target structure. However, IBM does assist with the creation of the XML schema that defines the target structure. To generate an XML schema from the i2 Analyze schema, you use the supplied schema converter. The tool has two distinct outputs: v An XSD file that contains the generated XML schema v A set of Java classes that map from the XML schema to the i2 Analyze schema. To transform the XML from the provider to the XML that the converter requires, you must create and apply a mapping from one to the other. There are several third-party applications that can assist with this task, through a range of different technologies. XML schema definitions It is not straightforward to convert arbitrary XML data into a form suitable for i2 Analyze. Developer Essentials makes the process easier by providing a tool for expressing the i2 Analyze schema as an XSD file. With a suitable schema definition, it is possible to transform records from the data source into a form that is compatible with i2 Analyze. During development of a data acquisition solution, you use the schema converter to create a target for the XML mapping process. To run the tool, issue the following command at a command prompt. setup -t generatemappingjar -x i2analyzeschema -z XsdFiles -o JarFiles i2analyzeschema is the path to an existing i2 Analyze schema file. XsdFiles and JarFiles are the paths to directories where the command saves its output. Note: The JAR files that the schema converter creates alongside the XSD files are involved later in the development process. The XML converter uses the classes in the JAR file to convert from the transformed XML to i2 Analyze items. The generated XML schema definition contains a flattened representation of the i2 Analyze schema. All of the item types and property group types in the schema are defined as complex types in the XML schema definition. All of the property types in the i2 Analyze schema are defined as elements within the complex types. To use the support in Developer Essentials, all XML data from the data source must be transformed to match the XML schema definition. After transformation, the data is ready to be returned from the external data adapter. 20 Data Acquisition Guide

25 XSD reference The XSD files that are generated from an i2 Analyze schema vary depending on the schema contents. However, the files all have the same basic structure. This example represents a syntactically complete (but functionally useless) XSD file, as generated by the schema converter. The following facts affect the structure of the file: v The original i2 Analyze schema contains a single link type named Employment. v The Employment link type contains two property types, and a property group type named EmploymentDetails. v The EmploymentDetails property group type contains four property types. <?xml version="1.0" encoding="utf-8" standalone="yes"?> <xs:schema version="1.0" xmlns:xs=" <xs:element name="queryresult" type="queryresult"/> <xs:complextype name="employment"> <xs:all> <xs:element name="_1_itemid" type="xs:string"/> <xs:element name="_2_fromendid" type="xs:string"/> <xs:element name="_3_toendid" type="xs:string"/> <xs:element name="_4_linkdirection" type="xs:string"/> <xs:element name="_5_linkstrength" type="xs:string"/> <xs:element name="employment_additionalinformation" type="xs:string" minoccurs="0"/> <xs:element name="employment_employmentdetails" type="employment_employmentdetails" minoccurs="0"/> <xs:element name="employment_uniquereference" type="xs:string" minoccurs="0"/> </xs:all> </xs:complextype> <xs:complextype name="employment_employmentdetails"> <xs:all> <xs:element name="employment_employmentenddate" type="xs:datetime" minoccurs="0"/> <xs:element name="employment_employmentstartdate" type="xs:datetime" minoccurs="0"/> <xs:element name="employment_employmenttype" type="xs:string" minoccurs="0"/> <xs:element name="employment_role" type="xs:string" minoccurs="0"/> </xs:all> </xs:complextype> <xs:complextype name="queryresult"> <xs:all> <xs:element name="employments"> <xs:complextype> <xs:sequence> <xs:element name="employment" type="employment" minoccurs="0" maxoccurs="unbounded"/> </xs:sequence> </xs:complextype> </xs:element> </xs:all> </xs:complextype> </xs:schema> Chapter 3. Data conversion 21

26 Implications The following features of the example are common to all XSD files generated by the schema converter: v There is always an element that is named QueryResult, with a type that is also named QueryResult. v The QueryResult type is a container for all complextypes that represent item types. v When a complextype represents an item type (rather than a property group type), the names of its first few elements begin with numbers. These numbered elements represent the mandatory property types that all item types must contain. v The elements that represent the property types and property group types of an item type have names that begin with the name of the item type. v The elements that represent the property types of a property group type have names that begin with the name of the containing item type. XML content that conforms to any schema generated with the schema converter has a single root element named QueryResult. The root element can contain any number of elements that are named after item types in the original i2 Analyze schema. XML data that leaves the external data adapter must validate against the generated XSD file. Conversion of XML to i2 Analyze items In a data acquisition project, the XML converter receives data from an external provider that was transformed into platform-compatible XML format. The converter is then responsible for turning the input into entities and links for i2 Analyze. The supplied components of Developer Essentials include support for converting the XML that they receive into i2 Analyze items. To perform the conversion, the supplied components use the Java classes that the schema converter generates. To take advantage of this support, you must ensure that the JAR file that contains the classes is on the class path when you deploy the solution. Modification of i2 Analyze items Before the XML converter in a data acquisition project returns items to a caller, it sends those items to an item editor. The item editor provides an opportunity to augment or modify the content of the items that the XML converter created. Data from an external provider does not always contain all the information that a typical item in i2 Analyze requires. In particular, external providers often provide no values for some or all of the following content: v Information about the originator of the data, which would normally be displayed in the Source information section of a form in the Intelligence Portal v Grading information for each of the property values v The security settings that determine what access and grant levels users receive if items are added to the Analysis Repository 22 Data Acquisition Guide

27 The XML converter calls your implementation of the IExternalDataItemEditor interface, where your code has full access to the contents of an item. In the edit method, you can supply any of the content from the list. Note: You can also use the edit method to modify values that came from the external data provider. After the item editor, there is no further opportunity to change the results that the XML converter returns. Items at this stage are likely to be presented to users, and might be added to the Analysis Repository. Chapter 3. Data conversion 23

28 24 Data Acquisition Guide

29 Chapter 4. Development process for data access on-demand Developing a subset exploration service with i2 Analyze Developer Essentials involves writing some interface implementations, and calling or generating some others. Schematically, a subset exploration service for i2 Analyze that was created with Developer Essentials looks like this diagram: The process of using Developer Essentials to develop a subset exploration service involves the following stages: v Determine which of the standard i2 Analyze operations ("search", "dumbbell search", "network search", "expand", "get latest items") the data source supports. v Implement the pieces of the external data adapter that query the data source on behalf of the client. v Execute the schema converter against the i2 Analyze schema to create an XML schema definition and JAXB classes. v Implement the pieces of the external data adapter that transform retrieved XML data to match the XML schema. v Add the JAXB classes to the project so that the XML converter can use them to create i2 Analyze items. v Implement an external data item editor to add source, security, and grading information to the created items. v Build the project and deploy the solution to an instance of i2 Analyze. Copyright IBM Corp. 2012,

30 Supported operations The tools and code libraries that IBM supplies for data access on-demand provide maximum support when the external data source can return XML results. However, the framework is still useful even when XML output is not available. A data source that is further from the assumptions just increases the amount of custom code that you must write. i2 Analyze provides several ways for clients to request information from an external data source. An external data subset exploration service must support some, but not necessarily all, of these ways. Clients can perform five different operations on an external data source through an external data subset exploration service: GetLatestItems Returns the items with specified identifiers. Search Runs a search for items with data that matches some specified criteria, and returns a summary of the search results to the client. DumbbellSearch Runs a search for entity-link-entity structures with data that matches some specified criteria, and returns a summary of the search results to the client. NetworkSearch Runs a search for complex structures of items with data that matches some specified criteria, and returns a summary of the search results to the client. Expand Returns links and entities that are connected to the entities with specified identifiers. In i2 Analyze Developer Essentials, the supplied implementation of the IExternalDataSubsetExplorationService interface has built-in support for "get latest items" operations. However, the code delegates requests for the other operations to your implementation of IExternalDataAdapter. Before you can implement IExternalDataAdapter, you must determine which of "search", "dumbbell search", "network search", and "expand" the data source supports. v To support "search", the data source must at least be able to retrieve records with content that matches a string. v To support "dumbbell search", the data source must be able to retrieve simply related records, where each record in the relationship fulfills some criteria. v To support "network search", the data source must be able to retrieve collections of related records, where each record in the collection fulfills some criteria. v To support "expand", the data source must be able to retrieve records that are conceptually connected to a specific, identified record. 26 Data Acquisition Guide

31 Data source queries Data transformation To retrieve information from an external source in a data access on-demand solution, you must map from calls to IExternalDataAdapter methods, to calls to the data source API. In a complete implementation of an external data adapter method, your code must perform the following tasks: 1. Interpret the request from the client 2. Perform queries against the data source 3. Transform the results to a prescribed format 4. Return the transformed results to the caller Note: If you decide not to support one or more of the operations, then it is valid to throw a standard Java UnsupportedOperationException from the method implementation. You must also configure a settings file appropriately during deployment. Performing a data source query in response to a client request is the part of the process for which i2 Analyze Developer Essentials provides the least assistance. The wide variety of potential data sources means that there is no approach to mapping that works in the general case. To assist with later tasks, Developer Essentials places requirements on the data that you retrieve from the source. The output of the querying phase must be in XML format, either because the data source provides it that way, or because you generated it. The data from the source is then ready to be transformed so that it corresponds with the i2 Analyze schema. It is not strictly necessary for the XML retrieved from the data source to use the same schema for the results of every supported operation. If the schema is consistent between operations, however, subsequent stages of implementing the external data adapter become easier. A data access on-demand solution must transform the XML data from an external data adapter to match the schema definition that is generated from the i2 Analyze schema. This XML data must also be contained in the return values of the external data adapter methods. The difficulty of transforming XML to the structure defined in the generated XSD file ("QueryResult format") depends in part on the consistency of the XML. If the data source returns data in the same basic structure for all possible queries, then you can use the same transformation in all cases. The difficulty of creating the transformation depends on the complexity of the returned XML, and the similarity of the XML to the target structure. You might choose one of several approaches, depending on your familiarity with the technologies involved: v Create your own XSLT translation from one form to the other v Write XML DOM manipulation code as part of the implementation v Use a third-party tool to construct the mapping and generate code Chapter 4. Development process for data access on-demand 27

32 Creation and deployment The methods of IExternalDataAdapter all return the transformed data in Source objects that must contain a correctly formed QueryResult element. The size and precise contents of the element, however, are for you to decide: v The question of how many QueryResult elements to create depends on the size of the result set and the characteristics of the data source. v There is no requirement for a QueryResult to be self-contained. A link and its ends do not have to be represented in the same element. The supplied external data connector reassembles separate QueryResult elements into a single result before it performs any further processing. Therefore, you can base your implementation decisions solely on the throughput of the external data adapter. Your choice does not affect the rest of the external data service. When the components of an external data service are all written or assembled, you can build the service and deploy it to a running instance of i2 Analyze. When the external data service is successfully deployed, the Intelligence Portal automatically detects the new service and determines its capabilities. The external data source becomes selectable by name in the user interface of the Intelligence Portal. 28 Data Acquisition Guide

33 Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Copyright IBM Corp. 2012,

34 Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact: IBM United Kingdom Limited Hursley House Hursley Park Winchester, Hants, SO21 2JN UK Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of a fee. The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or any equivalent agreement between us. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. 30 Data Acquisition Guide