Essential Import Service

Transcription

1 Essential Import Service Interface Specification Version 2.0 March 2017

2 Contents Operation of the service Default callback listener Importing Microsoft Excel and CSV documents Operation of the service Spreadsheets and CSVs Helper components HTML request form Service requests Service responses Response to initial request Callback response Extensions for Essential Intelligence Platform Importing XML documents Operation of the service XML Imports Helper components HTML request form Service requests Service responses Response to initial request Callback response EAS

3 Operation of the service Import Activities, Import Specifications (the semantic mapping of source data to the Essential Meta Model) and Target Environments must all be configured in the Import Utility web-based, graphical user interface. The service interface provides a REST-based HTTP approach for invoking imports by selecting the Import Activity and Target Environment in a request that includes the source data to import. Content type (encoding) of multipart/form-data is required for compressed files and when using HTML test page W3C Default content type (application/x-www-form-urlencoded) is also supported for uncompressed CSV files Supported source data formats: Excel binary.xls documents Excel Workbook.XLSX documents CSV files XML documents CSV files are converted to Excel documents during the import process. Import Specifications are defined in terms of Worksheets and Columns, neither of which are available in a CSV file. To define an Import Specification for a CSV file, import an example CSV into Excel and save it as one of the Excel binary formats, e.g..xlsx. During this process, if you re-name the default Worksheet for the CSV (note that CSV files can only be represented as a single worksheet), be sure to use the 'sheet' parameter in the request to specify this worksheet name. An import activity may take several minutes to complete. Therefore, the service supports a long-running process by accepting a callback URL to which final success / failure messages are sent. On request, the parameters are evaluated and any immediate problems are reported (e.g. invalid import activity). The service responds with an HTTP 201 (Accepted) message if all the request parameters are found to be valid and the request has been accepted for processing EAS

4 Default Callback Listener When an import is completed, the service invokes the callback URL that you can specify as part of the request A default callback listener is included as part of the Import Utility, which simply reports the messages to standard out Normally to be found in the Tomcat catalina.log file. This listener is available on the /callback URI of the Import Utility, e.g. localhost:8080/essential_import_utility/callback See the Callback Response documentation for each of the services for more information about the schema and content of these responses EAS

5 Essential Import Service Microsoft Excel and CSV Imports

6 Operation of the service Spreadsheets and CSVs Once the Import Utility WAR file has been deployed, the Import Utility is configured in the usual way, including defining the Import Activities, Target Environments etc. via the web application GUI. With the source and target environments defined and Import Specifications configured, use the service interface to invoke a run of an Import Activity, selecting it by name. When it has completed, the results of the import are available from the Manage History section of the selected Import Activity GUI. Remember to enable logging when defining the new Import Activity! To perform a test or a live execute of an import, simply select the desired target environment in the service request. Test environments are returned to their original start state once the import has completed. The updated, resulting state can be downloaded from the logs, via Manage History EAS

7 HTML Request Form A simple HTML page servicerequestform.html is available for testing the operation of the Import Service and to demonstrate the parameters of the service request. This can be found in the root of the web application and at the URL using a browser, e.g. localhost:8080/essential_import_utility/servicerequestform.html EAS

8 Service requests Requests are made using HTTP POST method with the following parameters to URI: /autoimport Parameter Interpretation activity environment uid pwd publish source response sheet batchid The name of the import activity to use for the import The name of the target repository environment in which to import The user ID for the client requestor. This must be a valid user account defined to the import utility The password for the specified user account. A Boolean flag (yes / no) specifying whether the Import Utility should publish the resulting repository to the target Essential Viewer environment when the import has completed successfully. Valid values are yes or no with a default of no The source Excel binary file (.xls or.xlsx) or CSV to be imported Optional URL on the consumer side providing a callback for success / failure messages Optional name of the worksheet to use when converting a CSV to Excel. Default value is Sheet1 Optional identifier that groups a sequence of requests into a batch. A failure during import to any of the requests with the same batch ID will stop the processing of any queued requests with that ID and will prevent the resulting repository being published EAS

9 Response to initial request A response to the request POST method is returned immediately using an HTTP status code and corresponding message (where required) Status Code Message Interpretation 202 n/a All specified request parameters are valid and the source content has been queued for import 403 Invalid username or password The specified username or password was not correct : Invalid Import Activity The specified Import Activity could not be found : Invalid Target Environment The specified Target Environment could not be found : Invalid Worksheet The Import Activity could not find the named worksheet in source 405 GET Method is not available Requests that use the GET method are not supported and will be rejected. All requests must use the POST method : Invalid source content The supplied source content was not in a supported format : Import Job refused The queue of import jobs is full and the request has been refused by the service EAS

10 Callback Response Provide a URL which is ready to receive HTTP POST XML document with following structure of tags in a request parameter called callback_message (URL encoded form post parameter) import_callback import_job status o value o message timestamp import_activity target_environment source_content batch_id publish Requested status o value o message EAS

11 Callback response message details The import_callback XML document provides feedback on the operation of an import as follows: Tag Values / format Semantics import_callback n/a The containing tag for the document import_job n/a Containing tag for details of an import status n/a Containing tag for status content status_value success exception Value to indicate whether the import succeeded or encountered an exception status_message Text When value=exception, details of the exception that was encountered timestamp Text date and time Date and time of the completion of the import import_activity Text The name of the Import Activity that was used target_environment Text The name of the Target Environment that was used source_content Text The name of the source content (file) that was imported batch_id Text Optional. If specified on the request, the batch identifier of the import job Publish n/a The containing tag for the details of a publish of import results Requested yes no Indicator showing whether publishing was requested status n/a Containing tag for status content status_value success exception Value to indicate whether the publish succeeded or encountered an exception status_message Text When value=exception, details of the exception that was encountered EAS

12 Example XML <?xml version="1.0" encoding="utf-8"?> <import_callback xmlns=" xmlns:xsi=" xsi:schemalocation=" import_callback.xsd"> <import_job> <status> <value>success</value> <message>import completed successfully</message> </status> <timestamp> t10:10:14</timestamp> <import_activity>example Import Activity</import_activity> <target_environment>example Live Repository</target_environment> <source_content>spreadsheet_import_v12.xlsx</source_content> <batch_id>1001</batch_id> </import_job> <publish> <requested>yes</requested> <status> <value>success</value> <message>publish completed successfully</message> </status> </publish> </import_callback> EAS

13 Callback message XML Schema See import_callback.xsd EAS

14 Extensions for Essential Intelligence Platform A separate service interface is provided for importing Microsoft Excel and CSV documents to an Essential Repository that is hosted on the Essential Intelligence Platform (EIP) Cloud environment In the EIP environment imports are defined in the same way as in open-source mode. Invoking the service interface, however, operates differently by sending a Data Update Pack (DUP) from the Import Utility to the selected EIP repository and invoking the EIP to apply that update pack Additional security is applied to such requests Additional parameters are required to capture the user account that should be used by the service to execute the DUP on the cloud EIP environment: New service endpoint Additional username and password parameters to authorise the request on the EIP cloud environment A simple HTML page servicerequestdupform.html is available for testing the operation of the Import Service and to demonstrate the parameters of the service request. This can be found in the root of the web application and at the URL using a browser, e.g. localhost:8080/essential_import_utility/servicerequestdupform.html EAS

15 EIP Mode Excel Imports Requests are made using HTTP POST method with the following parameters to URI: /autoimportdup Parameter Interpretation activity environment uid pwd publish source tenantuid tenantpwd response sheet batchid The name of the import activity to use for the import The name of the target repository environment in which to import The user ID for the client requestor. This must be a valid user account, within the tenant that has sufficient rights to use the import utility The password for the specified user account. A Boolean flag (yes / no) specifying whether the Import Utility should publish the resulting repository to the target Essential Viewer environment when the import has completed successfully. Valid values are yes or no with a default of no The source Excel binary file (.xls or.xlsx) or CSV to be imported The user ID of a valid account on the target EIP repository (and tenant). This can be (but does not have to) be the same account as that used to access the Import Service (the value of the UID parameter) The password of the valid tenant account for accessing the target EIP repository Optional URL on the consumer side providing a callback for success / failure messages Optional name of the worksheet to use when converting a CSV to Excel. Default value is Sheet1 Optional identifier that groups a sequence of requests into a batch. A failure during import to any of the requests with the same batch ID will stop the processing of any queued requests with that ID and will prevent the resulting repository being published EAS

16 Essential Import Service XML Imports

17 Operation of the service XML imports The XML import operates on the same concepts as the spreadsheet import services. However, import specifications are defined using XSL stylesheets, outside of the Essential Import Utility and there is no Import Activity development environment as there is for spreadsheets An XML import is defined by creating a new import activity on the Manage Imports page of the Import Utility web application, specifying the Import Activity Type as Automated XML Import The Source Repository that should be used as the basis for the import and containing all the Classes that are referred to in the Import Specification is selected in the same way as for a spreadsheet import. The Source Spreadsheet field is left empty and an XML transform is selected. This transform can take one of two forms but either must be defined outside of the import utility, e.g. using XML development tools: A simple, stand-alone XSL file (XML Stylesheet) that defines the transform from the source XML to a set of Python API calls to update the target repository A Zip archive that contains one or more XSL files to transform the source XML into a set of Python calls. This Zip archive can also contain Python libraries, e.g. standardfunctions.py to aid with the development of the import specification A Python script variable, includepath is available to aid with import statements in the Python (e.g. to execute library script files), providing a local-reference to the import pack for finding files, e.g.: standardfunctionsfile = includepath + "/library/standardfunctions.py When a Zip file is specified as the XML Transform, the Start File must be defined to specify which XSL file in the ZIP the import utility must use first. Ensure that the Log Imports option is selected to produce detailed logging of each operation of the XML import activities.

18 Operation of the service XML Imports The Open... button of the Manage Import Activities user interface presents the activity log for this XML import activity. From here a full trace of each operation of the XML import service for the selected Import Activity is available: Source XML document that was delivered in the service request, Import transformation (XSL) used by the import activity Generated import script (Python) Set of messages reported by the Import Activity The resulting repository with the import applied 2017 EAS 18

19 HTML Request Form A simple HTML page servicerequestxmlform.html is available for testing the operation of the Import Service and to demonstrate the parameters of the service request. This can be found in the root of the web application and at the URL using a browser, e.g. localhost:8080/essential_import_utility/servicerequestxmlform.html EAS

20 Service requests Requests are made using HTTP POST method with the following parameters to URI: /autoimportxml Parameter activity environment uid pwd publish source response Interpretation The name of the import activity to use for the import The name of the target repository environment in which to import The user ID for the client requestor. This must be a valid user account defined to the import utility The password for the specified user account. A Boolean flag (yes / no) specifying whether the Import Utility should publish the resulting repository to the target Essential Viewer environment when the import has completed successfully. Valid values are yes or no with a default of no The source XML file (.xml) to be imported Optional URL on the consumer side providing a callback for success / failure messages EAS

21 Response to initial request A response to the request POST method is returned immediately using an HTTP status code and corresponding message (where required) Status Code Message Interpretation 202 n/a All specified request parameters are valid and the source content has been queued for import 403 Invalid username or password The specified username or password was not correct : Invalid Import Activity The specified Import Activity could not be found : Invalid Target Environment The specified Target Environment could not be found. 405 GET Method is not available Requests that use the GET method are not supported and will be rejected. All requests must use the POST method : Invalid source content The supplied source content was not in a supported format : Import Job refused The queue of import jobs is full and the request has been refused by the service EAS

22 Callback Response Provide a URL which is ready to receive HTTP POST XML document with following structure of tags in a request parameter called callback_message (URL encoded form post parameter) import_callback import_job status o value o message timestamp import_activity target_environment source_content batch_id publish Requested status o value o message EAS

23 Callback response message details The import_callback XML document provides feedback on the operation of an import as follows: Tag Values / format Semantics import_callback n/a The containing tag for the document import_job n/a Containing tag for details of an import status n/a Containing tag for status content status_value success exception Value to indicate whether the import succeeded or encountered an exception status_message Text When value=exception, details of the exception that was encountered timestamp Text date and time Date and time of the completion of the import import_activity Text The name of the Import Activity that was used target_environment Text The name of the Target Environment that was used source_content Text The name of the source XML content (file) that was imported. This is available from the Import Activity Log for each operation of the service Publish n/a The containing tag for the details of a publish of import results Requested yes no Indicator showing whether publishing was requested status n/a Containing tag for status content status_value success exception Value to indicate whether the publish succeeded or encountered an exception status_message Text When value=exception, details of the exception that was encountered EAS

24 Example XML <?xml version="1.0" encoding="utf-8" standalone="yes"?> <import_callback xsi:schemalocation="service/import_callback.xsd" xmlns=" xmlns:xsi=" <import_job> <status> <value>success</value> <message>import Successful</message> </status> <timestamp> t15:55:13.072z</timestamp> <import_activity>test XML</import_activity> <target_environment>eas Test</target_environment> <source_content>/users/[...]/essentialimportutility/service/source/example Data xml</source_content> </import_job> <publish> <requested>no</requested> <status> <value>not requested</value> <message></message> </status> </publish> </import_callback> EAS

25 Callback message XML Schema See import_callback.xsd EAS

26 Essential Import Service Interface Specification