Consume XML Documents and Web-Services with SQL

Transcription

1 Schaumburg Consume XML Documents and Web-Services with SQL Birgitta Hauser / TOOLMAKER Advanced Efficiency GmbH Tel. (+49) Sasbach / Ortenau Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 2 1

2 Agenda Consuming XML Documents Annotated XML Document Decomposition XMLTABLE Table Function Accessing XML documents Converting the hierarchical structure into a relational represenatation Picking out only the relevant information DB2 XPath Syntax GET_XML_File Scalar Function Accessing (RESTFul) Web-Services RESTFul Web-Services Using http-functions for accessing web-services Combining http-functions with XMLTABLE for accessing and filtering the information provided in the web-service response Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 3 Well-formed XML Document Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 10 2

3 XML Well-formed Document Exactly 1 Root Element All other elements must be embedded within the root element XML-Declaration (XML Version and Encoding) can be specified first, before the Root Element All information is included in partly nested Elements All elements consist of an Start and End Tag (except Empty Elements) Can be nested ( sequence must be correct) Attributes can be used for describing elements more accurately Attributes must be specified within the start tag of an element Attribute values must be enclosed with double quotes ( x ) Namespaces allow universally unique element and attribute names All element, attribute, namespace names are case sensitive Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 11 XML Wellformed Document Declaration Root Element Start Tag Element End Tag Element Attribute Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 12 3

4 Decomposing XML Documents Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 13 Decomposing XML Documents Annotated XML Document Decomposition Based on XML Schemas Stored Procedures for (de)registering XML Schemas within the XML-Schema-Repository (XSR) Stored Procedure for decomposing an XML document based on schemas registered within the XSR and shredding the XML information directly into tables XMLTABLE Table Function Retrieves the content of an XML document as a result set (table) Specified within the FROM-clause of a SELECT-statement Returned data can be reduced based on additional WHERE conditions Works in conjunction with a sub-set of the XPath expressions Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 14 4

5 Annotated XML Document Decomposition Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 15 Stored Procedures for managing XML Schemas System Stored Procedures within Bibliothek SYSPROC Stored Procedures to maintain the DB2 XSR (XML Schema Repository) XSR_REGISTER XSR_ADDSCHEMADOC XSR_COMPLETE XSR_REMOVE XDBDECOMPXML Registers an XML Schema in the DB2 XML Schema Repository (DB2 XSR) for future use when validating or decomposing XML Documents Adds additional XML schema documents to an XML schema that you are in the process of registering. Completes the registration of an XML Schema in the DB2 XML Schema Repository (DB2 XSR) Removes the XML schema from the DB2 XML schema repository (DB2 XSR) Stored Procedures to decompose XML Documents Decomposes an XML document into specified DB2 objects using annotated decomposition Currently there is neither a Stored Procedure nor any other feature available that allows an XML schema to be created based on a DB2 for i table or SELECT statement Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 16 5

6 XML Schema for Annotaded Decomposition Default Schema Table Definition Table Schema Table Name Column Definition Column Name XML Element Name and data type Special XML Decomposition Annotations namespace Identifed by the db2-xdb: namespace prefix Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 17 Decompose XML Document Example SALESXML Table Definition XML Document /home/hauser/salesxxx.xml Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 21 6

7 Decompose XML Document Example Initialize Output File Decompose the XML document using the XDBDECOMPXML stored procedure based on the SALESXSD schema Display the content in SALESXML after decomposition Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 22 XMLTABLE Table Function Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 23 7

8 XMLTABLE Table Function Retrieving XML data in relational format XML Parser Reads an XML document located In an XML column in a database table In the IFS (Integrated File System) in composition with the GET_XML_FILE and XML_PARSE scalar functions (with http-functions in SYSTOOLS) returned by a RESTful WebService Hierarchical XML information is converted into a relational format Returns a table Must be specified within the FROM clause of a SELECT statement XMLTABLE Table Function can be used like any table or view <employee id="901"> <name><first>birgitta</first> <last>hauser</last></name> <department>programming </department> </employee>... additional employees id first last department 901 Birgitta Hauser Programming Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 24 XQuery and XPath Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 25 8

9 XQuery and XPath XQuery is a functional programming language designed by the World Wide Web Consortium (W3C) for querying and modifying XML data XPath is an expression language designed by the World Wide Web Consortium (W3C) for processing of XML data that conforms to the XQuery 1.0 and XPath 2.0 data model Tutorial: w3schools - DB2 XPath Sub-set of the language constructs in the XPath 2.0 recommendation used with the XMLTABLE SQL built-in table function for converting an XML value into a relational result set Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 26 XPath Syntax - Nodes Nodes XML documents are treated as trees of nodes Kinds of Nodes: element, attribute, text, namespace, processing-instruction, comment, document nodes Root Element The topmost element of the tree is called the root element Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 27 9

10 XPath Syntax - Nodes Expression Description Path Expression Result nodename / // Selects all nodes with the name "nodename" Selects from the root node Selects nodes in the document from the current node that match the selection no matter where they are bookstore /bookstore bookstore/book //book bookstore//book Selects all nodes with the name "bookstore" Selects the root element bookstore Note: If the path starts with a slash ( / ) it always represents an absolute path to an element! Selects all book elements that are children of bookstore Selects all book elements no matter where they are in the document Selects all book elements that are descendant of the bookstore element, no matter where they are under the bookstore Selects attributes //@lang Selects all attributes that are named lang Selects the current. node Selects the parent of.. the current node Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 28 XMLTABLE Basic Syntax XMLTABLE( row-xquery-expression-constant PASSING row-xquery-argument AS Identifier COLUMNS xml-table-column-definition, Def2,..., DefN) Row-XQuery-Expression-Constant Syntax: $xxx/rootelement/element1/ /ElementN $xxx: $ - mandatory xxx: Identifier (without double quotes) /RootElement/Element1/ : Start Element Row-XQuery-Argument XML column name within the specified table or CTE or Sub-Select BLOB-Locator (returned by GET_XML_FIlE function) on the XML document in the IFS Identifier Syntax: Name Name Must be specified in the Row-Xquery-Expression in upper case Must be specified as exact match in the Row-Xquery-Expression Attention: Row-Xquery-Expression and Identifier are case sensitive Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 29 10

11 XMLTABLE Basic Syntax XMLTABLE( row-xquery-expression-constant PASSING row-xquery-argument AS Identifier COLUMNS xml-table-column-definition, Def2,..., DefN) XML-Table-Column-Definition Multiple column definitions and be specified, separated by commas Syntax: ColName ColName DataTypeDefinition PATH XPathDefinition Column name in the result table cannot be qualified DataType Data type and length specification for the column in the result table CCSID for CHAR/VARCHAR not allowed DEFAULT clause for replacing NULL values can be specified Xpath Element Element name embedded in single quotes Elem1/Elem2/Element incl. path underneath the start Attibute name (must be prefixed incl. path underneath the start element Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 30 XMLTABLE Basic Syntax Example Table with XML Column Create Table EmplXML Column Document: DataType XML Insert XML data into XML Column XMLPARSE Checks and converts the XML-string into the internal XML format Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 31 11

12 XMLTABLE Basic Syntax Example XML Document - Structure Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 32 XMLTABLE - Basic Syntax Example 1 Table EmplXML must be specified it will be implcitly joined with the XMLTABLE UDTF Document: Column out of the EmplXML table containing the XML document Return only information within the Staff/Department/Employee/Name Element Column-Definition: Return First and Last Names only Column Names in the result table (=xml document/renamed) Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 33 12

13 XMLTABLE - Basic Syntax Example 2 - Nodes Return only information within the Staff/Department/Employee Element Return First and Last Name Element information located within Element Name Return , Birthday and Salary Element information located within Element Privacy Convert the Birthday data into a real date Convert the Salary data into a packed numeric values Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 34 XMLTABLE - Basic Syntax Example 3 Default Values If the XML document does not include salary information, Default value 0 is returned Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 35 13

14 XML-Table Basic Syntax Example 3 Attributes Attribute Definition: Attribute names must be prefixed with Default clauses can also be specified for Attributes Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 36 Access XML in Stream Files Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 37 14

15 GET_XML_FILE Function Returns data from a source physical file or stream file The result of the function is a BLOB-Locator Can be compared with a pointer that points to the data Can be used in SQL statements and with scalar functions like any character value Data is converted into UTF-8 Must be executed under commitment control The locator is freed when a COMMIT or ROLLBACK is performed Copy an IFS file into an XML column Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 38 Access Stream Files in composition with XMLTABLE Define IFS File to be accessed directly within the XMLTABLE function GET_XML_File: Retrieve a Locator/Pointer on the data stored in an stream file XMLPARSE: Check for a valid/well-formed XML document Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 39 15

16 Namespaces Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 40 XML Namespaces Namespaces in XML 1.0 (3 rd edition December 8 th 2009) published by W3C (World Wide Web Consortium) Definition universally unique Names For elements and attributes (Definition with xmlns:xx= URI ) Element/attribute (qualified) names are composed of: The Namespace identifier <MyNS:MyElem>Value</MyNS:MyElem> A colon (:) The element/attribute name New (XML) documents consisting of pieces from other documents even with duplicate element/attribute names can be created Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 41 16

17 XML Namespaces Are identified by a Uniform Resource Identifier (URI) URIs do not need to represent actual resources Any unique text string can serve as a namespace identifier Namespace declaration must be specified within a start tag of an element Namespace declaration Qualified element names Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 42 XMLTABLE and Namespaces Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 43 17

18 XMLTABLE XML Document with Namespaces Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 44 XMLTABLE XML Document with Namespaces Without Namespace declaration within the XMLTABLE function *: must be specified instead of the namespace within the node name Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 45 18

19 XMLTABLE with Namespaces XML Document with Namespaces Declaring the namespaces to be used with the XMLNAMESPACE scalar function All Element, Attribute and Node names must be qualified, i.e. prefixed with the appropriate namespace label Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 46 Query the Web with SQL Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 47 19

20 http Functions for DB2 for i for RESTFul Web-Services Prerequisites: 5770-SS1 DB2 PTF Group SF99701 Level 23 Java 1.6 or later (5761-JV1 Option 11, 12, 14, or 15) New http Functions http Functions added to the SYSTOOLS schema SYSTOOLS schema is NOT part of the system path/library list must be (manually) added to the SQL path/library list http Functions they are NOT subject to IBM service and support IBM maintenance for SYSTOOLS is to delete existing objects followed by the creation of the newer versions (without considering any modifications) Can be used as soon as DB2 PTF Group Level 23 is installed Java source is provided in /QIBM/ProdData/OS/SQLLIB/bin/systools_java_source.jar Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 48 http Functions for DB2 for i Scalar and Table Functions (names starting with http) Located in the SYSTOOLS schema Function Names http (Method) (DataType) (Verbose) Method: Data type: GET, PUT, POST, DELETE, HEAD Without method Method passed as parameter value of the request AND response messages CLOB (Character Large Object) / BLOB (Binary Large Object) Verbose: with Verbose: Table function with 2 columns 1st column: http header information 2nd column: response message without Verbose: Scalar Function response message only Examples: httpgetclob(), httpputblobverbose(), httpclob() Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 49 20

21 IBM i Navigator http Functions in Schema SYSTOOLS Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 50 SQL Scalar http Functions for RESTful Services - Overview User Defined Return Parameter Definition http- Function (UDF) Data Type Name Data Type Verb httpgetblob BLOB URL VarChar(2048) httpheader CLOB(10K) GET URL VarChar(2048) httpgetclob CLOB httpheader CLOB(10K) GET URL VarChar(2048) httpputblob BLOB httpheader CLOB(10K) PUT Data BLOB(5M) URL VarChar(2048) httpputclob CLOB httpheader CLOB(10K) Data CLOB(5M) PUT URL VarChar(2048) httppostblob BLOB httpheader CLOB(10K) POST Data BLOB(5M) URL VarChar(2048) httppostclob CLOB httpheader CLOB(10K) POST Data CLOB(5M) Description Retrieves a resource by a given URL vie HTTP GET. The resource is returned as binary LOB (BLOB). Retrieves a text-based resource by a given URL via HTTP GET. The resource is returned as character LOB (CLOB). Character set conversion into the database code page is performed if necessary. Creates (upload) a binary resource under a given URL using HTTP PUT. A response message is returned as binary LOB (BLOB) if issued by the server. Creates (upload) a textual resource under a given URL using HTTP PUT. A response message is returned as character LOB (CLOB) if issued by the server. Character set conversion is performed if necessary. Updates a binary resource under a given URL using HTTP POST. The response message is returned as binary LOB (BLOB). Updates a textual resource under a given URL using HTTP POST. The response message is returned as character LOB (CLOB). Character set conversion is performed if necessary Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 51 21

22 SQL Scalar http Functions for RESTful Services - Overview User Defined Return Function (UDF) Data Type httpdeleteblob BLOB httpdeleteclob CLOB httpblob httpclob BLOB CLOB Parameter Definition http- Name Data Type Verb Description URL VarChar(2048) httpheader CLOB(10K) DELETE URL httpheader URL httpmethod httpheader Data URL httpmethod httpheader Data VarChar(2048) CLOB(10K) DELETE VarChar(2048) VarChar(128) CLOB(10K) BLOB(5M) VarChar(2048) VarChar(128) CLOB(10K) BLOB(5M) any HTTP verb any HTTP verb URL VarChar(2048) httphead CLOB/XML httpheader CLOB(10K) HEAD Deletes a binary resource under the given URL using HTTP DELETE. The response message is returned as binary LOB (BLOB) if there is any. Deletes a textual resource under the given URL using HTTP DELETE. The response message is returned as character LOB (CLOB) if there is any. Character set conversion is performed if necessary. Performs an HTTP request with the provided verb. A request message can be provided as binary LOB (BLOB). A response message is returned as binary LOB (BLOB) if issued by the server. Performs an HTTP request with the provided verb. A request message can be provided as character LOB (CLOB). A response message is returned as character LOB (CLOB) if issued by the server. Character set conversion is performed if necessary. Verifies the HTTP header for the requested resource using HTTP HEAD. Just the HTTP header is returned as CLOB Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 52 SQL Scalar Functions for RESTful Services - Parameter Parameter URL Required for all functions VarChar(2048) httpheader Can be passed without value or *BLANKS CLOB(10K) Data Required for all PUT and POST functions BLOB(5M) or CLOB(5M) httpmethod PUT, POST, GET, DELETE Required for httpblob / httpclob Functions VarChar(128) Return Value Binary (BLOB) / Character (CLOB) Large Object Containing the resource representation for example as XML/JSON Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 53 22

23 Access Web-Service Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 54 Free Web Service Exchange Rates - Example Web Service European Central Bank Daily Exchange Rates Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 55 23

24 Consume Web-Services with SQL Example Exchange Rates Namespaces Access web-service with HTTPGetBLOB URL Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 58 Consume Web-Services with SQL wrapped with a View (Exchange Rates) Additional Currency Rates Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 59 24

25 Access Web-Service with embedded SQL Code Snippet Access Web-Service through an http function in conjunction with the XMLTABLE UDTF stored within the ExcRateV2 SQL View Cursor handling is performed like for any other table or view Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 60 Access Web-Service with RPG Example: Exchange Rates Access Web-Service Embedded in a SQL View RPGLE mit embedded SQL Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 61 25

26 Any Questions? Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 62 Don t Forget Your Session Surveys Sign in to the Online Session Guide ( Go to your personal schedule Click on the session that you attended Click on the Feedback Survey button located above the abstract Completing session surveys helps us plan future programming and provides feedback used in speaker awards. Thank you for your participation Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 63 26

27 References IBM i Information Center DB2 for i SQL reference IBM i Database XML Programming Accessing webservices using IBM DB2 for i HTTP UDFs and UDTFs ibm.com/partnerworld/wps/servlet/download/DownloadServlet?id=k4ixw2TAAIwiPCA$cnt&attachmentNa me=accessing_web_services_using_ibm_db2_for_i_udfs_and_udtfs.pdf&token=mtqxmdaxodmxmjmymg= =&locale=en_all_zz IBM DeveloperWorks Article Incorporate web services into your SQL queries Using RPG to exploit IBM DB2 XML support Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 64 Speaker s Biography Birgitta Hauser has been a Software and Database Engineer since 2008, focusing on RPG, SQL and Web development on IBM i at Toolmaker Advanced Efficiency GmbH in Germany. She graduated with a business economics diploma, and started programming on the AS/400 in She also works in consulting and education as a trainer for RPG and SQL developers. Since 2002 she has frequently spoken at the COMMON User Groups and other IBM i and Power Conferences in Germany, other European Countries, USA and Canada. In addition, she is co-author of two IBM Redbooks and also the author of several articles and papers focusing on RPG and SQL for a German Publisher, ipro Developer and IBM DeveloperWorks. In 2015 she received the John Earl Speaker Scholarship Award Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser Seite 65 27

28 Consume XML documents and Web Services with SQL! Yes, i can! Thank you! Birgitta Hauser Toolmaker Advanced Efficiency GmbH Hauser@sss-software.de Westendstraße 34 bha@toolmaker.de Kaufering Tel. (+49) Seite 66 Common Annual Conference Consume XML Documents and Web-Services with SQL - Birgitta Hauser 28