Consume XML Documents and Web-Services with SQL

Transcription

1 Schaumburg Consume XML Documents and Web-Services with SQL Birgitta Hauser / TOOLMAKER Advanced Efficiency GmbH Tel. (+49) Sasbach / Ortenau Seite 2 1

2 Agenda Consuming XML Documents XMLTABLE Table Function Accessing XML documents Converting the hierarchical structure into a relational represenatation Picking out only the relevant information DB2 XPath Syntax GET_XML_File Scalar Function Accessing (RESTFul) Web-Services Using http-functions for accessing web-services Combining http-functions with XMLTABLE for accessing and filtering the information provided in the web-service response Seite 3 Well-formed XML Document Seite 4 2

3 XML Well-formed Document Exactly 1 Root Element All other elements must be embedded within the root element XML-Declaration (XML Version and Encoding) can be specified first, before the Root Element All information is included in partly nested Elements All elements consist of an Start and End Tag (except Empty Elements) Can be nested ( sequence must be correct) Attributes can be used for describing elements more accurately Attributes must be specified within the start tag of an element Attribute values must be enclosed with double quotes ( x ) Namespaces allow universally unique element and attribute names All element, attribute, namespace names are case sensitive Seite 5 XML Wellformed Document Declaration Root Element Start Tag Element End Tag Element Attribute Seite 6 3

4 Decomposing XML Documents Seite 7 Decomposing XML Documents Annotated XML Document Decomposition Based on XML Schemas Stored Procedures for (de)registering XML Schemas within the XML-Schema-Repository (XSR) Stored Procedure for decomposing an XML document based on schemas registered within the XSR and shredding the XML information directly into tables XMLTABLE Table Function Retrieves the content of an XML document as a result set (table) Specified within the FROM-clause of a SELECT-statement Returned data can be reduced based on additional WHERE conditions Works in conjunction with a sub-set of the XPath expressions Seite 8 4

5 XMLTABLE Table Function Seite 9 XMLTABLE Table Function Retrieving XML data in relational format XML Parser Reads an XML document located In an XML column in a database table In the IFS (Integrated File System) in composition with the GET_XML_FILE and XMLPARSE scalar functions (with http-functions in SYSTOOLS) returned by a RESTful WebService Hierarchical XML information is converted into a relational format Returns a table Must be specified within the FROM clause of a SELECT statement XMLTABLE Table Function can be used like any table or view <employee id="901"> <name><first>birgitta</first> <last>hauser</last></name> <department>programming </department> </employee>... additional employees id first last department 901 Birgitta Hauser Programming Seite 10 5

6 XQuery and XPath Seite 11 XQuery and XPath XQuery is a functional programming language designed by the World Wide Web Consortium (W3C) for querying and modifying XML data XPath is an expression language designed by the World Wide Web Consortium (W3C) for processing of XML data that conforms to the XQuery 1.0 and XPath 2.0 data model Tutorial: w3schools - DB2 XPath Sub-set of the language constructs in the XPath 2.0 recommendation used with the XMLTABLE SQL built-in table function for converting an XML value into a relational result set. Seite 12 6

7 XPath Syntax - Nodes Nodes XML documents are treated as trees of nodes Kinds of Nodes: element, attribute, text, namespace, processing-instruction, comment, document nodes Root Element The topmost element of the tree is called the root element. Seite 13 XPath Syntax - Nodes Expression Description Path Expression Result nodename / // Selects all nodes with the name "nodename" Selects from the root node Selects nodes in the document from the current node that match the selection no matter where they are bookstore /bookstore bookstore/book //book bookstore//book Selects all nodes with the name "bookstore" Selects the root element bookstore Note: If the path starts with a slash ( / ) it always represents an absolute path to an element! Selects all book elements that are children of bookstore Selects all book elements no matter where they are in the document Selects all book elements that are descendant of the bookstore element, no matter where they are under the bookstore Selects attributes //@lang Selects all attributes that are named lang Selects the current. node Selects the parent of.. the current node Seite 14 7

8 XMLTABLE Basic Syntax XMLTABLE( row-xquery-expression-constant PASSING row-xquery-argument AS Identifier COLUMNS xml-table-column-definition, Def2,..., DefN) Row-XQuery-Argument XML column name within the specified table or CTE or Sub-Select BLOB-Locator (returned by GET_XML_FILE function) on the XML document in the IFS Result of an RESTful Web-Service (access in composition with http-functions in SYSTOOLS) Identifier Syntax: Name Must be specified in the Row-Xquery-Expression in upper case Name Must be specified as exact match in the Row-Xquery-Expression Row-XQuery-Expression-Constant Syntax: $xxx/rootelement/element1/ /ElementN ( Identifier xxx ) $xxx: $ - mandatory xxx: Identifier (without double quotes) /RootElement/Element1/ : Start Element Attention: Row-Xquery-Expression and Identifier are case sensitive Seite 15 XMLTABLE Basic Syntax XMLTABLE( row-xquery-expression-constant PASSING row-xquery-argument AS Identifier COLUMNS xml-table-column-definition, Def2,..., DefN) XML-Table-Column-Definition Multiple column definitions and be specified, separated by commas Syntax: ColName ColName DataTypeDefinition PATH XPathDefinition Column name in the result table cannot be qualified DataType Data type and length specification for the column in the result table CCSID for CHAR/VARCHAR not allowed DEFAULT clause for replacing NULL values can be specified Xpath Element Element name embedded in single quotes Elem1/Elem2/Element incl. path underneath the start Attibute name (must be prefixed Elem1/Elem2/@Attribute incl. path underneath the start element Seite 16 8

9 XMLTABLE Basic Syntax Example Table with XML Column Create Table EmplXML Column Document: DataType XML Insert XML data into XML Column XMLPARSE Checks and converts the XML-string into the internal XML format Seite 17 XMLTABLE Basic Syntax Example XML Document - Structure RDi XML Editor Seite 18 9

10 XMLTABLE - Basic Syntax Example 1 Cross Join Table EmplXML must be specified it will be implcitly joined with the XMLTABLE UDTF Document: Column out of the EmplXML table containing the XML document Return only information within the Staff/Department/Employee/Name Element Column-Definition: Return First and Last Names only Column Names in the result table (=xml document/renamed) Seite 19 XMLTABLE - Basic Syntax Example 1 Sub-Select Locate the XML Document with an SQL Sub-Select Document/D: Identifier / Name of the Sub-Select Return only information within the Staff/Department/Employee/Name Element Column-Definition: Return First and Last Names only Column Names in the result table (=xml document/renamed) Seite 20 10

11 XMLTABLE - Basic Syntax Example 2 - Nodes Return only information within the Staff/Department/Employee Element Return First and Last Name Element information located within Element Name Return , Birthday and Salary Element information located within Element Privacy Convert the Birthday data into a real date Convert the Salary data into a packed numeric values Seite 21 XMLTABLE - Basic Syntax Example 3 Default Values If the XML document does not include salary information, Default value 0 is returned Seite 22 11

12 XML-Table Basic Syntax Example 3 Attributes Attribute Definition: Attribute names must be prefixed with Default clauses can also be specified for Attributes Seite 23 Access XML in Stream Files Seite 24 12

13 GET_XML_FILE Function Returns data from a source physical file or stream file The result of the function is a BLOB-Locator Can be compared with a pointer that points to the data Can be used in SQL statements and with scalar functions like any character value Data is converted into UTF-8 Must be executed under commitment control The locator is freed when a COMMIT or ROLLBACK is performed Copy an IFS file into an XML column Seite 25 Access Stream Files in composition with XMLTABLE Define IFS File to be accessed directly within the XMLTABLE function GET_XML_File: Retrieve a Locator/Pointer on the data stored in an stream file XMLPARSE: Check for a valid/well-formed XML document Seite 26 13

14 Global Variables Seite 27 Global Variable What is a SQL Global Variable? It is a permanent object (*SRVPGM) located in a schema Is created with the SQL command CREATE VARIABLE Almost any SQL data type is allowed (Except ROWID and DATALINK) Cannot be created in the QTEMP library/schema Even though being global it is managed at job/connection level The same variable may have different values for different jobs/connections Initialization occurs with the first use in a job/connection Initialization can be based on a constant value a special register an other (SQL) global variable a scalar function or a User Defined Function (UDF) a SQL statement that returns a single value (row/column) Can be used in triggers and in view definitions Seite 28 14

15 Global Variable SQL Command CREATE VARIABLE Variable Name (upto 128 characters) must be specified Almost any data type allowed (except ROWID and DATALINK) Default Values can be specified (Default = NULL) Seite 29 Global Variables Examples Create Variable GblBossIdFix: GblBossId: Global Variable with data type Integer Default value = 101 Global Variable with data type Integer Default value depends on the manager s employee no Employee without an superior employee Seite 30 15

16 Global Variables When Global Variables are resolved? For static SQL statements and SQL routines global variables get resolved the first time all table references are resolved In views, triggers, and other global variables global variables get resolved when the object is created For dynamic statements global variables get resolved when the statement is initially prepared. How Global Variables can be changed? By executing a SQL SET, SELECT INTO or VALUES INTO statement By calling a stored procedure to which the global variable is passed as OUT or INOUT parameter Seite 31 Using Global Variables within Views - Examples Default value for the global variable is taken The global variable is modified Modified value vor the global variable is taken Create View with a RCTE (Recursive Common Table Expression) for an organization chart Start Value depends on the global variable GblBossId Page 32 16

17 Namespaces Seite 33 XML Namespaces Namespaces in XML 1.0 (3 rd edition December 8 th 2009) published by W3C (World Wide Web Consortium) Definition universally unique Names For elements and attributes (Definition with xmlns:xx= URI ) Element/attribute (qualified) names are composed of: The Namespace identifier <MyNS:MyElem>Value</MyNS:MyElem> A colon (:) The element/attribute name New (XML) documents consisting of pieces from other documents even with duplicate element/attribute names can be created Seite 34 17

18 XML Namespaces Are identified by a Uniform Resource Identifier (URI) URIs do not need to represent actual resources Any unique text string can serve as a namespace identifier Namespace declaration must be specified within a start tag of an element Namespace declaration Qualified element names Seite 35 XMLTABLE and Namespaces Seite 36 18

19 XMLTABLE XML Document with Namespaces Seite 37 XMLTABLE XML Document with Namespaces Without Namespace declaration within the XMLTABLE function *: must be specified instead of the namespace within the node name Seite 38 19

20 XMLTABLE with Namespaces XML Document with Namespaces Declaring the namespaces to be used with the XMLNAMESPACE scalar function All Element, Attribute and Node names must be qualified, i.e. prefixed with the appropriate namespace label Seite 39 Query the Web with SQL Seite 40 20

21 http Functions for DB2 for i for RESTFul Web-Services Prerequisites: 5770-SS1 DB2 PTF Group SF99701 Level 23 Java 1.6 or later (5761-JV1 Option 11, 12, 14, or 15) New http Functions http Functions added to the SYSTOOLS schema SYSTOOLS schema is NOT part of the system path/library list must be (manually) added to the SQL path/library list http Functions they are NOT subject to IBM service and support IBM maintenance for SYSTOOLS is to delete existing objects followed by the creation of the newer versions (without considering any modifications) Can be used as soon as DB2 PTF Group Level 23 is installed Java source is provided in /QIBM/ProdData/OS/SQLLIB/bin/systools_java_source.jar Seite 41 http Functions for DB2 for i Scalar and Table Functions (names starting with http) Located in the SYSTOOLS schema Function Names http (Method) (DataType) (Verbose) Method: Data type: GET, PUT, POST, DELETE, HEAD Without method Method passed as parameter value of the request AND response messages CLOB (Character Large Object) / BLOB (Binary Large Object) Verbose: with Verbose: Table function with 2 columns 1st column: http header information 2nd column: response message without Verbose: Scalar Function response message only Examples: httpgetclob(), httpputblobverbose(), httpclob() Seite 42 21

22 IBM i Navigator http Functions in Schema SYSTOOLS Seite 43 SQL Scalar http Functions for RESTful Services - Overview User Defined Return Parameter Definition http- Function (UDF) Data Type Name Data Type Verb httpgetblob BLOB URL VarChar(2048) httpheader CLOB(10K) GET URL VarChar(2048) httpgetclob CLOB httpheader CLOB(10K) GET URL VarChar(2048) httpputblob BLOB httpheader CLOB(10K) PUT Data BLOB(5M) URL VarChar(2048) httpputclob CLOB httpheader CLOB(10K) Data CLOB(5M) PUT URL VarChar(2048) httppostblob BLOB httpheader CLOB(10K) POST Data BLOB(5M) URL VarChar(2048) httppostclob CLOB httpheader CLOB(10K) POST Data CLOB(5M) Description Retrieves a resource by a given URL vie HTTP GET. The resource is returned as binary LOB (BLOB). Retrieves a text-based resource by a given URL via HTTP GET. The resource is returned as character LOB (CLOB). Character set conversion into the database code page is performed if necessary. Creates (upload) a binary resource under a given URL using HTTP PUT. A response message is returned as binary LOB (BLOB) if issued by the server. Creates (upload) a textual resource under a given URL using HTTP PUT. A response message is returned as character LOB (CLOB) if issued by the server. Character set conversion is performed if necessary. Updates a binary resource under a given URL using HTTP POST. The response message is returned as binary LOB (BLOB). Updates a textual resource under a given URL using HTTP POST. The response message is returned as character LOB (CLOB). Character set conversion is performed if necessary. Seite 44 22

23 SQL Scalar http Functions for RESTful Services - Overview User Defined Return Function (UDF) Data Type httpdeleteblob BLOB httpdeleteclob CLOB httpblob httpclob BLOB CLOB Parameter Definition http- Name Data Type Verb Description URL VarChar(2048) httpheader CLOB(10K) DELETE URL httpheader URL httpmethod httpheader Data URL httpmethod httpheader Data VarChar(2048) CLOB(10K) DELETE VarChar(2048) VarChar(128) CLOB(10K) BLOB(5M) VarChar(2048) VarChar(128) CLOB(10K) BLOB(5M) any HTTP verb any HTTP verb URL VarChar(2048) httphead CLOB/XML httpheader CLOB(10K) HEAD Deletes a binary resource under the given URL using HTTP DELETE. The response message is returned as binary LOB (BLOB) if there is any. Deletes a textual resource under the given URL using HTTP DELETE. The response message is returned as character LOB (CLOB) if there is any. Character set conversion is performed if necessary. Performs an HTTP request with the provided verb. A request message can be provided as binary LOB (BLOB). A response message is returned as binary LOB (BLOB) if issued by the server. Performs an HTTP request with the provided verb. A request message can be provided as character LOB (CLOB). A response message is returned as character LOB (CLOB) if issued by the server. Character set conversion is performed if necessary. Verifies the HTTP header for the requested resource using HTTP HEAD. Just the HTTP header is returned as CLOB Seite 45 SQL Scalar Functions for RESTful Services - Parameter Parameter URL Required for all functions VarChar(2048) httpheader Can be passed without value or *BLANKS CLOB(10K) Data Required for all PUT and POST functions BLOB(5M) or CLOB(5M) httpmethod PUT, POST, GET, DELETE Required for httpblob / httpclob Functions VarChar(128) Return Value Binary (BLOB) / Character (CLOB) Large Object Containing the resource representation for example as XML/JSON Seite 46 23

24 Access Web-Service Seite 47 Free Web Service Exchange Rates - Example Web Service European Central Bank Daily Exchange Rates Seite 48 24

25 Access Web Service with SQL Exchange Rates - Example Seite 49 Consume Web-Services with SQL Example Exchange Rates Namespaces Access web-service with HTTPGetBLOB URL Seite 50 25

26 Consume Web-Services with SQL wrapped with a View (Exchange Rates) Additional Currency Rates Seite 51 Access Web-Service with embedded SQL Code Snippet Access Web-Service through an http function in conjunction with the XMLTABLE UDTF stored within the ExcRateV2 SQL View Cursor handling is performed like for any other table or view Seite 52 26

27 Access Web-Service with RPG Example: Exchange Rates Access Web-Service Embedded in a SQL View RPGLE mit embedded SQL Seite 53 Any Questions? Seite 54 27

28 References IBM i Information Center Db2 for i SQL reference IBM i Database XML Programming Accessing webservices using IBM Db2 for i HTTP UDFs and UDTFs ibm.com/partnerworld/wps/servlet/download/DownloadServlet?id=k4ixw2TAAIwiPCA$cnt&attachmentNa me=accessing_web_services_using_ibm_db2_for_i_udfs_and_udtfs.pdf&token=mtqxmdaxodmxmjmymg ==&locale=en_all_zz IBM DeveloperWorks Article Incorporate web services into your SQL queries Using RPG to exploit IBM Db2 XML support Seite 55 Special Thanks to Deon (Kit) von Blerk For proof-reading/correcting the English translation of these presentations Holger Scherer For providing an AS/400 (IBM i-system) enabling the creation of the samples/code used in these presentations Your data is save! in the bunker Page 56 28

29 Speaker s Biography Birgitta Hauser has been a Software and Database Engineer since 2008, focusing on RPG, SQL and Web development on IBM i at Toolmaker Advanced Efficiency GmbH in Germany. She graduated with a business economics diploma, and started programming on the AS/400 in She also works in consulting and education as a trainer for RPG and SQL developers. Since 2002 she has frequently spoken at the COMMON User Groups and other IBM i and Power Conferences in Germany, other European Countries, USA and Canada. In addition, she is co-author of two IBM Redbooks and also the author of several articles and papers focusing on RPG and SQL for a German publisher, ipro Developer and IBM DeveloperWorks. In 2015 she received the John Earl Speaker Scholarship Award. Page 57 Consume XML documents and Web Services with SQL! Yes, i can! Tack så mycket! Birgitta Birgitta Hauser Hauser bha@toolmaker.de Hauser@sss-software.de Hauser@SSS-Software.de bha@toolmaker.de Seite 58 Data3 - Höstkonferensen Consuming XML-Documents and Web-Services with SQL - Birgitta Hauser Toolmaker Advanced Efficiency GmbH Westendstraße Kaufering Tel. (+49)