Unlock your XML potential with DB2 9

Transcription

1 IBM Software Group Unlock your XML potential with DB2 9 A DB2 Chat with the Lab Sept 20, IBM Corporation

2 Agenda Part I Value of managing XML data Background Usage scenarios and examples Part II purexml technology in DB2 9 purexml Overview XML Storage XML Indexes Querying XML: XQuery, SQL/XML XML Schema Support Shredding XML to Relational Performance 2

3 IBM Software Group Unlock your XML potential with DB2 9 A DB2 Chat with the Lab Sept 20, 2006 Part I Value of managing XML data - Rav Ahuja, DB2 Program Manager, IBM Toronto Lab 2006 IBM Corporation

4 What is XML? IBM Software Group DB2 Information Management Software XML extensible Markup Language Self-describing data structures XML tags describe each element and their attributes Benefits Flexible No fixed format or data types Structure can be easily changed Platform Independent Not tied to any operating system, language or software vendor XML can be easily exchanged Fully Unicode compliant!"#$% &' () *+),-),.) /%)/ ) ) 4

5 Importance of XML data? More and more XML data generated everyday XML is pervasive in all kinds of organizations Almost every sector has XML based standards 5

6 XML is Driving Many Industry Segments Today % & ' ( ' ( ) ( % & * + ' ( ( ',+ ' -.! ' * $!!! "!!! ' ) 0 " #! ( 4 (! # * / *! 0! " # $ "!" (. 1 ) $ 2 3 $! $!. 6

7 XML - the Fabric of Services Oriented Architectures 7

8 XML Solves Many Business Problems Today Integration of diverse data sources Information exchange between applications & organizations Forms and workflow processing Content and document management Message-based transactions, web services, SOA XML as business object / transaction record digital signatures, auditing, regulatory compliance XML as the better data model for multi-values, hierarchical and complex data 8

9 Business Benefits of managing XML data with DB2 9 Lower Development Costs Reduced code and development complexity Improved developer productivity Quicken solution development and gain cost savings Greater Business Agility Easily accommodate changes to data and schemas Update applications rapidly and reduce maintenance costs Respond quickly to dynamic conditions and get faster time to value Improved Business Insight Access to hidden gems (data) in unexploited documents Unprecedented application performance Gain competitive advantage through better and quicker information 9

10 Improving business flexibility with SOA and XML Needed integration across different products, IT infrastructures and business processes to respond flexibly to customer requests Needed to link products & processes to simplify and expedite orders, increase product customization and speed time-to-market Benefits Early Results Relational DB2 9 Significantly reduced order processing time Development Time CLOB: 8 hrs Shred: 2 hrs 30 min. Radically improved ability to handle increased business volumes Add field to schema Relative lines of I/O code Query Performance Query non-shredded XML 1 week hrs 1 week 5 min sec - 10 min ½ day 10

11 XML Data Management Needs Relational Maturity XML Data Needs Protection Backup and recovery features to ensure continuity Data is protected using database security Simplified XML Data Access Centrally store and access difficult to retrieve data SQL or XQuery can be used to retrieve data Join XML data with it s related relational data Search Speed Search documents quickly and efficiently using proven search optimization engine of mature database Optimize Existing Investments Use existing technology infrastructure and skills to store and manage both relational and XML 11

12 IBM Software Group Unlock your XML potential with DB2 9 A DB2 Chat with the Lab Sept 20, 2006 Part II purexml technology in DB2 9 - Matthias Nicola, DB2 XML Performance, IBM Silicon Valley Lab 2006 IBM Corporation

13 Agenda Overview purexml Storage XML Indexes Querying XML: XQuery, SQL/XML XML Schema Support Shredding XML to Relational Performance 13 Matthias Nicola, IBM SVL

14 DB2 9: purexml and Relational Capabilities Relational and XML data are stored differently, but closely linked SQL to manipulate relational data SQL/XML to publish relational data as XML XQuery or SQL/XML to query XML data Join Relational and XML data DB2 9 SERVER CLIENT DB2 Client / Client Application SQL/XML XQuery Relational Interface XML Interface Hybrid DB2 Engine DB2 Storage: XML 14 Matthias Nicola, IBM SVL

15 DB2 9: purexml and Relational Capabilities Relational and XML data are stored differently, but closely linked SQL to manipulate relational data SQL/XML to publish relational data as XML XQuery or SQL/XML to query XML data Join Relational and XML data DB2 9 SERVER CLIENT DB2 Client / Client Application SQL/XML XQuery Relational Interface XML Interface Hybrid DB2 Engine DB2 Storage: XML 15 Matthias Nicola, IBM SVL

16 +! $ $! 1 5! 6 XML Storage: Old and New 1 5 * / 7 $ 5! Extract selected elements/attr. XML DOC Fixed Mapping XML DOC XML DOC XML DOC XML DOC XML DOC CLOB Column i.e. XML as text Side Tables or Indexes (regular relational tables) Any sub-document level access requires XML parsing slow. Mapping prevents XML schema changes, and is often too complex. XML reconstruction is slow. Maximum flexibility and performance XML Column 16 Matthias Nicola, IBM SVL

17 Agenda Overview purexml Storage XML Indexes Querying XML: XQuery, SQL/XML XML Schema Support Shredding XML to Relational Performance 17 Matthias Nicola, IBM SVL

18 The XML Data Type & purexml Storage Tables can contain relational and/or XML columns Relational columns stored in tabular format XML columns stored in a parsed hierarchical format No XML parsing for query evaluation High Performance create table dept (deptid char(8),...,deptdoc xml); deptid deptdoc PR27 <dept> <emp> </emp> </dept> DB2 Storage: XML 18 Matthias Nicola, IBM SVL

19 Efficient Document Tree Storage <dept> <employee id="901"> <name>john Doe</name> <phone> </phone> <office>344</office> </employee> <employee id="902"> <name>peter Pan</name> <phone> </phone> <office>216</office> </employee> </dept> 7=901 1 John Doe =902 Reduces storage Fast comparisons & navigation 1 Peter Pan XML text represented as document tree dept Tag names encoded as unique integers employee employee id=901 name phone office id=902 name phone office John Doe Peter Pan Matthias Nicola, IBM SVL

20 XML Data as Trees on DB2 pages All benefits of DB2 tablespaces: Index Buffered in Buffer Pools Prefetching Logging & Recovery etc.... page page page table space 20 Matthias Nicola, IBM SVL

21 Agenda Overview purexml Storage XML Indexes Querying XML: XQuery, SQL/XML XML Schema Support Shredding XML to Relational Performance 21 Matthias Nicola, IBM SVL

22 XML Indexing Examples create unique index idx1 on customer(info) generate key using xmlpattern as sql double; create table customer( info XML); create index idx4 on customer(info) generate key using xmlpattern '/customerinfo/phone' as sql varchar(40); create index idx2 on customer(info) generate key using xmlpattern '/customerinfo/name' as sql varchar(40); create index idx3 on customer(info) generate key using xmlpattern '//name' as sql varchar(40); <customerinfo Cid="1004"> <name>matt Foreman</name> <addr country="canada"> <street>1596 Baseline</street> <city>toronto</city> <state>ontario</state> <pcode>m3z-5h9</pcode> </addr> <phone type="work"> </phone> <phone type="home"> </phone> <assistant> <name>peter Smith</name> <phone type="home"> </phone> </assistant> </customerinfo> 22 Matthias Nicola, IBM SVL

23 XML Indexes for High Query Performance Define 0, 1 or multiple XML Value Indexes per XML column Index any elements or attributes, incl. mixed content Index definition uses XPath to specify which elements/attributes to index (and which not to) Can index all elements/attributes, but not forced to do so Can index repeating elements 0, 1 or multiple index entries per document New XML-specific join and query evaluation methods, evaluate multiple predicates concurrently with minimal index I/O 23 Matthias Nicola, IBM SVL

24 XML Full Text Search DB2 Net Search Extender enhanced for XML XML-aware text index of full or partial documents create index idx5 for text on customer(info) ; Usage: select info from customer where contains(info, 'sections("/customerinfo/comment") "happy" ')= 1; Complex search criteria supported 37 languages, fuzzy search, stemming, etc. Queries can use a mix of full-text & regular XML indexes 24 Matthias Nicola, IBM SVL

25 Agenda Overview purexml Storage XML Indexes Querying XML: XQuery, SQL/XML XML Schema Support Shredding XML to Relational Performance 25 Matthias Nicola, IBM SVL

26 Two ways to query XML data in DB2 9 SQL/XML SQL as the primary language Optional: XQuery embedded in SQL SQL SQL/XML XQuery XQuery XQuery as the primary language Optional: SQL embedded in XQuery XQuery XQuery SQL 26 Matthias Nicola, IBM SVL

27 Querying XML using SQL or SQL/XML create table dept(deptid char(8), deptdoc xml) select deptid, deptdoc from dept where deptid = PR37 SQL select deptid, xmlquery( for $i in $d/dept let $j := $i//name return $j passing deptdoc as d ) SQL/XML XQuery from dept where deptid LIKE PR% and xmlexists( $d/dept[@bldg = 101] passing deptdoc as d ) 27 Matthias Nicola, IBM SVL

28 XMLTABLE: Return XML in tabular format <dept bldg="101"> <employee id="901"> <name> <first>john</first> <last>doe</last> </name> <office>344</office> </employee> <employee id="902"> <name> <first>peter</first> <last>pan</last> </name> <office>216</office> </employee> </dept> SELECT X.* FROM dept, XMLTABLE ( $d/dept/employee passing deptdoc as d ) COLUMNS empid INTEGER firstname VARCHAR(30) PATH name/first, lastname VARCHAR(30) PATH name/last, office INTEGER PATH office ) AS X empid firstname John Peter lastname Doe Pan SQL/XML XQuery office Matthias Nicola, IBM SVL

29 SQL/XML Publishing: Turn relational data into XML SELECT XMLELEMENT(NAME "Department", XMLATTRIBUTES (e.dept AS "name" ), XMLAGG( XMLELEMENT(NAME "emp", e.firstname) ) ) AS "dept_list" FROM employee e WHERE.. GROUP BY e.dept; SQL/XML dept_list firstname SEAN MICHAEL VINCENZO CHRISTINE lastname LEE JOHNSON BARELLI SMITH dept A00 B01 A00 A00 <Department name="a00"> <emp>christine</emp> <emp>vincenzo </emp> <emp>sean</emp> </Department> <Department name="b01"> <emp>michael</emp> </Department> 29 Matthias Nicola, IBM SVL

30 XQuery Support in DB2 9 create table dept(deptid char(8), deptdoc xml); for $d in db2-fn:xmlcolumn( dept.deptdoc )/dept let $emp := $d//employee/name where $d/@bldg > 95 XQuery order by $d/@bldg return <EmpList> {$d/@bldg, $emp} </EmpList> 30 Matthias Nicola, IBM SVL

31 XQuery Support in DB2 9 create table dept(deptid char(8), deptdoc xml); for $d in db2-fn:xmlcolumn( dept.deptdoc )/dept let $emp := $d//employee/name where $d/@bldg > 95 XQuery order by $d/@bldg return <EmpList> {$d/@bldg, $emp} </EmpList> for $d in db2-fn:sqlquery('select deptdoc from dept where deptid LIKE PR% ')/dept XQuery SQL for $d in db2-fn:sqlquery('select dept.deptdoc from dept, unit where dept.deptid=unit.id Optional: and unit.headcount > 200 )/dept SQL embedded 31 Matthias Nicola, IBM SVL

32 Agenda Overview purexml Storage XML Indexes Querying XML: XQuery, SQL/XML XML Schema Support Shredding XML to Relational Performance 32 Matthias Nicola, IBM SVL

33 XML Schema Support & Validation in DB2 9 Documents for zero, one, or many schemas per XML column: No Schema One Schema Schema V1 Documents Any mix you want! & Schema V2 w/ and w/o schema XML Schema Flexibility! 33 Matthias Nicola, IBM SVL

34 XML Schema Support & Validation in DB2 9 Schemas are registered & stored in the DB2 Schema Repository (XSR) for fast and stable access. Use of XML Schemas is optional, on a per-document basis No need for a fixed schema per XML column Validation per document (i.e. per row) Zero, one, or many schemas per XML column For example: different versions of the same schema, or schemas with conflicting definitions Mix validated & non-validated documents in 1 XML column 34 Matthias Nicola, IBM SVL

35 Agenda Overview purexml Storage XML Indexes Querying XML: XQuery, SQL/XML XML Schema Support Shredding XML to Relational Performance 35 Matthias Nicola, IBM SVL

36 Why Shredding? Shredding is bad if XML Schema & Mapping is complex XML Schema changes Fixed Mapping XML DOC Shredder Shredding may still be needed & OK if (regular relational tables) Existing SQL applications need the data Your BI & Reporting Tools only have relational APIs XML structure no longer relevant after shredding XML structure is simple & very regular XML schema changes are rare and benign 36 Matthias Nicola, IBM SVL

37 The New Shred in DB2 9 (XML Decomposition) Mapping from XML to relational: annotations to the XML schema aka Annotated Schema Shred uses XML schemas from the schema repository Replaces XML Extender shred - a lot faster! "Our tests of the new XML Decomposition in DB2 Viper showed an 8 times performance improvement over the DB2 V8 XML Extender." Dr. Andreas Birkendorf, DOUGLAS Informatik & Service GmbH 37 Matthias Nicola, IBM SVL

38 Other XML Features in DB2 9 XML Import, Export, Backup, Restore XML Type in Stored Procedures API Extensions (JDBC, ODBC/CLI,.NET, etc.) for SQL/XML and XQuery High Availability Data Replication (HADR) DB2 Developer Workbench Visual XQuery Builder GUI for XML Schema annotations for shredding and more 38 Matthias Nicola, IBM SVL

39 Agenda Overview purexml Storage XML Indexes Querying XML: XQuery, SQL/XML XML Schema Support Shredding XML to Relational Performance 39 Matthias Nicola, IBM SVL

40 DB2 9 Performance and Scalability Financial transaction processing scenario (FIXML) Customers Customer 1 n Account 1 1 n Holding n n Order n 1 1 Security Brokerage House DB Cus tacc.xsd FIXML (41 XSD files) Security.xsd Database: 36 million XML documents, 2kb to 20kb 23 XML Indexes High concurrency, queries/insert/update/delete 40 Matthias Nicola, IBM SVL

41 Constantly High XML Insert Throughput Order Documents: Inserts per Second concurrent clients inserting FIXML order documents, with XML indexes Number of Documents Inserted (in Millions) AIX 5.3, P-Series P5-560Q, 8 CPUs, TotalStorage DS8100, 41 Matthias Nicola, IBM SVL

42 XML Query Scalability Queries/sec Read-only Workload, XML Queries 36 Million Documents Queries/second %CPU Utilization %CPU Number of concurrent users 0 Throughput increases with number of users until machine capacity is max ed out. This is text-book scalability. 42 Matthias Nicola, IBM SVL

43 purexml Insert vs. Shred Performance 10,000 Customer Documents, 4Kb to 20Kb shredded to 87 columns in 12 tables. Seconds XML Insert vs. Shred Performance 100% purexml insert, 1 XML column, 1 table. XML DOC Fixed Mapping XML DOC 14.2% 3.4% DB2 V8 XML Extender Shred DB2 9 New Shred DB2 9 purexml Insert 6 (regular relational tables) XML Column 43 Matthias Nicola, IBM SVL

44 Questions? ibm.com/db2/xml 44 Matthias Nicola, IBM SVL

45 ibm.com/db2/xml 45