Information On Demand Conference 2007

Transcription

1 A New Way to Handle Complex Dynamic Data Type by Using pureml Samson Tai Manager of BetaWorks, Software Group

2 Introduction DB2 9 pureml support enables us to justify new applications which may not have been practical, such as office reports, patients medical records..

3 Agenda Characteristics of DB2 9 pureml Advantages of ML data type pureml for office reports / eforms /e-catalog / e- Patient Record Querying ML Creating indexes on ML Conclusion

4 Characteristics of DB2 9 pureml What is ML? ML is etensible Markup Language ML describes data CSV data example 47; John Doe; 58; Peter Pan; Database systems; 29; SQL; relational ML data example <book> <authors> <author id= 47 >John Doe</author> <author id= 58 >Peter Pan</author> </authors> <title>database systems</title> <price>29</price> <keywords> <keyword>sql</keyword> <keyword>relational</keyword> </keywords> </book>

5 Characteristics of DB2 9 pureml ML is flexible Easy to expand the data structure Schema Flexibility what ML is all about by self describing nature <book> <authors> <author id= 47 >John Doe</author> <author id= 58 >Peter Pan</author> </authors> <title>database systems</title> <price>29</price> <keywords> <keyword>sql</keyword> <keyword>relational</keyword> </keywords> </book> <book> <authors> <author id= 47 >John Doe</author> <author id= 58 >Peter Pan</author> </authors> <title>database systems</title> <price>29</price> <keywords> <keyword>sql</keyword> <keyword>relational</keyword> <keyword>hierachical</keyword> </keywords> <review>.. </review> <review>.. </review> </book>

6 Characteristics of DB2 9 pureml DB2 V8 ML extender vs DB2 9 pureml DB2 V8 ML extender supported two modes for storing ML with trade-offs Storing as a CLOB Parsing degrades query performance Shredding to relational tables / columns Schema Loses Schema Flexibility Flexibility Query Performance DB2 9 pureml realizes both Schema Flexibility and Performance

7 Characteristics of DB2 9 pureml Native ML data type support ML data type support in Application / API ML data type support in DDL ML data storage support Storing ML data in parsed hierarchical format (W3C DM) Optimal for query </dept> Schema Flexibility Schema Less One or Multiple Schema <dept> <emp> </emp> Application ML indexing create table dept (, doc xml) create index.. on dept(doc)

8 Advantages of ML data type

9 Advantages of ML data type Single schema Relational Data Meta Data definition in relational data 2. Insert Is mandatory Before inserting data Adding column D TABLE tbl 2. Insert TABLE tbl A B C A B C D INSERT INSERT INTO INTO tbl tbl (( A, A, B, B, C )) A B C A INSERT INSERT INTO INTO tbl tbl (A, (A, B, B, C, C, D) D) 1. Creating table CREATE CREATE TABLE TABLE tbl tbl (( A type, type, DB2 B type, DB2 Catalog type, C type) Meta type) Meta Data Data 1. Altering table ALTER ALTER TABLE TABLE tbl tbl ADD ADD COLUMN COLUMN (D (D type) type) DB2 DB2 Catalog Meta Meta Data Data

10 Advantages of ML data type Schema Less ML No ML data structure definition as a meta data DB2 pureml parse ML and store allows any ML schema in a ML column Less number of columns involved in programming IBM Basis Software for Ultra-RAD Group Lotus TABLE ML COLUMN software Inserting rows tbl doc... DOC INSERT INTO INTO tbl tbl (, (, doc doc )) (customerml) <Cust>.. INSERT INTO INTO tbl tbl (, (, doc doc )) (productml) INSERT INTO INTO tbl tbl (, (, doc doc )) (orderml) </Cust> <Product>.. </Product> <Order>.. </Order>

11 Advantages of ML data type Multiple Schema ML TABLE Multiple ML Schemas are allowed in a ML column tbl ~Mar31 Apr1~ May1~ Validating ML data by a ML schema Per Row basis when insert / update w/o affecting existing data Can restrict sequence of data, repeating limits, data types, etc Allows continuous evolution of data Schema Evolution Realizes new requirements quickly INSERT INTO tbl tbl (.., (.., doc doc )(ML) schema1.0 INSERT INTO tbl tbl (.., (.., doc doc )(ML) schema1.1 INSERT INTO tbl tbl (.., (.., doc doc )(ML) schema1.2.. ML COLUMN doc DOC <A> <B>..</B> </A> <A> <B>..</B> <B>..</B> </A> <A> <B>...</B> <B>...</B> <C><D>x</D></C> </A>

12 Advantages of ML data type - Sparse attributes in relational model Mostly sparse or absent values in many possible attributes Merchant Catalog Form Wage calculation How do you describe these data in relational model? product A B C D E F.. Size Color req. req. Weight Material Style Weave Power Thermal Fuel Req...

13 Advantages of ML data type Sparse attributes in ML ML allows More natural data representation For representing sparse attributes Less complex and expensive search To store in a column To justify new applications ML Column <product <product code= A > code= A > <size>s</size> <size>s</size> <color>blue</color> <color>blue</color> <material>pearl</material> <material>pearl</material> <style>flat</style> <style>flat</style> <weave>smooth</weave> <weave>smooth</weave> </product> </product> <product <product code= B > code= B > <size>large</size> <size>large</size> <weight>1200</weight> <weight>1200</weight> <power>200</power> <power>200</power> <thermal>flat</thermal> <thermal>flat</thermal> <fuel>coal</fuel> <fuel>coal</fuel> </product> </product> <product <product code= C > code= C > <color>red</color> <color>red</color> <weight>25</weight> <weight>25</weight> <material>wool</material> <material>wool</material> <style>slim</style> <style>slim</style> </product> </product> <product <product code= D > code= D > </product> </product>

14 Using pureml for office reports / eforms / ecatalog

15 pureml for office reports / eforms Requirements for Data Requirements Various formats / Schema Form1 Form2 Form Form N Form Form N Form NN Schema Change/Evolution Old adding new fields Form New Form Searching capability? Form Workflow support pureml satisfies all

16 pureml for office reports / eforms If you implemented it on RDB, Schema change involves database maintenance application Application change is dependent on database change TCO for database, DB access and delay of application change Old Adding New New fields database Change

17 pureml for office reports / eforms Schema flexibility of ML Schema flexibility reduces TCO application No change in DB side when schema change Allows continuous evolution of data and applications Old Adding New New fields database ML column

18 pureml for office reports / eforms Productivity improvement by ML RDB: Strong DB schema dependency in application development database application DB Schema Design / Maintenance Pre Post Using Data Layout IBM Software dependency Group Lotus software Space management ML: Least DB dependency in application development Allows continuous evolution of data and application Enable to justify projects by less TCO application database Space management DB Schema Design / Maintenance Using Data Layout Manages Evolution of data

19 pureml for office reports / eforms Storing spread sheets as ML Storing office spread sheets into pureml enables Eliminate manual gathering, copy pasting spread sheets Powerful searching and aggregation Better decision support by leveraging data belongs to individual person Ubiquitous opportunities for leveraging data Purchase estimation product price amount subtotal.ls file Save As ML Cell oriented ML.ML file Transform SLT (per format) <product> <price> <amount> <subtotal> </product> DB2 import Purchase Order Planning/Decision Support Query SQL/ML DB2 9 ML

20 pureml for e-catalog Similarly, an e-catalog usually contains the descriptions for many products Each product has its own set of attributes T-shirt: size, style, color, price TV set: brand, view-type, signal-type, screen-size, price The total number of attributes across all products can be huge E-catalog should efficiently support users to search for products of interest via constraints on attributes Find all the OIDs of T-shirts with size='m' and price<$25

21 Schemas for e-catalog Horizontal Schema: one big "fat" table H(OID, A1, A2,..., An) Conceptually easy Too many columns: impossible for real RDBMS Very sparse: a lot of null values, resulting in poor query processing High IBM processing Software and maintenance Group Lotus cost for software product changes Binary Schema: each attribute corresponds to one table Bi(OID, Ai) Dense A lot of joins are involved in search query: poor query performance

22 Schemas for e-catalog (Continued) Vertical Schema: one big "skinny" table V(OID, attribute_name, attribute_value) This is the schema used in many commercial e-commerce systems! Advantages High Flexibility Ease of schema evolution Low storage overhead (dense) Disadvantages Writing SQL queries against V is cumbersome A lot of joins are involved in search queries: query performance is no better than binary schema

23 Horizontal Schema OID A1 1 v1 2 v5 3 v9 OID A1 A2 A3 A4 A5 1 v1 v2 v3 v4 2 v5 v6 v7 v8 3 v9 v10 v11 4 v12 v13 v14 5 v15 v16 6 v17 v18 OID A4 4 v13 6 v17 Binary Schema OID AttrName Value 1 A1 v1 1 A2 v2 1 A3 v3 1 A5 v4 2 A1 v5 2 A2 v6 2 A3 v7 2 A5 v8 3 A1 v9 IBM OID Software A2 OIDGroup A3 Lotus software 1 v2 2 v6 3 v10 OID A5 1 v4 2 v8 3 v11 4 v14 5 v16 6 v18 1 v3 2 v7 4 v12 5 v15 Vertical Schema 3 A2 v10 3 A5 v11 ML Table OID xmldoc 1 A5 2 A4 3 A5

24 pureml for Patient Medical Records (Schema Flexibility) 饮食特点平日喜好进食 婚姻史未婚 结婚年龄 0 配偶健康情况健康 配偶死忘年龄 0 配偶死亡原因死因 月经情况 1 月经情况月经史初潮持续周期 IBM Software Group Lotus 年龄天数 software 月经量

25 pureml for Patient Medical Records (Hybrid Support)

26 Querying ML

27 Querying ML Query and SQL/ML overview Query Any to any manipulation Can apply ML predicates Path expression Returns ML sequence from ML column db2-fn:xmlcolumn() from SQL/ML db2-fn:sqlquery() SQL/ML select Can apply ML predicates WHERE MLEISTS() Relational predicates trival Returns ML sequence from ML column trivial from Query MLQUERY() from relational MLELEMENT(),etc Returns relational from ML FROM MLTABLE() from relational trivial path is the key concept

28 Querying ML A Simple Query Example Whose phone number = ? Searches an ML column with xpath Returns name element Wildcard is possible -e.g. for any phone IBM Software Group Lotus </employee> software xquery for $p in db2-fn:xmlcolumn( DEPT.DOC")/dept/employee/phone where $p[. =' '] return $p/../name <name>john Doe</name> Dept table, doc column <dept <dept bldg=101> bldg=101> <employee <employee id=901> id=901> <name>john <name>john Doe</name> Doe</name> <phone>408 <phone> </phone> 1212</phone> <phone>415 <phone> </phone> 1234</phone> <office>344</office> <office>344</office> </employee> </employee> <employee <employee id=902> id=902> <name>peter <name>peter Pan</name> Pan</name> <phone>408 <phone> </phone> 9918</phone> <office>216</office> <office>216</office> </employee> </dept> </dept>

29 Dept table, id, doc column Querying ML A simple SQL/ML Example List the name and phone number of the document Select qualified row(s) Searches an ML column with xpath Return results as relatinal <dept <dept bldg=101> bldg=101> <employee <employee id=901> id=901> <name>john <name>john Doe</name> Doe</name> <phone>408 <phone> </phone> 1212</phone> <phone>415 <phone> </phone> 1234</phone> <office>344</office> <office>344</office> </employee> </employee> <employee <employee id=902> id=902> <name>peter <name>peter Pan</name> Pan</name> <phone>408 <phone> </phone> 9918</phone> <office>216</office> <office>216</office> </employee> </employee> IBM Software Group Lotus </dept> </dept> software select x.* from dept t, xmltable('$d/dept/employee/phone' passing t.doc as d" columns "NAME" char(20) path '../name',"phone" char(20) path '. ' ) as x where id=101 NAME PHONE John Doe John Doe Peter Pan

30 Creating indexes on ML

31 Creating Indexes on ML ML Indexes Table dept ML indexes can be created On any elements, attributes Repeating elements Schema less docs Multiple schema docs IBM With Software namespacegroup Lotus software With wild card expression create unique index idx2 on dept(doc) generate key using xmlpattern as sql double; create index idx3 on dept(doc) generate key using xmlpattern '/dept/employee/phone' as sql varchar(35) DOC <dept <dept bldg= 101 > bldg= 101 > <employee <employee id= 901 > id= 901 > <name>john <name>john Doe</name> Doe</name> <phone>408 <phone> </phone> 1212</phone> <phone>415 <phone> </phone> 1234</phone> <office>344</office> <office>344</office> </employee> </employee> <employee <employee id= 902 > id= 902 > <name>peter <name>peter Pan</name> Pan</name> <phone>408 <phone> </phone> 9918</phone> <office>216</office> <office>216</office> </employee> </employee> </dept> </dept>

32 Conclusion pureml enhances the database applications into the area which may not have been practical. Semi-structured office reports / eforms is a typical use case of pureml. Schema flexibility with high performance is the key technology to reduce TCO, as no schema maintenance is required in DB side. This promotes rapid application development. pureml drives ubiquitous database applications for leveraging data