Design flexible data models leveraging Oracle 11gR2 XML indexing capabilities

Transcription

1 Design flexible data models leveraging Oracle 11gR2 XML indexing capabilities Aris Prassinos

2 Who am I Chief Engineer MorphoTrak, SAFRAN Group Oracle ACE Director Oracle Magazine Content Management Architect of the Year (2005) Oracle XML Customer Advisory Board member Slide 2 Aris Prassinos 2011

3 The problem Flexible schema application deployed to customers with different schema needs entities and relationships (e.g. Person, Incident) well defined customized attributes for each customer custom attributes may be multi-valued e.g. Aliases, Tattoos attributes may also be added by customers over time Full spectrum of queries range, arithmetic, aggregates, free-text / fuzzy several searchable attributes per entity searchable columns may change over time OLTP access with frequent inserts / updates / deletes near-relational performance needed for queries and updates indexing should not adversely affect write performance Slide 3 Aris Prassinos 2011

4 Two critical Oracle 11gR2 technologies Binary XML compact format to avoid XML text bloat reduces parsing, validation, conversion costs streaming for fast fragment extraction delta sliding updates exploits XML Schema information if available XML Index Structured Component project commonly searched elements as columns in a pivot table all leaf data having the same parent node stored in one row of pivot table transparent query rewrite using XQuery/XPath expression matching secondary indexes created on pivot table columns Slide 4 Aris Prassinos 2011

5 The solution? CREATE TABLE Person ( id NUMBER(10) NOT NULL, xml_data XMLType NOT NULL, CONSTRAINT pk_person PRIMARY KEY (id)) XMLTYPE xml_data STORE AS BINARY XML (RETENTION); CREATE TABLE Incident ( iid NUMBER(10) NOT NULL, id NUMBER(10) NOT NULL, xml_data XMLType NOT NULL, CONSTRAINT pk_incident PRIMARY KEY (iid), CONSTRAINT fk_incident FOREIGN KEY (id) REFERENCES Person(id) ON DELETE CASCADE) XMLTYPE xml_data STORE AS BINARY XML (RETENTION); Slide 5 Aris Prassinos 2011

6 Example XML schema <Person> <Height>70</Height> <Sex>Male</Sex> <SSN> </SSN> <Aliases> <Name><FirstName>Bob</FirstName></Name> <Name><FirstName>Mark</FirstName></Name> </Aliases> <Offenses> <Offense><Type>03</Type></Offense> <Offense><Type>04</Type></Offense> </Offenses> <Tattoos> <Tattoo><Description>Dragon and sword</description></tattoo> <Tattoo><Description>Dagger through rose</description></tattoo> </Tattoos> </Person> Slide 6 Aris Prassinos 2011

7 Query and Indexing requirements Some elements (e.g. Height) searchable via numeric range queries Aliases searchable via common nicknames (e.g. Robert / Bob) Free-text Tattoo description searching Some Collections are Optional (i.e. XML tags not present in the document) Note the different syntax for indexing Mandatory (e.g. Aliases) and Optional Collections (e.g. Offenses, Tattoos) Non-searchable elements (e.g. SSN) do not need to be indexed Slide 7 Aris Prassinos 2011

8 Structured XML Index creation process Register XML Index parameter Create XML domain index Create secondary indexes Gather stats on base tables Slide 8 Aris Prassinos 2011

9 Structured XML Index creation Step 1 DBMS_XMLINDEX.registerParameter('person_param', 'GROUP person_group XMLTable Person_pivot ''/Person'' passing XML_DATA Sex VARCHAR2(10) PATH ''Sex'', Height BINARY_DOUBLE PATH ''Height'', Aliases XMLType PATH ''Aliases'' VIRTUAL XMLTable Aliases_pivot ''/Aliases/Name'' PASSING Aliases FirstName VARCHAR2(30) PATH ''FirstName'' GROUP Offenses_group XMLTable Person_pivot ''/Person'' passing XML_DATA Offenses XMLType PATH ''<Offenses>{Offenses/Offense}</Offenses>'' VIRTUAL XMLTable Offenses_pivot ''for $i in /Offenses return if (empty($i/offense)) then <Type/> else $i/offense'' PASSING Offenses OffenseType VARCHAR2(20) PATH ''Type'' GROUP Tattoos_group XMLTable Person_pivot ''/Person'' passing XML_DATA Tattoos XMLType PATH ''<Tattoos>{Tattoos/Tattoo}</Tattoos>'' VIRTUAL XMLTable Tattoos_pivot ''for $i in /Tattoos return if (empty($i/tattoo)) then <Description/> else $i/tattoo'' PASSING Tattoos TattooDescription CLOB PATH ''Description'' '); Slide 9 Aris Prassinos 2011

10 Structured XML Index creation Step 2 CREATE INDEX person_index ON person(xml_data) INDEXTYPE IS XDB.XMLINDEX PARAMETERS('PARAM person_param'); Slide 10 Aris Prassinos 2011

11 Secondary Indexes Step 3 CREATE INDEX SexHeight_index ON Person_pivot(Sex, Height); CREATE INDEX OffenseType_index ON Offenses_pivot(OffenseType); CREATE INDEX TattooDescription_index ON Tattoos_pivot(TattooDescription) INDEXTYPE IS CTXSYS.CONTEXT PARAMETERS ('transactional'); ctxload -thes -name nicknames -file nicknames.txt exec ctx_ddl.create_section_group('namegroup','basic_section_group'); exec ctx_ddl.add_ndata_section('namegroup', 'FirstName', 'FirstName'); exec ctx_ddl.create_preference('nameds', 'MULTI_COLUMN_DATASTORE'); exec ctx_ddl.set_attribute('nameds', '', 'FirstName'); exec ctx_ddl.create_preference('namewl', 'BASIC_WORDLIST'); exec ctx_ddl.set_attribute('namewl', 'NDATA_THESAURUS', 'nicknames'); CREATE INDEX AliasFirstName_index ON Aliases_pivot(FirstName) INDEXTYPE IS Ctxsys.Context PARAMETERS('datastore nameds wordlist namewl section group namegroup transactional'); Slide 11 Aris Prassinos 2011

12 XML Index pivot tables and secondary indexes Slide 12 Aris Prassinos 2011

13 Two ways to query SELECT COUNT(*) FROM Person a, XMLTable( '/Person' passing XML_DATA Sex VARCHAR2(10) PATH 'Sex' ) x WHERE x.sex = 'Male'; SELECT COUNT(DISTINCT id) FROM my_application_view WHERE Sex = 'Male'; Slide 13 Aris Prassinos 2011

14 Application View View chaining needed for Optional Collections (e.g. Offenses, Tattoos) with a purely relational schema this would be handled via outer joins XML Index query rewrite does not currently support outer joins Slide 14 Aris Prassinos 2011

15 View chaining Slide 15 Aris Prassinos 2011

16 person_view_1 CREATE OR REPLACE VIEW person_view_1 AS SELECT a.id, x.height, x.sex, XMLCast(XMLQuery('$p/Person/SSN' PASSING a.xml_data AS "p" RETURNING CONTENT) AS VARCHAR2(11)) SSN, y.firstname FROM person a, XMLTable( '/Person' passing XML_DATA Height BINARY_DOUBLE PATH 'Height', Sex VARCHAR2(10) PATH 'Sex', Aliases XMLType PATH 'Aliases' ) x, XMLTable( '/Aliases/Name' PASSING Aliases FirstName VARCHAR2(30) PATH 'FirstName' ) y; Slide 16 Aris Prassinos 2011

17 person_view_2 CREATE OR REPLACE VIEW person_view_2 AS SELECT a.id, v.height, v.sex, v.ssn, v.firstname, y.offensetype FROM person a, person_view_1 v, XMLTable( '/Person' passing XML_DATA Offenses XMLType PATH '<Offenses>{Offenses/Offense}</Offenses>' ) x, XMLTable( 'for $i in /Offenses return if (empty($i/offense)) then <Type/> else $i/offense' PASSING Offenses OffenseType VARCHAR2(20) PATH 'Type' ) y WHERE a.id = v.id; Slide 17 Aris Prassinos 2011

18 person_view CREATE OR REPLACE VIEW person_view AS SELECT a.id, v.height, v.sex, v.ssn, v.firstname, v.offensetype, y.tattoodescription FROM person a, person_view_2 v, XMLTable( '/Person' passing XML_DATA Tattoos XMLType PATH '<Tattoos>{Tattoos/Tattoo}</Tattoos>' ) x, XMLTable( 'for $i in /Tattoos return if (empty($i/tattoo)) then <Description/> else $i/tattoo' PASSING Tattoos TattooDescription VARCHAR2(255) PATH 'Description' ) y WHERE a.id = v.id; Slide 18 Aris Prassinos 2011

19 Example Application View data Male Bob 03 Dragon and sword Male Bob 03 Dagger through rose Male Bob 04 Dragon and sword Male Bob 04 Dagger through rose Male Mark 03 Dragon and sword Male Mark 03 Dagger through rose Male Mark 04 Dragon and sword Male Mark 04 Dagger through rose Slide 19 Aris Prassinos 2011

20 Example Query SELECT DISTINCT id FROM person_view WHERE Sex = 'Male' AND Height BETWEEN 65 AND 75 AND OffenseType = '03' AND CONTAINS(TattooDescription, 'fuzzy(dagger) AND fuzzy(rose)')>0 AND CONTAINS(FirstName, 'NDATA(FirstName, Robert)')>0; Slide 20 Aris Prassinos 2011

21 Example Query Execution plan 0 SELECT STATEMENT 1 HASH UNIQUE 2 NESTED LOOPS 3 HASH JOIN 4 NESTED LOOPS 5 NESTED LOOPS 6 NESTED LOOPS 7 NESTED LOOPS 8 NESTED LOOPS 9 TABLE ACCESS BY INDEX ROWID ALIASES_PIVOT 10 DOMAIN INDEX ALIASFIRSTNAME_INDEX 11 TABLE ACCESS BY INDEX ROWID PERSON_PIVOT 12 INDEX RANGE SCAN SEXHEIGHT_INDEX 13 TABLE ACCESS BY USER ROWID PERSON 14 TABLE ACCESS BY INDEX ROWID PERSON_PIVOT 15 INDEX RANGE SCAN SYS78236_78237_RID_IDX 16 INDEX RANGE SCAN SYS78236_78237_RID_IDX 17 TABLE ACCESS BY INDEX ROWID PERSON_PIVOT 18 TABLE ACCESS BY INDEX ROWID TATTOOS_PIVOT 19 DOMAIN INDEX TATTOODESCRIPTION_INDEX 20 TABLE ACCESS BY INDEX ROWID OFFENSES_PIVOT 21 INDEX RANGE SCAN OFFENSETYPE_INDEX Slide 21 Aris Prassinos 2011

22 Piecewise updates UPDATE Person SET xml_data = updatexml(xml_data, '/Person/Height/text()', '71') WHERE id = 76; UPDATE Person SET xml_data = insertchildxml(xml_data, '/Person/Offenses', 'Offense', XMLType('<Offense><Type>05</Type></Offense>')) WHERE id = 76; UPDATE Person SET xml_data = deletexml(xml_data, '/Person/Offenses/Offense[Type="03"]') WHERE id = 76; With Binary XML / SecureFiles only the relevant portions of the data get updated on disk with the exception of appendchildxml() - use insertchildxml() instead Slide 22 Aris Prassinos 2011

23 Caveats (as of ) No online index creation Hash partitioning of base table not supported ALTER INDEX is a slow operation Not possible to use hints for secondary indexes Slide 23 Aris Prassinos 2011

24 Conclusion Bridges XML flexibility and relational design benefits simplified application design relational indexing of structured islands of data master-detail relationships via multilevel chaining of XMLTable optional strong typing by combining Binary XML with XML schema Handles full spectrum of use cases without sacrificing performance relational query performance relational insert performance near-relational fragment extraction performance using Binary XML streaming efficient piecewise updates with the use of Binary XML and SecureFiles Slide 24 Aris Prassinos 2011

25 Questions - Answers Slide 25 Aris Prassinos 2011