Model Management and Schema Mappings: Theory and Practice (Part II)

Transcription

1 IBM Research Model Management and Schema Mappings: Theory and Practice (Part II) Howard Ho IBM Almaden Research Center For VLDB07 Tutorial with Phil Bernstein, Microsoft Research 2006 IBM Corporation

2 The Integration Challenge Complex and heterogeneous environments Many different types of systems Many inter-related applications Escalating needs Variety, velocity, volume People are expensive 1

3 Outline Clio: Basic Features of a Schema Mapping System Schema Matching Schema Mapping Query Generation IBM Rational Data Architect Product MAUI: Advanced Features of a Schema Mapping System Nested Mapping Model Mapping-Based XML Transformation Engine Schema Integration Schema Evolution Clio2010: Mapping-Based Authoring of Data Flows ETL: Mapping ETL Conversion Web-Service Composition: Mapping for web-service data sources Mashups: Mapping Mashup Generation Reuse: Mapping Polymorphism Conclusions and Future Directions 2

4 Clio: A Schema Mapping System XML Schema DTD Relational User mapping Mapping Generation Wants data from S Understands T May not understand S Source schema S conforms to Logical mapping (internal) Xformation Query Generation Target schema T conforms to data Low-level mapping (SQL, SQL/XML, XQuery, XSLT and Java code) Logical mappings can be used for both target materialization or query rewriting 3

5 Major Features (and Challenges) Schemas can be arbitrarily different Element correspondences Human friendly Automatic discovery Support Nested Structures Nested Relational Model Nested Constraints Produce Correct Grouping Preserve data meaning Discover associations Use constraints & schema Create New Target Values and 4

6 Generate Transformation Queries (XQuery) <?xml version="1.0" encoding="utf-8"?> <statisticsdb> <citystatistics> <city/>, distinct ( FOR $x0 IN $doc/expensedb/grant, $x1 IN $doc/expensedb/company WHERE $x1/cid/text() = $x0/cid/text() RETURN <organization> <orgid> $x0/cid/text() </orgid>, <oname> $x1/cname/text() </oname>, distinct ( FOR $x0l1 IN $doc/expensedb/grant, $x1l1 IN $doc/expensedb/company WHERE $x1l1/cid/text() = $x0l1/cid/text() AND $x1/cname/text() = $x1l1/cname/text() AND $x0/cid/text() = $x0l1/cid/text() RETURN <funding> <fid> "Sk35(", $x0l1/amount/text(), ", ", $x1l1/cname/text(), ", ", $x0l1/cid/text(), ")" </fid>, <proj> "Sk36(", $x0l1/amount/text(), ", ", $x1l1/cname/text(), ", ", $x0l1/cid/text(), ")" </proj>, <aid> "Sk32(", $x0l1/amount/text(), ", ", $x1l1/cname/text(), ", ", $x0l1/cid/text(), ")" </aid> </funding> ) </organization> ), distinct ( FOR $x0 IN $doc/expensedb/grant, $x1 IN $doc/expensedb/company WHERE $x1/cid/text() = $x0/cid/text() RETURN <financial> <aid> "Sk32(", $x0/amount/text(), ", ", $x1/cname/text(), ", ", $x0/cid/text(), ")" </aid>, <amount> $x0/amount/text() </amount> </financial> ) </citystatistics> </statisticsdb> 5

7 Generate Transformation Scripts (XSLT) <?xml version="1.0" encoding="utf-8"?> <xsl:stylesheet version="1.0" xmlns:xsl=" <xsl:output method="xml" version="1.0" encoding="utf-8" indent="yes"/> <xsl:template match="/"> <result> <xsl:call-template name="q0"/> </result> </xsl:template> <xsl:template name="q0"> <xsl:element name="statisticsdb"> <xsl:attribute name="isroot">true</xsl:attribute> <xsl:element name="clioset"> <xsl:attribute name="id">sk_statisticsdb()</xsl:attribute> </xsl:element> </xsl:element> <xsl:for-each select="/expensedb/grant"> <xsl:variable name="x0" select="."/> <xsl:for-each select="/expensedb/company"> <xsl:variable name="x1" select="."/> <xsl:if test="$x1/cid=$x0/cid"> <xsl:element name="citystatistics"> <xsl:attribute name="inset">sk_statisticsdb()</xsl:attribute> <xsl:element name="city"/> <xsl:element name="clioset"> <xsl:attribute name="id">sk_statisticsdb_0_1(sk_statisticsdb())</xsl:attribute> </xsl:element> <xsl:element name="clioset"> <xsl:attribute name="id">sk_statisticsdb_0_2(sk_statisticsdb())</xsl:attribute> </xsl:element> </xsl:element> <xsl:element name="organization"> <xsl:attribute name="inset">sk_statisticsdb_0_1(sk_statisticsdb())</xsl:attribute> <xsl:element name="orgid"><xsl:value-of select="$x0/cid"/></xsl:element> <xsl:element name="oname"><xsl:value-of select="$x1/cname"/></xsl:element> <xsl:element name="clioset"> <xsl:attribute name="id">sk_statisticsdb_0_1_0_2(<xsl:value-of select="$x0/cid"/>, <xsl:value-of select="$x1/cname"/>, Sk_statisticsDB_0_1(Sk_statisticsDB())) </xsl:attribute> </xsl:element> </xsl:element> <xsl:element name="funding"> <xsl:attribute name="inset">sk_statisticsdb_0_1_0_2(<xsl:value-of select="$x0/cid"/>, <xsl:value-of select="$x1/cname"/>, Sk_statisticsDB_0_1(Sk_statisticsDB())) </xsl:attribute> <xsl:element name="fid"> Sk35(<xsl:value-of select="$x0/amount"/>, <xsl:value-of select="$x1/cname"/>, <xsl:value-of select="$x0/cid"/>) </xsl:element>

8 (Flat) Mapping Generation Source schema S Schema Correspondences Target schema T Mappings Source Concepts (relational views) Target Concepts (relational views) Step 1. Extraction of concepts (in each schema). Concept = one category of data that can exist in the schema Step 2. Mapping generation Enumerate all non-redundant maps between pairs of concepts [Popa, Velegrakis, Miller, Hernandez, Fagin. VLDB02] [Fagin, Kolatios, Miller, Popa. ICDT 03] [Haas, Hernandez, Ho, Popa, Roth. SIGMOD 05] 7

9 (Flat) Mapping Example m 1 maps proj to dept-projects m 1 : (p 0 in proj) (d in dept) (p in d.projects) p 0.dname = d.dname p 0.pname = p.pname m 2 : (p 0 in proj) (e 0 in p 0.emps) (d in dept) (p in d.projects) (e in d.emps) (w in e.workson) w.pid = p.pid p 0.dname = d.dname p 0.pname = p.pname e 0.ename = e.ename expression for e 0.salary = e.salary dept-empsworksonprojects The concept of project of a department m m 2 1 proj: Set [ dname pname emps: Set [ ename salary ] ] Two basic mappings (or source-to-target tgds or GLAV formulas) dept: Set [ dname budget emps: Set [ ename salary workson: Set [ pid ] ] projects: Set [ pid pname ] ] m 2 maps proj-emps to dept-emps-workson-projects The concept of project of an employee of a department 8

10 IBM Rational Data Architect Product 9

11 IBM Rational Data Architect Product Schema Matching, Schema Mapping and Query Generation Technologies from Clio Value Correspondences in the GUI Blue Lines: Confirmed by the users Gray Lines: Suggested by the schema matching algorithms Schema Matching Five different algorithms: two name-based (including thesaurus lookup) and three instance-based Users can choose Any weighted combination of the 5 schema matching algorithms Source or target One element (element/attribute or column) or a group of elements (subtree or table) The value k (for the top-k matches) The system returns the top-k matches for each element Current Release Source is relational (other IBM products support XML sources) Target can be relational (generates SQL) or XML (generates SQL/XML) Mapping is standardized within IBM, as an EMF in-memory object and as a serialized XML document 10

12 Outline Clio: Basic Features of a Schema Mapping System MAUI: Advanced Features of a Schema Mapping System Nested Mapping Model [Fuxman, Hernandez, Ho, Miller, Papotti, Popa. VLDB 06] Mapping-Based XML Transformation Engine Schema Integration Schema Evolution Clio2010: Mapping-Based Authoring of Data Flows Conclusions and Future Directions 11

13 New Nested-Mapping Engine for Clio Existing Clio engine is based on a flat mapping model Pros: easier to implement Cons: Fragmentation into many overlapping mappings Inefficiency in execution Redundancy in the output data No user-defined grouping semantics New Clio engine is based on a nested-mapping model Cons: more challenging to design and implement Pros: overcomes the above problems in the flat mapping model 12

14 (Flat) Mapping Example m 1 maps proj to dept-projects m 1 : (p 0 in proj) (d in dept) (p in d.projects) p 0.dname = d.dname p 0.pname = p.pname m 2 : (p 0 in proj) (e 0 in p 0.emps) (d in dept) (p in d.projects) (e in d.emps) (w in e.workson) w.pid = p.pid p 0.dname = d.dname p 0.pname = p.pname e 0.ename = e.ename expression for e 0.salary = e.salary dept-empsworksonprojects The concept of project of a department m m 2 1 proj: Set [ dname pname emps: Set [ ename salary ] ] Two basic mappings (or source-to-target tgds or GLAV formulas) dept: Set [ dname budget emps: Set [ ename salary workson: Set [ pid ] ] projects: Set [ pid pname ] ] m 2 maps proj-emps to dept-emps-workson-projects The concept of project of an employee of a department 13

15 Correlating Mapping Formulas m 1 : (p 0 in proj) (d in dept) (p in d.projects) p 0.dname = d.dname p 0.pname = p.pname proj tuples mapped only once m 2 : (p 0 in proj) (e 0 in p 0.emps) (d in dept) (p in d.projects) (e in d.emps) (w in e.workson) w.pid=p.pid p 0.dname = d.dname p 0.pname = p.pname e 0.ename = e.ename e 0.salary = e.salary Submapping, correlated to the parent mapping For every proj tuple, we map all employees, as a group. (Source grouping is preserved) Replace with n: (p 0 in proj) (d in dept) (p in d.projects) p 0.dname = d.dname p 0.pname = p.pname [ (e 0 in p 0.emps) (e in d.emps) (w in e.workson) w.pid=p.pid e 0.ename = e.ename e 0.salary = e.salary ] This is a nested mapping 14

16 Performance Flat query execution time / Nested query execution time KB 514 KB 312 KB Execution time for flat: 22mins Execution time for nested: 1.1s Flat query output size / Nested query output size Nesting Level 2111 KB 514 KB 312 KB Size of generated data (flat) including duplicates: 45MB Size of generated data (nested): 552KB Nesting Level The nested mapping generates much more efficient execution and less redundant data 15

17 Outline Clio: Basic Features of a Schema Mapping System MAUI: Advanced Features of a Schema Mapping System Nested Mapping Model Mapping-Based XML Transformation Engine [Jiang, Ho, Popa, Han. WWW 07] Schema Integration Schema Evolution Clio2010: Mapping-Based Authoring of Data Flows Conclusions and Future Directions 16

18 Performance in Executing Mappings Source schema S conforms to Logical mapping (internal) Query Generation Target schema T conforms to data Low-level mapping (SQL/XQuery/XSLT) Mappings are translated into general purpose query languages E.g., SQL, XQuery, XSLT Query optimization issues are left to the runtime engine of each of these languages to decide i.e., Q.O. decisions are not encoded in the queries. Idea: Execute mappings directly in our own runtime. 17

19 Mapping Execution Engine in Java Source schema S Logical mapping (internal) Target schema T conforms to data Mapping Execution Engine conforms to Motivation Grouping (on multiple levels) of transformed values. XQuery & XSLT have no specific constructs (hence algorithms) for such grouping tasks Scalability and efficiency Controllable memory resource usage Speed of transformation almost linear to input sizes Mapping-based Model high-level mapping semantics using IBM Mapping Specification Language (MSL) standard 18

20 Three-Phase Algorithm for Executing Mappings Phase 1 (Extract): tuple extraction Streaming holistic twig join Streaming and stored/indexed XML SQL queries through ODBC Relational source Phase 2 (Transform): generate an XML tree from each tuple Phase 3 (Merge): data merging A dynamic, scalable merge algorithm Hash-based vs. sort-based algorithms 19

21 XML-to-XML Comparative Results Running time in second Galax Quip MonetXQ Saxon Ours Data sizes (MB) 20

22 XML-to-XML Scalability Results Our Engine Running time (seconds DBLP data sizes (MB) 21

23 Outline Clio: Basic Features of a Schema Mapping System MAUI: Advanced Features of a Schema Mapping System Nested Mapping Model Mapping-Based XML Transformation Engine Schema Integration [Chiticariu, Hernandez, Popa, Kolaitis. VLDB 07 demo] Schema Evolution Clio2010: Mapping-Based Authoring of Data Flows Conclusions and Future Directions 22

24 Schema Integration Problem Given multiple overlapping schemas in the same domain, consolidate them into one. Output Input Source 1 Schema Source 2 Schema Schema Mappings Source n Schema Applications Provide a standard representation of the data (for unified querying, data warehousing, reference point, etc.) Metadata chaos reduction Schema integration is a hard problem Requires a lot of human interaction/feedback A series of recipes that arrive at a single integrated schema Related work [Pottinger, Bernstein. VLDB 03] [Chiticariu, Hernandez, Popa, Kolaitis. VLDB 07 demo] 23

25 Schema Integration [Chiticariu et al.] 1 User schema S 1 Schema Correspondences User schema S 2 2 Integrated schema I 1 3 Final schema I S 1 concepts S 2 concepts Integrated schema I n employee of a department B 1 B 4 dname budget dname budget ename B 3 B 2 sal address dname budget ename sal address pid pname department dname budget pid pname project of a department project of an employee of a department Step 1. Extraction of concepts from a hierarchy Step 2. Consider all possible ways of performing a hierarchical merge of the related concepts Generate initial set of candidate good integrated schemas Duplication-free enumeration algorithm Avoids duplicates by using constraints (Horn clauses) Polynomial-delay algorithm for enumerating satisfying assignments Step 3. Browse/Search/Refine set of candidate schemas Combination of partial enumeration with user constraints Based on the schemas seen so far, users express constraints on the future schemas to be generated 24

26 25

27 Outline Clio: Basic Features of a Schema Mapping System MAUI: Advanced Features of a Schema Mapping System Nested Mapping Model Mapping-Based XML Transformation Engine Schema Integration Schema Evolution Clio2010: Mapping-Based Authoring of Data Flows Conclusions and Future Directions 26

28 Schema Evolution The mapping Source Schema SS M ST Adapted Derived MTT T Target Schema Diff. Script Given or discovered T evolves Mapping Composition M ST = M ST M TT T When the target schema evolves, the system can generate an initial default mapping for T T automatically The user adds new mapping lines for new schema elements in T to complete the mapping M TT (for T T ) The new mapping M ST (S T ) is a composition of M ST (S T) and M TT (T T ) 27

29 Schema Evolution S evolves SS M ST T Diff. Script Given or discovered Derived MS S S S Adapted M S T = M S S M ST Mapping Composition When the source schema evolves, the system can generate an initial default mapping for S S automatically The user adds new mapping lines for new schema elements in S to complete the mapping M S S (for S S) The new mapping M S T (S T) is a composition of M S S (S S) and M ST (S T) 28

30 Related Theory and Algorithms Mapping Composition [Madhavan, Halevy. VLDB 03] [Fagin, Kolaitis, Popa, Tan. PODS 04] [Nash, Bernstein, Melnik. PODS 05] [Bernstein, Green, Melnik, Nash. VLDB 06] Mapping Inversion [Fagin. PODS 06] [Fagin, Kolaitis, Popa, Tan. PODS 07] Schema Evolution [Rahm, Bernstein. SIGMOD Rec. Dec 06] Mapping Adaptation under Evolving Schemas [Velegrakis, Miller, Popa. VLDB 03] [Yu, Popa. VLDB 05] Query rewrite [Yu, Popa. SIGMOD 04] 29

31 Outline Clio: Basic Features of a Schema Mapping System MAUI: Advanced Features of a Schema Mapping System Clio2010: Mapping-Based Authoring of Data Flows ETL: Mapping ETL Conversion Web-Service Composition: Mapping for web-service data sources Mashups: Mapping Mashup Generation Reuse: Mapping Polymorphism Conclusions and Future Directions 30

32 Mapping-Based Authoring of Data Flows Motivation Schema mappings are the building blocks for larger data transformation and integration applications The graph of mappings is a declarative specification of the flow of data Similar to ETL & mashups Need to take functional (web-service) data sources Need to extend Clio where multiple mappings can be defined, loaded (sharing a context), reused and managed 31

33 Clio2010 Clio2010 Automatically assemble a graph of initial, uncorrelated mappings into larger, richer mappings Support more complicated mapping composition and mapping merge Support functional data sources (e.g., web services) Compile the larger mappings into a global execution plan in Queries (XQuery) transformation scripts (XSLT) ETL flows (e.g., IBM WebSphere DataStage) mashups (e.g., IBM DAMIA) Support mapping reuse through mapping polymorphism framework 32

34 Outline Clio: Basic Features of a Schema Mapping System MAUI: Advanced Features of a Schema Mapping System Clio2010: Mapping-Based Authoring of Data Flows ETL: Mapping ETL Conversion [Hernandez et al. 07] Web-Service Composition: Mapping for web-service data sources Mashups: Mapping Mashup Generation Reuse: Mapping Polymorphism Conclusions and Future Directions 33

35 Orchid: Mapping ETL Conversion Round-tripping between ETL (Extract, Transform, Load) scripts and declarative mappings Generate ETL script from mappings Extract mapping information from ETL scripts Track and propagate mapping-related modifications Optimization of ETL scripts Remove redundant operations Push computation to other runtimes Rewrite ETL scripts EII Mapping in RDA ETL Flow Orchid 34

36 Mapping ETL Generation The developer inspects the connections between the sources, rules, and data warehouse model and generates a DataStage job to move data 35

37 ETL Mapping Abstraction The developer refines, tests and deploys the DataStage job to production. Need ETL Mapping: To reflect changes made in the ETL script back in the mapping. To extract mappings out of ETL scripts. 36

38 Outline Clio: Basic Features of a Schema Mapping System MAUI: Advanced Features of a Schema Mapping System Clio2010: Mapping-Based Authoring of Data Flows ETL: Mapping ETL Conversion Web-Service Composition: Mappings for web-service data sources [Alexe et al. 07] Mashups: Mapping Mashup Generation Reuse: Mapping Polymorphism Conclusions and Future Directions 37

39 Flows of Mappings getibmhotels : [ state, city ] Set [ hotel, geo, country, airport ] Source 1 Source 2 M 2 M 1 gethotelreviews : [ hotel, state, city ] Set [ image rooms rating reviews: Set [ review ] ] Target root: Set [ state, city, hotel, ratings: Set [ rating reviews: Set [ review ] ] M 3 38

40 Functional Data Sources Web Service Definition Argument Type Result Type Mapping between functional sources 39

41 Outline Clio: Basic Features of a Schema Mapping System MAUI: Advanced Features of a Schema Mapping System Clio2010: Mapping-Based Authoring of Data Flows ETL: Mapping ETL Conversion Web-Service Composition: Mapping for web-service data sources Mashups: Mapping Mashup Generation [Camacho et al. 07] Reuse: Mapping Polymorphism Conclusions and Future Directions 40

42 IBM DAMIA: A Mashup Tool IBM DAMIA Produces easily customizable flows Mashup fabric for data aggregation and transformation Web-based tool with userfriendly interface No scripting/programming skills required XQuery-like model 41

43 Generate DAMIA Flows from Clio Clio High level mappings Compiler SQL XSLT XQuery DAMIA flow 42

44 From XQuery to ETL The XQuery has a specific flow pattern, with several phases: Source extraction queries (projection, navigation, join) Result: sets of flat tuples Key generation Union Duplicate elimination Computation of target groups Generate target output (hierarchy) Back to hierarchical data We can visualize all this as a graph of operators we get the equivalent of an ETL job for XML 43

45 Doc 1 Student2, Doc 2 Student1 courseeval /Student1/* De-dup /Student2/* /courseeval/* // // // XML sid relational name sid π course π ûü eval_key=eval_key name grade sid sid name name eid = Key π π cname Key to Sk1( ) Eval_2 file Course_2 = F_course(sid, name) Key Key eid = Key Sk2( ) to Pid_0 = F_course ( ) Key Course_2 Eval_2 value_key π Pid_0 Key cname eid Key Pid_0 = F_course ( ) value_key Pid_0 π cname Key eid value_key Key value_key Course_1 Student_0 Eval_2 De-dup value_key Course_2 = Pid_0 value_key Nest-join Group-by Pid_0 concatenate Student, Course, Eval De-dup value_key relational XML... 44

46 Outline Clio: Basic Features of a Schema Mapping System MAUI: Advanced Features of a Schema Mapping System Clio2010: Mapping-Based Authoring of Data Flows ETL: Mapping ETL Conversion Web-Service Composition: Mapping for web-service data sources Mashups: Mapping Mashup Generation Reuse: Mapping Polymorphism [Wisnesky et al. 07] Conclusions and Future Directions 45

47 Mapping Reuse Motivaion: Existing mapping formalisms implicitly contain information that we need to be explicit Example: A Clio nested mapping expression implicitly defines a class of schemas for which it has an interpretation But in Clio mapping representation (XSML), every mapping instance requires concrete source and target schemas The flexibility to re-use mappings is lost Approach: We need a formal language for talking about mappings Require theory and tools to manipulate mappings as blocks Solution: Mapping Polymorphism 46

48 Clio Language Embedding into MF The bijection respects mapping semantics. Terms of Polymorphic Mapping Type Instantiation with concrete types Terms of Mapping Type Bijection Clio Nested Mapping Language Terms in MF MF also includes useful terms that manipulate mappings. 47

49 Outline Clio: Basic Features of a Schema Mapping System MAUI: Advanced Features of a Schema Mapping System Clio2010: Mapping-Based Authoring of Data Flows Conclusion and Related Work 48

50 Clio Innovations Over Time Product component Conference 99 SQL Assist Chocolate VLDB 2002 Cinnamon (CM 8.3) SIGMOD 2004 VLDB 2005 Criollo (RDA) SIGMOD VLDB 2001 & ICDE PODS SIGMOD 2000 demo demo VLDB 2006 Join-path generation Attribute matching XML support XSLT generation Deep union Mapping composition Schema integration SQL generation Clio research component XQuery generation Data viewer Hybrid algorithm SQL/XML generation Mapping language XQuery rewrite Schema evolution Java runtime engine Java code generation Nested mapping engine Flow mapping Web services 49

51 Conclusion Mappings define relationships between schemas of data sources Mappings and transformation queries are the new metadata Customers want I (for integration), not EAI, EII, ETL, etc. Consumability for information integration 50

52 Related Work: Very Small Subset Information Integration Survey [Haas ICDT 07] Beauty and the Beast: The Theory and Practice of Information Integration [Kolatis PODS 05] Schema mappings, data exchange, and metadata management [Halevy, Rajaraman, Ordille VLDB 06] Data Integration: The Teenage Years as part of their 10-year Best Paper Award on Information Manifold paper Schema Matching Survey by Erhard Rahm and Philip Bernstein in VLDB J. 01 Much work by Philip Bernstein, Anhai Doan, Alon Halevy, Jayant Madhavan Information Discovery project at IBM Almaden (Berthold Reinwald et al.) Data cleansing, conflict resolution, data quality, duplicate detection, entity resolution, data provenance Orchestra project at U Penn Schema Mapping Debugger [Alexe, Chiticariu, Tan. VLDB 06] A Benchmark for Schema Mapping System [Alexe, Tan, Velegrakis. 07] A new schema mapping GUI based on XQBE ideas [Ceri, Hernandez, Raffio et al. 07] Deep Web, peer-to-peer, ontology, etc 51

53 Acknowledgements: Clio Research Team The Project Founders: Laura Haas and Renee Miller (around 1999) Regulars: Clio Group: Mauricio Hernandez, Howard Ho, Lucian Popa, Ioana Stanoi Theory Group: Ron Fagin, Phokion Kolaitis, Alan Nash Past Visiting scientists: Stefan Dessloch, Paolo Papotti, Wang-Chiew Tan, Yannis Velegrakis, Cathy Wyss Past Interns and Post-docs: Bogdan Alexe, Alfredo Camacho, Laura Chiticariu, Ariel Fuxman, Michael Gubanov, Haifeng Jiang, Felix Naumann, Paolo Papotti, Ahmed Radwan, Alessandro Raffio, Michael Richmond, Shivkumar Shivaji, Yannis Velegrakis, Melanie Weis, Ryan Wisnesky, Lingling Yan, Cong Yu, Jindan Zhou 52