Semistructured Data and Mediation

Transcription

1 Semistructured Data and Mediation Prof. Letizia Tanca Dipartimento di Elettronica e Informazione Politecnico di Milano

2 SEMISTRUCTURED DATA FOR THESE DATA THERE IS SOME FORM OF STRUCTURE, BUT IT IS NOT AS PRESCRIPTIVE REGULAR COMPLETE AS IN TRADITIONAL DBMSs EXAMPLES WEB DATA XML DATA BUT ALSO DATA DERIVED FROM THE INTEGRATION OF HETEROGENEOUS DATASOURCES 1

3 AN EXAMPLE OF SEMISTRUCTURED DATA a page produced form a database 2

4 AN EXAMPLE OF SEMISTRUCTURED DATA 3

5 AN EXAMPLE OF SEMISTRUCTURED DATA 4

6 AN EXAMPLE OF SEMISTRUCTURED DATA GRAPH-BASED REPRESENTATION: THE IRREGULAR DATA STRUCTURE APPEARS MORE CLEARLY PROFESSOR PROFESSOR NAME NAME TEACHES TEACHES AGE AGE 37 COURSE YEARS NAME? LAST NAME NAME KING ABSTRACT DATABASES OBJECTS 45 CONCRETE VALUES 5

7 A SIMPLE XML DOCUMENT WITH ITS GRAPH BASED REPRESENTATION <producer> <mn-name>mercury</mn-name> <year>1999</year> <model> <mo-name>sable LT</mo-name> <front-rating>3.84</front-rating> <side-rating>2.14</side-rating> <rank>9</rank> </model> </producer> PRODUCER mn-name year Mercury 1999 mo-name Sable LT front-rating MODEL.. side-rating rank

8 INFORMATION SEARCH IN SEMISTRUCTURED DATABASES WE WOULD LIKE TO: INTEGRATE QUERY COMPARE DATA WITH DIFFERENT STRUCTURES ALSO WITH SEMISTRUCTURED DATA, JUST AS IF THEY WERE ALL STRUCTURED 7

9 DYNAMIC INTEGRATION OF SEMISTRUCTURED DATABASES AN OVERALL DATA REPRESENTATION SHOULD BE PROGRESSIVELY BUILT, AS WE DISCOVER AND EXPLORE NEW INFORMATION SOURCES 8

10 SEMISTRUCTURED BASED ON TEXT TREES GRAPHS LABELED NODES LABELED ARCS BOTH DATA MODELS THEY ARE ALL DIFFERENT AND DO NOT LEND THEMSELVES TO EASY INTEGRATION 9

11 Recall: MEDIATORS The term mediation includes: the processing needed to make the interfaces work the knowledge structures that drive the transformations needed to transform data to information any intermediate storage that is needed (Wiederhold) 10

12 Mediator-based approach IN TSIMMIS: UNIQUE, GRAPH-BASED DATA MODEL DATA MODEL MANAGED BY THE MEDIATOR WRAPPERS FOR THE MODEL-TO- MODEL TRANSLATIONS 11

13 TSIMMIS APPLICATION 1 APPLICATION 2 (LOREL) (LOREL) (OEM) (OEM) MEDIATOR 1 MEDIATOR 2 WRAPPER 1 WRAPPER 2 WRAPPER 3 DATASOURCE 1 (RDBMS) DATASOURCE 2 (LOTUS NOTES) DATASOURCE 3 (WWW) 12

14 OEM (Object Exchange Model) (TSIMMIS) Graph-based Does not represent the schema Directly represents data : self-descriptive <temp-in-farenheit,int,80> 13

15 Object structure <Object-id,label,type,value> Nested structure 14

16 OEM (Object Exchange Model) (TSIMMIS) 15

17 Typical complications when integrating Each mediator is specialized into a certain domain (e.g. weather forecast), thus Each mediator must know domain metadata, which convey the data semantics On-line duplicate recognition and removal (no designer to solve conflicts at design time here!!) 16

18 Query formulation Find books authored by Aho e.g. Datalog-like: Booktitle(X) :- library ( book( author:aho, title:x )) OK, but if this query must be produced at run-time, how does the system know that: A node library exists, which contains nodes book, which in turn contain fields author and title TSIMMIS uses the Dataguide: a-posteriori schema, progressively built while exploring the data sources 17

19 TSIMMIS s language is LOREL Lightweight Object REpository Language Object-based Similar to OQL, with some modifications appropriate for semistructured data select library.book.title where library.book.author = Aho from library (if more than one root is available) 18

20 WRAPPERS (translators) Convert queries into queries/commands which are understandable for the specific data source they can extend the query possibilities of a data source Convert query results from the source s format to a format which is understandable for the application 19

21 WRAPPERS BookTitle Author Editor The HTML Sourcebook J. Graham... Computer Networks A. Tannenbaum... Wrapper Database Systems Data on the Web R. Elmasri, S. Navathe S. Abiteboul, P.Buneman, D.Suciu... HTML page database table(s) (or XML docs) 20

22 Example: estraction of information from Information extraction HTML docs Source Format: plain text with HTML tags (no semantics) Target Format: relational table (possibly nested, NF 2 ) or XML (we add structure, i.e. semantics ) Wrapper Software module which performs an extraction step Intuition: use extraction rules which exploit the marking tags 21

23 A complex extraction process 20-30KB IN HTML >10 attributes with nesting 22

24 Problems Web sites change very frequently A layout change may affect the extraction rules Human-based maintenance of an ad-hoc wrapper is very expensive Better: automatic wrapper generation 23

25 Automatic wrapper generation... We can only use them when pages are regular to some extent OK when: Many pages sharing the same structure e.g. pages are dynamically generated from a DB data intensive web sites 24

26 Online library Name John Smith Paul Jones Authors Id #a1 Database Primer #a2 Books Title Computer Systems Id Author XML at Work #b1 #a1 #b2 #a1 HTML and Scripts #b3 #a2 #b4 #a2 JavaScripts #b5 #a2 Description Id Name #e1 #b1 John Smith #e2 This book Paul Jones #e3 #e4 #e5 An undergraduate #e6 Title A comprehensive #e7 Database Primer Books Editions Book Editions Year Price Details Details Year Price $ First #b1 First Edition, 2000 Paperback 30$ 1998 Second 20$ #b $ First Second Edition, Hard Cover $ #b $ First #b $ null First Edition, Paperback $ #b $ Second Description #b5 First Edition, 2000 Paperback 50$ 1999 null 30$ Computer Systems null $ An useful HTML XML at Work Second Edition, Hard Cover $ HTML and Script A must in Java Script null $ Source Dataset HTML Pages 25

27 The ROAD RUNNER project Page Class It is the collection of all pages generated by the same script from a common dataset Schema Derivation Given a set of HTML sample pages, belonging to the same class, find the underlying dataset structure (database schema) Solution: Wrapper Generator Underlying dataset structure Extraction rules 26

28 B A C D F E G First Edition, 1998 Database This book John Primer Second 2000 Smith Edition, An Computer First Edition, undergraduate 1995 Input HTML Systems Sources A <HTML><BODY><TABLE> First Edition, XML at Work <HTML><BODY><TABLE> comprehensive 1999 <TR> <TR> <TD><FONT>books.com<FONT></TD> <TD><FONT>books.com<FONT></TD> null 1993 <TD><A>John Smith</A></TD> HTML and <TD><A>Paul An useful HTML Jones</A></TD> </TR> Paul Jones Second Scripts </TR> <TR> <TR> Edition, 1999 <TD><FONT>Database Primer<FONT></TD> <TD><FONT>XML at Work<FONT></TD> <TD><B>First Edition, Paperback</B></TD>, JavaScripts <TD><B>First A must in Edition, null Paperback</B></TD>, 2000 Wrapper Output H 20$ 30$ 40$ 30$ 30$ 45$ 50$ Wrapper = Grammar <HTML><BODY><TABLE> <TR> <TD><FONT>books.com<FONT></TD> <TD><A> #PCDATA</A></TD></TR> (<TR> ( <TD><FONT> #PCDATA <FONT></TD> ( <TD><B> #PCDATA </B> )? </TD> ( <TR><TD><FONT> #PCDATA <FONT> <B> #PCDATA </B> </TD></TR> )+ )+ Target Schema SET ( TUPLE (A : #PCDATA; B : SET ( TUPLE ( C : #PCDATA; D : #PCDATA; E : SET ( TUPLE ( F: #PCDATA; G: #PCDATA; 27 H: #PCDATA)))

29 Model Management approach (Atzeni, Bernstein, others ) Given two different data models (e.g. OO and relational, or XML and OO, etc.) (when datasources are at least semistructured) Define general mappings from one model into another, which allow to Map SQL schema to XML schema Map data source to data warehouse Map OO classes to data source tables, To this end, one possibility is to use a metamodel

30 METAMODEL A METAMODEL IS AN ABSTRACT MODEL FOR THE SPECIFICATION OF CONCRETE MODELS TWO TYPES OF METAMODELS: 1. GENERAL ENTITIES WHOSE SPECIALIZATIONS BECOME OBJECTS IN THE TARGET MODEL, E.G. GSMM 2. ENTITIES DESCRIBING THE OBJECTS OF THE TARGET MODEL, E.G.GEOGRAPHIC DATA FILES (GDF) 29

31 Using a metamodel for integrated data representation TRANSLATION OF DIFFERENT MODELS INTO A UNIQUE FORMALISM EASY A-PRIORI COMPARISON BETWEEN THE DIFFERENT MODEL S FEATURES AUTOMATIC TRANSLATION DICTATED BY THE REPRESENTATION RULES OF THE CONCRETE MODEL INTO THE METAMODEL 30

32 GSMM (General Semistructured Meta-Model) GRAPH-BASED DOES NOT REPRESENT THE SCHEMA: SELF- DESCRIPTIVE AS TSIMMIS GSSM REPLACES THE CONCEPT OF SCHEMA WITH THAT OF CONSTRAINT INTRODUCTION OF FLEXIBILITY IN DATA REPRESENTATION 31

33 An XML document PRODUCERS.. <producers> <producer> <mn-name>mercury</mn-name> <year>1999</year> <model> <mo-name>sable LT</mo-name> <front-rating>3.84</frontrating> <side-rating>2.14</side-rating> <rank>9</rank> </model> </producer> </producers> PRODUCER.. mn-name year MODEL Mercury 1999 rank mo-name Sable LT front-rating side-rating

34 Its representation in GSMM <PRODUCERS,root,0, >.. sub-element of <PRODUCER,element,1, > sub-element of <mn-name,element,1, Mercury> sub-element of sub-element of.. <MODEL,element,3, > <year,element,2, 1999> sub-element of sub-element of sub-element of <mo-name,element,1, SableLT> <rank,element,5, 9> <front-rating,element,2, 3.84> sub-element of <side-rating,element,4, 2.14> 33

35 GSL General Semistructured Language RULE BASED WHERE A RULE IS A COLORED GRAPH RED SOLID FOR POSITIVE PREMISES RED DASHED FOR NEGATIVE PREMISES GREEN SOLID FOR POSITIVE CONSEQUENCES RULES REPRESENT: QUERIES CONSTRAINTS (EMPTY CONSEQUENCE) 34

36 GSL QUERIES G Q 1 : Find all the B nodes which have at least a C child and link them to a node R 1 R 1 A R 2 Q 1 R 1 B C B C B D Q 2 : Find all the B nodes that do not have children C and link them to a node R 2 Q 2 R 2 B C 35

37 CONSTRAINTS IN GSL G A C 1 : Whenever a B node has a child, this is a C node B B C 1 B x This is false in the instance { x = C } C D C 2 : Each A node has at least a B child C 2 A B This is true in the instance 36

38 APPLICATION OF THE GSMM METAMODEL THE METAMODEL ALLOWS THE CONSTRUCTION OF A GENERIC GRAPH WHICH REPRESENTS AN INSTANCE OF A CONCRETE MODEL PLUS CONSTRAINTS (also represented by means of graphs): HIGH LEVEL, OR META- CONSTRAINTS these dictate the sintactic rules of the object data model LOW LEVEL CONSTRAINTS as usual, these dictate the application domain semantics IT IS A METAMODEL OF THE FIRST KIND 37

39 XML EACH NODE LABEL HAS CARDINALITY 4: n i = <Ntag i, Ntype i, Norder i, Ncontent i > Ntype i says whether the node is the root, an element, a text, Norder i represents the node s position as a child of its parent node Ncontent i may assume a value of type PCDATA or value EACH EDGE LABEL HAS CARDINALITY 1: ej = < (nh, nk), ELj > EL j = < Etype j > Etype j {attribute of, sub-element of} 38

40 CONSTRAINT REPRESENTATION IN XML <TAG1,TYPE1,ORDER1,CONTENT1> <TAG1,TYPE1,ORDER1,CONTENT1> <E-type> <TAG2,TYPE2,ORDER2,CONTENT2> <E-type> <TAG2,TYPE2,ORDER2,CONTENT2> { E-type=SubElement-of TYPE1=element (TYPE2=element TYPE2=text), E-type=Attribute-of TYPE1=element TYPE2=attribute } HIGH LEVEL, OR META-CONSTRAINT: THE TYPE OF AN ARC DEPENDS ON THE TYPE OF THE DESTINATION NODES { TYPE1=root CONTENT1= } HIGH LEVEL, OR META-CONSTRAINT: THE GRAPH ROOT HAS TYPE root AND UNDEFINED CONTENT HIGH LEVEL CONSTRAINTS CONTRIBUTE TO THE DEFINITION OF THE STRUCTURE OF ANY XML DOCUMENT 39

41 An XML document PRODUCERS.. <producers> <producer> <mn-name>mercury</mn-name> <year>1999</year> <model> <mo-name>sable LT</mo-name> <front-rating>3.84</frontrating> <side-rating>2.14</side-rating> <rank>9</rank> </model> </producer> </producers> PRODUCER.. mn-name year MODEL Mercury 1999 rank mo-name Sable LT front-rating side-rating

42 Its representation in GSMM High-level constraints satisfied sub-element of <mn-name,element,1, Mercury> <PRODUCERS,root,0, > sub-element of <PRODUCERS,element,1, > sub-element of.. sub-element of.. <MODEL,element,3, > <year,element,2, 1999> sub-element of sub-element of sub-element of <mo-name,element,1, SableLT> <rank,element,5, 9> <front-rating,element,2, 3.84> sub-element of <side-rating,element,4, 2.14> 41

43 LOW-LEVEL, OR APPLICATION-LEVEL, CONSTRAINTS <producer,element,order1,content1> <producer,element,order1,content1> <Subelement-of> <Subelement-of> <model,element, ORDER2,CONTENT2> <year,element,order2,content2> { CONTENT2>1990 } Low-level constraint: For each PRODUCER element there is at least one MODEL element Low-level constraint: YEAR must be greater than 1990 LOW-LEVEL CONSTRAINTS EXPRESS THE DOCUMENT S SEMANTICS 42

44 Geographic Data Files (GDF) GDF is a standard, used for describing roadmaps CITY STREET REGISTER STREET SIGN ARCHIVE TRAFFIC CONTROL SYSTEMS CAR NAVIGATOR SYSTEMS (GPS).. IT IS A METAMODEL OF THE SECOND KIND 43

45 GDF STRUCTURE DATASET OF 82 ASCII CHARACTERS RECORDS ENTITIES ATTRIBUTES RELATIONS 11 INTEREST THEMES ROADS AND FERRIES BRIDGES AND TUNNELS RAILROADS RIVERS PUBLIC TRANSPORTATION ADMINISTRATION AREAS.. 3 DESCRIPTION LEVELS FOR EACH THEME 44

46 GDF LEVEL 0 TOPOLOGY A PLANAR GRAPH: POINT ARC NODE POLYGON EACH ELEMENT IS UNIQUELY IDENTIFIED BY AN ID TOPOLOGICAL RULES GOVERN INTERNAL STRUCTURE AND RELATIONSHIPS AMONG ELEMENTS 45

47 GDF LEVELS LEVEL 1 ELEMENTARY ENTITIES THIS IS THE BASIS FOR THE CITY STREET REGISTER ROAD ELEMENT JUNCTION TRAFFIC AREA LEVEL 2 COMPLEX ENTITIES THIS IS THE BASIS FOR THE GEOGRAPHIC INFORMATION SYSTEMS STREET INTERSECTION 46

48 GDF LEVELS VIALE ABRUZZI CITY MAP LEVEL 2 CL234 CS102 CL203 CL235 CS103 L525 S867 S868 L321 L322 L524 S869 S870 LEVEL 1 LEVEL 0 C622 C621 L818 L811 L812 C623 L817 L819 C624 C625 C626 L813 C627 L820 C628 L814 C629 L821 L815 C630 L816 C631 47

49 META-ENTITY DEFINITION ATTRIBUTE_TYPE_ENTITY_TYPE ATTRIBUTE_TYPE 1 COMPOUND_ATTRIBUTE_TYPE ENTITY_TYPE ATTRIBUTE_TYPE_RELATION_TYPE RELATION_TYPE RELATION_TYPE_ENTITY_TYPE 48

50 GDF META-SCHEMA 49

51 AN ORTHOGONAL APPROACH: A STANDARD FOR METADATA MULTIMEDIA CONTENT DESCRIPTION INTERFACE (MPEG-7) PROVIDES A SET OF TOOLS FOR DESCRIBING MULTIMEDIA CONTENT XML-SCHEMA BASED NOT BASED ON SPECIFIC APPLICATION DOMAINS, IT ENABLES EASY EXCHANGE AND REUSE OF MULTIMEDIA CONTENTS SPEECH TEXT IMAGES TEXT IT IS APPLIED IN REAL-TIME AS WELL AS IN NON-REAL- TIME SITUATIONS: OFF-LINE CONTENT STORAGE ON-LINE CONTENT STORAGE STREAM (NO PERMANENT STORAGE) 50

52 MPEG-7 TOP LEVEL ELEMENTS Content Description (abstract) Complete Description (abstract) Content Management (abstract) Content Entity Multimedia Content (abstract) Image Video Audio Audio-visual Multimedia Content Multimedia Collection Signal Ink Content Analytic Edited Video Content Abstraction (abstract) Semantic Description Model Description Summary Description View Description Variation Description User Description Media Description Creation Description Usage Description Classification Scheme Description Da: Martinez, Koenen, Pereira 51