modern database systems lecture 4 : semi-structured data

Transcription

1 modern database systems lecture 4 : semi-structured data Aristides Gionis spring 2018

2 in the previous lectures relational model and sql storage and indexing access cost analysis hash index b+ trees indexing external sorting join algorithms query optimization

3 relational model and relational database management systems (RDBMS) strengths and weaknesses?

4 relational model and relational database management systems (RDBMS) strengths easy to understand, well-established, standardized strong theoretical foundations e.g., well-founded semantics through relational algebra standard data access language through SQL common queries are well optimized (select, project, join, aggregates) support transaction management

5 relational model and relational database management systems (RDBMS) weaknesses lack of support for complex types SQL is limited when accessing complex data rigid database schema knowledge of the data structure is required to create and populate the database knowledge of the database structure is required to query the data

6 example 1 : gather data to create a new service Larisa is an ambitious Aalto CCIS student she wants to create a new web site for discovering and recommending tasty food around the world just bought domain foodiequest.com idea : collect and analyze data from different sources tourist guide websites (reviews) food markets (ingredients) restaurants own websites address, opening hours, food type, chef, menu, social media, etc.

7 how to pull data from restaurant websites?

8

9

10 how to pull data from restaurant websites? first need to create a crawler then, need to create a parser each website has different structure how to parse the data? enter the data manually (you must be joking!) text mining / language processing take advantage of html formatting

11 how to pull data from restaurant websites? text mining / language processing examples : parse World-acclaimed chef Joan Roca to INSERT INTO restaurant_chef VALUES ( El Celler de Can Roca, Joan Roca ); parse +358 (0) to INSERT INTO restaurant_phone VALUES ( olo, +358 (0) );

12 how to pull data from restaurant websites? take advantage of html formatting example : parse <b>lunch menu</b> <ul> <li>marbled fillet of beef with oyster and broccoli</li> <li>shallots with pork and pickled rhubarb</li> <li>skrei cod with beetroot and kale</li> <li>fermented carrot with sour cream and malt</li> </ul> to INSERT INTO restaurant_menu_item VALUES ( olo, lunch, Marbled fillet of beef with oyster and broccoli ); INSERT INTO restaurant_menu_item VALUES ( olo, lunch, Shallots with pork and pickled rhubarb ); etc

13 data extraction through ad hoc parsing advantages and disadvantages?

14 data extraction through ad hoc parsing advantages no need for a priori communication with data publishers disadvantages text processing can be a difficult task a lot of information may be missed or incorrectly parsed different sites may follow very different conventions difficult to write a general-purpose parser parser can be optimized for each website, but increases considerably the amount of work prone to errors due to changes

15 the data-exchange problem data exchange without a priori agreement on the structure of the data a solution : semi-structured data self-describing or schemaless data data structure (or schema) is directly described in the data itself using a simple syntax many ways to define such a syntax LISP label-value pairs, XML, JSON, RDF,

16 label-value pairs example { name: fafa s, address: Vilhonvuorenkatu 10, Helsinki, cuisine: mediterranean, } { name: Cafe Bar No 9, address: Uudenmaakatu 9, Helsinki 00120, Finland, cuisine: burger, bar: yes } { name: olo, tel: +358(0) , cuisine: finnish, cuisine: creative, chef: Jari Vesivalo }

17 nested structure example { restaurant: { name: fafa s, address: { street: Vilhonvuorenkatu, number: 10, zip: 00500, city: Helsinki }, cuisine: mediterranean, }, restaurant: { name: Cafe Bar No 9, address: { street: Uudenmaakatu, number: 9, city: Helsinki, zip: 00120, country: Finland }, cuisine: burger, bar: yes }, restaurant: { name: olo, tel: +358(0) , cuisine: finnish, cuisine: creative, chef: Jari Vesivalo } }

18 object references example { name: fafa s, address: Vilhonvuorenkatu 10, Helsinki, menu: &m, } { name: fafa s, address: Ruusulankatu 1, Helsinki, menu: &m, } &m{ dish: falafel and hummus, dish: falafel, pesto and goat cheese, dish: falafel, feta and eggplant }

19 XML example <restaurant> <name>fafa s</name> <address>vilhonvuorenkatu 10, Helsinki</address> <cuisine>mediterranean</cuisine> </restaurant> <restaurant> <name>cafe Bar No 9</name> <address>uudenmaakatu 9, Helsinki 00120, Finland</address> <cuisine>burger</cuisine> <bar>yes</bar> </restaurant> <restaurant> <name>olo</name> <tel>+358(0) </tel> <cuisine>finnish</cuisine> <cuisine>creative</cuisine> <chef>jari Vesivalo</chef> </restaurant>

20 abstract away the syntax graph representation restaurant restaurant name name address cuisine address cuisine bar fafa s mediterranean Cafe Bar No 9 burger yes street number zip city street number zip city

21 graph representation with object references name address menu &m menu name address fafa s fafa s Vilhonvuorenkatu Helsinki dish Ruusulankatu Helsinki dish dish falafel and hummus falafel, pesto and goat cheese falafel, feta and eggplant

22 graph representation semi-structured data can be represented as a graph also called data graph directed graph edge labels labels express relations between data objects node labels values or object references

23 the data graph algorithms for accessing semi-structured data may depend on the structure of the data graph graph with a cycle and no root rooted acyclic graph tree

24 representing relational databases r1: a b c r2: a1 b1 c1 a2 b2 c2 c c2 c3 c3 d d2 d3 d4 {r1: {row : {a: a1, b: b1: c: c1}, {row : {a: a2, b: b2: c: c2} } {r2: {row : {c: c2, d: d2}, {row : {c: c3, d: d3}, {row : {c: c4, d: d4} }

25 example 2 : creating a knowledge base Amir is a Macadamia student he also has an ambitious plan: he wants to gather and store all knowledge known to humankind! Helsinki has inhabitants Joe Shuster was a co-creator of the character Superman OPS 5113 was an American navigation satellite futile? restrict to all knowledge available in the internet challenge : gather and organize the knowledge in a database why? be able to perform database-type queries : sort all cities having more than inhabitants by density list best-selling novels of American creators of comics characters

26 example 2 : creating a knowledge base challenges for Amir to materialize his plan : challenge 1 : (design) how to store the knowledge base? option 1 : use a relational database create tables, decide on attributes, keys, types, etc. proceed to crawl the internet and populate the database what are the tables, attributes, etc? designing the database schema is an impossible task option 2 : use semi-structured data crawl the internet and generate label-value pairs for each fact encountered and can be parsed data schema described in the data itself

27 example 2 : creating a knowledge base city city name founder name timezone country country population Helsinki Gustav I of Sweden Stockholm Sweden UTC+01:00 Finland

28 example 2 : creating a knowledge base challenges for Amir to materialize his plan : challenge 2 : (implementation) how to populate the knowledge base? ideas? crowdsourcing wikipedia infoboxes

29 create a knowledge base by crowdsourcing follow wikipedia model create wiki service, seed it, and ask people to contribute/curate idea used by wikidata, DBpedia, Freebase

30 create a knowledge using wikipedia infoboxes idea used by DBpedia, yago yago developed in MPI (incorporates infoboxes and wordnet)

31 existing knowledge bases in the form of semi-structured data wikidata DBpedia extract information from Wikipedia (infoboxes, categories, etc.) crowd-sourced community effort open source Freebase initially seeded from high-quality data (wikipedia, MusicBrainz, etc.) then maintained mainly by community acquired by Google in 2010 yago based on wikipedia, includes WordNet

32 schema of semi-structured data

33 schema of semi-structured data we motivated semi-structured data as schema-less however, schema can be defined semi-structured data schema description of the structure of semi-structured data describes data types, relationships, constraints, specified using a formal language often uses the syntax of the data language itself

34 restaurant semi-structured data restaurant restaurant name name address cuisine address cuisine bar fafa s mediterranean Cafe Bar No 9 burger yes street number zip city street number zip city

35 schema example { } title: restaurant, description: basic information about restaurants, type: object, properties: { name: { description: the name of the restaurant, type: string } price_range: { description: price range, type: set, values: [ $, $$, $$$, $$$$ ] } address: { description: physical address of the restaurant, type: object, properties : { street: string, number: numerical, zip: string } } required : name }

36 benefits of defining a schema why have a schema? schema can be used to ensure that data items have intended types validate that data has intended structure ensure that data satisfies certain constraints integrate (merge) hetereogeous datasets by identifying mappings between schemas of different datasets query optimization

37 schema extraction extracting schema from the data can be automated data guide : a summary of the data graph data guide requirements accurate : every path in the data occurs in the data guide concise : every path in the data guide occurs exactly once

38 the data graph restaurant restaurant name name address cuisine address cuisine bar fafa s mediterranean Cafe Bar No 9 burger yes street number zip city street number zip city

39 data guide for restaurant dataset restaurant name cuisine chef bar address menu street number zip city dish

40 querying semi-structured data

41 desirable properties of a query language expressive power it should be able to define the operations expressed by the query language compositionality the output of a query can be used as the input of another query schema-conscious defining a query language for semi-structured data the query language should be able to exploit schema, when available program manipulation simple language with clear semantics, so that queries can be easily program-generated

42 path expression a sequence of edge labels e1.e2.. en applying path expression on a data graph returns a set of nodes {vn} such that (r, l1, v1), (r, l2, v2),,(r, ln, vn) are paths on the data graph each li matches the path expression e1.e2.. en r is root

43 path expression example Garcia- Molina Ullman Widom path expression book n1 date title 2008 Database systems biblio.book. returns db biblio book n2 date Smith {Garcia-Molina, Ullman, Widom, Smith} 1999 paper title Database systems n3

44 path expression example Garcia- Molina Ullman regular expressions are used to specify path expressions book n1 date title Widom 2008 Database systems disjunction : biblio.(book paper). wild card : biblio._. db biblio book n2 date Smith Kleene closure : biblio._* paper title Database systems n3

45 path expression used by most query languages of semi-structured data e.g., in XML it is called XPath allow to access the data graph at arbitrary depth a building block of a query language, not a query itself returns a set of nodes on the data graph, not a piece of semi-structured data does not construct new nodes cannot be used to perform join we need to use a standard query language to bind together nodes returned by path expressions

46 querying semi-structured data book db book biblio paper n1 date title n2 date title n3 Garcia- Molina Ullman Widom 2008 Database systems Smith 1999 Database systems % Query q1 select : X from biblio.book. X returns {: Garcia-Molina, : Ullman, : Widom, : Smith} q1 Garcia- Molina Ullman Widom Smith

47 querying semi-structured data Garcia- Molina Ullman % Query q2 select row: X from biblio._ X where Smith in X. db biblio book book n1 date title n2 date Widom 2008 Database systems Smith 1999 returns {row:{: Smith, date: 1999, title: Database systems }, } q2 row n2 date Smith 1999 paper title n3 Database systems row title Database systems

48 querying semi-structured data Garcia- Molina what does q3 return? book n1 date title Ullman Widom 2008 Database systems % Query q3 select : Y from biblio._ X, X. Y, X.title Z where matches(.*database.*,z) db biblio book n2 date Smith 1999 all s with a publication whose title contains the word database paper title Database systems n3

49 querying semi-structured data book db book biblio paper n1 date title n2 date title n3 Garcia- Molina Ullman Widom 2008 Database systems Smith 1999 Database systems % Query q4 select row:(select : Y from X. Y) from biblio.book X returns {row:{: Garcia-Molina, : Ullman, : Widom }, row:{: Smith }} q4 row row Garcia- Molina Ullman Widom Smith

50 query optimization

51 query optimization as with RDBMS, indices are used in semi-structured data to enable fast and efficient access to data in RDBMS an index is created on an attribute to locate objects with a particular value in that attribute when querying semi-structured data, the path to the object is important, not only its values

52 query optimization example consider the following query select X from biblio._*. X or it requires to reach all objects reachable by a path select X from biblio._*.: {name: X, affiliation: Aalto } narrow the search to parts of the database containing the string Aalto

53 indexing the simplest case assume that the data graph is a tree a simplex index : also a tree that summarizes path information each node of the tree is a hash table the root index node contains all labels found in the root data node in general, the index has one node for every sequence of labels leading to a non-leaf node in the data each entry in each hash table contains a list of pointers to the corresponding nodes in the data tree

54 a simple index Garcia- Molina biblio book db book biblio paper n1 date title n2 date title n3 Ullman Widom 2008 Database systems Smith 1999 Database systems book paper date title date title consider the path expressions biblio.book biblio.paper._* biblio._*.

55 query evaluation on a tree index follow path expression on the index when reaching end of path, follow pointers to the data objects X._* will lead to follow the whole subtree below X still it may be more efficient than scanning the whole data

56 indexing semi-structured data how to generalize the previous idea (simple index) on directed acyclic graphs (dags) or general graphs? some ideas: language-equivalent nodes : nodes with the same path from the root for indexing language-equivalent nodes can be merged bisumulation : creating equivalent classes of languageequivalent nodes index graph : index on graph with merged nodes extent : list of pointers from merged nodes to actual objects the index graph is always smaller than the data graph, in many cases much smaller

57 query optimization several indexing structures are used value index : find objects with a given incoming edge label and satisfying a predicate label index : locate all parents of an object reachable via an edge with a given edge label edge index : find all parent-child object pairs connected via a given edge label path index : find all objects reachable via a given path full-text index : string-matching index

58 query optimization other considerations tree vs. graph data : tree-shaped semi-structured data are easier to index restricted vs. full regular expressions : complex regular expressions are rarely encountered restricted regular path expressions can be easier to index

59 query optimization rules of thumb from the user perspective build indexes on attributes used to query the data build indices on attributes that lead to higher selectivity

60 summary semi-structured data provide a way to exchange data without prior agreement on the data structure many variants of the main idea : XML, RDF, JSON, abstraction via the data graph model no schema is required, but useful when available path expressions are used to access objects from the data graph standard query languages are used to bind together objects accessed by path expressions query optimization significantly more complex than in RDBMS

61 sources Book: Data on the Web, Serge Abiteboul, Peter Buneman, Dan Suciu Research paper: Lore: a database management system for semistructured data, Jason McHugh, Serge Abiteboul, Roy Goldman, Dallan Quass, Jennifer Widom MongoDB manual