EXTRACTING DATA FROM THE WEB

Size: px

Start display at page:

Download "EXTRACTING DATA FROM THE WEB"

Joseph Cameron Houston
6 years ago
Views:

1 EXTRACTING DATA FROM THE WEB Georg Gottlob Oxford University.

2 Talk Outline Motivation: need of information extraction Logical foundations of information extraction The Lixto Visual Wrapper The Diadem Project

3 Traditional data-based decision making in enterprises. Enterprise Data ETL Data Warehouse (entrepot de données) Analytics DECISION (e.g. pricing)

4 Traditional data-based decision making in enterprises. Enterprise Data ETL Data Warehouse (entrepot de données) Analytics DECISION (e.g. pricing) But often the most relevant data are outside the company, on the Web! Online data intelligence, online market intelligence, automatic web data extraction.

5 Online Market Intelligence (OMI) (surveillance du marché) - Electronics Retailer (détaillant d électronique - composants) : market overview, 20 competitors, 200,000 products/prices - Supermarket Chain: Price comparison; must quickly react to special offers (offres spéciales), new products, - Internet Travel Agency: Gives best price guarantee, wants to detect pricing attacks, - Road Construction Company: Find new public tenders ( appels d offre ) - Hedge Fund ( fonds de placement ): Obtain recent house price changes from real-estate agent s Web pages before the weekly index is published. Anticipating the Consumer price index (index des prix à la consommation). - Governmental/Policy Making.

6 The Web corporate news pages hotel reservation airline booking sites real estate markets environmental data bookmakers jobs ebay Quarterly reports in pdf retail prices tenders blogs governmental info news etc Enterprise Data ETL Data Warehouse (entrepot de données) Analytics DECISION (e.g. pricing)

The Web corporate news pages hotel reservation airline booking sites real estate markets environmental data bookmakers jobs ebay Quarterly reports in pdf retail prices tenders blogs

7 The Web corporate news pages hotel reservation airline booking sites real estate markets environmental data bookmakers jobs ebay Quarterly reports in pdf retail prices tenders blogs governmental info news etc Automatic web data extraction Data aggregation & integration & cleaning WEB ETL Enterprise Data ETL Data Warehouse (entrepot de données) Analytics DECISION (e.g. pricing)

8 Marketing & Business Intelligence Marketing Department goulet d'étranglement Business Objects report BI Tool Oracle 9 entrepot de données

9 The Wall Problem: Make web contents accessible to electronic data processing WEB HTML pages layout Corporate edp apps structured data, Databases, XML

10 WEB HTML pages layout Corporate edp apps structured data, Databases, XML Travail aliénant

11 Web wrapping Goal: Make web contents accessible to electronic data processing WEB HTML pages layout Corporate edp apps structured data, Databases, XML

12 Web wrapping Goal: Make web contents accessible to electronic data processing WEB HTML pages layout WRAPPER (adapteur) Corporate edp apps structured data, Databases, XML Wrappers: HTML select extract annotate XML

13 Enregistrement: hierarchie de données

14 Patterns:

15 Different approaches in the past Programming (Java, Perl, WebL, SQL+...) - very complicated & boring & expensive - testing very difficult Simple Screen scrapers ( gratte-écran ) - no complex data structures extracted Wrapper induction (apprentissage d adapteurs) - requires larger amounts of sample data - precision often not satisfactory - current systems text-based (not tree-based)

16 Different approaches in the past Programming (Java, Perl, WebL, SQL+...) - very complicated & boring & expensive - testing very difficult Simple Screen scrapers - no complex data structures extracted Wrapper induction - requires larger amounts of sample data - accuracy not satisfactory in all situations - current systems text-based (not tree-based)

commercial product since 2002 Fully automated extraction - for

17 Modern Solutions Semi-automatic tool (outils) - based on solid theory - modular knowledge representation - easy to use - commercial product since 2002 Fully automated extraction - for specific application domains - extracts from 1000s of websites - current research

18 Talk Outline Motivation: need of information extraction Logical foundations of information extraction The Lixto Visual Wrapper The Diadem Project

19 Web documents are trees! HTML: Hypertext Markup Language XML: Extensible Markup Language HTML, XML: Context free* languages. Represent a document by its parse tree (arbre syntaxique).

20 HTML Content Extractor Function f: HTML Parse tree Subtrees f Leaves of subtrees are among leaves of orig. tree

21 The Essence of Web Wrapping? Functional view: Wrapper defines functions f f: Tree P (Tree) t T subtrees(t) Equivalent logical view: Wrapper defines monadic predicates P over the nodes (arbre dom) of each input document

22 A HTML page <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <body> DBAI</h1> <table border="1" cellpadding="3" cellspacing="1"> <tr> <td>georg Gottlob</td> <td>gottlob@dbai.tuwien.ac.at</td> <td>18420</td> </tr> <tr> <td>christoph Koch</td> <td>koch@dbai.tuwien.ac.at</td> td <td>18449</td> </tr> </table> </body> </html> DBAI Georg Gottlob gottlob@ Christoph Koch koch@ Georg Gottlob h1 tr td gottlob@dbai.tuwien.ac.at html body table tr td td td td koch@dbai.tuwien.ac.at Christoph Koch 18420

23 Predicate employeetable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <body> DBAI</h1> <table border="1" cellpadding="3" cellspacing="1"> <tr> <td>georg Gottlob</td> <td>gottlob@dbai.tuwien.ac.at</td> <td>18420</td> </tr> <tr> <td>christoph Koch</td> <td>koch@dbai.tuwien.ac.at</td> td <td>18449</td> </tr> </table> </body> </html> DBAI Georg Gottlob gottlob@ Christoph Koch koch@ Georg Gottlob h1 tr td gottlob@dbai.tuwien.ac.at html body table tr td td td td koch@dbai.tuwien.ac.at Christoph Koch 18420

24 Predicate employee <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <body> DBAI</h1> <table border="1" cellpadding="3" cellspacing="1"> <tr> <td>georg Gottlob</td> <td>gottlob@dbai.tuwien.ac.at</td> <td>18420</td> td </tr> <tr> <td>christoph Koch</td> <td>koch@dbai.tuwien.ac.at</td> <td>18449</td> </tr> </table> DBAI </body> </html> Georg Gottlob gottlob@ Christoph Koch koch@ Georg Gottlob h1 tr td gottlob@dbai.tuwien.ac.at html body table tr td td td td koch@dbai.tuwien.ac.at Christoph Koch 18420

25 Predicate phone <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <body> DBAI</h1> <table border="1" cellpadding="3" cellspacing="1"> <tr> <td>georg Gottlob</td> <td>gottlob@dbai.tuwien.ac.at</td> <td>18420</td> </tr> <tr> <td>christoph Koch</td> <td>koch@dbai.tuwien.ac.at</td> <td>18449</td> </tr> </table> </body> </html> DBAI Georg Gottlob gottlob@ Christoph Koch koch@ td Georg Gottlob h1 tr td gottlob@dbai.tuwien.ac.at html body table tr td td td td koch@dbai.tuwien.ac.at Christoph Koch 18420

26 Expressiveness Yardstick: MSO MSO captures exactly the essence of data extraction: - Define sets of nodes of a document Expressiveness, complexity, semantics well understood: MSO over trees: perfect logical semantics MSO over trees: high expressive power (tree automata) MSO over trees: low data complexity Drawbacks: - hard to use, no visual specification, - high query complexity (cpl. de requetes) ( bad scalability, mauvais passage à l échelle).

27 MSO on strings and trees Rich theory: Büchi: MSO = REG over strings (chaînes de caractères) Thatcher and Wright, Rabin: MSO = REG over ranked trees (arbres bornés) = tree automata Brüggemann-Klein/Wood/Murata: MSO = REG over unranked trees Neven & Schwentick: Unranked Query Automata Courcelle: MSO in LinTime on tree-like structures (treewidth <= k, data complexity)

28 Ordered Trees as finite structures html html body body h1 table firstchild h1 table tr tr tr tr nextsibling td td td td td td td td td td td td Christoph Koch Georg Gottlob label h1 () label td () Christoph Koch root() leaf() Georg Gottlob

29 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton: auxiliary state roots even subtree roots odd subtree

30 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton:

31 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton:

32 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton:

33 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton:

34 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton:

35 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton:

36 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton:

37 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton:

38 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton:

39 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton:

40 MSO over Trees Extract from a binary tree all roots of sub-trees with an odd number of leaves: S x [ S(u) & ( leaf(x) S(x)) & x,y,z (((firstchild(x,y) & nextsibling(y,z)) (S(x) (S(y) S(z))))] Tree automaton:

41 Logic heaven DB theory heaven DB programming heaven Application design heaven

42 Logic heaven MSO DB theory heaven DB programming heaven Application design heaven

43 = Logic heaven MSO DB theory heaven Monadic Datalog DB programming heaven Application design heaven

44 = Logic heaven MSO DB theory heaven Monadic Datalog DB programming heaven Elog Application design heaven

45 = Logic heaven MSO DB theory heaven Monadic Datalog DB programming heaven Application design heaven Elog Lixto Visual Wrapper

46 Logic heaven MSO DB theory heaven Monadic Datalog DB programming heaven Application design heaven = Elog Lixto Visual Wrapper Suite

47 Monadic Datalog as a Wrapping Language root entry(x) :- root(r), firstchild(r,u), label[html](u), firstchild(u,v), label[body](v), firstchild(v,w),label[table](w), firstchild(w,x), label[tr](x). entry(x):- entry(y), nextsibling(y,x). html body name(x) :- entry(e), firstchild(e, X), label[td](x). (x) :- name(n), nextsibling(n, X), label[td](x). table phone(x) :- (m), nextsibling(m, X), label[td](x). tr tr td td td td td td

48 Monadic Datalog as a Wrapping Language root entry(x) :- root(r), firstchild(r,u), label[html](u), firstchild(u,v), label[body](v), firstchild(v,w),label[table](w), firstchild(w,x), label[tr](x). entry(x):- entry(y), nextsibling(y,x). html body name(x) :- entry(e), firstchild(e, X), label[td](x). (x) :- name(n), nextsibling(n, X), label[td](x). table phone(x) :- (m), nextsibling(m, X), label[td](x). tr tr td td td td td td

49 Monadic Datalog as a Wrapping Language root entry(x) :- root(r), firstchild(r,u), label[html](u), firstchild(u,v), label[body](v), firstchild(v,w),label[table](w), firstchild(w,x), label[tr](x). entry(x):- entry(y), nextsibling(y,x). html body name(x) :- entry(e), firstchild(e, X), label[td](x). (x) :- name(n), nextsibling(n, X), label[td](x). table phone(x) :- (m), nextsibling(m, X), label[td](x). tr tr td td td td td td

50 Monadic Datalog as a Wrapping Language root entry(x) :- root(r), firstchild(r,u), label[html](u), firstchild(u,v), label[body](v), firstchild(v,w),label[table](w), firstchild(w,x), label[tr](x). entry(x):- entry(y), nextsibling(y,x). html body name(x) :- entry(e), firstchild(e, X), label[td](x). (x) :- name(n), nextsibling(n, X), label[td](x). table phone(x) :- (m), nextsibling(m, X), label[td](x). tr tr td td td td td td

51 Monadic Datalog as a Wrapping Language root entry(x) :- root(r), firstchild(r,u), label[html](u), firstchild(u,v), label[body](v), firstchild(v,w),label[table](w), firstchild(w,x), label[tr](x). entry(x):- entry(y), nextsibling(y,x). html body name(x) :- entry(e), firstchild(e, X), label[td](x). (x) :- name(n), nextsibling(n, X), label[td](x). table phone(x) :- (m), nextsibling(m, X), label[td](x). tr tr td td td td td td

52 Monadic Datalog as a Wrapping Language root entry(x) :- root(r), firstchild(r,u), label[html](u), firstchild(u,v), label[body](v), firstchild(v,w),label[table](w), firstchild(w,x), label[tr](x). entry(x):- entry(y), nextsibling(y,x). html body name(x) :- entry(e), firstchild(e, X), label[td](x). (x) :- name(n), nextsibling(n, X), label[td](x). table phone(x) :- (m), nextsibling(m, X), label[td](x). tr tr td td td td td td

53 entry(x) :- root(r), firstchild(r,u), label[html](u), firstchild(u,v), label[body](v), firstchild(v,w),label[table](w), firstchild(w,x), label[tr](x). entry(x):- entry(y), nextsibling(y,x). name(x) :- entry(e), firstchild(e, X), label[td](x). (x) :- name(n), nextsibling(n, X), label[td](x). phone(x) :- (m), nextsibling(m, X), label[td](x). root html <?xml version="1.0"?> <peopledb> <entry> <name>georg Gottlob</name> body table <phone>18420</phone> tr tr </entry> <entry> <name>christoph Koch</name> td td td td td td <phone>18449</phone> </entry> </peopledb>

54 Monadic Datalog over XML author fc paper ns fc title paperdb ns paper ns fc chandra ns merlin fc Conj. Queries Select titles of articles authored by Chandra and Merlin paper(x) root(r) & firstchild(r,x). paper(x) paper(y) & nextsibling(y,x). output(x) paper(p) & firstchild(p,a) & firstchild(a,z) & label[chandra](z) & nextsibling(z,v) & label[merlin](v) & nextsibling(a,t) & firstchild(t,x).

55 How expressive is monadic Datalog? It was known that over arbitrary structures: Monadic Datalog Π 1 -MSO Full Datalog = P (in presence of order) Theorem [G. & Koch 2002]: Over trees, monadic Datalog = MSO A unary query is definable in MSO iff it is definable via a monadic datalog program.

56 How complex is Monadic Datalog? Theorem [G. & Koch 2002]: Monadic Datalog over trees has combined complexity: O( data * query ) Query Complexity: P-complete and linear-time.

57 Proof idea: 1.) Transform datalog program + input tree in linear time into a ground propositional logic program (programme Datalog instancié) Exploit functional dependencies: nextsibling(x,y) has only a linear number of ground instances: nextsibling(n i,n j ), etc. Decouple independent atoms of rule bodies p(x) q(x) & r(y) & nextsibling(x,z) & s(z). p(x) q(x) & r & nextsibling(x,z) & s(z). r r(y). 2.) Execute ground program in linear time by using well-known algorithms: [Beeri&Bernstein][Dowling&Gallier] [Minoux]

58 = Logic heaven MSO DB theory heaven Monadic Datalog DB programming heaven Application design heaven Elog Lixto Visual Wrapper

59 item description and link to detailpage one record # of bids date next page link price info

60 ELOG [Baumgartner, Flesca, G. VLDB 01] Examples of Special predicates: subelem(s,x,path, ) before(x,y,..) after(x,y, ) property(x,attribute, Op,Value..) document(url,d) getdocumentfromhref(x,d), etc. Xpath-like expression distance tolerance,etc. Additional features: Stratified negation, string processing ontological concepts phonenumber(x) ranges: H(S,X) :- body(..)[1,5] object hierarchies

<?xml version="1.0" encoding="utf-8"?> <document> <record> <number>409449118</number> <item>98 Degrees - Notebook - New</item> <picture/> <price>2.

61 <?xml version="1.0" encoding="utf-8"?> <document> <record> <number> </number> <item>98 Degrees - Notebook - New</item> <picture/> <price>2.99</price> <currency>$</currency> <bids>-</bids> </record> <record> <number> </number> <item>notebook - Compaq Presario 1207</item> <price>730.00</price> <currency>au $</currency> [...]

62 ELOG Program for ebay pages

63 = Logic heaven MSO DB theory heaven Monadic Datalog DB programming heaven Application design heaven Elog Lixto Visual Wrapper (outil: suite logicielle)

64 Lixto Visual Wrapper Architecture Web Visual Wrapper Generator similarly structured pages Example page(s) Extraction Module Extractionprogram XML Further processing: tracking changes, delivering ( ,sms)... ( transformatio server)

65 Product Architecture Transformation Server LiXto Extraction Engine

66 SHORT DEMO

67 Talk Outline Motivation: need of information extraction Logical foundations of information extraction The Lixto Visual Wrapper The Diadem Project: Fully automatic data extraction

68 Need for Automatic Extraction Technology (2) All search engine providers need it! Many work on it. Keywords: Vertical search, object search, semantic search. Raghu Ramakrishnan, Yahoo!, March 2009: no one really has done this successfully at scale yet Alon Halevy, Google, Feb. 2009: Current technologies are not good enough yet to provide what search engines really need. [ ] any successful approach would probably need a combination of knowledge and learning.

71 The Blackbox we are constructing URL BLACKBOX Application domain with thousands of websites Application relevant Structured data (XML or RDF) To achieve this, we combine a host of annotators with a new knowledge-based approach.

72 How to achieve it? Combine existing and new low level annotators with high level AI and reasoning.

73 <table>113 <tr>115 <tr> 134 <td>119 I m interested in radiobuttons <table>124 <tr>125 <tr>126 <td>129 <td>130 Buying Renting <td>135 Maximum price <select>136 <option>137 <option>138 <td>139 <td>140 GBP EUR

74 Bottom-up (low-level) annotation [(02873,227) (03900,417)] [(02873,227) (03900,417)] Geo-Price-Searchbox ISA Monochromatic Rectangle Occurs in Georaphic query form (formulaire de requete géo.) Occurs in ISA ISA Price search facility. 105 Postcode input field.. Active map (carte active).

75 Top-down reasoning Property Search Facility Property List part-of 1 m Single Property Description Specially highlighted property

76 Bottom-up processing Top-down reasoning Phenomenology [(02873,227) (03900,417)] 105 [(02873,227) (03900,417)] Postcode input field Occurs in Georaphic search facility. ISA Monochromatic Rectangle Occurs in ISA ISA. Geo-Price-Searchbox Active map. Price search facility. Property Search Facility Property List part-of 1 m Single Property Description Specially highlighted property

Phenomenological Record Segmentation data area p a img div img a img div img a img div img p 900 860 900 500 900 set of

77 Phenomenological Record Segmentation data area p a img div img a img div img a img div img p set of uniform, non-overlapping records maximise sequence of evenly segmented (same distance pivot) minimise irregularity of records 77

78 precision recall precision recall data areas records attributes Real Estate (100 pages) precision 98 data areas records attributes recall Used Car (100 pages) (voitures d occasion) 90 price postcode location bathroom bedroom reception legal type 78

79 OPAL Form Interpretation Form Patterns Example Small set of ubiquitous patterns ranges, dates, options, etc. Ontology by instantiation 79

OPAL-TL Example Price range two successive fields in the same group

$concept_minmax<c,c M,A> { concept<c M >(N 1 ) child(n 1,G),child(N$ $concept<c>(n 1 ),N 2 @range_connector{e,d}, (A 1 A, N 2 @A 1 {d})$ concept<c M >(N 1 ) child(n ( 1,G),child(N 2,G),adjacent(N 1,N 2 ),

80 OPAL-TL Example Price range two successive fields in the same group at least one price type range connector in between TEMPLATE concept_minmax<c,c M,A> { concept<c M >(N 1 ) child(n 1,G),child(N 2,G),adjacent(N 1,N 2 ), N 2 ) N concept<c M >(N 2 ) child(n 1,G),child(N 2,G),follows(N 2,N 1 ), concept<c>(n 1 ),N (A 1 A, N 1 {d}) concept<c M >(N 1 ) child(n ( 1,G),child(N 2,G),adjacent(N 1,N 2 ), N (N ) (N 80

81 1 Precision Recall F-score UK Real Estate (100) UK Used Car (100) ICQ (98) Tel-8 (436) Dragut et al., VLDB, 2009 Airfare Auto Book Job US R.E.

82 Short Demo diadem-3min43.m4v 82

Declarative Information Extraction, Web Crawling, and Recursive Wrapping with Lixto

Declarative Information Extraction, Web Crawling, and Recursive Wrapping with Lixto Robert Baumgartner 1, Sergio Flesca 2, and Georg Gottlob 1 1 DBAI, TU Wien, Vienna, Austria {baumgart,gottlob}@dbai.tuwien.ac.at