Databases and the World Wide Web

Transcription

1 Databases and the World Wide Web Paolo Atzeni D.I.A. - Università di Roma Tre thanks to the Araneus group: G. Mecca, P. Merialdo, A. Masci, V. Crescenzi, G. Sindoni, Outline Databases and database research Web-based Information Systems The Araneus project Querying Web sites Web-site design Integration on the Web The Road Runner Project 1

2 Database technology Developed with business applications in mind (banks, accounting, personnel, inventory, reservations, ) for integrated databases but within a controlled environments (one organization or a portion of it) 3 Database technology Major achievements (Silberschatz et al and 1996) relational systems (physical independence, usability, flexibility) transaction management (reliability) distribution 4 2

3 The current trend Widen / increase / enlarge, from various perspectives new types of data more users (not only humans) more cooperating systems more diverse systems (distributed, heterogeneous, autonomous) 5 The need for cross-fertilization database technology was developed within the domain of business applications; other domains have other requirements (also very different from each other) database technology and X technology can be complementary, with potential mutual benefit basic problems have been solved; specific areas may have specific problems, for which the general solutions need not be satisfactory 6 3

4 A ten-year goal for database research The Asilomar report (Bernstein et al. Sigmod Record 1999, The information utility: make it easy for everyone to store, organize, access, and analyze the majority of human information online 7 General themes for database research Development of plug and play database systems Scalability of federated databases Integration of data and processes Integration of structured and semistrucured data 8 4

5 WIS: Web-based Information Systems Complexity of data Data-intensive WIS WIS Web-presence sites Service-oriented sites Complexity of functions 9 Database approaches to the Web bottom-up: accessing information from Web sources top-down: designing and maintaining Web sites global: integrating existing sites and offering the information through new ones 5

6 Outline Databases and database research Web-based Information Systems The Araneus project Querying Web sites Web-site design Integration on the Web The Road Runner Project The Araneus Project Started in 1996 Goals: extend database-like techniques to Web data management About 15 people involved (researchers, developers, students) Several prototype systems 12 6

7 The Araneus Approach identification of sites of interest sites are wrapped to extract information sites are navigated and data are extracted data from the sites are integrated into a single database a new site is generated from the integrated database Wrapper Layer Site Generation Layer Araneus Web Site Development System (Homer) Integration Layer Mediator Layer Wrapper Layer SQL and Java Araneus Query System (Ulixes) Wrapper Layer Araneus Wrapper Toolkit (Minerva) 13 Example Applications The Integrated Web Museum a site integrating data coming from the Uffizi, Louvre and Capodimonte Web sites The ACM Sigmod Record XML Version XML+XSL Version of an existing (and currently running) Web site The Integrated Patent Search Site allows integrated searches in patent Web sites 14 7

8 Integration of Web Sites: The Integrated Web Museum Integrates data coming from several Virtual Museums from the Web uffizi.firenze.it, louvre.fr and Capodimonte Data are re-organized: Uffizi, paintings organized by rooms Louvre, Capodimonte, works organized by collections Integrated Museum, data organized by author 15 The Integrated Web Museum DB 16 8

9 Web Site Re-Engineering: ACM Sigmod Record in XML The ACM Sigmod Record Site (acm.org/sigmod/record) a site in HTML The ACM Sigmod Record Site, XML Edition a site in XML with XSL Stylesheets

10 19 Integrated Patent Search 20 10

11 Integrated Patent Search: Result 21 Building Applications in Araneus Phase A: Reverse Engineering Existing Sites Phase B: Data Integration Phase C: Developing New Integrated Sites 22 11

12 Building Applications in Araneus Phase A: Reverse Engineering First Step: Deriving the logical structure of data in the site ADM Scheme Second Step: Wrapping pages in order to map physical HTML sources to database objects Third Step: Extracting Data from the Site by Queries and Navigation 23 Modeling Web Sites: The ARANEUS Data Model ADM - ODMG-like model - Pages are URL-identified nested objects - Heterogeneous Union-Types Page-type: describes a set of homogeneous pages Site-scheme: set of page-types connected by links 24 12

13 Proceedings Sigmod Record On-Line acm.org/sigmod HomePage Abstract & IT OrdinaryIssue about 1000 pages integrated with other data dblp.uni-trier.de 25 HomePage Sigmod Record Proceedings OrdinaryIssue Abstract & IT 26 13

14 The Integrated Web Museum 27 The Integrated Web Museum 28 14

15 Building Applications in Araneus Phase A: Reverse Engineering First Step: Deriving the logical structure of data in the site ADM Scheme Second Step: Wrapping pages in order to map physical HTML sources to database objects Third Step: Extracting Data from the Site by Queries and Navigation 29 Wrapping Web Sites: The Araneus Wrapper Toolkit The need for wrappers March, 98 Internet wrapper Date: TEXT HTML ADM Pages are often logically homogeneous but physically heterogeneous 30 15

16 The Araneus Wrapper Toolkit <table> <tr> <td><a href="9903/index.html">march 1999</a></td> <td><font size=-1>vol 28, No 1</font> <B>(Semantic Interoperability in Global Information Systems)</B></td> </tr> <tr>... </tr> </table> $OrdinaryIssues := <table>( <tr><td><a*>$date</a></td> <td><font*>vol $Volume, No $Number</font> [<b>( $Description ) </b>] </td> </tr>)+ </table> 31 The Araneus Wrapper Toolkit Advantages Procedural Languages Flexible in presence of irregularities and exceptions Allows document restructurings Grammars Concise and declarative Simple in the case of strong regularity Drawbacks Tedious even with document strongly structured Difficult to maintain Not flexible in presence of irregularities Allow for limited restructurings 32 16

17 The Araneus Wrapper Toolkit: Minerva + Editor Minerva is a Wrapper generator It couples a declarative, grammar-based approach with the flexibility of procedural programming: grammar-based procedural exception handling 33 Building Applications in Araneus Phase A: Reverse Engineering First Step: Deriving the logical structure of data in the site ADM Scheme Second Step: Wrapping pages in order to map physical HTML sources to database objects Third Step: Extracting Data from the Site by Queries and Navigation 34 17

18 Queries over Web Sites: Query Interfaces: Ulixes Example of SQL Query: Titles and Years of Papers Published by Codd at SIGMOD " CREATE VIEW TitlesOfCoddPapers (Title, Year) OVER AS SELECT ProceedingsPage.SectionList.Articles.Title, HomePage.YearList.Year FROM HomePage.YearList.NumberList.ToIsssues -> ProceedingsPage.SectionList.Articles WHERE ProceedingsPage.SectionList.Articles.Authors LIKE %Codd% 35 Queries over Web Sites: Query Interfaces: Polyphemus 36 18

19 Building Applications in Araneus Phase A: Reverse Engineering Existing Sites Phase B: Data Integration Phase C: Developing New Integrated Sites 37 Building Applications in Araneus Phase C: Web Site Development Model-Based Development in Araneus CASE-Tool Approach (Homer) 38 19

20 Model-Based Development in Araneus Clear Separation between Site Design and Site Implementation: high-level models for site design tools for site implementation Overall Goal: users should concentrate on design issues they should not write any line of code Flexibility in Site Implementation: Transparent Generation of HTML, XML (or WML) sites Independence from the actual page-generation tool 39 Model-Based Development in Araneus Homer, a CASE-Tool for Web Site Development The Araneus Site Design Methodology High-Level Models: Modeling data: relational (or object-relational) Modeling hypertext: ADM Modeling presentation: Telemachus Styles 40 20

21 Presentation Modeling: Telemachus Requirements: precise notion of style for pages platform independence rapid prototyping and flexible maintenance working with sample pages ( templates ) Telemachus Styles: Attribute Styles: formatting directives for attributes in pages Page Styles: collections of attribute styles plus header and footer 41 Telemachus: attribute styles In the database: CoursePage Course: TEXT;... Table: Courses CourseName... Operating Systems... In the hypertext: Course: [<FONT color= red > <B>] [</B></FONT>] In the page: <FONT color= red ><B> Operating Systems </B></FONT> 42 21

22 Telemachus: attribute styles Course: [<TABLE border="0" rows= 2" cols= 3"> <TR> <TD><IMG src= dotyel.gif width="30"></td> <TD><FONT color= white"><b>course Name:</B></FONT> <TD><FONT color= yellow face= Arial ><B>] [ </B></FONT></TD> </TR> <TR><TD></TD><TD COLSPAN= 2 ><HR></TD></TR> </TABLE><HR>] 43 Telemachus: page styles one attribute style for each attribute in the page header and footer Header Footer 44 22

23 Telemachus: page styles Style for CoursePage Header: [<HTML STYLE="font-size:12pt"> <BODY background= bicolor.gif"> <CENTER> <TABLE width="100%" BORDER="0"> ] Course: [<B>] [</B>] Description: [<BR><B>Description</B>] [] ToInstructor: [<BR>] [<BR>] InstructorName : [... Footer: <FONT FACE="Times New Roman" COLOR="yellow"><B>] [</B></FONT></A>] [<TABLE>... </BODY> </HTML>] Easily implemented with any tool (including XSL) Users don t want to write so much code 45 Telemachus: templates Sample pages with placeholders Users work with Templates and Telemachus generates styles from them

24 Presentation Design with Telemachus Site Template user editing Telemachus user editing Telemachus Page Templ. 1 Es: PersonPage Page Templ. 2 Es: CoursePage Page Templ. N Es:GroupPage Page Style 1 Page Style 2 Page Style n This approach is strongly based on ADM; 47 Homer: A Case Tool for Web Sites Putting All Together: different models and different activities: database design (relational or object-relational); hypertext design (ADM); presentation design (Telemachus Styles); a precise Design Methodology; Homer: a Case Tool for the Design Process: graphical interface to assist users in the design; automatic production of code for page generation

25 Homer: A Case Tool for Web Sites Homer Step 1: ER Scheme user editing DB Scheme Step 2: ADM Scheme user editing Homer Step 3: Page Templ. Site Generation Code 49 Homer: A Case Tool for Web Sites Step 1: User Draws an Entity-Relationship Scheme; 50 25

26 Homer: A Case Tool for Web Sites 51 Homer: A Case Tool for Web Sites Step 2: ER Scheme ==> ADM Scheme; 52 26

27 Homer: A Case Tool for Web Sites Step 2: ER Scheme ==> ADM Scheme; 53 Homer: A Case Tool for Web Sites Homer Engine: all intermediate models (ER, Relational...) are seen as subsets of ADM design steps are transformations (views) nested-relational algebra with URL-invention view composition Implementation: HTML, XML with XSL stylesheets (WML) Penelope, JSP,

28 After Araneus Current Research (of my group;unfortunately,not really mine): automatic data extraction Web data sources theroadrunner Project 55 RoadRunner: motivation Web: standard platform for next generation IS the largest knowledge base ever developed Intended to be browsed by humans Hard to compute over Web: the modern legacy system Tools for data extraction, integration and transformation 56 28

29 RoadRuner: the Ultimate Goal URL of a data-intensive site Es: BookTitle The HTML Sourcebook Computer Networks Database Systems Author J. Graham A. Tannenbaum R. Elmasri, S. Navathe Editor RoadRunner Data on the Web S. Abiteboul, P. Buneman, D. Suciu HTML pages Automatic Wrappers database tables (or XML docs) 57 Background Manual wrappers are largely used today (Es: shopping agents) Several Recent Works on Automatic Wrapper Generation [Kushmerick et al, IJCAI 97] WIEN: machine learning background [Adelberg SIGMOD 98] NoDoSE: real-time databases background [Embley et al, SIGMOD 99] conceptual modeling background 58 29

30 Automatic Wrapper Generators Different inspirations, different techniques, some common features Rougly speaking, in all systems: one HTML page at a time (multiple-record document) some a priori knowledge about the page organization wrapper inference is based on labeled examples provided to the system 59 Wrapper Generation from Examples FC Kaiserslautern Kaiserslautern Club Location User Interaction or Domain-Specific Heuristics 60 30

31 Originality and Contributions Goal: Automating the process to the largest extent No a priori knowledge about the structure No need for labeled examples Main Ideas: HTML pages = encodings of a database wrapper generation = decoding process by comparing homogeneous pages (similarities and differences) 61 Overall System Architecture Classifier site URL Page clusters {C 1,C 2,,C n } Expander per C i =1 Matcher per C i >1 Wrappers 62 31