Extracting semistructured data from the Web: An XQuery Based Approach

Transcription

1 EurAsia-ICT 2002, Shiraz-Iran, Oct. Extracting semistructured data from the Web: An XQuery Based Approach Gies Nachouki Université de Nantes - Facuté des Sciences, IRIN, 2, rue de a Houssinière, BP Nantes cedex 03 nachouki@irin.univ-nantes.fr Mohamed Quafafou Université de Nantes - Facuté des Sciences, IRIN, 2, rue de a Houssinière, BP Nantes cedex 03 quafafou@irin.univ-nantes.fr Abstract This paper describes work in progress concerning the extraction of information from the web. This work is a part of frameworks consisting to extract, interconnect and access heterogeneous data sources. In this paper, we present a new approach for information extraction from the web. In this approach the web is viewed as a arge database containing XML documents. The XQuery anguage is used in order to extract information from this database. An experimenta too has been deveoped in order to vaidate this proposa. 1. Introduction With the growth of the web, many data sources have been made accessibe through the web. These go from search engines, onine product cataogues, to bibiographic istings. The potentia use of such information sources has ead to the research on how to encapsuate such sources to make them avaiabe to computer programs. This research has ead to the introduction of programs caed wrappers. Faced to the constant evoution of the web these wrappers need to be maintained and renewed often and since each Htm page has its own structure then an infinite number of wrappers must be generated. To face this probem it become interesting to think about another approach more suitabe for the web environment. In this paper, we present a new approach which don t need wrappers generation for each usefu HTML page but needs ony to convert and fiter non-usefu information in this page. The information extraction is based in our proposa on the XQuery anguage. Section 2 presents some reated work in this domain. Section 3 introduces our approach. Section 4 presents the XQuery anguage. Section 5 shows some resuts given by the experimenta too. 2. Reated Work Many researchers have tacked probems reated to information extraction from the web. Those researches have ead to the introduction of programs caed wrappers. A wrapper for the web can be seen as a procedure aowing to extract structured data contained in a set of party but simiary structured web pages[6]. In the construction of wrappers, three main points of view can be find in the iterature: Most wrapper construction methods are automatic or semi-automatic based on inductive earning [6,5]. Structure discovery or extraction can aso be used to resove the wrapper generation probem [3]. When considering the probem of reation extraction or mining, an approach is to consider that their exists a duaity between patterns and reations [2,7]. Many other wrapper induction aso exist such as supervised interactive wrapper generation [1],buiding knowedged based wrappers (JEDI [4]). The originaity of our approach is that we doesn t need to buid any wrappers to extract information from the web Page 511

2 3. Information extraction in our approach The Extraction of information from the web foows two steps (figure 1): (1) data fiter and conversion; (2) data extraction. The first step consists to convert an Htm page to an XHTML document after fiter of some non-usefu tags. The aim of the XHTML document is to permit to query an HTML document using XQuery anguage. The fiter process permits to reduce the current document by regrouping together some reated information over the same tag. For exampe consider the foowing fragment of an Htm page: <TD aign=center><font Size=+2 Coor=red> <B> Semistructured data and XML </B> </Font> </TD> This fragment is reduced as foows: <TD>Semistructured data and XML</TD>. This step shows that the web can be considered as a arge XML database containing XML documents and information extraction from this database can be performed using XML query anguage e.g. XQuery anguage. In order to iustrate the first step we consider the foowing exampe. Exampe 1. Consider the foowing HTML page given by the URL: com. This page gives information about Hotes in Nantes city (name, address, teephone number, ). This URL is obtained by a user interface (not shown in this paper). Users specify a key word through this interface (in our exampe the key word is Hotes Nantes guide and this interface returns, based on a searcher engine (Googe engine searcher), a ist of addresses that contains this key word. For each seected URL (by user) the system returns the corresponding XHTML document. In the iterature a ot of software are deveoped in order to convert HTML pages to XHTML documents (for exampe, the Tidy software which can be found at the foowing address: In our exampe the XHTML document corresponding to the URL is given in figure 2 as a tree document: Input: HTML pages returned by an engine searcher Fiter / Conversion XHTML documents Step1 Output: XML document we formed and vaid Extractor User queries Step2 Figure1. XQuery approach for extracting semistructured data from HTML pages The second step consists to extract usefu information. The process of data extraction from an XHTML document is based on XQuery anguage: a user formuates his query and the extractor processes it and returns to the user information about it in an XML document. A view of the content of each XHTML document (tree view) is provided in order to hep users to determine the path to access data in this document. A generator interface is provided in order to generate users queries (not shown in this paper). Figure 2. A part of the XHTML document after conversion After conversion of the HTML page to an XHTML document, we begin the phase consisting to fiter some non-usefu attributes and eements from the XHTML document. The foowing tabe (tabe1) gives a part of the ist of eements and attributes containing information supposed non-usefu for users. Tabe 1. A part of Non-usefu eements and attributes in an XHTML document Eements Attributes From Bgcoor img Aign Br Width Hr Name A Stye Page 512

3 Script Meta stye Map Link object At Border Cospan Cespacing Cepadding Height The name of usefu nodes in the XHTML document (after fiter) is not important for users because nosemantic is assigned to them in the rea word. So we have chosen to give them the same tag name. Figure 2 gives the fina XHTML document after fitering and renaming nodes. XQuery Object Set of casses that represents the basic type of the anguage Xpath Set of casses using XPath functionaity incuded in XQuery Functions Set of casses which impements XQuery functions Expressions Contains casses that describe a the XQuery expressions Xutis Set of utiity casses 4.2. XQuery Parser The XQuery Parser is deveoped for a exica and syntactic anaysis of XQuery queries. XQuery parser is generated using JavaCC [8] (Java Compier Compier deveoped by SUN) by projection of the XQuery Grammar on JavaCC utiity Queries Processing Figure 3. A part of the fina XHTML document after conversion, fitering and renaming. The second step of our approach is based on XQuery anguage. We present in the foowing section, the XQuery modeing, parser and processing. 4. XQuery anguage XQuery is designed by the W3C XML Query Working Group. XQuery is a query anguage for XML XQuery modeing This section shows an UML schema modeing XQuery anguage. This schema is impemented using Java programming anguage and is composed of six packages (tabe2): Mode, Object, XPath, Functions, Expressions and XUti. Tabe 2. XQuery modeing (UML packages) Packages Description Mode This package describes the data mode of This section provides an agorithm for processing XQuery queries. The input of this agorithm is a query compiant with XQuery specification and the output of this agorithm is an XML document that represents the resut of this query. The agorithm is composed of two principa steps: anaysis and evauation. The anaysis step checks the correctness of a query and returns an XQuery expression. The next step consists of evauating the returned expression by the parser in order to obtain the fina resut. Agorithm : XQuery anaysis and evauation Input: An XQuery query (Xq): <Resuts> For (or Let) Expression1 Where Expression2 Return Expression3 Output: An XML document corresponding to the resut of the query Step1 (Anaysis): This step consists to anaysis the query Xq given in input to the parser. It returns an XQuery expression Expr instance of the cass XqueryExpression (defined in the package Expressions): Expr = Parser.Start (Xq ) Step2 (Evauation): This step consists to evauate Expr returned by the XQuery parser. The evauation of this expression is reaised by the foowing instruction: Page 513

4 Resut = Expr.Eva(context). We distinguish severa substeps: Step2,1: This step evauates the For and Let causes in Expression1 of the query. Users can formuate each of those two causes as foows: (M1): For (or Let) $X in (:=) document( name )/path or (M2): For (or Let) $X in (:=) document( name )/path [Predicate] Foreach (For or Let cause expression) do Switch (Expr) Case 1 :ForCause Creation of a ist L1 where, L1= [(X,node1), (X,node2), ] containing a the nodes referred by the cause For (and satisfying Predicate in the case of M2). X is a variabe containing, for each iteration, one node; Case 2 :LetCause Creation of a ist L2 where, L2 contains the ony node returned by the cause LET (and satisfying Predicate in the case M2). End Switch EndFor Step2,2: The cause Where is optiona in XQuery anguage. Predicates in Expression2 of this cause consit of one of the foowing expressions (ist not exhaustive) : XEquaExpr XessExpr XgreaterExpr where XEquaExpr : E1 = E2, XessExpr : E1 < E2, XgreaterExpr : E1 > E2 and E1 = $X/path1; E2 = $Y/path2 The evauation of Expression2 foows the two foowing steps: - Creation of a ist L, L = L1Χ L2 (Χ is a product Cartesian), L1 and L2 are two ists obtained from step2,1 - Restriction of L to the eements which respect the predicates in the cause Where Step2,3: this step constructs new eements incuding vaues corresponding to the resut of the query as specified in the Expression3 of the cause Return. Two cases are distinguished: Case 1: Expression3 references to the nodes in the ist L (L1 or L2). The evauation consists to return the vaues of these nodes in the new buiding eement; Case 2: Expression3 is an expression of type Expression1 In this case step21 is caed for the evauation of this expression. In order to iustrate the second step of our approach we consider the foowing exampe. Exampe 2. Consider the XHTML document given in figure2. The structure of this document is represented in figure 3 as DOM tree document. The foowing query permits to seect a the hotes given in this page. <Resuts>For $a in document("hotes")/htm/body/// <Hote>$a</Hote> head tite Hote 1 htm Fi gure 3. DOM tree of the XHTML hotes Nantes guide Hotes is the name of the XHTML document. The path htm/body/// is the path to a node containing subnodes where each one describes one hote. 5. Experimenta resuts body We have deveoped an experimenta too in order to vaidate this proposa. The main features of this too are to extract, interconnect and access heterogeneous data sources. We use this system in this paper in order to show ony how to extract and query one (or severa) HTML web page(s). Users non-famiiar with XQuery anguage can use a generator interface in order to generate queries. Hote n Page 514

5 In this section we show two exampes. The first one seects information from one HTML page and the second one joins between two HTML pages. Queries are generated using the generator interface. The first exampe seects from the HTML page (URL: /index. a.htm) a the conferences incuding a database domain. The XQuery query is given as foows. <Resuts> FOR $a IN document("fitered.xm")/htm/body// where contains($a,"database") <Conferences>$a/text()</Conferences> Figure 5. Resut of the query: Restaurant and Hote in the route de Vannes street in Nantes City 6. Concusion Figure 4. Resut of the query: Database Conference from DBLP conferences HTML page. The second exampe seects from the HTML pages (URLs: fies/hote_resto/ resutat.asp?fichier=resto&s=-2, com ) the hotes and the Restaurants which are in the Route de Vannes street. The XQuery query is given as foows. <Resuts> For $a in document("hotes")/htm/body/// where contains($a,"route de Vannes") <Hote>$b</Hote>, For $b in document("restaurants")/htm/body////////////// where contains($b,"route de Vannes") <Restaurant>$b</Restaurant> In this paper, we have introduced a new approach to information extraction from the web. This method is based on XQuery anguage. In our proposa, the web is viewed as a database containing XML documents. Users can retrieve information from one or severa HTML pages using this anguage. Many research works can foow this work, such as trying to extract mutimedia information. 7. References [1] Baumgartner R et a Visua Web Information Extraction with LIXTO Proc of the 27 th VLDB Conf in ecture notes in Computer Sciences Springer Verag2000 [2] Brin S Extracting patterns and reations from the word wide web in ecture notes in Computer Sciences vo 1590 Vaencia Spain Springer 1998 [3] Chang CH et a Iepad: Information extraction based on pattern discovery in Proc of the ACM WWW10 Conf- ACM Press New York 2001 [4] Gardarin et a XML based component for federated mutipe heterogeneous data sources in ecture notes in Computer Sciences vo 1728 Springer Verag Paris 1999 [5] Hsu C Initia resuts on wrapping semi structured web pages with finite state transducers and contextua rues in Papers from the 1998 workshop on AI and Information Integration AAAI Press 1998 [6] Kushmeric N et a Wrapper induction for information extraction in Int Joint Conference on Artificia Inteigence 1997 [7] Sundaresan N et a Mining the web for reations in Proc of the ACM WWW9 Conf Esevier Science 2000 Page 515