W3QS: A Query System for the World-Wide Web

Transcription

1 W3QS: A Query System for the Word-Wide Web David Konopnicki konop&s.technion.ac.il Computer Science Department Technion, Haifa, Israe Oded Shmuei oshmu@cs.technion.ac.il Computer Science Department Technion, Haifa, 32000, Israe Abstract The Word-Wide Web (WWW) is an ever growing, distributed, non-administered, goba information resource. It resides on the wordwide computer network and aows access to heterogeneous information: text, image, video, sound and graphic data. Currenty, this weath of information is difficut to mine. One can either manuay, sowy and tediousy navigate through the WWW or utiize indexes and ibraries which are buit by automatic search engines (caed knowbots or robots). We have designed and are now impementing a high eve SQL-ike anguage to support effective and fexibe query processing, which addresses the structure and content of WWW nodes and their varied sorts of data. Query resuts are intuitivey presented and continuousy maintained when desired. The anguage itsef integrates new utiities and existing Unix toos (e.g. grep, awk). The impementation strategy is to empoy existing WWW browsers and Unix toos to the extent possibe. 1 Introduction The WWW was started to faciitate the sharing of data of various formats by physicists at CERN. The WWW supports pre-existing services (e.g. ftp) and many data formats (e.g. GIF for images and MPEG for movies). The WWW is organized as a set of HTTP (Hypertext Transmission Protoco) servers, where HTTP is a network protoco. A hypertext fie format, HTML (Hypertext Markup Language), is used to construct inks between documents, supporting a hypertext data orga.nization. Fies and services are identified over the Permission to copy without fee a or part of this materia is granted provided that the copies are not made or distributed for direct COmmercia advantage, the VLDB copyright notice and the tite Of the pubication and its date appear, and notice is given that copying is by permission of the Very Large Data Base Endowment. To COPY otherwise, or to repubish, requires a fee and/or specia permission from the Endowment. Proceedings of the 21st VLDB Conference Zurich, Switzerand, 1995 network using URLs (Universa Resource Locator). One of the popuar toos for browsing the WWW is Mosaic which was deveoped by NCSA. Mosaic is a WWW cient. Mosaic presents a point and cick graphica user interface. Through Mosaic one may navigate through the WWW and obtain access to various services. Mosaic is basicay a WWW browser and other browsers exist, e.g. Lynx which supports a vto0 termina interface, perwww written in Per, and the increasingy popuar Netscape[S], a commercia browser which aso has a pubic domain version. The WWW can be viewed as a gigantic database (mosty read-ony). The browsers enabe roaming through sites of interest, however they are not query processors. In order to get a piece of information one basicay needs to know where the data is ocated. To faciitate the search, there are certain indexes that are maintained over the WWW (no centra contro). These indexes are constructed by robots (e.g. Lycos[3], WWWW [], WebCrawer [14]) that) occasionay scan the WWW and construct indexes of interesting keywords. These indexes can be usefu in ocating information by using the browsers or other automatic toos. Some indexes may contain abstracting information in addition to keywords. Certain robots are used to maintain road maps to the WWW and others speciaize in certain topics, e.g. mathematics. Sti, there is no high eve query anguage for ocating, fitering and presenting WWW-hed information. In fact, the situation right now is anaogous to that of a huge fie system, or a document retrieva system, with many usefu indexes but without a convenient faciity for querying this information. One is thus forced to retrieve information manuay through browsing and indexes, or write specia purpose programs to obtain specific pieces of information. Our main goa is to design and construct a high eve querying and dispay faciity for the WWW. This incudes: 1. Specifying the syntax and semantics of a high eve SQL-ike query anguage. 54

2 2. Providing as part of the anguage, a simpe interface to user written programs and Unix utiities Providing advanced dispay faciities for gathered information, incuding specia graphics, HTML browser presentation (e.g. Mosaic), and Unix directory tree representation. Providing a view maintenance faciity at a much higher eve than robot maintained indexes. At the present time, we have competed the design of the W3QL query anguage. The buk of this paper is devoted to presenting the anguage and its capabiities. A prototype system wi be constructed in the coming months. In the impementation, the foowing aspects wi be emphasized: 1. Utiizing standard Unix services (e.g., fie, grep, awk) for identification, fitering and formatting. This way, the amount of software which is written from scratch wi be reduced. 2. Utiizing WWW robot constructed indexes and avaiabe ibraries for optimizing query processing. Reated work Due to the space imitations, we ony briefy mention works that are very cosey reated to this paper. Improvements to hypertext management, such as query processing and views, were proposed in [8] and [2]. Anaysis of fies that have a strict inner-structure is treated in []. One can view the parsing, pus query evauation, of [] as a possibe component of the condition ibrary and the unparsing as a possibe component of the formatting ibrary (these ibraries are components of W3QS). S evera proposas for structurespecifying queries exist. The Gruphog graphic anguage is introduced in [4] to specify searched patterns. Appication of the Graphog query anguage to different systems, such as NoteCards, gibis and HAM are given in [4]. In [2], Beeri and Kornatzky define a ogic-based anguage to state structure-specifying queries on a hypertext structure. Their formuae uniformy treat structura aspects, content aspects and booean operations. Unike [a], we separate structure and content specifications and concentrate ony on forward inks; we do not ater the goba hypertext structure. A query anguage on a dynamicay changing hypertext structure is proposed in [12], but this anguage requires an understanding of the interna organization of each hypertext node. Robot indexing initiatives are described in [] [14]. An approach to information mining in the WWW is proposed in [5], but without defining a muti-purpose query anguage. Paper organization The paper is organized as foows. In section 2 we give an overview of the WWW and expain the difficuties in searching for data in the WWW. In section 3 we present the principes of WSQL, our query anguage. In section 4 we give exampes of W3QL queries. In section 5 we describe the query system architecture. In section 6 we give a forma definition of the query anguage. In section 7 we present the ibrary of functions used to impement search agorithms over the WWW and two programs (SQLPRINT and SQLCOND) used by the query system. Section 8 presents concusions. 2 Difficuties in Searching the WWW The Word-Wide Web is succincty described as a Wide-area hypermedia information retrieva initiative aiming to give universa access to a arge universe of documents [o]. The interaction with the WWW is mainy based on hypertext navigation. Documents are usuay presented to the user in browser cients. Cicking on an emphasized word of the hypertext eads to a request to a HTTP server, and the server then returns a new document. The documents are identified by their URL [Uniform Resource ocators). A URL has three parts: The first part specifies the method of access to the data (e.g. ftp. HTTP), the second part is a network address of a server and the third part is a fie name. For exampe, corresponds to the access page (the home page) of the WWW server at the Technion. Parameters may be passed to a HTTP server; for exampe: queries the Webcrawer database with the keyword fower [14]. Documents written in the HTML format are standard ASCII fies containing formatting codes [7]. For exampe, the HTML document in figure 1 is presented by the Mosaic browser as in figure 2. Currenty, ac- ctitle>technion - Computer Science Department Home Page<TITLE> <PRE> <IHG SRC="Gifs/csd.gif"> <IPIE> tihg SRC="Gifs/technion.gif"> <H>Hecome to the Computer Science Department - Technion</H> <P> The symbo <IHG SRC="/Gifs/neu.xbm"> denotes new materia. <HP>Academic Information</H?> <UL> <LI><A HREF="department.htm"><B>About the Department</B><A> <LI><A HREF="sege.htm">Facuty Members and Research Interests</A>... Figure 1: Exampe of an HTML fie at 55

3 Wecome to the Computer Science Department - Technion Academic Information 0 About the Department. Facuty Members d Research Interests Figure 2: Dispay of the HTML fie in the Mosaic browser cess to the WWW is based on navigationay-oriented browsers. This eads to the we-known ost in cyberspace phenomenon. Users are confronted with a arge, unfamiiar, heterogeneous and constanty changing network. They have no systematic way to obtain information, because of the foowing reasons: There is no reiabe road map for the WWW. The WWW is constanty growing and it becomes more and more difficut to ocate specific information. It is difficut to anayze obtained information. The data found on the WWW is heterogeneous. Some fies contain text, whie others contain images, sounds or videos. These fies are stored in various formats. Therefore, it is currenty impossibe to verify automaticay whether a fie satisfies a specific condition (For exampe, Find a the images that contain a tree or Find a the artices written by A. Einstein ). In a hypertext environment, the organization of documents conveys information. Nevertheess, it is cumbersome for users to search for information reated to the organization of the hypertext. For exampe, if a hypertext document is composed of an index which points to different book chapters, and the chapters contain references to a bibiography, it is difficut to search for two chapters that have references to the same artice in the bibiography. Some systems address these probems as foows: Indexes are buit to aow searching for documents, usuay based on keyword matching. These indexes are buit by humans or by automatic toos caed robots or knowbots [5][14][11]. This approach is usefu, but it has some drawbacks: - In indexing, there is, perhaps unavoidabe, repication of information. - Indexes summarize the data, i.e. maintain the portions of the information which are considered important [9]. It is compicated to summarize images, graphics or sound data. So, indexes are most appropriate for text data. - Indexes do not do we in capturing the hypertext structure of indexed data. - Indexes become rapidy obsoete. There are some toos that hep navigating the WWW. They are based on a graphica representation of some part of the network [6]. For exampe, a graph may represent the part of the network that the user is exporing. However this approach is imited: - This technique is an aid to navigation, but it does not provide a powerfu information retrieva faciity. - A graph can be usefu for ony a very sma portions of the WWW. The whoe WWW graph is too intricate for graphic representation to be usefu. To summarize, whie some hep exists, there is no comprehensive faciity for querying the WWW. 3 WSQL: Decarative WWW Resource Finding There are two basic types of hypertext queries [8]. Content queries. These queries are based on the content of a singe node of the hypertext. A condition that such a node must satisfy to be seected is stated. 0 Structure-specifying queries. The information conveyed in the hypertext organization itsef is queried. The entity seected by this type of query is a set of nodes (and inks) from the hypertext structure that satisfy a given graph pattern [12][4][2]. Of course, one may combine these basic types and state more compex queries [2]. 3.1 Content queries in W3QL The information found on the WWW is mosty stored as unstructured data (fies) whie database systems are mainy concerned with structured data (such as tabes). However, in order to give a database-ike access to the WWW, information that is found in fies must be queried. Three basic sorts of fies exist: 1. Fies that have a strict inner structure, such as BibTeX fies or a Unix environment fie, are in fact fie representations of the content of a database 56

4 (in the BibTeX case. a bibiographic database). In such fies. the semantics of the data is ceary inked to the syntax of the fie. For exampe, the content of a BibTeX fied is defined by the name of the fied. Therefore. the database schema corresponding to such fies is naturay conveyed by the grammar of the anguage in which the fie is written [I]. Semz-structured fies are text fies that contain formatting codes (e.g. Latex or HTML). In such fies, most of the semantic information is not coded in a forma way. Therefore, it is difficut to give a database abstraction for such fies. However, it is possibe to use the formatting codes to anayze their semantic content. For exampe, Latex fies contain a tite attribute that gives some indications about the subject of the fie, and HTML fies have specific attributes that give a semantic meaning to some of their parts (e.g. the REL attribute in anchors). Therefore, a group of standard attributes may be defined for semi-structured fie format. The non-structured information of these fies may be accessed using natura anguage anaysis techniques. In raw fies the reation between the meaning of the fie and its inner structure is difficut or impossibe to ascertain. Such are executabe fies, pure text fies and image and sound fies. The WWW contains mosty semi-structured fies. A standard W3QS program, caed SQLCOND, is used to evauate booean expressions which are simiar to the where-cause of a SQL query. SQLCOND wi enabe utiizing the information that is conveyed in fie formats. Using SQLCOND, a user can seect nodes from the WWW that satisfy certain conditions. For exampe: node.format = Latex and node. author = A. Einstein. Specia attention must be paid to HTML fies. In particuar, we woud ike to state conditions about the anchors of HTML fies in order to buid structurespecifying queries. 3.2 Structure-specifying Queries in W3QL The WWW can be viewed as a graph: each URL constitutes a node, and there is an edge from a node a to a node b if: 1. the fie with URL a is in HTML format, and 2. the fie with URL a contains at east one anchor that points to node b. A path is a set of nodes {VI,..., vk} such that (Vi, %+1), 1 I i < k, are edges of the graph. A graph pattern is a graph in which the nodes and the edges are annotated with conditions. These conditions correspond to the content queries defined above. The answer to a structure-specifying query is a set of subgraphs of the WWW, where: 1. Each subgraph must be simiar to the query pattern. Simiarity is defined formay beow. 2. The nodes and the edges of the subgraph must satisfy the conditions specified for their corresponding images (see figure 3). Browsers aow an extended use of HTML and of the HTTP protoco which is known as form competion. Forms are HTML fies that contain eements, such as menus or answer fieds, that the user can compete (see figure 4). Competed forms are returned to the server. This technique is used, for exampe, to register users or to send queries to onine databases. Search the Web To search the WebCrawer database. type in your search key-xords here. This database ia indexed by content. That means that the contents of documents are indexed, not just their tites and URLs. Type 85 many reevant keywords 85 possibe: it wi hep to uniquey Identify what you re ooking for. a m 0 AND vords together -r Of rarutn to return: (1 Figure 4: Exampe of a form Three main components are invoved in supporting structure-specifying queries: Pattern definition. A pattern definition subanguage is used to describe the searched patterns. The sub-anguage is simpe enough to be practica but it is abe to express non-trivia queries. We intend to buid a graphica interface for pattern specification. Search engine. Finding sets of nodes in the WWW that satisfy a pattern is not a trivia task. We do not intend to sove this probem as a genera proposition. But, we do define a ibrary of functions that faciitate the impementation of search agorithms. We aso provide some basic vania-favored search agorithms, for exampe breadth first search. The engine wi use avaiabe indexes. Form competion. We need to dea with forms and other menu driven nodes in the WWW. A search engine must be abe to find its way automaticay through menus. Soving this probem as a genera proposition is beyond the scope of this work, ID 57

5 /! Structure-specifying query: nodez.author= A.EinJtein I answer: / URL. <TITLE> Good Artices </TITLE> URL: \author{a.einstein} I Figure 3: Exampe of a structure-specifying query but we define some basic toos that automaticay compete forms. The idea is to utiize keywords and experience gained in the past form competion activities. 3.3 Views: Deaing with Changing Information We appy the concept of database views to the WWW. Views simpify the information shown to the user, etting the user ony focus on what he/she needs. A view faciity woud greaty simpify work with the WWW. We intend that views be automaticay refreshed in order to refect changes. For exampe, a user can maintain a hypertext bibiography pointing to a the artices whose subject is Physics. This bibiography wi be maintained as new artices appear (or disappear) on the network. Currenty, information changes are difficut to ocate on the WWW. Therefore, maintaining truy up-todate views of the WWW is neary impossibe. So, we imit change monitoring to sites which are reated to the query answer, and periodicay start new searches in order to update the ist of sites reevant to the query. This approach to view maintenance resuts in fairy accurate views and reasonabe costs. 3.4 Query Resuts The defaut format for a query answer is a tabe. The tabe contains the URLs of the nodes and inks attribute vaues that satisfy the query, organized in rows. This tabe is stored in a fie and is presented using a browser. This aows the user to easiy utiize the documents returned as the answer for the query (see Tabe 1). node.url ink nodez.url DavidFies.htm REV= artice Reativity.tex OdedFies.htm REV= artice Reativity.tex OdedFies.htm REV= artice BackBody.tex Tabe 1: Exampe of an answer tabe. Resut of a search of a the home pages, at the Technion site, that point to Latex fies whose author is A. Einstein (We show here reative URLs, the compete URLs of these nodes begins with http : //cs. t echnion. ac. i). The fies that constitute the answer to the query are saved in a directory structure (as describded in section 6). The directory structure may be operated upon using standard Unix commands. These can be specified in the Seect Cause of the query. For exampe, cp */I FirstResut wi make a copy the first resut row fies into the directory FirstResut. A standard W3QS program, caed SQLPRINT, aows the formatting of resuts as a projection on fies attributes. For exampe, SELECT SQLPRINT n. author where n is a Latex fie, means that the query resut contains ony the author attribute of the Latex fie. 4 Exampes Here are some exampes of queries and views that can be expressed in WSQL. 4.1 Search for Artices The query asks for artices written in Latex format by A. Einstein. The query states that the search is performed on some indexes. These indexes are hed at known sites on the WWW. Generay, these indexes are queried by competing a form, and the answer is 58

6 an HTML page which contains the URLs of pages that might contain the information. 1 SELECT cp n2/* resut; 2 FROM n,,n2; 3 WHERE 4 n in ImportantIndexes.ur; 5 FILL n.form AS IN ImportantIndexes.fi 6 WITH keyword = " A. Einstein99; 7 SQLCOND (n2.format = Latex) AND (n2.author = A. Einsteins'); Line 4. The search is done using indexes. The fie ImportantIndexes.ur contains the ist of the URLs of the indexes being searched. The URL of the fie that corresponds to n must be in this fie. Line 2: The structure-specifying query is composed of: n, the node that corresponds to the queried index, 11 the edge to the artice returned by the index, and n2 the node that contains the artice. Line 5: Indexes are queried by competing a form. The form is competed automaticay by using the information found in the fie ImportantIndexes.fi, and the keyword portion of the query. Line 7: The condition to be seected. that n2 must satisfy in order Line 1: The seected artices are saved as fies in the user s directory resut. 4.2 Dynamicay Maintained Views This exampe refines the previous one. Now, a ist of pointers to the artices is maintained, instead of saving the artices themseves. The ist is defined as a view which is updated every week. 1 SELECT CONTINUOUSLY SQLPRINT n2.url; 2 FROM n,,n2; 3 WHERE 4 n IN ImportantIndexes.ur; 5 FILL n.form AS IN ImportantIndexes.fi 6 WITH keyword = A.Einstein"; 7 RUN earnformat IF n.form UNKNOWN IN ImportantIndexes.fi; 8 SQLCOND (n2.format = "Latex") AND (n2.author = "A.Einstein"); 9 EVALUATED EVERY week; Line 7: If a new index is inserted in ImportantIndexes.ur and its format is unknown, the program earnf ormat is caed. This program asks the user t,o describe how the form must be competed in order to compete it automaticay the next time this form is encountered. Line 9: This view is re-evauated every week. Line 1: A fie containing the URLs of the artices is returned to the user using the SQLPRINT program. This ist is continuousy updated. 4.3 Search for Hypertext Patterns This query refers to the hypertext structure described in figure 5. The query returns a the artices cited in the first chapter of the book. Each chapter incude severa pointers to the bibiography. For exampe, <A HREF= > CReativity </A> means that the ink [Reativity] ref 2 in the fie References.htm. the abeed ink ooks ike: eads to the abe In the reference fie <A HREF= NAHE= refz > [Reativity, A. Einstein1 </A>. The ink [Reativity, A. Einstein1 points to t,he artice Reat iv. t ex. SELECT cp art/* resut; FROM ind,,chap,2,ref,3,art; WHERE SQLCOND (ind.ur = " AND (chap.ur = /.Chapter-.htm/) AND (12.HREF = /.\#$13.NAME/); USING BFS Line 2: The from-cause describes the hypertext pattern that is searched for, ind is the index fie, 11 is the pointer to the chapter, chap is the chapter fie, 12 is the edge to the reference fie, ref is the reference fie, 13 is the edge to the artice, art is the artice that is returned in ine 1. Line 4-5: The URL of the index is given. Line 6: We use a Per1 [15] reguar expression. The URL of the chapter ends with the string Chapter- 1.htm (. means any sequence of characters ). Line 7: The HREF argument of the ink to the reference must satisfy the Per1 reguar expression. This expression is interpreted as foows:. means that HREF begins any sequence of characters, #$13. NAME is the abe part of the URL (the prefix $ is used specify that a variabe substitution must occur in the reguar expression). In our exampe, 12.HREF = whie 13.1~0 = cgref2. If the abe of the ink to the artice, 13, is found in a ink 12 from chapter 1, this means that the artice is cited in chapter 1 and so it must be returned. Line 8: The pattern is searched using the BFS agorithm. 59

7 URL URL In each chapter, the references point to the bibiography. The index contain pointers to the chapters and to the bibiography. ; 12 I I In the bibiography the references ref 4 point to the artices. L.-- Figure 5: Exampe of an HTML book 5 The WSQS Architecture W3QS architecture refects the foowing design principes: A the different modues shoud be easy to modify or enhance. The system shoud use existing UNIX utiities as much as possibe. The system must present an intuitive interface to the naive user but aso provide a fu programming environment for the more sophisticated user. Advanced users shoud be abe to use the query system to test information retrieva agorithms, or to manage hypertext information on their servers. 5.1 The System The main modues of the system are (see figure 6): The query processor receives the query, and uses the agorithm cause (for exampe: USING BFS) to obtain a search program (1) which is found in the RSP ibrary. The remote search program (RSP) executes the search. The RSP uses search processes to fetch information from the WWW (3). The RSP uses programs from the condition ibrary (that contains, for exampe, SQLCOND) to seect the information that corresponds to the query (4). When the RSP is done, it stores reevant information on the oca disk (5). When the search ends, the query processor uses the format ibrary (that contains, for exampe, SQLPRINT) to return the resut to the user (6,7). The ibraries contain programs which are used during query processing. The functionaity of WSQS can be extended by adding new programs to the ibraries. The RSP ibrary contains search programs, the condition ibrary contains condition evauation programs, and the format ibrary contains resut-formatting programs. 6 The W3QL Query Language W3QL is a SQL-ike anguage. Its syntax and semantics are described beow. 6.1 Syntax The core grammar for W3QL queries is described in tabe 2. Capita etters denote grammar terminas and ower case etters denote non-terminas. * means zero or more repetitions of a construct. [: 1 means optiona. node-name and inkname are C strings. unix-program is the name of an executabe fie. fiename is the name of a UNIX fie. The arguments are separated by banks. regexp is a Per1 reguar expression as defined in P51- assignment has the form x=y. For exampe, topping = Anchovies or time =

8 QUERY PROCESSOR Word Wide Web Figure 6: The system architecture time-unit is a period of time, e.g. week or 5 minutes. 1, query:= SELECT [CONTINUOUSLY] [seect-cause] statement:= Unix-program argument*; step:= node-name, ink-name, 1 (node-name, ink-name), ink-name, condition-cause = condition* condition:= statement = regexp; 1 node-name IN fie-name; 1 FILL node-name AS IN fie-name WITH assignment* ; 1 RUN Unix-program IF node-name UNKNOWN IN fie-name; asorithm-cause:= statement Tabe 2: Grammar of the query anguage 6.2 Semantics Condition Cause The condition-cause imposes conditions on the nodes and edges of the from-cause, and aso provides the RSP navigation directives. The foowing condition types are possibe: 1. unix-program argument* = regxp. To evauate this type of condition the RSP invokes an externa UNIX program. The program input is defined by the arguments. An argument can be a node name, a ink name or a node name with.form appended to it. The RSP runs the program, the arguments are repaced by fie names. Here are some simpe exampes: The statement diff ni n2 is evauated by running dif f tempi temp2, where temp and temp2 contain the data extracted from the WWW nodes that are mapped to n and n2. The.form termination is used to state that a condition appies to a node that contains a form. The program output is then compared to a Per1 reguar expression. If the output satisfies the reguar expression the condition is true. 61

9 2 node name IN fiename. This condition is true if r,hf node name is mapped to a URL that, is found m t,hr fie fiename This type of condit#ion can be 11set1 m two ways. Its can be a. directive to thr RSP meaning that t,he RSP must begin the search from a specific ist of nodes, or it may be a condition, meaning that, some node in the searched pattern must. he ocated in a specific domain of the nep;work a,s defined by the content of fie-name. to a set of paths is buit by identifying the same name (see figure 8). the nodes having 3. FILL nodename AS IN fiename WITH assignment*. The WWW node mapped to node-name contains a form that must. be fied automaticay by thr RSP using the data found in tie-name and the assignments. For exampe. if This graph can be described ar either. Godzia s Pizza -- Internet Deivery Type m your street address: 1 Type in your phone number:! Which toppings woud you ike? 1. 0 Pepperoni. 2. E Sausage Anchovies. To order your pizza. press this button: -1. Service: Figure 8: Exampe of a pattern graph. In order to pose queries without, knowing exao[y how the hypertext is organized, we aow the definit8ron of a specia sort of paths caed unbounded ergh paths (see figure 9). Unbounded ength paths are represented as sef-oops in the graph pattern. Figure 7: A form at /Mosaac/Docs/fi-out-forms/exampe-3.htm the form in figure 7 was competed in the past, the condition FILL n AS 0rderpizza.f i WITH topping= Anchovies means that, a the entries from the ast. competion activity (address a.nd phone number), that were saved in the fie orderpizza. f i, remain the same, ony the topping is possiby different. 4. RUN unix-program IF node-name UNKNOWN IN fiename: The RSP cas an externa program if it encounters an unknown form. This program may immediatey ask the user to compete the form! may save this form as an unknown form to be competed ater by the user, or may try to guess, using the form defa.ut vaues and the RSP s experience in form competion activities. Thx graph is descnbded as: F(h2 Figure 9: Exampe of a pattern-specifying an unbounded ength path. graph with Formay, a pattern-specifying graph is a directed graph G(V, E) where: There is at most one edge between any two nodes. There is at most one edge from a node to itsef (sef-oop). A subgraph of the WWW G (V, E ) is said to be simiar to a graph pattern G(V, E) if there is a mapping F such that: F: Vu E c--t 2 UE if w E V has no sef-oop: F(v) = {v }, ZI E V The Pattern Graph if v E V has a sef-oop: F(v) = {wi,..,u;},u~ E The from-cause of the query describes the pattern V and for 1 5 i 5 n - 1, (zi:, w:+r) E E. This graph that the RSP searches for. A pattern graph is aso defines First(F(w)) = vi, Last(F(v)) = v& described by a set of paths. The graph corresponding (if F(v) = {.u )? First(F(u)) = Last(F(v)) = v ). 62

10 / SELECT grep zoo ni/* > zoo.txt; G can be mapped onto G : Fin)={n } F(n2 1=inZ.n3,n4 } F(n3)=jnS k F()={ ) F (121= } F(3)={14 } I stand for (n,n2).(,,2,n2j... / QUERY A 1 /i /:t [\ - 2 pp21 2 i \ - I\ Figure 10: Exampe of a ma.pping.. for a i. 212 E I/ s.t. 211 # v2, F(v1) n F(U2) = 0. Thr srmamics of a W3QL query is as foows. Loca~.~ a subgraphs of the WWW that are simiar t,o the pattern specified in the from-cause. Each subgraph, whose nodes satisfy the where-cause, defines a row in the returned t,abe. The row contains the nodes and inks used t,o exhibit simiarity. The foowing query searches for images in GIF format a.ppearing in HTML fies foowing ony inks with t,he REL argument equas to exampe. REL is an HTML argument which is used t,o add meaning to hypertext inks. SELECT cp n3/* resut; FROM (ni,11),12,n3; WHERE SQLCOND (n.format = HTML) AND (i.rel= exampe ) (n3. name= *. gif > ; Seect Cause AND This optiona part of the query defines the form in which t,he query resuts shoud to be processed. Once the RSP fiuishes searching the network, the RSP saves t,he nodes t,hat were extracted from the WWW and t#ha.t are needed to be processed, in the oca disk, organized as a directory tree (see figure 11). The seect cause may run any UNIX program on these fies so as to buid the resut of the query. For exampe, the foowing query saves a the ines from the seected nodes, which correspond to the node name n, that cont,ain the string zoo? in the fie zoo.txt,. Figure 11: The fie tree buit for the pattern: n,,(n2.12),13,n3. The RSP found two sets of nodes that, form a soution to the query. The fies tha.t constitute the first set are: n/. n2/p/, n3/1 (underined in the figure). The fies that, constitute the, second set are: n1/2, n2/p2/1. n2/p2/2, n3/2. Not,e t#hat, p is a singe node path and p2 is a two node pa.th (both corresponding to (n2,12) in the query) Agorithm Cause The agorithm-cause prescribes, to the query processor, which procedure shoud execute the search. These procedures wi be impemented using the functions defined in the RSP Construction Kit. If the agorithm cause is not found in a query, a defaut search agorithm wi be used. 7 The RSP construction kit and SQLCOND 7.1 The RSP Construction Kit Functions The agorithm ibrary incudes data types and functions to assist programmers in easiy impementing WWW search agorithms. The ibrary has three groups: Structure-specifying graph functions. These functions buid and use an interna representation of the structure-specifying graph. The functions incude mapping a node to a URL, testing whether a URL satisfies a condition, etc. Access to, and description of parts of, the WM W. These functions hep manage the processes that extract information from the WWW, instruct, a process to bring a page, examine a page to find its inks, etc. Extension to the ibrary. The extension to the ibrary incudes functions to cone RSPs and 63

11 functions for inter-rsp communicat,ion in order to deveop distributed search agorithms. Fio,ure 12 shows an exampe of a simpe search h agorithm impemented using the RSP Construction Kit,. In t,his case, t,he structure-specifying graph is sinpy a singe node. SEARCH(PNODE *pn, WNODE *start, int eve, int imit) /* PNODE is a pattern specifying graph node, WNODE is a WWW node */ if (eve >= imit) return; Assign(pn,start) ; /* start satisfies the conditions */ If (TestCondition( AddToResut (pn) ; whie (NextAnchor(start)) SEARCH(pn,WWWGet(Anchor(start)),eve+,imit): 7.2 SQLCOND Figure 12: A simpe search agorithm The SQLCOND program uses the attributes of informa.t,ion found in a WWW node. This program uses t,r fie name and the fie UNIX utiity (as in [9]) to. gu& the t,ype of the fie (e.g. Latex, Postscript). Then. a t,abe is used to identify the attributes of the fie. For exampe: Format, 1 Att,ribute 1 Begin Token ) End Token Latex ( author iauthor I 1 We use the parsing capabiity of Per1 to make a simpe and quick anaysis of the information found in fies. This is a naive, yet effective, mechanism that ma,y, somet,imes, resut in wrong answers. In that respect, our query processing is approximate. SQLCOND permits stating conditions about node cont,ents, or join conditions, for exampe n. author = n2. author. In particuar, SQLCOND permits stating conditions on the different nodes of an unbounded engt,h path. The conditions that, can be stated are a subset of the condition causes defined in Per1 [15]. 8 Concusions We have designed and are currenty impementing W3QS, a, high eve querying and dispaying faciity for t,he WWW. We have specified the syntax and semantics of a. high eve SQL-ike query anguage cart W3QL. W3QL is based on interfacing to user written programs and utiizing standard Unix services (e.g., fie, grep, awk and W3QS standard programs) for identification. fitering a.nd formatting. This m&s W3QL extensibe and customizabe. This aso makes t,he soft,ware construction task manageabe. We pan t,o provide W3QS with adva.nced dispay faciities for ext,racted information, incuding qwcia graphics. HTML browsers visua present,ation! and Unix directory tree representation. W3QS wi incude a. view maintenance faciity at a higher eve t,han robot maintained indexes. We pan to ut,iixe WWW robot-constructed indexes in query processiug optimization. At the present time, we have competed the design of the query anguage. A prototype W3QS system C1 be constructed in the coming months. The construction status and other deveopments concerning W3QL may hefoundinhttp:// This paper itsef is accessibe in this address. References PI PI PI PI [51 PI ARITEBOUL. S., CLUET, S., AND MILO, T., Querying and updating the fie. In Proceeding of the 19th VLDB conference (Dubin, Ireand, 1993). BEERI, C., AND KORNATZKY, Y., A ogica query anguage for hypertext systems. In PTOceeding of the European Conference on Hypertext (1990), Cambridge University Press, pp CARNEGIE MELLON UNIVERSITY., Lycos, The Cataog of the Internet. At CONSENS, M. P., AND MENDELZON, A. O., Expressing structura hypertext queries in graphog. In Hypertext 89 (1989). DE BRA, P. M. E., AND POST! R. D. 3., Sea,rching for arbitrary information in the www: The fish search for mosaic. In Eectronic Proceedings of th.e Second Word- Wide Web Conference 94: Mosaic and the Web At /Proceedings/WWW2_Proceedingshtm, D~MEL, P., Webmap: A graphica hypertext navigation too. In Eectronic Proceedings of the Second Word- Wide Web Conference 94: Mosaic and the Web At /Proceedings/WWW2-Proceedings.htm, [71 GRAHAM, I. S., Htm- documentation and stye guide. At /NewHTML/htmindex.htm

12 [t;] FIALASZ. F G., R.efections on notecards: Seven issues for the next, generation of hypermedia syst>ems. Commu.nication of the ACM 31, 7 (1988). [$I] HARDY, D. R., AND SCHWARTZ, M. F., CUSt,omized information extraction as a basis for resource discovery. Tech. Rep. CU-CS , University of Coorado , Mar [I()] -I!GHES. Ii., Entering the word-wide web: A guide to cyberspace. At ftp://ftp.eit.com. directory /pub/web.guide, [I MCBRYAN! 0. A.. Genv a,nd wwww: Toos for taming the web. In Proceedings of the First Internationa Word Wide Web Conference (May 1994). 0. Nierstrasz CERN. [12] MINOHARA, T., AND WANATABE, R., Queries on structure in hypertext. In Foundation of data organiza.tion and agorithms, FODO 93 (1993), Lomet,, Ed.. Springer-Vera.g, pp [13j NETSCAPE COMMUNICATION CORPORATION., The Yetscape Home Page. At [14] PINKERTON, B., Finding what peope want: Experiences with the webcrawer. In Eectronic Proceedings of the Second Word- Wide Web Conference 94: Mosaic and the Web At, /Proceedings/WWW2-Proceedings.htm, [15] s CHWARTZ, R. I,., Learning Per. A Nutshe Handbook. O Reiy & Associates,Inc, 1993, ch. Regu1a.r expressions, pp