WEB LAB - Subset Extraction

Transcription

1 WEB LAB - Subset Extractin Fall 2005 Authrs: Megha Siddavanahalli Swati Singhal Table f Cntents: Sl. N. Tpic Page N. 1 Abstract 2 2 Intrductin 2 3 Backgrund 2 4 Scpe and Cnstraints 3 5 Basic Subset Extractin 5 6 Design Features 5 7 Design f Subset Extractin Applicatin 6 8 Results 9 9 Future Wrk References Acknwledgements Appendix 1 Discussin with Prf. 11 Jayavel Shanmugasundaram abut usage f views 1

2 Abstract: Infrmatin retrieval and strage is critical and ne f ur majr cncerns. The gal f subset extractin is t facilitate access t the terabytes f data in the archive. Is it practical t scan thrugh the entire data each time user sends in a query? What are the perfrmance cnstraints? What are the strage cnstraints? A substantial amunt f prperties f subset extractin vary frm user t user and it is nt pssible t cme up with a static set f requirements. Hence we need t design a system which can adapt t different requirements f the user and prvide ptimal perfrmance. The fllwing reprt discusses a practical design and implementatin f a subset extractin system. Intrductin: A subset is defined as the part f the infrmatin in the database which satisfies the user query. Subset extractin is the prcess which prvides the functinality f querying the database and creates a subset which satisfies the query. The size f the subset may vary fr different queries. A user may generate a query which cvers the entire database r a query which is a small sectin f the database. The size may vary frm terabytes t even kilbytes f infrmatin. Our prblem is t design a system which can grw and shrink depending n the size f the subset required. Strage space is critical when we are talking abut large data sets and care must be taken t have a system which utilizes the space in an ptimal manner. Nte that we d nt want t replicate the data we have in the database every time a query is executed; hence we require a system that is scalable. Several mdels were cnsidered. Backgrund: The underlying database used fr the subset extractin fr the Web Labratry prject is described belw The underlying database stres the DAT files and is shwn in the fllwing sectin. The database is made unique t every crawl; every crawl has its wn database. The crawls are given a unique crawl id. The archive database: The archive database is used t stre the internet archive data in the frm f a database fr easy querying. The database has a 2 tier schema: 2

3 Each f the crawls has its wn set f tables. Diagrammatic representatin f the tables is as fllws: Each crawl is identified by the CrawlID and CrawlDate Scpe and Cnstraints: Fr the subset extractin prcess infrmatin retrieval and strage are critical. The slutin t data strage is using Relatinal Database Management Systems (RDBMS). The factrs that influence the subset extractin and strage f the data extracted after retrieval are: 1. The subset schema shuld be scalable depending n the user queries. The user may query n general and brad tpics which may prduce a large set f dcuments. The Web Labratry prject database size ranges in terabytes and there is cnstant incming data frm the internet archive. T create a subset f this database requires a system that can supprt thusands f researchers and prvide strage fr the subsets. A subset is nthing but a part f the database relatin r a cmbinatin f relatins. It is a replica f infrmatin already existing in the database. This replicatin leads t data redundancy. Data redundancy leads t prblems f data incnsistency since any change in the database will have t be reflected in the replicas therwise 3

4 data ges in incnsistent state. Since the data stres is critical and f high imprtance we need t cme up with a mdel which is scalable and takes care f the prblems f data redundancy and incnsistency. 2. Prviding users ptins t stre the retrieved data subset n the system. User has the ptin t use the data retrieved by a particular query fr further research and need nt run the same query again t retrieve the data. A user may run a query, which may require hurs t run and create a subset since the underlying database is very big in size. It is nly reasnable t stre the subset fr the future use by the user and drp it when user n lnger requires it. This saves the prcessr time and user time and makes the system mre efficient. Als a user may find it tedius t run the same query again every time. The system must be built in such a way s as t take care f the needs f the end users and make the system easy and cnvenient t use. 3. An efficient mechanism is required fr lading and retrieving data recrds t and frm the database in utilizing the data fr research purpses. 4. Anther challenge in subset extractin is t stre the data which spans ver different crawls. The pages in the database are stred fr different crawls and the same URL maybe stred in several crawl but each cpy f the URL will have a different time assciated with it. User queries may be cmplex and may require infrmatin that requires the jin f tw r mre tables, scanning different tables in a single crawl r scanning different crawls and underlying tables fr each crawl. These are cmplex peratins and an efficient query language wuld be required t create a subset which spans ver different tables. 5. Sme f the key features that a subset shuld pssess are scalability, efficiency and minimum strage requirements. The subset shuld prvide all the functinality that wuld make the system mre user friendly. 4

5 Basic Subset Extractin: This applicatin is used t allw users t access the database. The users may just specify parameters t search, fr example url, dmain, date These inputs will then be cnverted int an SQL query and a subset f the database is created using this query. A unique subset id is assigned t each such subset generated and returned t the user. This subset id can uniquely identify the SQL query results t the users. The subset is stred fr future queries. The user can use this subset id in further research and need nt run the query again. Design features: The subset is stred in frm f regular views with each subset being assigned a unique subset id. The views are virtual tables. They are references t the tables in the database and hence the system is scalable since there is n verhead with data strage. Any changes made t the database are reflected in the views and hence this eliminates the prblem f data incnsistency. The subset infrmatin table keeps the recrd f all subsets generated alng with the query used t generate them. The user can specify the subset id and the view will be given t the user. Query the archive Subset extractin When the user runs queries n the archive the result will be a series f pages that were stred in the archive. Prperties f these pages They can span ver mre than ne crawl In a single crawl multiple pages with the same URL can be present They have the fllwing prperties Page ID, URL ID, IP Address, Arc Time, MIME type 5

6 Design f the Subset Extractin Applicatin: Architecture Diagram: Basic Access DLL - Uses ADO t cnnect t the database Applicatin - Used t create and access subsets Subset Extractin - Uses BADLL fr database cnnectin Scidata Database Applicatin: Applicatins are used t allw users t access the database. Fr the specific task f subset extractin, these applicatins shuld Allw user t specify parameters Cnvert these parameters int an SQL query Return a subset ID t the user that can uniquely identify the SQL query results t the users MSSQL Database: The subset extractin lgic adds the fllwing table t the database that allws it t manage the subsets TABLE SUBSET_INFORMATION Name Data Type Descriptin Subset_ID Integer Simple integral cunt incremented fr each new subset View_Table_Name Varchar(20) Name f the view table that crrespnds t the subset. SQL_Query Varchar(2000) The user query that was used t generate this subset 6

7 Subset ID: The primary key f this table will be the subset ID that is used t uniquely identify them and this will the parameter that is passed back t the user View_Table_Name: If the subset is just a database view, then it will be assciated with a view table that is created when the subset it generated. The View_Table_Name will be used t iterate ver the data in a subset that is a simple view SQL_Query: The query that was used t generate the view is als stred s that the view can be recreated when the database crashes r if a view is drpped unintentinally DLL The Subset Extractin functinality has been cnverted t a dll that can be invked n SciData1 t extract subsets. The DLL has the exact same functinality described in the functins in the design dcument and the class diagram belw. 7

8 Class Diagram View_Subset Recrd: _RecrdsetPtr getelementat(int :in,bstr: in, int &: ut) getelementat(int :in, BSTR :in, char & :ut) getelementat(int :in, BSTR: in, flat&: ut) getelementat(int :in, BSTR :in, BSTR& :ut) getnofrws(int &:ut) getnofclumns(int &: ut) CreateSubset(BA_DBCnnectr &: in, BSTR:in, int & :ut) CreateSubset(BA_DBCnnectr &: in, int subsetid: in) View_Subset_Factry Static class: n attributes CreateSubset(BA_DBCnnectr &cnnectin : in,bstr sqlquery : in, int &subsetid: ut,_recrdsetptr &recrd: ut) CreateSubset(BA_DBCnnectr &cnnectin : in,int subsetid : in,_recrdsetptr &recrd: ut) GetCunter(BA_DBCnnectr &: in, int &subsetid : in, _RecrdsetPtr &recrd: in): BSTR Abstract_Subset getelementat(int :in,bstr: in, int &: ut) getelementat(int :in, BSTR :in, char & :ut) getelementat(int :in, BSTR: in, flat&: ut) getelementat(int :in, BSTR :in, BSTR& :ut) getnofrws(int &:ut) getnofclumns(int &: ut) 8

9 Fr details n the implementatin refer the design dcument fr Subset Extractin API n the gfrge site. Web Service The functins are als exprted as web services using ATL prgramming. T crss platfrm and prgramming language, we wrap Subset Extractin DLL API and exprt its functins by Subset_WS t give users mre alternatives and flexibilities t utilize ur service. Subset_WS is a set f API functins implemented by ATL Server Web Service in Micrsft.Net Framewrk. Users can invke Subset_WS API functins in their native prgramming languages t d basic manipulatins thrugh ur web service. Fr details please refer the design dcument fr Subset Extractin Web Service n the gfrge site. Results: This prgram is able t generate a view subset fr the query given by the user The user can just prvide the subset id and reuse an existing subset by acquiring a pinter t the view. It can give the user view details such as number f rws and number f clumns It can iterate thrugh the subset and access any element at any given psitin f the subset. The subset infrmatin is stred fr future reference. Views are virtual and maintain a reference t the database tables. Any changes t the database tables is als reflected in the view hence there is n issue f data incnsistency. This applicatin is scalable since there is n strage verhead assciated with the string f subsets. The functinality is prvided in the frm f a DLL fr users t use The functinality is als exprted as a web service that users can remtely use 9

10 Future Wrk: The future wrk in this area invlves scaling the subset extractin t handle different kinds f subsets fr example file subsets. Currently views serve the purpse in an efficient manner but the user may want t save the subset generated n the client machine. We wuld require the subset t be in frm f files. The subset extractin applicatin prvides the basic functinality t the user, in future we wuld require t increase the functinalities prvided t the user t make the interface easy t use and cnvenient. Prvide the functinality t generate web graphs and full text indexing t facilitate search using query wrds. References: 1] Web Labratry Prject - 2] Raghu Ramakrishnan, University f Wiscnsin and Jhannes Gehrke, Crnell University Database Management Systems Third Editin 3] Inside Micrsft SQL Server 2000, Delaney, et. al., Micrsft Press, ISBN ] Mastering SQL Server 2000, Gunderly, et. al., Sybex Press, ISBN ] Jim Gray f MS Research and Gerd Heber f the Crnell Fracture Grup: ftp://ftp.research.micrsft.cm/pub/tr/tr pdf Acknwledgements: This wrk is a part f the Web Labratry, which is a jint prject f Crnell University and the Internet Archive. Other members f the team are: William Arms, Blazej Kt, Mindau and Shantanu. Prfessr Jayavel Shanmugasundaram in guiding us. This wrk is funded in part by Natinal Science Fundatin grants , , and

11 Appendix 1: Meeting with Prfessr Jayavel Shanmugasundaram regarding subset extractin: 1. Is using materialized r index views a slutin fr subset extractin? N. Because what if a user s query cnsists f mst f the database then the entire database will have t replicated. This wuld nt be a gd slutin. Indexed views in MSSQL are like materialized views in Oracle. They are nt virtual but make a cpy f the data required. 2. Wuld default views be a gd slutin fr string subsets? Default views in the current case wuld be the ideal way f string the subsets. Default views can be any number since they d nt use any space. They just reference the tables in the database. 3. Are the default views reliable? They are like the database tables and get erased nly when they are drpped. Hence they are stred fr as lng as the user wishes the view t exist. Updating the views: There is n prblem f data incnsistency since the views get updated as sn as the tables they refer t get updated. Scalability is nt an issue in case f default views since they d nt cntain real data. They are virtual and d nt ccupy memry space. 4. A user may want t query n the subset after creating the views. Is there a way we culd assciate certain views with the users wh created them? The view created by a user culd have the name starting with the user id. There shuld be a way t give each user his wn namespace and all the subsequent queries culd be first searched in the views that already exist. 11