A Web Site Classification Approach Based On Its Topological Structure

Transcription

1 Internatonal Journal on Asan Language Processng 20 (2): A Web Ste Classfcaton Approach Based On Its Topologcal Structure J-bn Zhang,Zh-mng Xu,Kun-l Xu,Q-shu Pan School of Computer scence and Technology,Harbn Insttute of Technology No.92,West Da-Zh Street,Nangang Dstrct,Harbn , Chna Phn: , Fax: , zjbxgg@ht.edu.cn,xuzm@ht.edu.cn Abstract: Automatc web ste classfcaton has a wde applcaton prospect; however, there are few researches on t. Dfferent from pure texts, web stes are the combnaton of a large number of web pages va hyperlnks, so text classfcaton methods are not sutable to classfy them drectly. Ths paper proposes a web ste classfcaton approach based on ts topologcal structure. Gven a web ste, frstly we represent ts topologcal structure as a drected graph, and from whch we extract a strongly connected sub-graph ncludng the ste s home page. Secondly, we use an mproved PageRank algorthm on the sub-graph to select some topc-relevant resources, and represent them as a topc vector of the ste. Fnally we use an SVM classfer to classfy the ste n term of ts topc vector. Some experments are conducted for web ste classfcaton. Expermental results show our approach acheved better performance than tradtonal super page-based web ste classfcaton approach. Key words: web ste classfcaton,topologcal structure of web ste,hyperlnk analyss,topc vector of web ste We thank all students of Natural Language Processng Research Center n Harbn Insttute of Technology for dscussng some ssues about ths paper. Research for ths artcle was supported by the Natonal Natural Scence Foundaton of Chna( , ).

2 76 J-bn Zhang,Zh-mng Xu,Kun-l Xu,Q-shu Pan 1 Introducton Wth the rapd development of Internet, the nformaton of network grows explosvely. Accordng to the statstcal data released by Google, Google currently has ndexed over one trllon web pages and ths fgure s stll rapdly ncreasng every day. Internet has already become the most mportant source of nformaton and knowledge n scentfc research, educaton and other felds. Due to ts mass, varable, and non-semantc characterstcs, t s not easy for people to fnd the nformaton they want quckly and accurately. How to fnd the nformaton we need from such a huge source has become an mportant objectve we need to study. For the moment, there are two knds of servces that can help us to retreve nformaton n the Internet: search engnes le Google and drectory servces le Yahoo! and DMOZ. Search engnes usually return some web pages matched wth queres. However people sometmes need to fnd some web stes related wth a certan subject. For example, when people want to buy somethng, they wll try to fnd the retaler s web stes nstead of web pages whch only contan descrptons of commodtes. Drectory servces supply a navgaton mechanc of web stes by collectng a number of web stes and manually classfyng them nto dfferent drectores. But they spend lots of manual edtoral work to mantan drectory servces. The technology of web nformaton navgaton, especally automatc classfcaton of web nformaton s becomng the research focus. Consderng that automatc web ste classfcaton s sgnfcant to mantenance drectory servces, ths paper manly studes automatc web ste classfcaton approaches. Because a web ste s the combnaton of a large number of web pages va hyperlnks, whch has rcher structure nformaton than sngle web page, text classfcaton approaches are not sutable to classfy t drectly. Ths paper proposes a web ste classfcaton approach based on ts topologcal structure. Gven a web ste, frstly we represent ts topologcal structure as a drected graph, and from whch we extract a strongly connected sub-graph ncludng the ste s home page; secondly, we use an mproved PageRank algorthm on the sub-graph to select some topc-relevant resource, and represent them as a topc vector of ths ste; fnally we use a SVM classfer to classfy the ste n term of ts topc vector. The rest of ths paper s organzed as follows. Secton 2 gves a summarzaton of prevous research on web page and web ste classfcaton. In secton 3 we descrbe our web ste classfcaton approach. In secton 4, we contact some experments to test our web ste classfcaton approach, and the concluson s gven n the last secton.

3 A Web Ste Classfcaton Approach Based On Its Topologcal Structure 77 2 Related work Automatc classfcaton of web pages has been studed for a long tme, some text classfcaton algorthms le Naïve Bayes(McCallum 1998;Mtchell 1996), KNN(Lam 1998;Masand 1992), and SVM(Joachms 1998;Kwok 1998) have been successfully appled. Apart from the content of web pages, Chakrabart(Chakrabart 1998) and Craven (Craven 1999) mproved the accuracy of web page classfcaton by ntroducng hyperlnk analyss. However, there s a lttle research on web ste classfcaton, the dffculty n whch s that a web ste conssts of many pages, and each page has ts own topc, a ste s topc can not be reflected by a sngle web page. A famous web ste classfcaton method s super page-based method (Ester 2002), whch represents a web ste as a sngle vrtual web page combned by all ts pages, and Perre (Perre 2001) mproved t by ntroducng web pages meta data, such as ttle, keyword, and so on. Terveen(Terveen 1999) represented a web ste as a drected graph and combned content and hyperlnk analyss to classfy t. Ester(Ester 2002; Ester 2004) gave an emprcal study on web ste classfcaton, and proposed several solutons of web ste classfcaton; on the bass of the research of Ester, Kregel(Kregel 2004) ntroduced a method that represented a web ste as a topc-frequency vector. In addton, YongHong Tan(Tan 2004) used a mult-scale tree model to represent a web ste, De-yu Fu proposed a key resource-based web ste classfcaton method(fu 2006), and Bao-l Dong employed a hybrd vector space model to recognze the subject of web stes(dong 2005). 3 Web ste classfcaton approach based on ts topologcal structure In ths secton, we manly dscuss our web ste classfcaton approach based on ts topologcal structure. Ths approach manly ncludes several phases: represent a web ste s topologcal structure as a drected-graph, extract some topc-relevant resources from the ste s topologcal graph, represent extracted topc-relevant resource as a topc vector, and use the topc vector to classfy the ste. 3.1 Representng a web ste s topologcal structure as a drected-graph In ths secton, we represent the topologcal structure of a web ste as a drected graph. Some defntons about the drected graph are gven as follows: Defnton 1. Drected graph: A drected graph s an ordered par D=<V, E>. D represents the topologcal structure of a web ste. V s a set of vertces, each vertex of V s a page; E s a set of drected edges, whch s a subset of V V, each drected edge e= (u, v) means a hyperlnk e from page u to page v.

4 78 J-bn Zhang,Zh-mng Xu,Kun-l Xu,Q-shu Pan Defnton 2. Degree: A drected graph D=<V, E>. For each vertex v V, the number of edges lnked from V s defned as the out-degree of v, and the number of edges lnked to v s defned as the n-degree of v. The sum of n-degree and out-degree of v s defned as the degree of v. Defnton 3. Path: A path n a drected graph D=<V, E> from v m to v n s defned as a sequence of vertces {v m, v m1, v m2, v n }, whch ncludes edges (v m, v m1 ), (v m1, v m2 ) (v m, v n ). Defnton 4. Sub-graph: There are two drected graphs: D=<V, E> and D =<V, E >, D s called as a sub-graph of D f V` V and E` E. Defnton 5. Strongly connected graph: A drected graph D=<V, E> s called as a strongly connected graph f there s a path between any par of two vertces. Fg.1 s an example of a web ste, where the ste s represented as a drected graph, f page A can reach page B va nner hyperlnks, then there s a path from A to B. If any par of pages n ths graph has a path to connect them, then we call t strongly connected. Fg.1 A topologcal graph of a web 3.2 Extractng topc-related resource from web ste s topologcal graph After the topologcal graph of a web ste s bult, we put the emphass on extractng topc-related resource from t. Accordng to some lteratures pont of vew, a web ste s home page may be the most topc-relevant to the ste(dong 2005), and a ste s pages wth the same topc usually have a compact lnk structure(lu 2006). In addton, web ste desgners generally hope that outgong-lnked pages should be topc-relevant to the current page, so we can assume a par of pages n a ste s topc-relevant f there s a hyperlnk between them(ester 2004). Accordng to ths assumpton, we can nfer that a par of pages n a ste should be topc-relevant f there s a path between them.

5 A Web Ste Classfcaton Approach Based On Its Topologcal Structure 79 Consderng the above all, we thnk that a ste s topc-relevant resource should be located on a strongly connected sub-graph ncludng the ste s home page, on whch we can use hyperlnk analyss technology to select mportant topc-relevant sources. The PageRank(Page 1999) algorthm s often used to compute the mportance of web pages, whch regards the entre web as a drected graph, and ranks pages through hyperlnk analyss. Ths paper wll use an mproved PageRank algorthm n a ste s sub-graph to rank pages to select mportant topc-relevant sources from t Improved PageRank Algorthm PageRank s the earlest and the most successful algorthm appled to the hyperlnk analyss on commercal search engnes, whch nterprets a hyperlnk from page A to page B as a vote, by page A, for page B. If pages that cast votes are mportant, they wll make pages voted to be mportant. A smplfed verson of PageRank defned by Larry s as follows: PR PR( P) N ( s) (1) 1 C( P ) where s s a page, PR(x) means the rank score of page x, N s the n-degree of s, P s the page lnked to s, and C(P ) s the out-degree of page P. In formula (1), the rank score of page P s dvded by ts out-degree, and each page lnked from P s dstrbuted wth the same rank score. There s a small problem wth formula (1). Assumed that there are two or more pages lnked to each other but to no other pages, and there s a hyperlnk lnked to one of them; after some teratons, rank scores are accumulated nto them but never dstrbuted out from them. Ths scenaro s called rank snkng. To solve rank snkng problem, Larry modfed the orgnal PageRank formula as follows: PR s) (1 d) d PR( P ) N 1 C( P) ( (2) where d s usually set as 0.85, t s the probablty that users contnue to vew pages lnked from the current page s, (1- d) s the probablty that users leave the current page s and skp to other web pages. In PageRank algorthm, each page s dstrbutes ts rank score to pages lnked from s averagely. But the average dstrbuton scheme of rank scores among pages s not sutable for the demand of web ste classfcaton. For a web ste, we am to select topc-relevant resource from ts sub-graph, so we consder that the rank scores should be dstrbuted accordng to page smlarty. If one page A s more smlar to pages lnked to A, A wll get more rank scores, otherwse t wll get less. Here, we use an

6 80 J-bn Zhang,Zh-mng Xu,Kun-l Xu,Q-shu Pan mproved PageRank formula to dstrbute rank scores among pages, whch s shown as follows(yuan 2007) : N sm( P, s) PR( s) (1 d) d PR( P ) (3) M 1 sm( P, Q ) where sm(p,s) s the page smlarty, Q j s a page lnked from P, M s the number of pages lnked from P. Fg.2 s an example of the mproved PageRank formula. Assumed that page A has two hyperlnks lnked to page B and page C respectvely, PR(A) s 1, sm(a, B) s 0.8, and sm(a, C) s 0.4. The rank score dstrbuted from A to B s 1*0.8/ ( ) =0.6666, and C gets 1*0.4/ ( ) = Computaton of lnk-based page smlarty j1 j The mproved PageRank formula uses the page smlarty to dstrbute the rank scores among pages. In general, the smlarty between pages can be computed accordng to ther contents(wang 2003). Consderng the computaton cost of content-based page smlarty, we use the computaton methods of lnk-based page smlarty, whch only analyze hyperlnks among pages nstead of ther contents. In Fg.2, Page A has two hyperlnks lnked to page B and page C respectvely. Accordng to Lterature(Ester 2004), f B and C are both lnked to or from the same page, they may have the same topc. The more are the pages lnked to or from both B and C, they are more topc-relevant. In other words, they are more smlar. sm(a,b)=0.8 sm(a,c)=0.4 B A C Fg.2 The mproved PageRank formula Accordng to Lterature (Wang 2003), we number all the pages n a ste as {1, 2, 3 n}; for each page s, we construct two vectors: V s out, V s n. If the th page has a hyperlnk lnked to s, then the th tem of V s n s 1, otherwse t s 0. Smlarly, f th page has a hyperlnk lnked from s, the th tem of V s out s 1, otherwse t s 0. Consderng the above all, Lterature (Wang 2003) gave the n-lnk-smlarty, out-lnk-smlarty and lnk-based smlarty between page A and page B as follows:

7 A Web Ste Classfcaton Approach Based On Its Topologcal Structure 81 Smlarty Smlarty v v n n n A B ( A, B) n n va vb v v out out out A B ( A, B) out out va vb (, ) n out Smlarty A B Smlarty (, ) A B Smlarty (, (6) ) A B In the above smlarty formulas, the more common pages are lnked to A and B, the bgger s Smlarty n (A,B); the more common pages are lnked from A and B, the bgger s Smlarty out (A,B). For a gven web ste, we frstly extract the strongly connected sub-graph ncludng the ste s home page, and then we use formulas (3) and (6) to compute the rank score of each page, rank these pages accordng to ther rank scores, and fnally select some hgh-scored pages as topc-relevant resource of the ste. 3.3 Represent the Topc Vector (4) (5) After rankng the pages n the sub-graph, some topc-relevant resources on the sub-graph are selected. Now we should consder how to represent extracted pages and ther hyperlnks. Accordng to lteratures pont of vew(hodgson 2001), 61% anchor texts of hyperlnks can reflect the topc of pages they lnk to. So we vew anchor text of hyperlnks as a ste s structure feature of stes, and vew content text of pages as content feature of stes. Under vector space model, we combne content feature and structures feature of a ste to a mxed vector, called a topc vector, whch s shown as follows: v ( w, w w, w, w w ), l m n (7) ' ' ' 1 2 m 1 2 n ' w s the weght of the structure feature w s the weght of the content feature term t j. Here, we use Informaton where v s a l-dmenson mxed vector, term t ' and j Gan (IG) method to select content and structure feature tems, and use tradtonal entropy weghtng method to weght structure feature tems (anchor text tems). where a N 1 TF TF log( TF 1) 1 log log( N j) 1 n n (8) a s the weght of term n the ste k. TF s the frequency of

8 82 J-bn Zhang,Zh-mng Xu,Kun-l Xu,Q-shu Pan term appearng n the ste k. N s the number of all tranng stes. n s the numbers of stes whch nclude term. But when we want to weght content feature tems, we should consder not only frequency nformaton of terms but also ther locaton nformaton on pages. Some of HTML tags are mportant for reflectng topcs of pages, such as ttle, keyword, and descrpton, and they generally summarze the content of pages. In addton, the ttles, bold, talc nformaton n the body of pages are also mportant to reflect the topc of pages. So we put our emphass on consderng the mpact on pages topcs of a tag set, S= {ttle, keywords, descrptons, H1, H2, H3, B, U, I}, and enlarge the weghts of terms whch appear n tags of S. here we gve an mproved entropy weghtng formula to weght them, whch s shown as follows: w TF N w TF 1 S S a logw TF 1 1 log S log( N) j1 n n (9) where TF s the frequency of tem that appears n the ste k and locates on the tag β. w ttle keyword descrpton s the weghted coeffcent for the tag β, and let W W W H 1 H 2 H 3 W W W W U W I. 4 Experments In our web ste classfcaton experments, we use Google's navgaton ste ( as our data source, from whch we download 1127 web stes data from 16 categores, use 760 web stes data as our tranng samples, and use 367 web stes data as our testng samples. We use SVM model as our web ste classfer, and Informaton Gan method s used for feature selecton; n addton, we use tradtonal entropy weghtng method and the mproved entropy weghtng method to weght structure terms and content terms respectvely. All the experments were mplemented n C++ and tested on a PC equpped wth AMD Athlon processor and 1 GB man memory. In our web ste classfcaton experments, we use super page-based web ste classfcaton method as the baselne system, n whch we lmt the numbers of each ste s web pages under a maxmum of 50; for our web ste classfcaton method based on ts topologcal structure, we only select top 20 pages as each ste s topc-relevant data. Fg.3, Fg.4 and Fg.5 show the comparson of our web ste classfcaton method based on ts topologcal structure wth super page-based web ste classfcaton method on precson, recall and F1 value. Table 1 shows the comparson of these two web ste

9 A Web Ste Classfcaton Approach Based On Its Topologcal Structure 83 classfcaton methods on macro-averagng and mcro-averagng values. Expermental results show that our method acheves much better performance than super page-based web ste classfcaton method. Macro-averagng and mcro-averagng values can be ncreased nearly by 20% wth our method compared wth those wth super page-based method mltary agrculture metallurgy medcne prnt precson buldng fnery mechansm auto law logstcs envronment electronc gran spnnng superpage based on topology structure energy Fg.3 Comparson on precson mltary agrculture metallurgy medcne prnt buldng fnery superpage recall mechansm auto law logstcs envronment electronc gran spnnng energy based on topology structure Fg.4 Comparson on recall mltary agrculture metallurgy medcne prnt superpage F1 buldng fnery mechansm auto Fg.5 based on topology structure Comparson on F1 law logstcs envronment electronc gran spnnng energy

10 84 J-bn Zhang,Zh-mng Xu,Kun-l Xu,Q-shu Pan based on topologcal structure superpage MacroP MacroR MacroF1 McroP Table 1 Comparson on macro-averagng and mcro-averagng To nvestgate the performance of the mproved PageRank algorthm and tradtonal PageRank algorthm on web ste classfcaton, we conducted a comparson experment for them. The expermental results are showed n Table 2. Although the mproved PageRank algorthm decreases the MacroP than tradtonal PageRank algorthm, but t ncreases MacroR, MacroF1, and McroP values evdently. MacroP MacroR MacroF1 McroP Table 2 Improved PageRank PageRank The effect of Improved PageRank 5 Conclusons In ths paper, we propose a web ste classfcaton approach based on ts topologcal structure. The topologcal structure of a web ste can be represented as a drected graph. Assumed that the topc-relevant resource of a web ste s located on the strongly connected sub-graph ncludng the ste s home pages, we use an mproved PageRank algorthm based on lnk-based page smlarty, whch can effcently rank pages n the sub-graph. For effcently representng content feature and structure feature of a ste, we mx them nto a topc vector, and use an mproved entropy weghtng method to weght content terms accordng to ther frequency and locaton nformaton on pages. The expermental results of web ste classfcaton show that our web ste approach can acheve better performance than tradtonal web ste classfcaton approaches. 6 References Google: Search Engne. Yahoo: rectory Servce. DMOZ: Open Drectory Project.

11 A Web Ste Classfcaton Approach Based On Its Topologcal Structure 85 McCallum,A.and Ngam,K.,1998,A Comparson of Event Models for Naïve Bayes Text Classfcaton, Proceedngs of AAAI-98 Workshop on Learnng for Text Categorzaton. Mtchell, T. M.,1996,Machne Learnng. New York :McGraw Hll. Lam,W. and Ho,C.Y.,1998,Usng a Generalzed Instance Set for Automatc Text Categorzaton, proceedng of the 21st Ann Internatonal ACM SIGIR Conference on Research and Development n Informaton Retreval Melboume, AU, pp Masand,B., Lno,G. and Waltz,D.,1992,Classfyng News Stores Usng Memory Based Reasonng, proceedng of the 15th Annual ACM SIGIR Conference, Denmark: Copenhagen, pp Joachms,T.,1998,The Categorzaton wth Support Vector Machnes: Learnng wth Many Relevant Features, In European Conference on Machne Learnng (ECML),Chemntz,Germany, pp Kwok,J.T.Y.,1998,Automatc Text Categorzaton Usng Support Vector Machne, Proceedng of Internatonal Conference on Neural Informaton Processng, pp Chakrabart, S., Dom, B. and Indyk, P.,1998,Enhanced Hypertext Categorzaton Usng Hpyerlnks, Proceedng of the ACM SIGMOD Conference on Management of Data Seattle, Washngton, pp Craven, M., DPasquo, D., and Fretag, D., 1999,Learnng to Construct Knowledge Bases from the World Wde Web, In Artfcal Intellgence. Ester, M., Kregle, H.P., Schubert, M.,2002,Web Ste Mnng: A new way to spot Compettors, Customers and Supplers n the World Wde Web, Proceedng of 8th Internatonal Conference on Knowledge Dscovery and Data Mnng. Perre, J. M.,2001,On the Automated Classfcaton of Web Stes, Lnkopng Electronc Artcles n Computer and Informaton Scence,Vol. 6. Terveen,L., Hll,W., and Amento, B.,1999, Constructng, Organzng, and Vsualzng Collectons of Topcally Related Web Resources. ACM Trans. on Computer-Human Interacton,vol. 6,no.1,pp Ester, M., Kregel, H.P., Schubert,M.,2004,Accurate and Effcent Crawlng for Relevant Webstes, Proceedngs of the Thrteth nternatonal conference on Very large data bases, Aug, pp Kregel, H.P., Schubert, M.,2004,Classfcaton of Webstes as Sets of Feature Vectors, proceedngs of the IASTED Internatonal Conference DATABASES AND APPLICATIONS,Feb

12 86 J-bn Zhang,Zh-mng Xu,Kun-l Xu,Q-shu Pan Tan, Y.H., Huang, T.J.,and Gao, W.,2004, A Web Ste Representaton and Mnng Algorthm usng a Multscale Tree Model. Journey of Software, vol.15,no.9,pp Fu, D.Y., Da, C.Q., and Zhong, W.,2006, A Web Ste Categorzaton System Based on Key Resources, Journey of Harbn Insttute of Technology, vol.38,no.1, pp Dong, B.L., Q, G.N,and Gu, X.J.,2005, Specfc webste subject recognton based on the hybrd vector space model. Journal of Tsnghua unversty ( Sc & Tech), vol.45,pp Lu, Y., Wang, B., Yang, Z.F., and Zhang, X., 2006,Lnk Analyss n Web Key Resources Dscovery, Proceedngs of CNCCL,pp Page, L., Brn, S., and Motwan, R., 1999,The PageRank Ctaton Rankng: Brngng order to the Web, Techncal report, Stanford Dgtal Lbrares SIDL-WP Hodgson, J., 2001,Do HTML Tags Flag Semantc Content? IEEE Internet Computng, vol. 5,no.1,pp Wang, X.Y.,Xong, F.,Lng, B.,and Zhou, A.Y.,2003,A Smlarty-Based Algorthm for Topc Exploraton and Dstllaton, Journey of Software,vol.14,no.09, pp Yuan, F.Y.,and Zhang, Y.Y.,2007,The research and mprovement of relevance rankng method based on lnk analyss, Computer Engneerng and Desgn,vol.28, no.7,pp