Impact of a New Attribute Extraction Algorithm on Web Page Classification

Transcription

1 Impact of a New Attrbute Extracton Algorthm on Web Page Classfcaton Gösel Brc, Banu Dr, Yldz Techncal Unversty, Computer Engneerng Department Abstract Ths paper ntroduces a new algorthm for dmensonalty reducton and ts applcaton on web page classfcaton. A heterogeneous collecton of web pages s used as the dataset. Selected attrbutes for classfcaton are the textual content of pages. Usng the offered algorthm, hgh dmenson of attrbutes- words extracted from the pages- are proected onto a new hyper plane havng dmensons equal to the number of classes. Results show that processng tmes of classfcaton algorthms dramatcally decrease wth the offered reducton algorthm. Ths mostly reles on the number of attrbutes gven to classfers fall off. Accuraces of the classfcaton algorthms also ncrease compared to tests run wthout usng the proposed reducton algorthm. I. INTRODUCTION HE rapd and exponental growth of web pages on the T nternet maes dffcult to organze relevant pages together. People tend to organze data sources nto categores dependng on the content because t s easer to trac between categores and reach the desred data. Dewey and Lbrary of Congress Headngs are the examples of classfcaton systems used n lbrares to acheve categorzaton. Web pages are not logcally organzed n the nternet whch maes the data access and retreval dffcult. When we loo at the nternet, we can fnd web page classfcaton solutons n forms of web drectores le Dmoz or Yahoo!. The problem s that these drectores are formed by human effort- wth the hands and mnds of the edtors. The web s n fact several tmes bgger than the coverage of these drectores plus search engnes. That s why automaton of ths categorzaton process s mportant. Classfcaton algorthms may help edtors to classfy new comng web pages nto exstng categores. Ths s also nown as web page classfcaton. There are many solutons proposed to help solve ths problem. Dumas and Chen [1] offers herarchcal classfcaton usng SVM s on LooSmart web drectory. Cho and Peng [2] offers a dynamc and herarchcal classfcaton scheme. Mladenc uses Yahoo web drectory wth a Naïve Bayes classfer [3]. Holden and Fretas classfy ther web dataset usng ant-colony algorthm [4]. Many other researches can be found n lterature about web page classfcaton problem. One of the models wdely used n text classfcaton s the G.Brc s wth the Computer Engneerng Department, Yldz Techncal Unversty, Istanbul, Turey (e-mal: gosel@ce.yldz.edu.tr). B.Dr s wth the Computer Engneerng Department, Yldz Techncal Unversty, Istanbul, Turey (e-mal: banu@ce.yldz.edu.tr). vector space model n whch the documents are represented as vectors descrbed by a set of dentfers, for example, words as terms. Ths model s also nown as bag-of-words model, every document s ust a contaner for the words t contans. Thnng n the vector space, each term s a dmenson for the document vectors. The nature of ths representaton causes a very hgh-dmensonal and sparse feature space, whch s a common problem to deal wth when usng bag-of-words model. There are two effectve ways to overcome ths dmensonalty problem: Attrbute selecton and attrbute extracton. Attrbute selecton algorthms output a subset of the nput attrbutes, results n a lower dmensonal space. Instead of usng all words as attrbutes, attrbute selecton algorthms evaluate attrbutes on a specfc classfer to fnd the best subset of terms [5]. Ths results n reduced cost for classfcaton and better classfcaton accuracy. The most popular attrbute selecton algorthms nclude document frequency, ch statstc, nformaton gan, term strength and mutual nformaton [6]. Ch-square and correlaton coeffcent methods have been shown to produce better results than document frequency [7]. The lac of attrbute selecton algorthms s that the selecton procedure s evaluated on a certan classfer. The produced subset may not be sutable for another classfer to mprove ts performance. Attrbute extracton algorthms smplfes the amount of resource requred to descrbe a large set of data. The hghdmensonal and sparse structure of vector space model requres large amount of memory and computaton power. The am of attrbute extracton s to combne terms to form a new descrpton for the data wth suffcent accuracy. Attrbute extracton can be seen as proectng the hghdmensonal data nto a new, lower-dmensonal hyperspace. Mostly used technques are prncpal components analyss (PCA), Isomap and Latent Semantc Analyss (LSA). Latent Semantc Indexng s based on LSA and t s the mostly used algorthm n text mnng tass nowadays. Our algorthm also extracts attrbutes from the orgnal vector space, proects nto a new hyperspace wth dmensons equal to number of classes. In secton 2, we ntroduce our evaluaton dataset and the preprocessng steps. Secton 3 gves bref descrpton about related attrbute extracton algorthms and ntroduces the proposed method. In secton 4 we dscuss our expermental results. Secton 5 addresses conclusons and feature wor.

2 II. EVALUATION DATASET AND PREPROCESSING A. Evaluaton Dataset Evaluaton dataset s prepared from web pages under Dmoz [8] category World/Turce. We wor to develop an automated classfer for the Dmoz Tursh drectory. We used the dataset that we prepared for our study because the attrbute extracton algorthm we ntroduce n ths paper s a part of our system. World/Turce category ncluded unque web pages n Tursh at the tme we crawled. There are thrteen man categores n ths set: 1. Shoppng 2. News 3. Computers 4. Scence 5. Regonal 6. Recreaton 7. Busness 8. Reference 9. Arts 10. Games 11. Health 12. Sports 13. Socety Each category has several sub categores (total 758), whch are not consdered at ths tme. B. Preprocessng The dataset s crawled n raw HTML format. It s then parsed wth HTML Parser 1 to fetch the content of web pages. Fg.1. Dstrbuton of terms and documents n the dataset. We can see that most of the dataset elements contan less than 200 eywords. Term statstcs are calculated to see the outlyng documents. Fg.1 shows the dstrbuton of terms and documents, n other words, the number of words documents have and the number of documents wth a certan amount of terms. Ths plot shows us that some documents contan lots of eywords, whle some have very few. A flter smlar to Box-Plot s used to fnd the outlers n ths dstrbuton. The mean and standard devatons of the axes are calculated and a box s drawn on the dstrbuton wth the center (mean x, mean y ) and boundares (σ x, σ y ). The area wthn the box s used as the nput dataset; the samples outsde ths box are consdered as outlers and cleared. The center of the box and the boundares are also shown n Fg.1. The dataset fetched out of Dmoz s World/Turce category s n Tursh. Because Tursh s a suffx language, meanngs of the words change wth suffxes. To get rd of ths effect, all of the terms are stemmed nto ther roots by usng a Tursh stemmer tool named Zembere 2. Number of web pages Shoppng News Computers Scence Regonal Recreaton Busness Reference Classes Arts Games Health Sports Socety Fg.2. Dstrbuton of web pages among classes after preprocessng step. The fnal preprocess step s cleanng up the Tursh stop words. After the preprocessng step, we have documents (web pages) and unque eywords. Loong from the bag-of-words vew, we have a termdocument matrx wth rows and columns. Because any of the documents wll contan only a tny subset of our terms, our matrx s very sparse. The cells of the matrx contan tf-df values of terms calculated by (1), (2) and (3), where n s the number of occurrences of the consdered term n document d, D s the total number of documents, {d :t d } s the number of documents where term t appears.. n, tf, = (1) n df, D = log (2) { d : t d } tfdf, = tf, df (3) Our matrx leads us to a dmensonal vector space, 1 Avalable at 2 Avalable at

3 whch s hard to deal wth because runnng classfcaton algorthms on such a multdmensonal space s very tme and memory consumng. The dstrbuton of web pages among 13 classes s also uneven, whch causes another problem for classfcaton algorthms. Number of web pages per class after preprocessng step s gven n Fg.2. III. ATTRIBUTE EXTRACTION Attrbute extracton algorthms smplfes the amount of resource requred to descrbe a large set of data. The am of attrbute extracton s to combne terms to form a new descrpton for the data wth suffcent accuracy. In ths secton, commonly used extracton algorthms -prncpal component analyss, PCA and latent semantc analyss, LSA- and the proposed method are ntroduced. Our algorthm proects attrbutes nto a new hyperspace wth dmensons equal to number of classes. A. Related Research 1) Prncpal Component Analyss: PCA transforms correlate varables nto a smaller number of correlated varables- prncpal components. Invented by Pearson n 1901 [9], t s mostly used for exploratory data analyss. PCA s used for attrbute extracton by retanng the characterstcs of the dataset that contrbute most to ts varance, by eepng lower order prncpals, whch tend to have the most mportant aspects of data. Ths s done by a proecton nto a new hyper plane usng Egen values and Egen vectors. PCA s a popular technque n pattern recognton, but ts applcatons are not very common because t s not optmzed for class separablty [10]. It s wdely used n mage processng, 2) Latent Semantc Analyss: Patented n 1988, LSA s a technque that analyzes relatonshps between a document set and the terms that they contan by producng a set of concepts related to them [11]. As the name suggests, sngular value decomposton breas our matrx down nto a set of smaller components. LSA uses sngular value decomposton (SVD) method to fnd the relatonshps between documents. Sngular value decomposton breas term-document matrx down nto a set of smaller components. The algorthm alters one of these components (reduces the number of dmensons), and then recombnes them nto a matrx of the same shape as the orgnal, so we can agan use t as a looup grd. The matrx we get bac s an approxmaton of the term-document matrx we provded as nput, and loos much dfferent from the orgnal. LSI s mostly used for web page retreval, document clusterng purposes. It s also used for document classfcaton va votng, or nformaton flterng. Many algorthms are combned wth LSI to mprove performance by worng n a less complex hyperspace. B. Proposed Attrbute Extracton Method Our attrbute extracton algorthm nether uses Egen values, Egen vectors, nor sngular value decomposton. Weghts of terms and ther probablstc dstrbuton over the classes are taen nto account. We proect term probabltes to classes, and sum up those probabltes to get the mpact of each term to each class [15]. Assume we have I terms, J documents and K classes. Let n, be the number of occurrences of term t n document d and N be the total number of documents that contan t. We calculate the total number of occurrences of a term t n class c wth (4). We calculate the weght of term t (n a document d ) that affects class c wth (5). nc = n, d c,, (4) N w, = log, (5) ( nc ) + 1 log N We calculate the total effect of terms n a document over a class c wth (6). At the end, I terms are proected onto number of classes; we have K new terms n hand for a document. Repeatng ths procedure for all documents gves us a reduced matrx wth J rows (one row per document) and K columns (number of extracted features equals the number of classes). Fnally, we normalze new terms by usng (7). Y = w, w d, (6) Y newterm = (7) Y We gve a bref example to explan the algorthm comprehensbly. Assume we have 8 terms, 6 documents and 2 classes gven n Fg.3. The correspondng term-document matrx s n Table 1. The cells of Table 1 correspond to term weghts calculated wth (4). Fg.3. Sample dataset to explan the proposed extracton algorthm comprehensbly.

4 TABLE I TERM-DOCUMENT MATRIX OF THE SAMPLE DATASET D1 D2 D3 D4 D5 D6 t t t t t t t t We calculate term weghts over classes wth (5). Ths produces I by K matrx, gven n Table 2. After applyng (6) and (7), we have the reduced J by K matrx wth the extracted features of the processed documents. Result matrx s gven n Table 3. Ths matrx s vsualzed n Fg.4. We can also read the extracted values as the membershp probabltes of the documents to classes, as seen on Fg.4. For example, the content of D4 tells us that t s 84% Class2 and 16% Class1. Thus, the extracton method we propose may also be used as a stand-alone classfer. When we feed a new document, the system decdes whch class ths document belongs to by calculatng (6) and (7) wth the new document s content. TABLE II TERM WEIGHTS OVER CLASSES FOR THE SAMPLE DATASET C1 C2 t t t t t t t t TABLE III TERM WEIGHTS OVER CLASSES FOR THE SAMPLE DATASET C1 C2 D D D D D D Membershp probabltes 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0 D1 D2 D3 Documents D4 D5 D6 Class2 Class1 Fg.4. Vsualzaton of Table III. Extracted values can be taen as the membershp probabltes of the documents to the classes. IV. EXPERIMENTAL RESULTS We prepare our evaluaton dataset and test our attrbute extracton algorthm s performance usng 5 dfferent classfers. We use 10-fold cross-valdaton for testng. We use Naïve Bayes as a smple probablstc classfer, whch s based on applyng Bayes' theorem wth strong ndependence assumptons [12]. We choose J48 Tree, an mplementaton of Qunlan s C4.5 decson tree algorthm [13] for a basc tree based classfer, and a random forest wth 100 trees to construct a collecton of decson trees wth controlled varatons [14]. We choose radal bass functons networ for a functon based classfer and K-nearest neghbor algorthm wth K=10 to test nstance-based classfers. We evaluate the classfers performance wth ther classfcaton accuracy whch s calculated by (8) and F- Measure by usng (9).The classfcaton performance results are gven n Table 4. We can see that K-nearest neghbor classfer fals because our documents are unevenly dstrbuted between classes. Regonal and Busness classes have much more nstances than the other classes; ths s the reason why an nstance-based learner fals. RBF networ also fals because the dstrbuton of the attrbutes can not be easly modeled wth Gaussan local models. TP + TN accuracy = (8) TP + FP + FN + TN 2 TP F = (9) 2 ( TP + FP + FN ) TABLE IV RESULTS FOR THE CLASSIFIERS EVALUATED ON OUR DATASET WITHOUT USING ATTRIBUTE EXTRACTION Classfcaton F-Measure Accuracy Naïve Bayes 55.7% J48 Tree (C4.5) 57.7% Functons Networ 32.4% K-Nearest Neghbor 31.8% Random Forest (wth 100 trees) 61.4% TABLE IV RESULTS FOR THE CLASSIFIERS EVALUATED ON OUR DATASET USING THE PROPOSED ATTRIBUTE EXTRACTION METHOD Classfcaton F-Measure Accuracy Naïve Bayes 67.5% J48 Tree (C4.5) 58.7% Functons Networ 70.6% K-Nearest Neghbor 74.3% Random Forest (wth 100 trees) 75.5% 0.753

5 We gve the classfcaton performance results of the classfers usng our proposed attrbute extracton algorthm n Table 5. We can see that our attrbute extracton algorthm ncreases the classfcaton accuracy by 11.8% when usng Naïve Bayes, 1.0% usng J48, 38.2% usng RBF networ, 42.5% usng K-nearest neghbor, and 14.1% usng a random forest. Another outcome of our method s that t decreases classfcaton tmes on both classfers dramatcally. Fg.5 compares the classfcaton accuraces and Fg.6 compares F-measures wth and wthout usng our extracton method. We can see that both classfcaton accuraces and F-measures ncrease. Our method hghly mproves the performance of K-nearest neghbor classfer on unevenly dstrbuted dataset. The performance of RBF networ s also boosted because the fnal attrbutes tend to have a Gaussan le dstrbuton. Classfcaton Accuracy wthout attrbute extracton wth attrbute extracton appled 74,3% 75,5% 70,6% 67,5% 58,7% 61,4% 55,7% 57,7% Naïve Bayes J48 Tree (C4.5) 32,4% 31,8% Functons Networ K-Nearest Neghbor Random Forest (wth 100 trees) Fg.5. Comparson of classfcaton accuraces wth and wthout usng the proposed attrbute extracton algorthm. F-Measure 0,675 0,591 0,589 0,572 Naïve Bayes wthout attrbute extracton J48 Tree (C4.5) wth attrbute extracton appled 0,740 0,753 0,705 0,221 0,245 Functons Networ K-Nearest Neghbor 0,583 Random Forest (wth 100 trees) Fg.6. Comparson of F-measures of the classfers wth and wthout usng the proposed attrbute extracton algorthm. V. CONCLUSION AND FUTURE WORK We ntroduce a new algorthm for attrbute extracton and ts applcaton on web page classfcaton. We use a heterogeneous collecton of web pages crawled from Dmoz s World/Turce as the dataset. Selected attrbutes for classfcaton are the textual content of the web pages, n other words, preprocessed terms accordng to vector-space model. Usng the offered algorthm, hgh dmenson of attrbutes- terms extracted from the pages- are proected onto a new hyper plane havng dmensons equal to the number of classes. Results show that processng tmes of classfcaton algorthms dramatcally decrease wth our reducton algorthm. Ths mostly reles on the number of attrbutes gven to classfers fall off. Accuraces of the classfcaton algorthms also ncrease compared to tests run wthout usng the proposed reducton algorthm, especally on the classfers that fal on unevenly dstrbuted datasets. We plan to compare our attrbute extracton method wth other algorthms le LSA and PCA as a future wor. As we see that our method can also act as a classfer, we program to test our method as a classfer wth other classfcaton algorthms. Furthermore, we have n vew to mprove our method to ft herarchcal classes. REFERENCES [1] S.Dumas and H.Chen, Herarchcal Classfcaton of web Content, In Proc. of SIGIR-00, 23rd ACM Internatonal Conference on Research and Development n Informaton Retreval, Athens, Greece, 2000, pp [2] B.Cho, X.Peng, Dynamc and Herarchcal Classfcaton of Web Pages, Onlne Informaton Revew, vol.28, no.2, pp , [3] D.Mladenc, Turnng Yahoo nto an Automatc Web-Page Classfer, 13 th European Conference on Artfcal Intellgence Young Researcher Paper, [4] N.Holden, A.A.Fretas, Web Page Classfcaton wth an Ant Colony Algorthm, n Parallel Problem Solvng from Nature PPSN VIII, vol.3242, Sprnger Berln-Hedelberg, 2004, pp [5] Y.Ymng, J.O.Pedersen, A comparatve study on feature selecton n text categorzaton, n 14 th Internatonal Conference on Machne Learnng, 1997, pp [6] J.Zhu, H. Wang and X.Zhang, Dscrmnaton-Based Feature Selecton for Multnomal Naïve Bayes Text Classfcaton, n 2006 Proc. ICCPOL2006, LNAI 4285, pp [7] R.Jensen, Q.Shen, Computatonal Intellgence and Feature Selecton Rough and Fuzzy Approaches, IEEE-Wley, New Jersey, 2008, pp [8] Drectory Mozlla Proect, DMOZ, avalable at: [9] K.Pearson, On Lnes and Planes of Closest Ft to Systems of Ponts n Space, Phlosophcal Magazne2(6):pp , [10] K.Fuunaga, Introducton to Statstcal Pattern Recognton, Academc Press, London,1990. [11] T.K. Landauer, S.T. Dumas, Soluton to Plato's Problem: The Latent Semantc Analyss Theory of Acquston, Inducton and Representaton of Knowledge. Psychologcal Revew, 1997, vol:104, no:2, pp [12] A. McCallum, K. Ngam, "A Comparson of Event Models for Nave Bayes Text Classfcaton". In Proc. AAAI/ICML-98 Worshop on Learnng for Text Categorzaton, 1998, pp [13] J.R. Qunlan, C4.5: Programs for Machne Learnng. Morgan Kaufmann Publshers, [14] L.Breman, "Random Forests". Machne Learnng, vol:45, no:1, pp.5-32, [15] Z.Kaban, B.Dr, Recognzng Type and Author n Tursh Documents Usng Artfcal Immune Systems, 16 th Sgnal Processng and ts Applcatons Congress, Turey, 2008.