A Taxonomy Fuzzy Filtering Approach

Transcription

1 JOURNAL OF AUTOMATIC CONTROL, UNIVERSITY OF BELGRADE, VOL. 13(1):25-29, 2003 A Taxonomy Fuzzy Flterng Approach S. Vrettos and A. Stafylopats Abstract - Our work proposes the use of topc taxonomes as part of a flterng language. Gven a taxonomy, a classfer s traned for each one of ts topcs. The user s able to formulate logcal rules combnng the avalable topcs, e.g. (Topc1 AND Topc2) OR Topc3, n order to flter related documents n a stream. Usng the traned classfers, every document n the stream s assgned a belef value of belongng to the topcs of the flter. These belef values are then aggregated usng logcal operators to yeld the belef to the flter. In our study, Support Vector Machnes and Naïve Bayes classfers were used to provde topc probabltes. Aggregaton of topc probabltes based on fuzzy logc operators was found to mprove flterng performance on the Reuters text corpus, as compared to the use of ther Boolean counterparts. Fnally, we deployed a flterng system on the web usng a sample taxonomy of the Open Drectory Proect. Index Terms- Fuzzy Aggregaton, Taxonomy Flterng, Web Flterng Systems. I. INTRODUCTION The prmary way of nteractvely fndng nformaton on the web s by makng a query n a search engne and then browsng a ranked lst of possbly related web pages. Alternatvely, we can browse a manually organzed topc taxonomy to fnd pages related to the query that we have n mnd. Although web taxonomes may be very large, they cover a small porton of the web relatve to search engnes, prmarly because they rely on human effort. Content-based flterng s the task of analysng a stream of nformaton based on a semantc structure, whch can be automatcally derved or pre-specfed [1],[2]. Text/Hypertext categorzaton promses not only to help mantan updated and large web taxonomes, but to be used n the context of content-based flterng [3]-[8]. The dea s to use topc classfers that have been traned usng the porton correspondng to the well-structured web taxonomy n order to organze the results of a query addressed to the much larger but unclassfed web porton ndexed by a search engne. Bascally, as regards the nterface used to nclude topc nformaton n the query results, t can be topc or lst-orented. In topc-orented nterfaces, results are organzed n a flat or herarchcal taxonomy, whle, n lstorented nterfaces, the orgnal query lst s enrched wth topc meta-data. Our work proposes the use of topc taxonomes as part of a flterng language. The user s able to formulate logcal rules (flters) combnng the avalable topcs, e.g. ( Topc1 AND Topc2) OR Topc3, n order to flter related documents or to provde relevance feedback as well [9]-[10]. School of Electrcal and Computer Engneerng Natonal Techncal Unversty of Athens Zographou, Athens, Greece vrettos@cslab.ntua.gr, andreas@cs.ntua.gr Typcally, classfcaton s a YES/NO assgnment, so the Boolean model s a good canddate for the flterng task. Nevertheless, Boolean flterng provdes no orderng, whch s a drawback to both retreval effectveness and man-machne nteracton. If perfect classfers were avalable, Boolean flterng would be enough. That s because all the true postve documents of the stream and only them would be retreved. In that case, Boolean flterng would yeld recall and precson equal to 1. Unfortunately, no perfect classfers are avalable yet and even the best performng classfers n laboratory text corpora mght have poor results n real, nosy envronments, lke the web. In such cases, rankng accordng to some sutable measure of classfcaton accuracy s able to mprove retreval performance, ether by mprovng recall through the retreval of false negatve documents that were not ncluded n the answer set, or by mprovng precson through the orderng of true postve documents hgher n the rank, above false postve ones. In ths work, Support Vector Machnes (SVM) and Naïve Bayes (NB) classfers are used to provde topc probabltes. Aggregaton of topc probabltes based on fuzzy logc s found to mprove flterng performance on the Reuters text corpus wth respect to Boolean flterng. Fnally, fuzzy aggregaton s embedded n a web flterng system based on a sample taxonomy of the Open Drectory Proect. Support Vector Machnes II. PRODUCING TOPIC PROBABILITIES The SVM model was proposed by Vapnk n 1979 and ganed much popularty n the recent years due to ts strong theoretcal and emprcal ustfcaton [11]. In the smplest lnear form, a SVM s a hyperplane that separates a set of postve examples from a set of negatve examples wth maxmum margn (the mnmal dstance τ from the separatng hyperlane to the closest data pont).a separatng hyperplane s a lnear functon capable of separatng the tranng data n the classfcaton problem wthout error [12]. Suppose that the tranng data consst of m samples (documents) that belong or not to a gven category : (d 1,y 1 ),, (d m,y m ), d R d, y {+1,-1}, whch can be separated by a hyperplane decson functon Dd ( ) = ( wd ) + w (1) wth approprate coeffcents w and w 0. A separatng hyperplane satsfes the constrants that defne the separaton of data samples: y[( w d ) + w ] 1, = 1, K, m (2) It s ntutvely clear that a larger margn corresponds to better generalzaton. maxmzng the margn τ s equvalent to mnmzng the norm of w. An optmal hyperplane s one that satsfes condton (2) above and addtonally mnmzes 0 0

2 26 VRETTOS, S., STAFYLOPATIS, A., A TAXONOMY FUZZY FILTERING APPROACH wth respect to both w and w 0. 2 η ( w) = w (3) The data ponts that exst at the margn, or equvalentlythe data ponts for whch (2) s an equalty, defne the locaton of the decson surface and are called the support vectors. In the case of tranng data that cannot be separated wthout error, t would be desrable to separate the data wth a mnmal number of errors. To do so, we penalze examples that fall on the wrong sde of the decson boundary ntroducng postve slack varables ξ, =1,,m, to quantfy the nonseperable data n the defnng condton of the hyperplane. For a tranng sample d the slack varable ξ s the devaton from the margn border correspondng to the gven category. Slack varables greater than zero correspond to nonseparable ponts, whle slack varables greater than one correspond to msclassfed samples.to releve the problem of nonlnear separablty, a nonlnear mappng of the tranng data nto a hgh dmensonal feature space s usually performed accordng to Cover s theorem on the separablty of patterns [13]. The constraned optmzaton problem s solved usng the method of Lagrange multplers. In ths work, we used the OSU SVM Classfer Matlab Toolbox to tran classfers [14]. For each test example d, an SVM classfer outputs a score that s the dstance of d from the hyperplane learned for separatng postve from negatve examples. The sgn of the score ndcates whether the example s classfed as postve or negatve. In our approach, we want to have a measure of confdence (belef) n the predcton To provde an accurate measure of confdence, a parametrc approach was proposed by Platt for SVM, whch conssts of fndng the parameters of a sgmod functon, mappng the scores nto probablty estmates [15]. Naïve Bayes classfers The decson of whether an unseen document d belongs or not to a category c s based on the estmated belef of d belongng to class c. Bayes theorem can be used to estmate the probablty Pr(c d ), that a document d s n class c. Pr ( ) ( d c ) Pr( c ) Pr c d = (4) c Pr d c Pr c l= 1 ( ) ( ) where Pr(c ) s the pror probablty that a document s n class c and Pr(d c ) s the lkelhood of observng Prˆ c, the estmate of Pr(c ), can be document d n class c. ( ) calculated from the fracton of the tranng documents that s assgned to ths class. The probablty of observng a document lke d n class c s based on the nave assumpton that a word's occurrence n class c s ndependent of the occurrences of the other words. Therefore Pr(d c ) s: n Pr d c = Pr t c, t d (5) ( ) ( ) k = 1 k k where t k represents the k th term of the collecton document. The estmaton of Pr(d c ) s now reduced to the estmaton of Pr(t k c ) (Laplace estmator), whch s the lkelhood of observng t k n class c : l l ( t c ) Pr = n k n 1 + tf ( tk, c ) + tf ( tl, c) where tf(t k, c ) s the number of occurrences of the word t k n category c and n s the number of the terms of the corpus. III. l = 1 FUZZY LOGIC In ths secton, we brefly revew some basc concepts n fuzzy sets and fuzzy logc, that wll be used n the proposed web flterng approach. Let X be a space of obects and x be an element of X. A classcal set A s defned as a collecton of elements x A, such that each x can ether belong or not belong to the set A. Therefore, we can represent a classcal set A by a set of ordered pars (x,0) or (x,1), whch ndcates that x A or x A, respectvely. Extendng the defnton of the classcal set, a fuzzy set s defned as a set of elements that may belong to the set by a membershp degree value between 0 and 1. More formally, a fuzzy set A n X s {,, } defned as a set of ordered pars ( µ ( )) A = x x x A, where µ ( x) s the membershp functon (MF) for the fuzzy set A. Usually, X s referred to as the unverse of dscourse and may consst of dscrete (ordered or unordered) or contnuous spaces. Smlar to classcal set operatons of unon, ntersecton and complement, can be defned for fuzzy sets accordngly. The unon of two fuzzy sets A and B s a fuzzy set C, denoted C = A B or C OR B, whose MF s related to those of A and B by µ x = max µ ( x), µ x = µ x µ x (7) ( ) ( ( )) ( ) ( ) C A B A B The ntersecton of two fuzzy sets A and B s a fuzzy set C, denoted C = A Bor C AND B, whose MF s related to those of A and B by ( x) = mn ( ( x), ( x) ) = ( x) ( x) µ µ µ µ µ (8) C A B A B The complement of fuzzy set A, denoted by A, s defned as µ A( x) = 1 µ A( x) (9) IV. EVALUATION AND RESULTS To evaluate fuzzy aganst Boolean flterng, we used the Reuters corpus [16]. We consder the flat topc taxonomy that conssts of the 10 most frequently assgned topc categores. Usng the labels of the topcs we are able to formulate flters of the form (Earn) AND (Trade), (Acq) OR (Money-Fx), and use them to fnd relevant documents n a stream of data. To specfy the exact flters to use and measure ther effectveness as regards the logcal operators, we consdered the ''ModApte'' splt, a standard commonly used parttonng of the Reuters corpus nto tranng and test sets. A search n the test set of the ''ModApte'' splt yelded 213 documents that belong to 2 or more of the specfed topcs. These documents consttute the set F to be fltered and are mapped to 19 mult-category vectors n the table CM, where cm = 1 mples that category c exsts n multcategory vector m (Table I). (6)

3 JOURNAL OF AUTOMATIC CONTROL, UNIVERSITY OF BELGRADE 27 Table I The category multcategory matrx cm m 1 m m 19 c 1 cm 11 cm 1 cm 1,19 c cm 1 cm cm,19 c 10 cm 10,1 cm 10, cm 10,19 For every mult-category vector m, =1 19, of the flterng set, the logc operator AND s used to combne traned classfers of all categores havng cm = 1, n order to fnd documents n F that belong to all of them. In the same way, the logc operator OR s used to combne traned classfers of all categores havng cm = 1, n order to fnd documents n F that belong to at least one of them. Fnally, the logc operator NOT s used n conuncton wth OR to combne traned classfers of all categores havng cm = 1, n order to fnd documents n F that do not belong to any of them. As a result, we form 19 flters and obtan ther relevant documents n F for every logcal operator. To tran NB classfers, a term-document matrx was created after removng the nfrequent and the most commonly used Englsh words [17]. We reduced dmensonalty, by applyng the Document Frequency (DF) method on the tranng set leavng the 300 most nformatve features [18]. We traned and valdated one two class NB classfer for every topc usng the ''ModApte'' splt, takng as postve examples all the samples that belong to the topc and as negatve examples all the samples that do not. In that way, a document s assgned to a class when ts probablty of belongng to that class s larger than the probablty of belongng to all other classes. In ths work, however, we consdered that d s categorzed under category c when Pr(c d )>h where h s a threshold selected to optmze classfcaton accuracy on the flterng set F. In Table II, the classfcaton accuracy of NB on the flterng/test set s dsplayed. For the tranng of SVM, a term-document matrx was created after removng the nfrequent and the most commonly used Englsh words [17]. To reduce dmensonalty, we appled Prncpal Component Analyss (PCA) on the tranng set leavng the 300 most nformatve features. PCA lead to better classfcaton results than DF on the test set. We traned and valdated one two class SVM classfer for every topc usng the ''ModApte'' splt, takng as postve examples all the samples that belong to the topc and as negatve examples all the samples that do not. To obtan the best possble classfcaton accuracy we optmzed the hyperparameters of non-lnear SVM on the flterng set F. Table II shows the classfcaton accuracy of SVM on the flterng/test set. The output of each SVM was transformed to probablty usng maxmum lkelhood, as descrbed n Secton II. Operators are used to create flters that decde of the membershp of a document to a conuncton, a dsuncton or a negaton of topcs. Table II Classfcaton accuracy on the test/flterng set Topc SVM NB Samples n F Earn Acq Money-fx Crude Gran Trade Interest Shp Wheat Corn Average For every logc operator, we summarze and compare the performance of Boolean and fuzzy flterng over all flters. In the case of Boolean flterng, every document d n F was assgned a crsp value {0,1} dependng on whether t belongs to class c or not. For every flter, the related decsons were aggregated usng Boolean logc. The classc notons of precson (Pr) and recall (Re) are used to measure classfcaton effectveness. For categorzaton nto category c, let tp, fp, tn, fn be the true postve, false postve, true negatve and false negatve documents respectvely. Precson s defned as the estmated probablty that, f a random document d s categorzed under category c, ths decson s correct, that s: tp Pr = (10) tp + fp Analogously, recall s defned as the estmated probablty that, f a random document d should be categorzed under category c, ths decson s actually taken, that s: tp Re = (11) tp + fn Precson and recall are related n an nverse manner. Hgh levels of precson can be acheved by keepng recall low and vce versa. Because no orderng s avalable, Boolean flterng was evaluated by averagng recall and precson of a logcal operator over all flters (Table III). On the contrary, fuzzy flterng provdes orderng, so a standard recall-precson dagram of a logcal operator can be constructed [19]. In all cases, fuzzy aggregaton succeeded n mprovng retreval performance (Fg. 1 and 2, Table III). In the case of OR and NOT operators the mprovement was due to hgher precson. Ths means that true postve documents are placed hgh n the ranked answer set. In the case of AND operators fuzzy aggregaton managed to mprove both recall and precson. Table III Average recall and precson for all flters and operators SVM NB Operator Precson Recall Precson Recall AND OR NOT

4 28 VRETTOS, S., STAFYLOPATIS, A., A TAXONOMY FUZZY FILTERING APPROACH pages. Queres are forwarded to ODP and the results are parsed and gven topc probabltes accordng to the taxonomy. Fnally, these probabltes are aggregated accordng to the flters and the results are fed back to the clent. Clent Sded Flterng Server Sded Flterng Clent User Clent User Server Classfers Interface Classfers Interface Indexng Server Indexng Fg. 1. Results for NB classfers Taxonomy: Bookmarks Search Engne-(s) Taxonomy: Drectory Search Engne-(s) Fg. 3. Clent vs server sded flterng systems CLIENTS (Html Java Applcatons Applets) SERVLETS RMI HTML WRAPPER TAXONOMY BUILDER QUERY- FILTER TCP/IP SEARCH ENGINES Fg. 4. The archtecture of the flterng system Fg. 2. Results for SVM classfer V. AN EXAMPLE FILTERING SYSTEM Generally, an Informaton Flterng system can be on the server or on the clent sde (Fg. 3). The proposed fuzzy flterng approach can be used both ways. In clent sded flterng, the taxonomy may be n the form of user s bookmarks, for example. The system creates the topc classfers that the user uses to flter the results of a search engne accordng to the descrbed method. In server sded flterng, the taxonomy may be n the form of a web drectory. In order to provde an example applcaton, we have developed a server sded flterng system on the web usng the Open Drectory Proect (ODP) [20]. The system bascally conssts of an Html Wrapper ( and a Taxonomy Bulder located on a server (Fg. 4). In that way we can take advantage of powerful servers for the tranng of classfers and the parsng of large quanttes of html pages. Clents are n the form of html pages, applets and ava applcatons that communcate wth the server usng servlets or RMI (Remote Method Invocaton). We created NB classfers for 10 topcs related to the Computers/Artfcal Intellgence drectory of the ODP, usng about 40 web pages as tranng examples for each topc. Through the nterface (html clent), the user s able to create queres and flters n order to retreve and flter web VI. CONCLUSIONS We have descrbed and evaluated a framework that can take advantage of a topc taxonomy as part of a flterng language. We used SVM and NB classfers on the Reuters corpus to create flters that decde of the membershp of a document to a conuncton, dsuncton or negaton of topcs. Fuzzy aggregaton of the estmated topc probabltes proved to exhbt superor performance than Boolean aggregaton for all knds of flters. Fnally, we deployed a flterng system based on ths framework usng a sample taxonomy of the Open Drectory Proect. REFERENCES [1] 1. B.U. Oztekn, G. Karyps and V. Kumar, Expert Agreement and Content Based Rerankng n Meta Search Envronment usng Mearf, Proc. WWW2002, Honolulu, May [2] S. Vrettos and A. Stafylopats, A Fuzzy Rule-Based Agent for Web Retreval Flterng, Proc. of Frst Asa-Pasfc Conference on Web Intellgence (WI 2001), Maebash Cty, Japan, October 2001, pp [3] Y. Yang, An evaluaton of statstcal approaches to text categorzaton, Journal of Informaton retreval, vol. 1,1999, pp [4] Y. Yang and X. Lu, A re-examnaton of text categorzaton methods, Proc. of SIGIR-99, 22nd ACM Internatonal Conference on Research and Development n Informaton Retreval, [5] H. Chen and S.T. Dumas, Brngng order to the Web: Automatcally categorzng search results, Proc. of ACM SIGCHI Conference on Human Factors In Computng Systems(CHI), 2000, pp

5 JOURNAL OF AUTOMATIC CONTROL, UNIVERSITY OF BELGRADE 29 [6] S. Dumas, E. Cutrell and H. Chen, Optmzng Search by Showng Results In Context,, Proc. of SIGCHI,Seattle, WA, USA,March 31,2001, pp [7] C. Chekur, M. Goldwasser, P. Raghavan and E. Upfal, Web search usng automated classfcaton, Proc. of Sxth Internatonal World Wde Web Conference, Poster POS725,Santa Clara, Calforna,Aprl [8] S. Chakrabart, B. Dom, R. Agrawal and P. Raghavan, Scalable feature selecton, classfcaton and sgnature generaton for organzng large text databases nto herarchcal topc taxonomes, The VLDB Journal, vol. 7, 1998, pp [9] Vrettos, S. and Stafylopats A, A Fuzzy logc Framework for Web Page Flterng, Proc. 6th Semnar on Neural Networks Applcatons n Electrcal Engneerng NEUREL 2002, September 2002, pp [10] M.A. Hearst, Improvng Full-Text Precson on Short Queres usng Smple Constrants, Proc. of SDAIR,Las Vegas, NV, Aprl,1996. [11] Susan Dumas, John Platt, Davd Heckerman, and Mehran Saham, Inductve learnng algorthms and representatons for text categorzaton, Proc. of ACM Conference on Informaton and Knowledge Management, 1998, pp [12] Vladmr Cherkassky and Flp Muler, Learnng from Data, (Jown Wley & Sons, 1998). [13] Smon Haykn, Neural Networks a Comprehensve Foundaton (Prentce Hall, 1999). [14] OSU SVM Classfer Matlab Toolbox (ver 3.00) [15] John C. Platt, Probabltes for SV Machnes, Advances n Large Margn Classfers, (The MIT Press, Cambrdge, 2000). [16] The Reuters collecton: [17] 17. T. Joachms, A probablstc Analyss of the Roccho Algorthm wth TFIDF for Text Categorzaton, Proc. of the 14th Internatonal Conference on Machne Learnng ICML97, 1997, pp [18] Ymng Yang and Jan O. Pedersen, A Comparatve Study on Feature Selecton n Text Categorzaton, Proc. of ICML-97, 14th Internatonal Conference on Machne Learnng, [19] Rcardo Baeza-Yates and Berther Rbero Neto, Modern Informaton Retreval, (ACM Press Books, 1999). [20] Open Drectory Proect. [