Expert Systems with Applications

Transcription

1 Expert Systems with Applictions 36 (2009) Contents lists ville t ScienceDirect Expert Systems with Applictions journl homepge: A new document representtion using term frequency nd vectorized grph connectionists with ppliction to document retrievl Tommy W.S. Chow *, Hijun Zhng, M.K.M. Rhmn Deprtment of Electronic Engineering, City University of Hong Kong, 83 Tt Chee Avenue, Kowloon, Hong Kong rticle info strct Keywords: Grph representtion Multiple fetures Document retrievl Self-orgnizing mp This pper presents new document representtion with vectorized multiple fetures including term frequency nd term-connection-frequency. A document is represented y undirected nd directed grph, respectively. Then terms nd vectorized grph connectionists re extrcted from the grphs y employing severl feture extrction methods. This hyrid document feture representtion more ccurtely reflects the underlying semntics tht re difficult to chieve from the currently used term histogrms, nd it fcilittes the mtching of complex grph. In ppliction level, we develop document retrievl system sed on self-orgnizing mp (SOM) to speed up the retrievl process. We perform extensive experimentl verifiction, nd the results suggest tht the proposed method is computtionlly efficient nd ccurte for document retrievl. Ó 2009 Elsevier Ltd. All rights reserved.. Introduction Internet ccess, such s World Wide We (WWW), hs mde document retrievl incresingly demnding s collection nd serching of documents hs ecome n integrl prt of mny people s lives. Accurcy nd speed re two key mesurements of effective retrievl methodologies. Existing document retrievl systems use sttisticl methods nd nturl lnguge processing (NLP) pproches comined with different document representtion nd query structures. Document retrievl hs creted mny interests in the informtion retrievl community. Document retrievl refers to finding similr documents for given user s query. A user s query cn e rnged from full description of document to few keywords. Most of the extensively used retrievl pproches re keywords sed serching methods, e.g., in which untrined users provide few keywords to the serch engine finding the relevnt documents in returned list. Another type of document retrievl is to use query document to serch similr ones. Using n entire document s query performs well in improving retrievl ccurcy, ut it is more computtionlly demnding compred with the keywords sed method. In ddition to retrievl tsk, document clssifiction nd clustering hs lso ecome importnt in orgnizing the mssive mount of document dt, which lso uses similr feture extrction pproches to fcilitte the clssifiction nd clustering process. Until now, most conventionl models use rough document fetures, such s terms in documents s feture units. Usully the connections * Corresponding uthor. E-mil ddress: eetchow@cityu.edu.hk (T.W.S. Chow). mong terms re overlooked which results in losing importnt semntic informtion of documents. Thus, there is need of developing more effective document representtion scheme to enhnce the performnce of relevnt document dt mining. Most currently used methods of document representtion in text dt mining re sed on vector spce, proilistic nd lnguge models. The vector spce model (VSM) (Slton & McGill, 983), the most populr nd widely used tf-idf scheme, uses sic voculry of words or terms for feture description. The term frequency (tf) is the numer of occurrences of ech term, nd the inverse document frequency (idf) is function of the numer of document where term took plce. A term weighted vector is then constructed for ech document using tf nd idf. Similrity etween two documents is then mesured using cosine distnce or ny other distnce functions (Zoel & Mofft, 998). Thus, this VSM scheme reduces ritrry length of term vector in ech document to fixed length. But lengthy vector is required for descriing the frequency informtion of terms, ecuse the numer of words involved is usully huge. This cuses significnt increse of computtionl urden mking the VSM model imprcticl for lrge corpus. In ddition, VSM scheme revels little sttisticl structure out document. To overcome these shortcomings, reserchers hve proposed severl dimensionlity reduction methods such s ltent semntic indexing (LSI) (Deerwester & Dumis, 990), proilistic ltent semntic indexing (PLSI) (Hofmnn, 999), ltent Dirichlet lloction (LDA) (Blei, Ng, & Jordn, 2003) nd exponentil fmily hrmonium model (EFHM) (Welling, Rosen-Zvi, & Hinton, 2004). LSI mps the documents nd terms to ltent spce representtion y employing liner projection to compress the feture vector of the VSM model into low dimension. In ddition /$ - see front mtter Ó 2009 Elsevier Ltd. All rights reserved. doi:0.06/j.esw

2 2024 T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) to feture compression, the LSI model is useful in encoding the semntics (Berry, Dumis, & O Brien, 995). A step forwrd in proilistic models is PLSI tht defines proper genertive model of dt to model ech word in document s smple from mixture distriution nd develop fctor representtions for mixture components. By relizing overfitting prolems nd lck of description t the level of documents, Blei et l. (2003) introduced further extension in this regrd, ltent Dirichlet lloction. LDA is viewed s three-level hierrchicl Byesin model, in which ech document is modeled s finite mixture over n underlying set of topics. Using proilistic pproch then provides n explicit representtion of document. Compred with LDA, exponentil fmily hrmonium model is n lterntive two-lyer model using exponentil fmily distriutions nd the semntics of undirected models. EFHM is le to reduce the feture dimension significntly using few ltent vriles to represent document. Aprt from these proilistic models, lnguge model (Ponte & Croft, 998), n lterntive to the trditionl tf.idf relevnce models, hs ecome quite populr. The relevnce of document to given query is rnked y using sttisticl techniques nd underlying lnguge model of the document. Moreover, Erkn (2006) introduced lnguge model-sed document representtion using rndom wlks for document clustering. Speeding up retrievl system for rel time ppliction is nother eqully importnt issue. In ddition to the ove mentioned techniques for reducing feture dimension tht re le to improve the retrievl speed in some extent, other ttempts such s clustering methods nd novel system structures re suggested to speed up the retrievl opertion. Rooney, Ptterson, Glushk, nd Dorynin (2006) introduced n ide tht clustering of lrge document corpus could e used for speeding up document retrievl. In order to reduce the serching effort, the scheme is to nrrow the serching scope y compring query to group of documents tht re clustered ccording to the document nture. Fuzzy concept employing clustering techniques (Horng, Chen, Chng, & Lee, 2005; Rldvn, Tutuncu, & Allhverdi, 2007) ws lso used in document retrievl. Other thn clustering techniques, new file structure (Du, Ghnt, Mly, & Shrrock, 989) ws lso suggested to speed up the retrievl process. Despite the progress on the re of document retrievl, most reported techniques re lrgely sed on typicl term frequency informtion of g of words model. This pproch ignores the connections mong terms. In ll the ove mentioned pproches, it is noticed tht they ll use independent word s feture unit. These feture schemes re rough representtion of document. For exmple, two documents contining similr term frequencies my e contextully different when the sptil distriution of terms re very different, i.e., school, computer, nd science mens very different when they pper in different prts of document compred to the cse of school of computer science tht pper together. In ddition, with the evolution of nturl lnguge, there re incresing comintoril words emerged such s computer network, neurl network, complex network, nture network, etc. Thus, only using term frequency informtion from the g of words model is not the most effective wy to ccount contextul similrity tht includes the word inter-connections nd sptil distriution of words throughout the document. The semntics my e very different whether considering the term-connections or not. To ddress the ove shortcomings nd improve the retrievl ccurcy, we in this pper introduce grphs for document representtion tht resulting in more semntic informtion to e included. It is worth mentioning tht grph representtion for document is not new. An interesting ppliction of grph representtion descriing words links with perspective of evolving complex network for humn lnguge study cn e found in Dorogovtsev nd Mendes (200), Cncho nd Sole (200). In Schenker, Lst, Bunke, nd Kndel (2003), Schenker, Lst, Bunke, nd Kndel (2004), different directed grphs with few most frequent terms s nodes were defined to represent document, k- Nerest Neighor lgorithm (k-nn) with different grph mtching distnces sed on mximum common sugrph ws pplied to we document clssifiction. Although it is quite successful to enhnce the clssifiction ccurcy, grph mtching cn e ccomplished in polynomil time mking it imprcticl for lrge dt sets. Aprt from the computtion time limittion, there my e difficulties in finding mximum common sugrph (sugrph isomorphism) etween two documents. It is much difficult to define resonle common sugrphs tht re le to ctch the semntic similrity etween documents ecuse portions in documents re usully not exctly similr (generlly, quite few prts re similr). In this pper, in order to void time consuming mtching process, first, we extrct term-connections from grph representtions with extensive feture extrction methods. Ech document is then projected into feture vector spce forming term-connection-frequency (tcf) together with term frequency (tf). This pproch enles more semntic informtion to e utilized for document dt mining. In ppliction to retrievl process, we employ SOM to ccelerte the serching opertions y mtching ech document to topologiclly ordered neurons, nd we further use query feedck to improve the retrievl ccurcy. In summry, the contriution of this pper is twofold. First, we propose new composite vector for representing document comined with trditionl term frequency nd term-connection-frequency extrcted from grphs. Multiple fetures re le to express more semntic informtion of the word inter-connections nd sptil distriution of words throughout the document. As result, it enhnces the document retrievl ccurcy. Second, Vectorized grph connectionists fcilitte the mtching of complex grph especilly when the system hndles lrge dtsets. The vectorized grph uses fixed length vector, resulting in sustntil reduction in computtionl cost. We employ SOM together with relevnce feedck pproch to improve the computtionl efficiency nd the document retrievl ccurcy. The method using vectorized multiple fetures cn serve s unified feture extrction frmework for performing oth document retrievl nd document clssifiction. The remining sessions of this pper re orgnized s follows. Undirected nd directed grph representtions of documents re introduced in Section 2. In Section 3, three extrction schemes for term-connection fetures re descried in detils, multiple fetures re projected into vector spce. Section 4 presents the SOM implementtion for multiple fetures of documents. SOM sed retrievl system is descried in Section 5. Extensive simultion results followed y discussions re presented in Section 6. The pper ends with conclusions nd future work propositions in Section Grph representtions of documents 2.. Directed grph representtion In our work, we use grphs to represent ech document in corpus. It is quite strightforwrd to pply directed grph to express the semntics using terms in sequence ppering in the document. First, we remove the stop words (set of common words such s in, the, re, etc.) which deliver little discriminte informtion. Then, we use the rest of terms to form directed grph. A directed grph ~ G for document is denoted y ~ G ¼ð ~ V; ~ E; ~ /; ~ hþ, where, ~ V represents set of vertices (i.e. terms), ~ E is set of edges or connections etween terms, ~ / : ~ V! ~ LV ssigns n ttriute (i.e. term frequency) to ech vertex of ~ V, similrly, ~ h : ~ E! ~ LE ssigns n ttriute (i.e. term-connection-

3 T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) frequency) to ech edge of ~ E. For exmple, Fig. illustrtes how such grph would look like for sentence we found it significntly more expensive for sending money to Mexico, ut slightly less for sending money to the United Kingdom. Note tht we use only single vertex for ech term even if term ppers more thn once in the document. In n erly implementtion, we used single vertex to represent term chin consisting of two nd three words tht pper together throughout the document, ut lter found tht using only single vertex for ech term is sufficient nd improves the performnce of our ppliction. Ech vertex is leled with term frequency mesure tht indictes how mny times the relted term ppers in the document. Similrly, ech edge is leled with term-connection-frequency mesure tht indictes how mny times the connected terms pper together in the document. Here, connected mens tht two terms re djcent to ech other in the specified sequence in the document Undirected grph representtion Using directed grph ccording to the sequence of words is le to represent the semntics of document. In mny cses the sequence of words is convertile, lthough it conveys the sme semntics for humn lnguge. For exmple, computer science cn e expressed s science of computer, which delivers the sme mening. Thus, in this pper we propose to use n undirected grph for representtion of ech document. Similrly, first we remove the stop words, nd develop n undirected grph G for ech document denoted y G =(V,E,/,h) where nottions re similr to directed grph G. Fig. 2 illustrtes the sme exmple discussed in the previous section y using undirected grph. Likewise, ech vertex is leled with term frequency mesure tht indictes how mny times the relted term ppers in the document. Ech edge is leled with term-connection-frequency mesure tht indictes how mny times the connected terms pper together in the document. Here, connected mens tht two terms re djcent to ech other without differing the word sequence. 3. Multiple fetures extrction In this section, we descrie the multiple fetures (terms nd term-connections) extrction pproches to extrct more informtion from ech document for etter document nlysis. Fig.. Directed grph s n exmple: we found it significntly more expensive for sending money to Mexico, ut slightly less for sending money to the United Kingdom. (Here, we, it, more, for, to, ut, less, for, the re stop words tht re removed.). Fig. 2. Undirected grph s n exmple: we found it significntly more expensive for sending money to Mexico, ut slightly less for sending money to the United Kingdom (here, we, it, more, for, to, ut, less, for, the re stop words tht re removed). 3.. Term-frequency-sed feture extrction First, extrct ll the words from ll documents except for stop words in dtse nd pply stemming lgorithm to ech word. Here, Porter stemming lgorithm (Porter, 980) is pplied to extrct stem of ech word, nd stems re used s sic fetures insted of originl words. Thus, send, sent nd sending re ll considered the sme word. Store the stemmed words together with the informtion of term-frequency f t nd the document-frequency f t d. Then, construct the voculry sed on term-frequency fetures. We use term-weighting mesure in clculting the weight of ech word, which is similr to VSM (Slton & Buckley, 996) p W t ¼ ffiffiffi f t idf ; ðþ where the inverse-document-frequency idf ¼ log 2, nd N is the totl numer of documents in the corpus. Then, the words re sorted in descending order ccording to the weights nd the first N t words re selected to construct the voculry. The choice of N t depends on the dtse Term-connection-frequency-sed feture extrction Feture extrction of terms-connection-frequency is sed on the word voculry, which is constructed in Section 3.. We use terms in the word voculry to uild directed or undirected grph for ech document. Bsed on grph representtions, if we directly use grph mtching methods to clculte the semntic similrity like references (Schenker et l., 2004), much time nd storge spce will e wsted for lrge dtsets ecuse the djcent mtrix of ech document is so sprse. The djcent mtrix A k (k =,2,...,N) for grph G k (or ~ G k ) is denoted y A k ¼½A k ij Š NtNt where A k ij ¼ f k;tc ij represents the term-connection-frequency etween term i nd term j in document k. Then, we clculte the totl term-connection-frequency djcent mtrix for ll the documents (i.e. A ¼ P N k¼ Ak ). We lso store the document frequency f tc d;ij for term-connection etween term i nd term j in the dtse. We then use three schemes to extrct the term-connection-frequency to construct term-connection sed voculry. The first one, the simplest wy, is top term-connection-sed method tht is to select the most frequent N tc term-connections from mtrix A. Second, we use the sme weighting mesure to clculte the weight of ech term-connection for pir of terms W tc ij qffiffiffiffiffi ¼ f tc ij idf tc ij ; N f t d ð2þ

4 where B t nd B tc re the projection mtrixes with dimen- nd m tc F re feture from term-frequency nd term-connection-frequency respectively. The projected fetures in F h re ordered ccording to their sttisticl importnce. In our ppliction, oth m t F nd mtc F re set to e 00, respectively. (6) Sve the projection mtrixes for mking the fetures of new query document. The multiple fetures of query document re extrcted in the sme wy except for steps (), (2) nd (4) tht re computed only once over the dtse T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) where the inverse-document-frequency idf tc N ij ¼ log 2. Then, we f tc d;ij sion N sort the term-connections y using the weights in descending order t m t F nd N tc m tc F, respectively, mt F the dimension of the projected nd select the first N tc term-connections. Finlly, we use similr entropy-sed mesure (Lochum & Streeter, 989) to weight ech term-connection entropy ij ¼ XN k¼ A k ij log f tc 2 d;ij f tc d;ij A k ij 3.3. Projection to vector spce! : ð3þ After fetures re extrcted from documents, we cn use pproprite dt reduction methods to otin lower dimensionl feture due to the curse of dimension. Here, we pply PCA, populr tool to project higher dimensionl dt into lower dimensionl feture without losing much sttisticl informtion, to construct document histogrm vector comined with term-frequency feture nd term-connection-frequency feture. The overll procedures of extrcting multiple fetures re summrized s follows. The ove extrcted hyrid feture vector cn ctully e used in vrious document pplictions such s document clssifiction, ctegoriztion nd retrievl. In this pper, we only used document retrievl s n ppliction exmple to provide insights to the sptil structure of documents nd exhiit its performnce. Other pplictions will e esily comined to this frmework using ove hyrid document feture. () Extrct words from ll the documents in the corpus excepting for stop words nd pply stemming to ech word. Clculte the weight of ech word ccording to Eq. (), nd select the first N t words to construct term-frequency-sed voculry. (2) Build grph for ech document using selected words s nodes nd clculte the totl djcent mtrix A. Select the first N tc term-connections (or the indexes of edges in grph) sed on ove mentioned three schemes (i.e. top tcf-sed, weighted tcf-sed, nd entropy tcf-sed) to construct the term-connection-frequency-sed voculry. (3) Clculte term histogrms nd term-connection histogrms for documents tht represent the multiple fetures of documents. Ech element of the histogrms indictes the numer of times tht the corresponding term or term-connection ppers in document. Finlly, we normlize the histogrm s follows. h i H ¼ h t h t 2 h t N t h tc h tc 2 h tc ; h t i ¼ f t ðn t i Þ; htc j ¼ f tc ðn tc ; ð4þ j Þ where n t i is the frequency of ith term in the voculry, similrly, n tc j is the frequency of jth term-connection in the voculry, f t n t i nd f tc n tc j re normliztion functions. In fct, there re mny lterntive normliztion pproches such s men normliztion, mximum normliztion, or trditionl tf-idf (i.e. VSM) normliztion. In this pper, we use tf-idf normliztion method on the histogrm vector h t i ¼ nt i N P Nt log 2 ; i¼ nt ðf t Þ i i! h tc j N log 2 : ð5þ j ¼ P ntc Ntc j¼ ntc j f tc d j (4) Use the normlized histogrm to construct the PCA projection mtrix. We use the MATLAB toolox (Esen, 2005) to compute the PCA projection mtrix. (5) Project the normlized histogrm into the lower dimensionl PCA feture y using PCA projection mtrix. The PCA fetures re computed s follows F h ¼ F t h F tc h ; h i F t h ¼ ht h t 2 h t N t h F tc h ¼ htc h tc 2 h tc N tc B t ; i B tc : N tc ð6þ 4. SOM implementtion for document retrievl Self-orgnizing mp (SOM) (Kohonen, 997) is verstile unsupervised neurl network used for dimension reduction, vector quntiztion nd visuliztion. It is le to preserve topologiclly ordered output mp, where input dt re mpped into smll numer of neurons. In this wy, SOM cn form mpping from document corpus to topologiclly ordered neurons. There re mny ttempts to employ SOM for document feture projection to reduce the dt dimensionlity (Ampzis & Perntonis, 2004; Honkel, Kski, Lgus, & Kohonen, 997). SOM hs een used for document orgniztion nd we mining (Antonio et l., 2008; Georgkis, Kotropoulos, Xfopoulos, & Pits, 2004). Document clustering nd rowsing using SOM re introduced in Is, Kllimni, nd Lee (2008), Freemn nd Yin (2005). In this pper, we employ SOM to speed up the retrievl process. SOM consists of M neurons locted t regulr low dimensionl grid tht is usully in 2-D grid. The lttice of the grid is either hexgonl or rectngle. The SOM lgorithm is itertive. Ech neuron i contins d-dimensionl feture vector w i ¼ ½w i w i2 w id Š T. At ech trining step t, smple dt vector x(t) is rndomly chosen from trining set. Distnces etween x(t) nd ll the feture vectors in the grid re computed. The winning neuron, denoted y c, is the neuron with the feture vector closest to x(t) c ¼ rg mxðfðxðtþ; w i ÞÞ; i 2f; 2;...; Mg; ð7þ i where F() is distnce function to compute the similrity etween x(t) nd w i. In this pper, we use cosine distnce to define the similrity X t W t X tc W tc FðxðtÞ; w i Þ¼C þð CÞ kx t kkw t k kx tc kkw tc k ; where h i T; X t ¼ ðxðtþþ ; ðxðtþþ m t F W t ¼ w i ; w im t T; F h X tc ¼ ðxðtþþ m t þ; ðxðtþþ i T; m t F F þmtc F W tc ¼ w iðm t F þþ ; w iðm t T; F þmtc F Þ where indictes the dot product opertion, nd C(0 6 C 6 ) is weight prmeter to lnce the importnce of term-frequency feture nd term-connection-frequency feture. The first prt of the ð8þ

5 T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) expression computes the similrity sed on tf feture, nd the second prt computes the similrity sed on tcf feture. It is worth noting tht the expression provides flexiility to users to chnge the vlue of C to lnce this similrity mesure ccording to their expecttions. In this pper, the effect of the prmeter C is lso included nd studied in Section 6. Here, the neighorhood kernel function is tken s Gussin function, nd the lerning rte decreses monotoniclly with itertion. We do not include the detils of the trining process of SOM. Reders re referred to ny stndrd ook such s Kohonen (997). 5. SOM-sed retrievl system frmework 5.. Pre-processing of the retrievl system The pre-processing for document retrievl cn e summrized s follows. () Sve previously constructed voculry se nd PCA projection mtrix for multiple fetures. (2) Trin the SOM with ll the documents of the dtse. (3) Sve the indexes of trining dt with their winning neurons, which is used for the document retrievl. (4) Sve the feture vectors (i.e. weights) of the SOM neurons. Sve the SOM inputs of ll documents constructed during trining process, which is used for relevnce feedck. Thus the trined SOM is redy to perform retrievl tsk for ny new query document Document retrievl A document is represented y histogrm vector comined with term-frequency feture nd term-connection-frequency feture. Ech document is indexed ginst its winning neuron t the grid of SOM. This ssocition etween document nd neuron tht is constructed y the pre-processing stge is prepred for document retrievl. The overll SOM-sed retrievl system cn e summrized s follows. () For given query document, extrct its multiple fetures. Compute the projected feture using pre-stored voculry se nd PCA projection mtrix. (2) Mtch the projected feture to find the most similr neurons on the SOM grid nd return their ttched documents. (3) Go through the sorted neurons in descending order nd dd their ttched documents into the retrievl list until t lest user-defined N ret documents re ppended. Here, the totl numer of documents cn e lrger thn N ret. (4) Sort the documents in the retrievl list y compring the query ccording to Eq. (8). Return the first N ret documents to users Relevnce feedck In order to mke this system prcticl, we lso provide n inference to llow users to rowse through the preliminrily retrieved documents nd give the relevnce feedck to the retrievl system. In this study, we use query modifiction (Chow, Rhmn, & Wu, 2006) for relevnce feedck opertion. The modified query dt X new is otined y verging ll the fetures of the query nd relevnt documents X new ¼ N R þ X q þ XNR X r!; ð9þ r¼ where N R is the numer of relevnt documents, X q is the feture vector of the query document, nd X r is the feture vector of the rth relevnt document. After query modifiction, the retrievl process cn e summrized s follows. () Mtch the modified query feture vector X new with the neurons on the SOM grid. (2) Sort the neurons in descending order ccording to distnce with the query X new. (3) Go through the sorted neurons in descending order nd dd their ttched documents into the retrievl list until t lest user-defined N ret documents re ppended. (4) Sort the documents in the retrievl list y compring the new query with the preliminry retrievl results in step (3) nd then return the first N ret documents to users. In fct, this relevnce feedck process cn e executed in severl times. In this study, we use this process only once. 6. Experimentl results nd discussion 6.. Dtse nd experimentl setup In this study, the document dtse, Html_CityU, which consists of 25 ctegories, were used for ll simultions. Ech ctegory includes 400 documents mking totl numer of 0,000 documents. The corpus ws split into trining set nd test set tht is used for query. One thousnd test documents were rndomly selected from the 25 ctegories, i.e The remining 9000 documents were used for trining. In order to provide more rel-life testing pltform, we estlished this dtse consisting of documents with size rnged from few hundred words to over 20 thousnd words. For ech ctegory, 400 documents were retrieved from Google using set of keywords. Some of the keywords re shred mong different ctegories, ut the set of keywords for ctegory is different from tht of other ctegories. The dtse cn e found online t Prmeters of the SOM lgorithm re set s follows. The size of the SOM grid ws set t 30 30; the initil lerning rte l 0 ws set to 0.3; the initil rdius of the neighorhood function ws set to hlf-length of the SOM squre grid; the numer of totl trining itertions ws set to 27,000 (i.e. three epochs multiply 9000 documents); nd the lnce weight prmeter C for term-frequency nd term-connection-frequency feture ws set to. All the ove prmeters were found to deliver good performnce. But it ws lso noticed tht mild devition from these settings would not hve noticele effect on the overll performnce. After SOM trining, the test set ws used to verify the performnce of this work. All the simultions were performed on PC with Intel Core GHz nd 2 GB memory. The feture extrction progrms were written in Jv progrmming lnguge, nd ll the document retrievl progrms were tested in Mtl Results nd discussion In this section, we present our simultion results using multiples fetures nd SOM-sed retrievl system. In our comprtive study, we compre the results of SOM with tht of direct method y using single feture nd multiple fetures. In direct method (tht is similr to LSI), the query dt with projected feture vector is compred directly with the documents in the trining set y using cosine distnce like Eq. (8). To quntify the retrievl results, we used verged precision nd recll vlues for ech query document from the test set nd retrieving unto 360 documents. The precision nd recll mesure re defined s follows

6 2028 T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) No: of correctly retrieved documents ¼ ; No: of totl retrieved documents ð0þ No: of correctly retrieved documents Recll ¼ No: of totl documents in relevnt ctegory : ðþ First, we summrize the results y using undirected grph for document representtion with different feture extrction pproches in Tles 3. Retrieved documents, the most similr trining documents from the dtsets for every query, vry from to 360, the precision nd recll vlues in the cse of 0, 40, nd 360 documents retrieved re listed. Performnces of different retrievl pproches like SOM, RF in SOM, nd direct method re compred in different feture extrction schemes. In order to show the effect of term-connection-frequency, we summrized the vlues of precision nd recll with different numer of retrieved documents when term-connection-frequency (tcf) is used s single feture (i.e. lnce weight prmeter C = 0). We list the results with term-frequency (tf) s single feture (i.e. C = ). From Tle, sed on top-frequency feture extrction method, it is oserved tht different retrievl tools using fetures comined tf with tcf chieve significnt improvement of retrievl ccurcy with 0 retrieved documents. The results sed on the pproch of multiple fetures consistently delivers etter results thn those using single feture (either tf or tcf) when the numer of retrieved documents increses from 0 to 360. Similr results re lso otined for the recll mesurement. On the other hnd, through relevnce feedck SOM delivers the est retrievl results mong three different query types, nd pure SOM pproch performs etter thn direct method. For weighted-frequency sed feture extrction method shown in Tle 2, using multiple Tle Retrievl results with top-frequency-sed feture extrction method y using undirected grph for document representtion. Feture extrction scheme Query type Feture No. of retrieved documents (%) Recll (%) Top-frequency-sed method SOM tf + tcf tf tcf RF in SOM tf + tcf tf tcf Direct method tf + tcf tf tcf Tle 2 Retrievl results with weighted-frequency-sed feture extrction method y using undirected grph for document representtion. Feture extrction scheme Query type Feture No. of retrieved documents (%) Recll (%) Weighted-frequency-sed method SOM tf + tcf tf tcf RF in SOM tf + tcf tf tcf Direct method tf + tcf tf tcf Tle 3 Retrievl results with entropy-sed feture extrction method y using undirected grph for document representtion. Feture extrction scheme Query type Feture No. of retrieved documents (%) Recll (%) Entropy-sed method SOM tf + tcf tf tcf RF in SOM tf + tcf tf tcf Direct method tf + tcf tf tcf

7 T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) fetures otins out 2.5% improvement compred with only using tf feture. RF in SOM encourgingly delivers etter performnce with 92.40% precision nd 2.57% recll for 0 documents retrieved. From Tle 3, using multiple fetures consistently perform well sed on entropy feture extrction pproch. SOM nd RF in SOM chieve 79.44% nd 8.09% in precision mesurement, respectively even for 360 documents retrieved. Fig. 3 visully summrizes the precision results ginst numer of retrieved documents nd the precision results ginst recll vlues with top-frequency-sed feture extrction scheme. Fig. 3 shows the results of using different fetures for document retrievl, i.e. single tf feture nd comined tcf with tf feture, nd it lso shows the results of the first retrievl s well s the retrievl results fter relevnce feedck for ech feture comintion. It is oserved tht retrievl precision decreses with the increse of numer of retrieved documents. nd tcf chieves out 98% precision when only one document retrieved. Using multiple fetures is le to otin the highest improvement of ccurcy when 20 documents re retrieved. The shrp slope pproximtes right-ngle in Fig. 3() due to the reltionship etween precision nd recll mesurement. Numer of retrieved documents vrying from 50 to 200 shows interesting results. The precision vlue decreses in n insignificnt rte using tf nd tcf fetures, whilst it increses slightly for the cse of using only tf. Similr visul results re shown in Figs. 4 nd 5 y using weighted-frequency-sed feture extrction nd entropy-sed feture extrction scheme, respectively, with different feture comintions. In summry, ccording to ove quntittive nd visul results, it is oserved tht using multiple fetures is le to otin significnt improvement of retrievl ccurcy compred with using term-frequency feture only in ll the cses. This indictes tht the ddition of term-connection feture is significnt in providing discriminte informtion for document retrievl. Compred with single term-frequency feture, using multiple fetures improves No. of Retrieved Document Recll Fig. 3. Retrievl results sed on top-frequency feture extrction using undirected grph: () precision ginst no. of retrieved document nd () precision ginst recll No. of Retrieved Document Recll Fig. 4. Retrievl results sed on weighted-frequency feture extrction using undirected grph: () precision ginst no. of retrieved document nd () precision ginst recll.

8 2030 T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) t lest 2% quntittively in precision. It is more interesting to note tht SOM pproch delivers etter performnce thn direct method fter comining tf nd tcf fetures compred with the cse of using single feture. This is elieved to e minly ttriuted to the choice of normliztion methods (here, we used VSM normliztion). The use of relevnce feedck (i.e. RF-SOM) is useful s it is le to consistently improve the retrievl ccurcy from out 2% to 3% compred with the first retrievl using SOM pproch without relevnce feedck. In our study, top-frequency-sed method performs etter thn weighted-frequency-sed method, when we used SOM pproch without relevnce feedck together with undirected grph for document representtion. After relevnce feedck is conducted, weighted-frequency-sed method shows etter results thn top-frequency-sed feture extrction scheme. Interesting results re oserved for entropy feture extrction pproch. Entropy-sed method exhiits the worst performnce compred with other two feture extrction pproches when few documents re retrieved. As the numer of retrieved documents increses to 360, entropy pproch delivers the est performnce compred to other two methods No. of Retrieved Document Recll Fig. 5. Retrievl results sed on entropy feture extrction using undirected grph: () precision ginst no. of retrieved document nd () precision ginst recll. Tle 4 Retrievl results with top-frequency-sed feture extrction method y using directed grph for document representtion. Feture extrction scheme Query type Feture No. of retrieved documents (%) Recll (%) Top-frequency-sed method SOM tf + tcf tf tcf RF in SOM tf + tcf tf tcf Direct method tf + tcf tf tcf Tle 5 Retrievl results with weighted-frequency-sed feture extrction method y using directed grph for document representtion. Feture extrction scheme Query type Feture No. of retrieved documents (%) Recll (%) Weighted-frequency-sed method SOM tf + tcf tf tcf RF in SOM tf + tcf tf tcf Direct method tf + tcf tf tcf

9 T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) The reported results using recll mesurement lso deliver the similr performnce with the ove comprtive nlysis y using different types of fetures nd retrievl pproches. Tles 4 6 summrize the retrievl results y using directed grph for document representtion with different feture extrction pproches. Figs. 6 8 illustrte the visul retrievl results Tle 6 Retrievl results with entropy-sed feture extrction method y using directed grph for document representtion. Feture extrction scheme Query type Feture No. of retrieved documents (%) Recll (%) Entropy-sed method SOM tf + tcf tf tcf RF in SOM tf + tcf tf tcf Direct method tf + tcf tf tcf No. of Retrieved Document Recll Fig. 6. Retrievl results sed on top-frequency feture extrction using directed grph: () precision ginst no. of retrieved document nd () precision ginst recll No. of Retrieved Document Recll Fig. 7. Retrievl results sed on weighted-frequency feture extrction using undirected grph: () precision ginst no. of retrieved document nd () precision ginst recll.

10 2032 T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) No. of Retrieved Document Recll Fig. 8. Retrievl results sed on entropy feture extrction using undirected grph: () precision ginst no. of retrieved document nd () precision ginst recll. Tle 7 Query time with different methods. Query type SOM RF in SOM Direct method Query time (s) including different feture extrction methods. Similr comprtive retrievl results re otined y using SOM-sed retrievl system. Using multiple fetures consistently performs etter thn using single feture with precision improvement from out 2% to 3%. SOM pproch s well s relevnce feedck exhiits etter performnce thn direct method. For using directed grph for document representtion, the top-frequency-sed nd the weightedfrequency-sed feture extrction schemes deliver etter retrievl results compred with entropy-sed pproch. It is oserved tht weighted-frequency-sed feture extrction method exhiits etter performnce thn top-frequency-sed pproch in directed grph representtion. By comprtive studies nd nlyzing the cses of different document representtions, it is elieved tht using either undirected grph or directed grph is dependent on the testing dtsets. Our comprtive study nd nlysis indicte tht the superior performnce delivered y SOM-sed retrievl system is ttriuted to the co-existence of the SOM properties of self-orgnizing, topologicl ordering nd non-liner projection. In order to show the serching speed of the retrievl system, Tle 7 summrizes the verge execution time from different retrievl pproches on the sme simultion pltform. It is ovious tht the SOM pproch is much fster thn direct method with improvement of retrievl Tle 8 Retrievl results with SOM using undirected grph for document representtion from 0-fold cross-vlidtion sets. Cross-vlidtion set Query type No. of retrieved documents (%) Recll (%) SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM

11 T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) speed of out 36%. Relevnce feedck with SOM lso improves the retrievl speed y out 25%. In order to ensure the generliztion of the proposed system, we lso exmined the retrievl results from 0-fold cross-vlidtion sets. Ech vlidtion set includes 000 documents without ny overlpped dt. Tles 8 nd 9 list the precision nd recll results from 0-fold cross-vlidtion sets y using different document representtions. Similr results re otined from 0 vlidtion sets, which indicte the stility of the system. Finlly, we studied the roustness of the proposed system over SOM initiliztion. We summrized the precision results in different trining sessions in Tles 0 nd y using different grphs for document representtions. It is oserved tht SOM pproch is le to deliver similr retrievl results under different initiliztions. Tle 9 Retrievl results with SOM using directed grph for document representtion from 0-fold cross-vlidtion sets. Cross-vlidtion set Query type No. of retrieved documents (%) Recll (%) SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM Tle 0 results with SOM from different trining sessions using undirected grph for document representtion (%). Trining sessions Query type No. of retrieved documents SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM Tle results with SOM from different trining sessions using directed grph for document representtion (%). Trining sessions Query type No. of retrieved documents SOM RF in SOM SOM RF in SOM SOM RF in SOM SOM RF in SOM

12 2034 T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) vg C vg C Fig. 9. Effect of weight prmeter C: () for undirected grph nd () for directed grph Prmeter study This section studies the effect of prmeter of lnce weight C. Bsed on the ssumption tht using comined multiple fetures is le to deliver superior performnce, there must exist n optiml weight C to lnce the effect of term-frequency feture nd term-connection-frequency feture. We used the verge precision mesure to evlute the effect of different vlue of C in Eq. (8). The verge precision mesure is defined s follows. P Nmx ret i¼ precisionðiþ vg C ¼ ; ð2þ N mx ret where N mx ret is the mximum numer of retrieved documents, in our experiments, N mx, ret = 360, precision(i) is denoted s the verge precision when retrieving i documents. We included the verge precision results when C = 0.00, 0.05, 0.0,..., in Fig. 9. It is ovious tht there exists n optiml C tht is le to optimlly comine the multiple fetures ccording to our experiments. The optiml vlue of C, round 0.7 in this study, is dependent upon different dt sets. Users cn specify the reltive emphsis etween term-frequency feture nd term-connection-frequency feture y choosing n pproprite vlue of C ccording to the nture of documents. 7. Conclusion A new document representtion using multiple fetures including term frequency nd vectorized grph connectionists is proposed. This representtion turns complex grph mtching process into fixed length vector tht contins more semntic informtion, nd it cn serve unified feture extrction frmework for vrious document mining tsks. Different feture extrction methods re extensively exmined in this study. We then develop SOM-sed retrievl system in the ppliction level of the new document representtion. Experimentl results show tht vectorized multiple fetures extrcted from different grphs of document representtion re le to enhnce the retrievl ccurcy. SOM is used to speed up the retrievl process nd lso improves the ccurcy of retrievl in our experiments. With the support of relevnce feedck, the SOM-sed system consistently further enhnces the retrievl ccurcy. It is suggested tht when deling with lrge dtset like our pplictions, SOM-retrievl system is le to sve much computtion cost nd cn e prcticl tool for rel time ppliction. References Ampzis, N., & Perntonis, S. (2004). LSISOM: A ltent semntic indexing pproch to self-orgnizing mps of document collections. Neurl Processing Letters, 9(2), Antonio, S. A. et l. (2008). We mining sed on growing hierrchicl selforgnizing mps: Anlysis of rel citizen we portl. Expert Systems with Applictions, 34, Berry, M. W., Dumis, S. T., & O Brien, G. W. (995). Using liner lger for intelligent informtion retrievl. SIAM Review, 37(4), Blei, D., Ng, A., & Jordn, M. (2003). Ltent Dirichlet lloction. Journl of Mchine Lerning Reserch, 3, Cncho, R. F. I., & Sole, R. V. (200). The smll-world of humn lnguge. Proceedings of the Royl Society B: Biologicl Sciences, 268(482), Chow, T. W. S., Rhmn, M. K. M., & Wu, S. (2006). Content sed imge retrievl y using tree-structured fetures nd multi-lyer SOM. Pttern Anlysis nd Applictions, 9(), 20. Deerwester, S., & Dumis, S. (990). Indexing y ltent semntic nlysis. Journl of the Americn Society of Informtion Science, 4(6), Dorogovtsev, S. N., & Mendes, J. F. F. (200). Lnguge s n evolving word we. Proceedings of the Royl Society B: Biologicl Sciences, 268(485), Du, D. H. C., Ghnt, S., Mly, K. J., & Shrrock, S. M. (989). An efficient file structure for document retrievl in the utomted office environment. IEEE Trnsctions on Knowledge nd Dt Engineering, (2), Erkn, G. (2006). Lnguge model-sed document clustering using rndom wlks. In Proceedings of the humn lnguge technology conference of the North Americn Chpter of the ACL, New York, Americ, pp Esen HØgh-Rsmussen (2005). BBTools Mtl toolox for lck-ox computtions. Neuroiology Reserch Unit, Copenhgen University Hospitl. URL: Freemn, Richrd T., & Yin, Hujun (2005). We content mngement y selforgniztion. IEEE Trnsctions on Neurl Networks, 6(5), Georgkis, A., Kotropoulos, C., Xfopoulos, A., & Pits, I. (2004). Mrginl medin SOM for document orgniztion nd retrievl. Neurl Networks, 7(3), Hofmnn, T. (999). Proilistic ltent semntic indexing. In Proceedings of the twenty-second nnul interntionl SIGIR conference. Honkel, T., Kski, S., Lgus, K., Kohonen, T. (997). WEBSOM self-orgnizing mps of document collections. In Proceedings of WSOM 97, workshop on self-orgnizing mps, Espoo, Finlnd, Helsinki University of Technology, Neurl Networks Reserch Centre, pp Horng, Yih- Jen, Chen, Shyi-Ming, Chng, Yu-Chun, & Lee, Chi-Hong (2005). A new method for fuzzy informtion retrievl sed on fuzzy hierrchicl clustering nd fuzzy inference techniques. IEEE Trnsction on Fuzzy Systems, 3(2), Is, D., Kllimni, V. P., Lee, L. H. (2009). Using the self-orgnizing mp for clustering of text documents. Expert Systems with Applictions, 36(5),

13 T.W.S. Chow et l. / Expert Systems with Applictions 36 (2009) Kohonen, T. (997). Self-orgnizing mps. Berlin, Germny: Springer-Verlg. Lochum, K. E., & Streeter, L. A. (989). Compring nd comining the effectiveness of ltent semntic indexing nd the ordinry vector spce model for informtion retrievl. Journl of Informtion Science, 6, Ponte, J. M., Croft, W. B. (998). A lnguge modeling pproch to informtion retrievl. In Proceedings. of the 2st nnul interntionl ACM SIGIR conference on reserch nd development in informtion retrievl, Melourne, Austrli, pp Porter, M. F. (980). An lgorithm for suffix stripping. Progrm, 4(3), Rldvn, Srcoglu, Tutuncu, Keml, & Allhverdi, Novruz (2007). A fuzzy clustering pproch for finding similr documents using novel similrity mesure. Expert Systems with Applictions, 33, Rooney, N., Ptterson, D., Glushk, M., & Dorynin, V. (2006). A sclle document clustering pproch for lrge document corpor. Informtion Processing nd Mngement, 42, Slton, G., & Buckley, C. (996). Term weighting pproches in utomtic text retrievl. Informtion Processing nd Mngement, 32(4), Slton, G., & McGill, M. (Eds.). (983). Introduction to modern informtion retrievl. McGrw-Hill. Schenker, A., Lst, M., Bunke, H., Kndel, A. (2003). Clssifiction of we document using grph model. In Proceedings of the 7th interntionl conference on document nlysis nd recognition (ICDAR 03). Schenker, A., Lst, M., Bunke, H., & Kndel, A. (2004). Clssifiction of we documents using grph mtching. Interntionl Journl of Pttern Recognition nd Artificil Intelligence, 8(3), Welling, M., Rosen-Zvi, M., & Hinton, G. (2004). Exponentil fmily hrmoniums with n ppliction to informtion retrievl. Advnces in neurl informtion processing systems (vol. 7, pp ). Cmridge, MA: MIT Press. Zoel, J., & Mofft, A. (998). Exploring the similrity spce. ACM SIGIR Forum, 32(), 8 34.