Hierarchical Semantic Perceptron Grid based Neural Network CAO Huai-hu, YU Zhen-wei, WANG Yin-yan Abstract Key words 1.

Transcription

1 Herarchcal Semantc Perceptron Grd based Neural CAO Hua-hu, YU Zhen-we, WANG Yn-yan (Dept. Computer of Chna Unversty of Mnng and Technology Bejng, Bejng 00083, chna) Abstract A herarchcal semantc perceptron grd archtecture based neural network has been proposed n ths paper, the semantc spottng s a key ssue of ths archtecture, for fndng soluton to ths problem, frstly we has formulated t, then a Semantc neural network classfer frame has been proposed. And eperment results of ths scenaro were presented, to evaluate the effectveness of ths scenaro, we compare ths scenaro wth the SVM, NNet, KNN and NB, the average epermental results of the scenaro are obvously superor to other conventonal approaches. Key words: semantc perceptron, grd, neural network, semantc classfer, machne learnng. Introducton Knowledge and Semantc Web technologes are evolvng the Grd towards the Semantc Grd to facltate knowledge reuse and collaboraton wthn a communty of practce. The Semantc grd s about the way that knowledge s acqured, used, retreved, publshed and mantaned to assst e-scentsts to acheve ther partcular goals and objectves -knowledge s understood as nformaton appled to acheve a goal, solve a problem or enact a decson. It nvolves all three conceptual layers of the Grd: knowledge, nformaton and computaton/data.[,3] For buldng a semantc perceptron grd, above all, semantc spottng must be eactly acheved to form basc semantc grd nodes. In ths paper, we propose a herarchcal semantc perceptron grd archtecture based neural network(spgn). It uses contet at the lower layer to select the eact meanng of key words, and employs a combnaton of contet, co-occurrence statstcs and thesaurus to group the dstrbuted but semantcally related words wthn a semantc to form basc semantc nodes. The semantc nodes are then used to nfer the semantc wthn an nput document. For fndng soluton to semantc spottng problem, we has formulated t, a semantc neural network classfer(snnc) frame has been proposed. Eperments on 0000 data set demonstrate that the SNNC s able to capture the semantcs, and t performs well on semantc spottng task.. Problem descrpton and theoretcal framework Semantc spottng s the problem of dentfyng the presence of a predefned Semantc n a tet document. More formally, gven a set of n Semantcs together wth a collecton of documents, the task s to determne for each document the probablty that one or more semantcs s present n the document. Semantc spottng may be used to automatcally assgn subject codes to newswre stores, flter electronc emals and onlne news, and pre-screen document n nformaton retreval and nformaton etracton applcatons. Semantc spottng, and ts related problem of tet categoraton, has been a hot area of research for over a decade. A large number of technques have been proposed to tackle the problem, ncludng: regresson model, nearest neghbor classfcaton, Bayesan probablstc model, decson tree, nductve rule learnng, on-lne learnng, and, support vector machne (Yang & Lu, 999; Teras & Hartmann, 993)[6]. Most of these methods are word-based and consder only the relatonshps between the features and semantcs, but not the relatonshps among features. Semantc spottng theoretcal framework model s shown n Fg.. The model has two basc components: gatng networks and epert networks. The gatng networks, located at the ntermedate level nodes of the tree, receve the nput (m) (a vector of m nput features representng a document) and produce scalar output that weghts the contrbuton of the chld networks. The epert networks, located at the leaf nodes, receve the nput (m) to produce an estmate of the output. Ths model may be seen as a cascade of networks that works n a bottom-up fashon: the nput s frst presented to the eperts that generate an output, then the output of the eperts are combned by the second level gates, generatng a new output. Fnally the outputs of the second level gates are combned by the root gate to produce the approprate result y (n) (a vector y of n components where n s the number of outputs). The nodes n the tree represent the conve sum of the output of the chld nodes whch s computed as y (n) = j g j y j (n) where g j s the output of the gate and y j (n) s the output of the correspondng chld node. In the orgnal model proposed by Jordan and Jacobs all the networks n the tree are lnear (perceptrons). The epert network produces ts output y as a generaled lnear functon of the nput : y = f (U ) () where U s a weght matr and f s a fed contnuous non lnear functon. The vector s assumed to nclude a fed component wth value one to allow for an ntercept term. For bnary classfcaton problems, f ( ) s the logstc functon, n whch case the epert outputs are nterpreted as the log odds of success

2 under a Bernoull probablty model. Other models (e.g., mult-way classfcaton, countng, rate estmaton and survval estmaton) are handled by makng other choces for f ( ). The gatng networks are also generaled lnear functons. The th output of the gatng network s the softma functon of the ntermedate varable ψ (Brdle 989, McCullagh and Nelder 989) [7]. y y (n) y (n+) Root Gatng y Second Level Gatng y y 3 Global Epert y, Epert y, Epert y,3 Epert Local percepton Input document m g = Fg.. Semantc spottng theoretcal framework model e Ψ j =,..., l e Ψ Where l s the number of chld nodes of the gatng network, and the ntermedate varable ψ s defned as: Ψ = X V T (3) Where v s a weght vector and T s the transpose operaton. The g s are postve and sum to one for each. They can be nterpreted as provdng a soft parttonng of the nput space. The output vector at each nontermnal node of the tree s the weghted sum of the output of the chldren below that nontermnal. For eample, the output at the th nontermnal n the second layer of the two-level tree n fg.. s: ( n) ( n) =. j (4) y g j y. j Where j =,..., l, l s the number of chld nodes connected to the gate, y.j (n) s the output of epert j whch s a chld of gate, and g.j s the jth output of the gate. Note that snce both the g s and the y s depend on the nput, the output s a nonlnear functon of the nput. The theoretcal framework model s very fleble. One may choose a functon f (Eq..) for the gate and epert decson modules that s approprate for the applcaton. Moreover, snce gates and eperts depend only upon the nput, one may choose ether bottom-up or top-down processng whchever s approprate for the problem. In our varaton of the theoretcal framework model we use a bnary classfcaton functon at the gates. Also, we tran and use our model top-down. The choce of ths drecton was made based upon the number of categores. As wll be descrbed n detal later, we have a collecton of 6 categores n the dataset and moreover am towards a scalable categoraton procedure that can handle large classfcaton schemes. Abottom-up approach would be very neffcent gven that there s an epert module for each category. The computatonal requrements are especally severe when the decson module at each epert node s a neural network. In contrast our top-down approach along wth a bnary classfcaton at each gate restrcts the number of epert networks to be actvated for a gven document. It may be observed that n studes usng the bottom-up approach the number of categores was small as for eample 6 for handwrtng recognton and 5 for speech recognton (Waterhouse 997)[8]. In ths study we choose the more effcent drecton ()

3 and postpone the eploraton of bottom-up processng for future research. In our model, gven a bnary functon a gate s traned to yeld a value of f the eample document s categored wth any of ts descendent concepts. Durng testng, the categoraton task starts at the root node and ts gate decdes whether the most general concept s present n the document. If ths s true, then all the second level nodes make ther decsons and the process repeats untl t reaches the leaf nodes. Observe that only the eperts connected to gates that output the value are actvated thus reducng the response tme durng classfcaton. A key dfference depcted s at the gatng networks whch use bnary functons. To gve a statstcal nterpretaton of our model we defne Z,..., Z j as the path of gates from the root node to the gate j that s the parent of an epert ε k that assgns category k. Let (m) be the nput features that represent a document, and y (n) the output vector of the categores assgned to the document. The probablstc nterpretaton of our herarchcal model s as follows: P( P( y ( n) j = ) = n k = =,..., P( j = =, ) P( ) P( ε k = = =, Gven an approprate set of features and a tranng set of manually categored documents, the backpropagaton network learns to make the approprate decsons. Observe that for eperts and gates the set of postve eamples s dfferent. The set of postve eamples for the eperts s a subset of the postve eamples of any ancestor gate. As a consequence, two dentcal neural networks traned wth these dfferent subsets learn dfferent probablstc functons. The backpropagaton neural network as an epert node learns to use the nput to estmate the desred output value (category), whle as a gate t computes the confdence value of the combned outputs of ts chldren. 3. Implementaton of herarchcal semantc perceptron grd based neural network It may be observed that network archtecture and feature selecton methods must be studed n combnaton. In fact, at a basc level feature selecton s one of the factors that defne the confguraton of the network. Gven the many comple combnatons n terms of feature selecton methods and numbers of nodes n the dfferent layers for both the epert and gatng networks we approach the problem n stages[9]. We follow a top-down approach that optmes the gates and then the eperts. Frst we focused our attenton on the gatng networks. We used eperts wth 5 nput features and 50 nodes n the hdden layers. We then eplored 5, 0, 5, 50, 00, and 50 nput features for the gatng networks wth hdden layer that had twce the number of nput nodes. We also tred all three dfferent feature selecton methods (Mutual Informaton, Odds Rato and Correlaton Coeffcent). Ths eperment was done only on the hgh frequency categores because they allow approprate tranng of the neural networks and also because the varance between dfferent tranng runs s smaller than the varance for lower frequency categores. Interestngly the dfferences between the three feature selecton methods on the gatng networks are not sgnfcant. It s possble for ths slght decrease to be caused by the lmt n the number of teratons (, 000) that a network was allowed to run durng tranng. Usually a larger network needs more teratons on the tranng set to converge to an optmal value. 3.. Semantc neural network classfer frame In order to assess the advantage ganed by eplotng the herarchcal structure of the classfcaton scheme, we bult a flat neural network classfer. We decded to buld a flat modular classfer that s mplemented as a set of 03 ndvdual epert networks. In ths model the eperts are traned ndependently usng the optmal feature set and the category one for each ndvdual category []. The thresholdng step s performed by optmng the F value of each epert usng the entre tranng set. These are the values that we report n the net secton for the flat neural network classfer. Observe also that the use of ths model allows us to assess the contrbuton of addng the herarchcal structure,.e., our frst research queston. Neural networks can learn nonlnear mappngs from a set of tranng patterns. The parameters are adjusted to the tranng data n a number of teratons. We have used a back-propagaton algorthm that mnmes quadratc error. Fg.. gves a pctoral representaton of a neural network for document classfcaton. The nput layer has as many unts (neurons) as features retaned. The number of hdden unts s determned by optmng the performance on a cross-valdaton set. There s one output unt for each possble class. Hert [] and Rpley [] gve an ntroducton to neural networks that goes out of the scope ) )... (5) 3

4 of ths paper. Ths learnng process needs to be montored usng a cross-valdaton set to avod overfttng the data. We have used several cross-valdaton sets as descrbed later. T T T C C C 3 Fg.. Neural for semantc classfcaton 3. Feature selecton In tet categoraton the set of possble nput features conssts of all the dfferent words that appear n a collecton of documents. Ths s usually a large set snce even small tet collectons could have hundreds of thousands of features. Reducton of the set of features to tran the neural networks s necessary because the performance of the network and the cost of classfcaton are senstve to the se and qualty of the nput features used to tran the network (Yang and Honovar 998) [4,5]. A frst step towards reducng the se of the feature set s the elmnaton of stop words, and the use of stemmng algorthms. Even after that s done the set of features s typcally too large to be useful for tranng a neural network. Two broad approaches for feature selecton have been presented n the lterature: the wrapper approach, and the flter approach (John et al. 994) [0]. The wrapper approach attempts to dentfy the best feature subset to use wth a partcular algorthm. For eample, for a neural network the wrapper approach selects an ntal subset and measures the performance of the network; then t generates an mproved set of features and measures the performance of the network. Ths process s repeated untl t reaches a termnaton condton (ether a mnmal value of performance or a number of teratons). The flter approach, whch s more commonly used n tet categoraton, attempts to assess the merts of the feature set from the data alone. The flterng approach selects a set of features usng a preprocessng step, based on the tranng data. In ths paper we use the flter approach, but we plan to eplore the wrapper approach n future research. We select three methods that have been used n prevous works: correlaton coeffcent, mutual nformaton, and odds rato. We elmnate those stems that occur n less than 5 documents n the tranng collecton. Snce feature selecton s done for each category, based on ts one (eplaned later) we also remove stems that occur n less than 5% of the postve eample documents. We then rank the remanng stems by the feature selecton measure and select a pre-defned number of top ranked stems as the feature set. 3.3 Tranng set selecton A supervsed learnng algorthm requres the use of a tranng set n whch each element has already been correctly categored. One would epect that the avalablty of a large tranng set (such as OHSUMED) wll be benefcal for tranng the algorthm. In practce ths does not seem to be the case. The problem occurs when there s also a large collecton of categores wth each assgned to a relatvely small number of documents. Ths then creates a stuaton n whch each category has a small number of postve eamples and an overwhelmng number of negatve eamples. When a machne learnng algorthm s traned to learn the assgnment functon wth such an unbalanced tranng set, the algorthm wll learn that the best decson s to not assgn the category. The overwhelmng amount of negatve eamples hdes the assgnment functon. To overcome ths problem an approprate set of tranng eamples must be selected. We call ths tranng subset the category one. The method for creatng the category one s a KNN approach n whch the category one conssts of the set of K nearest neghbors for each postve eample of the category. Ths method wll produce varable sed category ones. We eplored several values of K (0, 50, 00 and 00). Our man concern wth ths method was to obtan a tranng set large enough to tran a neural network wthout overfttng. 4. Performance measures Table Contngency table for class j Assgned YES NO Correct YES NO a j c j b j d j 4

5 Table descrbes the possble outcomes of a bnary classfer. The system YES/NO results refer to the classfer output and the real YES/NO refers to what the correct output s. The perfect classfer would have a value of for a j jand d j, and 0 for b j and c j. Usng table we defne 3 performance measures common n the semantc classfcaton lterature. The tradeoff between recall and precson s controlled by settng the classfers parameters. To descrbe the performance both values should be provded. Another common performance measure s the F-measure defned by Rjsbergen [] F ( r, p) ( β = β β + ) pr p + r The most commonly used F-measure n tet classfcaton s F : pr F ( r, p) = p + r When dealng wth multple classes there are two possble ways of averagng these measures, namely, macro average and mcro average. In the macro averagng, one contngency table as Table 3 per class s used, the performance measures are computed on each of them and then averaged. In mcro averagng only one contngency table s used, an average of all the classes s computed for each cell and the performance measures are obtaned theren. The macro average weghts equally all the classes, regardless of how many documents belong to t. The mcro average weghts equally all the documents, thus favorng the performance on common classes. Dfferent classfers wll perform dfferent n common and rare categores. Learnng algorthms are traned more often on more populated classes thus rskng local overfttng. 5 Eperments and Results We summare here the steps followed n the preparaton of the eperments, and the results theren. a. Preparng the data: A lst of 57 stop-words were removed from the document collecton. A standard stemmng algorthm was also used. b. Vectoraton and weghtng: The resultng documents were represented as vectors, usng TF IDF weghtng. Other weghtng schemes were tred and ths was found optmum. The Smart tet processng package [9] was used for ths stage. All the eperments descrbed use a cosne normalaton. c. archtecture: The selected terms were used as nput features to the network. We tred networks wth 500, 000 and 000 nput features. The network has 90 output unts, one for each class. d. Tranng: We generated 5 cross-valdaton sets wth a number of random documents each. These documents were set asde and the network was traned on the remanng ones. As a rule of thumb, t s common to use 5-0% of the tranng set, we tred usng 500 and 000 documents, the smaller cross-valdaton set resulted n better performance. The learnng rate (η ) was reduced when the error found a mnmum on the cross-valdaton set. Reducngη durng learnng s a commonly used technque, smlar to coolng down n smulated annealng. In table we show the mcro and macro average F performance for networks wth 500, 000 and 000 nput features. In all cases the best results obtaned were for 00 hdden unts. In table 3 the results obtaned by other authors and by us are dsplayed. We show n ths table the best results for each nput dmensonalty. The results n table and 3 were obtaned usng χ ( t). We compare the performance of the classfer wth the results of other authors that used Support Vector Machnes (SVM), kth Nearest Neghbors (KNN) and Naïve Bayes (NB) classfers. The mcro averaged precson of neural networks s the best for all the methods compared, ths s partcularly mportant for many applcatons where precson s the most mportant varable. The F performance for neural networks, n partcular M a F, s not as good as the one for SVM or KNN, and ths s due to a lower recall level. The tradeoff between hgh recall or hgh precson s a well known problem and must be consdered for each applcaton. Neural networks performance has a degree of randomness, nherent to the learnng process and by dstnct ntal condtons, so t s not possble to determne the method performance by a sngle eperment. Several tranngs are needed, preferably wth dfferent cross-valdaton sets and dfferent ntal weghts. We have used 5 to 0 ndependent eperments to determne each value. ma Pr 5

6 Table. Performance of the network wth 500 and 000 nput features. 50 hdden 00 hdden 50hdden Macro F 500nputs Mcro F 500nputs 0.54+/ / / / / / Macro F 000nputs Mcro F 000nputs 0.93+/ / / / / /-0008 Macro F 000nputs Mcro F 000nputs 0.83+/ / / / / / Table 3 Performance of dfferent classfers mr mp mf maf SVM NNet KNN NB NN500np. NN000np. NN000np / / / / / / / / / / / / Concluson In ths paper, we have proposed a herarchcal semantc perceptron grd archtecture based neural network, we has formulated the semantc spottng problem, and we has proposed a semantc neural network classfer frame. The eperment results of ths scenaro were presented, to evaluate the effectveness of ths scenaro, we compare ths scenaro wth the SVM, NNet, KNN and NB, the average epermental results of the scenaro are obvously superor to other conventonal approaches. Future work ncludes tryng these methods on new data sets, we are partcularly nterested n the classfcaton of documents n other languages, other neural network schemes should also be tested. REFERENCES. Foster, I., C. Kesselman, and S. Tuecke, The Anatomy of the Grd: Enablng Scalable Vrtual Organatons. Internatonal Journal of Hgh Performance Computng Applcatons, 00. 5(3): p SLONIM, N. AND TISHBY, N. 00. The power of word clusters for tet classfcaton. In Proceedngs of ECIR-0, 3rd European Colloquum on Informaton Retreval Research (Darmstadt, Germany, 00). 3. S. Lawrence and G. Gles. Accessblty and dstrbuton of nformaton on the web. Nature, 400:07 09, Y. Yang. An evaluaton of statstcal approaches to tet categoraton. Informaton Retreval Journal, May, Y. Yang and X. Lu. A re-eamnaton of tet categoraton methods. In ACM SIGIR Conference on Research and Development n Informaton Retreval (SIGIR 99), Alan, H., Dasmahapatra, S., Gbbns, N., Glaser, H., Harrs, S., Kalfoglou, Y., O'Hara, K., and Shadbolt, N. Managng Reference: Ensurng Referental Integrty of Ontologes for the Semantc Web. 4th Internatonal Conference on Knowledge Engneerng and Knowledge Management, Span, October, Bontcheva, K. Talorng the Content of Dynamcally Generated Eplanatons. M. Bauer, P.J. Gmytrasewc, J. Vassleva (eds). User Modellng 00: 8th Internatonal Conference, UM00, Lecture Notes n Artfcal Intellgence 09, Sprnger Verlag, Cerf, V. G., et al., Natonal Collaboratores: Applyng Informaton Technologes for Scentfc Research, Natonal Academy Press: Washngton, D.C., Cravegna, F.: "Adaptve Informaton Etracton from Tet by Rule Inducton and Generalsaton" n Proceedngs of 7th Internatonal Jont Conference on Artfcal Intellgence (IJCAI 00), Seattle, August 00." 0. Andreyev Y V, Belsky YL, Dmttrev A S, et al. :neural networks mplementaton[j ] Informaton processng usng dynamcal chaos. IEEE Trans Neural s, 7.(996): J. Hert, A. Krogh, and R. Palmer. Introducton to the theory of neural computaton. Addson-Wesley, Redwood, CA, 99.. B.D. Rpley. Pattern recognton and neural networks. Cambrdge Unversty Press, Cambrdge,