Making Tree Kernels practical for Natural Language Learning

Transcription

1 Mking Tree Kernels prcticl for turl Lnguge Lerning Alessndro Moschitti eprtment of Computer Science University of Rome Tor ergt Rome, Itly Abstrct In recent yers tree kernels hve been proposed for the utomtic lerning of nturl lnguge pplictions. Unfortuntely, they show () n inherent super liner complexity nd (b) lower ccurcy thn trditionl ttribute/vlue methods. In this pper, we show tht tree kernels re very helpful in the processing of nturl lnguge s () we provide simple lgorithm to compute tree kernels in liner verge running time nd (b) our study on the clssifiction properties of diverse tree kernels show tht kernel combintions lwys improve the trditionl methods. Experiments with Support ector Mchines on the predicte rgument clssifiction tsk provide empiricl support to our thesis. 1 Introduction In recent yers tree kernels hve been shown to be interesting pproches for the modeling of syntctic informtion in nturl lnguge tsks, e.g. syntctic prsing (Collins nd uffy, 2002), reltion extrction (Zelenko et l., 2003), med Entity recognition (Cumby nd Roth, 2003; Culott nd Sorensen, 2004) nd Semntic Prsing (Moschitti, 2004). The min tree kernel dvntge is the possibility to generte high number of syntctic fetures nd let the lerning lgorithm to select those most relevnt for specific ppliction. In contrst, their mjor drwbck re () the computtionl time complexity which is superliner in the number of tree nodes nd (b) the ccurcy tht they produce is often lower thn the one provided by liner models on mnully designed fetures. To solve problem (), liner complexity lgorithm for the subtree (ST) kernel computtion, ws designed in (ishwnthn nd Smol, 2002). Unfortuntely, the ST set is rther poorer thn the one generted by the subset tree (SST) kernel designed in (Collins nd uffy, 2002). Intuitively, n ST rooted in node n of the trget tree lwys contins ll n s descendnts until the leves. This does not hold for the SSTs whose leves cn be internl nodes. To solve the problem (b), study on different tree substructure spces should be crried out to derive the tree kernel tht provide the highest ccurcy. On the one hnd, SSTs provide lerning lgorithms with richer informtion which my be criticl to cpture syntctic properties of prse trees s shown, for exmple, in (Zelenko et l., 2003; Moschitti, 2004). On the other hnd, if the SST spce contins too mny irrelevnt fetures, overfitting my occur nd decrese the clssifiction ccurcy (Cumby nd Roth, 2003). As consequence, the fewer fetures of the ST pproch my be more pproprite. In this pper, we im to solve the bove problems. We present () n lgorithm for the evlution of the ST nd SST kernels which runs in liner verge time nd (b) study of the impct of diverse tree kernels on the ccurcy of Support ector Mchines (SMs). Our fst lgorithm computes the kernels between two syntctic prse trees in O(m + n) verge time, where m nd n re the number of nodes in the two trees. This low complexity llows SMs to crry out experiments on hundreds of thousnds of trining instnces since it is not higher thn the complexity of the polynomil ker-

2 nel, widely used on lrge experimenttion e.g. (Prdhn et l., 2004). To confirm such hypothesis, we mesured the impct of the lgorithm on the time required by SMs for the lerning of bout 122,774 predicte rgument exmples nnotted in PropBnk (Kingsbury nd Plmer, 2002) nd 37,948 instnces nnotted in Frmeet (Fillmore, 1982). Regrding the clssifiction properties, we studied the rgument lbeling ccurcy of ST nd SST kernels nd their combintions with the stndrd fetures (Gilde nd Jurfsky, 2002). The results show tht, on both PropBnk nd Frmeet dtsets, the SST-bsed kernel, i.e. the richest in terms of substructures, produces the highest SM ccurcy. When SSTs re combined with the mnul designed fetures, we lwys obtin the best figure clssifier. This suggests tht the mny frgments included in the SST spce re relevnt nd, since their mnul design my be problemtic (requiring higher progrmming effort nd deeper knowledge of the linguistic phenomenon), tree kernels provide remrkble help in feture engineering. In the reminder of this pper, Section 2 describes the prse tree kernels nd our fst lgorithm. Section 3 introduces the predicte rgument clssifiction problem nd its solution. Section 4 shows the comprtive performnce in term of the execution time nd ccurcy. Finlly, Section 5 discusses the relted work wheres Section 6 summrizes the conclusions. 2 Fst Prse Tree Kernels The kernels tht we consider represent trees in terms of their substructures (frgments). These ltter define feture spces which, in turn, re mpped into vector spces, e.g. R n. The ssocited kernel function mesures the similrity between two trees by counting the number of their common frgments. More precisely, kernel function detects if tree subprt (common to both trees) belongs to the feture spce tht we intend to generte. For such purpose, the frgment types need to be described. We consider two importnt chrcteriztions: the subtrees (STs) nd the subset trees (SSTs). 2.1 Subtrees nd Subset Trees In our study, we consider syntctic prse trees, consequently, ech node with its children is ssocited with grmmr production rule, where the symbol t left-hnd side corresponds to the prent node nd the symbols t right-hnd side re ssocited with its children. The terminl symbols of the grmmr re lwys ssocited with the leves of the tree. For exmple, Figure 1 illustrtes the syntctic prse of the sentence "Mry ct to school". The root A lef Mry A subtree S P S ct P PP PP PP I I school to school Figure 1: A syntctic prse tree. We define s subtree (ST) ny node of tree long with ll its descendnts. For exmple, the line in Figure 1 circles the subtree rooted in the P node. A subset tree (SST) is more generl structure. The difference with the subtrees is tht the leves cn be ssocited with non-terminl symbols. The SSTs stisfy the constrint tht they re generted by pplying the sme grmmticl rule set which generted the originl tree. For exmple, [S [ ]] is SST of the tree in Figure 1 which hs two non-terminl symbols, nd, s leves. Mry S P ct ct Mry Figure 2: A syntctic prse tree with its subtrees (STs). P ct P ct P P P ct ct P ct P Figure 3: A tree with some of its subset trees (SSTs). P P ct ct Mry Given syntctic tree we cn use s feture representtion the set of ll its STs or SSTs. For exmple, Figure 2 shows the prse tree of the sentence "Mry ct" together with its 6 STs, wheres Figure 3 shows 10 SSTs (out of 17) of the subtree of Figure 2 rooted in. The

3 high different number of substructures gives n intuitive quntifiction of the different informtion level between the two tree-bsed representtions. 2.2 The Tree Kernel Functions The min ide of tree kernels is to compute the number of the common substructures between two trees T 1 nd T 2 without explicitly considering the whole frgment spce. For this purpose, we slightly modified the kernel function proposed in (Collins nd uffy, 2002) by introducing prmeter σ which enbles the ST or the SST evlution. Given the set of frgments {f 1, f 2,..} = F, we defined the indictor function I i (n) which is equl 1 if the trget f i is rooted t node n nd 0 otherwise. We define K(T 1, T 2 ) = (n 1, n 2 ) (1) n 2 T2 n 1 T1 where T1 nd T2 re the sets of the T 1 s nd T 2 s nodes, respectively nd (n 1, n 2 ) = F i=1 I i(n 1 )I i (n 2 ). This ltter is equl to the number of common frgments rooted in the n 1 nd n 2 nodes. We cn compute s follows: 1. if the productions t n 1 nd n 2 re different then (n 1, n 2 ) = 0; 2. if the productions t n 1 nd n 2 re the sme, nd n 1 nd n 2 hve only lef children (i.e. they re pre-terminls symbols) then (n 1, n 2 ) = 1; 3. if the productions t n 1 nd n 2 re the sme, nd n 1 nd n 2 re not pre-terminls then (n 1, n 2 ) = nc(n 1 ) j=1 (σ + (c j n 1, c j n 2 )) (2) where σ {0, 1}, nc(n 1 ) is the number of the children of n 1 nd c j n is the j-th child of the node n. ote tht, since the productions re the sme, nc(n 1 ) = nc(n 2 ). When σ = 0, (n 1, n 2 ) is equl 1 only if j (c j n 1, c j n 2 ) = 1, i.e. ll the productions ssocited with the children re identicl. By recursively pplying this property, it follows tht the subtrees in n 1 nd n 2 re identicl. Thus, Eq. 1 evlutes the subtree (ST) kernel. When σ = 1, (n 1, n 2 ) evlutes the number of SSTs common to n 1 nd n 2 s proved in (Collins nd uffy, 2002). Additionlly, we study some vritions of the bove kernels which include the leves in the frgment spce. For this purpose, it is enough to dd the condition: 0. if n 1 nd n 2 re leves nd their ssocited symbols re equl then (n 1, n 2 ) = 1, to the recursive rule set for the evlution (Zhng nd Lee, 2003). We will refer to such extended kernels s ST+bow nd SST+bow (bg-ofwords). Moreover, we dd the decy fctor λ by modifying steps (2) nd (3) s follows 1 : 2. (n 1, n 2 ) = λ, 3. (n 1, n 2 ) = λ nc(n 1 ) j=1 (σ + (c j n 1, c j n 2 )). The computtionl complexity of Eq. 1 is O( T1 T2 ). We will refer to this bsic implementtion s the Qudrtic Tree Kernel (QTK). However, s observed in (Collins nd uffy, 2002) this worst cse is quite unlikely for the syntctic trees of nturl lnguge sentences, thus, we cn design lgorithms tht run in liner time on verge. function Evlute Pir Set(Tree T 1, T 2 ) returns OE PAIR SET; LIST L 1,L 2 ; OE PAIR SET p ; begin L 1 = T 1.ordered list; L 2 = T 2.ordered list; /*the lists were sorted t loding time*/ n 1 = extrct(l 1 ); /*get the hed element nd*/ n 2 = extrct(l 2 ); /*remove it from the list*/ while (n 1 nd n 2 re not ULL) if (production of(n 1 ) > production of(n 2 )) then n 2 = extrct(l 2 ); else if (production of(n 1 ) < production of(n 2 )) then n 1 = extrct(l 1 ); else while (production of(n 1 ) == production of(n 2 )) while (production of(n 1 ) == production of(n 2 )) dd( n 1, n 2, p ); n 2 =get next elem(l 2 ); /*get the hed element nd move the pointer to the next element*/ end n 1 = extrct(l 1); reset(l 2); /*set the pointer t the first element*/ end end return p ; end Tble 1: Pseudo-code for fst evlution of the node pir sets used in the fst Tree Kernel. 2.3 A Fst Tree Kernel (FTK) To compute the kernels defined in the previous section, we sum the function for ech pir n 1, n 2 T1 T2 (Eq. 1). When the productions ssocited with n 1 nd n 2 re different, we cn void to evlute (n 1, n 2 ) since it is 0. 1 To hve similrity score between 0 nd 1, we lso pply the normliztion in the kernel spce, i.e. K (T 1, T 2 ) = K(T 1,T 2 ) K(T. 1,T 1 ) K(T 2,T 2 )

4 S S Arg0 S S Arg1 S ArgM Mry Arg. 0 Predicte P ct Arg. 1 PP I to school Arg. M Mry P ct I to PP school Figure 4: Tree substructure spce for predicte rgument clssifiction. Thus, we look for node pir set p ={ n 1, n 2 T1 T2 : p(n 1 ) = p(n 2 )}, where p(n) returns the production rule ssocited with n. To efficiently build p, we (i) extrct the L 1 nd L 2 lists of the production rules from T 1 nd T 2, (ii) sort them in the lphnumeric order nd (iii) scn them to find the node pirs n 1, n 2 such tht (p(n 1 ) = p(n 2 )) L 1 L 2. Step (iii) my require only O( T1 + T2 ) time, but, if p(n 1 ) ppers r 1 times in T 1 nd p(n 2 ) is repeted r 2 times in T 2, we need to consider r 1 r 2 pirs. The forml lgorithm is given in Tble 1. ote tht: () The list sorting cn be done only once t the dt preprtion time (i.e. before trining) in O( T1 log( T1 )). (b) The lgorithm shows tht the worst cse occurs when the prse trees re both generted using only one production rule, i.e. the two internl while cycles crry out T1 T2 itertions. In contrst, two identicl prse trees my generte liner number of non-null pirs if there re few groups of nodes ssocited with the sme production rule. (c) Such pproch is perfectly comptible with the dynmic progrmming lgorithm which computes. In fct, the only difference with the originl pproch is tht the mtrix entries corresponding to pirs of different production rules re not considered. Since such entries contin null vlues they do not ffect the ppliction of the originl dynmic progrmming. Moreover, the order of the pir evlution cn be estblished t run time, strting from the root nodes towrds the children. 3 A Semntic Appliction of Prse Tree Kernels An interesting ppliction of the SST kernel is the clssifiction of the predicte rgument structures defined in PropBnk (Kingsbury nd Plmer, 2002) or Frmeet (Fillmore, 1982). Figure 4 shows the prse tree of the sentence: "Mry ct to school" long with the predicte rgument nnottion proposed in the Prop- Bnk project. Only verbs re considered s predictes wheres rguments re lbeled sequentilly from ARG0 to ARG9. Also in Frmeet predicte/rgument informtion is described but for this purpose richer semntic structures clled Frmes re used. The Frmes re schemtic representtions of situtions involving vrious prticipnts, properties nd roles in which word my be typiclly used. Frme elements or semntic roles re rguments of predictes clled trget words. For exmple the following sentence is nnotted ccording to the AR- REST frme: [ T ime One Sturdy night] [ Authorities police in Brooklyn ] [ T rget pprehended ] [ Suspect sixteen teengers]. The roles Suspect nd Authorities re specific to the frme. The common pproch to lern the clssifiction of predicte rguments reltes to the extrction of fetures from the syntctic prse tree of the trget sentence. In (Gilde nd Jurfsky, 2002) seven different fetures 2, which im to cpture the reltion between the predicte nd its rguments, were proposed. For exmple, the Prse Tree Pth of the pir, ARG1 in the syntctic tree of Figure 4 is P. It encodes the dependency between the predicte nd the rgument s sequence of nonterminl lbels linked by direction symbols (up or down). An lterntive tree kernel representtion, proposed in (Moschitti, 2004), is the selection of the miniml tree subset tht includes predicte with only one of its rguments. For exmple, in Figure 4, the substructures inside the three frmes re the semntic/syntctic structures ssocited with the three rguments of the verb to bring, i.e. S ARG0, S ARG1 nd S ARGM. Given feture representtion of predicte r- 2 mely, they re Phrse Type, Prse Tree Pth, Predicte Word, Hed Word, Governing Ctegory, Position nd oice.

5 guments, we cn build n individul OE-vs-ALL (OA) clssifier C i for ech rgument i. As finl decision of the multiclssifier, we select the rgument type ARG t ssocited with the mximum vlue mong the scores provided by the C i, i.e. t = rgmx i S score(c i ), where S is the set of rgument types. We dopted the OA pproch s it is simple nd effective s showed in (Prdhn et l., 2004). ote tht the representtion in Figure 4 is quite intuitive nd, to conceive it, the designer requires much less linguistic knowledge bout semntic roles thn those necessry to define relevnt fetures mnully. To understnd such point, we should mke step bck before Gilde nd Jurfsky defined the first set of fetures for Semntic Role Lbeling (SRL). The ide tht syntx my hve been useful to derive semntic informtion ws lredy inspired by linguists, but from mchine lerning point of view, to decide which tree frgments my hve been useful for semntic role lbeling ws not n esy tsk. In principle, the designer should hve hd to select nd experiment ll possible tree subprts. This is exctly wht the tree kernels cn utomticlly do: the designer just need to roughly select the interesting whole subtree (correlted with the linguistic phenomenon) nd the tree kernel will generte ll possible syntctic fetures from it. The tsk of selecting the most relevnt substructures is crried out by the kernel mchines themselves. 4 The Experiments The im of the experiments is twofold. On the one hnd, we show tht the FTK running time is liner on the verge cse nd is much fster thn QTK. This is ccomplished by mesuring the lerning time nd the verge kernel computtion time. On the other hnd, we study the impct of the different tree bsed kernels on the predicte rgument clssifiction ccurcy. 4.1 Experimentl Set-up We used two different corpor: PropBnk ( ce) long with PennTree bnk 2 (Mrcus et l., 1993) nd Frmeet. PropBnk contins bout 53,700 sentences nd fixed split between trining nd testing which hs been used in other reserches, e.g. (Gilde nd Plmer, 2002; Prdhn et l., 2004). In this split, sections from 02 to 21 re used for trining, section 23 for testing nd sections 1 nd 22 s developing set. We considered totl of 122,774 nd 7,359 rguments (from ARG0 to ARG9, ARGA nd ARGM) in trining nd testing, respectively. Their tree structures were extrcted from the Penn Treebnk. It should be noted tht the min contribution to the globl ccurcy is given by ARG0, ARG1 nd ARGM. From the Frmeet corpus ( frmenet), we extrcted ll 24,558 sentences of the 40 Frmes selected for the Automtic Lbeling of Semntic Roles tsk of Sensevl 3 ( We mpped together the semntic roles hving the sme nme nd we considered only the 18 most frequent roles ssocited with verbl predictes, for totl of 37,948 rguments. We rndomly selected 30% of sentences for testing nd 70% for trining. Additionlly, 30% of trining ws used s vlidtionset. ote tht, since the Frmeet dt does not include deep syntctic tree nnottion, we processed the Frmeet dt with Collins prser (Collins, 1997), consequently, the experiments on Frmeet relte to utomtic syntctic prse trees. The clssifier evlutions were crried out with the SM-light-TK softwre vilble t which encodes ST nd SST kernels in the SMlight softwre (Jochims, 1999). We used the defult liner (Liner) nd polynomil (Poly) kernels for the evlutions with the stndrd fetures defined in (Gilde nd Jurfsky, 2002). We dopted the defult regulriztion prmeter (i.e., the verge of 1/ x ) nd we tried few cost-fctor vlues (i.e., j {1, 3, 7, 10, 30, 100}) to djust the rte between Precision nd Recll on the vlidtion-set. For the ST nd SST kernels, we derived tht the best λ (see Section 2.2) were 1 nd 0.4, respectively. The clssifiction performnce ws evluted using the F 1 mesure 3 for the single rguments nd the ccurcy for the finl multiclssifier. This ltter choice llows us to compre our results with previous literture work, e.g. (Gilde nd Jurfsky, 2002; Prdhn et l., 2004). 4.2 Time Complexity Experiments In this section we compre our Fst Tree Kernel (FTK) pproch with the Qudrtic Tree Kernel (QTK) lgorithm. The ltter refers to the nive evlution of Eq. 1 s presented in (Collins nd uffy, 2002). 3 F 1 ssigns equl importnce to Precision P nd Recll R, i.e. f 1 = 2P R P +R.

6 Figure 5 shows the lerning time 4 of the SMs using QTK nd FTK (over the SST structures) for the clssifiction of one lrge rgument (i.e. ARG0), ccording to different percentges of trining dt. We note tht, with 70% of the trining dt, FTK is bout 10 times fster thn QTK. With ll the trining dt FTK terminted in 6 hours wheres QTK required more thn 1 week. Hours y = x x y = x x % Trining t FTK QTK Figure 5: ARG0 clssifier lerning time ccording to different trining percentges. µseconds y = 0.04x x y = 0.14x umber of Tree odes FTK QTK Figure 6: Averge time in seconds for the QTK nd FTK evlutions. Accurcy ST SST 0.80 ST+bow SST+bow 0.78 Liner Poly % Trining t Figure 7: Multiclssifier ccurcy ccording to different trining set percentges. 4 We run the experiments on Pentium 4, 2GHz, with 1 Gb rm. The bove results re quite interesting becuse they show tht (1) we cn use tree kernels with SMs on huge trining sets, e.g. on 122,774 instnces nd (2) the time needed to converge is pproximtely the one required by SMs when using polynomil kernel. This ltter shows the miniml complexity needed to work in the dul spce. To study the FTK running time, we extrcted from PennTree bnk the first 500 trees 5 contining exctly n nodes, then, we evluted ll 25,000 possible tree pirs. Ech point of the Figure 6 shows the verge computtion time on ll the tree pirs of fixed size n. In the figures, the trend lines which best interpoltes the experimentl vlues re lso shown. It clerly ppers tht the trining time is qudrtic s SMs hve qudrtic lerning time complexity (see Figure 5) wheres the FTK running time hs liner behvior (Figure 6). The QTK lgorithm shows qudrtic running time complexity, s expected. 4.3 Accurcy of the Tree Kernels In these experiments, we investigte which kernel is the most ccurte for the predicte rgument clssifiction. First, we run ST, SST, ST+bow, SST+bow, Liner nd Poly kernels over different trining-set size of PropBnk. Figure 7 shows the lerning curves ssocited with the bove kernels for the SMbsed multiclssifier. We note tht () SSTs hve higher ccurcy thn STs, (b) bow does not improve either ST or SST kernels nd (c) in the finl prt of the plot SST shows higher grdient thn ST, Liner nd Poly. This ltter produces the best ccurcy 90.5% in line with the literture findings using stndrd fetures nd polynomil SMs, e.g. 87.1% 6 in (Prdhn et l., 2004). Second, in tbles 2 nd 3, we report the results using ll vilble trining dt, on PropBnk nd Frmeet test sets, respectively. Ech row of the two tbles shows the F 1 mesure of the individul clssifiers using different kernels wheres the lst column illustrtes the globl ccurcy of the multiclssifier. 5 We mesured lso the computtion time for the incomplete trees ssocited with the predicte rgument structures (see Section 3); we obtined the sme results. 6 The smll difference (2.4%) is minly due to the different tretment of ARGMs: we built single ARGM clss for ll subclsses, e.g. ARGM-LOC nd ARGM-TMP, wheres in (Prdhn et l., 2004), the ARGMs, were evluted seprtely.

7 We note tht, the F 1 of the single rguments cross the different kernels follows the sme behvior of the globl multiclssifier ccurcy. On Frmeet, the bow impct on the ST nd SST ccurcy is higher thn on PropBnk s it produces n improvement of bout 1.5%. This suggests tht (1) to detect semntic roles, lexicl informtion is very importnt, (2) bow give higher contribution s errors in POS-tgging mke the word + POS frgments less relible nd (3) s the Frmeet trees re obtined with the Collins syntctic prser, tree kernels seem robust to incorrect prse trees. Third, we point out tht the polynomil kernel on flt fetures is more ccurte thn tree kernels but the design of such effective fetures required noticeble knowledge nd effort (Gilde nd Jurfsky, 2002). On the contrry, the choice of subtrees suitble to syntcticlly chrcterize trget phenomenon seems esier tsk (see Section 3 for the predicte rgument cse). Moreover, by combining polynomil nd SST kernels, we cn improve the clssifiction ccurcy (Moschitti, 2004), i.e. tree kernels provide the lerning lgorithm with mny relevnt frgments which hrdly cn be designed by hnd. In fct, s mny predicte rgument structures re quite lrge (up to 100 nodes) they contin mny frgments. ARGs ST SST ST+bow SST+bow Liner P oly ARG ARG ARG ARG ARG ARGM Acc Tble 2: Evlution of Kernels on PropBnk. Roles ST SST ST+bow SST+bow Liner P oly gent theme gol pth mnner source time reson Acc roles Tble 3: Evlution of the Kernels on Frmeet semntic roles. Finlly, to study the combined kernels, we pplied the K 1 + γk 2 formul, where K 1 is either the Liner or the Poly kernel nd K 2 is the ST Corpus Poly ST+Liner SST+Liner ST+Poly SST+Poly PropBnk Frmeet Tble 4: Multiclssifier ccurcy using Kernel Combintions. or the SST kernel. Tble 4 shows the results of four kernel combintions. We note tht, () STs nd SSTs improve Poly (bout 0.5 nd 2 percent points on PropBnk nd Frmeet, respectively) nd (b) the liner kernel, which uses fewer fetures thn Poly, is more enhnced by the SSTs thn STs (for exmple on PropBnk we hve 89.4% nd 88.6% vs. 87.6%), i.e. Liner tkes dvntge by the richer feture set of the SSTs. It should be noted tht our results of kernel combintions on Frmeet re in contrst with (Moschitti, 2004), where no improvement ws obtined. Our explntion is tht, thnks to the fst evlution of FTK, we could crry out n dequte prmeteriztion. 5 Relted Work Recently, severl tree kernels hve been designed. In the following, we highlight their differences nd properties. In (Collins nd uffy, 2002), the SST tree kernel ws experimented with the oted Perceptron for the prse-tree rernking tsk. The combintion with the originl PCFG model improved the syntctic prsing. Additionlly, it ws lluded tht the verge execution time depends on the number of repeted productions. In (ishwnthn nd Smol, 2002), liner complexity lgorithm for the computtion of the ST kernel is provided (in the worst cse). The min ide is the use of the suffix trees to store prtil mtches for the evlution of the string kernel (Lodhi et l., 2000). This cn be used to compute the ST frgments once the tree is converted into string. To our knowledge, ours is the first ppliction of the ST kernel for nturl lnguge tsk. In (Kzm nd Torisw, 2005), n interesting lgorithm tht speeds up the verge running time is presented. Such lgorithm looks for node pirs tht hve in common lrge number of trees (mlicious nodes) nd pplies trnsformtion to the trees rooted in such nodes to mke fster the kernel computtion. The results show n increse of the speed similr to the one produced by our method. In (Zelenko et l., 2003), two kernels over syntctic shllow prser structures were devised for the extrction of linguistic reltions, e.g. personffilition. To mesure the similrity between two

8 nodes, the contiguous string kernel nd the sprse string kernel (Lodhi et l., 2000) were used. In (Culott nd Sorensen, 2004) such kernels were slightly generlized by providing mtching function for the node pirs. The time complexity for their computtion limited the experiments on dt set of just 200 news items. Moreover, we note tht the bove tree kernels re not convolution kernels s those proposed in this rticle. In (Shen et l., 2003), tree-kernel bsed on Lexiclized Tree Adjoining Grmmr (LTAG) for the prse-rernking tsk ws proposed. Since QTK ws used for the kernel computtion, the high lerning complexity forced the uthors to trin different SMs on different slices of trining dt. Our FTK, dpted for the LTAG tree kernel, would hve llowed SMs to be trined on the whole dt. In (Cumby nd Roth, 2003), feture description lnguge ws used to extrct structurl fetures from the syntctic shllow prse trees ssocited with nmed entities. The experiments on the nmed entity ctegoriztion showed tht when the description lnguge selects n dequte set of tree frgments the oted Perceptron lgorithm increses its clssifiction ccurcy. The explntion ws tht the complete tree frgment set contins mny irrelevnt fetures nd my cuse overfitting. 6 Conclusions In this pper, we hve shown tht tree kernels cn effectively be dopted in prcticl nturl lnguge pplictions. The min rguments ginst their use re their efficiency nd ccurcy lower thn trditionl feture bsed pproches. We hve shown tht fst lgorithm (FTK) cn evlute tree kernels in liner verge running time nd lso tht the overll converging time required by SMs is comptible with very lrge dt sets. Regrding the ccurcy, the experiments with Support ector Mchines on the PropBnk nd Frmeet predicte rgument structures show tht: () the richer the kernel is in term of substructures (e.g. SST), the higher the ccurcy is, (b) tree kernels re effective lso in cse of utomtic prse trees nd (c) s kernel combintions lwys improve trditionl feture models, the best pproch is to combine sclr-bsed nd structured bsed kernels. Acknowledgments I would like to thnk the AI group t the University of Rome Tor ergt. Mny thnks to the EACL 2006 nonymous reviewers, Roberto Bsili nd Giorgio Stt who provided me with vluble suggestions. This reserch is prtilly supported by the Presto Spce EU Project#: FP References Michel Collins nd igel uffy ew rnking lgorithms for prsing nd tgging: Kernels over discrete structures, nd the voted perceptron. In ACL02. Michel Collins Three genertive, lexiclized models for sttisticl prsing. In proceedings of the ACL97, Mdrid, Spin. Aron Culott nd Jeffrey Sorensen ependency tree kernels for reltion extrction. In proceedings of ACL04, Brcelon, Spin. Chd Cumby nd n Roth Kernel methods for reltionl lerning. In proceedings of ICML Wshington, US. Chrles J. Fillmore Frme semntics. In Linguistics in the Morning Clm. niel Gilde nd niel Jurfsky Automtic lbeling of semntic roles. Computtionl Linguistic, 28(3): niel Gilde nd Mrth Plmer The necessity of prsing for predicte rgument recognition. In proceedings of ACL02, Phildelphi, PA. T. Jochims Mking lrge-scle SM lerning prcticl. In B. Schölkopf, C. Burges, nd A. Smol, editors, Advnces in Kernel Methods - Support ector Lerning. Junichi Kzm nd Kentro Torisw Speeding up trining with tree kernels for node reltion lbeling. In proceedings of EMLP 2005, Toronto, Cnd. Pul Kingsbury nd Mrth Plmer From Treebnk to PropBnk. In proceedings of LREC-2002, Spin. Hum Lodhi, Crig Sunders, John Shwe-Tylor, ello Cristinini, nd Christopher Wtkins Text clssifiction using string kernels. In IPS02, ncouver, Cnd. M. P. Mrcus, B. Sntorini, nd M. A. Mrcinkiewicz Building lrge nnotted corpus of english: The Penn Treebnk. Computtionl Linguistics, 19: Alessndro Moschitti A study on convolution kernels for shllow semntic prsing. In proceedings ACL04, Brcelon, Spin. Smeer Prdhn, Kdri Hcioglu, leri Krugler, Wyne Wrd, Jmes H. Mrtin, nd niel Jurfsky Support vector lerning for semntic rgument clssifiction. Mchine Lerning Journl. Libin Shen, Anoop Srkr, nd Arvind Joshi Using LTAG bsed fetures in prse rernking. In proceedings of EMLP 2003, Spporo, Jpn. Ben Tskr, n Klein, Mike Collins, phne Koller, nd Christopher Mnning Mx-mrgin prsing. In proceedings of EMLP 2004 Brcelon, Spin. S... ishwnthn nd A.J. Smol Fst kernels on strings nd trees. In proceedings of eurl Informtion Processing Systems.. Zelenko, C. Aone, nd A. Richrdell Kernel methods for reltion extrction. Journl of Mchine Lerning Reserch. ell Zhng nd Wee Sun Lee Question clssifiction using support vector mchines. In proceedings of SI- GIR 03, ACM Press.