Fast Methods for Kernel-based Text Analysis

Transcription

1 Proceedings of the 41st Annua Meeting of the Association for Computationa Linguistics, Juy 2003, pp Fast Methods for Kerne-based Text Anaysis Taku Kudo and Yuji Matsumoto Graduate Schoo of Information Science, Nara Institute of Science and Technoogy Abstract Kerne-based earning (e.g., Support Vector Machines has been successfuy appied to many hard probems in Natura Language Processing (NLP. In NLP, athough feature combinations are crucia to improving performance, they are heuristicay seected. Kerne methods change this situation. The merit of the kerne methods is that effective feature combination is impicity expanded without oss of generaity and increasing the computationa costs. Kerne-based text anaysis shows an exceent performance in terms in accuracy; however, these methods are usuay too sow to appy to arge-scae text anaysis. In this paper, we extend a Basket Mining agorithm to convert a kerne-based cassifier into a simpe and fast inear cassifier. Experimenta resuts on Engish BaseNP Chunking, Japanese Word Segmentation and Japanese Dependency Parsing show that our new cassifiers are about 30 to 300 times faster than the standard kerne-based cassifiers. 1 Introduction Kerne methods (e.g., Support Vector Machines (Vapnik, 1995 attract a great dea of attention recenty. In the fied of Natura Language Processing, many successes have been reported. Exampes incude Part-of-Speech tagging (Nakagawa et a., 2002 Text Chunking (Kudo and Matsumoto, 2001, Named Entity Recognition (Isozaki and Kazawa, 2002, and Japanese Dependency Parsing (Kudo and Matsumoto, 2000; Kudo and Matsumoto, It is known in NLP that combination of features contributes to a significant improvement in accuracy. For instance, in the task of dependency parsing, it woud be hard to confirm a correct dependency reation with ony a singe set of features from either a head or its modifier. Rather, dependency reations shoud be determined by at east information from both of two phrases. In previous research, feature combination has been seected manuay, and the performance significanty depended on these seections. This is not the case with kerne-based methodoogy. For instance, if we use a poynomia kerne, a feature combinations are impicity expanded without oss of generaity and increasing the computationa costs. Athough the mapped feature space is quite arge, the maxima margin strategy (Vapnik, 1995 of SVMs gives us a good generaization performance compared to the previous manua feature seection. This is the main reason why kerne-based earning has deivered great resuts to the fied of NLP. Kerne-based text anaysis shows an exceent performance in terms in accuracy; however, its inefficiency in actua anaysis imits practica appication. For exampe, an SVM-based NE-chunker runs at a rate of ony 85 byte/sec, whie previous ruebased system can process severa kiobytes per second (Isozaki and Kazawa, Such sow execution time is inadequate for Information Retrieva, Question Answering, or Text Mining, where fast

2 anaysis of arge quantities of text is indispensabe. This paper presents two nove methods that make the kerne-based text anayzers substantiay faster. These methods are appicabe not ony to the NLP tasks but aso to genera machine earning tasks where training and test exampes are represented in a binary vector. More specificay, we focus on a Poynomia Kerne of degree d, which can attain feature combinations that are crucia to improving the performance of tasks in NLP. Second, we introduce two fast cassification agorithms for this kerne. One is PKI (Poynomia Kerne Inverted, which is an extension of Inverted Index in Information Retrieva. The other is PKE (Poynomia Kerne Expanded, where a feature combinations are expicity expanded. By appying PKE, we can convert a kerne-based cassifier into a simpe and fast iner cassifier. In order to buid PKE, we extend the PrefixSpan (Pei et a., 2001, an efficient Basket Mining agorithm, to enumerate effective feature combinations from a set of support exampes. Experiments on Engish BaseNP Chunking, Japanese Word Segmentation and Japanese Dependency Parsing show that PKI and PKE perform respectivey 2 to 13 times and 30 to 300 times faster than standard kerne-based systems, without a discernibe change in accuracy. 2 Kerne Method and Support Vector Machines Suppose we have a set of training data for a binary cassification probem: (x 1, y 1,..., (x L, y L x j R N, y j {+1, 1}, where x j is a feature vector of the j-th training sampe, and y j is the cass abe associated with this training sampe. The decision function of SVMs is defined by y(x = sgn y j α j φ(x j φ(x + b, (1 where: (A φ is a non-iner mapping function from R N to R H (N H. (B α j, b R, α j 0. The mapping function φ shoud be designed such that a training exampes are ineary separabe in R H space. Since H is much arger than N, it requires heavy computation to evauate the dot products φ(x i φ(x in an expicit form. This probem can be overcome by noticing that both construction of optima parameter α i (we wi omit the detais of this construction here and the cacuation of the decision function ony require the evauation of dot products φ(x i φ(x. This is critica, since, in some cases, the dot products can be evauated by a simpe Kerne Function: K(x 1, x 2 = φ(x 1 φ(x 2. Substituting kerne function into (1, we have the foowing decision function. y(x = sgn y j α j K(x j, x + b (2 One of the advantages of kernes is that they are not imited to vectoria object x, but that they are appicabe to any kind of object representation, just given the dot products. 3 Poynomia Kerne of degree d For many tasks in NLP, the training and test exampes are represented in binary vectors; or sets, since exampes in NLP are usuay represented in socaed Feature Structures. Here, we focus on such cases 1. Suppose a feature set F = {1, 2,..., N} and training exampes X j (j = 1, 2,..., L, a of which are subsets of F (i.e., X j F. In this case, X j can be regarded as a binary vector x j = (x j1, x j2,..., x jn where x ji = 1 if i X j, x ji = 0 otherwise. The dot product of x 1 and x 2 is given by x 1 x 2 = X 1 X 2. Definition 1 Poynomia Kerne of degree d Given sets X and Y, corresponding to binary feature vectors x and y, Poynomia Kerne of degree d K d (X, Y is given by K d (x, y = K d (X, Y = (1 + X Y d, (3 where d = 1, 2, 3,.... In this paper, (3 wi be referred to as an impicit form of the Poynomia Kerne. 1 In the Maximum Entropy mode widey appied in NLP, we usuay suppose binary feature functions f i (X j {0, 1}. This formaization is exacty same as representing an exampe X j in a set {k f k (X j = 1}.

3 It is known in NLP that a combination of features, a subset of feature set F in genera, contributes to overa accuracy. In previous research, feature combination has been seected manuay. The use of a poynomia kerne aows such feature expansion without oss of generaity or an increase in computationa costs, since the Poynomia Kerne of degree d impicity maps the origina feature space F into F d space. (i.e., φ : F F d. This property is critica and some reports say that, in NLP, the poynomia kerne outperforms the simpe inear kerne (Kudo and Matsumoto, 2000; Isozaki and Kazawa, Here, we wi give an expicit form of the Poynomia Kerne to show the mapping function φ(. Lemma 1 Expicit form of Poynomia Kerne. The Poynomia Kerne of degree d can be rewritten as K d (X, Y = c d (r P r (X Y, (4 r=0 where P r (X is a set of a subsets of X with exacty r eements in it, c d (r = d ( d rm=0 =r ( 1 r m m ( r m. Proof See Appendix A. c d (r wi be referred as a subset weight of the Poynomia Kerne of degree d. This function gives a prior weight to the subset s, where s = r. Exampe 1 Quadratic and Cubic Kerne Given sets X = {a, b, c, d} and Y = {a, b, d, e}, the Quadratic Kerne K 2 (X, Y and the Cubic Kerne K 3 (X, Y can be cacuated in an impicit form as: K 2 (X, Y = (1 + X Y 2 = ( = 16, K 3 (X, Y = (1 + X Y 3 = ( = 64. Using Lemma 1, the subset weights of the Quadratic Kerne and the Cubic Kerne can be cacuated as c 2 (0 = 1, c 2 (1 = 3, c 2 (2 = 2 and c 3 (0=1, c 3 (1=7, c 3 (2=12, c 3 (3=6. In addition, subsets P r (X Y (r = 0, 1, 2, 3 are given as foows: P 0 (X Y = {φ}, P 1 (X Y = {{a}, {b}, {d}}, P 2 (X Y = {{a, b}, {a, d}, {b, d}}, P 3 (X Y = {{a, b, d}}. K 2 (X, Y and K 3 (X, Y can simiary be cacuated in an expicit form as: function PKI cassify (X r = 0 # an array, initiaized as 0 foreach i X foreach j h(i r j = r j + 1 end end resut = 0 foreach j SV resut = resut + y j α j (1 + r j d end return sgn(resut + b end Figure 1: Pseudo code for PKI K 2 (X, Y = = 16, K 3 (X, Y = = Fast Cassifiers for Poynomia Kerne In this section, we introduce two fast cassification agorithms for the Poynomia Kerne of degree d. Before describing them, we give the baseine cassifier (PKB: y(x = sgn y j α j (1 + X j X d + b. (5 The compexity of PKB is O( X SV, since it takes O( X to cacuate (1 + X j X d and there are a tota of SV support exampes. 4.1 PKI (Inverted Representation Given an item i F, if we know in advance the set of support exampes which contain item i F, we do not need to cacuate X j X for a support exampes. This is a naive extension of Inverted Indexing in Information Retrieva. Figure 1 shows the pseudo code of the agorithm PKI. The function h(i is a pre-compied tabe and returns a set of support exampes which contain item i. The compexity of the PKI is O( X B + SV, where B is an average of h(i over a item i F. The PKI can make the cassification speed drasticay faster when B is sma, in other words, when feature space is reativey sparse (i.e., B SV. The feature space is often sparse in many tasks in NLP, since exica entries are used as features. The agorithm PKI does not change the fina accuracy of the cassification.

4 4.2 PKE (Expanded Representation Basic Idea of PKE Using Lemma 1, we can represent the decision function (5 in an expicit form: y(x = sgn y j α j ( d r=0 If we, in advance, cacuate w(s = c d (r P r (X j X + b. (6 y j α j c d ( s I(s P s (X j (where I(t is an indicator function 2 for a subsets s d r=0 P r (F, (6 can be written as the foowing simpe inear form: y(x = sgn s Γ d (X w(s + b. (7 where Γ d (X = d r=0 P r (X. The cassification agorithm given by (7 wi be referred to as PKE. The compexity of PKE is O( Γ d (X = O( X d, independent on the number of support exampes SV Mining Approach to PKE To appy the PKE, we first cacuate Γ d (F degree of vectors w = (w(s 1, w(s 2,..., w(s Γd (F. This cacuation is trivia ony when we use a Quadratic Kerne, since we just project the origina feature space F into F F space, which is sma enough to be cacuated by a naive exhaustive method. However, if we, for instance, use a poynomia kerne of degree 3 or higher, this cacuation becomes not trivia, since the size of feature space exponentiay increases. Here we take the foowing strategy: 1. Instead of using the origina vector w, we use w, an approximation of w. 2. We appy the Subset Mining agorithm to cacuate w efficienty. 2 I(t returns 1 if t is true,returns 0 otherwise. Definition 2 w : An approximation of w An approximation of w is given by w = (w (s 1, w (s 2,..., w (s Γd (F, where w (s is set to 0 if w(s is triviay cose to 0. (i.e., σ neg < w(s < σ pos (σ neg < 0, σ pos > 0, where σ pos and σ neg are predefined threshods. The agorithm PKE is an approximation of the PKB, and changes the fina accuracy according to the seection of threshods σ pos and σ neg. The cacuation of w is formuated as the foowing mining probem: Definition 3 Feature Combination Mining Given a set of support exampes and subset weight c d (r, extract a subsets s and their weights w(s if w(s hods w(s σ pos or w(s σ neg. In this paper, we appy a Sub-Structure Mining agorithm to the feature combination mining probem. Generay speaking, sub-structures mining agorithms efficienty extract frequent sub-structures (e.g., subsets, sub-sequences, sub-trees, or subgraphs from a arge database (set of transactions. In this context, frequent means that there are no ess than ξ transactions which contain a sub-structure. The parameter ξ is usuay referred to as the Minimum Support. Since we must enumerate a subsets of F, we can appy subset mining agorithm, in some times caed as Basket Mining agorithm, to our task. There are many subset mining agorithms proposed, however, we wi focus on the PrefixSpan agorithm, which is an efficient agorithm for sequentia pattern mining, originay proposed by (Pei et a., The PrefixSpan was originay designed to extract frequent sub-sequence (not subset patterns, however, it is a trivia difference since a set can be seen as a specia case of sequences (i.e., by sorting items in a set by exicographic order, the set becomes a sequence. The basic idea of the PrefixSpan is to divide the database by frequent sub-patterns (prefix and to grow the prefix-spanning pattern in a depth-first search fashion. We now modify the PrefixSpan to suit to our feature combination mining. size constraint We ony enumerate up to subsets of size d. when we pan to appy the Poynomia Kerne of degree d.

5 1 Subset weight c d (r In the origina PrefixSpan, the frequency of each subset does not change by its size. However, in our mining task, it changes (i.e., the frequency of subset s is weighted by c d ( s. Here, we process the mining agorithm by assuming that each transaction (support exampe X j has its frequency C d y j α j, where C d = max(c d (1, c d (2,..., c d (d. The weight w(s is cacuated by w(s = ω(s c d ( s /C d, where ω(s is a frequency of s, given by the origina PrefixSpan. Positive/Negative support exampes We first divide the support exampes into positive (y i > 0 and negative (y i < 0 exampes, and process mining independenty. The resut can be obtained by merging these two resuts. Minimum Supports σ pos, σ neg In the origina PrefixSpan, minimum support is an integer. In our mining task, we can give a rea number to minimum support, since each transaction (support exampe X j has possiby non-integer frequency C d y j α j. Minimum supports σ pos and σ neg contro the rate of approximation. For the sake of convenience, we just give one parameter σ, and cacuate σ pos and σ neg as foows ( #of positive exampes σ pos = σ, σ neg = σ #of support exampes ( #of negative exampes #of support exampes After the process of mining, a set of tupes Ω = { s, w(s } is obtained, where s is a frequent subset and w(s is its weight. We use a TRIE to efficienty store the set Ω. The exampe of such TRIE compression is shown in Figure 2. Athough there are many impementations for TRIE, we use a Doube-Array (Aoe, 1989 in our task. The actua cassification of PKE can be examined by traversing the TRIE for a subsets s Γ d (X. 5 Experiments To demonstrate performances of PKI and PKE, we examined three NLP tasks: Engish BaseNP Chunking (EBC, Japanese Word Segmentation (JWS and. s!#"$"&% '(#* '+,'-+. '-+. '0 Figure 2: Ω in TRIE representation Japanese Dependency Parsing (JDP. A more detaied description of each task, training and test data, the system parameters, and feature sets are presented in the foowing subsections. Tabe 1 summarizes the detai information of support exampes (e.g., size of SVs, size of feature set etc.. Our preiminary experiments show that a Quadratic Kerne performs the best in EBC, and a Cubic Kerne performs the best in JWS and JDP. The experiments using a Cubic Kerne are suitabe to evauate the effectiveness of the basket mining approach appied in the PKE, since a Cubic Kerne projects the origina feature space F into F 3 space, which is too arge to be handed ony using a naive exhaustive method. A experiments were conducted under Linux using XEON 2.4 Ghz dua processors and 3.5 Gbyte of main memory. A systems are impemented in C Engish BaseNP Chunking (EBC Text Chunking is a fundamenta task in NLP dividing sentences into non-overapping phrases. BaseNP chunking deas with a part of this task and recognizes the chunks that form noun phrases. Here is an exampe sentence: [He] reckons [the current account deficit] wi narrow to [ony $ 1.8 biion]. A BaseNP chunk is represented as sequence of words between square brackets. BaseNP chunking task is usuay formuated as a simpe tagging task, where we represent chunks with three types of tags: B: beginning of a chunk. I: non-initia word. O: outside of the chunk. In our experiments, we used the same settings as (Kudo and Matsumoto, We use a standard data set (Ramshaw and Marcus, 1995 consisting of sections of the WSJ corpus as training and section 20 as testing.. '-+. +-/. '-(#*

6 5.2 Japanese Word Segmentation (JWS Since there are no expicit spaces between words in Japanese sentences, we must first identify the word boundaries before anayzing deep structure of a sentence. Japanese word segmentation is formaized as a simpe cassification task. Let s = c 1 c 2 c m be a sequence of Japanese characters, t = t 1 t 2 t m be a sequence of Japanese character types 3 associated with each character, and y i {+1, 1}, (i = (1, 2,..., m 1 be a boundary marker. If there is a boundary between c i and c i+1, y i = 1, otherwise y i = 1. The feature set of exampe x i is given by a characters as we as character types in some constant window (e.g., 5: {c i 2, c i 1,, c i+2, c i+3, t i 2, t i 1,, t i+2, t i+3 }. Note that we distinguish the reative position of each character and character type. We use the Kyoto University Corpus (Kurohashi and Nagao, 1997, 7,958 sentences in the artices on January 1st to January 7th are used as training data, and 1,246 sentences in the artices on January 9th are used as the test data. 5.3 Japanese Dependency Parsing (JDP The task of Japanese dependency parsing is to identify a correct dependency of each Bunsetsu (base phrase in Japanese. In previous research, we presented a state-of-the-art SVMs-based Japanese dependency parser (Kudo and Matsumoto, We combined SVMs into an efficient parsing agorithm, Cascaded Chunking Mode, which parses a sentence deterministicay ony by deciding whether the current chunk modifies the chunk on its immediate right hand side. The input for this agorithm consists of a set of the inguistic features reated to the head and modifier (e.g., word, part-of-speech, and infections, and the output from the agorithm is either of the vaue +1 (dependent or -1 (independent. We use a standard data set, which is the same corpus described in the Japanese Word Segmentation. 3 Usuay, in Japanese, word boundaries are highy constrained by character types, such as hiragana and katakana (both are phonetic characters in Japanese, Chinese characters, Engish aphabets and numbers. 5.4 Resuts Tabes 2, 3 and 4 show the execution time, accuracy 4, and Ω (size of extracted subsets, by changing σ from 0.01 to The PKI eads to about 2 to 12 times improvements over the PKB. In JDP, the improvement is significant. This is because B, the average of h(i over a items i F, is reativey sma in JDP. The improvement significanty depends on the sparsity of the given support exampes. The improvements of the PKE are more significant than the PKI. The running time of the PKE is 30 to 300 times faster than the PKB, when we set an appropriate σ, (e.g., σ = for EBC and JWS, σ = for JDP. In these settings, we coud preserve the fina accuracies for test data. 5.5 Frequency-based Pruning The PKE with a Cubic Kerne tends to make Ω arge (e.g., Ω = 2.32 miion for JWS, Ω = 8.26 miion for JDP. To reduce the size of Ω, we examined simpe frequency-based pruning experiments. Our extension is to simpy give a prior threshod ξ(= 1, 2, 3, 4..., and erase a subsets which occur in ess than ξ support exampes. The cacuation of frequency can be simiary conducted by the PrefixSpan agorithm. Tabes 5 and 6 show the resuts of frequency-based pruning, when we fix σ=0.005 for JWS, and σ= for JDP. In JDP, we can make the size of set Ω about one third of the origina size. This reduction gives us not ony a sight speed increase but an improvement of accuracy (89.29% 89.34%. Frequency-based pruning aows us to remove subsets that have arge weight and sma frequency. Such subsets may be generated from errors or specia outiers in the training exampes, which sometimes cause an overfitting in training. In JWS, the frequency-based pruning does not work we. Athough we can reduce the size of Ω by haf, the accuracy is aso reduced (97.94% 97.83%. It impies that, in JWS, features even with frequency of one contribute to the fina decision hyperpane. 4 In EBC, accuracy is evauated using F measure, harmonic mean between precision and reca.

7 Tabe 1: Detais of Data Set Data Set EBC JWS JDP # of exampes 135, , ,355 SV # of SVs 11,690 57,672 34,996 # of positive SVs 5,637 28,440 17,528 # of negative SVs 6,053 29,232 17,468 F (size of feature 17,470 11,643 28,157 Avg. of X j B (Avg. of h(i (Note: In EBC, to hande K-cass probems, we use a pairwise cassification; buiding K (K 1/2 cassifiers considering a pairs of casses, and fina cass decision was given by majority voting. The vaues in this coumn are averages over a pairwise cassifiers. 6 Discussion There have been severa studies for efficient cassification of SVMs. Isozaki et a. propose an XQK (expand the Quadratic Kerne which can make their Named-Entity recognizer drasticay fast (Isozaki and Kazawa, XQK can be subsumed into PKE. Both XQK and PKE share the basic idea; a feature combinations are expicity expanded and we convert the kerne-based cassifier into a simpe inear cassifier. The expicit difference between XQK and PKE is that XQK is designed ony for Quadratic Kerne. It impies that XQK can ony dea with feature combination of size up to two. On the other hand, PKE is more genera and can aso be appied not ony to the Quadratic Kerne but aso to the genera-stye of poynomia kernes (1 + X Y d. In PKE, there are no theoretica constrains to imit the size of combinations. In addition, Isozaki et a. did not mention how to expand the feature combinations. They seem to use a naive exhaustive method to expand them, which is not aways scaabe and efficient for extracting three or more feature combinations. PKE takes a basket mining approach to enumerating effective feature combinations more efficienty than their exhaustive method. 7 Concusion and Future Works We focused on a Poynomia Kerne of degree d, which has been widey appied in many tasks in NLP Tabe 2: Resuts of EBC PKE Time Speedup F1 Ω σ (sec./sent. Ratio ( PKI PKB Tabe 3: Resuts of JWS PKE Time Speedup Acc.(% Ω σ (sec./sent. Ratio ( , , , ,820 PKI PKB Tabe 4: Resuts of JDP PKE Time Speedup Acc.(% Ω σ (sec./sent. Ratio ( , , ,262 PKI PKB Tabe 5: Frequency-based pruning (JWS PKE time Speedup Acc.(% Ω ξ (sec./sent. Ratio ( , PKB Tabe 6: Frequency-based pruning (JDP PKE time Speedup Acc.(% Ω ξ (sec./sent. Ratio ( , , ,360 PKB

8 and can attain feature combination that is crucia to improving the performance of tasks in NLP. Then, we introduced two fast cassification agorithms for this kerne. One is PKI (Poynomia Kerne Inverted, which is an extension of Inverted Index. The other is PKE (Poynomia Kerne Expanded, where a feature combinations are expicity expanded. The concept in PKE can aso be appicabe to kernes for discrete data structures, such as String Kerne (Lodhi et a., 2002 and Tree Kerne (Kashima and Koyanagi, 2002; Coins and Duffy, For instance, Tree Kerne gives a dot product of an ordered-tree, and maps the origina ordered-tree onto its a sub-tree space. To appy the PKE, we must efficienty enumerate the effective sub-trees from a set of support exampes. We can simiary appy a sub-tree mining agorithm (Zaki, 2002 to this probem. Appendix A.: Lemma 1 and its proof c d (r = ( ( d r ( r ( 1 r m m. m =r m=0 Proof. Let X, Y be subsets of F = {1, 2,..., N}. In this case, X Y is same as the dot product of vector x, y, where x = {x 1, x 2,..., x N }, y = {y 1, y 2,..., y N } (x j, y j {0, 1} x j = 1 if j X, x j = 0 otherwise. (1 + X Y d = (1 + x y d can be expanded as foows where τ( = (1 + x y d = k k N = k n 0 = ( ( d N x j y j =0 =0 j=1 ( d τ(! k 1!... k N! (x1y1k 1... (x N y N k N. Note that x k j j is binary (i.e., x k j j {0, 1}, the number of r-size subsets can be given by a coefficient of (x 1 y 1 x 2 y 2... x r y r. Thus, c d (r = =r ( ( k k d r=! k 1!... k r! k n 1,n=1,2,...,r = = =r =r References ( ( d r m=0 ( r (r ( r (r ( ( d r ( ( 1 r m m r. m Junichi Aoe An efficient digita search agorithm by using a doube-array structure. IEEE Transactions on Software Engineering, 15(9. Michae Coins and Nige Duffy Convoution kernes for natura anguage. In Advances in Neura Information Processing Systems 14, Vo.1 (NIPS 2001, pages Hideki Isozaki and Hideto Kazawa Efficient support vector cassifiers for named entity recognition. In Proceedings of the COLING-2002, pages Hisashi Kashima and Teruo Koyanagi Svm kernes for semi-structured data. In Proceedings of the ICML-2002, pages Taku Kudo and Yuji Matsumoto Japanese Dependency Structure Anaysis based on Support Vector Machines. In Proceedings of the EMNLP/VLC-2000, pages Taku Kudo and Yuji Matsumoto Chunking with support vector machines. In Proceedings of the the NAACL, pages Taku Kudo and Yuji Matsumoto Japanese dependency anayisis using cascaded chunking. In Proceedings of the CoNLL-2002, pages Sadao Kurohashi and Makoto Nagao Kyoto University text corpus project. In Proceedings of the ANLP-1997, pages Huma Lodhi, Craig Saunders, John Shawe-Tayor, Neo Cristianini, and Chris Watkins Text cassification using string kernes. Journa of Machine Learning Research, 2. Tetsuji Nakagawa, Taku Kudo, and Yuji Matsumoto Revision earning and its appication to part-of-speech tagging. In Proceedings of the ACL 2002, pages Jian Pei, Jiawei Han, and et a Prefixspan: Mining sequentia patterns by prefix-projected growth. In Proc. of Internationa Conference of Data Engineering, pages Lance A. Ramshaw and Mitche P. Marcus Text chunking using transformation-based earning. In Proceedings of the VLC, pages Vadimir N. Vapnik The Nature of Statistica Learning Theory. Springer. Mohammed Zaki Efficienty mining frequent trees in a forest. In Proceedings of the 8th Internationa Conference on Knowedge Discovery and Data Mining KDD, pages