Text Classification and Clustering Using Kernels for Structured Data

Transcription

1 Text Mining SVM Conclusion Text Classification and Clustering Using, DGFS Institut für Kognitionswissenschaft Universität Osnabrück February 2005

2 Outline Text Mining SVM Conclusion 1 Text Mining Typical applications How to represent texts 2 Graphs, trees, sequences 3 SVM and kernel methods 4 Kernels for sequences, trees, and graphs 5 Discussion

3 Text Mining SVM Conclusion TFIDF Structured Information Ontologies Graphs Text Mining Data mining extracts useful knowledge from given data Text (web) mining is data mining for (hyper) texts. Typical text mining tasks: Answering of queries (e.g. Google) Automated answering of questions Automatic text clustering and ontology building Part-of-speech tagging Automatic annotation of corpora Text and classification (e.g. topic detection)

4 Text Mining SVM Conclusion TFIDF Structured Information Ontologies Graphs Representation of Texts Text mining using bag of words representation Features: Single words Word stems Lemmas Multi-word/phrasal features Feature/attribute values: Occurence of term (binary) Number of occurences Weighted with inverse document frequency (TFIDF)

5 Text Mining SVM Conclusion TFIDF Structured Information Ontologies Graphs Is TFIDF Good Enough? Web documents generally link to other documents I.e. the structure of the web can be described by a graph Using this link information was shown to improve performance How to use link information? How to combine with TFIDF information?

6 Text Mining SVM Conclusion TFIDF Structured Information Ontologies Graphs Is TFIDF Good Enough? Docments can often be described as trees Header and body Body might consists of several sections Sections can contain tables containing text Text in headings is more important Anchor text of links

7 Text Mining SVM Conclusion TFIDF Structured Information Ontologies Graphs Structure on the Text/Sentence/Word Level The structure of sentences can be incorporated using so-called parse trees Even without tree structure: Sentences are sequences of words Words are sequences of characters Text are sequences of sentences (and not word vectors) POS-tagging Different writings of same word: Viagra vs.

8 Text Mining SVM Conclusion TFIDF Structured Information Ontologies Graphs Representation of Sentences by Trees (picture of parse tree was deleted for copyright reasons - P.G.) Determine type of question [Hermjacob, 2001]: proper-person ( Who is...?, Name the president... ) at-location, proper-place temporal/ distance/ monetary quantity Improved performance using information from parse trees.

9 Text Mining SVM Conclusion TFIDF Structured Information Ontologies Graphs Ontologies for Tokens Knowledge about interrelation of concepts (e.g. words) Concept hierarchy Directed acyclic graph Ontologies How to incorporate into learning process? Feature vectors can get very long Activation of features

10 Text Mining SVM Conclusion TFIDF Structured Information Ontologies Graphs Document Graphs [Bunke et al. 2005]: Represents a document by a graph Nodes = stemmed words Arcs = neighborship relation in title, link, or text Report increased classification performance compared to bag-of-words Example: Title: YAHOO NEWS A link: More News Text: Reuters News Service Reports (picture of document graph has been deleted for copyright reasons, see graphs on graph-matching slide - P.G.)

11 Text Mining SVM Conclusion TFIDF Structured Information Ontologies Graphs Use of Structural Information Current classifiers and clusterers mainly use TFIDF representation Incorporation of non-vectorial information might be useful Link structure (graph) Document structure (tree) Sentence structure (sequence or tree) Ontologies (DAGS) How to use such information in a learning system? Augment vectors with additional features Use structured information directly Structure kernels

12 Text Mining SVM Conclusion Kernels Sequences Ontologies Graphs SVM and Kernel Methods Support vector machines [Vapnik 95] Basic idea: linear separation of the data Contrast to perceptron: hyperplane in the middle between classes

13 Kernels Text Mining SVM Conclusion Kernels Sequences Ontologies Graphs φ If not linearly separable: use transformation φ to a suitable feature space Example: φ(x) = (x 1 x 2 1 )T Definition Kernel is scalar product in feature space: k(x, x ) = φ(x),φ(x )

14 Text Mining SVM Conclusion Kernels Sequences Ontologies Graphs Kernel instead of Projection Consider k instead of φ: Problem to be solved depends only on k(x i, x j ) Solution classifier f(x) depends only on k(x j, x) where x j is so-called support vector. Interpretation of Kernels Scalar product corresponds to angle between projected vectors k(x, x ) can be interpreted as similarity measure with some additional properties

15 Text Mining SVM Conclusion Kernels Sequences Ontologies Graphs Advantages of Using Kernels Projection into high-dimensional space without explicitely computing it Useful kernels available: Polynomial RBF Methods for constructing new kernels Many methods can be kernelized kernel-pca non-linear dependencies of attributes Kernels can be defined for graphs, trees, sequences

16 Text Mining SVM Conclusion Kernels Sequences Ontologies Graphs Kernel for Sequences Applicable to texts, sentences, and words String kernel: Based on common subsequences Penalty for gaps, with parameter λ Application: Text classification Recognition of words Similar idea for tree kernels cat and cart: attribute φ(x) φ(x ) c λ 1 λ 1 a λ 1 λ 1 t λ 1 λ 1 ca λ 2 λ 2 ct λ 3 λ 4 at λ 2 λ 3 cat λ 3 λ 4 cart 0 λ 4 I.e. k(x, x ) = 3λ 2 + λ 4 + λ 5 + 2λ 7

17 Text Mining SVM Conclusion Kernels Sequences Ontologies Graphs Kernels for Using Ontologies Similar to string kernel: Consider all superconcepts of x and x Using discount factor λ Compute e.g. shortest distances to activated superconcepts k(x, x ) = λ λ λ λ λ λ λ 8+8 No need for explicitely computing the vectors.

18 Text Mining SVM Conclusion Kernels Sequences Ontologies Graphs Graph Kernels Typical graph kernels: Based on label sequences No real subgraphs loss of information [Jain, Geibel, Wysotzki 2005]: Use similarity measure based on largest common subgraphs Graph match is computed using recursive neural network Ontological knowledge can be incorporated

19 Text Mining SVM Conclusion Conclusion Many succesful kernel methods (SVM, kernel-pca) Kernels exists for structures, trees, and graphs Allow to include ontological information Plugins for kernel methods Own research: Used similarity measure for graphs with SVM Some theoretical issues need to be resolved (positive semi-definiteness) Previous application domains: microbiology, Planned: text clustering and classification Similarity measure might be used for building ontologies Topic hierarchies for documents Term ontologies