UNICAL, 21/10/2004. Tutorial goals

Transcription

1 Workshop Data Warehousing and Data Mining TEXTEXT MINING INING An Overview of Concepts, Techniques and Applications Ing. Andrea Tagarelli Tutorial goals Introduce you to major aspects of the Knowledge Discovery Process, when available data is textual and unstructured Provide a systematization to the many many concepts around this area, according to the following lines the process the methods the applications Important issues that will be not covered in this tutorial: The problem of Dimensionality Reduction Text categorization Learning techniques Evaluation Visualization techniques Tools for Text Mining etc.

2 Tutorial Outline 1. Introduction Motivations Basic concepts in Knowledge Discovery from textual data 2. Deeper into Text Mining: Functions Models Storage techniques Index term Identification Index term Weighting 3. Applications Tutorial Outline 1. Introduction Motivations Basic concepts in Knowledge Discovery from textual data 2. Deeper into Text Mining: Functions Models Storage techniques Index term Identification Index term Weighting 3. Applications

3 The reason for Text Mining Amount of infor mation Collections of Text Str uct ur ed Data 0 Per centage A non-exhaustive list: Text sources Web pages E-books News articles Insurance claims Patent portfolios IRC Technical documents Scientific articles Customer complaint letters Contracts Transcripts of phone calls with customers

4 Problems with textual data (I) The known KDD problems and challenges extend to textual data large (textual) data collections high dimensionality overfitting changing data and knowledge noisy data understandability of mined patterns etc. Problems with textual data (II) But there are new problems Text is not designed to be used by computers Complex and poorly defined structure and semantics But much harder, ambiguity in speech, morphology, syntax, semantics, pragmatics Multilingualism lack of reliable and general translation tools

5 What is Text Mining? Peoples first thought: Make it easier to find things on the Web. But this is information retrieval! The foundation of most commercial text mining products is all this unproperly named stuff Information retrieval engine Web spider/search Text classification Information extraction (only sometimes) The metaphor of extracting ore from rock: Does make sense for extracting documents of interest from a huge pile. But does not reflect notions of Data Mining in practice. Rather: finding patterns across large collections discovering heretofore unknown information Text Mining: definitions Text mining mainly is about somehow extracting the information and knowledge from text 2 definitions: Any operation related to gathering and analyzing text from external sources for business intelligence purposes Discovery of knowledge previously unknown to the user in text Text mining is the process of: compiling, organizing, and analyzing large document collections to support the delivery of targeted types of information to analysts and decision makers and to discover relationships between related facts that span wide domains of inquiry.

6 Text Mining: contributing areas Natural Language Processing Information Retrieval Text Mining Information Extraction Data Mining Tutorial Outline 1. Introduction and basic concepts Motivations Basic concepts in Knowledge Discovery from textual data 2. Deeper into Text Mining: Functions Models Storage techniques Index term Identification Index term Weighting 3. Applications

7 Major Functions Indicative reveals elements of the content, upon which the relevancy of the complete original text can be decided useful for Document browsing systems Document retrieval systems Informative represents a real substitute of the content of the full-text (or part), without references to its original text useful for Question-answering systems Document retrieval systems Browsing systems Usually part of hypertext and hypermedia systems Allow users to skim text collections in the search for valuable information Users need not to generate descriptions of what they want or specify in advance the topics of interest but can just indicate documents they find relevant Useful when a user has no clear need cannot express his need accurately or is a casual user of the information

8 Question-answering systems Also known as Information Extraction systems Retrieve specific information from the documents, by extracting or inferring answers from text representation Template types: Slots in template typically filled by a substring from the document Some slots may have a fixed set of pre-specified possible fillers that may not occur in the text itself Some slots may allow multiple fillers Assumes slots always in a fixed order Extraction patterns: Specify an item to extract for a slot, e.g. using a regular expression pattern May require preceding (pre-filler) pattern to identify proper context, and succeeding (post-filler) pattern to identify the end of the filler Information Retrieval systems (I) Select documents from a collection in response to a user s query Search request is formulated in natural language Rank these documents according to their relevance to the query Matching between document representation and query representation Return a list of possible relevant texts, the representations of which best match the request representation

9 Information Retrieval systems (II) Retrieval models Boolean model Vector space model Probabilistic model Network model Logic-based model Differences with respect to representation of textual contents, representation of information needs, and their matching Tutorial Outline 1. Introduction Motivations Basic concepts in Knowledge Discovery from textual data 2. Deeper into Text Mining: Functions Models Storage techniques Index term Identification Index term Weighting 3. Applications

10 Boolean model (I) Compares the boolean query statement with the term sets used to identify the textual content (index terms) Query has the form of an expression containing index terms boolean operators (AND, OR and NOT) defined upon the terms Document matches condition or not This model is employed in many commercial systems Professional searchers still like boolean queries: you know exactly what you re getting Boolean model (II) Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Term-document incidence matrix Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony Brutus Caesar Calpurnia Cleopatra mercy worser if play contains word, 0 otherwise To answer query: Idea: query satisfaction = overlap measure take the vectors for Brutus, Caesar and Calpurnia (complemented) bitwise AND AND AND =

11 Boolean model (III): Problems Very rigid: AND means all; OR means any Difficult to express complex user requests Difficult to control the number of documents retrieved All matched documents will be returned Difficult to rank output All matched documents logically satisfy the query Difficult to perform relevance feedback If a document is identified by the user as relevant or irrelevant, how should the query be modified? Boolean model (IV): Problems Overlap measure doesn t consider: term frequency in document term scarcity in collection (document mention frequency) length of documents

12 Vector Space model (I) Documents and queries are represented in a m-dimensional vector space Index term set of the collection: V = {w 1,, w m } Document: d = [d 1, d 2,, d m ] binary or weighted components Collection: D = {d 1, d 2,, d N } The relevancy of a document to a query is computed as a proximity measure Vector Space model (II) Desiderata for proximity If d 1 is near d 2, then d 2 is near d 1 If d 1 is near d 2, and d 2 near d 3, then d 1 is not far from d 3 No document is closer to d than d itself Distance between vectors d 1 and d 2 is the length of the vector d 1 - d 2 Euclidean distance Why is this not a great idea? We still haven t dealt with the issue of length normalization Long documents would be more similar to each other by virtue of length, not topic However, we can implicitly normalize by looking at angles

13 Cosine similarity Vector Space model (III) Distance between vectors d 1 and d 2 captured by the cosine of the angle θ between them. t 3 d 2 Note that this is similarity, not distance sim( d j,d k d ) = d j j d k d k = n i= 1 n i= 1 w w 2 i, j The denominator involves the lengths of the vectors So the cosine measure is also known as the normalized inner product i, j w i, k n i= 1 w 2 i, k t 2 θ d 1 t 1 Vector Space model (IV) A vector can be normalized (given a length of 1) by dividing each of its components by the vector's length This maps vectors onto the unit circle: n i d j = = w 1 i, 2 j = Then, longer documents don t get more weight For normalized vectors, the cosine is simply the dot product: cos( d j, dk ) = d j 1 d k

14 Vector Space model (V): Advantages Allows simple and efficient implementation for large document collections Query becomes a vector in the same space as the documents Can consider both local (tf) and global (idf) word occurrence frequencies Provides partial matching and natural measure of scores/ranking no longer Boolean Tends to work quite well in practice despite the simplifying assumptions Vector Space model (VI): Problems Missing syntactic information (e.g., phrase structure, word order, proximity information) Missing semantic information (e.g., word sense) Assumption of term independence Bag-of-words model Assumption that term vectors are pair-wise orthogonal Lacks the control of a Boolean model (e.g., requiring a term to appear in a document) Given a two-term query A B, may prefer a document containing A frequently but not B, over a document that contains both A and B, but both less frequently

15 Tutorial Outline 1. Introduction Motivations Basic concepts in Knowledge Discovery from textual data 2. Deeper into Text Mining: Functions Models Storage techniques Index term Identification Index term Weighting 3. Applications Storage of text representations (I) Desiderata for a data structure: Ability to represent concepts and relationships Ability to support the location of these concepts in the document collection Inverted index For each term, stores the ids of all documents that are indexed by that term The complete inverted index is first represented by an array of indexed documents then transposed (to obtain a term- document matrix)

16 Doc 1 I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. Storage of text representations (II) Each document is parsed to extract words, and these are saved with the Document ID Doc 2 So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious Term Doc # I 1 did 1 enact 1 julius 1 caesar 1 I 1 was 1 killed 1 i' 1 the 1 capitol 1 brutus 1 killed 1 me 1 so 2 let 2 it 2 be 2 with 2 caesar 2 the 2 noble 2 brutus 2 hath 2 told 2 you 2 caesar 2 was 2 ambitious 2 Storage of text representations (III) After all documents have been parsed the inverted file is sorted by terms Term Doc # I 1 did 1 enact 1 julius 1 caesar 1 I 1 was 1 killed 1 i' 1 the 1 capitol 1 brutus 1 killed 1 me 1 so 2 let 2 it 2 be 2 with 2 caesar 2 the 2 noble 2 brutus 2 hath 2 told 2 you 2 caesar 2 was 2 ambitious 2 Term Doc # ambitious 2 be 2 brutus 1 brutus 2 capitol 1 caesar 1 caesar 2 caesar 2 did 1 enact 1 hath 1 I 1 I 1 i' 1 it 2 julius 1 killed 1 killed 1 let 2 me 1 noble 2 so 2 the 1 the 2 told 2 you 2 was 1 was 2 with 2

17 Multiple term entries in a single document are merged (Local) frequency information are added Storage of text representations (IV) Term Doc # ambitious 2 be 2 brutus 1 brutus 2 capitol 1 caesar 1 caesar 2 caesar 2 did 1 enact 1 hath 1 I 1 I 1 i' 1 it 2 julius 1 killed 1 killed 1 let 2 me 1 noble 2 so 2 the 1 the 2 told 2 you 2 was 1 was 2 with 2 Term Doc # Freq ambitious 2 1 be 2 1 brutus 1 1 brutus 2 1 capitol 1 1 caesar 1 1 caesar 2 2 did 1 1 enact 1 1 hath 2 1 I 1 2 i' 1 1 it 2 1 julius 1 1 killed 1 2 let 2 1 me 1 1 noble 2 1 so 2 1 the 1 1 the 2 1 told 2 1 you 2 1 was 1 1 was 2 1 with 2 1 Storage of text representations (V) The file is commonly split into a Dictionary and a Postings file Term Doc # Freq ambitious 2 1 be 2 1 brutus 1 1 brutus 2 1 capitol 1 1 caesar 1 1 caesar 2 2 did 1 1 enact 1 1 hath 2 1 I 1 2 i' 1 1 it 2 1 julius 1 1 killed 1 2 let 2 1 me 1 1 noble 2 1 so 2 1 the 1 1 the 2 1 told 2 1 you 2 1 was 1 1 was 2 1 with 2 1 Term N docs Tot Freq ambitious 1 1 be 1 1 brutus 2 2 capitol 1 1 caesar 2 3 did 1 1 enact 1 1 hath 1 1 I 1 2 i' 1 1 it 1 1 julius 1 1 killed 1 2 let 1 1 me 1 1 noble 1 1 so 1 1 the 2 2 told 1 1 you 1 1 was 2 2 with 1 1 Doc # Freq

18 Storage of text representations (VII) n-gram structure breaks terms into smaller string units of n characters allows searching morphologically different terms Signature file contains signatures (bit patterns) representing the index terms documents are split into logical blocks each containing a fixed number of index terms Hashed word-signature in the same block are OR ed together Block signatures are then concatenated to create the document signature Tutorial Outline 1. Introduction Motivations Basic concepts in Knowledge Discovery from textual data 2. Deeper into Text Mining: Functions Models Storage techniques Index term Identification Index term Weighting 3. Applications

19 Index term Identification (I) What terms in a document do we index? All words or only important ones? A few words are very common 2 most frequent words (e.g., the, of ) can account for about 10% of word occurrences Most words are very rare Half the words in a corpus appear only once, called hapax legomena (Greek for read only once ) Search for terms that capture text semantics avoiding intensive manual processing (hand-coding) Involves: Index term Identification (II) Feature definition Text is usually represented as a bag of words, i.e. a collection of independent concepts concepts are words, word stems, word phrases concept weights can be binary or frequency-based Paragraph, sentence, word order is disrupted Syntactic structure is broken Feature selection and extraction reduce concept space dimensionality

20 Index term Identification (III) Feature selection and extraction Lessical and Morphological analysis processing of punctuation, numbers, case folding, etc. removal of stopwords stemming, lemmatization part-of-speech tagging Discourse Semantics analysis: Anaphora literal anaphor/pronominal anaphor This notebook weights even less than its predecessor textual ellipsis referential meronymy Pragmatics asservative, commissive, directive, declarative, expressive, interrogative sentences Selection of of natural language index terms (I) Cooper s concordance of Wordsworth was published in The applications of full-text retrieval are legion: they include résumé scanning, litigation support and searching published journals on-line. Cooper s vs. Cooper vs. Coopers Full-text vs. full text vs. {full, text} vs. fulltext résumé vs. resume Punctuation Ne er: use language-specific, handcrafted locale to normalize State-of-the-art: break up hyphenated sequence U.S.A. vs. USA - use locale Numbers Generally, don t index as text Creation dates for docs 3/12/91 Mar. 12, B.C. B Case folding Reduce all letters to lower case exception: upper case in mid-sentence General Motors Fed vs. fed SAIL vs. sail

21 Selection of Spell correction of natural language index terms (II) Look for all words within edit distance (maximum) k (Insert/Delete/Replace) at query time Data Minino Data Mining (edit distance: 1) eterogeneiti heterogeneity (edit distance: 2) Expensive and slows the query (upto a factor of 100) Invoke only when index returns zero matches What if documents contain mis- spellings? Selection of Removal of Stopwords of natural language index terms (III) Terms that are so common that they re ignored for indexing Function words that serve grammatical purposes and don t refer to objects or concepts. e.g., the, a, an, of, to language-specific

22 Selection of Stemming of natural language index terms (IV) Reduce terms to their roots before indexing mainly eliminate plurals, tenses, gerund forms, prefixes and suffixes automate(s), automatic, automation all reduced to automat Language- dependent for example compressed and compression are both accepted as equivalent to compress. for exampl compres and compres are both accept as equival to compres. Selection of of natural language index terms (V) Porter s stemmer algorithm Commonest algorithm for stemming English Conventions + 5 phases of reductions phases applied sequentially each phase consists of a set of commands sample convention: of the rules in a compound command, select the one that applies to the longest suffix sample rule: if word ends with ation replace with ate sses ss ies i ational ate tional tion

23 Selection of Lemmatization of natural language index terms (VI) Reduce inflectional/variant forms to base form am, are, is be car, cars, car's, cars' car the boy's cars are different colors the boy car be different color Selection of of natural language index terms (VII) Part-of-speech Tagging Labelling each word in a sentence with its proper grammatical category The representative put chairs on the table The/AT representative/nn put/vbd chairs/nns on/in the/at table/nn AT=determiner article, NN =noun singular, VBD=verb past tense, NNS=noun plural, IN=preposition There are several tag sets ranging mainly in granularity and complexity: Brown tag set, Penn Treebank tag set and several approaches: Markov models, Transformation-based learning, Decision trees

24 Selection of of natural language index terms (VIII) Part-of-speech Tagging (some) Public taggers Eric Brill s tagger: (Perl and C implementations) TreeTagger: MontyTagger: (Python and Java implementations) QTAG: (Java implementation) Makes sense as an intermediate task for others e.g. with shallow parsing for creating linguistically motivated index terms detecting slot-filler candidates in Information Extraction detecting answer candidates in Question Answering Assignment of Thesaurus of controlled language index terms Generalizes terms that have related meaning, but unrelated surface forms, into more uniform index terms Puts words that are synonyms and are intersubstitutable into equivalence classes Words may have many senses: polysemous words Word Sense Disambiguation techniques are needed Index such equivalences, or expand query?

25 Tutorial Outline 1. Introduction Motivations Basic concepts in Knowledge Discovery from textual data 2. Deeper into Text Mining: Functions Models Storage techniques Index term Identification Index term Weighting 3. Applications Index Term Weighting (I) Distribution patterns of words give significant information about the property of being content bearing Zipf s law Rank (r): the numerical position of a word in a list sorted by decreasing frequency (f ) 1 Zipf (1949) discovered that: If probability of word of rank r is p r and N is the total number of word occurrences: p r = f N = A r f r A 0.1

26 Index Term Weighting (II) Luhn (1958) suggested that both extremely common and extremely uncommon words were not very useful for indexing Most discriminative concepts have low to medium frequency Index Term Weighting (III) Need for considering frequency of a word in a document Weighting term frequency (tf) It still doesn t consider: Term scarcity in collection (document mention frequency) Length of documents and queries (not normalized) Weighting should depend on the term overall Suggest looking at collection frequency (cf) but document frequency (df) may be better

27 Index Term Weighting (IV) tf x idf measure combines: term frequency (tf) measure of term density in a document inverse document frequency (idf) measure of informativeness of term: its rarity across the whole corpus could just be raw count of number of documents the term occurs in (idf i = 1/df i ) but by far the most commonly used version is: idf log( n / df ) i = i Index Term Weighting (V) Assign a tf.idf weight to each term i in each document d tf i, d w = tf log( n / df i, d i, d i = frequency of term i in document d n = total number of documents df i = the number of documents that contain term i increases with the number of occurrences within a document increases with the rarity of the term across the whole corpus )

28 Length normalization Index Term Weighting (VI) Documents have different sizes Long and verbose texts usually use the same terms repeatedly have numerous different terms Variations in length can be normalized to compensate the effect that the tf factors are large for long texts and small for short ones obscuring the real term importance tf max i, d j tf j, d j tf i, d 2 ( tf ) ( tf idf j, d j tf i, d idf j, d i j ) 2 Tutorial Outline 1. Introduction Motivations Basic concepts in Knowledge Discovery from textual data 2. Deeper into Text Mining: Functions Models Storage techniques Index term Identification Index term Weighting 3. Applications

29 Applications (I) The main application areas cover two aspects: Knowledge discovery mining proper Information distillation mining on the basis of some pre- established document structure, to identify documents relevant to a target information Typical usage: Extract relevant information from documents Classify and manage documents according to their content Organize repositories of document-related metainformation for search and retrieval Applications (II) Text summarization Word sense disambiguation Hierarchical categorization of Web pages Text filtering CRM & marketing (e.g., cross-selling, recommendation) Product recommendation Information delivery at organizations for Knowledge Management Personalizing information access Filtering news items in Usenet newsgroups Detecting spam messages

30 Applications Text Summarization Generate a summary of a text s content short text: essential and coherent Use profiles to structure the important content in semantically welldefined fields Mostly applied to ease information access, e.g. most useful keywords are extracted from a set of documents (e.g., a cluster) to describe it documents in a collection are abstracted to avoid reading the full content documents retrieved from search are summarized to allow the user a faster identification of those relevant to the query High-level summary or survey of all main points? Approaches based on size of the text unit used in the summary Keyword summaries Sentence summaries Applications Word Sense Disambiguation Assign a word with the right sense with respect to the context in which the word appears An effective approach: Choosing word meanings from an existing sense inventory by exploiting measures of semantic relatedness WSD is an example of the more general issue of resolving natural language ambiguities For instance: bank may have (at least) two senses in English: the Bank of England (a financial institution) the bank of river Thames (a hydraulic engineering artefact) which of above senses the occurrence of bank has in last week I borrowed some money from the bank

31 Applications Hierarchical Categorization of Web pages Under hierarchical catalogues (hosted by popular Web portals), a searcher may first navigate in the hierarchy of categories and then restrict his/her search to a particular category of interest Category-pivoted categorization should allow new categories to be added and obsolete ones to be deleted Peculiarities: Hypertextual nature of the documents Hyperlink analysis Hierarchical structure of the category set Decomposing the classification as a branching decision at an internal node Applications Text Filtering (I) Classify a stream of incoming documents dispatched in an asynchronous way by an information producer to an information consumer typical case: a newsfeed (producer: news agency, consumer: newspaper) Desiderata of a filtering system should block the delivery of the documents the consumer is likely not interested in filtering can be seen as a case of single- labeled TC may be installed at the producer end to route the documents to the interested consumer only Builds and updates a profile for each consumer

32 Applications Text Filtering (II) or at consumer end to block the delivery of documents deemed uninteresting A single profile is needed Adaptive filtering a profile is initially specified by the user and is updated by using feedback information provided by the user on the relevance of the delivered messages Applications Customer Relationship Management Incorporates both the distillation and discovery aspects of TM Designed to specifically help companies better understand what their customers want and what think about the company itself Method: 1. Select a suitable set of documents and convert them to a common standard format 2. Extract relevant features and derive a database of documents which are grouped according to the similarity of their content, by exploiting clustering techniques 3. Use categorization tools to assign new incoming customer feedback to the identified categories

33 Applications Product Recommendation Content-based According a personal profile accounting for a set of categories (DVD, computer games, music, etc.) and subcategories (genres) Starting with preferred items authors, titles, brands Recommendation of new releases of course it is not text-content based, but on the purchasing history Collaborative or social According to other customers purchases Customers who bought this book also bought Based on previous annotations by other users and generating a user segmentation A trend is combining both ideas Applications Detecting Spam Spam is, more properly, unsolicited bulk It has been producing a considerable damage to Internet Service Providers Internet users (connection costs) and the whole Internet backbone Spam detection is a Text Categorization problem Two classes: spam and legitimate It relatively easy to Represent messages as vectors of concept weights Perform some feature selection Learn a classifier but evaluation is not so simple because it is a problem in which missclassification costs and class distribution are not symmetric