APPLICATION OF FP TREE GROWTH ALGORITHM IN TEXT MINING

Transcription

1 APPLICATION OF FP TREE GROWTH ALGORITHM IN TEXT MINING Project Report Submitted In Partial Fulfillment Of The Requirements for the Degree Of Master of Computer Application Of Jadavpur University By Jagdish Panjwani Department: Computer Science and Engineering Master of Computer Application III Roll Number: MCA Registration Number: of Under the guidance of Mrs. Chitrita Chaudhuri Reader, Department of Computer Science and Engineering Jadavpur University Department of Computer Science and Engineering Faculty of Engineering and Technology Jadavpur University Kolkata , India

2

3 Department of Computer Science and Engineering Faculty of Engineering and Technology Jadavpur University Kolkata , India CERTIFICATE OF APPROVAL The foregoing project work is hereby accepted as a credible study of an Engineering subject carried out and presented in a manner satisfactory to warrant it s accepted as a prerequisite to the degree for which it has been submitted. It is understood that by this approval the undersigned do not necessarily endorse or approve any statement made, opinion expressed or conclusion drawn therein, but approve the thesis only for the purpose for which it is submitted (Signature of Examiners)

4

5 CONTENTS Chapter Page No Preface 1 Date Mining Concepts Introduction 1.2 The Knowledge Discovery Process 1.3 An overview of Text Mining 1.4 Applications of Data Mining 2 Text Mining Definition of Text Mining: 2.2 What kind of patterns can be discovered? 2.3 Is all that is discovered interesting and useful? 2.4 Text Preprocessing 2.5 Extraction of relevant words from a corpus. 3 Different techniques for finding frequent 8 patterns 3.1 The Apriori Algorithm for finding frequent patterns 3.2 The Apriori Algorithm Using Vertical Data Format 3.3 FP Growth algorithm for frequent pattern generation 4 Study of FP-Tree Growth Algorithm Definition of FP tree 4.2 Formation of FP tree 4.3 Process of mining FP tree 4.4 Complexity of FP Growth Algorithm 4.5 Advantages of FP-growth over Apriori Algorithm

6 5 The Implementation Process of Extraction of Frequent Patterns 20 6 Analysis and Comparison of different algorithm and their results 24 7 Conclusion and Future Extensions 28 8 Bibliographic references 29

7 Preface The Project has been developed in an effort to identify the most frequently occurring keywords and key-phrases from a text document. Nowadays, Text mining is considered as one of the most important part of data mining. Text databases are rapidly growing due to the increasing amount of information available in electronic form, such as electronic publications, various kinds of electronic documents, , and the World Wide Web. The keywords and key phrases extracted from a text database may be used for classification of documents, in search engines where the user types keywords and key phrases and the search engine searches through a vast repository of documents to find the most relevant documents. To extract keywords and key phrases from text documents several data mining techniques have been explored in this project. Some of these such as tf-idf measures are related directly to text mining and others such as FP growth algorithm are usually associated with Market Basket Data Analysis. In the present work different techniques have been tried out to classify texts occurring in topical corpus. The results have been analyzed and presented after benchmarking with other competitive methods like Apriori using horizontal format and Apriori using vertical format.

8 Chapter1 Data Mining Concepts Data Mining consists of finding interesting trends or patterns in large datasets to guide decisions about future activities. There is a general expectation that is able to identify these patterns with minimal user input. The patterns identified can give a data analyst useful and unexpected results which can be more carefully investigated in future. 1.1 Introduction Data Mining is the process of discovering new correlations, patterns, and trends by digging into large amounts of data stored in warehouses. It is related to the subareas of artificial intelligence called knowledge discovery and machine learning. Data mining can also be defined as the process of extracting knowledge hidden from large volumes of raw data i.e. the nontrivial extraction of implicit, previously unknown, and potentially useful information from data. Alternative names of Data Mining are Knowledge discovery in databases (KDD), knowledge extraction, data/pattern analysis, etc

9 1.2 The Knowledge Discovery Process The Knowledge Discovery Process is an iterative sequence of following steps: 1. Data cleaning: noisy and inconsistent data are removed. 2. Data integration: multiple data sources may be combined. 3. Data selection: data relevant to the analyst are retrieved. 4. Data transformation: data are transformed into forms appropriate for mining. 5. Data mining: intelligent methods are applied in order to extract data patterns. 6. Pattern evaluation: to identify the truly interesting patterns representing knowledge based on some interestingness measures. 7. Knowledge presentation: where visualization and knowledge representation techniques are used to present the mined knowledge to the user. The results of any step in the knowledge discovery process might lead us back to an earlier step to redo the process with the new knowledge gained. 1.3 An overview of Text Mining: Text mining deals with automatic extraction of hidden knowledge from text documents. Text documents contain word descriptions - 2 -

10 for objects. These word descriptions are usually not simple keywords but rather long sentences or paragraphs, such as summary reports, notes, or other documents. By mining text data, one may uncover general and concise descriptions of the text documents, keyword, frequent patterns or content associations. To do this, standard data mining methods or algorithms need to be employed. The details of text mining and frequent pattern generation will be discussed in the subsequent chapters. 1.4 Applications of Data Mining Data mining is the principle of sorting through large amounts of data and picking out relevant information. It is usually used by business intelligence organizations, and financial analysts, but it is increasingly used in the sciences to extract information from the enormous data sets generated by modern experimental and observational methods, it has been described as "the nontrivial extraction of implicit, previously unknown, and potentially useful information from data" and "the science of extracting useful information from large data sets or databases"

11 Chapter 2 Text Mining: Extraction of relevant words from a corpus. One of the newest areas of data mining is text mining. However, in Text Mining, patterns are extracted from natural language text rather than databases. Text mining refers to a collection of methods used to find patterns and create intelligence from text data. The most common use of text mining procedure is in search engine technology. A user types in a word or phrase, which may include misspellings, and the search engine searches through a vast repository of documents to find the most relevant documents. 2.1 Definition of Text Mining: Text Mining can be defined as the nontrivial extraction of implicit, previously unknown, and potentially useful information from textual data. 2.2 What kind of patterns can be discovered? - 4 -

12 The patterns that can be discovered may be descriptive that describe the general properties of the existing data, and predictive that attempt to do predictions based on inference on available data. 2.3 Is all that is discovered interesting and useful? Text mining allows the discovery of knowledge potentially useful and unknown. Whether the knowledge discovered is interesting or not, is very subjective and depends upon the user and the application. The user may put a measurement or constraints on the patterns such that the patterns satisfying the constraints will be considered as interesting one. 2.4 Text Preprocessing Before mining the text data, it undergoes certain preprocessing stages: Any kind of data on which mining is to be done is converted into text format. Term extraction: in this process the entire text is split into a set of tokens. A blank space may be used as a delimiter. Stop words removal: Certain words occur very frequently in text data. Examples are "the", "a", of, etc. These words are referred to as "stopwords" [3]. The stopwords are words removed from the text document because they have no meaningful information. Stemming: it means identifying a word by its root, as for example words like technology and technologies have the same root technology

13 2.5 Extraction of relevant words from a corpus. Our goal is to identify and extract keywords(unigrams) which are considered to be relevant in certain documents of a corpus*. The statistical method used is called tf-idf (term frequencyinverse document frequency) which determines how important a word is to a document. The importance of a word increases with the number of times that particular word appears in the document. The measure of it is given by term-frequency tf which will be 0 if the word is not present in the document, otherwise it will be a nonzero value. The tf value of a word w in document d will be tf(d,w)={ 0 1+log(1+log(freq(d,w))) if freq(d,w)=0 otherwise where freq(d,w) is the number of times the word w occurs in the document. On the other hand, if a particular word appears in most of the documents of the corpus, it will less likely to be a keyword in the relevant document. As for example, for the word the term frequency in a document may be very high, but this word is not a keyword. Again the word the is more likely to be found in almost all the documents of a corpus, thus it is considered as a common word. So in order to reflect this in the statistical weight of a word, a new term called inverse document frequency (idf) is introduced which reduces the score of a word if it appears in almost all the documents. *a corpus is a collection of documents consisting of both relevant and non-relevant document

14 Thus, idf of a word w can be given as idf(w) = log( ( 1+ D ) / {d: w d} ) D : total number of documents in the corpus. d : number of document containing the term w. The tf value is expected to be high for a keyword in the relevant documents and a low value in the non relevant document. The idf value Therefore, the tf-idf score of a word w in document d is given by tf-idf(d,w) = tf(d,w) * idf(w) A high score of tf-idf is obtained by a high term frequency in relevant documents and a high inverse document frequency. The words with high tf-idf score are considered as keywords while the words with low tf-idf values are filtered out as common term. The output obtained after applying this process are the frequently occurring unigrams. With these unigrams, key-phrases are generated using one of the data mining algorithms for frequent pattern generation which will be discussed in the next chapter

15 Chapter 3 Different techniques for finding frequent patterns Frequent patterns are patterns that occur frequently in data. Frequent patterns help in mining associations, correlations, and many other interesting relationships among data. Moreover, it helps in data classification, clustering, and other data mining tasks as well. The different data mining algorithms used are discussed below: 3.1 The Apriori Algorithm for finding frequent patterns The input to this algorithm is a transaction database, each entry is a transaction of the form <transid, set if items>. This is also known as horizontal data format. At first the horizontal format table is scanned to find out the most frequent items. An item is said to be frequent if it appears in at least minsup transactions where minsup is a user-defined threshold value. It is a level-wise search, first it starts with frequent words or unigrams and subsequently generating bigrams, trigrams and so on. Any n-gram phrase is obtained by using the prior knowledge of (n- 1)-gram phrase. In general any n-gram is obtained from (n-1)- grams already generated. The candidates for n-grams is obtained - 8 -

16 by joining (n-1)-grams with itself. Any two (n-1)-grams can be joined if and only if all the words except the last one are matched. If it matches, then the candidate n-gram will be the common words followed by the last words of the two (n-1)-grams arranged in order. This procedure is known as joining. Now all the candidates generated in the join step may not be frequent. The candidate n-gram which exceeds the threshold value will be considered as frequent. To determine this count requires a full scan of database which is time consuming. So in order to reduce the candidate set, we use the Apriori property which states that any candidate n-gram cannot be the frequent n-gram if any of its subset is not frequent. The candidates whose subset is not frequent are eliminated. This is known as pruning. Once n-grams are obtained, (n+1)-grams are generated and the process continues until no more frequent patterns are generated. The Apriori Algorithm: Input: (transaction database, minsup count) for each item, check if it is a frequent itemset (appears atleast in minsup transactions ) (a set of items collectively known as itemset) k=1 repeat for each new frequent itemset I k with k items generate all itemsets I k+1 with k+1 items,such that I k is a subset of I k+1 Scan all transactions once and check if the generated k+1 itemsets are frequent k=k+1 Until no new frequent itemsets are generated - 9 -

17 return U I k Drawbacks of the algorithm: 1. It generates a large number of candidate sets which may not be a frequent pattern. 2. To determine the support count for candidate sets it requires a repeated scan of database. 3.2 The Apriori Algorithm Using Vertical Data Format The Apriori algorithm is applied on the vertical data format instead of horizontal format. The vertical data format contains entries like <item,set of transactions containing the item>. Let us consider an example of horizontal data format of four transactions. Transaction Items 1 a,b,c,d 2 b,c,d 3 a,c 4 c,d Table 1: Horizontal data format The corresponding vertical data format will be

18 Item Transactions a 1,3 2 b 1,2 2 c 1,2,3,4 4 d 1,2,4 3 Count Table 2: vertical data format Let the minimum support count be 2. The frequent 2 item set are obtained by intersecting the transaction sets of every pair of frequent single items. Item Transactions Count {a,c} 1,3 2 {b,c} 1,2 2 {b,d} 1,2 2 {c,d} 1,2,4 3 Table 3: Frequent 2 item sets Note item sets {a,b} and {a,d} are not frequent so they are rejected. Using the Apriori property, frequent 3 item sets are generated

19 Items Transactions Count {b,c,d} 1,2 2 Table 4: Frequent 3 item sets Advantage over horizontal data format: Repeated scan of database is not required here as the support count is stored in vertical data format. 3.3 FP Growth algorithm for frequent pattern generation FP Growth algorithm is another interesting algorithm which overcomes the major problems associated with Apriori algorithm. It follows a divide-and-conquer strategy. The original database is compressed or transformed into a tree well known as FP-tree, which holds all the information regarding frequent patterns. The compressed database or FP tree is divided into a set of conditional databases for each frequent item and mines each such database separately to generate frequent patterns. In this project, I have used this algorithm for generating frequent key phrases from text documents. The detail is discussed in the next chapter

20 Chapter 4 Study of FP Tree Growth Algorithm FP-Growth algorithm [1] is an efficient technique for mining frequent patterns from a text document. It overcomes the two major problems of Apriori algorithm. As in Apriori algorithm 1. It do not generates large no of candidate items. 2. No repeated scan of original database is required. It uses divide and conquer technique. Initially, the entire text database is transformed into a tree called FP-tree (frequent pattern tree) which holds all the information of database.as a result we do not have to scan the database again and again as in Apriori. It requires two scan of the database. In the first scan, it finds out the frequent words and their support count and in the second scan, it sorts each transaction according to their descending support count. 4.1 Definition of FP tree FP tree [4]can be defined as follows:

21 It consists of one root labeled as "null" and a set of item prefix subtrees as the children of the root.each node in the subtree consists of three fields - item-name, count, and node-link A frequent-item header table is maintained for efficient access of FPtree.It consists of three fields namely - item-name, support count and head of node-link 4.2 Formation of FP tree Algorithm for construction of FP tree: Input: Transaction DB, minimum support threshold. Output: FP-Tree 1. Collect the set of frequent items F and their support. Sort F in support order as prefix. 2. Create the root T of an FP-Tree, and label it as "null". Select and sort F in transaction according to the order of prefix. 3. Let the item list be [p P], p is the first item and P is remainder. for each item list call inserttree(items, T); 4. function inserttree([p P], T) if T has child N and N.itemName = p.itemname then N.count++; else create node N = p, N.count=1, be linked to T, node-link to the nodes with the same itemname; if P is nonempty then call inserttree(p, N);

22 Let us consider a transaction database. Items Transaction 1 f,a,c,d,g,i,m,p 2 a,b,c,f,l,m,o 3 b,f,h,j,o 4 b,c,k,s,p 5 a,f,c,e,l,p,m,n Let minimum support count to be 3. The frequent items are Item Count f 4 c 4 a 3 b 3 m 3 p 3 The transaction items are ordered according to descending order of their count

23 Transaction Items 1 f,c,a,m,p 2 f,c,a,b,m 3 f,b 4 c,b,p 5 f,c,a,m,p Using the above table FP tree is constructed: Header Table Item frequency head f 4 c 4 a 3 b 3 m 3 p 3 {} f:4 c:1 c:3 a:3 b:1 b:1 p:1 m:2 b:1 p:2 m:1 FP tree and its associated header table

24 4.3 Process of mining FP tree Frequent patterns are obtained for each entry in the header table starting from the lowest entry. For each item in the header, traverse the tree by following the links and form the set of prefix paths which is known as conditional pattern base.with this pattern base construct the FP tree called conditional FP tree and mine recursively on such a tree. The patterns are generated by concatenating the suffix pattern with the frequent pattern generated for conditional FP tree. The process continues until either the tree contains only the root or the tree has a single path.if the tree contains only a single path, then all the combinations of that path are generated and considered to be frequent patterns. Algorithm for FP Growth: Input: FP-Tree, minimum support threshold, without DB. Output: The complete set of frequent patterns. Method: Call FP-growth (FP-Tree, null) Procedure FP-growth (Tree, α) { 1. if Tree contain a single path P then 2. for each combination (denote as β) of the nodes in P do 3. generate pattern β α with support = minimum support in β 4. else for each ai in the Header Table of Tree do { 5. generate pattern β = ai α with support = ai.support 6. construct β's conditional pattern base and β's conditional FP-Tree Treeβ; 7. if Treeβ null then 8. call FP-growth (Treeβ, β); } } The above algorithm is used to mine FP tree. The set of conditional patterns generated are shown below:

25 Conditional pattern bases Item conditional pattern c f:3 a fc:3 b fca:1, f:1, c:1 m fca:2, fcab:1 p fcam:2, cb:1 The conditional FP tree generated is shown below: Item p Conditional pattern-base {(fcam:2), (cb:1)} Conditional FP-tree {(c:3)} p m {(fca:2), (fcab:1)} {(f:3, c:3, a:3)} m b {(fca:1), (f:1), (c:1)} Empty a {(fc:3)} {(f:3, c:3)} a c {(f:3)} {(f:3)} c f Empty Empty The frequent patterns thus generated are p, cp;

26 m, fm, cm, am, fcm, fam, cam, fcam; a, fa, ca, fca; c,fc. 4.4 Complexity of FP Growth Algorithm Time complexity of FP tree algorithm depends on searching of paths in FP tree i.e; the number of items in the path or in other words it depends on the depth of the tree and the number of items in the header table. So the time complexity is O(No. of items in header table * maximum depth of tree ). 4.5 Advantages of FP-growth over Apriori Algorithm 1.No repeated scans of database is required which is time consuming. 2.Avoids costly candidate generation. 3.FP tree contains all the information regarding mining frequent patterns.once the FP tree is constructed it never refers to original database

27 Chapter 5 The Implementation Process of Extraction of Frequent Patterns The project details out the process of extraction of frequent patterns from a corpus. A corpus is a collection of topical documents as well as non-topical documents. The ratio of topical documents and non-topical documents is assumed to be 1:10. All the documents are first converted into text format. The entire process of mining is described below: The corpus which is to be mined is first converted or transformed into text format. Text Preprocessing: All the symbols other than alphanumeric characters which do not play an important role in text mining are removed from the corpus. However symbols like ( and ) are kept to extract abbreviations. Again hyphens are also kept as it appears in relevant terms like DB-2, etc. The abbreviations within parenthesis are extracted as it may represent relevant terms, as for example, world wide web (www). Again some abbreviations whose matching words are not found, in

28 that case, user will have to decide how many words are to be considered. Once the abbreviation are extracted it is removed from the relevant documents. The string representing numeric figures are eliminated from the document. Again assuming that words cannot be of length greater than 29 they are also removed. The next step is called stemming. Stemming means replacing a word by it s root word. Without going into much complexity, only the words ending with letter s is dealt here. As for example, words like database and databases are assumed to be the same word. Again words ending with ies are replaced with letter y, for example technologies is replaced with technology. Evaluation of tf-idf score: After text preprocessing is over, the relevant files gets reduced and contains only a collection of words. For each of these words, tfidf[chap2] score is computed. Now, the term frequency tf of a keyword is expected to be high in relevant documents and low in non-relevant documents. Again for a common word the tf value is expected to be high throughout the corpus. The idf value should be low for a keyword and high for a common word. Thus for any word two term frequency averages are computed one over the relevant document(avg1) and other over non relevant documents(avg2) and the difference of their averages(avg=avg1-avg2) is evaluated. It is observed that the difference Avg is high for keywords which when multiplied by a high idf value will raise the tf-idf score. On the other hand for a common word the difference Avg is a low value and idf is also low enough which significantly reduces the tf-idf score. By trial and error method a threshold value is set such that the words with tf-idf score exceeding the threshold value is considered

29 to be keywords and the rest are considered as common words. Finally all the words except the keywords are removed from the relevant documents. Process of mining: The reduced file is converted to vertical data format(as it is easier to construct in case of text file)[chap3]. In the vertical data format instead of word, its hash code is kept which is merely an integer value. The reason behind integer hash code is that integer allows faster manipulation than strings. The hash function for a word of length n that is used here is h(word,n) = word[n] x (prime) n%10 + word[n-1] x (prime%10) (n-1)% word[0] x (prime) 0. where word[i] represents the ascii value for that character. The prime number chosen is 11. The words are stored in the hashtable. If a word comes out with a hash code such that there is already an entry in the hash-table, such a situation is called collision. In that case the prime number is replaced by the next immediate higher prime number. This process continues until there is no collision. To find frequent key phrases from a text document we are adopting one of the frequent pattern generation mechanism called FP tree growth algorithm, which is mostly used in Association Rule Mining. As in FP tree growth algorithm, the input required should be in the horizontal format so the vertical data format is then converted into horizontal data format. Here in analogy to association rule mining, each sentence in text document is considered as a transaction and the words in the sentence as items. The transaction number representing a sentence in a document consists of two fields

30 namely, document number and sentence number. Finally, with horizontal data table FP tree is constructed using the algorithm for the formation of FP tree. The FP tree is then mined using the FP Growth algorithm which generates frequent patterns related to the relevant documents[chap 4]

31 Analysis and Comparison of different algorithm and their results Chapter 6 This is the analysis that has been done on four different data sets, showing how time needed by the three different approaches i.e. fp-growth, apriory on vertical format file and apriory on horizontal format file, for mining up to frequent 5-itemsets. Data Set I : ebook of Data Mining Concepts and Techniques (1726KB). Total Sentences : 2127 Total Words : 8830 Unique words : 513 Time in milliseconds Support FP- Apriori(Vertical) Apriori(horizontal) Growth Data Set I time (in ms) sp fp-growth algorithm apriory on vertical apriory on horizontal min. support count

32 Data Set II : ebook of Image Analysis for Face Recognition (58KB). Total Sentences : 399 Total Words : 2911 Unique words : 522 Support FP- Growth Time in milliseconds Apriori(Vertical) Apriori(horizontal) Data Set II time (in ms) sp fp-growth algorithm apriory on vertical apriory on horizontal min. support count

33 Data Set III : an article on X-ray (21KB). Total Sentences : 80 Total Words :750 Unique words :382 Time in milliseconds Support FP-Growth Apriori(Vertical) Apriori(horizontal) Data Set III time (in ms) sp fp-growth algorithm apriory on vertical apriory on horizontal min. support count

34 Data Set IV : ebook of Linux Programming Unleashed (1499KB). Total Sentences : 9645 Total Words : Unique words :741 Time in milliseconds Support FP-Growth Apriori(Vertical) Apriori(horizontal) Data Set IV time (in ms) sp fp-growth algorithm apriory on vertical apriory on horizontal min. support count

35 Chapter 7 Conclusion and Future Extensions The results from the last chapter clearly indicate that the FP growth algorithm largely outperforms both varieties of Apriori algorithm in the matter of time complexity in case of large corpus. This is easily explained, as in this algorithm the database scan has been minimized drastically. The space complexity achieved is also a plus-point of this algorithm, as the whole database is being compressed into a FP-tree and this leads to further reduction in corpus-handling time as efficient and time-tested tree-handling routines can be utilized. However, the link-based data structure and the corresponding modules for handling this structure leave the stamp of their functional deficiency while handling small or moderate corpus. Some thoughts have been devoted to search for improvement techniques for a very large corpus. The FP growth algorithm used for frequent pattern generation may not work if the entire FP tree cannot be loaded in the main memory. This particular problem may be solved using Dynamic FP growth algorithm[2], which is a variation of FP growth algorithm. Here instead of loading the FP tree in main memory it may be stored in the disk and only the portion of it that is required may be brought into the main memory

36 Chapter 8 Bibliography [1] J. Han, M. Kamber, Data Mining Concepts and Techniques, Morgan Kaufmann Publishers, San Francisco, USA, 2001, ISBN [2] Cornelia Gyorodi, Robert Gyorodi, T.Cofecy & S.Holban- Mining association rules using Dynamic FP-trees. [3] [4] ao_10.pdf