International Journal of Advancements in Research & Technology, Volume 3, Issue 3, March-2014 ISSN

Transcription

1 International Journal of Advanements in Researh & Tehnology, Volume 3, Issue 3, Marh-204 ISSN Phrase Based Doument Retrieving y Comining Suffix Tree index data struture and Boyer- Moore faster string searhing B.Ganga, Researh student, gangaalu@yahoo.om ABS TRACT Phrase has een onsidered as a more informative feature term for improving the effetiveness of doument retrieval.this paper propose an Algorithm A Phrase Based Doument Retrieval to retrieve the similar douments y omining two exiting suffix tree,index data struture and The Boyer-Moore Algorithm, faster string searhing. The suffix tree is onstruted ased on E. Ukkonen, on-line Constrution Of Suffix Trees For Strings, a most effiient string-mathing. On the onstruted suffix, The Boyer-Moore Algorithm is applied to hek the presene of pattern i.e. the input phrase in order and without order to retrieve the similar douments. Furthermore, y studying the property of suffix tree and Boyer-Moore, we onlude that suffix tree data struture store huge douments and Boyer-Moore heks the presene of pattern fastly. This onlusion suffiiently explains why the Phrase Based Doument Retrieval works muh etter than the other doument retrieval. KEYWORDS: Suffix tree, Boyer-M oore, Doument retrieval.. INTRODUCTION Phrase identifiation is an important task of Doument Retrieval. Most of the retrieval tehniques are designed and ased on words or keywords and its ourrene. Conepts are often expressed as phrases onsisting of multiple words whose meaning is sustantially different from the meaning of the individual words. For example the phrase Artifiial Intelligene is different than individually Artifiial and Intelligene, and Operating System is different than the individual words Operating and System. Composite expressions are prevalent in natural language, hene there is a need in the information retrieval systems to have a methodology that identifies phrases and do a retrieval ased on it. Doument representation is a major prolem in the information retrieval. The Phrase Based Doument retrieval omine advantages of two Ukkonen s, suffix tree, for strings and oyer-moore string searh. Suffix tree is used to represents the doument as suffix of the tree and the Boyer-Moore, string searhing is used to searh input phrase in onstruted suffix in order and without order to retrieve the similar douments together. The purpose of implementing suffix tree is, it is effiient data strutures to store the large text, faster and numer of nodes reated is less when ompared to suffix trie, the previous and The Boyer-Moore Algorithm, is faster string searhing. This paper emphasizes on phases present in douments and do a retrieval ased on it.the Phrase Based Doument Retrieval Algorithm represent eah doument as suffix trees, a data struture where phrases of the douments are stored as suffixes of tree and Boyer Moore is used to hek the presene of pattern i.e. the input phrase in order and with out order. Suffix tree is onstruted y using E.Ukkonen, on-line Constrution Of Suffix Trees For Strings and enhaned to represent suffix tree for any type of doument namely Portale Doument Format, MS Format Files like MS Word, MS PowerPoint, MS Exel and Text files. And here edges of suffix tree are suffix i.e. phrases of the douments. And one suffix tree is onstruted, several operations an e performed quikly, for instane loating a sustring in, loating mathes for a regular expression or pattern et. suffix tree retrieval performane is etter ompare to other form ased retrieval and searhing eomes faster with suffix tree onstrution. The Boyer-Moore string searh is a partiularly effiient, and has served as a standard enhmark for string searh ever sine. This s exeution time an e su-linear, as not every harater of the string to e searhed needs to e heked. Generally speaking, the gets faster as the target string eomes larger..ojetives of the work The ojetives inlude Retrieving similar douments whih has input phrase in order and without order y omining two methodogies, the suffix tree and Boyer-Moore. And enhaned to retrieve the douments of different form suh as Portale Doument Format, MS Format files and Text files. The approah was an experiment to oserve the proposed methodology s aility i.e. using phrase to improve the retrieval. The rest of this paper is organized as follows: Setion 2 disuss the literature survey. Setion 3 starts with design of Phrase Based Doument Retrieval. Setion 4 illustrates the testing and experimental results. Finally Setion 5 summarizes the ontriutions of work.

2 International Journal of Advanements in Researh & Tehnology, Volume 3, Issue 3, Marh-204 ISSN LITERATURE SURVEY 2. Doument Retrieval Doument Retrieval is defined as the mathing of some user stated query against a set of free-text reords []. These reords ould e any type of mainly unstrutured text, suh as newspaper artiles, real estate reords or paragraphs in a manual. User queries an range from multi-sentene, full desriptions of an information need to a few words. and at most one parent node. An ordered tree data struture is used to store an assoiative array where the keys are strings. It s regarded as faster than a hash tale ut less spae-effiient. 2.2 Doument Retrieval System Doument Retrieval system onsists of a dataase of douments, a lassifiation to uild a full text index, and a user interfae to aess the dataase. The system finds information to given riteria y mathing text reords (douments) against user queries, as opposed to expert systems that answer questions y inferring over a logial knowledge dataase. And two main lasses of indexing shemata for doument retrieval systems are form ased (or word ased), and ontent ased indexing. The doument lassifiation sheme (or indexing ) in use determines the nature of the doument retrieval system. Form ased doument retrieval addresses the exat syntati properties of a text, omparale to sustring mathing in string searhes. The text is generally unstrutured and not neessarily in a natural language, the system ould for example e used to proess large sets of hemial representations in moleular iology. A suffix tree is an example for form ased indexing. The ontent ased approah exploits semanti onnetions etween douments and parts thereof, and semanti onnetions etween queries and douments. Most ontent ased doument retrieval systems use an inverted index. 2.3 Index Data Strutures Searh engine arhitetures vary in the way indexing is performed and in methods of index storage to meet the various design fators. Types of indies inlude [2]: 2.3. Suffix Tree Figuratively strutured like a tree, supports linear time lookup. Built y storing the suffixes of words. The suffix tree is a type of trie. Tries support extendale hashing, whih is important for searh engine indexing. Used for searhing for patterns in DNA sequenes (Deoxyrionulei aid is a nulei aid that ontains the geneti instrutions used in the development and funtioning of all known living organisms and some viruses) and lustering Tree A tree is a widely-used data struture (way of storing and organizing data) that emulates a hierarhial tree struture with a set of linked nodes. And it is an ayli onneted graph where eah node has a set of zero or more hild nodes, Figure 2. Tree (Data Struture) Inverted index In information tehnology, an inverted index is an index data struture storing a mapping from ontent suh as words or numers, to its loation in data ase file allowing full text searh. And it stores a list of ourrenes of eah atomi searh riterion, typially in the form of a hash tale or inary tree Ngram index Stores sequenes of length of data to support other types of retrieval or text mining Term doument matrix Used in latent semanti analysis, stores the ourrenes of words in douments in a two-dimensional sparse matrix (a matrix populated primarily with zeros). 2.4 Suffix Trees 2.4. Tries and Trees A trie (from retrieval), is a multi-way tree struture useful for storing strings over an alphaet. It has een used to store large ditionaries of English (say) words in spelling-heking programs and in natural-language "understanding" programs. For example the given the data is BCABC then the orresponding trie would e as shown in Figure 2.2. The advantage of suffix trie is that if you have an input text of length n, and a searh string of length m, a traditional rute fore searh will take as many as nm harater omparison to omplete, the suffix trie demolishes this performane y requiring just m harater omparisons, regardless of the length of the text eing searhed. The disadvantage in trie is that more spae is wasted as a lot of nodes near the edge of the trie will have most su tries set to nil, numer of nodes reated is more.

3 International Journal of Advanements in Researh & Tehnology, Volume 3, Issue 3, Marh-204 ISSN a a C BCABC a a a a Fig 2.3 Suffix tree for String BCABC Three Main [4][5] used to onstrut suffix trees are Weiner,MCreight, spae effiient linear-time, onstruted in 97 and E.ukkonen, On-line onstrution of suffix trees,995. Weiner was the first to show that suffix tree an e uilt in linear time. Figure 2.2 Trie for string BCABC Suffix Trees Introdution The suffix tree for a given lok of data retains the same topology as the suffix trie, ut it eliminates nodes that have only a single desendant.the tree has the same general shape as trie just far fewer nodes. By eliminating every node with just a single desendant, the ount is redued. Suffix Tree Mehanisms Suffix Tree Mehanisms start at longest suffix and work our way down to shortest suffix. Eah suffix ends at a node whih is of 3 types. First one is Leaf nodes whih are defined as all the suffixes that are longer than the suffix defined y the ative point. Seond is expliit node whih is the non-leaf nodes where 2 or more edges part way. Third is impliit node whih is the non- leaf nodes whose prefixes all ends in the middle of the edges. MCreight introdued a more spae effiient linear-time in 97 MCreight s original for onstruting a suffix tree had a few disadvantages, priniple among them was the requirement that the tree e uilt in reverse order, and meaning haraters were added from the end of the input. This ruled the out for the on -line proessing, making it muh more diffiult to use for appliations suh as data ompression. Suffix is represented y defining its harater. As first harater starts at node 0 i.e. Suffix ojet defines the last harater in a string y starting at a speifi node then following the string of haraters in the input sequene pointed to y the first harater index and last harater index memers Ukkonen developed a simpler to understand linear-time in 995. Ukkonen s was slightly modified version of the that works from left to right [5]. For a given string of text, T, Ukkonen s starts with an empty tree, then progressively adds eah of the n suffixes of T to the suffix tree. For example, when reating the suffix tree for BANANAS, is inserted into the Tree, then BA, then BAN, and so on. When BANANAS is finally inserted, the tree is omplete. AN B BA A BAN Definitions The suffix tree for the string S of length n is defined as a tree suh that the paths from the root to the leaves have a one to one relationship with the suffixes of S and edges spell non- empty strings, And all internal nodes (exept the root ) have at least two hildren[3]. The suffix tree for the string BCABC is as follows and ompared to suffix trie the numer of nodes reated is less. Figure 2.4 Progressively Building the Suffix Tree Algorithm for Updating the Suffix [4] Build suffix tree T for string S [...m] Begin N

4 International Journal of Advanements in Researh & Tehnology, Volume 3, Issue 3, Marh-204 ISSN String Mathing Algorithm Given a pattern of length m and a ody of text of length n, return true if the pattern is found in text, or false otherwise. Inluding omputational iology, omputer siene, and mathematis []. o Appliations Ovious find text in a doument, on a In the ase of a mismath, the omputes a new alignment for the target string ased on the mismath. This is where the gains onsiderale effiieny Comparisons of Brute Fore, Horspool, Boyer-Moore String Searh - Numer of Comparisons Made x 0 Brute Fore Horspool Boyer-Moore 4 2 numer of omparisons ) Build the tree in m phases, one for eah harater. At the end of phase i, we will have tree Ti, whih is the tree representing the suffix S [...i] is an online onstrution. 2) In eah phase i, we have i extensions, one for eah harater in the urrent suffix. At the end of extension j, we will have ensured that S [j..i] is in the tree Ti. End The egins with an impliit suffix tree ontaining the first harater of the string. Then it steps through the string adding suessive haraters until the tree is omplete. This order addition of haraters gives Ukkonen's its "on-line" property, earlier s proeeded akward from the last harater Text length x 0 wepage, et Brute Fore Fig 2.5 Comparisons of Brute Fore, Horspool, BoyerMoore The rute fore onsists in heking, at all positions in the text etween 0 and n-m, whether an ourrene of the pattern starts there or not. Then, after eah attempt, it shifts the pattern y exatly one position to the right. The rute fore requires no preproessing phase, and a onstant extra spae in addition to the pattern and the text. During the searhing phase the text harater omparisons an e done in any order. The time omplexity of this searhing phase is O (mn) (when searhing for am- in an for instane). The expeted numer of text harater omparisons is 2n Horspool s Algorithm Step For a given pattern P of length m, ompute the shift tale. Step 2 - Align the pattern against the eginning of the text Step3 Starting with the last harater in the pattern, ompare text and pattern. If pattern mathes, return suess If pattern does not math, use the mismathed harater from the text as. Lookup in the shift tale, and align the pattern to that position. Repeat step 3 until the end of string (or a math is found) Boyer-Moore The B-M takes a akward approah: the target string is aligned with the start of the hek string, and the last harater of the target string is heked against the orresponding harater in the hek string [7]. In the ase of a math, then the seond-to-last harater of the target string is ompared to the orresponding hek string harater. (No gain in effiieny over rute-fore method) 3. DESIGN 3. Theoretial akground The design is onsidered to e one of the most important phases of software development. The first phase is the development of a suffix tree, data struture for storing the douments of different format suh as Portale Doument Format, MS Format files and Text files.in onstruted suffix tree, the edges are stored in hash tale. Hashing is a very effiient way to store and retrieve data [9]. The seond phase is the development of the Phrase Based Doument Retrieval. For the input, phrase either in order or without order is taken.and the last phase is the development of the proper user interfae to assist the user in using th e Phrase Based Doument retrieval appliation [0]. 3.2 Prolem Formulation The fous of work is to omine the advantages of two methodogies suffix tree and Boyer-Moore in doument retrieval. As a result the Phrase Based Doument retrieval Algorithm represent eah doument as suffix trees, where phrases of the douments are stored as suffixes of tree in hash tale for effiient storing and retrieval and Boyer Moore is used to hek the presene of pattern i.e. the input phrase in order and with out order. 3.3 Phase :Algorithm for Construting The Suffix Tree Suffix Tree is onstruted for the doument y using E.Ukkonen, On-line onstrution of suffix trees and enhaned to onstrut the suffix tree for doument of different types. The onstruted suffix tree store the edges

5 International Journal of Advanements in Researh & Tehnology, Volume 3, Issue 3, Marh-204 ISSN heese 3 in a hash map, using a hash key ased on their starting node numer and the first harater of the sentenes. This implementation ompetently handles huge amount of data. A simple suffix tree for File.pdf [8] is shown in figure 3.. File.pdf Cat ate heese.mouse ate heese too.cat ate mouse too. Root Root ate Proposed Algorithm Comines Suffix Tree and Boyer-Moore too for Douments with input phrase in order 7. The suffix tree onstruted for every doument is otained. 2. The suffixes are sorted ignoring ase. 3. Then The Boyer-Moore string searh is applied. 4. Input phrase is defined as pattern and stored suffix as text. 5. Then The Boyer-Moore string searh is applied. Figure 3. Example of a Suffix Tree with Sentenes Algorithm for Construting the Suffix Tree The general arhiteture of Phrase Based Doument retrieval. Read the Doument. retrieving input phrase in order in shown in figure3.2 The 2. Tokenize the Doument into sentenes. general desription of the is, it takes input as set of 3. Start onstruting the suffix tree for eah sentene. douments in the form of Portale Doument Format, MS i. For Every sentene, add suffix is done to add the suffix Format files and Text files and input phrase and implements to onstrut suffix tree. suffix tree whih represent doument as suffix and ii In add_suffix then Boyer-Moore is applied and output is If the node is expliit node otained as the set of douments whih ontains the input Then the suffix are added into EDGE_KEY phrase in order. Else if the node is impliit node Then The suffix are added into EDGE_KEY lass And the split edge is done. iii. A node is reated. iv. The details of the suffix are added into edge lass. v. To add the suffix an edge is inserted. vi. Canonize is done to move to next smaller suffix. 4. The onstruted suffix is displayed. 3.4 Phase 2: Algorithm for Retrieving the Douments with input phrase in order. Cat ate heese Cat ate mouse h Mouse ate heese 4 5 ate heese too heese too On the onstruted suffix, Boyer Moore is applied to hek the presene of pattern i.e. the input phrase in order. The Boyer-Moore string searh (math a pattern of length n in a text of length m) 8 ate Mouse 9 0 Mouse i. Calulate the length of pattern and Text. ii. Preproess the pattern for the right-to-left-san and adharater-shift rules y finding the right-most positions of all haraters in the pattern. iii. Align p and t, starting on index and shift p to the left, until we reah the end of t. iv. San the pattern from right to left, omparing the aligned haraters at the urrent position in the text x and at the urrent position in the pattern y. v. If the pattern is longer than the text, we have no math. vi. In the ase of a mismath, we do the shifting Retrieve the right-most index of the mismathing textharater in the pattern. If the mismathing harater in the text is not in the pattern, shift until we are aligned ehind the mismath-position. Else we shift the pattern to the right until the right-most ourrene of x in the pattern is under the mismath position in the text. vii. If the haraters are equal and the pattern has een sanned ompletely from right to left, we have a math. We store the math and shift the pattern one position to the right.

6 International Journal of Advanements in Researh & Tehnology, Volume 3, Issue 3, Marh-204 ISSN Diretor y with set of Doume Douments are Represente d Figure 3.2 General Arhiteture of Phrase Based Doument retrieval retrieves Input Phrase in Order Algorithm for Retrieving the Douments with input phrase without order The general arhiteture of Phrase Based Doument retrieval retrieving input phrase without order in shown in figure3.3.the general desription of the is, it takes input as set of douments in the form of Portale Doument Format, MS Format files and Text files and input phrase and implements suffix tree whih represent doument as suffix and then Boyer-Moore is applied where input phrase is tokenized into words, input phrase ontaining all words are heked in suffixes and output is otained as set of douments whih ontains the input phrase without order. Diretor y with set of Doumen Phrase (in order) Douments are Represente d Phrase (with out order) Otain one file Otain one file Implement Suffix Tree Boyer- Moore is applied Stringmathing Boyer- Moore is applied Stringmathing Suffix Tree Is Construted Doument Containing Phrase (with order) Suffix Tree Is Construted Figure 3.3 General Arhiteture of Phrase Based Doument retrieval retrieves Input Phrase without Order Algorithm for Retrieving the Douments with input phrase without order. The suffix tree onstruted for every doument is otained. 2. The suffixes are sorted ignoring ase. 3. The input phrase is tokenized into words and stored. 4. Eah word in the input is defined as pattern individually and stored suffix as text. 5. Then The Boyer-Moore string searh is applied as shown in for retrieving the Douments with input phrase in order. And then following steps are also implemented to searh input phrase present in any order. Begin Chek for math and store the math. Chek if input phrase is equal to store math with same suffix. If so then store the name of the doument after heking for the presene of any dupliation of doument name. Else Chek in next suffix End. 3.5 Doument Colletion Input is set of douments in form PDF files, MS Format Files suh as MS Word, MS Exel, MS PowerPoint and Text File. Implement Suffix Tree Doument Containing Phrase (with out order ) 4. TESTING AND EXPERIMENTAL RESULTS The development and deployment of The Phrase Based Doument retrieval is done in Elipse mix of Java.4 and Java 5 VMs[][2]. For reading Portale Doument Format PDF Box, java API and for reading MS Format files suh as MS Word, MS Exel, MS PowerPoint Apahe POI java API is used. Comparative study of Phrase Based Doument retrieval Algorithm and previous Suffix tree and Boyer- Moore Algorithm. Previous Suffix tree onstrut suffix tree for strings and numers of nodes reated are more as suffixes are onstruted for strings and Boyer-Moore Algorithm searh pattern inside a text where as Phrase Based Doument retrieval Algorithm Comines advantages of oth Previous s Suffix Tree, Boyer-Moore and enhaned to represent different types of douments, searhing is done in douments and retrieve them ased on input phrase in order and without order. Numers of nodes reated are less as suffixes are onstruted for sentenes in douments. 4. Performane Metris The performane of Phrase Based Doument retrieval Algorithm is evaluated using performane oeffiient suh as Numer and types of Files availale for retrieval. Size of Files availale for retrieval. Time taken to retrieve the similar douments

7 International Journal of Advanements in Researh & Tehnology, Volume 3, Issue 3, Marh-204 ISSN The graphial representation of the performane of Phrase Based Doument retrieval Algorithm is shown in figure 4. and 4.2. testing and further improvements are required whih are given elow. 5. Limitations of the work The prolems oserved were diffiulty to retrieve the douments when input phrase is huge. The time taken to retrieve the douments also grows in diret proportion with size of douments. Figure 4.2 Size of Douments in MB with respet to Numer of Douments 5. CONCLUSION AND FUTURE WORK The projet was aout retrieving similar douments whih has input phrase in order and without order y omining two methodogies, the suffix tree and Boyer-Moore and enhaned to retrieve the douments of different form suh as Portale Doument Format, MS Format files and Text files. The approah was an experiment to oserve the proposed methodology s aility to improve the retrieval, whih partially sueeded with the given samples. Thorough 5.2 Future Enhanements The size of suffix tree is growing in diret proportion to size of the douments. And the time taken to retrieve the douments also grows in diret proportion. To provide more effiient way of retrieval suffix tree an e onverted into suffix arrays. And some page ranking an e applied to rank the retrieved douments. REFERENCES [] Fe 03, 2009 [2] Fe Figure 4. Time taken to retrieve douments in minutes with 03, respet to Numer of Douments [3]E.Ukkonen, On-Line Constrution of Suffix Trees, Algorithmia, vol.4, no.3, pp , Septemer, 995. [4] R.Giegerih and S.Kurtz, From Ukkonen to MCreight and Weiner: A Unifying View OF Linear _ Time Suffix Tree Constrution,Algorithmia,volume9, pages33 353,997. [5] Mark Nelson, Fast String searhing with suffix Trees, Dr. Do's Journal, August, 99. [] Dan Gus field, Algorithms on Strings, Sequenes and Trees, Camridge University Press, Seond Edition, 999. [7]. Moore_string_searh_, June4, [8]. Hung Chim and Xiaotie Deng, Effiient Phrase Based Doument Similarity For Clustering, IEEE Transations On knowledge And Data Engineering, vol 20, Sep 2008 [9] Her Shildt, Java programming ookook, MGraw-Hill Osorne Media, First Edition,Novemer 5, [0] Dr. Satyaraj Pantham, Pure JFC Swing, Sams, First Edition, Nov 999. [] Herert Shildt, Java Complete Referene, MGraw-Hill Osorne Media, 5th edition, [2] Marh 4, Aout The Author B.Ganga native of Tamilnadu. I have reeived my B.E degree in omputer siene and engineering from Karunya Ins titute of Tehnology, Tamilnadu and M.Teh in s oftware Engineering from M.S.Ramaiah Institute of Tehnology, Ba n g a lo r e. I have s even years of leturing experiene in engineering ollege and presently doing researh in data mining. My area of interes t helping human kind through tehnology.