AN EFFECTIVE SEARCH TOOL FOR LOCATING RESOURCE IN NETWORK

Transcription

1 AN EFFECTIVE SEARCH TOOL FOR LOCATING RESOURCE IN NETWORK G.Mohammad Rafi 1, K.Sreenivasulu 2, K.Anjaneyulu 3 1. M.Tech(CSE Pursuing), Madina Engineering College,Kadapa,AP 2. Professor & HOD Dept.Of CSE, Madina Engineering College,Kadapa,AP 3. Asst.Prof & HOD,Dept. of Computer Applications, Sri Sai Institute Of Technology &Science, Rayachoty,Kadapa,AP Abstract: A good search engine ensures that users find what they're looking for, first time, regardless of the format or location of the information. This means that a wide variety of information can be effectively dispersed and made available to staff, without the need for complex navigation systems or filing conventions. Most intranets evolve over time, and search functionality need not be a daunting task. A search tool can be implemented quickly, and then refined as the intranet grows and the needs of the organization change. Importantly, a flexible search engine that is costeffective and expands to suit growing requirements can be a much better. It is important to recognize that every intranet is different, with its own objectives, requirements and environment. INTRODUCTION The desktop search is a tool using certain keywords to search various data sources like the web browser histories, the archives, the text documents or the metadata of mp3 files from the local disk storage. After the search tool is installed, desktop search engine used by the search tool will parse all files stored on local disks and maintain an index database for achieving reasonable performance especially when the local storages have kept data of several hundred gigabytes or even terabytes. The key techniques used by the desktop search engine are the ability to parse all files in various formats and the ability to do full text search to the parsed contents. Search all contents in the intranet: It is convenient to find information that is hidden in the myriad files in the local hard drives with the help with desktop search tool. However, for some enterprise applications, sometimes the useful information like meeting minutes or schedule of certain works may be 36

2 concealed in other person's computer in metadata about documents. Lucene lets the intranet. us add searching capabilities to applications. Some Salient features of Intranet based search engine are: It is FAST For fast search it creates indexes of the shared folders. Creation of indexes considerably reduces the search time for a query. The application is DISTRIBUTED Documents to be searched may not reside on your local machine. Being distributed it can make queries to other machines in the network. The application uses P2P sharing No special requirements for server just install and get connected. Each peer serves others. 1. LUCENE PACKAGE: Lucene is a high performance, scalable Information Retrieval (IR) library. Information retrieval refers to the process of searching for documents, information within documents or Fig: Typical components of search application, the shaded components show which parts Lucene handles. Lucene provides a simple yet powerful core API that requires minimal understanding of full-text indexing and searching. Because Lucene is a Java library, it doesn t make assumptions about what it indexes and searches, which gives it an advantage over a number of other search applications. Its design is compact and simple, allowing Lucene to be easily embedded into desktop applications. 37

3 Lucene can index and make searchable documents where they any data that can extract text from. As appear. The quality of a seen in the following figure, Lucene search is typically doesn t care about the source of the data, its format, or even its language, as long as we can derive text from it. This means described using precision and recall metrics. Recall measures how well the we can index and search data stored in search system finds files: web pages on remote web servers, documents stored in local file systems, relevant whereas documents, precision simple text files, Microsoft Word measures how well the documents, XML or HTML or PDF system filters out the files, or any other format from which irrelevant documents. you can extract textual information. Search Query: This is the 2. IMPLEMENTATION: process of consulting the search index and retrieving We have used Lucene Package for the documents matching indexing the content of files. the Query, sorted in the 3.1 Indexing and searching: requested sort order. Index document: During the indexing step, the document is added to the index. Lucene provides everything necessary for this step, and works quite a bit of magic under a surprisingly simple API. Components for searching: Searching is the process of looking up words in an index to find There are three common models of search: Pure boolean model -- Documents either match or does not match the provided query, and no scoring is done. In this model there are no relevance scores associated with matching documents; a query simply identifies a subset of the overall corpus as matching the query. 38

4 Vector space model -- Both 3.2 The core indexing classes: queries and documents are modeled as vectors in a very high dimensional space, where In our Indexer class, we need the following classes to perform each unique term is a the simplest indexing dimension. Relevance, or procedure: similarity, between a query and a document is computed by a IndexWriter vector distance measure Directory between these vectors. Analyzer Probabilistic model -- Computes the probability that a Document document is a good match to a Field query using a full probabilistic approach. Lucene s approach combines the vector space and pure Boolean models. Lucene returns documents which we next must render in a very consumable way for users. Render Results: Once we have the raw set of documents that match the query, sorted in the right order, we then render them to the user in an intuitive, consumable manner. Fig: Classes used when indexing documents with Lucene IndexWriter: IndexWriter is the central component of the indexing process. This class creates a new index or opens an existing one, and then adds, removes or updates documents in the index. 39

5 Directory: The Directory class represents the location of a Lucene index. It s an abstract class that allows its subclasses to store the corresponding value, and a bunch of options, described in Section, that control precisely how Lucene will index the Field s value index as they see fit. In our Indexer example, we created an 3.3 The core searching classes: FSDirectory, which stores real files The basic search interface that in a directory in the filesystem, and Lucene provides is as passed it to IndexWriter s straightforward as the one for constructor. Analyzer: Before text is indexed, it s passed through an Analyzer. The indexing. Only a few classes are needed to perform the basic search operation: Analyzer, specified in the IndexSearcher IndexWriter constructor, is in charge of extracting those tokens out of text Term that should be indexed, and Query eliminating the rest. TermQuery Document: A Document represents a collection of fields. It can be like TopDocs virtual document a chunk of data, IndexSearcher: IndexSearcher is to such as a web page, an searching what IndexWriter is to message, or a text files that you indexing: the central link to the want to make retrievable at a later index that exposes several search time. Fields of a document represent methods. the document or meta-data associated with that document. Term: A Term is the basic unit for searching. Similar to the Field Field: Each Document in an index object, it consists of a pair of string contains one or more named fields, elements: the name of the field and embodied in a class called Field. the word (text value) of that field. Each field has a name and 40

6 Query: Lucene comes with a Lucene to translate a PDF number of concrete Query into a Lucene document. subclasses. The most basic Lucene Query: TermQuery. Other Query XPDF is an open source types are BooleanQuery, tool that is licensed under PhraseQuery, PrefixQuery, the GPL. It's not a Java PhrasePrefixQuery, RangeQuery, tool, but there is a utility FilteredQuery, and SpanQuery. called pdftotext that can translate PDF files into text TermQuery: TermQuery is the most basic type of query supported files on most platforms from the command line. by Lucene, and it s one of the primitive query types. It s used for matching documents that contain fields with specific values. JPedal is a Java API for extracting text and images from PDF documents. TopDocs: The TopDocs class is a simple container of pointers to the top N ranked search results documents that match a given query Simple Text Extractor Library for use with PDF documents. Relies on PDFBox. In order to index PDF documents we need to first parse them to extract text that we want to index from them. Here are some of the PDF parsers: PDFBox is a Java API from Ben Litchfield that will let us access the contents of a PDF document. It comes with integration classes for Fig : Indexing with Lucene breaks down into three main operations: extracting text from source documents, analyzing it and saving it to the index. 41

7 During indexing, the text is first extracted from the original content and its task. It typically uses port 20 for data transfer and port 21 to listen to used to create an instance of Document; commands. Though having data containing Field instances hold the content. The text in the fields is then analyzed, to produce a stream of tokens. Finally, those tokens are added to the index in a segmented architecture. transferred over port 20 is not always the case as it can also be a different port as well. That is where the confusing part for many people comes into play. There are two modes to FTP, namely active and passive mode. These two modes are 3.4 FTP Protocol: File Transfer Protocol (FTP) is a standard network protocol used to copy a file from one host to another over a initiated by the FTP client, and then acted upon by the FTP server. Some of the terminology regarding File Transfer Protocol TCP/IP-based network, such as the Internet. FTP is built on client-server 1. Control connection: The architecture and utilizes separate control and data connections between the client communication path between the USER-PI and SERVER-PI and server applications which solve the for the exchange of problem of different end host commands and replies. This configurations (i.e. Operating System, file names). FTP is used with user-based connection follows the Telnet Protocol. password authentication or with anonymous user access. 2. Data connection: A full duplex connection over which data is FTP itself uses the TCP transport transferred, in a specified protocol exclusively, or in other words, it mode and type. The data never uses UDP for its transport needs. Typically an application layer protocol transferred may be a part of a file, an entire file or a number will use one or the other. One notable of files. The path may be exception to that is DNS or Domain between a server-dtp and a Name System. FTP also is odd in the user-dtp, or between two fact that it uses two ports to accomplish server-dtps. 42

8 3. Data port: The passive data The query entered will be fired transfer process "listens" on the data port for a connection from the active transfer process in simultaneously on both local machine and remote machine and the results are displayed obtained from both local and order to open the data remote machine. By selecting a connection. particular file we can retrieve a file from the remote machine using FTP Protocol. 4. DTP: The data transfer process remote machine. By selecting a establishes and manages the particular file we can retrieve a file from data connection. The DTP can the remote machine using FTP Protocol. be passive or active. 5. End-of-Line: The end-of-line sequence defines the separation of printing lines. The sequence is Carriage Return, followed by Line Feed. 6. EOF: The end-of-file condition that defines the end of a file being transferred. 7. EOR: The end-of-record condition that defines the end of a record being transferred. 8. Error recovery: A procedure that allows a user to recover from certain errors such as failure of either host system or transfer process. In FTP, error recovery may involve restarting a file transfer at a given checkpoint. 43

9 3.5 Project Screenshots Fig: Indexing Files Fig : Settings for Indexing Fig : Results of search query moblin Fig : Search Settings 44

10 4 CONCLUSION: The desktop search can be sped up with the help of other computers in the intranet. Rather than searching the files based on name of the files we have developed an application that searches based on the content of the files and most importantly indexes the contents of the file using Lucene package rather than the files itself. We made sure that simplicity is maintained for user. The semantics of local and remote search procedures are properly abstracted and user feels that he is getting the results from remote machine with the same semantics as that of a local query. As this is a distributed application it can be further enhanced to search on different subnets in an organization by incorporating the range of I.P Addresses. REFERENCES: [1] Wei Lun Huang,Tzao Lin Lee, Chiao Szu Liao,Dept. of Comput. Sci. & Inf. Eng., Nat. Taiwan Univ, Taipei, "Desktop search in the intranet with integrated desktop search engines"computer Systems Architecture Conference, ACSAC th Asia-Pacific Aug page(s): 1 4 2/security-special-report-who-sees-your-data.htm, Apr [3] Ferragina, P., and Grossi, R. Fast incremental text indexing. Proc. 6th ACM SIAM Symposium on Discrete Algorithms (SODA), pages , [4] Erik Hatcher, Otis Gospodnetic, Michael McCandless; Lucene In Action, Second Edition, Manning Publication. [5] Apache-Luceneweb site, [6] Google desktop web site, March [7] Beagle desktop search engine web site, Apr First Author 1.G.MohammadRafi Pursuing M.Tech(CSE), MADINA Engineering College, Kadapa, Kadapa(Dt), Andhra Pradesh. I am interest to research the Network & Wireless Network areas. Second Author 2. K.Sreenivasulu HOD Dept.Of CSE, Madina Engineering College,Kadapa,AP Third Author 3.K.Anjaneyulu Asst.Prof & HOD,Dept. of Computer Applications, Sri Sai Institute Of Technology &Science, Rayachoty,Kadapa,AP [2] Computerweekly web site, 006/04/25/