LUCENE - QUICK GUIDE LUCENE - OVERVIEW

Transcription

1 LUCENE - QUICK GUIDE Copyright tutorialspoint.com LUCENE - OVERVIEW Lucene is simple yet powerful java based search library. It can be used in any application to add search capability to it. Lucene is open-source project. It is scalable and high-performance library used to index and search virtually any kind of text. Lucene library provides the core operations which are required by any search application. Indexing and Searching. How Search Application works? Any search application does the few or all of the following operations. Step Title Description 1 Acquire Raw Content First step of any search application is to collect the target contents on which search are to be conducted Build the document Analyze the document Indexing the document User Interface for Search Build Query Search Query Next step is to build the documents from the raw contents which search application can understands and interpret easily. Before indexing process to start, the document is to be analyzed as which part of the text is a candidate to be indexed. This process is called analyzing the document. Once documents are built and analyzed, next step is to index them so that this document can be retrived based on certain keys instead of whole contents of the document. Indexing process is similar to indexes in the end of a book where common words are shown with their page numbers so that these words can be tracked quickly instead of searching the complete book. Once a database of indexes is ready then application can make any search. To facilitate user to make a search, application must provide a user a mean or u0ser interface where a user can enter text and start the search process. Once user made a request to search a text, application should prepare a Query object using that text which can be used to inquire index database to get the relevant details. Using query object, index database is then checked to get the relevant details and the content documents. 8 Render Results Once result is received the application should decide how to show the results to the user using User Interface. How much information is to be shown at first look and so on. Apart from these basic operations, search application can also provide administration user interface providing administrators of the application to control the level of search based on the user profiles. Analytics of search result is another important and advanced aspect of any search application. Lucene's role in search application Lucene plays role in steps 2 to step 7 mentioned above and provides classes to do the required operations. In nutshell, lucene works as a heart of any search application and provides the vital operations pertaining to indexing and searching. Acquiring contents and displaying the results is left for the application part to handle. Let's start with first simple search application using lucene

2 search library in next chapter. Environment Setup LUCENE - ENVIRONMENT SETUP This tutorial will guide you on how to prepare a development environment to start your work with Spring Framework. This tutorial will also teach you how to setup JDK, Tomcat and Eclipse on your machine before you setup Spring Framework: Step 1 - Setup Java Development Kit JDK: You can download the latest version of SDK from Oracle's Java site: Java SE Downloads. You will find instructions for installing JDK in downloaded files, follow the given instructions to install and configure the setup. Finally set PATH and JAVA_HOME environment variables to refer to the directory that contains java and javac, typically java_install_dir/bin and java_install_dir respectively. If you are running Windows and installed the JDK in C:\jdk1.6.0_15, you would have to put the following line in your C:\autoexec.bat file. set PATH=C:\jdk1.6.0_15\bin;%PATH% set JAVA_HOME=C:\jdk1.6.0_15 Alternatively, on Windows NT/2000/XP, you could also right-click on My Computer, select Properties, then Advanced, then Environment Variables. Then, you would update the PATH value and press the OK button. On Unix Solaris, Linux, etc., if the SDK is installed in /usr/local/jdk1.6.0_15 and you use the C shell, you would put the following into your.cshrc file. setenv PATH /usr/local/jdk1.6.0_15/bin:$path setenv JAVA_HOME /usr/local/jdk1.6.0_15 Alternatively, if you use an Integrated Development Environment IDE like Borland JBuilder, Eclipse, IntelliJ IDEA, or Sun ONE Studio, compile and run a simple program to confirm that the IDE knows where you installed Java, otherwise do proper setup as given document of the IDE. Step 2 - Setup Eclipse IDE All the examples in this tutorial have been written using Eclipse IDE. So I would suggest you should have latest version of Eclipse installed on your machine. To install Eclipse IDE, download the latest Eclipse binaries from Once you downloaded the installation, unpack the binary distribution into a convenient location. For example in C:\eclipse on windows, or /usr/local/eclipse on Linux/Unix and finally set PATH variable appropriately. Eclipse can be started by executing the following commands on windows machine, or you can simply double click on eclipse.exe %C:\eclipse\eclipse.exe Eclipse can be started by executing the following commands on Unix Solaris, Linux, etc. machine: $/usr/local/eclipse/eclipse After a successful startup, if everything is fine then it should display following result:

3 Step 3 - Setup Lucene Framework Libraries Now if everything is fine, then you can proceed to setup your Lucene framework. Following are the simple steps to download and install the framework on your machine. Make a choice whether you want to install Lucene on Windows, or Unix and then proceed to the next step to download.zip file for windows and.tz file for Unix. Download the suitable version of Lucene framework binaries from At the time of writing this tutorial, I downloaded lucene zip on my Windows machine and when you unzip the downloaded file it will give you directory structure inside C:\lucene as follows.

4 You will find all the Lucene libraries in the directory C:\lucene Make sure you set your CLASSPATH variable on this directory properly otherwise you will face problem while running your application. If you are using Eclipse then it is not required to set CLASSPATH because all the setting will be done through Eclipse. Once you are done with this last step, you are ready to proceed for your first Lucene Example which you will see in the next chapter. LUCENE - FIRST APPLICATION Let us start actual programming with Lucene Framework. Before you start writing your first example using Lucene framework, you have to make sure that you have setup your Lucene environment properly as explained in Lucene - Environment Setup tutorial. I also assume that you have a little bit working knowledge with Eclipse IDE. So let us proceed to write a simple Search Application which will print number of search result found. We'll also see the list of indexes created during this process. Step 1 - Create Java Project: The first step is to create a simple Java Project using Eclipse IDE. Follow the option File -> New -> Project and finally select Java Project wizard from the wizard list. Now name your project as LuceneFirstApplication using the wizard window as follows:

5 Once your project is created successfully, you will have following content in your Project Explorer: Step 2 - Add Required Libraries: As a second step let us add Lucene core Framework library in our project. To do this, right click on your project name LuceneFirstApplication and then follow the following option available in context menu: Build Path -> Configure Build Path to display the Java Build Path window as follows:

6 Now use Add External JARs button available under Libraries tab to add the following core JAR from Lucene installation directory: lucene-core Step 3 - Create Source Files: Now let us create actual source files under the LuceneFirstApplication project. First we need to create a package called com.tutorialspoint.lucene. To do this, right click on src in package explorer section and follow the option : New -> Package. Next we will create LuceneTester.java and other java classes under the com.tutorialspoint.lucene package. LuceneConstants.java This class is used to provide various constants to be used across the sample application. package com.tutorialspoint.lucene; public class LuceneConstants { public static final String CONTENTS="contents"; public static final String FILE_NAME="filename"; public static final String FILE_PATH="filepath"; public static final int MAX_SEARCH = 10; TextFileFilter.java This class is used as a.txt file filter. package com.tutorialspoint.lucene; import java.io.file; import java.io.filefilter; public class TextFileFilter implements FileFilter public boolean accept(file pathname) { return pathname.getname().tolowercase().endswith(".txt");

7 Indexer.java This class is used to index the raw data so that we can make it searchable using lucene library. package com.tutorialspoint.lucene; import java.io.file; import java.io.filefilter; import java.io.filereader; import java.io.ioexception; import org.apache.lucene.analysis.standard.standardanalyzer; import org.apache.lucene.document.document; import org.apache.lucene.document.field; import org.apache.lucene.index.corruptindexexception; import org.apache.lucene.index.indexwriter; import org.apache.lucene.store.directory; import org.apache.lucene.store.fsdirectory; import org.apache.lucene.util.version; public class Indexer { private IndexWriter writer; public Indexer(String indexdirectorypath) throws IOException{ //this directory will contain the indexes Directory indexdirectory = FSDirectory.open(new File(indexDirectoryPath)); //create the indexer writer = new IndexWriter(indexDirectory, new StandardAnalyzer(Version.LUCENE_36),true, IndexWriter.MaxFieldLength.UNLIMITED); public void close() throws CorruptIndexException, IOException{ writer.close(); private Document getdocument(file file) throws IOException{ Document document = new Document(); //index file contents Field contentfield = new Field(LuceneConstants.CONTENTS, new FileReader(file)); //index file name Field filenamefield = new Field(LuceneConstants.FILE_NAME, file.getname(), Field.Store.YES,Field.Index.NOT_ANALYZED); //index file path Field filepathfield = new Field(LuceneConstants.FILE_PATH, file.getcanonicalpath(), Field.Store.YES,Field.Index.NOT_ANALYZED); document.add(contentfield); document.add(filenamefield); document.add(filepathfield); return document; private void indexfile(file file) throws IOException{ System.out.println("Indexing "+file.getcanonicalpath()); Document document = getdocument(file); writer.adddocument(document); public int createindex(string datadirpath, FileFilter filter) throws IOException{ //get all files in the data directory File[] files = new File(dataDirPath).listFiles();

8 for (File file : files) { if(!file.isdirectory() &&!file.ishidden() && file.exists() && file.canread() && filter.accept(file) ){ indexfile(file); return writer.numdocs(); Searcher.java This class is used to search the indexes created by Indexer to search the requested contents. package com.tutorialspoint.lucene; import java.io.file; import java.io.ioexception; import org.apache.lucene.analysis.standard.standardanalyzer; import org.apache.lucene.document.document; import org.apache.lucene.index.corruptindexexception; import org.apache.lucene.queryparser.parseexception; import org.apache.lucene.queryparser.queryparser; import org.apache.lucene.search.indexsearcher; import org.apache.lucene.search.query; import org.apache.lucene.search.scoredoc; import org.apache.lucene.search.topdocs; import org.apache.lucene.store.directory; import org.apache.lucene.store.fsdirectory; import org.apache.lucene.util.version; public class Searcher { IndexSearcher indexsearcher; QueryParser queryparser; Query query; public Searcher(String indexdirectorypath) throws IOException{ Directory indexdirectory = FSDirectory.open(new File(indexDirectoryPath)); indexsearcher = new IndexSearcher(indexDirectory); queryparser = new QueryParser(Version.LUCENE_36, LuceneConstants.CONTENTS, new StandardAnalyzer(Version.LUCENE_36)); public TopDocs search( String searchquery) throws IOException, ParseException{ query = queryparser.parse(searchquery); return indexsearcher.search(query, LuceneConstants.MAX_SEARCH); public Document getdocument(scoredoc scoredoc) throws CorruptIndexException, IOException{ return indexsearcher.doc(scoredoc.doc); public void close() throws IOException{ indexsearcher.close(); LuceneTester.java

9 This class is used to test the indexing and search capability of lucene library. package com.tutorialspoint.lucene; import java.io.ioexception; import org.apache.lucene.document.document; import org.apache.lucene.queryparser.parseexception; import org.apache.lucene.search.scoredoc; import org.apache.lucene.search.topdocs; public class LuceneTester { String indexdir = "E:\\Lucene\\Index"; String datadir = "E:\\Lucene\\Data"; Indexer indexer; Searcher searcher; public static void main(string[] args) { LuceneTester tester; try { tester = new LuceneTester(); tester.createindex(); tester.search("mohan"); catch (IOException e) { e.printstacktrace(); catch (ParseException e) { e.printstacktrace(); private void createindex() throws IOException{ indexer = new Indexer(indexDir); int numindexed; long starttime = System.currentTimeMillis(); numindexed = indexer.createindex(datadir, new TextFileFilter()); long endtime = System.currentTimeMillis(); indexer.close(); System.out.println(numIndexed+" File indexed, time taken: " +(endtime-starttime)+" ms"); private void search(string searchquery) throws IOException, ParseException{ searcher = new Searcher(indexDir); long starttime = System.currentTimeMillis(); TopDocs hits = searcher.search(searchquery); long endtime = System.currentTimeMillis(); System.out.println(hits.totalHits + " documents found. Time :" + (endtime - starttime)); for(scoredoc scoredoc : hits.scoredocs) { Document doc = searcher.getdocument(scoredoc); System.out.println("File: " + doc.get(luceneconstants.file_path)); searcher.close(); Step 4 - Data & Index directory creation I've used 10 text files named from record1.txt to record10.txt containing simply names and other details of the students and put them in the directory E:\Lucene\Data. Test Data. An index directory path should be created as E:\Lucene\Index. After running this program, you can see the list of index files created in that folder. Step 5 - Running the Program: Once you are done with creating source, creating the raw data, data directory and index directory, you are ready for this step which is compiling and running your program. To do this, Keep

10 LuceneTester.Java file tab active and use either Run option available in the Eclipse IDE or use Ctrl + F11 to compile and run your LuceneTester application. If everything is fine with your application, this will print the following message in Eclipse IDE's console: Indexing E:\Lucene\Data\record1.txt Indexing E:\Lucene\Data\record10.txt Indexing E:\Lucene\Data\record2.txt Indexing E:\Lucene\Data\record3.txt Indexing E:\Lucene\Data\record4.txt Indexing E:\Lucene\Data\record5.txt Indexing E:\Lucene\Data\record6.txt Indexing E:\Lucene\Data\record7.txt Indexing E:\Lucene\Data\record8.txt Indexing E:\Lucene\Data\record9.txt 10 File indexed, time taken: 109 ms 1 documents found. Time :0 File: E:\Lucene\Data\record4.txt Once you've run the program successfully, you will have following content in your index directory: LUCENE - INDEXING CLASSES Indexing process is one of the core functionality provided by Lucene. Following diagram illustrates the indexing process and use of classes. IndexWriter is the most important and core component of the indexing process. We add Documents containing Fields to IndexWriter which analyzes the Documents using the Analyzer and then creates/open/edit indexes as required and store/update them in a Directory. IndexWriter is used to update or create indexes. It is not used to read indexes. Indexing Classes: Following is the list of commonly used classes during indexing process. Sr. Class & Description

11 No. 1 IndexWriter This class acts as a core component which creates/updates indexes during indexing process. 2 Directory This class represents the storage location of the indexes. 3 Analyzer Analyzer class is responsible to analyze a document and get the tokens/words from the text which is to be indexed. Without analysis done, IndexWriter can not create index. 4 Document Document represents a virtual document with Fields where Field is object which can contain the physical document's contents, its meta data and so on. Analyzer can understand a Document only. 5 Field Field is the lowest unit or the starting point of the indexing process. It represents the key value pair relationship where a key is used to identify the value to be indexed. Say a field used to represent contents of a document will have key as "contents" and the value may contain the part or all of the text or numeric content of the document. Lucene can index only text or numeric contents only. Introduction LUCENE - INDEXWRITER This class acts as a core component which creates/updates indexes during indexing process. Class declaration Following is the declaration for org.apache.lucene.index.indexwriter class: public class IndexWriter extends Object implements Closeable, TwoPhaseCommit Field Following are the fields for org.apache.lucene.index.indexwriter class: static int DEFAULT_MAX_BUFFERED_DELETE_TERMS -- Deprecated. use IndexWriterConfig.DEFAULT_MAX_BUFFERED_DELETE_TERMS instead. static int DEFAULT_MAX_BUFFERED_DOCS -- Deprecated. Use IndexWriterConfig.DEFAULT_MAX_BUFFERED_DOCS instead. static int DEFAULT_MAX_FIELD_LENGTH -- Deprecated. See IndexWriterConfig. static double DEFAULT_RAM_BUFFER_SIZE_MB -- Deprecated. Use IndexWriterConfig.DEFAULT_RAM_BUFFER_SIZE_MB instead. static int DEFAULT_TERM_INDEX_INTERVAL -- Deprecated. Use IndexWriterConfig.DEFAULT_TERM_INDEX_INTERVAL instead.

12 static int DISABLE_AUTO_FLUSH -- Deprecated. Use IndexWriterConfig.DISABLE_AUTO_FLUSH instead. static int MAX_TERM_LENGTH -- Absolute hard maximum length for a term. static String WRITE_LOCK_NAME -- Name of the write lock in the index. static long WRITE_LOCK_TIMEOUT -- Deprecated. Use IndexWriterConfig.WRITE_LOCK_TIMEOUT instead. Class constructors S.N. 1 Constructor & Description IndexWriter Directoryd, Analyzera, booleancreate, IndexDeletionPolicydeletionPolicy, IndexWriter. MaxFieldLengthmfl Deprecated. use IndexWriterDirectory, IndexWriterConfig instead. 2 IndexWriterDirectoryd, Analyzera, booleancreate, IndexWriter. MaxFieldLengthmfl Deprecated. use IndexWriterDirectory, IndexWriterConfig instead. 3 IndexWriterDirectoryd, Analyzera, IndexDeletionPolicydeletionPolicy, IndexWriter. MaxFieldLengthmfl Deprecated. use IndexWriterDirectory, IndexWriterConfig instead. 4 IndexWriter Directoryd, Analyzera, IndexDeletionPolicydeletionPolicy, IndexWriter. MaxFieldLengthmfl, IndexCommitcommit Deprecated. use IndexWriterDirectory, IndexWriterConfig instead. 5 IndexWriterDirectoryd, Analyzera, IndexWriter. MaxFieldLengthmfl Deprecated. use IndexWriterDirectory, IndexWriterConfig instead. 6 IndexWriterDirectoryd, IndexWriterConfigconf Constructs a new IndexWriter per the settings given in conf. Class methods S.N. 1 Method & Description void adddocumentdocumentdoc Adds a document to this index. 2 void adddocumentdocumentdoc, Analyzeranalyzer Adds a document to this index, using the provided analyzer instead of the value of getanalyzer.

13 3 void adddocumentscollection < Document > docs Atomically adds a block of documents with sequentially assigned document IDs, such that an external reader will see all or none of the documents. 4 void adddocumentscollection < Document > docs, Analyzeranalyzer Atomically adds a block of documents, analyzed using the provided analyzer, with sequentially assigned document IDs, such that an external reader will see all or none of the documents. 5 void addindexesdirectory... dirs Adds all segments from an array of indexes into this index. 6 void addindexesindexreader... readers Merges the provided indexes into this index. 7 void addindexesnooptimizedirectory... dirs Deprecated.use addindexesdirectory... instead. 8 void close Commits all changes to an index and closes all associated files. 9 void closebooleanwaitformerges Closes the index with or without waiting for currently running merges to finish. 10 void commit Commits all pending changes added & deleted documents, segment merges, added indexes, etc. to the index, and syncs all referenced index files, such that a reader will see the changes and the index updates will survive an OS or machine crash or power loss. 11 void commitmap < String, String > commituserdata Commits all changes to the index, specifying a commituserdata Map String > String. 12 void deleteall Delete all documents in the index. 13 void deletedocumentsquery... queries Deletes the documents matching any of the provided queries. 14

14 14 void deletedocumentsqueryquery Deletes the documents matching the provided query. 15 void deletedocumentsterm... terms Deletes the documents containing any of the terms. 16 void deletedocumentstermterm Deletes the documents containing term. 17 void deleteunusedfiles Expert: remove any index files that are no longer used. 18 protected void doafterflush A hook for extending classes to execute operations after pending added and deleted documents have been flushed to the Directory but before the change is committed newsegments N filewritten. 19 protected void dobeforeflush A hook for extending classes to execute operations before pending added and deleted documents are flushed to the Directory. 20 protected void ensureopen 21 protected void ensureopenbooleanincludependingclose Used internally to throw an AlreadyClosedException if this IndexWriter has been closed. 22 void expungedeletes Deprecated. 23 void expungedeletesbooleandowait Deprecated. 24 protected void flushbooleantriggermerge, booleanapplyalldeletes Flush all in-memory buffered updates addsanddeletes to the Directory. 25 protected void flushbooleantriggermerge, booleanflushdocstores, booleanflushdeletes NOTE: flushdocstores is ignored now hardwiredtotrue; this method is only here for backwards compatibility.

15 26 void forcemergeintmaxnumsegments Forces merge policy to merge segments until there's <= maxnumsegments. 27 void forcemergeintmaxnumsegments, booleandowait Just like forcemergeint, except you can specify whether the call should block until all merging completes. 28 void forcemergedeletes Forces merging of all segments that have deleted documents. 29 void forcemergedeletesbooleandowait Just like forcemergedeletes, except you can specify whether the call should block until the operation completes. 30 Analyzer getanalyzer Returns the analyzer used by this index. 31 IndexWriterConfig getconfig Returns the private IndexWriterConfig, cloned from the IndexWriterConfig passed to IndexWriterDirectory, IndexWriterConfig. 32 static PrintStream getdefaultinfostream Returns the current default infostream for newly instantiated IndexWriters. 33 static long getdefaultwritelocktimeout Deprecated.use IndexWriterConfig.getDefaultWriteLockTimeout instead 34 Directory getdirectory Returns the Directory used by this index. 35 PrintStream getinfostream Returns the current infostream in use by this writer. 36 int getmaxbuffereddeleteterms Deprecated.use IndexWriterConfig.getMaxBufferedDeleteTerms instead. 37 int getmaxbuffereddocs Deprecated.use IndexWriterConfig.getMaxBufferedDocs instead.

16 38 int getmaxfieldlength Deprecated.use LimitTokenCountAnalyzer to limit number of tokens. 39 int getmaxmergedocs Deprecated.use LogMergePolicy.getMaxMergeDocs directly. 40 IndexWriter.IndexReaderWarmer getmergedsegmentwarmer Deprecated.use IndexWriterConfig.getMergedSegmentWarmer instead. 41 int getmergefactor Deprecated.use LogMergePolicy.getMergeFactor directly. 42 MergePolicy getmergepolicy Deprecated.use IndexWriterConfig.getMergePolicy instead. 43 MergeScheduler getmergescheduler Deprecated.use IndexWriterConfig.getMergeScheduler instead. 44 Collection<SegmentInfo> getmergingsegments Expert: to be used by a MergePolicy to a void selecting merges for segments already being merged. 45 MergePolicy.OneMerge getnextmerge Expert: the MergeScheduler calls this method to retrieve the next merge requested by the MergePolicy. 46 PayloadProcessorProvider getpayloadprocessorprovider Returns the PayloadProcessorProvider that is used during segment merges to process payloads. 47 double getrambuffersizemb Deprecated.use IndexWriterConfig.getRAMBufferSizeMB instead. 48 IndexReader getreader Deprecated.Please use IndexReader.openIndexWriter, boolean instead. 49 IndexReader getreaderintterminfosindexdivisor

17 Deprecated.Please use IndexReader.openIndexWriter, boolean instead. Furthermore, this method cannot guarantee the reader anditssub readers will be opened with the terminfosindexdivisor setting because some of them may have already been opened according to IndexWriterConfig.setReaderTermsIndexDivisorint. You should set the requested terminfosindexdivisor through IndexWriterConfig.setReaderTermsIndexDivisor int and use getreader. 50 int getreadertermsindexdivisor Deprecated.use IndexWriterConfig.getReaderTermsIndexDivisor instead. 51 Similarity getsimilarity Deprecated.use IndexWriterConfig.getSimilarity instead. 52 int gettermindexinterval Deprecated.use IndexWriterConfig.getTermIndexInterval. 53 boolean getusecompoundfile Deprecated.use LogMergePolicy.getUseCompoundFile. 54 long getwritelocktimeout Deprecated.use IndexWriterConfig.getWriteLockTimeout. 55 boolean hasdeletions 56 static boolean islockeddirectorydirectory Returns true iff the index in the named directory is currently locked. 57 int maxdoc Returns total number of docs in this index, including docs not yet flushed stillintherambuffer, not counting deletions. 58 void maybemerge Expert: asks the mergepolicy whether any merges are necessary now and if so, runs the requested merges and then iterate testagainifmergesareneeded until no more merges are returned by the mergepolicy. 59 void mergemergepolicy. OneMergemerge Merges the indicated segments, replacing them in the stack with a single segment. 60 void messagestringmessage Prints a message to the infostream ifnon null, prefixed with the identifying information for

18 this writer and the thread that's calling it. 61 int numdeleteddocssegmentinfoinfo Obtain the number of deleted docs for a pooled reader. 62 int numdocs Returns total number of docs in this index, including docs not yet flushed stillintherambuffer, and including deletions. 63 int numramdocs Expert: Return the number of documents currently buffered in RAM. 64 void optimize Deprecated. 65 void optimizebooleandowait Deprecated. 66 void optimizeintmaxnumsegments Deprecated. 67 void preparecommit Expert: prepare for commit. 68 void preparecommitmap < String, String > commituserdata Expert: prepare for commit, specifying commituserdata Map String > String. 69 long ramsizeinbytes Expert: Return the total size of all index files currently cached in memory. 70 void rollback Close the IndexWriter without committing any changes that have occurred since the last commit orsinceitwasopened, ifcommithasn tbeencalled. 71 String segstring 72 String segstringiterable < SegmentInfo > infos

19 73 String segstringsegmentinfoinfo 74 static void setdefaultinfostreamprintstreaminfostream If non-null, this will be the default infostream used by a newly instantiated IndexWriter. 75 static void setdefaultwritelocktimeoutlongwritelocktimeout Deprecated.use IndexWriterConfig.setDefaultWriteLockTimeoutlong instead 76 void setinfostreamprintstreaminfostream If non-null, information about merges, deletes and a message when maxfieldlength is reached will be printed to this. 77 void setmaxbuffereddeletetermsintmaxbuffereddeleteterms Deprecated.use IndexWriterConfig.setMaxBufferedDeleteTermsint instead. 78 void setmaxbuffereddocsintmaxbuffereddocs Deprecated.use IndexWriterConfig.setMaxBufferedDocsint instead. 79 void setmaxfieldlengthintmaxfieldlength Deprecated.use LimitTokenCountAnalyzer instead. Note that the behvaior slightly changed - the analyzer limits the number of tokens per token stream created, while this setting limits the total number of tokens to index. This only matters if you index many multi-valued fields though. 80 void setmaxmergedocsintmaxmergedocs Deprecated.use LogMergePolicy.setMaxMergeDocsint directly. 81 void setmergedsegmentwarmerindexwriter. IndexReaderWarmerwarmer Deprecated.use IndexWriterConfig.setMergedSegmentWarmer org. apache. lucene. index. IndexWriter. IndexReaderWarmer instead. 82 void setmergefactorintmergefactor Deprecated.use LogMergePolicy.setMergeFactorint directly. 83 void setmergepolicymergepolicymp Deprecated.use IndexWriterConfig.setMergePolicyMergePolicy instead. 84 void setmergeschedulermergeschedulermergescheduler Deprecated.use IndexWriterConfig.setMergeSchedulerMergeScheduler instead

20 85 void setpayloadprocessorproviderpayloadprocessorproviderpcp Sets the PayloadProcessorProvider to use when merging payloads. 86 void setrambuffersizembdoublemb Deprecated.use IndexWriterConfig.setRAMBufferSizeMBdouble instead. 87 void setreadertermsindexdivisorintdivisor Deprecated.use IndexWriterConfig.setReaderTermsIndexDivisorint instead. 88 void setsimilaritysimilaritysimilarity Deprecated.use IndexWriterConfig.setSimilaritySimilarity instead 89 void settermindexintervalintinterval Deprecated.use IndexWriterConfig.setTermIndexIntervalint 90 void setusecompoundfilebooleanvalue Deprecated.use LogMergePolicy.setUseCompoundFileboolean. 91 void setwritelocktimeoutlongwritelocktimeout Deprecated.use IndexWriterConfig.setWriteLockTimeoutlong instead 92 static void unlockdirectorydirectory Forcibly unlocks the index in the named directory. 93 void updatedocumenttermterm, Documentdoc Updates a document by first deleting the documents containing term and then adding the new document. 94 void updatedocumenttermterm, Documentdoc, Analyzeranalyzer Updates a document by first deleting the documents containing term and then adding the new document. 95 void updatedocumentstermdelterm, Collection < Document > docs Atomically deletes documents matching the provided delterm and adds a block of documents with sequentially assigned document IDs, such that an external reader will see all or none of the documents. 96

21 96 void updatedocumentstermdelterm, Collection < Document > docs, Analyzeranalyzer Atomically deletes documents matching the provided delterm and adds a block of documents, analyzed using the provided analyzer, with sequentially assigned document IDs, such that an external reader will see all or none of the documents. 97 boolean verbose Returns true if verbosing is enabled (i.e., infostream! 98 void waitformerges Wait for any currently outstanding merges to finish. Methods inherited This class inherits methods from the following classes: java.lang.object Introduction LUCENE - DIRECTORY This class represents the storage location of the indexes and generally it is a list of files. These files are called index files. Index files are normally created once and then used for read operation or can be deleted. Class declaration Following is the declaration for org.apache.lucene.store.directory class: public abstract class Directory extends Object implements Closeable Field Following are the fields for org.apache.lucene.store.directory class: protected boolean isopen protected LockFactory lockfactory -- Holds the LockFactory instance implementslockingforthisdirectoryinstance. Class constructors S.N. 1 Constructor & Description Directory Class methods S.N. Method & Description 1

22 1 void clearlockstringname Attempt to clear forcefullyunlockandremove the specified lock. 2 abstract void close Closes the store. 3 static void copydirectorysrc, Directorydest, booleanclosedirsrc Deprecated. Should be replaced with calls to copydirectory, String, String for every file that needs copying. You can use the following code: IndexFileNameFilter filter = IndexFileNameFilter.getFilter(); for (String file : src.listall()) { if (filter.accept(null, file)) { src.copy(dest, file, file); 4 void copydirectoryto, Stringsrc, Stringdest Copies the file src to Directory to under the new file name dest. 5 abstract IndexOutput createoutputstringname Creates a new, empty file in the directory with the given name. 6 abstract void deletefilestringname Removes an existing file in the directory. 7 protected void ensureopen 8 abstract boolean fileexistsstringname Returns true iff a file with the given name exists. 9 abstract long filelengthstringname Returns the length of a file in the directory. 10 abstract long filemodifiedstringname Deprecated. 11 LockFactory getlockfactory Get the LockFactory that this Directory instance is using for its locking implementation. 12

23 12 String getlockid Return a string identifier that uniquely differentiates this Directory instance from other Directory instances. 13 abstract String[] listall Returns an array of strings, one for each file in the directory. 14 Lock makelockstringname Construct a Lock. 15 abstract IndexInput openinputstringname Returns a stream reading an existing file. 16 IndexInput openinputstringname, intbuffersize Returns a stream reading an existing file, with the specified read buffer size. 17 void setlockfactorylockfactorylockfactory Set the LockFactory that this Directory instance should use for its locking implementation. 18 void synccollection < String > names Ensure that any writes to these files are moved to stable storage. 19 void syncstringname Deprecated. use synccollection instead. For easy migration you can change your code to call synccollections. singleton(name) 20 String tostring 21 abstract void touchfilestringname Deprecated. Lucene never uses this API; it will be removed in 4.0. Methods inherited This class inherits methods from the following classes: java.lang.object Introduction LUCENE - ANALYZER

24 Analyzer class is responsible to analyze a document and get the tokens/words from the text which is to be indexed. Without analysis done, IndexWriter can not create index. Class declaration Following is the declaration for org.apache.lucene.analysis.analyzer class: public abstract class Analyzer extends Object implements Closeable Class constructors S.N. 1 Constructor & Description protected Analyzer Class methods S.N. 1 Method & Description void close Frees persistent resources used by this Analyzer. 2 int getoffsetgapfieldablefield Just like getpositionincrementgapjava. lang. String, except for Token offsets instead. 3 int getpositionincrementgapstringfieldname Invoked before indexing a Fieldable instance if terms have already been added to that field. 4 protected Object getprevioustokenstream Used by Analyzers that implement reusabletokenstream to retrieve previously saved TokenStreams for re-use by the same thread. 5 TokenStream reusabletokenstreamstringfieldname, Readerreader Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method. 6 protected void setprevioustokenstreamobjectobj Used by Analyzers that implement reusabletokenstream to save a TokenStream for later re-use by the same thread. 7 abstract TokenStream tokenstreamstringfieldname, Readerreader Creates a TokenStream which tokenizes all the text in the provided Reader.

25 Methods inherited This class inherits methods from the following classes: java.lang.object Introduction LUCENE - DOCUMENT Document represents a virtual document with Fields where Field is object which can contain the physical document's contents, its meta data and so on. Analyzer can understand a Document only. Class declaration Following is the declaration for org.apache.lucene.document.document class: public final class Document extends Object implements Serializable Class constructors S.N. 1 Constructor & Description Document Constructs a new document with no fields. Class methods S.N. 1 Method & Description void clearlockstringname Attempt to clear forcefullyunlockandremove the specified lock. 2 void addfieldablefield Adds a field to a document. 3 String getstringname Returns the string value of the field with the given name if any exist in this document, or null. 4 byte[] getbinaryvaluestringname Returns an array of bytes for the first oronly field that has the name specified as the method parameter. 5

26 5 byte[][] getbinaryvaluesstringname Returns an array of byte arrays for of the fields that have the name specified as the method parameter. 6 float getboost Returns, at indexing time, the boost factor as set by setboostfloat. 7 Field getfieldstringname Deprecated. Use getfieldablejava. lang. String instead and cast depending on data type. 8 Fieldable getfieldablestringname Returns a field with the given name if any exist in this document, or null. 9 Fieldable[] getfieldablesstringname Returns an array of Fieldables with the given name. 10 List<Fieldable> getfields Returns a List of all the fields in a document. 11 Field[] getfieldsstringname Deprecated. Use getfieldablejava. lang. String instead and cast depending on data type. 12 String[] getvaluesstringname Returns an array of values of the field specified as the method parameter. 13 void removefieldstringname Removes field with the specified name from the document. 14 void removefieldsstringname Removes all fields with the given name from the document. 15 void setboostfloatboost Sets a boost factor for hits on any field of this document. 16 String tostring Prints the fields of a document for human consumption.

27 Methods inherited This class inherits methods from the following classes: java.lang.object Introduction LUCENE - FIELD Field is the lowest unit or the starting point of the indexing process. It represents the key value pair relationship where a key is used to identify the value to be indexed. Say a field used to represent contents of a document will have key as "contents" and the value may contain the part or all of the text or numeric content of the document. Lucene can index only text or numeric contents only.this class represents the storage location of the indexes and generally it is a list of files. These files are called index files. Index files are normally created once and then used for read operation or can be deleted. Class declaration Following is the declaration for org.apache.lucene.document.field class: public final class Field extends AbstractField implements Fieldable, Serializable Class constructors S.N. 1 Constructor & Description Field Stringname, booleaninternname, Stringvalue, Field. Storestore, Field. Indexindex, Field. TermVectortermVector Create a field by specifying its name, value and how it will be saved in the index. 2 FieldStringname, byte[]value Create a stored field with binary value. 3 FieldStringname, byte[]value, Field. Storestore Deprecated. 4 FieldStringname, byte[]value, intoffset, intlength Create a stored field with binary value. 5 FieldStringname, byte[]value, intoffset, intlength, Field. Storestore Deprecated. 6 FieldStringname, Readerreader Create a tokenized and indexed field that is not stored.

28 7 FieldStringname, Readerreader, Field. TermVectortermVector Create a tokenized and indexed field that is not stored, optionally with storing term vectors. 8 FieldStringname, Stringvalue, Field. Storestore, Field. Indexindex Create a field by specifying its name, value and how it will be saved in the index. 9 FieldStringname, Stringvalue, Field. Storestore, Field. Indexindex, Field. TermVectortermVector Create a field by specifying its name, value and how it will be saved in the index. 10 FieldStringname, TokenStreamtokenStream Create a tokenized and indexed field that is not stored. 11 FieldStringname, TokenStreamtokenStream, Field. TermVectortermVector Create a tokenized and indexed field that is not stored, optionally with storing term vectors. Class methods S.N. 1 Method & Description void clearlockstringname Attempt to clear forcefullyunlockandremove the specified lock. 2 Reader readervalue The value of the field as a Reader, or null. 3 void settokenstreamtokenstreamtokenstream Expert: sets the token stream to be used for indexing and causes isindexed and istokenized to return true. 4 void setvaluebyte[]value Expert: change the value of this field. 5 void setvaluebyte[]value, intoffset, intlength Expert: change the value of this field. 6 void setvaluereadervalue Expert: change the value of this field.

29 7 void setvaluestringvalue Expert: change the value of this field. 8 String stringvalue The value of the field as a String, or null. 9 TokenStream tokenstreamvalue The TokesStream for this field to be used when indexing, or null. Methods inherited This class inherits methods from the following classes: org.apache.lucene.document.abstractfield java.lang.object LUCENE - SEARCHING CLASSES Searching process is again one of the core functionality provided by Lucene. It's flow is similar to that of indexing process. Basic search of lucene can be made using following classes which can also be termed as foundation classes for all search related operations. Searching Classes: Following is the list of commonly used classes during searching process. Sr. No. Class & Description 1 IndexSearcher This class act as a core component which reads/searches indexes created after indexing process. It takes directory instance pointing to the location containing the indexes. 2 Term This class is the lowest unit of searching. It is similar to Field in indexing process. 3 Query Query is an abstract class and contains various utility methods and is the parent of all types of queries that lucene uses during search process. 4 TermQuery TermQuery is the most commonly used query object and is the foundation of many complex queries that lucene can make use of. 5 TopDocs TopDocs points to the top N search results which matches the search criteria. It is simple container of pointers to point to documents which are output of search result.

30 LUCENE - INDEXSEARCHER Introduction This class acts as a core component which reads/searches indexes during searching process. Class declaration Following is the declaration for org.apache.lucene.search.indexsearcher class: public class IndexSearcher extends Searcher Field Following are the fields for org.apache.lucene.index.indexwriter class: protected int[] docstarts protected IndexReader[] subreaders protected IndexSearcher[] subsearchers Class constructors S.N. 1 Constructor & Description IndexSearcherDirectorypath Deprecated. Use IndexSearcherIndexReader instead. 2 IndexSearcherDirectorypath, booleanreadonly Deprecated. Use IndexSearcherIndexReader instead. 3 IndexSearcherIndexReaderr Creates a searcher searching the provided index. 4 IndexSearcherIndexReaderr, ExecutorServiceexecutor Runs searches for each segment separately, using the provided ExecutorService. 5 IndexSearcherIndexReaderreader, IndexReader[]subReaders, int[]docstarts Expert: directly specify the reader, subreaders and their docid starts. 6 IndexSearcherIndexReaderreader, IndexReader[]subReaders, int[]docstarts, ExecutorServiceexecutor Expert: directly specify the reader, subreaders and their docid starts, and an ExecutorService. Class methods

31 S.N. 1 Method & Description void close Note that the underlying IndexReader is not closed, if IndexSearcher was constructed with IndexSearcherIndexReaderr. 2 Weight createnormalizedweightqueryquery Creates a normalized weight for a top-level Query. 3 Document docintdocid Returns the stored fields of document i. 4 Document docintdocid, FieldSelectorfieldSelector Get the Document at the nth position. 5 int docfreqtermterm Returns total docfreq for this term. 6 Explanation explainqueryquery, intdoc Returns an Explanation that describes how doc scored against query. 7 Explanation explainweightweight, intdoc Expert: low-level implementation method Returns an Explanation that describes how doc scored against weight. 8 protected void gathersubreaders(list allsubreaders, IndexReader r) 9 IndexReader getindexreader Return the IndexReader this searches. 10 Similarity getsimilarity Expert: Return the Similarity implementation used by this Searcher. 11 IndexReader[] getsubreaders Returns the atomic subreaders used by this searcher. 12 int maxdoc

32 Expert: Returns one greater than the largest possible document number. 13 Query rewritequeryoriginal Expert: called to re-write queries into primitive queries. 14 void searchqueryquery, Collectorresults Lower-level search API. 15 void searchqueryquery, Filterfilter, Collectorresults Lower-level search API. 16 TopDocs searchqueryquery, Filterfilter, intn Finds the top n hits for query, applying filter if non-null. 17 TopFieldDocs searchqueryquery, Filterfilter, intn, Sortsort Search implementation with arbitrary sorting. 18 TopDocs searchqueryquery, intn Finds the top n hits for query. 19 TopFieldDocs searchqueryquery, intn, Sortsort Search implementation with arbitrary sorting and no filter. 20 void searchweightweight, Filterfilter, Collectorcollector Lower-level search API. 21 TopDocs searchweightweight, Filterfilter, intndocs Expert: Low-level search implementation. 22 TopFieldDocs searchweightweight, Filterfilter, intndocs, Sortsort Expert: Low-level search implementation with arbitrary sorting. 23 protected TopFieldDocs searchweightweight, Filterfilter, intndocs, Sortsort, booleanfillfields Just like searchweight, Filter, int, Sort, but you choose whether or not the fields in the returned FieldDoc instances should be set by specifying fillfields. 24 protected TopDocs searchweightweight, Filterfilter, ScoreDocafter, intndocs

33 Expert: Low-level search implementation. 25 TopDocs searchafterscoredocafter, Queryquery, Filterfilter, intn Finds the top n hits for query, applying filter if non-null, where all results are after a previous result after. 26 TopDocs searchafterscoredocafter, Queryquery, intn Finds the top n hits for query where all results are after a previous result after. 27 void setdefaultfieldsortscoringbooleandotrackscores, booleandomaxscore By default, no scores are computed when sorting by field usingsearch(query, Filter, int, Sort). 28 void setsimilaritysimilaritysimilarity Expert: Set the Similarity implementation used by this Searcher. 29 String tostring Methods inherited This class inherits methods from the following classes: org.apache.lucene.search.searcher java.lang.object Introduction LUCENE - TERM This class is the lowest unit of searching. It is similar to Field in indexing process. Class declaration Following is the declaration for org.apache.lucene.index.term class: public final class Term extends Object implements Comparable, Serializable Class constructors S.N. 1 Constructor & Description TermStringfld Constructs a Term with the given field and empty text. 2 TermStringfld, Stringtxt

34 Constructs a Term with the given field and text. Class methods S.N. 1 Method & Description void adddocumentdocumentdoc Adds a document to this index. 2 int comparetotermother Compares two terms, returning a negative integer if this term belongs before the argument, zero if this term is equal to the argument, and a positive integer if this term belongs after the argument. 3 Term createtermstringtext Optimized construction of new Terms by reusing same field as this Term - avoids field.intern overhead. 4 boolean equalsobjectobj 5 String field Returns the field of this term, an interned string. 6 int hashcode 7 String text Returns the text of this term. 8 String tostring Methods inherited This class inherits methods from the following classes: java.lang.object Introduction LUCENE - QUERY Query is an abstract class and contains various utility methods and is the parent of all types of queries that lucene uses during search process. Class declaration

35 Following is the declaration for org.apache.lucene.search.query class: public abstract class Query extends Object implements Serializable, Cloneable Class constructors S.N. 1 Constructor & Description Query Class methods S.N. 1 Method & Description Object clone Returns a clone of this query. 2 Query combinequery[]queries Expert: called when re-writing queries under MultiSearcher. 3 Weight createweightsearchersearcher Expert: Constructs an appropriate Weight implementation for this query. 4 boolean equalsobjectobj 5 void extracttermsset < Term > terms Expert: adds all terms occurring in this query to the terms set. 6 float getboost Gets the boost for this clause. 7 Similarity getsimilaritysearchersearcher Deprecated. Instead of using "runtime" subclassing/delegation, subclass the Weight instead. 8 int hashcode 9 static Query mergebooleanqueriesbooleanquery... queries Expert: merges the clauses of a set of BooleanQuery's into a single BooleanQuery.

36 10 Query rewriteindexreaderreader Expert: called to re-write queries into primitive queries. 11 void setboostfloatb Sets the boost for this query clause to b. 12 String tostring Prints a query to a string. 13 abstract String tostringstringfield Prints a query to a string, with field assumed to be the default field and omitted. 14 Weight weightsearchersearcher Deprecated. never ever use this method in Weight implementations. Subclasses of Query should use createweightorg. apache. lucene. search. Searcher, instead. Methods inherited This class inherits methods from the following classes: java.lang.object Introduction LUCENE - TERMQUERY TermQuery is the most commonly used query object and is the foundation of many complex queries that lucene can make use of. Class declaration Following is the declaration for org.apache.lucene.search.termquery class: public class TermQuery extends Query Class constructors S.N. 1 Constructor & Description TermQueryTermt Constructs a query for the term t. Class methods

37 S.N. 1 Method & Description void adddocumentdocumentdoc Adds a document to this index. 2 Weight createweightsearchersearcher Expert: Constructs an appropriate Weight implementation for this query. 3 boolean equalsobjecto Returns true iff o is equal to this. 4 void extracttermsset < Term > terms Expert: adds all terms occurring in this query to the terms set. 5 Term getterm Returns the term of this query. 6 int hashcode Returns a hash code value for this object. 7 String tostringstringfield Prints a user-readable version of this query. Methods inherited This class inherits methods from the following classes: org.apache.lucene.search.query java.lang.object Introduction LUCENE - TOPDOCS TopDocs points to the top N search results which matches the search criteria. It is simple container of pointers to point to documents which are output of search result. Class declaration Following is the declaration for org.apache.lucene.search.topdocs class: public class TopDocs extends Object implements Serializable

38 Field Following are the fields for org.apache.lucene.search.topdocs class: ScoreDoc[] scoredocs -- The top hits for the query. int totalhits -- The total number of hits for the query. Class constructors S.N. 1 Constructor & Description TopDocsinttotalHits, ScoreDoc[]scoreDocs, floatmaxscore Class methods S.N. 1 Method & Description getmaxscore Returns the maximum score value encountered. 2 static TopDocs mergesortsort, inttopn, TopDocs[]shardHits Returns a new TopDocs, containing topn results across the provided TopDocs, sorting by the specified Sort. 3 void setmaxscorefloatmaxscore Sets the maximum score value encountered. Methods inherited This class inherits methods from the following classes: java.lang.object LUCENE - INDEXING PROCESS Indexing process is one of the core functionality provided by Lucene. Following diagram illustrates the indexing process and use of classes. IndexWriter is the most important and core component of the indexing process.

39 We add Documents containing Fields to IndexWriter which analyzes the Documents using the Analyzer and then creates/open/edit indexes as required and store/update them in a Directory. IndexWriter is used to update or create indexes. It is not used to read indexes. Now we'll show you a step by step process to get a kick start in understanding of indexing process using a basic example. Create a document Create a method to get a lucene document from a text file. Create various types of fields which are key value pairs containing keys as names and values as contents to be indexed. Set field to be analyzed or not. In our case, only contents is to be analyzed as it can contain data such as a, am, are, an etc. which are not required in search operations. Add the newly created fields to the document object and return it to the caller method. private Document getdocument(file file) throws IOException{ Document document = new Document(); //index file contents Field contentfield = new Field(LuceneConstants.CONTENTS, new FileReader(file)); //index file name Field filenamefield = new Field(LuceneConstants.FILE_NAME, file.getname(), Field.Store.YES,Field.Index.NOT_ANALYZED); //index file path Field filepathfield = new Field(LuceneConstants.FILE_PATH, file.getcanonicalpath(), Field.Store.YES,Field.Index.NOT_ANALYZED); document.add(contentfield); document.add(filenamefield); document.add(filepathfield); return document; Create a IndexWriter IndexWriter class acts as a core component which creates/updates indexes during indexing process. Create object of IndexWriter. Create a lucene directory which should point to location where indexes are to be stored. Initialize the IndexWriter object created with the index directory, a standard analyzer having version information and other required/optional parameters. private IndexWriter writer; public Indexer(String indexdirectorypath) throws IOException{ //this directory will contain the indexes Directory indexdirectory = FSDirectory.open(new File(indexDirectoryPath)); //create the indexer

40 writer = new IndexWriter(indexDirectory, new StandardAnalyzer(Version.LUCENE_36),true, IndexWriter.MaxFieldLength.UNLIMITED); Start Indexing process private void indexfile(file file) throws IOException{ System.out.println("Indexing "+file.getcanonicalpath()); Document document = getdocument(file); writer.adddocument(document); Example Application Let us create a test Lucene application to test indexing process. Step Description 1 Create a project with a name LuceneFirstApplication under a package com.tutorialspoint.lucene as explained in the Lucene - First Application chapter. You can also use the project created in Lucene - First Application chapter as such for this chapter to understand indexing process. 2 Create LuceneConstants.java,TextFileFilter.java and Indexer.java as explained in the Lucene - First Application chapter. Keep rest of the files unchanged. 3 Create LuceneTester.java as mentioned below. 4 Clean and Build the application to make sure business logic is working as per the requirements. LuceneConstants.java This class is used to provide various constants to be used across the sample application. package com.tutorialspoint.lucene; public class LuceneConstants { public static final String CONTENTS="contents"; public static final String FILE_NAME="filename"; public static final String FILE_PATH="filepath"; public static final int MAX_SEARCH = 10; TextFileFilter.java This class is used as a.txt file filter. package com.tutorialspoint.lucene; import java.io.file; import java.io.filefilter; public class TextFileFilter implements FileFilter public boolean accept(file pathname) { return pathname.getname().tolowercase().endswith(".txt"); Indexer.java This class is used to index the raw data so that we can make it searchable using lucene library.

41 package com.tutorialspoint.lucene; import java.io.file; import java.io.filefilter; import java.io.filereader; import java.io.ioexception; import org.apache.lucene.analysis.standard.standardanalyzer; import org.apache.lucene.document.document; import org.apache.lucene.document.field; import org.apache.lucene.index.corruptindexexception; import org.apache.lucene.index.indexwriter; import org.apache.lucene.store.directory; import org.apache.lucene.store.fsdirectory; import org.apache.lucene.util.version; public class Indexer { private IndexWriter writer; public Indexer(String indexdirectorypath) throws IOException{ //this directory will contain the indexes Directory indexdirectory = FSDirectory.open(new File(indexDirectoryPath)); //create the indexer writer = new IndexWriter(indexDirectory, new StandardAnalyzer(Version.LUCENE_36),true, IndexWriter.MaxFieldLength.UNLIMITED); public void close() throws CorruptIndexException, IOException{ writer.close(); private Document getdocument(file file) throws IOException{ Document document = new Document(); //index file contents Field contentfield = new Field(LuceneConstants.CONTENTS, new FileReader(file)); //index file name Field filenamefield = new Field(LuceneConstants.FILE_NAME, file.getname(), Field.Store.YES,Field.Index.NOT_ANALYZED); //index file path Field filepathfield = new Field(LuceneConstants.FILE_PATH, file.getcanonicalpath(), Field.Store.YES,Field.Index.NOT_ANALYZED); document.add(contentfield); document.add(filenamefield); document.add(filepathfield); return document; private void indexfile(file file) throws IOException{ System.out.println("Indexing "+file.getcanonicalpath()); Document document = getdocument(file); writer.adddocument(document); public int createindex(string datadirpath, FileFilter filter) throws IOException{ //get all files in the data directory File[] files = new File(dataDirPath).listFiles(); for (File file : files) { if(!file.isdirectory() &&!file.ishidden() && file.exists()

42 && file.canread() && filter.accept(file) ){ indexfile(file); return writer.numdocs(); LuceneTester.java This class is used to test the indexing capability of lucene library. package com.tutorialspoint.lucene; import java.io.ioexception; public class LuceneTester { String indexdir = "E:\\Lucene\\Index"; String datadir = "E:\\Lucene\\Data"; Indexer indexer; public static void main(string[] args) { LuceneTester tester; try { tester = new LuceneTester(); tester.createindex(); catch (IOException e) { e.printstacktrace(); private void createindex() throws IOException{ indexer = new Indexer(indexDir); int numindexed; long starttime = System.currentTimeMillis(); numindexed = indexer.createindex(datadir, new TextFileFilter()); long endtime = System.currentTimeMillis(); indexer.close(); System.out.println(numIndexed+" File indexed, time taken: " +(endtime-starttime)+" ms"); Data & Index directory creation I've used 10 text files named from record1.txt to record10.txt containing simply names and other details of the students and put them in the directory E:\Lucene\Data. Test Data. An index directory path should be created as E:\Lucene\Index. After running this program, you can see the list of index files created in that folder. Running the Program: Once you are done with creating source, creating the raw data, data directory and index directory, you are ready for this step which is compiling and running your program. To do this, Keep LuceneTester.Java file tab active and use either Run option available in the Eclipse IDE or use Ctrl + F11 to compile and run your LuceneTester application. If everything is fine with your application, this will print the following message in Eclipse IDE's console: Indexing E:\Lucene\Data\record1.txt Indexing E:\Lucene\Data\record10.txt Indexing E:\Lucene\Data\record2.txt Indexing E:\Lucene\Data\record3.txt Indexing E:\Lucene\Data\record4.txt Indexing E:\Lucene\Data\record5.txt Indexing E:\Lucene\Data\record6.txt Indexing E:\Lucene\Data\record7.txt

43 Indexing E:\Lucene\Data\record8.txt Indexing E:\Lucene\Data\record9.txt 10 File indexed, time taken: 109 ms Once you've run the program successfully, you will have following content in your index directory: LUCENE - INDEXING OPERATIONS In this chapter, we'll discuss the four major operations of indexing. These operations are useful at various times and are used throughout of a software search application. Indexing Operations: Following is the list of commonly used operations during indexing process. Sr. No. Operation & Description 1 Add Document This operation is used in the initial stage of indexing proces to create the indexes on the newly available contents. 2 Update Document This operation is used to update indexes to reflect the changes in the updated contents. It is similar to recreating the index. 3 Delete Document This operation is used to update indexes to exclude the documents which are not required to be indexed/searched. 4 Field Options Field options specifies a way or controls the way in which contents of a field are to be made searchable. LUCENE - ADD DOCUMENT OPERATION Add document is one of the core operation as part of indexing process. We add Documents containing Fields to IndexWriter where IndexWriter is used to update or create indexes.

44 Now we'll show you a step by step process to get a kick start in understanding of add document using a basic example. Add a document to an index. Create a method to get a lucene document from a text file. Create various types of fields which are key value pairs containing keys as names and values as contents to be indexed. Set field to be analyzed or not. In our case, only contents is to be analyzed as it can contain data such as a, am, are, an etc. which are not required in search operations. Add the newly created fields to the document object and return it to the caller method. private Document getdocument(file file) throws IOException{ Document document = new Document(); //index file contents Field contentfield = new Field(LuceneConstants.CONTENTS, new FileReader(file)); //index file name Field filenamefield = new Field(LuceneConstants.FILE_NAME, file.getname(), Field.Store.YES,Field.Index.NOT_ANALYZED); //index file path Field filepathfield = new Field(LuceneConstants.FILE_PATH, file.getcanonicalpath(), Field.Store.YES,Field.Index.NOT_ANALYZED); document.add(contentfield); document.add(filenamefield); document.add(filepathfield); return document; Create a IndexWriter IndexWriter class acts as a core component which creates/updates indexes during indexing process. Create object of IndexWriter. Create a lucene directory which should point to location where indexes are to be stored. Initialize the IndexWricrter object created with the index directory, a standard analyzer having version information and other required/optional parameters. private IndexWriter writer; public Indexer(String indexdirectorypath) throws IOException{ //this directory will contain the indexes Directory indexdirectory = FSDirectory.open(new File(indexDirectoryPath)); //create the indexer writer = new IndexWriter(indexDirectory, new StandardAnalyzer(Version.LUCENE_36),true, IndexWriter.MaxFieldLength.UNLIMITED);

45 Add document and start Indexing process Following two are the ways to add the document. adddocumentdocument - Adds the document using the default analyzer specifiedwhenindexwriteriscreated. adddocumentdocument, Analyzer - Adds the document using the provided analyzer. private void indexfile(file file) throws IOException{ System.out.println("Indexing "+file.getcanonicalpath()); Document document = getdocument(file); writer.adddocument(document); Example Application Let us create a test Lucene application to test indexing process. Step Description 1 Create a project with a name LuceneFirstApplication under a package com.tutorialspoint.lucene as explained in the Lucene - First Application chapter. You can also use the project created in EJB - First Application chapter as such for this chapter to understand indexing process. 2 Create LuceneConstants.java,TextFileFilter.java and Indexer.java as explained in the Lucene - First Application chapter. Keep rest of the files unchanged. 3 Create LuceneTester.java as mentioned below. 4 Clean and Build the application to make sure business logic is working as per the requirements. LuceneConstants.java This class is used to provide various constants to be used across the sample application. package com.tutorialspoint.lucene; public class LuceneConstants { public static final String CONTENTS="contents"; public static final String FILE_NAME="filename"; public static final String FILE_PATH="filepath"; public static final int MAX_SEARCH = 10; TextFileFilter.java This class is used as a.txt file filter. package com.tutorialspoint.lucene; import java.io.file; import java.io.filefilter; public class TextFileFilter implements FileFilter public boolean accept(file pathname) { return pathname.getname().tolowercase().endswith(".txt"); Indexer.java

46 This class is used to index the raw data so that we can make it searchable using lucene library. package com.tutorialspoint.lucene; import java.io.file; import java.io.filefilter; import java.io.filereader; import java.io.ioexception; import org.apache.lucene.analysis.standard.standardanalyzer; import org.apache.lucene.document.document; import org.apache.lucene.document.field; import org.apache.lucene.index.corruptindexexception; import org.apache.lucene.index.indexwriter; import org.apache.lucene.store.directory; import org.apache.lucene.store.fsdirectory; import org.apache.lucene.util.version; public class Indexer { private IndexWriter writer; public Indexer(String indexdirectorypath) throws IOException{ //this directory will contain the indexes Directory indexdirectory = FSDirectory.open(new File(indexDirectoryPath)); //create the indexer writer = new IndexWriter(indexDirectory, new StandardAnalyzer(Version.LUCENE_36),true, IndexWriter.MaxFieldLength.UNLIMITED); public void close() throws CorruptIndexException, IOException{ writer.close(); private Document getdocument(file file) throws IOException{ Document document = new Document(); //index file contents Field contentfield = new Field(LuceneConstants.CONTENTS, new FileReader(file)); //index file name Field filenamefield = new Field(LuceneConstants.FILE_NAME, file.getname(), Field.Store.YES,Field.Index.NOT_ANALYZED); //index file path Field filepathfield = new Field(LuceneConstants.FILE_PATH, file.getcanonicalpath(), Field.Store.YES,Field.Index.NOT_ANALYZED); document.add(contentfield); document.add(filenamefield); document.add(filepathfield); return document; private void indexfile(file file) throws IOException{ System.out.println("Indexing "+file.getcanonicalpath()); Document document = getdocument(file); writer.adddocument(document); public int createindex(string datadirpath, FileFilter filter) throws IOException{ //get all files in the data directory File[] files = new File(dataDirPath).listFiles(); for (File file : files) {

47 if(!file.isdirectory() &&!file.ishidden() && file.exists() && file.canread() && filter.accept(file) ){ indexfile(file); return writer.numdocs(); LuceneTester.java This class is used to test the indexing capability of lucene library. package com.tutorialspoint.lucene; import java.io.ioexception; public class LuceneTester { String indexdir = "E:\\Lucene\\Index"; String datadir = "E:\\Lucene\\Data"; Indexer indexer; public static void main(string[] args) { LuceneTester tester; try { tester = new LuceneTester(); tester.createindex(); catch (IOException e) { e.printstacktrace(); private void createindex() throws IOException{ indexer = new Indexer(indexDir); int numindexed; long starttime = System.currentTimeMillis(); numindexed = indexer.createindex(datadir, new TextFileFilter()); long endtime = System.currentTimeMillis(); indexer.close(); System.out.println(numIndexed+" File indexed, time taken: " +(endtime-starttime)+" ms"); Data & Index directory creation I've used 10 text files named from record1.txt to record10.txt containing simply names and other details of the students and put them in the directory E:\Lucene\Data. Test Data. An index directory path should be created as E:\Lucene\Index. After running this program, you can see the list of index files created in that folder. Running the Program: Once you are done with creating source, creating the raw data, data directory and index directory, you are ready for this step which is compiling and running your program. To do this, Keep LuceneTester.Java file tab active and use either Run option available in the Eclipse IDE or use Ctrl + F11 to compile and run your LuceneTester application. If everything is fine with your application, this will print the following message in Eclipse IDE's console: Indexing E:\Lucene\Data\record1.txt Indexing E:\Lucene\Data\record10.txt Indexing E:\Lucene\Data\record2.txt Indexing E:\Lucene\Data\record3.txt Indexing E:\Lucene\Data\record4.txt

48 Indexing E:\Lucene\Data\record5.txt Indexing E:\Lucene\Data\record6.txt Indexing E:\Lucene\Data\record7.txt Indexing E:\Lucene\Data\record8.txt Indexing E:\Lucene\Data\record9.txt 10 File indexed, time taken: 109 ms Once you've run the program successfully, you will have following content in your index directory: LUCENE - UPDATE DOCUMENT OPERATION Update document is another important operation as part of indexing process.this operation is used when already indexed contents are updated and indexes become invalid. This operation is also known as re-indexing. We update Documents containing Fields to IndexWriter where IndexWriter is used to update indexes. Now we'll show you a step by step process to get a kick start in understanding of update document using a basic example. Update a document to an index. Create a method to update a lucene document from an updated text file. private void updatedocument(file file) throws IOException{ Document document = new Document(); //update indexes for file contents writer.updatedocument( new Term(LuceneConstants.CONTENTS, new FileReader(file)),document); writer.close(); Create a IndexWriter IndexWriter class acts as a core component which creates/updates indexes during indexing process. Create object of IndexWriter. Create a lucene directory which should point to location where indexes are to be stored. Initialize the IndexWricrter object created with the index directory, a standard analyzer having version information and other required/optional parameters.

49 private IndexWriter writer; public Indexer(String indexdirectorypath) throws IOException{ //this directory will contain the indexes Directory indexdirectory = FSDirectory.open(new File(indexDirectoryPath)); //create the indexer writer = new IndexWriter(indexDirectory, new StandardAnalyzer(Version.LUCENE_36),true, IndexWriter.MaxFieldLength.UNLIMITED); Update document and start reindexing process Following two are the ways to update the document. updatedocumentterm, Document - Delete the document containing the term and add the document using the default analyzer specifiedwhenindexwriteriscreated. updatedocumentterm, Document, Analyzer - Delete the document containing the term and add the document using the provided analyzer. private void indexfile(file file) throws IOException{ System.out.println("Updating index for "+file.getcanonicalpath()); updatedocument(file); Example Application Let us create a test Lucene application to test indexing process. Step Description 1 Create a project with a name LuceneFirstApplication under a package com.tutorialspoint.lucene as explained in the Lucene - First Application chapter. You can also use the project created in EJB - First Application chapter as such for this chapter to understand indexing process. 2 Create LuceneConstants.java,TextFileFilter.java and Indexer.java as explained in the Lucene - First Application chapter. Keep rest of the files unchanged. 3 Create LuceneTester.java as mentioned below. 4 Clean and Build the application to make sure business logic is working as per the requirements. LuceneConstants.java This class is used to provide various constants to be used across the sample application. package com.tutorialspoint.lucene; public class LuceneConstants { public static final String CONTENTS="contents"; public static final String FILE_NAME="filename"; public static final String FILE_PATH="filepath"; public static final int MAX_SEARCH = 10; TextFileFilter.java This class is used as a.txt file filter. package com.tutorialspoint.lucene;

50 import java.io.file; import java.io.filefilter; public class TextFileFilter implements FileFilter public boolean accept(file pathname) { return pathname.getname().tolowercase().endswith(".txt"); Indexer.java This class is used to index the raw data so that we can make it searchable using lucene library. package com.tutorialspoint.lucene; import java.io.file; import java.io.filefilter; import java.io.filereader; import java.io.ioexception; import org.apache.lucene.analysis.standard.standardanalyzer; import org.apache.lucene.document.document; import org.apache.lucene.document.field; import org.apache.lucene.index.corruptindexexception; import org.apache.lucene.index.indexwriter; import org.apache.lucene.store.directory; import org.apache.lucene.store.fsdirectory; import org.apache.lucene.util.version; public class Indexer { private IndexWriter writer; public Indexer(String indexdirectorypath) throws IOException{ //this directory will contain the indexes Directory indexdirectory = FSDirectory.open(new File(indexDirectoryPath)); //create the indexer writer = new IndexWriter(indexDirectory, new StandardAnalyzer(Version.LUCENE_36),true, IndexWriter.MaxFieldLength.UNLIMITED); public void close() throws CorruptIndexException, IOException{ writer.close(); private void updatedocument(file file) throws IOException{ Document document = new Document(); //update indexes for file contents writer.updatedocument( new Term(LuceneConstants.FILE_NAME, file.getname()),document); writer.close(); private void indexfile(file file) throws IOException{ System.out.println("Updating index: "+file.getcanonicalpath()); updatedocument(file); public int createindex(string datadirpath, FileFilter filter) throws IOException{ //get all files in the data directory File[] files = new File(dataDirPath).listFiles(); for (File file : files) {

51 if(!file.isdirectory() &&!file.ishidden() && file.exists() && file.canread() && filter.accept(file) ){ indexfile(file); return writer.numdocs(); LuceneTester.java This class is used to test the indexing capability of lucene library. package com.tutorialspoint.lucene; import java.io.ioexception; public class LuceneTester { String indexdir = "E:\\Lucene\\Index"; String datadir = "E:\\Lucene\\Data"; Indexer indexer; public static void main(string[] args) { LuceneTester tester; try { tester = new LuceneTester(); tester.createindex(); catch (IOException e) { e.printstacktrace(); private void createindex() throws IOException{ indexer = new Indexer(indexDir); int numindexed; long starttime = System.currentTimeMillis(); numindexed = indexer.createindex(datadir, new TextFileFilter()); long endtime = System.currentTimeMillis(); indexer.close(); Data & Index directory creation I've used 10 text files named from record1.txt to record10.txt containing simply names and other details of the students and put them in the directory E:\Lucene\Data. Test Data. An index directory path should be created as E:\Lucene\Index. After running this program, you can see the list of index files created in that folder. Running the Program: Once you are done with creating source, creating the raw data, data directory and index directory, you are ready for this step which is compiling and running your program. To do this, Keep LuceneTester.Java file tab active and use either Run option available in the Eclipse IDE or use Ctrl + F11 to compile and run your LuceneTester application. If everything is fine with your application, this will print the following message in Eclipse IDE's console: Updating index for E:\Lucene\Data\record1.txt Updating index for E:\Lucene\Data\record10.txt Updating index for E:\Lucene\Data\record2.txt Updating index for E:\Lucene\Data\record3.txt Updating index for E:\Lucene\Data\record4.txt Updating index for E:\Lucene\Data\record5.txt Updating index for E:\Lucene\Data\record6.txt

52 Updating index for E:\Lucene\Data\record7.txt Updating index for E:\Lucene\Data\record8.txt Updating index for E:\Lucene\Data\record9.txt 10 File indexed, time taken: 109 ms Once you've run the program successfully, you will have following content in your index directory: LUCENE - DELETE DOCUMENT OPERATION Delete document is another important operation as part of indexing process.this operation is used when already indexed contents are updated and indexes become invalid or indexes become very large in size then in order to reduce the size and update the index, delete operations are carried out. We delete Documents containing Fields to IndexWriter where IndexWriter is used to update indexes. Now we'll show you a step by step process to get a kick start in understanding of delete document using a basic example. Delete a document from an index. Create a method to delete a lucene document of an obsolete text file. private void deletedocument(file file) throws IOException{ //delete indexes for a file writer.deletedocument(new Term(LuceneConstants.FILE_NAME,file.getName())); writer.commit(); System.out.println("index contains deleted files: "+writer.hasdeletions()); System.out.println("index contains documents: "+writer.maxdoc()); System.out.println("index contains deleted documents: "+writer.numdoc()); Create a IndexWriter IndexWriter class acts as a core component which creates/updates indexes during indexing process. Create object of IndexWriter. Create a lucene directory which should point to location where indexes are to be stored. Initialize the IndexWricrter object created with the index directory, a standard analyzer having version information and other required/optional parameters.

53 private IndexWriter writer; public Indexer(String indexdirectorypath) throws IOException{ //this directory will contain the indexes Directory indexdirectory = FSDirectory.open(new File(indexDirectoryPath)); //create the indexer writer = new IndexWriter(indexDirectory, new StandardAnalyzer(Version.LUCENE_36),true, IndexWriter.MaxFieldLength.UNLIMITED); Delete document and start reindexing process Following two are the ways to delete the document. deletedocumentsterm - Delete all the documents containing the term. deletedocumentsterm[] - Delete all the documents containing any of the terms in the array. deletedocumentsquery - Delete all the documents matching the query. deletedocumentsquery[] - Delete all the documents matching the query in the array. deleteall - Delete all the documents. private void indexfile(file file) throws IOException{ System.out.println("Deleting index for "+file.getcanonicalpath()); deletedocument(file); Example Application Let us create a test Lucene application to test indexing process. Step Description 1 Create a project with a name LuceneFirstApplication under a package com.tutorialspoint.lucene as explained in the Lucene - First Application chapter. You can also use the project created in EJB - First Application chapter as such for this chapter to understand indexing process. 2 Create LuceneConstants.java,TextFileFilter.java and Indexer.java as explained in the Lucene - First Application chapter. Keep rest of the files unchanged. 3 Create LuceneTester.java as mentioned below. 4 Clean and Build the application to make sure business logic is working as per the requirements. LuceneConstants.java This class is used to provide various constants to be used across the sample application. package com.tutorialspoint.lucene; public class LuceneConstants { public static final String CONTENTS="contents"; public static final String FILE_NAME="filename"; public static final String FILE_PATH="filepath"; public static final int MAX_SEARCH = 10; TextFileFilter.java This class is used as a.txt file filter.

54 package com.tutorialspoint.lucene; import java.io.file; import java.io.filefilter; public class TextFileFilter implements FileFilter public boolean accept(file pathname) { return pathname.getname().tolowercase().endswith(".txt"); Indexer.java This class is used to index the raw data so that we can make it searchable using lucene library. package com.tutorialspoint.lucene; import java.io.file; import java.io.filefilter; import java.io.filereader; import java.io.ioexception; import org.apache.lucene.analysis.standard.standardanalyzer; import org.apache.lucene.document.document; import org.apache.lucene.document.field; import org.apache.lucene.index.corruptindexexception; import org.apache.lucene.index.indexwriter; import org.apache.lucene.index.term; import org.apache.lucene.store.directory; import org.apache.lucene.store.fsdirectory; import org.apache.lucene.util.version; public class Indexer { private IndexWriter writer; public Indexer(String indexdirectorypath) throws IOException{ //this directory will contain the indexes Directory indexdirectory = FSDirectory.open(new File(indexDirectoryPath)); //create the indexer writer = new IndexWriter(indexDirectory, new StandardAnalyzer(Version.LUCENE_36),true, IndexWriter.MaxFieldLength.UNLIMITED); public void close() throws CorruptIndexException, IOException{ writer.close(); private void deletedocument(file file) throws IOException{ //delete indexes for a file writer.deletedocuments( new Term(LuceneConstants.FILE_NAME,file.getName())); writer.commit(); private void indexfile(file file) throws IOException{ System.out.println("Deleting index: "+file.getcanonicalpath()); deletedocument(file); public int createindex(string datadirpath, FileFilter filter) throws IOException{ //get all files in the data directory

55 File[] files = new File(dataDirPath).listFiles(); for (File file : files) { if(!file.isdirectory() &&!file.ishidden() && file.exists() && file.canread() && filter.accept(file) ){ indexfile(file); return writer.numdocs(); LuceneTester.java This class is used to test the indexing capability of lucene library. package com.tutorialspoint.lucene; import java.io.ioexception; public class LuceneTester { String indexdir = "E:\\Lucene\\Index"; String datadir = "E:\\Lucene\\Data"; Indexer indexer; public static void main(string[] args) { LuceneTester tester; try { tester = new LuceneTester(); tester.createindex(); catch (IOException e) { e.printstacktrace(); private void createindex() throws IOException{ indexer = new Indexer(indexDir); int numindexed; long starttime = System.currentTimeMillis(); numindexed = indexer.createindex(datadir, new TextFileFilter()); long endtime = System.currentTimeMillis(); indexer.close(); Data & Index directory creation I've used 10 text files named from record1.txt to record10.txt containing simply names and other details of the students and put them in the directory E:\Lucene\Data. Test Data. An index directory path should be created as E:\Lucene\Index. After running this program, you can see the list of index files created in that folder. Running the Program: Once you are done with creating source, creating the raw data, data directory and index directory, you are ready for this step which is compiling and running your program. To do this, Keep LuceneTester.Java file tab active and use either Run option available in the Eclipse IDE or use Ctrl + F11 to compile and run your LuceneTester application. If everything is fine with your application, this will print the following message in Eclipse IDE's console: Deleting index E:\Lucene\Data\record1.txt Deleting index E:\Lucene\Data\record10.txt Deleting index E:\Lucene\Data\record2.txt Deleting index E:\Lucene\Data\record3.txt

56 Deleting index E:\Lucene\Data\record4.txt Deleting index E:\Lucene\Data\record5.txt Deleting index E:\Lucene\Data\record6.txt Deleting index E:\Lucene\Data\record7.txt Deleting index E:\Lucene\Data\record8.txt Deleting index E:\Lucene\Data\record9.txt 10 File indexed, time taken: 109 ms Once you've run the program successfully, you will have following content in your index directory: LUCENE - FIELD OPTIONS Field is the most important and the foundation unit of indexing process. It is the actual object containing the contents to be indexed. When we add a field, lucene provides numerous controls on the field using Field Options which states how much a field is to be searchable. We add Documents containing Fields to IndexWriter where IndexWriter is used to update or create indexes. Now we'll show you a step by step process to get a kick start in understanding of various Field options using a basic example. Various Field Options Index.ANALYZED - First analyze then do indexing. Used for normal text indexing. Analyzer will break the field's value into stream of tokens and each token is searcable sepearately. Index.NOT_ANALYZED - Don't analyze but do indexing. Used for complete text indexing for example person's names, URL etc. Index.ANALYZED_NO_NORMS - Varient of Index.ANALYZED. Analyzer will break the field's value into stream of tokens and each token is searcable sepearately but NORMs are not stored in the indexes.norms are used to boost searching but are sometime memory consuming. Index.Index.NOT_ANALYZED_NO_NORMS - Varient of Index.NOT_ANALYZED. Indexing is done but NORMS are not stored in the indexes. Index.NO - Field value is not searchable. Use of Field Options Create a method to get a lucene document from a text file. Create various types of fields which are key value pairs containing keys as names and values as contents to be indexed. Set field to be analyzed or not. In our case, only contents is to be analyzed as it can contain data such as a, am, are, an etc. which are not required in search operations.

57 Add the newly created fields to the document object and return it to the caller method. private Document getdocument(file file) throws IOException{ Document document = new Document(); //index file contents Field contentfield = new Field(LuceneConstants.CONTENTS, new FileReader(file)); //index file name Field filenamefield = new Field(LuceneConstants.FILE_NAME, file.getname(), Field.Store.YES,Field.Index.NOT_ANALYZED); //index file path Field filepathfield = new Field(LuceneConstants.FILE_PATH, file.getcanonicalpath(), Field.Store.YES,Field.Index.NOT_ANALYZED); document.add(contentfield); document.add(filenamefield); document.add(filepathfield); return document; Example Application Let us create a test Lucene application to test indexing process. Step Description 1 Create a project with a name LuceneFirstApplication under a package com.tutorialspoint.lucene as explained in the Lucene - First Application chapter. You can also use the project created in EJB - First Application chapter as such for this chapter to understand indexing process. 2 Create LuceneConstants.java,TextFileFilter.java and Indexer.java as explained in the Lucene - First Application chapter. Keep rest of the files unchanged. 3 Create LuceneTester.java as mentioned below. 4 Clean and Build the application to make sure business logic is working as per the requirements. LuceneConstants.java This class is used to provide various constants to be used across the sample application. package com.tutorialspoint.lucene; public class LuceneConstants { public static final String CONTENTS="contents"; public static final String FILE_NAME="filename"; public static final String FILE_PATH="filepath"; public static final int MAX_SEARCH = 10; TextFileFilter.java This class is used as a.txt file filter. package com.tutorialspoint.lucene; import java.io.file; import java.io.filefilter;

58 public class TextFileFilter implements FileFilter public boolean accept(file pathname) { return pathname.getname().tolowercase().endswith(".txt"); Indexer.java This class is used to index the raw data so that we can make it searchable using lucene library. package com.tutorialspoint.lucene; import java.io.file; import java.io.filefilter; import java.io.filereader; import java.io.ioexception; import org.apache.lucene.analysis.standard.standardanalyzer; import org.apache.lucene.document.document; import org.apache.lucene.document.field; import org.apache.lucene.index.corruptindexexception; import org.apache.lucene.index.indexwriter; import org.apache.lucene.store.directory; import org.apache.lucene.store.fsdirectory; import org.apache.lucene.util.version; public class Indexer { private IndexWriter writer; public Indexer(String indexdirectorypath) throws IOException{ //this directory will contain the indexes Directory indexdirectory = FSDirectory.open(new File(indexDirectoryPath)); //create the indexer writer = new IndexWriter(indexDirectory, new StandardAnalyzer(Version.LUCENE_36),true, IndexWriter.MaxFieldLength.UNLIMITED); public void close() throws CorruptIndexException, IOException{ writer.close(); private Document getdocument(file file) throws IOException{ Document document = new Document(); //index file contents Field contentfield = new Field(LuceneConstants.CONTENTS, new FileReader(file)); //index file name Field filenamefield = new Field(LuceneConstants.FILE_NAME, file.getname(), Field.Store.YES,Field.Index.NOT_ANALYZED); //index file path Field filepathfield = new Field(LuceneConstants.FILE_PATH, file.getcanonicalpath(), Field.Store.YES,Field.Index.NOT_ANALYZED); document.add(contentfield); document.add(filenamefield); document.add(filepathfield); return document; private void indexfile(file file) throws IOException{ System.out.println("Indexing "+file.getcanonicalpath());

59 Document document = getdocument(file); writer.adddocument(document); public int createindex(string datadirpath, FileFilter filter) throws IOException{ //get all files in the data directory File[] files = new File(dataDirPath).listFiles(); for (File file : files) { if(!file.isdirectory() &&!file.ishidden() && file.exists() && file.canread() && filter.accept(file) ){ indexfile(file); return writer.numdocs(); LuceneTester.java This class is used to test the indexing capability of lucene library. package com.tutorialspoint.lucene; import java.io.ioexception; public class LuceneTester { String indexdir = "E:\\Lucene\\Index"; String datadir = "E:\\Lucene\\Data"; Indexer indexer; public static void main(string[] args) { LuceneTester tester; try { tester = new LuceneTester(); tester.createindex(); catch (IOException e) { e.printstacktrace(); private void createindex() throws IOException{ indexer = new Indexer(indexDir); int numindexed; long starttime = System.currentTimeMillis(); numindexed = indexer.createindex(datadir, new TextFileFilter()); long endtime = System.currentTimeMillis(); indexer.close(); System.out.println(numIndexed+" File indexed, time taken: " +(endtime-starttime)+" ms"); Data & Index directory creation I've used 10 text files named from record1.txt to record10.txt containing simply names and other details of the students and put them in the directory E:\Lucene\Data. Test Data. An index directory path should be created as E:\Lucene\Index. After running this program, you can see the list of index files created in that folder. Running the Program: Once you are done with creating source, creating the raw data, data directory and index directory, you are ready for this step which is compiling and running your program. To do this, Keep

60 LuceneTester.Java file tab active and use either Run option available in the Eclipse IDE or use Ctrl + F11 to compile and run your LuceneTester application. If everything is fine with your application, this will print the following message in Eclipse IDE's console: Indexing E:\Lucene\Data\record1.txt Indexing E:\Lucene\Data\record10.txt Indexing E:\Lucene\Data\record2.txt Indexing E:\Lucene\Data\record3.txt Indexing E:\Lucene\Data\record4.txt Indexing E:\Lucene\Data\record5.txt Indexing E:\Lucene\Data\record6.txt Indexing E:\Lucene\Data\record7.txt Indexing E:\Lucene\Data\record8.txt Indexing E:\Lucene\Data\record9.txt 10 File indexed, time taken: 109 ms Once you've run the program successfully, you will have following content in your index directory: LUCENE - SEARCH OPERATION Searching process is one of the core functionality provided by Lucene. Following diagram illustrates the searching process and use of classes. IndexSearcher is the most important and core component of the searching process. We first create Directorys containing indexes and then pass it to IndexSearcher which opens the Directory using IndexReader. Then we create a Query with a Term and make a search using IndexSearcher by passing the Query to the searcher. IndexSearcher returns a TopDocs object which contains the search details along with document IDs of the Document which is the result of