Introduc)on to Lucene. Debapriyo Majumdar Information Retrieval Spring 2015 Indian Statistical Institute Kolkata

Transcription

1 Introduc)on to Lucene Debapriyo Majumdar Information Retrieval Spring 2015 Indian Statistical Institute Kolkata

2 Open source search engines Academic Terrier (Java, University of Glasgow) Indri, Lemur (C++, and Java too, UMass & CMU) Zettair (University of Melbourne) Apache project (non-academic) Lucene Apache license, legally easier for commercial use Lucene Java search engine library, with many features Ports/integration to other languages available (C/C++, C#, Python, Ruby, ) Other projects on top of Lucene: Solr and others Used by: LinkedIn, Twitter, CiteSeer, 2

3 Lucene: overview Documents Lucene document User: document building Lucene: Analyzing Text query User: query building Lucene query Lucene: Analyzing UI User Results display User: reading search results Tokens Lucene: Indexing Index and dic)onary Lucene: Searching Search results User = Programmer 3

4 Lucene document building A document is a collection of Fields Document: CV of a student, several fields Last_Name: Banerjee First_Name: Arkadeep Age: 24 Gender: M Institute: ISI Kolkata Location: Kolkata Description: Arkadeep is a highly motivated student who simply loves challenge. He is A field can be a text, a number, a range 4

5 Building document import org.apache.lucene.document.document; import org.apache.lucene.document.field; Document doc = new Document(); doc.add(new StringField( Last_Name, Banerjee, )); doc.add(new StringField( First_Name, Arkadeep, )); doc.add(new IntField( Age,24, ); doc.add(new TextField( Description,description, )); Now, let s understand the fields 5

6 Building document new StringField( Last_Name, Banerjee,Field.Store.Yes)); new StringField( First_Name, Arkadeep,Field.Store.Yes)); new NumericField( Age,24,Field.Store.Yes); new TextField( Description,desc, Field.Store.No)); Field.Store Field.Index NO : Don t store the field value in the index ANALYZED : Tokenize with an Analyzer YES : Store the field value in the index NOT_ANALYZED : Do not tokenize NO : Do not index this field Other options 6

7 Indexing StandardAnalyzer analyzer = new StandardAnalyzer(); FSDirectory dir = FSDirectory.open(new File( directory_path )); IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_40, analyzer); IndexWriter w = new IndexWriter(index, config); w.adddocument(document); w.close(); IndexWriter: primary class for indexing Stores the index in a directory RAMDirectory also possible (in memory) 7

8 Lucene document building A document is a collection of Fields Document: CV of a student, several fields Last_Name: Banerjee First_Name: Arkadeep Age: 24 Gender: M Institute: ISI Kolkata Location: Kolkata Description: Arkadeep is a highly motivated student who simply loves challenge. He is A field can be a text, a number, a range What would we like to search with? 8

9 Lucene Analyzer Tokenizes the text in the Fields Common Analyzers WhitespaceAnalyzer Splits tokens on whitespace SimpleAnalyzer Splits tokens on non-letters, and then lowercases StopAnalyzer Same as SimpleAnalyzer, but also removes stop words StandardAnalyzer Most sophisticated analyzer that knows about certain token types, lowercases, removes stop words,...

10 Analysis example The quick brown fox jumped over the lazy dog WhitespaceAnalyzer [The] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dog] SimpleAnalyzer [the] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dog] StopAnalyzer [quick] [brown] [fox] [jumped] [over] [lazy] [dog] StandardAnalyzer [quick] [brown] [fox] [jumped] [over] [lazy] [dog]

11 Analysis example 2 XY&Z Corporation xyz@example.com WhitespaceAnalyzer [XY&Z] [Corporation] [-] [xyz@example.com] SimpleAnalyzer [xy] [z] [corporation] [xyz] [example] [com] StopAnalyzer [xy] [z] [corporation] [xyz] [example] [com] StandardAnalyzer [xy&z] [corporation] [xyz@example.com]

12 Searching int k = 10; StandardAnalyzer analyzer = new StandardAnalyzer(); IndexReader reader = DirectoryReader.open(FSDirectory.open(new File( ".//index"))); IndexSearcher searcher = new IndexSearcher(reader); Query q = new QueryParser("Description", analyzer).parse(query); TopDocs docs = searcher.search(q, k); ScoreDoc[] hits = docs.scoredocs; IndexReader and IndexSearcher 12

13 Query and QueryParser QueryParser parser = new QueryParser(Version.LUCENE_40, LastName, new StandardAnalyzer()); Query query = parser.parse(q); QueryParser Need to parse the query in the same way the documents were indexed Tell the query which field should it use (field based search) Use the same analyzer

14 TopDocs and ScoreDoc TopDocs docs = searcher.search(q, k); ScoreDoc[] hits = docs.scoredocs; Search returns TopDocs Reference to the top ranked documents returned by search TopDoc has ScoreDoc(s) Each ScoreDoc is a single document

15 GeXng the results System.out.println("Found " + hits.length + " hits."); for(int i=0;i<hits.length;++i) { int docid = hits[i].doc; Document d = searcher.doc(docid); System.out.println((i + 1) + ". " + d.get("first_name") + " " + d.get("last_name")); } Get the required fields from the documents

16 Adding/dele)ng Documents void adddocument(document d); void adddocument(document d, Analyzer a); Important: Need to ensure that Analyzers used at indexing time are consistent with Analyzers used at searching time // deletes docs containing term or matching // query. The term version is useful for // deleting one document. void deletedocuments(term term); void deletedocuments(query query);

17 Index format Each Lucene index consists of one or more segments A segment is a standalone index for a subset of documents All segments are searched A segment is created whenever IndexWriter flushes adds/ deletes Periodically, IndexWriter will merge a set of segments into a single segment Policy specified by a MergePolicy Segments are grouped into levels Segments within a group are roughly equal size (in log space) Once a level has enough segments, they are merged into a segment at the next level up Explicitly invoke optimize() to merge segments

18 Searching a changing index Directory dir = FSDirectory.open(...); IndexReader reader = IndexReader.open(dir); IndexSearcher searcher = new IndexSearcher(reader); Above reader does not reflect changes to the index unless you reopen it. Reopening is more resource efficient than opening a new IndexReader. IndexReader newreader = reader.reopen(); If (reader!= newreader) { reader.close(); reader = newreader; searcher = new IndexSearcher(reader); }

19 Near- real- )me search IndexWriter writer =...; IndexReader reader = writer.getreader(); IndexSearcher searcher = new IndexSearcher(reader); // Now let us say there s a change to the index using writer writer.adddocument(newdoc); // reopen() and getreader() force writer to flush IndexReader newreader = reader.reopen(); if (reader!= newreader) { reader.close(); reader = newreader; searcher = new IndexSearcher(reader); }

20 Query Syntax Query expression java java junit java OR junit +java +junit java AND junit )tle:ant )tle:extreme subject:sports (agile OR extreme) AND java )tle: junit in ac)on )tle: junit ac)on ~5 java* java~ lastmodified:[1/1/09 TO 12/31/09] Document matches if Contains the term java in the default field Contains the term java or junit or both in the default field (the default operator can be changed to AND) Contains both java and junit in the default field Contains the term ant in the )tle field Contains extreme in the )tle and not sports in subject Boolean expression matches Phrase matches in )tle Proximity matches (within 5) in )tle Wildcard matches Fuzzy matches Range matches

21 Programma)cally constructed queries TermQuery, TermRangeQuery NumericRangeQuery PrefixQuery BooleanQuery PhraseQuery, WildcardQuery

22 Source and acknowledgements Slides by Manning and Nayak: The Lucene tutorial website: Apache lucene: 22