Building Search Applications

Transcription

1 Building Search Applications Lucene, LingPipe, and Gate Manu Konchady Mustru Publishing, Oakton, Virginia.

2 Contents Preface ix 1 Information Overload Information Sources Information Management Tools Search Engines Entity Extraction Organizing Information Tracking Information Visualization Social Network Visualization Stock Price and News Visualization Tag Clouds Applications Spam Detection Usage and Management Customer Service Employee Surveys Other Applications 18 2 Tokenizing Text Character Sets Tokens Lucene Analyzers ' WhitespaceAnalyzer SimpleAnalyzer Analyzer Design 27

3 2.2.4 StandardAnalyzer PorterAnalyzer StandardBgramAnalyzer 32 3δ Other Analyzers 2.3 LingPipe Tokenizers όό IndoEuropeanTokenizer Filtered Tokenizers Regular Expression Tokenizer Character-based Ngram Tokenizer A LingPipe Tokenizer in a Lucene Analyzer A Lucene Analyzer in a LingPipe Tokenizer Gate Tokenizer A Gate Tokenizer in a Lucene Analyzer Tokenizing Problems Text Extraction WordNet Word Stems and WordNet Summary 69 1 Indexing Text with Lucene Databases and Search Engines Early Search Engines Web Search Engines and IR Systems Generating an Index Term Weighting Term Vector Model Inverted Index Creating an Index with Lucene Field Attributes Boosting 89 «3.5 Modifying an Index with Lucene A Database Backed Index Deleting a Document Updating a Document Maintaining an Index 98

4 3.7.1 Logs Transactions Database Index Synchronization Lucene Index Files Performance Index Tuning Parameters Evaluation of Parameters Memory-Based Index Index Performance with a Database Index Scalability Index Vocabulary Date Fields Metadata Document Metadata Multimedia Metadata Metadata Standards Summary 124 Searching Text with Lucene Lucene Search Architecture Search Interface Design Search Behavior Intranets and the Web Searching the Index Generating Queries with QueryParser Expanded Queries Span Queries Query Performance Organizing Results Sorting Results Scoring Results Customizing Query-Doc Similarity Filtering Queries Range Filter Security Filter 161

5 4.7.3 Query Filter Caching Filters Chained Filters 4.8 Modifying Queries Spell Check Finding Similar Documents Troubleshooting a Query Summary Tagging Text 5.1 Sentences Sentence Extraction with LingPipe Sentence Extraction with Gate Text Extraction from Web Pages Part of Speech Taggers Tag Sets Markov Models Evaluation of a Tagger POS Tagging with LingPipe Rule-Based Tagging POS Tagging with Gate Markov model vs Rule-based Taggers Phrase Extraction Applications Finding Phrases Likelihood Ratio Phrase Extraction using LingPipe Current Phrases Entity Extraction Applications Entity Extraction with Gate Entity Extraction with LingPipe Evaluation Entity Extraction Errors Summary

6 6 Organizing Text: Clustering Applications Creating Clusters Clustering Documents Similarity Measures Comparison of Similarity Measures Using the Similarity Matrix Cluster Algorithms Global Optimization Methods Heuristic Methods Agglomerative Methods Building Clusters with LingPipe Debugging Clusters Evaluating Clusters Summary Organizing Text: Categorization Categorization Problem Applications for Document Categorization Categorizing Documents Training the Model Using the Model Categorization Methods Character-based Ngram Models Binary and Multi Classifiers TF/IDF Classifier K-Nearest Neighbors Classifier Naïve Bayes Classifier Evaluation Feature Extraction Summary Searching an Intranet and the Web Early Web Search Engines Web Structure 298

7 8.2.1 A Bow-Tie Web Graph Hubs Authorities PageRank Algorithm PageRank vs. Hubs & Authorities Crawlers Building a Crawler Search Engine Coverage Nutch Nutch Crawler Crawl Configuration Running a Re-crawl Search Interface Troubleshooting Summary Tracking Information News Monitoring Web Feeds NewsRack Sentiment Analysis Automatic Classification An Implementation with LingPipe Detecting Offensive Content Detection Methods Plagiarism Detection Forms of Plagiarism Methods to Detect Plagiarism Copy Detection using SCAM Other Applications Summary Future Directions in Search Improving Search Engines Adding Human Intelligence Special Features 372

8 OpenSearch Specialized Search Engines Using Collective Intelligence to Improve Search Tag-Based Search Engines Question & Answer Q&A Engine Design Performance Summary 392 Appendix A Software 393 Appendix Β Bayes Classification 403 Appendix C The Berkeley DB 407 Index 417