Modern Information Retrieval

Transcription

1

2 Modern Information Retrieval The Concepts and Technology behind Search Ricardo Baeza-Yates Berthier Ribeiro-Neto Second edition Addison-Wesley Harlow, England Reading, Massachusetts Menlo Park, California New York Don Mills, Ontario Amsterdam Bonn Sydney Singapore Tokyo Madrid San Juan Milan Mexico City Seoul Taipei

3 To be filled by Pearson

4 To Helena, Rosa, and our children Amo los libros exploradores, libros con bosque o nieve, profundidad o cielo Un libro, un libro lleno de contactos humanos, de camisas, un libro sin soledad, con hombres y herramientas, un libro es la victoria. de Oda al Libro (I) y (II), en Odas Elementales, Pablo Neruda Ilovebooksthatexplore, books with a forest or snow, depth or sky Abook,abookfull of human contacts, of shirts, abookwithoutsolitude, with people and tools, abookisthevictory. from Ode to the Book (I) and (II), in Elemental Odes, Pablo Neruda território de homens livres que será nossopaís eserápátria de todos. Irmãos, cantai esse mundo que não verei, mas virá um dia, dentro de mil anos, talvez mais... não tenho pressa. de Cidade Prevista no livro ARosadoPovo,1945. Carlos Drummond de Andrade territory of free men that will be our country and will be the nation of all Brothers, sing that world which I ll not see, but which will come one day, in a thousand years, maybe more... no hurry. from Prevised City in the book The Rose of the People, Carlos Drummond de Andrade

5

6 Contents Preface to the Second Edition Preface to the First Edition Author s Acknowledgements to the Second Edition Author s Acknowledgements to the First Edition Publisher s Acknowledgements xix xxi xxiii xxv xxvii 1 Introduction Information Retrieval Early Developments Information Retrieval in Libraries and Digital Libraries IR at the Center of the Stage The IR Problem The User s Task Information versus Data Retrieval The IR System Software Architecture of the IR System The Retrieval and Ranking Processes The Web A Brief History The e-publishing Era How the Web Changed Search Practical Issues on the Web Organization of the Book Focus of the Book Book Contents The Book Web Site: A Teaching Resource Bibliographic Discussion User Interfaces for Search 21 by Marti Hearst 2.1 Introduction How People Search vii

7 viii CONTENTS Information Lookup versus Exploratory Search Classic versus Dynamic Model of Information Seeking Navigation versus Search Observations of the Search Process Search Interfaces Today Getting Started Query Specification Query Specification Interfaces Retrieval Results Display Query Reformulation Organizing Search Results Visualization in Search Interfaces Visualizing Boolean Syntax Visualizing Query Terms within Retrieval Results Visualizing Relationships Among Words and Documents Visualization for Text Mining Design and Evaluation of Search Interfaces Trends and Research Issues Bibliographic Discussion Modeling IR Models Modeling and Ranking Characterization of an IR Model A Taxonomy of IR Models Classic Information Retrieval Basic Concepts The Boolean Model Term Weighting TF-IDF Weights Document Length Normalization The Vector Model The Probabilistic Model Brief Comparison of Classic Models Alternative Set Theoretic Models Set-Based Model Extended Boolean Model Fuzzy Set Model Alternative Algebraic Models Generalized Vector Space Model Latent Semantic Indexing Model Neural Network Model Alternative Probabilistic Models BM Language Models Divergence from Randomness Bayesian Network Models

8 CONTENTS ix 3.6 Other Models The Hypertext Model Web based Models Structured Text Retrieval Multimedia Retrieval Enterprise and Vertical Search Trends and Research Issues Bibliographic Discussion Retrieval Evaluation Introduction The Cranfield Paradigm A Brief History Reference Collections Retrieval Metrics Precision and Recall Single Value Summaries: P@n, MAP, MRR, F User-Oriented Measures DCG: Discounted Cumulated Gain BPREF: Binary Preferences Rank Correlation Metrics Reference Collections The TREC Collections Other Reference Collections Other Small Test Collections User-Based Evaluation Human Experimentation in the Lab Side-by-Side Panels A/B Testing Crowdsourcing Evaluation using Clickthrough Data Practical Caveats Trends and Research Issues Bibliographic Discussion Relevance Feedback and Query Expansion Introduction A Framework for Feedback Methods Explicit Relevance Feedback Relevance Feedback for the Vector Model: Rocchio Method Relevance Feedback for the Probabilistic Model Evaluation of Relevance Feedback Explicit Feedback Through Clicks Eye Tracking and Relevance Judgements User Behavior Clicks as a Metric of User Preferences Implicit Feedback Through Local Analysis

9 x CONTENTS Implicit Feedback Through Local Clustering Implicit Feedback through Local Context Analysis Implicit Feedback Through Global Analysis Query Expansion based on a Similarity Thesaurus Query Expansion based on a Statistical Thesaurus Trends and Research Issues Bibliographic Discussion Documents: Languages & Properties 203 with Gonzalo Navarro and Nivio Ziviani 6.1 Introduction Metadata Document Formats Text Multimedia Graphics and Virtual Reality Markup Languages SGML HTML XML RDF: Resource Description Framework HyTime Text Properties Information Theory Modeling Natural Language Text Similarity Document Preprocessing Lexical Analysis of the Text Elimination of Stopwords Stemming Keyword Selection Thesauri Organizing Documents Taxonomies Folksonomies Text Compression Basic Concepts Statistical Methods Statistical Methods: Modeling Statistical Methods: Coding Dictionary Methods Preprocessing for Compression Comparing Text Compression Techniques Structured Text Compression Trends and Research Issues Bibliographical Discussion

10 CONTENTS xi 7 Queries: Languages & Properties 257 with Gonzalo Navarro 7.1 Query Languages Keyword-Based Querying Beyond Keywords Structural Queries Query Protocols Query Properties Characterizing Web Queries User Search Behavior Query Intent Query Topic Query Sessions and Missions Query Difficulty Trends and Research Issues Bibliographical Discussion Text Classification 283 with Marcos Gonçalves 8.1 Introduction A Characterization of Text Classification Machine Learning The Text Classification Problem Text Classification Algorithms Unsupervised Algorithms Clustering Naive Text Classification Supervised Algorithms Decision Trees The k-nn Classifier The Rocchio Classifier Probabilistic Naive Bayes Document Classification The SVM Classifier Ensemble Classifiers Final Remarks on Supervised Algorithms Feature Selection or Dimensionality Reduction Term Class Incidence Table Term Document Frequency TF-IDF Weights Mutual Information Information Gain Chi Square Impact of Feature Selection Evaluation Metrics Contingency Table Accuracy and Error Precision and Recall

11 xii CONTENTS F-measure and F Cross-Validation Standard Collections Organizing the Classes Building Taxonomies Trends and Research Issues Bibliographic Discussion Indexing and Searching 339 with Gonzalo Navarro 9.1 Introduction Inverted Indexes Basic Concepts Full Inverted Indexes Searching Ranking Construction Compressed Inverted Indexes Structural Queries Signature Files Suffix Trees and Suffix Arrays Structure: Tries and Suffix Trees Searching for Simple Strings Searching for Complex Patterns Construction Compressed Suffix Arrays Sequential Searching Simple Strings: Horspool Complex Patterns: Automata and Bit-Parallelism Faster Bit-Parallel Algorithms Regular Expressions Multiple Patterns Approximate Searching Searching Compressed Text Multi-dimensional Indexing Trends and Research Issues Bibliographic Discussion Parallel and Distributed IR 401 with Eric Brown 10.1 Introduction A Taxonomy of Distributed IR Systems Data Partitioning Collection Partitioning Collection Selection Inverted Index Partitioning Partitioning other Indexes Parallel IR

12 CONTENTS xiii Introduction Parallel IR on MIMD Architectures Parallel IR on SIMD Architectures Cluster-based IR Distributed IR Introduction Indexing Query Processing Web Issues Federated Search Retrieval in Peer-to-Peer Networks Trends and Research Issues Bibliographic Discussion Web Retrieval 449 with Yoelle Maarek 11.1 Introduction A Challenging Problem The Web Characteristics Structure of the Web Graph Modeling the Web Link Analysis Search Engine Architectures Basic Architecture Cluster-based Architecture Caching Multiple Indexes Distributed Architectures Search Engine Ranking Ranking Signals Link-based Ranking Simple Ranking Functions Learning to Rank Learning the Ranking Function Quality Evaluation Web Spam Managing Web Data Assigning Identifiers to Documents Metadata Compressing the Web Graph Handling Duplicated Data Search Engine User Interaction The Search Rectangle Paradigm The Search Engine Result Page Educating the User Browsing

13 xiv CONTENTS Flat Browsing Structure Guided Browsing and Web Directories Beyond Browsing Hypertext and the Web Combining Searching with Browsing Web Query Languages Dynamic Search Related Problems Computational Advertising Web Mining Metasearch Trends and Research Issues Beyond Static Text Data Current Challenges Bibliographical Discussion Web Crawling 519 with Carlos Castillo 12.1 Introduction Applications of a Web Crawler General Web Search Topical Crawling Web Characterization Mirroring Web Site Analysis A Taxonomy of Crawlers Types of Web Pages Architecture and Implementation Crawler Architecture Practical Issues Parallel Crawling Scheduling Algorithms Selection Policy Revisit Policy Politeness Policy Combining Policies Evaluation Evaluating Network Usage Evaluating Long-term Scheduling Trends and Research Issues Crawling the Hidden Web Crawling with the Help of Web Sites Distributed Crawling Bibliographic Discussion

14 CONTENTS xv 13 Structured Text Retrieval 549 with Mounia Lalmas 13.1 Introduction Structuring Power Explicit vs. Implicit Structure Static vs. Dynamic Structure Single Hierarchy vs. Multiple Hierarchies Early Text Retrieval Models Model Based on Non-Overlapping Lists Model Based on Proximal Nodes Ranking Structured Text Results XML Retrieval Challenges in XML Retrieval Indexing Strategies Ranking Strategies Removing Overlaps XML Retrieval Evaluation Document Collections Topics Retrieval Tasks Relevance Measures Query Languages Characteristics Classification of XML Query Languages Examples of XML Query Languages Trends and Research Issues Bibliographic Discussion Multimedia Information Retrieval 591 by Dulce Ponceleón and Malcolm Slaney 14.1 Introduction What is Multimedia? Multimedia IR Text IR versus Multimedia IR The Challenges The Semantic Gap Feature Ambiguity Machine-generated Data Content-based Image Retrieval Color-Based Retrieval Texture Salient Points Audio and Music Retrieval Fingerprinting Speech Recognition Speaker Identification

15 xvi CONTENTS Spoken Document Retrieval Audio Basics Retrieving and Browsing Video Video Abstracts Static Summaries Mosaics and Salient Stills Dynamic Summaries Interactive Summaries Visual vs. Audio Browsing Evaluating Summaries Fusion Models: Combining it All Naming Faces Naming Images Naming Audio Combining Audio and Video for AVSR Combining Audio and Video for Multimedia Segmentation A Video Segmentation Example Segmentation Schemes for Video Video Segmentation with Edges Speech Segmentation Segmentation Evaluation Compression and MPEG Standards Intensity and Sampling Color Lossy Compression Lossless Compression Temporal Redundancy Motion Prediction MPEG Standards Trends and Research Issues Bibliographic Discussion Enterprise Search 645 by David Hawking 15.1 Introduction Characteristics and Applications of Enterprise Search Enterprise Search Software Workplace Search Enterprise Search Tasks Examples of Search-Supported Tasks Search Types Studying Enterprise Search Architecture of Enterprise Search Systems Gathering Extracting Indexing

16 CONTENTS xvii Indexing Textual Annotations Query Processing Presentation of Search Results Security Models Federation/Metasearch Enterprise Search Evaluation Published Test Collections for Enterprise Search Internal Enterprise Search Evaluations Enterprise Search Tuning What is it Reasonable to Expect? Potential Reasons for Dissatisfaction Context and Personalization Controls and Levers for Contextualization Contextualization: Local, Enterprise or Global? Privacy of Profiles Defining, Creating and Maintaining a Profile User Modeling Implicit Measures Information Filtering Social Recommender Systems Trends and Research Issues Bibliographic Discussion Library Systems 687 by Edie Rasmussen 16.1 The Information Environment in the Library Online Public Access Catalogues OPACs and Bibliographic Records Information Retrieval from the ILS Integrating the Hybrid Library OPACs and End Users ILS: Vendors and Products IR Systems and Document Databases Bibliographic and Full-text Databases Content of Database Records The Online Industry: Database Vendors Information Retrieval from Document Databases Information Retrieval in Organizations Trends and Research Issues Bibliographic Discussion Digital Libraries 713 by Marcos Gonçalves 17.1 Introduction Defining Digital Libraries A General Architecture Fundamentals

17 xviii CONTENTS Digital Objects and Collections Metadata and Catalogs Repositories/Archives Services Social-Economical Issues Social Issues Economical Issues Software Systems Greenstone Eprints DSpace Fedora Open Digital Libraries The 5S Suite DL Case Studies The Networked DL of Theses and Dissertations The National Science Digital Library The ETANA-DL Archaeological Digital Library Trends and Research Issues Evaluation Integration Other Research Challenges Bibliographic Discussion A Open Source Search Engines 739 with Christian Middleton A.1 Introduction A.2 Search Engines A.2.1 Preliminary Selection of Search Engines A.2.2 Features A.2.3 Evaluation A.3 Methodology A.3.1 Document Collections A.3.2 Evaluation Tests A.3.3 Experimental Setup A.4 Experimental Results A.4.1 Test A Indexing A.4.2 Test B Incremental Indexing A.4.3 Test C Search Performance A.4.4 Global Evaluation A.5 Conclusions B Biographies 757 References 765