INFORMATION RETRIEVAL: SEARCHING IN THE 21ST CENTURY

Size: px

Start display at page:

Download "INFORMATION RETRIEVAL: SEARCHING IN THE 21ST CENTURY"

Adrian Rich
6 years ago
Views:

1 INFORMATION RETRIEVAL: SEARCHING IN THE 21ST CENTURY Ayşe Göker City University London, UK John Davies BT, UK A John Wiley and Sons, Ltd., Publication

3 INFORMATION RETRIEVAL

5 INFORMATION RETRIEVAL: SEARCHING IN THE 21ST CENTURY Ayşe Göker City University London, UK John Davies BT, UK A John Wiley and Sons, Ltd., Publication

6 This edition first published John Wiley & Sons, Ltd. Except for Chapter 3, Multimedia Resource Discovery 2009 Stefan Rüger. Registered office John Wiley & Sons, Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at The right of the authors to be identified as the authors of this work has been asserted in accordance with the Copyright, Designs and Patents Act All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Library of Congress Cataloging-in-Publication Data Information retrieval : searching in the 21st century / [edited by] Ayşe Göker, John Davies. p. cm. Includes bibliographical references and index. ISBN Information retrieval. I. Göker, Ayşe. II. Davies, J. (N. John) ZA3075.I dc A catalogue record for this book is available from the British Library. ISBN: (H/B) Set in 9/11 Times New Roman by Laserwords Private Ltd, Chennai, India. Printed and bound in Great Britain by Antony Rowe Ltd, Chippenham, Wiltshire.

7 Dedication I would like to thank my husband, parents and wider family for all their support to my efforts to reach my full potential. This book is one of the tangible outcomes of this process. My father, in particular, would have loved to have seen it. Ayşe Göker

9 Contents Foreword Preface About the Editors List of Contributors Introduction xiii xv xvii xix xxi 1 Information Retrieval Models 1 Djoerd Hiemstra 1.1 Introduction Terminology What is a model? Outline Exact Match Models The Boolean model Region models Discussion Vector Space Approaches The vector space model Positioning the query in vector space Term weighting and other caveats Probabilistic Approaches The probabilistic indexing model The probabilistic retrieval model The 2-Poisson model Bayesian network models Language models Google s PageRank model Summary and Further Reading 15 Exercises 16 References 18 2 User-centred Evaluation of Information Retrieval Systems 21 Pia Borlund 2.1 Introduction Background Chapter outline The MEDLARS Test 23

10 viii Contents Description of Lancaster s test of MEDLARS Evaluation characteristics The Okapi Project The objectives of Okapi Okapi at TREC The impact of Okapi The Interactive IR Evaluation Model The cognitive IR approach The three parts of the IIR evaluation model Summary 31 Exercises 34 References 35 3 Multimedia Resource Discovery 39 Stefan Rüger 3.1 Introduction Basic Multimedia Search Technologies Piggy-back text retrieval Automated annotation Content-based retrieval Fingerprinting Challenges of Automated Visual Indexing Added Services Video summaries New paradigms in information visualisation Visual search and relevance feedback Browsing: Lateral and Geotemporal Summary 55 Exercises 57 References 59 4 Image Users Needs and Searching Behaviour 63 Stina Westman 4.1 Introduction Image Attributes and Users Needs Image attributes Image attributes in queries Attributes beyond queries Image needs Image Searching Behaviour Search process Search strategies Relevance criteria New Directions for Image Access Social tagging Images in context Visualisations Workspaces Summary 78 Exercise 80 References 81

11 Contents ix 5 Web Information Retrieval 85 Nick Craswell and David Hawking 5.1 Introduction Distinctive Characteristics of the Web Web data Web structure User behaviour User interaction data Three Ranking Problems Retrieval Selective crawling Index organisation Other Web IR Issues Stemming Treatment of near-duplicate content Spelling suggestions Spam rejection Adult content filtering genre classification Query-targeted advertisement generation Snippet generation Context and web information retrieval Evaluation of Web Search Effectiveness TREC-9 Web Track: Realistic queries, rich link structure, traditional IR task Evaluation using web-specific tasks Future directions for web IR evaluations Comparing results lists in context Evaluation by commercial web search companies Summary 97 Exercises 98 References 99 6 Mobile Search 103 David Mountain, Hans Myrhaug and Ayşe Göker 6.1 Introduction: Mobile Search Why Now? Technological drivers Predicted demand for mobile search Information for Mobile Search Linking information to physical space The storage of information The ownership of information Designing for Mobile Search Characteristics of mobile usage Filters as a framework for mobile search Manual versus automatic filtering Using filters to push information Case Studies Oslo airport AmbieSense The Swiss Alps WebPark Summary 125 Exercises 127 References 129

12 x Contents 7 Context and Information Retrieval 131 Ayşe Göker, Hans Myrhaug and Ralf Bierig 7.1 Introduction What is Context? Whose context? Context in Information Retrieval Context in the wider sense Perceptions of context in related fields Example: context and images Context Modelling and Representation Context modelling User models and their relationship to context Past, present and future contexts Context and Content Representation of context Capturing context Searching with context information Context templates Related Topics Personalisation and context Mobility and context Evaluating Context-aware IR Systems Principles of methodology Summary 150 Exercises 151 References Text Categorisation and Genre in Information Retrieval 159 Stuart Watt 8.1 Introduction: What is Text Categorisation? Purpose of categorisation How to Build a Text Categorisation System The classifier component The machine learning component The feature selection component Evaluating Text Categorisation Systems Genre: Text Structure and Purpose An overview of genre Text categorisation and genre The importance of layout Related Techniques: Information Filtering Applications of Text Categorisation Summary and the Future of Text Categorisation 174 Exercises 175 References Semantic Search 179 John Davies, Alistair Duke and Atanas Kiryakov 9.1 Introduction Limitations of current search technology Ontologies Knowledge bases and semantic repositories 182

13 Contents xi 9.2 Semantic Web Semantic web and semantic search Basic semantic web standards: RDF(S) and OWL Metadata and Annotations Semantic Annotations: the Fibres of the Semantic Web Semantic Annotation of Named Entities Named entities Semantic annotation model and representation Semantic Indexing and Retrieval Indexing with respect to lexical concepts Indexing with respect to named entities Retrieval as spreading activation over semantic network Semantic Search Tools Searching through document-level RDF annotations QuizRDF Exploiting massive background knowledge TAP Character-level annotations and massive world knowledge KIM Squirrel Other approaches Summary 208 Exercises 210 References The Role of Natural Language Processing in Information Retrieval: Searching for Meaning and Structure 215 Tony Russell-Rose and Mark Stevenson 10.1 Introduction Natural Language Processing Techniques Named entity recognition Information extraction WordNet Word sense disambiguation Evaluation Applications of Natural Language Processing in Information Retrieval Text mining Question answering Discussion Summary 226 Exercises 228 References Cross-Language Information Retrieval 233 Daqing He and Jianqiang Wang 11.1 Introduction Major Approaches and Challenges in CLIR Identifying Translation Units Tokenisation Stemming Phrase identification Stop-words Obtaining Translation Knowledge Obtaining bilingual dictionaries and corpora Extracting translation knowledge 238

14 xii Contents Dealing with out-of-vocabulary terms Using Translation Knowledge Translation disambiguation Weighting translation alternatives Using translation probabilities in term weighting Interactivity in CLIR Interactive CLIR Query translation in interactive CLIR Document selection in interactive CLIR Evaluation of CLIR Systems Cranfield-based evaluation framework Evaluations on interactive CLIR Current CLIR evaluation frameworks Summary and Future Directions Current achievements in CLIR Future directions for CLIR Further reading 246 Exercises 247 References Performance Issues in Parallel Computing for Information Retrieval 255 Andrew MacFarlane 12.1 Introduction Why Parallel IR? Review of Previous Work Distribution Methods for Inverted File Data On-the-fly distribution Inverted file replication Inverted file partitioning Tasks in Information Retrieval The indexing task The probabilistic search task The passage retrieval task The routing/filtering task The index update task A Synthetic Model of Performance for Parallel Information Retrieval Empirical Examination of Synthetic Model Comparative results using indexing models Comparative results using search models Comparative results using passage retrieval models Comparative results using term selection models Comparative results using index update model Summary and Further Research 269 Exercises 270 References 271 Solutions to Exercises 273 Index 285

15 Foreword In the forty years since I started working in the field, and indeed for some years before that (almost since Calvin Mooers coined the term information storage and retrieval in the 1950s), there have been a significant number of books on information retrieval. Even if we ignore the more specialist research monographs and the readers of previously published papers, I can find on my shelves or in my mental library many books that attempt (probably with the IR student in mind) to construct a coherent and systematic way of defining and presenting information retrieval as a field of study and of application. Often such a book is the work of a single author, or perhaps a pair working together. Such works can clearly have an advantage in respect of coherence; the field is necessarily presented from a single viewpoint. On the other hand, they can also suffer for the same reason. The IR field is rich (more so now than it has ever been), and it is difficult within a single viewpoint to do justice to this richness. Readers, on the other hand, have to be constructed out of the materials to hand: the published papers, each of which has taken its own view, probably with a much narrower field of vision, and different from that of the other chosen papers. The present book attempts the tricky task of combining the breadth of vision of multiple authors with the coherence of a single integrated work. The richness of the field is apparent in the range of chapters: from formal mathematical modelling to user context, from parallel computation to semantic search. The topics covered also vary greatly in their historical association with the field. Categorisation,for example, has been around as an IR technique for quite a long time though Stuart Watt brings a new perspective. Mobile search (David Mountain, Hans Myrhaug and Ayşe Göker), however, is a relatively recent development. The use of formal models (information retrieval models, Djoerd Hiemstra) goes back almost to the beginning, as does experimental evaluation (user-centred evaluation of information retrieval systems, Pia Borlund), though in both cases there have been huge changes in the past decade. This same decade has witnessed the huge growth of the World Wide Web, and the developing dominance of web search engines (web information retrieval, Nick Craswell and David Hawking) as the glue which holds the web together. For many people today, IR is web search. It is true that there has been a huge amount of influence in both directions: search engines are largely based on techniques from both the IR research community and from previous operational systems, while IR research and practice in other environments has learnt a great deal from the forcing-house that is the web search space. This dominance of the web as the domain of interest is well reflected in many of the chapters in the present volume. It is important, however, to remember that IR is not all about web search, and that the web space presents both problems and opportunities which differ from those in other domains. The desktop, the enterprise, specialist collections such as scientific papers are all examples of different domains for which search functionality is a fundamental requirement. There are references to several of these throughout the book, but specific domains with their own chapters are multimedia resource discovery (Stefan Rüger) and image users needs and searching behaviour (Stina Westman). The user theme is taken further in the context and information retrieval (Ayşe Göker, Hans Myrhaug and Ralf Bierig). More generic problem areas are addressed in cross-language information retrieval (Daqing He and Jianqiang Wang), in semantic search (John Davies, Alistair Duke and Atanas Kiryakov) and in the

INFORMATION RETRIEVAL: SEARCHING IN THE 21ST CENTURY

INFORMATION RETRIEVAL: SEARCHING IN THE 21ST CENTURY Ayşe Göker City University London, UK John Davies BT, UK A John Wiley and Sons, Ltd., Publication INFORMATION RETRIEVAL INFORMATION RETRIEVAL: SEARCHING