CONTENT-BASED IMAGE AND VIDEO RETRIEVAL

Transcription

1 CONTENT-BASED IMAGE AND VIDEO RETRIEVAL

2 MULTIMEDIA SYSTEMS AND APPLICATIONS SERIES Recently Published Titles: Consulting Editor Borko Furht Florida Atlantic University ELECTRONIC BUSINESS AND EDUCATION: Recent Advances in Internet Infrastructures, edited by Wendy Chin, Frederic Patricelli, Veljko Milutinovic^; ISBN: INFRASTRUCTURE FOR ELECTRONIC BUSINESS ON THE INTERNET by Veljko Milutinovic; ISBN: DELIVERING MPEG-4 BASED AUDIO-VISUAL SERVICES by Hari Kalva; ISBN: CODING AND MODULATION FOR DIGITAL TELEVISION by Gordon Drury, Garegin Markarian, Keith Pickavance; ISBN: CELLULAR AUTOMATA TRANSFORMS: Theory and Applications in Multimedia Compression, Encryption, and Modeling, by Olu Lafe; ISBN: COMPUTED SYNCHRONIZATION FOR MULTIMEDIA APPLICATIONS, by Charles B. Owen and Fillia Makedon; ISBN: STILL IMAGE COMPRESSION ON PARALLEL COMPUTER ARCHITECTURES, by Savitri Bevinakoppa; ISBN: INTERACTIVE VIDEO-ON-DEMAND SYSTEMS: Resource Management and Scheduling Strategies, by T. P. Jimmy To and Babak Hamidzadeh; ISBN: MULTIMEDIA TECHNOLOGIES AND APPLICATIONS FOR THE 21st CENTURY: Visions of World Experts, by Borko Furht; ISBN: INTELLIGENT IMAGE DATABASES: Towards Advanced Image Retrieval, by Yihong Gong; ISBN: BUFFERING TECHNIQUES FOR DELIVERY OF COMPRESSED VIDEO IN VIDEO-ON-DEMAND SYSTEMS, by Wu-chi Feng; ISBN: HUMAN FACE RECOGNITION USING THIRD-ORDER SYNTHETIC NEURAL NETWORKS, by Okechukwu A. Uwechue, and Abhijit S. Pandya; ISBN: MULTIMEDIA INFORMATION SYSTEMS, by Marios C. Angelides Schahram Dustdar; ISBN: MOTION ESTIMATION ALGORITHMS FOR VIDEO COMPRESSION, by Borko Furht, Joshua Greenberg and Raymond Westwater; ISBN: VIDEO DATA COMPRESSION FOR MULTIMEDIA COMPUTING, edited by Hua Harry Li, Shan Sun, Haluk Derin; ISBN: and

3 CONTENT-BASED IMAGE AND VIDEO RETRIEVAL by Oge Marques Borko Furht Florida Atlantic University Boca Raton, FL, U.S.A. SPRINGER SCIENCE+BUSINESS MEDIA, LLC

4 ISBN ISBN (ebook) DOI / Library of Congress Cataloging-in-Publication Data A CLP. Catalogue record for this book is available from the Library of Congress. Copyright 2002 by Springer Science+Business Media New York Originally published by Kluwer Academic Publishers in 2002 Softcover reprint of the hardcover 1st edition 2002 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo-copying, recording, or otherwise, without the prior written permission of the publisher. Printed on acid-free paper.

5 Contents Preface Acknowledgments 1. INTRODUCTION 1 2. FUNDAMENTALS OF CONTENT-BASED IMAGE AND VIDEO RETRIEVAL 7 1. Basic Concepts 7 2. A Typical CBIVR System Architecture 9 3. The User's Perspective Summary DESIGNING A CONTENT-BASED IMAGE RETRIEVAL SYSTEM Feature Extraction and Representation Feature Classification and Selection Color-Based Features Color Models Representation of Color Properties Other Parameters Additional Remarks Texture-Based Features Shape-Based Features Specialized Features Similarity Measurements Dimension Reduction and High-dimensional Indexing Clustering The Semantic Gap Learning Relevance Feedback (RF) 30 IX XIll

6 vi CONTENT-BASED IMAGE AND VIDEO RETRIEVAL 8. Benchmarking CBVIR Solutions Design Questions Summary DESIGNING A CONTENT-BASED VIDEO RETRIEVAL SYSTEM The Problem The Solution Video Parsing Shot Boundary Detection Scene Boundary Detection Video Abstraction and Summarization Key-frame Extraction "Highlight" Sequences Video Content Representation, Indexing, and Retrieval Video Browsing Schemes Examples of Video Retrieval Systems VideoQ Screening Room Virage Summary A SURVEY OF CONTENT-BASED IMAGE RETRIEVAL SYSTEMS 47 by Remco C. Veltkamp and Mirela Tanase 1. Introduction Criteria Systems ADL (Alexandria Digital Library) AMORE (Advanced Multimedia Oriented Retrieval Engine) ASSERT BDLP (Berkeley Digital Library Project) Blobworld CANDID (Comparison Algorithm for Navigating Digital Image Databases) C-bird (Content-Based Image Retrieval from Digital libraries) CBVQ (Content-Based Visual Query) Chabot CHROMA (Colour Hierarchical Representation Oriented Management Architecture) 57

7 Contents vii 3.11 DrawSearch FIDS (Flexible Image Database System) FIR (Formula Image Retrieval) FOCUS (Fast Object Color-based Query System) ImageRETRO (Image RETrieval by Reduction and Overview) ImageRover ImageScape JACOB (Just A COntent Based query system for video databases) LCPD (Leiden 19th Century Portrait Database) MARS (Multimedia Analysis and Retrieval System) MetaSEEk MIR (Multimodal Information Retrieval System) NETRA Photobook Picasso PicHunter PicSOM PicToSeek QBIC (Query By Image Content) Quicklook Shoebox SIMBA (Search Images By Appearance) SMURF (Similarity-based Multimedia Retrieval Framework) SQUID (Shape Queries Using Image Databases) Surfimage SYNAPSE (SYNtactic APpearance Search Engine) TODAI (Typographic Ornament Database And Identification) VIR Image Engine VisualSEEk Web SEEk WebSeer WISE (Wavelet Image Search Engine) Summary and Conclusions CASE STUDY: MUSE 103 l. Overview of the System The User's Perspective The RF Mode 109

8 viii CONTENT-BASED IMAGE AND VIDEO RETRIEVAL 3.1 Features Probabilistic Model The RFC Mode More and Better Features Clustering Learning A Numerical Example Display Update Strategy Experiments and Results Testing the System in RF t-.fode Preliminary Tests Increasing Database Size Improving the Color-Based Feature Set Evaluating the Influence of the Number of Images per Iteration Testing the Relationship Between the User's Style and System Performance A Note About Convergence Testing Features and Distance Measurements Goals and Methodology Color-Based Methods Shape or Texture Only Combining Color, Texture, and Shape Distance Measures Testing the Clustering Algorithm Testing the System in RFC Mode Preliminary Tests Tests Using a Small Database Increasing Database Size Mixed Mode Tests Summary Future Work 159 References 163 Index 181

9 Preface The main objective of this book is to excite the curiosity of its readers and inspire new technological breakthroughs in the exciting field of image and video retrieval systems. Only a few years ago there was a small group of researchers and practitioners in the field of content-based image and video retrieval techniques. Today, this area has become one of the emerging areas of research in the multimedia field, with a large number of applications. These new applications, such as digital libraries, video-on-demand systems, geographic information systems, astronomical research, satellite observation systems, criminal investigation systems and others, need to store large collections of multimedia data consisting of images, video, and audio data. The challenge is to exploit these large multimedia databases by content. Another challenge is the World Wide Web, which is a very large distributed information database. Most current search and retrieval systems use keywords only. However, in order to realize the full potential of the Web and search and retrieve multimedia data from the Web effectively requires adding the content-based retrieval techniques. This book addresses the basic concepts and techniques for designing content-based image and video retrieval systems. It also discusses a variety of design choices for the key components of these systems. The book gives a comprehensive survey of the content-based image retrieval systems including several content-based video retrieval systems. The survey includes both research and commercial content-based retrieval systems. Finally, the book presents a detailed case study of designing MUSE - a content-based image retrieval system developed at Florida Atlantic University in Boca Raton, Florida. The book also contains pointers to two hundred representative bibliographic references on this field - ranging from survey papers to de-

10 x CONTENT-BASED IMAGE AND VIDEO RETRIEVAL scription of recent work in the area, to entire books - and more than seventy Web sites. A Web page for the book, containing color versions of representative images, updated useful links, and supplemental materials is available at: 0GE MARQUES AND BORKO FURHT

11 To my son Nicholas. - Oge Marques To my daughter Tanya. - Borko Furht

12 Acknow ledgments Our thanks to Susan Lagerstrom-Fife and Jennifer Evans from Kluwer Academic Publishers, for their support to this project. We are deeply indebted to Dr. Remco C. Veltkamp and Mirela Tanase, for contributing with their excellent and comprehensive survey to what became Chapter 5 of this book. We would also like to acknowledge the support of our institution - Florida Atlantic University - and the funding provided by CAPES (Brasilia-DF, Brazil) along the development of the MUSE Project, reported in Chapter 6. Thanks to the members of the Visual Information Retrieval group at Florida Atlantic University. Special thanks to James A. Russo, for his technical expertise, enthusiasm and willingness to always help, and Hugo Vieira Neto, for patiently reviewing earlier versions of portions of this manuscript. Finally, and most importantly, a very heartfelt thank-you to our wives and children, for their constant support, encouragement, patience and understanding.