MPEG-7 Audio and Beyond

Transcription

1 MPEG-7 Audio and Beyond Audio Content Indexing and Retrieval Hyoung-Gook Kim Samsung Advanced Institute of Technology, Korea Nicolas Moreau Technical University of Berlin, Germany Thomas Sikora Communication Systems Group, Technical University of Berlin, Germany

2

3 MPEG-7 Audio and Beyond

4

5 MPEG-7 Audio and Beyond Audio Content Indexing and Retrieval Hyoung-Gook Kim Samsung Advanced Institute of Technology, Korea Nicolas Moreau Technical University of Berlin, Germany Thomas Sikora Communication Systems Group, Technical University of Berlin, Germany

6 Copyright 2005 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) (for orders and customer service enquiries): Visit our Home Page on All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or ed to permreq@wiley.co.uk, or faxed to This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA , USA Wiley-VCH Verlag GmbH, Boschstr. 12, D Weinheim, Germany John Wiley & Sons Australia Ltd, 42 McDougall Street, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Cataloging in Publication Data Kim, Hyoung-Gook. Introduction to MPEG-7 audio / Hyoung-Gook Kim, Nicolas Moreau, Thomas Sikora. p. cm. Includes bibliographical references and index. ISBN (cloth: alk. paper) ISBN X (cloth: alk. paper) 1. MPEG (Video coding standard) 2. Multimedia systems. 3. Sound Recording and reproducing Digital techniques Standards. I. Moreau, Nicolas. II. Sikora, Thomas. III. Title. TK K dc22 British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN (HB) ISBN X (HB) Typeset in 10/12pt Times by Integra Software Services Pvt. Ltd, Pondicherry, India Printed and bound in Great Britain by TJ International Ltd, Padstow, Cornwall This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production.

7 Contents List of Acronyms List of Symbols xi xv 1 Introduction Audio Content Description MPEG-7 Audio Content Description An Overview MPEG-7 Low-Level Descriptors MPEG-7 Description Schemes MPEG-7 Description Definition Language (DDL) BiM (Binary Format for MPEG-7) Organization of the Book 10 2 Low-Level Descriptors Introduction Basic Parameters and Notations Time Domain Frequency Domain Scalable Series Series of Scalars Series of Vectors Binary Series Basic Descriptors Audio Waveform Audio Power Basic Spectral Descriptors Audio Spectrum Envelope Audio Spectrum Centroid Audio Spectrum Spread Audio Spectrum Flatness Basic Signal Parameters Audio Harmonicity Audio Fundamental Frequency 36

8 vi CONTENTS 2.7 Timbral Descriptors Temporal Timbral: Requirements Log Attack Time Temporal Centroid Spectral Timbral: Requirements Harmonic Spectral Centroid Harmonic Spectral Deviation Harmonic Spectral Spread Harmonic Spectral Variation Spectral Centroid Spectral Basis Representations Silence Segment Beyond the Scope of MPEG Other Low-Level Descriptors Mel-Frequency Cepstrum Coefficients 52 References 55 3 Sound Classification and Similarity Introduction Dimensionality Reduction Singular Value Decomposition (SVD) Principal Component Analysis (PCA) Independent Component Analysis (ICA) Non-Negative Factorization (NMF) Classification Methods Gaussian Mixture Model (GMM) Hidden Markov Model (HMM) Neural Network (NN) Support Vector Machine (SVM) MPEG-7 Sound Classification MPEG-7 Audio Spectrum Projection (ASP) Feature Extraction Training Hidden Markov Models (HMMs) Classification of Sounds Comparison of MPEG-7 Audio Spectrum Projection vs. MFCC Features Indexing and Similarity Audio Retrieval Using Histogram Sum of Squared Differences Simulation Results and Discussion Plots of MPEG-7 Audio Descriptors Parameter Selection Results for Distinguishing Between Speech, Music and Environmental Sound 91

9 CONTENTS vii Results of Sound Classification Using Three Audio Taxonomy Methods Results for Speaker Recognition Results of Musical Instrument Classification Audio Retrieval Results Conclusions 100 References Spoken Content Introduction Automatic Speech Recognition Basic Principles Types of Speech Recognition Systems Recognition Results MPEG-7 SpokenContent Description General Structure SpokenContentHeader SpokenContentLattice Application: Spoken Document Retrieval Basic Principles of IR and SDR Vector Space Models Word-Based SDR Sub-Word-Based Vector Space Models Sub-Word String Matching Combining Word and Sub-Word Indexing Conclusions MPEG-7 Interoperability MPEG-7 Flexibility Perspectives 166 References Music Description Tools Timbre Introduction InstrumentTimbre HarmonicInstrumentTimbre PercussiveInstrumentTimbre Distance Measures Melody Melody Meter Scale Key 181

10 viii CONTENTS MelodyContour MelodySequence Tempo AudioTempo AudioBPM Application Example: Query-by-Humming Monophonic Melody Transcription Polyphonic Melody Transcription Comparison of Melody Contours 200 References Fingerprinting and Audio Signal Quality Introduction Audio Signature Generalities on Audio Fingerprinting Fingerprint Extraction Distance and Searching Methods MPEG-7-Standardized AudioSignature Audio Signal Quality AudioSignalQuality Description Scheme BroadcastReady IsOriginalMono BackgroundNoiseLevel CrossChannelCorrelation RelativeDelay Balance DcOffset Bandwidth TransmissionTechnology ErrorEvent and ErrorEventList 226 References Application Introduction Automatic Audio Segmentation Feature Extraction Segmentation Metric-Based Segmentation Model-Selection-Based Segmentation Hybrid Segmentation Hybrid Segmentation Using MPEG-7 ASP Segmentation Results 250

11 CONTENTS ix 7.3 Sound Indexing and Browsing of Home Video Using Spoken Annotations A Simple Experimental System Retrieval Results Highlights Extraction for Sport Programmes Using Audio Event Detection Goal Event Segment Selection System Results A Spoken Document Retrieval System for Digital Photo Albums 265 References 266 Index 271

12

13 Acronyms ADSR AFF AH AP ASA ASB ASC ASE ASF ASP ASR ASS AWF BIC BP BPM CASA CBID CM CMN CRC DCT DDL DFT DP DS DSD DTD EBP ED EM EMIM Attack, Decay, Sustain, Release Audio Fundamental Frequency Audio Harmonicity Audio Power Auditory Scene Analysis Audio Spectrum Basis Audio Spectrum Centroid Audio Spectrum Envelope Audio Spectrum Flatness Audio Spectrum Projection Automatic Speech Recognition Audio Spectrum Spread Audio Waveform Bayesian Information Criterion Back Propagation Beats Per Minute Computational Auditory Scene Analysis Content-Based Audio Identification Coordinate Matching Cepstrum Mean Normalization Cyclic Redundancy Checking Discrete Cosine Transform Description Definition Language Discrete Fourier Transform Dynamic Programming Description Scheme Divergence Shape Distance Document Type Definition Error Back Propagation Edit Distance Expectation and Maximization Expected Mutual Information Measure

14 xii ACRONYMS EPM Exponential Pseudo Norm FFT Fast Fourier Transform GLR Generalized Likelihood Ratio GMM Gaussian Mixture Model GSM Global System for Mobile Communications HCNN Hidden Control Neural Network HMM Hidden Markov Model HR Harmonic Ratio HSC Harmonic Spectral Centroid HSD Harmonic Spectral Deviation HSS Harmonic Spectral Spread HSV Harmonic Spectral Variation ICA Independent Component Analysis IDF Inverse Document Frequency INED Inverse Normalized Edit Distance IR Information Retrieval ISO International Organization for Standardization KL Karhunen Loève KL Kullback Leibler KS Knowledge Source LAT Log Attack Time LBG Linde Buzo Gray LD Levenshtein Distance LHSC Local Harmonic Spectral Centroid LHSD Local Harmonic Spectral Deviation LHSS Local Harmonic Spectral Spread LHSV Local Harmonic Spectral Variation LLD Low-Level Descriptor LM Language Model LMPS Logarithmic Maximum Power Spectrum LP Linear Predictive LPC Linear Predictive Coefficient LPCC Linear Prediction Cepstrum Coefficient LSA Log Spectral Amplitude LSP Linear Spectral Pair LVCSR Large-Vocabulary Continuous Speech Recognition map Mean Average Precision MCLT Modulated Complex Lapped Transform MD5 Message Digest 5 MFCC Mel-Frequency Cepstrum Coefficient MFFE Multiple Fundamental Frequency Estimation MIDI Music Instrument Digital Interface MIR Music Information Retrieval MLP Multi-Layer Perceptron

15 ACRONYMS xiii M.M. MMS MPEG MPS MSD NASE NMF NN OOV OPCA PCA PCM PCM PLP PRC PSM QBE QBH RASTA RBF RCL RMS RSV SA SC SCP SDR SF SFM SNF SOM STA STFT SVD SVM TA TPBM TC TDNN ULH UM UML VCV VQ Metronom Mälzel Multimedia Mining System Moving Picture Experts Group Maximum Power Spectrum Maximum Squared Distance Normalized Audio Spectrum Envelope Non-Negative Matrix Factorization Neural Network Out-Of-Vocabulary Oriented Principal Component Analysis Principal Component Analysis Phone Confusion Matrix Pulse Code Modulated Perceptual Linear Prediction Precision Probabilistic String Matching Query-By-Example Query-By-Humming Relative Spectral Technique Radial Basis Function Recall Root Mean Square Retrieval Status Value Spectral Autocorrelation Spectral Centroid Speaker Change Point Spoken Document Retrieval Spectral Flux Spectral Flatness Measure Spectral Noise Floor Self-Organizing Map Spectro-Temporal Autocorrelation Short-Time Fourier Transform Singular Value Decomposition Support Vector Machine Temporal Autocorrelation Time Pitch Beat Matching Temporal Centroid Time-Delay Neural Network Upper Limit of Harmonicity Ukkonen Measure Unified Modeling Language Vowel Consonant Vowel Vector Quantization