Information Retrieval for Music and Motion

Transcription

1 Meinard Miiller Information Retrieval for Music and Motion With 136 Figures, 39 in color and 26 Tables ^y Springer

2 Contents 1 Introduction Music Information Retrieval Outline of Part I Further Notes Motion Retrieval Outline of Part II Further Notes General Concepts 10 Part I Analysis and Retrieval Techniques for Music Data 2 Fundamentals on Music and Audio Data Music Representations Score Representation Sound, Waveform, and Audio Representation MIDI Representation Fourier Transform Signals and Signal Spaces Fourier Representations Discrete Fourier Transform Further Notes Digital Filters Convolution Filters Frequency Response Filter Specifications Examples Further Notes 48

3 XII Contents 3 Pitch- and Chroma-Based Audio Features Pitch Features Local Energy (STMSP) and Onset Features Chroma and CENS Features Further Notes 65 4 Dynamic Time Warping Classical DTW Variations of DTW Step Size Condition Local Weights Global Constraints Approximations Multiscale DTW Subsequence DTW Further Notes 82 5 Music Synchronization Synchronization Tasks A Multiscale Approach to Audio Synchronization Audio Features Local Cost Measure Resolution Levels and J-Neighborhood Experimental Results Enhancing Cost Matrices Onset-Based Score-Audio Synchronization Audio Features Local Similarity Measure Matching Model and Alignment Efficiency and Anchor Matches Experimental Results Further Notes Audio Matching Diagonal Audio Matching Audio Features Basic Matching Procedure Ill Global Variations in Tempo and Key Efficient Implementation Experimental Results for Diagonal Audio Matching Representative Matching Results Dependence on Query Length Codebook Selection for CENS Features Codebook Selection by Unsupervised Learning Codebook Selection Based on Musical Knowledge 125

4 Contents XIII 6.4 Index-Based Audio Matching Exact Matches Fault Tolerance Mechanisms Retrieval Scenario and Ranking Experimental Results for Index-Based Audio Matching Diagonal vs. Index-Based Matching Comparison of Codebooks Running Times Further Notes 137 Audio Structure Analysis General Strategy and Notation Enhancing Similarity Matrices Path Extraction Global Structure Analysis Experiments General Results Running Time Behavior Transpositions Further Notes 165 SyncPlayer: An Advanced Audio Player Overview SyncPlayer Server SyncPlayer Client Further Notes 180 Part II Analysis and Retrieval Techniques for Motion Data 9 Fundamentals on Motion Capture Data Motion Capture Data Similarity Aspects Kinematic Chains Formal Definition Forward Kinematics Animated Kinematic Chains Further Notes Rotations Basic Definitions and Properties Euler Angles Quaternions Further Notes 208

5 XIV Contents 10 DTW-Based Motion Comparison and Retrieval Local Distance Measures Quaternion-Based Pose Distance D Point Cloud Distance Examples DTW-Based Motion Warping Motion Retrieval Based on Self-Similarity Further Notes Relational Features and Adaptive Segmentation Relational Features A Basic Example Generic Features Threshold Selection Example for Some Feature Set Adaptive Segmentation Further Notes Index-Based Motion Retrieval Indexing, Queries, and Hits Inverted File Index Exact Queries and Exact Hits Fuzzy Queries and Fuzzy Hits Adaptive Fuzzy Hits QBE Motion Retrieval Query and Retrieval Mechanism Indexing Stage Experimental Results Ranking Strategies Further Applications Accelerating DTW-Based Motion Alignment Keyframe-Based Preselection Toward Scene Descriptions Further Notes Problems and Limitations Related Work Motion Templates Basic Definitions and Properties MT Learning Procedure: Overview MT Learning Procedure: Technical Details Reference-Based WMT Warping WMT Unwarping Reference-Based WMT Averaging Reference-Free WMT Averaging 273

6 Contents XV 13.4 Experimental Results Motion Class Database WMT Computation and Time Complexity Examples Further Notes MT-Based Motion Annotation and Retrieval MT-Based Matching MT-Based Annotation MT-Based Retrieval Keyframe-Based Preprocessing Comparison to Other Retrieval Methods Further Notes 293 References 297 Index 309