A Brief Overview of Audio Information Retrieval. Unjung Nam CCRMA Stanford University

Transcription

1 A Brief Overview of Audio Information Retrieval Unjung Nam CCRMA Stanford University 1

2 Outline What is AIR? Motivation Related Field of Research Elements of AIR Experiments and discussion Music Classification System December 2000 by Unjung Nam 2

3 What is AIR? Audio Information Retrieval (AIR): Audio Information: Speech, Music, Natural sounds, etc. To develop various methods in order to recognize the audio information Audio Human Speech? Music? Animal Sound? Clock Alarm? : Computer December 2000 by Unjung Nam 3

4 What is AIR? Applications Speech-related retrieval Recognizing and Transcribing the content of Radio programs, Telephone conversations, Recorded Meetings Music-related retrieval Music similarity, Music style classification, Instrument recognition Others audio retrieval applications Alarms, animal sounds, natural sounds, etc. December 2000 by Unjung Nam 4

5 What is AIR? Muscle Fish Audio Retrieval December 2000 by Unjung Nam 5

6 Motivation Multimedia Information stored on computer systems increases due to Internet. Multimedia database is classified/retrieved in a manual process which is often subjective and inaccurate when describing audio. Multimedia database should be handled by the methods of automatic analysis, segmentation, indexing and retrieval. December 2000 by Unjung Nam 6

7 Related Field of Research Automatic Speech Recognition Overview of the process Audio/Video Classification Video Mail Retrieval Computer Vision Image Retrieval, Face Recognition Multimedia Database Management December 2000 by Unjung Nam 7

8 Automatic Speech Recognition: Overview December 2000 by Unjung Nam 8

9 Audio/Video Classification: Video Mail Retrieval December 2000 by Unjung Nam 9

10 Computer Vision: Image Retrieval December 2000 by Unjung Nam 10

11 Computer Vision: Face Recognition December 2000 by Unjung Nam 11

12 Multimedia Database Management: Characteristics of Multimedia Information Retrieval Content based retrieval Automatic Indexing Similarity Matching Similar content may have different representation Data filtering rather than exact matching (data selection) Browsing and Relevance feedback No ideal mathematical model for defining similarity, human feedback is required December 2000 by Unjung Nam 12

13 Elements of AIR Building Classification Database Audio Signal Feature Extraction Feature Classification/Clustering Retrieval of Best Matching Audio Audio Input Feature Extraction Projection to Model Space Find Matching Model Space Classification Model Space Retrieval of Best Match December 2000 by Unjung Nam 13

14 Feature Extraction Time domain feature modules Short-Time Energy and Average Magnitude Short-Time Average Zero-Crossing Rate Linear Prediction Pulse metric, etc. Spectral domain feature models STFT Spectral Centroid Harmony Analysis MFCC Constant Q, etc. December 2000 by Unjung Nam 14

15 Classification/Clustering Methods Deterministic Methods Minimum Distance Classifier k-nearest neighbor (k-nn) Discriminant functions Generalized Discriminators, etc. Statistical Methods Class-related Probability Functions Minimum Error Classification Likelihood-based MAP Classification Approximating a Bayes Classifier Parameterization and Probability Estimation: Hidden Markov Model, etc. December 2000 by Unjung Nam 15

16 Experiments: Music Classification System Genre category File No. Filename Duration MATLAB Graphic notation (secs.) Jazz 1 dance.wav 11 go 2 rocky.wav 13 gx 3 band.wav 46 g+ Pop/Rock 4 queen.wav 3 gd 5 susqnet.wav 16 * 6 pop.wav 26 v 7 latin.wav 26 * 8 latin2.wav reggae.wav 26 d 10 reggae2.wav 26 ^ Classic 11 phantom.wav 6 ro 12 quartet4.wav 10 rv 13 quartet3.wav 7 rx 14 quartet2.wav 8 r* 15 quartet1.wav 10 r^ 16 musicnight.wav 14 r< 17 angel.wav 9 r> 18 piacel.wav 8 r+ 19 piavio.wav 7 rx 20 piavio1.wav 7 rs Test Signal b3.wav 7 k. December 2000 by Unjung Nam 16

17 Experiments: Feature Modules Spectral Centroid Center Gravity of Spectrum: Brightness of a sound Sound File Preprocessing frame STFT Input & N Windowing Number of N Spectral Centroid The individual centroid of a spectral frame is defined as the average frequency weighted by amplitudes, divided by the sum of the amplitudes, or: Spectral Centroid Here, F [k] is the amplitude corresponding to bin k in DFT spectrum. N k = 1 = N k = 1 kf[ k] F[ k] December 2000 by Unjung Nam 17

18 Experiments: Spectral Centroid Jazz Pop/Rock Classic December 2000 by Unjung Nam 18

19 Experiments: Spectral Centroid The weighted average spectral centroid of each frames in 20 sound files. Green: Blue: Red: jazz pop/rock classic pop/rock higher than classical classical fluctuating alot Spectral Centroid December 2000 by Unjung Nam 19

20 Experiments: Feature Modules Short-Time Energy Function Amplitude variation over the time Rhythm and periodicity information Sound File Input Preprocessing & Windowing frame N Number of N Energy Change En = [ x( m) w( n m ] 2 1 ) N m 1, w( x) = 0, 0 x N 1, otherwise. Where x(m) is the discrete time audio signal, n is time index of the short-time energy, and w(m) is a rectangle window December 2000 by Unjung Nam 20

21 Experiments: Short-Time Energy Function Jazz: dance.wav December 2000 by Unjung Nam 21

22 Experiments: Short-Time Energy Function The pop/rock music samples show the most fluctuating energy while classical music samples shows stable energy fluctuation. The energy changes of jazz samples seem to show medium fluctuation. December 2000 by Unjung Nam 22

23 Experiments: Feature Modules Short-Time Average Zero Crossing Rate Zero-Crossing Rate (ZCR) is a measure of how often the signal crosses zero per unit time. occur if successive samples have different signs. The rate at which zero-crossings occur is a simple measure of the frequency content of a signal. w(n) is a rectangle window of length N 1 Zn = sgn 2 where m sgn [ x( m) ] sgn[ x( m 1) ] [ x( n) ] 1, = 1, x( n) 0, x( n) < 0, w( n m), December 2000 by Unjung Nam 23

24 Experiments: Short-Time Average ZCR Classic: quartet4.wav December 2000 by Unjung Nam 24

25 Experiments: Short-Time Average ZCR It doesn t seem to show any indication of classifying the three different music genres. December 2000 by Unjung Nam 25

26 Experiments: Classification/Clustering Methods K-means clustering algorithm K-means cluster analysis programs begin by creating the K clusters according to some arbitrary procedure. The program calculates the means or centroids of each of the clusters. If one of the observations is closer to the centroid of another cluster, then the observation is made a member of that cluster. K-Nearest Neighbour Classifier (KNN) to classify a feature space with a given set of sample data by evaluating the k nearest sample points of each point in the feature space. December 2000 by Unjung Nam 26

27 Experiments: Classification/Clustering Methods K-means clustering KNN classifier December 2000 by Unjung Nam 27

28 Experiments: Classification/Clustering Methods Feature vectors of the 20 input files get extracted in each frame and got plotted. red: classical blue: pop/rock green: jazz The first figure: spectral centroid in x-axis and short-time energy in y axis. The second: shorttime energy in x-axis and short-time ZCR in y- axis.the third figure: 3 dimensional space December 2000 by Unjung Nam 28

29 Experiments: Classification/Clustering Methods The means of feature vectors of 20 music samples are plotted in 3 dimensional space December 2000 by Unjung Nam 29

30 Experiments: Classification/Clustering Methods The feature vectors of the test signal b3.wav gets plotted as black dots in 2 dimensional space. December 2000 by Unjung Nam 30

31 Experiments: Classification/Clustering Methods The nearest neighbours are determined using Euclidean distance. Each mean of the 20 sound samples gets the predicted class labels as an index 1 to 20. Each of the feature vectors of the test signal is assigned to one of 20 means. The test signal in the figure above determined that the number of feature vectors assigned to 13 is the greatest. The following describes the result in MATLAB. Test 13. Quartet3.wav >> classfier ans = class is 13 >> classpoint classpoint = >> % 27 feature vectors assigned to 13 th class December 2000 by Unjung Nam 31

32 Experiments: Discussion Though the test signal and the quartet3.wav are not fell into a same category, they sounded similar in terms of rhythm and tempo information. It seems that this system doesn t effectively classify the timbre information. The number of feature modules are limited in this system. The variance factor of the feature vectors is not considered. Need to experiment with more samples. December 2000 by Unjung Nam 32