MUSI-6201 Computational Music Analysis

Transcription

1 MUSI-6201 Computational Music Analysis Part 9.2: Music Similarity and Mood Recognition alexander lerch November 11, 2015

2 temporal analysis overview text book Chapter 8: Musical Genre, Similarity, and Mood (pp ) sources: slides (latex) & Matlab github repository lecture content music similarity k-means clustering and SOMs mood and emotion linear regression

3 temporal analysis overview text book Chapter 8: Musical Genre, Similarity, and Mood (pp ) sources: slides (latex) & Matlab github repository lecture content music similarity k-means clustering and SOMs mood and emotion linear regression

4 temporal analysis overview text book Chapter 8: Musical Genre, Similarity, and Mood (pp ) sources: slides (latex) & Matlab github repository lecture content music similarity k-means clustering and SOMs mood and emotion linear regression

5 temporal analysis overview text book Chapter 8: Musical Genre, Similarity, and Mood (pp ) sources: slides (latex) & Matlab github repository lecture content music similarity k-means clustering and SOMs mood and emotion linear regression

6 temporal analysis overview text book Chapter 8: Musical Genre, Similarity, and Mood (pp ) sources: slides (latex) & Matlab github repository lecture content music similarity k-means clustering and SOMs mood and emotion linear regression

7 similarity and mood introduction commonalities with genre classification similar set of features ambiguous ground truth unclear value/impact of low level and high level features differences to genre classification mood: often not a classification but regression similarity: distance measure instead of categorizing into classes

8 similarity and mood introduction commonalities with genre classification similar set of features ambiguous ground truth unclear value/impact of low level and high level features differences to genre classification mood: often not a classification but regression similarity: distance measure instead of categorizing into classes

9 audio similarity introduction perception of music similarity multi-dimensional (melodic, rhythmic, sound quality,... ) user dependent associative, may also depend on editorial data may be context dependent genres are clusters of musical similarity genre classification is a special case of audio similarity measures instead of assigning (genre) labels, the similarity/distance between (pairs) of files is measured

10 audio similarity introduction perception of music similarity multi-dimensional (melodic, rhythmic, sound quality,... ) user dependent associative, may also depend on editorial data may be context dependent genres are clusters of musical similarity genre classification is a special case of audio similarity measures instead of assigning (genre) labels, the similarity/distance between (pairs) of files is measured

11 audio similarity introduction perception of music similarity multi-dimensional (melodic, rhythmic, sound quality,... ) user dependent associative, may also depend on editorial data may be context dependent genres are clusters of musical similarity genre classification is a special case of audio similarity measures instead of assigning (genre) labels, the similarity/distance between (pairs) of files is measured

12 audio similarity K-Means clustering example simple k-means example goal: minimize intra-cluster variance distance: Euclidean procedure: 1 initialization: randomly select K points in the feature space as initialization. 2 assignment: assign each observation to the cluster with the mean/centroid of the closest cluster. 3 update: compute mean/centroid for each cluster. 4 iteration: go to step 2 until the clusters converge.

13 audio similarity K-Means clustering example simple k-means example goal: minimize intra-cluster variance distance: Euclidean procedure: 1 initialization: randomly select K points in the feature space as initialization. 2 assignment: assign each observation to the cluster with the mean/centroid of the closest cluster. 3 update: compute mean/centroid for each cluster. 4 iteration: go to step 2 until the clusters converge.

14 audio similarity K-Means clustering example simple k-means example goal: minimize intra-cluster variance distance: Euclidean procedure: 1 initialization: randomly select K points in the feature space as initialization. 2 assignment: assign each observation to the cluster with the mean/centroid of the closest cluster. 3 update: compute mean/centroid for each cluster. 4 iteration: go to step 2 until the clusters converge.

15 audio similarity K-Means clustering example simple k-means example goal: minimize intra-cluster variance distance: Euclidean procedure: 1 initialization: randomly select K points in the feature space as initialization. 2 assignment: assign each observation to the cluster with the mean/centroid of the closest cluster. 3 update: compute mean/centroid for each cluster. 4 iteration: go to step 2 until the clusters converge.

16 audio similarity K-Means clustering example matlab source: matlab/displaykmeans.m data points

17 audio similarity K-Means clustering example matlab source: matlab/displaykmeans.m init assignment

18 audio similarity K-Means clustering example matlab source: matlab/displaykmeans.m update centroids assignment

19 audio similarity K-Means clustering example matlab source: matlab/displaykmeans.m update centroids assignment

20 audio similarity K-Means clustering example matlab source: matlab/displaykmeans.m update centroids assignment

21 audio similarity K-Means clustering example matlab source: matlab/displaykmeans.m convergence

22 audio similarity visualization in a 2D space problem feature space is high-dimensional cannot be visualized find mapping to 2D preserving (high-dimensional) distance metrics

23 audio similarity visualization in a 2D space problem feature space is high-dimensional cannot be visualized find mapping to 2D preserving (high-dimensional) distance metrics

24 audio similarity visualization example: SOM 1/2 1 create a map with neurons 2 train for each training sample find BMU (best matching unit) adapt BMU and neighbors toward training sample W v (t + 1) = W v (t) + θ(v, t)α(t) ( D(t) W v (t) ) θ(v, t): depends on distance from BMU α(t): learning restraint D(t) training sample

25 audio similarity visualization example: SOM 1/2 1 create a map with neurons 2 train for each training sample find BMU (best matching unit) adapt BMU and neighbors toward training sample W v (t + 1) = W v (t) + θ(v, t)α(t) ( D(t) W v (t) ) θ(v, t): depends on distance from BMU α(t): learning restraint D(t) training sample

26 audio similarity SOM 2/2 from 1 1 E. Pampalk, Islands of Music, Diploma Thesis, Technische Universität Wien, 2001.

27 mood recognition introduction objective:identify mood/emotion of a song terminology: Music Mood Recognition and Music Emotion Recognition usually used simultaneously What is the difference between mood and emotion

28 mood recognition introduction objective:identify mood/emotion of a song terminology: Music Mood Recognition and Music Emotion Recognition usually used simultaneously What is the difference between mood and emotion emotion: temporary, evanescent (directly) related to eternal stimuli mood: longer term, stable diffuse affect state

29 mood recognition challenges ground truth data verbalization of emotions/moods usually misleading not easily quantifiable/categorizable change over time? research focus can established basic emotions (happiness, anger, fear,... ) used for music perception aroused vs. transported moods?

30 mood recognition challenges ground truth data verbalization of emotions/moods usually misleading not easily quantifiable/categorizable change over time? research focus can established basic emotions (happiness, anger, fear,... ) used for music perception aroused vs. transported moods?

31 mood recognition challenges ground truth data verbalization of emotions/moods usually misleading not easily quantifiable/categorizable change over time? research focus can established basic emotions (happiness, anger, fear,... ) used for music perception aroused vs. transported moods?

32 mood recognition models classification into label clusters Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Rowdy Amiable/Good Natured Literate Witty Volatile Rousing Sweet Wistful Humorous Fiery Confident Fun Bittersweet Whimsical Visceral Boisterous Rollicking Autumnal Wry Aggressive Passionate Cheerful Brooding Campy Tense/Anxious Poignant Quirky Intense Silly mood model, circumplex model most common 2 2 J. A. Russel, A Circumplex Model of Affect, Journal of Personality and Social Psychology, vol. 39, no. 6, pp , 1980, issn: (Electronic); (Print). doi: /h

33 mood recognition models classification into label clusters mood model, circumplex model most common 2 2 J. A. Russel, A Circumplex Model of Affect, Journal of Personality and Social Psychology, vol. 39, no. 6, pp , 1980, issn: (Electronic); (Print). doi: /h

34 mood recognition mood model: regression modeling mapping (N-dimensional) observation (feature) to 2-dimensional coordinate (valence/arousal) training find model to minimize error between data points and prediction

35 linear regression introduction to regression 1/2 fit a linear function to a series of points (x j, y j ) y n = m x n + b

36 linear regression introduction to regression 2/2 minimize error between model and data (here: least squares) e 2 n = (y n mx n b) 2 E = (y n mx n b) 2

37 linear regression introduction to regression 2/2 minimize error between model and data (here: least squares) e 2 n = (y n mx n b) 2 E = (y n mx n b) 2 E b = 2(y n mx n b) = 0 E m = 2x n(y n mx n b) = 0

38 linear regression introduction to regression 2/2 minimize error between model and data (here: least squares) e 2 n = (y n mx n b) 2 E = (y n mx n b) 2 E b = 2(y n mx n b) = 0 2 y n + 2 mx n + 2 b = 0 E m = 2x n(y n mx n b) = 0 2 x ny n + 2 mx 2 n + 2 bx n = 0

39 linear regression introduction to regression 2/2 minimize error between model and data (here: least squares) e 2 n = (y n mx n b) 2 E = (y n mx n b) 2 E b = 2(y n mx n b) = 0 2 y n + 2 mx n + 2 b = 0 mxn + b = y n E m = 2x n(y n mx n b) = 0 2 x ny n + 2 mx 2 n + 2 bx n = 0 mx 2 n + bx n = x ny n

40 linear regression introduction to regression 2/2 minimize error between model and data (here: least squares) e 2 n = (y n mx n b) 2 E = (y n mx n b) 2 E b = 2(y n mx n b) = 0 2 y n + 2 mx n + 2 b = 0 mxn + b = y n E m = 2x n(y n mx n b) = 0 2 x ny n + 2 mx 2 n + 2 bx n = 0 mx 2 n + bx n = x ny n m x n + N b = y n m x 2 n + b x n = x ny n

41 linear regression introduction to regression 2/2 minimize error between model and data (here: least squares) e 2 n = (y n mx n b) 2 E = (y n mx n b) 2 E b = 2(y n mx n b) = 0 2 y n + 2 mx n + 2 b = 0 mxn + b = y n E m = 2x n(y n mx n b) = 0 2 x ny n + 2 mx 2 n + 2 bx n = 0 mx 2 n + bx n = x ny n m x n + N b = y n m x 2 n + b x n = x ny n m = N x ny n x n yn N xn 2 ( x n) 2 yn b = N m xn N

42 mood recognition range of results 5 mood clusters: 40 60% classification rate mood model: absolute prediction error (unit circle)

43 summary lecture content 1 what are typical use cases for music similarity and mood recognition 2 discuss advantages and disadvantages of using classes vs the circumplex model for mood recognition

44 summary lecture content 1 what are typical use cases for music similarity and mood recognition 2 discuss advantages and disadvantages of using classes vs the circumplex model for mood recognition