Optimizing feature representation for speaker diarization using PCA and LDA

Transcription

1 Optimizing feature representation for speaker diarization using PCA and LDA Jean-Francois Bonastre

2 Outline Speaker Diarization what is it? Baseline Speaker Diarization System Dimensionality Reduction Principal Component Analysis Linear Discriminant Analysis Proposed Speaker Diarization System Experiments and Results Conclusions 2

3 Speaker Diarization what is it? The goal is to separate the conversation into R clusters each cluster, hopefully, contains single speaker data. Additional clusters can be added for nonspeech, simultaneous speech, etc Number of speakers, R, can be known; otherwise has to be estimated. 3

4 Speaker Diarization what is it? (cont.) Pirate 1 Pirate 2 Pirate 3 4

5 Baseline Diarization System General Blocks 5

6 Baseline Diarization System (cont.) Fix duration HMM clustering 6

7 Baseline Diarization System (cont.) Fix duration HMM clustering (cont.) A A A A A A A A A A K K K1 K 2 KK A kk 0 0 n p s k s k n1 A qk 0 0 p sn q sn 1 k

8 Baseline Diarization System (cont.) We approximate the log-likelihood for some input vector x n SOM r as: d 1 T L x SOM m x m x 2 2 * * l l log 2 * Fix duration HMM clustering (cont.) n r r n r n T m min m x m x l l l r l1,, L r n r n r Problem: in order to obtained a good clustering results a good initialization of the hyper-states is required. 8

9 Baseline Diarization System (cont.) Weighted segmental K-means initialization Weighted K-Means: Given the data into K clusters N X x C k k 1 n n 1 1. Represent each cluster by a centroid 2. Estimate the clusters: 3. Update the centroids: K c k C x : k arg min x c 1 ck wn x n k n n p p 1,, K separate the data wn x C n: xnc k n k 4. If termination conditions are not met, return to

10 Baseline Diarization System (cont.) Weighted segmental K-means initialization (cont.) Weighted Segmental K-Means: Given N data segments: 1. Calculate the average of each segment. 2. Apply the weighted K-Means algorithm for the means. 3. Assign each segment to the cluster which the average vector belongs. 10

11 Baseline Diarization System (cont.) 1 1 x 1 x 2 Weighted segmental K-means initialization (cont.) Seg. 1 1 x K x 1 x 2 Seg. 2 x 2 N N x x K Seg. N N x K N x, w K x, w x, w K K N N N 11

12 Baseline Diarization System (cont.) Weighted segmental K-means initialization (cont.) x1 x2 xn Weighted K-Means Algorithm C1 Ck 12

13 Dimensionality Reduction There are 2 main goals in dimensionality reduction: 1. Orthogonal projection of data to a lower linear space, such that the variance of the projected data is maximized (PCA unsupervised ). 2. Projection of data on a lower linear space, such that within-class variance is minimize while the between-class variance maximize (LDA supervised). 13

14 Dimensionality Reduction (cont.) Principle Component Analysis (PCA) K Optimization criterion: t t J u u Su k 1 ukuk n1 k1 N 1 S n n N x x x x Unsupervised, no labeled data is required Linear Discriminant Analysis (PCA) t u S u K b Optimization criterion: t J u k 1 t ukuk u S u w k1 t S S K N x x x x b k k k k1 K x x x x b n k n k k1 n Supervised, labeled data is required k t t 14

15 Proposed Speaker Diarization System General Blocks 15

16 Experiments and Results Evaluation Setup LDC America CallHome 108 conversations, 8kHz About 2.1sec average speech segment length Features 12 th order MFCC 3-states HMM 5+1 Iterations (20+10 tied states) NIST evaluation criterion with 0.5sec window (66.2% speech coverage) Principle Component Analysis (PCA) Find the sub-space with the highest variance, i.e., sub-space spanned be eigen-vector with the MAXIML eigen-values. The order of the projection, P, should be found on evaluation set: 1P 12 (Features vector size) 16

17 Experiments and Results (cont.) Principle Component Analysis (PCA) (cont.) DER as a function of PC s number Diarization time as a function of PC s number 17

18 Experiments and Results (cont.) Linear Discriminant Analysis (LDA) Find the most discriminant sub-space, i.e., MAXIMIZE the inter-class means variance and MINIMIZE intra-classes variances. The order of the projection, P, should be found on evaluation set: 1P 12 (Features vector size) The classes have to be well defined which is not so at diarization!!! Assumption: after first 5 iterations, the classes are reasonably separated 5 additional iterations performed with the projected features 18

19 Experiments and Results (cont.) Linear Discriminant Analysis (LDA) (cont.) DER as function of the number of LD vectors DER improvement as a function of the baseline DER 19

20 Experiments and Results (cont.) PCA as pre-processing and LDA as post-processing DER as a function of PC s and LD vectors 20

21 Experiments and Results (cont.) NIST 2005 LDA Experiments NIST conversations, 8kHz About 0.7sec average speech segment length DER as a function of LD vectors 0.5sec window, 40.0% coverage DER as a function of LD vectors 0.2sec window, 69.6% coverage 21

22 Conclusions We presented a way to combine the PCA (unsupervised) as a pre-processing and LDA (supervised) as post-processing into speaker diarization system. PCA does not gain in the terms of DER some gain can be achieved in the diarization speed. LDA can be applied as a post-processing with additional diarization step, only if reasonable results are achieved at the initial diarization. The LD vectors number should be found on a reliable development set. This approach can be applied to many other diarization combinations. Other discriminant approaches, such as PLDA and ICA should be explored. 22

23 23