Automatic Control of a Digital Reverberation Effect using Hybrid Models

Transcription

1 Automatc Control of a Dgtal Reverberaton Effect usng Hybrd Models Emmanoul Theofans Chourdaks 1 and Joshua D. Ress 1 1 Queen Mary Unversty of London, Mle End Road, London E14NS, Unted Kngdom Correspondence should be addressed to Emmanoul Theofans Chourdaks (e.t.chourdaks@qmul.ac.uk) ABSTRACT Adaptve Dgtal Audo Effects are sound transformatons controlled by features extracted from the sound tself. Artfcal reverberaton s used by sound engneers n the mxng process for a varety of techncal and artstc reasons, ncludng to gve the percepton that t was captured n a closed space. We propose a desgn of an adaptve dgtal audo effect for artfcal reverberaton that allows t to learn from the user n a supervsed way. We perform feature selecton and dmensonalty reducton on features extracted from our tranng data set. Then a user provdes examples of reverberaton parameters for the tranng data. Fnally, we tran a set of classfers and compare them usng 10-fold cross valdaton to compare classfcaton success ratos and mean squared errors. Tracks from the Open Multtrack Testbed are used n order to tran and test our models. 1. INTRODUCTION Dgtal Audo Effects (DAFx) are transformatons on an audo sgnal, or a set of audo sgnals, where the transformaton depends on a set of control parameters. In general, users of DAFx control these parameters themselves and they tend to change these parameters over tme based on how the audo sounds. They assgn specfc audo features (or ther changes) to specfc parameters (or changes). Unknowngly, they are dong a form of classfcaton where the samples are features of the audo, and the classes are parameter sets. Our goal s to smulate ths process usng a supervsed learnng approach to tran classfers so that they automatcally assgn effect parameter sets to audo features. Ths way, we can tran our reverberaton effect to decde how to choose ts parameters based just on the observed audo. In order to acheve ths, we perform factor analyss to select the best features from a 31-dmensonal feature space from 8 features found n the lterature. Pre-processng s appled on the resultng features as well as on the parameters whch we cluster nto sets usng k-means. We then compare 4 dfferent classfers on the classfcaton task where our samples are vectors of audo features and classes are the parameter-set clusters. The tranng data conssts of the control parameters provded by the user wth a smple nterface that allows her to control a smple reverberaton effect. valdaton. Testng s done usng cross- 2. PREVIOUS WORK There has been a lot of research n Adaptve Dgtal Audo Effect for automatc multtrack mxng but n almost all cases they focus on achevng a pre-specfed goal. Parameter automaton and ntellgent control have been appled to many of the most popular audo effects (e.g. gan and faders [1], equalzaton [2], pannng [3] and dynamc range compresson [4]), but to the best of the authors knowledge t has not been attempted on artfcal reverberaton. Furthermore, all of the above mentoned approaches except [1] whch use Lnear Dynamcal Systems to estmate mxng weght coeffcents, use fxed rules, rather than arbtrary rules that are learned from tranng data. On the other hand, to the knowledge of the authors, there are no publshed works on Automatc Applcaton of Reverberaton. Important work however, s found n [5] where the authors try to assgn descrptve terms to reverberaton parameters. Smlar work can be found n [6] where they use a Reverberaton Effect among others for ther S.A.F.E semantc audo project whch allows users to assgn hgh level descrptve terms to low level audo feature changes that are caused by effect parameter changes. In a smlar fashon, [7] create a map of hgh level descrptve terms that correspond to 1

2 x L[n] d 1,c 1 y L[n] x[n] Feature extracton φ[] Buffer 1/2 d 2,c 2 d 3,c 3 d a + m /2,g 7 g c G D θ[] Φ d 4,c 4 d a + m /2,g 8 g c G 1/2 d 5,c 5 Decoder J Classfer x R[n] d 6,c 6 y R[n] c [] Fg. 1: Raf & Pardo s Algorthmc Reverberator. Flter boxes are represented by ther control parameters. d ν,g ν for ν = represent Comb Flter delays and gans respectvely, d a all-pass flter delays, g c low-pass flter gan and G s a dry/wet mxer gan. User can control the parameters shown n bold. Reverberaton Effect y[n] Fg. 3: Reverb applcaton x[m] c Feature Extracton φ[] Buffer Fg. 2: Tranng of classfer models Φ C Classfer model estmaton low level reverb parameters. Relevant work n [8] performs classfcaton for drum sounds n order to control effect parameters, but stll reles on fxed rules. 3. EFFECT ARCHITECTURE Our proposed desgn uses the tradtonal adaptve DAFX desgn [9] lmted to one track and can be seen n detal n Fgure 3. It conssts of an artfcal reverberator effect where the values of the parameters are decded by a classfer model. The classfer model can be traned onlne or off-lne. The archtecture of the model tranng process can be seen n Fgure 2 where φ s the feature vector of the -th frame, Φ s a matrx of features whch conssts of the vertcal concatenaton of the feature vectors (as row vectors), from frame 1 to frame. In a smlar fashon, C s a matrx whch conssts of control parameters. θ s the classfer parameter vector, D a dctonary that maps class labels to reverberator parameter sets, and c are the selected effect parameters. Note that, several features requre the accumulaton of a number of samples n a buffer (.e. spectral features) to be computed. In such cases, latency equal to sze of the buffer the sze of the frame s ntroduced. Smlarly, some classfer models requre the accumulaton θ[] D of several values before beng able to make a decson and therefore ntroduce latency equal to number of prevous values buffer sze sze of each frame. Consequently, our archtecture, although mplementable n real tme, can ntroduce latency that depends on the features chosen, as well as the models used. One therefore should be careful n ther choce of features and classfer model. For our reverberaton effect we use the algorthmc desgn descrbed n [5] (Fgure 1) because of ts smplcty n desgn and the fact that t can be traned drectly from measurements of the reverberaton. However, our archtecture s not lmted to just ths partcular reverberaton model. The parameters of the reverberaton and what they affect can be seen n Table 1. Parameter Controls Affects d 1 Comb flter array Reverberaton Tme Echo Densty Central Tme d a All-pass flter array Echo Densty Central Tme g 1 Comb flter array Reverberaton Tme Clarty Central Tme g c Low-pass flter array Spectral Centrod G Dry/Wet mx Reverberaton Tme Clarty Table 1: Algorthmc Reverberaton Parameters Page 2 of 8

3 4. FEATURE EXTRACTION 5. CLASSIFICATION Applcaton of reverberaton to a track can depend on the nstrumentaton, the type of musc and the percussveness of the track, among others. For our task we use 8 dfferent features. Ther names and ther role can be seen n Table 2 [10], [11], [12]. The reason for choosng these features s because they have been used extensvely n the lterature for classfcaton of nstruments based on the above characterstcs. Before extractng the features from our audo, we frst splt the audo nto 23ms frames (1024 samples at 44.1Hz, commonly used n the lterature for audo-based machne-learnng tasks) usng the onset-based audo segmentaton method descrbed n [13] whch s based on the Spectral Contrast feature. The reason for choosng ths knd of segmentaton, as shown n the same paper, s that t appears to gve hgher classfcaton accuraces for at least the musc-genre classfcaton task. We then concatenate our features nto a 31 dmensonal vector 1 for each frame. Next, we use Prncpal Component Analyss [14] to flter out nonseparable or nosy features and reduce our feature vectors dmensonalty. We use Horn s Parallel Analyss [15], [16] to fnd the optmal number of Prncpal Components to keep. Feature ZeroCrossngRate Used n Source Identfcaton 13 MFCCs Genre Classfcaton 12 Spectral Contrast Coeffcents Root Mean Square Crest Factor Spectral Centrod Spectral Roll-off Spectral Flux Genre Classfcaton Voce/Musc Dscrmnaton Audo Actvty Detecton Genre Classfcaton Genre Classfcaton Table 2: Used features and ther usage n the lterature. 1 MFCCs and Spectral Contrast features have 13 and 12 dmensons respectvely. We use classfcaton on the audo features n order to control the values of the reverberaton effect parameters. Gven audo segments wth appled reverb we do the followng: 1. Assgn a dfferent label for every dfferent set of reverb parameters assgned. The labels wll consttute our class labels. Keep a dctonary structure comprsed of the parameter sets and the labels to whch they correspond. 2. Segment the audo segments to frames and calculate a feature vector for each frame. 3. Tran a classfer model that classfes the features extracted to the labels gven above. For the applcaton of reverberaton: 1. Segment the audo, to whch we want to apply reverb, to frames and calculate a feature vector for each frame. 2. Classfy the new features to the class labels found n the tranng phase. 3. Use the dctonary structure derved n the tranng phase to change from class labels to reverberaton effect parameters. More specfcally, for the tranng phase, we consder the vectors: c = [ d 1 d a g 1 g c G ] T φ = [φ 1 φ 2... φ M ] T The elements of c are the values of the parameters shown n Table 1 normalzed to (0,1), at frame. Correspondngly, the elements of φ are values of features of Table 2, normalzed agan to (0,1) at frame. We also consder the (unque) sets: C = {c : } then C s the number of classes. We then defne the varable: J = C k=1 k[c == C k ], = 1...M,k = 1...N Page 3 of 8

4 We wll call J our parameters label varable. Consstently, we wll call A wth values a 1...I our parameters varable and F wth values f 1...I our features varable. We compare 5 dfferent classfers: Gaussan Nave Bayes Classfer [17], All-vs-all Lnear Support Vector Machne (SVM) Classfer [18], Hdden Markov Model Maxmum A-posteror Classfer [17], a hybrd HMM classfer wth observatons taken from a set of SVMs [19] and one we wll call the Snkhole approach classfer, that uses the classfcaton results of all of the above and a votng scheme. The last three take nto consderaton past tme events whle the frst two consder just the current frame. Below we gve a small descrpton of each Gaussan Nave Bayes Classfer Gaussan Nave Bayes Classfer (In short GNBC) s a verson of the nave Bayes classfer for contnuous varables. Lke ts non-contnuous counterpart, t s based on the Bayes rule: p(x O) = p(x)p(o X) p(o) where X and O are random varables and O N (µ, Σ) where µ and Σ are the model s parameters. If we substtute O wth F and X wth J, then the probablty we have a specfc set of parameters s gven by Equaton 1. Gven a set of feature vectors {φ : = 1...I}, J can be calculated usng the Maxmum A-Posteror Decson rule: J = argmax k=1... C p(c k ) I j=1 p(φ j C k ) Inversely, gven a set of features φ and ther correspondng labels J, we can tran the Gaussan Nave Bayes Classfer (whch means to estmate the parameters µ, Σ) usng Maxmum lkelhood Estmaton AVA Lnear Support Vector Machne Classfer Support Vector Machnes (SVMs) are Maxmum Margn classfers. Ther goal s to fnd a boundary that dscrmnates the data ponts and retans the maxmum dstance from the closest ones to that boundary (the margn). Gven a set of feature vectors {φ : = 1...I} the decson boundary of a Lnear SVM that classfes these vectors nto two classes {+, } s gven by: y(φ ) = w T φ + b and the decson rule s: { + f y(φ φ s classfed as: ) 0 f otherwse Tranng of a Lnear SVM s an optmzaton problem whch can be solved usng varous methods [20]. Extendng the SVMs for the mult-class case usng bnary classfers can be done n two ways: One-vs-All and All-vs-All classes (abbrevated OVA and AVA) [18]. For the OVA case, we fnd a decson boundary that dscrmnates between a class k and the other K 1 classes, for every k. In the AVA case, we fnd the decson boundares for every par (k,l) of K classes. In ths work we (arbtrarly) choose to do AVA classfcaton Hdden Markov Model Maxmum Lkelhood Classfer Hdden Markov Models (HMMs) can be consdered as fnte state automata where each state can only be ndrectly observed. States are consdered hdden varables and each state s assgned one observaton varable. Frst-Order Hdden Markov Models are HMMs where the transton n one state depends only on the prevous state. An HMM of K states and Gaussan emssons s formally gven as the tuple: θ = A, µ, Σ, Π where A s the matrx of transton probabltes between states, µ and Σ are the mean vector and covarance matrx of the Gaussan PDF of the observatons, and Π s the matrx of the states pror probabltes. For our classfer we tran C HMMs one for each class usng feature (observaton) sequences of length L. For each feature sequence then that we want to classfy, we choose as our class label, the label of the HMM that gves us the Maxmum Lkelhood for that sequence. The decson rule s therefore gven by: J = argmaxl (φ L+1:,x L+1: θ k ) k=1...k where θ k are the parameters of the HMMs for the k-th class, x s the hdden state at the -th moment, φ s the Page 4 of 8

5 correspondng emsson and L (e, x) denotes the lkelhood of the emsson sequence e gven the state sequence x. Lkelhood L s calculated usng the FORWARD algorthm [17]. Each HMM n our case s traned usng the VITERBI algorthm [17]. The length of the sequence for each class s found by dong cross-valdaton on the tranng set and choosng the length of the chan that gves the best classfcaton accuracy ratos Hdden Markov Models wth SVM emssons The Gaussan HMM classfers descrbed above work suffcently, but they rely on Gaussan probablty densty functons (PDF) for ther emssons whch cannot dscrmnate really well. However, SVMs can learn to dscrmnate well even wth very few samples. We would lke to combne the dscrmnatng power of the SVMs wth the sequental nature of the HMMs but ths s not possble drectly because SVMs do not provde emsson PDFs. Fortunately, we can use some trcks to derve a PDF. Usng Platt s method [19] and followng the work done n [21] for our AVA Classfer descrbed n Secton 5.2 we have for a class C k and a sample feature vector φ [21]: K ] 1 p(c k φ ) = 1/[ (K 2) µ kl l=1,l k where µ kl s the probablty that the class s ether C k or C l, that s: µ kl = p(c k C l φ ) If we regard the class of f our HMM s state, then we can compute ts emsson probabltes by usng the Bayes rule: p(φ C k ) = α p(c k φ ) p(c k ) where p(c k ) can be estmated by countng occurrences of C k n our data and α s the normalzaton factor. Snce our model s states are the same as our classes, the process of classfcaton reduces to predctng the hdden sequence state of our HMM usng the VITERBI algorthm. Tranng of the HMM n that case s acheved by frst tranng the SVMs that wll provde the emsson probabltes, then normally tranng the HMM usng the VITERBI or the BAUM-WELCH (FORWARD- BACKWARD) [17] algorthm Snkhole Approach Classfer The classfers n the sectons above wll perform dfferently dependng of the nature of our features. The Gaussan-based classfers (GNB, HMM) wll perform better when our features are dstrbuted around a Gaussan bell, the Support Vector Machne-based classfers (AVA, HMM/SVM) wll perform better when our features can be dscrmnated by usng straght lnes. Smlarly, the Hdden Markov Model-based classfers (HMM, HMM/SVM) wll explot the sequental nature of our features. In some cases, one of the classfers above may be more sutable than the other for a set of features. We wll explot ths fact and as a fnal classfer-model approach, we wll use what we wll call a Snkhole approach. That s, we wll do classfcaton on our features usng all of the above classfers and then votng for the result. The decson process n ths case s as follows. For each feature vector φ do: 1. Classfy φ usng the GNB, AVA, HMM, and HMM/SVM classfers, gvng t the labels J (GNB), J (AVA), J (HMM), and J (HMM/SVM) respectvely. Those wll be our canddate labels. 2. If a canddate label appears more than the rest, pck that as our fnal label. 3. When there s a draw, pck the label at random from the canddates that are at draw. 6. TRAINING Tranng takes place on excerpts from 254 audo fles taken from the Open Multtrack Testbed [22]. Frst the audo data s segmented nto meanngful parts (.e. song phrases, parts of musc, etc.) usng a smlarty matrx and a novelty functon as found n [12]. A user s presented wth a smple GUI where she can lsten and apply reverb to the extracted parts. The segmented parts are splt nto frames usng the method descrbed n [13] and a tuple of features and parameters are extracted for each frame as descrbed n Secton 4. Features are fltered wth a low-pass flter and parameter values are compressed usng K-MEANS n order to reduce the number of dstnct parameter labels. The resultng data set s used to tran the models descrbed n Secton RESULTS In order to test our models, we splt our data nto 6 sets. Every set ncluded 90 audo segments except the last Page 5 of 8

6 whch ncluded 58. Every segment was a part of a Bass, Keyboards, Vocals, Percusson or Saxophone track. The segments were randomly splt nto sets. Because the tranng had been done by the user, the tranng data ncludes several labelng errors (that s, we ncorrectly appled reverb to some parts) whch we automatcally detect by fttng a Mnmum Covarance Determnant [23] and relabelng them by applyng an ntal classfcaton on the mslabeled features. We defne classfcaton accuracy as the normalzed (to 1.0) number of tmes the classfer has been correct on ts decsons, and mean squared error the average of the squares of the dfference between an estmated value of a parameter vector and the true value of that vector. We tested for classfcaton accuracy and mean squared error as such: Splt every set nto 10 parts. Use 9 parts for tranng and 1 for testng. Do ths for every combnaton of 10 parts. Use an ncreasng number of samples from the tranng set to tran the models, and test them aganst the full testng set. Store the classfcaton rato and the mean squared error versus the number of samples used for tranng. Fnally, use the averages of all combnatons to derve the Classfcaton Accuracy and Mean Squared Error values. # GNB AVA HMM HMM SVM SINK Table 3: Average Classfcaton Accuracy rates Usng the full tranng set, we can see the average Classfcaton Accuraces n Table 3, and the average Mean Squared Errors n Table 4. A comparson graph for classfcaton accuracy can be seen n Fgure 4 and for MSEs n Fgure 5. The hgh classfcaton accuracy result s # GNB AVA HMM HMM SVM SINK Table 4: Mean Squared Errors for the normalzed parameters. SINK-HOLE HMM/SVM HMM SVM GNB Set 1 Set 2 Set 3 Set 4 Set 5 Set 6 Fg. 4: Classfcaton accuracy scores. Insde the bars s the ndvdual accuracy rate for each model and each dataset. The total score (sum of the ratos) can be seen next to each bar. mportant because t represents the rate of agreement, between the automatc reverberaton effect and the user that t traned t, on the parameters of the reverberaton. What s more mportant though, s havng a low mean squared error. Mean squared error effectvely measures how far the estmated parameters are from the real ones. Ths means that whle the classfcaton accuracy may be hgh, so the effect and the users agree most of the tme, the dfferences on the parts they do not agree may be too hgh for the model to be useful. Therefore, the most useful model s the model wth the least mean squared error. In our case, the SVM and the SINK-HOLE approaches perform very smlarly both n classfcaton accuracy and mean squared error. Page 6 of 8

7 SINK-HOLE HMM/SVM HMM dea worth explorng s the combnaton wth reverberaton effect control usng perceptual parameters, as n [5], or descrptve terms, as n [6] and [7]. Especally the two later, snce they are dealng by desgn wth sets of parameters, we would expect ther method to complement ours very naturally. SVM GNB Set 1 Set 2 Set 3 Set 4 Set 5 Set 6 Fg. 5: Mean Squared Error values. The sum of the MSE values for each classfer model can be seen next to each bar. 8. CONCLUSION From our graphs we can see that for our datasets, the nonsequental models performed better than the sequental counterparts, performng comparably or even worse than the Nave Bayes classfer. Ths suggests that the Hdden Markov Models faled to capture correctly the temporal progresson of our data, possbly suggestng further exploraton wth dfferent models and confguratons. The best model so far seemed to be the All-vs-All Support Vector Machne classfer whch performed best regardng Classfcaton Accuracy and Mean Squared Error. In the context of our reverberaton effect, ths means that our model should provde correct estmates of the parameters 77.9% of the tmes (the average classfcaton accuracy over all sets). 9. LIMITATIONS/FUTURE WORK The orgnal Adaptve Dgtal Audo Effect archtecture [9] supports multtrack DAFx, whle our method has only been tested for effects appled on a sngle track. A logcal next step would then be to extend our archtecture to multtrack audo content. Also, our method s lmted to pre-traned sets of parameters, whch can be lmtng n the ntroducton of new unexplored audo. [24] descrbes a way to control contnuous control parameters usng dscrete states usng only two parameter states, somethng whch would ft naturally wth our approach. Another 10. REFERENCES [1] J. Scott, M. Prockup, E. M. Schmdt, and Y. E. Km. Automatc mult-track mxng usng lnear dynamcal systems. In SMC th Sound and Musc Computng Conference, [2] S. Hafez and J.D. Ress. Autonomous multtrack equalzaton based on maskng reducton. Journal of Audo Engneerng Socety, 63(5): , May [3] E. P. Gonzalez and J. D. Ress. A real-tme semautonomous audo pannng system for musc mxng. EURASIP Journal on Advances n Sgnal Processng, [4] Z. Ma, B. De Man, P. D. Pestana, D. A. A. Black, and J. D. Ress. Intellgent dynamc range compresson. Journal of the Audo Engneerng Socety, 63, [5] Z. Raf and B. Pardo. Learnng to control a reverberator usng subjectve perceptual descrptors. In 10th Internatonal Socety for Musc Informaton Retreval Conference (ISMIR 2009), [6] R. Stables, S. Enderby, De Man B., G. Fazekas, and Ress J.D. Safe: A system for the extracton and retreval of semantc audo descrptors. In 15th Int. Socety for Musc Informaton Retreval Conference, [7] P. Seetharaman and B. Pardo. Crowdsourcng a reverberaton descrptor map. In 22nd ACM Internatonal Conference on Multmeda, [8] J. Scott and Y. E. Km. Instrument dentfcaton nformed mult-track mxng. In 14th Internatonal Socety for Musc Informaton Retreval Conference, November [9] V. Verfalle, U. Zolzer, and D. Arfb. Adaptve dgtal audo effects (a-dafx): a new class of sound transformatons. EEE Transactons on Audo, Speech, and Language Processng, 14: , [10] G. Peeters. A large set of audo features for sound descrpton (smlarty and classfcaton) n the cudado project. Techncal report, IRCAM, [11] T. Ganchev, N. Fakotaks, and G. Kokknaks. Comparatve evaluaton of varous mfcc mplementatons on the speaker verfcaton task. In Proceedngs of the SPECOM-2005, [12] J. Foote. Automatc audo segmentaton usng a measure of audo novelty. In IEEE Internatonal Conference on Multmeda and Expo, volume 1, pages , Page 7 of 8

8 [13] K. West and S. Cox. Fndng an optmal segmentaton for audo genre classfcaton. In 6th Internatonal Conference on Musc Informaton Retreval, pages , [14] Y. Lu, I. Cohen, S. Zhou, X, and Q. Tan. Feature selecton usng prncpal component analyss. In Systems Scence, Engneerng Desgn and Manufacturng Informatzaton (ICSEM), volume 1, pages IEEE, [15] J. L. Horn. A ratonale and test for the number of factors n factor analyss. Psychometrka, 30(2): , June [16] A. Dnno. Gently clarfyng the applcaton of horns parallel analyss to prncpal component analyss versus factor analyss. for PCA vs FA.pdf, May [17] C. M. Bshop. Pattern Recognton and Machne Learnng. Sprnger, [18] M. Aly. Survey on multclass classfcaton methods. Neural networks, pages 1 9, [19] J. Platt. Probablstc outputs for support vector machnes and comparson to regularze lkelhood methods. In Advances n Large Margn Classfers, pages 61 74, [20] J. Platt. Fast tranng of support vector machnes usng sequental mnmal optmzaton. In Advances n Kernel Methods - Support Vector Learnng. MIT Press, [21] S. E. Krger, M. Schaffner, M. Katz, E. Andelc, and A. Wendemuth. Speech recognton wth support vector machnes n a hybrd system. In INTERSPEECH Eurospeech, 9th European Conference on Speech Communcaton and Technology, [22] B. De Man, M. Mora-Mcgnty, G. Fazekas, and J.D. Ress. The open multtrack testbed. In AES 137 th Conventon, Los Angeles, USA, [23] P.J. Rousseeuw and K. Van Dressen. A fast algorthm for the mnmum covarance determnant estmator. Technometrcs, 41(3):212, [24] J. Pazs and M. G. Lagoudaks. Bnary acton search for learnng contnuous-acton control polces. In ICML 09 Proceedngs of the 26th Annual Internatonal Conference on Machne Learnng, Page 8 of 8