SECOND FIDDLE IS IMPORTANT TOO: PITCH TRACKING INDIVIDUAL VOICES IN POLYPHONIC MUSIC

Transcription

1 SECOND FIDDLE IS IMPORTANT TOO: PITCH TRACKING INDIVIDUAL VOICES IN POLYPHONIC MUSIC Mert Bay 2, Andreas F. Ehmann 2, James W. Beauchamp 2, Pars Smaragds 1,2 and J. Stephen Downe 3 1 Department of Computer Scence, Unversty of Illnos at Urbana-Champagn 2 Department of Electrcal & Computer Eng., Unversty of Illnos at Urbana-Champagn 3 Graduate School of Lbrary and Informaton Scence, Unversty of Illnos at Urbana-Champagn {mertbay,aehmann,jwbeauch,pars,jdowne}@llnos.edu ABSTRACT Recently, there has been much nterest n automatc ptch estmaton and note trackng of polyphonc musc. To date, however, most technques produce a representaton where ptch estmates are not assocated wth any partcular nstrument or voce. Therefore, the actual tracks for each nstrument are not readly accessble. Access to ndvdual tracks s needed for more complete musc transcrpton and addtonally wll provde a wndow to the analyss of hgher constructs such as counterpont and nstrument theme mtaton durng a composton. In ths paper, we present a method for trackng the ptches (F0s) of ndvdual nstruments n polyphonc musc. The system uses a pre-learned dctonary of spectral bass vectors for each note for a varety of muscal nstruments. The method then formulates the trackng of ptches of ndvdual voces n a probablstc manner by attemptng to explan the nput spectrum as the most lkely combnaton of muscal nstruments and notes drawn from the dctonary. The method has been evaluated on a subset of the MIREX multple-f0 estmaton test dataset, showng promsng results. 1. INTRODUCTION One of the most mportant classes of nformaton to be retreved from musc s ts polyphonc ptch content. Recently, many researchers have attempted multple-f0 estmaton (MFE) [11, 14, 18]. (A revew of state-of-the-art MFE systems can be found n [2].) Many current MFE systems such as [14] restrct themselves to the estmaton of a certan number of F0s for each frame, whle gnorng whch nstrument/tmbre corresponds to each F0. Whle approaches for trackng solo melodc voce/tmbres exst [15], the ptch trackng of addtonal lnes and parts n polyphonc musc opens new possbltes. By exposng the ndvdual lnes produced by each nstrument n polyphonc musc, aspects such as counterpont, the appearance of letmotfs across nstruments n a pece, and hdden thematc Permsson to make dgtal or hard copes of all or part of ths work for personal or classroom use s granted wthout fee provded that copes are not made or dstrbuted for proft or commercal advantage and that copes bear ths notce and the full ctaton on the frst page. c 2012 Internatonal Socety for Musc Informaton Retreval. references across muscal peces can be uncovered. Therefore, nstruments and ther tmbres are very mportant components n musc and should be tracked along wth ther F0s. Moreover, knowng each nstrument s F0 track can be very benefcal for a varety of MIR user applcatons, such as musc transcrpton, score algnment, audo musc smlarty, cover song dentfcaton, actve musc lstenng, melodc smlarty, harmonc analyss, ntellgent equalzaton, and F0-guded source separaton. As stated prevously, most MFE systems produce lowlevel representatons of the polyphonc ptch content present n musc audo whch report only what fundamental frequences are present at each gven tme. However hgher-level representatons can attempt to track and lnk these fundamental frequences over tme to form notes such as descrbed n [3]. It s mportant to note that such trackng can also mprove the accuracy of MFE s based purely on ndvdual frames, as reported reported n [18]. In [12] a classfcaton approach s used to determne sngng voce portons n musc audo so as to buld the ptch-track correspondng to a vocal melody. In [5, 6], a frame level mult- F0 estmaton method s used followed by a constraned clusterng method that uses harmonc ampltude-based features to cluster ptches nto ptch tracks. In these cases, to buld nstrument or tmbre-specfc ptch tracks, bottom-up methods were used that frst produced frame-based ptch estmates and subsequently sometmes attempted to buld note-level representatons, whch may form solo phrases. In the method presented n ths paper, we use an alternatve approach. Instead of buldng tmbre tracks from the ptch content, our proposed approach uses tmbre nformaton to gude the formaton of nstrument-specfc ptch tracks. Ths paper s organzed as follows. Secton 2 detals the proposed method. Secton 3 descrbes the evaluaton datasets, measures, and results. The evaluaton results are dscussed n Secton 4 and conclusons and future work covered n Secton PROPOSED METHOD To make a system that tracks ptches attrbuted to dfferent muscal nstruments, we borrow an dea from the supervsed sound source separaton doman: usng a spectral lbrary [1, 13]. By tranng our system on example sounds

2 from dfferent nstruments, we can track them n complex sound mxtures. The proposed method s based on probablstc latent component analyss (PLCA) [17]. PLCA s a varant of non-negatve matrx factorzaton (NMF) and has been used wdely to model sounds n the spectral doman. Its probablstc nterpretaton makes t extensble to usng prors and statstcal technques. We use regular PLCA to buld dctonares of spectra ndexed by F0 and nstrument where the spectra are analyzed from recordngs of ndvdual notes of dfferent muscal nstruments. We extend the model of [16] to represent each source/nstrument not by only one spectral dctonary, but rather wth a collecton of dctonares, each of whch s an ensemble of spectral bass vectors that have the same F0. We model the nput musc sgnal s spectrum as a sum of bass vectors from F0-specfc spectrum dctonares for dfferent nstruments. Update rules are desgnated to calculate the model parameters, whch are estmated probabltes for the occurrence of each spectral bass vector, F0- spectrum dctonary, and nstrument n the nput mxture at a gven tme. Fnally, we perform the Vterb algorthm [8] to track the most lkely ptch sequence for each nstrument. A snusodal model s used for the tme-frequency representaton because of ts compactness and also for representng each note ndependent of the ntonaton errors or tunng dfferences between tranng and test set performers. We begn by performng a short-tme Fourer transform on the audo sgnal. Peaks n the spectrum are then determned usng a frequency-dependent threshold as descrbed n [7]. We then refne each peak s ampltude and frequency usng a sgnal dervatve method proposed by [4]. 2.1 Model We model the audo nput mxture s spectrum for each frame as a sum of nstrument tones where each tone s represented by a dctonary of spectral bass vectors that are learned n advance. It s assumed that all nstrument tones have harmoncally related frequences whch are nteger multples of an F0 frequency. It turns out that n a mxture of such tones there s a hgh probablty that harmoncs wll overlap. The nput spectrum can be modeled as a dstrbuton/hstogram over a range of frequences. E.g., each component s vewed as the probablty of occurrence of that partcular frequency. The magntude at any partcular frequency can be thought as an accumulaton of magntudes from varous nstruments due to component overlappng. The scaled verson of the nput mxture spectrum s modeled as a dscrete dstrbuton. The generatve process s modeled as follows: X t (f) = P t (f) = I P t () N K P t (p ) P t (z p)p p (f z) (1) n p z z p where X t (f j ) s the spectral magntude of the peak j at frequency f j for the observed nput mxture spectrum at tme t; P t (f) s an approxmaton of the nput spectrum; P t () s the estmated probablty of occurrence of nstrument at tme t, whereas P t (p ) s the estmated probablty that ptch p s produced by nstrument ; P (f z) s the learned spectral bass vector for the ptch p of nstrument ; and P t (z p) s the probablty (weght) of that bass vector. The above model explans the mxture magntude spectrum herarchcally as the sum of N ndvdual ptches from I dfferent nstruments where the dctonary correspondng to each ptch/nstrument has K elements. The ndependence relatonshps of the model can be represented by the graph I P Z F. The process that generates the frequency components n the observed magntude spectrum s as follows: Frst, an ndvdual nstrument lbrary s selected from a group of nstrument lbrares. Second, a spectrum dctonary s drawn for each ptch from the nstrument lbrary. Thrd, a spectral bass vector s drawn from a partcular F0-spectrum dctonary for the nstrument. Fourth, a spectral component at a partcular frequency s drawn from the spectral bass vector. Although t s possble that ths spectral component wll be generated by only a sngle nstrument, most often t only contrbutes a fracton to the magntude of the observed spectrum at that frequency. 2.2 Parameter Estmaton Parameters for the model θ = {P t (z p), P t (p ), P t ()} can be estmated usng an expectaton-maxmzaton (EM) algorthm. In the E-step, current parameter values θ old are used to calculate the posteror dstrbuton. P t (, p, z f, θ old ) = P t(f, p, z)p t (, p, z) P t (f) Because of the structure of the model, P t (f, p, z) = P t (f z) and P t (, p, z) = P t ()P t (p )P t (z p). We can wrte the posteror as P t (, p, z f, θ old ) = (2) P (f z)p t(z p)p t(p )P t() P t() P t(p ) P t(z p)p p (f z) (3) n p z zp The posteror s used to calculate the expectaton of the complete data log lkelhood Q Q(θ, θ old ) = X f P (, p, z f, θ old )log(p (, p, z, f θ) (4) f z zp n p In the M-step, new parameters are estmated by maxmzng the above functon accordng to θ, resultng n the followng update rules: P t (z p) new P t (p ) new P t () new f z z p f z z p f p p z z p f p p z z p f p p z z p f (5) (6) (7)

3 We then use the new estmates to calculate the posteror n an teratve manner and repeat untl convergence s acheved. 2.3 Sparsty Pror At any gven tme, we expect each nstrument actve to be playng a sngle ptch. Even though the parameter estmaton method would allow multple ptches per frame for each nstrument, n ths project our goal s to track a monophonc ptch contour for each nstrument. We also expect that not all nstruments are actve at the same tme. Enforcng sparsty constrants on note and nstrument probabltes P t (p ) s and P t() s renforce ths behavor. We use the followng pror (the normalzng constant s dropped for convenence) ( ) β P (φ) = (P (φ)) α (8) φ Addng the above pror to the expectaton of the complete data log-lkelhood and maxmzng t wth respect to P () and P (p ), we arrve at the followng update rules P (p ) new P (, p, z f)x f + βp(pt(p ))α (9) (P t(p )) α z z p f p p P t () new p p z z p f + βp(pt())α (10) (P t()) α p (We have to rescale explctly so that t sums up to one.) new P (p ) new P (p )new P () and P new ()new P () new (11) p p 2.4 Enforcng Contnuty P () Our goal s to track each nstrument s F0 through tme. We expect the ptch contour to be smooth, not changng drastcally from frame to frame except at note transtons. Usng the Vterb algorthm, we treat the ptches as hdden states and pose the nstrument trackng problem as one of nferrng the most lkely ptch state sequence for each nstrument. Above estmated ptch dstrbutons (whch maxmze the mxture lkelhood P (X t (f), p, z)) for each nstrument are consdered to be the emsson probablty of the hdden state n a hdden Markov model (HMM). Transton probabltes are modeled as normal dstrbutons gven by P (p t p t 1) = 1 2πσ e ( f 0t f ) 2 0t 1 2σ 2 (12) where p t denotes the hdden ptch state for nstrument at tme t. f 0t denotes the F0 assocated wth the hdden ptch state for nstrument at tme t. Transtons are calculated wthn the same nstrument. We emprcally choose σ = 7 + f/100hz. The above dstrbuton enforces the contnuty of the actve notes from frame to frame. 3. EVALUATION ON REAL WORLD DATA We traned a dctonary for each ptch of each nstrument usng the RWC muscal nstrument database [9] usng nonnegatve matrx factorzaton wth Kullback-Lebler dvergence whch s numercally equal to the regular PLCA method n 2 dmensons [17]. For each ptch from the RWC dataset, 20 representatve spectra were derved from 27 dfferent tones correspondng to 3 players, 3 dynamcs (pano, mezzo, forte), and 3 artculatons (normal, staccato, vbrato). In the ptch trackng stage, we lmt the number of nstrument lbrares to choose from by desgnatng the nstruments expected to be n the nput mxture as nput to the algorthm. We also lmt the search range for F0 (whch F0-spectrum dctonares to use) of each nstrument by only usng the peaks estmated by the snusodal model that are between 50 Hz to 2500 Hz as ptch canddates. Evaluatons are performed at frame level. Mult-F0 trackng problem s smlar to melody extracton problem extended to multple melodes as opposed mult-f0 estmaton problem where the estmated number of F0s for each frame can be dfferent than the ground-truth F0s, whch s not the case n the trackng problem, where the number of estmated F0s and the reference F0s are smply equal to total number of frames. We extended the evaluaton metrcs from MIREX melody extracton task to our problem. Comparng the voced (nonzero F0) and unvoced (zero F0) values for each frame of the estmated and ground-truth F0 tracks, the counts for true postves (TP), true negatves (TN), false postves (FP) and false negatves (FN) are calculated accordng to Table 1 Estmated voced unvoced Ground voced TP FN truth unvoced FP TN Table 1. Evaluaton TP s further break down nto ones wth correct F0 and the ones wth ncorrect F0 as T P = T P C + T P I. Estmated voced F0 s correct f t s wthn a quarter tone (+-2.93%) range of a postve ground-truth F0 for that frame. The precson, recall, F-measure, and accuracy for each nput test fle s then calculated over all frames and all nstruments as: T P C,t t Precson = (13) T P,t + F P,t t T P C,t t Recall = (14) T P,t + F N,t F-measure = Acc. = t t 2 precson recall precson + recall T P C,t+T N,t t (15) T P,t+F P,t+T N,t+F N,t (16)

4 where t s the frame ndex and s the nstrument ndex. We tested the proposed method on dfferent datasets. The ground-truths for these datasets were estmated usng monophonc ptch estmators (Wavesurfer, Praat and YIN) on the sngle-nstrument recordngs pror to mxng. The results of the monophonc ptch estmators are manually corrected where necessary. For prelmnary testng and development, we appled the method on a 11 second excerpt taken from a real world performance by bassoon, clarnet and oboe whch was taken from a MIREX multtrack dataset (standard woodwnd quntet transcrbed from L. van Beethoven Varatons for Strng Quartet, Op.18, N.5). The three separate tracks were mxed to monaural. The results can be seen n Table 2. The proposed method scored 0.83 accuracy on average. Fgure 1 shows the multple-f0 tracks for each nstrument. Wthout trackng, the F0 s would not be connected from frame to frame. Bassoon Clarnet Oboe Ave. Accuracy Precson Recall F-measure Table 2. Evaluaton performances for a 11-s woodwnd tro excerpt (bassoon, clarnet, oboe). non-overlappng 30-second sectons were chosen from the recordng. Isolated nstruments were mxed from solo tracks to polyphones rangng from 2 (duo) to 5 (quntet), resultng n a total of 20 tracks (4 dfferent polyphones tmes fve sectons). The average ptch-trackng results over all tracks for polyphony 2 to 5 are shown n Table 3. Accuracy Precson Recall F-measure Table 3. Average performance on the MIREX woodwnd quntet dataset. Fgure 2 shows a bar graph of the performance of the method for dfferent polyphones. Fgure 3 shows the average accuraces for dfferent nstruments n varous polyphones Accuracy Precson Recall F measure 0.1 Md note number Ground truth oboe Estmated oboe Ground truth clarnet Estmated clarnet Ground truth Bassoon Estmated Bassoon Fgure 2. Performance vs. polyphony for the MIREX woodwnd quntet dataset (Fve 30-s segments per polyphony) Bassoon Clarnet Horn Oboe Flute Tme (sec) Fgure 1. Ptch (n md note numbers) vs. tme usng the proposed system on the 11-s woodwnd tro excerpt (bassoon (lower), clarnet (mddle), oboe (upper)). Thn lnes represent the ground-truth. We then tested the proposed method on two datasets used n the MIREX multple-f0 task test set [2]. The frst one s a multtrack recordng of the woodwnd quntet mentoned above [2]. The pece s hghly contrapuntal as opposed to consstng of a lone melodc voce plus accompanment. The predomnant melodes alternate between nstruments. The F0 tracks often cross each other. Fve Ave. Accuracy Fgure 3. Ave. accuracy of ndvdual nstruments for dfferent polyphones for the MIREX woodwnd quntet dataset. (Fve 30-s segments) The second dataset we tested our method on s a recordng of a four-part J.S. Bach chorales created by [5] consst-

5 ng of bassoon, clarnet, saxophone, and vola. Four 30 seconds sectons were mxed from duo to quartet, resultng n 12 tracks (2 dfferent polyphones tmes 4 sectons). The average results over all tracks n ths dataset can be seen n Table 4. Accuracy Precson Recall F-measure Table 4. Average performance for the MIREX Bach chorales dataset Fgure 4 shows a bar graph of the performance of the method for dfferent polyphones. Fgure 5 shows the average accuraces of dfferent nstruments n varous polyphones. The RWC dataset of the MIREX multple-f0 task was not evaluated because RWC samples were used for tranng. Also the pano dataset was not used because our project s goal s to track dstnct tmbres Accuracy Precson Recall F measure Fgure 4. Ave. Performance vs. polyphony for the MIREX Bach chorales dataset. (Four 30-s segments) Ave. Accuracy 0.8 Bassoon Clarnet Saxophone Voln Fgure 5. Ave. accuracy of nstruments n dfferent polyphones for the MIREX Bach chorales quntet dataset.(four 30-s segments) 4. DISCUSSION The proposed method on average performed wth 83% accuracy on dentfyng the nstrument tracks for the tro case (Fgure 1) whch was used for the development of the algorthm. Accuraces were 52% and 59% for the MIREX woodwnd quntet and Bach chorales quartet datasets. By examnng the MIREX dataset results, we see that most problems are caused by an nactve nstrument followng the domnant nstrument s F0 track. Some nstruments n ths dataset are nactve durng 70-80% of the entre duraton of the nput. Accuracy-vs.-nstrument results for the MIREX woodwnd quntet (Fgure 3), ndcate that nstruments horn and clarnet have lower performance n the 4 and 5 polyphony cases, due to ther F0 tracks beng hghly sparse. In addton to remanng slent much of the tme, they often play soft notes when they are actve. The trackng system reports note probabltes for every non-slent frame, whch are then used n an HMM to estmate the F0 tracks. Vocng decsons are based strctly on the rms ampltude of the mxture nput sgnal to decde whether the sgnal s slent. Ths results n a hgh number of false postves when an ndvdual nstrument s slent n parts of a track, a condton whch happens often n the MIREX dataset. Ths behavor also results n a very low number of true and false negatves (snce the system reports very few negatves) resultng n a hgher recall than lower precson whch can be seen n Fgures 2 and 4. In the future, we would lke to explore methods to nfer whether nstruments are actve or not at any gven pont n tme. Another knd of error that frequently occurs s when a less domnant nstrument tracks a more domnant one that has a smlar tmbre. Ths s probably caused by the nstrument spectra havng sgnfcant correlaton wth each other. Lookng at the performance-vs.-polyphony results for each nstrument n Fgure 5, we see that nstruments lke voln, whch have a unque tmbre, have better average accuracy. Possble solutons nclude tranng the nstrument spectrum dctonares not n solaton but n combnaton wth other nstrument spectra, whch may assst the bass vectors behavng n a more dscrmnant way. Methods for dscrmnant non-negatve tensor factorzatons are explored n [19]. Ths ssue can also be addressed n the testng part. Ptch probabltes n EM teratons can be estmated to be as maxmally dfferent as possble, whle stll explanng the overall mxture by approprate use of prors. Another method that mght mprove ths ssue would be to use a factoral HMM to jontly estmate the ptch tracks. On average, the proposed method scored 0.53 for Accuracy on the MIREX dataset, whch s an mprovement over the only past result (0.21) for the multple-f0 trackng task evaluated at MIREX [10]. However, we note that the MIREX multple-f0 task dd not offer the opportunty to utlze nstrument names for the mxture nput test fles. One reason the proposed method works comparatvely well for the tro and the Bach chorales case (see Table 1 and Fgure 1) s that most nstruments were actve most of the tme. The encouragng results from ths case lead us to

6 beleve that the ptch trackng problems can be mproved effectvely by addressng the ssues dscussed above. Fnally, we note that the restrcton that sounds must consst soley of harmonc partals can be relaxed. E.g., ptch estmates for nstruments lke xylophone, whch do not have strct harmonc structures but do have predctable nharmonc structures, can be determned. 5. CONCLUSION A new method for ptch and nstrument trackng of ndvdual nstruments n polyphonc musc has been desgned and evaluated on an establshed dataset that has prevously been used for multple-f0 estmaton under MIREX. Current results are encouragng, but several problems need to be resolved (as descrbed n the Dscusson secton) for the method to be an effectve tool. As mentoned n the Introducton, knowng the F0 tracks can be very benefcal for a varety of MIR tasks. In the future, we would lke to explore the use of vocng detecton to determne whch nstruments are actve. We would also lke to perform nstrument dentfcaton n the front-end so the method does not requre pror knowledge of the nstrumentaton. We also plan to experment wth dfferent dscrmnant learnng methods and to re-nfer the dctonares based on the nput mxture. Fnally, we propose to explore usng an automatc key detecton algorthm and a more muscologcally nformed ptch transton matrx for the hdden Markov model. 6. REFERENCES [1] M. Bay and J. Beauchamp. Harmonc source separaton usng prestored spectra. Proc. Independent Component Analyss and Blnd Sgnal Separaton, pages , [2] M. Bay, A.F. Ehmann, and J.S. Downe. Evaluaton of multple-f0 estmaton and trackng systems. In Proc. 10th Int. Conf. for Musc Informaton Retreval (ISMIR 2009), pages , [3] W.C. Chang, A.W.Y. Su, C. Yeh, A. Roebel, and X. Rodet. Multple-f0 trackng based on a hgh-order hmm model. In Proc. 11th Int. Conf. Dgtal Audo Effects (DAFx-08), pages , [4] M. Desante-Catherne and S. Marchand. Hghprecson Fourer analyss of sounds usng sgnal dervatves. J. Audo Eng. Soc, 48(7/8): , [5] Z. Duan, J. Han, and B. Pardo. Harmoncally nformed mult-ptch trackng. In Proc. 10th Int. Conf. on Musc Informaton Retreval (ISMIR 2009), pages , [6] Z. Duan, J. Han, and B. Pardo. Song-level mult-ptch trackng by heavly constraned clusterng. In Proc. IEEE Int. Conf. on Acoustcs, Speech, and Sgnal Processng (ICASSP 2010), pages 57 60, [7] M.R. Every and J.E. Szymansk. Separaton of synchronous ptched notes by spectral flterng of harmoncs. IEEE Trans. Audo, Speech and Language Processng, 14(5): , [8] G.D. Forney Jr. The Vterb algorthm. Proceedngs of the IEEE, 61(3): , [9] M. Goto. Development of the RWC musc database. In Proc. 18th Int. Congress on Acoustcs (ICA 2004), volume 1, pages , [10] IMIRSEL. Multple fundamental frequency estmaton and trackng results wk. http: // Multple_Fundamental_Frequency_ Estmaton_%26_Trackng_Results, [11] A. Klapur. Multptch analyss of polyphonc musc and speech sgnals usng an audtory model. IEEE Trans. on Audo, Speech, and Language Processng, 16(2): , [12] R. Marxer, J. Janer, and J. Bonada. Low-latency nstrument separaton n polyphonc audo usng tmbre models. Proc. 10th Int. Conf. on Latent Varable Analyss and Sgnal Separaton, pages , [13] G. Mysore, P. Smaragds, and B. Raj. Non-negatve hdden markov modelng of audo wth applcaton to source separaton. Proc. 8th Int. Conf. on Latent Varable Analyss and Sgnal Separaton, pages , [14] A. Pertusa and J.M. Inesta. Multple fundamental frequency estmaton usng gaussan smoothness. In Proc. IEEE Int. Conf. on Acoustcs, Speech and Sgnal Processng,(ICASSP 2008), pages , [15] G.E. Polner, D.P.W. Ells, A.F. Ehmann, E. Gómez, S. Strech, and B. Ong. Melody transcrpton from musc audo: Approaches and evaluaton. IEEE Trans. on Audo, Speech, and Language Processng, 15(4): , [16] B. Raj and P. Smaragds. Latent varable decomposton of spectrograms for sngle channel speaker separaton. In Proc. IEEE Workshop on Applcatons of Sgnal Processng to Audo and Acoustcs, (WASPAA 2005), pages 17 20, [17] P. Smaragds, B. Raj, and M. Shashanka. A probablstc latent varable model for acoustc modelng. Proc. 20th Conf. on Neural Informaton Processng Systems (NIPS 2006), 148, [18] C. Yeh, A. Roebel, and X. Rodet. Multple fundamental frequency estmaton and polyphony nference of polyphonc musc sgnals. IEEE Trans. on Audo, Speech, and Language Processng, 18(6): , [19] S. Zaferou. Dscrmnant nonnegatve tensor factorzaton algorthms. IEEE Trans. on Neural Networks, 20(2): , 2009.