Performance Evaluation of TreeQ and LVQ Classifiers for Music Information Retrieval

Transcription

1 Performane Evaluaton of TreeQ and LVQ Classfers for Mus Informaton Retreval Matna Charam, Ram Halloush, Sofa Tsekerdou Athens Informaton Tehnology (AIT) 0.8 km Markopoulo Ave. GR Peana, Athens, Greee {sha, raha, sots}@at.edu.gr Abstrat. Classfaton algorthms are ganng more and more mportane n many felds suh as Artfal Intellgene, Informaton Retreval, Data Mnng and Mahne Vson. Many lassfaton algorthms have emerged, belongng to dfferent famles, among whh the tree-based and the lusterng-based ones. Suh extensve avalablty of lassfers makes the seleton of the optmal one per ase a rather omplex task. In ths paper, we am to address ths ssue by ondutng extensve experments n a mus nformaton retreval applaton, spefally wth respet to mus genre queres, n order to ompare the performane of two state-of-the-art lassfers belongng to the formerly mentoned two lasses of lassfaton algorthms, namely, TreeQ and LVQ, respetvely, usng a varety of mus features for suh a task. The deployed performane metrs are extensve: auray, preson, reall, F- measure, onfdene. Conlusons on the best performane of ether lassfer to support mus genre queres are fnally drawn. 1 Introduton Wth the explosve amount of mus data avalable on the Internet n reent years, there has been muh nterest n developng new ways to searh and retreve suh data effetvely. Most on-lne mus databases today, suh as Napster and mp3.om, rely on fle names or text labels to do searhng and ndexng, usng tradtonal text searhng tehnques. Although ths approah has proven to be useful and wdely aepted n the past, there are many reasons ths s not enough nowadays. As the amount of musal ontent nreases and the Web beomes an mportant mehansm for dstrbutng mus, we expet to see a rsng demand for mus searh serves. It would be ne to have more sophstated searh apabltes, namely, searhng by ontent. Mus Informaton Retreval Systems an be lassfed nto two types: ) systems that depend on human generated annotatons and desons (textual-based), and )

2 332 Matna Charam, Ram Halloush, Sofa Tsekerdou systems that depend on extratng nformaton from the audo sgnal (ontent-based). The frst type s manual and hene demands a lot of effort, t s tme onsumng and suseptble to errors. We wll onentrate on ontent-based mus nformaton retreval n the rest of the paper. Content-based mus nformaton retreval nvolves proesses suh as representatve mus feature extraton, lassfaton n apror known lasses, usually deployng tranng and testng (supervsed lassfaton) and smlartybased queryng. Thus, the hallengng aspets of settng up an effent mus nformaton retreval are: ) what features to selet as most representatve on the types of smlarty-based queres (e.g. mus genre queres), ) whh lassfers wll perform optmally for the applaton and types of queres at hand. In ths paper, we am to address the ssue of seletng the optmal ombnaton of representatve features and lassfers to address queres on mus genre, by ondutng extensve experments n a mus nformaton retreval applaton. The am s to ompare the performane of two state-of-the-art lassfers, namely, TreeQ [1] and LVQ [2] (Learnng Vetor Quantzaton), desrbed brefly n the sequel, usng a varety of mus features for suh a task. The deployed performane metrs are extensve: auray, preson, reall, F-measure, onfdene. Conlusons on the best performane of ether lassfer ombned wth spef mus feature vetors are fnally drawn. 2 LVQ Classfer: a short overvew In general, LVQ [6] s a supervsed verson of vetor quantzaton, whh s applable to pattern reognton, mult-lass lassfaton and data ompresson. LVQ algorthms dretly defne lass boundares based on prototypes, a nearestneghbour rule and a wnner-takes-t-all paradgm. The man dea, as shown n Fgure 1, s to over the nput spae of samples wth odebook vetors (CVs), eah representng a regon labeled wth a lass. A CV s loalzed n the entre of a deson regon, alled Vorono ell, n the nput spae. Fg. 1. LVQ spae parttonng nto deson regons by odebook vetors [2]. For the purpose of undertaken experments, the LVQ software pakage of [7] has been used, whh mplements all algorthms neessary for statstal lassfaton

3 Performane Evaluaton of TreeQ and LVQ Classfers for Mus Informaton Retreval 333 and pattern reognton. The performane of LVQ depends on the algorthm mplementaton, as well as the data used for tranng and testng, n terms of sze and the degree of representatve features extrated from suh data. The bas three mplementatons of LVQ are LVQ1, LVQ2 and LVQ3. In our experments, we have used the optmzed OLVQ1 mplementaton [1], an enhaned verson of LVQ1. The bas dea behnd ths algorthm s that eah lass s represented n terms of a set of odevetors m, eah of whh s a pont n the D-Dmensonal feature spae. Ths set s alled odebook. Several odebook vetors are assgned to eah lass. A feature vetor x s then assgned to the same lass to whh the nearest m belongs: m arg mn{ x m }, where OLVQ1 extends LVQ1 by modfyng the latter so that an ndvdual learnng rate a t s assgned to eah m. Agan, the dsrete tme learnng proess s gven from above equatons. For ther fastest onvergene, s optmally determned by: a a t 1 s a 1 t a 1 m s the nearest m to x Values for m that approxmately mnmze the mslassfaton errors n the above nearest neghbour lassfaton an be found as asymptot values n the followng learnng proess. Let x be a sample of nput and let m t represent sequenes of m n the dsrete tme doman. Startng wth properly defned ntal values, the followng equatons defne the bas LVQ1 proess, where 0 a t and t s the teraton step: 1 m 1 m a x m m 1 m a x m m 1 m, for, f x s lassfed orretly, f x s lassfed norretly 3 TreeQ Classfer: a short overvew For the purposes of the undertaken experments, the TreeQ software pakage [3], [4], [5] has been onsdered, mplementng the TreeQ mahne learnng algorthm. TreeQ s data-drven and therefore t an be used for any knd of data to fnd smlartes by learnng ther dfferenes. Espeally for the ase of audo data, the algorthm may be appled for speaker dentfaton, speeh and mus lassfaton, mus and audo retreval by smlarty, audo segmentaton. Gven labeled tranng data, the algorthm onstruts templates that haraterze these data, utlzng three man steps. Frst, t alulates spetral parameters for the audo data. Seond, t grows a quantzaton tree from labeled parameterzed data. Ths step learns those features that best haraterze a lass,.e. gven adequate tranng data, t learns the salent dfferenes amongst lasses and learns to gnore other nsgnfant dfferenes. Thrd, the produed tree s used to onstrut the

4 334 Matna Charam, Ram Halloush, Sofa Tsekerdou templates. As soon as the templates are onstruted (lass models), smlartes an be measured by alulatng dstanes between them and test data. The bas operaton of the system s llustrated n Fgure 2. A sutable orpus of audo examples must be aumulated and parameterzed nto feature vetors. The orpus must ontan examples of the lasses of audo to be dsrmnated. Next, a tree-based quantzer s onstruted. Ths s a supervsed operaton and requres the tranng data to be labeled wth a lass. The tree automatally parttons the feature spae nto regons, alled ells, whh have maxmally dfferent lass populatons. To generate an audo template, represented by a hstogram, for subsequent retreval, parameterzed data are quantzed usng the tree. An audo fle an be haraterzed by fndng nto whh ells the nput data vetors are most lkely to fall. A template s an estmate of the vetor ounts for eah ell, whh aptures the salent haratersts of the nput audo, sne sounds from dfferent lasses wll have very dfferent ounts n the varous hstogram bns, whle smlar audo data should have smlar ounts. To retreve audo by smlarty, a hstogram s further onstruted for the query audo. The query hstogram s ompared to the orpus hstograms, a smlarty measure s alulated for eah audo fle n the orpus, and fnally the query template s assoated wth a orpus template. Fg. 2. An overvew of the bas operatons of the TreeQ algorthm [1]. 4 Mus Informaton Retreval based on Genre Queres It s a fat that the rapd development of tehnology ontnuously realzes senaros that prevously seemed sene fton. Web-based mus statons lke pandora.om gve eah user the opportunty to spefy the knd of mus he wants to lsten to, and

5 Performane Evaluaton of TreeQ and LVQ Classfers for Mus Informaton Retreval 335 n the ontext of ambent ntellgene, pervasve systems wll dress the surroundngs wth mus based on the human s mood. Gven the above and a number of other emergng applatons of mus, lassfaton and nformaton retreval based on mus genres beomes not only mportant, but essental and fundamental. Our work addresses ths problem and ams to provde extended experments n order to push TreeQ and LVQ to ther lmts and dede whh gves the best results under what ontexts of use. Before presentng our expermental results, we onsdered t neessary to brefly desrbe the mus features used to parameterze the audo data,.e. the nput to the prevously presented lassfers. 4.1 Mus Feature Extraton Few lassfers dretly operate on raw data suh as pxels of an mage or samples of speeh waveforms. Most pattern reognton tasks are preeded by a pre-proessng transformaton that extrats nvarant features from raw data, suh as spetral omponents of aoustal sgnals. Thus, n our ase, desons need to be made on the types of representatve features to be used by lassfers to aheve optmal mus lassfaton based on genres. Suh feature extraton task parameterzes the raw mus data nto sequenes of representatve feature vetors. It s evdent that the seleton of the adequate pre-proessng method s equally vtal as to the seleton of the proper lassfer for optmal performane, thus, t requres areful onsderaton. For ths task, we have used the HTK Toolkt [8], whh supports Hdden Markov Models (HMMs) usng both Fast Fourer Transformaton (FFT) and Lnear Predtve Codng (LPC). The feature extraton proess s ontrolled by a ustomzable onfguraton fle that spefes all the onverson parameters towards extratng the desrable feature vetors. In the urrent nvestgaton, we have onsdered wdely known and used features, namely the mel-frequeny epstral oeffents (MFCCs) and the lnear predton odng oeffents (LPCs). Both are, n general, the parametersaton of hoe for many speeh reognton applatons, sne they attan good dsrmnaton apabltes and are flexble towards a number of manpulatons. In lnear predton analyss [9], the followng transfer funton s onsdered: where the flter oeffents H z p 0 a are hosen so as to mnmze the mean square flter predton error summed over the analyss wndow. On the other hand, epstral parameters are alulated from the log flter-bank ampltudes m j usng the Dsrete Cosne Transform (DCT), where N s the number of flter-bank hannels [9]: 2 N N j1 1 a z m j os j 0.5 N

6 336 Matna Charam, Ram Halloush, Sofa Tsekerdou 5 Expermental Setup and Performane Evaluaton In order to evaluate the performane of the two lassfers, namely, TreeQ and LVQ, n mus genre lassfaton and retreval, we performed a set of experments that are desrbed n the sequel. Intally, the expermental orpus has been reated arefully. Fve mus genres have been onsdered, namely, jazz, reggae, pop, post rok and eletro tehno. For eah genre, the orpus ontans twenty mus pees, summng up to a total of one hundred pees for all genres. Eah mus lp s about ten seonds long and has been seleted as the most representatve part of the entre mus fle. We have used the holdout samplng method n order to splt the orpus nto tranng and test data. Thus, seventy fve of the mus lps were used for tranng (the frst ffteen of eah genre) whle the remanng twenty fve were used for testng. All mus pees were later parameterzed usng the HTK toolkt, n MFCC and LPC feature vetors. Expermentaton on the lassfers (TreeQ, OLVQ1) performane followed usng the extrated feature vetors n dfferent ombnatons of features-lassfers. For the TreeQ experments, we frst obtaned the optmal wndow sze and the target rate of the algorthm. Ths was aheved by measurng the performane wth varyng values and ombnatons for these two parameters. An teratve proedure was used durng whh a new parameter was added, tested and deded to be mantaned only f the aqured performane was no worse than the best performane aheved that far. Ths was done for both MFCC and LPC features wth or wthout usng quantzaton nto hstograms. Due to lak of tme, the OLVQ1 experments were performed usng only MFCCs, wthout hstogram quantzaton. LVQ nvolved only two parameters, n (number of odebook vetors) and k (knn parameter), whose optmal values were obtaned n a smlar manner as the optmal wndow sze and target rate for TreeQ. In all the experments, the lassfer performane was measured usng Preson, Reall, Auray, F1 on preson and reall, and Confdene measures. Results are summarzed n Table 1. We observe that TreeQ outperforms OLVQ1, when LPC features are used, wth hstogram quantzaton, for all onsdered measures. Table 1. Optmal performane aheved by TreeQ and OLVQ1. Performane Metr Classfer TreeQ LVQ MFCC LPC - Hstograms MFCC Overall Auray(%) Average Auray (%) Average Preson (%) Average Reall (%) Average F-1 (%) Havng obtaned the optmal setup for eah lassfer, we reated learnng urves as shown n Fgures 3, 4 and 5. From these urves, we annot extrat vald onlusons about the learnng apablty of the two algorthms. For nstane, for TreeQ wth MFCCs, the urve s stll nreasng at the fnal steps whh mght ndate that a larger tranng set ould help us aheve better performane. On the

7 Performane Evaluaton of TreeQ and LVQ Classfers for Mus Informaton Retreval 337 other hand, at these fnal steps, the urve also seems to smoothen, so the mprovement of the performane usng more tranng data mght be nsgnfant. Fnally, omparng the two TreeQ urves wth the OLVQ1 urve, we may say that TreeQ seems to better learn that OLVQ1. However, ths s only an ndaton. More extensve experments wth a muh larger dataset (and thus bgger test and tranng data sets) need to be undertaken to draw valdated onlusons. OVERALL LEARNING CURVE % 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0, step Overall Auray Fg. 3. Overall learnng urve usng MFCCs wth TreeQ OVERALL LEARNING CURVE % 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0, step Overall Auray Fg. 4. Overall learnng urve usng LPCs wth TreeQ

8 338 Matna Charam, Ram Halloush, Sofa Tsekerdou OVERALL LEARNING CURVE 1 0,9 0,8 0,7 0,6 %Overall 0,5 0,4 0,3 0,2 0, step Auray Fg. 5. Overall learnng urve usng MFCCs wth LVQ 6 Conlusons and Future Work In ths paper, we have underlned the mportane of ontent-based retreval, whh we addressed by ondutng extensve experments for mus nformaton retreval, based on mus genre queres, n order to ompare the performane of two state-ofthe-art lassfers, TreeQ and LVQ. From the performane evaluaton, we ould not reah vald onlusons, however, we have dentfed performane hnts to extend the work further towards a ertan dreton. The learnng apablty of both algorthms needs to be further explored and hene we ntend to undertake more experments wth a larger dataset as ontnuaton of the urrently reported work. Referenes 1. Jonathan T. Foote, TreeQ Manual V0.8, September, T. Kohonen, H. Hynnnen, J. Kangas, H. Laaksonen, and K. Torkkola. LVQ-PAK: The learnng vetor quantzaton program pakage, Tehnal Report A30, Helsnk Unversty of Tehnology, Laboratory of Computer and Informaton Sene, FIN Espoo, Fnland, Jonathan T. Foote, Content-based retreval of mus and audo, Multmeda Storage and Arhvng Systems II, Proeedngs of SPIE, Jonathan T. Foote, An overvew of audo nformaton retreval, Multmeda Syst., Sprnger-Verlag New York, In., Seauus, NJ, USA, Mus retreval demo usng open-soure software pakage TreeQ by Jonathan T. Foote, 6. Foreastng wth artfal neural networks, 7. Helsnk Unversty of Tehnology Neural Networks Researh Centre, 8. The HTK Toolkt, 9. Steve Young et al., The HTK Book ( Mrosoft Corporaton, Cambrdge Unversty Engneerng Department)