AN INTEGRATED APPROACH TO MUSIC BOUNDARY DETECTION

Transcription

1 10th International Society for Music Inforation Retrieval Conference (ISMIR 2009) AN INTEGRATED APPROACH TO MUSIC BOUNDARY DETECTION Min-Yian Su, Yi-Hsuan Yang, Yu-Ching Lin, Hoer Chen National Taiwan University ABSTRACT Music boundary detection is a fundaental step of usic analysis and suarization. Existing works use either unsupervised or supervised ethodologies to detect boundary. In this paper, we propose an integrated approach that takes advantage of both ethodologies. In particular, a graph-theoretic approach is proposed to fuse the results of an unsupervised odel and a supervised one by the knowledge of the typical length of a usic section. To further iprove accuracy, a nuber of novel id-level features are developed and incorporated to the boundary detection fraework. Evaluation result on the RWC dataset shows the effectiveness of the proposed approach. 1. INTRODUCTION Popular songs usually coprise several usic sections such as intro, verse, chorus, bridge and outro. A usic boundary is the tie point where a section transits to another. Identifying such boundaries is iportant because it allows us to divide a song into seantically eaningful sections. This inforation can also be applied to usic suarization [1] and thubnailing [2] to facilitate usic browsing and structure-aware playback [3]. Boundary detection also serves as a front-end processor for usic content analysis since it provides a local description of each section rather than a global but coarse representation of the whole song [5]. Although there is a rich literature in usic theory about usic structure analysis for sybolic usic (e.g. [20]), usic boundary detection for usic signals is still a challenging task because precise pitch detection in polyphonic usic is not yet achievable. Under this condition, ost work on usic boundary detection utilizes the siilarity between short-ter (e.g., 23s) audio fraes within a song to identify the repetitive parts and divide a song into a nuber of sections [1 3, 6 8]. A ore recent work forulates boundary detection as a clustering proble and considers that the audio fraes of each cluster belong to the sae usic section [9]. The accuracy of this unsupervised approach, however, ay be liited because only the inforation of a song itself is exploited. For exaple, identifying repetitive parts cannot correctly identify the boundary between two adjacent usic sections that always occur successively in Perission to ake digital or hard copies of all or part of this work for personal or classroo use is granted without fee provided that copies are not ade or distributed for profit or coercial advantage and that copies bear this notice and the full citation on the first page International Society for Music Inforation Retrieval Figure 1. A scheatic diagra of the proposed usic boundary detection syste. a song. On the other hand, clustering-based ethods tend to produce over-segented results if the acoustic property of the fraes in a usic section varies greatly. Using histogras to gather statistic of spectral characteristics of neighboring audio fraes [9] does not necessarily solve the proble because the histogras of two adjacent fraes are usually siilar, aking boundary detection even ore difficult. To address the aforeentioned drawbacks, Turnbull et al forulate usic boundary detection as a supervised proble and train a binary classifier to classify whether a tie point is a boundary or not [10]. In this way, we can ine ore inforation fro a large nuber of training songs and identify features that are relevant to boundary detection. However, because a supervised syste is pre-trained by using the training data and fixed afterwards, it is not as adaptive to test songs as its unsupervised counterpart. The detection accuracy ay significantly degrade when the characteristics of the training data and a test song are considerably different. For instance, if the syste detects boundary according to the energy level in a certain frequency range, the syste ay not work for a song whose energy in that frequency range aintains high throughout the song. Based on the above observations, we propose to take advantage of both ethodologies by aggregating the results of an unsupervised odel and a supervised one. In this way, we can exploit the discriinative inforation provided by the training data and the song-specific 705

2 Poster Session 4 inforation of a test song at the sae tie. Moreover, to better capture the discriinative characteristics of a boundary, we further propose a nuber of novel id-level features, including novelty score, dissonance level and vocal occurrence. Coparing to low-level features such as the spectral properties, these id-level features carry ore seantic eaning that iproves usic boundary detection. A scheatic diagra of the proposed syste is shown in Fig. 1. An input song is partitioned by the beat onsets and represented by a set of low-level and id-level features. The probability of each beat onset of being a boundary is then coputed by both supervised and unsupervised ethods with the features extracted fro the subsequent beat interval. We then odel the beat onsets as the vertices of a directed graph, with the vertex weights deterined by the probability of being a boundary and the edge weights deterined based on the usic knowledge of the typical length of a usic section [7, 11]. Finally, we forulate usic boundary detection as a shortest path proble and identify the true boundaries by the Viterbi algorith [18]. The paper is organized as follows. Section 2 describes the feature representation of usic, including low-level and id-level features. Section 3 elaborates on the syste fraework and the adopted supervised and unsupervised approaches. Experiental result is presented in Section 4. Section 5 concludes the paper. 2. MUSICAL REPRESENTATION Before feature extraction, each song is converted to a standard forat (ono channel and 22,050 Hz sapling rate) and partitioned into several beat intervals by the beat onset detection algorith BeatRoot [12]. We adopt beat interval instead of frae as the basic tie unit because the characteristics of a song are ore likely to be consistent within a beat interval and because a usic boundary tends to occur at a beat onset [7]. 2.1 Low-level Features For low-level local features, we use 40-di Mel-scale cepstral coefficients (MFCCs), 24-di chroagra, and 52-di fluctuation patterns (FPs) [19] to represent the tibre, harony, and rhyth aspects of usic. We extract MFCCs and chroagra with a 40s and non-overlapping sliding window and aggregate the frae-level features within each beat interval by taking the ean and the standard deviation. FPs are coputed directly for each beat interval. These features have been found useful for usic boundary detection [10]. Note these features only capture the local property of usic. 2.2 Mid-level Features Below we describe three id-level features: novelty score, dissonance level, and vocal occurrence. While the first one is originally proposed by Cooper et al in [4], it has been used in an unsupervised setting rather than as a id-level feature in a supervised one. On the other hand, though the latter two features have been studied in the context of usic theory [21], few attepts have been ade to incorporate the to the task of usic boundary detection for raw audio signals Novelty Score The novelty score is coputed by two steps [4]. First, a siilarity atrix is constructed by easuring the siilarity of the low-level feature vectors of every two beats in a song. In this atrix, the two segents beside the boundary produce two adjacent square regions of high within-segent siilarity along the ain diagonal and two rectangular regions of low between-segent siilarity off the ain diagonal. As a result, each boundary produces a checkerboard pattern in the atrix and the beat interval that boundary occurs is the crux of this checkerboard. To identify these patterns, we correlate a Gaussian-tapered checkerboard kernel along the ain diagonal of the siilarity atrix to copute the so-called novelty scores, which easures both the dissiilarity between two different adjacent segents beside each potential boundary as well as the siilarity within these segents. We define the ter segent here to represent a set of consecutive beat intervals and the ter section as a segent which is seantically eaningful (such as verse, chorus or bridge). 1 In this work, we copute three novelty scores based on the three low-level features. Because the novelty scores of adjacent beats tend to be siilar, 2 we also divide the novelty score of a certain beat interval by the su of the novelty scores of neighboring beat intervals and use the noralized score as additional feature, resulting in a total of 6 features for each beat interval Dissonance Level It is known in usicology that the relaxation or release of tension plays an iportant role in the transition of usic sections. Because changes in tension often occur when dissonance giving way to consonance [13], we develop a novel feature based on the dissonance level of usic. We first define the dissonant intervals according to the relationship between the pitches of two notes that cause tension (e.g., Tritone and Minor Second [14]), and then copute the dissonance level as the weighted su of the corresponding dissonant intervals fro the unwrapped chroagra of a beat, q D + q y t =, (1) c where y t denotes the dissonance level of a beat t, q denotes the interval that has q seitones between the two notes, D is the set of dissonant intervals, c is the th bin of the chroagra, and k q is a constant corresponding to q, which is epirically set according to the ratio of frequencies of the two pitches in q. The denoinator is a noralization ter. 1 While a segent can be of arbitrary length, the length of a section often follows a typical pattern, see Section The novelty scores of adjacent beats are siilar because the subatrices of the siilarity atrix of these beats overlap a lot. k q c c 706

3 10th International Society for Music Inforation Retrieval Conference (ISMIR 2009) Figure 2. The dissonance level of a part of Billie Jean by Michael Jackson. The two red lines label a transition fro verse to bridge and a transition fro bridge to chorus. These boundaries occur right after high dissonance levels. We copute the dissonance level for each beat interval and obtain a sequence of dissonance levels. We copute the derivative fro the resulting sequence as the dissonant features to capture the changes in tension, p = p Δ yt = p = p y t 2, (2) where p denotes the window size. In this work, we set p to 1 and 2 and generate a two diensional dissonance level feature. Fig. 2 illustrates the relationship between usic boundary and dissonance level; clearly the usic boundaries occur right after peaks of dissonance level (the rise and relax of tension) Vocal Occurrence In pop/rock songs, the tie points that a vocalist sings often correspond to the usic boundaries. For exaple, if a beat onset falls in the iddle of a segent with pure instruent and another segent with singing voice, it is very likely a usic boundary. Furtherore, because a usic section is coprised of several usic phrases, 3 a transition of usic sections ust also be a transition of usic phrases. Therefore, if a beat onset falls in a short instruental interval between two vocal usic phrases, it is ore likely to be a usic boundary. In light of the above observation, we train a vocal/non-vocal classifier by support vector achine (SVM) [15], with MFCC as the feature representation, to estiate the probability of the vocal occurrence for each beat interval. If the su of these probabilities fro the beat intervals in a segent exceeds a threshold, we regard the segent as a vocal segent. More specifically, the vocal occurrence feature of a certain beat interval is coputed as follows. For a beat interval, if both of its neighboring segents are non-vocal, the vocal occurence is set to 0; if only one of the neighboring segents is non-vocal, the vocal occurrence is set to 1. When both neighboring segents are vocal, we set the vocal occurrence according to the following forula: z t 1 vt = 2w + 1 t + w v j j= t w, j t, (3) where z t is the vocal occurrence feature of beat interval t, 3 Several usic phrases constitute a usic section. Figure 3. Top: the possibility of vocal estiated by SVM for a part of Billie Jean by Michael Jackson. The two red lines label a transition fro intro to verse and a transition fro verse to bridge. The green circles label two obvious transition points of usic phrases. Botto: corresponding vocal occurrence feature. v t is the probability estiate of beat interval t generated by the vocal/non-vocal SVM classifier, and ω is the window size that represents the length of the segent. We vary the value of ω and generate a ulti-diensional feature vector. In this work we set the value of ω to 8 and 12. An illustrative exaple is shown in Fig. 3. The first red line labels a transition fro a non-vocal section (intro) to a vocal section (verse). The green circles label two obvious transition points of usic phrases, while the latter one is in fact a transition point of usic sections. We can see the corresponding vocal occurrence feature is highly correlated to usic boundaries. A pitfall of this feature is that it ay regard every phrase boundary as a section boundary and result in over segentation. The use of other features ay offset this istake. Representing the acoustic properties of usic by these low-level and id-level features, we then eploy the syste described below to detect boundaries. 3. SYSTEM DESCRIPTION In this section, we first introduce the supervised and unsupervised approaches adopted in our syste. Both approaches estiate the possibility of each beat onset of being a usic boundary. Second, we describe how we integrate these two estiations with the usic knowledge of typical section length. 3.1 Supervised Estiation We train a SVM classifier with polynoial kernel and probability estiates to obtain the possibility of a beat onset being a usic boundary. The label for a beat interval is arked 1 if a boundary occurs at that beat onset and 0 otherwise. Besides id-level features, we also use the low-level features to train the classifier because low-level features also contain soe relevant inforation. For exaple, a dru-fill is usually played when a usic section ends; this characteristic can be detected by FP. For a test song, the SVM odel 707

4 Poster Session 4 coputes the probability of the occurrence of a boundary at every beat onset. We utilize this probability as the output of the supervised approach. 3.2 Unsupervised Estiation As for the unsupervised part, we construct three siilarity atrices based on the kinds of low-level features and detect the peaks of the ean of the novelty scores fro these atrices. We then use these peaks to divide the test song into a nuber of segents [4]. The low-level features of a segent are integrated to one vector by taking the ean and the standard deviation and a distance atrix aong the segents is constructed by coputing the pairwise distance between these vectors. The noralized cut algorith [16] is then perfored on the distance atrix to group these segents into acoustic siilar clusters. At each beat interval, we further count the cluster indices of neighboring beat intervals within a predefined window size and establish two histogras: one for the beat intervals preceding to the beat onset, and the other for the subsequent beat intervals. The Euclidean distance of the resulting histogras can represent the possibility of a usic boundary occurs at the designated beat onset, and the ratio of this possibility value of a beat onset to the su of the possibility values of its neighboring ones is regarded as the estiation of the unsupervised approach. 3.3 Integration Because usic sections tend to have soe typical length (e.g., 8 or 16 bars) [7, 11], it should be beneficial to incorporate this knowledge to the usic boundary detection fraework. As Fig. 4 illustrates, we construct a directed graph G = (V, E) to integrate the estiates of supervised and unsupervised odels and to take advantages of this usic knowledge. In this graph, a vertex represents a beat onset, with the weight of it deterined by the weighted su of the estiates of supervised and unsupervised odels w = p + k p, (4) vi ui 1 si where w vi denotes the weight of a vertex i, p ui and p si are the probability estiates produced by an unsupervised odel and a supervised one respectively, and k 1 is a paraeter balancing the effect of the two odels. The usic knowledge of section length is incorporated as follows. If there exists the possibility that vertices v i and v j are two successive usic boundaries, we for an edge between these two vertices. The weight of the edge is deterined by the usic knowledge of the length of a usic section. We gather the statistics fro training data to obtain the probability of two beats with specific teporal distance being usic boundaries. That is, the weight of e ij equals to the weight of e n if j i equals to n. To achieve this goal, a histogra is constructed by siply counting the nuber of beats of each usic section fro the training data. Therefore, a path in this constructed graph can be regarded as a set of usic boundaries. We further define the weight of a path B as the su of the weights of its constituent edges and vertices, Figure 4. The directed graph G of a song, which has n beat onsets (vertices) and k 1 possible section lengths (possible jups). The vertex weights are deterined by the probability of being a boundary and the edge weights are deterined based on the usic knowledge of the typical length of a usic section [7, 11]. We assue that every usic section contains at least one beat interval., (5) w = w + k w B v B v 2 e B e where w v and w e are the weights of a vertex and an edge in B, and k 2 is a constant to balance the effects of vertices and edges. We regard w B as the probability of the associated beat onsets being correct usic boundaries. Because the path with axiu w B consists of vertices that are ost likely the usic boundaries, we forulate the proble as a shortest path proble and eploy the Viterbi algorith [18] to solve it, B * = arg ax w, (6) where B * denotes the optial solution. In practice, we only apply Viterbi to a feasible nuber of paths to reduce the coplexity. B B 4. EXPERIMENT 4.1 Experiental Setup We conduct an epirical evaluation on the RWC usic dataset [17], which contains 100 pieces of song that are originally produced for experient; ost of the pieces (80%) are recorded according to 1990s Japanese chart usic, while the rest reseble the 1980s Aerican chart usic. RWC dataset provides clear annotations of usic boundaries and is adopted in any literatures in usic boundary detection [6, 10]. We evaluate the perforance in ters of precision (the proportion of true boundaries aong the detected ones), recall (the proportion of true boundaries in the ground truth that are detected by the syste), and f-score (the haronic average of precision and recall). A detected boundary is considered correct if it falls within 1.5 seconds of the ground-truth, which is stricter than the one used in prior work [9] and should be reasonable for real-world applications. For the unsupervised ethods, we process each of the 100 songs independently and take the average result. For the supervised ethods, we evaluate the syste with 708

5 10th International Society for Music Inforation Retrieval Conference (ISMIR 2009) Approach Method Precision Recall F-score Supervised only N +D+V+L Unsupervised only Cluster-based (noralized cut) [9] Histogra-based Directly su Integrated with section length Viterbi algorith [18] Table 2. Evaluation result of different usical boundary detection ethods. Feature # Precision feature Recall F-score MFCC chroagra fluct. pattern local (L) difference [10] novelty (N) dissonance (D) vocal (V) N+L N+D+V N +D+V+L Table 1. Evaluation result of different features used in supervised usical boundary detection ethods. stratified five-fold cross validation: 20 rando songs are held out as test data and the rest are used for training. The evaluation is iterated five ties to get the average result. 4.2 Results We first evaluate the supervised approach with different feature representations, including low-level and id-level features. To copare the perforance against previous work, we also ipleent the difference feature and its derivative proposed in [10]. The difference feature is coputed by sliding a window along the audio signal and coparing the statistic of low-level features in the first half of the window with the ones in the second half. A beat onset is detected as a boundary if its probability estiate assigned by SVM exceeds a threshold. Instead of using a fixed threshold, we adaptively set the threshold of each song to be the ean plus one standard deviation of the probability estiates of the song. The evaluation result is shown in Table 1. The three low-level features bring about siilar accuracy, with FPs slightly worse than the other two, iplying that the characteristics of usic boundaries are represented ore in tibre and rhyth. The direct concatenation of the three low-level features, which are denoted as local (L) in the table, further iproves the f-score to We then copare four id-level features, including the difference feature proposed in [10]. It can be found that, with uch lower feature diension, the use of id-level features achieves siilar or superior perforance to that attained by low-level features. The novelty score, in particular, achieve an f-score of that significantly outperfor all other low-level or id-level features. We can also find that the difference feature does not perfor well, which possibly due to the disregard of the siilarity of the beats in each segent. The cobination of id-level and low-level features only brings about slight iproveent, which soewhat iplies that ost of the inforation carried by low-level features has already been well represented by the id-level features. The cobination of novelty score (N), dissonance level (D), vocal occurrence (V), and local features (L) achieves the highest f-score of We then copare the two unsupervised ethods described in Section 3.2. For the cluster-based ethod, we siply ark the boundary of two consecutive segents that are associated with different clusters as a usic boundary without soothing. The result is shown in Table 2. As expected, the clustering-based approach exhibits a rearkably high recall but a relatively low precision. For the histogra-based ethod, we consider the segents whose probability estiates exceed a threshold as boundaries. The threshold value is set in the sae way as in the supervised ethods. The perforance of the histogra-based ethod is slightly worse than the clustering-based one, showing that gathering statistics of neighboring fraes does not iprove the precision of boundary detection. Moreover, it can be noted that in our evaluation the unsupervised approaches generally outperfor the supervised counterparts, showing that the ability of the unsupervised approach to be adaptive to each test song is essential in boundary detection. Finally, we evaluate the perforance of integrating the result of unsupervised and supervised ethodologies. For coparison, we further ipleent a baseline ethod that siply sus up the supervised and unsupervised estiates with the sae weight as the one in proposed graph-theoretical fusion ethod without exploiting the usic knowledge of section length. The result is also shown in Table 2. It can be found that siply taking the average has achieved a higher f-score than any of the supervised-only or unsupervised one, showing that the two ethodologies are indeed copleentary and the fusion of the is plausible. The proposed graph-theoretical fusion further iproves the f-score to , which greatly outperfor the taking average baseline, especially in recall. This result shows the integration of the two ethodologies and the incorporation of usic knowledge are essential to usic boundary detection. A saple segentation result is displayed in Fig. 5. In this exaple, all the boundaries can be correctly detected by the proposed syste. Nevertheless, there is an over segentation proble because the characteristics of the segents of the sae usic section ay be incoherent. 709

6 Poster Session 4 Figure 5. The segentation result of Billie Jean by Michael Jackson. Top: the boundaries detected by the proposed syste. Botto: the anual annotation. To resolve this proble, we are working on incorporating ore usic knowledge and id-level features. 5. CONCLUSION In this paper, we have presented an integrated syste that cobines the inforation fro supervised approach, unsupervised approaches, and usic knowledge. We forulate usic boundary detection as a shortest path proble and eploy the Viterbi algorith to solve it. We also propose a nuber of novel id-level features to better capture the discriinative characteristics of usic boundaries. Experients conducted on the RWC dataset show significant iproveent over the state-of-the-art supervised-only and unsupervised-only ethods. 6. ACKNOWLEDGEMENT This work was supported by the National Science Council of Taiwan under the contract nuber NSC E MY3. The authors would like to thank the anonyous reviewers for valuable coents that greatly iproved the quality of this paper. 7. REFERENCES [1] W. Chai, Seantic segentation and suarization of usic, in IEEE Signal Processing Magazine, Vol. 23, No. 2, pp , [2] M. Levy, M. Sandler, and M. Casey, Extraction of high-level usical structure fro audio data and its application to thubnail generation, in Proc. ICASSP, pp , [3] M. Goto, A chorus-section detection ethod for usical audio signals and its application to a usic listening station, in IEEE Trans. Audio, Speech and Language Processing, Vol.14, No.5, pp , [4] M. Cooper and J. Foote, Suarizing popular usic via structural siilarity analysis, in Proc. IEEE Workshop Applications of Signal Processing to Audio and Acoustics, pp , [5] M. Casey, R. Veltkap, M. Goto, M. Lean, C. Rhodes, and M. Slaney, Content-based usic inforation retrieval: current directions and future challenges, in Proceedings of the IEEE, Vol. 96, No. 4, pp , [6] J. Paulus and A. Klapuri, Music structure analysis using a probabilistic fitness easure and a greedy search algorith, in IEEE Trans. Audio, Speech and Language Processing, Vol. 17, No. 6, pp , [7] N. C. Maddage, C. Xu, M. S. Kankanhalli, and X. Shao, Content-based usic structure analysis with applications to usic seantics understanding, in Proc. ACM Multiedia, pp , [8] L. Lu, M. Wang, and H. Zhang, Repeating pattern discovery and structure analysis fro acoustic usic data, in Proc. ACM SIGMM Int. Workshop on Multiedia Inforation Retrieval, [9] M. Levy and M. Sandler, Structural segentation of usical audio by constrained clustering, in IEEE Trans. Audio, Speech and Language Processing, Vol. 16, No. 2, pp , [10] D Turnbull, G. Lanckriet, E. Papalk, and M. Goto, A supervised approach for detecting boundaries in usic using difference features and boosting, in Proc. ISMIR [11] C. Rhodes et al, A Markov-chain Monte-Carlo approach to usical audio segentation, in Proc. ICASSP, [12] S. Dixon, Evaluation of the audio beat tracking syste BeatRoot, in Journal of New Music Research, Vol, 36, No. 1, pp , [13] DeLone et al, Aspects of Twentieth-Century Music. Englewood Cliffs, New Jersey: Prentice-Hall, 1975 [14] Wyatt and Keith, Harony & Theory. Hal Leonard Corporation, pp. 77, [15] N. Maddage et al, An SVM-based classification approach to usical audio, in Proc. ISMIR, [16] Ji. Shi and J. Malik, Noralized cuts and iage segentation, in Proc. CVPR, pp , [17] M. Goto, AIST annotation for RWC usic database, in Proc. ISMIR, [18] G. D. Foey, The Viterbi algorith, in Proceedings of the IEEE Vol. 61, No. 3, pp , [19] E. Papalk, Coputational odels of usic siilarity and their application in usic inforation retrieval, in PhD thesis, Vienna University of Technology, [20] F. Lerdahl and R. Jackendoff, An overview of hierarchical structure in usic, in Machine Models of Music, pp , [21] D. Pressnitzer, S. McAdas, Two phase effects in roughness perception, in The Journal of the Acoustical Society of Aerica, Vol. 105, No. 5, pp ,