MPEG-7 Based Music Metadata Extensions for Traditional Greek Music Retrieval

Transcription

1 MPEG-7 Based Music Metadata Extensions for Traditional Greek Music Retrieval Sofia Tsekeridou 1, Athina Kokonozi 2, Kostas Stavroglou 2, and Christodoulos Chamzas 2 1 Athens Information Technology, Building B7, 19.5 km Markopoulo Ave., Peania, Athens, Greece sots@ait.edu.gr 2 Cultural and Educational Technology Institute, 58 Tsimiski Str., Xanthi, Greece {athinako, kstavrog, chamzas}@ceti.gr Abstract. The paper presents definition extensions to MPEG-7 metadata ones, mainly related to audio descriptors, introduced to efficiently describe traditional Greek music data features in order to further enable efficient music retrieval. A number of advanced content-based retrieval scenarios have been defined such as query by music rhythm, query by example and humming based on newly introduced music features, query by traditional Greek music genre, chroma-based query. MPEG-7 DDL extensions and appropriate traditional greek music genre dictionary entries are deemed necessary for accounting for the specificities of the music data under question and for attaining efficiency in music information retrieval. The reported work has been undertaken within the framework of an R&D project targeting an advanced music portal offering a number of content-based music retrieval services, namely POLYMNIA[12]. 1 Introduction The significant research efforts and the advent, in our everyday life, of advanced technologies for capturing, management, storage, transmission and use of multimediacontenthasledtoakeeninterestofmajor business actors in enabling next generation multimedia services. To allow for intelligence and automation in the manipulation of content and services throughout their entire life-cycle, metadata descriptions are crucial [4]. This is also justified by the extensive work performed within standardisation bodies to define appropriate metadata descriptions, such as the MPEG-7 standard [5,2,1], that introduces generic multimedia content descriptions, both low-level and high-level (semantic) ones, the MusicXML standard [11], Dublin Core, etc. Metadata description models as well as metadata repositories should be easily extended to accommodate the real needs of emerging applications. The current work is focused on music information extraction, description and retrieval. In this application domain, a number of methods and systems have been proposed for novel music feature extraction and efficient retrieval. In [10], B. Gunsel et al. (Eds.): MRCS 2006, LNCS 4105, pp , c Springer-Verlag Berlin Heidelberg 2006

2 MPEG-7 Based Music Metadata Extensions 427 an analysis is presented of current solution limitations in music information retrieval, after an extensive state-of-the-art study. The main problems are found to exist due to the singularities of the music data itself, related with human perception and music recognition. Feature extraction algorithms do not account for the peculiarities of all kinds of music data, with traditional music being the most difficult to handle. This paper is based on joint work, undertaken under the POLYMNIA project, that led to the introduction of novel features for the estimation of music melody, meter and tempo, applicable to traditional greek music [13,14]. Although there has been an extensive work on music information analysis and retrieval methodologies, the use of metadata descriptions in such systems to allow for their future large scale usage within a Semantic Web context is still limited, with few attempts stemming from EU funded R&D Projects. It is evident however, that support for standardised metadata descriptions is beneficial for future promising system deployments. Since though, every content type and application is imposing its own constraints in metadata descriptions as well, it is also evident that the introduction of extended metadata descriptions to account for novel music features conveying traditional music characteristics is inevitable. This paper proposes MPEG-7 audio extensions, based on MPEG-7 DDL[6], to describe novel audio related features, used subsequently in music information retrieval. It is known that the MPEG-7 Audio Part has limited support for musical features mainly focusing on Melody descriptions. If one considers the variety of music kinds, especially in traditional music, it is evident that in such specific application domains, retrieval efficiency is attained if the specificities of the musical data are accounted for via the feature extraction and classification algorithms. Such novel features though that efficiently index specific music data require proper standardised-based description for efficient manipulation of music information. Thus, the current paper aims at introducing extensions in standardised descriptions to account for the description of novel musical features that enable advanced music information retrieval scenarios on traditional Greek music data. Such metadata extensions can be considered application and music type specific, although based on the MPEG-7 standard, and thus they can only be used within the specific system. However, the approach clearly illustrates application specific MPEG-7 usage and music type extensions that could contribute to standardised application specific extensions. Furthermore, by sharing the metadata extensions schema with other MPEG-7 based systems, interoperability can be achieved. In the sequel, we initially present the end-to-end music information retrieval system that enables a number of advanced music information retrieval scenarios, always referring to traditional greek music. We emphasize the need for efficient metadata generation and archiving, showcasing limitations found in relational databases for storage of XML data, and exhibiting the advantages of XML native databases. Finally, the introduced MPEG-7 metadata extensions are presented in more detail to efficiently describe novel musical features used in the considered retrieval scenarios and music content types.

3 428 S. Tsekeridou et al. 2 Music Analysis, Archiving and Retrieval System Within the POLYMNIA project and the activities related to content-based traditional greek music information retrieval, a number of music information retrieval scenarios have been specified, which include: Beat, tempo and rhythm tracking, Query by rhythm Timbral-based Similarity Retrieval, Query by example Music Genre Classification The feature extraction and classification, as well as assumptions made for the application domain and specific music content types are reported in [13,14]. The overall system architecture to enable the above mentioned scenarios is presented in Figure 1. The system is composed of two sub-systems: the music Music Authoring & Processing System (feature extraction, metadata generation) Music Data Retrieval System Feature Extraction Chroma, Rhythmic, Melodic Features MPEG -7 Metadata Engine Valid MPEG -7 metadata XUpdate XML -RPC Functionalities : Insertions, Updates Extended MPEG -7 Schema Registration exist Native XML DB Reference Mechanisms XQuery XML -RPC Functionalities : Retrieval Music Data Retrieval XSLT Portal web interface: (1) Query selection and submission (2) Results presentation DB of Music Data and Extracts Fig. 1. Music analysis and retrieval system architecture - exist DB content analysis, processing and metadata generation system, the main aim of which is to adequately process or update new or already existing music data in order to extract the newly defined features for the specified scenarios and music content type, and the music information retrieval system targeted at the end-user searching for music content. Both systems are based in the efficient use of standardised MPEG-7 metadata descriptions and defined extensions. The central component in the end-to-end system is the database that stores metadata descriptions and references the repository of original music content and extracts. At the beginning of this work, due to the significant evolution of relational databases that reached maturity in XML data management, the first metadata archiving approach has been the commercial relational database Oracle 10g with XML add-on functionalities (XML DB Manager). The decision has been based on thorough investigation performed at that time in [8,9] and used already in the work of [7]. Furthermore, advances in Oracle XML management allowed immediate registration of the metadata XML schema to the

4 MPEG-7 Based Music Metadata Extensions 429 relational DB, automatically creating a respective table schema (structured mapping), with data constraints applied, that further allowed efficient query and retrieval. This was more efficient than attaining unstructured mapping where entire XML documents are stored in single table rows. During testing, however, since both short-term and long-term analysis of audio signals involves generation of huge sequences of numbers representing a single audio feature, and thus the value of a single MPEG-7 element/descriptor, a limitation of Oracle 10g was found in storing more than 64K string values in a single table row, when structured mapping is used. Since this is the case for many audio descriptors, we started investigating alternate metadata archiving solutions, resulting, based on the progress of native XML DBs, in adopting the open source exist native XML DB solution[15], supporting W3C specifications such as XUpdate (to update metadata instances in the DB), XQuery (to efficiently retrieve information from XML data), XML namespaces, XPath, XML-RPC for the communication of XML data, etc. The efficiency in handling metadata descriptions with this solution has been significantly extended since no XML data management is now required to map XML data to a relational DB. 3 MPEG-7 Based Audio Metadata Extensions In this section, we introduce extensions in MPEG-7 metadata descriptions (mainly audio) to efficiently describe new audio features used within the three scenarios mentioned in the previous section. The feature extraction and classification methods, as well as assumptions made for the application domain and specific content type are reported in [13,14]. In these works it has been found that improved retrieval efficiency is attained by using additional long-term analysis to the conventional short-term audio analysis to extract long-term audio features, such as energy or spectral roll-off, in temporal windows of 300msec, by averaging short term derived such features. Thus, extensions to existing MPEG-7 audio descriptors describing short term features have been defined for the newly introduced long term ones to differentiate between the two definitions. These definitions are shown below, with respect to signal energy, zero crossings rate, spectral centroid, spectral roll-off, spectral flux, which are scalar values per long term window, and MFCC coefficients, derived with two alternate methodologies, and presenting vector values per long term window. The basic MPEG7 type, for the scalar cases, is AudioLLDScalarType, while for the vector case, is AudioLLDVectorType. For the latter case, an extra attribute is defined, to denote the method used to extract MFCCs to be used during retrieval to avoid the case of comparing differently derived MFCCs among the query and the archived data. Such long term audio features are indirectly used in the first scenario for global beat and meter estimation, as well as directly used in the second scenario to enable query by example retrieval. In the following, the namespace mpeg7 refers to MPEG-7 type definitions while the namespace polymnia to introduced extensions.

5 430 S. Tsekeridou et al. <complextype name="energytype"> <complextype name="zerocrossingstype"> <complextype name="spectralcentroidtype"> <complextype name="spectralrollofftype"> <complextype name="spectralfluxtype"> <complextype name="mfcctype"> <complexcontent><extension base="mpeg7:audiolldvectortype"> <attribute name="method" use="required"> <simpletype><restriction base="nmtoken"> <enumeration value="slaney"/> <enumeration value="aj"/> </restriction></simpletype> </attribute></extension></complexcontent> </complextype> An instance extract of the above type definitions is given below. The temporal decomposition of the audio signal refers to the long term analysis in this case. The example cases refer to the scalar valued Energy and the vector valued MFCCs. <mpeg7:temporaldecomposition gap="false" overlap="false"> <mpeg7:audiosegment>... <mpeg7:audiodescriptor xsi:type="energytype"> <mpeg7:seriesofscalar hopsize="pt3n1000f" totalnumofsamples="3443"> <mpeg7:raw> </mpeg7:raw> </mpeg7:seriesofscalar></mpeg7:audiodescriptor>... <mpeg7:audiodescriptor xsi:type="mfcctype" method="slaney"> <mpeg7:seriesofvector hopsize="pt3n1000f" totalnumofsamples="3443" vectorsize="36"> <mpeg7:raw mpeg7:dim=" "> </mpeg7:raw> </mpeg7:seriesofvector></mpeg7:audiodescriptor>... </mpeg7:audiosegment>... </mpeg7:temporaldecomposition> To enable the first scenario, that is query by beat and meter similarity (rhythmic similarity), global estimates of beat and meter values are derived over the

6 MPEG-7 Based Music Metadata Extensions 431 entire music recording as shown in [13,14]. For this reason, the following definition extensions have been introduced. The MPEG-7 MeterType is not altered but refers now to a global audio feature. Analogous definition for the BeatType does not exist in MPEG-7, introduced now. An attribute called certainty in the BeatType definition denotes the certainty with which the algorithm has estimated the global beat value. <complextype name="rhythmtype"> <complexcontent> <extension base="mpeg7:audiodstype"> <sequence> <element name="meter" type="mpeg7:metertype"/> <element name="beat" type="mpeg7-polymnia:globalbeattype"/> </sequence> </extension> </complexcontent> </complextype> <complextype name="globalbeattype"> <simplecontent> <extension base="float"> <attribute name="certainty" type="float" use="required"/> </extension> </simplecontent> </complextype> An instance extract to showcase the use of the above definitions is shown below. The RhythmType since it extends the MPEG-7 AudioDSType can instantiate the MPEG-7 AudioDescriptionScheme abstract element. <mpeg7:audiodescriptionscheme xsi:type="rhythmtype"> <Meter> <mpeg7:numerator>3</mpeg7:numerator> <mpeg7:denominator>4</mpeg7:denominator> </Meter> <Beat certainty=" "> </beat> </mpeg7:audiodescriptionscheme> Retrieval and ranking of results is performed in two stages by examining the beat values of DB entries, having the same meter with the query sample. Similarities are measured by estimating pairwise mean absolute errors in beat values as shown in the following XQuery and Php syntax. $xquery = "for $a in xcollection( db/polymnia ) let $b := ((". $beat."-abs((xs:decimal($a//*:beat) - ". $beat. "))) div ". $beat. ")[1] where xs:decimal($a//*:numerator) eq ".$num." and xs:decimal($a//*:denominator) eq ".$denom." and $b ge ".$certainty." order by abs(data($a//*:beat) - ". $beat. ") ascending Finally, for the third scenario, referring to traditional greek music data, a dictionary of terms for different genres and origins in Greece of such music is

7 432 S. Tsekeridou et al. maintained to define classification scheme entries in MPEG-7 descriptions. Furthermore, the following definition has been introduced, as child element of the MPEG-7 ClassificationType, to additionally associate semantic traditional greek music genre values (as well as numeric identifiers introduced in the respective dictionary) with a feature vector model of 32 global music features and their variations, that uniquely identifies the specific genre class. The attribute representative denotes whether the associated music recording is a representative recording of the specific music genre. In this way, retrieval is more efficiently achieved when comparing the Feature Vector Model of the query sample with only the representative Feature Vector Models of the DB. <complextype name="polymniagenretype"> <complexcontent> <extension base="mpeg7:controlledtermusetype"> <sequence> <element name="featurevectormodel" type="mpeg7:audiolldvectortype"/> </sequence> <attribute name="type" use="optional" default="main"> <simpletype> <restriction base="nmtoken"> <enumeration value="main"/> <enumeration value="secondary"/> </restriction> </simpletype> </attribute> <attribute name="representative" type="boolean" use="required"/> </extension> </complexcontent> </complextype> An instance extract referring to a case of Kalamatiano is shown below. <mpeg7:classification> <mpeg7:genre href="urn:polymnia:cs:contentcs:2005:1.1.4" representative="true"> <mpeg7:name xml:lang="en">kalamatiano</mpeg7:name> <mpeg7:featurevectormodel> </mpeg7:featurevectormodel> </mpeg7:genre> </mpeg7:classification> Retrieval and ranking of results is performed the feature vector model values of DB entries against that of the query sample. Similarities are measured by estimating pairwise mean square errors in feature vectors as shown in the following XQuery and Php syntax. $xquery = "for $a in xcollection( db/subpolymnia ) let $b := tokenize($a//*:genre/*:featurevectormodel, ) let $d := ("; $xquery1 = ""; for ($kl=0;$kl<32;$kl++) { if ($kl==31) { $xquery1.="(($b[". ($kl+1). "] - ". $pattern[$kl].") *

8 MPEG-7 Based Music Metadata Extensions 433 } } ($b[". ($kl+1). "] - ". $pattern[$kl]."))"; } else {$xquery1.="(($b[". ($kl+1). "] - ". $pattern[$kl].") * ($b[". ($kl+1). "] - ". $pattern[$kl].")) + "; 4 Conclusions The paper presents definition extensions to MPEG-7 metadata ones, mainly related to audio descriptors, to account for application specific requirements for traditional Greek music retrieval. The approach may reinforce generalization of the process in case a number of distributed music databases, using MPEG-7 based descriptions, may interoperate to offer the end-user with advanced music content search functionalities within a Semantic Web context. References 1. Manjunath, B.S., Salembier, P., Sikora, T.: Introduction to MPEG7: Multimedia Content Description Language. John Wiley and Sons. (2002) 2. Special Issue on MPEG-7. IEEE Trans. on Circuits and Systems for Video Technology. 11:6 (2001) 3. Special Issue on Multimedia Content Modeling and Personalization. IEEE Multimedia. 10:4 (2003) 4. van Beek, P., Smith, J.R., Ebrahimi, T., Suzuki, T., Askelof, J.: Metadata-driven Multimedia Access. IEEE Signal Processing Magazine. (2003) 5. Martinez, J.M.: Overview of the MPEG-7 Standard. ISO/IEC JTC1/SC29/WG11 N4509. (2001) 6. MPEG Committee, ISO/IEC FCD Information Technology - Multimedia Content Description Interface - Part 2 Description Definition Language. ISO/IEC JTC1/SC29/WG11 N4002. (2001) 7. Tsekeridou, S.: MPEG-7 MDS-based metadata and user profile model for personalized multi-service access in a DTV broadcast environment. IEEE Int. Conf. on Image Processing. (2005) 8. Westermann, U., Klas, W.: An Analysis of XML Database Solutions for the Management of MPEG-7 Media Descriptors. ACM Computing Surveys. 35:4 (2003) Kosch, H.: MPEG-7 and Multimedia Database Systems. SIGMOD Recor. 31:2 (2002) Byrd, D., Crawford, T.: Problems of Music Information retrieval in the Real World. Elsevier Information Processing and Management Journal. 38 (2002) The MusicXML Web site The POLYMNIA Project: Integrated System of Music Tools, Advanced Music Portal Pikrakis, A., Theodoridis, S.: A Novel HMM Approach to Melody Spotting in Raw Audio Recordings. ISMIR. (2005) Pikrakis, A., Antonopoulos, I., Theodoridis, S.: Music meter and tempo tracking from raw polyphonic audio. ISMIR. (2004) 15. The exist DB.