At the Crossroads of Web and Interactive Multimedia: an Approach to Merge the Two Realms

Transcription

1 At the Crossroads of Web and Interactive Multimedia: an Approach to Merge the Two Realms Stefano Ferretti, Paola Salomoni, Marco Roccetti, Silvia Mirri, Ludovico A. Muratori Dipartimento di Scienze dell Informazione, Università di Bologna, Bologna, Italy {sferrett, salomoni, roccetti, Abstract Multimedia and Web 2.0 can meet up. Despite the success they have obtained in last years, these two domains present some important limitations. In particular, current Web 2.0 based technologies, such as wikis, blogs, podcasts, commonly resort to sequential combinations of contents, rather than exploiting dynamic and synchronized compositions of discrete and continuous media resources. On the other hand, tools for multimedia content authoring and distribution usually lack cooperativeness, interactivity, use of adequate metatagging structures and application mash-up. This paper discusses how by merging these two worlds it is possible to inherit the advantages while removing their limitations. We describe a system exemplar we developed for the cooperative creation, manipulation and sharing of rich media contents. Keywords- Collaborative Multimedia Authoring; Pervasive Distribution; Annotation Systems; Collaborative e-learning. I. INTRODUCTION Multimedia and Web 2.0 are currently two separate worlds with poor intersections between each other [2]. Specifically, a variety of useful features of the multimedia domain are not exploited at all in the Web 2.0 field. Among these, it is worth mentioning structured multimedia authoring systems, (continuous and discrete) media synchronization languages for the rich media contents production, profiling protocols describing users and their devices, dynamic customization strategies, user-centered delivery, etc [6, 7, 9]. Nevertheless, multimedia lacks many of those characteristics that lead Web 2.0 to its wide success. The very first missing features, which are regarded as winning aspects of Web 2.0 technologies, include open participation and collaboration, easy-to-use online applications, interactive assembly from diverse sources. All these characteristics make the network as a platform shared by all users. Services can be easily combined as new mash-ups exploiting data coming from different sources into a single, integrated tool. Another important factor in Web 2.0 is the possibility, offered to the users, to associate new metadata and annotations to shared contents. This has led to the introduction of folksonomies, i.e. collaborative tagging approaches, which revolutionized the methods of labelling online contents, without the boundaries of using specific hierarchies and the constraints of exploiting a unified, regulated taxonomy. This aspect is still totally unexplored in the multimedia domain [2]. Despite these differences, there is room to merge these two worlds into a unique realm. In particular, since Web 2.0 is regarded as the platform of the future, Web 2.0 technologies can be put to good use to construct mature multimedia (2.0) applications. We will take into consideration important features of the Web 2.0 and show their melting point with multimedia. First, we show how Web 2.0 based collaborative editing can be exploited to produce rich multimedia contents. This allows to remove, on the Web 2.0 side, the constraint of being able to produce just limited, discrete media resources, and, on the multimedia side, the complete absence of cooperation in the authoring process. We will show how the specific solutions we present have been technically exploited for the development of a new collaborative, pervasive system for the dynamic editing of rich media contents. In particular, this system exploits an extended wiki-type editing interface. The extended wiki syntax we introduce allows to specify timing properties of several media resources to be synchronized and combined into a compound media presentation. Alternative resources can be associated to primary ones by users, hence augmenting the accessibility of collaboratively produced rich media contents. Second, we will discuss how tagging approaches should be employed for characterizing multimedia elements, in order to inherit from the two domains both the freedom (typical of Web 2.0) to associate to a content whatever semantic keyword, and the possibility (typical of multimedia) of profiling rich media contents depending on their structure and on the media elements that compose them. This would facilitate an open categorization as well as a proper manipulation of these contents. The idea is that once tags have been associated to contents, these can be customized and delivered to users which have been adequately profiled. Third, we will demonstrate how the strengths of Web 2.0 and multimedia can be overlapped to produce a mash-up of services that, on one side, allows users to exploit new collaborative applications for multimedia editing and, on the other side, promotes a pervasive delivery of produced rich media contents. Thanks to the service mash-up we devised, these contents can be dynamically manipulated using typical facilities of Web 2.0 and distributed multimedia. Hence, for instance, RSS feeds can be exploited to advertise the production (or modification) of new contents. Concurrently, customization and delivery facilities, which are typical features of multimedia for continuous media, can be automatically employed so as to guarantee a pervasive distribution of multimedia resources. Based on the use of a multimedia adaptation service we developed, multimedia contents are automatically customized, before being delivered to a specific user, depending on the specific user profile and his exploited device. In particular, resources created using our extended wiki syntax can be transformed into complex SMIL multimedia /09/$ IEEE 1

2 presentations, as well as single video streams, or even discrete XHTML documents [9]. This promotes a pervasive delivery and sharing of cooperatively authored multimedia contents. In rest of the paper, we first discuss the current approaches to author contents (Section II); we then describe how rich media contents can be edited collaboratively based on our solution (Section III), how they can be tagged (Section IV) and customized to guarantee a pervasive delivery (Section V). Some concluding remarks are then provided in Section VI. II. RICH MEDIA CONTENTS COLLABORATIVE EDITING The Web 2.0 phenomenon has caused an explosion of interest in collaborative, interactive, online authoring tools. The use of wikis for producing hypertext-based contents is now widespread. The same holds for other free editing applications, such as Google Docs & Spreadsheet, for instance. The strong success of these services is due to the fact that these are easyto-use online applications, accessible through a common Web browser. Neither ad-hoc software is required, nor any particular skills to interact with the application. This results in an ever increasing involvement and participation of Web users that become content producers, besides being consumers. A limitation of contents produced through these Web 2.0 applications is that they typically consist of single media resources, e.g., a video file authored offline and published on YouTube, or a limited set of discrete media resources. In the case of wikis, for instance, contents produced by collaborative users are mainly text-based HTML documents with fixed structures, some images and simple graphic styling. However, no continuous media can be employed or manipulated for producing compound multimedia contents. Needless to say, this greatly simplifies the problems concerned with the presentation of the content, guaranteeing a high level of pervasiveness. Indeed, the use of HTML makes almost all contents easily accessible through any browserequipped device. The collaborative editing software can be managed at the wiki administration level through CSS technologies, thus exempting users from the weight of designing the graphical appearance of edited contents. This would allow to structure contents based on different possible alternatives of presentation. This way, the styling operation can be completely transparent to the user, which is simply asked to edit the content. As to multimedia authoring, things run completely different. The process of editing multimedia contents is currently accomplished offline by a single, expert producer. A plethora of video editing software applications exist, such as those ones developed by Adobe, Apple, Avid, as well as open source projects (e.g., Kdenlive, Kino). When we go online, it seems that multimedia authoring almost remains at the level of creating Flash videos, or something similar to the typical Web 2.0 practice of grabbing some video with the cellphone and upload it on YouTube [2]. Some simple online services exist that allow to edit videos, e.g., InternationalRemix, JumpCut. Apart from their limited effectiveness, the main limitations of these online tools are that they explicitly focus on a single media and do not allow the collaborative editing of rich media contents, where different kinds of media elements can be exploited to create compound, synchronized presentations. Recent advances have been made with new tools for collaboratively producing videos, e.g., Kaltura [5] or new annotation features offered by YouTube, yet limiting the cooperative authoring to a single media resource, through simple editing features. As a matter of fact, the authoring, manipulation and distribution of rich media contents is one of the most interesting topics of research in multimedia. The idea is that several discrete and continuous media resources can be utilized as building blocks for creating a wider, interactive (and thus rich ) content that incorporates all the single elements. Thus, you can take a pre-recorded video and synchronize it with some audio, a set of some image slides, some flash animations and publish it as a Quicktime video, or a SMIL document. Then, alternative contents, such as captions, may be associated to primary media types (e.g., audio or video). This represents a valuable aspect since, in principle, it offers different options to enjoy the same content, enabling users to choose which is the modality they prefer and which better addresses their own preferences and needs. The problem here is to devise a solution which permits a collaborative editing of rich media contents, through an online, easy-to-use authoring system. III. WHERE WEB 2.0 AND MULTIMEDIA EDITING CAN MEET: PROPOSED SOLUTION Trying to find a meeting point for the Web 2.0 and the multimedia fields, we discuss the approach we have developed for the cooperative creation and sharing of rich media resources. Specifically, features have been built to add and synchronize: i) one or more video files (to be shown in sequence), ii) audio files, iii) one or more images, to be shown in sequence in a given region of the screen (different from the region associated to the video), iv) subtitles alternative to audio flows, and v) alternative annotations associated to any other media element present in the presentation. Table 1: Wiki-syntax and Corresponding SMIL Attributes Wiki-syntax SMIL attributes Tags $Ts begin= Ts _$Ts end= Ts ::Ts dur= Ts ^^^ <seq> +++ <par> VVVurlVVV <video src= url /> AAAurlAAA <audio src= url > IIIurlIII <img src= url > TTTurlTTT <text src= url > SSSurlSSS <textstream src= url > MMMurlMMM <animation src= url > To make this possible, users are provided with an easy-touse Wiki-type editing interface to collaboratively manipulate multimedia contents [4]. As in all wikis, a specific wikitext is in charge of creating and organizing contents. The original (DokuWiki) syntax has being extended to specify timing properties and synchronize media elements. The wiki-like editing interface provides a versioning tool, so that each user has a better control on the updates he introduces during the cooperative editing. The system manages and stores this 2

3 wikitext specification as a textual file. However, the language is sufficiently expressive to allow the collaborative creation of SMIL-based multimedia presentations. Table 1 reports the mapping between the Wiki syntax that the user may employ to add new media resources (e.g., audio, video, images, text) and to specify timing and synchronization relationships. The table shows the corresponding specifications in SMIL. opera mpeg player screen reader MS-IE b c Adaptation system Multimedia Repository mozilla Client a Figure 2. Content Customization. Figure 1. Wiki-like Editing Interface for the Collaborative Editing of Multimedia Contents. Figure 1 shows an example of screenshot of our developed cooperative editing interface. In the example, the edited source code refers to the attempt to add a video file together with an audio file, a text and some images. Timing specifications are defined to state when and how long these captions must be displayed. These subtitles can be employed upon direct request of a user, or even automatically activated during the content presentation, if the user profile states that, for instance, the user is deaf, or when the client terminal has no audio capabilities. The default form of presentation is as a SMIL presentation. Our system exploits an adaptation service in charge to customize contents, in order to guarantee pervasive and accessible multimedia distribution services [9]. Specifically, upon request for a specific content, a content adaptation strategy is performed depending on the types of the available media elements composing the rich media content, and on the user profile [9]. The wikitext specification is employed to produce the final content to be visualized. By comparing the needs of the user (based on the metadata in his profile) and the available media elements in the rich media content (based on its associated metadata), the content can be presented as a SMIL multimedia presentation, a single video file (containing all the media flows combined into a unique one), or even a XHTML hypertext document [5]. Thus, for instance, a deaf user, which asks for a content through the use of a common PC, will receive a high quality SMIL presentation with captions activated instead of audio flows (see Figure 2, case a). Instead, a blind user will receive a properly adapted XHTML document (see Figure 2, case b). This way, it is ensured that all the contents will be presented to the blind user as a linear sequence (thus avoiding cognitive overload); then, text can be converted at the client-side, by resorting to a screen reader or a Braille display. Finally, a mobile user joining the system through the use of a PDA (with no support for SMIL technologies), will receive a video version properly encoded to fit the technical characteristics of the exploited client device (see Figure 2, case c). IV. TAGGING RICH MEDIA CONTENTS The Web 2.0 has suddenly shifted the attention from the need to identify good taxonomies and hierarchical organizations for characterizing contents, to a broader use of general tags, leading to the creation of folksonomies. This important revolution has determined a new style of collaborative categorization of sites using freely chosen keywords, instead of creating a rigid and static range of categories [8]. The success of this approach is mainly due to its simplicity. In fact, upon creation to (or access to) a new Web resource, each user is let completely free to associate to that content whatever keyword he/she wants. This philosophy is in contrast with the approach adopted by the multimedia research community that seeks for a unified ontology. For instance, MPEG-7 includes in its standard a controlled vocabulary to characterize and annotate multimedia contents. This categorization could be used to retrieve contents from digital archives. As a matter of facts, such metadata are currently not exploited at all [2]. The fact is that tagging a Web content is much simpler than tagging a rich media content. Indeed, when a user adds a tag to a photo in Flickr, it is quite obvious that the tag relates to the content depicted on the image. Everyone knows it is some kind of picture. In general, each content in Web 2.0 is encoded using a single type of media, and its associated tags refer to the semantics of the content. Conversely, in the multimedia domain, tags associated to contents may refer not only to the semantic meaning of the resource, but also to other kinds of information. A clear example is represented by all those tags that refer to the accessibility issues of a multimedia content, e.g., ACCMD [9]. Indeed, the structure of the rich media content and tags exploited to represent this structure permit to characterize the level of accessibility of the content itself. For instance, a tag is provided in ACCMD which details all the features of the rich media content. ACCMD tags inside it allow to specify that certain contents can be, for instance, accessed only by a portion of the audience, when no alternatives for some primary media elements are available. When multiple alternatives are provided on the rich media content, instead, the <primary> tag defines which is the primary resource and describes its features; 3

4 <equivalentresource> elements allow to point to alternative resources for the primary one. These alternatives can be dynamically exploited if these may help the user to enjoy the content. Thus, for instance, when available as alternative resources, subtitles and additional textual descriptions of contents can be utilized by deaf / hard of hearing people, foreign users, or users which exploit devices unsupplied with audio capabilities [9]. Needless to say, to identify the best option among the wide range of possible presentations of rich media contents, information is needed to characterize (and hence, to tag) the user itself. Standards exist that cope with this issue. For instance, in the context of e-learning scenarios the IMS Learner Information Profile (IMS LIP) and the IMS Accessibility Learner Profile (IMS ACCLIP) standards describe users in terms accessibility constraints ( User preferences are detailed within an XML document. Within this profile, for instance, a <display> tag is in charge of describing how contents should be displayed to the user. Take as an example the case of a blind user; a <screenreader> element might be present which asserts that the user exploits a screen-reader. This information states that textual information can be sent to that user. On the other hand, the W3C CC/PP specification offers a vocabulary to fully represent client devices and all their possible computation and communication constraints ( Hence, CC/PP profiles contain information on the device in terms of hardware capabilities, e.g., display width and display height resolution, audio board presence, images support, Braille display presence, software capabilities, and so on. At the end of the story, due to the high complexity of rich media contents, it is not possible to directly transfer Web 2.0 based folksonomies to the multimedia domain and exploit them alone. Rather, several levels of tagging strategies should be adopted, to cover all the possible domains of additional information to be associated to rich media contents. With this in view, we have devised a solution that tackles the problem as follows. According to our approach, each multimedia content has different levels of tags. Tags concerned with the semantic meaning of the collaboratively produced content can be freely added by users, thank to our Wiki-like editing interface. This approach inherits the easy-to-use Web 2.0 categorization methodology for the semantic domain. We then characterize the structure of the rich media content using ACCMD metatags that specify which typologies of (primary and alternative) media resources are available in a rich media content and those that can be used to present the final content. The wikitext interpreter is able to analyze the structure of the rich media content, find all media elements composing it and store this information. Our extended wiki interface provides users with an additional button that allows to specify, upon insertion of a new content, if this is a primary resource or an alternative one. Figure 3 shows how the ACCMD related to a given resource may change upon insertion of new media elements to the specific rich media content. Just as an example, assume that a user adds some caption to a given audio file through our system. After the insertion of the text through the Wiki interface, the user is asked to specify whether the inserted text is a primary or an alternative resource. As soon as the user specifies that the text is an alternative resource for the audio file, the ACCMD code for that audio resource is updated (see Figure 3). Also captions get their ACCMD code, in which they are defined as a secondary resource and the hastext attribute is set to true, stating that this is a text-based resource. <primary hasauditory="true" hastext="false" hasvisual="false"/> Audio <primary hasauditory="true" hastext="false" hasvisual="false"> <equivalentresource> captions.txt </equivalentresource> </primary> Audio + <secondary hasauditory="false" hastext="true" hasvisual="false"/> Caption Figure 3. Evolution of the ACCMD code for an audio file, through the collaborative editing process promoted by our system. At time of distribution and presentation, all the information contained within the ACCMD associated to a rich media content is matched with the profile of the user. A user profiling approach is exploited that considers both accessibility information as well as technical characteristics of his/her device [9]. This is accomplished by employing both the IMS ACCLIP and CC/PP standards, coupled into a unique profile. The ACCLIP-based information describes user preferences that relate to accessibility constraints (e.g., preferred/required input/output modalities or preferred content alternatives), while CC/PP based information is exploited to profile client devices. The idea is that when some specific user requirement, detailed in the ACCLIP derived elements, corresponds to a feature availability on the exploited device, detailed in related CC/PP derived elements (e.g., a request for a screen reader, properly installed on the device), then the feature can be exploited to meet the user s needs. Thus, contents can be adapted to be exploited through that feature (e.g., captions and text can be sent to a blind user, since the screen reader will transform that textual information into an audio stream). Conversely, if some specific requirement, detailed in ACCLIP derived elements, does not correspond to a feature availability on the device, detailed in CC/PP derived elements (e.g., a request for the use of a screen reader, not available on the device), the feature cannot be exploited and, if possible, alternative solutions are needed. Thus, based on the types of media the user can enjoy, the ACCMD of the rich media content, created during the collaborative editing activity, is inspected. Only those media resources composing the rich media content, which match the constraints imposed by the user, are selected. These resources are then adapted (if needed), composed, synchronized and then delivered to the user [9]. 4

5 V. SERVICE MASH-UP FOR A PERVASIVE CONTENT DISTRIBUTION The success of Web 2.0 is mainly due to the wide mash-up of services already available, which are utilized, combined and merged to create new attractive online applications. By exploiting facilities offered by AJAX technologies, services can be created that combine data coming from RSS feeds of the preferred news sites, photos from Flickr, videos from YouTube, bookmarks from Del.icio.us, events from Eventful, music from myspace or LastFM, places from Google Maps. The idea is that of enabling ordinary people to create useful applications by combining different online information sources. In this sense, Yahoo s Pipes tool is a prominent example. Like Unix pipes, this online tool allows the combination and aggregation of Web modules and commands using a graphical user interface, in few simple steps. Another important example is OpenLaszlo, an open source platform for the development and delivery of rich Internet applications. This platform exploits an XML programming language that can be compiled to obtain a binary SWF Flash or a DHTML file, to be enjoyed via a browser. Following this philosophy, we claim that new multimedia (2.0) applications should be thought as a combination and aggregation of different online services, instead of a standalone product. Based on this, our system has been designed as a mash-up of services for the production, customization and distribution of multimedia contents (see Figure 4). opera mpeg player screen reader MS-IE rendering wiki RSS feed mashup Web media mng ( Another service is in charge of converting a wide set of continuous media formats, based on the open source library for media conversion FFmpeg ( Another Web service is in charge of taking the SMIL code (produced by the extended Wiki editing service) and transcode it into an XHTML document or a single video. Finally, we also included an external Web service for converting text from a specific language to another one (i.e., To adapt contents, the adaptation service first inspects the user profile and the ACCMD associated to the rich media content to determine which typologies of media should be delivered to a given user. The best type of presentation is identified (e.g., SMIL presentation, single video, XHTML document) [9]. Media elements to be composed are selected and combined, based on the timing specification provided by users during the content editing using the Wiki-like interface. Encoding formats for single media elements are selected, based on the client profile and, when needed, media elements are adapted to fit the client s needs. The produced content is then delivered to the user. A detailed description of the adaptation service, together with a complete performance evaluation, can be found in [9]. VI. CONCLUSIONS Web 2.0 and multimedia can indeed meet together. Taking the best of the two worlds, it becomes possible to create highly collaborative and interactive multimedia (2.0) applications. The melting point can be found in the mash-up of functionalities where technologies of Web 2.0 such as those based on rich Internet applications and social tagging are integrated with others coming from the multimedia field, such as manipulation of rich media contents, pervasive distribution, ubiquitous communication. The developed system confirms our claims. mozilla Client editing storage tagging discovery Figure 4. Service Mash-up. Multimedia adaptation In particular, we already mentioned that our rich media contents editing system is based on the use of an extended wiki. This way, (discrete and continuous) media elements can be cooperatively combined and synchronized using our wiki-type interface. Then, open Web 2.0 services can be freely combined to it. Thus, tags can be associated thanks to features provided by a plugin of the employed wiki engine. Moreover, classic discovery mechanisms for discovering rich media contents and tracking their changes might be employed, such as RSS feeds and/or other Web 2.0 publish-subscribe mechanisms. As a matter of fact, the adaptation service we developed in our system is another mash-up of transcoding Web services. In the current version of our system, we exploit different Web services for specific transcoding operations. In particular, a Web service has been explicitly developed to manage and transform several image formats, based on the use of the open source library for image conversion ImageMagick REFERENCES [1] T. Berners-Lee, M. Fischetti, Weaving the Web. The original design and ultimate destiny of the World Wide Web, by its inventor, Harper San Francisco, [2] S. Boll, MultiTube--Where Web 2.0 and Multimedia Could Meet, IEEE MultiMedia, 14(1): 9-13 January, [3] S. Downes, E-learning 2.0, ACM elearn Magazine, (October 2005) vol. 10 (2005) [4] S. Ferretti, S. Mirri, M. Roccetti, P. Salomoni, Notes for a Collaboration: On the Design of a Wiki-type Educational Video Lecture Annotation System, Proc. IEEE International Conference on Semantic Computing, Irvine, CA, IEEE, September 2007, [5] Kaltura Open Source Video Platform, [6] T. Laakko, T. Hiltunen Adapting Web Content to Mobile User Agents, IEEE Internet Computing, 9(2):46-53, [7] R. Mohan, J. Smith, C.S. Li, Adapting Multimedia Internet Content for Universal Access, IEEE Transactions on Multimedia, 1999, [8] T. O'Reilly, Web 2.0 Compact Definition: Trying Again, 2006, [9] P. Salomoni, S. Mirri, S. Ferretti, M. Roccetti, A Multimedia Broker to support Accessible and Mobile Learning through Learning Objects Adaptation, ACM Transactions on Internet Technology, ACM Press, 8(2):9-23, February [10] A. Savidis, C. Stephanidis, Developing inclusive e-learning and e- entertainment to effectively accommodate learning difficulties, ACM SIGACCESS Accessibility and Computing, 83 (Sep. 2005),