Mobile Multimedia Services Content Adaptation

Transcription

1 Mobile Multimedia Services Content Adaptation Maija Metso 1 A, Mikko Löytynoja 1, Jari Korva 2, Petri Määttä 2, Jaakko Sauvola 1 1 MediaTeam Oulu Infotech Oulu, University of Oulu P.O. Box 4500 FIN University of Oulu, Finland {mikko.loytynoja, jaakko.sauvola}@oulu.fi 2 VTT Electronics Kaitoväylä 1 P.O.Box 1100 FIN Oulu, Finland {jari.korva, petri.maatta}@vtt.fi Abstract This paper introduces theoretical backgrounds and necessary architecture to integrate mobility with adaptive multimedia services. The presented approach allows for anytime - anywhere access to mobile services by adapting media (text, images and video) and user interfaces for a wide variety of fixed and mobile devices. Furthermore, the proposed Princess platform supports a dynamic service creation and delivery process by offering tools for the content and connection providers, including billing, push type of delivery, and personalisation. The implemented platform has been validated with a set of pilot services. 1. Introduction The diversity of multimedia presentation environment sets strict requirements for multimedia applications and systems. The trend is to offer services anytime, anywhere, and with any device. Emerging growth of mobile services defines more requirements for the content and service providers. Content, terminal capabilities, and underlying network set demands for separate service creation process, and mobile services require support for new billing and profiling mechanisms based on the user and the service at hand. Several attempts have been made to standardize the presentation environment and the presentation format for mobile service delivery. Wireless Application Protocol (WAP) [1] is a model for providing two-way data communication to a GSM phone. Markup languages such as the XML (Extensible Markup Language) [2] and its applications like SMIL (Synchronized Multimedia Integration Language) [3] developed by the World Wide Web Consortium (W3C), can be applied in modelling structured, document-like multimedia presentations. The research community has tackled the problem of heterogeneous communication infrastructures e.g. by adapting service contents to match the terminal and network capabilities [4, 5]. Recently, several commercial products have been launched in this area [6]. However, the problem of mobile service adaptation cannot be solved by mere image transcoding and other such lower level technologies: These need to be incorporated with intelligent tools assisting the users with consistent user interfaces as well as tools to facilitate content and connection providers with economically reasonable business models. In this paper, we describe our approach realized within the Princess project ( ) [7]. We propose middleware architecture supporting end-to-end services. Our architecture supports service mobility, multimedia push services, efficient user profiling, different payment models, adaptation of multimedia contents and presentation information, as well as content providing procedure. The unique architecture consists of two operational components: digital media server (DMS) and service broker (SB). 2. Characteristics of Developed Mobility Platform Mobile users, who require access to services, need to use a heterogeneous set of mobile and fixed terminal devices and access networks in order to optimise the service experience in each location. An important concept in supporting this is adaptation in other words, adapting the service to the computing environment, user, and usage context. Three software architectures have been proposed for building adaptive services to heterogeneous computing environments [4, 5]. These have been entitled on the basis of where the adaptation is carried out, as client-, server- and intermediary-based approaches. An example of the clientbased model is the Web, where it is the terminal software s task to adapt the service to that specific device. This approach has the drawback of requiring support for same protocols and presentation formats on all clients. A serverbased approach requires that the adaptation for all supported terminals be done on the content provider's server. From the content provider s point of view, it is vital that the heterogeneous computing environment does not implicate manual creation of many versions of the service. Therefore, the proposed Princess architecture [8] is based on an intermediary approach, where the system acts as an independent component between the client and the content provider. It adapts both the media and user interface for different terminals and network connections without setting specific requirements for the terminals or content providers. In the proposed approach, adaptation has been divided into two separate, nonetheless closely related topics: media and A Maija Metso s current address: Solid Information Technology, Elektroniikkatie 6, FIN Oulu, maija.metso@solidtech.com

2 user interface. Multimedia adaptation is based on the multimedia content model [9]. It is implemented in a digital media server (Fig. 1). Because underlying terminal and network types may vary, the DMS adapts the service accordingly before sending contents to the user. This is described in Section 2.2. User Terminal Request Adapted Response G W G W G User Agent 1 W User Agent 2 Adapted Media Service Broker Service Catalog User Agent 3 User Agent n Service Request Media Server Core Adapted Digital Media Server Adaptation Manager Figure 1. Internal structure. Service Request Media Service Manager Media Database CP Tools User interface adaptation for different computing environments is a task of matching characteristics of different interaction styles, input/output devices, and UI description formats together. In a networked environment, the task is also affected by network characteristics: bandwidth, latencies, error handling, and generally, allowed communication protocols. In our approach, the UI adaptation is implemented in service broker (SB) as gateways (GW), one for each terminal type, performing required conversions between the internally used formats and protocols, and the ones supported by the terminals. Section 2.3 describes this procedure more carefully. Two approaches have been applied to adapt the system s behaviour to the user s needs: The first one is the modelling of user interests as a tree-based profile and the second is a service specific, rule-based approach (Section 2.4). Adapting to usage context, namely the user s location, has been studied but not implemented in the current version (Chapter 4). Managing user profiles with mobile users sets new requirements to the architecture. In general, there are two kinds of mobility: user moving with a terminal and user moving between terminals. Our approach is based on an assumption that the terminal s physical mobility is managed by the access networks (e.g. GSM or WLAN) so it is not addressed. The latter is addressed with a centralised repository for user information. This functionality is implemented in the SB module (Fig. 1). The centralised nature of the SB also enables keeping track of a user's actions. This, in turn, facilitates creation of a user profile as well as a flexible micro billing approach where the user pays for the service, e.g. objects consumed, not the time spent on accessing them. This functionality is described in Section 2.5. To support the content providers, the developed platform includes a tool that can be used to develop new services. The multimedia presentations are created using content provider interface (CPI). The actual content is stored in a database, along with URLs to the multimedia objects. The service database is hosted by the DMS. The DMS also provides so-called platform services to the content providers to use in their service applications. The services are described in Section 2.6. The proposed platform is implemented using Java and XML technologies. Intercommunication between the modules is implemented using Java RMI object communication interface. An XML-based presentation model, specifically an XHTML [11], was chosen as a structured information transfer language for its flexibility (Section 2.1). In this chapter, we start with content modelling as the basis for the adaptation process, and after that, defined types of adaptation are described followed by other characteristics of the proposed platform. 2.1 Content and Presentation Models The multimedia content model (abbreviation MMCM) presented in [9] is used to define metadata of the multimedia presentation to support the content adaptation process. The presentation model is a concrete data format that describes relations of the user interface and media objects [10]. Figure 2. Multimedia Content Model MMCM. In the MMCM (Fig. 2), the presentation is divided into four semantic layers and each layer of the MMCM contains metadata about the structure of the presentation and relationships between different parts of the presentation. These parameters are used for adaptation purposes. Instead of describing the abstract level, i.e. MMCM, the presentation model concerns data format. In selection of the presentation model, the main criteria were adaptability (capability to support many alternate presentations of the same content and user interface), extensibility, and easy development. The best match to these requirements was XHTML [11] with our own proprietary extensions made using XML namespace mechanism: Semantic significance of document fragments is identified using priority-attribute A content that can be subscribed by a user is identified using subscribe-attribute of a-element Pricing information is attached to elements or operations using price-attribute

3 Service may save user-specific properties to the SB using property-element. [8] 2.2 Adaptation of Multimedia Presentations The automatic adaptation of pre-generated content is performed within the DMS, i.e. the content is created using the content provider interface (CPI) described at Section 2.6. The purpose of the adaptation is to scale, compress, and/or convert the requested presentation to the optimal format and size with respect to the spatial-temporal requirements. Other requirements for the adaptation come from terminal capability; network QoS, and user profile. The tasks of the adaptation manager are: 1. Create the MMCM version of the presentation to help represent and process the multimedia containing both static and streaming components. 2. Adapt multimedia presentations using shared media analysis, scaling and conversion modules through a common interface. While creating the MMCM, the adaptation manager parses information such as length, size, title and author of the presentation for presentation layer, size of subscript for subscript layer, mime-type of object for object layer, and location of binary or textual content for primitive layer. In the actual adaptation process, the (X)HTML, video, and image content is adapted to fit a narrow bandwidth and small screen size as in the case of a PDA device. In adaptation of images, the images are divided into three classes: photograph, graphics, and original. In the case of photographs and graphics, the images are converted to a smaller (in bytes) size where the photographs are compressed more roughly than graphic images. If an image is labelled as original, no adaptation is done. In the case of video or audio, we currently support MPEG and Real Video/Audio. If the original video is in MPEG-1 format, it is adapted by extracting key frames from the original presentation. In this process, we exploit ImageMagick freeware [12]. Key frames can be merged to one montage image, see Fig. 4 at Chapter 4. Our system can also modify presentations to SMIL format and the RealServer 7.02 [13] is used to transmit SMIL documents to the end user. To summarise the tasks of the adaptation manager, it can (1) scale or remove images from the original presentation or (2) transform video presentation to a summary, i.e. extract key frames from video presentation. This kind of adaptation speeds up web browsing over a mobile/slow (< 44 kbps) network connection. 2.3 Adaptation of the User Interface Adaptation modules have been developed for Web and WAP browsers, short message service (SMS), as well as a speech-driven, Java-based graphical user interface. [8] [14] The user interface adaptation has been implemented in the platform as gateways, which consist of four modules located at the edge of the service broker: Protocol handler that communicates with the client. UI adapter, which is responsible for adapting the user interface both at conceptual and technical levels by doing conversions between different classes of user interfaces and performing the adaptation from the internal presentation model to the terminal-specific format. Internal services that provide generic implementations of session control (login, service selection, and logout) and configuring user properties. Session management The modularity makes inclusion of new terminal types flexible. For example, from the platform s point of view, WAP and Web browsers use the same protocol (HTTP) and session tracking technologies the only component that needs to be implemented is the UI adapter. [8] 2.4 Adaptation to User s Profile Two approaches have been applied in order to model a user's preferences as a profile: use of explicitly declared data as well as use of inferred data collected from a user's behaviour. Both of these types of profiles are used to adapt system behaviour and service content to the user's needs. The simplest approach is to explicitly ask the user what he/she needs and use this information with a rule-based approach to personalise the service content. This method is not generic, as creating an extensive profile requires many questions to be answered. Thus, content providers can implement this functionality in services assisted by a couple of features of the platform: a presentation can contain dynamic elements, which are replaced dynamically by alternative contents depending on user profile. Moreover, the SB manages the profile information on behalf of the service. For a more generic profiling method, we have added a profiling component inside the service broker. The profile consists of a tree of concepts, which have been added a factor denoting the user's interest in a certain topic. Similarly, the service catalogue contains descriptive keywords of each service. This enables inferring the user's interest by combining information about the topics addressed by the service with information about the time spent with the services. This profile has been used for bringing interesting services to the attention of the user by ordering the service catalogue based on measuring distances between the user's profile and the service profiles. 2.5 System management There are three basic methods used for system control and management in the SB: Service catalogue, centralized

4 billing scheme, and user management based on use of software user agents. The internal service catalogue is a class structure that stores available services and essential information about them and their location. When a new service is founded in DMS, the updated list of services is offered to a SB. An internal process then updates the service catalogue and makes the new services available to users. Correspondingly, if a DMS is shut down, the catalogue update process deletes the services from the catalogue. The access to services, personal configurations (e.g. address data and billing data), and service properties is made through a control panel interface. The billing method in our platform was defined as one based on a per-service approach because time-based billing is, especially in cellular networks, inconvenient and unfair since the connection speeds may be very slow. The pricing of services is defined in the Princess presentation model (Section 2.1), i.e. a price tag is attached to each service (e.g. to a picture or a video). The tags are processed in the SB where the prices and time of event information are collected for the total bill. This way the micro billing scheme could be added to e.g. normal phone billing. In the design phase of the architecture, our aim was to use existing Internet and networking technologies for communication between the different software modules as much as possible. Additionally, we implemented a light version of the agent platform on a piece of commercial software, i.e. ObjectSpace s Voyager ORB [15], where the software agents only handle user management. The other parts like adaptation modules and gateways are of a static type, as in a conventional client-server model, and not using software agents. In the user registration process, the users are added to the user database of the SB; user information like name, address and additional information, e.g. preferred user groups and preferences are added into the SB repository. This information is not only for authentication use but also for describing a user s likes and dislikes for adaptation of services. The content provider defines the exact set of preference parameters but the user can update some of the user preferences too. Beside the SB, the DMS core offers tools to manage adaptation profiles, current users, and running services. Adaptation profiles define the parameters to adapt the multimedia presentations. User management shows the users current service, session parameters, and suspended services. The management tool also logs all the events inside the DMS and shows currently running platform services and user services. 2.6 Services The service manager (Fig. 1) provides two kinds of services: platform services and user services. The platform services are transparent to the users. They help the content providers to create the user services. Service subscription is one example of platform services. It allows the service to push presentations that the user has subscribed even if the user is not currently using the platform. This is done using either GSM SMS messages or . Other examples of platform services are ones used by video surveillance service to create MPEG videos from stored images and automatic movement detection from surveillance video. The user services are the ones that are available to the end users. Instead of implementing services e.g. in Java and compiling them after each modification, a tool for the content provider (content provider interface or CPI) was created [16]. It is used to create new services and to update and modify the content of existing ones. By using the CPI, the user services defined with the presentation model (Section 2.1) are stored in the service database. The database is ObjectStore PSE Pro 6.0 [17]. The use of the object database allows high-performance persistent storage of service data. It also supports updating of the contents of the service database remotely and online. In addition to an editor used to create presentations, the CPI provides some tools to check the presentation s validity and to preview the presentation. 3. Experiments The key issues in our approach are the adaptation process and the flexibility to work with many user terminals and on many computer systems. In addition, the support for personalization and micro billing makes the platform interesting for further development. In the following, a pilot service is briefly introduced, namely an ice hockey service. Our latest pilot service, an ice-hockey service, is created using the CPI. The material is from the ice-hockey world championships 2000 and from Finnish major league finals during spring In addition, the service consists of player presentations and hockey-related news, which can be subscribed. The users can join and create user groups, which are used to send messages to other group members via SMS or . Fig. 3 shows the main page of the icehockey service seen on a desktop PC. Figure 3. Ice-hockey service with the user belonging to a user group of a local hockey team s fans.

5 The ice-hockey service provides users with different ways to watch the highlights of the games. The users can choose to watch a key frame presentation of the games or, if they have enough bandwidth, to see the actual videos. The users can also hear only the audio part of the video when their terminal does not support video or image presentations. Some of the stored games are provided as indexed SMIL presentations. Fig. 4 shows the montage image created automatically from ice-hockey video. A montage image is provided to a terminal that is unable to show video, or to a terminal with a slow network connection, therefore unable to transmit video in reasonable time. As can be seen in Fig. 4, the montage provides enough information about the video s events and the needed transmission time is an order of magnitude shorter than the time needed to transmit the video. Figure 4. Montage created from ice-hockey video. Fig. 3 shows the adaptation to the user profile. Because the user has registered as a fan of a local ice-hockey team, the main page of the ice-hockey service contains news about the team (shown under Personal profile ) and some advertisements for the local area (below the ice-hockey image). Princess #1 Web #1 Princess #2 Web #2 Princess #3 Web #3 Princess #4 Web #4 Princess #5 Web #5 Princess #6 Web # s Page loaded Page complete Figure 5. Times to load the same presentations from the Web and using our platform with a laptop and 33.6 kbps modem. Fig. 5 shows measurements of loading the same presentation from the Web and through the developed platform. The platform was running with Windows 2000 on a 500 MHz Pentium III. The first part of the bar shows how long it takes to load the presentation without any binary objects, i.e. images. The two topmost bars show the times to load the ice-hockey service main page shown in Fig. 3. The second presentation is also similar to the first one, i.e. containing only very few small images. It can be seen that the overhead caused by our platform is not significant. The presentations from 3 to 6 contained many images and the overhead to analyse the presentation is much bigger. However, the adaptation of the images shortens the time to download them, and the overall time to get the presentation is much less than without adaptation. From the user's point of view, the ice-hockey service offers a time and place independent way to follow games, and it also shortens download times when retrieving video from the server. The user gets updates by using the push service, which provides information of game situations to his/her GSM. 4. Future Work One of the emerging service application types within the wireless Internet world is a service model based on navigation and location. In the near future, personalisation combined with navigational services will form the base for the whole service providing industry. In the Princess project, we have been searching for similar existing service platforms that support the location-based service applications. It seems that the main general research goal is to get a standardised (e.g. XML-based) method for the transmission of geographical data and, on the other hand, to integrate the geographical information data into general services like news or weather services. Many of the current efforts are considering the XML meta-language and other standardized and well-known methods as bases for the new services. The OpenGIS consortium is developing GML [18], and another presentation language following the XML-specification is Navigation Mark-up Language, which was presented in the public note submitted by Fujitsu Limited to the W3C [19]. In our approach, we used Internet-compatible methods like an XML-based presentation language, so that in the future the developed platform could be extended to support also the standardised location-based services. The choice of XML ensures also that our platform will be compatible with different networks and other services also in the future. Security aspects have not been under deep consideration at this stage of the research. Providing end-to-end security using intermediary-approach is not generally possible, as the intermediary component must be able to interfere with the communication between the service and the client. Partial solutions can be provided but this inherently limits the applicability of this approach to medium and low security services. In this sense, Princess architecture could

6 be made more flexible so that it would allow adaptation components to be used more easily also in a server-based approach. 5. Summary The proposed architecture has successfully shown its abilities to adapt services based on the properties of the client terminal and network connection. It is able to support terminals ranging from SMS and WAP to full-fledged HTML-capable browsers with a variety of display sizes and colour properties. In addition to adaptation features, the platform is also coupled with personalisation features as well as tools for content providers. The approach allows for extension by new future technologies. In this paper, we presented the architecture for adaptation of multimedia services for mobile and fixed connections. The presented approach considers adaptation from the content and user interface points of view, user management, i.e. billing and profiling services for users, and service management issues. The evaluated architecture consists of two main components: service broker (SB) and digital media server (DMS). The SB not only maintains user profiles and a service catalogue but also contains user interface adaptation for different terminal types and user preferences. The main task of the DMS is media adaptation and support for the content provider with a set of tools and platform services, which ease and automate the service creation process. The platform has been evaluated by developing several pilot services emphasising different characteristics of our approach: adaptability, personalisation, and the service creation process. 6. Acknowledgments We gratefully acknowledge the support of the Academy of Finland, the National Technology Agency (TEKES) and our industrial partners: Nokia Mobile Phones, Nokia Networks, Sonera, and CCC. We also wish to thank the personnel of the Princess project, especially Antti Koivisto and Johan Plomp. 7. References [1] Wireless Application Protocol Forum, "WAP 1.2 Specification Suite", [2] World Wide Web Consortium, "The Extensible Markup Language 1.0 Recommendation", [3] L. Bouthillier, Synchronized Multimedia on the Web, Web Techniques Magazine, Volume 3, Issue 9, September [4] A. Fox, S. D. Gribble, Y. Chawathe and E. A. Brewer, "Adapting to Network and Client Variation Using Infrastructural Proxies: Lessons and Perspectives", IEEE Personal Communications, Vol. 5, No 4, [5] J. R. Smith, R. Mohan and C-S. Li, "Scalable Multimedia Delivery for Pervasive Computing", ACM Multimedia 99 Vol. 10, pp , Orlando, FL, USA, [6] W-Y. Ma, I. Bedner, G. Chang, A. Kuchinsky and H. Zhang, "A Framework for Adaptive Content Delivery in Heterogeneous Network Environments", Proceedings of the Multimedia Computing and Networking (MMCN), San Jose, CA, January [7] MediaTeam Oulu, University of Oulu, "MediaTeam Oulu", [8] J. Korva, J. Plomp, P. Määttä and M. Metso, "On-line Service Adaptation for Mobile and Fixed Terminal Devices", Proceedings of the MDM '01, pp , Hong Kong, January [9] M. Metso, A. Koivisto and J. Sauvola, "Content Model for Mobile Adaptation of Multimedia Information", Proc. 3rd IEEE Workshop on Multimedia Signal Processing, pp , Copenhagen, Denmark. [10] J. Forstadius, J. Ala-Kurikka, A. Koivisto and J. Sauvola, Model of Adaptive Multimedia Services, Proceedings of SPIE International Symposium on Convergence IT and Communications, Denver, CO, USA, August 2001, to appear. [11] World Wide Web Consortium, "XHTML 1.0: The Extensible HyperText Markup" Language, [12] ImageMagick, [13] RealServer, [14] J. Peltola, J. Plomp and T. Seppänen, "A dictionaryadaptive speech driven user interface for a distributed multimedia platform", Euromicro workshop on multimedia and telecommunications, Milan, [15] ObjectSpace Voyager ORB, "Overview", [16] T. Polvi, A. Koivisto and J. Sauvola, Authoring Adaptive Multimedia Services, Proceedings of SPIE International Symposium on Convergence IT and Communications, Denver, CO, USA, August 2001, to appear. [17] Object Design, ObjectStore PSE Pro, [18] OpenGIS Consortium, Geography Markup Language (GML) v1.0, "OpenGIS recommendation", [19] NaVigation Markup Language, "Technical report",