Community TV: An Approach to Interaction for Groups and Single Users on Internet Video

Transcription

1 Community TV: An Approach to Interaction for Groups and Single Users on Internet Video Norbert Braun Zentrum für Graphische Datenverarbeitung (ZGDV e.v.) Rundeturmstrasse Darmstadt Germany Matthias Finke Thomas Rieger Technische Universität Darmstadt Rundeturmstrasse Darmstadt Germany Abstract - We describe an interactive access of video content via Internet for multimedia computer clients. The users are integrated within one community of interest, having the same interaction and content manipulation options. More than that, a community awareness is established and displayed to every user within the community in order to form a experience of watching Video together with other people within a virtual world. The system architecture is shown and the basic processing of community interests and behaviour as well as the system s interaction metaphors like conversation, is discussed. The prototype of the Community TV system implementing the architecture and functionality is presented. Key Words: Group behaviour, internet community, internet video broadcast, multimedia, lifelike avatar 1. Introduction It is obvious that features like collaborative work, group chat, virtual presence or conversational interaction have a growing impact on manipulation of internet content today. In our opinion the joint experience of content, environments or stories is a major factor of successful end user applications, particularly when those applications run within a wide area network like the Internet. Within this network it is evident that groups of users are using the same application and exchange each others opinions ore just virtually live together in virtual environments. If content is presented the collaborative manipulation of this content is another feature claimed by most users. This paper presents a internet based system called Community TV that possesses the features collaborative work, group chat, virtual presence of singles and groups as well as conversational and direct manipulative interaction on video content. Within this system, user groups exchange opinions on the video content using a virtual humanlike avatar to present group behaviour and speech in a conversational way. Users can also chat with each other in a conversational way via humanlike avatar using speech and emotions. Content is presented with integrated interaction facilities like video hyperlinks, with on the fly content annotation possibilities (presented to every user in the group) as well as with conversational interacting on the content. Hence the system shows a group of avatars representing the system, the group, and single users. The approach presented in this paper is compared to related work in paragraph 2. An overview of the system is given in paragraph 3; interaction possibilities on the content are discussed in paragraph 4. The conversational interaction between the group and single users is introduced in paragraph 5. The system prototype is described in paragraph 6. Lastly a conclusion and a view on future work are given in paragraph Related Work Interactive Internet Broadcast with direct manipulative interaction in combination with conversational interaction was shown in [1]. This system uses an avatar to commentate the video and to act as the broker between user and video in case of conversational interaction. Interaction on the video is further possible with graphical and acoustical video hyperlinks. The

2 system is construed for single user presentations without group functionality. Real Networks is market leader in generic interactive video systems [2]. Their system is using SMIL [3] (a declarative html-like structure to synchronize multimedia streams within an html document) to add graphical video hyperlinks, web sites and other multimedia content to a video presentation. The graphical hyperlink approach is extended by [4] to the audio part of the video. Acoustical hyperlinks (so called audio hyperlinks) are played in parallel to the audio objects within a video. This broadens the hyperlink metaphor on both the graphical and the acoustic part of a video. The Telebuddy project in [5] uses one physical avatar to represent online connected users in physical environments like expositions. The users are able to speak and gesture (presented by the avatar) and they can watch and listen to the environment (recorded by the avatar). The users were able to text-chat among themselves. The system models a selection of user input that is presented by the avatar, but it does not model a conjoint group behaviour. 3. Community TV System overview The Community TV System consists of a server application, a client application and the communication protocols; see Figure 1. The server application contains the major group related intelligence and some features related to live video streaming and content tracking. The authoring of video-related interaction possibilities and conversational interaction related to the video is done separately and temporally before the video content presentation. Videograbber VCR / TV Behaviordatabase Video Hyperlinkdatabase Server / Router Pixel Handler Video Server Tracker Chat Manager Behavior Manager Video Hyperlink Processor Video Avatar Processor Complex Info Manager over TCP/IP WWW Figure 1: Server Architecture The client applications implement some real time content manipulation functionality as well as the rendering of video and humanlike avatars Server Every user who wants to be connected to a video session has to connect via a client onto the server. The server administrates all users, scheduling them to user groups (communities) related to certain video presentations. The system uses pixel manipulation within the raster image to render additional graphic content into the video stream In case of a separate presentation of the additional graphic content, a based coding is used. For each video stream there are the following components: Raster image handler: Any single information that is provided by any community client or by the server (see below Complex information manager) is converted into a separate matrix. The matrix is indicated using a unique ID number that is served along with default parameters concerning location, color and size. Video server: The Internet streamer. Tracker: For content manipulation purposes every user of a community has the possibility to mark (a limited number of) objects within the broadcasted video. The tracker generates for each object a unique object ID and tracks the object whenever it is visible in the video scene. The coordinates combined with the object s ID are sent to the complex information manager for further processing. Chat manager: The chat manager connects single users of a group to enable a one-to-one conversation. The chat manager itself has no behaviour-generator as the behaviour creation is done on the client side; see paragraph 5. The information generated by the chat manager is transferred towards the raster image handler or offered -coded for any client within the community. Behaviour manager: The conjoint group behaviour, a matching of behaviour and text input of every user of a certain community, is generated within this module; see paragraph 5. The information generated by the behaviour manager is transferred towards the raster image handler or offered coded for any client within the community. Video hyperlink processor: The interaction requests of the community are weighted. If the hits on the hyperlink exceed a critical mass the hyperlink is processed. The module gives information on the status of hyperlinks to the community clients; see paragraph 4.1. The processing of the hyperlink could be a change in the broadcasted video presentation as well as the presentation of additional multimedia information. The information

3 generated by the video hyperlink processor is transferred towards the raster image handler or offered coded for any client within the community. Video avatar processor: The video avatar is controlled by pre-authored behaviour and textual speech components. This behaviour and text are played out first hand in relation to the video. Users have the possibility to conversationally interact with the video using the avatar as an information broker; see paragraph 4.2. The conversation contributions of the several users of a community are processed and correlated. The behaviour/text answers of the avatar, as well as user query related system responses in the form of video navigation or additional information presentation, are processed to every client within the community to provide a coherent video broadcast. The information generated by the video avatar processor is transferred towards the raster image handler or offered -coded for any client within the community. Complex information manager: This manager acquires content dependent meta data. This data could be input from the behaviour manager, tracked objects with additional information, video annotations, as well as content from external information servers. This manager prepares the meta data to be transferred towards the raster image handler module, and in addition, is offered in for any client within the community The client consists of several modules for presenting animated artificial characters, video stream presentation, and content manipulation. The pixel and content manipulation module of the client is used for real time manipulation of pixels within a video frame. The video frame that is itself a raster image will be superimposed with the server-generated matrices, which are of transparent character. Before the actual manipulation, the server-generated matrix is adapted according to personalized user filters. In this context user filters include the following manipulation settings: Disable or enable the presentation of a particular information User-specified offset for information location on the client screen Appearance properties, for instance color or size If the connection between client and server has a low bandwidth, then the transfer of the matrices (therefore the visualization) of the animated artificial characters can be disabled. In this case, the client has to render the avatars himself on the basis of the server-provided protocol; see next paragraph. Behaviour Controller Communication Speech Synthesis Text to Phonem Figure 2: Context user filter Input Sensors Controller Behaviour- Library Phonem to Audio Synchronisation Presentation Manager Animation- Component Animation Element Library Figure 3: Virtual human management Avatar Library Each one of the animated artificial characters are controlled by its own instance of a virtual human manager, see Figure 3. The behaviour controller provides an interface on abstract levels [7]; see paragraph 3.1. These levels are motivations, tasks, features, and geometric control. Within the behaviour manager those behaviour instructions are processed to the fundamental geometry (so called morphing targets) and transmitted to the presentation manager that presents the avatar; see Figure 4.

4 Figure 4: Examples of avatar facial expressions generated by the presentation manager An important requirement for the Avatar to achieve a realistic appearance is speech output with lip-sync. Since the dialogue with the user is not known in advance, prerecorded animation sequences with lip animation can not serve as a solution. Instead, a realtime lip-sync with the audio output generated by a speech synthesis module [8], [9], is implemented. Text strings to be spoken by the different avatars are passed to the speech synthesis. The lip-sync presentation is handled by the presenter manager server connection The protocols between client and server are separated in control & animation channels and the video presentation channel. is selected for conveying the control & animation information among the client-applications and the system server. In our context it is important to be independent of proprietary solutions and specific platforms, ideally using deployment of web services. is, therefore, the natural choice as it is widespread, independent of any platform and supported by most web enabled services. For an application on the Internet, an appropriate page is changed into a stream and passed on to the parser. Hence, each client possesses therefore an - Parser to decode the several avatar behaviours and text, as well as an - Generator to directly connect to another client s user chat. The defined instructions are consolidated together in a DTD (Document Type Declaration). 4. Interaction on the Video content The interaction possibilities are aligned to the properties of the continuous medium that is presented. Direct manipulation is used in the form of temporal video hyperlinks [6] to permit time-dependent interaction on content that is actually visible within the video s presentation. The usage of conversational interaction is enabled to allow time-independent interaction on content regardless of the content s playback time; for example, on content that was presented or is yet to be presented by the video Temporal Video Hyperlinks The system is able to show acoustical as well as graphical annotations on the video content. Graphical annotations are rendered as polygons around the annotated video content. The user can hear acoustic annotations as a sound played in parallel to the acoustical content. The annotations show their progress in time in order to enhance the user s interaction efficiency. Figure 5: Annotation of graphical video content Figure 5 shows how the progress is graphically shown as a change in the annotation s color. The users of the system are watching synchronous to the video presentation they see the hyperlinks at the exact same time. If a user is interacting on a hyperlink, the interaction request is weighted by the system. If a critical mass is exceeded (adjustable by the author of the hypermedia interaction) the hyperlink is performed on every client of a community watching the particular video broadcast. The system gives a feedback on the weight status of every hyperlink to the community clients Conversational interaction on the video content As mentioned in paragraph 3.1, users have the possibility to conversationally interact with the video using the avatar as an information broker. They use emotions and phrases as input information that is transferred to the video avatar processor. Within the video avatar processor the input of the user community is weighted. Themes that seem very interesting to a certain number of users are chosen for a presentation using the avatar as presenter. The number of users is defined in relation to the pre-authored meta information on the video content, see Figure Community-based content manipulation Every user within a community is enabled to annotate the video frames and to set tracking points within the video. Additional information can be added to the tracked objects. The annotations and the tracking points are sent to the video server which integrates them into the video frames by means of a matrix, sending them with the frames to the community clients. (The tracking

5 points are automatically processed by the tracker on the server; see paragraph 3.1.) The annotations and the tracked objects (marked by a polygon) are observable on every client of the community in order to allow collaborative work on the content. Every user of the community is able to access additional information that is related to tracked objects by simply clicking on the object. Input Input representation of all the users of the community. The speech output of the avatar representing the group behaviour is generated as a weighted evaluation of the text inputs of the community. Text Input Text Input Text Parsing Content DB Thematic Evaluation Figure 7: Video with community avatar and additional information Server Behaviour DB Output Avatar Behaviour & Speech Output Content Generation Behaviour Generation (Group Behaviour) Avatar Behaviour & Speech Figure 6: Video avatar behaviour generation The behaviour is a composition of conversation-related speech acts, emotional input of the users and parsing of the text inputs. Within the first evaluation of Community TV, emotions are expressed via emoticons. The text output of the avatar is a summary of the text inputs of the users. Within the first evaluation of Community TV, content judgments are searched and a general judgment of the content is calculated. This general judgment is expressed within typical (preauthored) phrases like Cool stuff or This is tedious. With this community-related avatar, every user has a fast impression of the general content estimation of the community, see Figure Chat functionality and group behaviour The chat functionality and the user group behaviour are rendered on the client side by instances of the presentation manager. The single user chat allows the user to get a direct connection with another registered user of the group session. The behaviour of the user is processed to behaviour instructions and sent as behaviour tags (in the case of rendering the avatar beside the video image; see paragraph 3.2) or as a raster image (in case of rendering the avatar within the video image), the speech input is processed to text. Both are sent to the chat opponent. There, the avatar s picture is displayed lip-synced to the text that is converted to speech. Every user has the possibility to express his emotions about the actual video content via emotions and phrases. This user input is transferred to the behaviour manager on the server. Within the behaviour manager, a group behaviour is generated as a conjoint behaviour 6. Prototype The prototype was developed at ZGDV e.v. and at the Technical University of Darmstadt. The system architecture supports video transmission via Internet. The video content is provided using a VCR with a digital video camera connected as a life content source. The video channel between the Community TV client workstations and the Community TV server is established via Internet using streaming technology. The integration of the information channel within the system prototype has been done by establishing a local network based on a TCP/IP protocol. The application running on the server is completely developed into C++ in regard of the real time requirements of the Community TV system. The client application is implemented as java applets. For handling the video content concerning the pixel manipulation requirements sun s Java Media Framework [10] was used. The avatar animation is done using Java 3D [11]. The protocol browsers and parsers were implemented using the Sax parser [12].

6 7. Conclusion and future work The approach discussed in this paper shows how interaction metaphors like conversational interaction between system and user are expanded to the interaction between the system and groups of users, as well as to the interaction among the users. With the Community TV system collaborative watching of video content is possible and collaborative estimation of video content is introduced. The system is implemented as a prototype; it is now within a phase of evaluation. Future work will be done by improving the group functionality on the server side and by advancing the client s user interface including user navigation. A major concern is the development of an integrated authoring system that helps the author to arrange the interactive components of a video presentation. New ways of dramaturgy and rhetoric have to be examined in order to allow an immersive and enjoyable conjoint television experience. In addition, the transmission of video content could be performed by broadcast systems, such like TV stations. Visualization and Computer Vision, Prague, Czech Republic, [8] Faculte Polytechnique de Mons, MULTITEL-TCTS Lab, The MBROLA Project Towards a Freely Available Multilingual Synthesizer, Mons, Belgium, [9] Institut für Kommunikationsforschung und Phonetik IKP, German TTS front end for the MBROLA synthesizer, Bonn, Germany, [10] Java Media Framework, USA, 2001 [11] Java 3D, USA, [12] Sax Parser, Java Technology &, USA, References [1] Braun, N., and M. Finke, Interaction of Video on Demand Systems with Human-like Avatars and Hypermedia, Proceedings of the 7th International Workshop on Interactive Distributed Multimedia Systems and Telecommunication Services, Enschede, The Netherlands, [2] Hefta-Gaub, B., The Real Media Platform Architectural Overview, Real Networks Conference, San Francisco, USA, [3] World Wide Web Consortium W3C., Synchronized Multimedia Integration Language (SMIL) 1.0 Specification [4] Braun, N. and R. Doerner, Using Sonic Hyperlinks for Web-TV, International Conference on Auditory Displays ICAD Proceedings, Glasgow UK, [5] Dechau, J., Finke, M., Gerfelder, N., Ide, R., Kirste, T. and U. Spierling, The Telebuddy : Collective Tele- Presence and Tele-Conversation through Physical Avatars, Proceedings of the Interactive Application of Mobile Computing Conference, Rostock-Warnemünde, Germany, [6] Braun, N., and R., Doerner; Temporal Hypermedia for Multimedia Applications in the World Wide Web, Proceedings of the 3rd International Conference on Computational Intelligence and Multimedia Applications, Neu Delhi, India, [7] Rieger, T., Alexa, M., Hellenschmidt, M., and Berner U. An Animation System for User Interface Agents, Proceedings of the 9-th International Conference in Central Europe on Computer Graphics,