D4.1: Report on Initial Consensus Standards for Platform and Interaction

Transcription

1 D4.1: Report on Initial Consensus Standards for Platform and Interaction 0

2 Project ref. no. Project acronym Project full title Security IST M4 MultiModal Meeting Manager Restricted Contractual date of delivery M12 (28 February 2003) Actual date of delivery 17 March 2003 Deliverable number Deliverable name Type D41 Report on Initial Consensus Standards for Platform and Interaction Report Status and version V2.0 Number of pages 12 WP contributing to deliverable WP/Task responsible Other contributors Editor(s) EC project officer Key words Abstract WP4 IDIAP USFD, EPFL, TUM, TNO, UniGe, VUT Brno Pierre D. Wellner (IDIAP) Domenico Perrotta Information access, multimodal information The purpose of this document is to describe the initial consensus standards for systems that will allow M4 partners to share data files with each other and to build integrated applications that demonstrate M4 results. The initial data to be shared by the project are multimodal meeting recordings from IDIAP that include dozens of gigabytes of audio, video, and XML per meeting. We also intend to store the ICSI meeting data which includes audio files and manual transcriptions, but no video. Additional data, including annotations and transcriptions (both automatic and manual) will be derived from the original data and also stored in these systems. 1

3 Report on Initial Consensus Standards for Platform and Interaction Workpackage 4: Demonstration and Evaluation Deliverable D4.1 First draft for comments & contributions 1 Workpackage summary: Participants: Aim in first period Outline of the deliverable Introduction Database Architecture Functions of Media File Server Browsing Playing Retrieving Adding Data rates and formats Video Audio Annotations and XML Documents Synchronization Organization of files on media file server Video Filenames and timestamps Recording session directories Naming of files in session directories Access control Backup Demonstration Plans for next period References

4 1 Workpackage summary: 1.1 Participants: IDIAP (WP leader), CH USFD, UK EPFL, CH TUM, DE TNO, NL UniGE, CH UT, NL VUT Brno, CZ 1.2 Aim in first period WP4 consists of the construction of a demonstration system using components developed in the other WPs, along with formal and informal evaluation of the system as a whole and its component technologies. The aim in the first period is to build initial consensus on common standards and infrastructure that will be used to integrate and share components from the other WPs, including the sharing of raw data files and recognition results. This report is due as deliverable D4.1 (Report on initial consensus standards for platform and interaction) at Milestone M2.1 (month m12, End of February, 2003). 1.3 Outline of the deliverable The purpose of this document is to describe the initial consensus standards for systems that will allow M4 partners to share data files with each other and to build integrated applications that demonstrate M4 results. The initial data to be shared by the project are multimodal meeting recordings from IDIAP that include dozens of gigabytes of audio, video, and XML per meeting. We also intend to store the ICSI meeting data which includes audio files and manual transcriptions, but no video. Additional data, including annotations and transcriptions (both automatic and manual) will be derived from the original data and also stored in these systems. 2 Introduction This WP will bring together the diverse technologies developed in the other projects to create integrated multimodal offline meeting management tools. It will also be the goal of this WP to address the portability and standardization issues, as well as the final evaluation of the system. 3 Database Architecture The M4 database will consist of two separate components: the Media File Server and the SQL Database. The Media File Server stores data files and provides a simple means to access these files. The SQL Database stores higher level descriptions about these data files and provides a query-based interface. These two components will work together to support processing, access, and demonstrations with M4 data. 3

5 M4 Database clients Media File Server SQL Database FTP HTTP RTSP/ RTP details TBD Media files Figure 1: Top level architecture for M4 database system to support integration, and demonstration systems. Interfaces to the media file server will use the standard internet protocols FTP, HTTP, and RTSP/RTP (file transfer protocol, hypertext transfer protocol, real time streaming protocol, and real-time transport protocol) These standards are supported by a wide range of application-building tools, and offer the greatest flexibility for integration, sharing of data, and building demonstration systems. The SQL database will store interval annotations about the media files in a form that can be queried, and will support XML-based interfaces for input and output. Version 0 of the system developed in this period focuses on the Media File Server rather than on the SQL Database for two reasons: 1) There is an urgent need for M4 partners to share, store and retrieve recorded meeting data. The Media File Server can support this immediately, and be enhanced later. 2) The SQL Database component depends on the Media File Server component whereas it is possible to implement stand-alone functionality for the Media File Server without an SQL Database. 4 Functions of Media File Server The system must support four basic tasks: browsing, playing, retrieving, and adding of recorded multimodal data files. 4.1 Browsing Each recorded session has a directory on the file server and a dynamically generated "home page" that displays all available files including a jpeg images for each video file [Figure 2]. Every file is downloadable from this page by FTP or HTTP from 4

6 Figure 2: Web interface to home page of one recording on the media file server. All audio and video files can be downloaded separately or can be played back synchronously (see Figure 3). 4.2 Playing The session "home page" has images and buttons to stream any audio or video file using RealPlayer on Unix of Windows, and a "Synchronized Play" button that dynamically generates a SMIL 1 presentation from all user-selected "checked" media clips to display them simultaneously in sync [Figure 2]. Start time offsets and durations are part of the file names to allow the media file server (and other software) to account for varying start times of concurrently recorded audio and video files. 1 SMIL (pronounced smile ) is the World Wide Web Consortium (W3C) Synchronized Multimedia Integration Language. 5

7 Figure 3: Synchronized playback of three video streams and several audio streams. In this example, hand annotations and automatic annotations are also presented as links for navigating within the recording. 4.3 Retrieving Data needs to be retrieved in at least the following three ways: 1) Manually, for selected segments of interest to researchers. 2) Automatically, for bulk processing. 3) As part of user interfaces and demonstration applications. Data can be downloaded for processing on local computers using either HTTP or FTP. To download over HTTP, users can right-click-save on the desired filename extension in a web browser. To use anonymous FTP, simply connect to ftp://mmm.idiap.ch with any FTP client. 4.4 Adding Tools are available to rename and format recorded data so it can be displayed, previewed, and retrieved using the web and SMIL user interfaces on this server. At this time, however, new data files are still added manually by server administrators. If partners have multimodal media data to contribute, they can contact mmmadmin@idiap.ch for help with making it available on this server. 6

8 5 Data rates and formats The Media File Server does not impose strict constraints on data rates or formats. Data is expected to come from a variety of sources in a variety of formats. Like any web server, the Media File Server must be flexible enough to store and serve many different kinds of data and make it available in a consistent manner. Applications and systems that make use of data on the media file server sometimes limit the formats they can support, however. 5.1 Video The IDIAP meeting room currently records three synchronized video streams in PAL Digital Video format (but may produce additional streams in the future). This Video is transmitted over 1394/FireWire and stored on disk in Microsoft AVI format using the AVI-IO program and the Adaptec DVSoft dvsd codec. The result is an AVI file with a single video stream per file and no audio at approximately 25 frames per second at 720(w) x 576(h). File sizes are about 12 Gigabytes per hour per camera, or up to approximately 36 Gigabytes/hour for three cameras. This AVI format can be read by a wide variety of MS Windows and Linux utilities (including mplayer and MainActor on Linux). An alternative is to use the DivX codec, which is a widely used version of MPEG-4 that has open source implementations and is supported on Windows and mplayer on Linux. The compression provided by DivX creates files up to 50 times smaller, and yet vision researchers have found its quality to be sufficient. Although initial files on the media file server have been stored using the DVSoft dvsd format, the project consensus is to use DIVX for the future. 5.2 Audio The IDIAP meeting room records up to 24 channels of audio into separate wave files at 16 KHz 16 bits per sample PCM, or a total of 256Kbps or about 109 Megabytes per hour per channel, or up to approximately 2.6 Gigabytes per hour for 24 channels. In the future additional streams of audio (e.g. telephone) may also be included. The ICSI meeting data consists of up to 16 channels, each in its own file, also at 16 khz 16 bits per sample. Other formats could also be supported by the Media File Server, as long as they can be handled by the Streaming Server, but external programs that access files on the Media File Server (e.g. speech processing software) may require 16 khz 16 bit audio. 5.3 Annotations and XML Text files may also be included in the raw data or (more frequently) the derived data. These should be formatted as XML, are expected to be much smaller than the video data, and are usually smaller than the audio data as well. The Media File Server will not impose any particular formatting restrictions on this XML, but restrictions TBD will be imposed by other systems that make use of this data, including, for example, the programs that load XML data into the SQL Database. Note that after the SQL Database has been implemented and supports the loading of XML files into its tables, then the owners of these files may choose to delete the XML from the Media File Server in order to avoid potential inconsistency with records in the SQL Database. 5.4 Documents The documents used by participants during a meeting may include an agenda, slides, articles, etc. They may be projected on a screen or printed before the meeting. After a meeting, all the documents may be collected to be added to the meeting archives. Some of the documents will be annotated, including temporal annotations such as times at which slide changes occurred and/or structural annotations describing the physical and logical composition of documents. 7

9 6 Synchronization Multimodal data files come from different sources, but the Media File Server will store them in such a way that all time-stamped data files can be synchronized with other time-stamped data files, even if stored on other instances of the Media File Server. All audio files from a single recording session at the IDIAP meeting room are guaranteed to start exactly at the same time, and every audio sample in each file is synchronized with all the others (see. By convention, the start time for audio recordings is at SMPTE time-code of 0:0:10:0 (0 hours: 0 minutes: 10 seconds: 0 frames) where a frame is 40 milliseconds. Each video frame recorded in the IDIAP meeting room is also labeled with the SMPTE code in the vertical interval, and then encoded in the first two lines of every frame, thus providing that frame s offset from 10 seconds before the start of the audio recordings. A utility written at IDIAP extracts this time code from the first two lines of a video frame, and the source code for this will be made available. Using this information, each video file can be named with its offset from the start of the recording session, as described in more detail below. 7 Organization of files on media file server As discussed, the Media File Server imposes few hard constraints on the formats of data that it can store, but it does specify the way that files are organized and named. 7.1 Video Video files larger than a few hundred megabytes are broken into smaller files so that they can be downloaded over the Internet within a reasonable amount of time. Each clip is labeled with its timestamp and duration (see below). A one minute video clip is about 200 Megabytes, and should take about 25 minutes to download at a bandwidth of approximately 1mbps. Because the video data is so large, it is important to also provide low bit-rate previews of video data to help vision researchers find the data they need without having to download it all. So in addition to full quality video, there will also be a highly compressed video file available for previewing. For previewing, we have selected RealMedia format because of the wide accessibility of Real players available on both Windows and Unix platforms, and the ability to integrate Real players with web pages. The compressed preview files (both video and audio) is available by ftp for downloading to a local machine, and is also be available for streaming over RTSP/RTP to a local player without the need to download the entire file. It is also possible to preview a session as a single unit without regard to how the video files might be broken into segments. JPEG key frames can be dynamically extracted from the video files based on time offset using a web-based CGI program on the media file server that takes as arguments the video file name and a timestamp. This can be used, for example, by any HTML page that needs to display video keyframes. 7.2 Filenames and timestamps Every recording session is stored in a directory with a unique name, and the various media files associated with the recording session are stored in the directory. File naming conventions should ensure uniqueness and also provide timestamp information if available. We allow files to be stored on multiple servers and be accessed using HTTP or FTP with standard URL naming conventions. It is possible, for example, to store original and derived files on different file servers and still be associated with each other through their timestamps. The purpose of file naming conventions that include timestamps is for use by programs that automatically extract desired segments of media files. 7.3 Recording session directories Every recording session has a directory with a unique name that can include a timestamp formatted according to conventions below. Time stamp names are based on the timestamp 8

10 format defined in IETF RFC 3339 but we do not include the : character because it is illegal within a Windows filename. NameOfRecordingSession_yyyy-mm-ddThhmmss.#Z The timestamp following the underscore is an absolute timestamp that indicates when the recording session was started in UTC. yyyy is a four-digit year, mm is the month, dd is the day of the month, hh is the hour, mm is the minute, ss is the seconds, and the # represents a string of digits for the fractional part of a second (e.g. 3 digits for milliseconds, 1 digit for tenths of seconds). The Z represents a Universal Coordinated Time (UTC) time zone. The hours, minutes, seconds, and milliseconds are optional and assumed to be zero if left out. Missing or 0 values for the date indicate that this information is unknown. The NameOfRecordingSession string should not include spaces to facilitate storing these files on standard file servers. It will begin with a directory name or string that identifies the source (e.g. ICSI, IDIAP, LDC or IIUF ), and each source may adopt further documented naming conventions to provided additional information in the file name such as the number of participants. Note that the filename is not intended to provide complete information about the recording session, however. If necessary, more detailed labels can be included in a separate XML file in the directory and in the associated records to be stored in the SQL Database. 7.4 Naming of files in session directories Files stored in session directories include a timestamp that is relative to the start of the recording session absolute timestamp. They are named according to the following convention: NameOfSource_relativeTimeOffset_duration.extension Where relativetimeoffset and duration have a similar format as described for absolute timestamps above: yyyy-mm-ddthhmmss.#.extension but there is no Z. For offsets and durations shorter than a day, it s ok to include only seconds and parts of seconds, or only minutes and seconds. To make the duration units unambiguous the T must be included. Any time durations shorter than one second must specify 0 seconds and include the point after the 0. NameOfSource must be unique within the recording session, and can either be an original raw data source (e.g. Camera1 or Microphone3), or it can be a derived source (e.g. RealProducer or JPEGKeyframeExtractor) Example: Camera1_T001022_T avi is an AVI video file of duration 40 seconds and 300 milliseconds that begins 10 minutes and 22 seconds after the start of the recording session. The recording session directory can include subdirectories. In this case, the name of the subdirectory or source does not have to include a timestamp. 8 Access control Each session (directory) has an owner who has an account on the system, and the directory can only be written by that owner. Anyone with an owner account on the Media File Server can create additional session directories which can either contain original data or data derived from another session. Owner accounts with permission to create and write to recording sessions will be manually created by the media file server administrator. Uploading of files to the media file server will not be supported through standard ftp because of security risks. For version 0, uploading will require manual assistance from an administrator, but in future versions, we also plan to support upload through the standard HTTP POST mechanism over https (e.g. as used by RFC 1867). Read access will be allowed for any partner, and implemented using standard HTTP and FTP authentication. In version 0, we allow any partner to have read access to all data and not attempt to hide any of it. In future versions, we will provide session owners with the ability to protect their data from read-access, for example by creating un-indexed un-guessable random session identifiers. 9

11 9 Backup The original data, directory structures and time stamped filenames must be backed up in case of accidents. We will keep the original DV tapes and copies of the raw data files on DVD, but this is not sufficient because it will not include all the post processing. The solution we have adopted is to maintain an on-line replicated mirror of the data on a separate file server using a utility such as rsync that runs approximately once per week. Although this approach doubles the cost of online storage hardware, it provides good data protection with minimal cost of labor and removable media. If we need to duplicate the entire database for someone, we have multiple options to consider, including making a copy onto DVDs or copying it onto a new file server or disk to be taken away. 10 Demonstration Meeting Browser: To highlight of the capabilities SVG and SMIL to create a demonstrator, we propose a Video Meeting Browser using RealPlayer driven by SVG (W3C s Scalable Vector Graphics) within Adobe s SVG Viewer. The browser is composed of a viewing video and slider for scrolling to RealPlayer Synchronised controls Control via JavaScript Dialog Zone SVG viewer Physical layout Agenda display any position. The agenda is shown as a list of topics each displayed in its own colour. Clicking at any position seeks within the top video. Live version at Speaker turns are an essential cue for browsing. Here, speaker turns are displayed as colour blocks, and clicking on these sets the audio and video player to the beginning of the speaker 10

12 turn. Whenever playing, the timeline is synchronised, and active speakers are highlighted at their physical location. Likewise, dialog transcripts appear in a roll-over fashion. This initial browser shows that important information can be displayed and highlighted using these technologies. Additional features being considered include the following: Complete interactivity: Clicking on any item triggers the playing at this location (mostly done) Displaying physical Smart Meeting Room equipment for better presentation of the scene. This could be completed by an interactive view for synthesising unseen views. Display statistics: e.g. speaker talkativity, topic importance, clickable pie-charts, etc. Add scalability. It is clear that the above interface works for short meetings, but for longer meetings with a high number of speaker turns, zooming facilities and hierarchical browsing should be considered. Enhance symbolic coding: In the proposed interface, both participants and topics are colour-coded. This may be confusing. There is a need to define efficient symbolic coding for intuitive use of the interface. The use of a metaphor in this context can prove useful. Support interactive annotations, so users can add their own annotations to the data while browsing (e.g. validation of turns, topics, and keyframe/milestone marking). Remarks: All the above can clearly be generated automatically from the data stored in a DB that would embed the data model proposed earlier. The data would be read and formatted on request by a servlet or a CGI script. For now, it uses random data. Reuse of the code is made possible with the use of an open source Java SVG interpreter such as the Apache Batik package. 11 Plans for next period 1. Definition of specific XML tags and attributes (i.e. DTDs and Schemas) for media file annotations as well as related database schemas. 2. Implementation of processing XML data in and out of SQL database. 3. Implementation of preliminary dummy applications that integrates preliminary outputs of other WPs. 12 References References: 1. W3C SVG web page : 2. W3C SMIL page: 3. Adobe SVG zone: 4. RealNetworks (player and sersver) home page : 5. SVG Essentials. J. David Eisenberg O Reilly Publishers. February Apache s Batik homepage: 7. Open source MPEG-4 DivX project 11