Multimedia Sape J. Mullender Huygens Systems Research Laboratory Universiteit Twente Enschede 1
What is Multimedia? Videophone CD Interactive One socket, many services (video) phone TV video on demand Teletext Computer Computer-supported cooperative work 2
Good Multimedia Two classes of media Static: text, images, 3D graphics Continuous: audio, video, sensor data Media are integrated Operating system supports all media Applications can process all media 3
Continuous Media Issues in processing continuous media Minimize Latency Minimize Jitter Achieve necessary Throughput Late data is useless data 4
Bad data Corrupted data (transmission errors) or late data (retransmissions) is useless and will be dropped on the floor. But if the amount of data lost is small, e.g., one video frame or 20 ms of audio, It is barely noticeable. 5
Small Data Units There are two reasons for using small units of data: Reduction of end-to-end latency Loss of an occasional unit is not noticed Error recovery through retransmission is not useful: retransmitted data would arrive too late 6
Some Numbers Acceptable end-to-end latency for audio Acceptable length of missing audio unit Acceptable end-to-end latency for video Acceptable length of missing video (one frame) 40 ms 10 ms 40 ms 25 ms Telephone-grade audio, 8-bit samples at 8 KHz CD-quality stereo, 2 16-bit samples at 44.1 KHz Uncompressed video, 25 fps, 640 480 24-bit pixels JPEG-compressed video, 25 frames per second MPEG-compressed video, 25 frames per second 64 Kbp 1.4 Mb 200 Mb 8 Mb 4 Mb 7
Audio and Video over ATM networks An ATM cell contains 48 bytes Telephone-grade audio audio CD-quality stereophonic audio Uncompressed full-colour video JPEG-compressed video, 25 frames per second MPEG-compressed video, 25 frames per second 6 ms/cell 275 µs/cell 2 µs/cell 60 µs/cell 200 µs/cell 8
Cells and Frames If cells were sent separately, hosts might have to deal with up to half a million interrupts per second. Not a good idea. ATM Adaptation Layers group cells in larger units called frames. There were four types of AAL: AAL1: Continuous bit-rate transmission AAL2: Variable bit-rate transmission AAL3: Connection-oriented data AAL4: Connection-less data AAL3 and AAL4 have been merged: AAL3/4. AAL5, the Simple and Efficient Adaptation Layer (SEAL) has been standardized. The need for AAL2 is unclear so it never got standardized. 9
Buffering Necessary for jitter elimination Smaller buffers improve latency Bigger buffers improve throughput 10
Workstation Architecture 11
Problems with bus-based architectures Processing audio and video implies very low cache hit rates Memory is a bottleneck (Pegasus File Server) CPU is involved in all data transfers 12
The Desk-Area Network 13
Using the Desk-Area Network Memory no longer a central resource CPU manages connections, not transfers Devices can interact directly, without CPU mediation 14
The Pegasus Project Sept 1993 Sept 1996: Pegasus I University of Twente and University of Cambridge Nov 1996 Nov 1999: Pegasus II University of Cambridge, University of Glasgow, University of Twente, Swedish Institute for Computer Science, APM Ltd. Cambridge 15
Pegasus Architecture 16
ATM Camera 17
ATM Camera CCD Arrary Scan / Digitise Output Buffers Optional Compress ATM fabric 18
Tiles 19
ATM Display 20
ATM Display 160Mbps 960Mbps Video Frame Buffer ATM fabric CPU Memory 21
Display Management 22
Experience with ATM Devices All positive: Simple, potentially cheap hardware Does not interfere with processors on the bus Window management mechanisms in hardware, policies still under control of the user Integrates well with existing systems (using ATM networks) ATM will become more standardized than processor buses 23
Multimedia and Real time Real-time systems Hard deadlines Bounded load Run times known (and bounded) Schedule guarantees deadlines Multimedia systems Soft deadlines Load is not known a priori Run times often estimated Statistical guarantees at best 24
Example Application Federico is an application that runs on Nemesis and shows its capabilities: Process Multimedia Streams in Real Time Synchronize Multimedia Streams Provide Quality-of-Service ( ) Control to Applications Capture, Transport and Render Multimedia Streams with No Discernible Latencies 25
Goal Federico is an application that controls multiple cameras in a room. Based on the camera input and the input from several microphones, Federico controls the cameras (pan/tilt/zoom) and, in a continuous process, selects one of the camera outputs for transmission to a remote site. It is Federico s goal to provide to remote viewers, through the lenses of alternating cameras, an effective overview of what is happening in the room. To achieve this goal, Federico tracks people as they move about and locates speakers in space by analysing the microphone inputs. 26
Situation 27
Federico Architecture 4 ATM audio streams Audio filtering & timing Stream of sightings Audio skeptic Better stream of sightings Geometry queries Geometry queries Geometry server Federico Camera selection Stream of sightings Better stream of sightings Camera tracker Video skeptic Camera control Camera commands Pan/tilt/zoom control... Geometry queries Camera control Pan/tilt/zoom control Camera tracker Stream of sightings Video skeptic MM stream Data stream RPC 28
Face Tracking 29
Face Tracking 30
Face Tracking 31
Quality of Service Applications tell the system what resources they need as a function of the quality they can deliver. The system provides resources to applications. The allocation may change over time, but the applications are notified of these changes beforehand. Applications adapt to the resources allocated to them. 32
Adaptation Minor: frame rate, frame size, stereo/mono. Major: compression, hardware support Continuous adaptation is hardly useful 33
Multimedia Streams A multimedia stream is usually composed of synchronized substreams, e.g., lip-synchronized audio + video. These can be coded in a single data stream. But that makes adaptation to changing resources harder 34
Separate Streams Pegasus encodes a set of synchronized multimedia data streams separately and adds a single control/synchronization stream. 35
How applications adapt Typically, each stream type has just a few QoS settings, audio, for example, could have 44 KHz stereo, 20 KHz mono, and 8 KHz mono. The applications have code for each setting and switch between them when QoS settings change. Applications consist of management threads that are scheduled conventionally and media threads that are scheduled periodically 36
Resource Allocation Each application has a small number of QoS settings. Each QoS setting has a value indicating its QoS and a list of resource quantities necessary to achieve that QoS. Typical resources: CPU bandwidth, Network bandwidth, Special devices (compression device, rendering device, etc.) and device bandwidth (e.g., for an ATM camera). The best resource allocation is that which maximizes the sum of the Qualities of Service of the applications. 37
Calculating Resource Allocation Typically, a machine will have no more than eight applications with, perhaps, three or four QoS setting each. Calculating the optimal resource allocation for a machine is straightforward. However, in a distributed setting, there are complicating factors. 38
Resource Allocation in a Distributed System Different resource allocators linked together by applications. Closure may become very large...... and may involve multiple management domains. 39
Deliverable and Desirable Applications operating under different resource allocators are responsible for obtaining allocations in each allocation domain that can be combined. A setting in an allocation domain is deliverable if the resource allocator can provide it (given its obligations to other applications) A setting is desirable if the application can make of that setting, given what is deliverable in the other allocation domains of concern. Application tells allocator which settings are desirable; allocator tells application which settings are deliverable (and which one is currently delivered). 40
Deliverable and Desirable A setting is desirable in an allocation domain if it is deliverable in all of the others. A setting is deliverable in an allocation domain if the requested resources can be made available to the application. The resources actually allocated are the maxima of the settings that are both desirable and deliverable. 41
Matchmaking between theory and practice Here, the choice of a resource-allocation algorithm was dictated to a large extent by practical factors: QoS settings are discrete rather than continuous. A few settings suffice. Separate audio and video streams make separate QoS management possible Each management domain wants to have its own QoS manager (aka, resource allocator). The system must end up with reasonable APIs. 42