Multimedia. Sape J. Mullender. Huygens Systems Research Laboratory Universiteit Twente Enschede

Transcription

1 Multimedia Sape J. Mullender Huygens Systems Research Laboratory Universiteit Twente Enschede 1

2 What is Multimedia? Videophone CD Interactive One socket, many services (video) phone TV video on demand Teletext Computer Computer-supported cooperative work 2

3 Good Multimedia Two classes of media Static: text, images, 3D graphics Continuous: audio, video, sensor data Media are integrated Operating system supports all media Applications can process all media 3

4 Continuous Media Issues in processing continuous media Minimize Latency Minimize Jitter Achieve necessary Throughput Late data is useless data 4

5 Bad data Corrupted data (transmission errors) or late data (retransmissions) is useless and will be dropped on the floor. But if the amount of data lost is small, e.g., one video frame or 20 ms of audio, It is barely noticeable. 5

6 Small Data Units There are two reasons for using small units of data: Reduction of end-to-end latency Loss of an occasional unit is not noticed Error recovery through retransmission is not useful: retransmitted data would arrive too late 6

7 Some Numbers Acceptable end-to-end latency for audio Acceptable length of missing audio unit Acceptable end-to-end latency for video Acceptable length of missing video (one frame) 40 ms 10 ms 40 ms 25 ms Telephone-grade audio, 8-bit samples at 8 KHz CD-quality stereo, 2 16-bit samples at 44.1 KHz Uncompressed video, 25 fps, bit pixels JPEG-compressed video, 25 frames per second MPEG-compressed video, 25 frames per second 64 Kbp 1.4 Mb 200 Mb 8 Mb 4 Mb 7

8 Audio and Video over ATM networks An ATM cell contains 48 bytes Telephone-grade audio audio CD-quality stereophonic audio Uncompressed full-colour video JPEG-compressed video, 25 frames per second MPEG-compressed video, 25 frames per second 6 ms/cell 275 µs/cell 2 µs/cell 60 µs/cell 200 µs/cell 8

9 Cells and Frames If cells were sent separately, hosts might have to deal with up to half a million interrupts per second. Not a good idea. ATM Adaptation Layers group cells in larger units called frames. There were four types of AAL: AAL1: Continuous bit-rate transmission AAL2: Variable bit-rate transmission AAL3: Connection-oriented data AAL4: Connection-less data AAL3 and AAL4 have been merged: AAL3/4. AAL5, the Simple and Efficient Adaptation Layer (SEAL) has been standardized. The need for AAL2 is unclear so it never got standardized. 9

10 Buffering Necessary for jitter elimination Smaller buffers improve latency Bigger buffers improve throughput 10

11 Workstation Architecture 11

12 Problems with bus-based architectures Processing audio and video implies very low cache hit rates Memory is a bottleneck (Pegasus File Server) CPU is involved in all data transfers 12

13 The Desk-Area Network 13

14 Using the Desk-Area Network Memory no longer a central resource CPU manages connections, not transfers Devices can interact directly, without CPU mediation 14

15 The Pegasus Project Sept 1993 Sept 1996: Pegasus I University of Twente and University of Cambridge Nov 1996 Nov 1999: Pegasus II University of Cambridge, University of Glasgow, University of Twente, Swedish Institute for Computer Science, APM Ltd. Cambridge 15

16 Pegasus Architecture 16

17 ATM Camera 17

18 ATM Camera CCD Arrary Scan / Digitise Output Buffers Optional Compress ATM fabric 18

19 Tiles 19

20 ATM Display 20

21 ATM Display 160Mbps 960Mbps Video Frame Buffer ATM fabric CPU Memory 21

22 Display Management 22

23 Experience with ATM Devices All positive: Simple, potentially cheap hardware Does not interfere with processors on the bus Window management mechanisms in hardware, policies still under control of the user Integrates well with existing systems (using ATM networks) ATM will become more standardized than processor buses 23

24 Multimedia and Real time Real-time systems Hard deadlines Bounded load Run times known (and bounded) Schedule guarantees deadlines Multimedia systems Soft deadlines Load is not known a priori Run times often estimated Statistical guarantees at best 24

25 Example Application Federico is an application that runs on Nemesis and shows its capabilities: Process Multimedia Streams in Real Time Synchronize Multimedia Streams Provide Quality-of-Service ( ) Control to Applications Capture, Transport and Render Multimedia Streams with No Discernible Latencies 25

26 Goal Federico is an application that controls multiple cameras in a room. Based on the camera input and the input from several microphones, Federico controls the cameras (pan/tilt/zoom) and, in a continuous process, selects one of the camera outputs for transmission to a remote site. It is Federico s goal to provide to remote viewers, through the lenses of alternating cameras, an effective overview of what is happening in the room. To achieve this goal, Federico tracks people as they move about and locates speakers in space by analysing the microphone inputs. 26

27 Situation 27

28 Federico Architecture 4 ATM audio streams Audio filtering & timing Stream of sightings Audio skeptic Better stream of sightings Geometry queries Geometry queries Geometry server Federico Camera selection Stream of sightings Better stream of sightings Camera tracker Video skeptic Camera control Camera commands Pan/tilt/zoom control... Geometry queries Camera control Pan/tilt/zoom control Camera tracker Stream of sightings Video skeptic MM stream Data stream RPC 28

29 Face Tracking 29

30 Face Tracking 30

31 Face Tracking 31

32 Quality of Service Applications tell the system what resources they need as a function of the quality they can deliver. The system provides resources to applications. The allocation may change over time, but the applications are notified of these changes beforehand. Applications adapt to the resources allocated to them. 32

33 Adaptation Minor: frame rate, frame size, stereo/mono. Major: compression, hardware support Continuous adaptation is hardly useful 33

34 Multimedia Streams A multimedia stream is usually composed of synchronized substreams, e.g., lip-synchronized audio + video. These can be coded in a single data stream. But that makes adaptation to changing resources harder 34

35 Separate Streams Pegasus encodes a set of synchronized multimedia data streams separately and adds a single control/synchronization stream. 35

36 How applications adapt Typically, each stream type has just a few QoS settings, audio, for example, could have 44 KHz stereo, 20 KHz mono, and 8 KHz mono. The applications have code for each setting and switch between them when QoS settings change. Applications consist of management threads that are scheduled conventionally and media threads that are scheduled periodically 36

37 Resource Allocation Each application has a small number of QoS settings. Each QoS setting has a value indicating its QoS and a list of resource quantities necessary to achieve that QoS. Typical resources: CPU bandwidth, Network bandwidth, Special devices (compression device, rendering device, etc.) and device bandwidth (e.g., for an ATM camera). The best resource allocation is that which maximizes the sum of the Qualities of Service of the applications. 37

38 Calculating Resource Allocation Typically, a machine will have no more than eight applications with, perhaps, three or four QoS setting each. Calculating the optimal resource allocation for a machine is straightforward. However, in a distributed setting, there are complicating factors. 38

39 Resource Allocation in a Distributed System Different resource allocators linked together by applications. Closure may become very large and may involve multiple management domains. 39

40 Deliverable and Desirable Applications operating under different resource allocators are responsible for obtaining allocations in each allocation domain that can be combined. A setting in an allocation domain is deliverable if the resource allocator can provide it (given its obligations to other applications) A setting is desirable if the application can make of that setting, given what is deliverable in the other allocation domains of concern. Application tells allocator which settings are desirable; allocator tells application which settings are deliverable (and which one is currently delivered). 40

41 Deliverable and Desirable A setting is desirable in an allocation domain if it is deliverable in all of the others. A setting is deliverable in an allocation domain if the requested resources can be made available to the application. The resources actually allocated are the maxima of the settings that are both desirable and deliverable. 41

42 Matchmaking between theory and practice Here, the choice of a resource-allocation algorithm was dictated to a large extent by practical factors: QoS settings are discrete rather than continuous. A few settings suffice. Separate audio and video streams make separate QoS management possible Each management domain wants to have its own QoS manager (aka, resource allocator). The system must end up with reasonable APIs. 42