Chapter 2: Representation of Multimedia Data Chapter 3: Multimedia Systems Communication Aspects and Services Multimedia Applications and Communication Protocols Quality of Service and Resource Management Synchronization Multimedia Operating Systems Chapter 4: Multimedia Systems Storage Aspects 3.1: Multimedia Applications and Communication Classification and requirements of multimedia applications Control protocols: the H.x Protocol Family Control Protocols: Session Initiation Protocol SIP Streaming Multimedia Data Transfer Protocols: RTP and RTCP Multimedia Applications Important term: Quality of Service (QoS): the network provides the application with a level of performance needed by the application to work Multimedia applications: mainly audio and video transmission ( continuous media ) Page 1 Page 2 Classification of Applications Transport and Network Layer Classes of multimedia applications 1.) Streaming stored audio and video 2.) Streaming live audio and video 3.) Real-time interactive audio and video Fundamental characteristics Typically delay sensitive end-to-end delay jitter But most times also loss tolerant: infrequent losses cause minor glitches Requirements to communication Transfer protocols with small delay but also weak reliability (TCP, UDP, or something else?) Control protocols for signaling between communicating applications (e.g. phone ringing ) Quality of Service guarantees within the network (IP layer): routers and resource reservations Page 3 Multimedia applications have high requirements to protocols: 1. Transport protocols Deliver as much data as possible in short time audio and video data typically have a stream-like behavior (16 kbit/s for compressed audio, 64 kbit/s for PCMaudio in telephony, 2 Mbit/s for MPEG-coded video) New transfer protocols needed RTP 2. Control protocols Deliver data with regard to negotiated policies (throughput, delay) and/or signaling information Control protocols needed H.323, SIP 3. Quality of Service (chapter 3.2) Deliver data as fast as possible and with low jitter real-time communication demands low end-to-end delays, typically less than 200 msec. End-to-end delay is limited by the routers, thus e.g. routing strategies have to be modified Network protocol enhancements needed scheduling, resource reservations, traffic shaping Page 4
Transfer and Control Protocols A main protocol family is the H.x standards by the ITU Contains video coding standards for video conferences, similar to MPEG Also: audio coding standards based on PCM H.323 is a control protocol for management of a communication session, comprising several control sub-protocols Developed by ITU, driven by telecommunication needs Alternative for session management: Session Initiation Protocol (SIP) Only one protocol, not a protocol family Developed by IETF: integrated with the Internet Additionally: RTP/RTCP as transfer protocols H.x and SIP both are not defining transfer protocols RTP as a special transfer protocol basing on UDP Standards of ITU The ITU has standardized everything needed in cooperative computing: G.711, G.722, G.723, G.728, G.729 for audio coding with 5.3 64 kbit/s H.261, H.263, H.264, for video coding similar to MPEG H.245 for controlling media streams H.450 for negotiation of communication resources H.235 for authentication and encryption H.225.0 for connection setup and termination, packetizing of data streams, signaling, H.323 for controlling and coordination and several more, e.g. T.x for data transfer User Interface Audio Video Configuration Audio Codecs G.711 G.722 H.323 Video Codecs H.261 H.263 H.225.0 Layer Network Interface H.245 H.450 H.235 Page 5 Page 6 H.323 Components Session Initiation Protocol (SIP) Not only client terminals (telephones, video phones, NetMeeting, ) speak H.323, but also other system components: Gatekeeper: address translation (phone numbers to IP addresses), admission control and bandwidth management for multipoint connections, call authorization, call signal routing Gateway: integration with other voice networks Multipoint control unit (MCU): coordinates several terminals taking part in a conference Proxy: e.g. used to pass a firewall H.323 terminal can be workstations as well as more specalized end systems, e.g. IP phones The gateway enables an integration with existing systems like ISDN or older POTS (Plain Old Telephony System) Instead of H.323, also the simpler, Internet-oriented SIP can be used: Defined by IETF SIP long-term vision All telephone calls and video conference calls take place over the Internet People are identified by names or e-mail addresses, rather than by phone numbers You can reach the callee, no matter where the callee roams, no matter what IP device the callee is currently using SIP is an application layer signaling protocol that defines initiation, modification and termination of interactive multimedia communication sessions between multiple users Call setup Agree on media type and encoding Maps logical address identifier to current IP address Call management: add new media streams during call, change encoding during call, invite others, transfer and hold calls Bases upon HTTP concepts (message syntax, SIP URLs, responses, ) Page 7 Page 8
Setting up a Call to a known IP Address Example of a SIP Message µ Alice s SIP invite message indicates her port number & IP address. Indicates the encoding that Alice prefers to receive (PCM µ-law) Bob s 200 OK message indicates his port number, IP address & preferred encoding (GSM) SIP messages can be sent over TCP or UDP; here sent over RTP/UDP Default SIP port number is 506. INVITE sip:bob@domain.com SIP/2.0 Via: SIP/2.0/UDP 167.180.112.24 From: sip:alice@hereway.com To: sip:bob@domain.com Call-ID: a2e3a@pigeon.hereway.com Content-Type: application/sdp Content-Length: 885 c=in IP4 167.180.112.24 m=audio 38060 RTP/AVP 0 Notes: HTTP message syntax sdp = session description protocol Call-ID is unique for each call Here we don t know Bob s IP address. Intermediate SIP servers will be necessary. Alice sends and receives SIP messages using the SIP default port number 506. Alice specifies in Via: header that SIP client sends and receives SIP messages over UDP Page 9 Page 10 SIP Architecture SIP Example User Agent E.g. a VoIP phone SIP Registrar Users register their SIP and IP address with the registrar (like a DNS server) SIP Proxy Responsible for routing SIP messages to a callee Interprets, rewrites or translates a request message before forwarding it Location Holds information about the current location of a mobile user User Agent SIP Components Location Proxy Redirect Proxy Registrar Redirect Can pass back a reference to a temporary location/device of a mobile user Gateway PSTN Caller jim@umass.edu places a call to keith@upenn.edu (1) Jim sends INVITE message to umass SIP proxy (2) Proxy forwards request to upenn registrar server (3) upenn server returns redirect response, indicating that it should try keith@eurecom.fr (4) umass proxy sends INVITE to eurecom registrar (5) eurecom regristrar forwards INVITE to 197.87.54.21, which is running keith s SIP client SIP proxy umass.edu 1 8 SIP client 217.123.56.89 2 3 SIP registrar upenn.edu SIP registrar eurecom.fr (6-8) SIP response sent back (9) Messages sent directly between clients, e.g. with RTP 4 7 9 6 5 SIP client 197.87.54.21 Page 11 Page 12
Comparison with H.323 Internet Multimedia: Simplest Approach H.323 is a complete, vertically integrated suite of protocols for multimedia conferencing: signaling, registration, admission control, transport and codecs SIP is a single component. Can be combined with any other protocols and services. H.323 comes from the ITU (telephony) SIP comes from IETF: Borrows much of its concepts from HTTP. SIP has a Web flavor, whereas H.323 has a telephony flavor First: how can application level streaming be realized? Audio or video stored in files Files are transferred as HTTP object Received in entirety at client Then passed to player H.323 is complex SIP uses the KISS principle: Keep it simple and stupid But: both need some transfer protocols on transport and application layer to exchange the media stream Audio and video are not really streamed: Long delays until playout! Page 13 Page 14 Streaming from a Streaming Solution: RTSP Separation of web server and streaming Browser GETs metafile with audio/video server contact information Player contacts audio/video server using the metafile information streams audio/video to player This architecture allows for non-http protocol between server and media player Used here: e.g. RTSP Page 15 HTTP Does not focus on multimedia content No commands for fast forward, etc. Real-time Streaming Protocol RTSP Client/server application layer protocol For user to control display: rewind, fast forward, pause, resume, repositioning, etc What it doesn t do: Does not define how audio/video is encapsulated for streaming over network Does not restrict how streamed media is transported; it can be transported over UDP or TCP Does not specify how the media player buffers audio/video Page 16
RTSP Example Streaming Multimedia Scenario: Metafile communicated to web browser Browser launches player Player sets up an RTSP control connection and a data connection to streaming server What transport protocol to use for transferring the multimedia information? UDP sends at rate appropriate for client (oblivious to network congestion!) Often send rate = encoding rate = constant rate Buffering of received data and short playout delay (2-5 seconds) to compensate for network jitter Error recovery: if time permits TCP Send at maximum possible rate under TCP Data rate fluctuates due to TCP congestion control Larger playout delay: smooth TCP delivery rate HTTP/TCP passes more easily through firewalls Page 17 Page 18 Possible Transport Protocol Real-Time Protocol (RTP) A transport protocol for multimedia has to deal with e.g.: Data loss: IP packet lost due to network congestion (router buffer overflow), packet loss rates between 1% and 10% can be tolerated. Delay: IP packet can arrive too late for playout at receiver Delay is caused by processing/queueing in network as well as by the end-system Typical maximum tolerable delay: 400 ms What to do? Use UDP to avoid TCP congestion control (delays) for time-sensitive traffic Client-side buffering and adaptive playout delay to compensate for network delay side matches stream bandwidth to available client-to-server path bandwidth Chose among pre-encoded stream rates Dynamic server encoding rate Internet Multimedia is a bag of tricks! Provide a standardized transport protocol which supports such tricks: RTP RTSP still would have to use the unreliable UDP or the slow TCP better define a new transport protocol for combining speed with reliability: Real-Time Transport Protocol (RTP) RTP specifies a packet structure for packets carrying audio and video data RTP packet provides Payload type identification Packet sequence numbering Time-stamping RTP runs in the end systems RTP packets are encapsulated in UDP segments Interoperability: if two Internet phone applications run RTP, then they may be able to work together Page 19 Page 20
RTP runs on Top of UDP RTP Header RTP libraries provide a transport-layer interface that extend UDP: Port numbers, IP addresses Payload type identification Packet sequence numbering Time-stamping Transport Layer Page 21 Ver.: Version number of the RTP protocol in use P: packet size was padded to a multiple of 32 bit X: an extension header is used CC: indicates the number of sources M: User-specific mark. Can e.g. mark the beginning of a word on an audio channel. Contributing Source Identifier: used by mixers in the studio. The mixed flows are listed here. Page 22 RTP Header RTP Header Payload Type (7 bits) Indicates type of encoding currently being used. If the sender changes encoding in middle of transmission, it informs the receiver through this payload type field Payload type 0: PCM µ-law, 64 kbps Payload type 3, GSM, 13 kbps Payload type 26, Motion JPEG Payload type 31, H.261 Payload type 33, MPEG2 video Sequence Number (16 bits) Increments by one for each RTP packet sent, and may be used to detect packet loss and to restore packet sequence Timestamp field (32 bits long) Reflects the sampling instant of the first byte in the RTP data packet For audio, timestamp clock typically increments by one for each sampling period (for example, each 125 µsecs for a 8 KHz sampling clock) If application generates chunks of 160 encoded samples, then timestamp increases by 160 for each RTP packet when source is active. Timestamp clock continues to increase at constant rate when source is inactive. The timestamp gives the receiver the relative time (with respect to the first data) when to playout the data Synchronization Source Identifier field (32 bits long) Identifies the source of the RTP stream Each stream in a RTP session should have a distinct identifier Page 23 Page 24
Combining Streams RTP and QoS Often necessary: Synchronization of different media streams, e.g. in videoconferencing: audio and video are transmitted as two independent streams: synchronization has to take place when streams are played Guidelines for human perception of synchronization: Media combination Mode, Application Maximum time difference Video / Audio Lips synchronization ± 80 ms Audio / Audio Tightly coupled (e.g., stereo) ± 10 ms Loosely coupled (e.g., background music) ± 500 ms Audio / Image Tightly coupled (e.g., music with scores) ± 5 ms Loosely coupled (e.g., slide show) ± 500 ms Audio / Text Text annotation ± 240 ms RTP only adds some information to the UDP header needed for kind of reliability RTP does not provide any mechanism to ensure timely delivery of data or provide other quality of service guarantees RTP encapsulation is only seen at the end systems: it is not seen by intermediate routers. Routers providing best-effort service do not make any special effort to ensure that RTP packets arrive at the destination in a timely matter. Usage of (and reaction to) the information in the RTP header are left over to the application Synchronization specification is an essential part of the description of a multimedia object; RTP timestamp and M-flag can be used to pass sychronization information to the receiver Page 25 Page 26 Real-Time Control Protocol (RTCP) RTCP Works in conjunction with RTP Each participant in RTP session periodically transmits RTCP control packets to all other participants Each RTCP packet contains sender and/or receiver reports report statistics useful to application Statistics include number of packets sent, number of packets lost, interarrival jitter, etc. Feedback can be used to control performance Sender may modify its transmissions based on feedback To limit traffic, each participant reduces his RTCP traffic as the number of conference participants increases RTCP controls the data flow: Feedback to the sender about QoS on receiver side Data losses, delay and jitter are reported Note: RTCP does not provide corrective actions - this is left over to the application Sender Receiver Application Application RTP / RTCP RTP / RTCP UDP UDP IP IP RTP RTP RTP RTP RTP RTP RTCP RTCP RTCP RTCP Still a problem: quality of an IP transmission; how to improve QoS in the Internet? Page 27 Page 28