Multimedia Applications: Streaming Hamid R. Rabiee Mostafa Salehi, Fatemeh Dabiran, Hoda Ayatollahi Spring 2011
Outline What is Streaming Technology? Issues in Video Streaming over the Internet Bandwidth Loss rate Delay Streaming types Streaming stored multimedia Streaming live multimedia Streaming through a web server Streaming through a streaming server Real Time Streaming Protocol (RTSP) 2
What is Streaming Technology? Streaming multimedia allows the user to begin viewing video clips without completely downloading the entire file. After a brief initializing and buffering, the file begins to stream. Streaming process Advantages: At Sender: Partition video into packets At Receiver: After X-second (e.g. 5s to 15s) delay -> begin playback while video is still being downloaded At Receiver : Simultaneous delivery and playback (with short delay) Low delay before viewing Minimum storage requirements Disadvantages: Any data that is lost or arrived late is useless 3
Issues in Video Streaming over the Internet Internet only offers best-effort service => Provide no guarantees on 1. Bandwidth 2. Loss rates 3. Delay 1. Bandwidth Available bandwidth is dynamic Internet doesn t provide bandwidth reservation If transmit faster than available bandwidth Congestion occurs, packet loss, and severe drop in video quality If transmit slower than available bandwidth Sub-optimal video quality Goal: Match video sending bit rate with available bandwidth = Rate Control Video bit rate may be adapted by Varying the quantization Varying the frame rate Varying the spatial resolution Adding/dropping layers (for scalable coding) 4
Issues in Video Streaming over the Internet : Overcoming Bandwidth Problem Rate control schemes Source-based rate control Source is responsible for adapting the video transmission rate Applying in unicastvideo Probe-based approach: source probes the network for available bandwidth by adjusting sending rate in a way that Match the rate of pre-compressed video bitstream to the target rate Packet loss ratio p < threshold Pth If p< Pth then increase transmission rate If p> Pth then decrease transmission rate Fig.1 An architecture for the source-based rate control system Model-based approach: It is based on the TCP throughput model to determine video stream s sending rate 1.22 MTU λ = RTT ρ λ = Throughput TCP, MTU= MaximumTransmit Unit ρ = Packet loss ratio, RTT= RoundTrip Time Applying in multicast video Sender uses single channel to transport video (single-channel multicast) Only probe-based approach can be employed All receivers share one channel => efficient but cannot provide various receivers demand with different bandwidths of 5
Issues in Video Streaming over the Internet : Overcoming Bandwidth Problem (cont.) Receiver-based rate control Receiver is responsible for adapting the video receiving rate by adding/dropping channels Useful in multicasting scalable video; each video layer correspond to one channel in multicast tree Source Base layer Enh layer 1 Enh layer 2 Client with high-rate connection Client with medium-rate connection Client with low-rate connection Client with low-rate connection Applying approaches Probe-based Fig.2 Example of Receiver-Driven Layered Multicast No congestion detected => receiver probes for available bandwidth => joining a layer => receiving rate increased Congestion detected => receiver drops a layer => receiving rate reduced Model-based The same as in the sender-based rate control Hybrid rate control Receivers regulate the receiving rate of video streams by adding/dropping channels while sender adjusts the transmission rate of each channel based on the receivers feedback Examples: Destination set grouping, Layered multicast scheme 6
Issues in Video Streaming over the Internet : Overcoming Loss Rate Problem 2. Packet Loss Can damage pictures in a video sequence Goal: Overcome packet loss via error control Forward Error Correction (FEC) A video stream chopped into segments, each segment is packetized into k packet A block code applied to k packets => n-packet block generated By receiving any k packets in n-packet block, the segment can be recovered Advantages: Low delay (as compared to retransmits) Doesn t require feedback channel Works well (if appropriately matched to channel) Disadvantages: Overhead Channel loss characteristics are often unknown and time-varying FEC may be poorly matched to channel. Therefore, it is often ineffective (too little FEC) or inefficient (too much FEC) 7
Issues in Video Streaming over the Internet : Overcoming Loss Rate Problem FEC Mechanism 1 2 3 4 Original stream 1 1 2 2 3 3 4 Redundancy Internet 1 1 2 LOSS 3 4 Received stream 1 2 3 4 Reconstructed stream 8
Issues in Video Streaming over the Internet : Overcoming Loss Rate Problem (cont.) Original Stream Interleaving Mechanism 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 5 9 13 2 6 10 14 3 7 11 15 4 8 12 16 Interleaved Stream Received Stream 1 5 9 13 2 6 10 14 LOSS 4 8 12 16 1 2 4 5 6 8 9 10 12 13 14 16 Reconstructed Stream 9
Issues in Video Streaming over the Internet : Overcoming Loss Rate Problem (cont.) Retransmission When receiver detects the loss of packet N: if ( Tc + RTT + Ds < Td (N)) then send the request for packet N to sender { Tc = Current time, RTT= Round trip time, Ds= Slack term, Td(N)= time of scheduling packet N for display } Two approaches in video streaming with time-sensitive data Delay-constrained retransmission Only retransmit packets that can arrive in time Priority-based retransmission Retransmit important packets before unimportant packets Advantages: Only resends lost packets, efficiently uses bandwidth Easily adapts to changing channel conditions Disadvantages: Latency (round-trip-time (RTT)) Requires a back-channel (not applicable in broadcast, multicast, or point-topoint w/o back-channel) For some applications, usefulness decreases with increasing RTT 10
Issues in Video Streaming over the Internet : Overcoming Loss Rate Problem (cont.) Error concealment Performed by receiver when packet loss has already occurred Approaches to apply error concealment Spatial interpolation: Missing pixels are reconstructed using neighboring spatial information Temporal interpolation: loss data is reconstructed using data in previous frames Error-resilient video coding Using Multiple Description Coding (MDC): a raw video sequence is compressed into multiple streams Robust to packet loss: if receiver gets one description, it can reconstruct video with acceptable quality Enhanced video quality: if receiver gets multiple description, a better reconstruction can be made by combining them 11
Issues in Video Streaming over the Internet : Overcoming Delay Problem 3. Delay (Delay Jitter) Delay jitter: End-to-end delay may fluctuate from packet to packet Streaming video needs bounded end-to-end delay so that packets can arrivein time for decoding and playback Jitter: Variation in the end-to-end delay Receiver should decode and display frames at the same rate Each frame has its own specific playout time Playout time: Deadline by which it must be received/decoded/displayed If video packet arrives late (after its plaback deadline) => it is useless, can be considered as lost If subsequent frames depend on the late frame, then effects can propagate Goal: Overcome delay jitter => Playback buffer 12
Issues in Video Streaming over the Internet : Overcoming Delay Problem (cont.) Approach: Add playback buffer at decoder side to compensate for jitter Corresponds to adding an offset to the playback time of each packet If (packet delay < offset) then OK Buffer packet until its playback time If (packet delay > offset) then problem Playback buffer typically has duration of 5-15 secs Compensates for delay jitter and enables retransmission of lost packets 13
Issues in Video Streaming over the Internet : Overcoming Delay Problem (cont.) Client-side buffering: constant bit rate video transmission variable network delay (jitter) client video reception buffered video constant bit rate video playout at client client playout delay time 14
Issues in Video Streaming over the Internet : Overcoming Delay Problem (cont.) Jitter: Some Definitions Input: t 0,, t n Delay Jitter: Delay jitter J For every k: t 0 + kx - t k J X = (t n -t 0 ) / n Rate Jitter Rate Jitter A I k = t k -t k-1 For every k and j: I j -I k A Jitter and buffering delay versus jitter 15
Issues in Video Streaming over the Internet : Overcoming Delay Problem (cont.) Packet delivery, time-varying delay (jitter), and playout delay: 16
Issues in Video Streaming over the Internet : Overcoming Delay Problem (cont.) Effect of Different Playout Delays Playout delays: TD1 < TD2 < TD3 17
Streaming Stored Multimedia Streaming stored media: Audio/video file is stored in a server Users request audio/video file on demand. Audio/video is rendered within, say, 10 s after request. Interactivity (pause, repositioning, etc.) is allowed. Media player: removes jitter decompresses error correction graphical user interface with controls for interactivity Plug-ins may be used to embed the media player into the browser window. 18
Streaming Stored Multimedia (cont.) 1. video recorded 2. video sent network delay 3. video received, played out at client time streaming: at this time, client playing out early part of video, while server still sending later part of video 19 19
Streaming Live Multimedia Broadcasting the media, live over the Internet. The process involves a camera for the media, an encoder to digitize the content, a media publisher where the streams are made available to potential end-users a content delivery network to distribute and deliver the content. playback buffer is required (as with the streaming stored multimedia) playback can lag tens of seconds after transmission still have timing constraint Interactivity Features fast forward impossible rewind, pause possible Examples: Internet radio talk show Live sporting event 20
Interactive, Real-Time Multimedia End-end delay requirements: audio: < 150 msec good, < 400 msec OK includes application-level (packetization) and network delays higher delays noticeable, impair interactivity Session initialization how does callee advertise its IP address, port number, encoding algorithms? Applications: IP telephony, video conference, distributed interactive worlds 21
Interactive, Real-Time Multimedia (cont.) Applications: PC-2-PC phone instant messaging services are providing this PC-2-phone Dialpad Net2phone Videoconference with Webcams 22
Interactive, Real-Time Multimedia (cont.) Internet Phone, an example of interactive multimedia speaker s audio: alternating talk spurts, silent periods. 64 kbps during talk spurt pkts generated only during talk spurts 20 msec chunks at 8 Kbytes/sec: 160 bytes data application-layer header added to each chunk. Chunk+header encapsulated into UDP segment. application sends UDP segment into socket every 20 msec during talkspurt. 23
Interactive, Real-Time Multimedia (cont.) Internet Phone (cont.) Network loss: IP datagram lost due to network congestion (router buffer overflow) Delay: IP datagram arrives too late for playout at receiver => considered as lost delays: processing, queueing in network; end-system (sender, receiver) delays Typical maximum tolerable delay: 400 ms loss tolerance: depending on voice encoding, losses concealed, packet loss rates between 1% and 10% can be tolerated. 24
Interactive, Real-Time Multimedia (cont.) Internet Phone: Fixed Playback Delay Sender generates packets every 20 msec during talk spurt. First packet received at time r First playback schedule: begins at p Second playback schedule: begins at p packets packets generated loss packets received playout schedule p' - r playout schedule p - r time r p p' 25
Interactive, Real-Time Multimedia (cont.) Adaptive Playout Delay Goal: minimize playout delay, keeping late loss rate low Approach: adaptive playout delay adjustment: Estimate network delay, adjust playout delay at beginning of each talk spurt. Silent periods compressed and elongated. Chunks still played out every 20 msec during talk spurt. t r p r i i i d = timestamp of the ith packet = the time packet i is receivedby receiver i = the time packet i is played at receiver t i i = network delay for ith packet = estimate of average network delay after receiving ith packet Dynamic estimate of average delay at receiver (u is a fixed constant ): d i = ( 1 u) d i 1 + u( ri ti ) 26
Interactive, Real-Time Multimedia (cont.) Adaptive Playout Delay (cont.) Also useful to estimate the average deviation of the delay, v i : v i = ( 1 u) vi 1 + u ri ti di The estimates d i and v i are calculated for every received packet, although they are only used at the beginning of a talk spurt. For first packet in talk spurt, playout time is: p = t + d + i where K is a positive constant. Remaining packets in talkspurt are played out periodically i i Kv i 27
Streaming from a Web server Audio and video files stored in Web servers naïve approach browser requests file with HTTP request message Web server sends file in HTTP response message content-type header line indicates an audio/video encoding browser launches media player, and passes file to media player media player renders file Major drawback: media player interacts with server through intermediary of a Web browser 28
Streaming from a Web server (cont.) Alternative: set up connection between server and player Web browser requests and receives a meta file (a file describing the object) instead of receiving the file itself; Content-type header indicates specific audio/video application Browser launches media player and passes it the meta file Player sets up a TCP connection with server and sends HTTP request. Some concerns: Media player communicates over HTTP, which is not designed with pause, ff, rwnd commands May want to stream over UDP These concerns may not be real problems more later 29
Streaming from a streaming server This architecture allows for non-http protocol between server and media player Can also use UDP instead of TCP. 30
Streaming Multimedia: UDP or TCP? UDP server sends at rate appropriate for client (oblivious to network congestion!) often send rate = encoding rate = constant rate then, fill rate = constant rate - packet loss short playback delay (2-5 seconds) to compensate for network delay jitter error recover: time permitting TCP send at maximum possible rate under TCP fill rate fluctuates due to TCP congestion control larger playback delay: smooth TCP delivery rate HTTP/TCP passes more easily through firewalls 31
Real Time Streaming Protocol (RTSP) Asignaling and control protocol for multimedia streaming ininternet To control the data delivery in amultimedia streaming session by conveying VCR-stylecommands (like play, mute) between communicating partners It is typically used in conjunction with RTP which conveys the actual multimedia data. Itis arequest-response protocol similar tohttp,but stateless Serverand player use RTSPtosend control info toeach other Foruser to control display: rewind, fastforward, pause, resume, repositioning, etc What it doesn t do: Does not define how audio/video is encapsulated for streaming over network Does not restrict how streamed media is transported; it can be transported over UDP or TCP Does not specify how the media player buffers audio/video 32
RTSP Request Has the form: Request-Method SP Request-URL SP RTSP-Version <CR><LF> (generic-header request-header entity-header <CR><LF>) <CR><LF> [message body] Request-Method is: DESCRIBE -retrieves the description of a media object from a server; SETUP - prepares the streaming session; PLAY - starts the delivery of multimedia data; PAUSE -streaming is paused, session is still active, but no packet is sent; TEARDOWN -session is terminated and resources are freed. 33
RTSP Request (cont.) Request-header can have the following fields (selection): Accept : MIME types of resources accepted by client Accept-Encoding : encoding accepted by client Accept-Language : language accepted by client Authorization : user-agent wishes to authenticate itself with a server From: Referer: the URL of document referingthis URL User-Agent : client software 34
RTSP Response Has the form: Http-Version SP Status-Code SP Reason-Phrase<CR><LF> (generic-header response-header entity-header <CR><LF>) <CR><LF> [message body] Response-header has the following fields (selection): Location : redirect the client to a location other than Request-URL for completion of the request Retry-After : indicate to client how long the service is expected to be unavailable Server : information about software used by the server to handle the request 35
RTSP: out of band control FTP uses an out-of-band control RTSP messages are also sent out-of-band: channel: A file is transferred over one channel. Control information (directory changes, file deletion, file renaming, etc.) is sent over a separate TCP connection. The out-of-band and in-band channels use different port numbers. The RTSP control messages use different port numbers than the media stream, and are therefore sent out-of-band. The media stream, whose packet structure is not defined by RTSP, is considered inband. If the RTSP messages were to use the same port numbers as the media stream, then RTSP messages would be said to be interleaved with the media stream. 36
RTSP initiates and controls delivery Client obtains a description of the multimedia presentation, which can consist of several media streams. Web browser HTTP GET presentation desc. SETUP Web server The browser invokes media player (helper application) based on the content type of the presentation description. Presentation description includes references to media media player PLAY media stream PAUSE media server streams, using the URL method rtsp:// Player sends RTSP SETUP request; server sends RTSP SETUP response. Player sends RTSP PLAY request; server sends RTSP TEARDOWN PLAY response. Media server pumps media stream. client server Player sends RTSP PAUSE request; server sends RTSP PAUSE response. Player sends RTSP TEARDOWN request; server sends RTSP TEARDOWN response.
Meta file example <title>twister</title> <session> </session> <group language=en lipsync> <switch> <track type=audio e="pcmu/8000/1" src = "rtsp://audio.example.com/twister/audio.en/lofi"> <track type=audio </switch> e="dvi4/16000/2" pt="90 DVI4/8000/1" src="rtsp://audio.example.com/twister/audio.en/hifi"> <track type="video/jpeg" </group> src="rtsp://video.example.com/twister/video"> 38
RTSP session Each RTSP has a session identifier, which is chosen by the server. The client initiates the session with the SETUP request, and the server responds to the request with an identifier. RTSP port number is 554. RTSP can be sent over UDP or TCP. Each RTSP message can be sent over a separate TCP connection. The client repeats the session identifier for each request, until the client closes the session with the TEARDOWN request. 39
RTSP: exchange example C: SETUP rtsp://audio.example.com/twister/audio RTSP/1.0 Transport: rtp/udp; compression; port=3056; mode=play S: RTSP/1.0 200 1 OK Session 4231 C: PLAY rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 npt = Normal play time Range: npt=0- C: PAUSE rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=37 C: TEARDOWN rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 S: 200 3 OK 40
RTSP: streaming caching Caching of RTSP response messages Proxy cache may hold only segments of makes little sense. a given media stream. But desirable to cache media streams Proxy cache may start serving a closer to client. client from its local cache, and Much of HTTP/1.1 cache control has then have to connect to origin been adopted by RTSP. server and fill missing material, Cache control headers can be put in RTSP SETUP requests hopefully without introducing gaps at client. and responses: When original server is sending a If-modified-since:, Expires:, Via:, Cache-Control: stream to client, and stream passes through a proxy, proxy can use TCP to obtain the stream; but proxy still sends RTSP control messages to origin server. 41
Next Session Multimedia Protocols 42