Multimedia networking: outline

Computer Network Architectures and Multimedia Guy Leduc Chapter 4 Multimedia Applications & Transport Sections 9.1 to 9.4 from Computer Networking: A Top Down Approach, 7 th edition. Jim Kurose, Keith Ross Addison-Wesley, April 2016. Also 7.4.2 and 7.4.7 from Computer Networks - 4th edition Andrew S. Tanenbaum Prentice-Hall International, 2003 4: Multimedia App. & Transp. 4-1 Multimedia networking: outline 4.1 multimedia networking applications 4.2 streaming stored video 4.3 voice-over-ip 4.4 protocols for real-time conversational applications 4: Multimedia App. & Transp. 4-2 1

Multimedia: audio! PCM (Pulse Code Modulation): analog audio signal sampled at constant rate " telephone: 8,000 samples/sec " CD music: 44,100 samples/sec! each sample quantized, i.e., rounded " each quantized value represented by bits, " e.g., rounded to one of 2 8 =256 values " 8 bits/sample! receiver converts bits back to analog signal: " some quality reduction audio signal amplitude quantization error sampling rate (N sample/sec) quantized value of analog value analog signal time 4: Multimedia App. & Transp. 4-3 Multimedia: audio Examples:! Telephony:! 8,000 samples/sec, 8 bits/sample: 64 kbps! CD music:! 44,100 samples/sec, 16 bits/sample: 705.6 kbps! Stereo: 1.411 Mbps Other example rates! MP3: 96, 128, 160 kbps! Internet telephony: 5.3 kbps and up audio signal amplitude quantization error sampling rate (N sample/sec) quantized value of analog value analog signal time 4: Multimedia App. & Transp. 4-4 2

More on Audio Compression MP3 (MPEG 1 audio layer 3) takes masking effects into account and does not encode masked signals. Can compress stereo CD down to 96-128 kbps. From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-5 Multimedia: video video: sequence of images displayed at constant rate e.g. 25 images/sec digital image: array of pixels each pixel represented by bits coding: use redundancy within and between images to decrease # bits used to encode image spatial (within image) temporal (from one image to next)...... frame i temporal coding example: instead of sending complete frame at i+1, send only differences from frame i spatial coding example: instead of sending N values of same color (all purple), send only two values: color value (purple) and number of repeated values (N) frame i+1 4: Multimedia App. & Transp. 4-6 3

Multimedia: video! CBR: (constant bit rate): video encoding rate fixed! VBR: (variable bit rate): video encoding rate changes as amount of spatial, temporal coding changes! examples: " MPEG 1 (CD-ROM) 1.5 Mbps " MPEG2 (DVD) 3-6 Mbps " MPEG4 (often used in Internet, < 1 Mbps) spatial coding example: instead of sending N values of same color (all purple), send only two values: color value (purple) and number of repeated values (N)...... frame i temporal coding example: instead of sending complete frame at i+1, send only differences from frame i frame i+1 4: Multimedia App. & Transp. 4-7 Video - Digital Systems Consider a rectangular 4:3 grid of pixels, such as VGA: 640 x 480 XGA: 1024 x 768 Pixel = 8 bits for each of the RGB colours 25 frames per sec With XGA : 24 bits/pixel x 1024 x 768 x 25 frames/sec = 472 Mbps! Needs compression! From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-8 4

Data compression Encoding/decoding schemes Video on Demand (VoD) Encoding can be slow (done once) Decoding must be fast (done many times) Asymmetrical schemes Real-time multimedia (e.g. videoconference) Symmetrical schemes Lossy compression Encode/decode is not neutral When acceptable, leads to better compression ratios Two main compression schemes: Entropy encoding (lossless) Source encoding (lossy) From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-9 Entropy encoding Lossless Three typical examples: Run-length encoding repeated symbols are encoded as Special symbol + number of occurrences Statistical encoding short codes for frequent symbols (Huffman encoding) Look up table e.g. CLUT (Colour Look Up Table) define the table of the colours actually used send table index instead of a 24-bit colour value From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-10 5

Lossy Source encoding Three main examples: Differential encoding sequence of values are encoded by representing the differences from the previous values makes sense if differences are encoded with less bits lossy when there are large jumps between two values and a fixed number of bits per difference lossless if variable-length encoding is used Transformation e.g. Fourier or DCT Transform lossy since only the first amplitudes are sent Variant of Look Up Table with approximations to closest value From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-11 JPEG Joint Photographic Experts Group ISO/IEC and ITU standard for compressing still pictures Compression ratio 20:1 is typical Roughly symmetrical scheme (decoding as long as encoding) Lossy sequential mode: 6 steps Block preparation Discrete Cosine transform Quantization Differential quantization Run-length encoding Statistical Output encoding From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-12 6

Block preparation Discrete Cosine transform JPEG - Step 1 Quantization Differential quantization Run-length encoding Statistical Output encoding Step 1: block preparation Translate RGB into luminance (Y) and 2 chrominance (I,Q) values gives better compression we get 3 matrices of pixels Average square blocks of 4 pixels for I and Q lossy but unnoticeable Subtract 128 from each element (0 is middle) Divide up frame into 8x8 blocks From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-13 Block preparation Discrete Cosine transform JPEG - Step 2 Quantization Differential quantization Run-length encoding Statistical Output encoding Step 2: DCT (Discrete Cosine Transformation) to each block Sort of 2 dimensional Discrete Fourier Transform, but more compact Advantage: most of the spectral power in the first few terms Output: block of 8x8 elements (coefficient of DCT) Slightly lossy in practice (round-off errors) From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-14 7

JPEG - Step 3 Block preparation Discrete Cosine transform Quantization Differential quantization Run-length encoding Statistical Output encoding Step 3: Quantization Apply sort of low pass filter to coefficients (lossy) From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-15 JPEG - Steps 4, 5 and 6 Step 4: Differential quantization Replace upper-left (DC) coefficient by its difference with corresponding element of previous block Step 5: Run-length encoding Applied to a zig-zag scanning pattern Step 6: Statistical output encoding (Huffman) From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-16 8

MPEG Motion Picture Experts Group - ISO standard Audio and video MPEG-1 Video-recorder quality (CD-ROM) 1.2 Mbps output MPEG-2 Broadcast quality 4-6 Mbps output is typical but higher for HDTV MPEG-4 Medium-resolution videoconferencing with low frame rate 10 frames/sec From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-17 MPEG-1 Audio signal Clock Video signal Audio encoder Video encoder System multiplexer MPEG-1 output Audio and video encoders work independently Timestamps included in both flows for synchronization at receiver Audio compression (MP3) Also, exploitation of redundancy in the 2 channels of a stereo stream From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-18 9

MPEG-1 - Video compression Exploit spatial and temporal redundancies Spatial redundancy: like JPEG But adds temporal redundancy Many common parts in the following three consecutive frames! From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-19 MPEG-1 - Video compression (2) Temporal redundancy: four kinds of frames: I, P, B, D I frames (Intracoded) self-contained JPEG-encoded still pictures should appear periodically in the output (initial synch, resynch on error, fast forward or rewind) P frames (Predictive) block-by-block difference with the previous frame search for a macroblock (Y,I,Q) in previous frame which is equal or slightly different encode the offset in position and difference B frames (Bidirectional): same as P but search also in next I or P frame D frames (DC-coded): block averages for fast forward (low resolution) Example of part of an MPEG sequence I B B P B B I From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-20 10

MPEG-2 Similar to MPEG-1 Better quality (10 x 10 DCT coefficients instead of 8 x 8) Several resolution levels (lowest one is comparable to MPEG-1) Several profiles (e.g. no B frames to simplify encoding) Usually 3-4 Mbps, but can go up to 100 Mbps (HDTV) From From Computer Computer Networks, Networking, by Tanenbaum by Kurose&Ross Prentice Hall 4: Multimedia App. & Transp. 4-21 Multimedia networking: 3 application types streaming, stored audio, video streaming: can begin playout before downloading entire file stored (at server): can transmit faster than audio/video will be rendered (implies storing/buffering at client) e.g., YouTube, Netflix, Hulu conversational voice/video over IP interactive nature of human-to-human conversation limits delay tolerance e.g., Skype streaming live audio, video e.g., live sporting event 4: Multimedia App. & Transp. 4-22 11

MM Networking Applications Fundamental characteristics: typically delay sensitive end-to-end delay delay jitter loss tolerant: infrequent losses cause minor glitches antithesis of data, which are loss intolerant but delay tolerant Jitter is the variability of packet delays within the same packet stream QoS (Quality of Service) refers to performance metrics such as delay, bandwidth, jitter and loss 4: Multimedia App. & Transp. 4-23 Multimedia Over Today s Internet TCP/UDP/IP: best-effort service no guarantees on delay, bandwidth, jitter, loss (if UDP)??????? But you said multimedia apps require QoS and level of performance to be? effective!??? Today s Internet multimedia applications use application-level techniques to mitigate (as best possible) effects of delay, loss 4: Multimedia App. & Transp. 4-24 12

Chapter 4: outline 4.1 multimedia networking applications 4.2 streaming stored video 4.3 voice-over-ip 4.4 protocols for real-time conversational applications 4: Multimedia App. & Transp. 4-25 Internet multimedia: simplest approach Media player jitter removal decompression error concealment graphical user interface with controls for interactivity audio or video stored in file files transferred as HTTP object received in entirety at client then passed to player audio, video not streamed in this scenario: no, pipelining, long delays until playout! 4: Multimedia App. & Transp. 4-26 13

Internet multimedia: streaming approach browser GETs metafile browser launches player, passing metafile player contacts server server streams audio/video to player 4: Multimedia App. & Transp. 4-27 Streaming from a streaming server allows for non-http protocol between server and media player UDP or TCP for step (3), more shortly 4: Multimedia App. & Transp. 4-28 14

Cumulative data Streaming Multimedia: client rate(s) 1.5 Mbps encoding 28.8 Kbps encoding Q: how to handle different client receive rate capabilities? A: server stores, transmits multiple copies of video, encoded at different rates (info found in meta-file) 4: Multimedia App. & Transp. 4-29 Streaming stored video: 1. video Recorded (e.g., 30 frames/ sec) 2. video sent network delay (fixed in this example) 3. video received, played out at client (30 frames/sec) time streaming: at this time, client playing out early part of video, while server still sending later part of video 4: Multimedia App. & Transp. 4-30 15

Streaming stored video: challenges! continuous playout constraint: once client playout begins, playback must match original timing " but network delays are variable (jitter), so will need client-side buffer to match playout requirements! other challenges: " client interactivity: pause, fast-forward, rewind, jump through video " video packets may be lost, retransmitted 4: Multimedia App. & Transp. 4-31 Streaming stored video: revisited Cumulative data constant bit rate video transmission variable network delay client video reception buffered video constant bit rate video playout at client client playout delay time client-side buffering and playout delay: compensate for network-added delay, delay jitter 4: Multimedia App. & Transp. 4-32 16

Client-side buffering, playout variable fill rate, x(t) buffer fill level, Q(t) playout rate, e.g., CBR r video server client application buffer, size B client CBR = Constant Bit Rate VBR = Variable Bit Rate 4: Multimedia App. & Transp. 4-33 Client-side buffering, playout variable fill rate, x(t) buffer fill level, Q(t) playout rate, e.g., CBR r video server client application buffer, size B client 1. initial fill of buffer until playout begins at t p 2. playout begins at t p, 3. buffer fill level varies over time as fill rate x(t) varies and playout rate r is constant 4: Multimedia App. & Transp. 4-34 17

Client-side buffering, playout variable fill rate, x(t) buffer fill level, Q(t) playout rate, e.g., CBR r video server client application buffer, size B x < r: buffer may empty, causing freezing of video playout until buffer again fills initial playout delay tradeoff: buffer starvation less likely with larger delay, but larger delay until user begins watching 4: Multimedia App. & Transp. 4-35 Streaming multimedia: UDP server sends at rate appropriate for client often: send rate = encoding rate = constant rate transmission rate can be unaware of network congestion level short playout delay (2-5 seconds) to remove network jitter error recovery: application-level, time permitting encapsulation of audio/video chunks in RTP (Real-Time Transport Protocol, RFC 3550) and then in UDP see later for details needs a control connection in parallel to pause, resume reposition, etc: Real-Time Streaming Protocol (RTSP, RFC 2326) issue: UDP may not go through firewalls 4: Multimedia App. & Transp. 4-36 18

User Control of Streaming Media: RTSP HTTP does not target multimedia content no commands for fast forward, etc. RTSP Real-Time Streaming Protocol client-server application layer protocol user control: rewind, fast forward, pause, resume, repositioning, etc. What it doesn t do: doesn t define how audio/video is encapsulated for streaming over network doesn t restrict how streamed media is transported (UDP or TCP possible) doesn t specify how media player buffers audio/video 4: Multimedia App. & Transp. 4-37 RTSP: out-of-band control FTP uses an out-ofband control channel: file transferred over one TCP connection control info (directory changes, file deletion, rename) sent over separate TCP connection out-of-band, inband channels use different port numbers RTSP messages also sent out-of-band: RTSP control messages use different port numbers than media stream: out-of-band port 554 media stream is considered in-band 4: Multimedia App. & Transp. 4-38 19

RTSP example Scenario: metafile communicated to web browser (1) browser launches player (2) player sets up an RTSP control connection, data connection to streaming server (3) 4: Multimedia App. & Transp. 4-39 Metafile Example <title>twister</title> <session> <group language=en lipsync> <switch> <track type=audio e="pcmu/8000/1" src = "rtsp://audio.example.com/twister/audio.en/lofi"> <track type=audio e="dvi4/16000/2" pt="90 DVI4/8000/1" src="rtsp://audio.example.com/twister/audio.en/hifi"> </switch> <track type="video/jpeg" src="rtsp://video.example.com/twister/video"> </group> </session> 4: Multimedia App. & Transp. 4-40 20

RTSP Operation 4: Multimedia App. & Transp. 4-41 RTSP exchange example C: SETUP rtsp://audio.example.com/twister/audio RTSP/1.0 Transport: rtp/udp; compression; port=3056; mode=play S: RTSP/1.0 200 1 OK Session 4231 C: PLAY rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=0- C: PAUSE rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=37 C: TEARDOWN rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 S: 200 3 OK 4: Multimedia App. & Transp. 4-42 21

Streaming multimedia: TCP multimedia file retrieved via HTTP GET send at maximum possible rate under TCP variable rate, x(t) video file TCP send buffer server issue: fill rate fluctuates much due to TCP congestion control and TCP retransmissions (inorder delivery) larger playout delay: smooth TCP delivery rate HTTP/TCP passes more easily through firewalls TCP receive buffer client application playout buffer 4: Multimedia App. & Transp. 4-43 Video Streaming and CDNs: context " video traffic: major consumer of Internet bandwidth Netflix, YouTube: 37%, 16% of downstream residential ISP traffic ~1B YouTube users, ~75M Netflix users " challenge: scale - how to reach ~1B users? single mega-video server won t work (why?) " challenge: heterogeneity " different users have different capabilities (e.g., wired versus mobile; bandwidth rich versus bandwidth poor) " solution: distributed, application-level infrastructure 2-44 Application Layer 22

Streaming multimedia: DASH DASH: Dynamic, Adaptive Streaming over HTTP server: divides video file into multiple chunks each chunk stored, encoded at different rates manifest file: provides URLs for different chunks client: periodically measures server-to-client bandwidth consulting manifest, requests one chunk at a time chooses maximum coding rate sustainable given current bandwidth can choose different coding rates at different points in time (depending on available bandwidth at time) 4: Multimedia App. & Transp. 4-45 Streaming multimedia: DASH DASH: Dynamic, Adaptive Streaming over HTTP intelligence at client: client determines when to request chunk (so that buffer starvation, or overflow does not occur) what encoding rate to request (higher quality when more bandwidth available) where to request chunk (can request from URL server that is close to client or has high available bandwidth) 4: Multimedia App. & Transp. 4-46 23

Content distribution networks challenge: how to stream content (selected from millions of videos) to hundreds of thousands of simultaneous users? option 1: single, large mega-server single point of failure point of network congestion long path to distant clients multiple copies of video sent over outgoing link quite simply: this solution doesn t scale 4: Multimedia App. & Transp. 4-47 Content distribution networks challenge: how to stream content (selected from millions of videos) to hundreds of thousands of simultaneous users? option 2: store/serve multiple copies of videos at multiple geographically distributed sites (CDN) enter deep: push CDN servers deep into many access networks close to users used by Akamai, 1700 locations bring home: smaller number (10 s) of larger clusters in POPs near (but not within) access networks used by Limelight Google uses both, in addition to its mega data centers responsible for serving dynamic content 4: Multimedia App. & Transp. 4-48 24

Content Distribution Networks (CDNs) " CDN: stores copies of content at CDN nodes e.g. Netflix stores copies of MadMen " subscriber requests content from CDN directed to nearby copy, retrieves content may choose different copy if network path congested where s Madmen? manifest file 2-49 Application Layer Content Distribution Networks (CDNs) over the top Internet host-host communication as a service OTT challenges: coping with a congested Internet from which CDN node to retrieve content? viewer behavior in presence of congestion? what content to place in which CDN node? 25

CDN: simple content access scenario Bob (client) requests video http://video.netcinema.com/6y7b23v actually stored in a KingCDN content distribution server 1. Bob gets URL for video http:// video.netcinema.com/6y7b23v from netcinema.com web page 1 2 6. request video from 5 KingCDN server, streamed via HTTP netcinema.com 3. netcinema s DNS returns a1105.kingcdn.com 3 2. resolve video.netcinema.com via Bob s local DNS that relays to netcinema s authoritative DNS server 4 4&5. Resolve a1105.kingcdn.com via KingCDN s authoritative DNS, which returns IP address of KingCDN distribution server with video netcinema authoritative DNS KingCDN content distribution server KingCDN authoritative DNS 4: Multimedia App. & Transp. 4-51 CDN cluster selection strategy challenge: how does CDN DNS select good CDN node to stream to client CDN learns the IP address of the client s local DNS via the client s DNS lookup CDN can then implement a selection strategy to dynamically direct clients to a suitable server cluster or data center Possible strategies: 1. pick IP address of CDN node geographically closest to client 2. pick IP address of CDN node with shortest delay (or min # hops) to client (CDN nodes periodically ping access ISPs, reporting results to CDN DNS) 3. always pick the same IP address, but make sure this IP address is an IP anycast address associated with all CDN nodes see next slide alternative: let client decide! give client a list of several CDN servers client pings servers, picks best Netflix approach 4: Multimedia App. & Transp. 4-52 26

IP anycast An IP anycast address is an IP unicast address, but this address is assigned to multiple devices that are geographically distributed Like a private address, but an anycast address is routable, i.e. reachable from anywhere in the Internet When a source sends a packet to an IP anycast address, the packet reaches one of the devices associated with this address The choice is made by the network, not by the source, because it results from a decision taken at the interdomain routing protocol (BGP) level In other words, BGP decides by applying its usual routing rules based on preferences, AS-Path length, Hot Potato criterion, etc. So, don t confuse anycast and multicast! An anycast address is a unicast address And nothing can tell you if a unicast address is actually anycast or not! B Source 1 CDN cluster (IP anycast) A C D Source 2 CDN cluster (same IP anycast) 4: Multimedia App. & Transp. 4-53 IP anycast in a CDN Role of BGP The CDN assigns the same IP address range to each of its clusters Therefore, it s an anycast address The CDN relies on standard BGP to advertise this IP address range from each of the different cluster locations When a BGP router receives multiple route advertisements for this same IP address range, it treats them as providing several paths to the same physical location and picks the best 1. BGP: AS-PATH A to CDN prefix 2. BGP: AS-PATH BA to CDN prefix 3. BGP: AS-PATH C to CDN prefix 1 B 2 In D, BGP selects the best path between BA and C to reach the CDN prefix CDN cluster (IP prefix) A 1 C 3 D Client A, B, C, D are Autonomous Systems (AS) CDN cluster (same IP prefix) 4: Multimedia App. & Transp. 4-54 27

Case study: Netflix 30% downstream US traffic in 2011 owns very little infrastructure, uses 3 rd party services: own registration, payment servers Amazon (3 rd party) cloud services: Netflix uploads studio master to Amazon cloud create multiple version of movie (different encodings) in cloud upload versions from cloud to CDNs Cloud hosts Netflix web pages for user browsing three 3 rd party CDNs host/stream Netflix content: Akamai, Limelight, Level-3 4: Multimedia App. & Transp. 4-55 Case study: Netflix Amazon cloud upload copies of multiple versions of video to CDNs Akamai CDN Netflix registration, accounting servers 1 1. Bob manages Netflix account 2. Bob browses Netflix video 2 3. Manifest file returned for requested video 3 4. DASH streaming Limelight CDN Level-3 CDN 4: Multimedia App. & Transp. 4-56 28

Chapter 4: outline 4.1 multimedia networking applications 4.2 streaming stored video 4.3 voice-over-ip 4.4 protocols for real-time conversational applications 4: Multimedia App. & Transp. 4-57 Voice-over-IP (VoIP) VoIP end-end-delay requirement: needed to maintain conversational aspect higher delays noticeable, impair interactivity < 150 msec: good > 400 msec: bad includes application-level (packetization,playout), network delays session initialization: how does callee advertise IP address, port number, encoding algorithms? value-added services: call forwarding, screening, recording emergency services: 112 (Europe), 911 (North America) 4: Multimedia App. & Transp. 4-58 29

VoIP characteristics speaker s audio: alternating talk spurts, silent periods. 64 kbps during talk spurt pkts generated only during talk spurts 20 msec chunks at 8 Kbytes/sec: 160 bytes of data so, 20 msec of packetization delay application-layer header added to each chunk chunk+header encapsulated into UDP (or TCP) segment application sends segment into socket every 20 msec during talkspurt 4: Multimedia App. & Transp. 4-59 VoIP: packet loss, delay network loss: IP datagram lost due to network congestion (router buffer overflow) delay loss: IP datagram arrives too late for playout at receiver delays: processing, queueing in network; end-system (sender, receiver) delays typical maximum tolerable delay: 400 ms loss tolerance: depending on voice encoding, loss concealment, packet loss rates between 1% and 10% can be tolerated 4: Multimedia App. & Transp. 4-60 30

Delay jitter Cumulative data constant bit rate transmission variable network delay (jitter) client reception buffered data constant bit rate playout at client client playout delay time end-to-end delays of two consecutive packets: difference can be more or less than 20 msec (transmission time difference) 4: Multimedia App. & Transp. 4-61 VoIP: fixed playout delay receiver attempts to playout each chunk exactly q msecs after chunk was generated chunk has timestamp t: play out chunk at t+q chunk arrives after t+q: data arrives too late for playout, data lost tradeoff in choosing q: large q: less packet loss small q: better interactive experience 4: Multimedia App. & Transp. 4-62 31

VoIP: fixed playout delay sender generates packets every 20 msec during talk spurt first packet received at time r first playout schedule: begins at p second playout schedule: begins at p p a c k e t s p a c k e t s g e n e r a t e d l o s s p a c k e t s r e c e i v e d playout schedule p - r playout schedule p - r t i m e r p p ' 4: Multimedia App. & Transp. 4-63 Adaptive playout delay (1) goal: low playout delay, low late loss rate approach: adaptive playout delay adjustment: estimate network delay, adjust playout delay at beginning of each talk spurt silent periods compressed and elongated chunks still played out every 20 msec during talk spurt adaptively estimate packet delay: (EWMA - exponentially weighted moving average, recall TCP RTT estimate): d i = (1-α)d i-1 + α (r i t i ) delay estimate after ith packet small constant, e.g. 0.1 time received - time sent (timestamp) measured delay of ith packet 4: Multimedia App. & Transp. 4-64 32

Adaptive playout delay (2) also useful to estimate average deviation of delay, v i : v i = (1-β)v i-1 + β r i t i d i estimates d i, v i calculated for every received packet, but used only at start of talk spurt for first packet in talk spurt, playout time is: playout-time i = t i + d i + Kv i remaining packets in talkspurt are played out periodically Q: does it require clock synchronization? 4: Multimedia App. & Transp. 4-65 Adaptive playout delay (3) Q: How does receiver determine whether packet is first in a talk spurt? if no loss, receiver looks at successive timestamps difference of successive stamps > 20 msec -> talk spurt begins with loss possible, receiver must look at both time stamps and sequence numbers difference of successive stamps > 20 msec and sequence numbers without gaps -> talk spurt begins 4: Multimedia App. & Transp. 4-66 33

VoIP: recovery from packet loss (1) Challenge: recover from packet loss given small tolerable delay between original transmission and playout each ACK/NAK takes ~ one RTT alternative: Forward Error Correction (FEC) send enough bits to allow recovery without retransmission (recall two-dimensional parity) simple FEC for every group of n chunks, create redundant chunk by exclusive OR-ing n original chunks send n+1 chunks, increasing throughput by factor 1/n can reconstruct original n chunks if at most one lost chunk from n+1 chunks, with playout delay called erasure code 4: Multimedia App. & Transp. 4-67 VoIP: recovery from packet loss (2) increasing throughput: by factor 1/n increasing playout delay: need enough time to receive all n+1 packets tradeoff: increase n, less bandwidth waste increase n, longer playout delay increase n, higher probability that 2 or more chunks will be lost 4: Multimedia App. & Transp. 4-68 34

VoIP: recovery from packet loss (3) FEC: Reed-Solomon (RS) scheme RS is a more sophisticated error correcting code, which can be used as erasure code An (n,k) RS code encodes k source packets into n > k packets Systematic code: the n transmitted packets contain verbatim copies of the k source packets + n-k new packets no decoding if no source packet loss! Optimal code: Original k packets can be recovered provided that any k packets among n are received Linear code: coding/decoding represented by matrix operations: x is the vector of k source packets G is a n x k matrix y is the vector of n transmitted packets y = G x Decoding: y vector of any k received packets G is the k x k submatrix of G with rows corresponding to these packets x = G -1 y 4: Multimedia App. & Transp. 4-69 VoIP: recovery from packet loss (4) 2nd FEC scheme # piggyback lower quality stream # send lower resolution audio stream as redundant information # e.g., nominal stream PCM at 64 kbps and redundant stream GSM at 13 kbps. # non-consecutive loss, receiver can conceal the loss # generalization: can also append (n-1)st and (n-2)nd low-bit rate chunks 4: Multimedia App. & Transp. 4-70 35

VoIP: recovery from packet loss (5) Interleaving to conceal loss audio chunks divided into smaller units for example, four 5 msec units per 20 ms audio chunk packet contains small units from different chunks if packet lost, still have most of every chunk no redundancy overhead, but increases playout delay 4: Multimedia App. & Transp. 4-71 Voice-over-IP: Skype proprietary applicationlayer protocol (inferred via reverse engineering) encrypted msgs P2P components: " clients: skype peers connect directly to each other for VoIP call " super nodes (SN): skype peers with special functions " overlay network: among SNs to locate SCs " login server Skype login server Skype clients (SC) supernode (SN) supernode overlay network 4: Multimedia App. & Transp. 4-72 36

P2P voice-over-ip: skype skype client operation: 1. joins skype network by contacting SN (IP address cached) using TCP 2. logs-in (username, password) to centralized skype login server Skype login server 3. obtains IP address for callee from SN, SN overlay " or client buddy list 4. initiate call directly to callee 4: Multimedia App. & Transp. 4-73 Skype: peers as relays Problem: both Alice, Bob are behind NATs NAT prevents outside peer from initiating connection to insider peer inside peer can initiate connection to outside relay solution: Alice, Bob maintain open connection to their SNs " Alice signals her SN to connect to Bob " Alice s SN connects to Bob s SN " Bob s SN connects to Bob over open connection Bob initially initiated to his SN 4: Multimedia App. & Transp. 4-74 37

Chapter 4: outline 4.1 multimedia networking applications 4.2 streaming stored video 4.3 voice-over-ip 4.4 protocols for real-time conversational applications: RTP/RTCP, SIP 4: Multimedia App. & Transp. 4-75 Real-Time Protocol (RTP) RTP specifies packet structure for packets carrying audio, video data RFC 3550 RTP packet provides payload type identification packet sequence numbering time stamping RTP runs in end systems RTP packets encapsulated in UDP segments interoperability: if two Internet phone applications run RTP, then they may be able to work together 4: Multimedia App. & Transp. 4-76 38

RTP runs on top of UDP RTP libraries provide transport-layer interface that extends UDP: port numbers, IP addresses payload type identification packet sequence numbering time-stamping 4: Multimedia App. & Transp. 4-77 RTP Example consider sending 64 kbps PCM-encoded voice over RTP application collects encoded data in chunks, e.g., every 20 msec = 160 bytes in a chunk audio chunk + RTP header form RTP packet, which is encapsulated in UDP segment RTP header indicates type of audio encoding in each packet sender can change encoding during conference RTP header also contains sequence numbers, timestamps 4: Multimedia App. & Transp. 4-78 39

RTP and QoS RTP does not provide any mechanism to ensure timely data delivery or other QoS guarantees RTP encapsulation is only seen at end systems (not by intermediate routers) routers provide best-effort service, making no special effort to ensure that RTP packets arrive at destination in timely manner 4: Multimedia App. & Transp. 4-79 RTP entities End System SSRC = 53 Different encodings Multiple streams Single stream Translator Mixer SSRC = 19 SSRC = 19 CSRC = 53 77 End System SSRC = 77 End system: application that actually generates/consumes the content carried in RTP packets SSRC: Synchronisation Source identifier Translator: intermediate system that changes the encoding scheme without altering the timing. It may also convert multicast into multiple unicast streams Mixer: intermediate system that receives multiple streams and combines them in some manner. The new stream has its own timing (new SSRC) CSRC: Contributing Source identifier 4: Multimedia App. & Transp. 4-80 40

RTP Header payload type sequence number type time stamp Synchronization Source ID Miscellaneous fields Payload Type (7 bits): Indicates type of encoding currently being used. If sender changes encoding in middle of conference, sender informs receiver via payload type field Payload type 0: PCM µ-law, 64 kbps Payload type 3: GSM, 13 kbps Payload type 7: LPC, 2.4 kbps Payload type 26: Motion JPEG Payload type 31: H.261 Payload type 33: MPEG2 video Sequence Number (16 bits): incremented by one for each RTP packet sent, detects packet loss and restores packet sequence 4: Multimedia App. & Transp. 4-81 RTP Header (2) payload type sequence number type time stamp Synchronization Source ID Miscellaneous fields Timestamp field (32 bytes): sampling instant of first byte in this RTP data packet for audio, timestamp clock typically increments by one for each sampling period (for example, each 125 µsecs for 8 KHz sampling clock) if application generates chunks of 160 encoded samples, then timestamp increases by 160 for each RTP packet when source is active. Timestamp clock continues to increase at constant rate when source is inactive SSRC field (32 bits): identifies source of RTP stream. Each stream in RTP session should have distinct SSRC. 4: Multimedia App. & Transp. 4-82 41

Real-Time Control Protocol (RTCP) works in conjunction with RTP each participant in RTP session periodically transmits RTCP control packets to all other participants each RTCP packet contains sender and/or receiver reports report statistics useful to application: # packets sent, # packets lost, interarrival jitter, etc. feedback can be used to control performance sender may modify its transmissions based on feedback 4: Multimedia App. & Transp. 4-83 RTCP: multiple multicast senders sender RTP RTCP RTCP RTCP receivers! each RTP session: typically a single multicast address; all RTP / RTCP packets belonging to session use multicast address! RTP, RTCP packets distinguished from each other via distinct port numbers! to limit traffic, each participant reduces RTCP traffic as number of conference participants increases 4: Multimedia App. & Transp. 4-84 42

RTCP: packet types Receiver Report (RR) packets: fraction of packets lost, last sequence number, average interarrival jitter Sender Report (SR) packets: SSRC of RTP stream, current time, number of packets sent, number of bytes sent Source Description (SDES) packets: e-mail address of sender, sender's name, SSRC of associated RTP stream provide mapping between the SSRC and the user/host name 4: Multimedia App. & Transp. 4-85 RTCP: stream synchronization RTCP can synchronize different media streams within a RTP session e.g., videoconferencing app: each sender generates one RTP stream for video, one for audio timestamps in RTP packets tied to the video, audio sampling clocks not tied to wall-clock time each RTCP sender-report packet contains (for most recently generated packet in associated RTP stream): timestamp of RTP packet wall-clock time for when packet was created receivers use association to synchronize playout of audio, video 4: Multimedia App. & Transp. 4-86 43

RTCP: bandwidth scaling RTCP attempts to limit its traffic to 5% of session bandwidth Example one sender, sending video at 2 Mbps RTCP attempts to limit its traffic to 100 kbps RTCP gives 75% of rate to receivers; remaining 25% to sender 75 kbps is equally shared among receivers: with R receivers, each receiver gets to send RTCP traffic at 75/R kbps sender gets to send RTCP traffic at 25 kbps participant determines RTCP packet transmission period by calculating average RTCP packet size (across entire session) and dividing by allocated rate 4: Multimedia App. & Transp. 4-87 SIP: Session Initiation Protocol [RFC 3261] SIP long-term vision: all telephone calls, video conference calls take place over Internet people are identified by names or e-mail addresses, rather than by phone numbers you can reach callee (if callee so desires), no matter where callee roams, no matter what IP device callee is currently using 4: Multimedia App. & Transp. 4-88 44

SIP Services SIP provides mechanisms for call setup: for caller to let callee know she wants to establish a call so caller and callee can agree on media type, encoding to end call determine current IP address of callee: maps mnemonic identifier to current IP address call management: add new media streams during call change encoding during call invite others transfer, hold calls 4: Multimedia App. & Transp. 4-89 Example: setting up a call to known IP address Alice 167.180.112.24 193.64.210.89 INVITE bob@193.64.210.89 c=in IP4 167.180.112.24 m=audio 38060 RTP/AVP 0 port 5060 Bob # Alice s SIP invite message indicates her port number, IP address, encoding she prefers to receive (PCM µlaw) port 5060 200 OK c=in IP4 193.64.210.89 m=audio 48753 RTP/AVP 3 Bob's terminal rings # Bob s 200 OK message indicates his port number, IP address, preferred encoding (GSM) port 38060 ACK port 5060 µ Law audio # SIP messages can be sent over TCP or UDP; here sent over RTP/UDP GSM port 48753 # default SIP port number is 5060 time time 4: Multimedia App. & Transp. 4-90 45

Setting up a call (more) codec negotiation: suppose Bob doesn t have PCM µlaw encoder Bob will instead reply with 606 Not Acceptable Reply, listing his encoders Alice can then send new INVITE message, advertising different encoder rejecting a call Bob can reject with replies busy, gone, payment required, forbidden media can be sent over RTP or some other protocol 4: Multimedia App. & Transp. 4-91 Example of SIP message INVITE sip:bob@domain.com SIP/2.0 Via: SIP/2.0/UDP 167.180.112.24 From: sip:alice@hereway.com To: sip:bob@domain.com Call-ID: a2e3a@pigeon.hereway.com Content-Type: application/sdp Content-Length: 885 c=in IP4 167.180.112.24 m=audio 38060 RTP/AVP 0 Notes: HTTP message syntax sdp = session description protocol Call-ID is unique for every call # Here we don t know Bob s IP address. -> Intermediate SIP servers needed # Alice sends, receives SIP messages using SIP default port 5060 # Alice specifies in Via: header that SIP client sends, receives SIP messages over UDP 4: Multimedia App. & Transp. 4-92 46

Name translation and user location caller wants to call callee, but only has callee s name or e-mail address need to get IP address of callee s current host: user moves around DHCP protocol user has different IP devices (PC, smartphone, car device) result can be based on: time of day (work, home) caller (don t want boss to call you at home) status of callee (calls sent to voicemail when callee is already talking to someone) Service provided by SIP servers 4: Multimedia App. & Transp. 4-93 SIP Registrar one function of SIP server: registrar when Bob starts SIP client, client sends SIP REGISTER message to Bob s registrar server Register Message: REGISTER sip:domain.com SIP/2.0 Via: SIP/2.0/UDP 193.64.210.89 From: sip:bob@domain.com To: sip:bob@domain.com Expires: 3600 4: Multimedia App. & Transp. 4-94 47

SIP Proxy another function of SIP server: proxy Alice sends invite message to her proxy server contains address sip:bob@domain.com proxy responsible for routing SIP messages to callee Bob possibly through multiple proxies Bob sends response back through the same set of proxies proxy returns Bob s SIP response message to Alice contains Bob s IP address SIP proxy analogous to local DNS server plus TCP setup 4: Multimedia App. & Transp. 4-95 SIP example: jim@umass.edu calls keith@poly.edu 2. UMass proxy forwards request to Poly registrar server UMass SIP proxy 1. Jim sends INVITE message to UMass SIP proxy. 128.119.40.186 1 8 2 3 3. Poly server returns redirect response, indicating that it should try keith@eurecom.fr 4. Umass proxy forwards request to Eurecom registrar server 4 7 6-8. SIP response returned to Jim 5. eurecom registrar forwards INVITE to 197.87.54.21, which is running Keith s SIP client Note: also a SIP ack message from Jim, which is not shown 4: Multimedia App. & Transp. 4-96 9 Poly SIP registrar 9. Data flows between clients 6 5 Eurecom SIP registrar 197.87.54.21 48

Comparison with former H.323 H.323 is another signaling protocol for real-time, interactive applications H.323 is a complete, vertically integrated suite of protocols for multimedia conferencing: signaling, registration, admission control, transport, codecs SIP is a single component. Works with RTP, but does not mandate it. Can be combined with other protocols, services H.323 comes from the ITU (telephony) SIP comes from IETF: borrows much of its concepts from HTTP SIP has Web flavor, whereas H.323 has telephony flavor SIP uses the KISS principle: Keep It Simple Stupid 4: Multimedia App. & Transp. 4-97 Chapter 4: Summary Principles audio and video coding multimedia applications types over IP streaming stored audio video, real-time conversational voice/video UDP versus TCP streaming making the best of best effort service DASH: Dynamic, Adaptive Streaming over HTTP CDN: Content Distribution Networks adaptive playout delay loss recovery (FEC, retransmissions) and concealment Protocols RTSP RTP/RTCP SIP 4: Multimedia App. & Transp. 4-98 49

How should the Internet evolve to better support multimedia? Laissez-faire just put more capacity where needed no major changes in network, let apps handle it content distribution networks, application-layer multicast Differentiated services philosophy: fewer changes to Internet infrastructure, yet provide 1st and 2nd class service Integrated services philosophy: fundamental changes in Internet so that apps can reserve end-to-end bandwidth requires new, complex software in hosts & routers What s your opinion? 4: Multimedia App. & Transp. 4-99 50