Streaming Stored Multimedia

Transcription

1 Introduction to Multimedia Networking amd QoS Classify multimedia applications Identify the network services the apps need Making the best of best effort service Streaming Stored Multimedia vs. Interactive Applications Adaptive Playback and Smart Error Recovery Algorithms Some Common Protocols: RTSP, RTP/RTCP, SIP QoS Interserv Diffserv RSVP Multimedia and Quality of Service QoS network provides application with level of performance needed for application to function. Multimedia applications: network audio and video ( continuous media ) 1 2 MM Networking Applications Classes of MM applications: 1) Streaming stored audio and video 2) Streaming live audio and video 3) Real-time interactive audio and video Jitter is the variability of packet delays within the same packet stream Fundamental characteristics: Typically delay sensitive end-to-end delay delay jitter But loss tolerant: infrequent losses cause minor glitches Antithesis of data, which are loss intolerant but delay tolerant. Streaming Stored Multimedia Streaming: media stored at source transmitted to client streaming: client playout begins before all data has arrived timing constraint for still-to-be transmitted data: in time for playout 11 14

2 Streaming Stored Multimedia: What is it? Streaming Stored Multimedia: Interactivity 1. video recorded 2. video sent network delay 3. video received, played out at client streaming: at this time, client playing out early part of video, while server still sending later part of video time VCR-like functionality: client can pause, rewind, FF, push slider bar 10 sec initial delay OK 1-2 sec until command effect OK RTSP often used (more later) Streaming Live Multimedia Examples: Internet radio talk show Live sporting event Streaming playback buffer playback can lag tens of seconds after transmission still have timing constraint Interactivity fast forward impossible rewind, pause possible! Interactive, Real-Time Multimedia applications: IP telephony, video conference, distributed interactive worlds end-end delay requirements: audio: < 150 msec good, < 400 msec OK includes application-level (packetization) and network delays higher delays noticeable, impair interactivity session initialization how does callee advertise its IP address, port number, encoding algorithms? 17 18

3 Multimedia Over Today s Internet TCP/UDP/IP: best-effort service no guarantees on delay, loss?????? But you said multimedia apps requires QoS and level of performance to be?? effective!?? Today s Internet multimedia applications use application-level techniques to mitigate (as best possible) effects of delay, loss? Challenges TCP/UDP/IP suite provides best-effort, no guarantees on expectation or variance of packet delay Streaming applications delay of 5 to 10 seconds is typical and has been acceptable, but performance deteriorate if links are congested (transoceanic) Real-Time Interactive requirements on delay and its jitter have been satisfied by over-provisioning (providing plenty of bandwidth), what will happen when the load increases? Challenges (more) Most router implementations use only First-Come-First-Serve (FCFS) packet processing and transmission scheduling To mitigate impact of best-effort protocols, we can: Use UDP to avoid TCP and its slow-start phase Buffer content at client and control playback to remedy jitter Adapt compression level to available bandwidth 21 How should the Internet evolve to better support multimedia? Laissez-faire no major changes more bandwidth when needed content distribution, application-layer multicast application layer Integrated services philosophy: Fundamental changes in Internet so that apps can reserve end-to-end bandwidth Requires new, complex software in hosts & routers Incorporate resource reservation (bandwidth, processing, buffering), and new scheduling policies Set up service level agreements with applications, monitor and enforce the agreements, charge accordingly Differentiated services philosophy: Fewer changes to Internet infrastructure, yet provide 1st and 2nd class service. Use two traffic classes for all packets and differentiate service accordingly Charge based on class of packets Network capacity is provided to ensure first class packets incur no significant delay at routers 22

4 Streaming Stored Multimedia Internet Multimedia: Simplest Approach Application-level streaming techniques for making the best out of best effort service: client side buffering use of UDP versus TCP multiple encodings of multimedia Media Player jitter removal decompression error concealment graphical user interface w/ controls for interactivity audio, video not streamed: audio or video stored in file files transferred as HTTP object received in entirety at client then passed to player no, pipelining, long delays until playout! Internet multimedia: Streaming Approach Streaming from a streaming server browser GETs metafile browser launches player, passing metafile player contacts server server streams audio/video to player 25 This architecture allows for non-http protocol between server and media player Can also use UDP instead of TCP. 26

5 User Control of Streaming Media: RTSP HTTP Does not target multimedia content No commands for fast forward, etc. RTSP: RFC 2326 Client-server application layer protocol. For user to control display: rewind, fast forward, pause, resume, repositioning, etc What it doesn t do: does not define how audio/video is encapsulated for streaming over network does not restrict how streamed media is transported; it can be transported over UDP or TCP does not specify how the media player buffers audio/video RTSP Example Scenario: metafile communicated to web browser browser launches player player sets up an RTSP control connection, data connection to streaming server Metafile Example <title>twister</title> <session> <group language=en lipsync> <switch> <track type=audio e="pcmu/8000/1" src = "rtsp://audio.example.com/twister/audio.en/lofi"> <track type=audio e="dvi4/16000/2" pt="90 DVI4/8000/1" src="rtsp://audio.example.com/twister/audio.en/hifi"> </switch> <track type="video/jpeg" src="rtsp://video.example.com/twister/video"> </group> </session> RTSP Operation 29 30

6 RTSP Exchange Example C: SETUP rtsp://audio.example.com/twister/audio RTSP/1.0 Transport: rtp/udp; compression; port=3056; mode=play S: RTSP/1.0 Session 4231 C: PLAY rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=0- C: PAUSE rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=37 C: TEARDOWN rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 S: RTSP/1.0 Session Streaming Multimedia: Client Buffering constant bit rate video transmission variable network delay client playout delay client video reception buffe ered vide eo constant bit rate video playout at client Client-side buffering, playout delay compensate for network-added delay, delay jitter time 32 Streaming Multimedia: Client Buffering variable fill rate, x(t) buffered video constant drain rate, d Client-side buffering, playout delay compensate for network-added delay, delay jitter 33 Streaming Multimedia: UDP or TCP? UDP server sends at rate appropriate for client (oblivious to network congestion!) often send rate = encoding rate = constant rate then, fill rate = constant rate - packet loss short playout delay (2-5 seconds) to compensate for network delay jitter error recover: time permitting TCP send at maximum possible rate under TCP fill rate fluctuates due to TCP congestion control larger playout delay: smooth TCP delivery rate HTTP/TCP passes more easily through firewalls 34

7 Streaming Multimedia: client rate(s) 1.5 Mbps encoding 28.8 Kbps encoding Q: how to handle different client receive rate capabilities? 28.8 Kbps dialup 100Mbps Ethernet A: server stores, transmits multiple copies of video, encoded at different rates APPLICATION KER RNEL DASH - Dynamic Adaptive Streaming over HTTP DASH HTTP AAL3/ 4 TCP Synchronization Service SIP RTSP RSVP RTCP AAL5 DCCP MPLS ATM/Fiber Optics UDP IP Version 4, IP Version 6 Ethernet/WiF i Media encaps (H.264, MPEG- 4) RTP Layer 5 (Session) Layer 4 (Transport) Layer 3 (Network) Layer 2 (Link/MAC) 35 Adaptive Streaming Concept Adaptive Streaming technologies enable Optimal streaming video viewing experience for diverse range of devices over broad set of connection speeds Adaptive streaming technologies share Production of multiple files from the same source file to distribute to viewers watching on different powered devices via different connection speeds Distribution of files adaptively, changing stream that is delivered to adapt to changes in effective throughput and available CPU cycles on playback stations Transparent operation to the user so that the viewer clicks one button and all streams switch/adapt behind the scenes. Adaptive Streaming One Approach of Adaptive Streaming 1. Server sends first the high important video information (e.g., I frames) And after the high importance video information is sent, lower importance video information follows (e.g., P and B frames) if bandwidth and time allows Second Approach of Adaptive Streaming 1. At server video is encoded in multiple bitrates and depending on the device Bandwidth it adjusts at what rate Other approaches exist

8 Adaptive HTTP Streaming System Client-centric approach Server Can be standard web server Media segment can be prepared in-line or off-line Client Sends series of HTTP GET segment requests and receives segments Performs rate adaptation before sending a new GET segment request Client has best view of network conditions No session state in network Redundancy Scalability Faster innovation and experimentation But, relies on client for operational metrics Only client knows what really happens Terms and Definitions of Adaptive HTTP Streaming DASH Client Need Media Presentation Description (MDP) which provides metadata For requesting (GET request) media segments For rate adaptation purpose Segment which may include media data or metadata to decode Need DASH Thomas Stockhammer, Qualcomm, DASH Design Principles and Standards

9 Information Classification DASH uses MPD (Media Presentation Descriptor) and Index Information as metadata for DASH Access Client Initialization and Media Segments for Media Engine Reuse of existing container format Media Presentation Data Model MDP - description of accessible segments and corresponding timing MDP Information Includes redundant information of media streams to initially select or reject groups or representations Includes access and timing information Content addressing via HTTP-URLs Byte range for each accessible segment Media start time and duration Media Segments (1) Contain information to map segment into media presentation timeline for switching and synchronous presentation with other representations Can be short (~ 1-10seconds) Can be long (~10sec 2 hours)

10 Media Segments (2) Segment Indexing Media segment duration Short duration Long duration advantages live high switching granularity on segment level - Small number of files - Small number of URLs - Coarse switching granularity - Flexible request sizes disadvantages - Large number of files - Large number of URLs - Need for large segment index - Not appropriate for live streaming Provides information in ISO box structure on Accessible units of data (e.g., frames) in media segment Byte range in segments (easy access through HTTP GET) Accurate presentation duration (seamless switching) Presence of representation access positions Provides compact bitrate-over-time to client Can be used for intelligent request schedule Generic data structure Hierarchical structuring for efficient access Media Segment with Segment Index MPEG-DASH Summary Is rich and simple at the same time Supports both un-chunked and chunked Supports both separate and combined AV Index formats for efficient byte range operation ISO base media file format w/common encryption Currently best candidate for open standard for adaptive streaming

11 Real-time interactive applications PC-2-PC phone instant messaging services are providing this, e.g. Yahoo messenger, googletalk Skype PC-2-phone Dialpad Net2phone Skype videoconference with Webcams Voice Over IP (VoIP) Delivering phone calls over IP Computer to computer Analog phone to/from computer Analog phone to analog phone Motivations for VoIP Cost reduction Simplicity Advanced applications Web-enabled call centers Collaborative white boarding Do Not Disturb, Locate Me, etc. Voic sent as Traditional Telecom Infrastructure Corporate/Campus Private Branch Exchange 7043 External line Telephone switch Another switch Corporate/Campus 7043 PBX VoIP Gateway VoIP Gateways External line Another campus 8151 PBX VoIP Gateway Corporate/Campus LAN Internet LAN Internet LAN IP Phone Client 55 56

12 Skype Niklas Zennström and Janus Friis in 2003 Developed by KaZaA Instant Messenger (IM) with voice support Based on peer-topeer (P2P) networking technology Skype Network Architecture Login server is the only central server (consisting of multiple machines) Both ordinary host and super nodes are Skype clients Any node with a public IP address and having sufficient resources can become a super node Skype maintains their own super nodes Challenges of Firewalls and NATs Firewalls Often block UDP traffic NAT Cannot easily initiate traffic to a host behind a NAT since there is no unique address for the host Skype must deal with these problems Discovery: client exchanges messages with super node Traversal: sending data through an intermediate peer We will talk more about this later Interactive Multimedia: Internet Phone Introduce Internet Phone by way of an example speaker s audio: alternating talk spurts, silent periods. 64 kbps during talk spurt pkts generated only during talk spurts 20 msec chunks at 8 Kbytes/sec: 160 bytes data application-layer header added to each chunk. Chunk+header encapsulated into UDP segment. application sends UDP segment into socket every 20 msec during talkspurt

13 Internet Phone: Packet Loss and Delay network loss: IP datagram lost due to network congestion (router buffer overflow) delay loss: IP datagram arrives too late for playout at receiver delays: processing, queueing in network; end-system (sender, receiver) delays typical maximum tolerable delay: 400 ms loss tolerance: depending on voice encoding, losses concealed, packet loss rates between 1% and 10% can be tolerated. Delay Jitter constant bit rate transmission variable network delay (jitter) client playout delay client reception buffe ered dat ta constant bit rate playout at client Consider the end-to-end delays of two consecutive packets: difference can be more or less than 20 msec time Internet Phone: Fixed Playout Delay Receiver attempts to playout each chunk exactly q msecs after chunk was generated. chunk has time stamp t: play out chunk at t+q. chunk arrives after t+q: data arrives too late for playout, data lost Tradeoff for q: large q: less packet loss small q: better interactive experience Fixed Playout Delay Sender generates packets every 20 msec during talk spurt. First packet received at time r First playout schedule: begins at p Second playout schedule: begins at p packets packets generated packets received loss playout schedule p - r playout schedule p' - r time r 63 p p' 64

14 Adaptive Playout Delay, I Goal: minimize playout delay, keeping late loss rate low Approach: adaptive playout delay adjustment: Estimate network delay, adjust playout delay at beginning of each talk spurt. Silent periods compressed and elongated. Chunks still played out every 20 msec during talk spurt. t i =timestampof the ith packet r = i p thetimepacket iis received by receiver thetimepacket iis played at receiver r t = network delay for ith packet i i d = estimate of average network delay after receiving ith packet i = i Dynamic estimate of average delay at receiver: d i = ( 1 u) di 1+ u( ri ti) where u is a fixed constant (e.g., u =.01). Adaptive playout delay II Also useful to estimate the average deviation of the delay, v i : v i = ( 1 u) vi 1+ u ri ti di The estimates d i and v i are calculated for every received packet, although they are only used at the beginning of a talk spurt. For first packet in talk spurt, playout time is: p = t + d + Kv where K is a positive constant. i i Remaining packets in talkspurt are played out periodically i i Adaptive Playout, III Q: How does receiver determine whether packet is first in a talkspurt? If no loss, receiver looks at successive timestamps. difference of successive stamps > 20 msec -->talk spurt begins. With loss possible, receiver must look at both time stamps and sequence numbers. difference of successive stamps > 20 msec and sequence numbers without gaps --> talk spurt begins. Recovery from packet loss (1) forward error correction (FEC): simple scheme for every group of n chunks create a redundant chunk by exclusive OR-ing the n original chunks send out n+1 chunks, increasing the bandwidth by factor 1/n. can reconstruct the original n chunks if there is at most one lost chunk from the n+1 chunks Playout delay needs to be fixed to the time to receive all n+1 packets Tradeoff: increase n, less bandwidth waste increase n, longer playout delay increase n, higher probability that 2 or more chunks will be lost 67 68

15 Recovery from packet loss (2) Recovery from packet loss (3) 2nd FEC scheme piggyback lower quality stream send lower resolution audio stream as the redundant information for example, nominal stream PCM at 64 kbps and redundant stream GSM at 13 kbps. Whenever there is non-consecutive loss, the receiver can conceal the loss. Can also append (n-1)st and (n-2)nd low-bit rate chunk Interleaving chunks are broken up into smaller units for example, 4 5 msec units per chunk Packet contains small units from different chunks if packet is lost, still have most of every chunk has no redundancy overhead but adds to playout delay Summary: Internet Multimedia: bag of tricks use UDP to avoid TCP congestion control (delays) for time-sensitive traffic client-side adaptive playout delay: to compensate for delay server side matches stream bandwidth to available client-to-server path bandwidth chose among pre-encoded stream rates dynamic server encoding rate error recovery (on top of UDP) FEC, interleaving retransmissions, time permitting conceal errors: repeat nearby data 71 Real-Time Multimedia Protocols There are several related protocols which support the realtime traffic over Internet RTP (Real-Time Protocol) Used for real-time data transport (extends UDP, sits between application and UDP) RTCP (Real-time Control Protocol) Used to exchange control information between sender and receiver, works in conjunction with RTP SIP (Session Initiation Protocol) Provides mechanisms for establishing calls over IP RTSP (Real-Time Streaming Protocol) Allows user to control display (rewind, FF, pause, resume, ) RSVP (Resource Reservation Protocol) Intended to add determinism to connectionless information, provides QoS to some extent. 72

16 Real-Time Multimedia Protocols Real Time Protocol (RTP) Real Time Protocol (RTP) RTP (RFC 3550 (2003)) provides payload type identification, packet sequence numbering, timestamping Dilemma: RTP as a transport layer sublayer or part of application layer At this point RTP is an application library executed in user space RTP does not ensure real-time delivery, it provides means for: Jitter elimination/reduction (playout buffer) Synchronization of audio-video streams that belong to same session Multiplexing of audio/video streams Translation of audio/video streams from one encoding to another 75 RTP and QoS RTP does not provide any mechanism to ensure timely delivery of data or provide other quality of service guarantees. RTP encapsulation is only seen at the end systems: it is not seen by intermediate routers. Routers providing best-effort service do not make any special effort to ensure that RTP packets arrive at the destination in a timely matter. 76

17 RTP Packet RTP Header RTP Payload Type RTP Header Timestamp field (32 bytes long). Reflects the sampling instant of the first byte in the RTP data packet. The granularity of the timing is application specific! For audio, timestamp clock typically increments by one for each sampling period (for example, each 125 usecs for a 8 KHz sampling clock) if application generates chunks of 160 encoded samples, then timestamp increases by 160 for each RTP packet when source is active. Timestamp clock continues to increase at constant rate when source is inactive. 79 SSRC field (32 bits long). Identifies the source of the RTP stream. Each stream in a RTP session should have a distinct SSRC. 80

18 RTP Timestamp and Seq # Audio: Timestamp clock rate for audio 8000 Hz (= 125 μs) One RTP packet caries 20 ms of audio samples, each RTP packet sent by separate UDP datagram to avoid packetization delay Timestamp increments by 20,000 μs/125 μs = 160 (even if the packet is not sent after silence suppression) Packet rate = 1 sec/20 ms = 50 Hz Number of bits per RTP payload for uncompressed audio = 160*8 = 1280, for compressed audio typically 8 times less. Sequence number increments by one for each RTP packet RTP Timestamp and Seq # Video Timestamp clock rate for video is 90,000 Hz One RTP packet caries one video frame (it is possible that one frame is spread between several RTP packets) RTP packet rate = 30 Hz (sometimes 25 Hz) Timestamp increments by (1/30)/(1/90000) = 90000/30 = 3000, or 9000/25 = 3600 Number of bits per RTP payload for uncompressed video conferencing 352x240x12 = 1,000,000, for compressed video typically 100 times less. Sequence number increments by one for each RTP packet RTP packets It is advisable to keep coded slice sizes as close to, but never bigger than, the MTU size (largest size of a packet that can be transmitted without being plit/recombined on the transport and network layer), because: 1. It optimizes the payload/header overhead relationship 2. Minimizes the loss probability of a (fragmented) coded slice due to the loss of a single fragment on the network/transport layer and the resulting discarding of all other fragments belonging to the coded slice MTU sizes: ~1500 bytes for wireline IP links (max. size of an Ethernet packet) RTP packets - example DVD quality video transmission: 30 frames/s, 720x480 resolution, 3 bytes per pixel 31,104,000 bytes/s raw rate 311,040 bytes/s compressed data rate (100x compression) MTU = 1500 bytes: 311,040/1460 = 213 packets/s -> 319,500 bytes/s (required throughput including overhead) 83 84

19 RTP Source Identifiers Synchronization source identifier (SSRC) Uniquely defines the multimedia source. The initial value is determined randomly. Contributing source identifier (CSRC) Used by mixers to identify sources in a mix. Real-Time Control Protocol (RTCP) RTP provides basic transport layer for real time applications It does not provide any mechanism for error and flow control, congestion control, quality feedback and synchronization. RTCP is added as a companion to RTP to provide end-to-end monitoring, data delivery, and QoS 85 RCTP provides 1. Feedback on application/network performance 2. Correlation/Synchronization information of different media streams generated by the same sender 3. Convey sender Identity 86 Real-Time Control Protocol (RTCP) Each participant in RTP session periodically transmits RTCP control packets to all other participants. Each RTCP packet contains sender and/or receiver reports report statistics useful to application Statistics include number of packets sent, number of packets lost, interarrival jitter, etc. Feedback can be used to control performance Sender may modify its transmissions based on feedback RTCP - Packet 87 88

20 RTCP - Header RCTP Packet Type RTCP Packets Receiver report packets: fraction of packets lost, last sequence number, average interarrival jitter. Sender report packets: SSRC of the RTP stream, the current time, the number of packets sent, and the number of bytes sent. Source description packets: address of sender, sender's name, SSRC of associated RTP stream. Provide mapping between the SSRC and the user/host name. RTCP Packets Synchronization of Streams RTCP can synchronize different media streams within a RTP session. Consider videoconferencing app for which each sender generates one RTP stream for video and one for audio. Timestamps in RTP packets tied to the video and audio sampling clocks not tied to the wall-clock time Each RTCP sender-report packet contains (for the most recently generated packet in the associated RTP stream): timestamp of the RTP packet wall-clock time for when packet was created. Receivers can use this association to synchronize the playout of audio and video

21 RTCP Sender Report RTCP Receiver Report RCTP Source Description RCTP Bye Message 95 96

22 RTCP Bandwidth Scaling - For an RTP session there is typically a single multicast address; all RTP and RTCP packets belonging to the session use the multicast address. - RTP and RTCP packets are distinguished from each other through the use of distinct port numbers. - To limit traffic, each participant reduces his RTCP traffic as the number of conference participants increases. RTCP Bandwidth Scaling RTCP attempts to limit its traffic to 5% of the session bandwidth. Example Suppose one sender, sending video at a rate of 2 Mbps. Then RTCP attempts to limit its traffic to 100 Kbps. RTCP gives 75% of this rate to the receivers; remaining 25% to the sender The 75 kbps is equally shared among receivers: With R receivers, each receiver gets to send RTCP traffic at 75/R kbps. Sender gets to send RTCP traffic at 25 kbps. Participant determines RTCP packet transmission period by calculating avg RTCP packet size (across the entire session) and dividing by allocated rate SIP Session Initiation Protocol Comes from IETF SIP long-term vision All telephone calls and video conference calls take place over the Internet People are identified by names or addresses, rather than by phone numbers. You can reach the callee, no matter where the callee roams, no matter what IP device the callee is currently using. Session Initiation Protocol (SIP) RFC 3261, June 2002 SIP is used to set up, modify and terminate an RTP-based multimedia session (like IP telephone conversation or videoconferencing). There are several issues related to this: How to determine the current IP address of the callee? He/she may have several devices like Internet phone on PC (one at home, one at office), tablet, real phone/cell phone. The person can have dynamic IP address (DHCP), or the person is not currently available (not signed in), or doesn t want to answer at this moment. How can we route the call if we even do not know where the person is at the moment. How can we be notified once the person is located, or the person is available

23 Session Initiation Protocol (SIP) RFC 3261, June 2002 SIP is used to set up, modify and terminate an RTP-based multimedia session (like IP telephone conversation or videoconferencing). There are several issues related to this: Once the connection is established, how can we define the encoding type for the audio/video stream, or change the encoding type during the session, How can we invite new participants during the call, perform call transfer and call holding. All these isues are addressed in SIP. An alternative protocol to SIP is H.323, which is more complex and more difficult to implement/configure. SIP is a simple protocol well suited for Internet. 101 SIP - Functionalities User Location and Registration End points (telephones) notify SIP proxies of their location; SIP determines which end points will participate in a call. User Availability SIP is used by end points to determine whether they will answer a call. User Capabilities SIP is used by end points to negotiate media capabilities, such as agreeing on a mutually supported voice codec. Session Setup SIP tells the end point that its phone should be ringing; SIP is used to agree on session attributes used by the calling and called party. Session Management SIP is used to transfer calls, terminate calls, and change call 102 parameters in mid-session (such as adding a 3-way conference). SIP SIP: architectural elements Proxy Server Redirect Server Registrar Server IP network User Agent User Agent SIP Gateway PSTN 103

24 Proxy Server User Agent Redirect Server Registrar Server Proxy Server Proxy Server Redirect Server Registrar Server User Agent IP network SIP Gateway An application that initiates, receives and terminates calls. User Agent Clients (UAC) An entity that initiates a call. User Agent Server (UAS) An entity that receives a call. Both UAC and UAS can terminate a call. User Agent IP network SIP Gateway An intermediary program that acts as both a server and a client to make requests on behalf of other clients. Requests are serviced internally or by passing them on, possibly after translation, to other servers. Interprets, rewrites or translates a request message before forwarding it. Redirect Server Registrar Server Proxy Server Redirect Server Registrar Server Proxy Server Redirect Server Registrar Server User Agent IP network SIP Gateway A server that accepts a SIP request, maps the address into zero or more new addresses and returns these addresses to the client. Unlike a proxy server, the redirect server does not initiate its own SIP request. User Agent IP network SIP Gateway A server that accepts REGISTER requests. The register server may support authentication. A registrar server is typically co-located with a proxy or redirect server and may offer location services.

25 Location server SIP Gateway Proxy Server Redirect Server Registrar Server Proxy Server Redirect Server Registrar Server User Agent Location Server IP network SIP Gateway A location server is used by a SIP redirect or proxy server to obtain information about a called party s possible location(s). It is not a SIP element and the protocols between the SIP (proxy/redirect) server and the location server is not SIP User Agent IP network SIP Gateway An application that interfaces a SIP network to a network using another signaling protocol It can also terminate the media path:»a SIP/PSTN gateway terminates both the signaling and the media path» a SIP/H323 gateway terminates only the signaling path It is just a special type of user agent (supporting lots of users) SIP call setup SIP principles - Client Server User Agent IP network user@xy.it ( audio G.729 ) User Agent Proxy can be easily introduced in the call flows... Proxy Server Client Server ACK Proxy Server Server Client 100 PROGRESS ACK SIP - SDP (UDP) Client ACK ACK Server RTP Streams User Agent User Agent Media (UDP)

26 SIP Principles - Addressing Addresses are associated to the user and not the specific terminal (like addresses) URL - Uniform Resource Locator sip:user@domain sip:stefano.salsano@uniroma2.it sip:salsano@ sip:1212@gateway.uniroma2.it sip: @gateway.com;user=phone scheme SIP URLs sip:salsano@coritel.it:5060;user=ip; user-name host port sip: @gateway.com;user=phone list of parameters The user parameter is used to determine if a telephone number is present in the user-name portion of the URL A SIP URL is resolved to an IP address by using DNS lookups and SIP servers and at the time of the call SIP Principles - Addressing Thanks to user based addressing SIP allows for Personal Mobility The user can access the service from different points and can associate different terminals with different capabilities (phone, videophone, answering machine) Registrar Server REGISTER sip:uniroma2.it SIP/2. 0 From: sip:stefano.salsano@uniroma2.it To: sip:stefano.salsano@uniroma2.it Contact: <sip: > Expires: 3600 User Agent X SIP Registration 1 REGISTER 3 Location Server 2 Registrar Server This registration example establishes presence of user with address sip:stefano.salsano@uniroma2.it and binds this address to user s current location

27 SIP URL resolution with DNS & proxy DNS SRV Query? uniroma2.it 1 User Agent Y DNS server 3 2 sip:stefano.salsano@uniroma2.it IP address of uniroma2.it SIP proxy server sip: Location Server 7 uniroma2.it incoming proxy server User Agent Y wants to call sip:stefano.salsano@uniroma2.it User Agent X SIP call using a destination proxy DNS server User Agent Y uniroma2.it proxy server Internet Location Server User Agent Y wants to call s.salsano@uniroma2.it s.salsano@uniroma2.it could be at User Agent X or Z User Agent Y will contact the destination proxy User Agent Z User Agent X User Agent Y SIP call using a destination proxy uniroma2.it proxy server ACK (*) Media Session User Agent X (Destination) proxy server functionality Serves as rendezvous point at which callees are globally reachable Performs routing function, i.e., determines to which hop the SIP message should be relayed Allows the routing logic to be programmable Forking: several destination for a request may be tried, sequentially or in parallel BYE A proxy server has no media capability (*) There can be a direct communication after the Message

28 Forking proxy operation Outbound proxy - Proxy Chaining Location Server Forking proxy server branch 1 branch 2 branch 3 SIP PC SIP mobile phone SIP Phone DNS server Outbound proxy server LAN User Agent Y uniroma2.it proxy server Internet User Agent Y wants to call sip:stefano.salsano@uniroma2.it It has to use its own Outbound proxy server... User Agent stefano.salsano@uniroma User Agent Z User Agent Y Outbound proxy - Proxy Chaining ACK Outbound proxy server uniroma2.it proxy server ACK User Agent X Outbound proxy - Proxy Chaining Some reasons to use outbound proxy: locally important call processing (i.e. emergency calls) firewall interaction manages accounting for local users IP phones needs to know Outbound GW IP phones needs to know Outbound GW address Media Session BYE BYE

29 More on Proxy Chaining SIP call using a redirect proxy Location Server User Agent Y Outbound proxy server uniroma2.it central proxy server uniroma2.it local proxy servers DNS server uniroma2.it redirect server Internet User Agent X 1 Internet User Agent X Note: Signaling (in blue) may take a different path from Media (in black) 2 User Agent Z User Agent Y User Agent Y wants to call s.salsano@uniroma2.it User Agent Y will contact the destination redirect server The redirect server just give back the address SIP call using a redirect proxy Proxy vs. Redirect Server User Agent Y 302 Moved temporarily ACK uniroma2.it redirect server ACK Media Session BYE User Agent X A SIP server may either proxy or redirect a request Which of the two method applies is a configuration issue. It may be statically configured or dynamically determined (Call Processing Language - CPL). Redirection useful if a user moves or changes his provider (PSTN: The number you have dialed is not available) Proxy is useful if forking, authentication & accounting, firewall control are needed. In general proxying grants more control to the server

30 SIP methods (RFC 3261) : invites a party to participate in a session; may include SDP descriptions; res are use to change session state ACK: confirms the reception of an response (i.e. the session establisment) BYE: to leave a session OPTIONS: query about a server capabilities CANCEL: cancels a pending request (i.e. a pending ) REGISTER: to register an address with a SIP server; may convey user data (e.g. call processing scripts) Additional SIP methods INFO: carries session related control information generated during a session; does not change session state (RFC 2976). UPDATE: updates session parameters and can be sent before the is completed (RFC 3311). MESSAGE: mainly used to convey Instant Messages (RFC 3428). REFER: asks the recipient to refer to a resource provided in the request (RFC 3515). PRACK: carries a provisional response in a reliable way (RFC 3262). SUBSCRIBE: requests to be notified of a future event (RFC 3265, RFC 2848). NOTIFY: notifies about an event (RFC 3265, RFC 2848). PUBLISH: requests a server to publish event state (RFC 3903) SIP response codes SIP Message Syntax 1xx Informational (Provisional) 2xx Success (Final) 3xx Redirection (Final) 4xx Client error (Final) 5xx Server error (Final) 6xx Global failure (Final) Many header fields from http New ones are SIP specific Payload contains a media description SDP - Session Description Protocol ann@lucent.com SIP/2.0 From: Rosenberg <sip:rosen@bell-labs.com> Subject: SIP will be discussed, too To: Netravali <sip:ann@lucent.com> Call-ID: @ Contact: <sip:rosen@bell-labs.com> Content-type: application/sdp CSeq: 4711 Content-Length: 187 v=0 o=user IN IP s=mbone Audio i=discussion of Mbone Engineering Issues e=mbone@somewhere.com c=in IP /127 t=0 0 m=audio 3456 RTP/AVP 0 a=rtpmap:0 PCMU/8000

31 Principi del SIP Independenza della descrizione della sessione Il protocollo SIP è disaccoppiato dalla descrizione delle sessioni La sessione è descritta dal Session Description Protocol - SDP SIP trasporta gli elementi SDP come Payload SDP - Session Description Protocol Il protocollo SDP (Session Description Protocol, RFC 2327) consente ai terminali di descrivere i tipi di media (audio, video ) e le loro codifiche Non è in realtà un protocollo, ma una sintassi di descrizione SDP descrive gli indirizzi IP, le porte TCP/UDP, i tipi di codifica SIP trasporta le informazioni di SDP in modo trasparente SIP syntax request response method URL SIP/2.0 SIP/2.0 status reason Via: SIP/2.0/protocol host:port From: user <sip:name_from@source_domain> To: user <sip:name_to@dest_domain> Call-ID: local_id@host Cseq: seq_num method Content-Length: length of body Content-Type: type of body Header: parameter; par1=value; par2= value ;par3= folded into next line blank line V=0 o=origin_user timestamp timestamp IN IP4 host c=in IP4 media destination address t=0 0 m=media type port RTP/AVP payload types a=rtpmap:<payload type> <encoding name>/<clock rate> header body SIP requests A dialog is identified by the combination of 3 fields: From: To: Call-ID: Response messages do not reverse To: and From: Requests from callee to caller reverse To: and From: Caller and callee keep their own Cseq space Either side can send more s or BYE

32 Jonathan Rosenberg SIP call with a destination proxy Lucent.com proxy server ACK Media Session BYE Arun Netravali J.Rosenberg Lucent.com proxy server ACK Media Session BYE SIP messages A.Netravali Request URI sip:ann@lucent.com SIP/2.0 Via: SIP/2.0/UDP :5060 To: Netravali <sip:ann@lucent.com> From: Rosenberg <sip:rosen@bell-labs.com> Subject: SIP will be discussed, too Call-ID: @ CSeq: 4711 Contact: <sip:rosen@bell-lab.com> Content-type: application/sdp Content-Length: 183 v=0 o=user IN IP s= c=in IP t=0 0 m=audio 3456 RTP/AVP 0 a=rtpmap:0 PCMU/8000 J.Rosenberg Lucent.com proxy server ACK Media Session BYE Proxy adds the Via header SIP messages A.Netravali Branch: first part is computed as hash of To, From, Call-ID and Cseq second part is the forking number ann@lucent.com SIP/2.0 Via: SIP/2.0/UDP xy.lucent.com;branch=23.1 Via: SIP/2.0/UDP :5060 To: Netravali <sip:ann@lucent.com> From: Rosenberg <sip:rosen@bell-labs.com> Subject: SIP will be discussed, too Call-ID: @ CSeq: 4711 Contact: <sip:rosen@bell-lab.com> Content-type: application/sdp Content-Length: 183 v=0 o=user IN IP s= c=in IP t=0 0 m=audio 3456 RTP/AVP 0 a=rtpmap:0 PCMU/8000 J.Rosenberg Lucent.com proxy server ACK Media Session BYE SIP messages A.Netravali To: and From: are not exchanged A Tag:is added No SDP information is carried All Via:headers are copied SIP/2.0 Via: SIP/2.0/UDP xy.lucent.com;branch=23.1 Via: SIP/2.0/UDP :5060 To: Netravali <sip:ann@lucent.com>;tag= From: Rosenberg <sip:rosen@bell-labs.com> Call-ID: @ CSeq: 4711 Content-Length: 0

33 J.Rosenberg Lucent.com proxy server ACK Media Session BYE SIP messages A.Netravali Topmost Via:is removed SIP/2.0 Via: SIP/2.0/UDP :5060 To: Netravali From: Rosenberg Subject: SIP will be discussed, too Call-ID: CSeq: 4711 Content-Length: 0 J.Rosenberg Lucent.com proxy server ACK Media Session BYE A Tag:is added SIP messages A.Netravali To: and From: are not exchanged Contact:will be used to send messages bypassing proxy All Via:headers are copied SIP/2.0 Via: SIP/2.0/UDP xy.lucent.com;branch=23.1 Via: SIP/2.0/UDP :5060 To: Netravali <sip:ann@lucent.com>;tag= From: Rosenberg <sip:rosen@bell-labs.com> Subject: SIP will be discussed, too Call-ID: @ CSeq: 4711 Contact: <sip:netravali@ > Content-type: application/sdp Content-Length: 183 v=0 o=user IN IP s= c=in IP t=0 0 m=audio 3456 RTP/AVP 0 a=rtpmap:0 PCMU/8000 J.Rosenberg Lucent.com proxy server ACK Media Session BYE SIP messages A.Netravali Topmost Via:is removed SIP/2.0 Via: SIP/2.0/UDP :5060 To: Netravali <sip:ann@lucent.com>;tag= From: Rosenberg <sip:rosen@bell-labs.com> Subject: SIP will be discussed, too Call-ID: @ CSeq: 4711 Contact: <sip:netravali@ > Content-type: application/sdp Content-Length: 183 v=0 o=user IN IP s= c=in IP t=0 0 m=audio 3456 RTP/AVP 0 a=rtpmap:0 PCMU/8000 J.Rosenberg Lucent.com proxy server ACK Media Session BYE The ACK message is sent end-to-end SIP messages A.Netravali Tag:set by receiver is copied ACK ann@lucent.com SIP/2.0 Via: SIP/2.0/UDP :5060 To: Netravali <sip:ann@lucent.com>;tag= From: Rosenberg <sip:rosen@bell-labs.com> Subject: SIP will be discussed, too Call-ID: @ CSeq: 4711 ACK Content-type: application/sdp Content-Length: 0

34 J.Rosenberg Lucent.com proxy server ACK Media Session BYE The BYE message is sent end-to-end SIP messages A.Netravali Tag:Same Tag:is used BYE SIP/2.0 Via: SIP/2.0/UDP To: Rosenberg From: Netravali Call-ID: CSeq: 2000 BYE Content-Length: 0 NAT & Firewall Problem NAT & Firewall are employed to prevent hackers or unauthorized persons to access to the internal network Voice and video over IP are not NAT & Firewall friendly Provide secure two-way communication connection cross the NAT & Firewall Firewall Problem NAT Problem Firewall Problem for VoIP NAT Problem for VoIP A Media (A) 3.Media Internet 4. 2.OK 5.Media (B) B Internet? 2. OK 3. Media 1. User A is able to call User B since the firewall allows inside to outside sessions 2. User B is able to respond back to User A at the VoIP signaling layer 3. PROBLEM: Media traffic sent by User B from outside will be blocked since it uses a different socket than the VoIP signaling 4. PROBLEM: If User B tries to initiate a call to User A, it will be blocked by firewall 5. PROBLEM: If symmetric RTP is not used, the RTP fails to get back inside from B (S-RTP = the UA uses the same socket/port for sending and receiving the RTP) 1. User A sends an invite to User B, NAT translate the layer 3 address, but not the layer 5 (SIP,SDP) addresses 2. User B receives the invite and responds back to the NAT address 3. PROBLEM: User B tries to send RTP to User A with IP:Port (c= :m= 8000), but this fails since it can not route to User A Notes: VoIP devices on the Internet cannot make calls to private address (where to send them?) do not know the type of NAT being used (cone, symmetric and so on), so they do not know about what kinds of bindings to use do not know if the bindings are still open

35 Sip trace SIP Signaling SDP Signaling Internal IP address 001 SIP/ Via: SIP/2.0/UDP :5060;branch=a71b6d57-507c77f2 003 Via: SIP/2.0/UDP :5060;received= ;rport= From: 005 To: sip: 006 Contact: sip: 007 Call-ID: 008 CSeq: Content-Length: 138 External IP address 010 Content-Type: application/sdp seen by SIP proxy 011 User-Agent: HearMe SoftPHONE from outside v=0 014 o=deltathree0 0 IN IP s=deltathree 016 c=in IP t=0 0 Internal IP address for RTP stream 018 m=audio 3456 RTP/AVP a=ptime: a=x-ssrc:00aea3c0 Symmetric RTP Classical RTP is unidirectional (i.e.two RTP sessions, one in each direction) Endpoints use UDP port symmetry to establish bi-directional traffic Sending and receiving ports for the RTP and RTCP traffic should be the same on the endpoint behind the NAT/Firewall Usage require that endpoints use UDP port symmetry to establish bi-directional traffic Solution to NAT Traversal Let clients be aware of their external IP:PORT Ask the NAT Universal Plug and Play (UPnP) Ask someone outside the NAT Simple Traversal of UDP Through NATs (STUN) Hole Punching Traversal Using Relay NAT (TURN) Interactive Connectivity Establishment (ICE) Make NAT & Firewall SIP friendly Application Layer Gateway Universal Plug and Play (UPnP) Proposed by Microsoft Client talks with NAT gateway and ask about IP and ports Will NOT work with cascading NAT

36 Universal Plug and Play (UPnP) Solution to NAT/Firewall Traversal Source : What is my IP:Port assigned : Will NOT work with cascading NAT for security issue Let clients be aware of their external IP:PORT Ask the NAT Universal Plug and Play (UPnP) Ask someone outside the NAT Simple Traversal of UDP Through NATs (STUN) Hole Punching Traversal Using Relay NAT (TURN) Interactive Connectivity Establishment (ICE) Make NAT & Firewall SIP friendly Application Layer Gateway STUN Simple Traversal of UDP Through NATs Types of NATs Full Cone (Address) Restricted Cone Increasing Port Restricted Cone security Symmetric Not suitable for Symmetric NAT Types of NATs: Full Cone Client A :8000 NAT translation table LAN side addr WAN side addr , , All the incoming traffic can get through from the pinhole to client A if they know the IP : Port mapping :7868 Client C :7865 Client B :6988 Client D

37 Types of NATs: (Address) Restricted Cone NAT translation table LAN side addr WAN side addr , ,9000 Client A : Traffic from B with different source ports can get through : :2134 Filter traffic only by IP : block incoming traffic from other IP (client C and D) address. Incoming traffic from same authorized IP but different ports will be accepted :7868 Client C :7865 Client B :6988 Client D Types of NATs: Port Restricted Cone Client A :8000 NAT translation table LAN side addr WAN side addr , , : :2134 Filter by both IP and Port: set up one to many mapping :7868 Client C :7865 Client B :6988 Client D Types of NATs: Port Restricted Cone (con t) Types of NATs: Symmetric Only one entry is set up in the table for :8000 to different clients outside Client A NAT translation table LAN side addr WAN side addr , , : :7868 Client C :7865 Client A NAT translation table LAN side addr WAN side addr , , , , : :7868 Client C : : :2134 Client B : :9001 Client B Filter by both IP and Port: set up only one entry for multiple remote clients :6988 Client D Filter by both IP and Port, NAT assign a mapping for each source-destination pair :6988 Client D

38 External Query STUN Ask a server on the Internet what I look like Compared the returned answer (external address) with my own address (local internal address) Put my real address in signaling to allow media traffic in This works IF: The client send and receive RTP on the same port (why?) SIP request must be sent immediately. After a while the mapping might change In the case of Address Restricted Cone or Port Restricted Cone it must send out data to the other end first Symmetric NAT case NAT translation table LAN side addr WAN side addr , , , , 9001 STUN Solution 1. Send 3. Incoming query to media 2. STUN Put get assigned server through IP:Port to ask with IP:Port the in SDP informed assigned IP:Port by NAT Useless for symmetric NAT since holes punctured by STUN can not be used by others Assigned for STUN by NAT Assigned for RTP by NAT, RTP is only authorized to get through using 9001 but not 9000 due to the NAT/Firewall combination STUN (Cont ) With the information sent by STUN, client can determine If it is on the open Internet If it is behind a firewall that blocks UDP If it is behind a NAT and what type of NAT it is behind Will NOT work for symmetric NAT Typical in Large Enterprise STUN: Automatic Detection of NAT Environment [1/2] STUN Client Environment Port1 STUN Server Port2 IP1 Port2 STUN Server Port1 IP2 Test I Test II Test III Test IV

39 STUN: Automatic Detection of NAT Environment [2/2] UDP Hole Punching Symmetric NAT No UDP Blocked Same IP and Port as Test I? Yes Test IV Resp? No Port Restricted NAT Yes No Test I Resp? Yes Test III Restricted NAT Same Yes IP and Port as original? No Test II No Resp? Yes Full Cone NAT Test II Resp? Yes Open Internet No Sym UDP Firewall method for establishing bidirectional UDP connections between host in private networks using NAT third party hosts on the public network are used to establish UDP port states to be used for direct communications between the hosts. Once port state has been established, port state may be maintained by either normal traffic, or by keep alive packets Full Cone-Full Cone Full cone/restrictedrestricted

40 TURN TURN Solution Solve Symmetric NAT case by allowing Media Flows Through TURN Server directly Not the case with STUN servers Increase voice latency Increase probability of packet loss Few SIP clients support TURN today (complex and not-yet a standard) No free TURN server available (only commercial) Skype seems to support TURN NAT translation table LAN side addr WAN side addr , , 9000 Media makes use of the hole punctured by TURN directly Assigned for both TURN and SIP by NAT TURN Overview TURN Works similar to STUN 1. Client sends IP/port request to TURN server 2. TURN server provides one that is a local interface 3. TURN server receives media on that IP/port 4. Forward it to IP/port where TURN request came from 5. Will get routed back to client :6554 Client N AN TA T Give me IP :3884 RTP to :8877 NAT rewrites Source to :8877 TURN Server RTP to :3884 Interactive Connectivity Establishment A Client Has Many Addresses at Which It Can Receive Media o Local interfaces o IP Addresses learned from STUN o IP Addresses learned from TURN Which One(s) Will Work When Talking to a Specific Peer? NO WAY TO KNOW AHEAD OF TIME Try Each of Them o Each side uses a connectivity check to see if It can reach a specific address provided by the peer o Choose The Best Address That Works

41 ICE Learns about the network topology in which the clients exist and the various sets of network addresses by which these devices can communicate Framework to unify the various NAT traversal techniques STUN, TURN and Realm Specific IP (RSIP) Benefits from the collective functionality of each while avoiding any one protocol's drawback ICE Properties ICE Always will find a means for communicating if one physically exists Always finds the communications path with fewest relays Always finds the communication path cheapest for the service provider Does not require any knowledge of topology, NAT types, or anything Can guarantee that the phone won t ring unless audio works when you pickup TURN/ STUN TURN/ STUN TURN/ STUN TURN/ STUN Proxy Proxy Proxy Proxy NAT NAT NAT NAT X:Y Caller Callee X:Y Caller Caller gets STUN And TURN addresses From server STUN: A:B TURN: C:D Callee

42 TURN/ STUN TURN/ STUN TURN/ STUN TURN/ STUN Proxy Proxy Proxy Proxy NAT NAT NAT NAT STUN: A:B TURN: C:D Local: X:Y Caller O=IN IP4 C M=audio D RTP/AVP 0 A=candidate: UDP A:B A=candidate: UDP C:D A=candidate: UDP X:Y Callee STUN: A:B TURN: C:D Local: X:Y Caller Callee gets STUN And TURN addresses From server STUN: E:F TURN: G:H Callee U:V TURN/ STUN TURN/ STUN TURN/ STUN C:D G:H TURN/ STUN Proxy Proxy Proxy Proxy NAT NAT NAT NAT STUN: A:B TURN: C:D Local: X:Y Caller O=IN IP4 G M=audio H RTP/AVP 0 A=candidate: UDP E:F A=candidate: UDP G:H A=candidate: UDP U:V Callee STUN: E:F TURN: G:H Local: U:V STUN: A:B TURN: C:D Local: X:Y Caller X:Y Media starts flowing immediately to the c/m value of the peer U:V Callee STUN: E:F TURN: G:H Local: U:V

43 TURN/ STUN TURN/ STUN TURN/ STUN TURN/ STUN Proxy Proxy Proxy Proxy NAT NAT NAT NAT STUN: A:B TURN: C:D Local: X:Y Caller O=IN IP4 A M=audio B RTP/AVP 0 A=candidate: UDP A:B A=candidate: UDP C:D A=candidate: UDP X:Y Callee STUN: A:B TURN: C:D Local: X:Y Caller O=IN IP4 E M=audio F RTP/AVP 0 A=candidate: UDP E:F A=candidate: UDP G:H A=candidate: UDP U:V Callee STUN: E:F TURN: G:H Local: U:V Solution to NAT Traversal TURN/ STUN TURN/ STUN Let clients be aware of their external IP:PORT Proxy Proxy Ask the NAT Universal Plug and Play (UPnP) Ask someone outside the NAT NAT A:B E:F NAT Simple Traversal of UDP Through NATs (STUN) Hole Punching STUN: A:B TURN: C:D Local: X:Y Caller X:Y Media starts flowing to the c/m value of the peer U:V Callee STUN: E:F TURN: G:H Local: U:V Traversal Using Relay NAT (TURN) Interactive Connectivity Establishment (ICE) Make NAT & Firewall SIP aware Application Layer Gateway

44 Application Layer Gateway Make Firewall/NAT SIP aware Analyze the address information inside the packet payload and dynamically open or close holes for media communications Needed to be updated for each new application which restricts it for large corporate networks No commercial SIP ALGs today Application Layer Gateway Solution Understanding the signaling messages and their relationship with resulting media flows ---- Media Friendly