Internet Streaming Media Reji Mathew NICTA & CSE UNSW COMP9519 Multimedia Systems S2 2006
Multimedia Streaming UDP preferred for streaming System Overview Protocol stack Protocols RTP + RTCP SDP RTSP SIP Encoder Side Issues Receiver Side Issues File Format : MP4 COMP9519 Multimedia Systems Overview Part 2 Slide 2 Reji Mathew
Introduction Facts on UDP (User Datagram Protocol) Datagram mode of communication Send messages with a minimum of protocol mechanism Unordered datagram service delivery and duplicate protection are not guaranteed UDP is unreliable But used when timeliness is more important you have control over when data is sent Preferred for streaming TCP is more reliable Used when reliability is more important (than timeliness) Eg, file download COMP9519 Multimedia Systems Overview Part 2 Slide 3 Reji Mathew
Introduction UDP/IP preferred (over TCP/IP) for streaming But multimedia data have other requirements Sequencing: to reorder packets at the receiver that are out of order to detect lost packets Intra-media synchronization: Inter-media synchronization: Payload identification: Communicate the codec being currently used Allow codec switching to adjust to bandwidth or user capabilities Frame indication: To indicate start and end of a frame or access unit Helps in delivery of data to higher layers COMP9519 Multimedia Systems Overview Part 2 Slide 4 Reji Mathew
Protocol Stack MPEG-4 File / Encoder packetize MPEG-4 Video RTCP RTP SDP RTSP SDP RTSP RTCP Decoder RTP UDP TCP TCP UDP IP COMP9519 Multimedia Systems Overview Part 2 Slide 5 Reji Mathew
RTP + RTCP Protocols for Enabling Streaming RTP the real-time transport protocol, to carry data that has real-time properties. RTCP to monitor the quality of service and to convey information about the participants in an on-going session. Ref : IETF Request for Comments: 3550 COMP9519 Multimedia Systems Overview Part 2 Slide 6 Reji Mathew
RTP Header Fields The first twelve bytes (octets) are present in every RTP packet. COMP9519 Multimedia Systems Overview Part 2 Slide 7 Reji Mathew
RTP - System Overview compressed bit stream Reorder packets Identify lost packets Detect payload type Detect participants Packets to Frames packetize RTP Attach RTP Header Time stamping Sequence numbering Payload ID Frame indication Source ID Decoder RTP UDP UDP IP COMP9519 Multimedia Systems Overview Part 2 Slide 8 Reji Mathew
RTCP Need some feedback on network performance Real-Time Control Protocol (RTCP) Used in conjunction with RTP Primary function feedback on quality of data distribution COMP9519 Multimedia Systems Overview Part 2 Slide 9 Reji Mathew
RTCP Control information exchanged between session members by RTCP packets Five RTCP packet types defined RTCP packets begin with a fixed part Similar to RTP data packets RTCP packets are sent periodically from all participants of a session multicast example RTCP packets are stackable compound packets concatenate multiple packets Usually sent over UDP (same as RTP) Using a port number = RTP_port + 1 COMP9519 Multimedia Systems Overview Part 2 Slide 10 Reji Mathew
RTCP RTCP packet types to carry control information RTCP SDES : Source description items sent by all session members includes a persistent identifier of sources in the session RTCP SR : Sender Report sent by active senders report of transmission and reception statistics RTCP RR : Receiver Report Sent by receivers (not active senders) report of reception statistics RTCP BYE : Packet to indicate end of participation RTCP APP : Application specific functions COMP9519 Multimedia Systems Overview Part 2 Slide 11 Reji Mathew
RTP + RTCP : System Overview MPEG-4 Video AMR Audio Synchronized MPEG-4 Video and AMR Audio Presentation packetize packetize RTCP RTP RTCP RTP Audio Decoder Video Decoder RTCP RTP RTCP RTP UDP UDP IP COMP9519 Multimedia Systems Overview Part 2 Slide 12 Reji Mathew
System Overview MPEG-4 File Receiver / Encoder Side Issues -Buffering input packetize data -Error concealment -Inter media synchronization RTCP RTP Sender Side Issues MPEG-4 Video -Conforming to Network Bandwidth -Adaptive Rate Control -Error Resilience Decoder -Packetization strategy RTCP RTP UDP UDP COMP9519 Multimedia Systems Overview Part 2 Slide 13 Reji Mathew IP
System Overview : Sender Side Issues Packetization Strategy Example : fragment and packetize MPEG-4 video Must meet network MTU limits Example : MTU = 1500 bytes Packets larger than MTU will result in IP fragmentation Resulting in overhead for each fragmented packet Loss of one fragment will corrupt the whole packet MPEG-4 : Try to allocate a single VOP (frame) per packet If packet is greater than MTU then break into multiple packets Can break any where - arbitrary byte positions Best to break at GOB or slice boundaries COMP9519 Multimedia Systems Overview Part 2 Slide 14 Reji Mathew
System Overview : Sender Side Issues Packetization Strategy (continued) Minimize number of packets required Less packets means less bits spend on packet headers Example : 20 byte IP Header 8 byte UDP Header 12 byte RTP Header Minimize dependency between packets So that even if one packet is lost, the next packet can still be decoded Example : MPEG-4 use VOP boundaries to start / stop packets COMP9519 Multimedia Systems Overview Part 2 Slide 15 Reji Mathew
System Overview : Sender Side Issues Fragmentation Strategy Minimize dependency between packets (continued) VOP 1 VOP 2 VOP 3 loss VOP 1 VOP 2 VOP 3 VOP 1 VOP 2 VOP 3 COMP9519 Multimedia Systems Overview Part 2 Slide 16 Reji Mathew
System Overview : Sender Side Issues Procedure to fragment and packetize live MPEG-4 coded video or stored MPEG-4 video. Goal try to packetize one Frame per RTP packet 1. Calculate Max allowable size of frame : Max_FrameSize = MTU Where = RTP_header + UDP_header + IP_header 2. Find size of frame N from the bit stream : FrameSize 3. IF (FrameSize <= Max_FrameSize) THEN include video frame in one RTP packet 4. ELSE Break Frame into multiple packets Best to break at GOB or slice boundaries Example break at a slice boundary to form 2 smaller packets 5. N = N + 1 : go back to Step 2 COMP9519 Multimedia Systems Overview Part 2 Slide 17 Reji Mathew
System Overview : Receiver Side Issues Buffering Input Data Constant Transmit Rate Mean Receive Rate Constant Play Out Rate Number of Packets Packets Generated Packets Received Received Data Played out T N T B : Network transfer delay : Buffer Delay T N T B Time COMP9519 Multimedia Systems Overview Part 2 Slide 18 Reji Mathew
System Overview : Receiver Side Issues Inter Media Synchronization Audio De-packetizer and Decoder Video De-packetizer and Decoder 1600 1760 1920 2080 2240 SR 1680 : 15:2:32.00 18000 108000 117000 SR 106200 : 15:2:32.01 Audio clock = 8000Hz (eg u-law) 1760 = 32.00 + 0.01 1920 = 32.00 + 0.03 2240 = 32.00 + 0.07 ---------------------------- Video clock = 90000Hz (eg MPEG-4) 108000 = 32.01 + 0.02 117000 = 32.01 + 0.12 COMP9519 Multimedia Systems Overview Part 2 Slide 19 Reji Mathew
Further Protocols Enabling Streaming SDP Description RTSP protocol for control over the delivery of data with real-time properties. IETF Request for Comments: 2326 SIP - Session Initiation Protocol COMP9519 Multimedia Systems Overview Part 2 Slide 20 Reji Mathew
SDP SDP includes Session name and purpose Time the session is active The media comprising the session Information to receive media (addresses, ports, formats) Information about bandwidth to be used Contact information of a person responsible for the session SDP is used by other signaling / initiation protocols SIP : Session Initiation Protocol RTSP : Real-time Streaming Protocol COMP9519 Multimedia Systems Overview Part 2 Slide 21 Reji Mathew
SDP v=0 o=mmvc 2890844526 2890842807 IN IP4 129.94.135.201 s=camera ONE i=video stream for realtime surveillance u=http://www.nicta.com/mmvc/demos/surveillancevideo.pdf e=rmathew@nicta.com.au (Reji Mathew) c=in IP4 225.0.0.37/2 t=0 0 a=recvonly m=video 20000 RTP/AVP 98 a=rtpmap:98 MP4V-ES/90000 a=fmtp:98 profile-level-id=1; config=000001b001000001 a=orient:portrait COMP9519 Multimedia Systems Overview Part 2 Slide 22 Reji Mathew
RTSP How to control streams Start, Stop, Pause, Fast Froward, Rewind internet VCR Solution RTSP Real Time Streaming Protocol Establishes and controls one or more continuous media streams - such as audio and video Similar in syntax and operation to HTTP/1.1 Client Server protocol Text based IETF RFC 2326 COMP9519 Multimedia Systems Overview Part 2 Slide 23 Reji Mathew
RTSP Protocol Operation Text based messages between client and server Messages can be : Requests Responses Example : Play PLAY rtsp://audio.example.com/twister.en RTSP/1.0 CSeq: 833 Session: 12345678 Range: smpte=0:10:20-;time=19970123t153600z Play Request (TCP/IP) Play Response (TCP/IP) Media Server RTSP/1.0 200 OK CSeq: 833 Date: 23 Jan 1997 15:35:06 GMT Range: smpte=0:10:22-;time=19970123t153600z Client COMP9519 Multimedia Systems Overview Part 2 Slide 24 Reji Mathew
Summary of RTSP Methods OPTIONS : SETUP : ANNOUNCE : DESCRIBE : PLAY : RECORD : REDIRECT : PAUSE : get available methods establish transport change description of media object get description of media start playback, reposition start recording redirect client to new server pause delivery, while keeping state SET PARAMETER : device or encoding control TEARDOWN : remove state / close COMP9519 Multimedia Systems Overview Part 2 Slide 25 Reji Mathew
RTSP : Describe Method Retrieves the description of a presentation, identified by the request URL, from a server. The DESCRIBE reply-response pair constitutes the media initialization phase of RTSP. C->S: DESCRIBE rtsp://nicta.com/twister RTSP/1.0 CSeq: 1 S->C: RTSP/1.0 200 OK CSeq: 1 Content-Type: application/sdp Content-Length: 164 v=0 o=- 2890844256 2890842807 IN IP4 172.16.2.93 s=rtsp Session i=an Example of RTSP Session Usage a=control:rtsp://nicta.com/twister t=0 0 m=audio 0 RTP/AVP 0 a=control:rtsp://nicta.com/twister/audio m=video 0 RTP/AVP 26 a=control:rtsp://nicta.com/twister/video COMP9519 Multimedia Systems Overview Part 2 Slide 26 Reji Mathew
RTSP : Setup Method Specifies the transport mechanism to be used for the streamed media C->S: SETUP rtsp://nicta.com/twister/audio RTSP/1.0 CSeq: 2 Transport: RTP/AVP;unicast;client_port=8000-8001 S->C: RTSP/1.0 200 OK CSeq: 2 S->C: RTSP/1.0 200 OK CSeq: 3 Transport: RTP/AVP;unicast;client_port=8002-8003; server_port=9004-9005 Session: 12345678 COMP9519 Multimedia Systems Overview Part 2 Slide 27 Reji Mathew unicast RTP/RTCP port pair on which the client has chosen to receive media data and control information Transport: RTP/AVP;unicast;client_port=8000-8001; server_port=9000-9001 Session: 12345678 RTSP session ID unicast RTP/RTCP port C->S: SETUP rtsp://nicta.com/twister/video RTSP/1.0 pair on which the server CSeq: 3 has chosen to send Transport: RTP/AVP;unicast;client_port=8002-8003 media data and control Session: 12345678 information Setup operates on each stream separately Same session ID used
RTSP : Signal Timing Diagram Media Client DESCRIBE Response (SDP) SETUP (Video stream) Response SETUP (Audio stream) Response PLAY Response PAUSE Media Server Response TEARDOWN Response COMP9519 Multimedia Systems Overview Part 2 Slide 28 Reji Mathew
SIP Session Initiation Protocol (SIP) Another session control protocol IETF RFC 3261 www.ietf.org/rfc/rfc3261.txt Protocol that can establish, modify and terminate multimedia sessions. Applications IP telephony, multimedia conferences SIP, like RTSP, Uses text-based request/response transaction model Requests contain Methods and Header fields Responses include 3 digit status codes (eg 200 OK ) Is Transport layer independent Uses other protocols for media delivery (eg RTP). COMP9519 Multimedia Systems Overview Part 2 Slide 29 Reji Mathew
SIP SIP Methods INVITE, ACK, CANCEL : for setting up sessions BYE : for terminating sessions REGISTER : for registering contact information OPTIONS : for querying servers about capabilities INVITE User agent client initiates a session with INVITE Invitation can be accepted by a user agent server SIP invitations convey session descriptions that allow participants to agree on a set of compatible media types Offer/answer model ACK Response confirmation to INVITE COMP9519 Multimedia Systems Overview Part 2 Slide 30 Reji Mathew
SIP SIP supports user mobility With proxy servers to help route requests to user s current location Registration function allowing users to upload current locations for use by proxy servers COMP9519 Multimedia Systems Overview Part 2 Slide 31 Reji Mathew
File Format MPEG-1 & MPEG-2 content typically exchanged as files that represent a stream ready to be delivered Embedded absolute time stamps Fragmentation of media for some preferred transport Random Access could be difficult MPEG-4 file format : MP4 life-cycle file format defined as part of MPEG-4 Based on Apple s QuickTime format ( mov files ) Due to MP4 s flexibility and extensibility Industry acceptance for exchange of MPEG-4 video and audio Basis for Motion JPEG 2000 file format Basis for 3GPP file format used in MMS Multimedia Messaging Service COMP9519 Multimedia Systems Overview Part 2 Slide 32 Reji Mathew
MP4 : Main File Format Concepts The Media data is stored separately from Meta data Media data : Metadata : Audio, Video samples Data describing the media Examples : timing info, number of bytes required for a frame Timing information specified by relative numbers (durations) rather than absolute numbers Allows editing to be easier eg insertion of a new frame Able to store media data distributed over several files Use URLs to point to media data stored at various locations COMP9519 Multimedia Systems Overview Part 2 Slide 33 Reji Mathew
MP4 : Main File Format Concepts The Media data is stored separately from Meta data Timing information specified by relative numbers (durations) rather than absolute numbers Able to store media data distributed over several files Locating media data by means of data offsets and length information Metadata tables mapping media sample number to location in a file Support streaming protocols through optional hint tracks Metadata information for packetization and header data Example hints for RTP streaming stored as a separate track COMP9519 Multimedia Systems Overview Part 2 Slide 34 Reji Mathew
MPEG-7 System : Client Server architecture MPEG-7 Indexing & Searching: Semantics-based (people, places, events, objects, scenes) Content-based (color, texture, motion, melody, timbre) Metadata (title, author, dates) Search Engine Query target sounds like, looks like MPEG-7 Database Query Response List of matching content clip1.mp4 clip2.mp4 Request for Content RTSP Media Server Streaming Media RTP/RTCP COMP9519 Multimedia Systems Overview Part 2 Slide 35 Reji Mathew