Multimedia Streaming Internet Streaming Media Reji Mathew NICTA & CSE UNSW COMP9519 Multimedia Systems S2 2006 preferred for streaming System Overview Protocol stack Protocols + SDP SIP Encoder Side Issues Receiver Side Issues File Format : MP4 COMP9519 Multimedia Systems Overview Part 2 Slide 2 Reji Mathew Introduction Facts on (User Datagram Protocol) Datagram mode of communication Send messages with a minimum of protocol mechanism Unordered datagram service delivery and duplicate protection are not guaranteed is unreliable But used when timeliness is more important you have control over when data is sent Preferred for streaming TCP is more reliable Used when reliability is more important (than timeliness) Eg, file download Introduction /IP preferred (over TCP/IP) for streaming But multimedia data have other requirements Sequencing: to reorder packets at the receiver that are out of order to detect lost packets Intra-media synchronization: Inter-media synchronization: Payload identification: Communicate the codec being currently used Allow codec switching to adjust to bandwidth or user capabilities Frame indication: To indicate start and end of a frame or access unit Helps in delivery of data to higher layers COMP9519 Multimedia Systems Overview Part 2 Slide 3 Reji Mathew COMP9519 Multimedia Systems Overview Part 2 Slide 4 Reji Mathew
Protocol Stack File / Encoder packetize Video + Protocols for Enabling Streaming the real-time transport protocol, to carry data that has real-time properties. SDP SDP to monitor the quality of service and to convey information about the participants in an on-going session. Ref : IETF Request for Comments: 3550 TCP TCP COMP9519 Multimedia Systems Overview Part 2 Slide 5 Reji Mathew IP COMP9519 Multimedia Systems Overview Part 2 Slide 6 Reji Mathew Header Fields - System Overview compressed bit stream packetize Reorder packets Identify lost packets Detect payload type Detect participants Packets to Frames Attach Header Time stamping Sequence numbering Payload ID Frame indication Source ID The first twelve bytes (octets) are present in every packet. IP COMP9519 Multimedia Systems Overview Part 2 Slide 7 Reji Mathew COMP9519 Multimedia Systems Overview Part 2 Slide 8 Reji Mathew
Need some feedback on network performance Real-Time Control Protocol () Used in conjunction with Primary function feedback on quality of data distribution Control information exchanged between session members by packets Five packet types defined packets begin with a fixed part Similar to data packets packets are sent periodically from all participants of a session multicast example packets are stackable compound packets concatenate multiple packets Usually sent over (same as ) Using a port number = _port + 1 COMP9519 Multimedia Systems Overview Part 2 Slide 9 Reji Mathew COMP9519 Multimedia Systems Overview Part 2 Slide 10 Reji Mathew packet types to carry control information SDES : Source description items sent by all session members includes a persistent identifier of sources in the session SR : Sender Report sent by active senders report of transmission and reception statistics RR : Receiver Report Sent by receivers (not active senders) report of reception statistics BYE : Packet to indicate end of participation APP : Application specific functions COMP9519 Multimedia Systems Overview Part 2 Slide 11 Reji Mathew + : System Overview Video AMR Audio packetize packetize IP Audio COMP9519 Multimedia Systems Overview Part 2 Slide 12 Reji Mathew Synchronized Video and AMR Audio Presentation Video
System Overview File Receiver / Encoder Side Issues -Buffering input packetize data -Error concealment -Inter media synchronization Sender Side Issues Video -Conforming to Network Bandwidth -Adaptive Rate Control -Error Resilience -Packetization strategy System Overview : Sender Side Issues Packetization Strategy Example : fragment and packetize video Must meet network MTU limits Example : MTU = 1500 bytes Packets larger than MTU will result in IP fragmentation Resulting in overhead for each fragmented packet Loss of one fragment will corrupt the whole packet COMP9519 Multimedia Systems Overview Part 2 Slide 13 Reji Mathew IP : Try to allocate a single VOP (frame) per packet If packet is greater than MTU then break into multiple packets Can break any where - arbitrary byte positions Best to break at GOB or slice boundaries COMP9519 Multimedia Systems Overview Part 2 Slide 14 Reji Mathew System Overview : Sender Side Issues Packetization Strategy (continued) Minimize number of packets required Less packets means less bits spend on packet headers Example : 20 byte IP Header 8 byte Header 12 byte Header Minimize dependency between packets So that even if one packet is lost, the next packet can still be decoded Example : use VOP boundaries to start / stop packets System Overview : Sender Side Issues Fragmentation Strategy Minimize dependency between packets (continued) VOP 1 VOP 2 VOP 3 loss VOP 1 VOP 2 VOP 3 VOP 1 VOP 2 VOP 3 COMP9519 Multimedia Systems Overview Part 2 Slide 15 Reji Mathew COMP9519 Multimedia Systems Overview Part 2 Slide 16 Reji Mathew
System Overview : Sender Side Issues Procedure to fragment and packetize live coded video or stored video. Goal try to packetize one Frame per packet 1. Calculate Max allowable size of frame : Max_FrameSize = MTU Where = _header + _header + IP_header 2. Find size of frame N from the bit stream : FrameSize 3. IF (FrameSize <= Max_FrameSize) THEN include video frame in one packet 4. ELSE Break Frame into multiple packets Best to break at GOB or slice boundaries Example break at a slice boundary to form 2 smaller packets 5. N = N + 1 : go back to Step 2 System Overview : Receiver Side Issues Buffering Input Data Number of Packets Constant Transmit Rate Packets Generated T N T B Packets Received Mean Receive Rate Time T N T B Received Data Played out Constant Play Out Rate : Network transfer delay : Buffer Delay COMP9519 Multimedia Systems Overview Part 2 Slide 17 Reji Mathew COMP9519 Multimedia Systems Overview Part 2 Slide 18 Reji Mathew System Overview : Receiver Side Issues Inter Media Synchronization Audio De-packetizer and Video De-packetizer and 1600 1760 1920 2080 2240 SR 1680 : 15:2:32.00 18000 108000 117000 SR 106200 : 15:2:32.01 COMP9519 Multimedia Systems Overview Part 2 Slide 19 Reji Mathew Audio clock = 8000Hz (eg u-law) 1760 = 32.00 + 0.01 1920 = 32.00 + 0.03 2240 = 32.00 + 0.07 ---------------------------- Video clock = 90000Hz (eg ) 108000 = 32.01 + 0.02 117000 = 32.01 + 0.12 Further Protocols Enabling Streaming SDP Description protocol for control over the delivery of data with real-time properties. IETF Request for Comments: 2326 SIP - Session Initiation Protocol COMP9519 Multimedia Systems Overview Part 2 Slide 20 Reji Mathew
SDP SDP includes Session name and purpose Time the session is active The media comprising the session Information to receive media (addresses, ports, formats) Information about bandwidth to be used Contact information of a person responsible for the session SDP is used by other signaling / initiation protocols SIP : Session Initiation Protocol : Real-time Streaming Protocol SDP v=0 o=mmvc 2890844526 2890842807 IN IP4 129.94.135.201 s=camera ONE i=video stream for realtime surveillance u=http://www.nicta.com/mmvc/demos/surveillancevideo.pdf e=rmathew@nicta.com.au (Reji Mathew) c=in IP4 225.0.0.37/2 t=0 0 a=recvonly m=video 20000 /AVP 98 a=rtpmap:98 MP4V-ES/90000 a=fmtp:98 profile-level-id=1; config=000001b001000001 a=orient:portrait COMP9519 Multimedia Systems Overview Part 2 Slide 21 Reji Mathew COMP9519 Multimedia Systems Overview Part 2 Slide 22 Reji Mathew How to control streams Start, Stop, Pause, Fast Froward, Rewind internet VCR Solution Real Time Streaming Protocol Establishes and controls one or more continuous media streams - such as audio and video Similar in syntax and operation to HTTP/1.1 Client Server protocol Text based IETF RFC 2326 COMP9519 Multimedia Systems Overview Part 2 Slide 23 Reji Mathew Protocol Operation Text based messages between client and server Messages can be : Requests s Example : Play Play Request (TCP/IP) Play (TCP/IP) Media Server /1.0 200 OK CSeq: 833 Date: 23 Jan 1997 15:35:06 GMT Range: smpte=0:10:22-;time=19970123t153600z PLAY rtsp://audio.example.com/twister.en /1.0 CSeq: 833 Session: 12345678 Range: smpte=0:10:20-;time=19970123t153600z COMP9519 Multimedia Systems Overview Part 2 Slide 24 Reji Mathew Client
Summary of Methods OPTIONS : SETUP : ANNOUNCE : DESCRIBE : PLAY : RECORD : REDIRECT : PAUSE : get available methods establish transport change description of media object get description of media start playback, reposition start recording redirect client to new server pause delivery, while keeping state SET PARAMETER : device or encoding control TEARDOWN : remove state / close COMP9519 Multimedia Systems Overview Part 2 Slide 25 Reji Mathew : Describe Method Retrieves the description of a presentation, identified by the request URL, from a server. The DESCRIBE reply-response pair constitutes the media initialization phase of. C->S: DESCRIBE rtsp://nicta.com/twister /1.0 CSeq: 1 S->C: /1.0 200 OK CSeq: 1 Content-Type: application/sdp Content-Length: 164 v=0 o=- 2890844256 2890842807 IN IP4 172.16.2.93 s= Session i=an Example of Session Usage a=control:rtsp://nicta.com/twister t=0 0 m=audio 0 /AVP 0 a=control:rtsp://nicta.com/twister/audio m=video 0 /AVP 26 a=control:rtsp://nicta.com/twister/video COMP9519 Multimedia Systems Overview Part 2 Slide 26 Reji Mathew : Setup Method Specifies the transport mechanism to be used for the streamed media C->S: SETUP rtsp://nicta.com/twister/audio /1.0 CSeq: 2 Transport: /AVP;unicast;client_port=8000-8001 S->C: /1.0 200 OK CSeq: 2 S->C: /1.0 200 OK CSeq: 3 Transport: /AVP;unicast;client_port=8002-8003; server_port=9004-9005 Session: 12345678 COMP9519 Multimedia Systems Overview Part 2 Slide 27 Reji Mathew unicast / port pair on which the client has chosen to receive media data and control information Transport: /AVP;unicast;client_port=8000-8001; server_port=9000-9001 Session: 12345678 session ID unicast / port C->S: SETUP rtsp://nicta.com/twister/video /1.0 pair on which the server CSeq: 3 has chosen to send Transport: /AVP;unicast;client_port=8002-8003 media data and control Session: 12345678 information Setup operates on each stream separately Same session ID used : Signal Timing Diagram Media Client DESCRIBE (SDP) SETUP (Video stream) SETUP (Audio stream) PLAY PAUSE Media Server TEARDOWN COMP9519 Multimedia Systems Overview Part 2 Slide 28 Reji Mathew
SIP Session Initiation Protocol (SIP) Another session control protocol IETF RFC 3261 www.ietf.org/rfc/rfc3261.txt Protocol that can establish, modify and terminate multimedia sessions. Applications IP telephony, multimedia conferences SIP, like, Uses text-based request/response transaction model Requests contain Methods and Header fields s include 3 digit status codes (eg 200 OK ) Is Transport layer independent Uses other protocols for media delivery (eg ). SIP SIP Methods INVITE, ACK, CANCEL : for setting up sessions BYE : for terminating sessions REGISTER : for registering contact information OPTIONS : for querying servers about capabilities INVITE User agent client initiates a session with INVITE Invitation can be accepted by a user agent server SIP invitations convey session descriptions that allow participants to agree on a set of compatible media types Offer/answer model ACK confirmation to INVITE COMP9519 Multimedia Systems Overview Part 2 Slide 29 Reji Mathew COMP9519 Multimedia Systems Overview Part 2 Slide 30 Reji Mathew SIP SIP supports user mobility With proxy servers to help route requests to user s current location Registration function allowing users to upload current locations for use by proxy servers File Format MPEG-1 & MPEG-2 content typically exchanged as files that represent a stream ready to be delivered Embedded absolute time stamps Fragmentation of media for some preferred transport Random Access could be difficult file format : MP4 life-cycle file format defined as part of Based on Apple s QuickTime format ( mov files ) Due to MP4 s flexibility and extensibility Industry acceptance for exchange of video and audio Basis for Motion JPEG 2000 file format Basis for 3GPP file format used in MMS Multimedia Messaging Service COMP9519 Multimedia Systems Overview Part 2 Slide 31 Reji Mathew COMP9519 Multimedia Systems Overview Part 2 Slide 32 Reji Mathew
MP4 : Main File Format Concepts The Media data is stored separately from Meta data Media data : Audio, Video samples Metadata : Data describing the media Examples : timing info, number of bytes required for a frame Timing information specified by relative numbers (durations) rather than absolute numbers Allows editing to be easier eg insertion of a new frame Able to store media data distributed over several files Use URLs to point to media data stored at various locations MP4 : Main File Format Concepts The Media data is stored separately from Meta data Timing information specified by relative numbers (durations) rather than absolute numbers Able to store media data distributed over several files Locating media data by means of data offsets and length information Metadata tables mapping media sample number to location in a file Support streaming protocols through optional hint tracks Metadata information for packetization and header data Example hints for streaming stored as a separate track COMP9519 Multimedia Systems Overview Part 2 Slide 33 Reji Mathew COMP9519 Multimedia Systems Overview Part 2 Slide 34 Reji Mathew MPEG-7 System : Client Server architecture MPEG-7 Indexing & Searching: Semantics-based (people, places, events, objects, scenes) Content-based (color, texture, motion, melody, timbre) Metadata (title, author, dates) Search Engine sounds like, looks like Query target MPEG-7 Database Query List of matching content clip1.mp4 clip2.mp4 Request for Content Media Server Streaming Media / COMP9519 Multimedia Systems Overview Part 2 Slide 35 Reji Mathew