Lecture 7: Internet Streaming Media. Reji Mathew NICTA & CSE UNSW COMP9519 Multimedia Systems S2 2007

Transcription

1 Lecture 7: Internet Streaming Media Reji Mathew NICTA & CSE UNSW COMP9519 Multimedia Systems S2 2007

2 Notes on Previous Lecture RTCP Packets SR and RR can be used for independent network management Payload independent monitors that receive only the RTCP packets to evaluate the performance of their networks for multicast distribution COMP9519 Multimedia Systems Lecture 7 Slide 2 Reji Mathew

3 Introduction The story so far. Video Coding MPEG-4 Audio Coding u-law, AMR Packetization and Transport of media RTP Transport Control / QoS Feedback - RTCP Lecture Outline Literature Review Client Architecture : An Example Next Further Protocols Enabling Streaming SDP - Description RTSP Signaling SIP COMP9519 Multimedia Systems Lecture 7 Slide 3 Reji Mathew

4 Client Architecture : An Example Reference Client architecture for MPEG-4 streaming, Haifeng Xu Diamand, J. Luthra, A. IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages Review of Streaming Video, RTP, MPEG-4 Client Architecture for streaming MPEG-4 video Multilevel buffer system COMP9519 Multimedia Systems Lecture 7 Slide 4 Reji Mathew

5 Client Architecture : An Example Multiple buffer architecture to allow efficient control of media processing and presentation. Three Modules Network Module : Packet buffer Codec Module : Frame Buffer Presentation Module : Image Buffer Client architecture for MPEG-4 streaming, Haifeng Xu Diamand, J. Luthra, A. IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages COMP9519 Multimedia Systems Lecture 7 Slide 5 Reji Mathew

6 Client Architecture : Multiple Buffers Packet Buffer Frame Buffer Image Buffer Client architecture for MPEG-4 streaming, Haifeng Xu Diamand, J. Luthra, A. IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages COMP9519 Multimedia Systems Lecture 7 Slide 6 Reji Mathew

7 Client Architecture : Multiple Buffers Packet Buffer Reorder packets Identify missing packets (allow for easy insertion) Ignores delayed packets (outside moving window) Frame Buffer Stores video packets (smoothing out network jitter) Allows dropping of frame (when lacking CPU resources) Identify frames to be dropped (B, P) Image Buffer Stores decoded frames for rendering Smoothing decoders speed COMP9519 Multimedia Systems Lecture 7 Slide 7 Reji Mathew

8 Client Architecture : Rendering Rendered frames rates with and without the image buffer Client architecture for MPEG-4 streaming, Haifeng Xu Diamand, J. Luthra, A. IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages COMP9519 Multimedia Systems Lecture 7 Slide 8 Reji Mathew

9 Client Architecture : Conclusion The MPEG-4 MLBA subsystem facilitates three player-related activities: precise A/V synchronization, client-based QoS management, and improved rendering performance through an image buffer. Client architecture for MPEG-4 streaming, Haifeng Xu Diamand, J. Luthra, A. IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages COMP9519 Multimedia Systems Lecture 7 Slide 9 Reji Mathew

10 Lecture Outline Session Description Protocol How to describe a multimedia session? SDP example RTSP Internet VCR controls Stop, Pause, Play, Fast Forward over the internet RTSP Methods Signal Timing Diagram Protocol stack for a total streaming system SIP Quick Overview MPEG-4 File Format What s so good about MP4? File Format and Structure Hinting COMP9519 Multimedia Systems Lecture 7 Slide 10 Reji Mathew

11 Introduction Example : An existing live multicast session Video and Audio streams Transport - RTP/UDP/IP, Control RTCP/UDP/IP A new client wanting to join the multicast session Needs to know multicast IP address and port Media streams in a session (e.g. video only or audio + video) Payload format (e.g. MPEG-4 video, AMR audio) Initialization data for video and audio decoders Transport protocol used Other information. Need a way to describe a multimedia session To enable new clients to easily join the session COMP9519 Multimedia Systems Lecture 7 Slide 11 Reji Mathew

12 SDP Session Description Protocol (SDP) IETF RFC For describing multimedia sessions To communicate the existence of a session To convey sufficient information to join a session Simple text format Defined to be general purpose Can be used for a wide range of network environments And applications COMP9519 Multimedia Systems Lecture 7 Slide 12 Reji Mathew

13 SDP SDP includes Session name and purpose Time the session is active The media comprising the session Information to receive media (addresses, ports, formats) Information about bandwidth to be used Contact information of a person responsible for the session SDP is used by other signaling / initiation protocols SIP : Session Initiation Protocol RTSP : Real-time Streaming Protocol COMP9519 Multimedia Systems Lecture 7 Slide 13 Reji Mathew

14 SDP session level description v= (protocol version) o= (owner and session identifier) s= (session name) i=* (session information) u=* (URI of description) e=* ( address) p=* (phone number) c=* (connection information) b=* (bandwidth information) z=* (time zone adjustments) k=* (encryption key) a=* (zero or more session attribute) t= (time the session is active) r=* (zero or more repeat times) SDP session description consists of a number of lines of text of the form <type>=<value> <type> is always exactly one character and is case-significant. media level description m= (media & transport address) i=* (media title) c=* (connection information) b=* (bandwidth information) k=* (encryption key) a=* (zero or more media attribute) COMP9519 Multimedia Systems Lecture 7 Slide 14 Reji Mathew

15 SDP Version number increased when a modification is made to the session data. Recommended that an NTP timestamp is used v=0 o=mmvc IN IP s=camera Originator ONE information i=video stream for Session realtime name surveillance u= (address of Session machine Information from which the session was created) e=rmathew@nicta.com.au (text description / title (Reji for session) Mathew) URL for more information about the session c=in IP /2 Contact person t=0 0 (person responsible Connection not necessarily Details the creator) a=recvonly <network type> <address type> <connection address> m=video IN (internet) RTP/AVP IP4 (IP v4) Time 98 address Information (multicast address/ttl) a=rtpmap:98 MP4V-ES/90000 Session level attribute <start time> <end time> (operate in receive only mode) a=fmtp:98 profile-level-id=1; (ntp time, 0,0 implies config=000001b permanent session) Media Media attribute Announcement (rtp map) a=orient:portrait <media> <port> <transport> <fmt list> <username> <session id> <version> <network type> <address type> <address> <payload type> (fmt : refers Media <encoding to media attribute name>/<clock format (format specific specific rate> (map RTP dynamic payload number to information) parameters) media format and clock rate) (e.g. (info Dynamic regarding payload mpeg-4 number) media, profile, level & initialization data) Media attribute (orientation) (only used in some applications, example landscape or portrait) COMP9519 Multimedia Systems Lecture 7 Slide 15 Reji Mathew

16 System Overview Example Using SDP to join a multicast session Request SDP file via HTTP Retrieve information from downloaded SDP file Receive RTP streams on SDP specifed address & port Decode and display specified media HTTP GET SDP file Media Server RTP Media Stream Client RTCP COMP9519 Multimedia Systems Lecture 7 Slide 16 Reji Mathew

17 System Overview Example Traffic Monitoring Continuous multicast streaming of video A client can receive the stream by downloading SDP via http (web browser) Provide SDP file to QuickTime player Player initializes and waits for stream data Easily support multiple client Demo Link COMP9519 Multimedia Systems Lecture 7 Slide 17 Reji Mathew

18 Lecture Outline Session Description Protocol How to describe a multimedia session? SDP example RTSP Internet VCR controls Stop, Pause, Play, Fast Forward over the internet RTSP Methods Signal Timing Diagram Protocol stack for a total streaming system SIP Quick Overview MPEG-4 File Format What s so good about MP4? File Format and Structure Hinting COMP9519 Multimedia Systems Lecture 7 Slide 18 Reji Mathew

19 RTSP Now we can describe and transport media streams But how to control these streams Start, Stop, Pause, Fast Forward, Rewind internet VCR Solution RTSP Real Time Streaming Protocol Establishes and controls one or more continuous media streams - such as audio and video Similar in syntax and operation to HTTP/1.1 Client Server protocol Text based IETF RFC 2326 COMP9519 Multimedia Systems Lecture 7 Slide 19 Reji Mathew

20 RTSP More Info Example PLAY rtsp://audio.example.com/twister.en RTSP/1.0 CSeq: 833 Session: Range: smpte=0:10:20-;time= t153600z Play Request (TCP/IP) Play Response (TCP/IP) Media Server RTSP/ OK CSeq: 833 Date: 23 Jan :35:06 GMT Range: smpte=0:10:22-;time= t153600z Client COMP9519 Multimedia Systems Lecture 7 Slide 20 Reji Mathew

21 RTSP Protocol Properties Media Transport Independent RTSP does not depend on the transport mechanism used to carry the continuous media Media transport can be via RTP or some other mechanism Transport-independent: RTSP may use either an unreliable datagram protocol (UDP) or a reliable stream protocol such as TCP Control of recording devices: Can control both recording only and playback only devices, As well as devices that can alternate between the two modes COMP9519 Multimedia Systems Lecture 7 Slide 21 Reji Mathew

22 RTSP Protocol Properties Extendable: New methods and parameters can be easily added to RTSP. Easy to parse Multi-server capable: Each media stream within a presentation can reside on a different server. HTTP-friendly: RTSP reuses HTTP concepts, so that the existing infrastructure can be reused. COMP9519 Multimedia Systems Lecture 7 Slide 22 Reji Mathew

23 RTSP Protocol Operation Text based messages between client and server Messages can be : Requests Responses Request Messages Request-line Message - header Message - header CRLF Message - body Contains URI of the resource The method to be applied to the resource The protocol version in use General and request headers May or may not be present COMP9519 Multimedia Systems Lecture 7 Slide 23 Reji Mathew

24 RTSP Protocol Operation Response Messages Request-line Message - header Message - header CRLF Message - body The protocol version in use Status code (3 digit code) Textual description of status code General and response headers May or may not be present Next : examples of Request and Response messages Corresponding to five important request methods Methods : Describe, Setup, Play, Pause, Teardown Unicast example COMP9519 Multimedia Systems Lecture 7 Slide 24 Reji Mathew

25 RTSP : Describe Method Retrieves the description of a presentation, identified by the request URL, from a server. The DESCRIBE reply-response pair constitutes the media initialization phase of RTSP. C->S: DESCRIBE rtsp://nicta.com/twister RTSP/1.0 CSeq: 1 v=0 o= IN IP s=rtsp Session i=an Example of RTSP Session Usage a=control:rtsp://nicta.com/twister t=0 0 m=audio 0 RTP/AVP 0 a=control:rtsp://nicta.com/twister/audio m=video 0 RTP/AVP 26 a=control:rtsp://nicta.com/twister/video COMP9519 Multimedia Systems Lecture 7 Slide 25 Reji Mathew Version Request line Presentation URL S->C: RTSP/ OK Response line: Version, Status code 200 with description Method CSeq: 1 Same CSeq number as request Content-Type: application/sdp Content-Length: 164 CSeq header field Sequence number for an RTSP request-response pair Must be present in all requests and responses SDP Information (message body) Aggregate control possible (session level control attribute)

26 RTSP : Setup Method Specifies the transport mechanism to be used for the streamed media C->S: SETUP rtsp://nicta.com/twister/audio RTSP/1.0 CSeq: 2 Transport: RTP/AVP;unicast;client_port= S->C: RTSP/ OK CSeq: 2 S->C: RTSP/ OK CSeq: 3 Transport: RTP/AVP;unicast;client_port= ; server_port= Session: COMP9519 Multimedia Systems Lecture 7 Slide 26 Reji Mathew unicast RTP/RTCP port pair on which the client has chosen to receive media data and control information Transport: RTP/AVP;unicast;client_port= ; server_port= Session: RTSP session ID unicast RTP/RTCP port C->S: SETUP rtsp://nicta.com/twister/video RTSP/1.0 pair on which the server CSeq: 3 has chosen to send Transport: RTP/AVP;unicast;client_port= media data and control Session: information Setup operates on each stream separately Same session ID used

27 RTSP : Play Method Tells the server to start sending data via the mechanism specified in SETUP request/response messages C->S: PLAY rtsp://nicta.com/twister RTSP/1.0 CSeq: 4 Range: npt=0- Session: S->C: RTSP/ OK CSeq: 4 Session: RTP-Info: url=rtsp://foo/twister/video; seq= ;rtptime= Aggregate control of all media streams in the presentation. The PLAY request starts streaming both audio and video streams. Range header used to specify time ranges (start / stop) for playback Normal play time (NPT) indicates the stream absolute position relative to the beginning of the presentation. COMP9519 Multimedia Systems Lecture 7 Slide 27 Reji Mathew

28 RTSP : Pause Method Causes the stream delivery to be interrupted (halted) temporarily. C->S: PAUSE rtsp://nicta.com/twister RTSP/1.0 CSeq: 5 Session: S->C: RTSP/ OK CSeq: 5 Session: Aggregate control of all media streams in the presentation. The PAUSE request halts the streaming of both audio and video streams. COMP9519 Multimedia Systems Lecture 7 Slide 28 Reji Mathew

29 RTSP : Teardown Method Stops the stream delivery for the given URI, freeing the resources associated with it C->S: TEARDOWN rtsp://nicta.com/twister RTSP/1.0 CSeq: 892 Session: S->C: RTSP/ OK CSeq: 892 COMP9519 Multimedia Systems Lecture 7 Slide 29 Reji Mathew

30 RTSP : Signal Timing Diagram Media Client DESCRIBE Response (SDP) SETUP (Video stream) Response SETUP (Audio stream) Response PLAY Response PAUSE Media Server Response TEARDOWN Response COMP9519 Multimedia Systems Lecture 7 Slide 30 Reji Mathew

31 Summary of RTSP Methods OPTIONS : SETUP : ANNOUNCE : DESCRIBE : PLAY : RECORD : REDIRECT : PAUSE : get available methods establish transport change description of media object get description of media start playback, reposition start recording redirect client to new server pause delivery, while keeping state SET PARAMETER : device or encoding control TEARDOWN : remove state / close COMP9519 Multimedia Systems Lecture 7 Slide 31 Reji Mathew

32 RTSP : Comparison with HTTP Similar to HTTP, but some important differences for RTSP are : Data delivery takes place out-of-band in a different protocol e.g. RTP Both the media client and media server can issue requests Client > Server : SETUP, PLAY,. Server > Client : ANNOUNCE, REDIRECT,. Requests are not stateless : client & server state machines client changes state on receipt of replies to requests server changes state on receiving requests Init Ready Playing Recording COMP9519 Multimedia Systems Lecture 7 Slide 32 Reji Mathew

33 Protocol Stack MPEG-4 File / Encoder packetize MPEG-4 Video RTCP RTP SDP RTSP SDP RTSP RTCP Decoder RTP UDP TCP TCP UDP IP COMP9519 Multimedia Systems Lecture 7 Slide 33 Reji Mathew

34 Lecture Outline Session Description Protocol How to describe a multimedia session? SDP example RTSP Internet VCR controls Stop, Pause, Play, Fast Forward over the internet RTSP Methods Signal Timing Diagram Protocol stack for a total streaming system SIP Quick Overview MPEG-4 File Format What s so good about MP4? File Format and Structure Hinting COMP9519 Multimedia Systems Lecture 7 Slide 34 Reji Mathew

35 SIP Session Initiation Protocol (SIP) Another session control protocol IETF RFC Protocol that can establish, modify and terminate multimedia sessions. Applications IP telephony, multimedia conferences SIP, like RTSP, Uses text-based request/response transaction model Requests contain Methods and Header fields Responses include 3 digit status codes (eg 200 OK ) Is Transport layer independent Uses other protocols for media delivery (eg RTP). COMP9519 Multimedia Systems Lecture 7 Slide 35 Reji Mathew

36 SIP SIP Methods INVITE, ACK, CANCEL : for setting up sessions BYE : for terminating sessions REGISTER : for registering contact information OPTIONS : for querying servers about capabilities INVITE User agent client initiates a session with INVITE Invitation can be accepted by a user agent server SIP invitations convey session descriptions that allow participants to agree on a set of compatible media types Offer/answer model ACK Response confirmation to INVITE COMP9519 Multimedia Systems Lecture 7 Slide 36 Reji Mathew

37 SIP INVITE Example INVITE SIP/2.0 Via:SIP/2.0/UDPpc33.atlanta.com;branch=z9hG4bK776asdhds Max-Forwards: 70 To: Bob From: Alice Call-ID: CSeq: INVITE Contact: Content-Type: application/sdp Content-Length: 142 (Alice's SDP not shown) COMP9519 Multimedia Systems Lecture 7 Slide 37 Reji Mathew

38 SIP SIP supports user mobility With proxy servers to help route requests to user s current location Registration function allowing users to upload current locations for use by proxy servers COMP9519 Multimedia Systems Lecture 7 Slide 38 Reji Mathew

39 SIP atlanta.com proxy biloxi.com proxy Alice INVITE INVITE Bob INVITE 200 OK 200 OK ACK 200 OK Media Session BYE 200 OK COMP9519 Multimedia Systems Lecture 7 Slide 39 Reji Mathew

40 SIP Response Example SIP/ OK Via: SIP/2.0/UDP server10.biloxi.com ;branch=z9hg4bknashds8;received= Via: SIP/2.0/UDP bigbox3.site3.atlanta.com ;branch=z9hg4bk77ef4c ;received= Via: SIP/2.0/UDP pc33.atlanta.com ;branch=z9hg4bk776asdhds ;received= To: Bob From: Alice Call-ID: CSeq: INVITE Contact: Content-Type: application/sdp Content-Length: 131 COMP9519 Multimedia Systems Lecture 7 Slide 40 Reji Mathew (Bob's SDP not shown)

41 Streaming Applications Video on Demand VoIP Video Conf (3G) IPTV MPEG SDP RTP RTCP RTSP SIP IP COMP9519 Multimedia Systems Lecture 7 Slide 41 Reji Mathew

42 Streaming Media Players Qucik Time Player Real Player Microsoft Media player Packet Video Players for various platforms PC Pocket PC 3G Phones COMP9519 Multimedia Systems Lecture 7 Slide 42 Reji Mathew

43 Lecture Outline Session Description Protocol How to describe a multimedia session? SDP example RTSP Internet VCR controls Stop, Pause, Play, Fast Forward over the internet RTSP Methods Signal Timing Diagram Protocol stack for a total streaming system SIP Quick Overview MPEG-4 File Format What s so good about MP4? File Format and Structure Hinting COMP9519 Multimedia Systems Lecture 7 Slide 43 Reji Mathew

44 File Format Now we can code and stream video and audio Next we look at file formats for exchanging multimedia MPEG-1 & MPEG-2 content typically exchanged as files that represent a stream ready to be delivered Embedded absolute time stamps Fragmentation of media for some preferred transport Random Access could be difficult Need a new file format that can be used when Storing captured media Editing stored media Combining multiple files Delivering as File Download or Streaming A life-cycle file format COMP9519 Multimedia Systems Lecture 7 Slide 44 Reji Mathew

45 File Format MPEG-4 file format : MP4 life-cycle file format defined as part of MPEG-4 Based on Apple s QuickTime format ( mov files ) Due to MP4 s flexibility and extensibility Industry acceptance for exchange of MPEG-4 video and audio Basis for Motion JPEG 2000 file format Basis for 3GPP file format used in MMS Multimedia Messaging Service COMP9519 Multimedia Systems Lecture 7 Slide 45 Reji Mathew

46 MP4 : Main File Format Concepts The Media data is stored separately from Meta data Media data : Metadata : Audio, Video samples Data describing the media Examples : timing info, number of bytes required for a frame Timing information specified by relative numbers (durations) rather than absolute numbers Allows editing to be easier eg insertion of a new frame Able to store media data distributed over several files Use URLs to point to media data stored at various locations COMP9519 Multimedia Systems Lecture 7 Slide 46 Reji Mathew

47 MP4 : Main File Format Concepts The Media data is stored separately from Meta data Timing information specified by relative numbers (durations) rather than absolute numbers Able to store media data distributed over several files Locating media data by means of data offsets and length information Metadata tables mapping media sample number to location in a file Support streaming protocols through optional hint tracks Metadata information for packetization and header data Example hints for RTP streaming stored as a separate track COMP9519 Multimedia Systems Lecture 7 Slide 47 Reji Mathew

48 MP4: File Structure The MP4 file format is composed of object-oriented structures called atoms. A unique tag and a length identify each atom. Metadata atoms : A hierarchy of metadata giving information such as index points, durations, and pointers to the media data. This collection of atoms is contained in an atom called the movie atom. Media Data Atoms : The media data is contained in one or more mdat or media data atoms, or could be located outside the MP4 file; referenced via URL s. COMP9519 Multimedia Systems Lecture 7 Slide 48 Reji Mathew

49 MP4: File Structure Metadata Hierarchy [5] Movie Atom moov Audio track trak Video track trak Hint track trak Other MPEG-4 track trak Sample table stbl Contains tables of sample timing and location information COMP9519 Multimedia Systems Lecture 7 Slide 49 Reji Mathew

50 MP4: File Structure Media data mdat Atom Contains interleaved, time ordered video and audio samples referred to as Access Units (AU) Several frames from the same track are often stored contiguously in chunks. Media data mdat Frame 1 Frame 2 Frame 3 Frame 1 Frame 2 Frame 4 Video chunk Audio chunk COMP9519 Multimedia Systems Lecture 7 Slide 50 Reji Mathew

51 MP4: File Structure Basic structure of MP4 file Metedata moov atom Each stream has a track trak atom which contain tables Tables with info on AU s Timing, size, location (offsets), synchronization points Tables used to point to AU in mdat Media Data mdat atom Contains AU of audio and video Putting it together See next slide COMP9519 Multimedia Systems Lecture 7 Slide 51 Reji Mathew

52 MP4: File Structure [4] COMP9519 Multimedia Systems Lecture 7 Slide 52 Reji Mathew

53 MP4: Hinting MP4 file format is a streamable format, as opposed to a streaming format. The file format facilitates streaming without ever being streamed directly File format does NOT define an on-the-wire protocol, and is never actually streamed over a transmission medium. Metadata in the file known as hint tracks provide instructions, telling a server application how to deliver the media data over a particular delivery protocol. There can be multiple hint tracks for one presentation, describing how to deliver over various delivery protocols. COMP9519 Multimedia Systems Lecture 7 Slide 53 Reji Mathew

54 MP4: Hinting Remember for streaming, need to format data into packets, with each packet containing a Header, and Samples or Media data Example RTP for MPEG-4 Hint Track Tables pointing to header information The header information is stored in mdat Example RTP header information Then points to appropriate fragment of sample data Example one video frame for an RTP packet See next slide COMP9519 Multimedia Systems Lecture 7 Slide 54 Reji Mathew

55 MP4: Hinting [4] COMP9519 Multimedia Systems Lecture 7 Slide 55 Reji Mathew

56 MP4: Hinting What to do to accommodate different transport protocols? Multiple hint tracks for one media Advantage to the media server? The server can be dum No knowledge of transport protocol required Header information already stored in the MP4 file No need to do media fragmentation Pointers to appropriate media fragments already created No knowledge of media codec required COMP9519 Multimedia Systems Lecture 7 Slide 56 Reji Mathew

57 MP4: Hinting Media Server No need to parse the coded media bitstream No need to perform fragmentation No need to create headers Low server complexity clip.mp4 retrieve Client Generic server design COMP9519 Multimedia Systems Lecture 7 Slide 57 Reji Mathew

58 References and Further Reading 1. IETF RFC 2327 SDP: Session Description Protocol 2. IETF RFC 2326 Real Time Streaming Protocol (RTSP) 3. IETF RFC 3261 SIP: Session Initiation Protocol 4. David Singer, QuickTime Engineering Quick Time etc. Introduction for UC Davis, Feb MPEG-4 Overview, ISO/IEC JTC1/SC29/WG11 N COMP9519 Multimedia Systems Lecture 7 Slide 58 Reji Mathew