Lec 16 Multimedia Systems II: DASH Advanced Features

Transcription

1 CS/EE 5590 / ENG 401 Special Topics (17804, 17815, 17803) Lec 16 Multimedia Systems II: DASH Advanced Features Zhu Li Course Web: Z. Li, Multimedia Communciation, 2016 Spring p.1

2 ReCap Lecture 15 Outline Advanced DASH Topics DASH FDH: DASH over Full Duplex HTTP/WebSocket DASH SAND: QoS, QoE and Traffic Coordination Summary Z. Li, Multimedia Communciation, 2016 Spring p.2

3 MPEG File Format (FF) Chocolate box approach Abstract the content into a hierarchical structure, meta data, media data Hiding the compression tech details The Solution ISOBMFF ISO Based Media File Format aka, Mp4 moov mdat moof hint Z. Li Multimedia Communciation, 2016 Spring p.3

4 ISOBMFF file types Plain File: Simple recording of plain media data, data first, header last mdat box, then moov box, e.g, foreman_320kbps_nf120.mp4 Progressive File: For progressive download or streaming Header first then media data Interleaved Chunks Fragmented File: Multiple moof segments followed by moov box Good for continous recording Segmented File: Self contained and playable fragments in signle file or in separate files For HTTP streaming (DASH) Tools: segment file, indexing Z. Li Multimedia Communciation, 2016 Spring p.4

5 DASH in a Nutshell Dynamic, Adaptive, Streaming over HTTP: An OTT solution HTTP De-facto Internet Transport Infrastructure Z. Li Multimedia Communciation, 2016 Spring p.5

6 Media Presentation Data Model MPD: a manifest of content available on HTTP server Accessible segments and their timing As a.xml file to be retrieved by clients at the start of DASH session Credits to figures on following slides: Christian and Ali, Over the Top Content Delivery: State of the Art and Challenges Ahead, ICME 2015 Tutorial Z. Li Multimedia Communciation, 2016 Spring p.6

7 Understanding DASH structure Three types of files Manifest (.mpd) o XML file describing the segments Initialization file o Contains headers needed to decode bytes in segments Segment Files o Contains playable media o Includes: 0 many video tracks 0 many audio tracks

8 DASH Manifest Manifest contains: Program Info (including title) 1 or more periods o Periods contain 1 adaptation set per video stream and o Periods contain 1 adaptation set per audio stream o Adaptation Sets contain: Content Composition nodes (for each video or audio track) 1 or more Representation node Each representation describes a single bitrate Representations contain data on finding the actual segments Different ways a representation can describe segments

9 Describing Representations SegmentBase Describes a stream with only a single Segment per bitrate Can be used for Byte Range Requests SegmentList A SegmentList will contain a specific list of each SegmentURL (individual HTTP packet with media data) Can be used for Byte Range Requests SegmentTemplate Defines a known url for the fragment with wildcards resolved at runtime to request a segments (see bbb.mpd) Alternatively, can specify a list of segments based on duration

10 SegmentList <Representation id="h264bl_hd" mimetype="video/mp4" codecs="avc1.42c01f" width="1280" height="720" startwithsap="1" bandwidth="514864"> <SegmentList timescale="1000" duration="10000"> <Initialization sourceurl="mp4-main-multi-h264bl_hd-.mp4"/> <SegmentURL media="mp4-main-multi-h264bl_hd-1.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-2.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-3.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-4.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-5.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-6.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-7.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-8.m4s"/>

11 SegmentTemplate fixed segment duration <AdaptationSet> <ContentComponent id="1" contenttype="video"/> <SegmentTemplate initialization="bigbuckbunny_720p_1800kbps_44khz_track1_dash.mp4"/> <Representation id="1" mimetype="video/mp4 codecs="avc f" width="1280" height="720 startwithsap="1" bandwidth=" "> <SegmentTemplate timescale="1000" duration="13809" media="bbb_seg_bigbuckbunny_720p_1800kbps_44khz_track1$number$.m4s" startnumber="1"/> </Representation> </AdaptationSet>

12 SegmentTemplate variable segment duration <AdaptationSet group="2" mimetype="video/mp4" par="16:9 minbandwidth=" maxbandwidth=" " minwidth="176" maxwidth="1680" minheight="99" maxheight="944 segmentalignment="true startwithsap="1"> <SegmentTemplate timescale="1000" initialization="dash/ateam-video=$bandwidth$.dash" media="dash/ateam-video=$bandwidth$-$time$.dash"> <SegmentTimeline> <S t="0" d="4171" /> <S d="2503" /> <S d="2961" /> <S d="2461" /> <S d="2127" r="2" />

13 Understanding MP4 H.264 structure Each segment is binary, with a series of boxes contained within First 8 bytes of every box defines Box type Number of bytes for the box.

14 Initialization Segment FTYP file type and compatibility FREE free space MOOV container of all metadata MVHD movie header 1.* TRAK container for individual stream data o TKHD track header o MDIA container for all media info Lots of child boxes which contain more detailed stream info

15 STYP - segment type SIDX - segment index.m4s Content Segments MOOF -movie fragment MFHD - movie fragment header TRAF - track fragment o TFHD - track fragment header o TFDT - track fragment decode time o TRUN - track fragment run MDAT - media data container (actual media data) Contains one or more samples (smallest described piece of content)

16 Preparing DASH content with MP4Box Download you favorable video from youtube US/firefox/addon/video-downloadhelper/ Encode for on-demand DASH content MP4Box dash out mydash/my.mpd dash-profile on-demand segment-name mydashvideo/seg myvideo.mp4 It will create A MPD file listing all segments A collection of video segment files in.m4s, the segmented file. Download Helper Z. Li Multimedia Communciation, 2016 Spring p.16

17 MPD Z. Li Multimedia Communciation, 2016 Spring p.17

18 DASH.js To playback DASH video is very simple, just use <video> tag Check out: <script src=" </script> <h4> DASH.js Player Demo </h4> <style> video { width: 640px; height: 360px; }</style>. <div> <video data-dashjs-player src=" manifest.mpd" controls></video> </div> Z. Li Multimedia Communciation, 2016 Spring p.18

19 ReCap Lecture 15 Outline Advanced DASH Topics DASH FDH: DASH over Full Duplex HTTP/WebSocket DASH SAND: Traffic Coordination MPEG MMT Summary Z. Li, Multimedia Communciation, 2016 Spring p.19

20 DASH FDH Introducing Push Based Operation Client req seg 1 Server Client req seg 1 to k Server seg 1 req seg 2 seg 2 seg n req seg n seg 1 seg k req seg (n-k+1) to n seg (n-k+1) seg n (a) Regular HTTP Benefit: Reduce the HTTP GET overhead Avoid TCP transmission window collapse (b) HTTP/2 Server Push Z. Li Multimedia Communciation, 2016 Spring p.20

21 HTTP 2.0 IN a NutShell One TCP connection Multiple streams requests are in streams, can be parallelized Binarized framing, streams can have flow control All for Push Service Z. Li Multimedia Communciation, 2016 Spring p.21

22 HTTP 2.0 Streams Streams are multiplexed into frames Frames are interleaved Can have flow control on streams E.g give main.css with priority 1, give me fig1.jpg with priority 5. Z. Li Multimedia Communciation, 2016 Spring p.22

23 DASH FDH with HTTP 2.0 Example of push-next Client request pushing 2 next segments Z. Li Multimedia Communciation, 2016 Spring p.23

24 Server Response: Push promise: 2 segments over stream 2 and 3 Segment 1 data DASH FDH over HTTP 2.0 Z. Li Multimedia Communciation, 2016 Spring p.24

25 DASH FDH over HTTP 2.0 Server Response: Segment 2, 3 over a different stream Response [Stream ID = 2]: HEADERS + END_STREAM + END_HEADERS :status = 200 DATA + END_STREAM {binary data for segment 2} Response [Stream ID = 3]: HEADERS + END_STREAM + END_HEADERS :status = 200 DATA + END_STREAM {binary data for segment 3} Z. Li Multimedia Communciation, 2016 Spring p.25

26 WebSockets Motivation Evolution of Web Applications Early days no real time requirement Now Dynamic, Media, and Real time applications like chat, streaming, collaboration HTTP is not designed for web apps Large overhead Hanging-GET is necessary for real-time server push Hence, the new solution, WebSockets Full Duplex Small overhead Multiplexing over a single TCP Easy to use API Specs RFC 6455: Z. Li Multimedia Communciation, 2016 Spring p.26

27 Establishing WebSocket Connection Upgrade over HTTP1.1 Z. Li Multimedia Communciation, 2016 Spring p.27

28 WebSocket Frames Support binary data channel Opcode: o 0x0 Continuation frame, 0x1 Connection close 0x2 Ping, 0x3 Pong, 0x4 Text frame, 0x5 Binary frame 0x7-0xF Reserved Z. Li Multimedia Communciation, 2016 Spring p.28

29 Message flow FDH DASH over WebSocket Z. Li Multimedia Communciation, 2016 Spring p.29

30 WebSocket Sub-Protocol for DASH DASH WebSocket Header as part of WebSocket Payload STREAM_ID: 8 bits, identify current stream MSG_CODE: 8 bits: o 1- get_mpd, 2 get_segment, 3-new_mpd, 4: new_segment o 255: cancel F: 3 bits: flags to signal different commands EXT_LENT: extension header for JSON coded parameters STREAM_ID MSG_CODE F EXT_LENGTH Extension (JSON encoded parameters) Application Data Z. Li Multimedia Communciation, 2016 Spring p.30

31 Request New MPD WebSocket DASH FDH Ops Client Request: STREAM_ID : 1 MSG_CODE: 1 EXT_LENGTH: 27 EXT: { mpd_uri :./example.mpd } Server Response: STREAM_ID : 1 MSG_CODE: 3 EXT_LENGTH: 0 {binary data with example.mpd} Z. Li Multimedia Communciation, 2016 Spring p.31

32 WebSocket DASH FDH Ops Request a new segment, followed by 2 pushed segmetns Client Request: STREAM_ID : 1 MSG_CODE: 2 EXT_LENGTH: 98 EXT: { segment_uri :./rep1/segment1.mp4, push_ directive : type=\ urn:mpeg:dash:fdh:2015:p ush-next\ ;2 } Server Response: STREAM_ID : 1 MSG_CODE: 4 EXT_LENGTH: 59 EXT: { push_policy : type=\ urn:mpeg:dash :fdh:2015:push-next\ ;2 } {binary data with segment1.mp4} Server Response: STREAM_ID : 1 MSG_CODE: 4 EXT_LENGTH: 0 {binary data with segment2.mp4} STREAM_ID : 1 MSG_CODE: 4 EXT_LENGTH: 0 {binary data with segment3.mp4} Z. Li Multimedia Communciation, 2016 Spring p.32

33 ReCap Lecture 15 Outline Advanced DASH Topics DASH FDH: DASH over Full Duplex HTTP/WebSocket DASH SAND: Traffic Coordination MPEG MMT Summary Z. Li, Multimedia Communciation, 2016 Spring p.33

34 DASH SAND DASH SAND Server and Network Assisted DASH (SAND) Objectives: Unidirectional/bidirectional, point-to-point/multipoint communication with and without session (management) between servers/cdns and DASH clients, Mechanisms for providing content-awareness and service-awareness towards the underlying protocol stack including server and/or network assistance, Various impacts on elements of the existing Internet infrastructure such as servers, proxies, caches and CDNs, QoS and QoE support for DASH-based services, Scalability in general and specifically for logging interfaces, and Analytics and monitoring of DASH-based services. Z. Li Multimedia Communciation, 2016 Spring p.34

35 DASH Boxes Client, Server, DANE. DASH SAND SAND Messages Parameters Enhancing Delivery (PED) messages that are exchanged between DANEs, Parameters Enhancing Reception (PER) messages that are sent from DANEs to DASH clients, Metrics and Status messages that are sent from DASH clients to DANEs. Z. Li Multimedia Communciation, 2016 Spring p.35

36 SAND Operations Where SAND sits Z. Li Multimedia Communciation, 2016 Spring p.36

37 PER:QoSInformation QoS Signalling with SAND Z. Li Multimedia Communciation, 2016 Spring p.37

38 Metics: Client to DANE Resource Coordination Allows for cross traffic sharing of common resources allocationstrategy: weighted, pricing.etc Parameter Type Cardinality Description SharedResourceAllocation object 1 operationpoints array 1..N List of suitable operation points for current play time. bandwidth integer 1 A bandwidth value expressed in a number of bits per second. This value shall be computed from the MPD by summing bandwidths of all components the client would use for working at this operation point. If playback rate is not 1, this bandwidth value shall be modified accordingly. quality integer 0..1 An optional value describing the quality of minbuffertime the current operation point. integer 0..1 An optional value in milliseconds extracted from the MPD regarding a minimal buffer time of the current operation point. weight integer 0..1 A user allocated optional value which indicates a weight of the request in the present message for the resource allocation process. The exact use of this value depends on the allocation strategy indicated by allocationstrategy. allocationstrategy urn 0..1 An optional identifier to indicate the resource allocation strategy preferred by the client for resource sharing. mpdurl string 0..1 If present, an URL to the MPD related to the present message. Z. Li Multimedia Communciation, 2016 Spring p.38

39 Resource Allocation PER:sharedResourceAllocation, DANE to Client Results of optimization process Parameter Type Cardinality Description SharedResourceAssignment object 1 Response message from the coordinator that indicates the results of the bandwidth sharing operation. clientid string 1 resourceprice int 0..1 bandwidth int 1 The clientid identifies the target receiver of this message. This field shall use the same value as the senderid of the SharedResourceAllocation message sent by the client. A price for the bandwidth resource to be used by the receiver in its utility to price tradeoff to determine an optimal operation point. The operation point selected will maximize the utility to price ratio. This fields contains the assigned bandwidth to the identified client. The unit is defined in bits per second. Resource Pricing in bottleneck coordination: Ying Li, Zhu Li, Mung Chiang, A. Robert Calderbank: Content-Aware Distortion-Fair Video Streaming in Congested Networks. IEEE Trans. Multimedia 11(6): (2009) Z. Li Multimedia Communciation, 2016 Spring p.39

40 Summary MP4Box/DASH.js MP4Box is a versatile tool for media file manipulation, it has functionality built in for preparing DASH streaming content DASH.js is a reference javascript DASH client implementation, by the DASH IF, and easy to modify and good for research PoC DASH FDH Improve throughput, efficiency by utilizing new transport HTTP 2.0/WebSocket has different APIs but fundamentally the same implementation Ref: DASH FDH specs DASH SAND Allows server and other DANE middle box to play a role New messaging to report QoS, QoE, and coordinate, accelerate traffic Ref: DASH SAND specs Z. Li Multimedia Communciation, 2016 Spring p.40