Adaptive HTTP Streaming and the MPEG-DASH standard

Transcription

1 Adaptive HTTP Streaming and the MPEG-DASH standard Jean Le Feuvre 1 Institut Mines-Télécom

2 Overview n HTTP Refresher n HTTP Streaming Principles Comparison of adaptive HTTP streaming solutions n MPEG DASH Study of the standard n Advanced Adaptive HTTP Streaming Metrics, Encryption, Latency, HEVC, Subtitles n Lab Session 2 Institut Mines-Télécom

3 HTTP Refresher 3 Institut Mines-Télécom

4 HTTP 1.1 n Basics Goal: delivery of files or bytes identified by a URL Textual protocol Request/answer model Request: Method + URL + Headers ( + data) Answer: Response Code + Headers ( + data) n Methods: HEAD: get info about URL GET: retrieves URL data POST: post data associated with a URI PUT: uploads a new resource with new URL DELETE: deletes the URL resource TRACE: get HTTP request received by the server OPTIONS: get HTTP options available at the server CONENCT: for use in SSL tunnels 4 Institut Mines-Télécom

5 HTTP 1.1 n Caching Servers may assign expiration date to resources Cache-Control: public: any caching entity may store Cache-Control: private: usually browser cache Clients may check for modifications on resources Avoids re-downloading the same resource 5 Institut Mines-Télécom

6 6 Institut Mines-Télécom

7 HTTP 1.1 n Resource Addressing Complete resource Byte-range: Closed: bytes=0-200 Open: bytes=1024- End ranges: bytes=-500 Multiple: bytes=0-200,-500 n Resource Packing Multipart messages A server may pack several resources in one GET response Content-Type: multipart/mixed;boundary=myfoobardelim 7 Institut Mines-Télécom

8 Optimizing HTTP 1.1 Latency RTT RTT Client Open SYN, ACK GET FOO FOO Server SYN ACK TCP Three-Way Handshake n Connecting to the server TCP Connection takes time At least 2 RTTs before geting the first byte RTT RTT Close Open SYN, ACK GET BAR BAR SYN ACK TCP Three-Way Handshake Close 8 Institut Mines-Télécom

9 Optimizing HTTP 1.1 Latency RTT RTT Client Open SYN, ACK GET FOO FOO Server SYN ACK TCP Three-Way Handshake Idle time n Persistent Connection Same connection used Saves one RTT for each subsequent request RTT GET BAR BAR 9 Institut Mines-Télécom

10 Optimizing HTTP 1.1 Latency RTT RTT Client Open SYN, ACK GET FOO FOO Server SYN ACK TCP Three-Way Handshake Idle time n Persistent Connection Same connection used Saves one RTT for each subsequent request But still 1 RTT for request RTT GET BAR BAR 10 Institut Mines-Télécom

11 Optimizing HTTP 1.1 Latency n Pipeline Requests Client Server Send subsequent requests while downloading one request RTT Open SYN, ACK GET FOO SYN ACK TCP Three-Way Handshake Same connection used No extra RTT for each subsequent request RTT GET BAR FOO BAR 11 Institut Mines-Télécom

12 Optimizing HTTP 1.1 Latency Client Server n Resource size must be known RTT RTT Open SYN, ACK GET FOO SYN ACK TCP Three-Way Handshake Wait for resource to be completely available at server side Problematic for consumption of resource while it is being produced (live) FOO 12 Institut Mines-Télécom

13 Optimizing HTTP 1.1 Latency Client Server n Chunk transfer RTT Open SYN, ACK GET FOO SYN ACK TCP Three-Way Handshake Available parts of the resource may right away RTT OK, Chunk1 ChunkN FOO Last Chunk 13 Institut Mines-Télécom

14 HTTP 1.1 Issues n No multiplexing All request are served sequentially Head of Line blocking Multiplexing can be simulated by using multiple connections to the server Resource costly Often limited by browser and/or server Hard to prioritize resources n No cancel If resource no longer needed while still being downloaded Connection has to be closed Bad for HTTP streaming Connection has to be re-setup Bad for latency n Headers cost Usually always similar headers for subsequent resources 14 Institut Mines-Télécom

15 HTTP/2 Design Principles n Binary format Data exchanged through frames n Transport Layer for HTTP1.1 Backward compatibility for applications Multiplexing of responses and requests Single connection to the server Persistent connection Header Compression All headers from HTTP1.1 are used n Resource management Each resource transferred is called a stream State machine for each stream at both client and server side 15 Institut Mines-Télécom

16 HTTP/2 Goodies n Stream Cancel A client may cancel any stream (pending or current) at any time n Server Push: a server may decide to push resources without the client asking Ex: speed up load HTML time by pushing CSS, Javascript n Stream Priority Weight streams to share available bandwidth accordingly n Stream Dependencies Indicate sequence order for sending resources n Flow Control how much bytes per stream can be stored at receiver s side 16 Institut Mines-Télécom

17 HTTP Streaming 17 Institut Mines-Télécom

18 HTTP Streaming Overview n Transport of media stream over HTTP/TCP n Why? Benefits? Simplified deployment compared to RTP No need for UDP or multicast Removes router configuration problems Reuse of Internet Infrastructures (servers, caches, proxy, browsers ) n Challenges HTTP was made for file delivery not stream, unlike RTP HTTP can suffer from Large RTT TCP congestion Need to support broad requirements Support live use cases Tune-in capabilities Support VoD use cases Play/Pause Rewind, Fast Forward 18 Institut Mines-Télécom

19 HTTP Streaming: Web Radio Example n Already deployed in 1999! Solutions/protocols: Ice-Cast, Shout-Cast n Principles Send an indefinite file over HTTP Content-length = 0 (not really standard) File is read by client but never stored n Limitations Single stream Elementary or multiplexed Hard to cache: same URL for different content at different times Difficult to support VOD Difficult to support bandwidth adaptation 19 Institut Mines-Télécom

20 Principles of VOD over HTTP n Use media files stored on an HTTP server n Let the client determine (the part(s) of) the file to download given the time range to be played n Use HTTP requests to download the file with «Byte-range» header for parts of files 20 Institut Mines-Télécom

21 VoD over HTTP: Example HTTP Server Client Meta Data V1 A1 V2 A2 V3 A3 Vk-& Ak-1 Vk Ak Vk+1Ak+1 Media Data V1 A1 V2 A2 V3 A3 Play Meta Data V1 A1 V2 A2 V3 A3 Vk-& Ak-1 Vk Ak Vk+1Ak+1 V1 A1 V2 A2 V3 A3 Vk-& Ak-1 Vk Ak Vk+1Ak+1 Seek Vk-& Ak-1 Vk Ak Vk+1Ak+1 21 Institut Mines-Télécom

22 Simple VOD over HTTP: limitations n Requires media-time/byte-offset mapping In-band mapping: download the file header first Out-of-band mapping: use an additional file n Requires extensions for Bandwidth adaptation VOD 22 Institut Mines-Télécom

23 Live Streaming over HTTP n n Problems Live data needs to be delivered frequently HTTP Infrastructures work best with files (cache, proxys) Difficult to cache indefinite duration files (web radios) Solution Divide the virtual file into small «Segments» of duration Δt Concatenation of segments makes a valid file Segments to download are described Solution 1: In-band: (n-1)-th segment indicates the n-th segment URL Solution 2: Out-of-band: by a playlist/manifest Explicitly naming of each URL Implicitly naming based on time or numbering, template URL Players start by buffering some segments Latency in the order of a segment duration (Nb Seg *Δt) Low latency mode: buffers less than a segment Players regularly download additional segments update the playlist 23 Institut Mines-Télécom

24 HTTP Live: Exemple HTTP Server Client Meta Data Meta Data Vk-& Ak-1 Vk Ak Vk+1Ak+1 V1 A1 V2 A2 V3 A3 Media Data Media Data Vk-& Ak-1 Vk Ak Vk+1Ak+1 V1 A1 V2 A2 V3 A3 V1 V2 Meta Data V1 A1 V2 A2 V3 A3 V1 A1 V2 A2 V3 A3 Meta Data Meta Data Vk-& Ak-1 Vk Ak Vk+1Ak+1 Vj-11 Aj-1 Vj Aj Vj+1 Aj+1 Meta Data Vk-& Ak-1 Vk Ak Vk+1Ak+1 Media Data Media Data Vk-& Ak-1 Vk Ak Vk+1Ak+1 Vj-1 Aj-1 Vj Aj Vj+1 Aj+1 Vk-& Ak-1 Vk Ak Vk+1Ak+1 24 Institut Mines-Télécom

25 Principles for Adaptive Streaming n Problems Avoid re-/buffering issues due to TCP congestion Offer different qualities to different clients n Solution Use of alternative files corresponding to different bitrates of the same segment Concatenation of files from different bitrates still makes a valid stream Alternatives indicated in a playlist Delivered independently n Increased client complexity Bandwidth/buffer estimation Adaptation logic Switching strategies (double download, abandonment ) 25 Institut Mines-Télécom

26 HTTP Adaptive Streaming Session Time 26 Institut Mines-Télécom

27 Rate 1 Examples and Problems in Bitstream Switching n-1 n n+1 n+2 t Rate 2 k-1 k k+1 k+2 I-frame δd δt n R 1,n-1 -> R 2,k Useless download of δd media Double decoding of δt media n Solution: Temporal Alignment of Segments Removes double download (δd=0) & double decoding (δt=0) Simplifies temporal description of segments 27 Institut Mines-Télécom

28 HTTP Streaming & Caching 28 Institut Mines-Télécom

29 HTTP Streaming and CDN 29 Institut Mines-Télécom

30 Adaptation Logic Challenges n Bandwidth-based adaptation Estimate the current bandwidth to select best quality Issues Congestion may happen Cache miss, cache hit, local cache alter estimation Server processing time unknown n Buffer-based adaptation Monitors amount of buffered media time to select best quality Can be seen as a smoothed version over time of bandwidthbased Issues Requires long buffer times (memory impact) Less reactive 30 Institut Mines-Télécom

31 HTTP Cache Origin Server Cache Server Client Request Segment with 1.5 Mbps bitrate Cache Hit Return Segment at 5 Mbps Request Segment with 2 Mbps bitrate Increase requested bitrate Request Segment with 2 Mbps bitrate Return Segment at 1.6 Mbps Cache Miss Return Segment at 1.6 Mbps Request Segment with 1.5 Mbps bitrate Decrease requested bitrate Cache Hit Return Segment at 5 Mbps Increase requested bitrate 31 Institut Mines-Télécom

32 Live Adaptation Logic Challenges n Latency Lower the amount of media buffer Consume segments while downloading them Compromise between buffer-based and bandwidth-based n Live edge Most recent segment produced If request too late Longer latency towards edge time If request too early 404! n Client logic: 404 or not 404? HTTP / OTT (no loss : TCP) Usually implies request too early File Datacast : (losses : multicast, broadcast) May imply request too early or lost segment! 32 Institut Mines-Télécom

33 Random Access vs Bitstream Switching t R 1 n-1 n n+1 n+2 I-frame closed GOP R 2 n-1 n n+1 n+2 I-frame open GOP n Tune in R 1 and R 2 : possible within the segment n Switching R 1 ->R 2 : possible within the segment R 2 ->R 1 : not possible within segment Depends on reference picture compatibility between R 1 and R 2 33 Institut Mines-Télécom

34 Commercial solutions n 3GPPs' Adaptive HTTP Streaming (AHS) n Adobes' Dynamic HTTP Streaming (HDS) n Apples' HTTP Live Streaming (HLS) n Microsoft s Smooth Streaming (MSS) n Open IPVT Forum (OIPF) 34 Institut Mines-Télécom

35 35 Institut Mines-Télécom

36 Apple HTTP Live Streaming n n n n n n Solution for Live, VOD & Adaptive streaming over HTTP Internet Draft published by IETF Supported on iphone, ipad and some desktop players Playlist format Based on M3U8 (extended MP3 playlists, not XML) Extended for live streaming, bitrate switching, MIME types: «audio/x-mpegurl» «video/x-mpegurl» «application/x-mpegurl» Media formats 10 seconds segments, typically 3 segments buffered AAC files for audio only MPEG-2 Transport Stream for AAC Audio & AVC Video WebVTT for subtitles Encryption Examples 36 Institut Mines-Télécom

37 Microsoft IIS Smooth Streaming n Windows Media approach Live, VOD and Adaptive Streaming Supported on platforms with Silverlight n Playlist format XML format n Media Segments Modified MP4 files: ISMV n Demo 37 Institut Mines-Télécom

38 HTTP Streaming Exercises n Exercises Analyze M3U8 Playlists Play M3U8 Playlists n Playlists USB Stick NRJ12 n Tools Text editor GPAC MP4Client player with GUI 38 Institut Mines-Télécom

39 MPEG-DASH Jean Le Feuvre Cyril Concolato 39 Institut Mines-Télécom

40 MPEG DASH Dynamic Adaptive Streaming over HTTP n ISO/IEC Support for Live, VOD and Adaptive Streaming Different profiles Live On Demand n Joint standard development with 3GPP Important industrial consortium: Qualcomm, Microsoft, Adobe, Ericsson, Apple, Freely available but not sure about royalty free 40 Institut Mines-Télécom

41 (Some) DASH Design Principles n DASH is not: system, protocol, presentation format, codec, client specification n DASH is an enabler It provides formats to enable efficient and high-quality delivery of streaming services over the Internet It is considered as one component in an e2e service System definition left to other organizations (SDOs, Fora, Companies, etc.) 41 Institut Mines-Télécom

42 (Some) DASH Design Principles n It attempts to be very good in what is to be addressed by the standard Enable reuse of existing technologies (containers, codecs, DRM etc.) Enable deployment on top of HTTP-CDNs (Web Infrastructures, caching) Enable very high user-experience (low start-up, no rebuffering, trick modes) Enable selection based on network and device capability, user preferences Enable seamless switching Enable live and DVD-kind of experiences Move intelligence from network to client, enable client differentiation Enable deployment flexibility (e.g., live, on-demand, time-shift viewing) Provide simple interoperability points (profiles) Enable industry requirements: Ad-Insertion, DRMs 42 Institut Mines-Télécom

43 DASH Scope n Playlist/manifest format: XML language MPD: Media Presentation Description n Segment formats: ISO Base media files MPEG-2 TS Extensible to other formats WebM WebVTT 43 Institut Mines-Télécom

44 DASH Description Model 44 Institut Mines-Télécom

45 DASH Description Model (2) 45 Institut Mines-Télécom

46 MPD (Media Presentation Description) n XML document Containing exactly one <MPD> element Containing in turn several <Period> elements n Description of the segments HTTP-URLs absolute or relative, templated or not. The MIME type of the MPD is defined: application/dash+xml n Out of scope Delivery of the MPD Actual playback is not controlled 46 Institut Mines-Télécom

47 Example MPEG-DASH Manifest <MPD type="static" minbuffertime="pt1.5s" mediapresentationduration="pt0h10m0.00s"> <ProgramInformation moreinformationurl=" <Title>Media Presentation Description for file ZZZ</Title> </ProgramInformation> <Period start="pt0s" duration="pt0h10m0.00s"> <AdaptationSet> <ContentComponent id="1" contenttype="video"/> <ContentComponent id="2" contenttype="audio" lang="und"/> <SegmentTemplate initialization="counter-10mn_i25_opengop_init.mp4"/> <Representation id="1" mimetype="video/mp4" codecs="avc d,mp4a.40.02" width="320" height="180" samplerate="44100" numchannels="1" lang="und" startwithsap="3" bandwidth="109952"> <SegmentTemplate timescale="1000" duration="9880" media="seg40_$number$.m4s" startnumber="1"/> </Representation> <Representation id="2" mimetype="video/mp4" codecs="avc d,mp4a.40.02" width="320" height="180" samplerate="44100" numchannels="1" lang="und" startwithsap="3" bandwidth="182078"> <SegmentTemplate timescale="1000" duration="9880" media="seg112_$number$.m4s" startnumber="1"/> </Representation> </AdaptationSet> </Period> </MPD> 47 Institut Mines-Télécom

48 Period n Media Presentation = one or more Periods. Period = time interval within which the media configuration does not change TV program ad movie Period elements are ordered in increasing start time. n Main (optional) n Main Content <AdaptationSet> elements 48 Institut Mines-Télécom

49 Adaptation Sets n 1 Period = one or more AdaptationSets n 1 AdaptationSet = N Representations Alternative but equivalent media, typically contain different encoded versions of the same source Language, media component type, picture aspect ratio, role, accessibility, viewpoint, rating. n Adaptationsets can be arranged Either one Representation from group 0, or the combination of at most one Representation from each non-zero group. 49 Institut Mines-Télécom

50 Representations n Media content Aligned within the period s boundaries. Consists of one or more Segments. Contains an initialization segment or all segments are self-initializing. May contain zero or more SubRepresentations. n Multiplexed Elementary Streams In one AdaptationSet Different components identified using <ContentComponent> Storage cost (e.g. duplicated audio streams) n Individual Elementary Stream One AdaptationSet par ES Possibly different Segment durations n Type identification 50 Institut Mines-Télécom

51 Sub Representations n Provide the ability for accessing a lower quality version of the Representation. Audio track in a multiplexed Representation. Lower frame rate for efficient fast-forward. 51 Institut Mines-Télécom

52 Subsets n Optional mechanism for the content creator to create collections of media presentations for various application/devices within the same MPD. Restrict the combination of active AdaptationSets. DASH client must use one subset. Empty subsets are not allowed. No subset can contain all groups. 52 Institut Mines-Télécom

53 Segments n n n n Segment = part of a bitstream Exact definition depending on the bitstream format (TS, ISOBMF, ) Sub-segment = part of a segment Several types of segments are defined A Segment is a unit that can be referenced by an HTTP-URL included in the MPD. and optionally with a byte range. Segments have an availability window Can be accessed by the HTTP-URL. Each representation has at most one SegmentInfo element which provides: Presence or absence of Initialization and Index Segment information. HTTP-URL and byte range for each segment. Segment availability start time and availability end time for live case. Approximated media start time and duration of each segment. Fixed or variable duration. 53 Institut Mines-Télécom

54 Initializations Segments n Each representation may have at most one Initialization Segment to initialize the media engines for play-out. n If no Initialization Segment URL is present, then each Media Segment is self-initializing. 54 Institut Mines-Télécom

55 Media Segments n Each representation has a list of consecutive Media Segments n Media segment information: URL, possibly restricted by a byte range. Index of segment. Approximate start time and duration. 55 Institut Mines-Télécom

56 Bitstream Switching Segment n Additional data needed for switching Insures that concatenation of segments coming from different representations is correct Example: DVB-CSA information at switch: system info (CAT) and key info (ECM/EMM) 56 Institut Mines-Télécom

57 DASH: Random Access Points n Historical, simple notion I / IDR frame n Does not cover more complex cases Open GOP: Frames from GOP N+1 use Frames from GOP N as references Gradual Decoding Refresh: No «I» or «IDR» frames but progressive reconstruction over N>1 frames n SAP: Stream Access Point I SAP : Point/position in the bitstream from which the decoding can start with data before I SAP T SAP : Presentation time from which the decoding can happen without data prior to I SAP T PTF : Smallest presentation time for all frames after I SAP T EPT : Presentation time of the frame at the I SAP position 57 Institut Mines-Télécom

58 SAP Types I SAP Frames in display order n 1: Closed GOP n 2: Closed GOP n 3: Open GOP n 4: GDR B B P T EPT =T PTF T SAP =T DEC 58 Institut Mines-Télécom

59 DASH: Segment Indexing n Allows random access into a media segment without downloading the segment Very hard to do with MPEG-2 TS Hard to do in ISOBMF n New indexing tool, the «sidx» box A table per segment Describes N entries (aka subsegments) N is configurable by the encoder/packager/indexer Each entry has: Duration and size SAP Type Time difference between the subsegment start and the SAP Indexing can be flat, hierarchical, or chained n «sidx» can be In the media segment (ISOBMF): Indexed Segment Outside of the segment (TS or ISOBMF): Indexing Segment Fetched via HTTP GET (w/ or w/o byte-range) 59 Institut Mines-Télécom

60 SIDX example for a segment SIDX - earliest PTS - for NbSubSeg - ref_type - size - duration - SAP_type - SAP_Offset SIDX - n SIDX describes the whole segment (no holes) 60 Institut Mines-Télécom

61 SIDX (Segment InDeXing) n Describes sub-segments and Stream Access Points (SAP)s in the segment. Byte offset and duration. access the sub-segments by the use of HTTP partial GET. 61 Institut Mines-Télécom

62 Summary of ISOBMFF Segments n n n n Initialization Segment ftyp box moov box with empty sample tables Movie fragment defaults Decoder configurations (for all bitrates!) Media Segment (or subsegment) «styp» Box 1 or more movie fragments (moof+mdat) Media data in the segment file Position (byte offset) of AU relative to the «moof» start A segment contains a least a Random Access Point Decoding time of the first sample of each track is given with the tfdt box for each fragment DTS are stored as delta not as absolute time in mp4 Indexed Segment Segment with «sidx» before the first fragment May have other «sidx» boxes in the segment Bitstream Swiching Segment Not used Vidéo Audio Vidéo Audio Vidéo Audio tfdt audio - tfdt video 62 Institut Mines-Télécom

63 Summary of MPEG-TS Segments n Segment (or subsegment) Sequence of TS packets (188 bytes) One program only An integer number of video frames Not necessarily integer number of audio frames n Initialization Segment (optional) PAT, PMT & PSI constant over the stream ECM if constant Optional If not present, PAT/PMT/ECM at the beginning of each segment n Bitstream Switching Segment (optional) Used to make concatenation of segments conformant with TS (PCR, buffer model ) n Indexing Segment ISOBMF Structure not stored in the TS styp, sidx pcrb: gives the PCR of the first byte of each segment 63 Institut Mines-Télécom

64 DASH: Types of URLs n String representing an HTTP URL SegmentBase, SegmentList n URL Template Enables constructing a URL from some segment parameters $RepresentationID$: ID of the representation $Number$: segment number $Bandwidth$: representation bandwidth $Time$: time of the segment start <SegmentTemplate startnumber="1" media="mp4-live-$representationid$-$number$.m4s" initialization="mp4-live-$representationid$-.mp4"/> 64 Institut Mines-Télécom

65 URL Addressing n URLs at each level of the MPD are resolved with respect to the BaseURL elements of levels above. n The base URL information may present on the following levels: MPD, Period, AdaptationSet, Representation n Alternative base URLs may be provided through the BaseURL element the identical segments are accessible at multiple locations. 65 Institut Mines-Télécom

66 DASH: Timelines (1/3) n MPD presentation timeline (T MPD ) T MPD = 0 at an arbitrary moment, typically when the first period starts (if period@start=0) n Period timeline (T Period ) Time relative to the beginning of the period (T Period =0 at the beginning of each period) n Mapping of Period time to MPD time: Linear mapping: T Period =0 T MPD = period@start period@start : time in the MPD time line from which the media in the period start to be processed n Media timeline (T media ) T media = 0 when the media presentation starts (typically when the first frame is presented) For ISOBMFF, movie timeline (i.e. with edit list) For TS, PTS n Mapping of Media Time to Period Time Linear mapping: T Period = T Media PTO PTO = Representation@presentationTimeOffset Consequences: If T period (FirstAU) < 0 (i.e. T Media (FirstAU) < PTO), the media is not presented (may need to be decoded) If T period (FirstAU) > 0, may have a gap in the presentation 66 Institut Mines-Télécom

67 DASH: Timelines (2/3) Segment Times n MPD Start Time of a Segment: Reference Time in the Period Time Line for a segment Used in live scenarios First segment in the period: T Period (segment(0)) = 0 Segment K in the period: T Period (segment(k)) = Sum(segmentDuration(j), 0, k-1) n Drift Constraint ABS(MPDStartTime(k) T period (FirstAU(k)) < 50%*segmentDuration 67 Institut Mines-Télécom

68 DASH: Timelines (3/3) Live case n Use of UTC Time line mpd@availabilitystarttime (AST): UTC time used to determine the availability of each segment To issue HTTP requests Synchronization server/client n Live edge If T UTC = now, find segment k such that? now >= AST + period@start + MPDStartTime(k) The segment will be available at : T UTC = AST + period@start + MPDStartTime(k) + duration(k) Adjust k depending on buffering constraints 68 Institut Mines-Télécom

69 DASH: SegmentTimeline n Goal Segments with variable duration Drift in segment availability Gaps in the content generation (black-out) n Principle Explicit signaling of each MPD StartTime Run-length encoding of segments with the same duration List of segments, for each MPD segment duration <0: duration not yet number of consecutive segments with the same duration <0: until the next element or until the end of the period 69 Institut Mines-Télécom

70 Which Segment addressing to choose? n List (playlist) : No <SegmentURL media="seg2.m4s"/> n Base (byte offsets): perfect for VoD <SegmentBase indexrangeexact="true" indexrange=" "/> n Number: ok <SegmentTemplate timescale="25000" media="seg_$number$.m4s" startnumber="1" duration="250000" initialization="seg_init.mp4"/> n Timeline: MSS compatible <SegmentTimeline> <S t="0" d="10000" r="59"/> </SegmentTimeline> 70 Institut Mines-Télécom

71 DASH: Live Issues n n Problem: UTC timing NTP servers can provide different UTC values!! Possible drifts between clients and servers Solution 1 Indicate the live-edge segment Using SegmentTimeline Live edge = last entry in the SegmentTimeline Problem: Requires frequent MPD updates Or requires client-guessing of segments not advertised n Solution 2 DASH 2015 Addition of <UTCTiming> to use adjust the provide UTC information NTP, SNTP servers HTTP servers: HTTP HEAD/GET to get o the «Date» header o An xs:datetime value in the body o An ISO 8601 value in the body o An RFC 5905 value in the body A date value in the MPD! 71 Institut Mines-Télécom

72 DASH: Switching n Based on the presentation times of 2 representations i.e. using the MPD Media Time, i.e. ISOBMFF Movie Time Not the decoding times (I/P/B <-> I/P) n Requires determining Random Access Points Presentation times of those RAPS Indicates if the concatenation of media segments is valid Otherwise, need to use an initialization segment or switching segment 72 Institut Mines-Télécom

73 DASH Descriptors n Basic Descriptor @id Common behavior: If 2 descriptors are present at the same level, with the only one of the two needs to processed n Derived descriptors Generic descriptors EssentialProperty: should not process if not understood Supplemental Property: additional information Role, Accessibility Define media role (main, caption, subtitle, alternate ) Mapping to HTML 5 Media-Specific descriptors Viewpoint, Frame-Packing, Audio Config Others Content Protection Asset Identifier 73 Institut Mines-Télécom

74 74 Institut Mines-Télécom

75 DASH ISOBMFF ondemand Profile n Netflix profile n Only one segment per representation Temporal alignment of sub-segments Sub-segments start with SAP Types 1, 2 or 3 n Mandatory Indexing using «sidx» All «sidx» boxes before the first «moof» n «Self-initializing» Segment Segment contains «moov» box: Segment = styp, moov, sidx, moof,mdat moof, mdat n Cannot be used for live streaming 75 Institut Mines-Télécom

76 DASH ISOBMFF Live Profile n N segments per representation Temporal alignment of segments Segments can start with SAP Type 1, 2 or 3 n Optional Indexing All «sidx» boxes before the first «moof» n Segment addressing Only SegmentTemplate Possibly with SegmentTimeline n Useful for Live or VoD scenarios Used by HBBTV 1.5! 76 Institut Mines-Télécom

77 DASH ISOBMFF Main Profile n N segments per representation Temporal alignment of segments Segments can start with SAP Type 1, 2 or 3 n Optional Indexing All «sidx» boxes before the first «moof» n Segment addressing SegmentTemplate or SegmentList Possibly with SegmentTimeline Byte-Range requests allowed n Useful for Live or VoD scenarios 77 Institut Mines-Télécom

78 78 Institut Mines-Télécom

79 DASH MPEG-2 TS Profiles n Main Profile N segments per representation Temporal alignment of segments Segments can start with SAP Type 1, 2 or 3 Optional Indexing n Simple Profile Same PSI in all representations Optional Indexing If CAS, ECM valid for the whole duration of a subsegment or segment if no sidx 79 Institut Mines-Télécom

80 80 Institut Mines-Télécom

81 New 2015 Profiles n ISOBMFF Extended ondemand and live Base: ondemand and live Adds support for xlink:href Content personalization Dynamic ad insertion n ISOBMFF Common Profile Combined use of ondemand et live profiles Profiles signaled at the MPD and Period levels Allows mixing a Live Period with an OnDemand Period Profiles adopted by HBBTV 2.0 et DVB-DASH profiles 81 Institut Mines-Télécom

82 DASH Exercises n Exercises Experiment with MP4 fragmentation/segmentation Generated DASH Sequences Different tools Different profiles Analyze MPD and segments n Tools GPAC MP4Box / MP4Client / DashCast 82 Institut Mines-Télécom

83 MP4Box DASH help n General DASH MP4Box h dash n Fragmentation MP4Box -frag 1000 file.mp4 n Segmentation MP4Box dash 2000 file.mp4 n DASH Profiles MP4Box dash 2000 profile live file.mp4 83 Institut Mines-Télécom

84 Some advanced topics n Advanced DASH n QoE/QoS: quality metrics, latency issues, switching issues n Encryption: CENC, Widevine, Marlin, PlayReady, FairPlay n Browser-perspective: HTML 5, Media Source Extension, Encrypted Media Extension, DASH.js n HLS + HEVC n Packaging On The Fly: benefits & challenges 84 Institut Mines-Télécom

85 DASH 2015 Extensions n Spatial Relationship Description Defines how AdaptationSets relate spatially 360 videos, panoramic videos, tiles, etc n Insertion of URL Parameters Reuse of MPD query string in segment URL Ex MPD URL: Segment Name without: With: p= Used for Authentification & Authorization OATC, OMAP, OAuth, SAML 85 Institut Mines-Télécom

86 DASH Quality Metrics n To report back the Quality of Experience (QoE) to the reporting server. n 3 conceptual observation points for measurements: 86 Institut Mines-Télécom

87 DASH Metrics n Collected information Start and duration of analysis Type of data Indicated in the MPD n Observation points TCP stack Segment assembly Player output n Network reports TCP: IP adresses and interfaces Opening/closing/connect times HTTP URL before and after redirects + byte-range Request/response times + code Download rates at different times n Switching reports Switching times (UTC and media), representations ID Buffer levels (segments) n Playback reports Actions: play/pause/stop/ Times 87 Institut Mines-Télécom

88 DASH Latency n Where is the latency? Confusion between Segments: DASH playlist media element Fragments: ISOBMF tool, at least one per Segment => latency due to fragment size Bitstream switching constraints RAP CDN Same content delivered to: 3G in subway Fibered couch potato DASH availabilitystarttime and minbuffertime Recommendations being written Separation of concerns: compute buffers from the lower layer n 88 Institut Mines-Télécom

89 Ad Insertion n XLink (dynamic referencing of external elements to a manifest), n DASH events: messages embedded directly in a MPD or transported in media segments n MPD Updates (refreshing the manifest at specific intervals). 89 Institut Mines-Télécom

90 Remote MPD Elements n Elements can be embedded in MPD with remote addressing using XLINK: n Unresolved, inconsistent, self-referencing will be considered invalid. n No conflict between internal and remote elements. n => Ad insertion 90 Institut Mines-Télécom

91 DASH Events n n n Problems How to signal an MPD change before an MPD Refresh E.g. CDN change How to deliver unforeseen data Alerts, etc Solution Event: Time and duration Type (URI) Payload EventStream Inband Event Defined at the period level One stream per event type Signaled by InbandEventStream Message delivered in the emsg box At the start of an ISOBMFF segment In PID 0x0004 in MPEG-2 TS Current Events Types MPD no longer valid MPD Patch (full replace or XML patch) 91 Institut Mines-Télécom

92 Common Encryption n Encrypted once, deliver licenses many ways Provides a mechanism for delivering same encrypted segments to various client supporting different proprietary DRM systems. n Can be applied to media segments, subsegments or individual samples. n Signals algorithm identifier, key identifier and initialization vector size. Using track encryption box or sample groups. n Use pssh box for opaque information of each specific DRM scheme. n A new standard: ISO/IEC Institut Mines-Télécom

93 HTML 5 Media Source n W3C Recommendation in progress Extension to the HTML5 recommendation Separation between networking and playback Enables adaptive streaming in Web Browsers n JavaScript API Controlling media decoding buffers Add, delete media data Supported by Google Chrome, MS IE 11, Firefox n JavaScript-based players (MP4-based) DASH.js DASH-JS YouTube Test player Akamaï 93 Institut Mines-Télécom

94 Media Source Architecture 94 Institut Mines-Télécom

95 DRM on the Web n EME: Encrypted Media Extensions W3C Recommendation in progress Extensions to the HTML5 recommendation Used in conjunction with Media Source Extension Enables the streaming of encrypted content and the secure decryption JavaScript API to access to a (proprietary) Content Decryption Module (CDM) Communication with the License Server is out-of-scope n Implementation of CDM is out-of-scope n In use by Netflix with Microsoft IE Institut Mines-Télécom

96 96 Institut Mines-Télécom

97 New formats n HEVC Carriage of HEVC in TS: done Carriage of HEVC in MP4: done Use of HEVC in DASH: done, GPAC demo Use of HEVC in HLS+TS: possible but not authorized n Subtitles WebVTT vs TTML formats Carriage of WebVTT & TTML in MP4: done Use of WebVTT & TTML in DASH (in MP4 or in separate): done Use of TTML in TS (HLS): not yet defined 97 Institut Mines-Télécom

98 Questions! Jean Le Feuvre Cyril Concolato 98 Institut Mines-Télécom