Multimedia Networking: File Formats E506

Transcription

1 Multimedia Networking: File Formats E506 1

2 Multimedia File Format Types n Files for storing media data Single elementary stream files Ex: audio only Multiplex of elementary streams Ex: Audio + video + subtitles n Files for storing associated data Presentation Information E.g. HTML Metadata Descriptive metadata (author, date ) DRM-related metadata (License, rights) Different formats: XML, MPEG-7, EXIF, n Mixed files 2

3 Usages of Multimedia Files Streaming Editing Recording Progressive Download Playback / Seek Search Archival Storage Extract 3

4 Typical file-based operations n Simple Playback Ordered extraction of Access Units from time 0 n Seeking Play from time T n Concatenating files A.k.a. splicing, e.g. ad-insertion n Splitting a file in two E.g. Shortening the duration of a file n Editing operations Adding / removing a stream Audio languages, subtitles Enhancement streams (Scalable, Multiview) Adding / removing Access Units Temporal layers (B-frames) 4

5 Examples of Multimedia File Types n DVD MPEG-2 Program Stream (VOB) + Navigation Information (IFO) n Blue Ray Disc MPEG-2 Transport Stream + other files n Flash Video FLV, F4V n Microsoft AVI, ASF, WMV, WMA n Music AAC, MP3, M4A, AC3, WebA n Video MP4, 3GP, AVC, PSP, MKV, WebM Ogg, ogv n 5

6 Container vs. codec n Codec Software/hardware/ algorithm to encode/decode a media stream n Container file May contain multiple streams (audio, video, subtitles, ) Streams may use different codecs (AVC H264, MP3, AAC, VP8, ) n Some container files are restricted to use some codecs E.g. WebM = (VP8 VP9 VP10) + (Opus Vorbis) 6

7 Creating and Reading Media Files Application Associated data Application Encoder Decoder Encoder File Creator Media File File Reader Decoder Encoder Decoder Media Access Units & Decoder configuration 7

8 Identifying File Types n n n Using the file extension Ex: mp3, mp4, avi, Basic, unreliable: may be changed, may be absent in URL Does not identify the number and types of media streams in the file Looking at the first bytes (a.k.a. sniffing, probing) Detection of header Ex: XML files Search for a registered magic number Ex: GIF89a Search for well-known bytes Ex: ftyp in MP4/ISOBMFF Sync marker in MPEG-2 TS Use of MIME type advertised by the server/protocol (ex: HTTP) <type> / <subtype> ( ; parameters )* 5 major types: audio, video, image, text, application Subtypes specific to a payload ( x- and vnd.* are proprietary) Ex: video/mp4, audio/ogg Parameters: name=value (e.g. codecs) 8

9 Storage of an elementary stream Simple approach n Principle Store Access Units consecutively in decoding order Ex: MP3 frames AU AU AU AU AU n Advantages Simplicity Low overhead Progressive download and playback n Problems Hard to seek into the file Hard to find the beginning of an AU Requires a deep parsing of the bitstream Specific for each stream format Hard to find the timing of an AU Ex: sum of durations of previous AU Hard to find media decoder configuration Most of the time in-band with RAP AUs. 9

10 Storage of an elementary stream Variations n n Adding a framing mechanism Means to determine beginning and end of an AU Ex: Use of Start Code (not emulated in the AU payload) SC L AU Use of AU header with length AU SC AU Examples H264: NAL Units with start code AAC: ADTS or LATM L AU SC L AU AU SC L AU AU n Problems Seeking still requires parsing (of start code or headers) and skipping of unneeded AU Start code or header format specific to the media format Still hard to find media decoder configuration 10

11 Start codes in AVC H264 and HEVC 0x AU 0x AU what is encoded what is written/read 0xfe4a b788 Wrong detection if the payload contains a byte sequence equivalent to the start code 0xfe4a b788 Emulation prevention byte: one byte is inserted in the payload to break any byte sequence involved in start-code detection 0x x x

12 Start codes in MP3, AAC+ADTS 0xfe L AU 0xfe L AU 0xa2b8fed8 Wrong detection if the payload contains a byte sequence equivalent to the start code Detection done by checking next payload start 0xfe L AU DATA L bytes If SC cur +Length!= start code SC cur is not start code 12

13 Structured Storage n Principle Use of framing mechanisms independent from the elementary stream type Separation between media data and framing data (length, positions) Separate storage of decoder configuration n Examples AVI, ASF, MP4, MKV n Pros Enables generic parsers, file format handlers Enables fast seeking, searching Enables easy editing n Cons Bigger file size Not so simple playback 13

14 Structured Storage Timing information T T T T Framing data (position, size, offset) F F F F Media data AU AU AU AU Decoder Configuration DecCfg n Framing and Timing data is small compared to media data Easy to scan and seek n Specific location for decoder configuration Easy to initialize 14

15 Specific aspects for multi-stream storage n Access Unit Storage Different access unit sizes Support for flexible interleaving n Framing/Timing Storage Different durations, time scales Relationships between streams (synchronization, decoding dependency) 15

16 Multiple Stream Storage No Interleaving 16

17 Multiple Stream Storage With Interleaving 17

18 Auxiliary Media File Formats 18

19 Redirection files n Principle Separate the actual location (server) of the media data from the location of the multimedia applications (web page) Enables delivery mode changes without changing the web page Similar to a playlist n Technology Real Media (old format) Microsoft ASX: Advanced Stream Redirector XML file (also used for WVX, WAX) Apple QuickTime Reference Specific signaling in MOV files 19

20 Playlists n List of media files to be played in sequence Ex: m3u, m3u8, pls, n Advanced usages Seamless playback Stream splicing (ad-insertion) Bitrate changes: HTTP Streaming Microsoft IIS Smooth Streaming Flash HTTP Streaming Apple HLS MPEG-DASH 20

21 Typical File Formats 21

22 The Flash File Formats n SWF Files MIME type: application/x-shockwave-flash Simple storage of audio/video mixed with graphics and script n FLV Files MIME type: video/x-flv Media-only (no script or graphics) Max 1 audio & 1 video streams, synchronized Tag-based storage à la SWF Video and Audio formats: same as SWF Used by Flash Media Server (v6) & played by Flash Player (v7) n F4V, F4P, F4A, F4B Files Used in Flash 9 Same as ISO base media file format 22

23 The Audio-Video Interleave (AVI) File Format n n n AVI Files Historical file format (Microsoft, 1992) MIME types: video/avi, video/msvideo, video/x-msvideo Hierarchically structured into chunks Type on 4 bytes a.k.a. 4 ASCII Character Codes: 4CC Length on 4 bytes Typical chunks RIFF hdlr Contains general metadata about the file: number of streams, types of streams, width & height of video, movi Contains the actual access units idx1 Contains random access information 23

24 The Advanced Streaming Format (ASF) File Format Header Object n Latest Microsoft File Format for Windows Media Extensions:.asf,.wma,.wmv files MIME types: video/x-ms-asf, application/vnd.ms-asf n Hierarchically structured into ASF Objects Header General Information on the file Specific Information on each stream Codecs, encryption, metadata Data Fixed-size data packets interleaved in transmission order Index optional, used for RAP File Properties Object Stream Properties Object 1 Stream Properties Object N<Other header objects> Data Object Data Packet 1 Data Packet M <Other top-level objects> Index Object 1 Index Object K Simple Index Object 1 Simple Index Object L 24

25 The OGG Format n Extensions.ogg,.ogv,.oga,.ogx MIME Type: video/ogg, audio/ogg, application/ogg 25

26 The Matroska File Format n Extensions.mkv,.mk3d,.mka,.mks MIME Type: video/x-matroska audio/x-matroska.webm Google s format Restriction of the Matroska format to VP8, VP9, VP10 video codecs and Vorbis audio codecs MIME Types: video/webm 26

27 Matroska File Structure n Structured into containers Header (version ) Metaseek container Fast search of signaling info Segment Info Identifying the file type Track type, codec, resolution Chapters Clusters Raw data, Access Units Cueing Timing information and offsets within the clusters Attachment Other data than timed data Tagging ex: ID3 Tag 27

28 Tools n MediaInfo n FFmpeg n File explorers n MP4Box n Handbrake n 28

29 Multimedia Networking: The ISO Base Media File Format Family of Standards 29

30 History n Development of the MPEG-4 Standard (1998) Requirement for support for more streams than just 1 audio & 1 video Call for Proposals Selected Base Technology: QuickTime n Development of Motion JPEG-2000 Standard Call for Proposals Selected Base Technology: QuickTime n Consequence Development of a common standard: The ISO Base Media File Format based on QuickTime File Format n Successful standard MPEG, JPEG, Flash, 3GPP, Microsoft, Sony, 30

31 Multiple Standard Specifications ISOBMFF MPEG-4 Part 12 = JPEG-2000 Part 12 MPEG-4 Part 14 (MP4) MPEG-4 Part 15 (AVC, HEVC) MPEG-21 Part 9 (MP21) MPEG- DASH MPEG-4 Part 30 (WebVTT, TTML) 3GPP / 3GPP2 Flash (F4V) n Core standard extended for Specific MPEG-4 Systems storage Specific H264 AVC, SVC, MVC, HEVC requirements Storage of XML or non-timed data HTTP Streaming features (MPEG-DASH) Storage of subtitles 31

32 ISOBMFF: Principles n Logical Structure Information organization n Physical Structure Byte organization Design rules 34

33 ISOBMFF: Logical Structure (1/4) n A file Contains Timed media data of a presentation (audio, video) Non-timed data (images, files) Network protocols helpers ( packet builders ) Meta-data describing the above and their relationships Defines a common timeline for all tracks for synchronization Declares its type and its compatibility FileTypeBox Main usage type (eg, AAC+H264 for download) Tools required by file reader to process the file 35

34 ISOBMFF: Logical Structure (2/4) n A track Corresponds to timed data of a specific media, May support streams with varying coding types (codec) Is decomposed into samples Of variable sizes and durations Usually 1 track ó 1 decoder except for hierarchical layered codecs where N tracks => 1 decoder Has associated decoder configuration(s): sample description(s) Most of the time, decoder config valid for the whole track duration Some configuration allow in-band configuration in the track May be linked to, grouped with or alternative to other tracks May be encrypted May have associated metadata or user data attached 36

35 ISOBMFF: Logical Structure (3/4) n A sample Represents AU used by the compression system Has properties: times (DTS, CTS), size, position, random access, decoder configuration May be described in terms of sub-samples E.g. an HEVC/AVC AU decomposed into NAL units May be associated to similar samples in sample groups E.g. samples that have the same dependencies on an other sample May have sample-specific auxiliary information E.g. samples that use the same encryption key 37

36 ISOBMFF: Logical Structure (4/4) n An item Represents data consumed as a whole and valid for the entire duration of the movie E.g. metadata: copyright information E.g. cover image May have associated properties (type, position, size ) May be encrypted and/or compressed If multiple items are used, the primary item is the entry point 38

37 ISOBMFF: Physical Structure n Data is stored in a basic structure called box No data outside of a box n Each box has length (4 or 8 bytes), type (4 printable chars), possibly version and flags, and type-specific data n Extensible format Unknown boxes can be skipped (syntactically) n Header information is a hierarchical set of boxes typically rooted by a moov or meta box 39

38 ISOBMFF: Design Principles n n Media extensibility No need to modify the core structure to define support for new coding formats Derived specifications define Sample entry boxes (give codec name and configuration) Sample content What is a RAP and other sample properties Run-length encoding Ex: if N consecutive samples share the same property, use compact description PropertyXXXBox Sample1 foo Sample2 bar Sample3 bar Sample4 bar Sample5 bar Sample6 foo Sample7 bar Sample8 bar Sample9 bar Flat representation Less compact Less processing PropertyXXXBox SampleCount 1 Value foo SampleCount 4 Value bar SampleCount 1 Value foo SampleCount 3 Value bar RLE representation More compact More processing 40

39 Identifying ISOBMFF files n n Extension not sufficient mp4, m4a, m4s, 3gp, f4v, ism, Magic number/sniffing: mandatory structure in files: ftyp for files with headers styp for files without headers (remaining segments of a file) ftyp / styp indicate brands Major brand: best use of the file Compatible brands Example: isom, avc1, isox (X=2 9), mp41, mp71, n MIME types (RFC 6381) video/mp4 : if it contains visual data audio/mp4 : otherwise, if it contains audio, application/mp4 : otherwise (in particular metadata, ) Sub-parameters codecs Comma-separated list of track information Uses the sample entry 4cc: avc1, mp4a, stpp, Additional codec-specific information (profiles, levels ) profiles brands 41

40 Important Boxes ftyp (File Type) One per file Version Compatibility with other file types mdat (Media Data): N per file moov (Movie): High-level Box for signalling information for the movie mvhd (Movie Header): Generic information about the movie trak (Track): High-level Box for signaling info for the track meta (Metadata): High-level Box for signaling info for metadata hdlr (Handler) Indication of the type of the track or the type of metadata dinf/dref (Data Information/Data Reference) Location of media data (local vs. remote) stbl (Sample Table) High level box for sample signalling information stsd (Sample Description) Low-level box for signalling elementary stream configuration stts (Sample To Time) Low-level box for signalling DTS for each sample Run-Length encoded stsz (Sample To Size) Low-level box for signalling the size of each sample Run-Length encoded And many more 42

41 ISOBMFF: Physical Structure Separate Data vs. Header storage Header information describing the properties of samples, of tracks is stored separately from the sample data (header vs payload) Movie header Track header Track header Video track information Audio track information Movie Media Data sample sample frame sample sample frame 43 43

42 Media data storage n n n n Unstructured, bag of bytes AUs of different tracks may be interleaved Cannot extract sample data without using header information Data bytes are stored in one or more boxes of specific types mainly mdat (or idat for item data in some cases) Usually in the same file as the moov or meta header may also be stored in a separate file that doesn t use boxes Sample data storage The bytes of a sample are stored contiguously (except if extractors are used, eg SVC) Bytes of multiple time-contiguous samples can be stored contiguously (called chunks) Chunks of different tracks can be interleaved Item data storage The bytes of an item can be stored in separate byte ranges (called extents) Item data and media data can be interleaved 44

43 Physical Storage of Samples Box type: moov Meta-data Box type: mvhd Creation date Duration Box type: iods ES descriptors Media-data Box type: mdat Box type: trak Box type: mdia Box type: minf Box type: stbl Box type: stsd Box type: mp4v Box type: esds Dec config Descr SL Config Descr Box type: tkhd TrackID Width, height Box type: stco Chunk i Chunk i+1 AU j AU k AU k+1 AUl Chunk (i+1) 45

44 Typical Box Hierarchy (1 track) ISO File ftyp moov mdat mvhd trak iods tkhd mdia... Data not box-structured mdhd minf hdlr vmhd dinf stbl dref stsd stts stsz 46

45 Typical Box Hierarchy (Untimed data) ISO File ftyp meta mdat hdlr iinf iloc ipro sinf 47

46 Dual Headed File: Timed and Untimed data File ftyp moov mdat meta mvhd trak hdlr iinf iloc ipro tkhd mdia tref sinf mdhd minf hdlr vmhd dinf dref stbl stsd stts ctts stsc stsz page 49

47 Progressive Download of ISO Files n Interleaving of a set of AUs N seconds of audio, N seconds of video, The contiguous (bytes in file) set of AUs for a given media is called chunk or run n Constrained box order ftyp, moov, meta first Optional pdin box for informing on initial delay before playback mdat last n Limitations Cannot cover live cases as the whole moov box is written first and only 1 moov box allowed per file 50

48 First Byte = Download start Exemples Movie Media Data Sample 1 Sample 1 Sample 2 frame Sample 2 frame Movie header Track header Track header Sample information Sample information Last byte = Download end Last Moov byte = Playback Start n Non-optimized file Playback cannot start until all sample info is known 51

49 Exemples First Byte = Download start Movie header Track header Track header Sample information Sample information Movie Media Data Sample 1 Sample N Sample 1 frame Sample N frame Last Moov byte = Playback Possible Last byte = Download end Last audio byte but no video Video + audio available, playback starts n Non-optimized file Synchronized playback cannot start until enough samples from both tracks are downloaded 52

50 First Byte = Download start Exemples Movie header Track header Track header Sample information Sample information Movie Media Data Sample 1 Sample 1 Sample 2 frame Sample 2 frame Last Moov byte = Playback Possible Last byte = Download end n Optimized file Video + audio available, playback starts Playback can start after downloading only a portion of the file 53

51 Exercises on Media Interleaving n For 2 tracks file with Audio AU duration 1024/44100 = 23,2199 ms Video AU duration 1/25/ = 40 ms n Basic interleaving pattern Storing {N audio AUs, N video AUs}, with N=10 Video AU time after 1 min of audio? Audio AU time after 1 min of video? n Time interleaving pattern Storing {500 ms audio AUs, 500 ms video AUs} Video AU time after 1 min of audio? Audio AU time after 1 min of video? n Propose another interleaving algorithm 54

52 Movie Fragments n Single moov box limitation Cannot be written until the entire data is known Problematic for capture/live recording Size can be quite large for few hours movies Problematic for memory optimization Solution Being able to regularly flush the header upon recording But only one header allowed n Introducing «Movie Fragments» Initial structure (moov + N trak) Same as regular file Plus indication that additional movie fragments exist Followed by a sequence of new boxes for signaling moof + N traf Each traf is made of K trun Each trun stores a contiguous (in bytes) set of samples Additional data for fragments still stored in mdat boxes 55

53 Movie Fragments Box Hierarchy ISO File ftyp moov mdat moof mdat mvex mfhd mehd traf traf trex trex tfhd trun trun 56

54 Structure of a fragmented file moov Video track trak trex Audio track trak trex Movie mdat traf traf moof trun trun trun Movie Fragment mdat traf traf moof trun trun trun Movie Fragment mdat 57

55 Interleaving in a movie fragments moov Video track Audio track trak trex trak trex Movie Movie Fragment Traf Video Traf Audio Traf Video Traf Audio moof trun trun trun trun mdat Movie Fragment moof Traf Video trun trun Traf Audio trun trun mdat 58

56 Sample information in fragmented files n Header compression Default sample information Duration, size, sample description sample properties (sync, depended, ) «sample_flags» Possibly coded in 3 places: moov/mvex/trex: default for all samples in the track moof/traf/tfhd: default for all truns in the moof moof/traf/trun: default for the first sample n Practically Some streaming systems (Smooth, CMAF) imposes defaults to be in tfhd No factorization between movie fragments 59

57 Movie vs Movie Fragments n moov mvhd trak chunk, sampleperchunk time, CTS offset, rap, size chunk offset sample groups (stbl) n moof mfhd traf «trun» In trun In trun sample groups (traf) n Notes Offsets in fragments may be From the beginning of the file From the beginning of the moof for the first traf, then subsequent From the beginning of the moof for each traf (MPEG-DASH) Media configuration (sample description) still in the moov (unless inband, in the sample data like avc3 ) 60

58 Hint Tracks n Special Track Dedicated to a delivery protocol Stores timed instructions to create protocol packets Linked to a media track Not (necessarily) containing media data n Examples Hint track for streaming of MP4 using RTP Hint track for delivery over FLUTE n Variant: Reception Hint Track Storage of delivery streams in ISO (MPEG-2 TS and RTP) 61

59 Hint track and other tracks Movie (Signalling) Video track moov trak Hint track trak Sample Data mdat sample hint header pointer sample frame sample hint header pointer sample frame 62

60 ISOBMFF: File Types & Usages (1/2) n Plain Files Simple recording of timed data (data first, header last) ISOBMFF Tools: mdat, moov, n Progressive Files Progressive download and playback (Header first, data last and interleaved) ISOBMFF Tools: storage using multiple interleaved chunks n Fragmented Files Files for long-running recording sessions (multiple blocks of header and data) ISOBMFF Tools: Movie fragments 63

61 ISOBMFF: File Types & Usages (2/2) n Segmented Files Self-contained fragments stored in single file or in separate files for HTTP streaming ISOBMFF Tools: segment files, indexing n Packaging files Storage of related timed or untimed data (e.g. JPEG or XML + audio/video) ISOBMFF Tools: moov and meta box n Files for generating transport-related streams Protocol-specific instructions to create streams from files ISOBMFF Tools: hint tracks (RTP, FLUTE, ) n Files recording transport-related streams Recording of protocol-specific packets into files for replay ISOBMFF Tools: reception hint tracks (RTP, MPEG-2 TS) 64

62 ISOBMFF Timelines n Media Timeline n Movie Timeline 65

63 ISOBMFF Media Timeline n Expressing times of each AU in its track n Boxes involved MovieHeaderBox mdhd : gives the timescale TimeToSampleBox stts : provides the DTS for each sample CompositionOffsetBox ctts : provides CTS for each sample, when needed cslg :additional info for specific CTS/DTS config TrackFragmentDecodeTimeBox tfdt : time anchor for movie fragments TrackRunBox trun : timing for movie fragments relative to tfdt 66

64 Basic Media Timeline Information n stts / trun coding DTS values given as sample_delta values DTS(0) = 0 DTS(i) = SUM(sample_delta, 0, i-1) Values are run-length encoded n ctts / trun coding Offset from the DTS Box version 0: offset >=0 Box version 1: offset >=0 or < 0 CTS = DTS + Composition Offset Values are run-length encoded 67

65 Media Time in Movie Fragments n Default behavior T M (0) = 0 T M (i+1) = T M (i) + Duration(i) n Special case In some scenarios (e.g. DASH), upon receiving of a first fragment, the end time of the previous sample is unknown! Need to insert the media time of the first sample in a fragment TrackFragmentBaseMediaDecodeTime tfdt If tfdt > T M (i), the duration of sample i-1 is extended Useful for variable frame rate streams, e.g. for subtitles 68

66 ISOBMFF Movie Timeline n n n Expresses times for the presentation of the movie from its tracks Main use for multi-tracks synchronization E.g. CTS track1 (AU 0 ) ó CTS track2 (AU 0 ) Useful for single track presentation when not all samples are presented E.g. start presenting track 1 from the 3rd second Boxes involved MovieHeaderBox mvhd : gives the timescale for movie times EditListBox edts : gives instructions for presenting a track SegmentIndexBox sidx : timing for random access points in segments Track media times (i.e. CTS) are mapped onto the Movie Timeline Default: simple, linear mapping T(movie, track(i), sample(k)) = CTS(track(i), sample(k)) * mvhd.timescale/mdhd.timescale Complex, non-linear mapping with the concept of Edit List 69

67 ISOBMFF Edit List n List of edit entries type, media time, movie duration n Types of edit entries dwell (frame freeze), empty (no presentation), Normal playback (with possible speed changes) n Media times do not necessarily align with frame start times! 0 0 empty normal dwell normal T 1 T 2 T 3 T(movie, track(i)) CTS (track(i)) 70

68 Typical Edit List example n n n Use case: File with 1 audio and 1 video stream Case 1: video with B frames Presentation timestamp of the first video frame: CTS video (AU 0 ) >0 Case 2: first audio AU not produced at the same time as the first video AU For each track DTS(AU 0 ) is by definition 0! Option 1: audio shift Video track: no edit list Audio track: One «empty» edit list entry from 0 to T(movie, F 0 ) One «normal» entry from CTS=0 to the end of the track Drawback: need to apply the edit for every new track (subtitles ) Option 2: video shift Audio track: no edit list Video track: one «normal» entry from CTS(F 0 ) to the end of the media 71

69 Sample Groups n n n Mechanism to store additional sample signaling A box providing descriptors (sgpd) Located in stbl: global for the entire track Located in traf: applicable only for the fragment A box providing which descriptor apply to which sample (sbgp) Located in stbl Located in traf Grouping types Each descriptor/group is identified by a 4CC Ex: rap, roll, prol Contains several entries Ex: one entry per «roll» distance The mapping sample->group can be parametrized ex: view priority in MVC In practice Smooth Streaming imposes that descriptions be on fragments only 72

70 Sample Auxiliary Information n Possibility to add data associated to a sample Not used directly by the decoder Ex: data for decrypting the sample Without having to define a new track Can be used in fragmented and non-fragmented mode n Tools Data type indicated by a 4CC (ex: «cenc») Box saiz indicates the additional data sizes Box saio indicates the additional data offsets Additional sample data shall be in the same file as sample data 73

71 Random Access Points n n n sync box If absent, all samples are RAP (e.g. audio streams) If present, RAP are signaled (I-frame, IDR) Sample Groups «rap»: non-idr intra frames Also indicates number of leading samples «roll»: nb samples to decode until perfect reconstruction is reached «prol»: audio, nb samples before this sample to decode to produce perfect reconstruction for this sample Independent and Disposable Samples is_leading: signaling for open GOP sample_depends_on: signaling for I frames or not sample_is_depended_on: signaling of reference frames sample_has_redundancy: redundant coding Can be used to signal duplicated samples for text streams 74

72 Refresher: leading sample n A sample after a Random Access Point in decode order, but presented before the RAP May be decodable (not often) Or not (OpenGOP) OpenGOP REF I 3 B 1 B 2 P 4 t GOP start Leading Samples not decodable if seek here IDR leading I 3 B 1 B 2 P 4 t GOP start Leading Samples decodable if seek here 75

73 Dependencies between tracks n Track reference: tref track N uses or refers to track(s) K Examples hint : hint track for one or more media track chap : chapter track for the referenced media tracks scal : scalable track using data from media tracks n Track group: trgr Tracks in the same group share a common feature e.g. may be subtitle tracks n Track Selection: tsel Provides selection information for alternate tracks E.g. for scalable tracks 76

74 ISOBMFF for NALU-based streams (part 15) n Principles Reuses the ISOBMFF structures Defines sample description entry format and sample format n Sample format Each NALU preceded by a 1, 2, or 4 bytes Exact size given in the sample description entry No start codes Random Access Points «SyncSample»: only IDR SampleGroup rap for all the others 77

75 AVC/HEVC specificities n Sample description entry formats avc1 / hvc1 All Parameter Sets in a box avcc / hvcc Decoder configurations cannot be changed without a new moov! avc3 / hev1 Parameter Sets in IDR or in the sample description entry n Specific boxes avcc ou hvcc profile tier level, chroma format & bit depth PPS & SPS lists HEVC: + VPS & SEI messages 78

76 Storage of Interactive Applications in ISO n Static Interactive Application Storage as metadata Ex: HTML, XML, n Dynamic, Timed Application Storage as track Ex: MPEG-4 Scene Description Streams, 3GPP DIMS Streams n Case of Images Image as Tracks Image as Metadata item 79

77 SVC/MVC specificities n n Sample entry Configuration svcc or mvcc mvdc, extends avcc avc2 : avc1 with NALU SVC or MVC avc4 : avc3 avec NALU SVC or MVC Each layer can be stored in a separate track svc1, mvc1 No base layers in these tracks svc2, mvc3, mvc4: SPS extension in the samples Possibility to split MVC and MVC+DEPTH mvdx: depth only mvcx: Multiview without depth if no mvdc Multiview with depth if mvdc present Handling NAL units Extractor NAL Unit Share data between tracks Copy 1 or N NAL from a sample in another track Source Track : via scal reference types Source Sample: same decode time +/- sample offset Aggregator NAL Unit Groups a set of NALs as a single NAL Simplifies NAL descriptions in sample group description 80

78 Extractors Movie Info Avc1 track moov sample SVC track scal reference sample Sample Data mdat Extractor NAL * Pointer NAL3 NAL4 NAL1 NAL2 81

79 HEVC Specificities n Tile Tracks Possibility to store independent (motion constrained) tiles in separate tracks entry type: hvt1 for tile tracks, hvc2 for base tile track Original sample build by concatenation of tile track samples Order given by sabt track references from base track to tile track Movie Info hvc2 track moov sample sabt reference hvt1 track sample hvt1 track sample Sample Data NAL1 NAL3 mdat SPS/PPS/VPS SEI Messages NAL2 NAL4 82

80 HEVC Tiling in ISOBMFF n n n n n n 1 tile base track (hvc2) for SPS/PPS/SEI N tracks (hvt1), one for each independent tile Decoding of base + 1 tile track: 1 tile base track (hvc2) for SPS/PPS/SEI N tracks (hvt1), one for each independent tile N tracks (hvc2) rewriting each HEVC tile track: Have their own decoder config (SPS/PPS/ ) Use inband data extractors to replace slice header Extractor reference data: slice payload Slice Data Slice Header: address!=0, pic_start!=0, 83 Extractor inband data, new slice Header: address==0, pic_start==0,