HPA Thursday, February 21, 13

Transcription

1 HPA 2013 File Based Production Architecture End to End Or what it feels like to be between a rock and a hard place presented by Andrew G. Setos and Jim DeFilippis

2 WHAT S THE BIG DEAL?

3 WHAT S THE BIG DEAL? Point: What is the problem? Seems like there are plenty of movies and TV shows coming out of Hollywood.

4 WHAT S THE BIG DEAL? Point: What is the problem? Seems like there are plenty of movies and TV shows coming out of Hollywood. Counterpoint: Today we are managing, but look at the trend, we are heading for a data tsunami

5 THE BIG ISSUES

6 THE BIG ISSUES

7 THE BIG ISSUES The Storage Challenge

8 THE BIG ISSUES The Storage Challenge Archiving the data

9 THE BIG ISSUES The Storage Challenge Archiving the data File Format anarchy

10 GROWTH OF DATA

11 GROWTH OF DATA The conversion to digital formats has fueled the increase of media data.

12 GROWTH OF DATA The conversion to digital formats has fueled the increase of media data. A TV show (HD) generate 30hrs of 2.7 TB. A season is 60TB. 3D doubles this storage; 4k quadruples the storage (1/4 Petabyte).

13 GROWTH OF DATA The conversion to digital formats has fueled the increase of media data. A TV show (HD) generate 30hrs of 2.7 TB. A season is 60TB. 3D doubles this storage; 4k quadruples the storage (1/4 Petabyte). Movies (DCI) generate 1-2 PB per title; 3D more than double (5-6TB); HFR can double or triple these numbers (10-15 TB).

14 GROWTH OF DATA The conversion to digital formats has fueled the increase of media data. A TV show (HD) generate 30hrs of 2.7 TB. A season is 60TB. 3D doubles this storage; 4k quadruples the storage (1/4 Petabyte). Movies (DCI) generate 1-2 PB per title; 3D more than double (5-6TB); HFR can double or triple these numbers (10-15 TB). Distribution. Each new on-line distributor requires their own file formats, usually at 1-4Mb/s. For a 1000hr library, this represents 0.5 to 2 PB for each distribution deal

15 FLASH: Terror-Bites of Media Files Threaten Hollywood! TV Movies

16 FLASH: Terror-Bites of Media Files Threaten Hollywood! Terabyte TV Movies

17 FLASH: Terror-Bites of Media Files Threaten Hollywood! Petabyte Terabyte TV Movies

18 FLASH: Terror-Bites of Media Files Threaten Hollywood! Exabyte Petabyte Terabyte TV Movies

19 FLASH: Terror-Bites of Media Files Threaten Hollywood! Exabyte Petabyte Terabyte 1,000, ,000 10,000 1, TV Movies

20 FLASH: Terror-Bites of Media Files Threaten Hollywood! Exabyte Petabyte Terabyte 1,000, ,000 10,000 1, TV Movies

21 FLASH: Terror-Bites of Media Files Threaten Hollywood! Exabyte Petabyte Terabyte 1,000, ,000 10,000 1, TV Movies

22 FLASH: Terror-Bites of Media Files Threaten Hollywood! Exabyte Petabyte Terabyte 1,000, ,000 10,000 1, TV Movies

23 FLASH: Terror-Bites of Media Files Threaten Hollywood! Exabyte Petabyte Terabyte 1,000, ,000 10,000 1, TV Movies

24 FLASH: Terror-Bites of Media Files Threaten Hollywood! Exabyte Petabyte Terabyte 1,000, ,000 10,000 1, TV Movies

25 FLASH: Terror-Bites of Media Files Threaten Hollywood! Exabyte Petabyte Terabyte 1,000, ,000 10,000 1, TV Movies *estimated and extrapolated from multiple sources of information

26 HOW TO STORE ALL THIS CONTENT?

27 HOW TO STORE ALL THIS CONTENT? Lots and lots of spinning drives?

28 HOW TO STORE ALL THIS CONTENT? Lots and lots of spinning drives? What about Solid State or Optical?

29 HOW TO STORE ALL THIS CONTENT? Lots and lots of spinning drives? What about Solid State or Optical? Data tape??

30 HOW TO STORE ALL THIS CONTENT? Lots and lots of spinning drives? What about Solid State or Optical? Data tape?? Hey, how about the Cloud?

31 THE CLOUD

32

33

34 RIGHT APPROACH TO STORAGE

35 RIGHT APPROACH TO STORAGE Each type of data storage has it s purpose.

36 RIGHT APPROACH TO STORAGE Each type of data storage has it s purpose. Need to assess the best solution for each stage of production

37 RIGHT APPROACH TO STORAGE Each type of data storage has it s purpose. Need to assess the best solution for each stage of production This is a dynamic situation as current technologies improve and new technologies become available.

38 STORAGE PRO/CON

39 STORAGE PRO/CON Type Use Case Pro Con

40 STORAGE PRO/CON Type Use Case Pro Con Solid State Memory (SSM) Acquisition reliable, fast, low power high cost, not suitable for long term storage

41 STORAGE PRO/CON Type Use Case Pro Con Solid State Memory (SSM) Acquisition reliable, fast, low power high cost, not suitable for long term storage Hard Disk Drive Post Production systems Playout systems Reliable (RAID configuration) Moderate cost Large bandwidth Fragile Power hungry

42 STORAGE PRO/CON Type Use Case Pro Con Solid State Memory (SSM) Acquisition reliable, fast, low power high cost, not suitable for long term storage Hard Disk Drive Post Production systems Playout systems Reliable (RAID configuration) Moderate cost Large bandwidth Fragile Power hungry Optical Distribution Long term storage low cost reasonable life time (20yrs) slow read/write access

43 STORAGE PRO/CON Type Use Case Pro Con Solid State Memory (SSM) Acquisition reliable, fast, low power high cost, not suitable for long term storage Hard Disk Drive Post Production systems Playout systems Reliable (RAID configuration) Moderate cost Large bandwidth Fragile Power hungry Optical Distribution Long term storage low cost reasonable life time (20yrs) slow read/write access Data Tape Long term storage Archive Lowest cost sufficient read/write access times for archive, need to be incorporated into a library with a migration strategy

44 STORAGE PRO/CON Type Use Case Pro Con Solid State Memory (SSM) Acquisition reliable, fast, low power high cost, not suitable for long term storage Hard Disk Drive Post Production systems Playout systems Reliable (RAID configuration) Moderate cost Large bandwidth Fragile Power hungry Optical Distribution Long term storage low cost reasonable life time (20yrs) slow read/write access Data Tape Long term storage Archive Lowest cost sufficient read/write access times for archive, need to be incorporated into a library with a migration strategy The Cloud Temporary/ad hoc storage available everywhere high cost limited bandwidth/ access

45 WHAT ABOUT ARCHIVING?

46 WHAT ABOUT ARCHIVING? Migration vs. fundamentally archival storage ( bits vs. atoms )

47 WHAT ABOUT ARCHIVING? Migration vs. fundamentally archival storage ( bits vs. atoms ) commercial interest: 90 yrs?

48 WHAT ABOUT ARCHIVING? Migration vs. fundamentally archival storage ( bits vs. atoms ) commercial interest: 90 yrs? Cultural interest: how to fund?

49 WHAT ABOUT ARCHIVING? Migration vs. fundamentally archival storage ( bits vs. atoms ) commercial interest: 90 yrs? Cultural interest: how to fund? lifetime of the player/read-back

50 WHAT ABOUT ARCHIVING? Migration vs. fundamentally archival storage ( bits vs. atoms ) commercial interest: 90 yrs? Cultural interest: how to fund? lifetime of the player/read-back time to migrate: outflow need be only equal to inflow.

51 WHAT ABOUT ARCHIVING? Migration vs. fundamentally archival storage ( bits vs. atoms ) commercial interest: 90 yrs? Cultural interest: how to fund? lifetime of the player/read-back time to migrate: outflow need be only equal to inflow. Hey, could the Internet could be a good content archive?

52 FUNDAMENTALLY ARCHIVAL MEDIA (ATOMS)

53 FUNDAMENTALLY ARCHIVAL MEDIA (ATOMS) Cave Paintings (30,000yrs)

54 FUNDAMENTALLY ARCHIVAL MEDIA (ATOMS) Cave Paintings (30,000yrs) Stone Tablets (5000yrs)

55 FUNDAMENTALLY ARCHIVAL MEDIA (ATOMS) Cave Paintings (30,000yrs) Stone Tablets (5000yrs) Photographic Film (1000+yrs)

56 DIGITAL FILE STORAGE (BITS)

57 DIGITAL FILE STORAGE (BITS) data tapes

58 DIGITAL FILE STORAGE (BITS) data tapes tape drive

59 DIGITAL FILE STORAGE (BITS) data tapes tape drive Archive System

60 DIGITAL FILE STORAGE (BITS) tape drive Archive System

61 DIGITAL FILE STORAGE (BITS) Archive System

62 DIGITAL FILE STORAGE It s about the media file: (BITS)

63 DIGITAL FILE STORAGE It s about the media file: (BITS)

64 ARCHIVING METHODS

65 ARCHIVING METHODS Digital File Storage Migration: vs. Fundamentally Archival Media:

66 ARCHIVING METHODS Digital File Storage Migration: vs. Fundamentally Archival Media: Pro

67 ARCHIVING METHODS Digital File Storage Migration: vs. Fundamentally Archival Media: Pro Continuous verification of content. Content available for use at all times Final version and components.

68 ARCHIVING METHODS Digital File Storage Migration: vs. Fundamentally Archival Media: Pro Continuous verification of content. Content available for use at all times Final version and components. Simple, durable, long lasting Store and forget

69 ARCHIVING METHODS Digital File Storage Migration: vs. Fundamentally Archival Media: Pro Cons Continuous verification of content. Content available for use at all times Final version and components. Simple, durable, long lasting Store and forget

70 ARCHIVING METHODS Digital File Storage Migration: vs. Fundamentally Archival Media: Pro Cons Continuous verification of content. Content available for use at all times Final version and components. on-going process: need to manage and plan ahead. Simple, durable, long lasting Store and forget

71 ARCHIVING METHODS Digital File Storage Migration: vs. Fundamentally Archival Media: Pro Cons Continuous verification of content. Content available for use at all times Final version and components. on-going process: need to manage and plan ahead. Simple, durable, long lasting Store and forget Read device availability in the future

72 ARCHIVING METHODS Digital File Storage Migration: vs. Fundamentally Archival Media: Pro Cons Continuous verification of content. Content available for use at all times Final version and components. on-going process: need to manage and plan ahead. Simple, durable, long lasting Store and forget Read device availability in the future High value content: Use both methods!

73 FILE FORMATS, WHY SO MANY?

74 FILE FORMATS, WHY SO MANY? Different requirements at each stage of production (capture, post, play-out, distribution)

75 FILE FORMATS, WHY SO MANY? Different requirements at each stage of production (capture, post, play-out, distribution) Competitive issues between manufacturers (camera formats, edit formats, playout formats)

76 FILE FORMATS, WHY SO MANY? Different requirements at each stage of production (capture, post, play-out, distribution) Competitive issues between manufacturers (camera formats, edit formats, playout formats) Diverse distribution file types required (BluRay, DVD, Netflicks, itunes, Hulu, broadcast/cable)

77 SAMPLE FILE TYPES/USES

78 SAMPLE FILE TYPES/USES Use Types Compression Comments

79 SAMPLE FILE TYPES/USES Use Types Compression Comments Acquisition Raw MXF 1A and Op-Atom DPX Uncompressed DNx Pro Res JPEG 2000 MPEG2, 4(AVC-I) hi-bit rate/lo compression Preserve maximum detail/resolution

80 SAMPLE FILE TYPES/USES Use Types Compression Comments Acquisition Raw MXF 1A and Op-Atom DPX Uncompressed DNx Pro Res JPEG 2000 MPEG2, 4(AVC-I) hi-bit rate/lo compression Preserve maximum detail/resolution Post Production IMX MXF QT DPX DNx ProRes MPEG 2, 4(AVC-I) Uncompressed JPEG 2k (ML) moderate bit rate ease of editing/ transcoding Avoid generational loss/artifacts

81 SAMPLE FILE TYPES/USES Use Types Compression Comments Acquisition Raw MXF 1A and Op-Atom DPX Uncompressed DNx Pro Res JPEG 2000 MPEG2, 4(AVC-I) hi-bit rate/lo compression Preserve maximum detail/resolution Post Production IMX MXF QT DPX DNx ProRes MPEG 2, 4(AVC-I) Uncompressed JPEG 2k (ML) moderate bit rate ease of editing/ transcoding Avoid generational loss/artifacts Distribution Flash, WMV QT, HLS MPEG-DASH DCP MPEG 2 and 4 VP8 VC 1 and 9 JPEG 2k Hi-compression/ Lo bit rate Visually loss-less

82 WILL THERE EVER BE ONE COMMON FILE FORMAT?

83 WILL THERE EVER BE ONE COMMON FILE FORMAT? Simple answer...no!

84 WILL THERE EVER BE ONE COMMON FILE FORMAT? Simple answer...no! However the goal should be to minimize the number of file types in the content creation process

85 WILL THERE EVER BE ONE COMMON FILE FORMAT? Simple answer...no! However the goal should be to minimize the number of file types in the content creation process Be mindful of the waterfall effect in the use of compression.

86 WILL THERE EVER BE ONE COMMON FILE FORMAT? Simple answer...no! However the goal should be to minimize the number of file types in the content creation process Be mindful of the waterfall effect in the use of compression. Ease of conversion, ideally no transcoding just un-wrap/ re-wrap of files

87 SO WHAT DO WE DO?

88 SO WHAT DO WE DO? Point: Let s go back to video tape and 35mm film!

89 SO WHAT DO WE DO? Point: Let s go back to video tape and 35mm film! Counterpoint: Too late. Need to invest in refining the digital media workflow!

90 SUMMARY

91 SUMMARY Storage, network, and compression decisions need to be carefully selected

92 SUMMARY Storage, network, and compression decisions need to be carefully selected Don t get hung up with past solutions for archiving; face realities

93 SUMMARY Storage, network, and compression decisions need to be carefully selected Don t get hung up with past solutions for archiving; face realities Storage challenges. For each stage in production, requirements will dictate different storage solutions.

94 SUMMARY Storage, network, and compression decisions need to be carefully selected Don t get hung up with past solutions for archiving; face realities Storage challenges. For each stage in production, requirements will dictate different storage solutions. File Format anarchy. Single unique media file may never happen; however one should attempt to limit the number of types and especially the number of codecs

95 QUESTIONS?

96 QUESTIONS? Thank you for your attention.