Archive and Preservation for Media Collections

Transcription

1 Archive and Preservation for Media Collections Leveraging Standards Based Technologies and the Cloud May 15, 2014 Brian Campanotti, P.Eng. Chief Technology Officer Front Porch Digital

2 What We Will Cover Today Challenges faced in the archive and preservation of large media collections (or any big data ) What is Content Storage Management? Considering the Cloud? Open standards helping to realize our lofty goals Deep dive in the Archive exchange Format (AXF) Brian Campanotti Chief Technology Officer Front Porch Digital

3 Media Storage and Preservation Challenges The size and scale of the data is immense! Many formats used in many different ways (archivists, news, post, etc.) Direct media device integration (editors, video servers, etc.) is key Economically feasible scalability content demands will grow! Redundancy and content protection is imperative Can we abstract the underlying storage technology?

4 Media Storage and Preservation Challenges Efficient storage infrastructure leveraging commodity technologies What about migration to newer storage technologies? Complex assets and collections should be stored as Objects not as files Will I be able to access my assets decades from now? Domain knowledge and expertise is imperative!

5 Content Storage Management (CSM) Most media companies have made the transition to files Content Storage Management (CSM) is core to these infrastructures CSM is the storage abstraction middleware CSM resides between devices which produce/consume content and commodity IT-centric storage CSM is an object-based, content-aware intelligent archive solution Environments are typically a combination of disk and data tape

6 Content Storage Management (CSM) Data tape archives offer better TCO, greener, more secure, scalable and high performance CSM solutions have direct integration with business systems linking processes, resources and metadata with the content repository CSM solutions include many content aware features o Direct device integration o In-path transcoding o Timecode based partial restore o Asset analytics o Metadata mining CSM solutions handle generation migration and leverage open standards for storage

7 Content Storage Management (CSM)

8 What about the Cloud? Security concerns are deep rooted Where is my data actually stored? Difficult philosophical transition from on-premises to services solutions Are the economic benefits tangible and can risk be mitigated? Network and bandwidth challenges Legacy mindset challenges Immature and non-domain specific solutions What is our exit strategy? Simple (dumb) storage is simply not enough... 8

9 CSM in the Cloud Benefits Simple o Removes customer burden (cost, risk, etc.) of operating on-prem CSM infrastructures Flexible o Infrastructure dynamically scales to meet demand pay as you go andyou can exit Managed o Operational and support challenges are managed by the platform Reliable o Leading CSM providers offer private-cloud CSMaaS leveraging proven technologies Current o Storage technology migration is automatic including fixity checks and provenance Preserve o Multiple copies of each asset are maintained with global replication options Secure o Private Tier III data centre nodes ensure protection including encrypted links/storage options

10 CSM as a Service (CSMaaS) Solutions Today

11 Your Cloud Services Checklist Leverage purpose built CSMaaS providers rather than a generic public cloud Ensure a physically partitioned user space Deploy private network links between sites where possible Leverage object storage technologies to ensure collection integrity Use UDP file transfer acceleration and encryption Leverage OAIS key concepts for preservation Employ fixity checks throughout every step of the workflow Enable continuous content health checks for data at rest Constrain physical locations of data and media assets Ensure you can easily disconnect from the service if your needs change Rely on open standards to ensure long-term accessibility

12 Open Standards in Storage and Preservation We must take steps to ensure long term accessibility to valuable file-based assets stored in digital archives in all industries Key objectives: o Ensure long term accessibility o Self-describing assets and self-describing media o Wrapped (encapsulated) to maintain metadata/file relationships oscalability for any number of elements of any size and type o Standardized regardless of storage media technology o Enable transportability between systems o Include key OAIS preservation features such as fixity, provenance, etc. o Streaming support to enable on-the-fly processing What choices are there today?

13 What About LTFS? The Linear Tape File System (LTFS) is a simple file system for linear data tape which makes tapes appear as removable storage There are no standards bodies which currently endorse LTFS although often incorrectly referred to as a standard LTFS is a great transport tool but has some significant limitations in storage and preservation applications o Relies on simple folder hierarchies to maintain asset relationships (no encapsulation) o No support for spanning (limited scalability) o Only applicable to modern data tape technologies o No preservation characteristics o Application specific in terms of implementation Cross vendor compatibility issues have been plaguing early adopters caused by vendor specific implementations and no centralized control

14 What About AXF? The Archive exchange Format (AXF) is a universal format for the wrapping (encapsulation), storage, transport and preservation of any type of file assets AXF is like an advanced ZIP which encapsulates any number of files of any type, metadata and a universal file system an object store model AXF is IT-centric and applies to all types and generations of storage technologies The first open format targeted at the storage, preservation and transport of file assets AXF fully defines (and constrains) implementations for increased interoperability AXF includes all of the functionality of LTFS, overcomes all of its limitations and adds significant features and benefits

15 AXF and LTFS Key Features Key Features AXF LTFS IT-centric design applicable to M&E as well as enterprise Applicable to cloud-based transfers and storage Encapsulates (wraps) files and metadata to protect key asset relationships Scales to store any number of files of any size and of any type Key support for spanning across media (data tape spanning, etc.) Universal format regardless of storage technology (disk, flash, data tape, etc.) Includes key preservation qualities (provenance, fixity, access control, etc.) Self-Describing Media: Maintains on-media index of all stored objects Self-Describing Objects: Maintains in-object index of all stored files

16 AXF in Media Applications Metadata Media Payload Ancillary Files Structured Asset Components Unstructured Proprietary Preservation Elements File System Open Access Control Provenance Fixity Context Reference Universal Storage Agnostic File System

17 AXF in Generic Collection Applications Generic IT Files/Folders Preservation Elements File System Access Control Provenance Fixity Context Reference Universal Storage Agnostic File System

18 AXF is a Universal Format Future Storage Technologies Spinning Disk (NTFS, FAT, etc.) Solid State Disk (NTFS, FAT, etc.) AXF Object Holographic (UDF, etc.) Blu-Ray (BDFS, UDF, etc.) Flash Media (NTFS, FAT, etc.) Data Tape (Block Based, LTFS, etc.) DVD (UDF, etc.)

19 Start of Media End of Media AXF Structural Overview Header Metadata Container Metadata Container File 1 AXF File Footer File 2 AXF File Footer File N AXF File Footer Footer 1 2 Storage Media (Disk, Data Tape, Flash, etc) Embedded File System Metadata Payload File Payload N Block/Chunk Aligned Structures VOL1 Header AXF Medium Identifier Object Index Object Index

20 Inside an Header Metadata Container Metadata Container File 1 AXF File Footer File 2 AXF File Footer File N AXF File Footer Footer Metadata Payload (Optional) File Payload Binary Structure Container Object Header XML Binary Structure Container Object Metadata Package Raw Payload File Data Binary Structure Container File Footer XML Raw Payload File Data Binary Structure Container File Footer XML Raw Payload File Data Binary Structure Container File Footer XML Binary Structure Container Object Footer XML Chunk Aligned

21 Constructing an Footer AXF File Footer File N AXF File Footer File 2 AXF File Footer File 1 Metadata Container Metadata Container Header File Collection + Metadata + Source Information Archive Cloud Disk and Flash

22 Handling an Archive Header Metadata Container Metadata Container File 1 AXF File Footer File 2 AXF File Footer File N AXF File Footer Footer Features and Benefits Block aligned for increased resiliency Provenance tracks the entire lifecycle of object forever Stream based transport and checksum verification on all files and structures Spanning across media for limitless scale Random access to all metadata and files Self-describing objects and media allow quick exchange between systems Cloud Disk and Flash

23 AXF Key Features Unlimited file and size scalability including media spanning Resilience to media damage and corruption Support for all storage media technologies Support for any file system or operating system Self-describing objects and self-describing media Object versioning and collection support Support for all file types not just media files Streaming and file-based asset transport and delivery

24 On the Horizon DNA Based Storage Ground breaking research by Nick Goldman and Ewan Birmeyat the European Bioinformatics Institute Data encoded using DNA's four bases (A, T, C and G) Projected shelf life in the 10,000+ year range 2PB of data storage per gram of DNA material First trials were able to independently encode and decode a payload which included: o MP3 recordings of Martin Luther King s speeches o Shakespeare s collection of sonnets of Time, Love, Beauty and Mortality Although likely more than a decade away from commercial availability we could immediately leverage the benefits of AXF for DNA storage!

25 AXF Today Front Porch Digital invented AXF and openly contributed it to SMPTE in 2011 The SMPTE AXF committee was founded by proactive manufacturers because of industry demand and limitations in legacy approaches Currently 50+ organizations entrusting AXF for long-term archive and preservation Universal access and exchange tools are available (such as AXF Explorer ) AXF is behind the LYNX CSMaaScloud-based asset transport, archive and preservation service today AXF is not just for storage! AXF will be published as a SMPTE standard this summer! AXF is the only transport, storage and preservation standard formally endorsed by any industry body

26 OpenAXF.org OpenAXF.org is the community portal for the AXF initiative The website includes news, documentation, videos and whitepapers providing additional technical details on AXF This is the focal point for the AXF community sign up now! Information and material is refreshed often so please visit regularly Work with other industry standards bodies is planned to widen adoption of AXF outside of media and entertainment industry Many vendors are on-board to support AXF once published

27 AXF Certified Technologies

28 Archive and Preservation for Media Collections Leveraging Standards Based Technologies and the Cloud THANK YOU! Brian Campanotti, P.Eng.