Archive exchange Format AXF

Transcription

1 Archive exchange Format AXF An Open Format for Universal Content Transport, Storage and Long Term Preservation Brian Campanotti CTO Front Porch Digital What is AXF? AXF is a universal format for the encapsulation, storage, transport and preservation of file assets AXF is like an advanced ZIP which encapsulates any number of files, metadata and a ubiquitous file system AXF packages can exist on block based storage as well as any file system on any media type AXF is the first open format targeted at the storage, preservation and transport of file based assets

2 Origins of AXF Developed based on the demanding needs of the media and entertainment industry o Very large files and file collections o Complex and important file relationships o Fundamental connection between assets and metadata o Randomly accessible at every levels o Valuable assets which must be preserved Although developed in the M&E space, the benefit of AXF is easily extended to any file based application AXF is currently at the heart of various storage, preservation and cloud transport applications today Where Did We Come From? Proprietary systems are rampant in M&E o Many data formats on media o Numerous interface protocols o Same system type is required for recovery o Media migration is not easy Ever increasing danger of orphaned archives High costs of implementation and operation o Individualized system integration requirements o Transfer costs resulting from inability to interchange media

3 User Demands Simplified movement of material between different storage and archive systems Flexibility in changing storage vendors, technology and middleware solutions Limitless scale (number of files and size) including the ability to span across media Universally accessible leveraging stand alone utilities Support for all media types now and into the future Highly resilient with in-built redundancy Universal despite the storage or transport technology AXF Technology

4 Storage and Preservation We must take steps to ensure long term accessibility to valuable file-based assets stored in digital archives in all industries Key goals of an ideal storage format o Ensure long term accessibility o Self-describing assets and self-describing media o Encapsulation to maintain important metadata/file relationships o Scalability for any number of elements of any size and type o Standardized regardless of storage media technology o Transportability and compatibility between systems o Preservation (OAIS) features such as fixity, provenance, etc. o Streaming support to enable on-the-fly processing What choices are there? Tape ARchive Format Tape ARchive (TAR) format has been around for many decades There is no true universal TAR implementation but many customized ones TAR is a legacy format which cannot support many of the core functionality required in the M&E space TAR does not address many of the storage format goals we have outlined Do we have any other choices?

5 Linear Tape File System The Linear Tape File System (LTFS) is a simple file system for linear data tape LTFS makes data tapes appear as removable storage There are no standards bodies which currently endorse LTFS although some work is underway within SNIA Unfortunately LTFS has some significant limitations with respect to the goals previously outlined LTFS is very useful for the physical transport of content but not for long term storage or preservation But why? Linear Tape File System LTFS offers no media encapsulation and relies on simple folder hierarchies to form important asset relationships LTFS does not scale well due to lack of support for spanning across storage media a significant problem in M&E LTFS only supports modern data tape technologies and is not applicable to any other storage technologies Neither LTFS nor TAR achieve 100% of our long term storage and preservation goals What other choices are there?

6 Archive exchange Format AXF is a file collection wrapper which can encapsulate any number of files of any type and size AXF does not overlap with MXF or other media wrapper approaches as these are simply files inside AXF objects AXF brings the same level of universal transport and interoperability to storage as MXF has for media It was designed to support all storage technologies now and into the future It is IT-centric and not tied to media applications alone What is an Object Store? Key to media and entertainment asset storage and preservation is the concept of an Object Store An Object Store encapsulates all related files (essence, metadata, ancillary files, etc.) into a single package maintaining critical asset relationships CSM solutions are Object Stores by definition (HSM systems are NOT) This concept becomes even more important when we extend the CSM functionality into the cloud and asset transport

7 CSM as an Object Store Archive File Collection + A Media Object Cloud Metadata + Source Information Disk and Flash 2012 Front Porch Digital Inc. AXF Characteristics Unlimited Storage Support o Any number of files, any size of files and media spanning Resilience to Media Damage and Corruption o Redundancy in all structures; payload can be independently recovered Support for Media with and without a File System o Raw data tape, LTFS data tape, spinning disk, flash media, optical, etc. Support for Any File System or Operating System o Embedded file system abstracts the underlying operating and file system

8 AXF Characteristics Self describing Objects and Self Describing Media o Enables simple transport of objects and media between systems Object Versioning and Collection Support o Supports complex relationships between objects; additions, updates, etc. Support for All File Types Not Just Media Files o IT-centric implementation based on experience in M&E big data handling Streaming and File Based Asset Transport and Delivery o Support for streaming de-encapsulation; in-path checksums for structures and files; cloud storage and delivery applications Archive exchange Format Metadata Structured File Payload Ancillary Files Asset Compone nts Unstructured Proprietary Open Preservati on Elements File System Access Control Provenance Fixity Context Reference Universal Storage Agnostic File System AXF Object 2012 Front Porch Digital Inc.

9 Universal Storage Format Future Storage Technologies Spinning Disk (NTFS, FAT, etc.) Holographic (UDF, etc.) Solid State Disk (NTFS, FAT, etc.) AXF Object Blu-Ray (BDFS, UDF, etc.) Flash Media (NTFS, FAT, etc.) Data Tape (Block Based, LTFS, etc.) DVD (UDF, etc.) 2012 Front Porch Digital Inc. Basic AXF Overview AXF Medium Identifier AXF Object 1 AXF Object 2 AXF Object N AXF Object Header Metadata Container Metadata Container File 1 AXF File Footer File 2 AXF File Footer File N AXF File Footer AXF Object Footer Metadata Payload File Payload 2012 Front Porch Digital Inc.

10 Inside an AXF Object AXF on Block Based Media

11 Constructing an AXF Object Metadata Container AXF Object Header File 1 Metadata Container File 2 AXF File Footer AXF File Footer + File N File Collection AXF File Footer AXF Object Footer Archive An AXF Object Cloud Metadata + Disk and Flash Source Information 2012 Front Porch Digital Inc. Handling an AXF Object AXF Object Footer File N AXF File Footer AXF File Footer File 2 AXF File Footer File 1 Metadata Container AXF Object Header Metadata Container Archive An AXF Object Block alignment for increased resiliency Provenance records track the entire lifecycle of object On-the-fly fixity verification on all files and structures Spanning across media for limitless scale (eg. data tape spanning) Random access to all metadata and files Fully contained AXF indices enable quick re-indexing Self-describing objects and media allow quick transport between systems Cloud Disk and Flash 2012 Front Porch Digital Inc.

12 Transporting an AXF Object AXF Object Footer AXF File Footer File N AXF File Footer File 2 AXF File Footer File 1 Metadata Container Metadata Container AXF Object Header WAN$ AXF Object Footer AXF File Footer File N AXF File Footer File 2 AXF File Footer File 1 Metadata Container Metadata Container AXF Object Header An AXF Object An AXF Object On-the-fly encapsulation and de-encapsulation AXF objects are re-aligned to the block/chunk size of target storage devices Provenance metadata automatically tracks all actions on the AXF object On-the-fly fixity verification on all files and structures Self-describing objects and media allow quick transport between systems Important file and metadata relationships are always maintained 2012 Front Porch Digital Inc. AXF Benefits AXF Objects can scale to any size and encapsulate any number of files with full support for media spanning No need to upgrade existing storage infrastructures AXF ensures long term compatibility and resiliency with self-describing features for both AXF objects and media AXF overcomes all the technical, operational and functional limitations of TAR and LTFS AXF is an IT-centric implementation and is not limited to media files alone (documents, imaging data, etc.) AXF satisfies all of our ideal storage format goals as well as enables effective transport of complex file collections

13 AXF in the Industry Manufacturers are beginning to release products which fully support AXF today Implementations are based on the SMPTE standards body draft technical specifications which is nearing completion Universal exchange tools such as AXF Explorer can be used to access AXF objects stored on data tape, disk, etc. AXF is also the basis for cloud-based asset transport, archive and preservation services being offered AXF is not just for storage Very important milestone for the media, archive and preservation communities and offers real-world proof of AXF Front Porch Digital is leading AXF innovation as its primary inventor The AXF Standard AXF is a SMPTE 31FS initiative which will standardize the way media Objects (and any other file collections) are transported, stored and preserved The SMPTE AXF committee was founded in 2006 because of industry demand and reactive manufacturers AXF draft standard (ST2034 Part 1) is nearing completion and is expected to enter ballot stage this year Part 2 of the standard will be an RP on AXF applications Many users and vendors are actively participating in committee activities

14 OpenAXF.org OpenAXF.org is the community portal for the AXF initiative The website includes news, documentation, videos and whitepapers providing additional technical details on AXF This is the focal point for the AXF community so please sign up now SMPTE is always looking for active participants (users, manufacturers, etc) to help bring other perspectives to AXF Visit smpte.org for more information Thank You For questions or comments please contact me at