1 st Int. Workshop on Standards and Technologies in Multimedia Archives and Records (STAR), Lausanne, /27

Transcription

1 1 st Int. Workshop on Standards and Technologies in Multimedia Archives and Records (STAR), Lausanne, /27 NTT Communication Science Labs. Noboru Harada

2

3 World is getting smaller! Archive s will possibly be inter-connected! Standardization related assets Liberally Multimedia contents User generated contents Audio archive World Wide Web Museum Medical data Movie/ Cinema Sharing resources reduces the cost - Tools or systems - Storage space - Knowledge Archive has different meaning to each other!

4 What? Everything needed to be archived (???) How long? Long enough time (???) Where???? How? Format?/Media? (???) And then? What to do with them? Digital Black hole???

5 Helps us to discuss issues using well-defined models and terms P R O D U C E R SIP Descriptive Info. Ingest Preservation Planning Data Management Archival AIP AIP DIP Storage Administration Descriptive Info. Access queries result sets orders C O N S U M E R MANAGEMENT SIP = Submission Information Package AIP = Archival Information Package DIP = Dissemination Information Package Information Package

6 Every archive has its own archiving policy Archiving policy / agreement may be difficult to be generalized because of its historical / cultural reasons Legacy data format should be treated as well as standardized format Multiple byte character set shall be supported All digital media contents and meta information should be treated in general way General and standardized approach Open Archival Information System (OAIS) Reference Model (ISO 14721) ISO/IEC Professional Archival Application Format (PA AF) File structure based approach is the must for enabling legacy data formats

7 SIP Import/Export with standardized SIP and DIP AIP DIP Query RDB Interfaces OS, File systems, (Cloud Computing?) Physical Storage RAID (HDD/SSD) LTO. Optical Disks Archiving/Preservation system

8 PA AF provides a standardized packaging format for digital files. It specifies metadata format to describe the original structure of digital files archived in a PA AF file necessary information to reverse the pre processing processes applied to digital files prior to archiving them in a PA AF file context information related to archived digital files and a file format for carriage of the metadata formats and digital files

9 April 2006 MPEG76 1 st proposal January 2008 April MPEG83 CD MPEG84 FCD January 2009 April MPEG86 FDIS PDAM1 IS MPEG87 FPDAM1 October MPEG90 FDAM1 ISO/IEC :2009 Professional Archival Application Format ISO/IEC :2010/Amd1 Conformance and reference software for PA-AF Liaison organizations: ISO/IEC JTC1 SWG-ARM (Sub Working Group Archival and Retrieval Mechanisms) ISO/IEC JTC1 SG-DCMP (Study Group on Digital Contents Management and Protection) ISO/IEC JTC1 SC29/WG1 (JPEG) D-Cinema Ad hoc Group The Society of Motion Picture and Television Engineers (SMPTE) ISO TC20/SC13 The Consultative Committee for Space Data Systems (CCSDS) and Data Archive Ingest (DAI) working group

10 ISO/IEC Professional Archival Application Format Build with MPEG tools and other standards General Packaging Format Preserve hierarchical folder structure and attributes Any Meta information can be contained as XML files (MPEG 7, Dublin core, METS, AES, etc.) Policy / coding scheme independent Cross platform support (OS, file system, multiple byte character set) Support file sub division based on the package file size Can serve as Information Packages defined in OAIS Reference Model (SIP, AIP, DIP)

11 Input files C:/folder1/ Index.html folder2/ file2.html file3.jpg file4.mp4 Standardization archive Profile Set of files to be stored Set of metadata to be stored Policy / Profile Content analysis Pre processing Lossless Compression Encryption / Protection Hash code calculation Recording Project archive Profile Set of files to be stored Set of metadata to be stored Content Description PA AF File split Verification PA AF Packaging tool (File format) Compression tools Protection Integrity check tools tools External tools Dissemination tools Trans coding Trans format Metadata search

12 MPEG 21 File Format (derivative of ISO base file format) MPEG 21 Digital Item Declaration (DIDL) 2 nd Ed. MPEG 21 Digital Item Identifier MPEG 7 Multimedia Description Schemes MPGE 21 Intellectual Property Management and Protection(IPMP) Base Profile MPEG 21 Rights Expression Language (REL) Other pre processing tools MPEG 4 Audio Lossless Coding (ALS) OpenSSL, Gzip, etc.

13 Archives can decide which level of services should be applied based on their own archiving policy Pre processing Lossless compression Protection Encryption Integrity check Usage governance Interoperable content searching Application specific context information Basic Function [Package input digital files into a PA AF file]

14 MPEG 21 File Format Archive File Header [Preservation Description Information] Content Information Application Specific Context Information (Content Information)... Content Information Archive Structure Information Model the structure of input files in its original file system MPEG 21 DIDL MPEG 21 DII Pre processing Information Describe all content processing applied to the Content Information MPEG 21 IPMP MPEG 21 REL Context Information Describe context information attached to the Content Information MPEG 7 MDS

15 C:/folder1/ Index.html folder2/ file2.html file3.jpg file4.mp4 (i.e. AAC) *Index.html can be treaded as an application specific context information MPEG-21 file format for the Professional Archival Application Format Ftyp mp21 meta hdlr mp21 XML MPEG-21 DID XML iloc item_count = 4; item_id:1 index.html item_id:2 folder2/file2html item_id:3 folder2/file3.jpg item_id:4 folder2/file4.mp4 entry_count = 4; iinf item_id:1 index.html item_id:2 folder2/file2.html item_id:3 folder2/file3.jpg item_id:4 folder2/file4.mp4 mdat index.html* folder2/file2.html folder2/file3.jpg folder2/file4.mp4

16 C:/folder1/ mets.xml Contents/ BXF01.jpg BXF02.jpg BXF03.jpg MPEG-21 file format for the Professional Archival Application Format Ftyp mp21 meta hdlr mp21 XML MPEG-21 DID XML iloc item_count = 4; item_id:1 mets.xml item_id:2 Contents/BXF01.jpg item_id:3 Contents/BXF02.jpg item_id:4 Contents/BXF03.jpg entry_count = 4; iinf item_id:1 mets.xml item_id:2 Contents/BXF01.jpg item_id:3 Contents/BXF02.jpg item_id:4 Contents/BXF03.jpg mdat mets.xml Contents/BXF01.jpg Contents/BXF02.jpg Contetns/BXF03.jpg

17 A music project often contains non audio files Folder structure based archiving is suitable for delivery or storage of a recorded music project Project_description.pts track1.bwf track2.bwf trackn.bwf tracking sheet.txt recording map.txt lyrics.jpg Lossless compression and archiving Complete reconstruction of original folder structure

18 Packaging Tool for the PA-AF An example input C:/folder1/ file1.xml file2.exe folder2/ file3.bwf file4.mp4 (e.g. AAC) file5.zip Un-packaging Tool for the PA-AF An example input C:/folder1/ file1.xml file2.exe folder2/ file3.bwf file4.mp4 (e.g. AAC) file5.zip Generate File information XML Parse File information XML Zip Zip ALS Lossless Encoding Zip Zip ALS Lossless decoding Pack all data into ISO base media file format Extract all data from the ISO base media file format Information Original file attributes Path name File name File type Encoding schemes

19 PA-AF reference software GUI / CUI application Configuration for external tools UI/Configuration wrapper Packager API Reader functions MP21File Format MP21 DID/DII PA-AF File Attribute PA-AF (.LIB/.DLL) Extractor API Writer functions MPEG-7 CI/MDS MP21 IPMP MP21 REL Editor/Viewer API Item functions Content manager functions Metadata manager functions Configuration API Application specific extension IF Configuration functions Processing tool manager functions PA-AF API High-level API Low-level API and Helper functions iconv.lib/.dll (Character set Conversion tool) xerces.lib/.dll (Metadata handling tool) External tools (executables) Example processing tools - Compression/Decompression - Encryption/Decryption - Hash generator - Integrity checking tool - Media viewer / Media player - Editing tool

20 PA AF reference software (GUI) Web archiving system for standardization related accets

21 PA AF is a standardized packaging format for archiving digital files. PA AF can serve as SIP, AIP, and DIP in the OAIS reference model Sharing the resources can reduce cost. MPEG A PA AF software is available at pc3.kaist.ac.kr/paaf