Migration of the digitised content Strategies and Case Studies

Transcription

1 Tecnología Documental y Workflow Migration of the digitised content Strategies and Case Studies Julián Bescós Toledo Workshop 23-25th June, 2002

2 Migration of the digitised content CONTENT 1. Definition and Purpose 2. Complex task and large field 3. Origin of migration 4. Best practices for preservation 5. Application of migration 6. Planning 7. Case Studies 8. Cost

3 Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost Definition Purpose Set of tasks to achieve periodic transfer of digital materials from one hard/soft configuration to another! Long term Preservation of the digital information created and stored using digital technology! Allow broad access Retrieve, display and use Consequence of the digital technology: Advantages It relieves the traditional conflict between preservation and access Users operate on images of original works It affords multiple and simultaneous uses from a single original Long distance access It yields additional, effective means of access: searching for words, manipulation of the digital display of digital materials, etc Drawbacks of the digital technology Reading information requires equipment and software which is in changing constantly Probably not available in a decade

4 Large Field Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost! Complex task and large field Transfer of digital information: kind of media, hard/soft soft, websites information Different organisations are concerned: : UNESCO, IFLA, ICA, RLG, LOC, etc. Focus on images of textual documents of archives and libraries

5 Origin of migration I Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost New devices, processes and software replace the methods to record, store and access Enhancement of service! Technology obsolescence Hardware More powerfull computers and higher density storage Elements for updating are not available ( increase of storage, memory, etc) Basic software! Media Operating systems Data base managers Degradation or Longevity Lifetime is rarely the constraining factor for DP Obsolescence of old storage media as newer and better media are available in the market

6 Origin of migration II Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost! Obsolescence of the Access software Access in new platform and media Not available long term programs New functions! In practice it is a combination of: Technology obsolescence New functionalities of the software Derived from information and communication technology Daily work of digitisation and storage that requires: Higher density storage Faster computers! It is a consequence of: The digital world of information and communication technology is still relatively young and inmature

7 Process of a Digital Information System Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost! Digitization By flatbed scanners By zenithal scanners! Storage In Magnetic disk for on line image service Indexing and search for navigation and access: in their own media or/and in a relational database Massive optical media: CD-R and DVD! Access and Consultation Functions for Searching, Navigation and Display of images: zoom, scroll,etc Image Processing Functions: stain removal, increase of contrast, reduction of ink bleeding through the paper, etc.! Diffussion Internet CD-ROM or DVD

8 Best practices for preservation I Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost! General principles Based on PC s and mainstream commercial equipment Hardware provided by first class IT companies Database managers of widespread use Consultations with institutions undertaking projects Based on elements and standard formats. Officials or the facto, like TIFF, JPEG, XML, etc. Modular, allowing a progressive installation and easy update of elements Selection of software: Functionalities Number of installations Maintenance Provided by a IT company settled in cultural environments Key factors: Server, operating system, database manager Backup policies

9 Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost! Digitization Capture systems: Best practices for preservation II Robust flatbed scanners (A3) Zenithal scanners. Limitations of cameras Use of standard compression formats. JPEG, CCITTGIV Ensure that digital images will allow a broad range of future use Capture the highest quality image technically possible and economically feasible for large-scale production Capture the informational content / physical appearance Fast and easy correction of errors! Criteria for holding selection Value Condition Use Acceptability of the digital object Access aids

10 Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost! Storage Media of wide use and low cost: Magnetic disk for on line image service (specially in high demand) Disks with redundancy Backup in tapes of high capacity (10/20GB) One or two units available as hotsawp It allows migration without personnel operation In a distributed network they may need to be stored online in multiple locations CD-R as backup for off line access in case of system failure In general there is little experience in storing massive quantities of culturally valuable materials! Backup and Recovery Best practices for preservation III Use industry standard backup and recovery procedures: Periodic backup to magnetic tape A copy held on site for near term recovery A copy off-site stored for disaster recovery

11 Application of migration I Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost Traditional approach of Computer Science! Migration of media Refreshing digital information by copying it from medium to medium Conversion of files to another format to be interpreted by new programs; to a reduced number of standard formats; progression of compression; decompress and compress! Migration of technology platform Server and PCs Periphericals Capture devices and CDR writers Operating system and database manager! Migration of the digitising and access software Maintenance of software in new platform New software versions for digitising and access Develoment of migration software Testing with samples of data Application to the database

12 Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost! Planning Planning for migration is difficult due to: the limited experience we cannot predict when media, soft and hard will become obsoleted No single strategy applies to all formats of digital information It varies in different applicational environments, for different formats of digital materials and for preserving different degrees of computation, display and retrieval It requires a unique new solution for each new format and process Automatic conversion is only partially possible In general there are no firm plans for migration, but to stay up to date with current technologies by migration the content Usually there is urgency involved in migration: due by the obsolescence of soft and hard! Schedule Planning Schedule New releases of software, databases,etc. can be expected every 2-3 years, with minor updates more often Migration from one storage media to another every 4-5 years, if not online Migration to new hardware and software occur less frequently but can be expected between 5-10 years

13 Practical Case Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost Archivo Archivo General General de de Indias, Indias, Sevilla Sevilla

14 Practical Case Definition/Purpose Preservation of digitised content Broad Access Large Field Origin of migration Technical Obsolescence Media Obsolescence of Access software Best practices for preservation General Principles Digitization Storage Application of migration Planning Schedule Case Studies Cost Reading ReadingRoom. Room.. Archivo Archivo General General de de Indias Indias

15 Archivo General de Indias Computarization system for Archivo General de Indias! Content 86 million of pages of original manuscripts realted to the Spanish Administration in America (XV-XIX XIX centuries), in bundles pages of inventories and catalogs! Information system integrating A textual Data Base with all descriptive information A Digital Image Archive by digitisation of 11 million of pages A Module for User Management Researcher autorization, access control and work in research room Documents movement control, statistics

16 Archivo General de Indias! Main Results 11 million digital images in descriptive entries Control of User management, Consultation room, documents movements and statistics Access by researchers and archivists from 40 workstations About 30% of present consultations are on the screen (1 million pages/year ) About 35% of printing are digital ( /year ) Access system in service since 1992 System migrated to ArchiDOC and ArchiGÉS in 1999: From OS/2 to Windows NT and Oracle Images stored in optical disks migrated to CD-R in 1999 From CD-R to magnetic disks for online access in 2000

17 Archivo General de Indias Digital Image Archive! Image Acquisition Digitized Images keep high fidelity with respect to the original manuscripts Digitization: : 100 dpi,, 16 gray levels DPCM compression by soft Compressed Image size of A4: Kbytes Storage for 1 bundle: : 2000 x 350 = 700 MB! Storage of digital images in 1997 Bundles Media Year beg. Number of disks Images IBM optical disks ( 200 MB) Plasmon optical disks ( 940 MB) CD-R R (640 MB) Backup in DAT magnetic tapes, Plasmon disks and CD-R

18 Archivo General de Indias Digital Image Archive! Display and Digital Restoration Restoration algorithms selected and applied by the user if desired: Blotches removal Reduction of Ink bleeding through the paper, etc Local contrast enhancement

19

20

21

22

23

24 Archivo General de Indias Migration from optical disks to CD-R R (1999) Project for the 2K year effect! CD-R writing per complete bundle Typically 4 WORM IBM disks ( 200 MB each) ) in 1 or 2 CD-R 1 WORM Plasmon disk ( 940 MB) in 1 or 2 CD-R! Reading of optical disks through the network No direct connectivity between optical disks and Windows NT! Decompression of the DPCM format! Compression on JPEG format! Temporary storage in magnetic disk! All images of the bundle are copied in CD-R! Verification of images by reading! 6000 CD-Rs Rs, and backup copy

25 Archivo General de Indias Migration of Images IBM Disks to CD-R Microchannel IBM PS/2 File system driver for OS/2 OS/2 1.3 and Lan Server TokenRing Microchannel Card IBM Optical Drives Token Ring Network Pentium PC Windows NT Token-Ring PCI Card 3GB disk SCSI interface CD-R Drives

26 Archivo General de Indias Migration of Images PC with i486 SCSI interface File system driver for OS/2 OS/2 3.0 Ethernet card Plasmon Drives Plasmon Disks to CD-R HUB Ethernet Network HUB Ethernet Network Pentium PC Windows NT Token-Ring PCI Card 3GB disk SCSI interface CD-R Drives Migration requirements of personnel and time! 3 operators during 4 months

27 Archivo General de Indias Migration from CD-R to magnetic disk in 2000 Project for online access and Internet! Just copy. Images are already with JPEG compression! 10 RAID cabinets of 350 GB each ( 8 disks x 50 GB )! 1 operator was required during 1 month for the copy from a CD- ROM tower to magnetic disks Transfer from different media Media Transfer rate Image IBM optical disk 60 KBs 6 seconds Plasmon optical disk 100 KB/s 3 seconds CD-R R 16x 2,5 MB/s <1 second Magnetic disk Bundle 4 hours 1 hour 5 minutes 80 MB/s 1 minute

28 Archivo General de Indias Servers and Image Storage Image Server RAID Cabinet 1 RAID Cabinet 2 RAID Cabinet 3 RAID Cabinet 4 RAID Cabinet 5 RAID Cabinet 6 RAID Cabinet 7 Domain Controler Server Data Base Server WEB Server RAID Cabinet 8 RAID Cabinet 9 RAID Cabinet 10 UPS UPS UPS UPS

29 Low cost Magnetic Disk Storage Storage GB (12 disks x 160 GB in RAID 5) Stored images 5 million Cost

30 Archivo General de Indias Reserved Im age Server RAID Cabinet 1 RAID Cabinet 2 R eserved for R AID C abinet 3 Auto Rep licated on line R emote D isk subsystem for Back up and Servic e UPS UPS UPS Domain C ontro ler Servers Data Base Server WEB Servers Red local

31 Komintern Archives Computarization system for the Komintern Archives in Moscow Project supported by the National Archives of France, Germany, Italy, Spain, Swiss, Sweden, US Library of the Congress, Soros Foundation, ICA and Council of Europe! Content of Komintern The Third International was founded in Moscow in 1919 to promote the Communist movement worldwide. It conducted and coordinated the action of about 50 Communist parties and affiliated organisations One of the richest sources of information about world history in the interwar period It contains 66 fonds, around files in 15 Km shelving! Aims Preservation and access to the documents of the Communist International ( opened to researchers in 1992) Copy of the data base for each country partner ( entries) Copy of digitized documents for each partner (1 million pages)! System installed in Servers,, 7 data entry worstations and 2 digitization workstations With OS/2, DB2 and ArchiDOC Images stored in 500 CD-R R in JPEG format

32 Komintern Archives! Migration in 2000 To Windows NT and Oracle Not required for images Next steps! Instalation of standalone worksations in each country site Access to images in 500 CD-R Access to images in magnetic disk! Web Access Policies in discussion: : legal, technical and financial conditions! Access from the research room in Moscow 20 Workstations for access with ArchiDOC Control of User Management, Consultation Room and Reprography with ArchiGES

33 Example of Project Cost Example for a Digital Information System! Objective Description, scanning and access to 250 bundles of old documents ( about 40 files/ bundle,, files, and pages)! Work 2 persons for archival work: organizing, describing, indexing,, etc. ( 15 files/day x 200 days/year year x 2 persons ) 2 persons for scanning ( 600 pags/day x 200 days x 2 ) 1 part time computer person The objective can be achieved in 2 years! Equipment One server with 160GB redundant magnetic storage with W2000, Oracle and ArchiDOC 2 scanning workstations 2 data entry workstations 2 consultation workstations

34 Cost Figures Example for a Digital Information System! Costs Personnel 2 years work 2 Archive experts scanner operators computer operator ( part time) Hardware Mainten. Server with storage for images Backup elements flatbed scanner zenithal scanner worksations Basic software Windows Oracle 8i Application software Software, installation, training, assistance, managing Total

35 Cost Factors Cost factors for Migration Hardware Magnetic disk storage for images! Hardware! Application software Sotware development for migration! Minimum computer operation

36 Summary Summary! Best practices for Digital Preservation Mainstream commercial equipment Use of standard formats Storage in magnetic disk with redundancy Backup policies Maintenance! Periodical Update Policy Hardware Media Basic sofware Application software