Accession Procedures Born-Digital Materials Workflow Initiating Author: Department: Sam Meister Archives & Special Collections Revision History Date Version Description Changed by 02/29/12 0.1 Draft 03/27/12 0.2 Revised Electronics Log changed to Born Digital Log 04/19/12 0.3 Post Digital Forensics Training revisions SM 07/16/12 0.4 Added appendices SM 07/31/12 0.5 Revisions based on DM feedback SM 08/23/13 0.6 Software screenshots added SM 10/31/14 0.7 Revisions and updates SM SM 1
Contents Purpose Steps 1. Receive New Transfer 2. Remove and Separate Electronic Media 3. Stabilize Physical Electronic Media 3.1. Assign Identifier 3.2. Create new folder location 3.3. Record media characteristics 3.4. Setup for disk imaging 3.5. Create disk image 3.5.1. Create disk image for 3.5 inch Floppy disks and Zip disks 3.5.2. Create disk image for 5.25 inch Floppy Disks 3.5.3. Common issues with Floppy disks and Zip disks 3.5.4. Create disk image for Hard Drives 3.5.5. Create Disk Image for Optical Media (CD, DVDs) 3.6. Move disk image reports to metadata folder 3.7. Document disk image process 3.8. Export Files from Disk Images 3.9. Generate Checksums Part 1 3.10. Extract audio files from Audio CDs 3.11. Run Virus Scan 3.12. Generate Checksums Part 2 3.13. Transfer copy of original data to secure storage 3.14. Verify data transfer 4. Stabilize Network File Transfers (Email, FTP) 4.1. Assign Identifier 4.2. Create new folder location 5. Initial Analysis 5.1. Generate file format identification report 5.2. Search for personal / private / sensitive information 5.2.1. Identity Finder 5.2.2. BitCurator Bulk Extractor 5.3. Extract filesystem metadata 6. Produce Accession Report 7. Transfer copy of files and metadata to Working Files storage location 8. Return Physical Media to Electronic Media Storage 2
Purpose The purpose of the following procedures is to guide the process of accessioning borndigital materials. These procedures provide a mechanism to perform and document tasks necessary to properly receive and transfer born-digital files to secure storage. The details of operating specific hardware and software to perform specific accession tasks will be revised as necessary in relation to changes in hardware and software infrastructure. These procedures should be evaluated for needed updates and revisions on an annual basis. Steps 1. Receive new transfer Transfer types: Network / Email (See Stabilize Network File Transfers) o Files received via attachments to email messages o Files received via network (FTP, Dropbox, etc) Physical media (See Remove and separate electronic media) o o Digitize o Return to donor - Files received on physical media transferred to local storage and physical media returned to donor (most likely to be the case with USB drive or portable hard drives) Do not return to donor - Media sent / given / donated to Archives without expectation of being returned to donor Physical analog materials are transferred, digitized, and physical materials are returned to donor 2. Remove and separate electronic media New accessions For new accessions this task should be performed as soon as possible if born-digital materials have been previously identified during initial collection surveys. If not already identified, physical electronic media may be discovered during initial inventory, or during additional processing activities. 3
Steps: Insert separation sheet to indicate materials have been removed Place physical electronic media in separate housing o Size of new housing container will depend on extent of physical media 1 2 floppy disks, CDs, DVDS, USB flash drives Folder or Envelope 5 and over floppy disks, CDs, DVDs, USB flash drives Box Hard Drives Box Computers Box (dependent on computer size) Label housing with Accession number Complete Electronic Media Form (See below) and include with media in new housing o Electronic Media Form elements Accession No. Enter Accession number for the accession Deposit Date Write date media received Format & Number of items Write number of items associated with each format type Example: CDRoms = 3 Received By / Separated By Write name of person who received media items Place media items and form in designated Electronic Media Storage location o Current physical storage location for electronic media is shelf location = A:13:137-140 New Accessions Update Accession Record o Check Includes Born-Digital Material box Enter A:13:137-140 in Location field Notify Digital Archivist about new born-digital materials awaiting processing o Send email with information from electronic media form 4
3. Stabilize Physical Electronic Media Retrieve media Bring media to Born Digital Workstation 3.1 Assign Identifier Open Born Digital Log Assign each electronic media item a unique identifier o o Identifier name structure = Accession number_001 Example 2007_038_001 Enter identifier in Born Digital Log Enter other accession information in Born Digital Log o Accession number o Transfer type o Date acquired (same as Deposit Date in Accessions database) o Received from (same as Donor in Accessions database) o Received by 5
3.2 Create new folder location This will be location for disk images, exported files, metadata, log information o o o Navigate to BornDigital folder (local folder on BornDigital workstation computer, not network storage location) If new Accession, then create new folder Folder name = AccessionNumber Example = 2007_038 Within each AccessionNumber folder create the following additional folders logs metadata objects disk image 6
Example of folder structure: files 2007_038 logs metadata 2007_038_001 2007_038_002 2007_038_003 objects disk_image 2007_038_001 2007_038_002 2007_038_003 files 2007_038_001 2007_038_002 2007_038_003 3.3 Record media characteristics Enter in Born Digital Log Media Storage Location Media Format o Magnetic: formats that include floppy disks, hard drives, zip disks o Optical: formats that include CDs, DVDs o Flash: formats that include USB drives Media Sub-Type o For optical media, media sub-type information may be printed on the front side of the disc Manufacturer o Look for any information on the media item that indicates the manufacturer or company Model Age o Look for any date information written on media item Condition o Enter Good unless media is damaged so that it is unreadable (e.g. multiple scratches on optical media, bent or warped disks) Media Label Text o Enter any text written on labels or directly on media item 7
3.4 Setup for Disk Imaging Enable write-protection Enabling write-protection ensures that data on media is not inadvertently modified or changed during the transfer process. Write-protection is enabled either manually (e.g. floppy disks) or by using write-blocker hardware. Floppy Disks o 3.5 inch Switch read / write tab to open 8
Read / write tab enabled Switch tab to write-protected open position (so you can see through the open part) 9
Setup 3.5 inch floppy drive 5.25 inch floppy disks Write-protection is enabled via disk imaging software for 5.25 inch floppy disks Setup 5.25 inch floppy drive 10
Plug in power cable here 11
Power cable Power cable 12
USB Interface Plug in USB cable here 13
USB cable USB cable 14
USB Drives / Hard Drives Configure hardware Connect write-blocker to workstation computer Power cable USB drive (from donor) USB cable (plug into computer) 3.5 Create Disk Image 15
3.5.1 Create disk image for 3.5 inch Floppy disks and Zip disks Enable write-protection on 3.5 inch floppy disk Insert 3.5 floppy disk into disk drive Open FTK Imager software Select Create disk image Select Source o Select Logical Drive for floppy disks 16
Select Drive or browse to source of image (e.g. A:, D:, E: drives) A: drive = floppy disks 17
In Create Image box select: Verify images after they are created Create directory listings of all files in the image Select Add 18
In Select Image Type box: o Select AFF o Click Next 19
In Select Image Destination box: o Click Browse and navigate to correct folder location 20
o In Image Filename field enter identifier for physical media Example: 2007_038_001 o Leave Image Fragment Size at 1500mb default o Select 0 for image compression size o Click Finish 21
In Create Image box select: o Start 22
During the Imaging process a box will display imaging progress and output 23
3.5.2 Create disk image for 5.25 inch Floppy Disks Make sure 5.25 Floppy disk drive and controller is setup correctly Insert 5.25 floppy disk drive Open DeviceSide Disk and Browse software Select Disk Type to find correct disk type that displays files 24
Select Browse Disk Contents to view files on disk Enter file path to folder location for accession item in Output Image Directory and change Output Image Filename to AccessionNumber_ItemNumber format 25
Select Capture Disk Image File 3.5.3 Common issues with Floppy disks and Zip disks Disks are format-dependent (e.g. disks formatted for a PC will not be able to be read in a Mac operating system) Disks are not always labeled with format information (PC, Mac, IBM, Apple, etc) o May have to experiment with multiple drives and operating systems to read data on disks 3.5.4 Create disk image for Hard Drives Make sure write-blockers are connected Open FTK Imager software Select Create disk image Select Source Physical (e.g. hard drives, flash memory drives) o Select Physical when goal is to create a forensic disk image to preserve entire computing environment Contents of a folder (logical image) o Select Contents of a folder when goal is to create a logical image of specific folder contents Follow above instructions for Imaging floppy disks to complete hard drive imaging 26
3.5.5 Create Disk Image for Optical Media (CD, DVDs) Open FTK Imager software Select Create disk image 27
Select Source o Select Logical Drive for CDs, DVDs 28
Select Drive or browse to source of image (e.g. A:, D:, E: drives) CDs or DVDs will likely be found in either the E: or F: drives 29
In Create Image box select: Verify images after they are created Create directory listings of all files in the image Select Add 30
In Select Image Destination box: o Click Browse and navigate to correct folder location 31
32
o In Image Filename field enter identifier for physical media Example: 2007_038_001 o Click Finish 33
Select Start 34
During the Imaging process a box will display imaging progress and results 35
3.6 Move disk image reports to metadata folder Open folder for disk image just created Disk image metadata 36
Move from disk_image folder to metadata folder 37
3.7 Document disk image process Enter in BornDigital Log o o o Disk Image Success? Enter Yes if successfully created disk image Disk Image Date Enter date disk image created Disk Image Software Select disk image software used from dropdown menu 38
3.8 Export Files from Disk Images Open FTK Imager Select Add evidence item 39
Select Source Image File Browse to folder location of disk image file 40
Select Disk image file Select Finish 41
42
Select Export files 43
44
Navigate to files folder location of disk image file 45
Select Ok 46
3.9 Generate Checksums Part 1 This action generates checksums for files from specific media items Select Export files hash 47
Navigate to metadata folder location Assign filename with relevant identifier Select Save 48
3.10 Extract audio files from Audio CDs Open CDex software Insert CD into drive 49
Select Options > Settings Select General > Directories & files Choose directory for audio files to extract to 50
Navigate to files folder Select OK Select Convert>Extract CD tracks to WAV files or press F8 51
When extracting process is complete audio files should be located in files folder 52
3.11 Run Virus Scan Navigate to folder location for accession files Right-click and select virus scan software Scan now Review virus scan results If virus found, download and save report from virus scan software to folder location 53
54
3.12 Generate Checksums Part 2 This action generate checksums for all files exported from all media items within a specific accession. This step should be performed after all media items have been disk imaged and files exported from those disk images. Open NARA File Analyzer In Action to perform on Files dropbox select Get MD5 Checksum by Name 55
Select the button in the Root Directory to Scan section Navigate to the objects folder within the specific accession folder Select Open 56
Select the button in the Auto-save Output directory section Navigate to the metadata folder within the specific accession folder Select Open 57
Select Auto-save results checkbox Enter checksum.md5 as the filename in the text next to Auto-save results checkbox Select Analyze button at bottom 58
Progress tab will open showing File Processing Status 59
When checksum generation process is complete a new tab MD5 will open displaying results 60
New checksum.md5 text file will automatically be saved to metadata folder 61
Open checksum.md5 file to confirm checksums were created 3.13 Transfer copy of original data to secure storage This step should only be performed by staff with approved access to secure network storage location Move copy of disk images, files, and metadata to secure network storage location o Current storage location BornDigital/originals Use Teracopy software to transfer copy of original data 3.14 Verify data transfer Verify checksums of files 62
4. Stabilize Network File Transfers (Email, FTP) Files received over a network via email or other transfer should be downloaded to Born Digital workstation and a virus check should be performed before transferring files to Library network storage 4.1 Assign Identifier Open Born Digital Log Assign each network transfer item a unique identifier o o Identifier name structure = Accession number_001 Example 2007_038_001 Enter identifier in Born Digital Log Enter other accession information in Born Digital Log o Accession number o Transfer type o Date acquired (same as Deposit Date in Accessions database) o Received from (same as Donor in Accessions database) o Received by 4.2 Create new folder location This will be location for metadata, log information, and files received via network transfer o o o Navigate to BornDigital folder (local folder on BornDigital workstation computer, not network storage location) If new Accession, then create new folder Folder name = AccessionNumber Example = 2007_038 Within each AccessionNumber folder create the following additional folders logs metadata objects files Example of folder structure: 2007_038 logs metadata 2007_038_001 objects files 2007_038_001 63
Move files to new files folder location o Email If files have been received as email attachments, then download files to Born Digital workstation computer o Other Network transfer (e.g. FTP, Dropbox, Google Drive) If files have been received via other network transfer, then download files to Born Digital workstation computer Go to Run Virus Scan 5. Initial Analysis 5.1 Generate file format identification report This step should be performed after all media items have been disk imaged and files exported from those disk images. Open DROID software 64
Save DROID Profile Select File > Save As Navigate to metadata folder Enter accession number as filename (example: 2007_038) Select Save 65
Select Add (green plus sign) Navigate to files folder within the specific accession folder to select files to be analyzed Select OK 66
Select Start 67
Select Export Select checkbox for profile Select Export profiles 68
Navigate to metadata folder Assign accession number plus format_report as filename (example: 2007_038_format_report.csv) Be sure Comma separated values (*.csv) is selected for Files of type Select Save 69
Delete DROID profile file from metadata folder 70
5.2 Search for personal / private / sensitive information 5.2.1 Identity Finder Open IdentityFinder software Enter password digital1 71
Go to Locations tab Select Custom folders 72
Select the button next the Folder box 73
Navigate to the files folder of the specific accession Select Ok 74
Select Add Make sure file path is correct under Folder location Select Ok 75
Go to Main tab Select Start 76
Status window will open displaying progress of search for personally identifiable information within the files 77
Search Summary window will display when search completes Select Save As 78
Save.idf file Identity Finder (.idf) file is a proprietary file that will allow for investigation and analysis of the report within the software interface, including filtering results and reviewing identified results within the context of the files. In Save As window navigate to metadata folder for specific accession For File name enter AccessionNumber_pii_report o Example 2007_038_pii_report Select Identity Finder (.idf) for Save As type Select Save 79
Save.csv file The purpose of saving a.csv file of the search results is to make sure a non-proprietary version of the report is available Select Save As In Save As window navigate to metadata folder for specific accession For File name enter AccessionNumber_pii_report o Example 2007_038_pii_report Select Text Export (Comma delimited) (*.csv) for Save As type Select Save 80
5.2.2 BitCurator Bulk Extractor Using the BitCurator Reporting Tool to generate reports on personal identifiable information and filesystem metadata requires a disk image as a starting point. The BitCurator tools should be used after all the disk images for a specific accession have been created. Restart computer and select Ubuntu o BitCurator will boot up Open Forensics Tools folder on Desktop Select BitCurator Reporting Tool 81
In BitCurator Reports window select Lauch BEViewer This will launch the Bulk Extractor tool 82
Select Run Bulk Extractor 83
To scan disk images: o Select Image File o Browse to folder location of disk image file and select disk image file (e.g. 2007_038_001.aff) To scan a set of files: o Select Directory of Files o Browse to folder location of set of files 84
85
Select the button next to Output Feature Directory and browse to metadata folder location for the media item or set of files 86
Add bulk to the end of the directory location after metadata 87
Select Submit Run 88
When complete Done button will be highlighted Output of tool will be located in new bulk folder within metadata folder 89
5.3 Extract filesystem metadata Open BitCurator Reports tool Select button next to Image file and browse to disk image location 90
91
Select button next to Bulk Extractor Feature Directory and browse to bulk folder location 92
Select button next to Output Directory and browse to metadata folder location 93
Select Run 94
Output Reports will be located in reports folder within metadata folder 95
6. Produce Accession Report Document o Total data size o Total number of files o Date range of files o File organization o Content types o Preservation issues o Materials to be restricted o Materials to be removed / deaccessioned 7. Transfer copy of files and metadata to Working Files storage location Create copy of files and metadata Move to designated working files storage location o Current working_files storage location = BornDigital/working_files 8. Return Physical Media to Electronic Media Storage Current physical storage location for electronic media is shelf location = o A:13:137-140 In Process Place electronic media into folder labeled with accession number Place folder with electronic media into In Process box based on accession number (aka most recent accession would be last folder in In Process box) 9. Update Born Digital Accession Status spreadsheet After all physical media items and/or network transfers for a specific accession have been processed through the above steps the Born Digital Accession Status spreadsheet should be updated to reflect the current status 96
Appendices A. Electronic Media Form B. Accession Report template C. Media Photography Form 97
Electronic Media Form Accession No. Deposit Date Received by Format Number of items ------------------------------------------------------------------------------------------------------------ Electronic Media Form Accession No. Deposit Date Received by Format Number of items 98
Accession Report Overview Accession Number Deposit date Transfer type Collection Number Creator Physical details Number of media Extent / Data size Number of files Formats Preservation issues Preservation actions Intellectual details File organization Date range Content types Privacy issues Donor restrictions Appraisal Report Author Report Date 99
Media Photography Form [Place media here] Accession No. Identifier Aspect: Front / Reverse / Side / Case 100