Accession Procedures Born-Digital Materials Workflow

Transcription

1 Accession Procedures Born-Digital Materials Workflow Initiating Author: Department: Sam Meister Archives & Special Collections Revision History Date Version Description Changed by 02/29/ Draft 03/27/ Revised Electronics Log changed to Born Digital Log 04/19/ Post Digital Forensics Training revisions SM 07/16/ Added appendices SM 07/31/ Revisions based on DM feedback SM 08/23/ Software screenshots added SM 10/31/ Revisions and updates SM SM 1

2 Contents Purpose Steps 1. Receive New Transfer 2. Remove and Separate Electronic Media 3. Stabilize Physical Electronic Media 3.1. Assign Identifier 3.2. Create new folder location 3.3. Record media characteristics 3.4. Setup for disk imaging 3.5. Create disk image Create disk image for 3.5 inch Floppy disks and Zip disks Create disk image for 5.25 inch Floppy Disks Common issues with Floppy disks and Zip disks Create disk image for Hard Drives Create Disk Image for Optical Media (CD, DVDs) 3.6. Move disk image reports to metadata folder 3.7. Document disk image process 3.8. Export Files from Disk Images 3.9. Generate Checksums Part Extract audio files from Audio CDs Run Virus Scan Generate Checksums Part Transfer copy of original data to secure storage Verify data transfer 4. Stabilize Network File Transfers ( , FTP) 4.1. Assign Identifier 4.2. Create new folder location 5. Initial Analysis 5.1. Generate file format identification report 5.2. Search for personal / private / sensitive information Identity Finder BitCurator Bulk Extractor 5.3. Extract filesystem metadata 6. Produce Accession Report 7. Transfer copy of files and metadata to Working Files storage location 8. Return Physical Media to Electronic Media Storage 2

3 Purpose The purpose of the following procedures is to guide the process of accessioning borndigital materials. These procedures provide a mechanism to perform and document tasks necessary to properly receive and transfer born-digital files to secure storage. The details of operating specific hardware and software to perform specific accession tasks will be revised as necessary in relation to changes in hardware and software infrastructure. These procedures should be evaluated for needed updates and revisions on an annual basis. Steps 1. Receive new transfer Transfer types: Network / (See Stabilize Network File Transfers) o Files received via attachments to messages o Files received via network (FTP, Dropbox, etc) Physical media (See Remove and separate electronic media) o o Digitize o Return to donor - Files received on physical media transferred to local storage and physical media returned to donor (most likely to be the case with USB drive or portable hard drives) Do not return to donor - Media sent / given / donated to Archives without expectation of being returned to donor Physical analog materials are transferred, digitized, and physical materials are returned to donor 2. Remove and separate electronic media New accessions For new accessions this task should be performed as soon as possible if born-digital materials have been previously identified during initial collection surveys. If not already identified, physical electronic media may be discovered during initial inventory, or during additional processing activities. 3

4 Steps: Insert separation sheet to indicate materials have been removed Place physical electronic media in separate housing o Size of new housing container will depend on extent of physical media 1 2 floppy disks, CDs, DVDS, USB flash drives Folder or Envelope 5 and over floppy disks, CDs, DVDs, USB flash drives Box Hard Drives Box Computers Box (dependent on computer size) Label housing with Accession number Complete Electronic Media Form (See below) and include with media in new housing o Electronic Media Form elements Accession No. Enter Accession number for the accession Deposit Date Write date media received Format & Number of items Write number of items associated with each format type Example: CDRoms = 3 Received By / Separated By Write name of person who received media items Place media items and form in designated Electronic Media Storage location o Current physical storage location for electronic media is shelf location = A:13: New Accessions Update Accession Record o Check Includes Born-Digital Material box Enter A:13: in Location field Notify Digital Archivist about new born-digital materials awaiting processing o Send with information from electronic media form 4

5 3. Stabilize Physical Electronic Media Retrieve media Bring media to Born Digital Workstation 3.1 Assign Identifier Open Born Digital Log Assign each electronic media item a unique identifier o o Identifier name structure = Accession number_001 Example 2007_038_001 Enter identifier in Born Digital Log Enter other accession information in Born Digital Log o Accession number o Transfer type o Date acquired (same as Deposit Date in Accessions database) o Received from (same as Donor in Accessions database) o Received by 5

6 3.2 Create new folder location This will be location for disk images, exported files, metadata, log information o o o Navigate to BornDigital folder (local folder on BornDigital workstation computer, not network storage location) If new Accession, then create new folder Folder name = AccessionNumber Example = 2007_038 Within each AccessionNumber folder create the following additional folders logs metadata objects disk image 6

7 Example of folder structure: files 2007_038 logs metadata 2007_038_ _038_ _038_003 objects disk_image 2007_038_ _038_ _038_003 files 2007_038_ _038_ _038_ Record media characteristics Enter in Born Digital Log Media Storage Location Media Format o Magnetic: formats that include floppy disks, hard drives, zip disks o Optical: formats that include CDs, DVDs o Flash: formats that include USB drives Media Sub-Type o For optical media, media sub-type information may be printed on the front side of the disc Manufacturer o Look for any information on the media item that indicates the manufacturer or company Model Age o Look for any date information written on media item Condition o Enter Good unless media is damaged so that it is unreadable (e.g. multiple scratches on optical media, bent or warped disks) Media Label Text o Enter any text written on labels or directly on media item 7

8 3.4 Setup for Disk Imaging Enable write-protection Enabling write-protection ensures that data on media is not inadvertently modified or changed during the transfer process. Write-protection is enabled either manually (e.g. floppy disks) or by using write-blocker hardware. Floppy Disks o 3.5 inch Switch read / write tab to open 8

9 Read / write tab enabled Switch tab to write-protected open position (so you can see through the open part) 9

10 Setup 3.5 inch floppy drive 5.25 inch floppy disks Write-protection is enabled via disk imaging software for 5.25 inch floppy disks Setup 5.25 inch floppy drive 10

11 Plug in power cable here 11

12 Power cable Power cable 12

13 USB Interface Plug in USB cable here 13

14 USB cable USB cable 14

15 USB Drives / Hard Drives Configure hardware Connect write-blocker to workstation computer Power cable USB drive (from donor) USB cable (plug into computer) 3.5 Create Disk Image 15

16 3.5.1 Create disk image for 3.5 inch Floppy disks and Zip disks Enable write-protection on 3.5 inch floppy disk Insert 3.5 floppy disk into disk drive Open FTK Imager software Select Create disk image Select Source o Select Logical Drive for floppy disks 16

17 Select Drive or browse to source of image (e.g. A:, D:, E: drives) A: drive = floppy disks 17

18 In Create Image box select: Verify images after they are created Create directory listings of all files in the image Select Add 18

19 In Select Image Type box: o Select AFF o Click Next 19

20 In Select Image Destination box: o Click Browse and navigate to correct folder location 20

21 o In Image Filename field enter identifier for physical media Example: 2007_038_001 o Leave Image Fragment Size at 1500mb default o Select 0 for image compression size o Click Finish 21

22 In Create Image box select: o Start 22

23 During the Imaging process a box will display imaging progress and output 23

24 3.5.2 Create disk image for 5.25 inch Floppy Disks Make sure 5.25 Floppy disk drive and controller is setup correctly Insert 5.25 floppy disk drive Open DeviceSide Disk and Browse software Select Disk Type to find correct disk type that displays files 24

25 Select Browse Disk Contents to view files on disk Enter file path to folder location for accession item in Output Image Directory and change Output Image Filename to AccessionNumber_ItemNumber format 25

26 Select Capture Disk Image File Common issues with Floppy disks and Zip disks Disks are format-dependent (e.g. disks formatted for a PC will not be able to be read in a Mac operating system) Disks are not always labeled with format information (PC, Mac, IBM, Apple, etc) o May have to experiment with multiple drives and operating systems to read data on disks Create disk image for Hard Drives Make sure write-blockers are connected Open FTK Imager software Select Create disk image Select Source Physical (e.g. hard drives, flash memory drives) o Select Physical when goal is to create a forensic disk image to preserve entire computing environment Contents of a folder (logical image) o Select Contents of a folder when goal is to create a logical image of specific folder contents Follow above instructions for Imaging floppy disks to complete hard drive imaging 26

27 3.5.5 Create Disk Image for Optical Media (CD, DVDs) Open FTK Imager software Select Create disk image 27

28 Select Source o Select Logical Drive for CDs, DVDs 28

29 Select Drive or browse to source of image (e.g. A:, D:, E: drives) CDs or DVDs will likely be found in either the E: or F: drives 29

30 In Create Image box select: Verify images after they are created Create directory listings of all files in the image Select Add 30

31 In Select Image Destination box: o Click Browse and navigate to correct folder location 31

32 32

33 o In Image Filename field enter identifier for physical media Example: 2007_038_001 o Click Finish 33

34 Select Start 34

35 During the Imaging process a box will display imaging progress and results 35

36 3.6 Move disk image reports to metadata folder Open folder for disk image just created Disk image metadata 36

37 Move from disk_image folder to metadata folder 37

38 3.7 Document disk image process Enter in BornDigital Log o o o Disk Image Success? Enter Yes if successfully created disk image Disk Image Date Enter date disk image created Disk Image Software Select disk image software used from dropdown menu 38

39 3.8 Export Files from Disk Images Open FTK Imager Select Add evidence item 39

40 Select Source Image File Browse to folder location of disk image file 40

41 Select Disk image file Select Finish 41

42 42

43 Select Export files 43

44 44

45 Navigate to files folder location of disk image file 45

46 Select Ok 46

47 3.9 Generate Checksums Part 1 This action generates checksums for files from specific media items Select Export files hash 47

48 Navigate to metadata folder location Assign filename with relevant identifier Select Save 48

49 3.10 Extract audio files from Audio CDs Open CDex software Insert CD into drive 49

50 Select Options > Settings Select General > Directories & files Choose directory for audio files to extract to 50

51 Navigate to files folder Select OK Select Convert>Extract CD tracks to WAV files or press F8 51

52 When extracting process is complete audio files should be located in files folder 52

53 3.11 Run Virus Scan Navigate to folder location for accession files Right-click and select virus scan software Scan now Review virus scan results If virus found, download and save report from virus scan software to folder location 53

54 54

55 3.12 Generate Checksums Part 2 This action generate checksums for all files exported from all media items within a specific accession. This step should be performed after all media items have been disk imaged and files exported from those disk images. Open NARA File Analyzer In Action to perform on Files dropbox select Get MD5 Checksum by Name 55

56 Select the button in the Root Directory to Scan section Navigate to the objects folder within the specific accession folder Select Open 56

57 Select the button in the Auto-save Output directory section Navigate to the metadata folder within the specific accession folder Select Open 57

58 Select Auto-save results checkbox Enter checksum.md5 as the filename in the text next to Auto-save results checkbox Select Analyze button at bottom 58

59 Progress tab will open showing File Processing Status 59

60 When checksum generation process is complete a new tab MD5 will open displaying results 60

61 New checksum.md5 text file will automatically be saved to metadata folder 61

62 Open checksum.md5 file to confirm checksums were created 3.13 Transfer copy of original data to secure storage This step should only be performed by staff with approved access to secure network storage location Move copy of disk images, files, and metadata to secure network storage location o Current storage location BornDigital/originals Use Teracopy software to transfer copy of original data 3.14 Verify data transfer Verify checksums of files 62

63 4. Stabilize Network File Transfers ( , FTP) Files received over a network via or other transfer should be downloaded to Born Digital workstation and a virus check should be performed before transferring files to Library network storage 4.1 Assign Identifier Open Born Digital Log Assign each network transfer item a unique identifier o o Identifier name structure = Accession number_001 Example 2007_038_001 Enter identifier in Born Digital Log Enter other accession information in Born Digital Log o Accession number o Transfer type o Date acquired (same as Deposit Date in Accessions database) o Received from (same as Donor in Accessions database) o Received by 4.2 Create new folder location This will be location for metadata, log information, and files received via network transfer o o o Navigate to BornDigital folder (local folder on BornDigital workstation computer, not network storage location) If new Accession, then create new folder Folder name = AccessionNumber Example = 2007_038 Within each AccessionNumber folder create the following additional folders logs metadata objects files Example of folder structure: 2007_038 logs metadata 2007_038_001 objects files 2007_038_001 63

64 Move files to new files folder location o If files have been received as attachments, then download files to Born Digital workstation computer o Other Network transfer (e.g. FTP, Dropbox, Google Drive) If files have been received via other network transfer, then download files to Born Digital workstation computer Go to Run Virus Scan 5. Initial Analysis 5.1 Generate file format identification report This step should be performed after all media items have been disk imaged and files exported from those disk images. Open DROID software 64

65 Save DROID Profile Select File > Save As Navigate to metadata folder Enter accession number as filename (example: 2007_038) Select Save 65

66 Select Add (green plus sign) Navigate to files folder within the specific accession folder to select files to be analyzed Select OK 66

67 Select Start 67

68 Select Export Select checkbox for profile Select Export profiles 68

69 Navigate to metadata folder Assign accession number plus format_report as filename (example: 2007_038_format_report.csv) Be sure Comma separated values (*.csv) is selected for Files of type Select Save 69

70 Delete DROID profile file from metadata folder 70

71 5.2 Search for personal / private / sensitive information Identity Finder Open IdentityFinder software Enter password digital1 71

72 Go to Locations tab Select Custom folders 72

73 Select the button next the Folder box 73

74 Navigate to the files folder of the specific accession Select Ok 74

75 Select Add Make sure file path is correct under Folder location Select Ok 75

76 Go to Main tab Select Start 76

77 Status window will open displaying progress of search for personally identifiable information within the files 77

78 Search Summary window will display when search completes Select Save As 78

79 Save.idf file Identity Finder (.idf) file is a proprietary file that will allow for investigation and analysis of the report within the software interface, including filtering results and reviewing identified results within the context of the files. In Save As window navigate to metadata folder for specific accession For File name enter AccessionNumber_pii_report o Example 2007_038_pii_report Select Identity Finder (.idf) for Save As type Select Save 79

80 Save.csv file The purpose of saving a.csv file of the search results is to make sure a non-proprietary version of the report is available Select Save As In Save As window navigate to metadata folder for specific accession For File name enter AccessionNumber_pii_report o Example 2007_038_pii_report Select Text Export (Comma delimited) (*.csv) for Save As type Select Save 80

81 5.2.2 BitCurator Bulk Extractor Using the BitCurator Reporting Tool to generate reports on personal identifiable information and filesystem metadata requires a disk image as a starting point. The BitCurator tools should be used after all the disk images for a specific accession have been created. Restart computer and select Ubuntu o BitCurator will boot up Open Forensics Tools folder on Desktop Select BitCurator Reporting Tool 81

82 In BitCurator Reports window select Lauch BEViewer This will launch the Bulk Extractor tool 82

83 Select Run Bulk Extractor 83

84 To scan disk images: o Select Image File o Browse to folder location of disk image file and select disk image file (e.g. 2007_038_001.aff) To scan a set of files: o Select Directory of Files o Browse to folder location of set of files 84

85 85

86 Select the button next to Output Feature Directory and browse to metadata folder location for the media item or set of files 86

87 Add bulk to the end of the directory location after metadata 87

88 Select Submit Run 88

89 When complete Done button will be highlighted Output of tool will be located in new bulk folder within metadata folder 89

90 5.3 Extract filesystem metadata Open BitCurator Reports tool Select button next to Image file and browse to disk image location 90

91 91

92 Select button next to Bulk Extractor Feature Directory and browse to bulk folder location 92

93 Select button next to Output Directory and browse to metadata folder location 93

94 Select Run 94

95 Output Reports will be located in reports folder within metadata folder 95

96 6. Produce Accession Report Document o Total data size o Total number of files o Date range of files o File organization o Content types o Preservation issues o Materials to be restricted o Materials to be removed / deaccessioned 7. Transfer copy of files and metadata to Working Files storage location Create copy of files and metadata Move to designated working files storage location o Current working_files storage location = BornDigital/working_files 8. Return Physical Media to Electronic Media Storage Current physical storage location for electronic media is shelf location = o A:13: In Process Place electronic media into folder labeled with accession number Place folder with electronic media into In Process box based on accession number (aka most recent accession would be last folder in In Process box) 9. Update Born Digital Accession Status spreadsheet After all physical media items and/or network transfers for a specific accession have been processed through the above steps the Born Digital Accession Status spreadsheet should be updated to reflect the current status 96

97 Appendices A. Electronic Media Form B. Accession Report template C. Media Photography Form 97

98 Electronic Media Form Accession No. Deposit Date Received by Format Number of items Electronic Media Form Accession No. Deposit Date Received by Format Number of items 98

99 Accession Report Overview Accession Number Deposit date Transfer type Collection Number Creator Physical details Number of media Extent / Data size Number of files Formats Preservation issues Preservation actions Intellectual details File organization Date range Content types Privacy issues Donor restrictions Appraisal Report Author Report Date 99

100 Media Photography Form [Place media here] Accession No. Identifier Aspect: Front / Reverse / Side / Case 100