Utilizing Digital Library Infrastructure to Build Modern Research Collections

Transcription

1 Utilizing Digital Library Infrastructure to Build Modern Research Collections Mark Phillips Wisconsin Association of Academic Librarians

2 What is a digital library? 2

3 After 15+ years surely we've figured it out. 3

4 Right? 4

5 We have systems for some or all of the following 5

6 Digitized content of all types 6

7 Books 7

8 Photographs 8

9 Letters 9

10 Maps 10

11 Newspapers 11

12 Yearbooks 12

13 Audio 13

14 Video 14

15 Articles 15

16 Slidedecks 16

17 Grey literature 17

18 Electronic Theses and Dissertations 18

19 Datasets 19

20 We are harvesting the web 20

21 All of this content I consider part of the digital library 21

22 We purchase resources from vendors 22

23 We subscribe to database 23

24 We provision access to closed access journals 24

25 We buy ebooks 25

26 But we don't really manage that content 26

27 We pay other people to do that 27

28 But this isn't what I want to talk about today 28

29 Instead I want to talk about the opportunities 29

30 And some of the challenges 30

31 When we think of building collections 31

32 Digitally 32

33 In many cases from the Web 33

34 Story #1 When we depend on pointing instead of collecting 34

35 Anyone can point to a web site or document, but we have a special name for those that select, acquire, organize, and preserve information and provide access to and services for that information: We call them Libraries. James Jacobs (2013) 35

36 NASA Technical Reports Server (NTRS) 36

37 Hundreds of thousands of freely available technical reports created or contracted by NASA 37

38 On March 19th this is what the site looked like. 38

39 39

40 We've built much of our infrastructure around linking to content that others maintain. 40

41 How many catalog records do we have that point to an open access resource hosted somewhere else? 41

42 worldcat.org has over 5,500 records pointing to NTRS 42

43 Instead of just pointing to resources we should be collecting these resources 43

44 Remember, lots of copies keeps stuff safe 44

45 But many of these resources we are trusting to one host. It is just a matter of time before something else disappears 45

46 So if we should be collecting this stuff and it is everywhere on the Web, do you want us to just start harvesting the Web? 46

47 Yes! 47

48 Story #2 End of Term Web Harvests 48

49 End of Term 2008 Presidential Web Archive 49

50 Collaborative Web archiving project 50

51 5 Partners 51

52 Library of Congress Internet Archive California Digital Library UNT Libraries Government Printing Office 52

53 Capture the transition of the executive branch and the changes in the federal Web during the process 53

54 Before the election After the election After the inauguration 54

55 EOT-2008 Stats 55

56 16 TB of WARC and ARC file 56

57 160,211,356 URIs 57

58 Largest TLDs # URLS # Subdomains gov 137,847,822 14,339 com 7,809,711 57,873 org 5,108,645 29,798 mil 3,555,425 1,677 edu 3,552,509 13,856 58

59 Largest Mime-types # Files text/html 105,590,929 image/jpeg 13,665,196 image/gif 13,031,046 application/pdf 10,320,163 59

60 Largest Mime-types # Files text/html 105,590,929 image/jpeg 13,665,196 image/gif 13,031,046 application/pdf 10,320,163 60

61 What is that? 61

62 We took a look 62

63 After seven weeks of machine time we had 4.5 million unique pdf samples 63

64 We learned a bunch of things 64

65 Year Number , , , , , , , , , , , , , , ,829 65

66 PDF Version Number , , , ,127, ,449, , , ,812 66

67 PDF Optimization 47% 53% yes no 67

68 How many pages per PDF 68

69 Shortest Document = 1 page (1,477,612 PDFs) 69

70 Longest Document = 17,584 pages (1 PDF) 70

71 60,874,402 pages in PDFs 71

72 Average PDF Length = 13.8 pages 72

73 Wait back up... 17,584 pages? 73

74 Really? 74

75 75

76 Page Distribution (1-10 pages per PDF) Pages Number 1 1,477, , , , , , , , , ,945 76

77 Page distribution by range Range Number 1 1,477, ,203, , , ,766 1,001+ 1,902 77

78 Encrypted PDFs 206,627 No Yes 4,197,422 78

79 Encryption Type Number yes (print:yes copy:yes change:no addnotes:yes) yes (print:yes copy:yes change:no addnotes:no) yes (print:yes copy:no change:no addnotes:no) yes (print:yes copy:yes change:yes addnotes:yes) yes (print:yes copy:no change:no addnotes:yes) 3292 yes (print:yes copy:no change:yes addnotes:yes) 1107 yes (print:no copy:no change:no addnotes:no) 797 yes (print:yes copy:yes change:yes addnotes:no) 553 yes (print:yes copy:no change:yes addnotes:no) 457 yes (print:no copy:yes change:no addnotes:no) 58 yes (print:no copy:yes change:yes addnotes:yes) 22 yes (print:no copy:no change:yes addnotes:yes) 8 yes (print:no copy:no change:no addnotes:yes) 7 yes (print:no copy:yes change:no addnotes:yes) 7 yes (print:no copy:no change:yes addnotes:no) 2 79

80 What domains have the most PDFs? 80

81 gpo.gov with 1,082,735 PDFs 81

82 Followed by: usda.gov house.gov army.mil bea.gov census.gov 82

83 With PDFs over 1 page? 83

84 gpo.gov = 594,430 84

85 usda.gov house.gov uscis.gov uscourts.gov army.mil tres.gov noaa.gov 85

86 What about 20+ pages 86

87 gpo.gov gao.gov epa.gov usda.gov army.mil noaa.gov 87

88 The list starts to change 88

89 We indexed the size of the items. 89

90 Inches square of the first page 90

91 Area over 500 inches square finds us this stuff 91

92 92

93 93

94 Maps, Posters, Charts 94

95 There are 121,490 of these in the archive 95

96 The goal is to better understand the content hidden in the Web archive 96

97 Surface this content and establish it as first order content in a digital library 97

98 We wanted to build usable collections out of this content. 98

99 Find the needles in the haystack 99

100 Here is a concrete example 100

101 In the 2008 EOT Crawls 101

102 We captured brac.gov 102

103 Defense Base Closure and Realignment Commission 103

104 104

105 The site contained over 14,000 documents 105

106 Over a million pages of documentation 106

107 Only 260 results in google site:brac.gov 107

108 108

109 109

110 110

111 111

112 Unique content that is now easily discoverable 112

113 Which means stuff gets found 113

114 114

115 And people want to use it for research 115

116 Creating a unique dataset for research 116

117 How do you create subject based collections like this? 117

118 We gave it a shot Environmental Policy Collection 118

119 We developed a collection policy 119

120 Defined the scope and the boundaries 120

121 We knew where we were going to put it 121

122 122

123 123

124 We knew how we were going to capture and process the content 124

125 125

126 We just had to decide on what 126

127 127

128 128

129 We gave each publication a proper home on the Web 129

130 130

131 Made it easy to cite 131

132 132

133 Made it easy to get the metadata for the item 133

134 134

135 We surrounded it with similar content 135

136 136

137 137

138 Which allows for browsing and unexpected discovery of resources 138

139 139

140 We added a number of APIs to the collection 140

141 141

142 And people began to use it 142

143 143

144 We should hit 100,000 item uses next month 144

145 600 items with 100,000 uses in three years Not bad. 145

146 This work has surfaced more questions than answers 146

147 Should we be doing this? 147

148 If a vendor offered this service we would probably subscribe to it. 148

149 To me it looks very similar to what libraries have always done. 149

150 How do we scale this activity? 150

151 How do we distribute the work and responsibility for building collections of born digital content? 151

152 How do we build the infrastructure that supports this kind of digital library work? 152

153 How do we empower collection libraries, subject experts, liaisons to made selection decisions and contribute to an actively growing collection of resources? 153

154 How do we make it efficient, scalable, and fun? 154

155 Anyone can point to a web site or document, but we have a special name for those that select, acquire, organize, and preserve information and provide access to and services for that information: We call them Libraries. 155

156