Updates on the BHL Global Cluster. biodiversity heritage library

Transcription

1 Updates on the BHL Global Cluster biodiversity heritage library anthony goddard phil cryer 1

2 Us? o We do this talk a lot.. generally our shirts match. 2

3 What is the BHL? BHL - The Biodiversity Heritage Library o digitization component of the Encylopedia of Life o a consortium of a global partners o aims to share historic biodiversity literature texts o provide open access of all content o free for all 3

4 Why do we need a cluster? All BHL data is at the Internet Archive in San Francisco o no redundancy o single point of failure (earthquake risk) o limited in how we could serve o no easy way to analyze data First global BHL cluster gives us o redundancy o no single point of failure o various new serving options o new ways to run analytics #win! 4

5 5

6 6

7 Use Linux and open source software running on commodity hardware to create a scalable, distributed filesystem. 7

8 software 8

9 hardware 9

10 10

11 # ls -lh /mnt/glusterfs/www/a/actasocietatissc26suom total 649M -rwxr-xr-x 1 www-data www-data 19M :55 actasocietatissc26suom_abbyy.gz -rwxr-xr-x 1 www-data www-data 28M :53 actasocietatissc26suom_bw.pdf -rwxr-xr-x 1 www-data www-data 1.3K :21 actasocietatissc26suom_dc.xml -rwxr-xr-x 1 www-data www-data 18M :05 actasocietatissc26suom.djvu -rwxr-xr-x 1 www-data www-data 1.3M :54 actasocietatissc26suom_djvu.txt -rwxr-xr-x 1 www-data www-data 14M :08 actasocietatissc26suom_djvu.xml -rwxr-xr-x 1 www-data www-data 4.4K :42 actasocietatissc26suom_files.xml -rwxr-xr-x 1 www-data www-data 20M :57 actasocietatissc26suom_flippy.zip -rwxr-xr-x 1 www-data www-data 285K :52 actasocietatissc26suom.gif -rwxr-xr-x 1 www-data www-data 193M :51 actasocietatissc26suom_jp2.zip -rwxr-xr-x 1 www-data www-data 5.7K :21 actasocietatissc26suom_marc.xml -rwxr-xr-x 1 www-data www-data 2.0K :21 actasocietatissc26suom_meta.mrc -rwxr-xr-x 1 www-data www-data :21 actasocietatissc26suom_metasource.xml -rwxr-xr-x 1 www-data www-data 2.2K :20 actasocietatissc26suom_meta.xml -rwxr-xr-x 1 www-data www-data 279K :42 actasocietatissc26suom_names.xml -rwxr-xr-x 1 www-data www-data 324M :28 actasocietatissc26suom_orig_jp2.tar -rwxr-xr-x 1 www-data www-data 34M :35 actasocietatissc26suom.pdf -rwxr-xr-x 1 www-data www-data 365K :28 actasocietatissc26suom_scandata.xml 11

12 initial population 12

13 13

14 14

15 the plan Internet2 - woohoo o This will take forever (it took longer) o We need more space (not 24TB) o something s overloading the network (oops) o this checksum is wrong (what the...) Lessons learned would we do it again? Probably not. 15

16 code: grabbyd 1 Internet Archive, San Francisco BHL Global, Woods Hole 16

17 code: grabbyd_reporting 17

18 code: bhl-sync Open source Dropbox model inotify lsyncd OpenSSH rsync 18

19 all of our created code is open sourced and available at bit.ly/bhl-bits 19

20 20

21 Replication Replication 21

22 BHL content distribution Internet Archive, San Francisco 1 BHL Global, Woods Hole? BHL China, Beijing 2 2? BHL, St. Louis BHL Europe, London BHL Australia, Melbourne 22

23 BHL content + local data Internet Archive, San Francisco BHL Global, Woods Hole BHL China, Beijing Content sourced from China, scanned by Internet Archive, replicated into BHL Global 23

24 BHL content + regional data Internet Archive, San Francisco BHL Global, Woods Hole? BHL Europe, Paris BHL Europe, London BHL Europe, Berlin Content sourced from BHL Europe partners may, or may not, be passed back to Internet Archive and BHL Global 24

25 other replication challenges deleting content - "going dark" new content coming in from other sources (localization of content) distributing modified content 25

26 fedora-commons integration Repository platform storage, access and management digital content a base for software developers to build tools for sharing free, community supported, open source software 26

27 fedora-commons integration Repository platform storage, access and management digital content a base for software developers to build tools for sharing free, community supported, open source software Maintains a persistent, stable, digital archive o provides backup, redundancy and disaster recovery o complements existing architecture by incorporating open standards o stores data in a neutral manner o shares data via OAI 27

28 BHL content distribution Internet Archive, San Francisco BHL Global, Woods Hole Fedora-commons BHL, St. Louis BHL Europe, London 28

29 BHL content distribution Internet Archive, San Francisco BHL Global, Woods Hole Fedora-commons OAI BHL node Fedora-commons 29

30 BHL content distribution Internet Archive, San Francisco BHL Global, Woods Hole Fedora-commons OAI BHL node Fedora-commons 30

31 computational services 31

32 thanks. anthony goddard phil cryer all code available bit.ly/bhl-bits presentation slides on slidesha.re/bhl-slides 32