Recent Developments in the CernVM-FS Server Backend

Transcription

1 Recent Developments in the CernVM-FS Server Backend René Meusel Jakob Blomer, Gerardo Ganis, Predrag Buncic, Seppo Heikkila ACAT Prague, 4th of September

2 1 Usage Statistics and Adoption 2 New Challenges and Features 3 File System History 4 Garbage Collection 5 Smart Stratum1 Servers 2

3 What is CernVM-FS? Scalable software distribution system! Infrequent atomic updates in a central location Read-only access on the clients HTTP based global data transfer! Minimal protocol requirements Aggressive hierarchical cache strategy Assumption: Coherent working set on physically close nodes (cf. software vs. data distribution) Accessible through a mounted file system (POSIX) FUSE module, NFS exported FUSE volume or Parrot 3

4 Who Uses CernVM-FS? All LHC experiments! CernVM 3! Operating system in CernVM-FS Others beyond the HEP community! Human Brain Project, BioMed, VLEMED, Stratum0s at CERN, RAL, NIKHEF, Fermilab, DESY, 4

5 Repository Statistics Repository Files Refer. Objects Volume ø File Size atlas.cern.ch 34'500'000 3'700' TiB 66.2 kib cms.cern.ch 30'600' ' TiB 33.1 kib lhcb.cern.ch '000 4'600' TiB 41.9 kib alice.cern.ch 5'900' ' TiB 90.7 kib ams.cern.ch 2'900'000 1'900' TiB 0.7 MiB alice-ocdb.cern.ch 700' ' TiB 0.2 MiB atlas-condb.cern.ch 8'000 7' TiB 60.8 MiB Mainly Software Software + Conditions Data Conditions Data Files and Volume as saved in the CernVM-FS catalogs Actual number of Referenced Objects is compressed and de-duplicated Based on latest revision - no history involved (Effective: August 2014) 5

6 New Challenges 6

7 New Challenges Large Files (> 200 MiB) Long term data preservation Rapidly changing repository content Increasing configuration distribution effort Instant repository update propagation 7

8 CernVM-FS Repository From POSIX File Filesystem to Content-Addressable Objects 8

9 From POSIX to Blob-Objects release x86 config default.conf specific.conf executable fancy_tool do_magic.sh x86_64 specific.conf fancy_tool armv7 fancy_tool fancy_tool_debug run.sh setup.py foo.bar 9

10 From POSIX to Blob-Objects release Data Object x86 config default.conf specific.conf executable fancy_tool do_magic.sh x86_64 specific.conf fancy_tool a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea f9d9ab0a 001e62aad4c6722f96a1c4a7a c02b4aad b b78efb04ba9f0b0ff38d055bd3d751c a4d8ef529369c3d910a6cf001f562d07fd 2ee4f4dfc208f374855dc78dbb c a11b16694d6abf72e412ead6b0721c80c7dc98a7 armv7 fancy_tool fancy_tool_debug run.sh setup.py foo.bar c6fb1c3da08b88ae3f afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd e4e8ce b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 9

11 From POSIX to Blob-Objects release x86 config default.conf specific.conf executable fancy_tool do_magic.sh x86_64 specific.conf fancy_tool armv7 fancy_tool fancy_tool_debug run.sh setup.py foo.bar Catalog: /release Sub-Catalog: /release/x86/config a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea f9d9ab0a 001e62aad4c6722f96a1c4a7a c02b4aad b b78efb04ba9f0b0ff38d055bd3d751c a4d8ef529369c3d910a6cf001f562d07fd Sub-Catalog: /release/x86_64 2ee4f4dfc208f374855dc78dbb c a11b16694d6abf72e412ead6b0721c80c7dc98a7 Sub-Catalog: /release/armv7 c6fb1c3da08b88ae3f afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd e4e8ce b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 Data Object 10

12 From POSIX to Blob-Objects release x86 config default.conf specific.conf executable fancy_tool do_magic.sh x86_64 specific.conf fancy_tool armv7 fancy_tool fancy_tool_debug run.sh setup.py foo.bar Catalog: /release Sub-Catalog: /release/x86/config a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea f9d9ab0a 001e62aad4c6722f96a1c4a7a c02b4aad b b78efb04ba9f0b0ff38d055bd3d751c a4d8ef529369c3d910a6cf001f562d07fd Sub-Catalog: /release/x86_64 2ee4f4dfc208f374855dc78dbb c a11b16694d6abf72e412ead6b0721c80c7dc98a7 Sub-Catalog: /release/armv7 c6fb1c3da08b88ae3f afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd e4e8ce b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 Data Object 11

13 From POSIX to Blob-Objects release x86 config default.conf specific.conf executable fancy_tool do_magic.sh x86_64 specific.conf fancy_tool armv7 fancy_tool fancy_tool_debug run.sh setup.py foo.bar 038f625d0790e06b0848a04bef90a51bd7b3ebecC 59bb67e545ac1951ac0f274ff63e8d2cc78ef420C a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea f9d9ab0a 001e62aad4c6722f96a1c4a7a c02b4aad b b78efb04ba9f0b0ff38d055bd3d751c a4d8ef529369c3d910a6cf001f562d07fd 11f58905f87d7ad5513aede20e21722b890eb9d6C 2ee4f4dfc208f374855dc78dbb c a11b16694d6abf72e412ead6b0721c80c7dc98a a4d8ef529369c3d910a6cf001f562d07fdC c6fb1c3da08b88ae3f afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd e4e8ce b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 Data Object Root Catalog Catalog 12

14 From POSIX to Blob-Objects 038f625d0790e06b0848a04bef90a51bd7b3ebecC Data Object 59bb67e545ac1951ac0f274ff63e8d2cc78ef420C a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea f9d9ab0a Root Catalog Catalog 001e62aad4c6722f96a1c4a7a c02b4aad b b78efb04ba9f0b0ff38d055bd3d751c a4d8ef529369c3d910a6cf001f562d07fd 11f58905f87d7ad5513aede20e21722b890eb9d6C 2ee4f4dfc208f374855dc78dbb c a11b16694d6abf72e412ead6b0721c80c7dc98a a4d8ef529369c3d910a6cf001f562d07fdC c6fb1c3da08b88ae3f afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd e4e8ce b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 13

15 From POSIX to Blob-Objects.cvmfspublished 038f625d0790e06b0848a04bef90a51bd7b3ebecC 59bb67e545ac1951ac0f274ff63e8d2cc78ef420C a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea f9d9ab0a Data Object Manifest Root Catalog Catalog 001e62aad4c6722f96a1c4a7a c02b4aad b b78efb04ba9f0b0ff38d055bd3d751c a4d8ef529369c3d910a6cf001f562d07fd 11f58905f87d7ad5513aede20e21722b890eb9d6C 2ee4f4dfc208f374855dc78dbb c a11b16694d6abf72e412ead6b0721c80c7dc98a a4d8ef529369c3d910a6cf001f562d07fdC c6fb1c3da08b88ae3f afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd e4e8ce b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 13

16 From POSIX to Blob-Objects.cvmfspublished 038f625d0790e06b0848a04bef90a51bd7b3ebecC 59bb67e545ac1951ac0f274ff63e8d2cc78ef420C a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea f9d9ab0a 001e62aad4c6722f96a1c4a7a c02b4aad b b78efb04ba9f0b0ff38d055bd3d751c Data Object Manifest Root Catalog Catalog Reference a4d8ef529369c3d910a6cf001f562d07fd 11f58905f87d7ad5513aede20e21722b890eb9d6C 2ee4f4dfc208f374855dc78dbb c a11b16694d6abf72e412ead6b0721c80c7dc98a a4d8ef529369c3d910a6cf001f562d07fdC c6fb1c3da08b88ae3f afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd e4e8ce b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 13

17 From POSIX to Blob-Objects Merkle tree! only.cvmfspublished needs to be signed Content-Addressable Storage! File de-duplication Trivial file integrity checks Flat Namespace! Perfect for HTTP caching Minimal storage API requirements (PUT, GET, [DELETE]).cvmfspublished 038f625d bb67e545ac a53bec... dae8d8c e62aad4c6... b b f58905f87d... 2ee4f4dfc a11b16694d6a... c6fb1c3da08b... 5c1c639b9a e4e8ce d27418cf... 17b9caf Stratum0 (backend storage) 14

18 From POSIX to Blob-Objects Merkle tree! only.cvmfspublished needs to be signed Content-Addressable Storage! File de-duplication Trivial file integrity checks Flat Namespace! Perfect for HTTP caching Minimal storage API requirements (PUT, GET, [DELETE]).cvmfspublished 038f625d bb67e545ac a53bec... dae8d8c e62aad4c6... b b f58905f87d... 2ee4f4dfc a11b16694d6a... c6fb1c3da08b... 5c1c639b9a e4e8ce d27418cf... 17b9caf Stratum1 Stratum0 (backend storage) Stratum1 14

19 From POSIX to Blob-Objects Merkle tree! only.cvmfspublished needs to be signed Content-Addressable Storage! File de-duplication Trivial file integrity checks Flat Namespace! Perfect for HTTP caching Minimal storage API requirements (PUT, GET, [DELETE]).cvmfspublished 038f625d bb67e545ac a53bec... dae8d8c e62aad4c6... b b f58905f87d... 2ee4f4dfc a11b16694d6a... c6fb1c3da08b... 5c1c639b9a e4e8ce d27418cf... 17b9caf Stratum1 Stratum0 (backend storage) Stratum1 14

20 Repository Storage RESTful backend storage - PUT, GET, (DELETE) POSIX compliant file systems Stratum0 (backend storage) Key-Value stores (using S3 API - Seppo Heikkila) Pluggable storage connector implementation! Facilitate implementation of native Key-Value store APIs Like: Basho Riak, Ceph, Amazon Dynamo, 15

21 File System History Named Repository Snapshots 16

22 File System History Client: Mount historic revisions of a repository Use legacy software in its contemporary environment Long term data preservation Stratum0: Rollback to previous revisions Undo - Overwrite broken revisions with previous version 17

23 File System History.cvmfspublished 038f625d bb67e545ac a53bec... dae8d8c e62aad4c6... b b f58905f87d... 2ee4f4dfc a11b16694d6a... c6fb1c3da08b... 5c1c639b9a e4e8ce d27418cf... Data Object Manifest Root Catalog Catalog Reference 18

24 File System History.cvmfspublished 038f625d bb67e545ac a53bec... dae8d8c e62aad4c6... b b f58905f87d... 2ee4f4dfc a11b16694d6a... c6fb1c3da08b... 5c1c639b9a e4e8ce d27418cf... a3767d11dc ae8e0fad52d... 3d3d8bbb77e9... d4b20e277a21... cfbeae3135ac... 4e519bb5d4ac b12e983d f0ad3... c82c149ea1ae... e0feb5d7265f... b c2... Data Object Manifest Root Catalog Catalog Reference 18

25 File System History.cvmfspublished 038f625d bb67e545ac a53bec... dae8d8c e62aad4c6... b b f58905f87d... 2ee4f4dfc a11b16694d6a... c6fb1c3da08b... 5c1c639b9a e4e8ce d27418cf... a3767d11dc ae8e0fad52d... 3d3d8bbb77e9... d4b20e277a21... cfbeae3135ac... 4e519bb5d4ac b12e983d f0ad3... c82c149ea1ae... e0feb5d7265f... b c2... 6dd8e bfb8b6eb62... c35b3a93d c098919c e0018f5... b d c8bb2d... Data Object Manifest Root Catalog Catalog Reference 18

26 File System History.cvmfspublished 038f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc a11b16694d6a... c82c149ea1ae... e0feb5d7265f... b c2... Data Object Manifest d98d921a f609e064cda b7c c6fb1c3da08b... Root Catalog 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... Catalog Reference 9642a79d a9f a70f

27 File System History.cvmfspublished 612c764ab4cb f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc a11b16694d6a... c82c149ea1ae... e0feb5d7265f... b c2... Data Object Manifest d98d921a f609e064cda b7c c6fb1c3da08b... Root Catalog 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... Catalog Reference History Index 9642a79d a9f a70f

28 File System History.cvmfspublished 612c764ab4cb f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc a11b16694d6a... c82c149ea1ae... e0feb5d7265f... b c2... Data Object Manifest d98d921a f609e064cda b7c c6fb1c3da08b... Root Catalog 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... Catalog Reference History Index 9642a79d a9f a70f

29 File System History.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc a11b16694d6a... c82c149ea1ae... e0feb5d7265f... b c2... Data Object Manifest d98d921a f609e064cda b7c c6fb1c3da08b... Root Catalog 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... Catalog Reference History Index 9642a79d a9f a70f

30 File System History.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 5c1c639b9a e4e8ce d27418cf... atlas.cern.ch.conf [ ]! CVMFS_REPOSITORY_TAG=v3.0 1ba1d0fe892d a79d a9f a70f

31 File System History.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 5c1c639b9a e4e8ce d27418cf... atlas.cern.ch.conf [ ]! CVMFS_REPOSITORY_TAG=v3.0 1ba1d0fe892d a79d a9f a70f

32 File System History.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 5c1c639b9a e4e8ce d27418cf... atlas.cern.ch.conf [ ]! CVMFS_REPOSITORY_TAG=v3.0 1ba1d0fe892d a79d a9f a70f

33 File System History.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 5c1c639b9a e4e8ce d27418cf... atlas.cern.ch.conf [ ]! CVMFS_REPOSITORY_TAG=v3.0 1ba1d0fe892d a79d a9f a70f

34 File System History.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d... 2ee4f4dfc f0ad3... c82c149ea1ae... mount + pin 3caf7e8664f8... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 5c1c639b9a e4e8ce d27418cf... atlas.cern.ch.conf [ ]! CVMFS_REPOSITORY_TAG=v3.0 1ba1d0fe892d a79d a9f a70f

35 Garbage Collection Permanently Removing Overwritten or Deleted Files in Volatile Repositories 20

36 Garbage Collection CernVM-FS backend initially designed as insert-only New use-case: LHC experiment s nightly build repositories high update rate (up to twice a day) large volume of newly staged files (10-100GiB) short lived revisions (stay online max. two weeks) Insert-only quickly fills up backend storage! 21

37 LHCb Nightly Builds Regular Files Referenced Objects Stored Objects 16M 12M 8M 4M 1 Revision 54 22

38 Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... Data Object Manifest Catalog Reference 9642a79d a9f a70f (schematic -not an actual repository) Root Catalog History Index 23

39 Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... Data Object Manifest Catalog Reference 9642a79d a9f a70f (schematic -not an actual repository) Root Catalog History Index 23

40 Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... Data Object Manifest Catalog Reference 9642a79d a9f a70f (schematic -not an actual repository) Root Catalog History Index 23

41 Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d a9f a70f

42 Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d a9f a70f

43 Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d a9f a70f

44 Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d a9f a70f

45 Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d a9f a70f

46 Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d a9f a70f

47 Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d a9f a70f

48 Garbage Collection.cvmfspublished 612c764ab4cb... v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d a9f a70f

49 Garbage Collection.cvmfspublished 612c764ab4cb... remove v f625d a3767d11dc dd8e f05c12cc bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c a53bec... 3d3d8bbb77e9... c35b3a93d a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d a9f a70f

50 Garbage Collection.cvmfspublished 612c764ab4cb... v f625d a3767d11dc f05c12cc bb67e545ac... 7ae8e0fad52d... acebb88f4c a53bec... 3d3d8bbb77e a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d a9f a70f

51 Garbage Collection.cvmfspublished 612c764ab4cb... v f625d a3767d11dc f05c12cc bb67e545ac... 7ae8e0fad52d... acebb88f4c a53bec... 3d3d8bbb77e a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in a3767 currently: stop-the-world GC no concurrent publish 9642a79d a9f a70f

52 Garbage Collection.cvmfspublished 612c764ab4cb... v f625d a3767d11dc f05c12cc bb67e545ac... 7ae8e0fad52d... acebb88f4c a53bec... 3d3d8bbb77e a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in a3767 currently: stop-the-world GC no concurrent publish 9642a79d a9f a70f

53 Garbage Collection.cvmfspublished 612c764ab4cb... v f625d a3767d11dc f05c12cc bb67e545ac... 7ae8e0fad52d... acebb88f4c a53bec... 3d3d8bbb77e a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in a3767 currently: stop-the-world GC no concurrent publish 9642a79d a9f a70f

54 Garbage Collection.cvmfspublished 612c764ab4cb... v f625d a3767d11dc f05c12cc bb67e545ac... 7ae8e0fad52d... acebb88f4c a53bec... 3d3d8bbb77e a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in a3767 currently: stop-the-world GC no concurrent publish 9642a79d a9f a70f

55 Garbage Collection.cvmfspublished 612c764ab4cb... v f625d a3767d11dc f05c12cc bb67e545ac... 7ae8e0fad52d... acebb88f4c a53bec... 3d3d8bbb77e a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in a3767 currently: stop-the-world GC no concurrent publish 9642a79d a9f a70f

56 Garbage Collection.cvmfspublished 612c764ab4cb... remove v f625d a3767d11dc f05c12cc bb67e545ac... 7ae8e0fad52d... acebb88f4c a53bec... 3d3d8bbb77e a078736f41... dae8d8c d4b20e277a21... c098919c ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d f0ad3... 3caf7e8664f8... 2ee4f4dfc c82c149ea1ae... d98d921a a11b16694d6a... e0feb5d7265f... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in a3767 currently: stop-the-world GC no concurrent publish 9642a79d a9f a70f

57 Garbage Collection.cvmfspublished 612c764ab4cb... v f625d f05c12cc bb67e545ac... acebb88f4c a53bec... 3d3d8bbb77e a078736f41... dae8d8c d4b20e277a21... ea0630b e62aad4c6... cfbeae3135ac e0018f fb016f0bda... b b e519bb5d4ac... b d68... f3bd1410d41d b12e983d c8bb2d... 3d13859e193f... 11f58905f87d... 3caf7e8664f8... 2ee4f4dfc d98d921a a11b16694d6a... f609e064cda4... b c b7c c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a e4e8ce d27418cf... (schematic -not an actual repository) remove content in a3767 currently: stop-the-world GC no concurrent publish 9642a79d a9f a70f

58 Garbage Collection Mark-and-Sweep implementation! Two-stage approach: Traverse preserved catalogs and log referenced objects Traverse condemned catalogs and match against log Full walk of the repository s catalog graph required Time consuming task 30

59 Smart Stratum1 Servers Automatic Stratum1 Ordering, Push Replication to Stratum1 Servers 31

60 Smart Stratum1 Servers Equip Stratum1 servers with RESTful API Automatic Stratum1 ordering on the client side Based on GeoIP database to determine closest replicas (to come in CernVM-FS Dave Dykstra - Fermilab/OSG) Triggered (instant) replication of new revisions Repository s private key for authentication 32

61 Wrap up 33

62 New Challenges Large Files (> 200 MiB) File chunking Long term data preservation Named repository snapshots Rapidly changing repository content Garbage collection Increasing configuration distribution effort Central bootstrapping configuration repository Instant repository update propagation Push replication from Stratum0 to Stratum1 34

63 Main New Features Alternative Storage Backends based on keyvalue stores through the S3 API (Seppo Heikkila) Named Snapshots and History for long term software accessibility and error recovery Garbage Collection for rapidly changing repositories (expected in CernVM-FS ) Transactional Repository Updates for better publishing performance and a robust backend 35

64 Other New Features Automatic Ordering of Stratum1 Mirrors based on geo-ip location (Dave Dykstra - OSG) Configuration Bootstrap Repositories to facilitate public key and configuration distribution (not yet released) Chunking of Large Files for better cache exploitation and traffic efficiency Fully Parallel File Processing to speed up snapshot publishing 36

65 37

66 Backup Slides 38

67 Repository Growth Data Volume Referenced Objects Directory Entries Example Repository: atlas.cern.ch Size approximately doubled in two years Maximal values: Data: 2.1 TiB Entries: 48.0 M Objects: ~3.8 M 07/ / / / / / / / / / / / / / / / / / / / / / / / / /

68 What is CernVM-FS? Stratum0 40

69 What is CernVM-FS? Stratum0 Stratum1 (replication) 40

70 What is CernVM-FS? Stratum0 Stratum1 (replication) 40

71 What is CernVM-FS? Local Cache Stratum0 Stratum1 (replication) 40

72 What is CernVM-FS? DESY RAL CERN Fermilab?????? (schematic - does not reflect the real setup)??? 41

73 Stratum 0 Release Manager Machine 42

74 Updating a Repository Stratum0 (backend storage) 43

75 Updating a Repository CVMFS Revision X (read only) Stratum0 (backend storage) 43

76 Updating a Repository Union File System (writable) CVMFS Revision X (read only) Stratum0 (backend storage) 43

77 Updating a Repository Union File System (writable) CVMFS Revision X (read only) Stratum0 (backend storage) 43

78 Updating a Repository Union File System (writable) CVMFS Revision X (read only) Stratum0 (backend storage) Synchronisation 43

79 Updating a Repository CVMFS Revision X + 1 (read only) Stratum0 (backend storage) 44

80 Updating a Repository Stratum1 CVMFS Revision X + 1 (read only) Stratum0 (backend storage) Stratum1 44

81 Updating a Repository Stratum1 CVMFS Revision X + 1 (read only) Stratum0 (backend storage) Stratum1 44

82 Updating a Repository Union File System (writable) CVMFS Revision X + 1 (read only) Stratum0 (backend storage) 45

83 Updating a Repository Union File System (writable) CVMFS Revision X + 1 (read only) Stratum0 (backend storage) 45

84 Updating a Repository Union File System (writable) Discard CVMFS Revision X + 1 (read only) Stratum0 (backend storage) 45

85 Updating a Repository CVMFS Revision X + 1 (read only) Stratum0 (backend storage) 46

86 47