Recent Developments in the CernVM-FS Server Backend

Recent Developments in the CernVM-FS Server Backend René Meusel Jakob Blomer, Gerardo Ganis, Predrag Buncic, Seppo Heikkila ACAT 2014 - Prague, 4th of September

1 Usage Statistics and Adoption 2 New Challenges and Features 3 File System History 4 Garbage Collection 5 Smart Stratum1 Servers 2

What is CernVM-FS? Scalable software distribution system! Infrequent atomic updates in a central location Read-only access on the clients HTTP based global data transfer! Minimal protocol requirements Aggressive hierarchical cache strategy Assumption: Coherent working set on physically close nodes (cf. software vs. data distribution) Accessible through a mounted file system (POSIX) FUSE module, NFS exported FUSE volume or Parrot 3

Who Uses CernVM-FS? All LHC experiments! CernVM 3! Operating system in CernVM-FS Others beyond the HEP community! Human Brain Project, BioMed, VLEMED, Stratum0s at CERN, RAL, NIKHEF, Fermilab, DESY, 4

Repository Statistics Repository Files Refer. Objects Volume ø File Size atlas.cern.ch 34'500'000 3'700'000 2.1 TiB 66.2 kib cms.cern.ch 30'600'000 4 800'000 0.9 TiB 33.1 kib lhcb.cern.ch 13 600'000 4'600'000 0.5 TiB 41.9 kib alice.cern.ch 5'900'000 240'000 0.5 TiB 90.7 kib ams.cern.ch 2'900'000 1'900'000 1.9 TiB 0.7 MiB alice-ocdb.cern.ch 700'000 700'000 0.1 TiB 0.2 MiB atlas-condb.cern.ch 8'000 7'800 0.5 TiB 60.8 MiB Mainly Software Software + Conditions Data Conditions Data Files and Volume as saved in the CernVM-FS catalogs Actual number of Referenced Objects is compressed and de-duplicated Based on latest revision - no history involved (Effective: August 2014) 5

New Challenges 6

New Challenges Large Files (> 200 MiB) Long term data preservation Rapidly changing repository content Increasing configuration distribution effort Instant repository update propagation 7

CernVM-FS Repository From POSIX File Filesystem to Content-Addressable Objects 8

From POSIX to Blob-Objects release x86 config default.conf specific.conf executable fancy_tool do_magic.sh x86_64 specific.conf fancy_tool armv7 fancy_tool fancy_tool_debug run.sh setup.py foo.bar 9

From POSIX to Blob-Objects release Data Object x86 config default.conf specific.conf executable fancy_tool do_magic.sh x86_64 specific.conf fancy_tool 966672a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea2261733431f9d9ab0a 001e62aad4c6722f96a1c4a7a3865496c02b4aad b53283980b78efb04ba9f0b0ff38d055bd3d751c 949324a4d8ef529369c3d910a6cf001f562d07fd 2ee4f4dfc208f374855dc78dbb7043811c994258 a11b16694d6abf72e412ead6b0721c80c7dc98a7 armv7 fancy_tool fancy_tool_debug run.sh setup.py foo.bar c6fb1c3da08b88ae3f35293672254afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd484874637 6176e4e8ce9343570b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 9

From POSIX to Blob-Objects release x86 config default.conf specific.conf executable fancy_tool do_magic.sh x86_64 specific.conf fancy_tool armv7 fancy_tool fancy_tool_debug run.sh setup.py foo.bar Catalog: /release Sub-Catalog: /release/x86/config 966672a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea2261733431f9d9ab0a 001e62aad4c6722f96a1c4a7a3865496c02b4aad b53283980b78efb04ba9f0b0ff38d055bd3d751c 949324a4d8ef529369c3d910a6cf001f562d07fd Sub-Catalog: /release/x86_64 2ee4f4dfc208f374855dc78dbb7043811c994258 a11b16694d6abf72e412ead6b0721c80c7dc98a7 Sub-Catalog: /release/armv7 c6fb1c3da08b88ae3f35293672254afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd484874637 6176e4e8ce9343570b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 Data Object 10

From POSIX to Blob-Objects release x86 config default.conf specific.conf executable fancy_tool do_magic.sh x86_64 specific.conf fancy_tool armv7 fancy_tool fancy_tool_debug run.sh setup.py foo.bar 038f625d0790e06b0848a04bef90a51bd7b3ebecC 59bb67e545ac1951ac0f274ff63e8d2cc78ef420C 966672a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea2261733431f9d9ab0a 001e62aad4c6722f96a1c4a7a3865496c02b4aad b53283980b78efb04ba9f0b0ff38d055bd3d751c 949324a4d8ef529369c3d910a6cf001f562d07fd 11f58905f87d7ad5513aede20e21722b890eb9d6C 2ee4f4dfc208f374855dc78dbb7043811c994258 a11b16694d6abf72e412ead6b0721c80c7dc98a7 949324a4d8ef529369c3d910a6cf001f562d07fdC c6fb1c3da08b88ae3f35293672254afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd484874637 6176e4e8ce9343570b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 Data Object Root Catalog Catalog 12

From POSIX to Blob-Objects 038f625d0790e06b0848a04bef90a51bd7b3ebecC Data Object 59bb67e545ac1951ac0f274ff63e8d2cc78ef420C 966672a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea2261733431f9d9ab0a Root Catalog Catalog 001e62aad4c6722f96a1c4a7a3865496c02b4aad b53283980b78efb04ba9f0b0ff38d055bd3d751c 949324a4d8ef529369c3d910a6cf001f562d07fd 11f58905f87d7ad5513aede20e21722b890eb9d6C 2ee4f4dfc208f374855dc78dbb7043811c994258 a11b16694d6abf72e412ead6b0721c80c7dc98a7 949324a4d8ef529369c3d910a6cf001f562d07fdC c6fb1c3da08b88ae3f35293672254afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd484874637 6176e4e8ce9343570b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 13

From POSIX to Blob-Objects.cvmfspublished 038f625d0790e06b0848a04bef90a51bd7b3ebecC 59bb67e545ac1951ac0f274ff63e8d2cc78ef420C 966672a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea2261733431f9d9ab0a Data Object Manifest Root Catalog Catalog 001e62aad4c6722f96a1c4a7a3865496c02b4aad b53283980b78efb04ba9f0b0ff38d055bd3d751c 949324a4d8ef529369c3d910a6cf001f562d07fd 11f58905f87d7ad5513aede20e21722b890eb9d6C 2ee4f4dfc208f374855dc78dbb7043811c994258 a11b16694d6abf72e412ead6b0721c80c7dc98a7 949324a4d8ef529369c3d910a6cf001f562d07fdC c6fb1c3da08b88ae3f35293672254afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd484874637 6176e4e8ce9343570b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 13

From POSIX to Blob-Objects.cvmfspublished 038f625d0790e06b0848a04bef90a51bd7b3ebecC 59bb67e545ac1951ac0f274ff63e8d2cc78ef420C 966672a53bec6b0e43137e187d9bc5dce05d8443 dae8d8c367149f4b71f5ea2261733431f9d9ab0a 001e62aad4c6722f96a1c4a7a3865496c02b4aad b53283980b78efb04ba9f0b0ff38d055bd3d751c Data Object Manifest Root Catalog Catalog Reference 949324a4d8ef529369c3d910a6cf001f562d07fd 11f58905f87d7ad5513aede20e21722b890eb9d6C 2ee4f4dfc208f374855dc78dbb7043811c994258 a11b16694d6abf72e412ead6b0721c80c7dc98a7 949324a4d8ef529369c3d910a6cf001f562d07fdC c6fb1c3da08b88ae3f35293672254afa59b5f9cc 5c1c639b9a39a3c77c790768c06b2dd484874637 6176e4e8ce9343570b55aea9d771fe65f018ccf9 8599d27418cf321a855d0c79091f1dfd5bec202d 17b9caf70786d9a8444e51d77ea7495b9a5e8ce5 13

From POSIX to Blob-Objects Merkle tree! only.cvmfspublished needs to be signed Content-Addressable Storage! File de-duplication Trivial file integrity checks Flat Namespace! Perfect for HTTP caching Minimal storage API requirements (PUT, GET, [DELETE]).cvmfspublished 038f625d0790... 59bb67e545ac... 966672a53bec... dae8d8c36714... 001e62aad4c6... b53283980b78... 11f58905f87d... 2ee4f4dfc208... a11b16694d6a... c6fb1c3da08b... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... 17b9caf70786... Stratum0 (backend storage) 14

Repository Storage RESTful backend storage - PUT, GET, (DELETE) POSIX compliant file systems Stratum0 (backend storage) Key-Value stores (using S3 API - Seppo Heikkila) Pluggable storage connector implementation! Facilitate implementation of native Key-Value store APIs Like: Basho Riak, Ceph, Amazon Dynamo, 15

File System History Named Repository Snapshots 16

File System History Client: Mount historic revisions of a repository Use legacy software in its contemporary environment Long term data preservation Stratum0: Rollback to previous revisions Undo - Overwrite broken revisions with previous version 17

File System History.cvmfspublished 038f625d0790... 59bb67e545ac... 966672a53bec... dae8d8c36714... 001e62aad4c6... b53283980b78... 11f58905f87d... 2ee4f4dfc208... a11b16694d6a... c6fb1c3da08b... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... a3767d11dc42... 7ae8e0fad52d... 3d3d8bbb77e9... d4b20e277a21... cfbeae3135ac... 4e519bb5d4ac... 9148b12e983d... 5427187f0ad3... c82c149ea1ae... e0feb5d7265f... b430882696c2... Data Object Manifest Root Catalog Catalog Reference 18

File System History.cvmfspublished 038f625d0790... 59bb67e545ac... 966672a53bec... dae8d8c36714... 001e62aad4c6... b53283980b78... 11f58905f87d... 2ee4f4dfc208... a11b16694d6a... c6fb1c3da08b... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... a3767d11dc42... 7ae8e0fad52d... 3d3d8bbb77e9... d4b20e277a21... cfbeae3135ac... 4e519bb5d4ac... 9148b12e983d... 5427187f0ad3... c82c149ea1ae... e0feb5d7265f... b430882696c2... 6dd8e8407467... 19bfb8b6eb62... c35b3a93d603... c098919c0431... 40202e0018f5... b71381923d68... 066575c8bb2d... Data Object Manifest Root Catalog Catalog Reference 18

File System History.cvmfspublished 038f625d0790... a3767d11dc42... 6dd8e8407467... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... c35b3a93d603... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... a11b16694d6a... c82c149ea1ae... e0feb5d7265f... b430882696c2... Data Object Manifest d98d921a3213... f609e064cda4... 9742b7c70978... c6fb1c3da08b... Root Catalog 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... Catalog Reference 9642a79d7589... 246819803a9f... 5772a70f8101... 18

File System History.cvmfspublished 612c764ab4cb... 038f625d0790... a3767d11dc42... 6dd8e8407467... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... c35b3a93d603... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... a11b16694d6a... c82c149ea1ae... e0feb5d7265f... b430882696c2... Data Object Manifest d98d921a3213... f609e064cda4... 9742b7c70978... c6fb1c3da08b... Root Catalog 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... Catalog Reference History Index 9642a79d7589... 246819803a9f... 5772a70f8101... 18

File System History.cvmfspublished 612c764ab4cb... v3.0 v2.1.1 038f625d0790... a3767d11dc42... 6dd8e8407467... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... c35b3a93d603... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... a11b16694d6a... c82c149ea1ae... e0feb5d7265f... b430882696c2... Data Object Manifest d98d921a3213... f609e064cda4... 9742b7c70978... c6fb1c3da08b... Root Catalog 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... Catalog Reference History Index 9642a79d7589... 246819803a9f... 5772a70f8101... 18

File System History.cvmfspublished 612c764ab4cb... v3.0 v2.1.1 038f625d0790... a3767d11dc42... 6dd8e8407467... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... c35b3a93d603... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... c82c149ea1ae... d98d921a3213... a11b16694d6a... e0feb5d7265f... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... atlas.cern.ch.conf [ ]! CVMFS_REPOSITORY_TAG=v3.0 1ba1d0fe892d... 9642a79d7589... 246819803a9f... 5772a70f8101... 19

File System History.cvmfspublished 612c764ab4cb... v3.0 v2.1.1 038f625d0790... a3767d11dc42... 6dd8e8407467... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... c35b3a93d603... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 2ee4f4dfc208... 5427187f0ad3... c82c149ea1ae... mount + pin 3caf7e8664f8... d98d921a3213... a11b16694d6a... e0feb5d7265f... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... atlas.cern.ch.conf [ ]! CVMFS_REPOSITORY_TAG=v3.0 1ba1d0fe892d... 9642a79d7589... 246819803a9f... 5772a70f8101... 19

Garbage Collection Permanently Removing Overwritten or Deleted Files in Volatile Repositories 20

Garbage Collection CernVM-FS backend initially designed as insert-only New use-case: LHC experiment s nightly build repositories high update rate (up to twice a day) large volume of newly staged files (10-100GiB) short lived revisions (stay online max. two weeks) Insert-only quickly fills up backend storage! 21

LHCb Nightly Builds Regular Files Referenced Objects Stored Objects 16M 12M 8M 4M 1 Revision 54 22

Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v2.1.1 038f625d0790... a3767d11dc42... 6dd8e8407467... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... c35b3a93d603... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... c82c149ea1ae... d98d921a3213... a11b16694d6a... e0feb5d7265f... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... Data Object Manifest Catalog Reference 9642a79d7589... 246819803a9f... 5772a70f8101... (schematic -not an actual repository) Root Catalog History Index 23

Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v2.1.1 038f625d0790... a3767d11dc42... 6dd8e8407467... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... c35b3a93d603... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... c82c149ea1ae... d98d921a3213... a11b16694d6a... e0feb5d7265f... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d7589... 246819803a9f... 5772a70f8101... 24

Garbage Collection.cvmfspublished 612c764ab4cb... v3.0 v2.1.1 038f625d0790... a3767d11dc42... 6dd8e8407467... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... c35b3a93d603... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... c82c149ea1ae... d98d921a3213... a11b16694d6a... e0feb5d7265f... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d7589... 246819803a9f... 5772a70f8101... 25

Garbage Collection.cvmfspublished 612c764ab4cb... v2.1.1 038f625d0790... a3767d11dc42... 6dd8e8407467... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... c35b3a93d603... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... c82c149ea1ae... d98d921a3213... a11b16694d6a... e0feb5d7265f... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d7589... 246819803a9f... 5772a70f8101... 25

Garbage Collection.cvmfspublished 612c764ab4cb... remove v2.1.1 038f625d0790... a3767d11dc42... 6dd8e8407467... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... 19bfb8b6eb62... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... c35b3a93d603... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... c82c149ea1ae... d98d921a3213... a11b16694d6a... e0feb5d7265f... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d7589... 246819803a9f... 5772a70f8101... 25

Garbage Collection.cvmfspublished 612c764ab4cb... v2.1.1 038f625d0790... a3767d11dc42... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... c82c149ea1ae... d98d921a3213... a11b16694d6a... e0feb5d7265f... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... (schematic -not an actual repository) remove content in v3.0 find objects that are referenced nowhere else 9642a79d7589... 246819803a9f... 5772a70f8101... 26

Garbage Collection.cvmfspublished 612c764ab4cb... v2.1.1 038f625d0790... a3767d11dc42... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... c82c149ea1ae... d98d921a3213... a11b16694d6a... e0feb5d7265f... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... (schematic -not an actual repository) remove content in a3767 currently: stop-the-world GC no concurrent publish 9642a79d7589... 246819803a9f... 5772a70f8101... 27

Garbage Collection.cvmfspublished 612c764ab4cb... v2.1.1 038f625d0790... a3767d11dc42... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... c82c149ea1ae... d98d921a3213... a11b16694d6a... e0feb5d7265f... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... (schematic -not an actual repository) remove content in a3767 currently: stop-the-world GC no concurrent publish 9642a79d7589... 246819803a9f... 5772a70f8101... 28

Garbage Collection.cvmfspublished 612c764ab4cb... remove v2.1.1 038f625d0790... a3767d11dc42... 49f05c12cc29... 59bb67e545ac... 7ae8e0fad52d... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... 95a078736f41... dae8d8c36714... d4b20e277a21... c098919c0431... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 5427187f0ad3... 3caf7e8664f8... 2ee4f4dfc208... c82c149ea1ae... d98d921a3213... a11b16694d6a... e0feb5d7265f... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... (schematic -not an actual repository) remove content in a3767 currently: stop-the-world GC no concurrent publish 9642a79d7589... 246819803a9f... 5772a70f8101... 28

Garbage Collection.cvmfspublished 612c764ab4cb... v2.1.1 038f625d0790... 49f05c12cc29... 59bb67e545ac... acebb88f4c66... 966672a53bec... 3d3d8bbb77e9... 95a078736f41... dae8d8c36714... d4b20e277a21... ea0630b16316... 001e62aad4c6... cfbeae3135ac... 40202e0018f5... 79fb016f0bda... b53283980b78... 4e519bb5d4ac... b71381923d68... f3bd1410d41d... 9148b12e983d... 066575c8bb2d... 3d13859e193f... 11f58905f87d... 3caf7e8664f8... 2ee4f4dfc208... d98d921a3213... a11b16694d6a... f609e064cda4... b430882696c2... 9742b7c70978... c6fb1c3da08b... 1ba1d0fe892d... 5c1c639b9a39... 6176e4e8ce93... 8599d27418cf... (schematic -not an actual repository) remove content in a3767 currently: stop-the-world GC no concurrent publish 9642a79d7589... 246819803a9f... 5772a70f8101... 29

Garbage Collection Mark-and-Sweep implementation! Two-stage approach: Traverse preserved catalogs and log referenced objects Traverse condemned catalogs and match against log Full walk of the repository s catalog graph required Time consuming task 30

Smart Stratum1 Servers Automatic Stratum1 Ordering, Push Replication to Stratum1 Servers 31

Smart Stratum1 Servers Equip Stratum1 servers with RESTful API Automatic Stratum1 ordering on the client side Based on GeoIP database to determine closest replicas (to come in CernVM-FS 2.1.20 - Dave Dykstra - Fermilab/OSG) Triggered (instant) replication of new revisions Repository s private key for authentication 32

Wrap up 33

New Challenges Large Files (> 200 MiB) File chunking Long term data preservation Named repository snapshots Rapidly changing repository content Garbage collection Increasing configuration distribution effort Central bootstrapping configuration repository Instant repository update propagation Push replication from Stratum0 to Stratum1 34

Main New Features Alternative Storage Backends based on keyvalue stores through the S3 API (Seppo Heikkila) Named Snapshots and History for long term software accessibility and error recovery Garbage Collection for rapidly changing repositories (expected in CernVM-FS 2.1.20) Transactional Repository Updates for better publishing performance and a robust backend 35

Other New Features Automatic Ordering of Stratum1 Mirrors based on geo-ip location (Dave Dykstra - OSG) Configuration Bootstrap Repositories to facilitate public key and configuration distribution (not yet released) Chunking of Large Files for better cache exploitation and traffic efficiency Fully Parallel File Processing to speed up snapshot publishing 36

Backup Slides 38

Repository Growth Data Volume Referenced Objects Directory Entries Example Repository: atlas.cern.ch Size approximately doubled in two years Maximal values: Data: 2.1 TiB Entries: 48.0 M Objects: ~3.8 M 07/2012 08/2012 09/2012 10/2012 11/2012 12/2012 01/2013 02/2013 03/2013 04/2013 05/2013 06/2013 07/2013 08/2013 09/2013 10/2013 11/2013 12/2013 01/2014 02/2014 03/2014 04/2014 05/2014 06/2014 07/2014 08/2014 39

What is CernVM-FS? Stratum0 40

What is CernVM-FS? Stratum0 Stratum1 (replication) 40

What is CernVM-FS? Local Cache Stratum0 Stratum1 (replication) 40

What is CernVM-FS? DESY RAL CERN Fermilab?????? (schematic - does not reflect the real setup)??? 41

Stratum 0 Release Manager Machine 42

Updating a Repository Stratum0 (backend storage) 43

Updating a Repository CVMFS Revision X (read only) Stratum0 (backend storage) 43

Updating a Repository Union File System (writable) CVMFS Revision X (read only) Stratum0 (backend storage) 43

Updating a Repository Union File System (writable) CVMFS Revision X (read only) Stratum0 (backend storage) Synchronisation 43

Updating a Repository CVMFS Revision X + 1 (read only) Stratum0 (backend storage) 44

Updating a Repository Stratum1 CVMFS Revision X + 1 (read only) Stratum0 (backend storage) Stratum1 44

Updating a Repository Union File System (writable) CVMFS Revision X + 1 (read only) Stratum0 (backend storage) 45

Updating a Repository Union File System (writable) Discard CVMFS Revision X + 1 (read only) Stratum0 (backend storage) 45

Updating a Repository CVMFS Revision X + 1 (read only) Stratum0 (backend storage) 46