A time machine for the OCDB

Transcription

1 A time machine for the OCDB Dario Berzano ALICE Offline Week - July 19, 2017

2 OCDB source: AliEn Primary source of OCDB Calibration data in multiple ROOT files One XML file mapping run ranges to years Accessed by using raw:// as OCDB storage in AliRoot Multiple files to be accessed remotely: slow, causes large memory footprint No time machine feature: cannot select the OCDB status at a given moment in time!2

3 OCDB source: CVMFS As authoritative as AliEn: live sync upon AliEn updates, and has caching for faster access Directory structure identical to AliEn, except for Monte Carlos (with no reason ): AliEn: /alice/simulation/2008/v4-15-release/{full,ideal,residual} CVMFS: /cvmfs/alice-ocdb.cern.ch/calibration/mc/{full,ideal,residual} Fixes to make it work for real: alisw/aliroot#232 + alisw/alidpg#31 (usable from v ) Accessible as an alternative to AliEn by simply doing the following two steps together: Storage set to raw:// (same as AliEn) Environment variable exported: OCDB_PATH=/cvmfs/alice-ocdb.cern.ch Has native snapshotting feature, but it s still not usable in practice, see later on!3

4 OCDB source: snapshots Snapshots: make sure all jobs of same set will use the exact same OCDB A single dummy Grid job is run first to see what OCDB objects are actually used Objects used only are stored in a ROOT file This single ROOT file is the OCDB source for all jobs of the same set Solves the access problem: ROOT file is made available in the input box, no remote access!4

5 Current proposals for time machine snapshots Current snapshots contain only the objects, at a given moment in time, that serve the purpose of a single set of jobs Time machine snapshots: access the full OCDB as it was at a given moment in time We are file-based: just filter in only the files ones available at the selected time Three major proposals: Work from Raffaele (and an earlier implementation from me): save list of files at the time Current proposal: rely on files timestamps Native CVMFS snapshots and tagging!5

6 Snapshotting by saving list of files OCDB experts will start a procedure when they want the snapshot to be taken The procedure saves the result of a find command on the whole tree to a file This file is then used by AliRoot to know what files to consider Implementation problems: Requires manual intervention Requires an additional package where to store the list of files!6

7 Snapshotting by using files timestamps OCDB users will request with AliRoot the given timestamp It can be as easy as AliCDBManager::SetTimestamp(), and that s it The CDB manager will filter out all files newer than that date Works both on current CVMFS and AliEn Requires at most two full days of development: we already have all that s needed Relies on two pillars: File creation timestamps are reliable No OCDB file is ever deleted (a new version is uploaded instead) Only disadvantage: we would have to use timestamps instead of names But: easy workarounds can be made, if names are really important!7

8 Snapshotting by using CVMFS native features Whenever a CVMFS upload occurs, a new snapshot is created: no file is ever deleted Snapshots can be given a name (they can be tagged) Always possible to mount a CMVFS snapshot/tag from the past, by setting the configuration variable CVMFS_REPOSITORY_TAG currently accessible only as root user, and affects all users on the node :-( Proposal from the CVMFS team: make all snapshots, or tags, or both, accessible as special directory trees: /cvmfs/alice-ocdb.cern.ch/.tags/myspecialtag/ This seems to be the ideal solution, however time is a tyrant: Needs implementation, and testing: feature is not yet implemented Needs deployment: wait for Grid sites to use a CVMFS version supporting this feature!8

9 Where do we go?!9