CMS evolution on data access: xrootd remote access and data federation. Giacinto DONVITO INFN-BARI

Transcription

1 CMS evolution on ata access: xroot remote access an ata feeration Giacinto DONVITO INFN-BARI

2 Outlook ò CMS computing moel: first assumptions an results ò Technology trens an impact on HEP Computing moels ò Xroot concepts an features ò CMS framework enhancements ò Xroot Reirector use cases an basic ieas ò Xroot Reirectors actual eployment ò Xroot first test an monitoring

3 Outlook ò CMS computing moel: first assumptions an results ò Technology trens an impact on HEP Computing moels ò Xroot concepts an features ò CMS framework enhancements ò Xroot Reirector use cases an basic ieas ò Xroot Reirectors actual eployment ò Xroot first test an monitoring

4 CMS computing moel: first assumptions ò The Computing Moels of the LHC experiments were esigne using Monarc (Moels of Networke Analysis at Regional Centers for LHC Experiments) Moel assumption: ò ò ò ò ò ò ò Resource are locate in istribute centers with a hierarchical structure ò ò ò ò Tier 0 Master computing center, full resource set (isks an tapes) Tier 1 National Centre, resource subset (isks an tapes) Tier 2 Major Centre (only isks) Tier 3 University Department The network will be bottleneck We will nee a hierarchical mass storage because we cannot affor sufficient local isk space The file catalogue will not scale We will overloa the source site if we ask for transfers to a large number of sites We nee to run job close to the ata to achieve efficient CPU utilization We nee a a priori structure an preictable ata utilization

5 CMS computing moel: first assumptions (it applies to all LHC exps, though) ~PBytes/sec Online System ~100 MBytes/sec 1 TIPS is approximately 25,000 SpecInt95 equivalents Tier 1 There is a bunch crossing every 25 nsecs. There are 100 triggers per secon Each triggere event is ~1 MByte in size France Regional Centre ~622 Mbits/sec or Air Freight (eprecate) Germany Regional Centre Tier 0 Offline Processor Farm ~20 TIPS Italy Regional Centre ~100 MBytes/sec CERN Computer Centre FermiLab ~4 TIPS ~622 Mbits/sec ~622 Mbits/sec Tier 2 Caltech ~1 TIPS Tier2 Centre Tier2 Centre Tier2 Centre Tier2 Centre ~1 TIPS ~1 TIPS ~1 TIPS ~1 TIPS Physics ata cache Institute Institute ~0.25TIPS Physicist workstations Institute ~1 MBytes/sec Institute Tier 4 Physicists work on analysis channels. Each institute will have ~10 physicists working on one or more channels; ata for these channels shoul be cache by the institute server [ Image courtesy of Harvey Newman, Caltech ]

6 CMS computing moel: first assumptions Other routes: T&!T! T! $$$ example: CMS Take CMS ata Cow as example CMS T&"T! transfers likely to be bursty an riven by analysis emans D&"4&"" MB4s for worst4best connecte T!+s )!""5* CMS T!"T& transfers almost entirely fairly continuous simulation transfers D &"MB4s )NOTE: aggregate input rate into T&+s comparable to rate from T"!* tape 280 MB/s (RAW, RECO, AOD) 225 MB/s (RAW) Tier M-SI2K 0.4 PB isk 4.9 PB tape 5 Gbps WAN WNs 225 MB/s (RAW) 280 MB/s (RAW, RECO, AOD) Tier-1s Tier-1 40 MB/s (RAW, RECO, AOD) Tier-0 Tier-1s Tier-2s AOD AOD 48 MB/s (MC) 240 MB/s (skimme AOD, Some RAW+RECO) 60 MB/s (skimme AOD, Some RAW+RECO) 12 MB/s (MC) Tier MSI2K 1.0 PB isk 2.4 PB tape ~10 Gbps WAN WNs Tier M-SI2K 0.2 PB isk 1 Gbps WAN ~800 MB/s (AOD skimming, ata reprocessing) Up to 1 GB/s (AOD analysis, calibration) NB: average sustaine throughputs. Courtesy: J.Hernanez (CMS) WNs CCR!""# $ Gran Sasso% &" June "# D' Bonacorsi!,

7 CMS computing moel: first results ò Data transfers among sites is much more reliable than expecte ò WAN network performances are growing fast an the network infrastructure is reliable ò Retrieving files from a remote sites coul be easier than retrieving it from a local hierarchical mass storage system ò It looks like we nee a clearer partition between isk an tape ò Using tape only like an archive ò Geographically istribute Job submission an matchmaking are working well ò The tool evelope to help the final users are easy an make the usage of the gri transparent to the user

8 CMS computing moel: first results From CERN From T1s It is easy to get out of a single T0/T1 up to MB/s sustaine of WAN transfer to all others CMS sites

9 CMS computing moel: first results A very small number of atasets are use very heavily

10 Outlook ò CMS computing moel: first assumptions an results ò Technology trens an impact on HEP Computing moels ò Xroot concepts an features ò CMS framework enhancements ò Xroot Reirector use cases an basic ieas ò Xroot Reirectors actual eployment ò Xroot first test an monitoring

11 Technology trens an impact on HEP Computing moels

12 Technology trens an impact on ò The WAN available banwith is comparable with the backbone available at LAN level ò Network flows are alreay larger than foreseen at this point in the LHC program, even with lower luminosity ò Some T2 s are very large ò All US ATLAS an US CMS T2 s have 10G capability. ò Some T1-T2 flows are quite large (several to 10Gbps) ò T2-T2 ata flows are also starting to become significant. ò The concept of transferring file regionally is substantially broken! HEP Computing moels

13 Technology trens an impact on HEP Computing moels ò The vision progressively moves away from all hierarchical moels to peer-peer ò True for both CMS an ATLAS ò For reasons of reuce latency, increase working efficiency ò The challenge in the next future will be the available IOPS on storage systems ò It is clear we nee to optimize the IO at the application level as the isk will not increase the performance too much in the future ò The hierarchical mass storage system coul not cope with ynamic ata stage-in request ò The cost per TB of the isk allow us to buil huge isk-only facilities

14 Outlook ò CMS computing moel: first assumptions an results ò Technology trens an impact on HEP Computing moels ò Xroot concepts an features ò CMS framework enhancements ò Xroot Reirector use cases an basic ieas ò Xroot Reirectors actual eployment ò Xroot first test an monitoring

15 Xroot concepts an features What Is xroot? ò A file access an ata transfer protocol ò Defines POSIX-style byte-level ranom access for ò Arbitrary ata organize as files of any type ò Ientifie by a hierarchical irectory-like name ò A reference software implementation ò Emboie as the xroot an cms aemons ò xroot aemon provies access to ata ò cms aemon clusters xroot aemons together

16 Xroot concepts an features What Isn t xroot? ò It is not a POSIX file system ò There is a FUSE implementation calle xrootfs ò An xroot client simulating a mountable file system ò It oes not provie full POSIX file system semantics ò It is not an Storage Resource Manager (SRM) ò Provies SRM functionality via BeStMan ò It is not aware of any file internals (e.g., root files) ò But is istribute with root an proof frameworks ò As it provies unique & efficient file access primitives

17 Xroot concepts an features Primary xroot Access Moes ò The root framework ò Use by most HEP an many Astro experiments (MacOS, Unix an Winows) ò POSIX preloa library ò Any POSIX compliant application (Unix only, no recompilation neee) ò File system in User SpacE ò A mounte xroot ata access system via FUSE (Linux an MacOS only) ò SRM, globus-url-copy, griftp, etc ò General gri access (Unix only) ò xrcp ò The parallel stream, multi-source copy comman (MacOS, Unix an Winows) ò xr ò The comman line interface for meta-ata operations (MacOS, Unix an Winows)

18 Xroot concepts an features What Makes xroot Unusual? ò A comprehensive plug-in architecture ò Security, storage back-ens (e.g., tape), proxies, etc ò Clusters wiely isparate file systems ò Practically any existing file system ò Distribute (share-everything) to JBODS (share-nothing) ò Unifie view at local, regional, an global levels ò Very low support requirements ò Harware an human aministration

19 Xroot concepts an features Protocol Driver (XRD) Authorization (bms, voms, etc) Protocol (1 of n) (xroot, proof, etc) Authentication (gsi, krb5, etc) lfn2pfn prefix encoing Let s take a closer look at xroot-style clustering Clustering (cms) Logical File System (ofs, sfs, alice, etc) Physical Storage System (ufs, hfs, hpss, etc) Replaceable plug-ins to accommoate any environment

20 Xroot concepts an features A Simple xroot Cluster Client 1: 4: open( /my/file ) Try open() at A xroot cms Manager (a.k.a. Reirector) 2: Who has /my/file? xroot cms xroot cms xroot cms A /my/file B C /my/file Data Servers

21 Xroot concepts an features Exploiting Stackability Client 1: 5: open( /my/file ) Try open() at ANL xroot cms Meta-Manager (a.k.a. Global Reirector) 8: open( /my/file ) 6: 7: open( /my/file ) Try open() at A Manager (a.k.a. Local Reirector) Data is uniformly available 2: Who has /my/file? By feerating three istinct sites Manager (a.k.a. Local Reirector) Manager (a.k.a. Local Reirector) xroot cms xroot cms xroot cms S S S xroot cms e xroot cms e xroot cms e r r r v v v C B /my/file e C /my/file e C /my/file e xroot cms xroot cms r xroot cms xroot cms r xroot cms xroot cms r s s s A /my/file C B /my/file A B A /my/file B ANL 3: Who has /my/file? SLAC 3: Who has /my/file? UTA 3: Who has /my/file? An exponentially parallel search! (i.e. O(2 Distribute Clusters n )) Feerate Distribute Clusters

22 Xroot concepts an features Feerate Distribute Clusters ò Unites multiple site-specific ata repositories ò Each site enforces its own access rules ò Usable even in the presence of firewalls ò Scalability increases as more sites join ò Essentially a real-time bit-torrent social moel ò Feerations are flui an changeable in real time ò Provie multiple ata sources to achieve high transfer rates ò Increase opportunities for ata analysis ò Base on what is actually available

23 Xroot concepts an features Copy Data Access Architecture ò The built-in File Resiency Manager rives ò ò Copy On Fault ò Deman riven (fetch to restore missing file) Copy On Request ò Pre-riven (fetch files to be use for analysis) Client open( /my/file ) xrcp x xroot://mm.org//my/file /my xroot cms Meta-Manager (a.k.a. Global Reirector) Manager (a.k.a. Local Reirector) Manager (a.k.a. Local Reirector) Manager (a.k.a. Local Reirector) xroot cms xroot cms xroot cms S S xroot cms e xroot cms e xroot cms r r v v C /my/file e C /my/file e C /my/file xroot cms xroot cms r xroot cms xroot cms r xroot cms xroot cms s s A /my/file B /my/file A /my/file B A /my/file B ANL SLAC UTA xrcp copies ata using two sources S e r v e r s

24 Xroot concepts an features Direct Data Access Architecture ò Use servers as if all of them were local ò Normal an easiest way of oing this ò Latency may be an issue (epens on algorithms & CPU-I/O ratio) ò Requires Cost-Benefit analysis to see if acceptable Client open( /my/file ) xroot cms Meta-Manager (a.k.a. Global Reirector) Manager (a.k.a. Local Reirector) Manager (a.k.a. Local Reirector) Manager (a.k.a. Local Reirector) xroot cms xroot cms xroot cms S S xroot cms e xroot cms e xroot cms r r v v C /my/file e C /my/file e C /my/file xroot cms xroot cms r xroot cms xroot cms r xroot cms xroot cms s s A /my/file B /my/file A B A /my/file B ANL SLAC UTA S e r v e r s

25 Xroot concepts an features Cache Data Access Architecture ò Front servers with a caching proxy server ò Client access proxy server for all ata ò Server can be central or local to client (i.e. laptop) ò Data comes from proxy s cache or other servers Client open( /my/file ) xroot xroot cms Meta-Manager (a.k.a. Global Reirector) Manager (a.k.a. Local Reirector) Manager (a.k.a. Local Reirector) Manager (a.k.a. Local Reirector) xroot cms xroot cms xroot cms S S xroot cms e xroot cms e xroot cms r r v v C /my/file e C /my/file e C /my/file xroot cms xroot cms r xroot cms xroot cms r xroot cms xroot cms s s A B A /my/file B A B ANL SLAC UTA S e r v e r s

26 Outlook ò CMS computing moel: first assumptions an results ò Technology trens an impact on HEP Computing moels ò Xroot concepts an features ò CMS framework enhancements ò Xroot Reirector use cases an basic ieas ò Xroot Reirectors actual eployment ò Xroot first test an monitoring

27 CMS framework enhancements ò The CMSSW 4_X contains ROOT 5.27/06b, which has significant I/O improvements over 3_x ò CMSSW 4_x also contains moest I/O improvements on top of out-of-the-box ROOT I/O ò New ROOT ae: ò Auto-flushing: All buffers are flushe to isk perioically, guaranteeing some level of locality. ò Buffer-resizing: Buffers are resize so approximately the same number of events are in each buffer. ò Rea coalescing: Nearby reas (but non-consecutive) are combine into one.

28 CMS framework enhancements ò Incremental improvements through 3_x an 4_x: Only one event tree, improve event rea orering,ttreecache became functional, caching non-event trees. ò Improve startup: While TTreeCache is being traine, we now rea all baskets for the first 10 events at once. So, startup is typically one large rea instea of many small ones. Improve startup latencies (simplifie) {Event Training CMSSW 3_X CMSSW 4_2 Branch Branch = iniviual rea = vectore rea 7 rea calls 21 squares rea Event 2 rea calls 27 squares rea We rea all branches for the first 20 events; compare to the entire file size, the banwith cost is minimal. On high latency links, this can cause a 5 minute reuction in time to rea the first 20 events.

29 CMS framework enhancements Upcoming Enhancements ò Real Asynchronous prefetch (using threas an oublebuffering). Brian s opinion: the current patch set is problematic an not usable by CMS. At least a year out. ò Aitional reuction in un-streaming time. Multi-threae un-streaming. ò ROOT implementation of things in CMSSW: multiple TTreeCaches per file. Improve startup latencies.

30 Outlook ò CMS computing moel: first assumptions an results ò Technology trens an impact on HEP Computing moels ò Xroot concepts an features ò CMS framework enhancements ò Xroot Reirector use cases an basic ieas ò Xroot Reirectors actual eployment ò Xroot first test an monitoring

31 Xroot Reirector use cases an basic ieas ò The goal of this activity is to ecrease the effort require for the en- user to access CMS ata. ò We have built a global prototype, base on Xroot, which allows a user to open (almost) any file in CMS, regarless of their location or the file s source. ò Goals: ò Reliability: Seamless failover, even between sites. ò Transparency: Open the same filename regarless of source site. ò Usability: Must be native to ROOT. ò Global: Cover as many CMS files as possible.

32 Xroot Reirector use cases an basic ieas ò The target of this project: ò En-users: event viewers, running on a few CMS files, sharing files with a group. ò T3s: Running meium-scale ntuple analysis ò None of these users are well-represente by CMS tools right now. ò So, a prototype is better than nothing

33 Xroot Reirector use cases an basic ieas Xroot Prototype ò Have a global reirector users can contact for all files. ò Can use any ROOT-base app to access the prototype infrastructure! Each file only has one possible URL ò Each participating site eploys at least 1 xroot server that acts like a proxy/oor to the external worl.

34 Xroot Reirector use cases an basic ieas

35 Xroot Illustration Reirector use cases an basic ieas Remote User 1. Attempt login 4. Login successful! 5. Open /store/foor 4. Open successful! 2. Request mapping for DN / VOMS 3. Result: User "brian" Xroot 6. Can brian open /store/foo? 7. Permit ò The authentication is always elegate to the site hosting the file LCMAPS Authfile (Site mapping policy)

36 Xroot Reirector use cases an basic ieas

37 Xroot Reirector use cases an basic ieas

38 Xroot Reirector use cases an basic ieas

39 Outlook ò CMS computing moel: first assumptions an results ò Technology trens an impact on HEP Computing moels ò Xroot concepts an features ò CMS framework enhancements ò Xroot Reirector use cases an basic ieas ò Xroot Reirectors actual eployment ò Xroot first test an monitoring

40 Xroot Reirectors actual eployment Site Integration ò We have yet to run into a storage system that cannot be integrate with Xroot. Techniques: ò POSIX:Works out-of-the-box. Example: Lustre. ò C API: Requires a new XrOss plugin. Example: HDFS, cap, REDDNet (?) ò Native implementation: Latest Cache has built-in Xroot support; combine with xroot.org cms to join feeration. ò Direct access: Let xroot rea the other system s files irectly from isk FNAL)

41 Xroot over HDFS Xroot reirector Local HDFS installation Native Xroot local oor ò In this case: we nee a native xroot oor reaing file using hfs stanar C library ò This is quite similar to the installation of a griftp oor

42 Xroot over xroot Xroot EU reirector Local Cache installation Dcache xroot oor Native Xroot local oor ò In this case: we nee a native xroot oor reaing file using xroot library ò This machine is configurate in proxy-moe ò This nees that the Cache instance alreay provie a Cache xroot enpoint

43 Xroot over posix Xroot EU reirector Local GPFS/ Lustre installation Native Xroot local oor ò In this case: we nee one or more native xroot oors that are clients of the local parallel file-system (GPFS/Lustre)

44 Xroot Reirectors actual eployment CMSSW Performance ò Note the use cases all involve wie-area access to CMS ata, whether interactively or through the batch system. ò WAN performance for the software is key to this project s success. ò Current monitoring shows 99% of the bytes CMS jobs requests are via vectore reas - many jobs have about 300 reas per file total. That s at least 30s of I/ O that a locally running job wouln t have. ò We plan to invest our time in I/O to make sure these numbers improve, or at least on t regress.

45 Xroot Reirectors actual eployment Xroot eployment ò US Xroot Reirector: ò Collecting ata from US Sites: ò Nebraska, Purue, Caltech, UCSD, FNAL ò EU Xroot Reirector: ò Collecting ata from EU Sites: ò INFN-Bari, INFN-Pisa, DESY, Imperial College (UK)

46 Xroot Reirectors actual eployment Xroot eployment ò Xroot provies per-file an per-connection monitoring. ò I.e., a UDP packet per login, open, close, isconnect, an summarizing every 5s of activity. ò This is forware to a central server, where it is summarize an forware to to several consumers. ò Also provies a perserver summary (connections, files open) that is fe irectly to MonaLisa.

47 Xroot Reirectors actual eployment Xroot istribute performance ò Using CMSSW 4_1_3 an a I/O boun analysis ò Job are running on T2_IT_LNL an ata are hoste in T2_IT_Bari ò ping time: 12ms ò CPU Efficiency rop from 68% to 50% ò ~30% of performance rop ò This is about to be one of the worst case

48 Xroot Reirectors actual eployment Xroot istribute performance

49 Conclusion ò Xroot reirector infrastructure is fulfilling few use cases that are not covere by official CMS tools ò It is alreay been use: ò from both final users ò Debugging coe, visualization etc. ò an computing center ò That o not have enough man power to fully support CMS locally ò Tuning CMSSW is very important in orer to provie goo performance when accessing ata remotely