Linac Coherent Light Source (LCLS) Data Transfer Requirements

Transcription

1 Linac Coherent Light Source (LCLS) Data Transfer Requirements Dr. Les Cottrell, SLAC HPC talk Stanford Feb 2018

2 LCLS-II, a major (~ B$) upgrade to LCLS is currently underway. Online in Video 2

3 Basic instrument layout: Optical laser pump, and x-ray laser probe Basement Level Optical Laser X-Rays X-ray hutch Sub-Basement Level 3

4 Example experiment #1 Molecular Movie Captures Ultrafast Chemistry in Motion Scientific Achievement Time-resolved observation of an evolving chemical reaction triggered by light. Method LCLS X-ray pulses were delivered at different time intervals, measuring the structural changes on an X-ray area detector. Significance and Impact Results pave the way for a wide range of X-ray studies examining gas phase chemistry and the structural dynamics associated with the chemical reactions they undergo. M.P. Minitti, J.M. Budarz, et al., Phys. Rev. Lett., 114, (2015) (COVER ARTICLE) 4

5 Next example: Catalysis H2O, CO2 Catalyst Porous substrate coated with precious metals Exhaust gases react with precious metals HC, CO, NOX Polluting gases Another related example 50% of the world s gasoline goes through fluid catalytic cracking 5

6 Example experiment #2: Catalytic converter transient dynamics resolved at the atomic scale Surface catalysis of CO oxidation to CO 2 Sub-picosecond transient states, monitored via appearance of new electronic states in the O K-edge x-ray absorption spectrum. H. Öström et al., Science

7 Data Analytics for high repetition rate Free Electron Lasers FEL data challenge: Ultrafast X-ray pulses from LCLS are used like flashes from a high-speed strobe light, producing stop-action movies of atoms and molecules Both data processing and scientific interpretation demand intensive computational analysis LCLS-II will increase data throughput by orders of magnitude by 2026, creating an exceptional scientific computing challenge LCLS-II represents SLAC s largest data challenge 7

8 LCLS-II experiments will present challenging computing requirements, in addition to the capacity increase 1. Fast feedback is essential (seconds / minute timescale) to reduce the time to complete the experiment, improve data quality, and increase the success rate 2. 24/7 availability 3. Short burst jobs, needing very short startup time Very disruptive for computers that typically host simulations that run for days 4. Storage represents significant fraction of the overall system, both in cost and complexity: 1. 1 Pbyte/day fast local buffer, 5-10 Pbytes storage for local processing, 10 yr long term offline tape storage Teraflops in 2020, grow to 1 Pflop in Throughput between storage and processing is critical Currently most LCLS jobs are I/O limited 6. Speed and flexibility of the development cycle is critical Wide variety of experiments, with rapid turnaround, and the need to tune data analysis during experiments These aspects are instrumental also for other SLAC facilities (eg CryoEM, UED, SSRL, FACET-2) 8

9 LCLS-II data flow: from data production, to online reduction, real-time analysis, and offline interpretation Megapixel detector X-ray diffraction image Intensity map from multiple pulses Interpretation of system structure / dynamics Currently seeking user input on acceptable solutions for data reduction & analysis 9

10 Critical Requirement for Offsite Resources for LCLS Computing Several experiments require access to leading edge computers for detailed data analysis. This has its own challenges, in particular: Need to transfer huge amounts of compressed data from SLAC to supercomputers at other sites Providing near real-time results/feedback to the experiment Has to be reliable, production quality MIRA at Argonne TITAN at Oak Ridge CORI at NERSC The requirements will need long term agreement between BES and ASCR 10

11 Requirement Today 20 Gbps from SLAC to NERSC =>70Gbps 2019 Experiments increase efficiency & networking 2020 LCLS-II online, data rate 120Hz=>1MHz LCLS-II starts taking data at increased data rate Tbps: Imaging detectors get faster Moore s law 11

12 Offsite Data Transfer: Needs and Plans ESnet6 upgrade SLAC plans NERSC plans LCLS-I LCLS-II 12 LCLS-II needs are compatible with SLAC and NERSC plans

13 Data Transfer testing Today using existing equipment DTNs at NERSC & SLAC With Lustre file systems at ends Today 100Gbps link Using widely used bbcp and xrootd tools Currently ~55Gbps, exploring limitations Upgrading border to 2*100Gbps in progress Zettar/zx: Provide HPC data transfer solution (i.e. SW + transfer system reference design): - state of the art, efficient, scalable high speed data transfer Over carefully selected demonstration hardware 13

14 Data transfer performance: Test bed: two clusters at SLAC with 5000 mile link <80Gbps Space efficient: 6U per cluster Energy efficient, 14

15 NG demonstration 2x100G LAG Other cluster or High speed Internet IP over InfiniBand IPoIB) Storage servers > 25TBytes in 8 SSDs n(2)*100gbe 100Gbps 100Gbps 4* 2 * 25GbE Data Transfer Nodes (DTNs) 4* 56 Gbps

16 Memory to Memory between clusters with 2*100Gbps No storage involved just DTN to DTN mem-to-mem Extended locally to 200Gbps Here repeated 3 times Note uniformity of 8* 25Gbps interfaces. Can simply use TCP, no need for exotic proprietary protocols Network is not a problem 16

17 Storage On the other hand, file-to-file transfers are at the mercy of the back-end storage performance. Even with generous compute power and network bandwidth available, the best designed and implemented data transfer software cannot create any magic with a slow storage backend 17

18 XFS READ performance of 8*SSDs in a file server measured by Unix fio utility 800% SSD busy 3500 Queue Size Read Throughput 20GBps 600% GBps 400% 200% GBps 15:16 15:18 15:16 15:18 Data size = 5*200GiB files similar to typical LCLS large file sizes 5GBps 0GBps 15:16 15:18 Note reading SSD busy, uniformity, plenty of objects in queue yields close to raw throughput available 18

19 XFS + parallel file system WRITE performance for 16 SSDs in 2 file servers SSD busy SSD write throughput 10GBps Queue size of pending writes 50 Write much slower than read File system layers can t keep queue full (factor 1000 less items queued than for reads) 19

20 Conclusion Network is fine, can drive 200Gbps, no need for proprietary protocols Insufficient IOPS for write < 50% of raw capability - Today limited to 80-90Gbps file transfer Work with local vendors State of art components fail, need fast replacements Worst case waited 2 months for parts Use fastest SSDs for write We used Intel DC P3700 NVMe 1.6TB drives Biggest also fastest but also most expensive 1.6TB $1677 vs 2.0TB $2655 ; 20% improvement 60% cost increase Parallel file system is bottleneck Needs enhancing for modern hardware & OS 20

21 Demonstration PetaByte in < 1.5days at ~70Gbps on <80Gbps shared link 1/3rd of ESnet's traffic 21

22 Impact on all ESnet traffic When running data transfer contributes ~ 1/3 of total ESnet traffic OSCARS LHCONE Other 200G 100G 12pm 3pm 6pm 9pm Mon 10 3am 6am 9am 22

23 Summary What is special: Scalable. Add more NICs, more DTNs, more storage servers, links as needed/available Power, & space efficient; low cost HA tolerant to loss of components Storage tiering friendly Reference designs Easy to use, production-ready software Proposed Future PetaByte Club A member of the Petabyte Club MUST be an organization that is capable of using a shared production point-to-point WAN link to attain a production data transfer rate >= 150PiB-mile/hour 23

24 Future Upgrade 200Gbps border at SLAC to 2x100Gbps to Esnet Spring 2018 Then onto SLAC to NERSC Very special environment, hard to modify Focus has been different to what we need: vector computing and communication-intensive problems at expense of faster cpus - but LCLS embarassingly parallel Not easy to use for cluster computing such as xrootd or zx Unless Cray supports porting application - unlikely Using the Burst Buffer (BB) is not easy from an application BB Will probably limit data transfers performance Normally used from batch so challenge for near-realtime Can bypass the NERSC DTNs and go directly to the Cori nodes which have direct access to the BB. However DTNs provided common ways to customize, more agile Harder to do for Cori nodes Complex supercomputer, components tend to be behind very latest technology curve Looking to exceed PB/day by SC

25 Questions