Computing and Networking at Diamond Light Source. Mark Heron Head of Control Systems

Transcription

1 Computing and Networking at Diamond Light Source Mark Heron Head of Control Systems

2 Harwell Science and Innovation Campus ISIS (Spallation Neutron Source) Central Laser Facility LHC Tier 1 computing Research Complex (for users of Diamond, ISIS and CLF) Diamond Light Source

3 Diamond Light Source

4 Science SR Examples Pharmaceutical manufacture & processing Casting aluminium Non-destructive imaging of fossils Structure of the Histamine H1 receptor

5 Beamlines or Instruments Operational period 25/2/2016 An Introduction to the Diamond Software Developers Away Day 2016

6 Data Size in PB Cumulative Cumulative Amount Amount of of Data Data Generated Generated By By Diamond Diamond Jan-07 Jan-08 Jan-09 Jan-10 Jan-11 Jan-12 Jan-13 Jan-14 Jan-15 Jan-16

7 10000 Data Rates Detector Performance (MB/s) No detector faster than ~10 MB/sec 2009 Pilatus 6M system 60 MB/s Hz Pilatus 6M 150 MB/s Hz Pilatus 6M 600 MB/sec 2013 ~10 beamlines with 10 GbE detectors (mainly Pilatus and PCO Edge) 2016 Percival detector 6GB/sec

8 Data Rates

9

10 Electron Microscope Life Science EMs 2x Titan Krios Electron Microscopes Gatan Quantum Detector 600MB/s 2x Physical Science EMs to come 2x further Life Science EMs to come

11 Example of an EM Processing 17,500 particles 10 hours data collection Average resolution 3.2 Å processed with Relion Dan Clare, Sonja Welsch

12 Typical User Setup

13 GDA User Interface Rich GUI clients widgets, views, or perspectives using Eclipse plugin framework Terminal Live Plotting Script Editor Analysis & Visualisation Log View

14 Data Flow

15

16

17 High throughput MX at DLS 13:11:18 < 2m20s 13:09:01

18

19 First Line Storage ~5PB Online storage 430TB, 12 Gbyte/sec, Lustre file system 880TB, 16 Gbyte/sec, GPFS file system 3.6PB, 30 Gbyte/sec, GPFS file system, uses GPFS clusters Commissioned start of TB, 1 Gbyte/sec general purpose NAS file system for slower beamlines Lustre Vs GPFS Lustre good for high aggregate cluster processing rates GPFS better for single point data rates GPFS we had problems of interactions between clients, addressed through clustering

20 Compute Cluster Structured as 6 clusters, with a range of capabilities 135 Nodes providing 1824 x86 CPU cores Between 32GB and 256GB RAM (2GB/core to 12GB/core) depending on age 118 GPU nodes Mostly used by MX and Tomography beamlines All accessed via one Univa Grid Engine interface Multiple queues

21 Network Core network All high data rate beamlines are connected to two core Switches, with 2x10 GbE links or 2x40GbE. Clusters and Storage connected to two core Switches, with 2, 4 or 8 x 10 GbE links. Cluster and Storage connected with InfiniBand Beamlines Most detectors and workstations connected with 1 GbE links High performance systems connected with 10 GbE or 40GbE Becoming the norm

22 Network Bandwidth Balance 10 Gbit/s 1 Gbit/s 10 Gbit/s 40 Gbit/s Beamline Switch Beamline Switch 10 Gbit/s 40 Gbit/s 400 Gbit/s Central Switch 400 Gbit/s 40x10 Gbit/s Cluster Switch Disks 80 Gbit/s GPFS 40 Gbit/s Lustre Cluster IB 10x56Gbit/s

23 High Density Computer Room 22 racks, each rack 20 kw peak cooling 2 x 20 kw peak power 320 kw redundant power From two separate sub-stations Power from on side is UPS and generator backed up. 320 kw cooling water. Primary cooling is from site chilled water 220 kw standby chiller in case of problems. Its full Options for next phase

24 cluster 40x20 core cluster clusters 3PB parallel file system

25 Realtime Data Storage and Analysis

26 Data Archive STFC tape storage Tape Libraries: 2 x SD ,000 slot robots 64 tape drives Potential capacity 100PB Data management and archive systems: Storage-D Storage Resource Broker I-RODS Storage Resource Manager ICAT Top CAT

27 Moving Data Off Site a Science DMZ

28 More of same Future Detectors, automation, processing pipelines.. Develop post visit processing services Making visit archives and application environment available to other systems SCARF,.. Working with STFC SCD on UKT0 and Ada Lovelace Centre

29 Thank you

30

31 3600 image data set - images to density 2 minutes

32 I02/VMXi - Pilatus2 25 Hz I03 - Pilatus3 100 Hz I04 - Pilatus2 25 Hz I Pilatus2 25 Hz I Pilatus2 30 Hz I23 - Pilatus2 12 Hz I24 - Pilatus3 100 Hz VMXm - TBC Aggregate: > 300 frames/s > 1800 MB/s sustained rate - sample exchange < 20s Worth noting that Eiger 16M at 133 frames/s will actually reduce file system load compared with 100 frame/s Pilatus 6M

33 Distributed Control Systems

34 Realtime Feedback & Feedforward TMFB Controller Dispersion FB Controller Orbit FB Controller Coupling FB Controller Tune FB Controller Emit. Emit Y pbpm Pos X,Y Pos X,Y Tunes X, Y Gap Tune FF Controller Optics FF Controller Orbit FF Controller Master Osc RF Amp ebpm TMBF ID Camera pbpms 172 ebpms Buttons Insertion Device Corrs local to ID Quads local to ID 248 Quads 96 Skew Quad 172 Corrs on Sext RF Cavity Strip lines

35 EPICS and GDA both need to write the data file. We use HDF5 links to create one logical file from multiple real files. Avoids file contention issues. Allows detector files to be highly optimised for performance. The header data is written directly by GDA. The detector data is written using EPICS HDF5 area detector plugin. Data Formats GDA Header Data.nxs file EPICS Detector Data.hdf5 file Detector Data.hdf5 file

36 Example: Tomography Tomography scans are demanding: Data rate ~ 500 MB/s. Data size > 100 GB. First operation is read data perpendicular to write direction. Classic matrix transpose problem Real challenge for typical cache design. Completely unsuited to running inside the GDA server. Image Frames Sinogram Frames

37 Tomography data file format Data must be optimised for reading sinograms. All frames are written to a single file. File is arranged in chunks of a fixed number of rows and a fixed number of frames. The chunk size matches the Lustre stripe size so is written to a different Lustre server. Data is in cache until all frames in a chunk are written.

38 New Detector Parallel Control & DAQ Large scale/high speed detector developments Percival 300Hz, 120Hz (6GB/s) Excalibur to use the same framework for scalability/swmr/vds TimePix3 Large Area Time Resolved Detector

39 Network Layout