Energy efficient real-time computing for extremely large telescopes with GPU

Transcription

1 Energy efficient real-time computing for extremely large telescopes with GPU Florian Ferreira & Damien Gratadour Observatoire de Paris & Université Paris Diderot 1 Project # funded by European Commission under program H2020-EU coordinated in H2020-FETHPC-2014

2 Observing stars from the ground Atmospheric turbulence Modify the trajectory of light rays when they cross the atmosphere Reduces astronomical images quality Similar to the effect of aberrations in an optical system Adaptive optics Compensate in real-time for the effect of optical aberrations on image quality Already in use on most 5-10m astronomical telescope to provide nominal image quality whatever the turbulence conditions 2

3 Adaptive optics Disturbed wavefront Compensate in real-time the wavefront perturbations Using a wavefront sensor to measure them Using a deformable mirror to reshape the wavefront Deformable mirror Real-time controller Beamsplitter Corrected wavefront Wavefront sensor High resolution camera Commands to the mirror must be computed in real-time (1ms rate) 3 Loop closed Loop open

4 Adaptive optics Example with observations of the moon using a 8m telescope Without AO With AO 4

5 European Extremely Large Telescope (E-ELT) 39m diameter telescope! Credit : ESO Primary mirror will be made of ~800 segments of 1.4m diameter Theoretical resolution in the near-infrared : 10 milliarcseconds, i.e. 2x105 smaller than the full moon (~30 arcminutes) About 1000 m2 of collecting area, i.e. 15 times more sensitive than the largest state of the art professional telescopes currently in operation 5

6 European Extremely Large Telescope 39m diameter telescope : 100m dome, 2800 tones structure 360, seismic safe (Chile) 1.2 G project, first light foreseen in 2024 European project led by ESO involving public labs and private companies AO system complexity scales as the square of telescope diameter Credits : ESO x25 more complex AO systems 6 Credits : ESO

7 Adaptive optics Disturbed wavefront Compensate in real-time the wavefront perturbations Using a wavefront sensor to measure them Using a deformable mirror to reshape the wavefront Deformable mirror Real-time controller Beamsplitter Corrected wavefront Wavefront sensor High resolution camera Commands to the mirror must be computed in real-time (1ms rate) 7 Loop closed Loop open

8 AO real-time controller Highly heterogeneous HPC facility 8

9 The Green Flash project Large European initiative Goal : prototype a generic RTC for the next generation of AO on extremely large telescopes 4 partners in Europe (2 academic partners + 2 SMEs), project lead : Observatoire de Paris, 3.8 M investment funded under the H2020 program (FET-HPC, project #671662) Assess various technologies (CPUs, GPUs, FPGAs) and find the best trade-off Assemble a full featured prototype in the lab by

10 GPUs for real-time computing Naive implementation : several copies and multiple kernel launches 10

11 Low latency data transfer to GPUs Critical issue: high latency data acquisition from the camera using an off-the-shelf frame grabber Typical approach : Double-copy mechanism Multiple kernel launches DDR Mem. DDR Mem. GPU FPGA DMA engine PCIe bus CPU High jitter in performance not compatible with time deterministic constraint 11 DDR Mem. 10 Gbe Frame-grabber Serial interface Pixel data

12 Low latency data transfer to GPUs Solution : using GPUdirect coupled to a persistent kernel strategy DDR Mem. DDR Mem. GPU FPGA DMA engine PCIe bus No more jitter in the execution CPU Efficient data distribution 12 DDR Mem. 10 Gbe Frame-grabber Serial interface Pixel data

13 Low latency data transfer to GPUs Enabled through a smart interconnect strategy Based on FPGA boards Using dedicated dev. tools (a.k.a. QuickPlay from Accelize) FPGA design made easy 13

14 GPUs for real-time computing Our solution : using GPUdirect coupled to persistent kernels 14

15 Full scale prototyping Relies on NVIDIA DGX-1 Being assessed in the lab today. First performance estimates are consistent with system specifications NVIDIA DGX-1 WFS pixels (4x10GbE) WFS pixels (4x10GbE) WFS pixels (4x10GbE) Fast intra-cluster com. (40 GbE) X86 X86 NIC GPU GPU NIC WFS pixels (4x10GbE) NIC GPU GPU NIC NIC GPU GPU NIC NIC GPU GPU NIC WFS pixels (4x10GbE) WFS pixels (4x10GbE) Fast intra-cluster com. (40 GbE) NVlink PCIe 15

16 Full scale prototyping Relies on NVIDIA DGX-1 Master / slaves strategy implemented on DGX-1 Very low jitter introduced by sync / comm. 16

17 Tomographic adaptive optics Small patches of interest in a large field of view 17

18 Tomographic adaptive optics Multiple guide stars using Lasers and multiple correctors 18

19 Tomographic Adaptive Optics New concept of Multi-Object AO Multiplexed-AO (several WFS, several DM) Credits : E. Marchetti, ESO 19 Credits : E. Marchetti, ESO

20 AO loop supervision High throughput HPC facility Optimize loop performance by regular control matrix update Control matrix is built through statistical analysis of real-time data 20

21 AO loop supervision AO loop supervisor pipeline : computing a tomographic reconstructor (ToR) for AO Ongoing collaboration with KAUST 21

22 AO loop supervision Computing the tomographic reconstructor on DGX-1 25 sec to compute the scale is well within specs! and 8 P100 GPUs perform almost 20x better than a single KNL P100 is more than 2x more efficient than KNL including Nvlink communications. 22

23 AO simulation ToR is also at the core of our AO simulation pipeline (based on pure linear algebra to simulate system behavior) 23

24 AO simulation Producing AO performance map for a given patch of the sky and given turbulence conditions 24

25 AO simulation Portable SW stack, studying performance scaling over several generations of HW : GPUs always win Regular performance increase at each new HW release Credits : Hatem Ltaief 25

26 Conclusions and future work GPUs help to design the largest telescopes critical systems Provide the required throughput for efficient simulations of large scale AO systems GPUs proposed to be at the core of telescope operations : As compute units in the real-time controller, coupled to a smart interconnect strategy with sensors Assembled in a standard cluster for regular tomographic matrix update What's next? Large scale simulations for system design trade-off and sky coverage studies Full scale prototyping of GPU-based real-time controller Study new supervision strategy with optimized linear algebra (H-matrix formalism) 26

27 That s it for today Thank you! Credits : ESO 27