Performance of Monte-Carlo simulation of adaptive optics systems of the EAGLE multi-ifu Instrument for E-ELT

Transcription

1 Performance of Monte-Carlo simulation of adaptive optics systems of the EAGLE multi-ifu Instrument for E-ELT Alastair G. Basden a, Timothy Butterley a, Mark A. Harrison a, Timothy J. Morris a, Richard M. Myers a, Richard W. Wilson a, Eddy Younger a, Thierry Fusco b, Brice Le Roux c and Benoit Neichel d. a Durham University, Department of Physics, South Road, Durham, DH1 3LE, UK; b ONERA (France) c Observatoire Astronomique de Marseille-Provence (France) d Observatoire de Paris Meudon (France) ABSTRACT A method that has been followed to produce performance estimates for the adaptive optics (AO) aspect of the EAGLE instrument proposed for the European Extremely Large Telescope (ELT) (E-ELT) using Durham Monte- Carlo simulation code is presented. These simulations encompass a wide range of possible configurations for EAGLE, including multi-object adaptive optics (MOAO), segmented multi-conjugate adaptive optics (MCAO) and other more novel techniques. Particular emphasis is placed on the techniques used to enable a good simulation turn-around rate, allowing the large parameter space associated with optimising high performance AO systems to be explored. Performance estimate results for some AO system configurations are also provided. This paper is available directly from the author, a.g.basden@durham.ac.uk. 1. INTRODUCTION The software simulation of adaptive optics systems on Extremely Large Telescopes (ELTs) is an essential part of developing a working system that is able to meet its design requirements. Such simulations fall broadly into two categories; analytical and Monte-Carlo. Analytical simulations are able to provide rapid results, though do not include fine details of the system, many of which are essential to include for ELT scale designs. Monte-Carlo codes are typically much more computationally intensive, though can include as much detail as is required, and are therefore usually considered to be more accurate. Most Monte-Carlo codes are not usable for ELT scale systems, as the time taken to simulate a system and the memory requirements increase dramatically with telescope size. However, the Durham adaptive optics (AO) simulation platform (DASP) has been developed specifically with ELT simulation in mind, and this paper discusses some of the techniques used to realise this. 2 provides an overview of the DASP architecture, 3 gives details of some of the optimised algorithms used for ELT simulation, 4 discusses the tools available as part of DASP, and 5 provides some results from EAGLE simulations. 2. THE DURHAM AO SIMULATION PLATFORM The DASP has been designed from the ground up for high performance. One of the significant difficulties of Monte-Carlo AO simulation is the increasing computational power required for simulation of larger telescopes. The DASP uses a modular approach, representing all parts of an AO system with modules. These modules are connected together to form a simulation with necessary data passing between them. The DASP has been designed around the parallel Message Passing Interface (MPI). This allows it to operate across multiple processors, which can be remote from each other. Additionally, DASP can make use of shared memory facilities, giving separate processors the ability to share data with minimal latency and contention when on a shared memory platform. With the advent of multi-core processors and symmetric multi-processing (SMP) systems, shared memory facilities are now very commen. The DASP has also been designed with the Further author information: (Send correspondence to A. Basden) A. Basden: a.g.basden@durham.ac.uk, Telephone:

2 ability for each constituent process to be multi-threaded where this is appropriate. Multi-threading is not always appropriate. On systems with only one processing core per central processing unit (CPU), there is no benefit. Additionally, not all algorithms are well suited for parallelisation. The nature of AO simulation means that many algorithms are performed on relatively small data-sets, and often, the overhead involved with using multiple threads for these algorithms is greater than the performance benefits achieved by having more than one thread. The DASP also has the capability to use hardware acceleration facilities where these are available. Parts of the simulation code have been ported into field programmable gate array (FPGA), which has been shown to give performance improvements of over 600 times. 1 The use of Cell processors 2 has also been integrated into the DASP, which gives performance improvements by making use of the symmetric processing units in these processors, typically running just under one order of magnitude faster than conventional code. Code for multicore processors has also been integrated into DASP. We are also currently investigating the use of graphical processing units using the NVIDIA CUDA architecture, which will allow a very high degree of parallelisation for implemented algorithms. Initial work in this area involves the wavefront reconstruction algorithms. 2.1 Data flow The flow of regular data through the DASP has been streamlined such that data transfer is minimised and uses the most appropriate medium available. Typical data flowing through a simulation includes atmospheric and corrected wavefronts, deformable mirror commands and wavefront sensor measurements. This data is passed between simulation objects, which then perform the necessary calculations. Within a given simulation, data can be passed between processes, or within a process. The processes between which data is passed may be located on the same (multi-core) CPU, on different CPUs within a shared memory architecture (SMP), or between different CPUs of separated memory platforms (I/O connected only). Passing data within a process simply involves passing the memory pointer to the data. Passing data between processors within a shared memory architecture again involves passing a pointer to the shared memory array. Passing data between processes on discrete platforms involves the MPI library, and a physical transfer of the data (meaning that both processes require separate memory to store the data). This last passing of data consumes a significant amount of I/O bandwidth, and so simulations are designed to minimise this. When there is a need to pass data between science objects infrequently (or once, for example to pass a newly created wavefront sensor noise covariance matrix to a maximum a-posteriori (MAP) reconstructor object), this is done using the TCP/IP control interface even for objects in the same process, as performance is not an issue. Having this ability (which is transparent to the user across multiple processes) is very useful when performing AO simulations, and ensures the same effect, regardless of simulation topology. The DASP is written using a Python framework which is used as a wrapper for compiled C and Fortran libraries. This approach allows flexibility and allows rapid prototyping of new concepts and algorithms. Additionally, the Python interpreter can be used to run user code while a simulation is running, either to change internal parameters or simulation behaviour or to display and analyse results. Standard high performance libraries are used, such as BLAS, LAPACK, ATLAS, FFTW, ACML etc. 3. ALGORITHM OPTIMISATION The DASP has been designed with ELT scale simulations in mind. Specific algorithms that have been optimised for this purpose are discussed here. 3.1 Tomographic wavefront reconstruction The traditional way to perform tomographic wavefront reconstruction is to compute the inverse of an interaction matrix, which measures the wavefront sensor response to changes in the deformable mirror (or mirrors) shape. This approach works well for small systems, but scales very poorly to ELT scale. A typical simulation of the EAGLE instrument for an ELT may contain as many as Shack-Hartmann wavefront sensor measurements (for example 9 laser guide star (LGS) wavefront sensor (WFS) with sub-apertures each). Similarly, the wavefront may be reconstructed for at least 4 atmospheric layers, up to a height of 15 km, over a 5 arc-minute field

3 of view. This could require the reconstruction of tens of thousands (say 40000) of phase points. A reconstruction matrix of this size would take about 32 GB storage, which is at the limit of current top of the range work stations. Additionally, it would take months of CPU time to compute this matrix from the inverse of the interaction matrix. The DASP currently has two reconstructors of this scale. The first uses a highly customised singular value decomposition (SVD) algorithm, applied to a sparse interaction matrix, used to compute a sparse reconstruction matrix. The sparsity of the (zonal) interaction matrix is typically at the one percent level, i.e. a few hundred MB. The SVD algorithm computes the full reconstruction matrix (stored in a memory mapped file), and has been optimised for paged memory access. This matrix is then sparsified to a user defined fraction, again typically at the one percent level, with a trade-off between storage space, execution time and reconstruction fidelity. The current time to perform this computation for a typical EAGLE simulation (44000 centroids and 9000 phase values) is about 4 hours, compared with nearly a week for an unoptimised version. Once the eigenvalue decomposition part of this algorithm has been computed, intermediate results are stored so that different conditioning parameters can be used to compute the reconstructor. Each such computation takes of order 30 minutes. The second reconstructor that we have for tomographic wavefront reconstruction is a MAP reconstructor. This uses prior information about the atmospheric phase and wavefront sensor noise sources. The sparsity of the reconstruction matrix is less than in the SVD case (and sometimes non-sparse), but similar techniques may still apply. Additionally, we use the matrix inversion lemma when computing the reconstructor matrix from the interaction matrix and covariance matrices, such that the matrix inversion calculation is performed on a significantly smaller matrix than would otherwise be the case. This inversion is still time consuming, however we are investigating the use of graphical processing unit (GPU) processors to perform this and preliminary results show that the time scale for a typical inverse is acceptable, of order 1 day. We also have a Fourier domain pre-conditioned gradient (FDPCG) algorithm 3 which does not need to inverse the interaction matrix. However, this algorithm is less well understood, particularly with regards to LGS effects. Development using GPU processors is also underway. 3.2 Shack-Hartmann sensor simulation Our Shack-Hartmann wavefront sensor algorithm takes atmospheric phase (partially corrected for closed loop simulations), computes the wavefront sensor images (including noise, such as photon-shot, readout noise and background noise), and applies a centroiding algorithm to measure the wavefront tilts in each sub-aperture. This algorithm has been implemented on a variety of computing platforms, including FPGA, 1 Cell processor and multi-core processors and, on each of these platforms, runs far faster than the conventional linear software approach allows. 3.3 Science PSF generation Our science image point spread function (PSF) generation algorithm takes atmospheric phase (usually partially corrected) and computes a high light level image, and the associated AO system performance estimates when this PSF is integrated over a long period of time. This algorithm, which among other things contains a large 2D fast Fourier transform (FFT) has, again, been implemented on FPGA, Cell and multi-core platforms, each showing significant performance improvements over a conventional linear software approach. 3.4 Atmospheric phase generation Atmospheric phase screen generation is another algorithm that can take a significant amount of processing power, particularly for ELT scale simulations, as a very large 2D FFT is usually performed (to create a large phase screen that can be translated across the telescope pupil). Our approach is to use an infinite phase screen generation method, 4 which significantly reduces the computation times to a level where this is negligible compared with other algorithms. There is therefore little point in implementing this algorithm in non-standard processing technologies, since the overall performance improvement of the simulation as a whole would be tiny (Amdahl s law 5 ).

4 Figure 1. A screenshot of the simulation builder tool, showing the layout of a simulation. The parallel optical paths are clearly shown. The box around one of the optical paths represents grouping this optical path is repeated (6 times in this case) when the simulation is built. Resource sharing is shown by the horizontal lines connecting objects. Data flow is represented by the lines connecting the bottom of one object to the top of another. Parallelisation in this example is shown as different colour surrounds to the objects (red and green, or different shades of gray-scale). The tool bar shows the various different options that can be used when creating a simulation. 3.5 DM models One of the key components for a simulation is the model of the deformable mirror (DM) surface that is to be used. For zonal simulations, the principal model that we use is a spline interpolated surface model. This provides a good model of the surface of continuous phase-sheet DMs, and we have compared this with FISBA interferometer maps of a Xinetics 97 actuator DM at Durham (Morris et al., in progress). This fitting error is lower than for a Gaussian actuator model, and can be computed in a significantly shorter amount of CPU time. 4. SIMULATION CREATION A large amount of time can usually be spent setting up a new simulation when multiple guide stars, deformable mirrors and science directions are involved. This is made particularly difficult when the simulation is to be spread across multiple computers, as the passing of data between them needs to be considered. The DASP includes a graphical simulation builder (Fig. 1) which significantly simplifies this task. Using this tool, simulation objects are simply placed and connected together to design the simulation. This also helps with visualisation of the simulation, since it is clear which objects are connected. The user can set which objects run on which computer or CPU, and any communication between them is automatically managed. Objects placed on a single CPU are run in a set order (defined by the builder), to avoid reduced performance due to thread context swapping, and to simplify debugging. This builder has been developed to simplify the task of creating complex ELT scale simulations. In a complicated simulation, there will be many optical paths, a number of which will be very similar, for example

5 Figure 2. A figure showing the benefits of grouping. On the left is a complex simulation put together without grouping. This contains 12 WFS (e.g. 3 NGS, 9 LGS). On the right is an identical simulation created using the grouping facility. each may contain a telescope pupil module (computing the atmospheric phase along a given line of sight), a deformable mirror object and a wavefront sensor object. A typical ELT simulation may contain six LGS, which would require this sequence to be repeated six times. However, by specifying this sequence once, and then defining it as a group, the builder will replicate the sequence as necessary. This significantly reduces the complexity of setting up a simulation, and Fig. 2 shows a comparison between a simulation designed using grouping, and one without. The Python code produced in each case is almost identical. 4.1 Resource sharing An additional feature of DASP is the ability to share resources between different simulation objects. This can significantly reduce the memory required by a simulation, as well as reducing the computation time. A typical simulation object will require temporary scratch memory which is used to compute intermediate results, before the final output for this object is known. However, this memory is only required while the object algorithm is being computed, and so can be reused by other objects on the same processor without affecting results, since there is only a single execution thread. Additionally, this scratch memory may be used to store intermediate results which can then be reused by other similar simulation objects. For example, to simulate the behaviour of a deformable mirror, multiple simulation mirror objects are required, one for each source direction to be viewed. But, by sharing resources, the first such object can compute the shape of the mirror surface, and the other objects need only to select the appropriate part of this surface for their source direction (if the mirror is not ground conjugate), and apply a wavelength scaling. This can significantly reduce the computation time required. The graphical simulation creation tool allows users to implement resource sharing with minimal effort, simply by connecting similar objects together. 4.2 Defining simulation parameters There are many parameters used to define a simulation. These include wavelengths (usually different for science and wavefront sensors), deformable mirror types (segmented, continuous etc.) and surface fit algorithms, wave-

6 Figure 3. A diagram of the simulation control tool. front sensor sub-aperture count, height of atmospheric layers, source locations, phase screen resolution, detector pixel scale, reconstructor algorithms and many other things. The DASP allows a parameter file to be created either using a graphical tool or a text editor, and both an XML and Python format can be used. The parameters required for a given simulation (dependent on the simulation objects used) can be obtained using the graphical tool, and the user can then set these to appropriate values. Most parameters used in a simulation are given sensible defaults if not defined by the user, with a warning message being printed if these are used in case it was not what the user intended. A batch of parameters can also be specified, and the appropriate version is then selected when the simulation is run with a command line argument to specify the batch number. This is useful for exploring the simulation parameter space, and allows one parameter file to cover this space. A typical example may be ten different values for wavefront sensor CCD readout noise, allowing the user to investigate the effect that this has on AO system performance. 4.3 Simulation control A simulation is run from the command line using an aosim command. A number of command line arguments are available, e.g. to select the number of iterations to run for (overriding the parameter file), to change the batch number, to alter individual parameters without having to edit the parameter file (useful for testing) and to start the simulation in a paused or running state. Once a simulation has been started, a user (or several) can connect to it using a graphical simulation control tool (Fig. 3. This tool then gives the user access to all parts of the simulation. Control is then split into a normal mode and an expert mode. In both modes, the simulation is queried by the tool, which then displays a list of the recommended display plots available within the simulation, as well as the recommended controls for that simulation. Typical examples of information for plotting includes wavefront sensor images, DM surface maps, atmospheric phase and science images (long and short exposure). Other information can be available in text form, such as AO performance estimates, or simulation object processor timings. Recommended controls for the simulation can include things like opening or closing the loop, zeroing the science images etc. The expert control mode also lets the user do almost anything else they wish with the running simulation, and is typically used during debugging. This may include for example altering CCD readout noise, updating a reconstruction matrix, plotting the X centroid of one sub-aperture as a function of time, or anything else. The ability to have such flexible control is achieved by allowing the user to send arbitrary Python text strings, which are then executed by the simulation, sending any requested result back to the control tool.

7 Scheduled commands (control, plot etc.) can be regular (every N iterations, N 1) or once-only. Having such a powerful control tool makes the task of debugging a simulation much easier than without, since any parameter can be queried or displayed without stopping the simulation, altering code, and re-running it. This control is extremely important for ELT scale simulations. 4.4 Partial simulation execution The DASP has the ability to save to disk the output of any science object, for an arbitrary length of time. This data can then be used as the input for other science objects. This facility is extremely useful for two-stage openloop AO simulations, where open-loop wavefront sensor measurements, and atmospheric turbulence can be saved during the first stage. The second stage then involves using these measurements to shape the deformable mirror and compute performance verification parameters along multiple lines of sight. This provides the opportunity to investigate reconstruction algorithms without the computational expense of re-running the wavefront sensing stage, and so improves simulation turn-around time. 5. EAGLE SIMULATIONS The EAGLE instrument is a multi-integral field unit (IFU) instrument for the European ELT. It will work with between 6-9 LGS (currently undetermined) and with between pick-off mirrors, each supplying light to a science camera via an open-loop DM. Additionally, there will be a closed loop DM (the European ELT (E-ELT) M4 mirror) which all of the wavefront sensors will see. This is therefore a mixed open and closed loop multiobject adaptive optics (MOAO) system. Initial simulations (presented here) used wavefront sensors with sub-apertures, though we are now implementing simulations that use sub-apertures per WFS. A virtual DM approach is used to shape the MOAO DMs, with wavefront phase at multiple heights being reconstructed, and then compressed along the line of sight of each DM, providing the actuator control signal that is to be sent to this DM. This means that the reconstruction only has to be performed once (per iteration) regardless of the number of MOAO pick-offs. Fig. 4 shows the setup for this simulation. The grouping ability of the simulation creation tool has been used, so this figure is significantly simpler than it would otherwise be. 5.1 Segmented MCAO designs A segmented multi-conjugate adaptive optics (MCAO) system has been proposed as an alternative to the MOAO design. The segmented design splits the field of view into segments (typically 6), each of which is assigned an MCAO system to perform wavefront correction (with a shared M4 DM). Fig. 5 shows such a design for simulation. In this figure, all optical paths encounter the M4 DM. There are then a number of DMs used for segmented MCAO. In the left hand grouping (which is of size N equal to the number of segments), a LGS for each segment is used for wavefront sensing, and the output from the N WFS is sent to the reconstructor. This performs a tomographic reconstruction, one layer of which is used to directly shape the M4 DM. The segment DMs are controlled by compressing the reconstructed phase along the principal line of sight for each segment. The right hand grouping then evaluates the AO performance at a number of positions in one segment. This concept has now been disregarded, as it is deemed more complex than an MOAO system to implement, and does not fully remove the open-loop DM control requirement. This being the case, simulation results are not provided here. 5.2 Simulation turnaround and results A typical EAGLE simulation with virtual DMs at ground level and 8km, 9 LGS and 10 science targets takes about 12 CPU hours to complete. This includes the time to generate the interaction matrix, and to apply the SVD algorithm to compute the reconstruction matrix (about 4 hours for this). The simulation can be split efficiently between about 9 CPUs (putting one LGS path on each), which could reduce this time to 6 hours (the SVD algorithm involves iteration, and cannot be parallelised). Alternatively, multiple simulations can be run with a single processor each, meaning the parameter space can be covered much more quickly, depending on the number of available processors.

8 Figure 4. A diagram showing the set-up of a Monte-Carlo simulation of the EAGLE instrument. Here the WFS optical paths are repeated for each WFS, and the pick-off paths are repeated for each pick-off. The M4 DM is common to all paths, while the MOAO DM receives its commands from the vdmuser object, which compresses the reconstructed wavefront along a line of sight. The output from the reconstructor is split into the M4 control and the virtual DM controls. Figure 5. A diagram showing the set-up of a segmented MCAO simulation. This looks similar to the previous figure, but fundamentally is very different.

9 Ensquared energy / 100 mas box GS 6 GS 5 GS 4 GS 3 GS Guide star position radius / arcsec Figure 6. A figure showing on-axis ensquared energy as a function of guide star separation for between 3 9 guide stars. Fig. 6 shows the ensquared energy on an on-axis pick-off and AO system as the number and position of guide stars is changed. Each point is the result of only one simulation (about 12 hours of CPU time), hence the variability of points. Each point has been selected from a sample of six for each configuration, each sample having different conditioning parameters for the SVD algorithm. This conditioning value is somewhat arbitrary, though the best value is always found within a fairly small range. We are currently in the process of performing these simulations using a MAP reconstructor, which does not contain this arbitrary condition value. 5.3 Code cross checking Cross checks of the DASP are being carried out with other Monte-Carlo and analytical AO simulation codes. These include YAO (Monte-Carlo) 6 and a code from ONERA (analytical). Agreement between these codes is good for a range of parameters for classical systems. The DASP has previously also been cross checked with a number of codes 7 (PAOLA, CIBOLA, Arizona analytic code, Arizona Monte-Carlo code) for a Gemini GLAO study, and good agreement was seen. 6. CONCLUSION We have provided details of the DASP, a Monte-Carlo code for simulation of AO systems, with specific information about how this platform has been developed for full ELT AO system simulation. Details of specific algorithms have been provided, and also some of the simulation tools available to help design and create new simulations. Details of simulations of the EAGLE instrument for the E-ELT have been provided along with some expected performance results, and the simulation turnaround time has also been provided. REFERENCES [1] Basden, A. G., Adaptive optics simulation performance improvements using reconfigurable logic, Applied Optics 46, (Feb. 2007). [2] Kahle, J. A., Day, M. N., Hofstee, H. P., Johns, C. R., Maeurer, T. R., and D., S., Introduction to the Cell multiprocessor, IBM Journal of Research and Development 49, (Sept. 2005). [3] Yang, Q., Vogel, C. R., and Ellerbroek, B. L., Fourier domain preconditioned conjugate gradient algorithm for atmospheric tomography, AO 45, (July 2006). [4] Assémat, F., Wilson, R., and Gendron, E., Method for simulating infinitely long and non stationary phase screens with optimized memory storage, Optics Express 14, (Feb. 2006). [5] Amdahl, G., Validity of the Single Processor Approach to Achieving Large-Scale Computing Capabilities, in [AFIPS Conference Proceedings, Volume 30, pp (1967)], (1967). [6] YAO Monte-Carlo AO simulation tool.

10 [7] Andersen, D. R., Stoesz, J., Morris, S., Lloyd-Hart, M., Crampton, D., Butterley, T., Ellerbroek, B., Jolissaint, L., Milton, N. M., Myers, R., Szeto, K., Tokovinin, A., Véran, J.-P., and Wilson, R., Performance Modeling of a Wide-Field Ground-Layer Adaptive Optics System, Pub. Astron. Soc. Pacific 118, (Nov. 2006).