PaBdataODI-8.6 Deliverable: D8.6. PaN-data ODI. Deliverable D8.6

Transcription

1 PaN-data ODI Deliverable D8.6 Draft: D8.6: Evaluation of coupling of prototype to multi-core architectures (Month 36 - October 2014) Grant Agreement Number Project Title RI PaN-data Open Data Infrastructure Title of Deliverable D8.6 Evaluation of coupling of prototype to multi-core architectures (Month 36 October 2014) Deliverable Number D8.6 Lead Beneficiary Deliverable Dissemination Level Deliverable Nature Contractual Delivery Date STFC Public Report 01 March 2014 (Month 30) Actual Delivery Date November 2014 The PaN-data ODI project is partly funded by the European Commission under the 7th Framework Programme, Information Society Technologies, Research Infrastructures. Page 1 of 15

2 Abstract Reports D8.1 to D8.5 have detailed much of the work necessary to support parallel and high speed writing and reading of Hierarchical Data Files (hdf) and some of the resulting software and applications have been used already in the D5.3 Virtual laboratories workpackage and report. The main focus of this report is to examine in more detail how this applies to tomography and reconstruction, one of the most resource hungry scientific processes and in great demand. Many of the initial requirements and those that have emerged in the previous scalabilty reports but since D8.5 there has been significant progress in the optimiing the use of computing clusters and high performance file systems and in the collaboration with the hdf5 group in the U.S. Keyword list PANData ODI, Scalability Document approval Approved for submission to EC by all partners on xx.xx.xx Revision history Issue Author(s) Date Description 1.0 Bill Pulford 10 October 2014 Initial Version 1.1 Diamond co-workers 13 November 2014 Complete version for discussion Acknowledgements: Jon Thompson (DLS), Ulrik Pedersen (DLS), Mark Basham (DLS), Frederik Ferner (DLS), Nick Rees (DLS), Heiner Billich (PSI/SLS), Frank Schluenen (DESY), Ka Wanelik (DLS) and the HDF group. Page 2 of 15

3 Table of contents 1. Introduction Scope of the report A brief description of the tomographic process The Tomographic Acquisition Process Schematic The Data Analysis Process The production of sinograms (the Radon Transform) Visualiing the reconstruction Acquisition and the reconstruction with different file systems Introduction Case 1 Low resolution Case 2 High resolution Some conclusions from the above Update on Hierarchical Data File Format (HDF5) support Software available: Introduction The PANData ODI project sets out to optimie coordination between research groups working at one or more different large experimental facilities across Europe and with the potential of expanding its scope across the scientific world. There are a number of components to the project such as common authentication, application software and federated searchable data storage systems. This report relates to a joint research activity, Work Package 8 Scalability, which concerns standardiation of file formats and research to identify supporting data storage architectures to optimie speeds and data storage capacity. The timeline for this workpackage: D8.1: Definition of phdf5 capable Nexus implementation Software Report Delivered Aug 2012 D8.2: Evaluation of Parallel file systems and MPI I/O implementations - Report Delivered Aug 2012 D8.3: Implementation of pnexus and MPI I/O on parallel file systems - D8.4: Examination of Distributed parallel file system - Month 21 June 2013 D8.5: Demonstrate capabilities on selected applications - Month 21 June 2013 o A demonstration application is distributed and is in daily use by many users and at a number of European facilities see DAWNScience. D8.6: Evaluation of coupling of prototype to multi-core architectures (Month 30 March 2014) - Report - To be delivered - This report Page 3 of 15

4 1.1. Scope of the report Reports D8.1 to D8.5 have detailed much of the work necessary to support parallel and high speed writing and reading of Hierarchical Data Files (HDF). Further work has been done since the delivery of the D8.5 report particularly in the exploitation of the applications, computing resources and supporting libraries and, after discussions with the project manager, it was decided to review this in more detail. Included are examinations of the performance of computing clusters and advanced parallel data storage available to DLS together when applied to the processes involved in tomography data acquisition and reconstruction. There are a number of scientific disciplines that can benefit by exploiting the enhanced technology relevant to the workpackage 8 including: a) Macromacromolecular Crystallography (MX) b) Spectroscopy c) Scattering such as Wide and Small Angle (WAXS and SAXS) d) Data from X-ray Free Electron Lasers. (XFEL) e) Tomography Tomography acquisition and reconstruction was selected for the report as the processes involved are probably easier to understand than most and link directly with the computing resources. 2. A brief description of the tomographic process. The scientific details of the tomographic process are covered in many publications (1,2,3) and it is only intended here to describe how this contributes to the high data volumes and consequent computer processing aspects covered by this report The Tomographic Acquisition Process Schematic. Page 4 of 15

5 Figure 1 The tomographic acquisition process The schematic above illustrates the tomography acquisition process and indicates its relative comprehensibility with respect to the underlying computing. The acquisition process essentially provides a microscope capable of producing a three dimensional reconstruction of the sample. These samples can be of many types ranging from biological cells to metallic objects. The X-ray beam is incident on the rotating sample that projects images via suitable optics onto the detector. The sample is scanned vertically in steps varying from micrometres to millimetres, depending on the sample, to give a stack of images that can be transformed with resource intensive applications to reconstruct a 3-dimensional image. The detectors typically used at DLS are detailed in Table 1. Of these the PCO Edge, although smaller, has a much higher repetition rate during an experiment and provides a serious challenge to the data rates and volumes needing to be supported by the file systems. The resolution of the resulting 3-d image is governed by the vertical scan step sie (-axis) and the sampling of the pixels in the x and y axes of the CCD camera. There are frequently two steps in the acquisition: a) The vertical step sie is increased and the sampling frequency of the image is reduced. This permits the rapid production of low resolution 3-d images for evaluation purposes. b) Full acquisition where the -axis scan steps are optimied for high resolution and all pixels of the detector are used. (Often the vertical scan ranges with the PCO 4000 and the PCO Edge being reduced from 6000 to 4000 and 2600 to 1800 respectively for practical purposes. Detector Image Sie Scan Sie Collection Time PCO PCO Edge 4000x bit pixel grayscale 2600x bit pixel grayscale ~80 GB data ~30 minutes ~32 GB ~2 minutes Table 1: The principal detectors used at the Diamond Light Source for Tomography and commonly used in other facilities. Most large facilities will run experiments that involve the use of these resource intensive detectors in parallel and it is the challenge to the infrastructure and file systems to support the multiplication in bulk input and output requirements. Moreover data processing and analysis frequently is done while data are being taken and being transferred to storage and archive; this results in additional concurrent read and write operations. 1 PCO 4000 and PCO Edge are high speed cameras developed and supplied by PCO AG,Donaupark 11, Kelheim, Germany Page 5 of 15

6 3. The Data Analysis Process 3.1. The production of sinograms (the Radon Transform) Image Stack (A) Operation Sinograms (B) Sinograms (B) The orthogonal slices of the image stack are transformed using the standard algorithm into a stack of sinograms of equal data volume to the original image stack. Notes: a) This transformation is high paralleliable and thus conducive to the scalability architecture. The sinogram stack is used as the base data to produce the reconstruction. The reconstruction algorithm used varies according to the sample under investigation and the conditions of the image data acquisition. Nevertheless the processing is normally also highly paralleliable. Reconstruction (C) Figure 2 The basic process of converting an acquired image stack to a reconstructed image The data acquisition process for tomography varies for each facility but is overall represented by the diagram below. The common components include the detector (often a PCO 4000 or Edge), its controlling hardware and firmware (EPICS Area detector, Lima or other), the acquisition software (GDA, SPEC or other) and the association of high performance data storage and computing clusters. The schematic below is based on that found at DLS. Major focus points of this report are: The high capacity data flow from the detector to the data storage. (process A) The initial processing to sinograms using the software - based directly in the detector controller at DLS - (process B) The use of the computing clusters to perform the 3-d reconstruction of the image from the sinograms. (process C) Page 6 of 15

7 The evaluation image processing used actually during the data acquisition and described in 2.1.a tends not to be hugely resource hungry and can be performed by lower performance hardware. The full reconstruction of 2.1.b provides the primary challenge to the file-system and the associated cluster hardware. Figure 3 A schematic diagram of acquisition and analysis architecture at DLS Page 7 of 15

8 The cluster resources The computing cluster hardware required to support the above processes tends to have similarities across most facilities; at DLS it consists of the following: Name Nodes CPUs CPU Total cores COM Intel Xeon E Intel Xeon E5420 Clock speed GH GH RAM Accelerators Network 32GB 16GB NVIDIA Tesla S1070 GPUs (2 per node) 1Gb/s Ethernet 1Gb/s Ethernet COM Intel Xeon E GH 32GB NVIDIA Tesla S1070 GPUs (2 per node) 1Gb/s Ethernet 2 2 Intel Xeon E GH 48GB 1Gb/s Ethernet 37 2 Intel Xeon X GH 24GB 1Gb/s Ethernet QDR Infiniband 20 2 Intel Xeon E GH 32GB 1Gb/s Ethernet QDR Infiniband COM Intel Xeon X GH 48GB NVIDIA Tesla M2090 GPUs (2 per node) 2Gb/s Ethernet QDR Infiniband 40 2 Intel Xeon E v GH 128GB 1Gb/s Ethernet FDR Infiniband Overall this comprises ~2000 cores of variants of x86 with ~80 nvidia GPU s. Page 8 of 15

9 Storage hardware The procurement of storage hardware changes mainly stimulated by the ever increasing volumes of acquired data. DLS currently has 4 beamline storage systems supporting ~ 30 beamlines. Identity File System Sie Details GPFS01 GPFS 1 Petabyte DDN SFA12K Lustre03 Lustre 0.5 Petabyte DDN SFA10K Lustre01 Lustre 0.5 Petabyte DDN S2A9900 Commodity XFS Sum to ~1 Petabyte Note: This report considers systems 1-3 (GPFS, Lustre03 and Lustre01). The XFS systems support those beamlines without the requirements for high performance computing Visualiing the reconstruction The final stage of a tomography experiment is the creation and visualiation of the reconstructed image. There are a number of software suites that provide this functionality, the ideal case being software that can read from the HDF5 (NeXuS) created by the Area Detector during the data collection process. At DLS we use a python pipeline which reads directly from the HDF5 files and allows standard reconstruction routines to process the data in a parallel fashion. The DAWN package[1] provides a convenient graphical user interface to this pipeline and is used frequently during the data acquisition process to evaluate data as soon as it has been collected on the beamline. The core requirement of the high performance file system and cluster resources is that either portions of or the entire reconstruction should be complete and visible as soon as possible after the scan to enable future data collections to be steered by the data collected. Although DAWN allows the data to be visualied in various ways, there are always the requirements for dedicated and specialied volumetric data analysis and visualiation tools; this requirement is filled at DLS with the use of the Avio Package ( Other other open source and commercial packages are available. Page 9 of 15

10 Figure 4 - An example of the DAWN Tomography GUI being used to preview slices from a data collection, and set up the parameters for a full reconstruction, in this case the view is of a salt solution droplet, with growing bubbles, and a salt crystal shown in this frame. Figure 5 - The full reconstructed volume being viewed by DAWN using its Tiff Stack visualisation Options, slicing across the entire dataset, to view the crystal of salt, solution and support from the side. Page 10 of 15

11 Figure 6 - The same volume as rendered by AVIZO and clearly showing the salt crystals and Bubbles in the salt solution. 4. Acquisition and the reconstruction with different file systems 4.1. Introduction The evolution and procurement of hardware is rapid consequently the intention of this section is not to produce definitive results for the operations required using different file storage and reconstruction architectures. It is rather to provide some underlying information concerning the technology in action and some hard won observations that may be useful for other facilities and valuable for decisions to be made for subsequent projects. This work is being done at DLS but uses technology that could be readily available at a cost and should be able to be run at collaborating facilities. At this point DLS has the compute clusters and data file storage systems described in section 3. This has the inbuilt advantage enabling a comparison of the properties of these commonly used high performance systems. The testing involved using the same sample and data processing in each case but switching the supporting computer hardware and data storage technology. There are two cases considered that correspond to low resolution and high resolution reconstruction (2.1.a and 2.1.b as above). Each test concerns two major factors that influence the performance of the operations: a) File copy speed, a measure of the data storage system and network technology. b) The reconstruction time, influenced by the processor performance. Page 11 of 15

12 Duration (s) Duration (s) PaBdataODI-8.6 Deliverable: D8.6 c) In each case the evaluation was done on the basis 10 sequentially run cloned processes for the above. (see the horiontal axes on each plot) In each test the data storage technology (Lustre01, Lustre03, GPFS01) is identified directly on the diagram Case 1 Low resolution Parameters: Detector = PCO Edge, Cluster = COM07,Resolution = 2560 x 2160 (111 scan steps), NXSie = 128k, Total file sie = 1.4Gb, Job scheduler = Univa Grid Engine (UGE) Times for copying 1.4G file GPFS01 lustre01 lustre Reconstruction Duration (Low Resolution) GPFS01 lustre01 lustre03 Page 12 of 15

13 Duration (s) Duration (s) PaBdataODI-8.6 Deliverable: D Case 2 High resolution Parameters: Detector = PCO Edge, Cluster = COM07,Resolution = 2560 x 2160 (3651 scan steps), NXSie = 848k, Total file sie = 39Gb, Job scheduler = Univa Grid Engine (UGE) Times for copying 39G file GPFS01 Lustre01 Lustre Reconstruction Duration (High Resolution) GPFS01 Lustre01 Lustre Page 13 of 15

14 4.2. Some conclusions from the above. General: The use of the latest HDF5 format and associated NeXuS metadata instead of the original TIFF image stacks have led to at least a100% improvement in performance. Technical: o An additional benefit of applying the NeXuS model is the potential to use single files that include everything necessary for further processing. The details of the necessary processing must be considered. An example is that I/O to a parallel HDF5 file (phdf) is not necessarily very efficient for low levels of parallelism mainly due to the time and resources needed during setup the processing pipelines. It has been found that only beyond a parallelism of 5 does using the parallel architecture result in rapid increases in performance that outweigh the setup time. There are strong arguments for using MPI for cluster jobs as it found to be both more portable and likely to exist on all clusters. o The current reconstruction software starts with an HDF5 file uses parallel read from many cluster nodes and separate processes using batch processing cluster management software UNIVA ( The combination of the separation of files and the properties of the cluster management software result in a stack of files, normally TIFF images. It would be highly desirable to write from these separate processes to HDF5 file. o The use of MPI and phdf5 with the new pipeline enable this and consequently greatly simplify processes such as archiving and management. There are technical issues associated with our HPC file systems that can result in inconsistent performance levels. This implies that there must be tight control over configuration management; we have started to use Jenkins ( for this purpose. Contributory issues observed include: o Operating System upgrades, hardware changes and software changes) can lead do hard to understand degradation of performance. o Performance can often depend on various factors such as current occupation level of the file system. Page 14 of 15

15 5. Update on Hierarchical Data File Format (HDF5) support. There have been some important developments of the phdf5 libraries since the previous D8.5 report, of these one of the most important has been the release of support for Single Writer and Multiple Reader (SWMR) for phdf files. These were not directly funded by PANData ODI but are deemed to be of fundamental importance to subsequent research by supplying a stable and freely available solution for high performance computing. The funded developments were all performed by the HDF5 group at the request of the facilities and should be integrated into the HDF release tree. Mar 2013: Feasibility study (Funded by Diamond: $67k with an estimated cost to complete the developments current foreseen: $344k) Aug 2013: SWMR internal library changes (Funded by Diamond: $103k) Oct 2013: SWMR API changes (Funded by Dectris: $38k) Nov 2013: HDF5 SWMR test infrastructure (Funded by ESRF: $60k) The current state of HDF5 development at DLS may be followed on the externally available web site: 6. Software available: - h5python, a version of python optimied to access HDF5 files and allow the use of additional tools such as numpy. A single write and multiple reader (SWMR) test application cbflib -> NeXus - contains a NeXuS data writer - a high performance library for directly reading and writing NeXuS files. o o The detailed control of the detector is delegated to plugins within the EPICS area detector architecture; the plugins are normally written in C or C++. The parallel hdf5writer is currently tailored to EPICS/Diamond requirements, however this is only superficial. The intention would be to abstract it out and publish it on our external website. The main issue is to abstract the TCP protocol from detector system to phdf5writer. Given the plugin code it should be relatively straightforward to integrate into the LIMA architecture (Lima.blissgarden.org/applications/tango/doc/index.html) Page 15 of 15