Overview of Science Data Processor. Paul Alexander Rosie Bolton Ian Cooper Bojan Nikolic Simon Ratcliffe Harry Smith

Transcription

1 Overview of Science Data Processor Paul Alexander Rosie Bolton Ian Cooper Bojan Nikolic Simon Ratcliffe Harry Smith

2 Consortium Management and System Engineering

3 Aims of the SDP Work Detailed design of complete SDP system to CDR Hardware platform Complete software stack including system and management layers Delivery is the design documentation supported by Prototyping required for verification of the design Requirements driven approach to SE

4 Management Structure

5 Management Team Lead: Paul Alexander PM: Harry Smith (acting) Deputy PM: Ian Cooper PE: Bojan Nikolic SE: Simon Ratcliffe PS: Rosie Bolton

6 Consortium Members Management Groupings Partner Workshare (%) Status University of Cambridge (Astrophysics & High Performance Full 8.3 Computing Groups) (UK) Netherlands Institute for Radio Astronomy Full 8.6 International Centre for Radio Astronomy Research (AUS) Full 7.5 SKA South Africa Full 7.2 STFC Laboratories (UK) Full 3.1 NIP Team Full 6.3 University of Manchester Max-Planck-Institut für Radioastronomie University of Oxford (Physics) University of Oxford (OeRC) (UK) Full 3.9 Chinese Universities Collaboration Full 22.0 New Zealand Universities Collaboration Full 2.7 Canadian Collaboration Full 12.2 Canadian Universities Collaboration CADC CANARIE Forschungszentrum Jülich (Germany) FULL/HPCF 2.1 Centre for High Performance Computing (SA) HPCF/FULL 3.6 ivec (AUS) HPCF 1.0 Centro Nacional de Supercomputación (ESP) HPCF 1.4 Fundación Centro de Supercomputación de Castilla y León (ESP) HPCF 1.0 Instituto de Telecomunicações (Portugal) Associate 3.1 University of Southampton (UK) * Associate 1.5 University College London (UK) Associate 1.5 University of Melbourne (AUS) Associate 1.0 French Universities Collaboration Associate 1.0 Universidad de Chile Associate 1.0

7 Consortium Industry Partnerss Organization Amazon AMD AVANTEK ARM Bull CISCO DDN Dell Geomerics GNODAL HP Intel Mellanox NAG NVIDIA Oracle Parallel Scientific SGI Thoughtworks Tilera Xyratex Area of Expertise SKA related verification projects E E accelerator options (GGPU), E E Accelerator software stacks E E ARM based architectures E E ARM based architectures, software stacks for E E processing cores File & object store architectures SKA related verification projects Storage system Architecture & streaming data handling, File & object store architectures, SKA related verification projects Storage system Architecture & streaming data handling, File & object store architectures E E Accelerator software stacks Interconnects and NICS Architecture, EE computing, BigData tools E E accelerator options (MIC), SKA related verification projects Interconnects and NICS Software models, software stacks for EE processing cores E E ARM based architectures, EE accelerator options (GGPU), E E Accelerator software stacks Storage system Architecture & streaming data handling, File & object store architectures Software models, File & object store architectures Storage system Architecture & streaming data handling, SKA related verification projects SKA related verification projects E E accelerator options (GGPU), E E Accelerator software stacks Storage system Architecture & streaming data handling, File & object store architectures

8 Original Proposed Schedule Motivation was the expectation of major system changes following re-baselining Follow standard SE approach of element PDRs following system PDR Baseline element architecture at PDR

9 Agreed Delivery Milestones Stage 1 CODE Description Date Value (M ) M1 Kick-off of the project Nov M2 Confirmation of Requirements Feb M3 M4 M5 Reconciliation of requirements with initial Functional Analysis Analysis of the impact of SDP requirements on performance critical algorithms (including baselinedependent averaging) Analysis of scaling of algorithms and components to computational scale of SKA March May June M6 Preliminary element architecture July M7 Key parameters for inputs into the System PDR (to be defined) Sep M8 Participation in system-level review Dec M9 Closure of stage 1 Dec

10 Agreed Delivery Milestones Stage 2 CODE Description Date Value (M ) M10 Kick-off of Stage 2 Jan M11 Element PDR Feb M12 Test Report: Component Interface Completeness Jun M13 Test Report: Key Performance Metrics of Hardware in the Open Architecture Lab (LAB) Sep M14 Test Report: Functional Completeness Dec M15 Test Report: Scaling of System Software Prototype on ISP HPC facilities March 2016 M16 Test Report: Performance vertical prototypes Jun M17 Test Report: Test of System Software Prototype with revised components from vertical prototyping 1 Aug M18 Element CDR submission Sep M19 CDR close-out and wind-up of work Nov

11 Work Breakdown and Scope

12 Scope Added WBS element to consider technical aspects of data delivery to users Designed to inform final decision on edge of observatory, does not imply adding additional responsibilities to the observatory Outside of costing envelope Assume ingest visibilities, pulsar candidates after search, other time-series data from CSP/SADT (TBC)

13 Design Approach Adopt Incremental and Iterative Design approach to the system engineering Fully exploited in the prototyping work. Distinguish between horizontal and vertical prototyping. Horizontal prototyping aims to provide a system-wide prototype Vertical prototyping provides detailed prototyping for performance and functionality of individual components

14 Design Approach Horizontal prototype of whole software system Be used to test and verify the system decomposition and specification of internal interfaces which emerge from the architectural analysis Test models for scalability to SKA1 and give consideration to scaling of the software system to SKA2. Prototype the system architecture to test for the required flexibility in architectural design Prototype and test the design models for loose versus tight coupling between software and hardware. Vertical prototyping of key components Close interaction with industry on emerging technologies Aim to roadmap into technology available for SKA1

15 Design Approach Open Architecture Lab Approach based on Laurence Livermore Hyperion initiative: Integrated work with industry partners Emphasis on a determining an appropriate scalable element Emphasis on system-level components of the software stack Create an evaluation and prototyping testbed for new hardware and software technologies to address: Petascale I/O technology scaling for SKA1 and future capacity to SKA2 Processor, memory, networking, storage, visualization, etc. Designed for future technology refresh, expansion, and upgrades Open source software stacks

16 Element Concept

17 Principles Behind Technical Solution Data parallelism provides a scaleable model through SKA1 to SKA2 Emphasis is on the framework to manage the throughput Hardware platform will be replaced on a short duty cycle c.f. any HPC facility The approach to data analysis for the SKA will evolve during operations as more is learnt about the system Require a framework in which observatory staff can develop efficient radio-astronomy specific code Integration with TM distinguishes the system from a normal HPC environment

18 Example Data Rates and Data Products Baseline Design Element Aperture Array Line experiment (e.g. EoR) 5 sq degrees; channels over 250 MHz bandwidth ~ 30 GB/s reducing quickly to ~ 1GB/s Up to 500 TB UV (Fourier) data; Images (3D) ~ 1.5 TB Imaging experiment with long baselines 50 km baseline with the low-frequency AA or SKA1_Survey 1.5 TB/s reducing to ~ 50 GB/s Up to 1000 TB/day to archive if we archive raw UV data Images (3D) ~ 27 TB Previous Data Rate (GB/s) BL Design (Ingest) (GB/s) LFAA Survey Mid Use Case (maximal) (GB/s)

19 Overall Architecture Imaging: Corner Turning Course Delays Ingest, flagging Visibility Steering Observation Buffer Gridding Visibilities Imaging Image Storage Incoming Data from Correlator, beamformer Switch Buffer store Ingest Processor Switch Buffer store UV Processor HPC Bulk Store Regional Data Centres Non-Imaging: Corner Turning Course Delays Ingest, flagging Beam Steering Observation Buffer Time-series Searching Search analysis Object/timing Storage Heterogeneous hardware architecture Homogeneous software stack

20 Imaging Processing Model Correlator Subtract current sky model from visibilities using current calibration model UV processors RFI excision and phase rotation Major cycle Grid UV data to form e.g. W-projection UV data store Image gridded data Deconvolveimaged data (minor cycle) Update current sky model Astronomical quality data Solve for telescope and image-plane calibration model Update calibration model Imaging processors

21 Use Case requirements and estimates (preliminary) SKA1 LOW CH2 EOR HI emission CH2 EOR source subtraction (continuum survey) CH3 HI absorption CH4 High redshift HI absorption Bmax (km) G(out) GB/s Nchan 2500 Varies with bl 80, ,000 GSamples/s Gridding ops / sample 6.3e3 63e3 13e3 3.2e4 PFlops/s (Gridding) UV Buffer, TBytes Observation length, hours Archive, 1000hrs of experiment (PBytes)

22 System Sizing SKA1 LOW / SURVEY (36 beams): Data rate out of correlator: 4670 GBytes/s (SURVEY), 842 GBytes/s (LOW) Max data rate into SDP: 995 GBytes/s (SURVEY: DRM Ch 3 H1 absorption, proportional to Nbeams, assuming 36) Max computing load (flops/s): 32 Pflops (SURVEY: DRM Ch 3 H1 absorption, proportional to Nbeams, assuming 36) Max UV buffer: 14 PBytes (SURVEY: DRM CH3 H1 absorption) SKA1 Mid: Data rate out of correlator: 1800 GBytes/s (BL design page 49) Max data rate into SDP: 255 GBytes/s (DRM CH3: H1 absorption, band 1) Max computing load: 10.0 Pflops/s (DRM CH3: H1 absorption, band 1) Max UV buffer: 11.0 PBytes (DRM CH3: H1 absorption, band 1)

23 Data Flow Telescope Manager Meta Data Science Data Processor Local M&C Master Controller Local M&C Database Science Data Processor Sky Models, Calibration Parameters... Multiple Reads Visibility processing Image Plane Processing Correlator / Beamformer Data Routing Ingest Data Routing Data Buffer Data Prodcuts Tiered Data Delivery Multiple Reads Time Series Search Time Series Processing

24 Data Flow Tiered Data Delivery Regional Centre Regional Centre SDP Core Facility South Africa Data routing Sub-set of Archive Sub-set of Archive Regional Centre Sub-set of Archive Cloud access SDP Core Facility Australia Astronomer Cloud

25 Example Implementation (current technology) 42U Rack Processing blade 1 Processing blade 2 Processing blade 3 Processing blade 4 Processing blade 5 Processing blade 6 Processing blade 7 Processing blade 8 Processing blade 9 Processing blade 10 Leaf Switch-1 56Gb/s Leaf Switch-2 56Gb/s Processing blade 11 Processing blade 12 Processing blade 13 Processing blade 14 Processing blade 15 Processing blade 16 Processing blade 17 Processing blade 18 Processing blade 19 Processing blade 20 56Gb/s To rack switches Processing Blade: Disk 1 1TB Disk 2 1TB Disk 3 1TB Host processor Multi-core X86 20 TFlop 2x56 Gb/s comms 4 TB storage <1kW power Disk 4 1TB M-Core - >10TFLOP/s PCI Bus Blade Specification GGPU, MIC,? M-Core - >10TFLOP/s Capable host (dual Xeon) Programmable Significant RAM

26 Functional Analysis

27 Software Stack SKA subsystems and service components High-level APIs and Tools UIF Toolkit SKA Common Software Application Framework Core Services Access Control Monitoring Archiver Live Data Access Logging System Alarm Service Configuration Management Scheduling Block Service Base Tools Communication Middleware Database Support Third-party tools and libraries Development tools Operating System

28