SKA SDP : A snapshot of recent technical directions and conclusions

Transcription

1 SKA SDP : A snapshot of recent technical directions and conclusions Chris Broekema ASTRON Netherlands Institute for Radio Astronomy

2 Highlights A whirlwind overview of recent prototyping and design work done within the SDP consortium. Note: these are invariably proofs-of-concept. While concepts are taking shape, unless otherwise stated, no down-selects have been made.

3 Algorithm Reference Library

4 Illustrative flow chart Describing the data relationships between components DRAFT 4

5 Algorithm Reference Library (ARL) A Python prototype of the algorithms All major calibration and imaging algorithms expressed in Python unencumbered by practicalities of I/O, memory management, optimisation Establlishes reference e.g. for later implementation by nonexperts Prototypes of functional components (referentially transparent) Synthesis components: Fourier transforms (predict, invert) using 2D, wprojection, faceting, wslicing, snapshots Model for LOW sky(s3) and LOW station beam (OSKAR) Testing is memory-limited: largest image made so far is 25K by 25K pixels Largely complete 97% coverage test suite 5

6 Algorithm Reference Library Synthesis framework allows experimenting with components 6

7 Dask graph for ICAL 11 ingests. 5 major cycles. 7

8 Execution framework: DaLiuge

9 Algorithms what s the best algorithm to get the desired answer Astronomer Pipeline Logic reduction components and sequence Component Parameters default parameter values of components Data Parallelisation hints about the potential of parallelism Parallel Execution HPC Software Engineer what is executed where Parallel Coding writing parallel code Code Optimisation optimise parallel code OS level S/W Engineer I/O Optimisation optimise I/O on hardware OS and hardware co-design optimise hardware for code to be run Computer H/W Engineer

10 Algorithms best algorithm to get the desired answer Logical Graph Template Pipeline Logic reduction components and sequence Astronomer Component Parameters default parameter values of components Logical Graph Data Parallelisation Operator hints about the potential of parallelism Parallel Execution Physical Graph Template what is executed where Parallel Coding HPC Software Engineer writing parallel code Code Optimisation optimise parallel code Physical Graph I/O Optimisation optimise I/O on hardware OS level S/W Engineer OS and hardware co-design optimise hardware for code to be run Computer H/W Engineer

11

12

13 SDP Integration Prototype (SIP) SDP Consortium meeting, SKA Engineering Meeting, Rotterdam, Friday 16 th June 2017 Ben Mort, Fred Dulwich (University of Oxford) Brian McIlwrath, Chris Pearson, David Terrett, Nijin Thykkathu (Rutherford Appleton Laboratory) Arjen Tamerus, Vlad Stolyarov, Verity Allan, John Taylor (University of Cambridge)

14 SDP System Integration Prototype (SIP) - activities Exploring high-risk areas Current activities Implemented TANGO interfaces to TM CSP interfaces (SPEAD, Pulsar search) Master Controller interface to tasks and execution frameworks Prototyping a number of test capabilities (pipelines) LMC services such as LTS using distributed Redis Deployment of SIP code onto AlaSKA-P3 Next Steps Prototyping of the SDP buffer Integration of LMC services (eg. LTS, Sky model, QA) with execution frameworks Example outcome: resilience in running LMC services. By moving towards a micro-services like architecture using container orchestration the risk of single key service failure for our control interfaces is largely mitigated 14

15 Aims of SIP End-to-end SDP system prototype: Prototype of all major external and internal interfaces Verification and testing of SDP (software) architecture Focused testing and analysis of technology choices Link / overlap with execution framework work and vertical prototyping Link to hardware prototyping (AlaSKA-P3) TM LFAA? SDP LMC services CSP Capabilities / Execution framework(s)

16 Master Controller: Platform as a Service Service that starts, monitors and stops tasks Manages CPU resources Restarts failed tasks Redundancy / failover Service discovery Abstract interface Multiple implementations for different sorts of task First implementation based on Docker swarm mode Mesos + Marathon & Apache Spark have now also been implemented SaaS PaaS IaaS (OpenStack)

17 Middleware Containers, logging & monitoring, visualisation

18 Containerisation a modern software paradigm *

19 Container: A choice of engines Docker Rkt LXC/LXD Others: Singularity, runc Docker currently dominates greatest set of tools, largest community and adoption

20 Container Orchestration: choices We must schedule the pipelines Container build & run and Orchestration Container orchestration Batch processing - Old School! Bespoke DALiuGE, Dask, AirFlow

21 SDP: Logging Architecture

22

23 SDP: Visualisation & Integration

24 Software-defined networks

25 ryu OpenFlow Network controller./analyzer SDP_controller All of the investigated switches worked to some degree. Many of the features we rely on are optional and are not guaranteed to be implemented by the switch vendor. The concept is viable./analyzer We rely on optional features that are not always implemented Host Y performance and quality of implemented features varies wildly The following OpenFlow features are used in our proof-of-concept application: A software-defined network adds robustness over a conventional network Set Field action destination MAC address Set Field action destination IP address Group Type=ALL: apply action instructions Adding the network to the processing platform is a potentially disruptive new paradigm that offers great additional features Group Type=ALL: output to multiple ports./generator Host X Host A SDN capable switch

26

27 Numerical precision