The challenges of computing at astronomical scale

Transcription

1 Netherlands Institute for Radio Astronomy The challenges of computing at astronomical scale Chris Broekema Thursday 15th February, 2018, New Zealand SKA Forum 2018, Auckland, New Zealand ASTRON is part of the Netherlands Organisation for Scientific Research (NWO)

2 Introduction A whirlwind tour through the trials and tribulations of more than a decade of compute system design for radio astronomy. What are we trying to achieve? Tools in our inventory That s the theory, how does it work in practice? Some future work 2

3 What is our business goal? Modern radio telescopes rely heavily on compute power 3

4 What is our business goal? Modern radio telescopes rely heavily on compute power Budgets are limited 4

5 What is our business goal? Modern radio telescopes rely heavily on compute power Budgets are limited Computing is only a small fraction of these 5

6 What is our business goal? Modern radio telescopes rely heavily on compute power Budgets are limited Computing is only a small fraction of these So we want to maximize the scientific value of our investment 6

7 Costs of a compute system Compute systems are cost bound by limited capital and, more recently, energy budgets. The total cost of a compute system can be divided into: Capital investment Development cost Porting of existing applications to a new platform Implementing new science capabilities Operational cost Operational staff cost Maintenance cost (both parts and labour) Energy consumption (excluding cooling) Cooling Infrastructure (rack- and floor-space) These combined are the total cost of ownership (TCO) of the system. 7

8 Costs of a compute system Compute systems are cost bound by limited capital and, more recently, energy budgets. The total cost of a compute system can be divided into: Capital investment Development cost Porting of existing applications to a new platform Implementing new science capabilities Operational cost Operational staff cost Maintenance cost (both parts and labour) Energy consumption (excluding cooling) Cooling Infrastructure (rack- and floor-space) These combined are the total cost of ownership (TCO) of the system. 8

9 Costs of a compute system Compute systems are cost bound by limited capital and, more recently, energy budgets. The total cost of a compute system can be divided into: Capital investment Development cost Porting of existing applications to a new platform Implementing new science capabilities Operational cost Operational staff cost Maintenance cost (both parts and labour) Energy consumption (excluding cooling) Cooling Infrastructure (rack- and floor-space) These combined are the total cost of ownership (TCO) of the system. 9

10 Costs of a compute system Compute systems are cost bound by limited capital and, more recently, energy budgets. The total cost of a compute system can be divided into: Capital investment Development cost Porting of existing applications to a new platform Implementing new science capabilities Operational cost Operational staff cost Maintenance cost (both parts and labour) Energy consumption (excluding cooling) Cooling Infrastructure (rack- and floor-space) These combined are the total cost of ownership (TCO) of the system. 10

11 Costs of a compute system Compute systems are cost bound by limited capital and, more recently, energy budgets. The total cost of a compute system can be divided into: Capital investment Development cost Porting of existing applications to a new platform Implementing new science capabilities Operational cost Operational staff cost Maintenance cost (both parts and labour) Energy consumption (excluding cooling) Cooling Infrastructure (rack- and floor-space) These combined are the total cost of ownership (TCO) of the system. 11

12 Costs of a compute system Compute systems are cost bound by limited capital and, more recently, energy budgets. The total cost of a compute system can be divided into: Capital investment Development cost Porting of existing applications to a new platform Implementing new science capabilities Operational cost Operational staff cost Maintenance cost (both parts and labour) Energy consumption (excluding cooling) Cooling Infrastructure (rack- and floor-space) These combined are the total cost of ownership (TCO) of the system. 12

13 Costs of a compute system Compute systems are cost bound by limited capital and, more recently, energy budgets. The total cost of a compute system can be divided into: Capital investment Development cost Porting of existing applications to a new platform Implementing new science capabilities Operational cost Operational staff cost Maintenance cost (both parts and labour) Energy consumption (excluding cooling) Cooling Infrastructure (rack- and floor-space) These combined are the total cost of ownership (TCO) of the system. 13

14 Costs of a compute system Compute systems are cost bound by limited capital and, more recently, energy budgets. The total cost of a compute system can be divided into: Capital investment Development cost Porting of existing applications to a new platform Implementing new science capabilities Operational cost Operational staff cost Maintenance cost (both parts and labour) Energy consumption (excluding cooling) Cooling Infrastructure (rack- and floor-space) These combined are the total cost of ownership (TCO) of the system. 14

15 Costs of a compute system Compute systems are cost bound by limited capital and, more recently, energy budgets. The total cost of a compute system can be divided into: Capital investment Development cost Porting of existing applications to a new platform Implementing new science capabilities Operational cost Operational staff cost Maintenance cost (both parts and labour) Energy consumption (excluding cooling) Cooling Infrastructure (rack- and floor-space) These combined are the total cost of ownership (TCO) of the system. 15

16 Costs of a compute system Compute systems are cost bound by limited capital and, more recently, energy budgets. The total cost of a compute system can be divided into: Capital investment Development cost Porting of existing applications to a new platform Implementing new science capabilities Operational cost Operational staff cost Maintenance cost (both parts and labour) Energy consumption (excluding cooling) Cooling Infrastructure (rack- and floor-space) These combined are the total cost of ownership (TCO) of the system. 16

17 Total value of a compute system The sum of the science results can considered the value of the system. These are difficult to quantify, especially a priori. We can identify some elements that impact system effectiveness in this respect. Peak performance Efficiency Robustness Programmability Flexibility Suitability / fitness for purspose The combination of these elements define the total value of ownership (TVO) 17

18 Total value of a compute system The sum of the science results can considered the value of the system. These are difficult to quantify, especially a priori. We can identify some elements that impact system effectiveness in this respect. Peak performance Efficiency Robustness Programmability Flexibility Suitability / fitness for purspose The combination of these elements define the total value of ownership (TVO) 18

19 Total value of a compute system The sum of the science results can considered the value of the system. These are difficult to quantify, especially a priori. We can identify some elements that impact system effectiveness in this respect. Peak performance Efficiency Robustness Programmability Flexibility Suitability / fitness for purspose The combination of these elements define the total value of ownership (TVO) 19

20 Total value of a compute system The sum of the science results can considered the value of the system. These are difficult to quantify, especially a priori. We can identify some elements that impact system effectiveness in this respect. Peak performance Efficiency Robustness Programmability Flexibility Suitability / fitness for purspose The combination of these elements define the total value of ownership (TVO) 20

21 Total value of a compute system The sum of the science results can considered the value of the system. These are difficult to quantify, especially a priori. We can identify some elements that impact system effectiveness in this respect. Peak performance Efficiency Robustness Programmability Flexibility Suitability / fitness for purspose The combination of these elements define the total value of ownership (TVO) 21

22 Total value of a compute system The sum of the science results can considered the value of the system. These are difficult to quantify, especially a priori. We can identify some elements that impact system effectiveness in this respect. Peak performance Efficiency Robustness Programmability Flexibility Suitability / fitness for purspose The combination of these elements define the total value of ownership (TVO) 22

23 Scientific value Our challenge is to design a system and associated software stack that offers the most scientific output for a given investment. Or, more formally, we want to find the maximum of: Scientific value of a system SV = TVO TCO 23

24 The tools we can use Reduce cost per Flop? (double precision, single precision, half precision?) Gbps? PByte? Nature publication? Nobel prize? Increase and / or protect value 24

25 The tools we can use Reduce cost per Flop? (double precision, single precision, half precision?) Gbps? PByte? Nature publication? Nobel prize? Increase and / or protect value 25

26 The tools we can use Reduce cost per Flop? (double precision, single precision, half precision?) Gbps? PByte? Nature publication? Nobel prize? Increase and / or protect value 26

27 The tools we can use Reduce cost per Flop? (double precision, single precision, half precision?) Gbps? PByte? Nature publication? Nobel prize? Increase and / or protect value 27

28 The tools we can use Reduce cost per Flop? (double precision, single precision, half precision?) Gbps? PByte? Nature publication? Nobel prize? Increase and / or protect value 28

29 The tools we can use Reduce cost per Flop? (double precision, single precision, half precision?) Gbps? PByte? Nature publication? Nobel prize? Increase and / or protect value 29

30 The tools we can use Reduce cost per Flop? (double precision, single precision, half precision?) Gbps? PByte? Nature publication? Nobel prize? Increase and / or protect value 30

31 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality 31

32 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality 32

33 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality 33

34 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality 34

35 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality Focus on energy efficiency Reduces operational cost, Or Allows larger installation Dynamic clocking / energy aware code 35

36 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality Focus on energy efficiency Reduces operational cost, Or Allows larger installation Dynamic clocking / energy aware code 36

37 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality Focus on energy efficiency Reduces operational cost, Or Allows larger installation Dynamic clocking / energy aware code 37

38 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality Focus on energy efficiency Reduces operational cost, Or Allows larger installation Dynamic clocking / energy aware code 38

39 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality Focus on energy efficiency Reduces operational cost, Or Allows larger installation Dynamic clocking / energy aware code Software environment Quality software environment reduces development cost Small install base often translates to poor software quality Recruitment 39

40 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality Focus on energy efficiency Reduces operational cost, Or Allows larger installation Dynamic clocking / energy aware code Software environment Quality software environment reduces development cost Small install base often translates to poor software quality Recruitment 40

41 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality Focus on energy efficiency Reduces operational cost, Or Allows larger installation Dynamic clocking / energy aware code Software environment Quality software environment reduces development cost Small install base often translates to poor software quality Recruitment 41

42 Reduce cost Drives the desire off-the-shelf components Economy of scale Limits operational and maintenance costs Software availability and quality Focus on energy efficiency Reduces operational cost, Or Allows larger installation Dynamic clocking / energy aware code Software environment Quality software environment reduces development cost Small install base often translates to poor software quality Recruitment 42

43 Increase and / or protect value Algorithmic advances 43

44 Increase and / or protect value Algorithmic advances Computational efficiency 44

45 Increase and / or protect value Algorithmic advances Computational efficiency Scientific improvements 45

46 Increase and / or protect value Algorithmic advances Computational efficiency Scientific improvements Hardware co-design focused on computational efficiency 46

47 Increase and / or protect value Algorithmic advances Computational efficiency Scientific improvements Hardware co-design focused on computational efficiency Amdahl s law 47

48 Increase and / or protect value Algorithmic advances Computational efficiency Scientific improvements Hardware co-design focused on computational efficiency Amdahl s law Amdahl s rules for a balanced system A system needs a bit of IO per second and one byte of main memory for each instruction per second 48

49 Increase and / or protect value Algorithmic advances Computational efficiency Scientific improvements Hardware co-design focused on computational efficiency Amdahl s law Amdahl s rules for a balanced system A system needs a bit of IO per second and one byte of main memory for each instruction per second Improve operational robustness 49

50 Increase and / or protect value Algorithmic advances Computational efficiency Scientific improvements Hardware co-design focused on computational efficiency Amdahl s law Amdahl s rules for a balanced system A system needs a bit of IO per second and one byte of main memory for each instruction per second Improve operational robustness Scheduling of compute tasks 50

51 Increase and / or protect value Algorithmic advances Computational efficiency Scientific improvements Hardware co-design focused on computational efficiency Amdahl s law Amdahl s rules for a balanced system A system needs a bit of IO per second and one byte of main memory for each instruction per second Improve operational robustness Scheduling of compute tasks Reduce required size of system (buffer expensive experiments) 51

52 Increase and / or protect value Algorithmic advances Computational efficiency Scientific improvements Hardware co-design focused on computational efficiency Amdahl s law Amdahl s rules for a balanced system A system needs a bit of IO per second and one byte of main memory for each instruction per second Improve operational robustness Scheduling of compute tasks Reduce required size of system (buffer expensive experiments) Hiding inefficiencies by multi-tennancy 52

53 Increase and / or protect value Algorithmic advances Computational efficiency Scientific improvements Hardware co-design focused on computational efficiency Amdahl s law Amdahl s rules for a balanced system A system needs a bit of IO per second and one byte of main memory for each instruction per second Improve operational robustness Scheduling of compute tasks Reduce required size of system (buffer expensive experiments) Hiding inefficiencies by multi-tennancy Security 53

54 Increase and / or protect value Algorithmic advances Computational efficiency Scientific improvements Hardware co-design focused on computational efficiency Amdahl s law Amdahl s rules for a balanced system A system needs a bit of IO per second and one byte of main memory for each instruction per second Improve operational robustness Scheduling of compute tasks Reduce required size of system (buffer expensive experiments) Hiding inefficiencies by multi-tennancy Security Explore use of non-conventional hardware 54

55 Co-design of communication and compute systems Audience participation bit: By default, what does an Ethernet switch do with network packets it does not recognize? By default, what does a (Linux) node do with network packets it does not recognize? 55

56 Co-design of communication and compute systems Audience participation bit: By default, what does an Ethernet switch do with network packets it does not recognize? By default, what does a (Linux) node do with network packets it does not recognize? 56

57 Challenges in networking for radio astronomy The combination of very high bandwidth communication with a mix of highly specialized custom designed hardware and general purpose systems is often challenging. These can be categorized into three areas: 1 Packets are forwarded on ALL ports, causing performance and packet loss (flood) 2 Packets are handled by the switch software, not hardware, causing significant performance and packet loss (loss) 3 Packets arrive on the correct node, but the wrong Ethernet port (flux) 57

58 Software-defined networks In a software-defined network the control-plane is seperated from the data-plane. A programmable network controller is added to the network that explicitly controls the behavior of the switch data-plane. Controller OpenFlow Protocol Secure Channel Group Table Flow Table Flow Table OpenFlow switch 58

59 Software-defined networks A software-defined network breaks the tigh coupling between sender and receiver. the network controller has explicit control over the data flow the network controller can rewrite IP headers to the sender does not need to know the address of the receiver, it could send to any address 59

60 Robustness of a software-defined telescope Configurable behavior for unknown packets: drop or forward header to controller. Neither will flood the network. Explicit control over the forwarding to physical ports. Control plane is seperated from the switch, mitigating performance issues for software handled traffic. 60

61 Added functionality Conceptually we can make a publish/subscribe system producers Station A A Network Controller OpenFlow B consumers Node 1 SUB B SUB C SUB C Station B B Switch C Node 2 C C Station C Node 3 61

62 So you ve carefully sized your system December last year the internet started to notice heavy and obvuscated development in the Linux kernel tree. On January 3rd Meltdown and Spectre were made public. For radio astronomy the impact may be immense: Meltdown mitigation makes context switches much more expensive Spectre mitigation impact is less obvious, but less likely to be avoidable I/O heavy workloads (such as ours) are context switch heavy Initial Lustre benchmarks showed 40% reduced throughput 62

63 So now what? The choice is simple, either: 1 Patch all systems and accept excessive performance impact 2 Or accept internal systems are trivially exploitable and secure only at the door 63

64 SKA SDP construction timeline 64

65 More Moore 65

66 More Moore 66

67 Beyond Moore 67

68 Conclusions consider how to maximize scientific value the real-world will interfere with your design SKA1 roll-out coincides with disruptive period in semi-conductor industry non-conventional hardware is coming: need to prepare 68

69 Questions 69

70 Backup slides I

71 Normal host sending a packet Packet received, with interface and destination IP Is destination IP in cache? No Enqueue packet Have we recently sent an ARP request for this IP? No Send an ARP request for this IP Yes Yes Retrieve MAC address from cache and write to Ethernet header Wait for ARP reply or timeout Send packet / Send all packets in queue which are for this IP Add IP and MAC to cache Reply ARP reply or timeout Timeout Reply to all packets in queue with ICMP host unreachable Figure: A host sending a packet II

72 LOFAR station sending a packet Packet received Retrieve static header from SRAM and write to Ethernet header Send packet Figure: LOFAR RSP board sending a packet III

73 Flooding A switch will forward a packet to all physical ports, unless it knows which port is connected to the destination host. In a radio telescope data flows are strictly uni-directional Dedicated processing boards have no need to implement protocols like ARP. IV

74 Loss due to packet handling in software The vast majority of switching is done by a dedicated ASIC. Switches also have a more general purpose CPU running the switch OS and tasks that cannot be handled by the ASIC. Packets not recognized by the ASIC are forwarded to the CPU for inspection. This is orders of magnitude slower. V

75 ARP-flux Multi-homed nodes (nodes with multiple Ethernet ports in the same network) are not uncommon in radio astronomy. A Linux will answer ARP requests for a MAC it hosts on any device it receives it on Switches will learn based on ARP replies and the port it is received on node eth :00:00:00:00:AA eth :00:00:00:00:AB ARP request:who-has ? ARP reply: is at 00:00:00:00:00:AB ARP request:who-has ? switch A B Port/MAC table: A: -/- B: -/- A: 00:00:00:00:00:AB B: -/- VI