APENet: LQCD clusters a la APE

Transcription

1 Overview Hardware/Software Benchmarks Conclusions APENet: LQCD clusters a la APE Concept, Development and Use Roberto Ammendola Istituto Nazionale di Fisica Nucleare, Sezione Roma Tor Vergata Centro Ricerce E. Fermi SM&FT 2006, Bari 21th September 2006

2 Overview Hardware/Software Benchmarks Conclusions Motivation The APE group has traditionally focused on the design and the development of custom silicon, electronics and software optimized for Lattice Quantum ChromoDynamics. The APENet project was started to study the mixing of existing off-the-shelf computing technology (CPUs, motherboards and memories for PC clusters) with a custom interconnect architecture, derived from previous experience of the APE group. The focus is on building optimized, super-computer level platforms suited for numerical applications that are both CPU and memory intensive.

3 Overview Hardware/Software Benchmarks Conclusions Requested Features The idea was to build a switch-less network characterized by: High bandwidth Low latency Natural fit with LQCD and numerical grid-based algorithm. Not necessarily first neighbour communication. Good performance scaling as a function of the number of processors. Very good cost scaling even for large number of processors.

4 Overview Hardware/Software Benchmarks Conclusions Development stages APENet history: Sept 2001 June 2002: Release of the first HW prototype Sept 2002 Feb 2003: 2nd HW version March Sep 2003: 3rd HW version development and production Dec 2003: Electrical validation Sept 2004: 16 nodes APENet prototype cluster March Nov 2005: 128 nodes APENet cluster June 2006: Production starts

5 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software APENet Main Features APENet is a 3D network of point-to-point links with toroidal topology. Each computing node has 6 bi-directional full-duplex communication channels Computing nodes are arranged in a 3D cubic mesh Data is transmitted in packets which are routed to the destination node Lightweight low level protocol Wormhole routing Dimension ordered routing algorithm 2 Virtual Channels per receiving channel to prevent deadlocks

6 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software How Does it Look Like Here it is shown a example in a logic description and a real implementation: APE 16 (3,1,1) APE 15 (2,1,1) APE 12 (3,0,1) APE 8 (3,1,0) APE 14 (1,1,1) APE 11 (2,0,1) APE 7 (2,1,0) APE 4 (3,0,0) APE 13 (0,1,1) APE 10 (1,0,1) APE 6 (1,1,0) APE 3 (2,0,0) APE 9 (0,0,1) APE 2 (1,0,0) APE 5 (0,1,0) Y+ APE 1 (0,0,0) Z X+ X Z+ Y

7 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software The interconnection card Altera Stratix EP1S30, 1020 pin package, fastest speed grade National Serializers/Deserializers DS90CR485/486, 48 bit 133 MHz Usage of a programmable device allows possible logic redesign and quick on-field firmware upgrade.

8 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software Functional blocks 6.4(x2) Gb/s X+ X- Y+ Y- Z+ Z- 8DiffPairs*800Mhz 48bit*133Mhz (e) (e) (e) (e) (e) (e) 64b*133Mhz Crossbar Switch Router (a) Local PCI (b) (c) (d) (a): PCI & FIFOs (b): Command Queue FIFO (c): Gather/Scatter FIFO (d): RDMA Table Dual Port RAM (e): Channel FIFOs EP1S30 PCI-X CORE PCI-X (64b*133MHz)

9 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software Functional blocks 6.4(x2) Gb/s X+ X- Y+ Y- Z+ Z- 8DiffPairs*800Mhz 48bit*133Mhz (e) (e) (e) (e) (e) (e) Crossbar Switch 64b*133Mhz Router EP1S30 (a) Local PCI (b) (c) PCI-X CORE (d) (a): PCI & FIFOs (b): Command Queue FIFO (c): Gather/Scatter FIFO (d): RDMA Table Dual Port RAM (e): Channel FIFOs PCI-X (64b*133MHz)

10 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software Functional blocks 6.4(x2) Gb/s X+ X- Y+ Y- Z+ Z- 8DiffPairs*800Mhz 48bit*133Mhz (e) (e) (e) (e) (e) (e) Crossbar Switch 64b*133Mhz Router EP1S30 (a) Local PCI (b) (c) PCI-X CORE (d) (a): PCI & FIFOs (b): Command Queue FIFO (c): Gather/Scatter FIFO (d): RDMA Table Dual Port RAM (e): Channel FIFOs PCI-X (64b*133MHz)

11 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software Functional blocks 6.4(x2) Gb/s X+ X- Y+ Y- Z+ Z- 8DiffPairs*800Mhz 48bit*133Mhz (e) (e) (e) (e) (e) (e) Crossbar Switch 64b*133Mhz Router EP1S30 (a) Local PCI (b) (c) PCI-X CORE (d) (a): PCI & FIFOs (b): Command Queue FIFO (c): Gather/Scatter FIFO (d): RDMA Table Dual Port RAM (e): Channel FIFOs PCI-X (64b*133MHz)

12 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software Functional blocks 6.4(x2) Gb/s X+ X- Y+ Y- Z+ Z- 8DiffPairs*800Mhz 48bit*133Mhz (e) (e) (e) (e) (e) (e) Crossbar Switch 64b*133Mhz Router EP1S30 (a) Local PCI (b) (c) PCI-X CORE (d) (a): PCI & FIFOs (b): Command Queue FIFO (c): Gather/Scatter FIFO (d): RDMA Table Dual Port RAM (e): Channel FIFOs PCI-X (64b*133MHz)

13 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software Functional blocks 6.4(x2) Gb/s X+ X- Y+ Y- Z+ Z- 8DiffPairs*800Mhz 48bit*133Mhz (e) (e) (e) (e) (e) (e) Crossbar Switch 64b*133Mhz Router EP1S30 (a) Local PCI (b) (c) PCI-X CORE (d) (a): PCI & FIFOs (b): Command Queue FIFO (c): Gather/Scatter FIFO (d): RDMA Table Dual Port RAM (e): Channel FIFOs PCI-X (64b*133MHz)

14 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software Functional blocks 6.4(x2) Gb/s X+ X- Y+ Y- Z+ Z- 8DiffPairs*800Mhz 48bit*133Mhz (e) (e) (e) (e) (e) (e) Crossbar Switch 64b*133Mhz Router EP1S30 (a) Local PCI (b) (c) PCI-X CORE (d) (a): PCI & FIFOs (b): Command Queue FIFO (c): Gather/Scatter FIFO (d): RDMA Table Dual Port RAM (e): Channel FIFOs PCI-X (64b*133MHz)

15 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software Software A new piece of Hardware needs new Software Device driver Application level library Application level test suites MPI library MPI level test suites

16 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software Submitting job A job submitting environment has been developed, aware of allowed network topologies on a given machine. A configuration file describes allowed topologies: APE_NET_TOPOLOGY APE_NET_PARTITION full APE_NET_PARTITION single APE_NET_PARTITION zline APE_NET_PARTITION zline Jobs are submitted with an mpirun derived script: aperun -topo zline0 cpi

17 Overview Hardware/Software Benchmarks Conclusions Main Features Hardware Software MPI Profiling It can be very hard to produce optimized parallel code. A tool for profiling MPI communication is under development, which can help understanding the communication requests of a certain code for a better tuning of its internal parameters. Informations are gathered regarding: communication vs computation time time spent per communication function transferred data size per communication function transferred data size transfer per rank...

18 Overview Hardware/Software Benchmarks Conclusions Bandwitdh Latency Applications Testbed APE Dual Xeon "Nocona" 3.4 GHz 1 GB RAM DDR TB NFS export Service 100 Mbit Network User access 1Gbit Network APENet network

19 Overview Hardware/Software Benchmarks Conclusions Bandwitdh Latency Applications Some notes about the assembling To host the machine we needed to rework the infrastructure: floor air condition power supply with UPS APENet assembling issues: 5% faulty devices 20% badly mounted devices 10% badly connected links General issues: 10% early dead disks

20 Overview Hardware/Software Benchmarks Conclusions Bandwitdh Latency Applications MPI Bandwidth benchmark Unidirectional Bandwidth 133 MHz Bidirectional Bandwidth 133 MHz Link Protocol Limit 600 Bandwidth (MB/s) K 4K 16K Message Size (bytes) 64K 256K 1M Peak Unidirectional Bandwidth 570MB/s Peak Bidirectional Bandwidth 720MB/s Link physical limit 590MB/s

21 Overview Hardware/Software Benchmarks Conclusions Bandwitdh Latency Applications MPI Latency benchmark One Way Latency Round Trip Latency Time (us) K Message Size (bytes) 4K 16K 64K Minimum Streaming Latency 1.9µs Minumum Round Trip Latency 6.9µs

22 Overview Hardware/Software Benchmarks Conclusions Bandwitdh Latency Applications Real life benchmark The performances (in floating point operation per second) of 4 LQCD core routines are shown. The tests are executed with a fixed local lattice size (8 4 ). Allowed network topologies: 4 CPU: CPU: CPU: CPU: CPU: Best results are achieved when the global lattice geometry reflects the physical network topology.

23 Overview Hardware/Software Benchmarks Conclusions Bandwitdh Latency Applications Application Scaling Block Dirac inversion Full Dirac inversion Dirac application 32 bit Dirac application 64 bit GFlops Number of CPUs 128

24 Overview Hardware/Software Benchmarks Conclusions Bandwitdh Latency Applications Application Scaling MFlops per CPU Number of CPUs Block Dirac inversion Full Dirac inversion Dirac application 32 bit Dirac application 64 bit

25 Overview Hardware/Software Benchmarks Conclusions Bandwitdh Latency Applications Application Scaling Percentage of Communication Time Block Dirac inversion Full Dirac inversion Dirac application 32 bit Dirac application 64 bit Number of CPUs

26 Overview Hardware/Software Benchmarks Conclusions Future work Development is still in progress both for enhancing performances and for bug fixing. Activity will be focused mainly on: Multiport introduction Reduce CPU usage for communication Reduce latency for medium-large size data transfers

27 Overview Hardware/Software Benchmarks Conclusions Conclusions Developing APENet has been a very interesting challenge and performances can be (still) compared with commercial interconnect systems in HPC. Technology is running fast out there, and as we are not market driven (and under-sized), it s very easy to become obsolete. The acquired know-how is going to be transferred in the next generation APE machines (going for PETAFlops?).

28 Overview Hardware/Software Benchmarks Conclusions APENet people R. Ammendola R. Petronzio D. Rossetti A. Salamon N. Tantalo P. Vicini