Fast-communication PC Clusters at DESY Peter Wegner DV Zeuthen

Transcription

1 Fast-communication PC Clusters at DESY Peter Wegner DV Zeuthen 1. Motivation, History, Cluster Schema 2. PC cluster fast interconnect 3. PC Cluster Hardware 4. PC Cluster Software 5. Operating 6. Future 7. Large cluster installations

2 1. Motivation for PC Clusters Parallel LQCD computing Typical High Performance Computing (HPC) or Supercomputing applications characterized by a demand on high - CPU (especially FPU) performance - Memory throughput - Bandwidth between different nodes Improved PC Architecture suitable for Supercomputing Increasing CPU tact rate (Moore s law), now at 2.4 GHz larger caches at full processor speed, vector units (SSE, SSE2), cache pre-fetch, fast memory interfaces (Rambus GB/s, DDR SDRAM dual channel) High external bandwidth, PCI. PCI-X (66 MHz/64-Bit, 133 MHz/64-Bit). High speed PCI cards (Myrinet, Gb/s optical-connection) and switches Gordon Bell, Jim Gray: By 2010, the cluster is likely to be the principal computing structure.

3 Motivation for PC Cluster, History Autumn 2000 Pentium4 systems (SSE instructions, Rambus memory, 66MHz 64-bit PCI) available, first benchmarks Dual Pentium4 systems, XEON expected for May 2001, First systems on CeBit (under non-disclosure) Official announcement end of May 2001: Intel XEON processor, i860 chipset, Supermicro motherboard P4DC6 the only combination available (until 2002) BA (Ausschreibung) July 2001 First information about the i860 problem Dual XEON test system delivered August 2001 Final decision end of August 2001(Lattice2001 in Berlin) Installation: December 2001 in Zeuthen, January 2002 in Hamburg

4 PC Cluster Schema

5 2. PC cluster fast interconnect Myrinet Network Card (Myricom, USA, Technical details: 200 MHz Risc processor 2 MByte memory 66MHz/64-Bit PCIconnection Gb/s opticalconnection, bidirectional Myrinet2000 M3F-PCI64B PCI card with optical connector Sustained bandwidth: MByte/sec (with optimal chipset support)

6 Myrinet Switch Enclosures providing the mechanical housing, power, cooling, and a built-in backplane network into which Myrinet-2000-switch line cards can plugged into. Max. number of line cards: 16 Max. number of ports: 128

7 Fast interconnect performance Myrinet

8 Fast interconnect performance QSnet (Quadrics Supercomputer World) Ping Bandwidth Bandwidth [Mbytes/sec] Elan3, Main-to-Elan Elan3, Main-to-Main Elan3, Elan-to-Elan Elan3, Elan-to-Main Tports, Main-to-Elan Tports, Main-to-Main Tports, Elan-to-Elan Tports, Elan-to-Main MPI Message size [bytes]

9 The i860 chipset PCI bandwidth problem Xeon Processor Xeon Processor P64H 64 bit PCI PCI Slots (66 MHz, 64bit) P64H 64 bit PCI AGP4X Graphics 800MB/s 800MB/s >1GB/s LAN Connection Interface + ATA 100 MB/s (dual IDE Channels) 10/100 Ethernet 400MHz System Bus 266 MB/s 3.2 GB/s MCH + + ICH2 + Dual Channel RDRAM* 3.2GB/s Intel Hub Architecture 6 channel audio 133 MB/s MRH MRH PCI Slots (33 MHz, 32bit) Up to 4 GB of RDRAM bus_read (send) = 227 MBytes/s bus_write (recv) = 315 MBytes/s of max. 528 MBytes/s 4 USB ports External Myrinet bandwidth: 160 Mbytes/s

10 3. PC Cluster Hardware Nodes Mainboard Supermicro P4DC6 2(1) x XEON P4, 1.7 GHz, 256 kbyte Cache 1 Gbyte (4x 256 Mbyte) RDRAM IBM 18.3 GB DDYS-T18350 U SCSI disk Myrinet 2000 M3F-PCI64B-2 Interface Network Fast Ethernet Switch Gigaline 2024M, 48x100BaseTX ports + GIGAline BaseSX-SC Myrinet Fast Interconnect Installation M3-E32 5 slot chassis, 2xM3-SW16 Line cards Zeuthen: 16 dual CPU nodes, Hamburg: 32 single CPU nodes

11 PC - Cluster Zeuthen schematic node8 node7 node6 node5 node4 node3 node2 node1 8 8 Myrinet Switch 8 8 Ethernet Switch node16 node15 node14 node13 node12 node11 node10 node9 Gigabit Ethernet ( private Network) Host PC DESY Zeuthen Network

12 4. PC - Cluster Software Operating system Linux (z.b. SuSE 7.2) Cluster tools: Communication software: Compiler: Batch system: Cluster management: Clustware (Megware) Monitoring of temperature, fan rpm, cpu usage, MPI - Message passing interface based on GM (Myricom low level communication library) GNU, Portland Group, KAI, Intel Compiler PBS (OpenPBS) Clustware, SCORE

13 Monitoring Tools Clustware from Megware Monitoring example: CPU utilization snapshot DESY HH

14 Monitoring Tools Monitoring example: CPU Utilization, Temperature, Fan speed, DESY Zeuthen

15 ... if (myid == numprocs-1) next = 0; else next = myid+1; MPI (Message Passing Interface) if (myid == 0) { printf("%d sending '%s' \n",myid,buffer); MPI_Send(buffer, strlen(buffer)+1, MPI_CHAR, next, 99, MPI_COMM_WORLD); printf("%d receiving \n",myid); MPI_Recv(buffer, BUFLEN, MPI_CHAR, MPI_ANY_SOURCE, 99, MPI_COMM_WORLD, &status); printf("%d received '%s' \n",myid,buffer); /* mpdprintf(001,"%d receiving \n",myid); */ } else { printf("%d receiving \n",myid); MPI_Recv(buffer, BUFLEN, MPI_CHAR, MPI_ANY_SOURCE, 99, MPI_COMM_WORLD, &status); printf("%d received '%s' \n",myid,buffer); /* mpdprintf(001,"%d receiving \n",myid); */ MPI_Send(buffer, strlen(buffer)+1, MPI_CHAR, next, 99, MPI_COMM_WORLD); printf("%d sent '%s' \n",myid,buffer); }...

16 5. PC Cluster Operating DESY Zeuthen: DESY Hamburg: Dan Pop (DV), Peter Wegner (DV) Hartmut Wittig (Theorie), (Andreas Gellrich (IT)) Maintenance contract with MEGWARE Software: Linux system, Compiler, MPI/GM, (SCORE) Hardware: 1 reserve node + various components MTBF: O(weeks) Uptime of the nodes ( ): Zeuthen 51 days, node8 node16 4 days break for line card replacement Hamburg 54 days Problems: Hardware failures of Ethernet Switch, node, SCSI disks, Myrinet card All components were replaced relatively soon. KAI compiler not running together with MPI/GM, (RedHat-SuSE Linux problem)

17 6. Future CPUs: XEON 2.4 GHz, AMD Athlon XP Processor Chipsets: Mainboards: I/O interfaces: Fast Network: Intel E7500, ServerWorks GC, AMD-760 MPX Chipset full PCI bandwidth Mainboard Supermicro P4DP6 PCI-X, PCI Express Myrinet, QsNet, Inifiniband(?),

18 Dual Intel Xeon 2.4GHz Processor 512KB L2 cache on-die Hyper-Threading enabled 400MHz Bus (3.2GB/s) Dual-Channel DDR Memory (16GB) 3.2GB/s Memory Bandwidth 3.2GB/s I/O Bandwidth 64-bit PCI/PCI-X I/O support Optional SCSI and RAID support GbE support 1u and 2u dense packaging

19 PC Cluster new chipset Intel E7500

20 PC Cluster: Future interconnect Infiniband Concept CPU CPU up to 6GB/s Bi-directional Host Bus Sys Mem Link: High Speed Serial 1x, 4x, and 12x Mem Cntlr HCA Switch: Simple, low cost, multistage network CPU CPU Host Bus Mem Cntlr Sys Mem HCA Link Link Link Switch Link TCA Link Link I/O Cntlr Host Channel Adapter: Protocol Engine Moves data via messages queued in memory Link Link TCA I/O Cntlr Target Channel Adapter: Interface to I/O controller SCSI, FC-AL, GbE,... PCI Express point-to-point channel connection no PCI bus bottleneck

21 PC Cluster: Future interconnect Infiniband Concept (IBM)

22 PC Cluster: Future interconnect (?) Infiniband Cluster IB Fabric 1 st Generation up to 32 nodes (2002) 2 nd Generation 1000s of nodes (2003?)

23 7. Large cluster installations Martyn F. Guest, Computational Science and Engineering Department,CCLRC Daresbury Laboratory CCLRC D Fermilab 2002: nodes, 2005: ~1000 nodes Low power Cluster: