Early experience with Blue Gene/P. Jonathan Follows IBM United Kingdom Limited HPCx Annual Seminar 26th. November 2007

Transcription

1 Early experience with Blue Gene/P Jonathan Follows IBM United Kingdom Limited HPCx Annual Seminar 26th. November 2007

2 Agenda System components The Daresbury BG/P and BG/L racks How to use the system Some early results References

3 IBM System Blue Gene /P Solution: Expanding the Limits of Breakthrough Science Offering Schematic Blue Gene/P Blue Gene/P continues Blue Gene s leadership performance in a spacesaving, power-efficient package for the most demanding and scalable high-performance computing applications Rack 32 Node Cards 1024 chips, 4096 procs Cabled 8x8x16 System 1 to 72 or more Racks Compute Card 1 chip, 20 DRAMs Chip 4 processors 13.6 GF/s 8 MB EDRAM Node Card (32 chips 4x4x2) 32 compute, 0-2 IO cards 13.6 GF/s 2.0 GB DDR Supports 4-way SMP 435 GF/s 64 GB 14 TF/s 2 TB Front End Node / Service Node System p Servers Linux SLES10 1 PF/s TB + HPC SW: Compilers GPFS ESSL LoadLeveler 12 IBM System Blue Gene /P Solution 2007 IBM Corporation

4 IBM System Blue Gene /P Solution: Expanding the Limits of Breakthrough Science Processor Memory Attribute Details IBM PowerPC MHz; four per node 2 GB SDRAM-DDR per node (Model ) Benefits Low power allows dense packaging; better processor-memory balance Wider application reach Networks Compute Nodes I/O Nodes (10GbE) Operating Systems Performance Power Cooling Dimensions (includes air duct) 20 3D Torus 5.1 GB/s; 3.5 usec latency Collective Network 1.7 GB/s; 2.5 usec latency Global Barrier/Interrupt Optical 10 Gigabit Ethernet (machine control and outside connectivity) GB Control Network (system boot, debug, monitoring Quad SMP processor chip; 1024 per rack Quad SMP processor. Configurable from 8 to 64 per rack. 8 is the default configuration Compute Node Lightweight proprietary kernel Customized Linux possible for small systems. I/O Node Linux Front End and Service Nodes SUSE LINUX SLES 10 Peak per rack 13.9 teraflops 40 kw power consumption per rack (maximum) VAC 3-phase; 175 amp service per rack Air conditioning ~13 tons/rack (minimum) Height 1956 mm Width 1220 mm Depth 966 mm Weight 782 Kg Service clearance 914 mm Raised floor height 16 minimum. 48 recommended IBM System Blue Gene /P Solution Special networks speed up internode communications; designed for MPI programming constructs; improve systems management Double FPU improves performance Increases relative I/O performance Kernel tailored to processor design; industry-standard distribution preserves familiarity to end user Highest available performance benefits capability customers Low power draw enables dense packaging Low cooling requirements enable extreme scale-up Design allows brickwall layout for better floor space utilization 2007 IBM Corporation

5 Data 6.8GB/s Data 6.8GB/s 6.8GB/s Blue Gene/P node 13.6GB/s read(each), 13.6GB/s write(each) PPC 450 FPU PPC 450 FPU PPC 450 FPU PPC 450 FPU L1 L1 L1 L1 Prefetching L2 Prefetching L2 Prefetching L2 Prefetching L2 Multiplexing switch Multiplexing switch 4MB edram L3 4MB edram L3 DDR-2 Controller DDR-2 Controller 4 symmetric ports for Tree, torus and global barriers DMA module allows Remote direct put / get JTAG Control Network DMA Torus Collective Barrier 6*3.4Gb/s bidirectional 3*6.8Gb/s bidirectional Arb 10Gb Ethernet 6*3.4Gb/s To 10Gb bidirectional physical layer (Shares I/O with Torus) 13.6GB/s external DDR2 DRAM bus 2*16B 425Mb/s 17

6 IBM System Blue Gene /P Solution: Expanding the Limits of Breakthrough Science BPC ASIC 4 cores, 8MB Cu Heatsink DRAM address termination (Vtt), EEPROM, decoupling, monitoring SDRAM-DDR2 1 of 20 sites. 16 IBM System Blue Gene /P Solution 2007 IBM Corporation

7 IBM System Blue Gene /P Solution: Expanding the Limits of Breakthrough Science Blue Gene Software Hierarchical Organization Compute nodes dedicated to running user application, and almost nothing else - simple compute node kernel (CNK) I/O nodes run Linux and provide a more complete range of OS services files, sockets, process launch, signaling, debugging, and termination Service node performs system management services (e.g., partitioning, heart beating, monitoring errors) - transparent to application software Front-end nodes, file system 10 Gb Ethernet 1 Gb Ethernet 21 IBM System Blue Gene /P Solution 2007 IBM Corporation

8 IBM System Blue Gene /P Solution: Expanding the Limits of Breakthrough Science Blue Gene/P Application Highlights Multiple Run Modes SMP: 1 MPI Process / Node, 1, 2, 3 or 4 Threads / Process Use for Mixed MPI / OpenMP programs. Use for MPI programs which need > 1 GB / process (only 1 core active per node) Dual Mode: 2 MPI Processes / Node, 1 or 2 Threads / Process Use for mixed MPI / OpenMP programs Use for MPI programs which need > 0.5 GB/process (only 2 cores active per node) Virtual Node Mode: 4 MPI Processes / Node Use for MPI only programs which need < 0.5 GB / process 24 IBM System Blue Gene /P Solution 2007 IBM Corporation

9 IBM System Blue Gene /P Solution: Expanding the Limits of Breakthrough Science Programming Models MPI only virtual node mode with enhancements Separate MPI process for each processor in the compute node DMA support for each MPI process Ensure network does not block when processor is computing Drive network harder Sharing of read-only or write-once data on each node Need programming language extension to identify read-only data Allow applications to overcome memory limits of virtual node mode MPI + OpenMP OpenMP within each node relies on cache coherence support Only master thread on each node initiates communication Get benefits of message aggregation Exploit multiple processors to service MPI call 25 IBM System Blue Gene /P Solution 2007 IBM Corporation

10 Blue Gene in Daresbury

11 Compiling on FEN macmini:~ jonathanfollows$ ssh ############################ W A R N I N G ########################### # This is a private computer facility. Access for any reason must be # # specifically authorised by the owner. Unless you are so authorised,# # your continued access and any other use may expose you to criminal # # and/or civil proceedings. # ###################################################################### Last login: Fri Nov 23 09:06: from host range btcentralplus.com jfo@bglogin2:~$ cat hello_mpi.f include 'mpif.h' integer rank, size, ierror, tag, status(mpi_status_size) character(12) message call MPI_INIT(ierror) call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror) call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror) tag = 100 if(rank.eq. 0) then message = 'Hello, world' do i=1, size-1 call MPI_SEND(message, 12, MPI_CHARACTER, i, tag, MPI_COMM_WORLD, ierror) enddo else call MPI_RECV(message, 12, MPI_CHARACTER, 0, tag, MPI_COMM_WORLD, status, ierror) endif print*, 'node', rank, ':', message call MPI_FINALIZE(ierror) end jfo@bglogin2:~$ mpixlf90 hello_mpi.f -o hello ** _main === End of Compilation 1 === Compilation successful for file hello_mpi.f.

12 Running on BG/P cat run_hello arguments = -np 32 -cwd /home/jfo -exe /home/jfo/hello #@ class = BGP #@ input = /dev/null #@ output = out/$(jobid).out #@ error = out/$(jobid).err #@ wall_clock_limit=01:00:00 #@ notification = complete #@ queue jfo@bglogin2:~$ llq llq: There is currently no job status to report. jfo@bglogin2:~$ llsubmit run_hello Use of uninitialized value in string eq at /home/loadp/bluegene-filter.pl line 120. llsubmit: Processed command file through Submit Filter: "/home/loadp/bluegene-filter.pl". llsubmit: The job "bglogin2.dl.ac.uk.594" has been submitted. jfo@bglogin2:~$ llq Id Owner Submitted ST PRI Class Running On bglogin jfo 11/23 09:08 I 50 BGP 1 job step(s) in queue, 1 waiting, 0 pending, 0 running, 0 held, 0 preempted

13 cat 594.out node 6 :Hello, world node 5 :Hello, world node 13 :Hello, world node 1 :Hello, world node 29 :Hello, world node 30 :Hello, world node 20 :Hello, world node 24 :Hello, world node 8 :Hello, world node 22 :Hello, world node 26 :Hello, world node 12 :Hello, world node 11 :Hello, world node 27 :Hello, world node 10 :Hello, world node 4 :Hello, world node 9 :Hello, world node 18 :Hello, world node 16 :Hello, world node 2 :Hello, world node 0 :Hello, world node 21 :Hello, world node 25 :Hello, world node 7 :Hello, world node 14 :Hello, world node 23 :Hello, world Trivial results

14 Marginally less trivial! # # Benchmarking PingPong # ( #processes = 2 ) # ( 30 additional processes waiting in MPI_Barrier) # #bytes #repetitions t[usec] Mbytes/sec

15 Alltoall # # Benchmarking Alltoall # ( #processes = 128 ) # #bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]

16 Barrier # # Benchmarking Barrier # ( #processes = 128 ) # #repetitions t_min[usec] t_max[usec] t_avg[usec] #=====================================================

17 ESSL for BG/P Just be careful because of the crosscompilation environment to pick up the correct libraries! Native libraries for Linux/POWER /usr/lib64/libessl.so /usr/lib64/libesslsmp.so Libraries for BG/P /opt/ibmmath/lib/libesslbg.a, libesslbg.so.1.3 /opt/ibmmath/lib/libesslmpbg.a, libesslsmpbg.so.1.3

18 References Machine status, request for access Other IBM information Search for Blue Gene Blue Gene/P Application Development and other titles

19 In conclusion. HPCx users should feel comfortable BG/L and BG/P are in the same machine room as HPCx, run by the same administrators, but otherwise separate systems If you ve used BG/L already, BG/P is the same - but if you re starting from scratch, BG/P is easier to port code to - and faster!