From Beowulf to the HIVE

Size: px

Start display at page:

Download "From Beowulf to the HIVE"

Gregory Cooper
5 years ago
Views:

1 Commodity Cluster Computing at Goddard Space Flight Center Dr. John E. Dorband NASA Goddard Space Flight Center Earth and Space Data Computing Division Applied Information Sciences Branch 1

2 The Legacy MasPar MP1/MP K custom proc. $1.5-2 Million 6.2 GFlop 3.7 sustained (PPM) 2

3 Beowulf I: Wiglaf Mhz 486DX4 $40, MFlops sustained (PPM) Dual 10Mhz ethernet 3

4 Beowulf II: Hrothgar Mhz Pentium $40, Mflops sustained (PPM) Dual 100Mhz hub-based ethernet 1999: Dual 100Mhz switched ethernet 4

5 Mhz Pentium Pro 100Mhz ethernet Communications thru 10 distribution nodes Beowulf III: Ecgtheow Beowulf Bulk Storage System Mhz Pentium Pro 100Mhz switched ethernet 72 port switch 5

6 thehive Highly-parallel Integrated Virtual Environment Sponsored by: Mission to Planet Earth and Code S Office of Space Science Advanced Technology 6

7 Goal To build a cluster for those who did not build it Requirements Reliability Ease of use Cost performance 7

8 Initial Assumptions Low cost One or few users Low contention for resources Implication No scheduling - free-for-all No accounting 8

9 Reliability Long uptime, short down time, & no loss of data Met with Commodity hardware Well used open source software Hot system backup RAID technology 9

10 Ease of Use Easy/Quick Maintenance Installation Upgrade Easy application development Met with Parallel maintenance Parallel command shell Message Passing Global Filesystem 10

11 Cost Performance Open Source Software Commodity Hardware 11

12 Open Source Needs Parallel Debugger Hardware Diagnostics Global parallel programming paradigm Low latency network protocol 12

13 The HIVE Concept Dvlp Dvlp Console execute nodes S T O R A G E Dvlp Dvlp = Developer s Dvlp Dvlp 13

14 thehive Highly-Parallel Integrate Virtual Environment Mhz Pentium Pro $210, Gflops sustained (PPM) 100Mhz switched ethernet 5-16 port switches Mhz Pentium Pro 100Mhz switched ethernet 72 port switch 14

15 What is thehive? Lots of fans A Beowulf-class Parallel Computer (i.e. Lots of PCs running LINUX) Highly-parallel Integrated Virtual Environment (HIVE) 15

16 The Spec To provide a high performance computing resource (5-10Gflop) for earth and space science applications at low cost (<$200K). To develop a user interface for a parallel computer that exhibits an ease-of-use comparable to a personal workstation or PC. 16

17 thehive 17

18 Architecture 1997 Administrative Host Q0 D0 User Host C0 C1 C2 C3 C4 5 fast ethernet switches b0 - b15 b16 - b31 b32 - b47 b48 - b63 64 dual processor nodes 18

19 Architecture 1998 Administrative Host Q0 D0 User Host 72 port switch 1 fast ethernet switch b0 - b15 b16 - b31 b32 - b47 b48 - b63 64 dual processor nodes 19

20 Architecture 1999 Administrative Host User Host 16 Dual PIII Xeon E B G DL 10 Quad PIII Xeon Fast Ethernet switches 64 dual P-Pro 66 dual P-Pro Myrinet Gigabit Ethernet 20

21 Architecture 2000 Console Host 66 dual P-Pro User Host 16 Dual PIII Xeon B G DL 10 Quad PIII Xeon Fast Ethernet switches Topaz Fast Ethernet Myrinet 16 Dual PIII Mhz SGI Gigabit Ethernet Myrinet Orka VALinux BaByBlue IBM thehive Fast Ethernet Fast Ethernet Myrinet Myrinet 16 Dual PIII Mhz 16 Dual PIII Mhz 21

22 Hardware compute nodes (b0-b63) 2 host nodes (Q0 and D0) Pentium Pro processors (200Mhz, 256K cache) 4 Gbyte of RAM (64 MB per node) 160 Gbyte Disk (2.5 GB per node) 5 100Mhz Full-duplex Fast ethernet switches (C0-C4) 6.4 Gbit/sec maximum inter-processor bandwidth 1600 Mbit/sec within a switch 400 Mbits/sec between switches 22

23 Hardware compute nodes (b0-b63) 2 host nodes (Q0 and D0) Pentium Pro processors (200Mhz, 256K cache) 28 Gbyte of RAM (448 MB per node) 896 Gbyte Disk (14 GB per node) 100Mhz Full-duplex Fast ethernet switch (72 port) 6.4 Gbit/sec maximum inter-processor bandwidth 6400 Mbit/sec within the switch 23

24 The Essence of a BEE 1 Motherboard 2 CPU Chips + Fans 64M Memory 3G EIDE Hard Drive 1 Floppy Drive 1 PCI Video Card 1 PCI Ethernet Card 1 Chassis + Power Supply 24

25 The Essence of the QUEEN 1 Motherboard 2 CPU Chips + Fans 64M Memory 10G EIDE Hard Drive 1 Floppy Drive 1 PCI Video Card 2 PCI Ethernet Card 1 EIDE CDROM Drive 1 Display Monitor 1 Keyboard 1 3 button Mouse 1 Case + Power Supply 25

26 Building thehive (unpacking) 26

27 Building thehive (chassis) 27

28 Building thehive (components) 28

29 Building thehive (thepile) 29

30 Building thehive (therack) 30

31 Inside thehive 31

32 thehive 32

The 1999 Extension 1999 16 Gateway Nodes - dual PIII Xeons - 8 GB ram 10

33 The 1999 Extension Gateway Nodes - dual PIII Xeons - 8 GB ram 10 Dell Nodes - quad PIII Xeons - 5 GB ram - 1Gb/s Myrinet - Fast Ethernet 33

2000 32-700Mhz Pentium III $270,000 1 Gb/s Myrinet 100Mhz Ethernet 200 GB Disk 16 GB RAM 1

34 Mhz Pentium III $270,000 1 Gb/s Myrinet 100Mhz Ethernet 200 GB Disk 16 GB RAM 1 Console node 2 Developer nodes 4 Storage nodes 16 Compute nodes Topaz SGI Commodity Cluster 34

35 Inside Topaz 35

36 BaByBlue IBM Commodity Cluster Mhz Pentium III $270,000 1 Gb/s Myrinet 100Mhz Ethernet 300 GB Disk 16 GB RAM 1 Console node 2 Developer nodes 2 Storage nodes 16 Compute nodes QuickTime and a Photo - JPEG decompressor are needed to see this picture. 36

37 Orka VALinux Commodity Cluster Mhz Pentium III $160,000 1 Gb/s Myrinet 100Mhz Ethernet 100 GB Disk 16 GB RAM 1 Console/Storage node 1 Developer node 16 Compute nodes QuickTime and a Photo - JPEG decompressor are needed to see this picture. 37

38 Inside Orka QuickTime and a Photo - JPEG decompressor are needed to see this picture. 38

39 Installing thesystem Key components & steps Install system on Queen Bootp - Remote boot information server The Floppy System on BEEs Partition Disks & install Initial System The Initial System on BEEs Install Standard & Recovery Systems The Standard & Recovery Systems on BEEs 39

40 How to use thehive Data parallel MIMD (i.e. SPMD) However one is not restricted to SPMD only. Drones are user hosts into thehive Identical file system on all BEEs SPMD commands Most shell commands available on BEES Ssh, Scp, Psh, Pcp, Hsh, and Hcp All /home directories are cross mounted on all nodes under /homes 40

41 Software LINUX operating system Simple command distribution Message passing (PVM, MPI) Distributed shared-memory interface (BSP) Programming Languages: C, C++, Fortran77, Fortran90, ace, and Lisp 41

42 Applications Areas Image Data Base Storage & Retrieval Ocean/Atmosphere Weather Prediction Astronomical Image Restoration Hyper-spectral image information extraction Physical System Modeling 42

43 Specific Applications PPM Fluid Dynamics Image Segmentation Parallel Adaptive Mesh Refinement Plasma Diagnostics and Electromagnetic Interferometry Community Climate Model (weather) 43

44 Image Segmentation 44

45 Performance PPM fluid dynamics Gflops Mandlebrot set calculation Gflops Local Communication Gbit/sec Global Exchange of Data Mbits/s 45

46 thehistory of thehive March 1, Planning began. April 1, Procurement began May 31, Procurement complete June 31, Majority of parts received. July 1, Assembly began July 22, Chassis assembly complete (~ 6 man-weeks) July 23, Software installation began August 7, Cabinet assembly and chassis installation began. August 15, Assembly complete ( ~ 2 additional man-weeks) August 31, Software installation complete. 46

47 Problems & Limitations Only moderate interprocessor bandwidth Long message latency LINUX was not designed for parallel processing Stability (software releases) Not much parallel software available No parallel installation software Hard to get users to write parallel code 47

48 The Future Fy00-3 small 16 processor testbeds (i.e. VALinux, IBM. & SGI ) Fy02 - a large 512 processor testbed (possibly heterogeneous) Fy04 - a 2048 processor testbed 48

49 Conclusions High performance Low Cost Very reliable Easy to use Flexible 49

50 50

51 51

52 52

53 53

54 54

55 55

56 56

57 57

58 58

AMD: WebBench Virtualization Performance Study

March 2005 www.veritest.com info@veritest.com AMD: WebBench Virtualization Performance Study Test report prepared under contract from Advanced Micro Devices, Inc. Executive summary Advanced Micro Devices,