Desktop Grids Are They Green? Dr James Osborne Dr Hugh Beedie

Size: px

Start display at page:

Download "Desktop Grids Are They Green? Dr James Osborne Dr Hugh Beedie"

Darcy Ross
5 years ago
Views:

1 Desktop Grids Are They Green? Dr James Osborne Dr Hugh Beedie

2 Contents What is a desktop grid? Who uses our desktop grid? Power consumption of a green Condor pool Power consumption of a green cluster Power consumption of other clusters Comparing computational efficiency Condor is green! (power consumption) Nov 07 Condor is green? (computational eff.) Sep 08

3 What is a Desktop Grid?

What is Grid Computing The Grid is an emerging infrastructure that will fundamentally change the way we think about and use computing The grid will

4 What is Grid Computing The Grid is an emerging infrastructure that will fundamentally change the way we think about and use computing The grid will connect multiple regional and national computational grids to create a universal source of computing power Definition from Foster and Kesselman (1998)

5 What is a Service Grid? Service Grids are more flexible and can accommodate a broader variety of applications than Desktop Grids, however their setup and maintenance require greater effort, highly skilled specialists, and dedicated resources. Definition adapted from EDGES website

6 What is a Desktop Grid? Desktop Grids are currently restricted solely to a subclass of compute-intensive applications but these easy-to-scale systems are able to collect 1 or 2 orders of magnitude more compute power by utilizing spare and donated resources at a fraction of the cost. Definition adapted from EDGES website

7 The ARC Spectrum HPC Tightly Coupled HTC Loosely Coupled Supercomputers NUMA Machines Million+ Large Clusters SMP H Thousand Million Small Clusters Campus Grids Thousand H Thousand

8 What is Condor? A specialised workload management system for compute-intensive jobs Users submit their jobs to Condor Condor places them into a queue Condor chooses where and when to run them Condor carefully monitors their progress Condor informs the user upon completion Definition adapted from Condor website

9 What is a Condor Job? A command line windows executable All files in a self-contained directory structure Condor runs jobs in a sandbox..\execute\... Condor runs jobs as user condor-reuse-vm1 One or more input files One or more output files A submit file

10 Central Manager master, collector, negotiator Execute Nodes 1600 Workstations Submit Nodes 30 Workstations master, schedd, shadow master, startd, starter

11 Who uses our Desktop Grid?

12 Diversity of User Base Architecture 1 Biosciences 8 Business 1 Chemistry 1 Computer Sci 7 Engineering 4 History Arch 2 Mathematics 2 Medicine 4 Optometry 3 Physics 2 Psychology 1 Social Sci 1 Total 37

13 Diversity of Applications Blast, Damfilt Dammin, Dmarel Energyplus Gasbor, Grinder Lea, Leadmix Matlab, Msvar Oxcal, Paup, Perl, Pest, R Sienna, Structure Artificial Neural Nets Econometric Modelling Fluid Dynamics Fourier Analysis Geological Modelling Image Processing Radiation Transport Travelling Salesman WIFI Roaming

14 Research Assistant Patrick Downes Velindre Cancer Centre Monte Carlo simulation Radiotherapy dose calculation Time on a single workstation = 3 Months Time on the Condor pool = 36 Hours Speed-up of 6000%

15 Structural Biophysics Group Donna Lammie OPTOM X-Ray Diffraction Determine shape of molecules Time on a single workstation = 2-3 Days Time on the Condor pool = 2-3 Hours Speed-up factor of 2000%

16 Cardiff s Condor Pool...is the equivalent of a 500,000 supercomputer costs 50,000 in equipment, power, and staff improves return on investment...is one of the largest pools in the UK and we plan to expand the pool is probably the most utilised pool in the UK by a factor of 10...has more users than other pool in the UK and we are working hard to keep it that way

17 Watts Up Pro Measures Watts, Volts, Amps, WattHrs, Cost, Avg Kwh, Mo Cost, Max Wts, Max Vlt, Max Amp, Min Wts, Min Vlt, Min Amp, Pwr Fct, Dty Cyc, Pwr Cyc Freq 1 second Duration 15 minutes

18 Power Consumption of a Green Condor pool

19 Cost of a Condor Pool Capital expenditure None, the machines have already been bought Power used to compute Some, additional power consumption above idle Power used to cool None, the machines are in a large open space Therefore POWER CONDOR = POWER NODE POWER NODE = CPU MAX - CPU IDLE

20 Experiment Measure power consumption of the chassis whilst the CPU is IDLE Measure power consumption of the chassis whilst the CPU is running CPUSoak

21 Power Supply Based on a C2D 1.8GHz PC with 2GB RAM 2 Cores total per chassis 1 Processor total per 65w TDP A standard modern power supply 350w (80% efficiency)

22 Power Consumption in off P3 1.0GHz 57.2w CPU MAX, 39.2w CPU IDLE 19 off P4 1.8GHz 86.4w CPU MAX, 49.6w CPU IDLE 705 off P4 3.0GHz 148.4w CPU MAX, 101.2w CPU IDLE 902 off PCs total 37.2 kw CPU MAX Number and type of machines from INSRV inventory

23 Power Consumption in 2006 Machines by Type P3 1.0GHz 20% Machines by Power P3 1.0GHz 9% P4 1.8GHz 2% P4 1.8GHz 2% P4 3.0GHz 78% P4 3.0GHz 89% In 2006 newer machines perform more calculations than their older counterparts In 2006 newer machines consume MORE power than their older counterparts

24 Power Consumption in off P4 3.0GHz 148.4w CPU MAX, 101.2w CPU IDLE 162 off C2D 1.8GHz 93.2w CPU MAX, 75.6w CPU IDLE 818 off PCs total 33.6 kw CPU MAX Number and type of machines from INSRV inventory

25 Power Consumption in 2008 Machines by Type C2D 1.8GHz 20% Machines by Power C2D 1.8GHz 8% P4 3.0GHz 80% P4 3.0GHz 92% In 2008 newer machines perform more calculations than their older counterparts In 2008 newer machines consume LESS power than their older counterparts

26 Power Consumption Watts Off Hibernate Standby Idle Office Condor Based on a P4 3.0GHz PC (Blue) Based on a C2D 1.8GHz PC (Red)

27 Condor Hardware 818 off Single Socket, Mixed Core nodes 656 off Intel P4 3.0GHz w 2GB RAM (80%) 162 off Intel C2D 1.8GHz w 2GB RAM (20%) 10/100Mbps Ethernet Interconnect 538 x10 9 FLOPS 33.6 x10 3 WATTS 16.0 x10 6 FLOPS / WATT Pilot project in April 2004, production project in January 2006 Figures based on approximately half of the total number of machines in the pool

28 Condor Power For one P4 node 101w PSU IDLE 148w PSU DGEMM For all nodes 78.6Kw PSU IDLE 112.2Kw PSU DGEMM For one C2D node 76w PSU IDLE For everything 33.6Kw 93w PSU DGEMM

29 Summary Newer machines consume less power Newer machines perform more calculations Condor is getting cheaper, faster, greener Condor is replaced for free every four years Condor and power saving can work together Easy to administer if you have deployment tools

30 Power Consumption of a Green Cluster

31 Capital expenditure Cost of a Cluster Large, the machines have not already been bought Power used to compute Large, total power consumption Power used to cool Some, the machines are in cooled racks Therefore POWER CLUSTER = POWER NODE + POWER COOL POWER NODE = CPU MAX

32 Hot Aisle Containment +30% for Cooling. Images courtesy of APC and Comtec

33 1U Twin Server, 1 Power Supply A Cluster Node

34 Experiment Measure power consumption of the chassis whilst the CPU is IDLE Measure power consumption of the chassis whilst the CPU is running DGEMM

35 Power Supply Based on 4 C2Q 3.0GHz PC with 32GB RAM 16 Cores total per chassis 4 Processors total per 80w TDP A high efficiency modern power supply 980w (92% efficiency)

36 Merlin Hardware 256 off Dual Socket, Quad Core nodes Intel Harpertown 3.0GHz w 16GB RAM Voltaire Infiniband Connect-X Interconnect x10 12 FLOPS x10 3 WATTS 198 x10 6 FLOPS / WATT Commissioned in June 2008

37 Merlin Power For two nodes 332w PSU IDLE 610w PSU DGEMM For the cooling Rack 30% 23.4Kw For all nodes 42.5Kw PSU IDLE 78.1Kw PSU DGEMM Nodes + Cooling 55.3Kw IDLE 101.5Kw DGEMM Commissioned in June 2008

38 Cost of Merlin Capital expenditure on Machine + Capital expenditure on Environment + Running costs (POWER NODE + POWER COOL ) = 600k per year for x10 12 FLOPS

39 Power Consumption of Other Clusters

40 Capital expenditure Cost of a Cluster Large, the machines have not already been bought Power used to compute Large, total power consumption Power used to cool Large, the machines are in a cooled room Therefore POWER CLUSTER = POWER NODE + POWER COOL POWER NODE = CPU MAX

41 Hot Aisle Cold Aisle +60 to +100% for Cooling. Images courtesy of APC and Comtec

42 Helix Hardware 72 off Dual Socket, Single Core nodes 54 off Intel Xeon 2.2GHz w 1.5GB RAM 18 off Intel Xeon 2.4GHz w 1.5GB RAM Dolphin Wulfkit Interconnect 323 x10 9 FLOPS 29.0 x10 3 WATTS 11.1 x10 6 FLOPS / WATT Commissioned in June 2003

43 Helix Power For one node 111w PSU IDLE 202w PSU DGEMM For the cooling Room 100% 14.5Kw For all nodes 8Kw PSU IDLE 14.5Kw PSU DGEMM Nodes + Cooling 16Kw IDLE 29Kw DGEMM Commissioned in June 2003

44 Cubric Hardware 75 off Dual Socket, Dual Core nodes AMD Opteron 2.2GHz w 4GB RAM Gigabit Ethernet Interconnect 530 x10 9 FLOPS 30.2 x10 3 WATTS 17.5 x10 6 FLOPS / WATT Commissioned in June 2006

45 Cubric Power For one node 175w PSU IDLE 252w PSU DGEMM For the cooling Room 60% 11.3Kw For all nodes 13.1Kw PSU IDLE 18.9Kw PSU DGEMM Nodes + Cooling 20.9Kw IDLE 30.2Kw DGEMM Commissioned in June 2006

46 Summary Room based cooling is less efficient Low density clusters are less efficient Smaller clusters are less efficient Older clusters are less efficient

47 Comparing Computational Efficiency

48 Condor is Green! The view as of November 2007

Based on a P4 3GHz PC with 512MB RAM Economic Viability Makes sound financial sense Hibernate saves 60 per year Condor = 30 per year (max) Dedicated = 150 per year Condor is 5 times cheaper Saving of

49 Based on a P4 3GHz PC with 512MB RAM Economic Viability Makes sound financial sense Hibernate saves 60 per year Condor = 30 per year (max) Dedicated = 150 per year Condor is 5 times cheaper Saving of Hibernate = Cost of 100W Electricity (Idle State) for 16 Hours out of 24 Cost of Condor = Cost of 150W Electricity (Condor State) Cost of 100W Electricity (Idle State) Cost of Dedicated = Cost of 150W Electricity (Condor State) + Cost of 100W Electricity (Air Con)

Based on a P4 3GHz PC with 512MB RAM Environmental Impact Makes sound environmental sense Hibernate saves 650Kg CO 2 per year Condor = 325Kg CO 2 per year (max) Dedicated = 1,625Kg CO 2 per year

50 Based on a P4 3GHz PC with 512MB RAM Environmental Impact Makes sound environmental sense Hibernate saves 650Kg CO 2 per year Condor = 325Kg CO 2 per year (max) Dedicated = 1,625Kg CO 2 per year Condor is 5 times greener Saving of Hibernate = Cost of 100W Electricity (Idle State) for 16 Hours out of 24 Cost of Condor = Cost of 150W Electricity (Condor State) Cost of 100W Electricity (Idle State) Cost of Dedicated = Cost of 150W Electricity (Condor State) + Cost of 100W Electricity (Air Con)

51 Based on 10,000 P4 3GHz PCs with 512MB RAM Across Campus Makes sound financial sense Hibernate would save 600,000 per year Hibernate 16 out of 24 hours Makes sound environmental sense Hibernate would save 6,500T CO 2 per year Rainforest required = 52Km 2 Rainforest required = 40% area of Cardiff Saving of Hibernate = Cost of 100W Electricity (Idle State) for 16 Hours out of 24 Cost of Condor = Cost of 150W Electricity (Condor State) Cost of 100W Electricity (Idle State) Cost of Dedicated = Cost of 150W Electricity (Condor State) + Cost of 100W Electricity (Air Con)

52 Condor is Green? The view as of September 2008

53 Clusters

54 Add Condor

55 Extrapolate

56 Interpolate

57 Average Case

58 Best Case

59 Summary Condor is not as green as we thought it was Condor is about as green as a 2 year old cluster Clusters should have a lifetime of about 4 years Clusters older than 4 years should be scrapped If you cannot afford a Cluster then use Condor Condor machines older than 2 years are bad Accelerators will muddy the waters in the future Graphics cards in desktops will be interesting

60 Acknowledgements Ian Merrick Helix Chemistry and Earth Sciences Spiro Stathakis Cubric Psychology Huw Lynes Merlin - ARCCA

Data Centre Energy & Cost Efficiency Simulation Software. Zahl Limbuwala

Data Centre Energy & Cost Efficiency Simulation Software Zahl Limbuwala BCS Data Centre Simulator Overview of Tools Structure of the BCS Simulator Input Data Sample Output Development Path Overview of