Introduction to High-Performance Computing

Transcription

1 Introduction to High-Performance Computing Simon D. Levy BIOL November 2010 Chapter 12

2 12.1: Concurrent Processing

3 High-Performance Computing A fancy term for computers significantly faster than your average desktop machine (Dell, Mac) For most computational modelling, High Productivity Computing (C. Moler) is more important (human time more costly than machine time). But there will always be applications (like Gromacs) for super-fast computers, so HPC is worth knowing about

4 Background: Moore s Law Moore s Law: Computing power (number of transistors, or switches, basic unit of computation) available at a given price doubles roughly every 18 months Morgan Sparks ( ) with an early transistor

5 Background: Moore s Law

6 Computer Architecture Basics Architecture is used in two different senses in computer science 1. Processor Architecture (Pentium architecture, RISC architecture, etc.): the basic instruction set (operations) provided by a given chip 2. Layout of CPU & Memory (& disk) We will use the latter (more common) sense

7 Computer Architecture Basics Cost per Byte CENTRAL PROCESSING UNIT (RANDOM ACCESS) MEMORY DISK Access Speed

8 Spreadsheet Example Double-click on (open) document: loads spreadsheet data and program (Excel) from disk into memory Type a formula (= A1*B3 > C2) and hit return: 1. Numbers are loaded into CPU s registers from memory 2. CPU performs arithmetic & logic to compute answer (ALU = Arithmetic / Logic Unit) 3. Answer is copied out to memory (& displayed) Frequently accessed memory areas may be stored in CPU s cache Hit Save: memory is copied back to disk

9 Sequential Processing From an HPC perspective, the important things are CPU, memory, and how they are connected. Standard desktop machine is (until recently!) sequential: one CPU, one memory, one task at a time: CPU Memory

10 Concurrent Processing The dream has always been to break through the von Neumann bottleneck and do more than one computation at a given time Two basic varieties Parallel Processing: several CPUs inside the same hardware box Distributed Processing: multiple CPUs connected over a network

11 Parallel Processing: A Brief History In general, the lesson is that it is nearly impossible to make money from special-purpose parallel hardware boxes 1980 s s: Yesterday s HPC is tomorrow s doorstop Connection Machine MasPar Japan s Fifth Generation The revenge of Moore s Law by the time you finish building the $$$$ supercomputer, the $$ computer is fast enough (though there was always a market for supercomputers like the Cray)

12 Supercomputers of Yesteryear Connection Machine CM-1 (1985) Cray YM-P (1988) MasPar MP-1 (1990)

13 Distributed Processing: A Brief (er) History 1990 s s: Age of the cluster Beowulf: lots of commodity (inexpensive) desktop machines (Dell) wired together in a rack with fast connections, running Linux (free, open-source OS) Cloud Computing: The internet is the computer (like Gmail, but for computing services)

14 Today: Back to Parallel Processing Clusters take up lots of room, require lots of air conditioning, and require experts to build, maintain, & program Cloud Computing sabotaged by industry hype (c.f. Larry Ellison rant) Sustaining Moore s Law requires increasingly sophisticated advanced in semiconductor physics

15 Today: Back to Parallel Processing Two basic directions Multicore / multiprocessor machines : lots of little CPUs inside your desktop/laptop computer Inexpensive special-purpose hardware like Graphical Processing Units

16 Multiprocessor Architectures Two basic designs Shared memory multiprocessor: all processors can access all memory modules Message-passing multiprocessor Each CPU has its own memory CPU s pass messages around to request/provide computation

17 Shared Memory Multiprocessor CPU CPU CPU Connecting Network Memory Memory Memory

18 Message-Passing Multiprocessor Connecting Network CPU CPU CPU Memory Memory Memory

19 MPI: A Message-Passing Interface Interface: A specification for how to use the software Implemented mainly in C and FORTRAN mpirun -np X mdrun_mpi_d -deffnm md_0_1

20 Which is better? Scalability is Everything $1000 today $100 today, plus a way of making $100 more every day in the future? Scalability is the central question not just for hardware, but also for software and algorithms (think economy of scale )

21 Processes & Streams Process: an executing instance of a program (J. Plank) Instruction stream: sequence of instructions coming from a single process Data stream: sequence of data items on which to perform computation

22 Flynn s Four-Way Classification 1. SISD: Single Instruction stream, Single Data stream. You rarely hear this term, because it s the default (though this is changing) 2. MIMD: Multiple Instruction streams, Multiple Data streams Thread (of execution): lightweight process executing on some part of a multiprocessor GPU is probably best current exemplar

23 Flynn s Four-Way Classification 3. SIMD: Single Instruction stream, Multiple Data streams -- same operation on all data at once (like Matlab, though it s not (yet) truly SIMD) 4. MISD: Disagreement exists on whether this category has any systems Pipelining is perhaps an example: think of breaking weekly laundry into two loads, drying first load while washing second

24 Communication Pure parallelism : like physics without friction It s useful as a first approximation to pretend that processors don t have to communicate results But then you have to deal with the real issues

25 Granularity & Speedup Granularity: ratio of computation time to communication time Lots of tiny little computers (grains) means small granularity (because they have to communicate a lot) Speedup: how much faster is it to execute the program on n processors vs. 1 processor?

26 Linear Speedup In principle, maximum speedup is linear: n times faster on n processors This gives a decaying (k/n) exponential curve of execution time vs. processors Super-linear speedup is sometimes possible, if each of the processors can access memory more efficiently than a single processor (recall cache concept)

27 12.2: Parallel Algorithms

28 Some Problems Are Embarrassingly Parallel Embarrassingly Parallel (C. Moler): A problem in which the same operation is applied to all elements (e.g., of a grid), with little or no communication among elements

29 The Parallel Data Partition (Master/Slave) Approach Master process communicates with user and with slaves 1. Partition data into n chunks and send each chunk to one of the p slave processes 2. Receive partial answers from slaves 3. Put partial answers together into single answer (grid, sum, etc.) and report it to user

30 The Parallel Data Partition (Master/Slave) Approach Slave processes communicate with master 1. Receive one data chunk from master 2. Run the algorithm (grid initialization, update, sum, etc.) on the chunk 3. Send the result back to the master

31 Parallel Data Partition: Speedup Ignoring communication time, the theoretical speedup for n data items running on p slave processes is As n grows large, this value approaches p: speedup is linear in # of slaves.

32 Simple Parallel Data Partition Is Inefficient Communication time: Master has to send and receive p messages out one at a time, like dealing a deck of cards. Idle time: Master and some slaves are sitting idle until all slaves have computed their results

33 The Divide-and-Conquer Approach We can arrange things so that each process only sends/receives only two messages: less communication, less idling Think of dividing a pile of coins in half, then dividing each pile in half, till each pile has only one coin (or some fixed minimum # of coins) Example: summing 256 values with 8 processors...

34 Divide Phase p 0 : x 0 x 255 p 0 : p 4 : x 0 x 127 x 128 x 255 p 0 : p 2 : p 4 : p 6 : x 0 x 63 x 64 x 127 x 128 x 191 x 192 x 255 p 0 : p 1 : p 2 : p 3 : p 4 : p 5 : p 6 : p 7 : x 0 x 31 x 32 x 63 x 64 x 95 x 96 x 127 x 128 x 159 x 160 x 191 x 192 x 223 x 223 x 255

35 Conquer Phase p 0 : x 0 x 31 p 1 : x 32 x 63 p 2 : x 64 x 95 p 3 : x 96 x 127 p 4 : x 128 x 159 p 5 : x 160 x 191 p 6 : x 192 x 223 p 7 : x 223 x 255 p 0 : p 2 : p 4 : p 6 : x 0 x 63 x 64 x 127 x 128 x 191 x 192 x 255 p 0 : x 0 x 127 p 4 : x 128 x 255 p 0 : x 0 x 255

36 The N-Body Problem Consider a large number N of bodies interacting with each other through gravity (e.g. galaxy of stars) As time progresses, each body moves based on the gravitational forces acting on it from all the others: where m 1, m 2 are the masses of two objects, r is the distance between them, and G is Newton s Gravitational Constant.

37 The N-Body Problem Problem: for each of the N bodies, we must compute the force between it and the other N -1 bodies. This is N*(N-1)/2 = (N 2 N)/2 computations, which is proportional to N 2 as N grows large. Even with perfect parallelism, we still perform 1/ p * N 2 computations; i.e., still proportional to N 2.

38 The N-Body Problem: Barnes-Hut Solution Division by r 2 means that bodies distant from each other have relatively low mutual force. So we can focus on small clusters of stars for the formula, and then treat each cluster as a single body acting on other clusters This is another instance of divide-and-conquer. We will treat space as 2D for illustration purposes.

39

40

41 Divide

42 Divide

43 Build Labelled Hierarchy of Clusters a

44 Build Labelled Hierarchy of Clusters a b d c 11 8

45 Build Labelled Hierarchy of Clusters a b d e c 11 8

46 Produces a Quad-Tree a 1 b c d e Each node (circle) stores the total mass and center-of-mass coordinates for its members. If two nodes (e.g. 1, 5) are more than some predetermined distance apart, we use their clusters instead (1, e)

47 Barnes-Hut Solution: Speedup Each of the N bodies is on average compared with log N other bodies, so instead of N 2 we have N * log N Can have each processor do the movement of its cluster in parallel with others (no communication)