CS 770G - Parallel Algorithms in Scientific Computing Parallel Architectures. May 7, 2001 Lecture 2

Transcription

1 CS 770G - arallel Algorithms in Scientific Computing arallel Architectures May 7, 2001 Lecture 2

2 References arallel Computer Architecture: A Hardware / Software Approach Culler, Singh, Gupta, Morgan Kaufmann Introduction to arallel Computing: Design and Analysis of Algorithms Kumar, Grama, Gupta, Karypis, Benjamin Cummings 2

3 erformance goals 3

4 Microprocessor performance 4

5 What is a arallel Computer? Almasi-Gotllib 1989: A parallel computer is a "collection of processing elements that communicate and cooperate to solve large problems fast". Why parallel architecture? Add new dimension to design space -- number of processors. In principle, achieve higher performance by using more processors How much additional performance is gained and at what additional cost depends on several factors. 5

6 Questions How large is the collection? How powerful are the individual processing elements (pe)? Can the number be increased in a straightforward manner? How do they communicate and cooperate? How is data transmitted between pe's? What interconnection topology? 6

7 Taxonomy of arallel Architectures I. By control mechanism - instruction stream and data stream II. III. IV. By process granularity - coarse vs fine grain By address space organization - shared vs distributed memory By interconnection network - dynamic vs static 7

8 (I) Control Mechanism (Flynn s taxonomy) SISD: Single Instruction stream Single Data stream, e.g. conventional sequential computers. SIMD: Single Instruction stream Multiple Data stream MIMD: Multiple Instruction stream Multiple Data stream MISD: Multiple Instruction stream Single Data stream 8

9 SIMD Multiple processing elements are under the supervision of a control unit Thinking Machine CM-2, Masar M-2, Quadrics SIMD extensions are now present in commercial microprocessors (MMX or Katmai in Intel x86, 3DNow in AMD K6 and Athlon, Altivec in Motorola G4) 9

10 MIMD Each processing elements is capable of executing a different program independent of the other processors Most multiprocessor can be classified in this category) 10

11 (II) rocess Granularity Coarse grain: Cray C90, Fujitsu small number of very powerful processors Fine grain: CM-2, Quadrics large number of relatively less powerful processors Medium grain: IBM S2, CM-5 between the two extremes. Commuication cost >> computational cost coarse grain Commuication cost << computational cost fine grain 11

12 (III) Address Space Organization Single/shared address space Uniform Memory Address:SM (UMA) Non Uniform memory Address (NUMA) Message passing Distributed memory 12

13 SM Architecture CU CU CU CU Cache Cache Cache Cache Bus or Crossbar Switch Memory I/O SM uses shared system resources (memory, I/O) that can be accessed equally from all the processors Cache coherence is maintained 13

14 NUMA Architecture Memory Memory Memory Memory CU CU CU CU Cache Cache Cache Cache Bus or Crossbar Switch Shared address space Memory latency varies whether you access local or remote memory Cache coherence is maintained using hardware or software protocol 14

15 Message-assing Architecture Memory Memory Memory Memory CU CU CU CU Cache Cache Cache Cache Communication network Local address space No cache coherence 15

16 (IV) Interconnection Networks Dynamic Switches and communication links. Communication links are connected to one another dynamically by switches. Static oint-to-point communication links. Message-passing computers. 16

17 Dynamic Interconnections Crossbar switching : Most expensive and extensive interconnection. 1 2 M1 M2 Bus connected : rocessors are connected to memory through a common datapath Multistage interconnection: Butterfly,Omega network, perfect shuffle, etc Butterfly 17

18 Static Interconnection Completely-connected Star-connected Linear array Mesh: 2D/3D mesh, 2D/3D torus Tree and fat tree network Hypercube network 18

19 Characteristics of Static Networks Diameter: maximum distance between any two processors in the network D=1 complete connection D=N-1 linear array D=N/2 ring D=2( N -1) 2D mesh D=2 ( (N/2)) 2D torus D=log N hypercube 19

20 Characteristics of Static Networks (cont.) Bisection width: the minimum number of communications links that have to be removed to partition the network in half. Channel rate: peak rate at which a single wire can deliver bits. Channel bandwidth: it is the product of channel rate and channel width. Bisection bandwidth B: it is the product of bisection width and channel bandwidth. 20

21 Linear Array, Ring, Mesh, Torus rocessors are arranged as a d-dimensional grid or torus 21

22 Tree, Fat-tree Tree network: there is only one path between any pair of processors. Fat tree network: increase the number of communication links close to the root. 22

23 Hypercube 1-D 2-D 3-D 23

24 Binary Reflected GRAY Code G(i,d) denotes the i-th entry in a sequence of Gray codes of d bits. G(i,d+1) is derived from G(i,d) by reflecting the table and prefixing the reflected entry with 1 and the original entry with 0. 24

25 Binary Reflected GRAY Code Mapping a linear array into an hypercube: A linear array (or ring) of 2^d processors can be embedded into a d-dimensional hpercube by mapping processor I onto processor G(I,d) of the hypercube Mapping a 2^r x 2^s mesh on an hypercube: processor(i,j)---> G(i,r) G(j,s) ( denote concatenation) 25

26 Example of BRG Code 1-bit 2-bit 3-bit 8p ring 8p hyper

27 Trade-off Among Different Networks Network Minimum latency Maximum Bw per roc Wires Switches Example Completely connected Constant Constant O(p*p) - - Crossbar Constant Constant O(p) O(p*p) Cray Bus Constant O(1/p) O(p) O(p) SGI Challenge Mesh O(sqrt p) Constant O(p) - Intel ASCI Red Hypercube O(log p) Constant O(p log p) - Sgi Origin Switched O(log p) Constant O(p log p) O(p log p) IBM S-2 27

28 Beowulf Cluster built with commodity hardware components C hardware (x86,alpha,owerc) Commercial high-speed interconnection (100Base-T, Gigabit Ethernet, Myrinet,SCI) Linux, Free-BSD operating system 28

29 Appleseed: owerc Cluster 29

30 Clusters of SM The next generation of supercomputers will have thousand of SM nodes connected. Increase the computational power of the single node Keep the number of nodes low New programming approach needed, MI+Threads (OpenMp,threads,.) ASCI White, CompaqSC, IBM S

31 Multithread Architectures The MTA system provides scalable shared memory, in which every processor has equal access to every memory location. No concerns about the layout of memory. Each MTA processor has up to 128 RISC-like virtual processors. Each virtual processor is a hardware stream with its own instruction counter, register set, stream status word and target and trap registers. A different hardware stream is activated every clock period. 31