Outline. Lecture 11: EIT090 Computer Architecture. Small-scale MIMD designs. Taxonomy. Anders Ardö. November 25, 2009

Size: px

Start display at page:

Download "Outline. Lecture 11: EIT090 Computer Architecture. Small-scale MIMD designs. Taxonomy. Anders Ardö. November 25, 2009"

Ralph Wiggins
5 years ago
Views:

1 Outline Anders Ardö EIT Electrical and Information Technology, Lund University 1 / 49 2 / 49 Taxonomy SISD (Single Instruction stream, Single Data stream) traditional uniprocessor SIMD (Single Instruction stream, Multiple Data stream) vector processors MISD (Multiple Instruction stream, Single Data stream) no commercial examples MIMD (Multiple Instruction stream, Multiple Data stream) multiprocessor Small-scale MIMD designs Symmetric shared memory MultiProcessors (SMP) with Uniform Memory Access time (UMA) and bus interconnect Often limited to processors Flynn (1966) 3 / 49 4 / 49

Distributed memory machines Uses an interconnection network to connect processor-memory nodes = NUMA Scalable to a large number of nodes Can be either shared or private address space Shared memory vs.

2 Distributed memory machines Uses an interconnection network to connect processor-memory nodes = NUMA Scalable to a large number of nodes Can be either shared or private address space Shared memory vs. Message-passing Message-passing: The programmer must explicitly distribute data No execution overhead between explicit communication Shared memory: The same data structures as in the sequential program can be used Shared memory access can lead to high communication overhead 5 / 49 6 / 49 The cache coherence problem A read operation from address X must see the latest value produced by a write to address X With several copies of X, this may be a problem Techniques: Hardware-based protocols: Transparent to the software system, but increases the complexity of the machine Software-based protocols: Requires the user/compiler to detect when it is safe to cache, but do not require sophisticated hardware. Hard to do = limited use Policies: Write-invalidate remove (invalidate) other processor s copy of a data item when it is written Write-update update other processor s copy of a data item when it is written 7 / 49 Cache Coherence Protocols Snooping Status for a block is stored in every cache that has a copy of the block. Caches monitor (snoop) the shared memory bus to update status and take actions. Popular with single shared memory. Directory based Status for a block is stored in one location (the directory). Messages used to update status. Popular with distributed shared memory. 8 / 49

3 Synchronization Consistency models Why synchronize? We need to know when it is safe for different processes to use shared data Issues for synchronization: How do we implement the LOCK operation? Uninterruptable instruction to fetch and update memory (atomic operation) User level synchronization operation using this primitive For large scale multiprocessors, synchronization can be a bottleneck; techniques to reduce contention and latency of synchronizations are needed Atomic exchange, Test-and-set, Fetch-and-add Sequential consistency Serializing Write operations must stall until performed! Relaxed consistency A relaxed consistency model allows memory operations to be observed out-of-order between synchronization operations Possible to obtain significant performance advantages 9 / / 49 Lecture 11 agenda Outline Chapter 4 in "Computer Architecture" 11 / / 49

4 TLP Thread Level Parallelism TLP types Allow multiple threads to share functional units of a processor. A modern processor often has more functional unit parallelism available than a single thread can use. Must be able to switch between threads FAST Must be able duplicate the independent state (PC, Register, page table) Example: Intel HyperThreading (support for 2 threads) Coarse multithreading thread switch on costly stalls Fine multithreading thread switch each instruction issue slot Simultaneous multithreading (SMT) several threads can issue instructions simultaneously (combines ILP and TLP) 13 / / 49 TLP in practice From: An Overview of OpenMP - Ruud van der Pas - Sun Microsystems NTU Talk January A first OpenMP example A code segment can be threaded by distributing its components (e.g., iterations in a loop) evenly across available cores. The threaded code segment will execute correctly only if the work in all its components is independent of each other. For-loop with independent iterations for (int i=0; i<n; i++) c[i] = a[i] + b[i]; For-loop parallelized using an OpenMP pragma #pragma omp parallel for for (int i=0; i<n; i++) c[i] = a[i] + b[i]; % cc -xopenmp source.c % setenv OMP_NUM_THREADS 5 % a.out 15 / 49 RvdP/V1 An Overview of OpenMP 16 / 49

From: An Overview of OpenMP - Ruud van der Pas - Sun Microsystems NTU Talk January 14 2009 Example 14 parallel execution TLP in practice OpenMP Application Program Interface documentation is 326

5 From: An Overview of OpenMP - Ruud van der Pas - Sun Microsystems NTU Talk January Example 14 parallel execution TLP in practice OpenMP Application Program Interface documentation is 326 pages! Automate Nema Labs FASThread Thread 0 i=0-199 a[i] + b[i] = c[i] Thread 1 i= a[i] + b[i] = c[i] Thread 2 i= a[i] + b[i] = c[i] Thread 3 i= a[i] + b[i] = c[i] Thread 4 i= a[i] + b[i] = c[i] for (i = 0; i < height_a; ++i) { for (j = 0; j < width_b; ++j) { sum = 0; for (k = 0; k < width_a; ++k) { a_value = a[i * width_a + k]; b_value = b[k * width_b + j]; sum += a_value * b_value; } c[i * width_b + j] = sum; } } RvdP/V1 An Overview of OpenMP 17 / / 49 Outline Multiprocessor categorization 19 / / 49

Clusters Popularity of clusters Loosely coupled desktop machines No shared memory High bandwidth, switch-based LAN Standard of-the-shelf components = cheap Easy to scale High availability... but.

6 Clusters Popularity of clusters Loosely coupled desktop machines No shared memory High bandwidth, switch-based LAN Standard of-the-shelf components = cheap Easy to scale High availability... but... A cluster with N nodes have N independent machines with N copies of the OS High administration cost... and... Major problem is power (servers and cooling) 21 / / 49 Cluster categorizations Cluster software High-availability (HA) clusters Load-balancing clusters Compute clusters Grid computing OSCAR Beowulf Windows Compute Cluster HPC - Linux cluster software Moab Cluster Builder High-Availability Linux... etc / / 49

Cluster - Pleiades Cluster back 25 / 49 Jeff Dean Google 26 / 49 Example Google-cluster Numbers Everyone Should Know L1 cache reference

Read 1 MB sequentially from memory Round trip within same datacenter seek Read 1 MB sequentially from disk Send packet CA->Netherlands->CA

5 ns 5 ns 7 ns 25 ns 100 ns 3,000 ns 20,000 ns 250,000 ns 500,000 ns 10,000,000 ns 20,000,000 ns 150,000,000 ns L1$ L2$ P P L1$ L1$ P One

7 Cluster - Pleiades Cluster back 25 / 49 Jeff Dean Google 26 / 49 Example Google-cluster Numbers Everyone Should Know L1 cache reference Branch mispredict L2 cache reference Mutex lock/unlock Main memory reference Compress 1K bytes with Zippy Send 2K bytes over 1 Gbps network Read 1 MB sequentially from memory Round trip within same datacenter seek Read 1 MB sequentially from disk Send packet CA->Netherlands->CA Architectural view of the storage hierarchy P 0.5 ns 5 ns 7 ns 25 ns 100 ns 3,000 ns 20,000 ns 250,000 ns 500,000 ns 10,000,000 ns 20,000,000 ns 150,000,000 ns L1$ L2$ P P L1$ L1$ P One server L1$ L2$ : 16GB, 100ns, 20GB/s : 2TB, 10ms, 200MB/s Local Rack Switch Local rack (80 servers) : 1TB, 300us, 100MB/s : 160TB, 11ms, 100MB/s Cluster Switch Cluster (30+ racks) 27 / 49 : 30TB, 500us, 10MB/s : 4.80PB, 12ms, 10MB/s 28 / 49

8 Example: Google Example: Google Web-pages (2009); 400+ million search queries a day (2006) Selection criteria: cost per query Cluster with PC s 80 servers per rack; ca 10 kw = cooling problems Servers commodity-class x86 PCs (best performance/$) Design principles Software reliability Replication for better throughput and availability Price/performance beats peak performance Using commodity PCs reduces the cost of computation Application characteristics Google indexing server (Pentium III, 2003) moderate CPI (1.1 out of a possible 3.0) not much exploitable ILP 5 % branch mispredict 0.4 % to 0.7 % L1 cache miss-rate lots of trivially parallelizable computation Promising emerging techniques: TLP, simultaneous multi-threading chip multiprocessors, CMP Use large number of inexpensive nodes Barroso, L.A., Dean, J., Hölzle, U.: Web Search for a Planet: The Google Cluster Architecture / / 49 Google design goals Outline Jeff Dean keynote at LADIS 2009: The 3rd ACM SIGOPS International Workshop on Large Scale Distributed Systems and Middleware servers 1000s of locations around the world Storage (10 18 bytes) and computation spans all locations Mix of strong and weak consistency across locations Automatic allocation of resources Move computation to where it s best done (capacity, price of power, cooling,...) 31 / / 49

9 Grid Grid computing Analogy: electric power grid The Grid refers to an infrastructure that enables the integrated, collaborative use of high-end computers, networks, databases, and scientific instruments owned and managed by multiple organizations. Grid applications often involve large amounts of data and/or computing and often require secure resource sharing across organizational boundaries, and are thus not easily handled by today s Internet and Web infrastructures. What we need is an infrastructure and standard interfaces capable of providing transparent access to all this computing power and storage space in a uniform way. distributed, large-scale cluster computing network-distributed parallel processing cluster of networked loosely coupled computers heterogeneous, and geographically dispersed lack of central control over the hardware major grid middlewares are Globus Toolkit, glite, and UNICORE 33 / / 49 Grid use Grid projects The user submits his request through a Graphic User Interface (GUI) just specifying high level requirements (the kind of application he wants to use, the operating system,...) and eventually providing input data The Grid finds and allocates suitable resources (computing systems, storage facilities,...) to satisfy the user s request The Grid monitors request processing The Grid notifies the user when the results are available and eventually presents them High Energy Physics; Large Hadron Collider; about 1 PetaByte per year Earth Observation: 100 GB raw images per day / / 49

10 Outline What is Cloud computing? "Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (for example, networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction." Draft NIST Working Definition of Cloud Computing / / 49 Cloud computing characteristics Cloud computing customers do not own the physical infrastructure pay only for resources used infrastructure "in the cloud" dynamically scalable - on demand resources reliable services delivered through data centers normally built on servers with virtualization technologies {software/platform/infrastructure} as a service critics of cloud computing cite its seemingly broad and vague definition. T. Sridhar: Cloud Computing, The Internet Protocol Journal, Vol. 12, No / / 49

Outline ASUS ESC 1000 desktop supercomputer ESC 1000 using Nvidia graphics processors 1.1 teraflops 3.

Quadro FX5800 GPU (240 CUDA parallel processing cores) http://news.softpedia.

shtml 41 / 49 42 / 49 Top 6 supercomputers Roadrunner Source: http://www.top500.

11 Outline ASUS ESC 1000 desktop supercomputer ESC 1000 using Nvidia graphics processors 1.1 teraflops 3.33GHz Intel LGA1366 Xeon W3580 CPU Nvidia processors 3 x Tesla c1060 Computing Processors = 960 graphics processing cores Quadro FX5800 GPU (240 CUDA parallel processing cores) 41 / / 49 Top 6 supercomputers Roadrunner Source: Site Computer Cores TFlops MW 1 Oak Ridge, US Jaguar XT LANL, US Roadrunner NICS Kraken XT FZJ, Germany Blue Gene NUDT, China Tianhe NASA, US Pleiades / / 49

12 Blue Gene Blue Gene 45 / 49 Blue Gene Blue Gene 46 / / / 49

13 Summary Thread Level Parallelism (TLP) - coarse, fine, SMT Clusters Loosely coupled Cooling Wiring GRID - distributed infrastructure and resources CLOUD - on demand resources 49 / 49

Multiprocessors and Thread-Level Parallelism. Department of Electrical & Electronics Engineering, Amrita School of Engineering

Multiprocessors and Thread-Level Parallelism Multithreading Increasing performance by ILP has the great advantage that it is reasonable transparent to the programmer, ILP can be quite limited or hard to