Online Course Evaluation. What we will do in the last week?

Size: px

Start display at page:

Download "Online Course Evaluation. What we will do in the last week?"

Brian Watts
5 years ago
Views:

1 Online Course Evaluation Please fill in the online form The link will expire on April 30 (next Monday) So far 10 students have filled in the online form Thank you if you completed it. 1 What we will do in the last week? Chapter 6: Parallel processors Today and Thursday Also, on Thursday: Give you a study guide to prepare for the Final Exam Non-cumulative (only 2 nd half will be tested) Will give more details on Thursday Final Exam date and time: Thursday on May 3, 3:30pm- 4:45pm I ll provide office hours next Tuesday, May 1 st : 3pm-5pm Use this opportunity to ask more questions before the exam 2 1

2 CSCI 402: Computer Architectures Chapter 6: Parallel Processors (1) Fengguang Song Department of Computer & Information Science IUPUI Today s Contents What is parallel processing Hardware Multithreading Multicore processors Classic Cluster Organization 4 2

3 Introduction Multiprocessor Machine? A computer system with at least two processors (VS Uniprocessor Machine ) The Goal: To connect multiple computers to get higher performance for improving: Scalability, Availability, Power efficiency We are now in the multicore era Type 1: Task-level parallelism The goal is to obtain high throughput for independent jobs Type 2: Parallel processing program To speed up a single program that runs on multiple processors Cluster : A set of computers connected over a local area network It can serve as search engines, web servers, databases, etc. Multicore microprocessor : A CPU containing multiple cores in a single die/chip/socket. 5 State of the Art Why do we care about Parallel Processing? All processors today are multicore! NumberOfCores is expected to increase constantly in the future! We expect to see 2 additional cores per chip every two years Almost all machines today are SMP: Shared Memory Processors Hence, any programmer who cares about speed must become Parallel Programmer Before 2004, you don t have to care Today, sequential programs are considered slow Unfortunately, no easy software tool or language to write correct and fast parallel programs (a challenging and time-consuming task) reason for software developers high salary? 6 3

4 Commonly Used Terms in Parallel Computing Hardware Serial: e.g., Pentium 4 Parallel: e.g., quad-core Xeon e5345 Software Sequential: e.g., matrix multiplication Concurrent: e.g., operating system Sequential and Concurrent software can run on serial and parallel hardware Challenge: How to making effective use of parallel hardware 7 We Have Already Seen Parallelism Before 2.11: Instructions Synchronization instructions (to do lock/unlock) 3.6: Computer Arithmetic Subword Parallelism (SIMD) 4.10: Instruction-Level Parallelism (ILP) Pipelining, multiple issues 5.10: Memory Hierarchies Cache Coherence (how to make sure every core reads the most recently written value) 8 4

5 Parallel Programming First of all, Parallel Programming is a challenging task The difficulty of parallelism is Not in hardware (e.g., adding more CPU cores) Parallel software is the problem It is rather difficult to use multiple processors to compute one task faster The goal is to get significant performance improvement Otherwise, just use a uniprocessor // since it s easier :) Difficulties in parallel software: Partitioning the problem Coordination Communications overhead Load balancing Fault tolerance Power/energy locality 9 Amdahl s Law The sequential part of your program will limit your speedup on parallel computers Question: 100 processors, how to get 90 speedup? T old = T parallelizable + T sequential T new = T parallelizable /#P + T sequential Speedup = 1 (1 F parallelizable ) +F parallelizable / #P = 90 Solving: F parallelizable = 99.9% Need sequential part to be <= 0.1% of total time T old There are such applications with plenty of parallelism 10 5

1 st Scaling Example An example workload: sum of 10 scalars, and sum of a pair of 10 10 matrices Assuming scalars cannot benefit from parallelism, but matrix can benefit Q:

On a single processor: Time = (10 + 100) t add = 110 t add 10 processors: Parallel Efficiency Time = 10 t add + 100/10 t add = 20 t add Speedup = 110t/20t = 5.

) 11 2nd Scaling Example Now, we make matrix size bigger: 100 100?

6 1 st Scaling Example An example workload: sum of 10 scalars, and sum of a pair of matrices Assuming scalars cannot benefit from parallelism, but matrix can benefit Q: What are the Speedups if we use 10 and 100 processors? On a single processor: Time = ( ) t add = 110 t add 10 processors: Parallel Efficiency Time = 10 t add + 100/10 t add = 20 t add Speedup = 110t/20t = 5.5 (55% of potential) Q: What if P = Infinity? 100 processors: Time = 10 t add + 100/100 t add = 11 t add Speedup = 110t/11t = 10 (Only 10% of potential only!) 11 2nd Scaling Example Now, we make matrix size bigger: ? On a single processor: Time = ( ,000) t add 10 processors: Time = 10 t add + 10,000/10 t add = 1,010 t add Speedup = 10,010t/1,010t = 9.9 (99% of potential) 100 processors: Time = 10 t add + 10,000/100 t add = 110 t add Speedup = 10,010/110 = 91 (91% of potential) Parallel Efficiency Lesson: Getting good speed-up on a fixed problem size is harder than by increasing the problem size. previous example: this example: 10 10,

7 Strong vs Weak Scaling Strong scaling: the problem size is fixed As in the first example Weak scaling: the problem size is proportional to number of processors (2x, 3x, 4x, ) E.g., 10 processors, matrix //100 ops on 10 proc Time = 10 t add + 100/10 t add = 20 t add 100 processors, sqrt(1000) sqrt(1000) matrix //1000 ops on 100 processors (still 10 operations per processor!) Time = 10 t add /100 t add = 20 t add Constant execution time! In fact, most often people solve bigger problems on bigger computers 13 A classification of parallel hardware #Instruction Streams Single Multiple Single SISD: Intel Pentium 4 MISD: No example today #Data Streams Multiple SIMD: SSE(?) instructions of x86 MIMD: threads on Intel Core i7 n SPMD parallel programs: Single Program Multiple Data n A single parallel program on a MIMD computer n Conditional code for different processors 15 7

8 SIMD Operate element-wise on vectors of data E.g., MMX, SSE, AVX instructions in x86 Store multiple data elements in 256-bit wide registers All processors (or function units) can execute the same instruction at the same time Each with different data address Also called data-parallel computing Works best for highly data-parallel applications 16 Another SIMD Example: Vector Processors Popular in 70s-80s Cray-1, MHz Scalar Unit + Vector Extensions Has a Load/Store Architecture Vector Registers Vector Instructions Hardwired Control Highly Pipelined Functional Units Interleaved Memory System No Data Caches No Virtual Memory 17 8

Vectors vs. Multimedia Extensions (such as MMX, SSE, AVX) Vector instructions have a variable vector width; Multimedia extensions have a fixed width.

9 Vectors vs. Multimedia Extensions (such as MMX, SSE, AVX) Vector instructions have a variable vector width; Multimedia extensions have a fixed width. Vector Length is stored in a register Vector instructions support strided access; Multimedia extensions do not support it. Vector units can be a combination of pipelined and arrayed functional units: 18 Vector and SIMD Instruction s Advantages Compact one short instruction encodes N operations Expressive, tells hardware that these N operations: are independent can use the same functional unit access disjoint registers access registers in the same pattern as previous instructions access memory in a known pattern (strided load/store) Scalable can run same object code on more parallel lanes 20 9

10 MIMD and Hardware Multithreading Hardware multithreading is a concept related to MIMD on a single core Create N threads running on a processor In order to increase resource utilization of a single core It is able to perform multiple threads of execution in parallel CPU has replicated registers, PCs Support fast switching between threads 3 Versions: Fine-grain multithreading Switch threads after each cycle Interleave instruction execution (often round-robin) If one thread stalls, others will be executed Con: a normal individual thread will be delayed by other threads instructions Coarse-grain multithreading Only switch on long pipeline stall (e.g., L2-cache misses) Simplifies hardware, but does not hide short stalls (e.g., data hazards, load-use) SMT (see the next slide) 25 Simultaneous Multithreading (SMT) Most commonly used in modern multiple-issue dynamically scheduled processor Can simultaneously schedule instructions from multiple threads No thread switching in every cycle Instructions from independent threads will execute whenever there are available function units Within each thread, dependencies are handled by scheduling and register renaming Example: Intel Pentium4 has HypterThreading Support two threads: duplicated registers, has shared function units and caches 26 10

11 Hardware Multithreading Examples stalls 27 MIMD and Shared Memory Multiprocessor (SMP) Offers a single physical address space across all processors (different from clusters) SMP: Synchronize access to shared variables with locks Memory access time: UMA (uniform) vs. NUMA (non-uniform) 29 11

12 4/24/18 30 The latest 7th generation Kaby Lake has a similar structure:

A Parallel Programming Example: Sum of an Array of Integers Problem: To sum 100,000 numbers on 100 processor on SMP Each processor has an ID, e.g., 0 Pn 99 1,000 numbers per processor First, partial sum by every processor (process Pn) sum[pn] = 0; //Pn=0, 1,2,,99 for (i = 1000*Pn; i < 1000*(Pn+1); ++i) //e.

reduction steps 33 Example: Sum Reduction size size=8 procs;//could be 9,10 size repeat barrier();//all processors wait till everyone comes here if (size % 2!

13 A Parallel Programming Example: Sum of an Array of Integers Problem: To sum 100,000 numbers on 100 processor on SMP Each processor has an ID, e.g., 0 Pn 99 1,000 numbers per processor First, partial sum by every processor (process Pn) sum[pn] = 0; //Pn=0, 1,2,,99 for (i = 1000*Pn; i < 1000*(Pn+1); ++i) //e.g., P0, P1, P2, sum[pn] = sum[pn] + A[i]; Next, need to add up those partial sums A reduction op (P numbers -> 1 number) Half the processors add pairs, then quarter, Need to synchronize between reduction steps 33 Example: Sum Reduction size size=8 procs;//could be 9,10 size repeat barrier();//all processors wait till everyone comes here if (size % 2!= 0 && Pn == 0) sum[0]=sum[0] + sum[size-1]; If the size is not even. /* Only when size is odd, then P0 picks up the last number*/ size = size / 2; /* the dividing line on who will do sums*/ if (Pn < size) sum[pn] = sum[pn] + sum[pn+size]; until (size == 1); size 34 13

CS 590: High Performance Computing. Parallel Computer Architectures. Lab 1 Starts Today. Already posted on Canvas (under Assignment) Let s look at it

Lab 1 Starts Today Already posted on Canvas (under Assignment) Let s look at it CS 590: High Performance Computing Parallel Computer Architectures Fengguang Song Department of Computer Science IUPUI 1