COL 380: Introduc1on to Parallel & Distributed Programming. Lecture 1 Course Overview + Introduc1on to Concurrency. Subodh Sharma

Transcription

1 COL 380: Introduc1on to Parallel & Distributed Programming Lecture 1 Course Overview + Introduc1on to Concurrency Subodh Sharma Indian Ins1tute of Technology Delhi

2 Credits Material derived from Peter Pacheco: An Introduc1on to Parallel Programming Grama, Gupta, Karypis, Kumar: Introduc1on to Parallel Compu1ng Herlihy & Shavit: The Art of Mul1processor Programming Hennesy & PaNerson: Computer Architecture

3 Course Overview Goal: Learn fundamentals of concurrency Architecture, Memory consistency, latency/throughput, models of computa1on... Learn to analyze, develop and debug parallel or distributed program Exposure to common programming paradigms MPI, OpenMP, CUDA etc.

4 Other Course Details! Look at the website: Course TAs: Divyanshu, Neha, Dipanjan: PhD candidates Raunak, Mohit, Abhishek: MS candidates Tenta1vely, you can find them in the Verifica1on Lab!

5 Other Course Details! 4-6 wrinen assignments, 4-5 lab assignments Programming intensive! Everyone should work individually Pre-requisi1es: data structures, algorithms and possibly OS!

6 Books to be referred Ananth Grama, Georg Karypis, Vipin Kumar and Anshul Gupta: Introduc1on to Parallel Compu1ng Peter S. Pacheco: An Introduc1on to Parallel Programming Maurice Herlihy and Nir Shavit: The Art of Mul1processor Programming

7 Intro to Concurrency

8 Concurrency Defini1on! Dic1onary: Simultaneous occurrence Wikipedia: Computa1ons are execu1ng concurrently More aptly: mul1ple tasks in progress simultaneously

9 Reasons for Adop1on of Concurrency? Incessant need for computa1onal power Decoding genes, accurate medical imaging, fast and accurate web search.. Concrete case: Weather forecas1ng for an area of 10^9 sq. miles divided into 1x1x1 cells i.e. 10^9 cells suppose each calcula1on requires 1000 Flops => to model 7 day weather at 1 minute interval requires ~115 days on a 10 Gflops machine!

10 Plateauing Moore s Curve Smaller transistors = faster processors Clock rates increased from 40 Mhz to 2 Ghz in 1.4 decade Faster processors = increased power consumption Increased power consumption = increased heat Already at limits to dissipate heat! Increased heat = unreliable processors. DRAM access time remains a bottleneck (~10%)

11 How Concurrency Helps? More processors, slower clock Share memory, mul1ple instruc1on issues Faster interconnects Re-write serial program to now run in parallel But wait.. why can t we build compilers to do that?

12 Automa1c Paralleliza1on Success is limited! Example: Find all primes from 1 to Given 10 processors to perform the task! One solu1on - Split the work evenly! Each thread tests a range of 10 9 void primeprint { int i = ThreadID.get(); // IDs in {0..9} for (j = i* , j<(i+1)*10 9 ; j++) { if (isprime(j)) print(j); } }

13 Example -- Issues Higher ranges have fewer primes! Larger numbers are harder to test uneven workloads Architecture insights?

14 Common forms of Concurrency Shared-memory Concurrency Distributed-memory Concurrency

15 Forms of Concurrency in Applica1ons? Task Parallelism Par11on tasks on to separate cores Data parallelism Par11on the data used in solving the problem among the cores Each core carries out similar opera1ons on it s part of the data

16 Example Task and Data Parallelism

17 Example Task and Data Parallelism

18 How Does H/w Exploit Forms of Concurrency? ILP: Instruc1on level parallelism via pipelining and specula1ve execu1on! Vector Architectures, GPUs: applying single instruc1on to a collec1on of data Thread-level Parallelism: Exploits both data and task parallelism Request-level Parallelism: Decoupled tasks

19 Flynn s Classifica1on SISD: Uniprocessor SIMD: Vector processors MISD: No commercial processor of this kind! MIMD: Mul1cores/Mul1processors

20 Differences between Parallel, Distributed, Concurrent? Concurrency: Defined earlier! Parallelism: Task are progressing simultaneously on physically different cores but coordina-ng closely. Distributed: Same as parallelism but loosely coupled.

21 The Present & Future Today Check the website for supplementary reading, Intro to Concurrency Pacheco s An Introduc1on to Parallel Programming sec1ons 1.2 to 1.6 Next Architecture basics leading to Parallelism in H/w Future Threads, Coordina1on, models of computa1on etc.