PART II. Parallel processing, threads,! messages and communications

Size: px

Start display at page:

Download "PART II. Parallel processing, threads,! messages and communications"

Ralph Williams
5 years ago
Views:

Workshop on High performance computing and GPGPU computing PART II Parallel processing,

1 Postgraduate course on Electronics and Informatics Engineering (M.Sc.) Training Course on Circuits Theory (prof. G. Capizzi)! Workshop on High performance computing and GPGPU computing PART II Parallel processing, threads,! messages and communications Dr. Christian Napoli, M.Sc.! Dpt. Mathematics and Informatics, University of Catania!!! -

2 Parallel processing Moore s Law and Parallel Processing!2

3 Moore s Law(s) CLOUD???! DISTRIBUITED! COMPUTING BOINC GRID GPGPU High Performance Computing!3

An interesting phenomenon that s known as Andy giveth, and Bill taketh away describe that, no matter how fast processors get, software consistently finds new ways to eat up the extra speed.

Most classes of applications have enjoyed free and regular performance gains for several decades, even without releasing new versions or doing anything special, because the CPU manufacturers

4 An interesting phenomenon that s known as Andy giveth, and Bill taketh away describe that, no matter how fast processors get, software consistently finds new ways to eat up the extra speed. Make a CPU ten times as fast, and software will usually find ten times as much to do (or, in some cases, will feel at liberty to do it ten times less efficiently). Most classes of applications have enjoyed free and regular performance gains for several decades, even without releasing new versions or doing anything special, because the CPU manufacturers (primarily) and memory and disk manufacturers (secondarily) have reliably enabled ever-newer and ever-faster mainstream systems. Clock speed isn t the only measure of performance, or even necessarily a good one, but it s an instructive one: We re used to seeing 500MHz CPUs give way to 1GHz CPUs give way to 2GHz CPUs, and so on. Today we re in the 3GHz range on mainstream computers.!4 The growth must eventually slow down and even end. Moore s Law(s) Free launch is over!

5 Parallel processing Moore s Law and Parallel Processing!5

6 Moore s Law(s)!6

7 Parallel processing Moore s Law and Parallel Processing!7

8 Parallel processing Moore s Law and Parallel Processing!8

9 Parallel processing Moore s Law and Parallel Processing How to join CPU feature growth and Moore s Law hot zone?!9

10 Parallel processing Moore s Law and Parallel Processing!10

11 Parallel processing Moore s Law and Parallel Processing!11

Once upon a time. 1945 - Neumann János Lajos model (CC) Wikimedia Commons / CC-SA-3.

12 Once upon a time Neumann János Lajos model (CC) Wikimedia Commons / CC-SA-3.0 In the late 40s John von Neumann (born Neumann János Lajos, since it was Hungarian) became a leading figure in the field of computing. He worked for the ENIAC project, when ENIAC was being modified to contain stored programs. Since the modified ENIAC was fully functional by 1948 and the EDVAC wasn't delivered to Ballistics Research Laboratory until 1949, one could argue that ENIAC was the first computer to use a stored program in production runs. John von Neumann also designed the instruction set or op codes for the modified ENIAC, and he should be given credit for this. The electronics of the new ENIAC ran 6 times slower, but this in no way degraded the ENIAC's performance, since it was still entirely I/O bound. On the other hand, complicated programs could be developed and debugged in days rather than weeks, which is one of the advantages of storing the entire program in the computers electronics. The crucial point is whether or not a new paper tape has to be produced, using a slow and error prone tape punch every time the program is altered in any way, and then read in mechanically. A program is typically modified many times before it reaches its final form. This architecture is the basis of modern computer design, unlike the earliest computers that were 'programmed' by altering the electronic circuitry. Although the single-memory, stored program architecture is commonly called von Neumann architecture as a result of von Neumann's paper, the architecture's description was based on the work of J. Presper Eckert and John William Mauchly, inventors of the ENIAC at the University of Pennsylvania.!12

Von Neumann s The term Von Neumann architecture, also known as the Von Neumann model or the Princeton architecture, derives from a 1945 computer architecture description by the mathematician and

This describes a design architecture for an electronic digital computer with subdivisions of a processing unit consisting of an arithmetic logic unit and processor registers, a control unit

13 Von Neumann s The term Von Neumann architecture, also known as the Von Neumann model or the Princeton architecture, derives from a 1945 computer architecture description by the mathematician and physicist John von Neumann and others, First Draft of a Report on the EDVAC. This describes a design architecture for an electronic digital computer with subdivisions of a processing unit consisting of an arithmetic logic unit and processor registers, a control unit containing an instruction register and program counter, a memory to store both data and instructions, external mass storage, and input and output mechanisms. The meaning of the term has evolved to mean a stored-program computer in which an instruction fetch and a data operation cannot occur at the same time because they share a common bus. This is referred to as the Von Neumann bottleneck and often limits the performance of the system.!13 (CC) Wikimedia Commons / CC-SA-3.0

14 Von Neumann s Central Processing Unit: the early ages IBM 650, (CC) Wikimedia Commons / CC-SA-3.0!14

15 Von Neumann s Model Von Neumann s MEMORY CU!! IR PC CPU!!!!! ALU I/O Central Processing Unit:!! - Arithmetic / Logic Unit!! - Control Unit:!!! = Program Counter!!! = Instruction Register!! Central Memory!! Input / Output devices!15

16 Von Neumann s!16

Von Neumann s Model Von Neumann s Von Neumann computer systems contain three main building blocks: the central processing unit (CPU), memory, and input/output devices (I/O).

17 Von Neumann s Model Von Neumann s Von Neumann computer systems contain three main building blocks: the central processing unit (CPU), memory, and input/output devices (I/O). These three components are connected together using the system bus. The most prominent items within the CPU are the registers: they can be manipulated directly by a computer program.!17

18 Von Neumann s Components of the Von Neumann s Model 1. Memory: Storage of information (data/program) 2. Processing Unit: Computation/Processing of Information 3. Input: Means of getting information into the computer. e.g. keyboard, mouse 4. Output: Means of getting information out of the computer. e.g. printer, monitor 5. Control Unit: Makes sure that all the other parts perform their tasks correctly and at the correct time.!18

19 Von Neumann s Components of the Von Neumann s Model Communication between memory and processing unit consists of two registers: Memory Address Register (MAR). Memory Data Register (MDR). To read, 1. The address of the location is put in MAR. 2. The memory is enabled for a read. 3. The value is put in MDR by the memory. To write, 1. The address of the location is put in MAR. 2. The data is put in MDR. 3. The Write Enable signal is asserted.!19 4. The value in MDR is written to the location specified.

20 CPU Data Path Von Neumann s Hardware units like ALU's, registers, memory, etc., are linked together into a data-path. The flow of bits around the data-path is controlled by the "gates" which allow the bits to flow (on) or not flow (off) through the data-path. The binary instructions (1 = on; 0 = off) that control the flow are called microinstructions.!20

!21 Von Neumann s Simplified x86 data path (1979 a.d.) The 8088, 8086, and 80286 are 16-bit CPUs. Internal registers are 16 bits in size.

21 !21 Von Neumann s Simplified x86 data path (1979 a.d.) The 8088, 8086, and are 16-bit CPUs. Internal registers are 16 bits in size. The 8086 is faster than the 8088 because of its 16-bit data bus; the 8088 has only an 8- bit data bus. The 16-bit data bus allows you to use EVEN and ALIGN on an 8086 processor to word-align data and thus improve datahandling efficiency. Memory addresses on the 8086 and 8088 refer to actual physical addresses. The 8086 and 8088 have 20 address pins, and 1 megabyte of addressable memory (which is the real mode segmented memory explained later) requires addresses of 20 bits in size.

22 The Intel generations The Intel 8080 core! 8086, (CC) Wikimedia Commons / CC-3.0!22

23 The Intel generations The Intel core! 80286, (CC) Wikimedia Commons / CC-3.0!23

24 The Intel generations The Intel PENTIUM! Pentium, (CC) Wikimedia Commons / CC-3.0!24

25 The Intel generations The Intel PENTIUM 4! Pentium, (CC) Wikimedia Commons / CC-3.0 For a more detailed history of CPUs:

26 :-) ONE ONLY SOLUTION! TO THE ENERGY DENSITY PROBLEM!26

27 Parallel processing False Parallelisms PROCESS 1 D! A! T! D! A! T! PROCESS 1 A PROCESS 2 A!27

28 Parallel processing False Parallelisms PROCESS 1 PROCESS 3 D! A! T! A PROCESS 2 PROCESS 3 D! A! T! A PROCESS 1!28

29 Parallel processing False Parallelisms D! A! A! PROCESS 1 PROCESS 2 PROCESS 14 T! A PROCESS 3 D! T! A!29

30 Pipeline D! A! A! PROCESS 1 PROCESS 2 PROCESS 4 T! A D! T! A TERMINAL! $ ls /usr/local/include grep mpi! mpi.h! mpicxx.h! mpif.h! mpio.h! mpiof.h!! $ mkdir newdir && cd newdir! $ ls /usr/local/include > new.txt && ls! new.txt!! $_ Broken Pipe!30 IT layout US layout

Pipeline Serial Execution, (CC) Wikimedia Commons /

0 Pipelined Execution, (CC) Wikimedia Commons / 0 IF!

Execution! MEM!Memory activation (not mandatory)! WB!

31 Pipeline Serial Execution, (CC) Wikimedia Commons / CC-BY-SA-3.0 Pipelined Execution, (CC) Wikimedia Commons / CC-BY-SA-3.0 IF!! ID!! EX! Instructions Fetch! Instructions Decode! Execution! MEM!Memory activation (not mandatory)! WB! Write Back Superscalar CPU with double pipeline,! (CC) Wikimedia Commons / CC-BY-SA-3.0!31

32 Pipeline MIPS Architecture!32

33 :-) MISLEADING CONCEPTS computers with pipelines!33

Parallel programming True Parallelisms SERIAL PROGRAMS Programs run on a single computer and/or CPU Problems are a discrete set of instructions

cores Problems are still a discrete set of instructions Problems instructions are spitted in parts Each part can be solved concurrently

34 Parallel programming True Parallelisms SERIAL PROGRAMS Programs run on a single computer and/or CPU Problems are a discrete set of instructions Instructions are executed one after another Only one instruction may execute at a moment PARALLEL PROGRAMS Programs run on a several CPUs and/or cores Problems are still a discrete set of instructions Problems instructions are spitted in parts Each part can be solved concurrently Instructions are then executed simultaneously An overall control mechanism is required Synchronizations becomes non-trivial Memory management policy often required!34

Parallel programming Very common parallel problems «Doh!

no effort is required to separate the problem into a number of several

and equally common design problems PCAM: a design methodology for parallel

35 Parallel programming Very common parallel problems «Doh!» In parallel computing an embarrassingly parallel problem is one for which no effort is required to separate the problem into a number of several parallel tasks. and equally common design problems PCAM: a design methodology for parallel programs!35 Starting with a problem specification, we develop a partition, determine communication requirements, agglomerate tasks, and finally map tasks to processors.

36 Threads & multithreading i = 0 i = 1 i = 2 i = 3 i = 4 s=vecmean (x,y) t = a[i]+b[i] t = a[i]+b[i] t = a[i]+b[i] t = a[i]+b[i] t = a[i]+b[i] t = t/2 t = t/2 t = t/2 t = t/2 t = t/2 s[i] = t s[i] = t s[i] = t s[i] = t s[i] = t In computer science, a thread of execution is the smallest sequence of programmed instructions that can be managed independently by an operating system scheduler. Multithreading is the ability of a program or an operating system process to manage its use by more than one user at a time and to even manage multiple requests by the same user without having to have multiple copies of the programming running in the computer i = 0 t = a[i]+b[i] t = t/2 s[i] = t i = 1 t = a[i]+b[i] t = t/2 s[i] = t i = 2 t = a[i]+b[i] t = t/2 s[i] = t i = 0 t = a[i]+b[i] t = t/2 s[i] = t i = 6 t = a[i]+b[i] t = t/2 s[i] = t i = 12 t = a[i]+b[i] t = t/2 s[i] = t i = 1 t = a[i]+b[i] t = t/2 s[i] = t i = 7 t = a[i]+b[i] t = t/2 s[i] = t i = 13 t = a[i]+b[i] t = t/2 s[i] = t i = 2 t = a[i]+b[i] t = t/2 s[i] = t i = 8 t = a[i]+b[i] t = t/2 s[i] = t i = 14 t = a[i]+b[i] t = t/2 s[i] = t i = 3 t = a[i]+b[i] t = t/2 s[i] = t i = 9 t = a[i]+b[i] t = t/2 s[i] = t i = 15 t = a[i]+b[i] t = t/2 s[i] = t i = 4 t = a[i]+b[i] t = t/2 s[i] = t i = 10 t = a[i]+b[i] t = t/2 s[i] = t i = 5 t = a[i]+b[i] t = t/2 s[i] = t i = 11 t = a[i]+b[i] t = t/2 s[i] = t!36!36!36

37 SINGLE TRHEAD Threads & CPU cores MULTITRHEAD MULTICORE SINGLE CORE!37

If the systems adopt a FIFO policy each time a thread finishes to compute it

38 Threads & CPU cores That s how multithread CPUs should look like But let suppose that each thread wants to write on the same memory positions. If the systems adopt a FIFO policy each time a thread finishes to compute it must wait the others before to write back the results Is that a good parallel execution?!38

39 Threads & troubles Suppose that each thread receives inputs and gives back outputs which are needed by all treads before to continue THREAD 1 THREAD 1 D! A! T! A TREAD 2 THREAD 3 D! A! T! A TREAD 2 THREAD 3 D! A! T! A THREAD 4 THREAD 4 The beginning, synchronization and termination of a multithread computation are called FORK, SYNC and JOIN!39

40 Threads & troubles The beginning, synchronisation and termination of a multithread computation are called FORK, SYNC and JOIN THREAD 1 THREAD 1 F! O! R! K TREAD 2 THREAD 3 S! Y! N! C TREAD 2 THREAD 3 J! O! I! N THREAD 4 THREAD 4 But what if some thread has to do more work or has latency?!40

41 Threads & troubles But what if some thread has to do more work or has latency? THREAD 1 THREAD 1 F! O! R! K TREAD 2 THREAD 3 S! Y! N! C TREAD 2 THREAD 3 J! O! I! N THREAD 4 THREAD 4 The early returning threads should wait the others because fork, sync and join are BARRIERS for the execution!41

42 Threads & troubles The early returning threads should wait the others because fork, sync and join are BARRIERS for the execution While waiting the threads does not execute any work wasting useful CPU time, energy and resources.!42

43 Threads & troubles The early returning threads should wait the others because fork, sync and join are BARRIERS for the execution Wasted CPU time: communication idle time waiting for barriers It is the overall time wasted by any single thread during the synchronization processes (communication) and the idle time and it is an effective part of what it is called total overhead!43

44 Threads & troubles Better to avoid unnecessary barriers: Wasted CPU time: idle time waiting for barriers A good distribution of the work among the threads is called balanced multithreading and aims to minimise the idle time.!44

Amdahl's law, also known as Amdahl's argument, is used to find the maximum expected improvement to an overall system when only part of the system is improved.

45 serial part Almost every problem has a serial portion F! O! R! K thread 1 thread 2 thread 3 thread 4 Andahl s Law S! Y! N! C thread 1 thread 2 thread 3 thread 4 serial part How worthy it is then to parallelise a problem? Amdahl's law, also known as Amdahl's argument, is used to find the maximum expected improvement to an overall system when only part of the system is improved. It is often used in parallel computing to predict the theoretical maximum speedup using multiple processors. The law is named after computer architect Gene Amdahl, and was presented at the AFIPS Spring Joint Computer Conference in J! O! I! N Δ t t = N N S + P = 1 S + P N P parallel portion N number of cores S=1 P!45 (CC) Wikimedia Commons / CC-SA-3.0

46 Bottlenecks in threads Bottlenecks should be taken into the overall count LOW! LOAD MEDIUM! LOAD HIGH! LOAD TOO! HIGH! LOAD APPLICATION! CAPABILITY!46

47 When the damn code doesn t want to speedup :-)!47 someone searches answers in the Big Book of Bottlenecks

48 Message Passing Immagine to demand several people to perform some tasks independently and asking to be called by the first who finishes ALICE BOB CHARLIE WINNER DANA Let now to complicate the problem. What if:! Task 2 of Charlie needs an output by task 2 of Dana Task 3 of Alice needs an output by task 4 of Bob!48

49 Message passing! Task 2 of Charlie needs an output by task 2 of Dana Task 3 of Bob needs an output by task 4 of Alice ALICE WINNER BOB CHARLIE DANA Alice sent a message to Bob which was expecting it! Also Charlie was expecting a message from Dana who sent it What if Dana couldn t sent the message?!49

50 Message passing What if Dana couldn t sent the message? ALICE WINNER BOB CHARLIE 1 2 DANA Alice sent a message to Bob which was expecting it! Also Charlie is stuck expecting a message from Dana Maybe Dana failed to send the message, or, maybe, bob was not listening for the message. Anyway how does it applies to HPC?!50

51 Message passing Message Passing among processes 1 Get(a) ab=a*b aab=a*ab 2 1 Get(b) bb=b*b. bbabb = bb*aab 2 Get(c) 3 wait() 3 c2=c*c The green checkpoints represent the barriers when certain processes synchronizes themselves by exchanging messages. Can t the processes access directly the values of the needed variables directly by reading them on the memory?!51

52 Message passing Can t the processes access directly the values of the needed variables directly by reading them on the memory? Only if the process are running within a shared memory paradigm But what id those processes are running on different machines which are linked with a network? In this case each process knows only the memory of his machine: the values of non local variables have to be passed as a message trough the network itself.!52

53 Message passing MESSAGE PASSING INTERFACE Message Passing Interface (MPI) is a standardized and portable message-passing system.the standard defines the syntax and semantics of a core of library routines useful to a wide range of users writing portable message-passing programs in Fortran or the C programming language. There are several welltested and efficient implementations of MPI, including some that are free or in the public domain. These fostered the development of a parallel software industry, and there encouraged development of portable and scalable large-scale parallel applications.!53

operating systems, including Solaris, AIX, HP-UX, GNU/Linux, Mac OS X, and Windows platforms.

54 Parallel programming!54 Open MultiProcessing Open MultiProcessing (OpenMP) is an API that supports multi-platform shared memory multiprocessing programming in C, C++, and Fortran, on most processor architectures and operating systems, including Solaris, AIX, HP-UX, GNU/Linux, Mac OS X, and Windows platforms. It consists of a set of compiler directives, library routines, and environment variables that influence run-time behavior. OpenMP uses a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications for platforms ranging from the standard desktop computer to the supercomputer. An application built with the hybrid model of parallel programming can run on a computer cluster using both OpenMP and Message Passing Interface (MPI), or more transparently through the use of OpenMP extensions for non-shared memory systems.

55 Hybrids Distributed GPU Programming Hybrid MPI/OpenMP!55

56 Hybrids Last frontier: Hybrid MPI/OpenMP + Distributed GPU Programming!56

57 QUESTION TIME CAN YOU REPEAT PLEASE??!57

58 Workshop on High Parallel Performance Architectures Computing and Programming and GPGPU Techniques Computing - PART - I II Thank You You will find the PDF edition in the didactic section of the author s website, visit To contact the author send an to If you want to share this presentation be sure to read and follow the CC-BY-NC-ND-4.0 license. Visit

PART III. GPU Cards and Architectures. Dr. Christian Napoli, M.Sc.! Dpt. Mathematics and Informatics, University of Catania!

PART III. GPU Cards and Architectures. Dr. Christian Napoli, M.Sc.! Dpt. Mathematics and Informatics, University of Catania! Postgraduate course on Electronics and Informatics Engineering (M.Sc.) Training Course on Circuits Theory (prof. G. Capizzi)! Workshop on High performance computing and GPGPU computing Postgraduate course