Scientific Programming in C XIV. Parallel programming

Transcription

1 Scientific Programming in C XIV. Parallel programming Susi Lehtola 11 December 2012

2 Introduction The development of microchips will soon reach the fundamental physical limits of operation quantum coherence thermal loads Clock rates (MHz GHz) will cease to increase Instead, new chips have an ever increasing amount of processor units Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 2/27

3 Parallel computing Using many processors requires specific techniques need to choose parallel algorithms High-performance parallel computing has been an everyday thing in physics for decades and will be ever more important in the future Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 3/27

4 Approaches There are two fundamentally different approaches to parallel computing shared memory parallellization distributed memory parallellization Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 4/27

5 Shared memory CPU1 CPU2 CPU3 CPU4 Memory As the name states, the memory is shared between the processors Don t need to worry about communication Only problem is to make sure that processes do not overlap... and that a parallellizable algorithm is used can be as easy as inserting a couple simple lines into the source code OpenMP Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 5/27

6 Distributed memory Memory CPU1 Memory CPU3 Network CPU4 Memory CPU2 Memory Every processor has its own memory No implicit access to other processors memory Need to handle communication between processes manually Message Passing Interface (MPI) standard Can also run on shared-memory systems Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 6/27

7 Hybrid parallellization When going to very large calculations, the overhead from MPI communication may become significant can be reduced by using shared memory parallellization within the cluster nodes better memory use due to less data duplication CPU1 CPU2 CPU3 CPU4 Memory CPU9 CPU10 CPU11 CPU12 Memory Network CPU13 CPU14 CPU15 CPU16 Memory CPU5 CPU6 CPU7 CPU8 Memory Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 7/27

8 Parallel programming As in optimization, it only makes sense to parallellize the CPU-intensive part Distributed memory parallellization can be also performed to get a larger amount of memory for the program Once you have a parallel algorithm, the implementation is simple If time is mostly spent on library routines, you can achieve speedups by using a parallellized version of the library shared-memory parallellized BLAS & LAPACK in, e.g., OpenBLAS and Intel MKL Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 8/27

9 Parallel programming, cont d Distributed memory programming using MPI is discussed on the course Tools for high-performance computing (Suurteholaskennan työkalut) lectured every other year next time in fall 2013 In the following I will give a short introduction to shared-memory parallellization using OpenMP Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 9/27

10 OpenMP The OpenMP API supports multi-platform shared-memory parallel programming in C/C++ and Fortran similar interface in C and Fortran defines a portable, scalable model with a simple and flexible interface for developing parallel applications on platforms from the desktop to the supercomputer Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 10/27

11 OpenMP, cont d Parallellizing with OpenMP can be as easy as changing f o r ( i =0; i <N; i ++) y [ i ]= f ( x [ i ] ) ; to #pragma omp p a r a l l e l f o r f o r ( i =0; i <N; i ++) y [ i ]= f ( x [ i ] ) ; and compiling the program with the -fopenmp flag (GCC). Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 11/27

12 Conditional compilation Since parallellization using OpenMP generally only induces very small changes to the program, it s wise to use conditional compilation when using OpenMP #i f d e f OPENMP #pragma omp p a r a l l e l f o r #e n d i f f o r ( i =0; i <N; i ++) y [ i ]= f ( x [ i ] ) ; If the -fopenmp flag is not given to the compiler, you will get the serial version of the code If it is given, you will get the parallel version of the code Parallellization induces some overhead, so use the serial version if you only want to use a single CPU core Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 12/27

13 Conditional compilation, cont d Test program: #i n c l u d e <s t d i o. h> #i f n d e f OPENMP #d e f i n e OPENMP 0 #e n d i f i n t main ( v o i d ) { p r i n t f ( %i \n, OPENMP ) ; r e t u r n 0 ; } Compile: $ gcc t e s t. c o s e r i a l. x $ gcc fopenmp t e s t. c o p a r a l l e l. x Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 13/27

14 Conditional compilation, cont d On Fedora 18: $. / s e r i a l. x 0 $. / p a r a l l e l. x $ gcc v e r s i o n gcc (GCC) ( Red Hat ) C o p y r i g h t (C) 2012 Free S o f t w a r e Foundation, I n c. On Red Hat Enterprise 5: $. / p a r a l l e l. x $ gcc v e r s i o n gcc (GCC) ( Red Hat ) C o p y r i g h t (C) 2006 Free S o f t w a r e Foundation, I n c. The OPENMP macro is defined as the date of the OpenMP standard implemented in the compiler (3.0 and 2.5, respectively). Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 14/27

15 Shared memory parallellization workflow Serial code thread 1 thread 2 thread 3 thread 4 Parallel code Serial code Parallellization only used in compute-intensive parts Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 15/27

16 OpenMP workflow A parallel code segment in OpenMP begins with #pragma omp p a r a l l e l At this moment the program will launch the set amount of threads defaults to the amount of cores available on the system if not overrided with OMP NUM THREADS environment variable void omp set threads (int nth) function in the code Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 16/27

17 OpenMP hello world #i n c l u d e <s t d i o. h> #i f d e f OPENMP #i n c l u d e <omp. h> #e n d i f i n t main ( v o i d ) { #i f d e f OPENMP / Amount o f cpus / c o n s t i n t ncpu=omp get num procs ( ) ; / Maximum number o f t h r e a d s / c o n s t i n t maxth=o m p g e t m a x t h r e a d s ( ) ; #e l s e c o n s t i n t ncpu =1; c o n s t i n t maxth =1; #e n d i f p r i n t f ( %i p r o c e s s o r s i n system, %i t h r e a d s a l l o w e d. \ n,\ ncpu, maxth ) ; Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 17/27

18 OpenMP hello world, cont d #i f d e f OPENMP #pragma omp p a r a l l e l #e n d i f { / P a r a l l e l r e g i o n / #i f d e f OPENMP / Amount o f t h r e a d s / c o n s t i n t nth=omp get num threads ( ) ; / I n d e x o f c u r r e n t t h r e a d / c o n s t i n t i t h=omp get thread num ( ) ; #e l s e c o n s t i n t nth =1; c o n s t i n t i t h =0; #e n d i f } } p r i n t f ( H e l l o from t h r e a d %i /% i. \ n, i t h, nth ) ; Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 18/27

19 OpenMP hello world, cont d $. / s e r i a l. x 1 p r o c e s s o r s i n system, 1 t h r e a d a l l o w e d. H e l l o from t h r e a d 0 / 1. $. / p a r a l l e l. x 8 p r o c e s s o r s i n system, 8 t h r e a d s a l l o w e d. H e l l o from t h r e a d 7 / 8. H e l l o from t h r e a d 1 / 8. H e l l o from t h r e a d 2 / 8. H e l l o from t h r e a d 0 / 8. H e l l o from t h r e a d 3 / 8. H e l l o from t h r e a d 5 / 8. H e l l o from t h r e a d 4 / 8. H e l l o from t h r e a d 6 / 8. $. / p a r a l l e l. x 8 p r o c e s s o r s i n system, 8 t h r e a d s a l l o w e d. H e l l o from t h r e a d 5 / 8. H e l l o from t h r e a d 3 / 8. H e l l o from t h r e a d 1 / 8. H e l l o from t h r e a d 4 / 8. H e l l o from t h r e a d 6 / 8. H e l l o from t h r e a d 2 / 8. H e l l o from t h r e a d 0 / 8. H e l l o from t h r e a d 7 / 8. Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 19/27

20 OpenMP hello world, cont d $ e x p o r t OMP NUM THREADS=4 $. / a. out 8 p r o c e s s o r s i n system, 4 t h r e a d s a l l o w e d. H e l l o from t h r e a d 0 / 4. H e l l o from t h r e a d 2 / 4. H e l l o from t h r e a d 3 / 4. H e l l o from t h r e a d 1 / 4. $. / a. out 8 p r o c e s s o r s i n system, 4 t h r e a d s a l l o w e d. H e l l o from t h r e a d 2 / 4. H e l l o from t h r e a d 0 / 4. H e l l o from t h r e a d 1 / 4. H e l l o from t h r e a d 3 / 4. Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 20/27

21 OpenMP hello world, cont d variables declared within the parallel block are private to each thread variables declared outside the parallel block (but within scope) are shared by default can be declared as private execution order is not guaranteed! may lead to race conditions if shared variables are modified Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 21/27

22 Race condition #i n c l u d e <omp. h> #i n c l u d e <s t d i o. h> i n t main ( v o i d ) { i n t i =0; #pragma omp i ++; p a r a l l e l } p r i n t f ( i=%i \n, i ) ; $ f o r ( ( i =0; i <8; i ++)); do. / a. out ; done i =7 i =8 i =8 i =6 i =8 i =7 i =7 i =6 A race condition occurs, leading to an incorrect result. Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 22/27

23 Race condition, cont d Correct functionality 1. thread A reads i=0 2. thread A increments i by 1 3. thread A saves i=1 to memory 4. thread B reads i=1 5. thread B increments i by 1 6. thread B saves i=2 Race condition: 1. thread A reads i=0 2. thread B reads i=0 3. thread A increments i by 1 4. thread A saves i=1 5. thread B increments i by 1 6. thread B saves i=1 Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 23/27

24 Race condition, cont d In this case, the race condition can be prevented in multiple ways. #pragma omp p a r a l l e l r e d u c t i o n (+: i ) i ++; Each thread calculates the final value of i, and the results are summed together Natural example of reduction: summation of large array #i n c l u d e < s t d l i b. h> double sum ( c o n s t double x, s i z e t N) { s i z e t i ; double r e s =0.0; #i f d e f OPENMP #pragma omp p a r a l l e l f o r r e d u c t i o n (+: r e s ) #e n d i f f o r ( i =0; i <N; i ++) r e s+=x [ i ] ; r e t u r n r e s ; } Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 24/27

25 Race condition, cont d #pragma omp #pragma omp i ++; p a r a l l e l atomic the memory location of i is protected against multiple writes Example: e x t e r n s i z e t i n d e x [ ] ; e x t e r n f l o a t a [ ], p = a, b ; / P r o t e c t a g a i n s t r a c e s among m u l t i p l e u p d a t e s. / #pragma omp atomic a [ i n d e x [ i ] ] += b ; / P r o t e c t a g a i n s t r a c e s w i t h u p d a t e s t h r o u g h a. / #pragma omp atomic p [ i ] = 1. 0 f ; Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 25/27

26 Race condition, cont d #pragma omp #pragma omp i ++; p a r a l l e l c r i t i c a l only one thread is allowed to simultaneously execute the statement i++ Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 26/27

27 Cheat sheet For more features see, e.g., the OpenMP cheat sheet Scientific Programming in C, fall 2012 Susi Lehtola Parallel computing 27/27