Introduction to HPC. Introduction to HPC, Random number generation and OpenMP. Jérôme Lelong, Christophe Picard, Laurence Viry

Transcription

1 Introduction to HPC, Random number generation and OpenMP Jérôme Lelong, Christophe Picard, Laurence Viry Université Grenoble Alpes CEMRACS 2017

2 Agenda 1 Introduction to HPC Motivation Hardware Software Measure of performances 2 Random number generation 3 OpenMP Parallel Region Worksharing constructs Advanced usage

3 Agenda 1 Introduction to HPC Motivation Hardware Software Measure of performances 2 Random number generation 3 OpenMP Parallel Region Worksharing constructs Advanced usage

4 Agenda 1 Introduction to HPC Motivation Hardware Software Measure of performances 2 Random number generation 3 OpenMP Parallel Region Worksharing constructs Advanced usage

5 Why HPC? Sequential programming is limited applications are parallel. access to memory is limited. engineering/cost limitations: it is easier to increase the number of units than the frequency of a processor. Performances are evolving Hardware is evolving: faster but how to make the most of it. Algorithms are more efficient. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

6 Top 500 J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

7 Goals of HPC Intensive computation Solve a problem faster. Models are more sophisticated. Improve the accuracy of models. Increase interactivity. Example Improve the rate: compute N problems simultaneously. Decrease response time: solve a problem N times faster. Increase the size of the problem: compute a problem N times larger. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

8 How to achieve parallelism Architecture: multicore, memory, network, accelerators, instructions. Compilers: dedicated library, automatic parallelism. Algorithms: tailored algorithms. Mathematics: adapted numerical methods, evolutionary methods. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

9 Agenda 1 Introduction to HPC Motivation Hardware Software Measure of performances 2 Random number generation 3 OpenMP Parallel Region Worksharing constructs Advanced usage

10 Parallel computational units Implicit parallelism Parallel execution of different processor instructions. Happens almost automatically. Can only be influenced indirectly by the programmer. Multi-core/Multi-CPU/Many-Core Found in commodity hardware today. Computational units share the same memory. Clusters Aggregation of compute units. Independent systems linked using fast network interconnection. Each system has its own memory. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

11 Parallel computational units Accelerators Offer specialize features Often have their own memory Often not autonomous Vector processors/vector Units Perform same operations on multiple pieces of data simultaneously. Need a well thought layout of the data J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

12 Introduction to HPC Curie J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

13 How to increase performances? HPC attempts to divide a task into sub-tasks and to execute them simultaneously on different processing units. Identify where parallelism will be the most effective. Know the set of technological constraints. Design solution adapted to the problem and the constraints. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

14 Limitations of sequential solutions Clock speed limitation Current leakage. Power consumption. Heat dissipation. Uncompatible with mobile devices Standard optimisations Instruction prefetching Instruction reordering Pipelined functions units Branch prediction Functional unit allocation No control from the programmer J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

15 Flynn classification Classification depends on two parameters: the nature of the data flow and the sequence of instructions applied to them. Single Instruction Multiple Instruction Multiple Data SIMD MIMD Single Data SISD SIMD J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

16 Memory is bottleneck Single access Block access Page access Serial access Register Cache Main memory Background Memory Archive Memory Local memory is much faster to access. When computing a[i] = b[i] + c[i], accessing data takes at least 10 times longer than computing the sum. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

17 Shared memory (1) All the processors share the same memory space. They communicate using shared variables. Each processing unit carry out its task independently but modification of shared variables are instantaneous. Two kinds of shared memories SMP (Symmetric MultiProcessor) All the processors share a link to the memory. Access to the memory is uniform. CPU CPU CPU CPU MEMORY J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

18 Shared memory (2) NUMA (NonUniform Memory Access) All the processors can access to the memory but not uniformly. Each processor has a preferred access to some part of the memory. CPU CPU CPU CPU CPU CPU CPU CPU MEMORY MEMORY MEMORY MEMORY CPU CPU CPU CPU CPU CPU CPU CPU Decrease the risk of bottleneck to memory access. Local memory cache on each processor to mitigate the effect of non-uniform access. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

19 Distributed memory Each processor has its own memory. There is no global memory space. Each processor communicate with the others using messages. Variables are local to each processor. Each processor work independently on its own set of variables. The speed of the resolution depends on the architecture: network, topology, processors. Scales easily. INTERCONNECT NETWORK CPU CPU CPU CPU MEMORY MEMORY MEMORY MEMORY J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

20 Agenda 1 Introduction to HPC Motivation Hardware Software Measure of performances 2 Random number generation 3 OpenMP Parallel Region Worksharing constructs Advanced usage

21 Program execution abstraction Process Usually, multiple processes, each with their own associated set of resources (memory, file descriptors, etc.), can coexist Thread Typically smaller than processes Often, multiple threads per process Threads within the same process can share resources J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

22 Parallel programming models Classification according to process interaction 1. Message passing Parallel processes exchange data by passing messages Examples: PVM, MPI 2. Shared memory Parallel threads share a global address space Examples: POSIX threads, OpenMP 3. Implicit Process interaction is not visible to the programmer Examples: use of dedicated packages in R, Python,... J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

23 Agenda 1 Introduction to HPC Motivation Hardware Software Measure of performances 2 Random number generation 3 OpenMP Parallel Region Worksharing constructs Advanced usage

24 Measure of performances Scalability A metric that indicates how efficiently a parallel program can make use of increasing amounts of parallel computing resources. Typically focuses on the time needed to solve the same problem on increasing amounts of resources. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

25 Scalability Speedup S(n) = t(1) t(n) with t(k) the time on a system with k processing units. Upper limit: S(n) = n. This is the ideal speedup on n cores. Lower limit: S(n) = 1. No parallelism. Efficiency: E(n) = t(1) nt(n) E(n) = 1: Ideal efficiency. E(n) < 0.5: Characterize some form of inefficiency. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

26 Predicting efficiency Amdahl s law S(n, α) = ( α + 1 α ) 1 n α the serial part of the program Example: for α = 0.1 and n = 8, then S(n, α) = 4.7 and E(n, α) = 0.59 The parallel efficiency is limited by the sequential irreductible part lim S(n, α) = α 1 n J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

27 Agenda 1 Introduction to HPC Motivation Hardware Software Measure of performances 2 Random number generation 3 OpenMP Parallel Region Worksharing constructs Advanced usage

28 Constraints of parallelism On each processing unit, random numbers must be independent of each other and possess good statistical properties. The sequences of random numbers created on each processing unit must be independent. Generating random numbers must not incur overhead in information exchange between the processing units. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

29 General properties RNGs are deterministic sequences with good statistical properties independence uniformity large period low footprint memory fast A RNG is a state machine with a finite number of states. Formally, a random number is built using the following elements an initial state s 0, called the seed. a transition function S on all possible states such that s k+1 = S(s k ), a function V defined on the state space and such that V (s k ) is the random number produced ate step k. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

30 Mersenne Twister x k+n = x k+m + (x u k x l k+1)a, k 0 Generate unsigned integers on w bits, between 0 and 2 w 1. n is an integer representing the degree of the recursion 1 m n (x u k x l k+1 ) means that the w r most significant bits of x k are concatenated with the r less significant bits of x k+1 where 0 r w 1. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

31 Mersenne Twister A is a binary matrix with size w w a 0 a a w 1 xa is compduted by bit shifts the period is 2 p 1 with p = nw r. MT19937 has the following characteristics A large period: , A small memory footprint: 624 words of 32 bits, 4 times faster than rand(). J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

32 Quality of good parallel RNG 1. Reproducibility: we want to be able to replay the same scenario keeping the same number of processing units. 2. Independence between the processus: each sequence on each processing unit should be independent. 3. On each processing unit, draws should be statistically i.i.d. 4. Communication between processors should be limited: after the initial step, RNGs should not communicate with each other. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

33 Two different approaches 1. Splitting: divide a generator cycle into sub segments. Each stream of the generator use its own sub segment. streams are independent but the period is reduced. 2. Parametrization: each stream use different parameters (seed, multiplier, modulus) in its generator, such that all the sequences are independent. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

34 Splitting I 1. With a unique RNG, the stream associated to processor k is given by k + n p where p is the total number of processor. x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 10 x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 20 x 21 x 22 x 23 x 24 x 25 x 26 Requires to know how to jump over p elements at each iteration as fast as in the sequential case. Random numbers used by each processor change as soon as the number of PU changes. Elements of each stream are regularly spaced in the original sequence. regularly spaced sub-sequences are strongly correlated. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

35 Splitting II 2. With a unique RNG, the index bloc (k 1) N/p,..., k N/p is associated to processor k. unit 1 unit 2 unit 3 unit 4 unit 5 N/p N/p N/p N/p N/p Need to know how to make jumps with size N/p efficiently at the initial step. Often, we can only handle jumps with size 2 i. The period is reduced. The distribution of random number on the different processors may be independent of the value of p. Long range correlation is transformed into short range correlation. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

36 Splitting of Mersenne Twister In the case of RNGs with large period such as Mersenne Twister (N = ), long range correlation is very weak. With 10 5 processors, the period is Really hard to implement without using dedicated libraries since we need to compute the n th power of the recursion matrix with n very large. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

37 Parametrization 1. Changing the seed for each processing unit does not ensure independence. 2. Create streams associated to different values of the recursion matrix. How to find the parameter set ensuring independence. Initialization may take a while, especially when a large number of generators is required. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

38 Parametrization of Mersenne Twister Dynamic Creator Mersenne Twister (DCMT) For a given triplet (w, m, n), we can compute sets of vectors a such that the associated generators are independent. For p = 521, m = 9, n = 17, r = 23 and w = 32, there are 2 16 = possible values for A and each generator has the maximum period of 2 p 1. a contains an id ensuring independence for two different values: the rest is filled at random. Computational cost to find an appropriate a is small but random and grows fast with the size of a. On very large clusters, there are not enough available generators (#30: 124,200 cores). J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

39 Calling DCMT DCMT is available in PNL. PnlRng* pnl_rng_dcmt_create_id(int id, ulong seed) Create a RNG with PNL_RNG_DCMT and identifier id. Two generators with two differents id s are independent. The created RNG must be intialized with pnl_rng_sseed. void pnl_rng_free(pnlrng **) Free a RNG. void pnl_rng_sseed(pnlrng *rng, unsigned long int s) Set the seed of rng from s. When s=0, a default seed is used. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

40 How to choose a parallel RNG? If we need only few independent RNGs The parametrization of A for Mersenne Twister offers the best compromise. Specific methods can be used to store generators in a binary format independent of the machine for reproducibility. If we need a large number of RNGs Splitting the Mersenne Twister generator of period is a good solution. Even split in 100, 000 sub streams, each still has a period of and the correlation between sub-sequences are weak. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

41 Available solutions: The SPRNG library C++ Library dedicated to parallel random number generation. Available generators LCG, CMRG, LFG (Lagged Fibonacci), MLFG (Multiplicative Lagged Fibonacci) Each stream is parametrized by a unique identifier. Easy to reproduce a run. Each process/thread creates its own instance of the generator. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

42 Available solutions: The RngStream library [L Ecuyer et Côté, 91] C++ Library dedicated to parallel random number generation. Available generators : multiplicative recursive generators. Each stream is parametrized by a unique identifier. Easy to reproduce a run. Each process/thread creates its own instance of the generator. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

43 Agenda 1 Introduction to HPC Motivation Hardware Software Measure of performances 2 Random number generation 3 OpenMP Parallel Region Worksharing constructs Advanced usage

44 What is OpenMP OpenMP provides high-level thread programming Multiple cooperating threads are allowed to run simultaneously Threads are created and destroyed dynamically in a fork join pattern J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

45 Fork Join Execution Model Parallelism is achieved by generating multiple threads running in parallel A fork F is when a single thread is made into multiple, concurrently executing threads A join J is when the concurrently executing threads synchronize back into a single thread OpenMP programs essentially consist of a series of forks and joins. thread 0 thread 1 thread 0 thread 0 F J thread 0 F thread 1 J thread 0 thread 2 thread 3 thread 2 J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

46 Getting started, things to remember Remember to include the header file # include <omp.h> Insert compiler directives in C++ syntax as # pragma omp... Compile and link with -fopenmp flag for BNU and Clang compilers. Remember to assign the environment variable OMP_NUM_THREADS, which specifies the total number of threads inside a parallel region. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

47 General code structure # include <omp.h> main () { int var1, var2, var3; /* serial code */ /*... */ /* start of a parallel region */ # pragma omp parallel private(var1, var2) shared(var3) { /*... */ } /* more serial code */ /*... */ /* another parallel region */ # pragma omp parallel { /*... */ } } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

48 Agenda 1 Introduction to HPC Motivation Hardware Software Measure of performances 2 Random number generation 3 OpenMP Parallel Region Worksharing constructs Advanced usage

49 Parallel region A parallel region is a block of code that executed by a pool of threads The following compiler directive creates a parallel region # pragma omp parallel {... } Clauses can be added at the end of the directive Most often used clauses default(shared) or default(none) public(list of variables) private(list of variables) J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

50 Hello world # include <omp.h> # include <cstdio> int main (int argc, char *argv[]) { int th_id, nthreads; # pragma omp parallel private(th_id) shared(nthreads) { th_id = omp_get_thread_num(); printf("hello World from thread %d\n", th_id); # pragma omp barrier if ( th_id == 0 ) { nthreads = omp_get_num_threads(); printf("there are %d threads\n",nthreads); } } return 0; } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

51 Important OpenMP library routines int omp_get_num_threads(), returns the number of threads inside a parallel region int omp_get_thread_num(), returns the index of a thread inside a parallel region void omp_set_num_threads(int), sets the number of threads to be used void omp_set_nested(int), turns nested parallelism on/off J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

52 Single execution # pragma omp single {... } The code is executed by one thread only, but no guarantee on which one Can introduce an implicit barrier at the end # pragma omp master {... } Code executed by the master thread and no implicit barrier at the end. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

53 Coordination and synchronization # pragma omp barrier Synchronization, must be encountered by all threads in a pool (or none) # pragma omp ordered { a block of codes } is another form of synchronization (in sequential order). Execute some code by all threads but one at a time # pragma omp critical { a block of codes } # pragma omp atomic { single assignment statement } atomic is more efficient. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

54 Data scope OpenMP data scope attribute clauses: shared private firstprivate reduction What are the purposes of these attributes define how and which variables are transferred to a parallel region (and back). define which variables are visible to all threads in a parallel region, and which variables are privately allocated to each thread. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

55 Some remarks When entering a parallel region, the private clause ensures each thread having its own new variable instances. The new variables are assumed to be uninitialized. A pointer cannot be declared private. A shared variable exists in only one memory location and all threads can read and write to that address. It is the programmer s responsibility to ensure that multiple threads properly access a shared variable. The firstprivate clause combines the behavior of the private clause with automatic initialization. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

56 Agenda 1 Introduction to HPC Motivation Hardware Software Measure of performances 2 Random number generation 3 OpenMP Parallel Region Worksharing constructs Advanced usage

57 Parallel for loop Inside a parallel region, the following compiler directive can be used to parallelize a for-loop: #pragma omp for Clauses can be added, such as schedule(static, chunk size) schedule(dynamic, chunk size) schedule(guided, chunk size) (non-deterministic allocation) schedule(runtime) private(list of variables) reduction(operator:variable) nowait J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

58 Load balancing Schedule static dynamic guided When to Use Even and predictable workload per iteration; scheduling may be done at compilation time, least work at runtime. Highly variable and unpredictable workload per iteration; most work at runtime Special case of dynamic scheduling; compromise between load balancing and scheduling overhead at runtime J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

59 Example code # include <omp.h> # define CHUNKSIZE 100 # define N 1000 main () { int i, chunk; float a[n], b[n], c[n]; for (i=0; i < N; i++) a[i] = b[i] = i * 1.0; chunk = CHUNKSIZE; # pragma omp parallel shared(a,b,c,chunk) private(i) { # pragma omp for schedule(dynamic,chunk) for (i=0; i < N; i++) c[i] = a[i] + b[i]; } /* end of parallel region */ } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

60 More on Parallel for loop The number of loop iterations can not be non-deterministic; break, return, exit, goto not allowed inside the for-loop. The loop index is private to each thread. A reduction variable is special During the for-loop there is a private copy in each thread At the end of the for-loop, all the local copies are combined together by the reduction operation Unless the nowait clause is used, an implicit barrier synchronization will be added at the end by the compiler #pragma omp parallel and #pragma omp for can be combined into #pragma omp parallel for J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

61 What can happen with this loop? What happens with code like this # pragma omp parallel for for (i=0; i<n; i++) { sum += a[i]*a[i]; } All threads can access the sum variable, but the addition is not atomic! Race condition between threads should be avoided. So-called reductions in OpenMP are important for performance and for obtaining correct results. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

62 The above code becomes sum = 0.0; # pragma omp parallel for reduction(+:sum) for (i=0; i<n; i++) { sum += a[i]*a[i]; } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

63 Inner product n 1 a i b i i=0 int i; double sum = 0.; /* allocating and initializing arrays */ /*... */ # pragma omp parallel for default(shared) private(i) \ reduction(+:sum) for (i=0; i<n; i++){ sum += a[i]*b[i]; } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

64 Different threads do different tasks Different threads do different tasks independently, each section is executed by one thread. # pragma omp parallel { # pragma omp sections { # pragma omp section funca (); # pragma omp section funcb (); # pragma omp section funcc (); } } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

65 Parallelizing nested for-loops Serial code for (i=0; i<100; i++){ for (j=0; j<100; j++){ a[i][j] = b[i][j] + c[i][j] } } Why not parallelize the inner loop? Why must j be private? J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

66 Parallelizing nested for-loops Parallelization # pragma omp parallel for private(j) for (i=0; i<100; i++){ for (j=0; j<100; j++){ a[i][j] = b[i][j] + c[i][j] } } Why not parallelize the inner loop? Why must j be private? J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

67 Nested parallelism When a thread in a parallel region encounters another parallel construct, it may create a new pool of threads and become the master of the new pool. # pragma omp parallel num_threads(4) { /*... */ # pragma omp parallel num_threads(2) { // } } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

68 Common mistakes Race condition int nthreads; # pragma omp parallel shared(nthreads) { nthreads = omp_get_num_threads(); } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

69 Common mistakes Deadlock # pragma omp parallel {... # pragma omp critical {... # pragma omp barrier } } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

70 Agenda 1 Introduction to HPC Motivation Hardware Software Measure of performances 2 Random number generation 3 OpenMP Parallel Region Worksharing constructs Advanced usage

71 Not all computations are simple Not all computations are simple loops where the data can be evenly divided among threads without any dependencies between threads An example is finding the location and value of the largest element in an array for (i=0; i<n; i++) { if (x[i] > maxval) { maxval = x[i]; maxloc = i; } } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

72 Not all computations are simple, competing threads All threads are potentially accessing and changing the same values, maxloc and maxval. OpenMP provides several ways to coordinate access to shared values # pragma omp atomic Only one thread at a time can execute the following statement (not block). We can use the critical option # pragma omp critical Only one thread at a time can execute the following block atomic may be faster than critical but depends on hardware J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

73 How to find the max value using OpenMP Write down the simplest algorithm and look carefully for race conditions. How would you handle them? The first step would be to parallelize as # pragma omp parallel for for (i=0; i<n; i++) { if (x[i] > maxval) { maxval = x[i]; maxloc = i; } } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

74 Then deal with the race conditions Write down the simplest algorithm and look carefully for race conditions. How would you handle them? The first step would be to parallelize as # pragma omp parallel for for (i=0; i<n; i++) { # pragma omp critical if (x[i] > maxval) { maxval = x[i]; maxloc = i; } } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

75 What can slow down OpenMP performance? Performance poor because we insisted on keeping track of the maxval and location during the execution of the loop. We do not care about the value during the execution of the loop, just the value at the end. This is a common source of performance issues, namely the description of the method used to compute a value imposes additional, unnecessary requirements or properties Idea: Have each thread find the maxloc in its own data, then combine and use temporary arrays indexed by thread number to hold the values found by each thread J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

76 Find the max location for each thread int maxloc[max_threads], mloc; double maxval[max_threads], mval; # pragma omp parallel shared(maxval,maxloc) { int id = omp_get_thread_num(); maxval[id] = -1.0e30; # pragma omp for for (int i=0; i<n; i++) { if (x[i] > maxval[id]) { maxloc[id] = i; maxval[id] = x[i]; } } } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

77 Combine the values from each thread # pragma omp flush (maxloc,maxval) # pragma omp master { int nt = omp_get_num_threads(); mloc = maxloc[0]; mval = maxval[0]; for (int i=1; i<nt; i++) { if (maxval[i] > mval) { mval = maxval[i]; mloc = maxloc[i]; } } } J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

78 Tomorrow... MPI J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

79 G. Marsaglia. Diehard. ftp://stat.fsu.edu/pub/diehard. D. E. Knuth. The Art of Computer Programming, Vol. 2: Seminumerical Algorithms, Second edition. Addison-Wesley, Reading, Massachusetts, P. L Ecuyer. Efficient and portable combined random number generators. Comm. of the ACM, 31: , L Ecuyer and Côté (1991). Implementing a Random Number Package with Splitting Facilities. ACM Transactions on Mathematics Software, 17(1):98-111, A. De Matteis and A. Pagnutti. Long-range correlations in linear and non-linear random number generators. Parallel Computing, 1: , J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65

80 M. Matsumoto and T. Nishimura. Dynamic creation of pseudorandom number generators. Monte Carlo and Quasi-Monte Carlo Methods 1998, Springer, 2000, pp H. Haramoto, M. Matsumoto, T. Nishimura, F. Panneton, and P. L Ecuyer. Efficient jump ahead for f2-linear random number generators. INFORMS J. on Computing, 20: , July H. Haramoto, M. Matsumoto, and P. L Ecuyer. A fast jump ahead algorithm for linear recurrences in a polynomial space. Proceedings of the 5th international conference on Sequences and Their Applications, SETA 08, pages , Berlin, Heidelberg, Springer-Verlag. J. Lelong, C. Picard, L. Viry Université Grenoble Alpes CEMRACS / 65