Little Motivation Outline Introduction OpenMP Architecture Working with OpenMP Future of OpenMP End. OpenMP. Amasis Brauch German University in Cairo

Transcription

1 OpenMP Amasis Brauch German University in Cairo May 4, 2010

2 Simple Algorithm 1 void i n c r e m e n t e r ( short a r r a y ) 2 { 3 long i ; 4 5 for ( i = 0 ; i < ; i ++) 6 { 7 a r r a y [ i ]++; 8 } 9 } Increments all values of a large array. Can this problem be divided into a set of sub-problems? No dependency among different elements.

3 Dividing Problems Sequential Approach (same problem). Parallel Approach. Only beneficial when executed in parallel. How?

4 Threads (Review) Threads. Each thread has a separate instance of stack, registers & PC. OS schedules threads according to scheduling algorithm. Threads can indeed run in parallel. When?

5 Threads (Review)

6 Application Divide our previous problem into m parts: new IncrementThread(0, n/m, data), new IncrementThread(n/m + 1, 2 * n/m + 1, data),..., new IncrementThread((m - 1) * n/m + 1, n, data). Problems?

7 Problems Not Generic: IncrementThread? Will there be a thread for everything you need to do? Coming up with a generic algorithm is not a trivial task. Is it always useful/possible to parallelize the problem (e.g.: single core CPU)? Does the parallelization degree depend on the environment (Machine, OS, etc.)? Code might get very big hard to maintain; (working with many threads is usually complex). Answer: OpenMP.

8 Outlining the Rest Introduction to OpenMP. OpenMP Architecture. Working with OpenMP. Main Components. Directives. Environment Variables. Programming Examples. Future of OpenMP. Pros/Cons. Hybrid Model. Conclusion.

9 Introduction OpenMP. Open: Open Specification. M: Multi. P: Processing. OpenMP is an API (Application Programming Interface), used for shared memory multiprocessing programming. Shared Memory ( Shared Memory Architecture): Several processors can access same memory. Multiprocessing: Computer systems with more than one CPU core. Supported languages: C/C++ and Fortran. Runs on most platforms.

10 Mini OpenMP History Motivated by missing standardization in shared memory computing. Each vendor had their own standard (no portability). Digital, Intel, IBM, SGI and KAI formed a forum to come up with a new standard (OpenMP) after a previous failed attempt (ANSI X3H5). First release (for Fortran) in October C/C++ edition released in October Today most major vendors support OpenMP.

11 Main Components Figure: Open Extensions: Parallel control structures are used to parallelize code. Work sharing is used for distributing work (i.e. breaking up loops). Data environment: defines the scope of variables within parallel regions. Synchronization describes the process of rejoining threads. Runtime routines allow an application to interact with the environment during runtime.

12 OpenMP Diagram Figure: OpenMP Architecture: OpenMP works with threads. User puts directives in code to indicate parallelization. Environment variables determine runtime behaviour (resource allocation, scheduling algorithm, number of threads).

13 Directives OpenMP directives have the following form (in C): #pragma omp < rest of pragma >. Example usage of directives: #pragma omp parallel {}: Parallelization of the code enclosed by the braces. #pragma omp parallel do/for: Parallelization of a do/for-loop. Can t we use parallel to parallelize a loop? Note: pragmas are ignored by unknowing compilers.

14 Parallel Control Figure: Left: Open MP & Threads: Program prints Thread no: i from each available thread, where i is the thread number. Recall the idea of using parallel to parallelize a loop Would execute the entire for-loop n times. Solution: Work sharing/data based parallel programming (i.e. parallel do/for).

15 Work Sharing 1 void i n c r e m e n t e r ( short a r r a y ) 2 { 3 long i ; 4 #pragma omp p a r a l l e l for 5 { 6 for ( i = 0 ; i < ; i ++) 7 { 8 a r r a y [ i ]++; 9 } 10 } 11 } Initial Problem (Incrementer). Using work sharing, the work is distributed among the threads (e.g.: Thread 1 takes first half of array, thread 2 the second half).

16 Work Sharing contd. Divide code into sections which can then be executed in parallel (Task based parallel programming). 1 #pragma omp s e c t i o n s // Starts creating section here. 2 #pragma omp s e c t i o n // This is a single section within all sections. 1 #pragma omp p a r a l l e l // Server application with many tasks. 2 { 3 #pragma omp s e c t i o n s // Starts creating threads here. 4 { 5 #pragma omp s e c t i o n // Listen to incoming connections. 6 { 7 l i s t e n T o C l i e n t s ( ) ; 8 } 9 #pragma omp s e c t i o n // Update IP address. 10 { 11 updatedynamicdns ( ) ; 12 } 13 #pragma omp s e c t i o n // Get video input. 14 { 15 videostreaming ( ) ; 16 } 17 } 18 }

17 Scheduling schedule(type, chunk size), where type can be either: static: iterations are divided into chunks of size provided by chunk size, scheduled to thread in round-robin fashion. dynamic: like static but scheduled dynamically across the threads. guided: scheduled dynamically. chunk size = 1: Remaining iterations are divided by number of threads ( decreases exponentially to 1). For a general size k: Remaining iterations are divided by number of threads ( decreases exponentially to k). runtime: Checks environment variable OMP SCHEDULE.

18 Data environment Sharing variables between threads using: shared. Maintaining own (private) variables: private. firstprivate initializes each new variable with the original value. lastprivate the thread which executes the last iteration writes its value into this variable. firstprivate and lastprivate can be combined. reduction(op:variable) takes all instances of the variable and reduces them to one after parallel execution, by applying an operator op on them (+, *, -, &, ˆ,, &&, or ) Note: Reduction Privacy.

19 Synchronization critical: Only executed by one thread. No other thread may run at the same time. ordered: Executes loop in its original sequence (no parallelization). When is it needed? flow dependency: arr[i] = i * arr[i-1]. barrier: Each thread waits for the other threads to complete when this directive is reached. nowait: Opposite of barrier, threads may continue once they are done.

20 Largest Element 1 int l a r g e s t ( int v a l u e s, long l e n g t h ) 2 { 3 long i ; // Private variables. 4 int l a r g e s t P r i v ; 5 short threadcount = omp get num threads ( ) ; // Shared variables. 6 int sharedarray [ threadcount ] ; 7 l a r g e s t P r i v = v a l u e s [ 0 ] ; // Initialize the largest variable. 8 #pragma omp p a r a l l e l for f i r s t p r i v a t e ( i, l a r g e s t P r i v ) // Parallel region. 9 { 10 for ( i = 1 ; i < length ; i++) 11 { 12 if ( l a r g e s t P r i v < v a l u e s [ i ] ) 13 { 14 l a r g e s t P r i v = v a l u e s [ i ] ; 15 } 16 } 17 s h a r e d A r r a y [ omp get thread num ( ) ] = l a r g e s t P r i v ; 18 #pragma omp b a r r i e r // Wait for all threads to finish first. 19 } 20 l a r g e s t P r i v = s h a r e d A r r a y [ 0 ] ; // Initialize the largest variable. 21 for ( i = 1 ; i < threadcount ; i ++) // Sequential search within the new array. 22 { 23 if ( l a r g e s t P r i v < s h a r e d A r r a y [ i ] ) 24 { 25 l a r g e s t P r i v = s h a r e d A r r a y [ i ] ; 26 } 27 } 28 return l a r g e s t P r i v ; 29 }

21 Finding the Golden Mean Figure: Computation of pi; Speedup related to CPU vs. Thread count: Rule-of-thumb: Number of Threads = Number of processors/cores.

22 Pros & Cons Advantages: Easy to program / easy to maintain. Supports parallelization and sequential code execution. Disadvantages: Runs only on shared memory computers. No GPU support. Needs a compiler with OpenMP support. Fails on systems with large numbers of CPUs.

23 Hybrid Model Hybridization of OpenMP and MPI (Message Passing Interface). MPI can be used for systems with a very large number of processors, but deals with distributed memory. OpenMP programs are much easier to implement and allows handling shared memory.

24 Hybrid Model contd. Figure: MPI & OpenMP Hybridization: Use MPI to implement the required program, OpenMP can be used to divide each of the processes further into threads.

25 End Thanks for Listening!

26 References topic=/com.ibm.xlcpp8a.doc/compiler/ref/ruomprun.htm OpenMP.ppt HowToGuide-OpenMP.pdf Research--Parallelism_in_Applications--Worn2002a

27 Test yourself 1 bool isprime ( int number ) int countprimes ( int from, int to ) 2 { { 3 int count = 2 ; int count = 0 ; 4 if ( number < 2) for ( ; from <= to ; from++) 5 { { 6 return false ; if ( isprime ( from ) ) 7 } { 8 while ( count < number ) count++; 9 { } 10 if ( number % count == 0) } 11 { return count ; 12 return false ; } 13 } 14 count++; 15 } 16 return true ; 17 } What should be parallelized? Which variables should be private/shared? Can we use reduction here? Are the runtime routines helpful for parallelizing this algorithm?

28 Test yourself 1 int main ( int argc, char argv ) 2 { 3 #pragma omp p a r a l l e l 4 { 5 p r i n t f ( " Thread no: %i", omp get thread num ( ) ) ; 6 } 7 return 0 ; 8 }