Teaching parallel programming on multi-core and many-core processing architectures

Transcription

1 Teaching parallel programming on multi-core and many-core processing architectures P.Bakowski (SMTR) This presentation includes two parts : 1. Teaching parallel programing for multi-core processors (ARM based architecture) 2. Teaching massively parallel programming on many-core architectures (GPU nvidia) The first part is based on multi-core ARM architectures integrated in modern SoCs and embedded systems. These systems operate under the control of Linux system. The parallel programming environment is provided via openmp mechanisms (directives and pragmas). The second part is based on nvidia GPUs and CUDA programming environment. The programming exercises on CUDA also include the use of opencv providing us with image capture and recording operations, and the use of opengl for graphic operations on the video buffer. Part1 : Teaching parallel programing with OpenMP on embedded systems 1. Introduction In this part we show how to exploit the Embedded Systems on low cost development boards to teach the parallel programming on multi-core SoCs ARM processors and multi-core SoCs The great majority of the smart-phones and the embedded multimedia is based on ARM processor architecture and the associated circuits integrated into Systems on Chip (SoC). ARM processors are designed by ARM Holding as models and sold to a number of system design societies. The SoCs are developed on the base on these models then fabricated in the licensed fabs. Historically, the ARM processor architecture has evolved from the simple micro-controllers with no cache memory to complex multi-core architectures including memory management and multimedia processing units. In our study we are concerned only by recent ARM architecture version 7 that is often implemented as multicore Cortex-A7,A9, and A15. The architectural performance of a processor can be roughly measured by the number of instructions that the processor executes in one clock cycle. During two decades, the technological evolution has allowed for the increase of the clock frequency from several tens of MHz for the ARM micro-controllers to 1 or 2 GHz for recent implementations of ARM cores. If we take the average number of instructions per clock cycle (example Cortex-A15) equal to 3 and the clock frequency of 1.5 GHz, we obtain the processor performance of 4.5 Giga instructions per second. This is quite close to the performance of modern Intel x86 processors. Great quality of ARM processors and ARM based SoCs is their low power consumption (~1W/1Giga instructions per second). This is the principal reason for the use of these circuits in smart-phones and tablets. As the ARM SoCs contain complete systems including multimedia units, memory blocks, and I/O controllers, they allow for the construction of smart-phones and tablets from only from a few building blocks. Such a high level of integration lowers the production cost and increases the reliability of the devices. Nowadays the same ARM SoCs are used to build low cost and open source development boards. The open source aspect is related mainly to the software that drives the programmable components of the system. In our case the development boards are under the control of Linux system. Linux kernels and the

2 related packages provide the system software targeting specific development boards with specific SoCs, The Linux kernel operates on simple or multi-core ARM processors and related memory management units. The additional modules drive different SoC blocks such as GPU/VPU or WiFi and webcam circuits. All together, the hardware in the form of small development boards with multi-core SoCs and the open source software (Linux plus packages) provides us with an excellent platform for the teaching the Embedded Multiprocessing technology. The hardware side: In our study we use two kinds of ARM SoCs and computer units; the first kind is based on Odroid development boards, including Odroid-U2/X2 and Odroid-XU. Odroid U2/X2 boards are based on quad-core ARM Cortex-9 architecture while Odroid-XU integrates quad-core Cortex-15 SoC. - the second kind of boards is based on RK3188 quad-core Cortex-9 Soc integrated into Radxa board and RKM MK802IV dongle ARM Cortex-9 and Cortex-15 multi-core architecture The ARM Cortex-A9 MPCore is a 32-bit multi-core processor providing up to 4 cache-coherent Cortex-A9 cores, each implementing the ARM v7 instruction set architecture. The architectural of this processor is illustrated below: Each core owns one I-Cache and one D-cache. The chip control unit contains bus snoop unit, cache-tocache transfer unit and coherence unit necessary to organize shared memory operations. Shared memory operations are essential for the communication between the threads. In addition up to 8MB of L2 cache may be added through the optional L2 cache controller This architecture is the basis of Exynos-4212 and Rockchip-RK3188 SoCs integrated in the boards used in our practical classes. Both SoCs implement the ARM Neon SIMD (Single Instruction, Multiple Data) engine that is used to process multimedia formats and digital signals. It accelerates the speed at which signal processing algorithms are performed leading to a big increase in performance. This engine is faster than a general purpose CPU because it can perform the same instruction on multiple sets of data in parallel. The second kind of boards is based on ARM Cortex-A15 MPCore (Exynos5 octa core). The performance of Cortex-A15 MPCore is 40% higher than this of Cortex-A9 MPCore.

3 Cortex-A15 MPCore integrates a low-latency level-2 cache controller for up to 4 MB per cluster The software side: The development of examples is based on OpenMP programming interface. At its core, OpenMP is a set of compiler directives and function calls to enable you to run sections of your code in parallel on a shared memory parallel computer (multi-core). 1.1 OpenMP operational mode OpenMP assumes you will work with threads, which are basically processes that share the same memory address space as the other processes in a group of threads for a single program. If one thread makes a change to a variable that all threads can see, then the next access to that variable will use the new value. As it turns out, this model is fairly easy to think about. Imagine all your variables in a big block of memory, and all CPUs can see all the variables. This situation is not strictly true, but it is a good first approximation. The threads all see this memory and can modify this memory. The threads can all perform IO operations, file, print, and so on. So things as simple as our HelloWorld application may in fact be able to run in parallel, though generally IO operations tend to need to serialize access to global system state (file pointers, etc). 1.2 Compiler directives OpenMP operates mostly via compiler directives. A compiler directive is a comment that the compiler can ignore if not building for OpenMP. #pragma omp......code... where the...code... is called the parallel region. That is the area of the code that you want to try to run in parallel, if possible. When the compiler sees the start of the parallel region, it creates a pool of threads. When the program runs, these threads start executing, and are controlled by what information is in the directives. Without additional directives, we simply have a bunch of threads. A way to visualize this is to imagine an implicit loop around your parallel region, where you have N CPU/core iterations of the loop. These iterations all occur at the same time, unlike an explicit loop. The number of cores is controlled by an environment variable, OMP_NUM_THREADS. If it is not set, it could default to 1 or the number of cores on your machine. Just to be sure, you may want to do the following. export OMP_NUM_THREADS=`grep 'processor' /proc/cpuinfo wc l `

4 Now we re ready to parallelize hello.c. As a first step, lets put in the explicit compiler directives, and do nothing else. #include "stdio.h" int main(int argc, char *argv[]) #pragma omp parallel printf("\n"); return(0); Notice that I enclosed the parallelize region in a block denoted by an opening and closing set of braces, that is: printf... Now let s compile this this program with the following command line %gcc fopenmp o HelloMulticore HelloMultiCore.c and run it: %./HelloMulticore To get something like : By adjusting the value of the OMP_NUM_THREADS environment variable, we can adjust the number of execution threads. If we set 1 thread, we get, one print statement: %./HelloMulticore We can set more threads (8) than cores (4) : export OMP_NUM_THREADS="8" %./HelloMulticore It should be pointed out as well, that we really should insert a preprocessor directive in our code, in order to be able to pull in function prototypes and constants for use with OpenMP: #include <omp.h> If you want you can enclose this in an ifdef construct: #ifdef _OPENMP #include <omp.h> #endif

5 This way, that code will only be used if the compiler has been told to use OpenMP. Before we do something useful with this, lets explore a few functions that might be helpful. We can use several OpenMP function calls to query and control our environment. The most frequently used functions are those that return the number of threads operating, and the current thread ID. There are several others that are useful. Our new program incorporates several of these, along with a few tricks I have found useful over the years. The new hello code (HelloMulticoreID.c)now looks like this: #include "stdio.h" #include <omp.h> int main(int argc, char *argv[]) #pragma omp parallel int NCPU,tid,NPR,NTHR; /* get the total number of CPUs/cores available for OpenMP */ NCPU = omp_get_num_procs(); /* get the current thread ID in the parallel region */ tid = omp_get_thread_num(); /* get the total number of threads available in this parallel region */ NPR = omp_get_num_threads(); /* get the total number of threads requested */ NTHR = omp_get_max_threads(); /* only execute this on the master thread! */ if (tid == 0) printf("%i : NCPU\t= %i\n",tid,ncpu); printf("%i : NTHR\t= %i\n",tid,nthr); printf("%i : NPR\t= %i\n",tid,npr); printf("%i : I am thread %i out of %i\n",tid,tid,npr); return(0); We can compile and run it with 8 threads. %gcc fopenmp o HelloMulticoreID HelloMultiCoreID.c export OMP_NUM_THREADS=8 %./ HelloMulticoreID 1 : I am thread 1 out of 8 2 : I am thread 2 out of 8 0 : NCPU = 8 0 : NTHR = 1 0 : NPR = 8 0 : I am thread 0 out of 8 7 : I am thread 7 out of 8 3 : I am thread 3 out of 8 4 : I am thread 4 out of 8 5 : I am thread 5 out of 8 6 : I am thread 6 out of 8 The first number you see there is the thread number or thread ID (tid variable in the program). Notice that the output does not come out necessarily in thread order. And if you examine it closely, you might notice that it doesn t come out in time order either, though that is pretty close. One of the tricks that I have learned and

6 use is to tag each line with either the thread ID or the time, and then sort it. %./ HelloMulticoreID sort n 0 : I am thread 0 out of 8 0 : NCPU = 8 0 : NPR = 8 0 : NTHR = 1 1 : I am thread 1 out of 8 2 : I am thread 2 out of 8 3 : I am thread 3 out of 8 4 : I am thread 4 out of 8 5 : I am thread 5 out of 8 6 : I am thread 6 out of 8 7 : I am thread 7 out of 8 Now we can tell what thread did what. Though the time ordering is still off. It s a simple matter of programming to replace the thread ID with a time value that can be sorted. When this happens, I usually change the print format lines to look something like this: "%.3f D[%i]... ",timestamp,tid,... Once I ve added this, I can see what happened, in the order that it happened, and still get the thread ID data. Items like this are helpful when debugging parallel programs. Now it s time to move on to where a majority of the power of OpenMP becomes apparent to end users. 1.2 Loops OpenMP helps you in a number of clever ways, allowing you to add threading to your program without thinking through all the details of thread setup and tear-down. It also will effectively re-engineer loops that you tell it to re-engineer for you. As usual, it helps to start out with an understanding of where your program is spending its time. Without that, you could wind up parallelizing a loop that takes very little time in the overall execution path, and ignore the time expensive code. Additionally, it is important to test various sizes of runs, so you understand which portions of your code scale well, and which portions may need assistance. Profiling is the best way to do this, we will use a poor mans profiler, basically timing calipers around sections of the code. This technique will give us approximate millisecond resolution data (Specifically, it is limited to the clock timer tick interrupt rate, and could be anywhere from 10 milliseconds to 1 millisecond). You will not get more accurate single shot data than the timer resolution. In order to get better resolution, you need to iterate enough so that you can calculate an average time per iteration. It is also worth noting that OS jitter (management interrupts for disk I/O, network I/O, and running an OS in general) will also impact your measurements some, so please take this into account if you would like more precise data Ray tracing example To start with, suppose you re writing a ray tracing program. Without going too much into the details of how ray tracing works, it simply goes through each pixel of the screen, and using lighting, texture, and geometry information, the color of that pixel is determined. The program goes on to the next pixel and repeats the process. The important thing to note here is that the calculation for each pixel is completely separate from the calculation of any other pixel, therefore making this program highly suitable for OpenMP. Consider the following pseudo-code: for(int x=0; x < width; x++) for(int y=0; y < height; y++) finalimage[x][y] = RenderPixel(x,y, &scenedata);

7 This piece of code simply goes through each pixel of the screen, and calls a function, RenderPixel, to determine the final color of that pixel. Note that the results are simply stored in an array. Simply put, the entire scene that is being rendered is stored in a variable, scenedata, whose address is passed to the RenderPixel function. Because each pixel is independent of all other pixels, and because RenderPixel is expected to take a noticeable amount of time, this small snippet of code is a prime candidate for parallelization with OpenMP. Consider the following modified pseudo-code: #pragma omp parallel for for(int x=0; x < width; x++) for(int y=0; y < height; y++) finalimage[x][y] = RenderPixel(x,y, &scenedata); The only change to the code is the line directly above the outer for loop. This compiler directive tells the compiler to auto-parallelize the for loop with OpenMP. If a user is using a quad-core processor, the performance of your program can be expected to be 300% increased with the addition of just one line of code, which is amazing. In practice, true linear or super linear speedups are rare, while near linear speedups are very common. There are a few important things you need to keep in mind when parallelizing for loops or any other sections of code with OpenMP. For example, take a look at variable y in the pseudo code above. Because the variable is effectively being declared inside the parallelized region, each processor will have a unique and private value for y. However, take the following buggy code example below: int x,y; #pragma omp parallel for for(x=0; x < width; x++) for(y=0; y < height; y++) finalimage[x][y] = RenderPixel(x,y, &scenedata); The above code has a serious bug in it. The only thing that changed is the fact that now, variables x and y are declared outside the parallelized region. When we use the compiler directive to declare the outer for loop to be parallelized with OpenMP, the compiler already knows by common sense that the variable x is going to have different values for different threads. However, the default scope for the other variables, y, finalimage, and scenedata, are all shared by default, meaning that these values will be the same for all threads. All threads have access to read and write to these shared variables. The code above is buggy because variable y should be different for each thread. Declaring y inside of the parallelized region is one way to guarantee that a variable will be private to each thread, but there is another way to accomplish this. int x,y; #pragma omp parallel for private(y) for(x=0; x < width; x++) for(y=0; y < height; y++) finalimage[x][y] = RenderPixel(x,y, &scenedata);

8 Instead of declaring variable y inside the parallel region, we can declare it outside the parallel region and explicitly declare it a private variable during the OpenMP compiler directive. This effectively makes each thread have an independent variable called y. Each thread will only have access to it s own copy of this variable. 2. Application examples In the following part we are building two complete matrix oriented applications. The first is dot product algorithm, the second is matrix multiplication Dot product The following OpenMP example is a program which computes the dot product of two arrays a and b (that is sum(a[i]*b[i]) ) using a sum reduction. The input variables a and b are shared tables of double values. #include <omp.h> #include <stdio.h> #include <stdlib.h> #define N 1000 int main (int argc, char *argv[]) double a[n], b[n]; double sum = 0.0; int i, n, tid; /* Start a number of threads */ #pragma omp parallel shared(a) shared(b) private(i) tid = omp_get_thread_num(); /* Only one of the threads do this */ #pragma omp single n = omp_get_num_threads(); printf("number of threads = %d\n", n); /* Initialize a and b */ #pragma omp for for (i=0; i < N; i++) a[i] = 1.0; b[i] = 1.0; /* Parallel for loop computing the sum of a[i]*b[i] */ #pragma omp for reduction(+:sum) for (i=0; i < N; i++) sum += a[i]*b[i]; /* End of parallel region */ printf(" exit(0); Sum = %2.1f\n",sum); The reduction clause specifies two things: 1. When control enters the parallel region, each thread in the region gets a thread-private copy of sum, initialized to the identity element for When control leaves the parallel region, the original sum is updated by combining its value with the final values of the thread-private copies, using +. Since + is associative, the final sum has the same value as it would for serial execution of the code.

9 2.2 Matrix Multiplication In general with multiple threads you do not get any performance advantage relative to serial execution; worse, usually you get a performance disadvantage or penalty. That is why we are looking how to maximize the parallel execution part of the program/algorithm. The core algorithm in the matmul.c code is the triply nested for loop. The main multiplication loop looks like this: /* matrix multiply * * c[i][j]= a_row[i] dot b_col[j] for all i,j * a_row[i] > a[i][0.. DIM 1] * b_col[j] > b[0.. DIM 1][j] * */ for(i=0;i<dim;i++) for(j=0;j<dim;j++) dot=0.0; for(k=0;k<dim;k++) dot += a[i][k]*b[k][j]; c[i][j]=dot; You will probably notice immediately that the interior loop is a sum reduction. It is basically a dot product between two vectors. That inner loop is executed N squared times. This means that the parallel region is set up and torn down N squared times. The setup and tear down are not free: that is, they do have a non-zero time cost. Which would become abundantly clear in the event of placing the parallelization directives around that loop. In general, for most parallelization efforts, you want to enclose the maximum amount of work within the parallel region. This rule suggests you really want to put the directives outside the outer most loop, or as high up in the loop hierarchy as possible. So we insert a simple #pragma parallel directive as follows: #pragma omp parallel for private(i,j,k,dot) shared(a,b,c) for(i=0;i<dim;i++) for(j=0;j<dim;j++) dot=0.0; for(k=0;k<dim;k++) dot += a[i][k]*b[k][j]; c[i][j]=dot; In this example, private(i,j,k,dot) tells the compiler which variables are private relative to each thread (i.e. not shared between threads), and shared(a,b,c) indicates which ones are shared across threads. Notice we haven t specified anything about dimension of the array. With a relatively simple code adjustment (that some compilers might do for you if you can coax them), you can see significantly better performance in parallel. What we do is increase the amount of work done in parallel. We do this by unrolling a loop. Our code now looks like this: #pragma omp parallel for private(i,j,k,dot) shared(a,b,c) firstprivate(dim) for(i=0;i<dim;i+=4) for(j=0;j<dim;j++) dot[0]=dot[1]=dot[2]=dot[3]=0.0; for(k=0;k<dim;k++)

10 dot[0] += a[i+0][k]*b[k][j]; dot[1] += a[i+1][k]*b[k][j]; dot[2] += a[i+2][k]*b[k][j]; dot[3] += a[i+3][k]*b[k][j]; c[i+0][j]=dot[0]; c[i+1][j]=dot[1]; c[i+2][j]=dot[2]; c[i+3][j]=dot[3]; This version of the program operates on four rows at a time (we unrolled the loop), thus increasing the amount of work done per iteration. We have also reduced the cache miss penalty per iteration by reusing some of the more expensive elements (the b[k][j]). In addition, we added the firstprivate(dim) OpenMP directive which specifies that each thread should have its own instance of a variable, and that the variable should be initialized with the value of the variable as it exists before the parallel construct. Of course, after all of these modifications, we check the results to make sure that the addition of parallelization has not also caused the addition of bugs! 2.3 Comparing sequential and parallel matrix multiplication The following is the complete code for matrix multiplication including the performance test. The comparison is done with sequential and parallel execution of the same algorithm. The code also contains several parameters to be used for testing and debugging. For example the initial test and debugging runs with small matrix size (10 rows x 10 columns). Note the use of DEBUG constant set to 1 (true) for a run with debugging operations. #include <omp.h> #include <stdio.h> #include <stdlib.h> #define NR_THREADS 4 #define DEBUG 0 #define NRA 440 #define NCA 440 #define NCB 440 // number of threads used // normal run // number of rows in matrix A // number of columns in matrix A // number of columns in matrix B /* Use smaller matrices for testing and debugging */ /* #define DEBUG 1 // debug run #define NRA 10 // number of rows in matrix A #define NCA 10 // number of columns in matrix A #define NCB 10 // number of columns in matrix B */ int main (int argc, char *argv[]) int tid, nthreads, i, j, k; double **a, **b, **c; double *a_block, *b_block, *c_block; double **res; double *res_block; double starttime, stoptime; a = (double **) malloc(nra*sizeof(double *)); /* matrix a to be multiplied */ b = (double **) malloc(nca*sizeof(double *)); /* matrix b to be multiplied */ c = (double **) malloc(nra*sizeof(double *)); /* result matrix c */

11 a_block = (double *) malloc(nra*nca*sizeof(double)); b_block = (double *) malloc(nca*ncb*sizeof(double)); c_block = (double *) malloc(nra*ncb*sizeof(double)); /* Result matrix for the sequential algorithm */ res = (double **) malloc(nra*sizeof(double *)); res_block = (double *) malloc(nra*ncb*sizeof(double)); for (i=0; i<nra; i++) /* Initialize pointers to a */ a[i] = a_block+i*nra; for (i=0; i<nca; i++) /* Initialize pointers to b */ b[i] = b_block+i*nca; for (i=0; i<nra; i++) /* Initialize pointers to c */ c[i] = c_block+i*nra; for (i=0; i<nra; i++) /* Initialize pointers to res */ res[i] = res_block+i*nra; /* A static allocation of the matrices would be done like this */ /* double a[nra][nca], b[nca][ncb], c[nra][ncb]; */ /* Spawn a parallel region explicitly scoping all variables */ #pragma omp parallel shared(a,b,c,nthreads) private(tid,i,j,k) num_threads(nr_threads) tid = omp_get_thread_num(); if (tid == 0) /* Only thread 0 prints */ nthreads = omp_get_num_threads(); printf("starting matrix multiplication with %d threads\n",nthreads); printf("initializing matrices...\n"); /*** Initialize matrices ***/ #pragma omp for nowait /* No need to synchronize the threads before the */ for (i=0; i<nra; i++) /* last matrix has been initialized */ for (j=0; j<nca; j++) a[i][j]= (double) (i+j); #pragma omp for nowait for (i=0; i<nca; i++) for (j=0; j<ncb; j++) b[i][j]= (double) (i*j); #pragma omp for /* We synchronize the threads after this */ for (i=0; i<nra; i++) for (j=0; j<ncb; j++) c[i][j]= 0.0; if (tid == 0) /* Thread zero measures time */ starttime = omp_get_wtime(); /* Do matrix multiply sharing iterations on outer loop */ /* If DEBUG is TRUE display who does which iterations */ /* printf("thread %d starting matrix multiply...\n",tid); */

12 #pragma omp for for (i=0; i<nra; i++) if (DEBUG) printf("thread=%d did row=%d\n",tid,i); for(j=0; j<ncb; j++) for (k=0; k<nca; k++) c[i][j] += a[i][k] * b[k][j]; /* If DEBUG is true, print the results. Use smaller matrices for this */ if (DEBUG) printf("result Matrix:\n"); for (i=0; i<nra; i++) for (j=0; j<ncb; j++) printf("%6.1f ", c[i][j]); printf("\n"); printf ("Done.\n"); exit(0); #pragmaomp for nowait for (i=0; i<nca; i++) for (j=0; j<ncb; j++) b[i][j]= (double) (i*j); #pragma omp for /* We synchronize the threads after this */ for (i=0; i<nra; i++) for (j=0; j<ncb; j++) c[i][j]= 0.0; if (tid == 0) /* Thread zero measures time */ starttime = omp_get_wtime(); /* Master thread measures the execution time */ /* Do matrix multiply sharing iterations on outer loop */ /* If DEBUG is TRUE display who does which iterations */ /* printf("thread %d starting matrix multiply...\n",tid); */ #pragma omp for for (i=0; i<nra; i++) if (DEBUG) printf("thread=%d did row=%d\n",tid,i); for(j=0; j<ncb; j++) for (k=0; k<nca; k++) c[i][j] += a[i][k] * b[k][j]; if (tid == 0) stoptime = omp_get_wtime(); printf("time for parallel matrix multiplication: %3.2f s\n", stoptime starttime); /*** End of parallel region ***/

13 starttime = omp_get_wtime(); /* Do a sequential matrix multiplication and compare the results */ for (i=0; i<nra; i++) for (j=0; j<ncb; j++) res[i][j] = 0.0; for (k=0; k<nca; k++) res[i][j] += a[i][k]*b[k][j]; stoptime = omp_get_wtime(); printf("time for sequential matrix multiplication: %3.2f s\n", stoptimestarttime); /* Check that the results are the same as in the parallel solution. Actually, you should not compare floating point values for equality like this but instead compute the difference between the two values and check that it is smaller than a very small value epsilon. However, since all values in the matrices here are integer values, this will work. */ for (i=0; i<nra; i++) for (j=0; j<ncb; j++) if (res[i][j] == c[i][j]) /* Everything is OK if they are equal */ else printf("different result %5.1f!= %5.1f in %d %d\n ", res[i][j], c[i] [j], i, j); /* If DEBUG is true, print the results. Usa smaller matrices for this */ if (DEBUG) printf("result Matrix:\n"); for (i=0; i<nra; i++) for (j=0; j<ncb; j++) printf("%6.1f ", c[i][j]); printf("\n"); printf ("Done.\n"); exit(0); To do: Analyze and test the above code using different options. Summary We have shown how to obtain, build, and use an OpenMP compiler for Linux machines. The OpenMP examples shown range from simple hello examples, through parallel matrix multiplication, and all demonstrate excellent performance. With this short presentation you get the sense that OpenMP is both powerful, and easy to use. There is plenty more to learn, but at least you have already started working with OpenMP. As SoC units surge to eight and even 16 or more cores, OpenMP is likely to become increasingly important for applications development.

14 References MIT.Press.»Using.OpenMP».2008 OpenMP Application Program Interface, OpenMP, 2013