Chapter 6 Parallel Loops

Transcription

1 Chapter 6 Parallel Loops Part I. Preliminaries Part II. Tightly Coupled Multicore Chapter 6. Parallel Loops Chapter 7. Parallel Loop Schedules Chapter 8. Parallel Reduction Chapter 9. Reduction Variables Chapter 10. Load Balancing Chapter 11. Overlapping Chapter 12. Sequential Dependencies Chapter 13. Strong Scaling Chapter 14. Weak Scaling Chapter 15. Exhaustive Search Chapter 16. Heuristic Search Chapter 17. Parallel Work Queues Part III. Loosely Coupled Cluster Part IV. GPU Acceleration Part V. Map-Reduce Appendices 45

2 46 BIG CPU, BIG DATA W e begin our study of tightly coupled multicore parallel programming with a simple Parallel Java 2 program to test numbers for primality. Recall that a number is prime if it is divisible only by itself and 1. PrimeSeq (Listing 6.1) is a sequential (non-parallel) program to test the numbers specified on the command line. The program illustrates several features that I ll include in all the parallel programs we study: The program is implemented as a subclass of class Task (in package edu.rit.pj2), and the program s code is in the body of the task s main() method. Unlike a typical Java program, the main() method is an instance method, not a static method. The Parallel Java 2 middleware expects this. The main() method is declared to throw any exception (line 9). If an exception is thrown anywhere in the program, this lets the exception propagate out of the main() method, which will terminate the program and print an error message with an exception stack trace. I do this because I m lazy and I don t want to write a handler for every exception. At line 12, the program makes sure the proper command line arguments are present; if not, the program prints an error message and exits. If I run the program with no arguments, this reminds me what the arguments should be. On line 41, the program exits by calling the terminate() method. The argument is an exit code that is returned to the operating system; a nonzero value says that an error occurred. The program must not call System.exit(); doing so interferes with the Parallel Java 2 middleware. I overrode the static coresrequired() method (lines 45 48) to return the value 1, indicating that this program will use one core. This is standard for a non-parallel program. If the coresrequired() method is not overridden, the Parallel Java 2 middleware will assume that the program will use all the cores on the node. The loop on lines is the heart of the program. The loop iterates over the command line argument array, converts each number from a String to a long, calls the isprime() method, and prints the number if isprime() says it s prime. The isprime() method uses trial division to test whether the number x is prime. The method tries to divide x by 2 and by every odd number p up through the square root of x. If any remainder is 0, then p is a factor of x, so x is not prime. If none of the remainders are 0, then x is prime. (There s no point in trying factors greater than the square root of x, because if there were such a factor, x would have another factor less than the square root of x, and we would have found that other factor already.) Trial division is not a very efficient algorithm for primality testing, but it suffices for this example.

3 Chapter 6. Parallel Loops package edu.rit.pj2.example; import edu.rit.pj2.task; public class PrimeSeq extends Task // Main program. public void main (String[] args) throws Exception // Validate command line arguments. if (args.length < 1) usage(); // Test numbers for primality. for (int i = 0; i < args.length; ++ i) if (isprime (Long.parseLong (args[i]))) System.out.printf ("%s%n", args[i]); // Test the given number for primality. private static boolean isprime (long x) if (x % 2 == 0) return false; long p = 3; long psqr = p*p; while (psqr <= x) if (x % p == 0) return false; p += 2; psqr = p*p; return true; // Print a usage message and exit. private static void usage() System.err.println ("Usage: java pj2 " + "edu.rit.pj2.example.primeseq <number>..."); terminate (1); // Specify that this task requires one core. protected static int coresrequired() return 1; Listing 6.1. PrimeSeq.java

4 48 BIG CPU, BIG DATA I ran the PrimeSeq program on tardis, a cluster parallel computer with ten nodes, each node having 12 cores. For now I ll confine myself to running multicore parallel programs on just one node of the cluster. (Later we will develop cluster parallel programs and run them on all the cluster nodes.) Because PrimeSeq is not a parallel program, though, it ran on only one core. I gave it 12 very large numbers to test, all of which happened to be prime. Here is the command and the program s output on one of the tardis nodes: $ java pj2 debug=makespan edu.rit.pj2.example.primeseq \ \ \ \ Job 1 makespan msec Note that I ran the program by typing the command java pj2. pj2 is the actual Java main program; it is a launcher for Parallel Java 2 programs. pj2 creates an instance of the specified class (class edu.rit.pj2.example-.primeseq in this case), which must be a subclass of class Task. pj2 then calls the task s main() method, passing in an array of the command line argument strings. I also included the option debug=makespan before the task class name. This sets the pj2 program s debug parameter to include the makespan debug printout. Makespan is the elapsed wall clock time from when the task starts running to when the task finishes running. With that option, the pj2 program measures the makespan and prints it as the final line of output. (You can turn on other debug printouts and set other pj2 parameters as well. Refer to the Javadoc documentation for the pj2 program.) The loop iterations executed sequentially one after another on a single core. The running time measurement says that the whole program took 32.6 seconds. From this I infer that each loop iteration each execution of the is- Prime() method took 2.7 seconds.

5 Chapter 6. Parallel Loops 49 However, for this program there s no need to do the loop iterations in sequence. Because no iteration depends on the results of any prior iteration, we say that the loop does not have any sequential dependencies. (Later we will study loops that do have sequential dependencies.) Therefore, we can execute all the loop iterations in parallel, each on a separate core. Doing so, we hope the program will finish in less time. PrimeSmp (Listing 6.2) is a parallel version of the primality testing program. It starts out the same as program PrimeSeq. But I replaced the normal, sequential for loop in the original program with a work sharing parallel for loop in the new program (lines 16 23). The pattern for writing a parallel for loop is parallelfor (lb, ub).exec (new Loop() public void run (int i) Loop body code for iteration i ); The parallel for loop begins with parallelfor instead of just for. (parallelfor() is actually a method of class Task.) Then come the lower and upper bounds of the loop index. The loop goes from lb to ub inclusive, so at line 16 I specified bounds of 0 through args.length-1 to loop over all the command line arguments. The statement so far creates a parallel loop object. Then I called the parallel loop s exec() method, passing in another object, namely the loop body. The loop body is an instance of a subclass of class Loop (in package edu.rit.pj2), which I created using Java s anonymous inner class syntax. I put the code for one loop iteration in the Loop class s run() method, whose argument is the loop index i; this is the same code as the loop

6 50 BIG CPU, BIG DATA body in the sequential version. The rest of the program is the same as the sequential version. I did not override the coresrequired() method in the PrimeSmp program. Thus, by default, the Parallel Java 2 middleware will assume that the program will use all the cores on the node. When the PrimeSmp program runs, the parallel for loop object that is created contains a hidden parallel thread team. There is one thread in the team for each core of the machine where the program is executing. The loop iterations are partitioned among the team threads, and each team thread calls the loop body s run() method repeatedly for a different subset of the loop indexes, concurrently with the other team threads. In other words, the work of the parallel for loop is shared among all the threads, rather than being executed by a single thread as in the sequential version. When the program runs on a multicore parallel computer, the Java Virtual Machine and the operating system schedule each team thread to run on a separate core, resulting in parallel execution of the loop iterations. At the end of the parallel for loop, each team thread waits until all the team threads have finished executing their subsets of the loop iterations, then the program proceeds to execute whatever comes after the parallel loop. This implicit thread synchronization is called a barrier. I ran the PrimeSmp program on tardis, with the same arguments as the previous example. A tardis node has 12 cores, so the parallel for loop has 12 team threads. The loop s iterations are divided among the team threads; thus, each team thread does one iteration. Here is the result: $ java pj2 debug=makespan edu.rit.pj2.example.primesmp \ \ \ \ Job 1 makespan 2857 msec Keep in mind that the number of threads in the parallel thread team is equal to the number of cores in the node, not the number of iterations in the loop. In this example, it s just a coincidence that the number of loop itera-

7 Chapter 6. Parallel Loops package edu.rit.pj2.example; import edu.rit.pj2.loop; import edu.rit.pj2.task; public class PrimeSmp extends Task // Main program. public void main (final String[] args) throws Exception // Validate command line arguments. if (args.length < 1) usage(); // Test numbers for primality. parallelfor (0, args.length - 1).exec (new Loop() public void run (int i) if (isprime (Long.parseLong (args[i]))) System.out.printf ("%s%n", args[i]); ); // Test the given number for primality. private static boolean isprime (long x) if (x % 2 == 0) return false; long p = 3; long psqr = p*p; while (psqr <= x) if (x % p == 0) return false; p += 2; psqr = p*p; return true; // Print a usage message and exit. private static void usage() System.err.println ("Usage: java pj2 " + "edu.rit.pj2.example.primesmp <number>..."); terminate (1); Listing 6.2. PrimeSmp.java

8 52 BIG CPU, BIG DATA tions is the same as the number of cores. Shortly we will see examples of parallel loops where the number of loop iterations is not the same as the number of cores. Notice three things about the parallel program. First, while the sequential program finished in 32.6 seconds, the parallel program finished in only 2.9 seconds. From this I infer that the 12 loop iterations were indeed performed simultaneously on 12 cores instead of one after another on a single core. That is, the parallel program yielded a speedup over the sequential program. To be precise, the speedup factor was = (Later we will study more about parallel program performance and the metrics with which we measure performance.) Second, the parallel program s running time (2.9 seconds) was a bit larger than the running time of one loop iteration (2.7 seconds). Why? Because the parallel program has extra overhead creating the 12 threads in the parallel thread team, partitioning the loop iterations among the team threads, synchronizing the threads at the end-of-loop barrier that the sequential program does not have. This extra overhead increases the parallel program s running time slightly. Third, while the parallel program did determine correctly that every number on the command line was prime, the parallel program reported the prime numbers in a different order from the sequential program. This happened because each number was printed by a separate team thread, and there was nothing in the program to force the team threads to print the numbers in any particular order. (In this example program, there s no need to synchronize the threads so as to make them do their printouts in a certain order.) Under the Hood Figure 6.1 shows in more detail what happens when the PrimeSmp program executes the parallel for loop code, parallelfor (0, args.length 1).exec (new Loop() public void run (int i) if (isprime (Long.parseLong (args[i]))) System.out.printf ("%s%n", args[i]); ); on a parallel computer with 12 cores. Keep in mind that all this happens automatically. I only had to write the code above. Still, it s helpful to understand what s going on under the hood. In the figure, the various objects are arranged from top to bottom, and time flows from left to right. Methods called on each object are depicted as gray boxes. The threads calling the methods are shown as thick lines. A solid

9 Chapter 6. Parallel Loops 53 Figure 6.1. Parallel for loop execution flow line means the thread is executing; a dashed line means the thread is blocked waiting for something to happen. Execution begins with the main program thread calling the main() method on the Task object, an instance of the PrimeSmp subclass. The main thread calls the task s parallelfor() method with a lower bound of 0 and an upper bound of 11, which are the index bounds of the args array with 12 command line arguments. The parallelfor() method creates a parallel for loop object, an instance of class IntParallelForLoop. Hidden inside the parallel for loop is a team of threads, one thread for each core. Each team thread has a different rank in the range 0 through 11. Initially, the team threads are blocked until the program is ready to execute the parallel for loop.

10 54 BIG CPU, BIG DATA The main thread creates a loop object, which is an instance of an anonymous inner subclass of class Loop. The main thread calls the parallel for loop s exec() method, passing in the loop object. The exec() method creates 11 additional copies of the loop object by calling the loop object s clone() method. The main thread unblocks the team threads, then blocks itself inside the exec() method until the team threads finish. At this point the parallel for loop begins executing. Each team thread calls the run() method on a different one of the loop objects. Because each team thread is executing on a different core, the run() method calls proceed in parallel. Each team thread passes a different index as the run() method s argument. However, across all the team threads, every loop index from 0 to 11 gets passed to some loop object s run() method. Each team thread, executing the code in its own run() method, tests one of the numbers on the command line for primality and prints the number if it s prime. The Parallel Java 2 middleware automatically collects the characters each thread prints on System.out (or System.err) in separate per-thread internal buffers. When the program terminates, the buffers contents are written to the program s standard output stream (or standard error stream), one buffer at a time. Thus, characters printed by different threads are not commingled. To emit a printout before the end of the program, after printing the characters, call System.out.flush() (or System.err.flush()). After returning from the loop object s run() method, each team thread waits at a barrier. When all the team threads have arrived at the barrier, the team threads block themselves, and the main thread is unblocked. The main thread resumes executing and returns from the parallel for loop s exec() method. The main thread continues executing the code in the task s main() method after the parallel for loop. Thus, the parallel program follows this pattern of execution: Sequential section (single main program thread) Parallel section (multiple team threads) Sequential section (single main program thread) This pattern, of one or more parallel sections interspersed within a sequential section, is found in almost every multicore parallel program. Only a portion of the program is executed in parallel; the rest of the program is executed sequentially. We will return to this observation when we study parallel program performance. What if we run the PrimeSmp program with 12 command line arguments on a parallel computer with more than 12 cores? The parallel team will have more threads than needed to handle all the loop indexes. In that case, team threads rank 0 through 11 call the loop objects run() methods as described

11 Chapter 6. Parallel Loops 55 above, and team threads rank 12 and higher merely proceed directly to the barrier without calling the run() method. What if we do a parallel for loop with more loop indexes than there are cores? In that case, each team thread will call its loop object s run() method more than once. We ll study how that works in the next chapter. Points to Remember Write a Parallel Java 2 program as a subclass of class Task. Put the program code in the task s main() method. The task s main() method must be an instance method, not a static method. You can parallelize a loop if there are no sequential dependencies among the loop iterations. Use the parallelfor() pattern to parallelize a for loop. The parallel for loop index goes from the lower bound to the upper bound inclusive. The parallel for loop contains a hidden team of threads. The number of threads in the team is the number of cores on the node (not the number of loop iterations). Put the loop body code in the run() method of the inner Loop subclass. To emit a printout before the end of the program, call System.out.flush() or System.err.flush(). To terminate the program other than by returning from the task s main() method, call terminate(). Don t call System.exit(). In a non-parallel program, override the static coresrequired() method to return the number of cores the program will use, namely 1. Use the java pj2 command to run your Parallel Java 2 program. Use the debug=makespan option to measure the program s running time.

12 56 BIG CPU, BIG DATA