Concurrent Programming

Transcription

1 University of Applied Sciences Fulda Department of Applied Computer Sciences Prof. Dr. Siegmar Groß Fachhochschule Fulda Fachbereich Angewandte Informatik Concurrent Programming Process Virtual Address Space Resources open files locks sockets. main () count () Text Identity pid uid gid. prt_result () Data Heap main Thread Specific Data thread id priority Thread Specific Data thread id priority Registers PC SP Thread 1 Registers PC SP Thread 2 Registers PC SP Stack Stack Stack S. R. Groß, Fulda, 2000

2 Concurrent Programming Slide 1 how can we achieve high-performance computing? new processors and memories with higher and even higher clock rates multiprocessor machines supercomputers with vector units supercomputers with massively parallel processors (MPP; machines with some 100 up to some 1000 processors) (heterogeneous) workstation networks for distributed computing (so-called virtual machines or workstation clusters) very cheap compared to supercomputers, if you can use existing workstation networks (e.g., most workstations are idle at night) you can increase the computing power in a simple way you can add vector processors and parallel machines to your virtual machine if you need them the programming task is sometimes supported by special compilers which automatically vectorize or parallelize sequential programs how can parallel processes communicate with each other? using shared memory using message-passing

3 Concurrent Programming Slide 2 some characteristics of a virtual machine usually it consists of machines of different manufacturers usually heterogeneous machines are based on different proccessor architectures (from PC's via various workstations up to supercomputers different binary formats) usually heterogeneous machines use different data formats (big endian, little endian, floating point representation, ) usually the machines have different load usually the machines are connected to different networks (Ethernet, Fast-Ethernet, ATM, ) usually the performance of a virtual machine depends on the load of the networks each program must be compiled on each different machine all data transfers must be done using a standard interface (e.g., each message must be coded on the sender site from the internal format of the machine to a network format and decoded on the receiver site vice versa) the programmer must take care of the performance and load of the machines (high-speed computers shouldn't wait for results of slow ones) depending on the load of the machines and the network, the performance of a virtual machine may vary very strongly

4 Concurrent Programming Slide 3 influences on the performance of a multiprocessor system the scheduling scheduler attempts to keep each process/thread on the same processor (known as processor affinity, processor binding) scheduler doesn't attempt to keep each process/thread on the same processor cache must be reloaded (at least the L1 cache) the processor cache sharing a single L2 cache can result in contention for the cache possibly decreased performance when executing multiple independent sequential programs (after each context switch the cache possibly contains the "wrong" data) possibly increased performance when executing parallel programs (if many processes want to access the same "line" in shared memory, only one had to fill the cache line and the others can immediately access the data) scheduling without processor affinity isn't that harmful

5 Concurrent Programming Slide 4 independent L2 caches main memory cache line cache cache cache line tag processor 1 cache line tag processor 2 each processor uses its own 2nd level cache possibly increased performance when executing multiple independent sequential programs possibly decreased performance when executing parallel programs (if one process modifies a cache line it must be invalidated or reloaded in any other cache which stores the same line known as false sharing) lack of processor affinity leads to reloading of caches increasing bus contention

6 Concurrent Programming Slide 5 organizing parallel processes / threads Master / Slave what should the master do? initializes the system generates all necessary processes/threads (static or dynamically depending on the task) provides the slaves with work collects the results and prints / stores them (depending on the task) perhaps synchronizes the work of the slaves what should all slaves do? perform the real work receive their work from the master (static or dynamically depending on their progress) provide themselves with work (e.g., from a work-pool) sometimes the number of active threads isn't predictable thus leading to decreasing performance if all slaves fight for the same resources (e.g., if an internet server creates a slave for each request) examples internet servers parallel multiplication of matrices

7 Concurrent Programming Slide 6 Workpile (work queue, producer / consumer) some processes/threads place orders into a queue other processes/threads take orders from the queue and carry them out there is a fixed number of processes/threads peak loads are handled through the queue if there is high load over a long time orders must be refused to prevent that the queue grows too big example: computation of a Mandelbrot set tree computation processes/threads are created dynamically in a tree-like structure depending on the progress of the computation especially appropriated if the workload isn't well-known in the beginning (e.g., Branch-and-Bound or Divide-and-Conquer-algorithms) example: Quicksort

8 Concurrent Programming Slide 7 workload allocation data decomposition, data parallelism identical processes/threads work on different data areas (Single-Program-Multiple-Data, SPMD) static or dynamic distribution of the work (using a virtual machine distributing the work dynamically usually performs best, because the different workloads of the machines and networks and different performances of the machines can be compensated) examples computing a Mandelbrot set adding two n-dimensional vectors (using p processes/threads each one performing n/p additions) function decomposition, function parallelism different processes/threads are doing different operations on the same data-stream (Multiple-Program-Single-Data, MPSD) example: graphics-pipeline

9 Concurrent Programming Slide 8 a combination of the above methods different processes/threads are doing different operations where some operations are performed in parallel on different data areas (Multiple-Program-Multiple-Data, MPMD) process granularity coarse-grained large number of sequential instructions takes a substantial time to execute fine-grained very few sequential instructions only a small amount of time to execute granularity is often defined as the size of computation between communication or synchronization points coarse granularity reduces the proportional costs of process creation and interprocess communication fine granularity possibly increases the number of concurrent processes using message passing it must be guaranteed that the communication time doesn't dominate the overall execution time

10 Concurrent Programming Slide 9 which speed-up is possible? maximum speed-up with n processors is n (linear speed-up) some limiting factors some parts of a computation can't be executed in parallel some organizational tasks must be executed sequentially (initializing the system, setting up parallel processes, etc.) communication time for sending/receiving messages time to synchronize the parallel computation extra computations in the parallel version, e.g., recomputing values locally assume that all serial parts are combined at the beginning and that all other parts can be executed in parallel with no overhead (f is the fraction which can't be executed in parallel) f t s t s (1 - f) t s 1 processor serial code parallel code n processors f t s t p (1 - f) t s / n

11 Concurrent Programming Slide 10 speed-up factor S( n) = ts n = ft + ( 1 f ) t / n 1+ ( n 1 ) f s s equation is known as Amdahl's law speed-up is limited to S( n) = 1 f n (with 5% serial and 95% parallel computation the maximum speed-up is 20) Amdahl's law represents the result for a problem with fixed size solved with an increasing number of processors to reduce the computation time Gustafson had stated that a larger multiprocessor systems usually allows a larger size of the problem to be solved the size of the problem will be scaled so that the parallel execution time is fixed the serial part of the code will generally not increase in the same way as the problem size

12 Concurrent Programming Slide 11 Gustafson's scaled speed-up factor S n s( ) let s be the time to execute the serial part and p the time for executing the parallel part of the computation execution time on a parallel computer with n processors: s + p (the total execution time is fixed; for algebraic convenience: s + p = 1) execution time on a serial computer: s + n p (all parallel parts must be executed sequentially) s np Ss ( n) = + s + p s n( s) + 1 s + 1 s s+ p=1 = n + ( 1 n) s (with Amdahl's law the speed-up factor for a parallel computer with 32 processors and 5% serial code is and with Gustafson's law the scaled speed-up factor is 30.45) Gustafson's law seems more appropriate because today's intention is to solve more complex problems in a fixed time

13 Concurrent Programming Slide 12 structure of a process Process Virtual Address Space Resources open files locks sockets. main () count () count () Text Identity pid uid gid. prt_result () global_cnt s_time Data Heap Registers PC SP j i p_cnt return address from count () Stack

14 Concurrent Programming Slide 13 using parallel processes to increase a local and a global variable Parent Process Virtual Address Space Child Process Virtual Address Space Resources main () Text Resources main () Text fork () fork () Identity count () Identity count () pid=238 prt_result () pid=240 prt_result () Data Data Registers PC SP Heap Stack Registers PC SP Heap Stack for (i = 0; i < NUM_CLD; ++i) fork_pid [i] = fork (); /* create child process */ switch (fork_pid [i]) case -1: fprintf (stderr, "line %d: \"fork ()\" failed: %s\n", LINE, strerror (errno)); exit (-1); break; case 0: /* child process */ count (&(shm_addr->cnt [i])); /* local variable as parameter */ exit (0); /* nothing more to do */ /* wait for all child processes and print result */ for (i = 0; i < NUM_CLD; ++i) if (waitpid (fork_pid [i], NULL, 0) == (pid_t) -1) fprintf (stderr, "line %d: \"waitpid ()\" failed: %s\n", LINE, strerror (errno));

15 Concurrent Programming Slide 14 coarse-grained locking void count (long *p_cnt) #ifdef USE_SEM /* coarse-grained locking results in better performance, but leads * to sequential execution, because the whole work is protected. */ ret = semop (sem_id, &p, (size_t) 1); for (i = 0; i < MAX_OUT_LOOP; ++i) printf ("Process %ld is counting \n", (long) getpid ()); for (j = 0; j < MAX_IN_LOOP; ++j) (*p_cnt)++; (shm_addr->global_cnt)++; ret = semop (sem_id, &v, (size_t) 1); #else #endif fine-grained locking void count (long *p_cnt) for (i = 0; i < MAX_OUT_LOOP; ++i) printf ("Process %ld is counting \n", (long) getpid ()); for (j = 0; j < MAX_IN_LOOP; ++j) #ifdef USE_SEM /* fine-grained locking results in poor performance, because the * execution time for "semop" is much larger than that for the * increment-operations and the function doesn't do anything else. */ ret = semop (sem_id, &p, (size_t) 1); (*p_cnt)++; (shm_addr->global_cnt)++; ret = semop (sem_id, &v, (size_t) 1); #else #endif } the program isn't suited for parallel programming

16 Concurrent Programming Slide 15 using threads Process Virtual Address Space Resources open files locks sockets. main () count () Text Identity pid uid gid. prt_result () Data Heap main Thread Specific Data Registers PC SP Thread 1 Registers Stack thread id priority Thread Specific Data thread id priority PC SP Thread 2 Registers PC SP Stack Stack

17 Concurrent Programming Slide 16 there are many different thread interfaces UNIX International threads (SunOS) C threads (Mach) DCE threads (Open Software Foundation, draft 4 of pthreads) pthreads (POSIX c-1995) Windows NT threads OS/2 threads each thread is a different stream of control that can execute its instructions independently, allowing a multithreaded process to perform numerous tasks concurrently or in parallel each thread has its own set of CPU registers, especially its own program counter and its own stack pointer threads can be executed independently from each other all threads use the same virtual address space and other resources of the surrounding process if a thread changes shared data, all threads "see" the changes if a thread calls exit (), all threads and the whole process will finish if a thread opens a file another thread can read from or write to the file (they can interfere with each other when the program isn't implemented carefully)

18 Concurrent Programming Slide 17 using POSIX threads #define _REENTRANT /* must precede any "#include"! */ #define _POSIX_C_SOURCE L /* standard: June 1995 */ #include <pthread.h> int main (void) pthread_attr_t attr; /* attributes for new threads */ pthread_mutex_t LockCnt; } ret = pthread_mutex_init (&LockCnt, NULL); ret = pthread_attr_init (&attr); ret = pthread_attr_setdetachstate (&attr, PTHREAD_CREATE_JOINABLE); #ifdef BOUNDED ret = pthread_attr_setscope (&attr, PTHREAD_SCOPE_SYSTEM); #endif /* create concurrent threads for counting */ for (i = 0; i < NUM_THR; ++i) ret = pthread_create (&thr_id [i], &attr, (void *(*) (void *)) count, &cnt [i]); /* wait for all threads and print result */ for (i = 0; i < NUM_THR; ++i) ret = pthread_join (thr_id [i], NULL); /* clean up all things */ ret = pthread_attr_destroy (&attr); #ifdef USE_MTX ret = pthread_mutex_destroy (&LockCnt); #endif return 0; void count (long *p_cnt) for (i = 0; i < MAX_OUT_LOOP; ++i) printf ("Process %ld: Thread %ld is counting \n", ); for (j = 0; j < MAX_IN_LOOP; ++j) #ifdef USE_MTX ret = pthread_mutex_lock (&LockCnt); (*p_cnt)++; global_cnt++; ret = pthread_mutex_unlock (&LockCnt); #else #endif }

19 Concurrent Programming Slide 18 summary (all times in seconds) Pentium, 200 MHz, Linux R5000, 180 MHz, IRIX Ultra-1, 143 MHz, SunOS SPARCstation 20, 50 MHz, 2 CPUs, SunOS sequential process parallel processes user-level threads kernel-level threads 7 not allowed why are threads faster than parallel processes? a process mainly consists of the following parts: code (text) data stack file I/O signal tables hardware context (CPU registers, ) if a process forks a child only the code will be shared if a context switch between processes takes place, all the tables have to be exchanged

20 Concurrent Programming Slide 19 processes can share information only through pipes, shared memory, and other such expensive things (operating system intervention is needed) threads reduce this overhead by sharing fundamental parts share data share filesystem info share open files share signals share PID with parent (not available with LinuxThreads) switching happens much more frequently and efficiently sharing information is quite easy and can take place without operating system intervention threads are implemented differently separate process and thread tables with different schedulers (the solution of most multithreaded operating systems) what happens when a thread executes fork (), execve (), or something similar? a) the process with all threads will be replaced / duplicated? b) only the calling thread will be replaced /duplicated? (POSIX requires that a call to fork creates a process with a single thread. In a multithreaded process the new process shall contain a replica of the calling thread and its entire address space. POSIX requires that a call to any exec function shall result in all threads being terminated and the new executable image being loaded and executed.) one common process / thread table with one scheduler (e.g., the solution of LinuxThreads)

21 Concurrent Programming Slide 20 user-level threads are unknown to the kernel are fully managed in user-space (thread table, scheduling) don't consume kernel resources can typically switch faster than kernel-level threads are well suited for threads which are waiting almost all the time can't take advantage of more than one processor an I/O-blocking thread may block all other threads a compute-bounded thread may monopolize the timeslice thus starving the other threads within the process kernel-level threads are managed in kernel-space consume kernel resources (each process gets a table of threads or each thread needs an entry in the process table) can use all processors of a symmetric multiprocessor I/O-blocking isn't a problem because the kernel knows about the threads are less likely to hog a timeslice because the kernel schedules the threads, thus they are suited for compute-bounded threads

22 Concurrent Programming Slide 21 concurrent processes in distributed (heterogeneous) environments 1. you don't use NFS (different HOME-directories on each machine) compile your program on every machine 2. you are working in a homogeneous environment with NFS (one HOME-directory for all machines) compile your program on one machine 3. you are working in a heterogeneous environment with an NFS domain for every machine type (one HOME-directory for each machine type) compile your program once in every NFS domain 4. you are working in a heterogeneous environment with one NFS domain for all machines (one HOME-directory for all machines) compile your program once on every machine type take care that programs for different machine types are stored in different directories, e.g. in $HOME/$SYSTEM_ENV}/$MACHINE_ENV}/bin (SYSTEM_ENV: SunOS, Linux, ; MACHINE_ENV: sparc, x86, )

23 Concurrent Programming Slide 22 message passing interface (master/slave, this is the master) int main (int argc, char *argv []) int ntasks, /* number of parallel tasks */ mytid, /* my task id */ num, /* number of entries */ i; /* loop variable */ char buf [BUF_SIZE + 1]; /* message buffer (+1 for '\0') */ MPI_Status stat; /* message details */ } MPI_Init (&argc, &argv); MPI_Comm_rank (MPI_COMM_WORLD, &mytid); MPI_Comm_size (MPI_COMM_WORLD, &ntasks); if (ntasks > MAX_TASKS) printf ("Error: Too many tasks. Try again with at most %d tasks.\n", MAX_TASKS); /* terminate all slave tasks */ for (i = 1; i < ntasks; ++i) MPI_Send ((char *) NULL, 0, MPI_CHAR, i, EXITTAG, MPI_COMM_WORLD); MPI_Finalize (); exit (ENTASKS); printf ("\n\nnow %d slave tasks are sending greetings.\n\n", ntasks - 1); /* request messages from slave tasks */ for (i = 1; i < ntasks; ++i) MPI_Send ((char *) NULL, 0, MPI_CHAR, i, SENDTAG, MPI_COMM_WORLD); /* wait for messages and print greetings */ for (i = 1; i < ntasks; ++i) MPI_Recv (buf, BUF_SIZE, MPI_CHAR, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &stat); MPI_Get_count (&stat, MPI_CHAR, &num); buf [num] = '\0'; /* add missing end-of-string */ printf ("Greetings from task %d:\n" " message type:\t%d\n" " msg length:\t%d characters\n" " message:\t%s\n\n", stat.mpi_source, stat.mpi_tag, num, buf); /* terminate all slave tasks */ for (i = 1; i < ntasks; ++i) MPI_Send ((char *) NULL, 0, MPI_CHAR, i, EXITTAG, MPI_COMM_WORLD); MPI_Finalize (); return 0;

24 Concurrent Programming Slide 23 heterogeneous environments can have problems with incompatible data types operating system cc double gcc long double cc gcc IRIX, mips 8 bytes, 53 bit 8 bytes, 53 bit 16 bytes, 107 bit 8 bytes, 53 bit SunOS 7, Sparc 8 bytes, 53 bit 8 bytes, 53 bit 16 bytes, 113 bit 16 bytes, 113 bit SunOS 7, x86 8 bytes, 53 bit 8 bytes, 53 bit 12 bytes, 64 bit 12 bytes, 53 bit Linux, x86 8 bytes, 53 bit 12 bytes, 64 bit only double is suited for heterogeneous environments (computing π with MPI in a heterogeneous environment with long double delivers strange results: up to infinity)