Copyright 2013 Thomas W. Doeppner. IX 1

Transcription

1 Copyright 2013 Thomas W. Doeppner. IX 1

2 If we have only one thread, then, no matter how many processors we have, we can do only one thing at a time. Thus multiple threads allow us to multiplex the handling multiple chores on a single processor) as well to handle multiple chores simultaneously on a multiprocessor. Furthermore, on both uniprocessors and multiprocessors, having a thread for each chore allows an important chore s execution to preempt that of a less-important chore. IX 2

3 One can certainly handle multiple chores with a single thread: one simply performs them sequentially. This approach has the advantage that one doesn t have to worry about synchronization, but, as shown on the slide, there are a number of things of which one isn t able to take advantage. IX 3

4 In this series of slides we present a simple example of an (almost) complete program one that multiplies two matrices using a number of threads. Our algorithm is not an example of an efficient matrix-multiplication algorithm, but it shows us everything that must be included to make a multithreaded C program compile and run. Our approach is to use the most straightforward technique for multiplying two matrices: each element of the product is formed by directly taking the inner product of a row of the multiplier and a column of the multiplicand. We employ multiple threads by assigning a thread to compute each row of the product. This slide shows the necessary includes, global declarations, and the beginning of the main routine. II-4

5 Here we have the remainder of main. It creates a number of threads, one for each row of the result matrix, waits for all of them to terminate, then prints the results (this last step is not spelled out). Note that we check for errors when calling pthread_create. (It is important to check for errors after calls to almost all of the pthreadroutines,butwenormallyomititintheslidesforlackofspace.)for reasons discussed later, the pthread calls, unlike Unix system calls, do not return -1 if there is an error, but return the error code itself (and return zero on success). However, the text associated with error codes is matched with error codes, just as for Unix-system-call error codes. So that the first thread is certain that all the other threads have terminated, it must call pthread_join on each of them. II-5

6 Here is the code executed by each of the threads. It s pretty straightforward: it merely computes a row of the result matrix. Note how the argument is explicitly converted from void * to int. II-6

7 In the producer-consumer problem we have two classes of threads, producers and consumers, and a buffer containing a fixed number of slots. A producer thread attempts to put something into the next empty buffer slot, a consumer thread attempts to take something out of the next occupied buffer slot. The synchronization conditions are that producers cannot proceed unless there are empty slots and consumers cannot proceed unless there are occupied slots.

8 Another synchronization construct is the semaphore, designed by Edsger Dijkstra in the 1960s. A semaphore behaves as if it were a nonnegative integer, but it can be operated on only by the semaphore operations. Dijkstra defined two of these: P (for prolagen, a made-up word derived from proberen te verlagen, which means try to decrease in Dutch) and V (for verhogen, increase in Dutch). Their semantics are shown in the slide. We think of operations on semaphores as being a special case of guarded execution a special case that occurs frequently enough to warrant a highly optimized implementation.

9 Here s a solution for the producer/consumer problem using semaphores note that it works only with a single producer and a single consumer.

10 Here is the producer-consumer solution implemented with POSIX semaphores.

11 An example of the use of our terminology comes from the simple matrixmultiplication example. Here we madeachoreoutof computing a row of the result and assigned one thread per chore. We could then utilize as many processors as we had threads (and hence chores). IX 11

12 We might rework our matrix-multiplication algorithm to be a bit more general, taking advantage of a technique known as work queues. Inparticular,we could subdivide a result matrix into units other than rows: perhaps individual elements or small groups of elements. Rather than using as many threads as chores, we might use a smaller number of threads (perhaps equal to the number of processors) and organize the chores into a queue. When a processor, executing using a particular thread as its context, needs work to do, it takes the next chore from the queue. IX 12

13 Now consider designing a server. In the simple approach, each client request is a chore and we have one thread per chore (creating as many threads as necessary). We can then utilize as many processors as we have chores. As we discuss soon, this approach works well for up to a moderate number of chores, but doesn t scale beyond this. IX 13

14 A technique which is essentially the same as that of work queues but with a different emphasis is known as thread pools. Herewe maintain a collection of threads ready to perform chores as they arise. This is a good approach for use on servers, since it scales fairly well. In some cases (for example, on Windows) one can place limits on the number of threads that are active at once. With scheduler activations, the number of active threads is limited by the very nature of the mechanism. IX 14

15 With the thread-pool approach (and others) one must confront the question of how many threads should be used. We certainly want enough threads so as to utilize all of the available processors. Putting this another way, if a chore is ready to be handled and there is a free processor, we should make certain that there s a thread available to be used. Thus the number of threads should be at least as great as the number of processors. Since, at various times, some threads executions will be blocked because of I/O or other synchronization, to make certain that we can take advantage of all the processors (even when that number isone),weshouldhavemorethreadsthanprocessors. IX 15

16 Having just established a lower bound on the number of threads we should have, is there an upper bound? I.e., are there problems if there are too many threads? Let s consider some possible issues. The first that might come to mind is the cost of time slicing; is it expensive to multiplex the execution of a number of threads on a smaller number of processors? The answer is normally no: time slicing typically involves something like a maximum of a hundred context switches (between) threads a second. Such context switches probably take no more than a few tens of microseconds. Thus the cost of time slicing is probably no worse than 0.1% of total processor time, an amount that is negligible. However, there are some non-negligible effects. If our threads access common data structures, synchronization is required. If not that many threads are competing for access, collisions are infrequent and the overhead of using mutexes, etc., is low. But if we have many threads competing, then the overhead might become significant. The cost of thousands of such collisions per second could cause an appreciable overhead in context switching and related activities. If we have large numbers of threads concurrently executing, then we might have problems similar to the thrashing that occurs in over-utilized virtual memory systems. The problem is that many subsystems, such as the kernel and the underlying hardware, use caching to improve performance. Such caches work well if there are not too many users. But if there are a lot of users, then there is not enough room to hold the working sets of all of them and, in many situations, the caches become useless. IX 16

17 In a naïve multithreaded implementation of malloc/free, there is one mutex protecting the heap, resulting in a bottleneck. IX 17

18 IX 18

19 IX 19

20 IX 20

21 IX 21

22 IX 22

23 IX 23

24 The code was run on an Intel(R) Core(TM)2 Quad CPU 2.40GHz. IX 24

25 IX 25

26 This program creates pairs of threads: one thread allocates storage, the other deallocates storage. They communicate using producer-consumer communication. IX 26

27 IX 27

28 To reduce the number of calls to sem_wait and sem_post, at each iteration the thread calls new allocsperiter (1024) times. IX 28

29 IX 29

30 The main routine creates npairs (16) of communicating pairs of threads. IX 30

31 The code was run on an Intel(R) Core(TM)2 Quad CPU 2.40GHz. IX 31

32 IX 32

33 Suppose we are processing a bi-directional stream of data: data in each direction is deposited into a buffer. It sour job to write the code to remove it from the buffer, do some processing, then deposit it in another buffer. Should this be done with one or with two threads? IX 33

34 A two-threaded solution is pretty straight-forward. Each thread might execute the code in the slide. A complete program implementing stream-relay with two threads may be found in util/threads.c IX 34

35 IX 35

36 This solution is probably not what we d want, since it strictly alternates between processing the data stream in one direction and then the other. IX 36

37 Looking at the two-threaded solution again, we highlight the sources of delay. IX 37

38 IX 38

39 IX 39

40 Here a simplified version of a one-threaded program to handle the relay problem. IX 40

41 This and the next three slides give a more complete version of the one-thread relay program. A complete program may be found in util/select.c. IX 41

42 IX 42

43 IX 43

44 IX 44

45 IX 45

46 IX 46

47 Note that epoll may be either edge-triggered or level-triggered, depending on parameters. Rather than returning a bit vector of file descriptors (as select does), it returns variable-sized vectors of event structures indicating what events have occurred. This potentially scales better than the select approach: if, say, there are 10,000 file descriptors and two events, it returns just the two events. A complete program for the stream-relay example using epoll with edge triggering may be found in util/epoll.c. Note that it is far more complicated than even util/select.c. IX 47

48 IX 48

49 IX 49

50 A complete program implementing the stream-relay example using four threads can be found in util/newthreads.cc. IX 50

51 Code equivalent to a two-threaded solution to the producer-consumer problem can be achieved using a single thread processing an event loop. As sketched in the slide, events occur either when data is available to be produced or when data can be consumed. When either event occurs, the producer and consumer routines are called repeatedly until neither can process any more data. Programs implementing the stream-relay example can be found in util/newselect.cc and in util/newepoll.cc (using level-triggering edge-triggering doesn t help here). IX 51

52 IX 52

53 IX 53

54 This is, of course, highly simplified! IX 54

55 IX 55

56 A more elaborate use of the two-level model is to allow multiple kernel threads per process. This deals with both the disadvantages described above and is the basis of the Solaris implementation of threading. It has some performance issues; in addition the notion of multiplexing user threads onto kernel threads is very different from the notion of multiplexing threads onto activities there is no direct control over when a chore is actually run by an activity. From an application s perspective, it is sometimes desired to have direct control over which chores are currently being run.

57 Capriccio and Knot were research projects at the University of California, Berkeley. A paper describing Capriccio is Capriccio: Scalable Threads for Internet Services, presented at SOSP A copy can be found at A paper describing Knot, its use of Capriccio, and its performance is Why Events Are A Bad Idea (for high-concurrency servers), presented at HotOS A copy can be found at IX 57

58 The Waterloo work is described in Comparing the Performance of Web Server Architectures, presented at EuroSys A copy can be found at pdf. IX 58

59 IX 59