EF HLD on SMP machines. Sub-farm design Inter-thread communications and error handling Future developments. Andrea NEGRI D.F.N.T. and I.N.F.N.

Transcription

1 EF HLD on SMP machines Multi-thread vs multi-process Sub-farm design Inter-thread communications and error handling Future developments Andrea NEGRI D.F.N.T. and I.N.F.N. Pavia 1

2 SMP Prototype We are testing the implementation of the current High Level Design of an Event Filter sub-farm on a Symmetric Multi Processor machine :4-CPU HP K220 as benchmark machine in Pavia running the version 11 of the HP-UX Operating System All the components of the sub-farm are implemented within a single multithreaded process in POSIX-10 thread standard This solution implies many advantages in term of simplicity and resources optimisation :On a SMP machine the multitask work required for a sub-farm can be realised in a similar fashion with either a multi-process program (several processes in communication) a multi-threaded single process program (MT) 2

3 1998Multi-thread vs Multi-process implementation The MT implementation is simpler, runs more quickly and leads to a better exploitation of the hardware parallelism than a multi-process one Ø A process is a heavy-weight, kernel level entity All parts of the process structure are in kernel space and can be accessed only via a (slow) system call The user code along with the data is in user space and can be accessed directly (fast) Ø A thread is a light-weight, user level entity any thread has its own private registers, stack and a light structure About all of the thread structure is in user space very fast access The rest of the process structure is shared by all the threads (address space, file descriptors, etc) The user code is global and can be executed by any thread Ø A thread uses only a fraction of the system resources used by a process creating a process is about 30 times slower that creating a thread synchronisation mechanisms are about 10 times slower context switching is about 5 times slower 3

4 4 ( The idea is to have a single program with many threads running different parts of the user code concurrently ë The MT implementation eases the communication and the synchronisation between the different components Ö Ô Ö Ô The multi-process solution entails the use of the traditional inter-process communication facilities (pipes, sockets, shared memories) Threaded applications can communicate via the inherently shared resources of the process All threads in a process share the state of the process They reside in exactly the same memory space, see the same code blocks, see the same data A global variable/structure for a multi-threaded single process program is like a shared memory for a multi-process program A traditional multi-process program generally needs a large effort to co-ordinate the various tasks In a MT program the concurrency and the synchronisation of the various tasks is easily obtained with specific system calls that co-ordinate in a natural manner the threads activity

5 5 Simpler signal handler Ø Given the asynchronous nature of the signal handling is difficult to write code that deals with these asynchronous events and does something non-trivial Ø By dedicating a thread to this purpose it is possible to handle asynchronous (external) signals in a simple synchronous fashion æ Single source for multiple platforms Ø Using POSIX-10 threads standard it is possible to insure the portability of the code on different platforms qto each component of the EF subfarm is assigned a thread in the 1x1 scheduling model ä The thread is scheduled directly by the kernel

6 ëthe Distributor and the Collector are implemented simply with two FIFOs ä global structures visible by all the threads ºThe Distributor-FIFO is filled by an injector thread (SFI) that is in communication with the EB The Processing Task threads (PT) ä ä ä take the events from the Distributor-FIFO process the events send the output events to the Collector-FIFO»A thread (SFO) empties the Collector-FIFO $Another thread makes the supervision of the sub-farm, co-ordinates start/stop Sub-farm design procedures and relates to outside world Supervisor EB Storage Distributor FIFO PT SFI PT PT Collector FIFO SFO 6

7 7 The FIFO buffering mechanism insures the data security during its passage through the sub-farm < During processing by PTs the input event is stored in the Distributor-FIFO and is deleted only after it has been rejected or accepted in the latter case the input event is deleted from the Distributor-FIFO only after the corresponding output event is copied in the Collector-FIFO < The Collector-FIFO retains a copy of the output event until it has been sent to the storage by the SFO thread The system is completely data driven The flow of the events in the subfarm is automatically regulated by the availability of the resources These resources are controlled by the level of occupancy of the two FIFOs Indeed are the FIFOs that co-ordinate directly the main tasks of the subfarm through the thread synchronisation objects contained in them

8 FIFO characteristic M Critical sections that contain modifications of the FIFO shared data are protected by mutexes, the fundamental objects that provide the synchronisation of the thread concurrency By protecting a section with a mutex, only one thread at a time can execute this piece of code This prevents race conditions, that appear when many threads try to modify the same shared data at the same time x The control of the level of occupancy of the FIFO (and hence the block/unblock behaviour in case of full or empty FIFO) can be provided by two different kind of semaphores r The standard POSIX semaphores r A generalisation of the POSIX semaphores implemented with mutexes and condition variables (CV) CVs create a safe environment to test the condition pthread_mutex_lock(&m) Thread sleeps on it when false pthread_cond_wait(&c,&m) Is awaken when it becomes true pthread_cond_signal(&c) pthread_cond_broadcast(&c) 8

9 POSIX semaphores and condition variables have a useful property when one wants to stop or pause the subfarm: they are async-safe, i.e. they can be interrupted by a signal (UNIX signal or pthread signal) This simplifies the procedures of sub-farm stop, single thread stop and thread cancellation Indeed in this way it is possible to avoid the lost-wakeup problem that shows up when at stop time one thread sleeps on a semaphore or condition variable The inter-thread communication can be done via global variables protected by a mutex and the associated condition variables x For example in case of subfarm stop a thread can obtain the stop information in a synchronous fashion (a thread can exit only after the end of the processing operations) by checking the global variable value at some defined points of the code as a consequence of a signal, if the thread is sleeping on a semaphore or on a condition variable pthread_kill() pthread_cond_signal() If one needs to suddenly stop a thread, it is possible to use the POSIX cancellation call that permits many options Cancellation state: determines the permission of cancellation Deferred cancellation (polling) Inter-threads communication Cancellation clean-up handlers: responsible for freeing resources, locks and re-establishing data values With these options it is possible to prevent cancellation during critical sections 9

10 1998Signal behaviour and error handling in MT UNIX signals can be divided in two categories N synchronous signals (in general error signals) signals generated as a consequence of a program error (SIGFPE, SIGSEGV, SIGBUS,...) these signals are delivered directly to the offending thread this is a desired behaviour because the thread making the error run itself the (per process) signal handling routines h do error recovering and prevent the loss of events U asynchronous signals signals used to curtail the program (SIGALRM, SIGKILL, SIGSTOP,...) signals that can be used to inform the program of the occurrence of an external event and hence can be used at various stages of communication the signal is delivered to the process the system threads library decides which thread will handle the signal threads have individual signal masks, that the library looks at to decide the recipient there is a single, per-process, handler routine for each signal it is possible to mask all the asynchronous signals of all the threads but one, that is the signal catcher for the whole process 10

11 Future Developments The general EF SMP MT framework with all components is up and running Further develop and test the error handling procedures Integrate Calorec++ as a real processing task Produce measurements to compare with Marseille implementation Scalability test next year on Exemplar V CPUs at CILEA (Milan) C++ implementation of the program (some advantages IFF good class design is achieved) 11