GDB-UPC for GCC-UPC SMP runtime. GDB-UPC features. GDB features. Multiprocessing support in GDB. GDB UPC for SMP runtime Intrepid Technology, Inc.

Transcription

1 GDB-UPC for GCC-UPC SMP runtime Author: Nenad Vukicevic The goal of the GDB-UPC project is to enable debugging of programs that use GCC-UPC s SMP based runtime running on a single computing node. Furthermore, as UPC threads in SMP environment are mapped onto processes created by a system call fork(), or pthreads, created by a library call pthread_create(), this goal can be subdivided into the following: 1. Utilize already existing GDB support for debugging programs with multiple threads in all-stop or non-stop mode of operation 2. Simplify and expand GDB-UPC s mechanisms to access variables with UPC shared qualifiers 3. Enable simple UPC threads startup synchronization 4. Provide an easy to use commands and environment for UPC users With addition of SMP debugging support to GDB-UPC, UPC users will be able to improve a development cycle and easily compile and debug programs on their workstations by using GCC-UPC as their compiler and GDB-UPC as a debugging. No other components or programs are required to be installed or configured. GDB-UPC features In the past, Intrepid teamed with HP to develop GDB support for UPC programs and some support and debugging of the same were added at that time. In addition, over the course of the last few months GDB-UPC has been upgraded and synchronized a few times to the latest CVS head version of GDB. With these updates some new GDB features became available that are useful in debugging UPC programs developed with GCC-UPC s SMP based runtime. Current GDB-UPC features can be further described as GDB specific features (available in GDB s release 7.1 and later OR in GDB scvs development head), and UPC language specific features that were added to GDB-UPC. GDB features Multiprocessing support in GDB In the past, GDB supported a single program that may have more then one thread of execution. The precise semantics of threads differ from one operating system to another, but in general the threads of a single program are akin to multiple processes - except that they share one address space (that is, they can all examine and modify the same variables). In Linux terminology this amounts to 1

2 a single process with multiple pthreads. The recent developments in GDB (available in GDB s release 7.1 and later) allow the control and debugging of threads that are mapped on top of regular Unix processes, where code is shared across all processes (inferiors in GDB s terminology) and data is private to each one of them. On the user level the same GDB thread s interface is kept in place, but GDB threads refer to processes or pthreads. In the future discussion a thread will refer to GDB thread, or UPC thread as label as such. The following GDB options are used to control debugging environment with multiple processes: 1. Debugging Forks GDB s Linux support has ability to control and debug both the parent and child processes after the fork system call. An option detach-on-fork controls the mode of operation which can be on, or off. To control and debug both processes after the fork, this mode must be off Non-stop Mode of operation For multi-threaded targets, GDB supports an optional mode of operation in which you can examine stopped program threads in the debugger while other threads continue to execute freely. This minimizes intrusion when debugging live systems, such as programs where some threads have realtime constraints or must continue to respond to external events. This is referred to as non-stop mode. An option non-stop is used to control the environment and can be on or off. In non-stop mode, an additional argument -a is added to some common commands (e.g. continue a) to apply command to all threads in the system. Mode.html#Non_002dStop-Mode 3. All-stop Mode of operation In all-stop mode, whenever your program stops under GDB for any reason, all threads of execution stop, not just the current thread. This allows you to examine the overall state of the program, including switching between threads, without worrying that things may change underfoot. Mode.html#All_002dStop-Mode 4. Schedule multiple processes Option schedule-multiple, controls of all of the threads are continued in all-stop mode. When on, execution commands (like continue ) resume all threads of all processes. By default it is off, and only the current thread is affected. 2

3 Mode.html#All_002dStop-Mode 5. Scheduler locking mode of operation Option scheduler-locking applies in All-stop mode of operation and prevents other threads from running while single stepping the current thread. The other commands like continue, until, or finish behave in the manner, other threads are free to run if this option is set to off. Mode.html#All_002dStop-Mode 6. Asynchronous command execution (background execution) GDB s execution commands have two variants: the normal foreground (synchronous) behavior, and a background (asynchronous) behavior. In foreground execution, GDB waits for the program to report that a thread has stopped before prompting for another command. In background execution, GDB immediately gives a command prompt so that you can issue other commands while your program runs. Execution.html - Background-Execution Event Notification in GDB - Observers Another aspect of GDB that is useful for development of multiprocessing debugging options is a GDB observer. An observer is an entity which is interested in being notified when GDB reaches a certain state, or certain events occur in GDB. The entity being observed is called the subject. To receive notifications, the observer attaches a callback to the subject. One subject can have several observers. Observers are only available to GDB developers, and they are called just before the control is about to be returned to the user program. There are many GDB observers that deal with the loading and unloading of DLLs, object files, creation of inferiors and threads, breakpoints, etc. Here is a sample list of observers that are useful in developing of multiprocessing debugging support: inferior_created called when GDB just connected to an inferior, at startup or with Unix fork() system call. new_thread called when new thread is being created thread_exit called when thread has exited new_objfile called when new object file or library loaded More information on GDB observes can be found in GDB Internals document - GDB- Observers. 3

4 UPC language support features As stated earlier, support for UPC language has been added to GDB as part of the previous project in cooperation with HP. The following features were added: DWARF debugging information extensions for UPC UPC shared type qualifier support UPC threads number indications UPC Debug Assistant (UDA) server with Debug Assistant (DA) plug-in library GDB-UPC Multiplexer (GUM) server that controls multiple UPC threads This debugging environment was relatively complex because of the distributed nature of UPC programs, as well as the presence of the two servers (UDA server and GUM) developed and maintained on their own, outside of the GDB-UPC or GCC-UPC source trees. GDB-UPC interfaces to the UDA server via TCP/IP socket, both as a client (requesting shared data info) and as a server (responding for symbols and values of local data). The UDA server is used as an interface between GDB and UDA plug-in by sending GDB s requests to UDA plug-in and providing call back interfaces for UDA plug-in requesting data from GDB. Debug Assistant (DA) plug-in library interface API was developed by James Cownie in collaboration with developers of several UPC implementations, including GCC-UPC and HP UPC. Each UPC implementation needs to provide its own UDA plug-in. GCC-UPC provided its own as part of the previous project with HP (see GDB-UPC Design Proposal, August 20, 2007). While the above approach might have some advantages in licensing terms for some of the UPC implementers, there are some disadvantages: 1. UDA plug-in is a version locked to the GCC-UPC compiler s runtime. 2. Presence of a UDA server makes this architecture complex for simple debugging cases 3. Access to shared data goes through a callback mechanism to GDB 4

5 GDB-UPC for GCC-UPC SMP GCC-UPC s SMP runtime uses system or library calls to create new UPC threads. The system call fork() is used to create a new process, while pthread_create() creates a new pthread. As the latest GDB is already capable to attach, control, and debug multiple threads, it is obvious that UPC SMP runtime debugging can be achieved with minimal intrusion to the core of GDB s functionalities. GDB-UPC changes for UPC Debug Assistant Further development of UPC Debug Assistant for GCC-UPC SMP is driven by the need to simplify its architecture, and place the relevant source code in its natural place. This amounts to the following: 1. Eliminate the need for the UDA server and use the UDA plug-in directly in GDB-UPC 2. Develop UPC debug assistant as a part of the GCC-UPC SMP runtime library, thus eliminating SMP runtime version knowledge in the UDA plugin. To achieve this, the following needs to be developed; 1. UDA plug-in as a shared library UDA plug-in will become part of the GCC-UPC compiler. It is configured and compiled in the same environment as GCC-UPC runtime, thus eliminating possible mismatch in configuration and versions. Appropriate interfaces and callbacks are similar to the already existing UDA plug-in. This is also described in Intrepid s GDB-UPC Design Proposal, August 20, 2007.GDB interface to Debug Assistant A new GDB interface to UDA plug-in must be developed with the following in mind: a. UDA plug-in is loaded as a shared library with particular interfaces and call back functions b. Location of UDA plug-in can be specified in configuration time, with environment variable, or simple placing the UDA plug-in on the library search path. 5

6 UDA server functionality is still supported but eventually will be phased out of GDB-UPC. DLL interface and call backs to GDB-UPC are compatible with the previously developed UDA plug-in library. GDB-UPC changes for SMP runtime support Some additional changes need to be made to accommodate GCC-UPC SMP runtime programs. UPC program recognition As GDB-UPC needs to be able to debug UPC as well as non-upc programs, a method must exists to identify loaded programs. A new object file observer, 6

7 new_objfile(), is used for this purpose as it is called every time a new program or library is loaded by GDB. Currently the presence of symbol THREADS indicates that this is a UPC program. To better diagnose the presence of UPC programs the future releases of GDB- UPC must do additional checking and verify that the object file used matches the specified or supported UDA plug-in. 1. Presence of symbol upc_main indicates the GCC-UPC compiled program 2. Presence of some specific UPC DWARF symbols can be used to indicate that the UPC program has been compiled correctly for GDB-UPC debugging. Otherwise, a warning can be issued. For proper UPC debugging, program must contain UPC symbols and must be compiled with UPC DWARF extension. UPC thread recognition GDB s thread interface announces each thread as they are being attached to the pool of managed threads. Some of the threads are actual UPC threads while some others might not be. For example, this GDB info threads command output shows three threads, where only two of them are UPC threads: 3 UPC Thread 1 0x d1d7 in sched_yield () 2 UPC Thread 0 0x d1d7 in sched_yield () * 1 process x c644 in wait () A new thread observer, new_thread(), is used to recognize a UPC thread by examining the new thread s data space for proper value of variable MYTHREAD. However, for the current GCC-UPC SMP runtime the following approach must be taken: First created thread is always a monitor thread with UPC thread id of negative one (-1) Other threads are assigned UPC thread numbers from 0 till THRADS-1 in sequential order as they are being created and attached for debugging Assignment of the UPC thread numbers are verified in the thread startup synchronization phase as MYTHRAD variable is valid at that time for each of the threads. Appropriate adjustments are done at that time. 7

8 Number of UPC threads (THREADS) GDB-UPC needs to know how many UPC threads are available in the system for various reasons (allocation of data structures, needed by UDA plug-in, etc.). Number of UPC threads in the program can be determined in multiple ways: 1. Value of THREADS variable if program is compiled for static number of threads 2. Value of THREADS variable after UPC run-time is initialized if the program is compiled for a dynamic number of threads 3. Debugged program argument value that specifies number of threads The number of UPC threads is determined in the new_thread() GDB observer when the UPC thread 0 is being created. This is the place where internal SMP runtime variables are already initialized and THREADS variable has the proper value. GDB to UPC thread number conversion GDB internals deal with GDB numbers that have values going from one (1) to the maximum number of threads in the system. At the same time, UPC threads start with number zero (0) to the value of the THREADS-1 variable. GDB s observer new_thread() is also used to map UPC and GDB threads. GDB s thread information structure is extended to record and hold a UPC thread number too. A less intrusive approach is to keep a separate record of mappings between these two numbers (simple hash tables). Expanding GDB s thread structure seems like a reasonable solution, even though the existing relationship between GDB and UPC thread numbers is a simple one (difference is always 2). Debug Assistant initialization For GDB-UPC to handle shared variables, Debug Assistant shared library library must be initialized. GDB-UPC initializes UDA plug-in in a new_thread() observer when the UPC thread 0 is created. Environment variable UDA_GUPC_PLUGIN_LIBRARY is used to specify a path for the UDA plug-in shared library. Appropriate errors are issued if plug-in library cannot be found or the interfaces are not compatible. 8

9 If this environment variable is not present, the old method of connecting to UDA server is tried. This interface will be phased out in next releases of GDB-UPC. Startup synchronization of UPC threads At the start of the UPC program debugging it is necessary to synchronize all of the UPC threads, before any of the user code is executed. GCC-UPC run-time uses a mechanism similar to MPICH2 in order to synchronize threads at startup. This is also a method used by Totalview debugger to attach to MPI and UPC based programs: More information can be also found in Intrepid s GDB-UPC Design Proposal, August 20, GDB-UPC for SMP based runtime is able to use a simplified version of this startup mechanism as SMP runtime is used on the same host and all the threads are by default attached to the debugger from the creation time. Two GCC-UPC SMP runtime variables are important for the startup synchronization: MPIR_debug_gate controls the UPC thread s gate. If the gate is lowered (value of 0), thread is not allowed to proceed and will spin around the gate with a processor yield in every iteration. MPIR_being_debugged signals to all threads that the program is being debugged and synchronization is required. To initiate startup synchronization, GDB-UPC simple sets the value of MPIR_being_debuged variable and to release all threads from the gate, a value of 1 is written to each thread s variable MPIR_debug_gate. To accomplish startup synchronization, the new_thread() observer is used to initiate startup synchronization, by writing a value of 1 in MPIR_being_debugged variable of each UPC thread. As startup synchronization might not be required in all the cases, the whole process can be initiated with a GDB variable (e.g. set upcstartgate on) that is checked by the new thread observer. A special GDB variable upcstartgate is used to enable or disable UPC startup synchronization (set upcstartgate on/off) By default, UPC startup synchronization is enabled. A new command upc-sync is introduced to complete thread synchronization and switch debugging mode into UPC mode (upcmode). It also does verification of UPC thread numbers to GDB thread numbers, and turns on collective features of breakpoints and single stepping. 9

10 UPC mode for context and commands The current implementation of GCC-UPC SMP run-time contains, beside UPC threads, a process called a monitor thread that is responsible for creating all UPC threads at startup. This thread has a unique GDB thread number and will be shown in every thread view during a debugging session. Also, as UPC threads are able to create other Unix processes, it is possible to have more than one non-upc thread in a debugging session. GDB-UPC supports a UPC mode of operation where certain GDB commands apply only on UPC related threads. At the same time, thread information is shown for UPC threads only. For example: - info threads displays UPC threads only - c continues all UPC threads - interrupt stops all UPC threads - thread apply all cmd applies cmd only to UPC threads A GDB variable upcmode (e.g. set upcmode on/off) is used to control the UPC mode of operation. In UPC mode, GDB s prompt is changed to (gdb-upc). Regardless of UPC mode all UPC threads are always marked as UPC threads with their correct UPC thread numbers. In UPC mode all UPC threads are referenced with their UPC numbers. The following table shows all the differences between UPC and non-upc (GDB mode) modes: Commands/Info GDB mode UPC mode prompt (gdb) (gdb-upc_ info thread command shows all threads Shows UPC threads only thread number gdb thread number (with UPC threads identified) UPC thread numbers c, n, interrupt commands current threads all threads async commands asynchronous if feature set and target support always asynchronous if target support Collective breakpoints and single stepping GDB manages threads independently of each other and each thread reports its events without having any knowledge of the the other threads state. For each thread event (e.g. breakpoint reached) an alert is sent to the user in form of the 10

11 message on the screen and N number of threads will result N messages on the screen. UPC collective breakpoint feature allows the UPC user to insert a breakpoint that is announced to users once all the threads reach it. Placing a breakpoint at the upc_barrier is a good example for collective breakpoint. A warning is issued if multiple threads reach different collective breakpoints which is indication of an error in the code logic or collective breakpoint was inserted in. Collective breakpoint feature is available only in non-stop mode of operation. At the same time GDB must be placed in the upcmode of operation and it is turned on by default on upc-sync command. Behavior of the following GDB commands is changed for better support of collective breakpoints: a) breakpoint Breakpoint inserted in all threads (default GDB behavior) is assumed to be a collective breakpoint (no special breakpoint option is required). A breakpoint inserted in specific thread (e.g b upc_main thread 4) is a noncollective breakpoint. b) contune, next, step These commands have no effect on threads stopped on collective breakpoint until all the threads reach it. For example, user might have a case where some of the threads reached collective breakpoint, while others reached thread specific breakpoints. Next continue command will continue only threads that are NOT stopped on the collective breakpoint. UPC mode of operation also supports collective single stepping where all the threads collectively step over the code. No special commands or stepping options are available for collective stepping. Instead, commands like next, step, etc. behave as collective stepping commands in UPC mode of operation. The following controls are used for collective breakpoint and single stepping: set/show breakpoint collective on/off Turns OFF or ON collective breakpoint feature. There might be a case where user wants to see breakpoint announcements from all the threads reaching the breakpoint (e.g. timing issue where some of the threads take a long time to reach a breakpoint). set/show breakpoint collective_stepping on/off Turns OFF or ON collective single stepping feature. This is option is useful in cases where user wants to debug a particular thread. maint collective clear [all,n] 11

12 Maintenance command to clear collective breakpoint state on all or particular thread. In the course of debugging there might be a situation where different collective breakpoints are reached by different threads. Most likely this is caused by placing a collective instead of thread specific breakpoint in the thread specific section of the code. For example, placing a collective breakpoint on line 42 in the following code: 40 if (!MYTHREAD) 41 { 42 thr_diag++; 43 } will cuase only thread 0 to reach it. In this case, use maint collective clear 0 to release thread 0 from the collective breakpoint on line 42 and allow it to continue and reach a possible collective breakpoint where all the other threads are stopped. UPC threads exit GDB s observer thread_exit() is used to announce an exit of a UPC thread, the same way they are announced when they are create. Once the last UPC thread is completed, GDB-UPC is switched back to GDB mode (non UPC mode). GDB-UPC non-stop v. all-stop mode in UPC mode GDB-UPC for SMP runtime supports GDB s non-stop and all-stop modes. The main difference between these two can be summarized in a way threads stop on debugging events. Some commands in non-stop mode is further modified to accommodate UPC mdoe of operation.: 12

13 Events or All-Stop Non-Stop Commands non-upc mode UPC mode Breakpoint stop All threads stop No effect on other threads No effect on other threads Continue All threads execute Only the current thread execute All threads continue, unless stopped on collective Single stepping Current thread single steps and other threads are allowed to run Only the current thread execute breakpoint All threads execute a step, unless stopped on collective breakpoint In both cases, the UPC mode of execution makes sure that the commands apply to the UPC threads only. 13

14 Contents Contents GDB-UPC for GCC-UPC SMP runtime... 1 GDB-UPC features... 1 GDB features... 1 Multiprocessing support in GDB... 1 Event Notification in GDB - Observers... 3 UPC language support features... 4 GDB-UPC for GCC-UPC SMP runtime... 5 GDB-UPC changes for UPC Debug Assistant... 5 GDB-UPC changes for SMP runtime support... 6 UPC program recognition... 6 UPC thread recognition... 7 Number of UPC threads... 8 GDB to UPC thread number conversion... 8 Debug Assistant initialization... 8 Startup synchronization of UPC threads... 9 UPC mode for context and commands... 9 Sample Debugging Session with GDB-UPC for SMP runtime Error! Bookmark not defined. 14