The PDBG Process-level Debugger for Parallel and Distributed Programs

Transcription

1 The PDBG Process-level Debugger for Parallel and Distributed Programs João Lourenço José C. Cunha {jml, Departamento de Informática Faculdade de Ciências e Tecnologia Universidade Nova de Lisboa Portugal Abstract In this paper we discuss several issues concerning the design and implementation of a debugger for parallel and distributed applications. This debugger uses a clientserver approach to isolate the debugging user-interface from the debugging services, by way of a two-level structured approach: the component-level to observe and act upon individual processes; and the coordination-level to observe the interprocess relations and act upon them. A formal specification of the expected behavior of a processes under debugging is presented, and the debugging functionalities required to support this model are evaluated and its implementation described. Special relevance is given to the tool-interfacing support functionalities of the debugger and how they were implemented. 1 On Parallel and Distributed Programs A program is the realization of the specification of a concept. When the specification is incomplete or incorrect it doesn t model adequately the initial concept, and there is a bug in the specification. From the concept specification to its implementation there can be other set of mismatches, these ones called implementation bugs. It s the aim of the programmer to ensure that the implementation fulfills the specification requirements, and there are specific tools to aid in this goal, such as testing and debugging tools. Testing tools help in the detection of implementation deviations to the specification and, once Submitted to SPDT 98. such a deviation is detected, a debugger can be used to help in the determination of the specific program factors that generated the detected malfunction. The emphasis of this paper is on debugging tools to parallel and distributed applications, with support for integration into full software engineering environments. functionalities The expected functionalities in a debugging tool for parallel and distributed applications can be summarized as: Inspection and control functionalities To locate the origin of a program malfunction, one must first stop program execution and then inspect the computation state and the program code. If some hypothesis is made concerning the cause of the bug, there is some action upon the computation state, changing it according to some debugging method. Finally the execution is resumed and the whole process is repeated once more until the real cause of the bug is found. This cycle is illustrated in Fig. 1. When the application being debugged is composed by many processes and/or threads, the inspection and control of individual components may become impracticable, and the need arises to look into the application components in a structured way. We are following a two-level structured approach to achieve this goal: 1. Component-level. At this level, the performed debugging actions act upon individual application components, such as processes and threads. 1

2 No Testing Start / Resume execution Provide testing values Observe program behavior Malfunction detected? Yes Apply corrections Formulate hypothesis Inspect program Stop execution Figure 1: The testing & debugging cycle 2. Coordination-level. at this level, the ability to define coordination groups of processes and/or threads is provided, and the inspection and control commands can act upon individual components or groups. the natural problems of intrusion and execution reproducibility, as well as the study of race conditions, fit into this coordination level [8]. in this paper we are focusing in the component-level of a parallel/distributed program, so we re not covering here the study of the interactions between the multiple components of a parallel/distributed application, being such study, in the context of our work, is still under investigation (see Sec. 5). interfacing with other tools and environments. the use of a debugging tool should always be complemented by the use of other software development tools, such as testing or visualization tools. the consistent integration of such tools provide an environment with all the previous functionalities, plus the new ones resulting from the exploitation of the interactions between tools. heterogeneity. the components of a distributed application may spawn through a large set of machines, with different hardware and operating systems, using multiple communication protocols and can rely on different programming models, of the logical, functional and procedural styles. a debugger for distributed applications must be flexible and adaptable in order to support all such levels of heterogeneity. Previous work In the context of the european community SEPP (Software Engineering for Parallel Processing) [13] and HPCTI (High-Performance Computing Tools for Industry) [14] projects of the COPERNICUS programme, we participated in the development of an integrated software engineering environment for parallel and distributed programs. We have developed a prototype for debugging of parallel and distributed applications: the DDBG tool [2, 3]. In this paper we present a debugging tool the PDBG debugger that improves DDBG, in order to overcome the limitations detected in this tool s architecture and its functionalities. Organization of the paper The organization of the paper is as follows. The requirements, properties and functionalities of PDBG are described in Sec. 2. The prototype implementation is presented in Sec. 3. In Sec. 4 some related work is described and compared with our approach. Finally, in Sec. 5 the conclusions are presented, together with same ongoing and future work. 2 A Debugger for Parallel and Distributed Programs Due to the large diversity of programming and computational models, with different levels of concurrency (process- and thread-level), many difficulties arise in the definition of a universal debugging interface that can conform with all those models. This suggests the specialization of the debugging functionalities, e.g. process or multi-threaded oriented, and the support of group-based coordination of components. In the rest of this section we ll model the behavior of a process under debugging and an appropriate process-level debugging tool is presented that conforms with the proposed model. 2.1 Process States and Transitions The behavior of a process under debugging the Process can be specified by the automata B = (S; A; ; fs 0 g; fs t g) 2

3 where S is the set of process states (corresponding to the nodes in the graph in Fig. 2), A is the set of actions over this process, is the transition function, s 0 = NON-EXISTENT and s t = TERMINATED are the initial and terminal states respectively. <Create> Non-existent load() Ready run() attach() continue() next() nish() step() return() Detached Stopped Running detach() <Breakpoint> <Signal> interrupt() killproc() <Exit> Terminated Figure 2: The process state transitions <Exit> The alphabet corresponds to the set of arc labels (some of which are presented in Fig. 2) and is defined as A = A s [ A d A s is the set of system actions, which are presented with angle brackets in the figure (e.g. <Create>), and A d is the set of debugging actions, which are presented in the figure suffixed with parenthesis (e.g. run(), detach()). The set of process states is S = fnon-existent; READY; RUNNING; STOPPED; DETACHED; TERMINATEDg where each state has the following properties: NON-EXISTENT. This is the initial state of every process in the system. READY. The process image has been loaded into memory, but its execution has not started yet. The process is in a ready_to_run state, under control of the debugger. RUNNING. The process is running (from the debugger point of view). Even when the process is scheduled out by the underlying operating system it s still kept in the RUNNING state. STOPPED. The process has been stopped as a result of an action (a debugging command or a system exception). There is a large number of actions related with the inspection/changing of the internal state of the process that can only be applied if the process is in this state, e.g. get_var(), list_stack(). DETACHED. The process is not known to the debugger environment and its internal state is not relevant. The main difference between this and the RUNNING state is that, when in the DETACHED state, the process is not aware of breakpoints and the occurrence of a system exception is processed in the conventional way, i.e. it doesn t force a transition to the STOPPED state. TERMINATED. This is the terminal state of every process in the system. The transition function is defined as : (S A)! S which is represented in Fig. 2 as arcs labelled with the corresponding action, e.g. (RUNNING; <signal>) = STOPPED. A transition i is silent if its initial and final states are equal, i.e. i (s i ; a) = s i ) silent( i ) s i 2 S; a 2 A For example: (READY; set_breakpoint()) = READY. For simplicity, the arcs associated with silent actions were omitted in the Fig. 2 (e.g. list_source(), set_var()). The language L(B) accepted by this automata is the set of words over A, such that such that (s; ) = s L(B) = fw 2 A : (s 0 ; w) = s t g (s; aw) = ((s; a); w) s 2 S; =<empty word> s 2 S; a 2 A; w 2 A A History of Actions is a word over A, representing a complete sequence of actions over the target process, from its initial to the terminal state, i.e. H a = [a 0 a 1 : : : a n?1 ] : [a 0 a 1 : : : a n?1 ] 2 L(B) For example, the sequence A = [load() run() <signal> get_var() killproc()] is a possible History of Actions. A History of States is the complete sequence of states of the target process for a given History of Actions, i.e. H s = [s 0 s 1 : : : s n ] : (s i ; a i ) = s i+1 For example, the sequence a i 2 H a ; s i 2 S S = [NON-EXISTENT READY RUNNING STOPPED STOPPED TERMINATED] is the History of States for the History of Actions A. 3

4 A History of Transitions is the complete sequence of different states of the target process for a given History of States, i.e. H t = [s 0 s 1 : : : s n ] : [s 0 s 0 s 1s 1 : : : s ns n ] = H s ^ 8 0 i < n; s i 6= s i+1 For example, the sequence T = [NON-EXISTENT READY RUNNING STOPPED TERMINATED] is the History of Transitions for the History of States S. 2.2 A Process-level Debugger The PDBG debugger is a distributed debugger that uses a client-server model to isolate the debugging functionalities from the user interface, and whose set of component-level services act upon the target processes in such a way that their behavior is conforming with the model presented above The Design Options and Architecture of PDBG The PDBG debugger is composed by three main entities: the Library to provide access to PDBG services, the Module to implement these services and the Driver to execute them (see Fig. 3). Client Tool Library Module Easy interfacing and integration. Any client tool can include the PDBG debugging library, having a simple and intuitive procedure-call based interface to the PDBG debugging functionalities. On the other hand, the capabilities of asynchronous interactions with the client tools (see Sec ) give PDBG an extended support for integration, allowing the multiple client tools to share a coherent view of the debugging system and of the target application being debugged. Extended support for heterogeneity. PDBG relies upon system-dependent existent debuggers (Component-level Debuggers) to actually act upon the target processes, thus providing independence regarding the machine and the operating system. The PDBG debugger provides a well defined interfacing mechanism that allows system-dependent component-level debuggers to be integrated into PDBG just by designing an appropriate debugger controlling module. This provides support to new execution environments and programming models, e.g. the PDBG debugger can control an heterogeneous target application with possibly multi-lingual components. Independence from the target application s communication layer/protocol. The PDBG internal architecture uses its own communication layer that can be mapped into distinct communication platforms, such as PVM, MPI or TCP/IP. Besides a degree of machine/os independence, this allows the debugger to use a different communication protocol from the one used in the application, reducing the interference on the application executing environment The PDBG Services Driver Process Application Driver Process Figure 3: The PDBG tool This software architecture results from the set of requirements that have been considered from the very beginning when designing the PDBG debugger, namely: The interactions between client tools and PDBG have two distinct facets: synchronous and asynchronous: Synchronous interactions result from the natural semantics in a (remote) procedure-call interface, such as the one provided in PDBG, and the client tool waits for the command to be completed and the output data to be affected to the returning parameters. For each procedure invocation a status code is returned indicating success or unsuccess in the processing of the requested service. Depending on the requested service, some specific data may also be returned, e.g. a variable contents inspection command must return the variable value too. 4

5 Asynchronous interactions result from deferred commands, or from commands that take an undetermined time to be executed, e.g. continue the execution of the process until a breakpoint is reached or the process terminates. In this case the client tool just receives an indication that the service request was accepted (or not) by the debugger and will be notified later when the service execution is completed. The services provided by PDBG can be grouped into four classes: 1. session control services. These services integrate (remove) a client tool into (from) a debugging session, launch a new process under PDBG control, and attach (detach) the PDBG debugger to (of) an existing free (controlled) process. 2. Process execution control services. These services act upon processes, directly controlling the execution state and flow path, e.g. start running a READY process or forcing a RUNNING process to stop. 3. Process internal state inspection and modification. When interacting with the target application, some of these services may affect the internal state of the process, e.g. changing a variable value. However, they never change the process execution state (as defined in Sec. 2.1). 4. Event processing and management services. This aims to support multiple coherent views (provided by multiple client tools) concerning the same target application. The client tools can register their interest in being notified about what s happening with the processes of the target application. This notification reaches the client tools as Events that will activate client handling functions. The complete description of the debugging commands supported by PDBG is outside the scope of this paper but, due to its relevance to understand the tool functionality, the topic 4. is explained in detail below Events in PDBG To support the asynchronous interactions with the client tools, two different types of Events have been defined: Transition events. The PDBG debugger generates a Transition Event whenever there is a non-silent transition in the state of the target process. In this way the sequence of Transition Events will match the History of Transitions as defined in Sec Action events. In a way similar to Transition Events, the PDBG debugger generates an Action Event either for each system- or tool-generated action (command) applied to a process under debugging. Thus, the sequence of Action Events will match the History of Actions as defined in Sec The event-related PDBG services fit into one of the following classes: Event Handling. Event handling in PDBG requires each client process to explicitly register its interest in a specific type of event. By default, events are not delivered to the client processes, but by calling sethandler(pid?list, ev?type, ev?handler) the invoker client process declares its interest in being notified of all events of the specified type: EV_TRANSITION to receive notifications on state transitions occurring in the processes identified by pid_list; or EV_ACTION to receive notifications on debugging- or system-actions issued over these same processes (in pid_list). On event occurrence the specified function ev_handler (from the invoker code) is called in the context of the invoker process 1. The execution of the handler is done concurrently with the normal execution of the debugger, i.e. the debugger doesn t wait for the handler to terminate before proceeding. This event handler function will receive as argument a structure containing the relevant data for the event. Event Notification. On event notification, the handler function receives the as input argument: event(ev?type, ev?arguments) The ev_type field in the structure identifies the event type: EV_TRANSITION or EV_ACTION. The ev_arguments field contains general event information, namely a unique event identifier, the event type, and the identifier of the process who generated the event, plus event specific information that depends on each event type, such as information on the process state, or on the name, input arguments and results of the PDBG service requests. The incorporation of the event type in this structure allows the same event handler for different types of events. 1 The implementation of handler invocation is dependent on the programming model being single- or multi-threaded, and is not part of the PDBG interface definition. 5

6 When a client tool registers a handler for debugging action events, it will receive a notification every time a command is completed, independently of the tool that sent the command. This event-notification feature provides the basic engine to provide a coherent view of the debugging system to the client tools, as they can be permanently informed about what s going on concerning the target application. 3 PDBG Implementation The design of the PDBG tool has taken into consideration the requirements and functionalities described in Sec. 2.2, concerning inspection and control, interfacing with other tools and heterogeneity. To better achieve this goals, a three-layer architecture has been defined: 1. Monitoring and Control layer. This is realized in DAMS (Distributed Applications Monitoring System). This layer is briefly described in Sec Component-level layer. The basic DAMS system can be extended with Service Modules which provide specialized services, such as performance monitoring, debugging or resource-management. In the case of PDBG, a new Module has been developed, and is described in Sec Coordination-level layer. A set of coordinationlevel modules, such as Replayer Module or a Racedetection Module will extend both, control and inspection functionalities of PDBG. This layer 3 is still under development and is not reported here. 3.1 The DAMS System The DAMS tool aims to provide a basic layer for distributed monitoring and control, that clearly separates the low-level mechanisms to support the implementation of application-level services (such as debugging, profiling, and resource management) from their implementation. The design and implementation of DAMS is neutral regarding the programming and computational model of the target application, as well as the communication and process management protocol(s) used. As a result, the DAMS architecture offers a very expressive environment to monitor and control heterogenous application components. The DAMS system is composed by a set of entities of the following types (see Fig. 4): 1. Service Libraries. These libraries are linked to the client processes and provide (remote) procedure-call based interfaces to access the DAMS services, hiding communication details between the client processes and the DAMS system. 2. The Service Manager. Is responsible for the global supervision of the DAMS system. In order to keep the architecture flexible and expansible, it does not handle the services requested by the client tools, but just forwards them to the appropriate Service Module. 3. Service Modules. Each Service Module encapsulates the functionalities provided by a certain class of services (e.g. debugging), and implements the higherlevel system-independent part of those services. All communication between different Service Modules and between the Service Modules and the Service Manager is done through a well defined interface but, for performance reasons, all the Service Modules are linked to the Service Manager. 4. The Local Managers. On each node of the physical architecture there is a Local Manager process with two main functions: implementing a communication channel between the Process Driver and the associated Service Module, and the managing the local driver processes. 5. The Process Drivers. For each Process that is an associated Process Driver controlling and monitoring its behavior 2. As needed, Process Drivers are dynamically launched by the Local Manager and attached to Processes, and are responsible for the systems-dependent and direct enforcement of commands to each associated Process. A well-defined protocol for handling requests to Process Drivers and collecting their answers is used, in order to facilitate the replacement of a Process Driver by a new one with the same functionalities. When performance is critical, it s possible for a Service Module and its associated Process Driver to communicate directly using their own protocol, bypassing the channel established through the Local Manager, but loosing some portability. In order to support each class of services, one must define a pair (Service Module, Process Driver), e.g. a Service Module and a Driver. 2 Depending on the semantics of the Process Drivers, it will be possible (or not) to have two or more Process Drivers sharing one Process. 6

7 DAMS Client Processes Library Machine A Graphical Interface Module Service Manager Local Manager Driver CLD Controler Component Level Debugger Driver CLD Component Level Controler Debugger Process 1 Process 2 Text Interface (console) Machine B Local Manager Driver CLD Component Level Controler Debugger Process 3 Service Request (and Reply) Event propagation Figure 4: The implementation PDBG in DAMS In Fig. 4 it is possible to find a scheme of the DAMS architecture which is instantiated with a debugging service. A more complete description of the DAMS system can be found in [4] 3.2 The PDBG Debugger The PDBG debugger has been implemented as a service provided by the DAMS system, as illustrated in Fig. 4, and supports a set of component-level services to debug sequential and distributed C programs, e.g. C/PVM and C/MPI programs. As stated before, to implement this new service on top of DAMS, the following components are needed: (i) a library the Library that is linked to the client tool; (ii) a module the Module linked to the Service Manager and executed in a multithreaded environment; and (iii) a process the Driver to control each process of the Application. The Library The Library implements the interface between the client tool and the debugging tool, by providing a remote procedure-call model to access PDBG. Currently there is an interface for applications written in C, and a C++ object-based interface is under development. To access a PDBG service, a client tool calls a (local) function in the Library. This function sends the arguments to the Module, by way of the DAMS internal communication layer, then waits for a reply from the Module and sets the return parameters of the service call with the reply data. The Module The Module is linked into the DAMS Service Manager, and as the Service Manager is multi-threaded, the Module must be thread-aware (e.g. reentrant). This module supports all the abstractions and functionalities presented in Sec When the Module receives a message with a service request (from the Service Manager that, in turn, has received it from the Library), it unpacks the message contents, identifies the requested service and calls the local function that implements the service. According to the service requested by the client tool, one or more new service requests can be issued to the Process Drivers, to execute the low-level system-dependent set of actions necessary to satisfy the service requested by the client tool. Once the results of all the services request to the Process Drivers are collected, they are packed into a reply message that is sent to the Library in the client tool (again, via the Service Manager). The Driver In the current PDBG implementation, the DAMS Process Driver is actually implemented as two processes: 1. The Component-level Debugger Controller receives the system-independent debugging commands from the Module and converts then into a set 7

8 of system dependent commands to send to the Component-level Debugger. 2. The Component-level Debugger is a system-dependent debugger that actually performs the inspection and control actions on the Process. Due to its portability, easy access and non-commercial status, we have adopted the GNU gdb as the Component-level Debugger. The main drawback of using gdb is the performance, as between 60 and 90% of the elapsed time in a command execution, from the moment the client tool calls the Library function until it returns, is spent inside gdb. The usage of a Driver to implement the system-dependent services provides PDBG with a very useful plug-in functionality, where a Driver (e.g. using the GNU gdb) can easily be replaced by another one (e.g. using dbx). 4 Related Work There are many current efforts on the field of parallel and distributed debugging and related topics. Because we cannot cover them all here, we have chosen three of those ongoing works, which we find relevant, to represent different approaches to the topic. They are briefly presented and compared with our own approach. The HPDF (proposed) Standard The High-Performance Forum (HPDF) [1] is a collaborative effort between researchers and industry, aiming to define a standard for parallel debuggers. As of Version 1 of the standard, a command based (nongraphical) interface has been prepared, specifying either syntax and semantics of the proposed services. The definition of graphical interfacing and complex I/O operations are still under work. According to HPDF, a parallel debugger is either a thread-oriented debugger, a process-oriented debugger or a hybrid debugger, and sets of required and recommended services have been defined for each of them. Actually PDBG is being developed alongside the HPDF standard, and the mapping between PDBG functionalities and the suggested ones for process-oriented debuggers has not been effectively made, but from the analysis of the proposed standard we strongly believe that PDBG basically provides a superset of the required functionalities, incorporating even many of the recommended ones. In this regard, the tool integration features of PDBG, presenting a unified model for the system- and usergenerated commands (e.g. system signals and client tool requests) with a well defined set of events and an implementation-independent operational semantics, is a major step-forward to the integration of parallel debuggers in more complete and complex program development environments [7, 9]. The OMIS (proposed) Standard The On-line Monitoring Interface Specification (OMIS) [11] aims to define an open interface to connect software development tools for parallel software, including e.g. debuggers and performance analyzers. Concerning debugging, OMIS has a more broad scope than HPDF, and addresses the definition of the communication protocol and formats of the information transfered between the tools and the application. The processing of the replies to OMIS service requests can be synchronous or asynchronous, depending on a callback handler being (or not) defined when the service was requested. In PDBG the communication between the client tool and the debugger is supported by a library (language dependent) that is linked to client tool code, and all service request are processed synchronously. This option is much more limited than the one proposed by OMIS and our plans are to evolve to an OMIS compliant protocol. The tool integration features of PDBG are not addressed by OMIS, since the callback feature of OMIS, used to asynchronously process the reply to a service request, cannot be used to process the replies of service requests from other processes, as the PDBG event handlers can do when processing the debugging-action [4]. The p2d2 Distributed Debugger The p2d2 distributed debugger [6] uses a client-server approach, with a well defined interface, promoting portability by isolating the system dependent code into a debugger server. There is an user-interface capable of displaying and controlling many processes, individually or associated in groups. The GNU gdb is used as a processdebugger, and to support asynchronous interactions between gdb and the user-interface, a callback method is provided, and at call-time the user must specify the function that will be called when the request is completed. The approach used in the development of the PDBG debugger,x was to concentrate in the debugging functionalities necessary to support tool integration, where the graph- 8

9 ical interface (currently under development) is considered just another client tool, that can co-exist with other client tools such as command-line debugger interfaces or full graphical parallel programming environments [7]. 5 Conclusions In this paper we have discussed several issues concerning the design and implementation of a debugging tool. In this section we identify the main contributions of our work, by discussing the most distinctive aspects of the our approach, of the PDBG model and of the DAMS architecture. Incremental development of debugging functionalities The clear distinction between component-level and coordination-level functionalities promotes the development and experimentation with distinct types of debugging services, ranging from process-level (as discussed in detail in this paper) to thread-level debugging (as part of ongoing work [10]). The above separation of concerns has also influenced the design of the DAMS architecture with its flexible support to incorporate new services. This enables the experimentation with coordination-level services related to the global observation and control of the distributed computations. The management of deterministic re-execution, distributed breakpoints, and evaluation of global predicates, are all examples of aspects easily supported by corresponding DAMS service modules and associated drivers. As these aspects are highly dependent on the specifics semantics of each parallel and distributed computation models, it becomes important to be able to implement them on top of a common infrastructure, as provided by DAMS. Support for heterogeneity At the machine and operating system levels, we have tested the debugging prototype on top of heterogeneous platforms including PCs with Linux, and Alpha processors with OSF/1, interconnected by Ethernet and FDDI LANs. A port to WindowsNT platforms is also under way. At the programming language level, support for distributed C/PVM and C/MPI programs is allowed. Programs written in other programming models can also be controlled and inspected under the DAMS environment, through the implementation of suitable service modules, and associated drivers. Work is under way to implement a debugging service for distributed Prolog programs, by integrating a process-level debugger for Prolog into the DAMS architecture, and defining a suitable service module supporting the high-level semantics of the debugging interface. An instrumentation and performance monitoring service for PVM-Prolog [12] has already been implemented under DAMS with reduced effort. support for distinct computational models can also be easily implemented, provided there are adequate component-level debuggers to be integrated into DAMS. For example, it is possible to support the debugging of a heterogeneous application with multiple components, by controlling multiple process-level or threadlevel debuggers in the same DAMS environment. This becomes important for the development of meta-computing environments, and is also a concern in a related project [5] that is helping us to assess the flexibility of our approach. Tool integration As a concrete example consider the following views: 1. a debugging graphical interface, allowing debugging commands to be put at the program source level; 2. a text window for the display of the program code; 3. a graphical display of the source program structure, e.g. using a graph-based model or a higher level graphical programming language, allowing debugging commands to be put at the program graph level; 4. a visualization tool for the display of process- or thread-based time-lines; and 5. a display of the process interactions, in the form of interprocess communication lines, for testing purposes and path examination. From the integration point of view, some of the above tools can be issuing requests to the PDBG debugger simultaneously, and such tools must share a coherent view of the transition and action histories that describe the evolution of the target computation under debugging. As an example, the [Step] button of a graphical interface must be disabled if another tool has issued a [Continue] command to the same process, and must be enabled when the process stops again. As another example, the visualization tools must update the currently displayed views in a way that is consistent with the current computation state. For example, a threadbased time-line must be stopped in the display when the thread reaches a breakpoint. 9

10 The above concerns have strongly influenced the eventbased model that we present in the paper, as a result of our previous experiments with tool integration in related projects [13, 14]. Work is under way to apply the current PDBG prototype to support the above dimensions of tool integration. Acknowledgements Thanks are due to João Vieira, Bruno Moscão e Daniel Pereira for their work in the DAMS system. The work reported in this paper was partially supported by the Portuguese CIÊNCIA and PRAXIS XXI Programmes Research Projects PROLOPPE (Contract 3/3.1/TIT/24/94) and SETNA-ParComp PRAXIX XXI (2/2.1/TIT/1557/95), and by the DEC EERP PADIPRO (Contract P-005). References [1] J. Brown, J. Francioni, and C. Pancake. White paper on formation of the high performance debugging forum. Available in whitepaper.html, February [2] J. C. Cunha, J. Lourenço, and T. Antão. A debugging engine for a parallel and distributed environment. In Proceedings of the 1st Austrian-Hungarian Workshop on Distributed and Parallel Systems (DAP- SYS 96), pages , Misckolc, Hungary, October Hungarian Academy of Sciences. [3] J. C. Cunha, J. Lourenço, and T. Antão. An experiment in tool integration: the DDBG parallel and distributed debugger. Euromicro Journal of Systems Architecture, 2 nd Special Issue on Tools and Environments for Parallel Processing, (Accepted for publication). [4] J. C. Cunha, J. Lourenço, J. Vieira, B. Moscão, and D. Pereira. A framework to support parallel and distributed debugging. In Proceedings of the International Conference on High-Performance Computing and Networking (HPCN 98), Amsterdam, The Netherlands, April (Accepted for publication). [5] J. C. Cunha, P. Medeiros, J. Lourenço, V. Duarte, J. Vieira, D. Pereira, and R. Vaz. The DOTPAR project: Towards a framework supporting domain oriented tools for parallel and distributed processing. In Proceedings of the International Conference on High-Performance Computing and Networking (HPCN 98), Amsterdão, Holanda, April (Accepted for publication). [6] Robert Hood. The p2d2 project: Building a portable distributed debugger. In Proceedings of the 2 nd Symposium on Parallel and Distributed Tools (SPDT 96), Philadelphia PA, USA, ACM. [7] P. Kacsuk, J. C. Cunha, G. Dózsa, J. Lourenço, T. Fadgyas, and T. Antão. A graphical development and debugging environment for parallel programs. Parallel Computing, 22(1997): , [8] D. Kranzlmüller, S. Grabner, and J. Volkert. with the MAD environment. Parallel Computing, 23(1997): , [9] J. Lourenço, J. C. Cunha, H. Krawczyk, P. Kuzora, M. Neyman, and B. Wiszniewsk. An integrated testing and debugging environment for parallel and distributed programs. In Proceedings of the 23 rd Euromicro Conference (EUROMICRO 97), pages , Budapeste, Hungary, September IEEE Computer Society Press. [10] João Lourenço and José C. Cunha. A thread-level distributed debugger. In Proceedings of the 3 rd International Conference on Vector and Parallel Processing (VecPar 98), Porto, Portugal, June (Accepted for publication). [11] T. Ludwig, R. Wismuller, V. Sunderam, and A. Bode. OMIS On-Line Monitoring Interface Specification (Version 2.0). Technical report, Lehrstuhl fur Informatik, Technical University of Munich (LRR-TUM), Munich, Germany, July [12] R. Marques and José C. Cunha. PVM-Prolog: Parallel logic programming in the PVM system. In Proceedings of the 3 rd USA PVM Users Group Meeting, Pittsburgh, USA, May [13] S.Winter et al. Software Engineering for Parallel Processing, Copernicus Programme. Progress Report 1, University of Westminster, October [14] S.Winter et al. High Performance Computing Tools for Industry, Copernicus Programme. Progress Report 3, University of Westminster, April