A Procedural Interface for Program Directing

Transcription

1 SOFTWARE PRACTICE AND EXPERIENCE, VOL. 25(7), (JULY 1995) A Procedural Interface for Program Directing ROK SOSIČ School of Computing and Information Technology, Griffith University, Nathan, Queensland 4111, Australia ( sosic@cit.gu.edu.au) SUMMARY Debugging and performance measurement tools are becoming more important in the development and maintenance of increasingly complex software. These tools belong to a class of programs, called directors. Directors are programs that monitor and control other programs. Through monitoring, directors analyze program execution at runtime. Monitoring provides tracing primitives as well as access to program s variables, dynamic data structures, and its internal state. Results of the monitoring analysis can be used by the director itself to change the future program behavior or presented to a human user for an interactive review and possible feedback actions. The future program behavior is changed through controlling primitives which provide an organized way to modify data values as well as the program code. This paper presents a library of directing commands which enable the construction of powerful directors. The interface has been implemented in a Unix environment as a runtime subsystem running in the directed program s address space. The paper provides the description of the interface and the basic programming techniques in building directors. Examples of novel applications, illustrating the use of the directing interface, are demonstrated by the directors for the visualization of program control and structured snapshots. KEY WORDS: debugging; directing; monitoring; procedural interface; program visualization INTRODUCTION Debugging and performance measurement tools are becoming more important in the development and maintenance of increasingly complex software. 1,2,3,4,5,6,7,8 These tools provide monitoring and controlling capabilities. Monitoring means collecting information about the dynamic behavior of programs. Some examples of monitoring include tasks such as profiling program performance, obtaining the value of a variable or dynamically constructing a graph of procedure calls at runtime. Except for possible changes in the program s timing, monitoring usually does not change the results of a monitored program. Complementary to monitoring, controlling involves modifying executing programs in order to change their future behavior. Examples of controlling tasks are setting the value of a variable, modifying the stack, dynamically loading a new function and so on. The term directing is used to denote together monitoring and controlling activities. 9 These activities are closely related, because most controlling operations require monitoring. For example, monitoring can ensure that modifications of the program preserve its consistency by detecting the critical sections. The modifications are performed only when the program is executing outside of its critical sections. Programs performing directing are called directors. Debuggers and program profilers are examples of directors. Programs being directed CCC /95/ Received 1 March by John Wiley & Sons, Ltd. Revised 29 December 1994

2 768 R. SOSIČ are called executors. An executor contains a user program which is constructed without consideration for directing tasks. Directing operates transparently and invisibly to the user program. This transparent operation significantly expands the applicability of directing, because any user program can be directed. For example, it is possible to sample programs without special sampling constructs embedded in their source code. Several primitives for building directors can be found in existing systems. They are usually provided by the operating system. In the Unix environment, the most common primitives are /proc, ptrace, and profil. /proc is a special directory which includes a file for each executing program. 10 The file contains the memory image of the corresponding process. A process can be manipulated through its /proc file using system primitives for manipulating files. The ptrace system call provides functionality similar to /proc, but with a different programming interface. The profil system call is conceptually different from /proc and ptrace. It provides elementary profiling capabilities. 11,12 Directing primitives provided by operating systems are too limited to accommodate more complex directing applications, such as directing in distributed environments. Another approach for providing directing primitives is to modify program code. 13,14 Modifications that insert monitoring primitives are often performed on the object code or the executable. While this approach achieves very good performance, it is usually restricted to batch operation. The code cannot be modified at runtime, which limits the director s capabilities. In the absence of adequate support in operating systems, a new directing interface is usually designed specifically for each new directing application. 4 Large parts of these interfaces are often duplicated and reinvented. For example, most directing applications: require a connection between the director and the executor; read and possibly modify the executor s address space; and require some mapping between a high-level program description and machine addresses. A unified directing interface would significantly speed up the development and sharing of new directing capabilities. This paper describes such a procedural interface which provides primitives for building directing applications. Although existing implementations of the interface run on derivatives of BSD Unix, the interface itself contains no Unix-specific constructs. It is expected that the directing interface can be implemented on other operating system platforms without significant conceptual difficulties. DIRECTING INTERFACE In the past, debuggers have been practically the only applications using directing interfaces. As a result, the interfaces have been restricted in their functionality and thus applicability. For example, the directors have been able to direct only their child processes or they have been able to read at most one word in a single operation. Close coupling between debuggers and the operating system has led to proliferation of system-specific features and incompatible interfaces. The limitations of existing debugging interfaces can be avoided by providing a more general, system-independent, directing interface. Two constraints, which must be satisfied by the directing platform, are: a uniform interface across different computer systems; and a simple construction of directors. Both conditions can be met by abstracting the interface from low-level system-dependent features. The directing interface consists of commands that are called from a director. The directing interface uses a client server paradigm. The director acts as a client that sends directing requests to a directing server. The server responds to requests by directly accessing and controlling the user program through the shared address space and control. Servicing of directing requests is transparent to the user program, because it does not require any support

3 A PROCEDURAL INTERFACE FOR PROGRAM DIRECTING 769 in its source code. The described directing interface is implemented entirely at the user level and requires only generic support from the operating system. The interface can be thus implemented on different platforms while maintaining the portability of directors. Additionally, the interface can be easily extended with new features, because such extensions do not require any modifications in the operating system. In traditional debugging interfaces, the debugger requires several files when dealing with a program. These files supply the mapping between the high-level program representation and the corresponding machine-level representation. The files include at least the executable code and often the source code and the object code as well. 15,16,17,18 Access to relevant files can become an issue in distributed environments, where the debugger and the program being debugged execute on different computers. If several linking formats are being used, which is a common case in distributed heterogeneous networks, then a debugger must be able to recognize all possible formats. Maintaining such a debugger can present a formidable challenge in a rapidly changing distributed network. Because debugging interfaces are implemented by operating systems, it is impractical to extend them with specialized and language-specific features. These limitations significantly reduce the portability of debuggers. To improve support for distributed environments, the directing interface provides some primitives for the mapping between the source code and the machine code. The platformdependent work, such as decoding symbol tables and handling low-level details, is done by the directing server at the executor s side. The mapping primitives in the directing server encourage the development of new directing capabilities, because they can be incorporated in the server and used by any director. Without a directing server, which provides a centralized repository for directing capabilities, a capability is limited to a single directing application. The directing interface is accessed by a director through directing procedures. These procedures handle directing commands by executing a remote procedure call to the executor. Remote procedure calls and directing commands are handled by the directing platform which provides the communication between the executor and the director. Directing commands are divided in the following groups: execution control, state access, breakpoints, tracing, and dynamic loading and linking. The execution control commands enable a director to attach to or detach from an executor and to start or stop its execution. The state access commands provide capabilities to obtain and modify the executor s state. The breakpoints commands manipulate breakpoints which can be set inside the executor by the director. The tracing commands provide primitives for a fine-grained tracing of the executor s behavior. The dynamic linking and loading commands enable the director to dynamically load and link new functions into the executor s program in order to change its behavior. The commands are described in more details below. The description uses type void * to denote a generic pointer. Execution control Commands for execution control are used by the director for establishing and maintaining a connection with the executor and for stopping and starting the executor. Commands attach and invoke provide two ways for connecting to the executor: void *attach(int pid, char *computer, char *user) void *invoke(char *command, char *computer, char *user) attach connects the director to an already running process. invoke first starts the program and then attaches to it. Both commands stop the executor. attach is useful for directing

4 770 R. SOSIČ programs that are already running. An example is the monitoring of a program that exhibits unexpected or erroneous behavior. An invoke example is a batch tool for performance analysis which starts a program in the background and then measures its performance. attach and invoke share parameters computer and user. These parameters are used for specifying executor s computer and for verifying director s privileges for accessing the executor. They are character strings providing the name of the computer and the user name under which the executor is running. Parameters pid and commands are specific to attach and invoke, respectively. pid provides the process identifier of the executor. command gives the command line which is used to start the executor. The executor s machine code must be accessible to the command. Upon successful completion, attach and invoke return a handle. The handle identifies the executor in subsequent directing commands. Multiple handles allow a director to direct several executors at the same time. The handle is maintained by the directing interface and is never explicitly manipulated by the director. attach and invoke greatly simplify the connection between the director and the executor, even if they execute on different computers. For example, a program can be invoked on a remote computer by: handle = invoke("hello","remote.cit.gu.edu.au","user"); The connection between the director and the executor, established by attach or invoke, is closed by terminate or detach: terminate(void *handle) detach(void *handle) terminate closes the connection and terminates the executor. detach closes the connection, but the executor and the director continue running independently of each other. With the connection between the executor and the director established by attach or invoke, the executor can be controlled by commands continue and connect: continue(void *handle) connect(void *handle) connect and continue are similar to attach and detach, except that they maintain the connection. After continue, the director and the executor run independently. After connect, the executor stops and waits for further directing commands. Because connect and continue maintain the connection, they are much cheaper than attach and detach. The choice between the connect continue pair and the attach detach pair depends on the application. In a directing application that samples the user program at regular intervals the choice depends on the frequency of sampling. If the program is sampled at a high rate, then the cost of attach detach significantly slows down the execution. On the other hand, if the program runs for days or months and is sampled at a low rate, then the cost of maintaining the connection with the program prevails. In this case, attach detach is a more suitable choice than connect continue. After invoke, attach, or connect, the executor is waiting for further directing commands. The most elementary commands are those which access the executor s state, as described in the next section.

5 A PROCEDURAL INTERFACE FOR PROGRAM DIRECTING 771 State access Commands for state access require a representation of the executor s state. The state consists of several components, usually data, instructions and registers, which all have different properties. Several approaches can be used to represent these state components. An elegant and most general approach is to compose the state from abstract memories. 17 Each abstract memory has its associated type, behavior and address space. One physical location can belong to several abstract memories. Some examples of abstract memories are code and data spaces, general-purpose registers, floating point registers, and extra registers. An alternative approach is to use a single address space to represent the entire state. Entities without an address, such as registers, are assigned otherwise unused ranges in the address space. An advantage of this approach is a uniform addressing method for all elements of the state. The directing interface uses two types of abstract memories for representing the state, the main memory and internal registers. Different routines are provided to access each type. Only for the purpose of tracing, which is described later, are the addresses of registers mapped to an unused portion of the main memory. Access to the internal registers of the executor s processor is provided by directing commands getstate and putstate: getstate(void *handle, int reg[]) putstate(void *handle, int reg[]) Command getstate returns array reg which contains the current values of user accessible registers. Command putstate sets these registers to the values from array reg. Basic routines for reading or modifying the main memory are getmem and putmem: getmem(void *handle, void *loc, int len, void *buf) putmem(void *handle, void *loc, int len, void *buf) They copy blocks of consecutive bytes between the executor and the director. Parameter buf specifies an address in the director, parameter loc specifies an address in the executor and parameter len provides the number of bytes to be copied. getmem copies from the executor to the director, putmem from the director to the executor. getmem and putmem require the executor s machine addresses. The mapping between symbolic and machine addresses is provided by the symbol table in the executor s object file. The usual method to obtain the mapping is to read and decode the object file. This method requires director s access to the executor s object file which can be difficult in distributed environments. The directing interface simplifies the mapping by providing command getsym: void *getsym(void *handle, char *name) getsym takes a symbolic name and returns the symbol s machine address. The following code segment from a director illustrates the use of getsym: addr = getsym(handle,"i"); /* get the address */ getmem(handle,addr,size,&value); /* get the value */ The segment copies the value of remote variable i into local variable value. Command getsym returns the address of variable i which is then used to obtain the actual value.

6 772 R. SOSIČ Variable size specifies the length of i in bytes. The name and the type of variable i are known from the executor s source code. The code above can be extended to provide a director that samples the executor at regular intervals: handle = attach(pid,null); /* attach to the process */ addr = getsym(handle,name); /* get the address */ while (sampling()) { getmem(handle,addr,size,&value); /* read the value */ continue(handle); /* run the executor */ printresult(value); sleep(delay); connect(handle); /* stop the executor */ } detach(handle); Such directors are helpful in studying algorithms with complex dynamic behavior. Without the directing capabilities, the sampling must be embedded in the algorithm itself. Such embedded sampling cannot be turned on and off dynamically and can significantly distort the resulting behavior. If the sampling is done by the director, then its cost and interference can be adjusted dynamically by regulating the sampling frequency. Time penalty is paid only for the sampling performed, otherwise the algorithm executes unperturbed. Block-copying routines as described above are not capable of dealing with dynamic data structures. Because these structures require dynamic allocation of memory, their addresses are not known in advance. Dynamic data structures are supported by commands getptr and putptr: void *getptr(void *handle, void *loc, int len) void *putptr(void *handle, void *buf, int len) These commands allocate necessary space from the heap before copying the data. getptr allocates a new block of memory in the director, copies data from the executor and returns the address of the copy in the director. putptr allocates a new block in the executor, copies data from the director and returns the address of the copy in the executor. Using getptr and putptr, the director can access or modify dynamic data structures in the executor. Two additional commands getstr and putstr are variations of getptr and putptr specialized for dealing with zero terminated strings: void *getstr(void *handle, void *loc) void *putstr(void *handle, void *buf) These commands determine the string size, before allocating the memory and copying the string. The handling of dynamic data structures is demonstrated by the director that copies a linked list from the executor (see Figure 1). The data structure to be copied is specified by struct list. This definition is the same in the executor and the director. The main copying procedure is copy list. Its parameters are the executor s handle and the name of the head pointer. Procedure copy list first obtains the value of the head pointer by calling getsym followed by getmem. By calling getptr repeatedly, copy list copies the elements of the list from the executor to the director, until the end of the list is detected.

7 A PROCEDURAL INTERFACE FOR PROGRAM DIRECTING 773 struct list { /* a linked list */ struct list *next; /* the next element */... /* other fields */ } struct list *copy_list (void *handle, char *s) { struct list *p,*head; } /* get the address of the head */ head = getsym(handle,s); /* copy the value of the head */ getmem(handle,head,sizeof(head),&head); /* if the list is empty, return */ if (head == NULL) return(null); /* allocate the space, copy the element, */ /* obtain the pointer */ p = head = getptr(handle,head,sizeof(struct list)); /* while there are more elements */ while (p->next!= NULL) { /* copy the next element */ p->next = getptr(handle,p->next,sizeof(struct list)); /* advance the pointer */ p = p->next; } /* return the result */ return(head); Figure 1. Director to copy a linked list At the end, copy list returns the head pointer of the copy. Modifications to executor s data structures are performed similarly to copying, except data is copied from the director to the executor, using putptr. Breakpoints Breakpoints provide a standard technique for monitoring the executor without slowing down its execution. The director implants breakpoints in the user program. Whenever a breakpoint is encountered during the execution, the executor stops and waits for further instructions from the director. Breakpoints are set and removed by commands setbreak and delbreak: setbreak(void *handle, void *loc) delbreak(void *handle, void *loc) Parameter loc specifies the breakpoint address.

8 774 R. SOSIČ Breakpoints require a mapping between symbolic locations in the source code and machine addresses, similar to the mapping between variable names and their addresses discussed in the previous section. The symbolic location of a source line is commonly specified by its file name and a line number. The directing command, which provides this mapping, is lineaddr: void *lineaddr(void *handle, char *filename, int line) lineaddr returns the address of a source line, specified by file filename and line number line. After the director sets one or more breakpoints, it can activate the user program and wait for a breakpoint to occur. Breakpoints are detected by command waitbreak which stops the director until a breakpoint is reported by the executor: void *waitbreak(void *handle) Upon return, waitbreak provides the address of the encountered breakpoint. The following code segment demonstrates the handling of breakpoints by the director: /* get line address */ addr = lineaddr(handle,filename,line); setbreak(handle,addr); /* set the breakpoint */ continue(handle); /* activate the executor */ waitbreak(handle); /* wait for the breakpoint */ lineaddr obtains the address of the source line which is used by setbreak to implant the breakpoint. After the breakpoint is set, the executor is activated by continue. The director detects a breakpoint by calling waitbreak. A variation of breakpoints are watchpoints which report access to specified variables. Because watchpoints can impose significant time penalty without special hardware support or program modifications, 19 there are currently no constructs in the directing interface to support them. Tracing, described next, can provide equivalent functionality. Tracing Directors perform tracing by reading events which describe the executor s behavior. Events are generated by the tracing mechanism in the executor. Tracing supplements breakpoints as a complementary monitoring mechanism. It is suitable for tasks that would otherwise require a large number of breakpoints or where the breakpoint location is hard to determine. An example requiring a large number of breakpoints is the monitoring of all function calls. Breakpoint locations might be hard to determine, if the executor exhibits unpredictable behavior. Tracing events can be provided at different levels of description. At the highest level are events corresponding to the concepts in the source programming language, such as function entries and exits. At the lowest level are events that give information about every executed machine instruction. High-level events have the advantage that they are the least resource consuming. However, their disadvantages are that they provide only the most rudimentary information about the program s behavior and that they are programming-languagedependent. Because programming languages provide different high-level abstractions, it is hard, if not impossible, to provide a unified model of high-level events, covering all programming languages. Instead, a low-level universal model is provided by the instruction

9 A PROCEDURAL INTERFACE FOR PROGRAM DIRECTING 775 set which is the same for all compiled programs. The directing platform delivers low-level events with one or more events describing each executed machine instruction. High-level abstractions can be built by the director from these low-level events. An event is reported for each state location that is changed during the execution, including a user register or a location in the main memory. An event contains the address of the changed location, the current value of the program counter, the instruction that changed the location, the location s value before and after the instruction, and some additional bookkeeping items. Events are generated only during the tracing mode which is enabled or disabled by command execute: execute(void *handle, int mode, int n) Parameter mode can be TRACE or EXECUTE. TRACE turns on the tracing facility, EXECUTE turns it off. Parameter n specifies the number of instructions to be traced. It has no effect in the EXECUTE mode. After the tracing is turned on, events can be obtained by command getevent: getevent(void *handle, int event[]) getevent returns the next event emitted by the executor. The director repeatedly calls getevent, until it reads an event denoting the end of tracing. Simple, yet powerful, directors for instruction tracing can be built from tracing commands. At the core of these directors is the segment for tracing one instruction: execute(handle,trace,1); /* trace one instruction */ do { getevent(handle,event); /* get next event */ print_event(event); } while (!end_of_tracing(event)); The director first requests the tracing mode and then repeatedly reads events until the end of tracing is reported. User supplied functions print event and end of tracing print out an event and check for the end of tracing event, respectively. The output from the instruction tracing director on a M68040 processor is illustrated on instruction "link fp,#n". This instruction implements function entry. It first pushes the existing frame pointer on the stack, gets the new value of the frame pointer from the stack pointer, and decrements the stack pointer by parameter n. A straightforward print event function produces the following four events for a "link fp,#0" instruction executed at address 0x5f44: # PC OPCODE ADDRESS NEWVAL OLDVAL 1. 5f44 4e56 ffffff3c 3fffcb0 3fffcb4 ; sp -= f44 4e56 3fffcb0 3fffcc0 3ec6 ; *sp = fp 3. 5f44 4e56 ffffff38 3fffcb0 3fffcc0 ; fp = sp 4. 5f44 4e56 ffffff3c 3fffcb0 3fffcb0 ; sp -= 0 Each line represents one event. An event provides a program counter, the instruction opcode, the address changed, and the values after and before the instruction. Addresses 0xffffff3c and 0xffffff38 are the assigned addresses for the stack and frame pointers. Comments at the end of each line describe the effect of the corresponding event. This effect can be

10 776 R. SOSIČ eventfilter() { /* procedure call */ if (call_inst(event)) return(1); /* procedure return */ if (return_inst(event)) return(2); /* unconditional jump */ if (jump_inst(event)) return(3); /* conditional branch */ if (branch_inst(event)) return(4); return(0); /* discard the event */ } Figure 2. Filter for tracing control flow observed in columns ADDRESS, which shows the changed location, NEWVAL, which shows the new value at the location, and OLDVAL, which shows the old value at the location. By incorporating event processing in the director, very long instruction traces can be analyzed on the fly. Similar directors have been used to analyze traces with several billion executed instructions. 20 Tracing generates large quantities of events, at least one event for each executed instruction. The delivery of events from the executor to the director imposes substantial time overhead up to four orders of magnitude. 9 To reduce the number of events and the associated time overhead, the directing interface provides filters. A filter is a function that is called for each event. If the filter evaluates to TRUE, then the event is reported to the director, otherwise it is discarded. Filter functions are provided by object files which are dynamically linked with the executor when requested by the director. Elementary commands to deal with filters are ldfilter and rmfilter: ldfilter(void *handle, char *filename) rmfilter(void *handle, char *filename) ldfilter loads and installs functions from object file filename as the currently active filter. filename must contains two functions, initfilter, which initializes the filter, and eventfilter, which evaluates events. During tracing, eventfilter is called for each event. If eventfilter returns 0, the event is discarded. Otherwise, the event is sent to the director. Command rmfilter is the inverse of ldfilter and removes filter filename from the executor. Following are two examples which demonstrate the use of filters. The first example is a filter that detects all control instructions, such as jumps and function calls. This filter has no internal state. The second filter example detects modifications to a linked list. This example requires the filter to maintain an internal state. The control instructions are: a procedure call, a procedure return, an unconditional jump, and a conditional branch. The filter that recognizes the control instructions is shown in Figure 2. If the filter detects a control instruction, it returns a non-zero value. Otherwise, it returns zero. Because initfilter is not necessary, only eventfilter is shown. The current event is contained in global variable event. Control instructions can be recognized

11 A PROCEDURAL INTERFACE FOR PROGRAM DIRECTING 777 initfilter() { initialize_table(table,head_pointer); } eventfilter() { if (ismalloc(event)) { /* memory allocation */ insert(table,address,size); return(1); } if (isfree(event)) { /* memory deallocation */ delete(table,address); return(2); } /* pointer modification */ if (isstore(event) && intable(table,event)) return(3); return(0); } Figure 3. Filter for tracing linked list by examining the code of the executed instruction. This recognition is done by functions call inst, return inst, jump inst, and branch inst which perform a simple opcode matching. Because these functions are architecture-dependent, their details are not shown. The director for instruction tracing, shown before, can be modified to trace only control instructions. The only change necessary is to install the above filter at the start of tracing by calling ldfilter. The filter can be used in a range of applications as illustrated later by the director for visualizing control flow. Different traces can be obtained by varying the filter. For example, watchpoints can be implemented by installing the filter that detects modifications to a memory location. Tracing can be extended to dynamic data structures, such as linked lists, which require a filter with an internal state. The following types of events are important in monitoring a linked list: an allocation of a new element, a deallocation of an existing element, and a modification of pointers. A filter without an internal state cannot capture these events, because the position and the number of allocated elements are not known in advance. Therefore, the filter must maintain an internal table with the information about the dynamically changing data structures. The filter for tracing a linked list is shown in Figure 3. The filter maintains a table of dynamically allocated elements. The table keeps the address and the size of each element. The table is manipulated by procedures initialize table, insert and delete. initialize table initializes the table. insert adds a new elements in the table. delete removes an element from the table. Procedures initfilter and eventfilter are the two filter routines. initfilter initializes the filter s internal table with the address of the head list pointer. The name of the pointer is known from the executor s source code and its address can be looked up in the executor s symbol table. Procedure eventfilter uses the following procedures to detect events: ismalloc, isfree, isstore, and intable. Function ismalloc detects if a new element is allocated, assuming that the allocation has

12 778 R. SOSIČ been done through function malloc. If that is the case, insert puts the address and the size of the allocated element, obtained from the event and from the executor s state, in the filter s table. Function isfree detects if an element is released, assuming that every release is done through function free. If an element is released, it is removed from the filter s table by delete. Function isstore detects write operations to the main memory. Detected write operations are checked by function intable which returns TRUE, if the written value points to an element in the filter s table, and FALSE otherwise. If the address accessed and the value written are both found in the table, then the instruction is treated as a change in the pointer value. This method of determining pointer modifications makes conservative decisions. It guarantees to detect all pointers, although some additional references might be proclaimed as pointers. Dynamic loading and linking It is often desirable to be able to perform the modification of the code of executing programs. For example, the director might switch between two versions of a function, a highly optimized version and a version that is slower but more suitable for monitoring. By dynamically choosing between these two versions, the director can exploit the tradeoff between the speed of execution and the ease of monitoring. A traditional approach to program modification is self-modifying code. Because directing moves the responsibility for maintaining the program consistency to the director, it alleviates some of the problems of self-modifying code. The directing interface provides separate commands for program loading and symbol linking. Loading commands operate on object files, linking commands operate on individual symbols. One reason for the separation of loading and linking is the ability to switch rapidly between different versions of the same function. Two object files, containing different versions of the same function, are loaded only once, but can be linked many times. This results in significant time savings for certain applications, The command for object file loading is ldobject: ldobject(void *handle, char *filename) It loads object file filename from secondary storage to the executor s address space. ldobject reads the program code and constructs the symbol table. The storage for program code and the table is allocated from the executor s heap. ldobject modifies external symbols in filename, so that they correspond to values in the executor s program. After ldobject, filename is linked with the executor s program, but the program itself has not been modified and cannot use symbols from filename. The modification of the existing program and the linking of new symbols is performed by link: link(void *handle, char *filename, char *symbol) link takes as parameters an already loaded object file and a symbol defined in the file. It modifies the user program, so that all existing references to symbol reflect its definition in filename. The old definition of the symbol is discarded. No special provisions, usually required for dynamic linking in the executor s source code, 21 are necessary. Because linking is done by modifying existing code, link can dynamically redefine any symbol. A sample directing code, which replaces the current implementation of function f with a version from object file f.o, is:

13 A PROCEDURAL INTERFACE FOR PROGRAM DIRECTING 779 main Figure 4. Visualization of control flow ldobject(handle,"f.o"); /* load an object file */ link(handle,"f.o","f"); /* link a symbol */ All subsequent calls to function f use the new implementation, although active instances on the stack execute until their termination. A command, related to dynamic loading and linking, is call: call(void *handle, char *procname, params...) It provides a rudimentary facility for remote procedure calls. Using call, the director can call functions that are loaded, but not necessarily linked, with the user program. call uses the C calling convention for the activation record to perform the call. The usage of call is illustrated by the code segment that requests the display of string "Hello, world!": /* copy string to the executor */ address = putstr(handle,"hello, world!\n"); /* call printf remotely */ call(handle,"printf",address); The director assumes that the string is not part of the executor and must be copied over by putstr which returns the address of the string in the executor. This address is used as a parameter in the remote call to printf. It is assumed that printf is already loaded in the executor s address space. APPLICATIONS This section describes two examples of novel applications, using basic directing commands. The applications are directors for program visualization and structured snapshots. These directors can be applied to any program in traditional, compiled programming environments. Program visualization Control flow gives an outline of the program execution by providing data about executed functions and statements. The director for animating the control flow uses visualization method, based on rows of squares (see Figure 4). Each row represents one function invocation on the stack. A row includes the following elements: 1. name. At the beginning of the row is the name of the function. 2. squares. Each square represents a sequential block of instructions. The control proceeds sequentially through the block. The block contains no jumps in or out.

14 780 R. SOSIČ do { 14. if (condition) insert(); 15. else delete(); 16. } while (condition); Figure 5. Program for visualization 3. arcs. Arcs represent jumps, either conditional or unconditional. Arcs above the squares are forward jumps, arcs below the squares are backward jumps. 4. marker. The marker, represented by a cross, denotes the position of the program counter. 5. line numbers. White squares contain numbers indicating line positions in source code. 6. gray squares. Gray squares denote parts of program that have not been executed so far. The following types of events are recognized by the director: a function call, a function return, an unconditional jump, and a conditional branch. A filter to detect these events was described before. Each event changes visualization as follows: 1. function call. The row representing the currently executed function is frozen. A marker of the calling function shows the position from which the function call was performed. A new row is activated representing the called function. 2. function return. When a function returns, its row is deleted. The graph of the calling function becomes active again. 3. unconditional jump. If the jump is encountered for the first time, then its source square, the destination square, the body square, and an arc between the source and destination squares are added to the graph. If the jump was encountered before, only the marker is moved. 4. conditional branch. A conditional branch is handled identically as an unconditional jump. In addition to the described events, the director keeps track of the program counter by moving the marker. If the control enters a gray square, it is changed to a white square. The director displays program execution by expanding the graph at runtime. This dynamic graph expansion is demonstrated on the program segment in Figure 5. Figure 6 illustrates a function call. At first, there is only main executing. When function delete is called in line 15, a new row of squares is started. The graph expands as more instructions from the current function are being executed. A sequence of graph expansions during execution of main is demonstrated in Figure 7. Figure 7(a) presents the graph at the start of main. Because no jumps have been performed so far, there is only one white square and one gray square. The white square represent the portion of the code that was executed and the rest is represented by the gray square. After the if-then statement in line 14 is entered for the first time, the graph expands to show an untaken jump to the else part (see Figure 7(b)). After the body of the if-then statement in line 14 is executed the else part is skipped (see Figure 7(c)). The execution of the insert procedure in line 14 is not illustrated. When the do loop is repeated, the graph is

15 A PROCEDURAL INTERFACE FOR PROGRAM DIRECTING 781 main main delete 19 (a) before a function call (b) after a function call Figure 6. Function call expanded with an arc that shows control being transferred to the beginning of the loop (see Figure 7(d)). At this point, three gray squares show that three parts of the code were not executed: the code after the do loop; the else part; and the part of code between if-then and else. The part, denoting the code between if-then and else, might be possibly empty. Nevertheless, its gray square indicates that control has never passed through the square. The visualization graph enables fast comprehension of program structure. Several features help the programmer in understanding the dynamic behavior of the program: 1. The row of squares representing one function is displayed at the time of a function call and deleted at the time of a function return. The rows provide information about the instantiated functions. 2. Arcs represent program structure in a visual way. They enable rapid detection of jumps, branches, and loops. 3. A marker, shown as a cross, points out the current position of the program counter. Each function has its own marker, so that the place of a function call is easily determined. The position of each marker is dynamically updated to reflect the progress in execution. 4. Square colors indicate if the corresponding code has been executed or not. White squares denote portions of code that were executed. These squares remain unchanged in the future. Gray squares illustrate unexecuted parts of the program. These squares may expand to a more complex graph in the future. The director uses the directing interface to get necessary information for generating the control flow graph. Because no modifications to the executor s code are required, the director can visualize an arbitrary program. Structured snapshots It is sometimes desirable to obtain the state of an executing program. For example, often significant computational resources are required by scientific simulations to reach a certain stage of the simulation. Once this initial stage is computed, it serves as a base for multiple parametric studies. Each parametric study is one simulation run which takes the initial stage as the starting point and uses its own set of parameters to continue the

16 782 R. SOSIČ main 10 main (a) start of execution (b) control enters if-then statement main (c) control skips else statement main (d) control returns to the start of the loop Figure 7. Row expansion simulation. The easiest but most resource-demanding approach to parametric studies is to compute the initial stage separately for each simulation run. A better approach is to compute the initial stage only once and then replicate the simulation state within several processes, each performing one simulation run. Traditionally, the initial process would save a simulation state as a file on a secondary storage. Every continuing simulation then takes this file as a starting point. This approach requires that programs include procedures to load and save simulation states. While efficient implementations are possible for continuous blocks of data, storing and loading dynamic data structures with a lot of pointers can be a non-trivial task. It can also impose significant time overhead, because machine addresses must be converted to symbolic addresses and vice versa. The initial and the continuing programs must provide calling points for loading and saving procedures which significantly limits times when they can be called. Another method to replicate process state is through a core dump. 22 The process is sent a signal that generates a core image of the process. The image can be examined and restarted as a different process. Although this method is simple and effective, it only deals with unstructured blocks of bytes. The method is thus unsuitable for heterogeneous networks and for applications that deal with structured data.

17 A PROCEDURAL INTERFACE FOR PROGRAM DIRECTING 783 Structured snapshots are snapshots that preserve the structure of data. For example, a structured snapshot is capable of traversing a linked list and replicating the list in another process, possibly occupying different memory locations. The directing interface provides a framework for an effective implementation of structured snapshots. A director for structured snapshots needs to support the following operations on a data structure: copying the structure from a process, copying the structure to a process, storing the structure to a disk, and loading the structure from a disk. If such a director is constructed for a particular data structure, these operations are supported on any executor using the structure. A possible approach for implementation has been demonstrated by a director for copying linked lists. This approach can be extended to deal with more complex data structures. Additional operations can be included in the director besides copying, such as a memory compaction by rearranging elements or calculations producing statistics about the structure. Another feature of the directing interface is that the point of a snapshot can be determined at runtime by using monitoring facilities. The state of complex data structures is often inconsistent during the execution. By using breakpoints or execution tracing, the director can ensure that a snapshot is taken only when the structure is consistent. IMPLEMENTATION The directing interface, described in the previous sections, is provided by the Dynascope directing platform. This section summarizes the implementation of Dynascope. A comprehensive discussion on the implementation of Dynascope is given elsewhere. 23 Dynascope runs on Next computers using a M68040 processor and on Sun computers using a Sparc processor. It supports directing in distributed environments, with directors and executors arbitrarily placed on different computers. Subject to the restriction of identical data representations, a limited heterogeneity is provided between Nexts and Suns. For example, the director can be on a Sun and the executor on a Next. Dynascope provides the same directing interface on both computers, so porting a director from Next to Sun or vice versa requires only a recompilation. The implementation of Dynascope is similar on both computing platforms. Dynascope uses the following capabilities of the operating system: access to a remote system, interprocess communication across the network, and the ability to asynchronously interrupt a process. These or equivalent capabilities are standard features on most modern operating systems. Dynascope consists of two components: the client library and the directing server (see Figure 8). The client library is straightforward. It handles the communication with the directing server, transmits directing commands, and returns responses from the directing server. If the executor is on a remote computer, the client library uses remote execution primitives for accessing the computer. The communication between the client library and the directing server is established through signals and shared files and uses sockets and the TCP/IP protocol. The central component of Dynascope is the directing server. Each user program has its own server. The server is implemented as a library, linked with the user program. There are no direct function calls between the user program and its directing server, so the server could be added at runtime if such addition is supported by the operating system. Another technique that would integrate the server with a user program is to modify the executable file directly. 14 These alternative options have not be pursued so far. To simplify the implementation, there is no system-wide master directing server, although one might be

18 784 R. SOSIČ director executor user program function call client library remote execution, signals, TCP/IP user program signal, breakpoint directing server Figure 8. Dynascope implementation provided in the future versions. It is assumed that the information about the executor, such as the executor s process identification or the target computer, is provided by the director s user. Significant elements of the server implementation concern: the organization of the server; the connection with the director; and the implementation of breakpoints and tracing. The directing server is organized as a thread which shares control with the user program. The server is activated by a signal from a director or a breakpoint in the user program. Upon activation, the server performs a thread switch which pre-empts the user program and starts servicing directing requests. The server returns control to the user program when the director requests its continuation. The directing server is transparent to the user program. If no directing is being performed, the server imposes no overhead, except for occupying the address space. The TCP/IP protocol is used for transmission of directing commands and their results. 24 In establishing the TCP/IP connection, the director acts as the protocol server. In the case of errors in the director or the executor, this arrangement has proved more robust than the protocol server in the executor. The director requests a connection by sending a dedicated signal to the executor. The directing server catches the signal and connects to the director s port. The TCP/IP connection is maintained until the director detaches from the executor or until either the director or the executor terminate. The protocol for transmitting directing commands and their results is an extension of the ftp-like protocol. 25 Breakpoints are implemented by implanting the breakpoint code in the user program. 26 The breakpoint code, when executed, jumps to the code that performs a thread switch and activates the directing server. Execution tracing is often carried out by simulating the processor. To reduce the complexity of the simulator, Dynascope uses surrogate execution instead of pure simulation. Surrogate execution exploits the fact that the processor running the simulation and the processor being simulated are the same. Each instruction to be executed is inspected to determine its opcode, operands and locations of results. If the instruction is simple, it is simulated, otherwise a surrogate instruction is executed. The surrogate instruction produces the same effect as the original instruction, but operands and results have different locations. Before the surrogate instruction is executed, its operands are set up to reflect the operands to the original instruction. After the execution, the results are copied to locations to imitate the effects of the original instruction. Surrogate execution significantly simplifies the trac-