XMEM. Extended C/C++ Dynamic Memory Control and Debug Library XMEM Manual Version 1.7 / September 2010 Jürgen Müller reserves all rights including all industrial property rights. All rights of disposal such as copying and passing on to third parties.
Disclaimer Copyright (c) 1987-2010, Jürgen Müller All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. - Neither the name of Jürgen Müller nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Written by: Jürgen Müller, GERMANY EMail: sxt@gmx.de XMEM Manual 2/57
Contents Introduction 7 Quick Start. 9 Memory Management Errors 10 Memory leaks.. 10 Invalid references to memory.. 10 NULL pointer access 11 Access to freed (previously released) memory.. 11 Accessing uninitialized allocated memory.11 Retaining pointers to reallocated memory.12 Memory overwrites before or after allocated memory block (buffer underflow, buffer overflow).. 12 Problems with running Out of memory 12 Problems with releasing memory.. 13 Mixing incompatible memory management functions. 13 Use of invalid pointer address values. 14 Miscellaneous problem sources. 14 XMEM Functionality.. 15 Overview. 15 Internals.. 15 Detected errors 16 Output and error reporting. 16 Log files.. 18 Collecting statistical data 22 Clean-up and final statistics. 22 XMEM Special Features.. 25 C++ 25 XMEM DLL / Dynamic Link Library (experimental).. 25 Call stack trace 26 Garbage collection 26 64 bit support 27 Microsoft Detours.. 27 Library replacement (experimental). 27 Multi threading (experimental). 28 AspectC++ (experimental). 28 XMEM Installation and Configuration. 29 Installation and usage. 29 XMEM Files.. 29 Compiler defines and command line settings. 30 General control options 30 XMEM. 30 XMEM_TRACE.30 C language specific options.. 31 XMEM_C_INTERFACE 31 C++ language specific options. 31 XMEM_CPP 31 XMEM_CPP_INIT 31 XMEM_CPP_CLIB_HEADER 31 XMEM_CRITICAL_OPERATOR.. 31 XMEM_NEWARRAY.. 31 XMEM_DELETEARRAY.. 32 XMEM_CPP_DELETE_STACK 32 XMEM_CPP_DELETE_STACK_SIZE.. 32 XMEM Manual 3/57
Function call redirection specific options. 32 XMEM_DETOURS.. 32 XMEM_ALLOC_LIB 32 Memory block guard region specific options. 32 XMEM_GUARD 32 BEGFILLCHAR. 32 ENDFILLCHAR. 33 ALLOCFILLCHAR 33 FREEFILLCHAR..33 XMEM_ALLOCFILL 33 XMEM_FREEFILL 33 XMEM_USED 33 Testing specific options 33 XMEM_FREENULL. 33 XMEM_TEST_NULL.. 33 XMEM_ALLOC_FAIL_WARN 33 XMEM_WARN_NULL 33 XMEM_WARN_BAD.. 34 XMEM_PEDANTIC. 34 XMEM_WITHIN. 34 XMEM_TEST_MCB 34 XMEM_STREOS.. 34 XMEM_STRESS.. 34 Source code location specific options.. 34 XMEM_WHERE 34 XMEM_GET_LOCATION. 34 Call stack tracing specific options.. 35 XMEM_CALLSTACK. 35 XMEM_STACKWALKER. 35 XMEM_MAX_CALLSTACK. 35 Standard library specific options. 35 XMEM_REALLOC_ILLEGAL. 35 XMEM_ALLOCA.. 35 XMEM_STR 35 XMEM_MEM.. 35 Output specific options. 35 XMEM_CALLTRACE. 35 XMEM_DETOURSCALLTRACE. 35 XMEM_VISIT. 36 XMEM_QUIET.. 36 XMEM_FORCE_OUTPUT.. 36 XMEM_OUTPUTDEBUGSTRING.. 36 XMEM_DIALOG36 XMEM_DIALOG_SHOW.. 36 XMEM_BROWSER. 36 Logging specific options.. 36 XMEM_LOGFILE. 36 XMEM_HTMLLOGFILE 36 XMEM_ERRLOGFILE 37 XMEM_SETTINGS. 37 XMEM_STATS.. 37 Clean-up specific options 37 XMEM_FREE_INCREASE_ORDER. 37 XMEM_FILES_SORT_BY_COUNT 37 XMEM_INTERNAL_FREE_SORT.. 37 DLL specific options.. 37 XMEM_DLL. 37 XMEM_DLL_API_EXPORT 37 Miscellaneous options.. 38 XMEM_THREAD.. 38 XMEM Manual 4/57
XMEM_ASPECTCPP. 38 StackWalker configuration. 38 STACKWALKER_SYMSRV 38 XMEM API Methods 39 Controlling operation mode.. 39 void xmem_set_memtrace(unsigned int flag) 39 int xmem_get_memtrace().. 39 void xmem_enable_memtrace().. 39 void xmem_disable_memtrace().. 39 void xmem_enable_memtrace_silent(). 39 void xmem_disable_memtrace_silent() 39 void xmem_main_return() 40 Displaying information. 40 size_t xmem_get_used_memory().. 40 void xmem_showstatus(). 40 void xmem_check_free().. 40 void xmem_check_free_virtual(unsigned int flag) 40 void xmem_message(const char *text). 40 size_t xmem_get_overhead() 40 void xmem_set_output_function(void ( stdcall *)(const char *pfunc)) 40 Controlling information printing and logging. 40 void xmem_enable_print() 40 void xmem_disable_print().. 40 void xmem_enable_log().. 41 void xmem_disable_log(). 41 void xmem_enable_free_warning(). 41 void xmem_disable_free_warning() 41 Pointer and memory block checking 41 void xmem_check_ptr(void *ptr).. 41 void xmem_overwrite()..41 Pointer and memory block locking 41 int xmem_lock_ptr(void *ptr).. 41 int xmem_unlock_ptr(void *ptr). 41 Setting alloc and free trace points. 41 unsigned long xmem_get_alloc_nr().. 41 void xmem_set_trace_alloc(unsigned long nr). 41 void xmem_reset_trace_alloc(). 42 void xmem_set_trace_free(unsigned long nr).. 42 void xmem_reset_trace_free().. 42 Resource limitation and execution failing.. 42 void xmem_set_max_heapsize(size_t size).. 42 size_t xmem_get_max_heapsize(). 42 void xmem_set_stress_ratio(unsigned int ratio).. 42 unsigned int xmem_get_stress_ratio().. 42 Tracking program scope. 42 void xmem_enter_scope() 42 void xmem_leave_scope().. 43 Program information. 43 void xmem_set_progname(char *progname). 43 char *xmem_get_progname() 43 XMEM Browser. 44 Stand-alone XMEM data collection.. 44 XMEM Browser windows 45 Examples, Target and Compiler Details 47 Known Problems and Limitations. 48 General issues 48 Trouble shooting. 48 Third party code. 48 XMEM Manual 5/57
Header file xmem.h include sequence 48 C/C++ macro issues. 48 C++ issues. 49 C++ operator calling at global scope 49 C++ class specific operator overloading 50 User defined memory management functions 51 Log files location. 51 Performance. 51 Handling of invalid address values 52 Implementation limitations. 52 Memory Debugging Tools.. 53 Links 54 Index 55 XMEM Manual 6/57
Introduction Memory management in C and C++ is and has ever been a great source of problems - not only for C/C++ beginners - and debugging memory errors can be a nightmare. Allocating memory on the heap (with malloc, calloc, alloca, realloc, _expand, xmalloc, xcalloc, xrealloc, strdup, new, new[], ), accessing memory blocks with pointers and, probably the biggest problem for many developers, freeing memory (with free, cfree, xfree, delete, delete[], ) at the right place and right time must be done very carefully. Memory errors are often very strange and are not easy to detect, because the initial reason for them and the location they occur are usually not directly related and can be in totally different parts of the software. Errors like memory leaks, invalid references to free memory, reading from uninitialized allocated memory, retaining pointers to reallocated memory or memory overwrites are well known problems to every C and C++ programmer. C++ has introduced an additional type of problems with the underlying allocation and deallocation of memory and its related object constructors and destructors, single object allocation or object array allocation, not spoken of such special things like objects destroying themselves ("delete this;"), static objects with their constructors called before entry to main() function and destructors called after exit of main() function or a mixture of C and C++ programming with the possibility of an erratic use of malloc / calloc with delete / delete[] respectively new / new[] with free. For real-world applications in the commercial and industrial area which often have to run on servers or at production lines for weeks or months without interruption and frequent shutdowns, leaking memory resources is probably one of the greatest problems: After some time even small memory leaking leads to growing memory consumption, decreasing of the system performance and producing non-deterministic errors and malfunctions where the reason for this behavior is usually not obvious and hard to find. The extended C/C++ Dynamic Memory Control and Debug Library XMEM is a replacement library for dynamic heap management functions like malloc / free / new / delete to find memory leaks and other memory related errors in C and C++ code. By using XMEM each call to a memory management function can be observed, controlled, traced and logged, and obvious or potential memory allocation and deallocation problems can be identified and displayed. An extensive logging mechanism helps to get overview about the applications memory consumption during its lifetime and to generate final memory call statistics and reports about memory leaks that can be used for further analyzing. XMEM is easy to use by simply including the XMEM header file xmem.h into the source files to be observed and to compile and link the XMEM implementation file xmem.c or xmem.cpp to the application. There are many XMEM options that can be configured by using compiler command line defines or editing the XMEM header file xmem.h. Please see chapter Installation and Configuration for usage details. XMEM Manual 7/57
The complete XMEM functionality is available for 32 bit and 64 bit Microsoft Windows operating systems (Windows 7, Windows Vista, Windows XP). Reduced XMEM functionality is available for many other operating systems like 16 bit DOS, 32 bit protected mode DOS and Windows, 16 bit Windows 3.1, 32 bit and 64 bit Linux. The XMEM core functionality should compile on almost every 16 bit, 32 bit and 64 bit system that has an ISO / IEC standard conforming C and C++ compiler. If necessary, the source code easily be ported to other compilers or operating systems. Please note the following limitations concerning the application of XMEM: The usage of XMEM for C and C++ program development cannot guarantee that a program is absolutely free of memory management related errors in any case. XMEM supports debugging, identification and fixing of memory management related problems. However, there are certain situations possible with unforeseeable boundary conditions where XMEM may fail and where such errors may not be recognized and therefore will still exist in a program, sooner or later leading to program misbehavior or runtime errors. XMEM Manual 8/57
Quick Start For those users who do not want to read documentation but want to start immediately here are the necessary steps to include XMEM into your program and make it run: Include xmem.h as last header file into all source files for which XMEM memory management shall be used. However, if previous include files contain calls to memory management functions, xmem.h should additionally be included as first header file. Add the source file xmem.c (respectively xmem.cpp for C++) to your project list of source files. Set the necessary compiler command line defines to enable the XMEM functionality, at least the define XMEM must be set (compiler preprocessing option -DXMEM). Rebuild your program. If everything is okay and the program has been successfully built you can run your program and examine the XMEM console output and the XMEM log files. Nevertheless please read the XMEM documentation since there are a lot of useful features and options listed which are not self describing! XMEM Manual 9/57
Memory Management Errors The various types of memory access errors and with dynamic memory management can be classified into several categories as shown below. XMEM detects all these error types and provides a lot of useful error information that supports the identification and isolation of the problems and their cause. Memory leaks A memory leak is the failure to allocate dynamic memory but not to free that memory if it is no longer used. The memory leak occurs if all pointers (= references) to that memory block are lost. This happens, for example, if there is constantly new memory allocated and assigned to the same pointer or by leaving the scope where the pointer to the allocated memory is valid (automatic pointer variable on the stack) and thus having no chance to free the memory later after leaving the function scope. There are also C standard library functions like strdup() returning pointers to dynamically allocated memory which have to be explicitly released by the user which in fact is often forgotten. Memory leaks increase the memory usage of the application, decrease performance and may lead to system crash. char *ptr1 = malloc(10); /* allocate first memory block */ char *ptr2 = malloc(20); /* allocate second memory block */ /* check ptr1 and ptr2 if valid address */ ptr1 = ptr2; /* assign new value to ptr1 */ /* -> address of first memory block is lost and cannot be freed */ /* -> memory leak */ char *ptr3 = strdup( string to copy ); /* library function returning allocated */ /* memory */ /* check ptr3 if valid address */ /* ptr3 needs to be released by user */ Figure: Example code for memory leaks Invalid references to memory Reading or writing to memory via uninitialized or incorrectly initialized pointers may be reasons for this type of memory error. Also freeing the same memory multiple times or reallocating memory via uninitialized or previously freed pointers may be reasons for this type of memory error. char *ptr1; /* uninitialized pointer */ char *ptr2 = malloc(10); /* check ptr2 if valid address */ = *ptr1; /* reading from uninitialized pointer */ ptr1 = realloc(ptr1, 20); /* reallocation via uninitialized pointer */ free(ptr2); /* free pointer, ptr2 is not overwritten */ free(ptr2); /* freeing already freed pointer twice */ Figure: Example code for invalid references to memory XMEM Manual 10/57
NULL pointer access Accessing memory via NULL pointers ( NULL pointer access ) will crash the program and must be avoided. char *ptr1 = NULL; struct s1 { struct s1 *next; } *ptr2 = NULL; *ptr1 = a ; /* writing to NULL pointer -> access violation */ = ptr2->next; /* dereferencing NULL pointer -> access violation */ Figure: Example code for NULL pointer access Access to freed (previously released) memory Reading or writing to memory via already freed pointer. Also freeing the same memory multiple times or reallocating memory via pointers that have previously been freed may be reasons for this type of memory error. char *ptr1 = malloc(10); /* check ptr1 if valid address */ free(ptr1); *ptr1 = x ; /* writing to already freed pointer */ free(ptr1); /* freeing already freed pointer */ ptr1 = realloc(ptr1, 10); /* reallocate already freed pointer */ Figure: Example code for access to freed memory Accessing uninitialized allocated memory Except for the allocation with calloc() or with operator new with constructor initializers, dynamically allocated memory is not initialized and may contain random data. Accessing such memory without prior initialization will produce non-deterministic behavior. struct s1 { char *ptr; }; struct s1 *ps1 = malloc(sizeof(struct s1)); /* check ps1 if valid address */ = *(ps1->ptr); /* reading from uninitialized allocated memory */ Figure: Example code for accessing uninitialized allocated memory XMEM Manual 11/57
Retaining pointers to reallocated memory Due to a possibly different address for the reallocated memory, all related pointers have to be actualized after successful reallocation. char *ptr1 = malloc(10); /* check ptr1 if valid address */ char *ptr2; ptr2 = ptr1; ptr1 = realloc(ptr1, 20); /* reallocating ptr1, returned pointer may be */ /* different */ /* check ptr1 if valid address */ *ptr2 = ; /* ptr2 may no be valid since it has not been updated after */ /* reallocating ptr1 */ Figure: Example code for access to reallocated memory via retained pointers Memory overwrites before or after allocated memory block (buffer underflow, buffer overflow) Writing to unallocated memory before (buffer underflow) or after (buffer overflow) allocated memory block due to incorrect array indices or pointer offsets. int length = 10; /* */ long *array; array = malloc(length * sizeof(long)); /* allocating space for array with */ /* 10 elements */ /* check array if valid address */ for (int i = 10; i >= 0; --i) { array[i - 1] = 0; /* i = 0 -> array[-1] -> invalid index */ } /* writing 4 bytes before allocated memory block, */ /* buffer underflow */ array[length] = 0; /* length = 10 -> array[10] -> invalid index */ /* writing 4 bytes after allocated memory block, */ /* buffer overflow */ Figure: Example code for memory overwrites before or after allocated memory block Problems with running Out of memory In case of an application running out of memory with an allocation function returning a NULL pointer, the application should check that return value and be able to handle such low memory situations. If not, the application may use a NULL pointer or another invalid address. char *ptr1 = malloc(10); /* out of memory allocation returns NULL pointer */ /* missing checking of ptr1 if valid address */ *ptr1 = a ; /* writing to memory with pointer that was not checked after */ /* allocation */ Figure: Example code for problems with running out of memory XMEM Manual 12/57
Problems with releasing memory Typical problems are freeing an uninitialized or invalid pointer, freeing a NULL pointer, freeing the same pointer twice or freeing pointers to stack memory. char *ptr1; char array[10]; free(ptr1); /* freeing uninitialized pointer */ ptr1 = malloc(10); /* check ptr1 if valid address */ free(ptr1); free(ptr1); /* freeing already freed pointer twice */ ptr1 =&array[0]; free(ptr1); /* freeing stack memory (memory is not allocated with memory */ /* management) */ Figure: Example code for problems with releasing memory Mixing incompatible memory management functions Such types of errors occur if either incompatible C or mixed C and C++ memory management functions are used. Memory allocated with C memory management functions (malloc / free / realloc) cannot be deallocated with C++ memory management functions (new / new[] / delete / delete[]) and vice versa. Also mixing C++ single-object heap management (new / delete) cannot be mixed with C++ array heap management (new[] / delete[]). There is also an incompatibility with the C memory management function alloca() and free(), a pointer allocated on the local stack with alloca() cannot be released with free() in fact it needs no explicit deallocation. char *ptr1; ptr1 = malloc(10); /* check ptr1 if valid address */ delete ptr1; /* freeing with incompatible function */ ptr1 = malloc(10); /* check ptr1 if valid address */ delete [] ptr1; /* freeing with incompatible function */ ptr1 = new char; /* check ptr1 if valid address */ free(ptr1); /* freeing with incompatible function */ ptr1 = new char [10]; /* check ptr1 if valid address */ free(ptr1); /* freeing with incompatible function */ ptr1 = new char [10]; /* check ptr1 if valid address */ delete ptr1; /* freeing with incompatible function */ ptr1 = alloca(10); /* allocate pointer on local stack */ /* check ptr1 if valid address */ free(ptr1); /* freeing with incompatible function, attempt to free stack pointer */ Figure: Example code for mixing incompatible memory management functions XMEM Manual 13/57
Use of invalid pointer address values There are pointer address values which are not possible or not allowed during normal operation. This is for example a memory block with an odd address (natural alignment is always on even address for 16 / 32 / 64 bit systems) or an address value that is outside of the allowed address range (depends on CPU and operating system address capabilities). /* example for MS Windows 32 bit with Intel CPU */ /* valid address range: 0x00x7fffffff (31 bit address) */ char *ptr1; ptr = (char *)0xfffffffe; /* assign address value outside allowed address range */ free(ptr1); /* freeing invalid pointer, will probably crash */ ptr1 = (char *)1001; /* assign odd address value */ free(ptr1); /* freeing invalid pointer, will probably crash */ Figure: Example code for problems with invalid address values Miscellaneous problem sources There are some other possibilities for problems where the behavior is sometimes not clearly defined or implementation dependent like an allocation size of 0 bytes or reallocating a NULL pointer. char *ptr1; ptr1 = malloc(0); /* allocate 0 bytes */ /* check ptr1 if valid address */ char *ptr2 = NULL; ptr2 = realloc(ptr2, 10); /* reallocate NULL pointer */ /* check ptr2 if valid address */ Figure: Example code for other problem sources XMEM Manual 14/57
XMEM Functionality Overview The memory errors described in the previous chapter are usually caused by programming mistakes. They typically result from things like misunderstanding, missing knowledge and practice or lack of dealing with the size and complexity of large real-world applications. Especially the demands on control flow and timing are hard to meet and it is easy to loose overview. Memory errors are often non deterministic, they show a strange behavior and are sometimes hard to reproduce. The place and time where a defect is introduced and the place and time where it occurs and appears to the user are not directly related, they can be in different parts of the application and may happen long time after being introduced. Therefore finding and solving such programming errors is very difficult and time consuming. The resulting effects of such memory management errors are crashing programs due to access violations, programs which slow down in execution speed because of growing memory consumption, finally claiming all available memory and stopping execution after some time or influencing simultaneously running programs, or programs giving incorrect results. XMEM helps to recognize, find, locate, identify and fix memory errors and shortens the problem solving time significantly. An XMEM instrumented program reports details like the name of the calling function, filename, line number and call stack at the time the error happens or becomes obvious. That provided information supports the debugging and identification of the problem root cause location. With XMEM, developing industrial strength C and C++ applications with reduced risk for memory related errors becomes faster and easier. Internals XMEM detects memory errors by performing a program runtime analysis. Including the XMEM header file xmem.h and compiling the source performs a source code instrumentation. This inserts additional code at appropriate places replacing calls to standard memory management functions with corresponding XMEM interception functions that redirect the call, collect runtime information and perform validation tasks for each call to the embedded memory management functions. If a memory error is detected it will be logged and immediately reported. Finally, upon program termination, XMEM performs a final check for remaining memory leaks and errors with writes to free memory. Also a final extensive statistics output with call counts and allocation size histogram is generated. The XMEM source code instrumentation requires the availability of the applications source code. However, many memory problems are caused by third-party libraries and other object code for which source code is usually not available. In this case object code instrumentation is necessary to intercept calls to memory management functions. For XMEM on 32 bit Windows this is possible with the software package Detours, which is developed and freely available from Microsoft Research (http://research.microsoft.com/detours/). XMEM Manual 15/57
By default XMEM remembers the name and the location of the calling function and the called memory management function, if this name resolution feature is supported by the compiler. As an additional feature on 32 bit / 64 bit Windows the complete function call stack can be stored and the exact location of the function call with filename and line number is resolved if the program is compiled with debug options. A similar functionality is also available for Linux except that the call stack info is less detailed. This function call chain provides detailed information about the potential source for the error and helps analyzing the problem. The Windows call stack functionality for C++ is based on the Dbghelp API and on "StackWalker" that can be found at CodePlex (http://stackwalker.codeplex.com) and which was functionally extended and performance optimized for XMEM specific needs. For Windows C code the implementation uses similar techniques. Detected errors XMEM supports program development error detection in the following problem areas: detect memory leaks detect incorrect or unsafe memory allocation (like size = 0 or realloc with NULL) detect incorrect memory deallocation detect multiple memory deallocation of same memory block detect memory re-use of previously deallocated memory detect potentially unused memory (memory blocks that are never written) detect memory overwrites before or after allocated memory block detect use of incompatible memory management functions (like allocating with new and deallocating with free) detect use of pointers with invalid address values detect potentially unsafe use of string and memory handling functions These error detections depend on the programming language being used (C or C++) and on the XMEM configuration settings that are enabled for the program. Output and error reporting When XMEM detects an error or a problem, it reports the related information immediately to the user and writes it to an error report log file respectively stores this information for each call for being used later, if necessary. The following information is collected and displayed: the source filename and the line number within the source file where the memory management function is called the name of the memory management function being called call parameters and return values XMEM Manual 16/57
an allocation count and time stamp that uniquely identifies each memory allocation the pointer to the allocated memory block that is returned to the calling function the name of the calling function (if supported by the compiler) the call stack with filename and line number (if compiler debug options and XMEM call stack tracing are enabled, depending on compiler and operating system, currently supported for 32 bit and 64 bit Windows and Linux) the memory address where the corruption respectively problem was detected This information enables the developer to track each memory block from its allocation until its deallocation or, if the memory block is not deallocated, until the memory leak is reported on final program clean-up. Additionally, each call to a memory management function, string or memory handling function can be logged and may be reported with the used parameters and the returning value and additional remarks if something strange happens. Figure: XMEM command line / console output XMEM supports different types of error reporting: command line prompt / console output ( DOS box, terminal, shell) message box (depending on the platform, currently only for Windows) output to the debugging console (debugger log window within compiler environment, stand alone debug viewer, secondary monitor), currently only for Windows sending output as Windows messages which can be received and displayed from an independent Windows application like XMEM Browser (see description below) output to a call log file and an error log file (see description below) XMEM Manual 17/57
Figure: XMEM message box output (Windows XP) Log files During program execution there are two different types of XMEM log files written, collecting structured call information or collecting error information. The log files are closed after each write and reopened for the next write so that in case of a crash the already existing log file contents is not lost. Log file containing error information: xmem_err.log: This log file is used to collect all error information that is reported during program execution as well as the final clean-up and statistics information. Log file containing structured function call information: xmem.log: This log file is used to collect trace information about each call with call parameters and corresponding statistics data. The data is stored in preformatted categories as comma separated values (CSV) and can easily be imported from spread sheet programs like Microsoft Excel. xmem_log.html: This log file is similar to xmem.log but written in HTML format with the data stored in a table. The HTML log file can be viewed with every web browser. The following two figures show error log files on a 32 bit and a 64 bit system. XMEM Manual 18/57
Figure: Error log file contents (32 bit Windows XP program, with call stack info) Figure: Error log file contents (64 bit Linux program) XMEM Manual 19/57
The log file and the HTML log file containing structured function call information reports the following data: date of log entry, time of log entry, unique call counter, name of the called memory management function, name of calling function, filename where the function call happens, line number within the file name, the resulting allocation pointer address returned, respectively the pointer address given as parameter for freeing, allocation size, total number of currently allocated memory, number of calls to allocation functions, number of calls to freeing functions, unique allocation number, also given as reference when pointer is freed The following figures show examples for log files and HTML log files. Figure: Log file contents (32 bit Windows XP program) XMEM Manual 20/57
Figure: Log file imported as CSV formatted data to MS Excel Figure: HTML log file contents (32 bit Windows XP program) XMEM Manual 21/57
Figure: HTML log file contents (64 bit Linux program, 8 byte pointer) Collecting statistical data During runtime statistical data is collected and processed for each call to a memory management function. The data includes the name of to the memory management function, the requested allocation block size and the name of the calling function together with its source filename. With this the number of calls to a memory management function and the number of calls from a specific function within a source file are calculated as well as the overall number of calls to memory management functions. Additionally the currently allocated memory size is calculated and the maximum peak values are stored. Clean-up and final statistics Upon final program termination an XMEM clean-up and statistics function can be called that reports all memory leaks and, if enabled, memory corruptions like writes to free memory. In C code this clean-up function has to be inserted manually at the end of the main() function, for C++ this clean-up is done automatically (with a static destructor) if this XMEM feature is enabled. The final clean-up function reports all memory leaks which means all at that time still allocated memory blocks. For each memory leak the previously described information XMEM Manual 22/57
is given. Additionally, previously freed memory blocks are checked whether they have been changed later due to re-use of pointers to deallocated memory. Finally the summary of the statistical information is reported. This includes the number of calls to each memory management function, the overall summary values, an allocation size histogram with categorized call sizes and the number of calls from a specific function and the number of calls from a specific file. Figure: Final memory allocation statistics XMEM Manual 23/57
Figure: Final clean-up on program exit XMEM Manual 24/57
XMEM Special Features C++ The XMEM memory tracing functionality can either be predefined for compilation or it can be controlled at program start within the main() function by a call to xmem_set_memtrace(). For C programs the clean-up of memory leaks must be explicitly called by manually inserting the necessary checking function xmem_check_free() into the source code near the end of the main() function. For C++ programs this can be automated by using a special XMEM class static constructor and destructor that performs initialization before entering main() and clean-up after leaving main() without the need to manually editing the main() source code. The redirection of C++ operators new / new[] and delete / delete[] is also enabled if compiled as C++ code. These XMEM C++ features can be enabled with the additional command line defines XMEM_CPP and XMEM_CPP_INIT. If XMEM_CPP is enabled, the define XMEM_WHERE must also be enabled. For correct handling of source location tracing for nested calls to C++ operator delete the command line define XMEM_CPP_DELETE_STACK must be enabled. There is a known critical issue with operator new if it is used at global scope outside of a function body. See chapter Known Problems and Limitations sub chapter C++ for further information and a work around for this problem. XMEM DLL / Dynamic Link Library (experimental) On 32 bit and 64 bit Microsoft Windows XMEM can be compiled as a Dynamic Link Library (DLL). This XMEM DLL file (xmem.dll) implements and exports the XMEM redirection functions that can be imported and called by their consuming applications (EXE programs and other DLLs). The consuming applications must be linked with the library xmem.lib. For the C++ operators new and delete a special XMEM stub file (xmemstub.cpp) is necessary in the calling application to redirect them to the XMEM DLL. This XMEM feature can be enabled and used with the additional command line defines XMEM_DLL and XMEM_DLL_API_EXPORT. There are some known issues with this XMEM DLL implementation. One is that there is no console output. Another problem can happen if the XMEM DLL is released earlier than other DLLs that use the xmem.dll, this may generate wrong warnings about memory leaks. There may also be problems with file stream pointers. XMEM Manual 25/57
Call stack trace For C++ programs running on 32 bit or 64 bit Microsoft Windows XMEM provides extensive call stack tracing functionality by using the freely available program package StackWalker if option XMEM_STACKWALKER is enabled. The original version of StackWalker can be found at CodePlex (http://stackwalker.codeplex.com). The C++ source file StackWalker.cpp must be compiled and linked to the project together with xmem.cpp. The program package StackWalker uses the Windows Dbghelp API and was extended and optimized for XMEM specific use. For performance improvement obsolete features not required for XMEM have been disabled. To speed-up time consuming operations like retrieving source information for stack addresses an internal look-up hash list has been introduced that is frequently reorganized according to hit count statistics. For C programs running on Microsoft Windows 32 bit and 64 bit operating systems a call stack functionality can be implemented by including the source file xmem_cstk_win.c (respectively xmem_cstk_win.cpp) into the project. This file contains different call stack trace implementations (from which one has to be chosen by manually editing the file) which are all based on the Dbghelp API. If option XMEM_STACKWALKER is disabled, the C call stack functions from xmem_cstk_win.cpp are used also used for C++. For C and C++ programs running on Linux a call stack functionality can be implemented by including the source file xmem_cstk_linux.c respectively xmem_cstk_linux.cpp) into the project. This file contains different call stack trace implementations (from which one has to be chosen by manually editing the file. Linux does not provide a commonly available library like Dbghelp for Windows that can be used to resolve execution addresses to get corresponding filename and line number. The Linux call stack trace info based on the backtrace library functionality returns by default only the filename and an execution address. This info can, for example, be used offline as input for the command line program ADDR2LINE that displays the filename and line number for a given address. This XMEM feature can be enabled with the additional command line define XMEM_CALLSTACK. Garbage collection The fact that XMEM is able to track the allocation and freeing of memory blocks, detect memory leaks and frees them on demand can be used to perform a kind of garbage collection. Enabling XMEM for selected program parts during run time and disabling it afterwards with immediate memory clean-up following ensures that there are no memory leaks remaining for that XMEM controlled code section. To perform garbage collection with XMEM, the instrumented source code can be compiled for quiet mode which means that XMEM runs completely silent in the background without issuing any warning message or file logging. With this feature enabled XMEM is used not as a debugging tool for testing but as part of the release version of a program. XMEM Manual 26/57
This XMEM feature can be enabled if only XMEM and XMEM_QUIET are defined as command line defines. 64 bit support The XMEM source code has been extended to make XMEM ready for 64 bit CPU s and operating systems with 64 bit (8 byte) pointers. Almost all Microsoft Visual Studio 2008 projects have been extended to support 64 bit Windows. XMEM is compiled as 64 bit application with the Microsoft Visual Studio 2008 x64 cross compiler. This 64 bit cross compiler is part of the Windows SDK for Windows Server 2008 and.net Framework 3.5. With this 64 bit cross compiler the XMEM test programs can be compiled on Windows 32 bit systems and executed on Windows 64 bit systems. The XMEM core functionality has been successfully tested with Microsoft Windows Vista Business 64 bit. The XMEM example projects in subdirectories with extension.linux should work on both 32 bit and 64 bit Linux systems. The XMEM core functionality has been successfully tested with OpenSuSE Linux 11.0 x86-64 on an Intel 64 bit CPU system. Microsoft Detours The program package Detours, developed and publicly provided by Microsoft Research, supports the instrumentation of object code during runtime to intercept and redirect function calls. XMEM uses this technology to intercept calls to memory management functions to redirect them to XMEM memory management functions for checking, tracking and logging purposes. With Detours this is even possible for library functions that are available only as object code, without source code and therefore without the possibility for source code instrumentation. This XMEM feature can be enabled with the additional command line define XMEM_DETOURS. Additionally option XMEM_GET_LOCATION can be used to determine function call source location. Option XMEM_DETOURSCALLTRACE can be used to enable the call trace printing. A program that is compiled for XMEM with Detours needs the Detours header files, library files and the runtime DLL. Detours only works with C++. This feature works only with Microsoft Windows 32 bit operating systems because Detours is only available for 32 bit Windows. Detailed information about Detours and the installation package itself can be found and downloaded from http://research.microsoft.com/sn/detours/. Library replacement (experimental) This XMEM feature implements memory management library functions that overwrite / replace the original libraries routines when linking the program object files with the libraries. This attempt is neither perfect nor works without problems and can usually only be forced by ignoring linker errors like multiple defined functions. Therefore this XMEM library replacement must be considered experimental. However, if XMEM memory management call interception with Detours cannot be used, library replacement might be an alternative. XMEM Manual 27/57
This XMEM feature can be enabled with the additional command line define XMEM_ALLOC_LIB. Additionally option XMEM_GET_LOCATION can be used to determine function call source location. This feature has only been tested with 32 bit Microsoft Windows XP. Multi threading (experimental) This XMEM feature implements the correct handling of concurrent calls to memory management functions in multi thread applications. Critical sections in XMEM are protected to ensure integrity of core structures. This feature has been tested only with Microsoft Windows 32 bit operating systems but is also implemented for 64 bit Windows and Linux. However, this feature should still be considered as experimental. This XMEM feature can be enabled with the additional command line define XMEM_THREAD. AspectC++ (experimental) Aspect-oriented programming is a development method that identifies source patterns, so called aspects, and performs certain transitions on them. This transformation replaces each pattern occurrence by a predefined code fragment that can be used to insert underlying source code. XMEM uses AspectC++ as a means to perform the source instrumentation of memory management functions by defining aspects to insert XMEM functions. In this package there are two different XMEM replacement aspects provided, xmemaspc1.ah and xmemaspc2.ah, which can be used with the AscpetC++ compiler ac++. For usage details see the XMEM AspectC++ example projects in this package. Detailed information about AspectC++ and the installation package itself can be found and downloaded from http://www.aspectc.org. This XMEM feature can be enabled with the additional command line define XMEM_ASPECTCPP. This feature has only been tested with 32 bit Microsoft Windows XP. XMEM Manual 28/57
XMEM Installation and Configuration Installation and usage XMEM comes as a zipped package that contains the XMEM core source files and several example projects. The XMEM core functionality is implemented in the following two files: xmem.c: Source code file that implements the complete XMEM functionality. For C++ there is the file xmem.cpp provided as a wrapper for xmem.c to ensure correct C++ compilation. xmem.h: Header file that defines XMEM prototypes and macros. If option XMEM is not specified as compiler define, the content of this header file is completely skipped. Note that xmem.h must be included as last header file after all other header files! If one of the previously included files contains calls to memory management functions, xmem.h should additionally be included as first header file. XMEM is easy to use by simply including the XMEM header file xmem.h into all source files for which XMEM memory management shall be used and by compiling and linking the XMEM implementation file xmem.c (respectively xmem.cpp for C++) to the application. There are additional XMEM files for special features (see description below). To enable the XMEM memory management functionality at least the compiler define XMEM must be set. This is usually done by writing -DXMEM as compiler command line parameter or specifying XMEM as preprocessor option. Initially the XMEM default settings from xmem.h can be used to make XMEM work for a specific application being tested for memory leaks. If this first step is done XMEM can be further configured and fine tuned either on the compiler command line or by editing xmem.h. See the following chapters for a detailed description of the XMEM files and the available compiler defines and command line settings. To ensure overall consistency all source files must be compiled with the same XMEM specific compiler defines. XMEM Files The XMEM core functionality is implemented in the following files: xmem.h: Header file that defines XMEM prototypes and macros. This header file must be included by each source file that needs to be XMEM instrumented. xmem.c: Source code file that implements the complete XMEM functionality. xmem.cpp: Source file that is used as C++ wrapper file for xmem.c which is simply included here. XMEM Manual 29/57
xmemstub.cpp: Source file that implements XMEM C++ stub functions necessary to interact with the XMEM DLL. These stub functions redirect C++ functions from the executable program to the XMEM DLL. This file is only necessary if a C++ compiled XMEM DLL is used, it is obsolete if the XMEM DLL is C compiled. StackWalker.h: Header file for the StackWalker package, only used by StackWalker.cpp and xmem.cpp. StackWalker.cpp: Source file for the StackWalker package that implements the XMEM C++ call stack functionality. xmem_cstk_win.c: Source file that implements the XMEM C call stack functionality and an alternative C++ functionality for 32 bit and 64 bit Windows. xmem_cstk_win.cpp: Source file that is used as C++ wrapper file for xmem_cstk_win.c which is simply included here. xmem_cstk_linux.c: Source file that implements the XMEM call stack functionality for Linux with GCC compiler. xmem_cstk_linux.cpp: Source file that is used as C++ wrapper file for xmem_cstk_linux.c which is simply included here. There are various other files in the XMEM directory which contain example code. Additional files from the compilation of the XMEM DLL project are xmem.dll and xmem.lib. Compiler defines and command line settings XMEM has many different features which can be enabled by specific defines. These defines can be set either by manually editing them in the header file xmem.h or by defining them as parameters on the compiler command line. Many options are predefined and set to default values. For additional info see the contents of xmem.h and xmem.c. Although defines have been tested for their relationships and side effects, there may be still situations where combinations of them may not work. The actual XMEM settings are displayed during compilation and are also written to the error log file. General control options XMEM Option to enable XMEM functionality. If defined, the include file contents of xmem.h is processed and source code instrumentation is performed. If XMEM is not defined, the include file xmem.h is completely skipped and the original source remains unchanged. XMEM_TRACE Option to define the initially set default memory trace method. Possible values: 0 = memory trace disabled 1 = memory trace enabled XMEM Manual 30/57
2 = memory trace enabled with allocation block copy on free, useful to catch memory overwrites after release During runtime this can be changed with operation control functions like xmem_set_memtrace(). For details see chapter Controlling operation mode. C language specific options XMEM_C_INTERFACE Option to enable C interface declaration extern C for XMEM C library replacement functions when compiling them as C++ code. C++ language specific options XMEM_CPP Option to enable replacement of C++ operators new / new[] and delete / delete[]. For some compilers the array operators new[] and delete[] are not supported, so these XMEM operator replacements can be switched off by setting the internal defines XMEM_NEWARRAY and / or XMEM_DELETEARRAY to 0. For correct operation the define XMEM_WHERE must be enabled because the global C++ memory allocation operator prototypes need to be different from those functions that are called from within object files or libraries. This can only be done with additional location information (filename, line number) added to the C++ memory allocation operators. However, to catch all calls to operator new, even those without location information - for example those in libraries, the operator new / new[] prototypes without location information parameters can be enabled with define XMEM_CRITICAL_OPERATOR. Option XMEM_GET_LOCATION can additionally be used to determine function call source location. XMEM_CPP_INIT Option to enable automatic XMEM initialization (setting memory trace option) before entering main() and de-initialization (clean-up and statistics) after leaving main() with a static XMEM C++ class constructor and destructor. XMEM_CPP_CLIB_HEADER With C++ code and option XMEM_CPP_CLIB_HEADER enabled the C++ equivalents for the standard C language header files are used (<cstdio> instead of <stdio.h>, for example), otherwise the standard C header files are used. With C code option XMEM_CPP_CLIB_HEADER is disabled. XMEM_CRITICAL_OPERATOR See XMEM_CPP for further information. XMEM_NEWARRAY See XMEM_CPP for further information. XMEM Manual 31/57
XMEM_DELETEARRAY See XMEM_CPP for further information. XMEM_CPP_DELETE_STACK Option to enable correct handling of source location tracing for nested calls to C++ operator delete. XMEM_CPP_DELETE_STACK_SIZE Option to set maximum size of nested calls to C++ operator delete that can be handled. Function call redirection specific options XMEM_DETOURS Option to enable library function call redirection to XMEM replacement functions with the Microsoft Research Detours package. The define value specifies which Microsoft Research Detours version is used for compilation. This feature works only with Microsoft Windows 32 bit operating systems. XMEM_ALLOC_LIB This feature is experimental. Option to enable library function call redirection with direct replacing / overwriting C memory management library functions with their XMEM replacements. Memory block guard region specific options XMEM_GUARD Option to enable detecting memory overwrites before or after allocated memory blocks (bounds checking). The size of the additional guard regions before and / or after the memory block as well as the fill characters are defined in xmem.c. OVHDBEG and OVHDEND define the number of guard bytes before and after the memory block, BEGFILLCHAR and ENDFILLCHAR define the fill characters before and after the memory block. Both defines can also be defined on the compiler command line. Note that the number of guard bytes before the memory block defined by OVHDBEG has to be chosen carefully since the required pointer address alignment of the system must be met for both the XMEM internal memory block pointer including the guard bytes and the pointer returned (first byte after the guard block). On 32 bit systems this is usually a 4 byte (or multiple of 4 bytes) alignment, on 64 bit systems it should be 8 bytes. BEGFILLCHAR See XMEM_GUARD for further information. XMEM Manual 32/57
ENDFILLCHAR See XMEM_GUARD for further information. ALLOCFILLCHAR See XMEM_ALLOCFILL for further information. FREEFILLCHAR See XMEM_FREEFILL for further information. XMEM_ALLOCFILL Option to enable initialization of allocated memory blocks with a predefined allocation fill character. The fill character pattern is defined in xmem.c, the define ALLOCFILLCHAR defines the fill character. This define can also be defined on the compiler command line. XMEM_FREEFILL Option to enable initialization of freed memory blocks with a predefined free fill character. The fill character pattern is defined in xmem.c, the define FREEFILLCHAR defines the fill character. This define can also be defined on the compiler command line. XMEM_USED Option to enable checking whether allocated memory blocks have been really used before they are freed. This is done by testing each byte for changes to the default ALLOCFILLCHAR pattern. Testing specific options XMEM_FREENULL Option to enable the setting of freed pointers to NULL. If such a pointer is accessed afterwards, a NULL pointer exception will occur. If the pointer is not overwritten with NULL, the address may still be valid and point to a free memory area that can be read from or written to. XMEM_TEST_NULL Option to enable XMEM internal NULL pointer checking and reporting. Since not all calls to memory management functions may check their return values XMEM can do this internally and exit application in case of error. XMEM_ALLOC_FAIL_WARN Option to enable additional explicit warning message in case of failed allocation operations. XMEM_WARN_NULL Option to enable a warning if a NULL pointer is going to be freed. XMEM Manual 33/57
XMEM_WARN_BAD Option to enable a warning if a possibly bad / invalid pointer is given as parameter to an XMEM replacement function. Note that this check cannot be performed absolutely reliable and perfect since XMEM may also be called with pointers that are not initially traced by XMEM. However, even such a hint ( false-positive ) may be considered useful during debugging memory errors. XMEM_PEDANTIC Option to enable checking for memory overwrites at additional places during program execution. XMEM_WITHIN Option to enable checking of pointers whether they are pointing to addresses at the beginning or inside an XMEM allocated and traced memory blocks. XMEM_TEST_MCB Option to enable validation of Memory Control Blocks (MCB). Either implemented by direct access to MCBs or via API functions (pointer and memory block size validation). This feature works only with Microsoft C/C++. XMEM_STREOS Option to enable string length limit checking performed inside string functions. Within a predefined number of characters (see xmem.c) the end of string (EOS) character \0 must be found, otherwise a warning message is printed. XMEM_STRESS Option to enable XMEM stress testing functions to simulate low memory situations and random allocation fails. Source code location specific options XMEM_WHERE Option to enable collecting and tracking of calling function name, file name and line number. It must be enabled if XMEM_CPP is enabled. XMEM_GET_LOCATION Option to specify function call source location detection (calling function name, file name and line number). This is used together with Detours (option XMEM_DETOURS), with library function call redirection (option XMEM_ALLOC_LIB) and with handling of redirected C++ operators (option XMEM_CRITICAL_OPERATOR). This feature works only with Microsoft Windows 32 bit / 64 bit operating systems. It is implemented via the Windows Debug Help API. XMEM Manual 34/57
Call stack tracing specific options XMEM_CALLSTACK Option to enable XMEM call stack tracing. This feature requires additional XMEM files being compiled and linked. The call stack trace is either implemented with StackWalker (C++ only) or with other C/C++ specific call trace functions (defined in xmem_cstk_win or in xmem_cstk_linux). XMEM_STACKWALKER Option to choose C++ call stack function. If enabled, StackWalker (StackWalker.cpp) is used. Valid if option XMEM_CALLSTACK is enabled. This feature works only with Microsoft Windows 32 bit / 64 bit operating systems. It is implemented via the Windows Debug Help API. XMEM_MAX_CALLSTACK Option to set the maximum call stack depth that can be handled. Valid if option XMEM_CALLSTACK is enabled. Standard library specific options XMEM_REALLOC_ILLEGAL Option to perform extended realloc check and to return NULL pointer if it fails. XMEM_ALLOCA Option to enable XMEM handling of alloca() stack memory allocation. XMEM_STR Option to enable extended checking of C library string functions strxxx(). Test cases are NULL pointer access, string length, array bounds checking for overwrites. XMEM_MEM Option to enable extended checking of C library memory functions memxxx(). Test cases are NULL pointer access, array size, array bounds checking for overwrites. Output specific options XMEM_CALLTRACE Option to enable tracing of calls to memory management functions. If enabled, each call to a memory management function will be reported with the function call parameters and the return values. This feature is useful for generating a detailed memory function trace. XMEM_DETOURSCALLTRACE Option to enable tracing of calls to memory management functions redirected with Detours. XMEM Manual 35/57
XMEM_VISIT Option to enable tracking of info about last called XMEM function. This info will be printed with each call to an XMEM replacement function. XMEM_QUIET Option to disable printing output. Each message is suppressed to run XMEM silently in the background. This option is being used for XMEM garbage collection applications. XMEM_FORCE_OUTPUT Option to force printing output for some messages even if XMEM_QUIET is enabled. XMEM_OUTPUTDEBUGSTRING Option to enable changing the output function. This feature works only with Microsoft Windows 32 bit and 64 bit operating systems. XMEM_DIALOG Option to enable displaying of XMEM messages in a Windows message box. This feature works only with Microsoft Windows operating systems. XMEM_DIALOG_SHOW Option to force displaying of XMEM messages in a Windows message box even if other options suppress printing output. May be used in combination with define XMEM_BROSWER to send strings as Windows messages. This feature works only with Microsoft Windows operating systems. XMEM_BROWSER Option to enable sending of XMEM output messages and statistical data as Windows message to other applications (like XMEM Browser). This feature has been tested only with Microsoft Windows 32 bit operating systems. Logging specific options XMEM_LOGFILE Option to enable log file writing. The log file contents is written as comma separated values (CSV) and can easily be imported from Microsoft Excel or other spread sheet and database programs. XMEM_HTMLLOGFILE Option to enable HTML log file writing. The log file contents is written as an HTML table that can be viewed with every browser. Requires XMEM_LOGFILE to be enabled. XMEM Manual 36/57
XMEM_ERRLOGFILE Option to enable error log file writing. Each printed warning message is additionally written to this error log file. XMEM_SETTINGS Option to enable XMEM settings being displayed and additionally written to error log file. XMEM_STATS Option to enable tracking of memory allocation sizes for statistical purposes. The requested sizes are classified, counted and printed as histogram data. Clean-up specific options XMEM_FREE_INCREASE_ORDER Option to enable final clean-up in increasing allocation number order. This option is set internally and enabled by default except for XMEM_QUIET being enabled. XMEM_FILES_SORT_BY_COUNT Option to control the displaying of files in the statistics summary. If XMEM_FILES_SORT_BY_COUNT is set to 1 files are sorted by number of counts to memory management functions. If XMEM_FILES_SORT_BY_COUNT is set to 0 files are sorted alphabetically. XMEM_INTERNAL_FREE_SORT Option to control the internal memory block checking and freeing. If XMEM_INTERNAL_FREE_SORT is set to 1 the allocated memory blocks are freed in ascending allocation number order. Note that this ordered freeing is much slower than freeing in random order (XMEM_INTERNAL_FREE_SORT = 0). DLL specific options XMEM_DLL Option to enable XMEM DLL functionality. The XMEM kernel functionality is compiled in a DLL and can be used from an EXE program and its DLLs. This feature is only available for Windows. XMEM_DLL_API_EXPORT Option to control XMEM API export and import. This option is valid if option XMEM_DLL is enabled. XMEM_DLL_API_EXPORT = 1: Exports XMEM function interface. This feature is used when the XMEM DLL is compiled. The exported function interface is imported from the program that uses the XMEM functionality. XMEM Manual 37/57
XMEM_DLL_API_EXPORT = 0: Imports XMEM function interface. This feature is used when an executable (or a DLL) that uses XMEM functionality is compiled. The XMEM DLL exported functions are imported and calls are directed to the XMEM DLL. This feature is only available for Windows. Miscellaneous options XMEM_THREAD This feature is experimental. Option to enable the handling of concurrent calls to memory management functions in multi thread applications. If enabled, critical sections in XMEM are protected to ensure integrity of core structures. This feature has only been tested with Microsoft Windows 32 bit operating systems. For 64 bit Windows or Linux systems it may also work but has not been tested. XMEM_ASPECTCPP This feature is experimental. Option to enable AspectC++ handling. This package defines two different sets of XMEM replacement aspects, xmemaspc1.ah and xmemaspc2.ah, to be used with the AscpetC++ compiler ac++. StackWalker configuration STACKWALKER_SYMSRV Option to enable StackWalker with Windows Debug Help API symbol server access. If enabled, XMEM / StackWalker downloads missing symbol information from "http://msdl.microsoft.com/download/symbols". XMEM Manual 38/57
XMEM API Methods This chapter gives an overview about the XMEM application programming interface (API). Controlling operation mode void xmem_set_memtrace(unsigned int flag) Set type of XMEM memory tracing: flag = 0: XMEM memory tracing disabled, that means that calls via the XMEM redirection functions are directly forwarded to the original C / C++ standard library memory management functions, there is no XMEM checking and tracing performed, the performance overhead is reduced to a minimum; by additionally enabling silent tracing (with xmem_enable_memtrace_silent(), see below) every displaying trace output will be disabled flag = 1: XMEM memory tracing enabled flag = 2: XMEM memory tracing enabled, extension to flag = 1, additionally, memory blocks are not really freed and released to the heap, instead the original memory block is kept allocated together with an additional shadow copy to detect incorrect writes to the original block after being freed, with this XMEM memory tracing type enabled the performance is low and resource requirements are extremely high (during execution no memory block is released, instead each memory block is duplicated, the overall memory consumption is constantly increasing) int xmem_get_memtrace() Get actual xmem_memtrace value. void xmem_enable_memtrace() Enable XMEM tracing. The previously stored xmem_memtrace value will be restored. void xmem_disable_memtrace() Disable XMEM tracing by setting the xmem_memtrace value to 0 (see description above). The actual xmem_memtrace value will be stored. void xmem_enable_memtrace_silent() Enable silent XMEM tracing. This is tracing without displaying trace output information. void xmem_disable_memtrace_silent() Disable silent XMEM tracing. XMEM Manual 39/57
void xmem_main_return() Function to signal the return from function main(). This supporting function should only be used at the end of the main() function body just before the return statement. It sets an XMEM internal flag so that any further attempts to re-init XMEM functions are suppressed. This can be necessary to avoid conflicts with memory clean-up especially with C++ destructors where the destruction sequence cannot be influenced. Displaying information size_t xmem_get_used_memory() Get number of currently allocated bytes. void xmem_showstatus() Show XMEM status report. void xmem_check_free() Show XMEM status report and check pointers for memory leaks and really free them. void xmem_check_free_virtual(unsigned int flag) Check pointers for memory leaks, depending on parameter flag detected memory leaks are either only reported or reported and freed. void xmem_message(const char *text) Print and log the message text from within the program. Can be used to embed additional trace comments. size_t xmem_get_overhead() Get allocation overhead in bytes caused by XMEM array bounds checking. This is the value of the additional space being allocated for each memory block to detect under-/ overwrites. void xmem_set_output_function(void ( stdcall *)(const char *pfunc)) Function that can be used to set an alternative output function pfunc. This function works only with Microsoft Windows 32 bit and 64 bit operating systems. Controlling information printing and logging void xmem_enable_print() Enable printing information during program execution. void xmem_disable_print() Disable printing information during program execution. XMEM Manual 40/57
void xmem_enable_log() Enable log file writing during program execution. void xmem_disable_log() Disable log file writing during program execution. void xmem_enable_free_warning() Enable warnings during memory block freeing. void xmem_disable_free_warning() Disable warnings during memory block freeing. Pointer and memory block checking void xmem_check_ptr(void *ptr) Check a specific pointer ptr whether it is tracked by XMEM and if it is valid. void xmem_overwrite() Check all allocated memory blocks for overwrites. Pointer and memory block locking int xmem_lock_ptr(void *ptr) Lock a specific pointer ptr. ptr is the return value of a previous memory allocation. The return value is 1 if lock was successful, otherwise 0. This feature avoids freeing this pointer during XMEM memory clean-up. This is useful to avoid problems with deallocations that cannot be controlled (for example, initiated by global static destructors called after leaving main() function). int xmem_unlock_ptr(void *ptr) Unlock a specific pointer ptr. The return value is 1 if lock was successful, otherwise 0. This enables freeing this pointer during XMEM memory clean-up. By default, XMEM allocated pointers are always unlocked. Setting alloc and free trace points unsigned long xmem_get_alloc_nr() Get actual number of memory allocations. void xmem_set_trace_alloc(unsigned long nr) Set allocation trace number to be observed. XMEM will show a message when allocation nr occurs. XMEM Manual 41/57
void xmem_reset_trace_alloc() Reset allocation trace number, disables allocation trace observation. void xmem_set_trace_free(unsigned long nr) Set free trace number to be observed. XMEM will show a message when deallocation nr occurs. void xmem_reset_trace_free() Reset free trace number, disables deallocation trace observation. Resource limitation and execution failing void xmem_set_max_heapsize(size_t size) Set allowed heap size limit for XMEM to size bytes. The value (size_t)-1 means no limitation by XMEM, all available memory, this is the default. This feature can be used to investigate the programs behavior with low / limited memory resources and to check whether error conditions like no memory for malloc calls returning NULL pointers are handled correctly by the program. size_t xmem_get_max_heapsize() Get heap size limit for XMEM. The value (size_t)-1 means no limitation by XMEM, all available memory, this is the default. void xmem_set_stress_ratio(unsigned int ratio) Set allocation fail ratio in percent (%) for XMEM. The value must be between 0% (no allocation is forced to fail, this is the default) and 100% (each allocation is forced to fail). The allocation failing is computed randomly so that no prediction can be done in advance. This means, for example, that for a ratio of 30% an average of 3 out of 10 allocations will fail. This feature can be used to investigate the programs behavior with sporadic allocation fails due to limited memory resources probably caused by other programs running simultaneously and to check whether error conditions like no memory returning NULL pointers are handled correctly. unsigned int xmem_get_stress_ratio() Get allocation fail ratio for XMEM. Tracking program scope void xmem_enter_scope() This function sets the starting point of a specific section of code. Scope sections can be nested. XMEM Manual 42/57
void xmem_leave_scope() This function sets the end of a specific section of code. When leaving a section all memory blocks that have been allocated since the start and which are still allocated are reported. Program information void xmem_set_progname(char *progname) This function can be used to set the program name. The program name is used during program execution. char *xmem_get_progname() This function can be used to retrieve the program name previously set with xmem_set_progname(). XMEM Manual 43/57
XMEM Browser Stand-alone XMEM data collection XMEM reports errors, warnings, summaries of memory leaks and statistics information to a command line box, message box or log files. If option XMEM_BROWSER is enabled, all this information is sent as Windows message by the running program and can be collected and displayed from an independent application. XMEM Browser is an example for such an application that receives and collects these messages and shows the various types of information on different tabbed pages. The structure of the messages and their handling can be found in xmem.c and in the XMEM Browser source code. XMEM Browser is a rough and quickly written proof of concept example, it is neither complete nor is the performance satisfying. XMEM Browser is programmed in C# and requires Microsoft.NET Framework 2.0 or higher to run on Microsoft Windows 32 bit and 64 bit operating systems. The Microsoft Visual Studio 2008 project comes with complete source code and can be used as an example for further development. Note that the XMEM Browser probably needs to be adapted to run on 64 bit Windows systems and work with 64 bit programs sending XMEM data. The main reason is that the Windows messages need to be changed for handling 64 bit (8 byte) pointer information. However, this has not yet been tested due to the non-availability of a 64 bit test system. XMEM Manual 44/57
XMEM Browser windows The following figures show some XMEM Browser screen shots with example output collected from an XMEM instrumented application. Figure: XMEM Browser message window Figure: XMEM Browser log window XMEM Manual 45/57
Figure: XMEM Browser call statistics window Figure: XMEM Browser size statistics window XMEM Manual 46/57