Dr. Watson(Postmortem Debugger) Mini Dump Analysis using WinDBG

Transcription

1 Dr. Watson(Postmortem Debugger) Mini Dump Analysis using WinDBG Debugging Steps using WinDBG 1) Open the.dmp file using WinDBG 2) Make sure your symbol server path is set correctly(file->symbol File Path), below is what I have set it to srv*c:\temp\symbols* 3) You should save the workspace to save your symbol path information unless you are using _NT_SYMBOL_PATH otherwise you will loose the information next time you start WinDBG. Remember to save it without opening a dump file to use it for the next debugging session 4) Run!analyze -v 5) 0:182>!analyze -v FAULTING_IP: kernel32!raiseexception+53 77e4bee7 5e pop esi EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 77e4bee7 (kernel32!raiseexception+0x ) ExceptionCode: e0434f4d (CLR exception) ExceptionFlags: NumberParameters: 1 Parameter[0]: e DEFAULT_BUCKET_ID: CLR_EXCEPTION ERROR_CODE: (NTSTATUS) 0xe0434f4d - <Unable to get error code text> MANAGED_STACK:!dumpstack -EE OS Thread Id: 0xdd0 (182) TEB information is not available so a stack size of 0xFFFF is assumed Current frame: ChildEBP RetAddr Caller,Callee ADDITIONAL_DEBUG_TEXT: Followup set via attribute from Frame ffffffff on thread ffffffff FAULTING_THREAD: 00000dd0 PRIMARY_PROBLEM_CLASS: CLR_EXCEPTION BUGCHECK_STR: APPLICATION_FAULT_CLR_EXCEPTION LAST_CONTROL_TRANSFER: from 79f071ac to 77e4bee7 STACK_TEXT: 15d9f418 79f071ac e0434f4d kernel32!raiseexception+0x53 15d9f478 79f9293a 75c9cb mscorwks!raisetheexceptioninternalonly+0x2a8 15d9f4b0 7a00e74a 15d9f508 7a3c441c 9ae216b2 mscorwks!unwindandcontinuerethrowhelperaftercatch+0x70 15d9f a858c 0874cd af628 mscorwks!jit_newfast+0x105 15d9f578 79e7c74b d9f5e0 15d9f608 mscorlib_ni+0x2e858c 15d9f588 79e7c6cc d9f714 mscorwks!calldescrworker+0x33 15d9f608 79f00eca d9f714 mscorwks!calldescrworkerwithhandler+0xa3 15d9f628 79f00e d9f714 mscorwks!dispatchcallbody+0x1e 15d9f68c 79f00f d9f714 mscorwks!dispatchcalldebuggerwrapper+0x3d 15d9f6c0 79f d9f714 mscorwks!dispatchcallnoeh+0x51

2 15d9f720 79ef d9f870 7a3b57d mscorwks!addtimercallback_worker+0x79 15d9f734 79ef31a3 15d9f810 15d9f7bc 79f91478 mscorwks!thread::doadcallback+0x32a 15d9f7c8 79ef30c3 15d9f810 9ae21bee 7a3b57d8 mscorwks!thread::shouldchangeaborttounload+0xe3 15d9f804 79ef d9f mscorwks!thread::shouldchangeaborttounload+0x30a 15d9f82c 79ef48cf f d9f870 mscorwks!thread::shouldchangeaborttounload+0x33e 15d9f844 79f f d9f870 mscorwks!managedthreadbase::threadpool+0x13 15d9f88c 79f0288c 106bcc mscorwks!addtimercallbackex+0x83 15d9f8a0 79f bcc ae21b36 mscorwks!addtimercallback+0x10 15d9f8dc 79f024cf 2a023e70 9ae21ac2 79f02030 mscorwks!threadpoolmgr::asynctimercallbackcompletion+0x64 15d9f928 79f0202a e619d1 mscorwks! UnManagedPerAppDomainTPCount::DispatchWorkItem+0xb2 15d9f93c 79f021a0 9ae21a7e f0207d mscorwks!threadpoolmgr::executeworkrequest+0xaf 15d9f994 79f95a2e c81c5f0 15d9fa48 mscorwks!threadpoolmgr::workerthreadstart+0x223 15d9ffb8 77e f61d mscorwks!thread::intermediatethreadproc+0x49 15d9ffec f959e8 106f61d kernel32!basethreadstart+0x34 6) RaiseTheExceptionInternalOnly method signature is RaiseTheExceptionInternalOnly(OBJECTREF, BOOL, BOOL) 7) 2nd parameter evaluates to whether the exception is re-thrown and the 3 rd parameter tells you if this is a stack overflow exception but in this case both of them are false 8) ExceptionCode: e0434f4d (CLR exception) which applies to any unhandled CLR exception e.g DivideByZero 9) mini dump reports 1204 threads, as shown below 0:182> ~ 0 Id: Suspend: 1 Teb: 7ffdf000 Unfrozen 1 Id: 968.c34 Suspend: 1 Teb: 7ffde000 Unfrozen 2 Id: 968.8e0 Suspend: 1 Teb: 7ffdd000 Unfrozen 3 Id: 968.a90 Suspend: 1 Teb: 7ffdc000 Unfrozen Id: b0 Suspend: 1 Teb: 7fbd6000 Unfrozen 1201 Id: 968.1bc4 Suspend: 1 Teb: 7fae4000 Unfrozen 1202 Id: ac Suspend: 1 Teb: 7faba000 Unfrozen 1203 Id: 968.1b7c Suspend: 1 Teb: 7fab8000 Unfrozen More than 200 threads have the following stack trace 10) 0:182> ~*k 1201 Id: 968.1bc4 Suspend: 1 Teb: 7fae4000 Unfrozen ChildEBP RetAddr 7b22ebec 7c827d0b ntdll!kifastsystemcallret 7b22ebf0 77e61d1e ntdll!ntwaitforsingleobject+0xc 7b22ec60 79e789c6 kernel32!waitforsingleobjectex+0xac 7b22eca4 79e7898f mscorwks!peimage::loadimage+0x1af 7b22ecf4 79e78944 mscorwks!clrevent::waitex+0x117 7b22ed08 79fbc82a mscorwks!clrevent::wait+0x17 7b22ed1c 79fbc6da mscorwks!svr::gcheap::waituntilgccomplete+0x34 7b22ed58 79fc51bf mscorwks!thread::raredisablepreemptivegc+0x1a0 7b22ed8c 79fc529b mscorwks!thread::usersleep+0xb1 7b22ee2c 0deb1401 mscorwks!threadnative::sleep+0xce WARNING: Frame IP not in any known module. Following frames may be wrong. 7b22ef18 793b0d1f 0xdeb1401 7b22ef ab mscorlib_ni+0x2f0d1f 7b22ef30 79e7c74b mscorlib_ni+0x2b40ab 7b22ef40 79e7c6cc mscorwks!calldescrworker+0x33 7b22efc0 79e7c8e1 mscorwks!calldescrworkerwithhandler+0xa3 7b22f0fc 79e7c783 mscorwks!methoddesc::calldescr+0x19c 7b22f118 79e7c90d mscorwks!methoddesc::calltargetworker+0x1f 7b22f12c 79eb300f mscorwks!methoddesccallsite::call_retargslot+0x18

3 7b22f2f8 79eb2f31 mscorwks!executecodewithguaranteedcleanuphelper+0x9b 7b22f3a ff7 mscorwks!reflectioninvocation::executecodewithguaranteedcleanup+0xf9 11) There are 2 GC threads (2 and 3) since this process is running on Windows Server 2003 with 2 CPUs so one GC thread per CPU. 0:182> ~2 kb 00b7fd88 7c827d0b 77e61d1e e ntdll!kifastsystemcallret 00b7fd8c 77e61d1e e ntdll!ntwaitforsingleobject+0xc 00b7fdfc 79e789c e8 ffffffff kernel32!waitforsingleobjectex+0xac 00b7fe40 79e7898f e8 ffffffff mscorwks!peimage::loadimage+0x1af 00b7fe90 79e78944 ffffffff mscorwks!clrevent::waitex+0x117 00b7fea4 79efa6c3 ffffffff mscorwks!clrevent::wait+0x17 00b7fee8 79efb5b f mscorwks!svr::t_join::join+0xae 00b7ff28 79efa9ba f8 mscorwks!svr::gc_heap::mark_phase+0x16f 00b7ff54 79efaf60 ffffffff f8 mscorwks!svr::gc_heap::gc1+0x46 00b7ff74 79efa72f f8 mscorwks!svr::gc_heap::garbage_collect+0x246 00b7ff98 79fc fc e fc857c mscorwks!svr::gc_heap::gc_thread_function+0x6a 00b7ffb8 77e f mscorwks!svr::gc_heap::gc_thread_stub+0x92 00b7ffec fc f kernel32!basethreadstart+0x34 0:182> ~3 kb 00c7fc9c 79efd1d0 55b4de f8 mscorwks!svr::gc_heap::mark_object_simple1+0xdf 00c7fce4 79efba ffcc c94 mscorwks!svr::gc_heap::mark_object_simple+0x254 00c7fd10 79fbb49d 09063c ffcc mscorwks!svr::gcheap::promote+0x115 00c7fd28 79fc3a38 00c7fdd c mscorwks!gcenumobject+0x2d 00c7fd48 79fc39e fbb475 00c7fdd0 mscorwks!module::enumstaticgcrefs+0x59 00c7fd74 79fc3ada 79fbb475 00c7fdd ea8 mscorwks!domainassembly::enumstaticgcrefs+0x38 00c7fd98 79fc3a9b 79fbb475 00c7fdd0 2a03d8b0 mscorwks!appdomain::enumstaticgcrefs+0x29 00c7fdb4 79fc399b 79fbb475 00c7fdd0 79efba5a mscorwks!systemdomain::enumallstaticgcrefs+0x4e 00c7fddc 79efb mscorwks!cnamespace::gcscanroots+0x1a2 00c7fe28 79efa9ba d30 mscorwks!svr::gc_heap::mark_phase+0x111 00c7fe54 79efaf60 ffffffff d30 mscorwks!svr::gc_heap::gc1+0x46 00c7fe74 79efa72f d30 mscorwks!svr::gc_heap::garbage_collect+0x246 00c7fe98 79fc c8276bf 00c7fef8 mscorwks!svr::gc_heap::gc_thread_function+0x6a 00c7ffb8 77e d mscorwks!svr::gc_heap::gc_thread_stub+0x92 00c7ffec fc d kernel32!basethreadstart+0x34 12) Mark_Phase signature is mark_phase (int gen_number, BOOL mark_only_p parameters passed are 2, 0, that means GC was triggered for generation 2 13)Thread # 345 has triggered garbage collection for generation 2 with a request to allocate memory from GC Heap(see highleted in red) 0:182> ~345 kb e0bc 7c827d0b 77e61d1e ntdll!kifastsystemcallret 2484e0c0 77e61d1e ntdll!ntwaitforsingleobject+0xc 2484e130 79e789c ffffffff kernel32!waitforsingleobjectex+0xac 2484e174 79e7898f ffffffff mscorwks!peimage::loadimage+0x1af 2484e1c4 79e78944 ffffffff mscorwks!clrevent::waitex+0x e1d8 79efafc5 ffffffff mscorwks!clrevent::wait+0x e1fc 79efad3d ffffffff 71ca dd4 mscorwks!svr::gc_heap::wait_for_gc_done+0x e224 79fd1cf d30 mscorwks!svr::gcheap::garbagecollectgeneration+0x1b5 2484e2b4 79ef98cf 14fa94e c mscorwks!svr::gc_heap::try_allocate_more_space+0x e2d0 79ef9b9c 14fa94e c mscorwks!svr::gc_heap::allocate_more_space+0x2e 2484e2f0 79e7510e 14fa94e c mscorwks!svr::gcheap::alloc+0x e30c 79e c mscorwks!alloc+0x e34c 79e858cb abbf001a d8 mscorwks!fastallocateobject+0x38

4 2484e3f0 793ab e mscorwks!jit_newfast+0x9e 2484e aaef0 f1dcc000 ffffffbc mscorlib_ni+0x2eb e b0d1f 3a5a ab mscorlib_ni+0x2eaef0 2484e ab fa94a0 2484e640 mscorlib_ni+0x2f0d1f 2484e630 79e7c74b e6c0 mscorlib_ni+0x2b40ab 2484e640 79e7c6cc 2484e e6e0 mscorwks!calldescrworker+0x e6c0 79e7c8e1 2484e e6e0 mscorwks!calldescrworkerwithhandler+0xa3 2484e7fc 79e7c c01c0 2484e92c 2484e84c mscorwks!methoddesc::calldescr+0x19c 2484e818 79e7c90d 796c01c0 2484e92c 2484e84c mscorwks!methoddesc::calltargetworker+0x1f 2484e82c 79eb300f 2484e84c abbf0a12 14fa94a0 mscorwks!methoddesccallsite::call_retargslot+0x e9f8 79eb2f ea8c abbf mscorwks!executecodewithguaranteedcleanuphelper+0x9b 2484eaa ff7 2484ea4c c8 0109dd28 mscorwks! ReflectionInvocation::ExecuteCodeWithGuaranteedCleanup+0xf9 2484eac ede 4a34402c 2484eaf0 2484eb1c mscorlib_ni+0x2b3ff mscorlib_ni+0x2b3ede Analysis 1. Application has thrown exception at the top of stack 2. In CLR 2.0, unhandled exceptions at the top of the stack on any thread will terminate the application 3. Exception has occurred after GC was triggered 4. Mark Phase was triggered for Generation 2 that means it was under severe memory pressure 5. since there are more than 1200 threads that itself amounts to 1.2 GB of virtual memory for just the user mode stack and each thread has user mode plus kernel mode stack although not much but kernel stack will also add up to another 3 virtual pages(12kb) 1200*12KB of physical memory, kernel stack is physical memory resident. There will be around 100 MB or so in MEM_IMAGE itself 6. Since this exception has occurred while starting a worker thread from managed thread pool and at the same GC was also triggered. So that means there was not enough memory 7. There is no thread leak, these many threads are by design(one thread per request, I don't know why) 8. Some of the interesting points There are more than 200 threads with preemptive GC disabled what that means is these threads can't be suspended and GC Threads will have to wait for these threads to return to preemptive GC mode in order to claim the memory. Managed threads should not be in a state with preemptive GC disabled because this is very rare as interpreted from the call stack Thread::RareDisablePreemptiveGC Below is the description of Preemptive GC from Preemptive GC: also very important. In Rotor, this is m_fpreemptivegcdisabled field of C++ Thread class. It indicates what GC mode the thread is in: "enabled" in the table means the thread is in preemptive mode where GC could preempt this thread at any time; "disabled" means the thread is in cooperative mode where GC has to wait the thread to give up its current work (the work is related to GC objects so it can't allow GC to move the objects around). When the thread is executing managed code (the current IP is in managed code), it is always in cooperative mode; when the thread is in Execution Engine (unmanaged code), EE code could choose to stay in either mode and could switch mode at any time; when a thread are outside of CLR (e.g, calling into native code using interop), it is always in preemptive mode. CLR rotor source code has the following comments before calling Thread::RareDisablePreemptiveGC We must do the following in this order, because otherwise we would be constructing the exception for the abort

5 without synchronizing with the GC. Also, we have no CLR SEH set up, despite the fact that we may throw a ThreadAbortException. Most likely it crashed because of not having enough memory to start a worker thread since it crashed with the call stack pointing to WokerThreadStart The real challenge is how will you find out the exception detail for sure?how will you make sure that this is indeed a out of memory exception? Since this is a mini dump with no information on CLR Data structures. Effective managed debugging requires a mini dump with full virtual memory because managed heap is created with virtualalloc but Dr. Watson as a postmortem debugger with default options creates a minidump which includes all thread contexts and 10 instructions