Encyclopedia of Crash Dump Analysis Patterns Detecting Abnormal Software Structure and Behavior in Computer Memory Dmitry Vostokov Software Diagnostics Institute OpenTask
2 Published by OpenTask, Republic of Ireland Copyright 2015 by Dmitry Vostokov Copyright 2015 by Software Diagnostics Institute All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior written permission of the publisher. You must not circulate this book in any other binding or cover, and you must impose the same condition on any acquirer. OpenTask books are available through booksellers and distributors worldwide. For further information or comments send requests to press@opentask.com. Product and company names mentioned in this book may be trademarks of their owners. A CIP catalog record for this book is available from the British Library. ISBN-13: 978-1-906717-21-6 (Paperback) First printing, 2015 Version 1.0
3 Summary of Contents Summary of Contents 3 Detailed Table of Contents 17 Preface 41 A 43 Abridged Dump 43 Accidental Lock 47 Activation Context 54 Active Thread 57 Activity Resonance 59 Affine Thread 61 Annotated Disassembly 64 B 65 Blocked DPC 65 Blocked Queue 66 Blocked Thread 69 Blocking File 82 Blocking Module 85 Broken Link 86 Busy System 88 C 97 C++ Exception 97
4 Caller-n-Callee 99 Changed Environment 102 Cloud Environment 106 CLR Thread 108 Coincidental Error Code 112 Coincidental Frames 114 Coincidental Symbolic Information 118 Corrupt Dump 124 Corrupt Structure 126 Coupled Machines 128 Coupled Modules 129 Coupled Processes 130 Crash Signature 136 Crash Signature Invariant 138 Crashed Process 139 Critical Section Corruption 140 Critical Stack Trace 148 Custom Exception Handler 149 D 154 Data Alignment 154 Data Contents Locality 155 Data Correlation 160
Deadlock 162 5 Debugger Bug 200 Debugger Omission 201 Design Value 202 Deviant Module 203 Deviant Token 210 Dialog Box 211 Directing Module 215 Disconnected Network Adapter 216 Disk Packet Buildup 218 Dispatch Level Spin 221 Distributed Spike 224 Distributed Wait Chain 232 Divide by Zero 234 Double Free 238 Double IRP Completion 258 Driver Device Collection 260 Dry Weight 261 Dual Stack Trace 262 Duplicate Extension 263 Duplicated Module 267 Dynamic Memory Corruption 272
6 E 292 Early Crash Dump 292 Effect Component 295 Embedded Comments 301 Empty Stack Trace 302 Environment Hint 306 Error Reporting Fault 307 Exception Module 310 Exception Stack Trace 312 Execution Residue 314 F 330 Fake Module 330 False Effective Address 334 False Function Parameters 335 False Positive Dump 338 Fat Process Dump 340 Fault Context 341 First Fault Stack Trace 342 FPU Exception 343 Frame Pointer Omission 345 Frozen Process 349 G 353
Ghost Thread 353 7 Glued Stack Trace 355 H 358 Handle Leak 358 Handle Limit 359 Handled Exception 365 Hardware Activity 374 Hardware Error 378 Hidden Call 386 Hidden Exception 391 Hidden IRP 397 Hidden Module 398 Hidden Parameter 400 Hidden Process 402 High Contention 404 Historical Information 415 Hooked Functions 416 Hooked Modules 422 Hooking Level 424 I 427 Incomplete Stack Trace 427 Incomplete Session 428
8 Inconsistent Dump 430 Incorrect Stack Trace 431 Incorrect Symbolic Information 437 Injected Symbols 442 Inline Function Optimization 445 Instrumentation Information 449 Instrumentation Side Effect 453 Insufficient Memory 456 Invalid Exception Information 502 Invalid Handle 506 Invalid Parameter 518 Invalid Pointer 521 IRP Distribution Anomaly 523 J 525 JIT Code 525 L 528 Last Error Collection 528 Last Object 530 Late Crash Dump 531 Lateral Damage 532 Least Common Frame 533 Livelock 535
Local Buffer Overflow 537 9 Lost Opportunity 540 M 542 Main Thread 542 Managed Code Exception 545 Managed Stack Trace 552 Manual Dump 553 Memory Fluctuation 562 Memory Leak 564 Message Box 588 Message Hooks 591 Missing Component 594 Missing Process 608 Missing Thread 609 Mixed Exception 614 Module Collection 619 Module Hint 622 Module Product Process 624 Module Variable 625 Module Variety 627 Multiple Exceptions 630 N 640
10 Namespace 640 Nested Exceptions 641 Nested Offender 648 Network Packet Buildup 651 No Component Symbols 652 No Current Thread 655 No Data Types 657 No Process Dumps 658 No System Dumps 659 Not My Version 660 NULL Pointer 662 O 668 OMAP Code Optimization 668 One-Thread Process 672 Optimized Code 674 Optimized VM Layout 676 Origin Module 678 Out-of-Module Pointer 680 Overaged System 681 P 682 Packed Code 682 Paged Out Data 685
Paratext 687 11 Pass Through Function 689 Passive System Thread 691 Passive Thread 695 Past Stack Trace 702 Patched Code 704 Pervasive System 705 Platform-Specific Debugger 706 Pleiades 708 Pre-Obfuscation Residue 709 Problem Exception Handler 710 Problem Module 712 Problem Vocabulary 713 Process Factory 714 Punctuated Memory Leak 719 Q 723 Quiet Dump 723 R 724 Random Object 724 Raw Pointer 727 Reduced Symbolic Information 728 Reference Leak 729
12 Regular Data 732 RIP Stack Trace 733 Rough Stack Trace 735 S 738 Same Vendor 738 Screwbolt Wait Chain 739 Self-Diagnosis 740 Self-Dump 745 Semantic Split 747 Semantic Structure 754 Shared Buffer Overwrite 758 Shared Structure 766 Small Value 767 Software Exception 769 Special Process 770 Special Stack Trace 775 Special Thread 776 Spike Interval 777 Spiking Thread 778 Stack Overflow 787 Stack Trace 808 Stack Trace Change 822
Stack Trace Collection 823 13 Stack Trace Set 839 Step Dumps 842 Stored Exception 843 String Hint 844 String Parameter 846 Suspended Thread 848 Swarm of Shared Locks 850 System Object 855 T 858 Tampered Dump 858 Technology-Specific Subtrace 871 Template Module 879 Thread Age 883 Thread Cluster 885 Thread Starvation 886 Top Module 892 Translated Exception 893 Truncated Dump 894 Truncated Stack Trace 897 U 898 Ubiquitous Component 898
14 Unknown Component 913 Unloaded Module 917 Unrecognizable Symbolic Information 921 Unsynchronized Dumps 926 V 927 Value Adding Process 927 Value Deviation 928 Value References 932 Version-Specific Extension 933 Virtualized Process 937 Virtualized System 945 W 951 Wait Chain 951 Waiting Thread Time 1001 Well-Tested Function 1010 Well-Tested Module 1011 Wild Code 1012 Wild Pointer 1014 Y 1016 Young System 1016 Z 1018 Zombie Processes 1018
Appendix A 1025 15 Reference Stack Traces 1025 Appendix B 1026.NET / CLR / Managed Space Patterns 1026 Contention Patterns 1027 Deadlock and Livelock Patterns 1028 DLL Link Patterns 1029 Dynamic Memory Corruption Patterns 1030 Executive Resource Patterns 1031 Exception Patterns 1032 Falsity and Coincidence Patterns 1033 Hooksware Patterns 1034 Insufficient Memory Patterns 1036 Meta-Memory Dump Patterns 1037 Module Patterns 1038 Optimization Patterns 1039 Process Patterns 1040 RPC, LPC and ALPC Patterns 1041 Stack Overflow Patterns 1042 Stack Trace Patterns 1043 Symbol Patterns 1044 Thread Patterns 1045
16 Wait Chain Patterns 1046 Appendix C 1047 Crash Dump Analysis Checklist 1047 Index 1051
17 Detailed Table of Contents Summary of Contents 3 Detailed Table of Contents 17 Preface 41 A 43 Abridged Dump 43 Accidental Lock 47 Activation Context 54 Active Thread 57 Mac OS X 57 Activity Resonance 59 Affine Thread 61 Annotated Disassembly 64 JIT.NET Code 64 B 65 Blocked DPC 65 Blocked Queue 66 LPC/ALPC 66 Comments 68 Blocked Thread 69 Hardware 69 Software 71
18 Comments 79 Timeout 81 Blocking File 82 Blocking Module 85 Comments 85 Broken Link 86 Busy System 88 C 97 C++ Exception 97 Windows 97 Comments 97 Mac OS X 98 Caller-n-Callee 99 Changed Environment 102 Comments 105 Cloud Environment 106 CLR Thread 108 Comments 111 Coincidental Error Code 112 Coincidental Frames 114 Coincidental Symbolic Information 118 Windows 118
Mac OS X 122 19 Corrupt Dump 124 Comments 125 Corrupt Structure 126 Coupled Machines 128 Coupled Modules 129 Coupled Processes 130 Semantics 130 Strong 131 Comments 132 Weak 133 Crash Signature 136 Crash Signature Invariant 138 Crashed Process 139 Critical Section Corruption 140 Critical Stack Trace 148 Custom Exception Handler 149 Kernel Space 149 User Space 151 D 154 Data Alignment 154 Page Boundary 154
20 Data Contents Locality 155 Data Correlation 160 Function Parameters 160 Deadlock 162 Critical Sections 162 Comments 169 Executive Resources 174 LPC 178 Managed Space 183 Mixed Objects 186 Kernel Space 186 User Space 191 Comments 198 Self 199 Comments 199 Debugger Bug 200 Debugger Omission 201 Design Value 202 Deviant Module 203 Comments 209 Deviant Token 210 Dialog Box 211 Directing Module 215
Disconnected Network Adapter 216 21 Disk Packet Buildup 218 Dispatch Level Spin 221 Distributed Spike 224 Comments 231 Distributed Wait Chain 232 Divide by Zero 234 Kernel Mode 234 User Mode 236 Windows 236 Mac OS X 237 Double Free 238 Kernel Pool 238 Comments 241 Process Heap 246 Windows 246 Comments 255 Mac OS X 257 Double IRP Completion 258 Driver Device Collection 260 Dry Weight 261 Dual Stack Trace 262 Duplicate Extension 263
22 Comments 266 Duplicated Module 267 Comments 271 Dynamic Memory Corruption 272 Kernel Pool 272 Comments 278 Managed Heap 282 Process Heap 285 Windows 285 Comments 286 Mac OS X 290 E 292 Early Crash Dump 292 Effect Component 295 Embedded Comments 301 Empty Stack Trace 302 Comments 305 Environment Hint 306 Error Reporting Fault 307 Exception Module 310 Exception Stack Trace 312 Comments 313 Execution Residue 314
23 Mac OS X 314 Managed Space 316 Comments 317 Unmanaged Space 318 Comments 329 F 330 Fake Module 330 False Effective Address 334 False Function Parameters 335 False Positive Dump 338 Fat Process Dump 340 Fault Context 341 First Fault Stack Trace 342 FPU Exception 343 Frame Pointer Omission 345 Frozen Process 349 G 353 Ghost Thread 353 Glued Stack Trace 355 H 358 Handle Leak 358 Handle Limit 359
24 GDI 359 Handled Exception 365.NET CLR 365 Kernel Space 370 User Space 371 Comments 373 Hardware Activity 374 Hardware Error 378 Comments 383 Hidden Call 386 Hidden Exception 391 Kernel Space 391 User Space 392 Hidden IRP 397 Hidden Module 398 Comments 399 Hidden Parameter 400 Hidden Process 402 High Contention 404.NET CLR Monitors 404 Critical Sections 407 Executive Resources 409 Comments 411
Processors 412 25 Historical Information 415 Comments 415 Hooked Functions 416 Kernel Space 416 Comments 419 User Space 420 Hooked Modules 422 Comments 423 Hooking Level 424 I 427 Incomplete Stack Trace 427 GDB 427 Incomplete Session 428 Comments 429 Inconsistent Dump 430 Comments 430 Incorrect Stack Trace 431 Comments 436 Incorrect Symbolic Information 437 Injected Symbols 442 Inline Function Optimization 445
26 Managed Code 445 Unmanaged Code 447 Instrumentation Information 449 Instrumentation Side Effect 453 Comments 455 Insufficient Memory 456 Committed Memory 456 Control Blocks 458 Handle Leak 459 Comments 463 Kernel Pool 468 Comments 476 Module Fragmentation 477 Comments 484 Physical Memory 485 PTE 488 Comments 489 Region 490 Reserved Virtual Memory 492 Session Pool 495 Stack Trace Database 496 Invalid Exception Information 502 Invalid Handle 506 General 506
Comments 509 Managed Space 510 27 Invalid Parameter 518 Process Heap 518 Invalid Pointer 521 General 521 IRP Distribution Anomaly 523 J 525 JIT Code 525.NET 525 Comments 527 L 528 Last Error Collection 528 Last Object 530 Late Crash Dump 531 Lateral Damage 532 Comments 532 Least Common Frame 533 Livelock 535 Local Buffer Overflow 537 Mac OS X 537 Windows 539
28 Lost Opportunity 540 M 542 Main Thread 542 Managed Code Exception 545 Managed Stack Trace 552 Manual Dump 553 Kernel 553 Comments 555 Process 558 Comments 561 Memory Fluctuation 562 Process Heap 562 Memory Leak 564.NET Heap 564 Comments 570 I/O Completion Packets 571 Page Tables 572 Process Heap 578 Comments 584 Regions 585 Message Box 588 Comments 590 Message Hooks 591
Missing Component 594 29 General 594 Static Linkage 598 User Mode 598 Missing Process 608 Comments 608 Missing Thread 609 Comments 613 Mixed Exception 614 Comments 618 Module Collection 619 General 619 Predicate 621 Module Hint 622 Comments 623 Module Product Process 624 Module Variable 625 Module Variety 627 Multiple Exceptions 630 Windows 630 Kernel Mode 630 Managed Space 635 User Mode 636
30 Mac OS X 638 N 640 Namespace 640 Nested Exceptions 641 Managed Code 641 Unmanaged Code 644 Nested Offender 648 Network Packet Buildup 651 No Component Symbols 652 No Current Thread 655 No Data Types 657 No Process Dumps 658 No System Dumps 659 Comments 659 Not My Version 660 Hardware 660 Software 661 NULL Pointer 662 Windows 662 Code 662 Data 664 Mac OS X 665 Code 665
Data 667 31 O 668 OMAP Code Optimization 668 One-Thread Process 672 Optimized Code 674 Comments 675 Optimized VM Layout 676 Origin Module 678 Out-of-Module Pointer 680 Overaged System 681 Comments 681 P 682 Packed Code 682 Paged Out Data 685 Paratext 687 Mac OS X 687 Comments 688 Pass Through Function 689 Comments 690 Passive System Thread 691 Kernel Space 691 Passive Thread 695
32 User Space 695 Comments 701 Past Stack Trace 702 Patched Code 704 Pervasive System 705 Platform-Specific Debugger 706 Pleiades 708 Pre-Obfuscation Residue 709 Problem Exception Handler 710 Comments 711 Problem Module 712 Problem Vocabulary 713 Process Factory 714 Punctuated Memory Leak 719 Q 723 Quiet Dump 723 R 724 Random Object 724 Raw Pointer 727 Reduced Symbolic Information 728 Reference Leak 729 Regular Data 732
RIP Stack Trace 733 33 Rough Stack Trace 735 S 738 Same Vendor 738 Screwbolt Wait Chain 739 Self-Diagnosis 740 Kernel Mode 740 Comments 740 Registry 741 User Mode 743 Comments 744 Self-Dump 745 Semantic Split 747 Semantic Structure 754 PID.TID 754 Comments 757 Shared Buffer Overwrite 758 Windows 758 Mac OS X 762 Shared Structure 766 Small Value 767 Comments 768 Software Exception 769
34 Special Process 770 Comments 774 Special Stack Trace 775 Comments 775 Special Thread 776.NET CLR 776 Spike Interval 777 Spiking Thread 778 Windows 778 Comments 783 Mac OS X 785 Stack Overflow 787 Windows 787 Kernel Mode 787 Comments 795 Software Implementation 797 User Mode 799 Comments 802 Mac OS X 804 Stack Trace 808 Windows 808 Database 808 File System Filters 813
35 General 815 I/O Request 819 Mac OS X 821 Stack Trace Change 822 Stack Trace Collection 823 I/O Requests 823 Managed Space 827 Predicate 830 Unmanaged Space 831 Comments 838 Stack Trace Set 839 Step Dumps 842 Stored Exception 843 String Hint 844 String Parameter 846 Suspended Thread 848 Swarm of Shared Locks 850 System Object 855 T 858 Tampered Dump 858 Technology-Specific Subtrace 871 COM Interface Invocation 871 Dynamic Memory 874
36 JIT.NET Code 876 Template Module 879 Thread Age 883 Thread Cluster 885 Thread Starvation 886 Normal Priority 886 Realtime Priority 888 Top Module 892 Translated Exception 893 Truncated Dump 894 Windows 894 Mac OS X 896 Truncated Stack Trace 897 Comments 897 U 898 Ubiquitous Component 898 Kernel Space 898 User Space 901 Unknown Component 913 Unloaded Module 917 Unrecognizable Symbolic Information 921 Unsynchronized Dumps 926
V 927 37 Value Adding Process 927 Value Deviation 928 Stack Trace 928 Value References 932 Comments 932 Version-Specific Extension 933 Virtualized Process 937 WOW64 937 Comments 944 Virtualized System 945 W 951 Wait Chain 951 CLR Monitors 951 Critical Sections 952 Executive Resources 955 General 959 Comments 963 LPC/ALPC 964 Modules 970 Mutex Objects 971 Named Pipes 973 Process Objects 975
38 Pushlocks 980 RPC 982 RTL_RESOURCE 986 Thread Objects 992 Window Messaging 996 Waiting Thread Time 1001 Kernel Dumps 1001 Comments 1006 User Dumps 1008 Comments 1009 Well-Tested Function 1010 Well-Tested Module 1011 Wild Code 1012 Wild Pointer 1014 Y 1016 Young System 1016 Z 1018 Zombie Processes 1018 Comments 1024 Appendix A 1025 Reference Stack Traces 1025 Appendix B 1026.NET / CLR / Managed Space Patterns 1026
Contention Patterns 1027 39 Deadlock and Livelock Patterns 1028 DLL Link Patterns 1029 Dynamic Memory Corruption Patterns 1030 Executive Resource Patterns 1031 Exception Patterns 1032 Falsity and Coincidence Patterns 1033 Hooksware Patterns 1034 Insufficient Memory Patterns 1036 Meta-Memory Dump Patterns 1037 Module Patterns 1038 Optimization Patterns 1039 Process Patterns 1040 RPC, LPC and ALPC Patterns 1041 Stack Overflow Patterns 1042 Stack Trace Patterns 1043 Symbol Patterns 1044 Thread Patterns 1045 Wait Chain Patterns 1046 Appendix C 1047 Crash Dump Analysis Checklist 1047 Index 1051