StackMine Performance Debugging in the Large via Mining Millions of Stack Traces

Transcription

1 StackMine Performance Debugging in the Large via Mining Millions of Stack Traces Shi Han 1, Yingnong Dang 1, Song Ge 1, Dongmei Zhang 1, and Tao Xie 2 Software Analytics Group, Microsoft Research Asia 1 North Carolina State University 2 June 6 th, 2012

2 Performance Issues in the Real World One of top user complaints Impacting large number of users every day High impact on usability and productivity High Disk I/O High CPU consumption Given limited time and resource before software release, development-site testing and debugging become insufficient to ensure satisfactory software performance. 2

3 Performance Debugging in the Large Pattern Matching Bug update Network Trace collection Problematic Pattern Repository How many issues are still unknown? Trace Storage Which trace file should I investigate first? Bug Database Bug filing Key to issue discovery Bottleneck of scalability Trace analysis 3

4 Problem Definition Input: Runtime traces collected from millions of users Output: Program execution patterns causing the most impactful performance problems inspected by performance analysts 4

5 Goal Conduct systematic discovery & analysis of program execution patterns Efficient handling of large-scale trace sets Automatic discovery of new patterns Effective prioritization of investigation 5

6 Challenges Internet Large-scale trace data TBs of trace files and increasing Millions of events in single trace stream Highly complex analysis Numerous program runtime combinations triggering performance problems Multi-layer runtime components from application to kernel being intertwined Combination of expertise Generic machine learning tools without domain knowledge guidance do not work well 6

7 Wait callstack ntdll!userthreadstart Browser! Main Browser!OnBrowserCreatedAsyncCallback BrowserUtil!ProxyMaster::GetOrCreateSlave BrowserUtil!ProxyMaster::ConnectToObject Intuition Wait callstack ntdll!userthreadstart Browser! Main What happens behind a typical ntdll!ldrloaddll UI-delay? An example of delayed rpc!proxysendreceive browser wow64!runcpusimulation tab creation - wow64cpu!waitformultipleobjects32 wow64cpu!cpupsyscallstub nt!accessfault nt!pagefault Time ReadyThread callstack nt!kiretiredpclist nt!executealldpcs nt!iopfcompleterequest nt!setevent UI thread CPU Wait Ready CPU Wait Ready CPU Worker thread Ready Unexpected long execution CPU Underlying Disk I/O CPU CPU sampled CPU sampled sampled callstack callstack callstack ntdll!userthreadstart ntdll!userthreadstart ntdll!userthreadstart Ntdll!WorkerThread ntdll!workerthread ole!cocreateinstance ole!outserializer::unmarshalatindex ole!counmarshalinterface ReadyThread callstack ntdll!userthreadstart rpc!lrpciocomplete user32!postmessage win32k!setwakebit nt!setevent Wait Callstacks ReadyThread Callstacks CPU Sampled Callstacks 7

8 Approach Formulated as a callstack mining and clustering problem Caused by Mainly represented by Performance Issues Problematic program execution patterns Callstack patterns Discovered by mining & clustering costly patterns 8

9 Approach Workflow Pattern clusters 709 Trace streams 1 23 ntdll!openfile Ranked cluster Cluster Hits Total Wait time (ms) 1 94, , , , ,536 3,051, nt!pageread Ranking 1392 AOI Pattern AOI Extraction Clustering 412 nt!accessfault AOI Extraction Sequence Pattern Ranking Mining Pattern Clustering Callstacks Sequence Pattern Mining 709 ntdll!openfile ntdll!userthreadstart - - ntdll!userthreadstart - - ntdll!userthreadstart - - user32!internalcallwinproc user32!internalcallwinproc - - user32!internalcallwinproc kernel32!loadlibrary - - ntdll!openfile - kernel32!loadlibrary nt!pageread nt!trap - nt!accessfault - nt!pageread Callstack patterns nt!accessfault 412 9

10 Approach AOI Extraction Trace streams 1 23 AOI Ranked cluster Cluster Hits Total Wait time (ms) 1 94, , , , ,536 3,051,307 AOI Extraction Callstacks - ntdll!userthreadstart - - ntdll!userthreadstart - - ntdll!userthreadstart - - user32!internalcallwinproc user32!internalcallwinproc - - user32!internalcallwinproc kernel32!loadlibrary - - ntdll!openfile - kernel32!loadlibrary nt!pageread nt!trap - nt!accessfault - Ranking Pattern clusters ntdll!openfile 259 nt!accessfault nt!pageread Pattern Clustering Sequence Pattern Mining 709 ntdll!openfile 259 nt!pageread Callstack patterns nt!accessfault

11 Approach AOI Extraction Motivation Runtime traces capture both Relevant executions for performance issue E.g., executions relevant to browser-tab creation Irrelevant executions for performance issues E.g., executions of concurrently executed IM Noisy data for mining Huge investigation scope induced 11

12 Approach AOI Extraction Critical Path Scenario start Time Scenario finish UI thread CPU Wait Ready CPU Wait Ready CPU Worker thread 1 Ready CPU Wait Ready CPU Disk I/O Worker thread 2 Ready CPU Critical path CPU(UI) CPU(WT1) CPU(WT2) CPU(WT1) CPU(UI) Disk I/O Ready CPU 12

13 Approach AOI Extraction Wait Graph Scenario start Time Scenario finish UI thread Wait Graph Worker thread 1 CPU Wait Ready CPU Ready CPU Wait Ready CPU Wait Ready CPU Disk I/O Worker thread 2 Execution irrelevant to Worker thread 1 s waits CPU Critical path 13

14 Approach Callstack Pattern Mining Trace streams 1 23 AOI Ranked cluster Cluster Hits Total Wait time (ms) 1 94, , , , ,536 3,051,307 AOI Extraction Callstacks - ntdll!userthreadstart - - ntdll!userthreadstart - - ntdll!userthreadstart - - user32!internalcallwinproc user32!internalcallwinproc - - user32!internalcallwinproc kernel32!loadlibrary - - ntdll!openfile - kernel32!loadlibrary nt!pageread nt!trap - nt!accessfault - Ranking Pattern clusters ntdll!openfile 259 nt!accessfault nt!pageread Pattern Clustering Sequence Pattern Mining 709 ntdll!openfile 259 nt!pageread Callstack patterns nt!accessfault

15 Approach Callstack Pattern Mining Frequent Costly Sequence Pattern Example Sequence Occurrence Wait time A B 4 ms A B C D F 1 ms A B C E F 3 ms Frequency Cost threshold threshold T T A B G F 2 ms 5 5 ms (or (or 50% 50% of of total) Frequent Costly sequence pattern miner All Patterns Support A, B, AB 10 F, AF, BF, ABF 6 Closed Patterns Support A, B, AB 10 F, AF, BF, ABF 6 Maximal Patterns Support A, B, AB 10 F, AF, BF, ABF 6 Non-consecutive frequent costly sub-sequence as callstack as pattern Frequent Costly maximal patterns are are compact

16 Approach Callstack Pattern Clustering Trace streams 1 23 AOI Ranked cluster Cluster Hits Total Wait time (ms) 1 94, , , , ,536 3,051,307 AOI Extraction Callstacks - ntdll!userthreadstart - - ntdll!userthreadstart - - ntdll!userthreadstart - - user32!internalcallwinproc user32!internalcallwinproc - - user32!internalcallwinproc kernel32!loadlibrary - - ntdll!openfile - kernel32!loadlibrary nt!pageread nt!trap - nt!accessfault - Ranking Pattern clusters ntdll!openfile 259 nt!accessfault nt!pageread Pattern Clustering Sequence Pattern Mining 709 ntdll!openfile 259 nt!pageread Callstack patterns nt!accessfault

17 Approach Callstack Pattern Clustering Motivation Same issue often reflected by variant patterns Defect often hidden in invariant parts of variant patterns Goal Precise measurement of issue impact for better prioritization Comprehensive issue representation with pattern variations for quick and precise fixing 17

18 0. Alignment based on edit distance model App_main InitComponents GetHashCode GetShortPathName SwapKernelStack MmAccessFault MiIssueHardFault IoPageRead Approach Callstack Pattern Clustering Similarity Model Match Insertion/ Deletion Substitution Match App_main InitComponents GetHashCode GetShortPathName wow64service wow64system wow64queryattr ExpandKernelStack MmAccessFault MiIssueHardFault IoPageRead 1. Common-purpose function: weight Uni() is small 2. Special-purpose function: weight Uni() is large 3. Variant part representing nonessential factors 4. Similar names implying relevant functionalities 5. Constant call path: weight FBi() + BBi() is small 18

19 Technical Highlights Machine learning for system domain Formulate the discovery of problematic execution patterns as callstack mining & clustering Systematic mechanism to incorporate domain knowledge Interactive performance analysis system Parallel mining infrastructure based on HPC + MPI Visualization aided interactive exploration 19

20 Evaluation Windows 7 Study Task: since Dec 2010, a continued effort to improve Windows performance Analysts from one performance analysis team for Microsoft Windows Hunt for hidden performance bugs that caused common impact on Windows Explorer UI response Based on over 6,000 trace streams Data 921 qualified out of 1,000 randomly sampled trace streams 181 million callstacks in total 140 million wait callstacks 41 million CPU sampled callstacks 20

21 Evaluation Windows 7 Study Research Questions RQ1. How much does StackMine improve practices of performance debugging in the large? RQ2. How well do the derived performance signatures capture performance bottlenecks? RQ3. How much does StackMine outperform alternative techniques? 21

22 Evaluation Windows 7 Study RQ1. Overall Improvement of Practices Traditional approach would take 20~60 days Using StackMine 18 hours 10 hours of automatic computation 140 million callstacks 12 highly impactful bugs AOI Extraction 689 thousand callstacks Feature team confirmation Callstack Pattern Mining 93 performance signatures 2,239 costly patterns Human analyst confirmation Pattern Clustering 1,251 pattern clusters Pattern Ranking Top 400 pattern clusters 8 hours of one human analyst s review 22

23 Evaluation Windows 7 Study RQ2. Performance Bottleneck Coverage Performance Bottleneck Coverage (PBC) of a set of performance signatures PBC = Total time of performance signatures Total time of collected trace streams The higher PBC achieved, the lower possibility that high-impact performance bugs remain not captured 58.26% PBC achieved by the 93 signatures 23

24 Number of required trace streams to investigate Evaluation Windows 7 Study RQ3. Comparison with Alternative Techniques StackMine requires only 7.2%, 5.8%, and 6.3% of trace streams required by the other three techniques Baseline-Random Greedy-Total Greedy-Max StackMine % 30% 40% 50% 60% Performance bottleneck coverage (%) 24

25 Impact We believe that the MSRA tool is highly valuable and much more efficient for mass trace (100+ traces) analysis. For 1000 traces, we believe the tool saves us 4-6 weeks of time to create new signatures, which is quite a significant productivity boost. Highly effective new issue discovery on Windows mini-hang - from Development Manager in Windows Continuous impact on future Windows versions 25

26 Conclusion The first formulation and real-world deployment of performance debugging in the large as a data mining problem on callstacks A mining-clustering mechanism for reducing costlypattern mining results based on domain-specific characteristics of callstacks Industrial impact on using StackMine in performance debugging in the large for Microsoft Windows 26

27 Acknowledgment Our partners in Microsoft product teams The researchers from Microsoft Research 27

28 Q&A Thank you! 28