Statistical Debugging for Real-World Performance Problems. Linhai Song Advisor: Prof. Shan Lu

Transcription

1 Statistical Debugging for Real-World Performance Problems Linhai Song Advisor: Prof. Shan Lu 1

2 Software Efficiency is Critical No one wants slow and inefficient software Frustrate end users Cause economic loss Software efficiency is increasingly important Hardware is not getting faster (per-core) Software is getting more complex Energy saving is getting more urgent Still Not Finished?

3 Performance Bugs Implementation mistakes causing inefficiency An example rows=0 causing no cache allocated MySQL Bug DB void ha_partition::start_bulk_insert(int rows) {. - if (!rows) - DBUG_VOID_RETURN; - rows= rows/m_tot_parts + 1; + rows= rows? rows/m_tot_parts + 1 : 0;. // fast path using caches } MySQL Bug X Slower

4 How to Diagnose Performance Bugs Difficult to avoid Lack performance documentation for APIs Workloads are quickly changing Diagnosis tools are needed The state of the art is preliminary Profilers void ha_partition::start_bulk_insert(int rows) {. - if (!rows) Not in profiling results - DBUG_VOID_RETURN; - rows= rows/m_tot_parts + 1; + rows= rows? rows/m_tot_parts + 1 : 0;. // fast path using caches } MySQL Bug 26527

5 How to Diagnose Functional Bugs The state of the art is mature Has been studied for decades Many successful techniques have been proposed Statistical debugging Input: Program: Symptom: Bad Good int i = 0; int j = 10; int k = fopen( ); if (p==null) printf( %s\n, p->str); failure Predicates B: p==null R: k > 0 S: i<j Predicates B: p!=null R: k > 0 S: i<j Statistical Model Rank Predicates Score 1 B:p==NULL

6 What Can We Learn? How about statistical debugging Input: Program: Symptom: Q1: How to identify failure runs? Q2: How to obtain inputs? Q3: How to design predicates? Q2? Bad int i = 0; int j = 10; int k = fopen( ); if (p==null) printf( %s\n, p->str); failure Good Q1? Predicates B: p==null R: k > 0 S: i<j Predicates B: p!=null R: k > 0 S: i<j Q3? Statistical Model Rank Predicates Score 1 B:p==NULL

7 Contributions Diagnosis process for performance bugs Performance problems are noticed by comparison Inputs are provided during reporting Statistical in-house performance diagnosis 3 popular predicates 2 widely used statistical models Statistical on-line performance diagnosis Same diagnosis capability with <10% overhead Not sacrifice diagnosis latency

8 Outline Overview Diagnosis process study In-house diagnosis study On-line diagnosis study Conclusion

9 Outline Overview Diagnosis process study In-house diagnosis study On-line diagnosis study Conclusion

10 Outline Overview Diagnosis process study In-house diagnosis study On-line diagnosis study Conclusion

11 Methodology Application and Bug Source App. Apache Software Type Command-line Utility + Server + Library Language C/Java MLOC 0.45 Bug DB History 13 y Tags N/A # Bugs 25 # Bug User Perceived 16 Chrome GUI Application C/C y N/A 10 5 GCC Compiler C/C y Compiletime-hog 11 9 Mozilla GUI Application C++/JS y perf MySQL Server Software C/C++/C# y S Total:

12 Q1: How to identify failure runs? How about statistical debugging Q1: How to identify failure runs? Input: Program: Bad Good int i = 0; int j = 10; int k = fopen( ); if (p==null) printf( %s\n, p->str); Symptom: failure Q1? Predicates B: p==null R: k > 0 S: i<j Predicates B: p!=null R: k > 0 S: i<j Statistical Model Rank Predicates Score 1 B:p==NULL

13 How Perf. Bugs are Observed Dominating MySQL within one code base cross multiple code bases Not using comparison Mozilla GCC Chrome Apache

14 How Perf. Bugs are Observed the same input with different configuration inputs with different sizes inputs with slightly different functionality MySQL within one code base cross multiple code bases Not using comparison Mozilla GCC Chrome Apache

15 How Perf. Bugs are Observed same applications different versions different applications MySQL within one code base cross multiple code bases Not using comparison Mozilla GCC Chrome Apache

16 How Perf. Bugs are Observed MySQL within one code base cross multiple code bases Not using comparison Mozilla GCC Chrome Apache

17 Q2: How to obtain inputs? How about statistical debugging Q1: How to identify failure runs? Q2: How to obtain inputs? Input: Program: Symptom: Q2? Bad int i = 0; int j = 10; int k = fopen( ); if (p==null) printf( %s\n, p->str); failure Good Predicates B: p==null R: k > 0 S: i<j Predicates B: p!=null R: k > 0 S: i<j Statistical Model Rank Predicates Score 1 B:p==NULL

18 Bad Inputs Provided in Bug Reports Cover all bugs /? n/? MySQL Mozilla GCC Chrome Apache

19 Good Inputs Provided in Bug Reports Good inputs provided ?/0?/1?/n MySQL Mozilla GCC Chrome Apache

20 Implications Performance bugs are observed differently Noticed through comparison Easy to tell successful runs from failure runs Case 1: through comparison Case 2: symptom is dramatic Statistical debugging is a natural fit

21 Outline Overview Diagnosis process study In-house diagnosis study On-line diagnosis study Conclusion

22 Outline Overview Diagnosis process study In-house diagnosis study On-line diagnosis study Conclusion

23 Design In-house diagnosis Predicate design Branch if (p) else. Predicate

24 Design In-house diagnosis Predicate design Branch Return if (p) else. n=fprintf( ); Predicate

25 Design In-house diagnosis Predicate design Branch Return Scalar-pair if (p) else. n=fprintf( ); int i, j, k; i = ; Predicate

26 Design In-house diagnosis Predicate design Branch Return Scalar-pair if (p) else. Statistical model design n=fprintf( ); int i, j, k; i = ; Predicate Model

27 Design In-house diagnosis Predicate design Branch Return Scalar-pair Statistical model design Basic model if (p) else. n=fprintf( ); int i, j, k; i = ; Predicate Model

28 Design In-house diagnosis Predicate design Branch Return Scalar-pair Statistical model design Basic model Delta-LDA if (p) else. n=fprintf( ); int i, j, k; i = ; Predicate Model

29 Experimental Methodology Benchmark selection 8 C bugs, 8 C++ bugs and 4 Java bugs Input design and other setting 10 failure and 10 successful runs Techniques under comparison CBI for C programs Pin for C++ programs Compared with profiling results from OProfile

30 Experimental Results Candidate Predicates Basic Model ΔLDA Profiler BugID Branch Return S-pair Branch Return S-pair Branch Mozilla / 1 - / - - Mozilla / 1 - / - - Mozilla Mozilla MySQL / - - / 1 1 MySQL / 1 - / - - MySQL / - - / 1 1 MySQL / 1 - / - - MySQL / 1 - / - - MySQL / 1 - / - - Apache Apache

31 Experimental Results Candidate Predicates Basic Model ΔLDA Profiler BugID Branch Return S-pair Branch Return S-pair Branch Mozilla / 1 - / - - Mozilla / 1 - / - - Mozilla Mozilla MySQL / - - / 1 1 MySQL / 1 - / - - MySQL / - - / 1 1 MySQL / 1 - / - - MySQL / 1 - / - - MySQL / 1 - / - - Apache Apache

32 Experimental Results Candidate Predicates Basic Model ΔLDA Profiler BugID Branch Return S-pair Branch Return S-pair Branch Mozilla / 1 - / - - Mozilla / 1 - / - - Mozilla Mozilla MySQL / - - / 1 1 MySQL / 1 - / - - MySQL / - - / 1 1 MySQL / 1 - / - - MySQL / 1 - / - - MySQL / 1 - / - - Apache Apache

33 Experimental Results Candidate Predicates Basic Model ΔLDA Profiler BugID Branch Return S-pair Branch Return S-pair Branch Mozilla / 1 - / - - Mozilla / 1 - / - - Mozilla Mozilla MySQL / - - / 1 1 MySQL / 1 - / - - MySQL / - - / 1 1 MySQL / 1 - / - - MySQL / 1 - / - - MySQL / 1 - / - - Apache Apache

34 Outline Overview Diagnosis process study In-house diagnosis study On-line diagnosis study Conclusion

35 Outline Overview Diagnosis process study In-house diagnosis study On-line diagnosis study Conclusion

36 Experimental Methodology Challenges in on-line diagnosis Diagnosis capability Low overhead Benchmarks and inputs Tool implementation CBI in sampling mode for return predicates LBR for branch predicates Rough sampling rate is 1/100

37 Experimental Results BugID Diagnosis Capability Overhead Requested Failure Runs Mozilla % 1000 Mozilla % 1000 Mozilla % 10 Mozilla % 1000 MySQL % 10 MySQL % 1000 MySQL % 10 MySQL % 1000 MySQL % 1000 MySQL % 1000 Apache % 1000 Apache % 10

38 Experimental Results BugID Diagnosis Capability Overhead Requested Failure Runs Mozilla % 1000 Mozilla % 1000 Mozilla % 10 Mozilla % 1000 MySQL % 10 MySQL % 1000 MySQL % 10 MySQL % 1000 MySQL % 1000 MySQL % 1000 Apache % 1000 Apache % 10

39 Conclusion and Future Works Study diagnosis process for perf. bugs Noticed through comparison Good and bad inputs are provided Study statistical debugging on perf. bugs Branch predicates + two statistical models Future works Analyze inefficient loops Provide detailed fix strategies

40 Thanks a lot! 40