Motivation. Overview. Scalable Dynamic Analysis for Automated Fault Location and Avoidance. Rajiv Gupta. Program Execution

Transcription

1 Motivation Scalable Dynamic Analysis for Automated Fault Location and Avoidance Rajiv Gupta Funded by NSF grants from CPA, CSR, & CRI programs and grants from Microsoft Research Software bugs cost the U.S. economy about $59.5 billion each year [NIST 02]. Embedded Systems Mission Critical / Safety Critical Tasks A failure can lead to Loss of Mission/Life. (Ariane 5) arithmetic overflow led to shutdown of guidance computer. (Mars Climate Orbiter) missed unit conversion led to faulty navigation data. (Mariner I) missing superscripted bar in the specification for the guidance program led to its destruction 29 seconds after launch. (Mars Pathfinder) priority inversion error causing system reset. (Boeing ) loss of engine & flight displays while in flight. (Toyota hybrid Prius) VSC, gasoline-powered engine shut off. (Therac-25) wrong dosage during radiation therapy.. Overview Fault Location Long-running Multi-threaded Program Execution Scalability Tracing + Logging Goal: Assist the programmer in debugging by automatically narrowing the fault to a small section of the code. Fault Dynamic Information Data dependences Control dependences Values Fault Location Dynamic Slicing Offline Fault Avoidance Environment Faults Online Execution Runs One failed execution & Its perturbations

2 Dynamic Information Approach Program Execution Dynamic Dependence Graph Detect execution of statement s such that Faulty code Affects the value computed by s; or Faulty code is Affected-by the value computed by s Data Control through a chain of dependences. Estimate the set of potentially faulty statements from s: Affects: statements from which s is reachable in the dynamic dependence graph. (Backward ) Affected-by: statements that are reachable from s in the dynamic dependence graph. (Forward ) è Intersect slices to obtain a smaller fault candidate set. Backward & Forward s Backward & Forward s Backward Failure inducing Input Failure Inducing Input Erroneous Output [Korel&Laski,1988] Forward [ASE-05] Erroneous Output [ASE-05] è For memory bugs the number of statements is very small (< 5).

3 Bidirectional s Pruning s Critical Predicate: An execution instance of a predicate such that changing its outcome repairs the program state. Found critical predicates in 12 out of 15 bugs Search for critical predicate: Brute force: 2 predicates to 155K predicates; After Filtering and Ordering: 1 to 7K predicates. Backward of CP + Bidirectional Combined Forward of [ICSE-0] CP û v û Confidence in v C(v): [0,1] 1 - any change in v will change 0 - all values of v produce same How? Value profiles. 1 û 1 1 [PLDI-0] Test Programs Dynamic Sizes Real Reported Bugs Injected Bugs Buggy Runs BS FS BiS flex 2.5.1(a) Nine logical bugs (incorrect ouput) Unix utilities grep 2.5, grep 2.5.1, flex 2.5.1, make.80. Six memory bugs (program crashes) Unix utilities gzip, ncompress, polymorph, tar, bc, tidy. Siemens Suite (numerous versions) schedule, schedule2, replace, print_tokens.. Unix utilities gzip, flex flex 2.5.1(b) NA flex 2.5.1(c) NA grep 2.5 NA grep 2.5.1(a) NA grep 2.5.1(b) NA 599 NA grep 2.5.1(c) NA make.80(a) make.80(b) gzip ncompress polymorph tar bc tidy

4 Combined s Evaluation of Pruning Buggy Runs BS BS^FS^BiS (%BS) flex 2.5.1(a) (.9%) flex 2.5.1(b) (7.5%) flex 2.5.1(c) 50 5 (10%) grep 2.5 NA 8 (7.4%*EXEC) grep 2.5.1(a) NA 25 (4.9%*EXEC) grep 2.5.1(b) NA 599 (5.%*EXEC) grep 2.5.1(c) NA 12 (0.9%*EXEC) make.80(a) (81.4%) make.80(b) (75.%) gzip (8.8%) ncompress (14.%) polymorph (14.%) tar (42.9%) bc (50%) tidy (29.1%) Siemen s Suite Program Description LOC Versions Tests print_tokens Lexical analyzer print_tokens2 Lexical analyzer replace Pattern replacement schedule Priority scheduler schedule2 Priority scheduler gzip Unix utility flex Unix utility Single error is injected in each version. All the versions are not included: No output or the very first output is wrong; Root cause is not contained in the BS (code missing error). Evaluation of Pruning Effectiveness Program BS Pruned Pruned / BS print_tokens % Backward [AADEBUG-05] 1% of Executed Statements print_tokens % replace % schedule % Erroneous output Failure inducing input Critical predicate Confidence Analysis schedule % gzip % flex % Combined [ASE-05,ICSE-0] % of Backward 11% of Exec. Pruned [PLDI-0] 41% of Backward 1% of Exec.

5 Effectiveness Execution Omission Errors Program-bug Inspected Stmts. mutt heap overflow 8 X= X= pine stack overflow pine heap overflow 10 mc stack overflow 2 squid heap overflow 5 bc heap overflow X= A = A<0 A = A<0 Inspect pruned slice. Dynamically detect an Implicit dependence. Incrementally expand the pruned slice. [PLDI-07] Slicing is effective in locating faults. No more than 10 static statements had to be inspected. =X Implicit dependence =X Scalability of Tracing Trace Sizes & Collection Overheads Dynamic Information Needed Dynamic Dependences for all slicing Values for Confidence Analysis for pruning slices è annotates the static program representation Whole Execution Trace (WET) Trace Size 15 Bytes / Instruction Program Running Time Dep. Trace Collection Time mysql 1 s 21 GB 288 s prozilla 8 s GB 240 s proxyc 10 s 45 MB 880 s mc 10 s 55 GB 418 s mutt 20 s 88 GB 28 s pine 14 s 15 GB 2088 s squid 15 s 88 GB 112 s Trace sizes are very large for even 10s of execution.

6 Compacting Whole Execution Traces Dependence Graph Representation Explicitly remember dynamic control flow trace. Infer as many dynamic dependences as possible from control flow (94%), remember the remaining dependences explicitly ( %). è Specialized graph representation to enable inference. Explicitly remember value trace. Use context-based method to compress dynamic control flow, value, and address trace. è Bidirectional traversal with equal ease [MICRO-04, TACO-05] 1: z=0 2: a=0 : b=2 4: p=&b 5: for i = 1 to N do : if ( i %2 == 0) then 7: p=&a endif endfor 8: a=a+1 9: z=2*(*p) 10: print(z) Input: N=2 1 1 : z=0 2 1 : a=0 1 : b=2 4 1 : p=&b 5 1 : for I=1 to N do 1 : if (i%2==0) then 8 1 : a=a : z=2*(*p) 5 2 : for I=1 to N do 2 : if (i%2==0) then 7 1 : p=&a 8 2 : a=a : z=2*(*p) 10 1 : print(z) Dependence Graph Representation Transform: Traces of Blocks T Input: N=2 1 1 : z=0 2 1 : a=0 1 : b=2 4 1 : p=&b 5 1 : for i = 1 to N do 1 : if ( i %2 == 0) then 8 1 : a=a : z=2*(*p) 5 2 : for i = 1 to N do 2 : if ( i %2 == 0) then 7 1 : p=&a 8 2 : a=a : z=2*(*p) 10 1 : print(z) <2,7> <5,><9,10> 1: z=0 2: a=0 : b=2 4: p=&b 5:for i=1 to N T <,8> F <4,8> <10,11> :if (i%2==0) then T F <5,7><9,12> 7: p=&a <7,12> <11,1> 8: a=a+1 <12,1> <5,8><9,1> 9: z=2*(*p) <1,14> 10: print(z)

7 Infer: Local Dependence Labels Transform: Local Dep. Labels (...,20)... (20,20) (20,20) 10,20,0 (10,10) (20,20) (0,0) (20,21)... =Y 10,20 (10,10) 21 Transform: Local Dep. Labels Group: Non-Local Dep. Edges 10,20 (10,10) (10,11) (20,21) =Y (20,20) (10,11) (20,21) 20 Y = Y = = Y = X 10 Y = (20,11) (20,11) (10,21) Y = (10,21) = Y = X Y = (20,11) Y = = Y = X (10,21) =Y 11,21 11,21 11,21

8 Compacted WET Sizes Slicing Times Program Statements Executed (Millions) WET Size (MB) Before / Before After After 00.twolf 90 10, bzip , vortex 09 8, parser 181.mcf ,70 10, gzip 50 9, li , gcc 5 5, go 85 10, Average 47 9, Bits / Instruction 41. [PLDI-04] vs. [ICSE-0] Dep. Graph Generation Times Reducing Online Overhead Offline post-processing after collecting address and control flow traces è 5x of execution time Online techniques [ICSM 2007] è Information Flow: 9x to18x slowdown è Basic block Opt.: x to10x slowdown è Trace level Opt.: 5.5x to 7.5x slowdown è Dual Core: 1.5x slowdown Online Filtering techniques è Forward slice of all inputs è User-guided bypassing of functions Record non-deterministic events online Less than 2x overhead Deterministic replay of executions Trace faulty executions off-line Replay the execution Switch on tracing Collect and inspect traces Trace analysis is still a problem The traces correspond to huge executions Off-line overhead of trace collection is still significant

9 Reducing Trace Sizes Beyond Tracing Checkpointing Schemes Trace from the most recent checkpoint Checkpoints are of the order of minutes. Better but the trace sizes are still very large. Checkpoint: capture memory image. Execute and Record (log) Events. [ISSTA-07] x Exploiting Program Characteristics Multithreaded and server-like [ISSTA-07, FSE-0] Examples : mysql, apache. Each request spawns a new thread. Do not trace irrelevant threads. Checkpoint log Upon Crash, Rollback to checkpoint. Reduce log and Replay execution using reduced log. Turn on tracing during replay. Reduced log x Trace è Applicable to Multithreaded Programs An Example Example Execution and log file A mysql bug load command will crash the server if database is not specified sql/mysql_load.cc: int mysql_load (THD *thd,...) { 150 if( 151 +strlen(thd->db) + < 152 FN_REFLEN)... } Without typing use database_name, thd->db is Null. Run mysql server User 1 connects to the server User 2 connects to the server User 1: show databases User 2: use test select * from b Time open path=/etc/my.cnf Wait for connection Create Thread 1 Wait for command Create Thread 2 Wait for command Recv show databases Handle command Recv use test; select * from b Handle command Recv load data Blue T0 Red T1 Green T2 Gray - Scheduler User 1: load data into table1 Handle -- (server crashes)

10 Execution Replay using Reduced log Execution Reduction Run mysql server User 1 connects to the server Time open path=/etc/my.cnf Wait for connection Create Thread 1 Effects of Reduction Irrelevant Threads Replay-only vs. Replay & Trace User 2 connects to the server Recv load data Handle -- (server crashes) How? By identifying Inter-thread Dependences Event Dependences - found using the log File Dependences - found using the log Shared-Memory Dependences - found using replay User 1: show databases User 2: show databases select * from b Naïve approach requires thread id of last writer of each address Space and time efficient detection o Memory Regions: Non-shared vs shared o Locality of References to Regions è Space requirement reduced by 4x è Time requirement reduced by 2x User 1: load data into table1 Experimental Results Experimental Results Program-bug Original Optimized Trace Sizes Num. of dependences

11 Experimental Results Debugging System Program-bug Orig. Logging OPT. Static Binary Analyzer Diablo Execution Times (seconds) Checkpoint + log Record Replay Jockey Control Dependence Application binary Slicing Module WET s Reduced Log Execution Engine Valgrind Instrument code Traces Compressed Trace Input Output Fault Avoidance Experiments Large number of faults in server programs are caused by the environment. 5 % of faults in Apache server. Types of Faults Handled Atomicity Violation Faults. Try alternate scheduling decisions. Heap Buffer Overflow Faults. Pad memory requests. Bad User Request Faults. Drop bad requests. Avoidance Strategy Recover first time, Prevent later. Record the change that avoided the fault. Program Type of Bug Env. Change # of Trials mysql-1 Atomicity Violn. Scheduler 1 10 mysql-2 Atomicity Violn. Scheduler 1 5 mysql- Atomicity Violn. Scheduler 1 5 mysql-4 Buffer Overflow. Mem. Padding pine-1 Buffer Overflow. Mem. Padding 1 25 pine-2 Buffer Overflow. Mem. Padding mutt-1 Bad User Req. Drop Req. 205 bc-1 Bad User Req. Drop Req. 290 bc-2 Bad User Req. Drop Req. 195 Time taken (secs.)

12 Summary Long-running Multi-threaded Program Execution Scalability Tracing + Logging Fault Fault Location Dynamic Slicing Offline Fault Avoidance Environment Faults Online