Combining the Logical and the Probabilistic in Program Analysis. Xin Zhang Xujie Si Mayur Naik University of Pennsylvania

Transcription

1 Combining the Logical and the Probabilistic in Program Analysis Xin Zhang Xujie Si Mayur Naik University of Pennsylvania

2 What is Program Analysis? int f(int i) {... } Program Analysis x may be null!... x.f = y; assert(n > 10);... Pre: i > 0 Post: ret=sqrt(i) 2

3 Conventional Logical Approach + Easy to specify Explainable With formal guarantees path(a, a). path(a, c) :- path(a, b), edge(b, c). Program Analysis 1. Each node connects to itself. 2. If there is a path from a to b, and there is an edge from b to c, then there is a path from a to c. 3

4 Conventional Logical Approach + Easy to specify + Explainable path(a, a). path(a, c) :- path(a, b), edge(b, c). Program Analysis path(1,1) edge(1,7) edge(7,2,a 0 ) path(1,7) edge(7,5) edge(2,8,c 0 ) path(1,2) path(1,5) edge(5,8) edge(8,3) path(1,8) edge(8,6) int f(int i) {... x.f = y;... } x may be null when i = 5. path(1,3) path(1,6) 4

5 Conventional Logical Approach + Easy to specify + Explainable + With formal guarantees path(a, a). path(a, c) :- path(a, b), edge(b, c). Program Analysis int f(int i){... assert(n > 10);... } i 5

6 Conventional Logical Approach + Easy to specify + Explainable + With formal guarantees path(a, a). path(a, c) :- path(a, b), edge(b, c). Program Analysis - Unable to handle uncertainty /* Read type and payload length first */ hbtype = *p++; n2s(p, payload); pl = p;... /* Enter response type, length and copy payload */ *bp++ = TLS1_HB_RESPONSE; s2n(payload, bp); memcpy(bp, pl, payload); Will this buffer overflow lead to a security exploit? CVE (Heartbleed) 6

7 Conventional Logical Approach + Easy to specify + Explainable + With formal guarantees path(a, a). path(a, c) :- path(a, b), edge(b, c). Program Analysis - Unable to handle uncertainty - Unable to adapt Bug Detection Verification Pointer Analysis 7

8 Conventional Logical Approach + Easy to specify + Explainable + With formal guarantees path(a, a). path(a, c) :- path(a, b), edge(b, c). Program Analysis - Unable to handle uncertainty - Unable to adapt - Unable to learn 8

9 Emerging Probabilistic Approach - Hard to specify - Hard to interpret - Without formal guarantees guarantees ML System + Able to handle uncertainty + Able to adapt + Able to learn 9

10 An Overarching Question + Easy to specify + Explainable + With formal guarantees? Program Analysis + Able to handle uncertainty + Able to adapt + Able to learn 10

11 Talk Outline } Motivation } A General Approach } Instance Applications } Solver } Empirical Results 11

12 Our Approach: Mixed Hard and Soft Constraints + Easy to specify + Explainable + With formal guarantees path(a, a). path(a, c) :- path(a, b), edge(b, c) 2.0 Program Analysis + Able to handle uncertainty + Able to adapt + Able to learn 12

13 Background: Mixed Hard and Soft Constraints 5 p(p3, p2) :- p(p1, p2), n(p3, p1) a5 Stochastic Logic Programs (SLP) [Muggleton, 1996] Probabilistic Relational Models (PRM) [Koller, 1999] Bayesian Logic (BLOG) [Milch et al., 2005] Logic Probability Markov Logic Network (MLN) [Richardson & Domingos, 2006] Encodes Handles Uncertainty Formal Semantics Probabilistic Soft Logic (PSL) [Kimmig et al., 2012] 13

14 Background: Mixed Hard and Soft Constraints Input relations: edge(a, b) Output relations: path(a, b) Hard Rules: path(a, a). path(a, c) :- path(a, b), edge(b, c). Soft Rules: path(a, b). weight 5 Solution: path 1, 1 = T, path 2, 2 = T, path 1, 2 = T, path 2, 1 = F, Grounding Solving Input tuples: edge(1, 7), edge(4, 7), MaxSAT Hard constraints: edge 1, 7 edge(4, 7)... path(1, 1) path(2, 2)... (path(1, 1) path(1, 1) edge(1, 1)) (path(1, 2) path(1, 1) edge(1, 2)) (path(1, 2) path(1, 2) edge(2, 2)) Soft constraints: ( path 1, 1 weight 5) path 1, 2 weight 5 14

15 Our Overall Methodology Offline Online Learning Markov Logic Network Inference 15

16 Talk Outline } Motivation } A General Approach } Instance Applications } Solver } Empirical Results 16

17 Static Bug Detection: Prevalent Approach Detected Races R1: Race on field org.apache.ftpserver.requesthandler.request org.apache.ftpserver.requesthandler: 9 org.apache.ftpserver.requesthandler: 18 R2: Race on field org.apache.ftpserver.requesthandler.request org.apache.ftpserver.requesthandler: 17 org.apache.ftpserver.requesthandler: 18 R3: Race on field org.apache.ftpserver.requesthandler.writer org.apache.ftpserver.requesthandler: 19 org.apache.ftpserver.requesthandler: 20 R4: Race on field org.apache.ftpserver.requesthandler.reader org.apache.ftpserver.requesthandler: 21 org.apache.ftpserver.requesthandler: 22 R5: Race on field org.apache.ftpserver.requesthandler.controlsocket org.apache.ftpserver.requesthandler: 23 org.apache.ftpserver.requesthandler: 24 Eliminated Races E1: Race on field org.apache.ftpserver.requesthandler. isclosed org.apache.ftpserver.requesthandler: 13 org.apache.ftpserver.requesthandler: 15 17

18 Static Bug Detection: Our Approach Detected Races R1: Race on field org.apache.ftpserver.requesthandler.request org.apache.ftpserver.requesthandler: 9 org.apache.ftpserver.requesthandler: 18 R2: Race on field org.apache.ftpserver.requesthandler.request org.apache.ftpserver.requesthandler: 17 org.apache.ftpserver.requesthandler: 18 R3: Race on field org.apache.ftpserver.requesthandler.writer org.apache.ftpserver.requesthandler: 19 org.apache.ftpserver.requesthandler: 20 R4: Race on field org.apache.ftpserver.requesthandler.reader org.apache.ftpserver.requesthandler: 21 org.apache.ftpserver.requesthandler: 22 R5: Race on field org.apache.ftpserver.requesthandler.controlsocket org.apache.ftpserver.requesthandler: 23 org.apache.ftpserver.requesthandler: 24 Eliminated Races E1: Race on field org.apache.ftpserver.requesthandler. isclosed org.apache.ftpserver.requesthandler: 13 org.apache.ftpserver.requesthandler: 15 18

19 Static Bug Detection: Our Approach Detected Races R1: Race on field org.apache.ftpserver.requesthandler.request org.apache.ftpserver.requesthandler: 9 org.apache.ftpserver.requesthandler: 18 Eliminated Races E1: Race on field org.apache.ftpserver.requesthandler. isclosed org.apache.ftpserver.requesthandler: 13 org.apache.ftpserver.requesthandler: 15 E2: Race on field org.apache.ftpserver.requesthandler.request org.apache.ftpserver.requesthandler: 17 org.apache.ftpserver.requesthandler: 18 E3: Race on field org.apache.ftpserver.requesthandler.writer org.apache.ftpserver.requesthandler: 19 org.apache.ftpserver.requesthandler: 20 E4: Race on field org.apache.ftpserver.requesthandler.reader org.apache.ftpserver.requesthandler: 21 org.apache.ftpserver.requesthandler: 22 E5: Race on field org.apache.ftpserver.requesthandler.controlsocket org.apache.ftpserver.requesthandler: 23 org.apache.ftpserver.requesthandler: 24 19

20 Instance Applications [CAV 17 Invited Tutorial] Static Bug Detection [FSE 15] Hard Rules analysis rules Soft Rules analysis rule i weight w i confidence of writer result j weight w j confidence of user Automated Verification [PLDI 14, POPL 16] analysis rules abstraction 1 abstraction n result i weight w i query resolution award abstraction j weight w j abstraction cost Interactive Verification [OOPSLA 17] analysis rules cause i weight w i cost of inspection result j weight w j reward of resolution 20

21 An Example: Datarace Analysis Input relations: next(p1, p2), mayalias(p1, p2), guarded(p1, p2) Output relations: program point p1 parallel(p1, p2), race(p1, p2) If is & p1 p2 & may p2 may p1 happen & p2 are in parallel, If p1 & p2 may happen in parallel, immediate successor access the same guarded by the andif they p2 & may p1 and access may p3 happen is successor the same in parallel, of p1, memory location, of p2. memory location. same lock. Rules: then and they p1 then & are p2 p3 not may & p2 guarded happen may happen by in parallel. in parallel. the same lock, p1 & p2 may parallel(p3, p2) :- parallel(p1, p1 then & p1 p2 p2), & may p2 happen in parallel. next may (p3, have p1). a datarace. have a datarace. (2) parallel(p1, p2) :- parallel(p2, p1). race(p1, p2) :- parallel(p1, p2), mayalias(p1, p2), guarded(p1, p2). 21

22 An Example: Datarace Analysis Input relations: next(p1, p2), mayalias(p1, p2), guarded(p1, p2) Output relations: parallel(p1, p2), race(p1, p2) Rules: parallel(p3, p2) :- parallel(p1, p2), next (p3, p1). (2) parallel(p1, p2) :- parallel(p2, p1). race(p1, p2) :- parallel(p1, p2), mayalias(p1, p2), guarded(p1, p2). race(x2, x1). weight 25 a = 1; if (a > 2) { // p1... // p2 } Hard Rule Soft Rule weight 5 22

23 An Example: Datarace Analysis 1 public class RequestHandler { 2 Request request; 3 FtpWriter writer; 4 BufferedReader reader; 5 Socket controlsocket; 6 boolean isconnectionclosed; 7 8 public Request getrequest() { 9 return request; 10 } 11 public void close() { 12 synchronized (this) { 13 if (isclosed) 14 return; 15 isclosed = true; 16 } 17 request.clear(); 18 request = null; 19 writer.close(); 20 writer = null; 21 reader.close(); 22 reader = null; 23 controlsocket.close(); 24 controlsocket = null; 25 } Source code snippet from Apache FTP Server 23

24 An Example: Datarace Analysis 1 public class RequestHandler { 2 Request request; 3 FtpWriter writer; 4 BufferedReader reader; 5 Socket controlsocket; 6 boolean isconnectionclosed; 7 8 public Request getrequest() { 9 return request; 10 } R1 11 public void close() { 12 synchronized (this) { 13 if (isclosed) 14 return; 15 isclosed = true; 16 } 17 request.clear(); 18 request = null; 19 writer.close(); 20 writer = null; 21 reader.close(); 22 reader = null; 23 controlsocket.close(); 24 controlsocket = null; 25 } Source code snippet from Apache FTP Server 24

25 An Example: Datarace Analysis 1 public class RequestHandler { 2 Request request; 3 FtpWriter writer; 4 BufferedReader reader; 5 Socket controlsocket; 6 boolean isconnectionclosed; 7 8 public Request getrequest() { 9 return request; 10 } 11 public void close() { 12 synchronized (this) { 13 if (isclosed) 14 return; 15 isclosed = true; 16 } 17 request.clear(); 18 request = null; 19 writer.close(); 20 writer = null; 21 reader.close(); 22 reader = null; 23 controlsocket.close(); 24 controlsocket = null; 25 } R2 R3 R4 R5 Source code snippet from Apache FTP Server 25

26 How Does Generalization Work? 17 request.clear();// x1 next(x2,x1) parallel(x1,x1) 18 request = null; // x2 19 writer.close(); // y1 guarded(x2, x1) mayaias(x2,x1) parallel(x2,x1) 20 writer = null; // y2 race(x2,x1) parallel(x1,x2) next(x2,x1) parallel(p3, p2) :- parallel(p1, p2), next (p3, p1). weight 5 race(x2, x1) weight 25 next(y1,x2) parallel(y1,x2) parallel(x2,x2) parallel(p1, p2) :- parallel(p2, p1). race(p1, p2) :- parallel(p1, p2), mayalias(p1, p2), guarded(p1, p2). next(y1,x2) parallel(x2,y1) parallel(y1,y1) next(y2,y1) mayaias(y2,y1) parallel(y2,y1) guarded(y2, y1) race(y2,y1) 26

27 How Does Generalization Work? 17 request.clear();// x1 next(x2,x1) parallel(x1,x1) 18 request = null; // x2 19 writer.close(); // y1 guarded(x2, x1) mayaias(x2,x1) parallel(x2,x1) 20 writer = null; // y2 race(x2,x1) parallel(x1,x2) next(x2,x1) parallel(p3, p2) :- parallel(p1, p2), next (p3, p1). weight 5 race(x2, x1) weight 25 next(y1,x2) parallel(y1,x2) parallel(x2,x2) parallel(p1, p2) :- parallel(p2, p1). race(p1, p2) :- parallel(p1, p2), mayalias(p1, p2), guarded(p1, p2). next(y1,x2) parallel(x2,y1) parallel(y1,y1) next(y2,y1) mayaias(y2,y1) parallel(y2,y1) guarded(y2, y1) race(y2,y1) 27

28 How Does Generalization Work? 17 request.clear();// x1 next(x2,x1) parallel(x1,x1) 18 request = null; // x2 19 writer.close(); // y1 guarded(x2, x1) mayaias(x2,x1) parallel(x2,x1) 20 writer = null; // y2 race(x2,x1) parallel(x1,x2) next(x2,x1) parallel(p3, p2) :- parallel(p1, p2), next (p3, p1). weight 5 race(x2, x1) weight 25 next(y1,x2) parallel(y1,x2) parallel(x2,x2) parallel(p1, p2) :- parallel(p2, p1). race(p1, p2) :- parallel(p1, p2), mayalias(p1, p2), guarded(p1, p2). next(y1,x2) parallel(x2,y1) parallel(y1,y1) next(y2,y1) mayaias(y2,y1) parallel(y2,y1) guarded(y2, y1) race(y2,y1) 28

29 How Does Generalization Work? 17 request.clear();// x1 next(x2,x1) parallel(x1,x1) 18 request = null; // x2 19 writer.close(); // y1 guarded(x2, x1) mayaias(x2,x1) parallel(x2,x1) 20 writer = null; // y2 race(x2,x1) parallel(x1,x2) next(x2,x1) parallel(p3, p2) :- parallel(p1, p2), next (p3, p1). weight 5 race(x2, x1) weight 25 next(y1,x2) parallel(y1,x2) parallel(x2,x2) parallel(p1, p2) :- parallel(p2, p1). race(p1, p2) :- parallel(p1, p2), mayalias(p1, p2), guarded(p1, p2). next(y1,x2) parallel(x2,y1) parallel(y1,y1) next(y2,y1) mayaias(y2,y1) parallel(y2,y1) guarded(y2, y1) race(y2,y1) 29

30 Talk Outline } Motivation } A General Approach } Instance Applications } Solver } Empirical Results 30

31 The Inference Problem Input relations: edge(a, b) Output relations: path(a, b) Hard Rules: path(a, a). path(a, c) :- path(a, b), edge(b, c). Soft Rules: path(a, b). weight 5 Existing Solvers Tuffy [VLDB 11] Alchemy [ML 06] CPI [UAI 08] RockIt [AAAI 13] Z3 [TACAS 08] Markov Logic Network Solver Soundness Optimality Scalability 31

32 The Nichrome Solver MLN instance work set candidate solution MaxSAT Solver Yes Checker No sound & optimal solution expanded work set 32

33 The Nichrome Solver MLN instance work set candidate solution MaxSAT Solver Yes Checker No Datalog SQL MaxSAT ILP SMT sound & optimal solution expanded work set Volt: A Lazy Grounding Framework for Solving Very Large MaxSAT Instances [SAT 2015] 33

34 The Nichrome Solver MLN instance work set candidate solution MaxSAT Solver Yes Checker No Datalog SQL MaxSAT ILP SMT sound & optimal solution expanded work set On Abstraction Refinement for Program Analyses in Datalog [PLDI 2014] 34

35 The Nichrome Solver MLN instance work set candidate solution MaxSAT Solver Yes Checker No Datalog SQL MaxSAT ILP SMT sound & optimal solution expanded work set Scaling Relational Inference Using Proofs and Refutations [AAAI 2016] 35

36 The Nichrome Solver MaxSAT instance work set Program Reasoning: Does head alias tail on line 50 in Complex.java? candidate solution MaxSAT Solver expanded work set Yes Checker No Query-Guided Maximum Satisfiability [POPL 2016] Datalog SQL MaxSAT ILP SMT sound & optimal solution Information Retrieval: Is Dijkstra most likely author of Structured Programming? 36

37 The Nichrome Solver MaxSAT instance work set all clauses involving query variables candidate solution MaxSAT Solver Yes Checker No Overestimates the effect of unexplored clauses max(work set summary set) - max(work set) = 0 sound & optimal solution expanded work set Query-Guided Maximum Satisfiability [POPL 2016] 37

38 Talk Outline } Motivation } A General Approach } Instance Applications } Solver } Empirical Results 38

39 Experimental Setup } Control Study: } Analyses: (1) Datarace analysis, (2) Pointer analysis } Benchmarks: 7 Java programs ( KLOC each) } Feedback: Automated [Zhang et.al, PLDI 14] } User Study: } Analyses: Information flow analysis } Benchmarks: 3 Android Apps } Feedback: 9 users 30 rules 18 input relations 18 output relations 76 rules 52 input relations 42 output relations 76 rules 50 input relations 42 output relations 39

40 Benchmarks Characteristics classes methods bytecode(kb) KLOC antlr K avrora 1, K ftp K hedc K luindex K lusearch K weblech K App App App Control Study User Study 40

41 Precision Results: Datarace Analysis 100% feedback on 5% reports % With 60% only up to 5% feedback, 70% of the false positives 40% are eliminated and 96% of true positives retained. 20% 0% avrora ftp hedc luindex lusearch weblech false reports eliminated true reports retained 41

42 Precision Results: Varying Amount of Feedback 100% 338 false and 700 true reports 80% 60% 40% 20% 0% 5% 10% 15% 20% false reports eliminated true reports retained 42

43 Solver Performance Results running Winner time of the peak 2015 memory Annual (seconds) MaxSAT Competition (MB) # clauses (M = million) Nichrome Eva500 Nichrome Eva500 Nichrome Eva500 ftp ,262 29K 3M hedc , K 4.8M weblech 4 timeout 363 timeout 922K 8.4M antlr 190 timeout 1,405 timeout 3.3M 13M avrora 178 timeout 1,095 timeout 2.6M 16.3M chart 253 timeout 721 timeout 1.8M 22.3M luindex 169 timeout 944 timeout 2.2M 11.9M 43

44 Solver Performance Results running time (seconds) 44 peak memory (MB) # clauses (M = million) Nichrome Eva500 Nichrome Eva500 Nichrome Eva500 ftp ,262 29K 3M hedc , K 4.8M weblech 4 timeout 363 timeout 922K 8.4M antlr 190 timeout 1,405 timeout 3.3M 13M avrora 178 timeout 1,095 timeout 2.6M 16.3M chart 253 timeout 721 timeout 1.8M 22.3M luindex 169 timeout 944 timeout 2.2M 11.9M Advisor Recommendation AR 4 timeout 4 timeout 2K 7.9M ER Entity 13Resolution K 4.8M IE 2 2, K 0.9M Information Extraction

45 Conclusions } Combining logical and probabilistic reasoning in program analysis provides best of both worlds } Our approach: extend conventional program analyses by augmenting logical rules with weights => Adopt semantics of MLN from the AI community } New solver that achieves accuracy and scalability by leveraging program analysis domain insights 45