Software Challenges: Abstraction and Analysis

Transcription

1 Software Challenges: Abstraction and Analysis Martin Bravenboer University of Massachusetts Amherst University of Oregon April 2, 2009 University of Waterloo

2 my research 1 the design and implementation of programming language features and software engineering tools for solving software development problems

3 my research 2 extensible and domain-specific languages brief overview program transformation no time : ( program analysis latest work, rest of the talk

4 examples of poor abstraction and language integration

5 example 1: embedding in string literals 3 String username = getparam("username"); String password = getparam("password"); String query = "SELECT id FROM users " + "WHERE name = " + username + " " + "AND password = " + password + " "; if (executequery(query).size() == 0) throw new Exception("bad user/password");

6 example 1: embedding in string literals 3 String username = getparam("username"); String password = getparam("password"); String query = "SELECT id FROM users " + "WHERE name = " + username + " " + "AND password = " + password + " "; if (executequery(query).size() == 0) throw new Exception("bad user/password"); password ilovewaterloo SELECT id FROM users WHERE name = martin AND password = ilovewaterloo

7 example 1: embedding in string literals 3 String username = getparam("username"); String password = getparam("password"); String query = "SELECT id FROM users " + "WHERE name = " + username + " " + "AND password = " + password + " "; if (executequery(query).size() == 0) throw new Exception("bad user/password"); password OR x = x SELECT id FROM users WHERE name = dr.evil AND password = OR x = x

8 example 2: implement a graphical user-interface 4

9 example 2: implement a graphical user-interface 4 north center south

10 example 2: implement a graphical user-interface 4 north center south east

11 example 2: implement a graphical user-interface 5 public class HelloWorld { public static void main(string[] ps) { JTextArea text = new JTextArea(20,40); JPanel panel = new JPanel(new BorderLayout(12,12)); panel.add(borderlayout.north, new JLabel("Hello World")); panel.add(borderlayout.center, new JScrollPane(text)); JPanel south = new JPanel(new BorderLayout(12,12)); JPanel buttons = new JPanel(new GridLayout(1, 2, 12, 12)); buttons.add(new JButton("Ok")); buttons.add(new JButton("Cancel")); south.add(borderlayout.east, buttons); panel.add(borderlayout.south, south);...

12 example 2: implement a graphical user-interface 5 public class HelloWorld { public static void main(string[] ps) { JTextArea text = new JTextArea(20,40); JPanel panel = new JPanel(new BorderLayout(12,12)); panel.add(borderlayout.north, new JLabel("Hello World")); panel.add(borderlayout.center, new JScrollPane(text)); JPanel south = new JPanel(new BorderLayout(12,12)); JPanel buttons = new JPanel(new GridLayout(1, 2, 12, 12)); buttons.add(new JButton("Ok")); buttons.add(new JButton("Cancel")); south.add(borderlayout.east, buttons); panel.add(borderlayout.south, south);...

13 example 2: implement a graphical user-interface 5 public class HelloWorld { public static void main(string[] ps) { JTextArea text = new JTextArea(20,40); JPanel panel = new JPanel(new BorderLayout(12,12)); panel.add(borderlayout.north, new JLabel("Hello World")); panel.add(borderlayout.center, new JScrollPane(text)); JPanel south = new JPanel(new BorderLayout(12,12)); JPanel buttons = new JPanel(new GridLayout(1, 2, 12, 12)); buttons.add(new JButton("Ok")); buttons.add(new JButton("Cancel")); south.add(borderlayout.east, buttons); Does not correspond to panel.add(borderlayout.south, south);... hierarchical structure of the user-interface.

14 example 2: implement a graphical user-interface 5 public class HelloWorld { General-purpose languages do not give public static void main(string[] you the means ps) { to write coherent sentences over the vocabulary of an API JTextArea text = new JTextArea(20,40); JPanel panel = new JPanel(new BorderLayout(12,12)); panel.add(borderlayout.north, new JLabel("Hello World")); panel.add(borderlayout.center, new JScrollPane(text)); JPanel south = new JPanel(new BorderLayout(12,12)); JPanel buttons = new JPanel(new GridLayout(1, 2, 12, 12)); buttons.add(new JButton("Ok")); buttons.add(new JButton("Cancel")); south.add(borderlayout.east, buttons); Does not correspond to panel.add(borderlayout.south, south);... hierarchical structure of the user-interface.

15 example 3: libraries as half-baked languages 6 public void testbuyswhenpriceequalsthreshold() { mainframe.expects(once()).method("buy").with(eq(quantity)).will(returnvalue(ticket)); auditing.expects(once()).method("bought").with(same(ticket)); agent.onpricechange(threshold); }

16 example 3: libraries as half-baked languages 6 public void testbuyswhenpriceequalsthreshold() { mainframe.expects(once()).method("buy").with(eq(quantity)).will(returnvalue(ticket)); auditing.expects(once()).method("bought").with(same(ticket)); } agent.onpricechange(threshold); general-purpose languages are not designed for writing coherent sentences in a domain-specific language

17 example 4: failed effort for specific combinations 7 class Foo { pointcut foo() : call(boolean *.if*()); }

18 example 4: failed effort for specific combinations 7 class Foo { pointcut foo() : call(boolean *.if*()); } $ajc Foo.java Foo.java:2 [error] Syntax error on token "if", invalid allowable token in pointcut or type pattern

19 example 4: failed effort for specific combinations 7 class Foo { pointcut foo() : call(boolean *.if*()); } $ajc Foo.java Foo.java:2 [error] Syntax error on token "if", invalid allowable token in pointcut or type pattern class Foo { pointcut foo() : } call(void *0.Ef());

20 example 4: failed effort for specific combinations 7 class Foo { pointcut foo() : call(boolean *.if*()); } $ajc Foo.java Foo.java:2 [error] Syntax error on token "if", invalid allowable token in pointcut or type pattern class Foo { pointcut foo() : } call(void *0.Ef()); $ajc Foo.java Foo.java:2 [error] Invalid float literal number pointcut foo() : call(void *0.Ef()); ^

21 improved abstraction with embedded domain-specific languages

22 example 1: preventing injection attacks 8 java and sql String q = "SELECT id FROM users " + "WHERE name = " + username + " " + "AND password = " + password + " "; if (executequery(q.tostring()).size() == 0)... SQL q = < SELECT id FROM users WHERE name = ${username} AND password = ${password} >; if (executequery(q.tostring()).size() == 0)...

23 example 1: preventing injection attacks 9 php and shell commands $command = "svn cat \"file name\" -r". $rev; system($command); $command = < svn cat "file name" -r${$rev} >; system($command->tostring()); java and ldap String q = "(cn=" + name + ")"; LDAP q = ( (cn=$(name)) );

24 example 2: implement a graphical user-interface 10

25 example 2: implement a graphical user-interface 10 north center south east

26 example 2: implement a graphical user-interface 11 public static void main(string[] ps) { JPanel panel = panel of border layout { north = label "Hello World" center = scrollpane of textarea { rows = 20 columns = 40 } south = panel of border layout { east = panel of grid layout { row = { button "Ok" button "Cancel" } } } };...

27 example 2: implement a graphical user-interface 11 public static void main(string[] ps) { JPanel panel = panel of border layout { north = label "Hello World" center = scrollpane of textarea { rows = 20 columns = 40 } south = panel of border layout { east = panel of grid layout { row = { button "Ok" button "Cancel" } } } };... Syntax reflects the hierarchical structure of the user-interface The interaction between the domain-specific and generalpurpose code is seamless

28 we need to fundamentally change the way we develop grammars

29 technical challenge: grammar composition 12 java grm sql grm

30 technical challenge: grammar composition 12 java grm sql grm developer java+sql grammar

31 technical challenge: grammar composition 12 java grm sql grm developer java+sql grammar generate java+sql parser

32 technical challenge: grammar composition 12 java grm sql grm developer generate java+sql grammar java+sql parser scanners do not compose complex stateful scanners and/or bugs lr grammars do not compose rewrite grammar to resolve conflicts

33 technical challenge: grammar composition 12 java grm sql grm java grm sql grm developer compose java+sql grammar java+sql grammar generate java+sql parser

34 technical challenge: grammar composition 12 java grm sql grm java grm sql grm developer compose java+sql grammar java+sql grammar generate generate java+sql parser java+sql parser

35 technical challenge: grammar composition 12 java grm sql grm java grm sql grm developer compose java+sql grammar java+sql grammar generate generate java+sql parser java+sql parser scanners do not compose scannerless parser lr grammars do not compose cfg, generalized lr

36 technical challenge: parse table composition 13 java grm sql grm

37 technical challenge: parse table composition 13 java grm sql grm compose java+sql grammar

38 technical challenge: parse table composition 13 java grm sql grm compose java+sql grammar generate java+sql parser

39 technical challenge: parse table composition 13 java grm sql grm java grm sql grm compose generate java+sql grammar java tbl sql tbl generate java+sql parser

40 technical challenge: parse table composition 13 java grm sql grm java grm sql grm compose generate java+sql grammar java tbl sql tbl generate compose java+sql parser java+sql parser

41 technical challenge: parse table composition 13 java grm sql grm java grm sql grm compose generate java+sql grammar java tbl sql tbl generate compose java+sql parser java+sql parser from source-level extensibility to binary extensibility algorithm: minimize DFA reconstruction

42 my research 14 extensible and domain-specific languages program transformation program analysis

43 my research 14 extensible and domain-specific languages embedding domain-specific languages [OOPSLA 2004, GPCE 2005, OOPSLA 2006, OOPSLA 2008, ATEM 2007] >100 citations best paper program transformation program analysis

44 my research 14 extensible and domain-specific languages embedding domain-specific languages [OOPSLA 2004, GPCE 2005, OOPSLA 2006, OOPSLA 2008, ATEM 2007] preventing injection attacks [GPCE 2007, SCP 2009] program transformation program analysis

45 my research 14 extensible and domain-specific languages embedding domain-specific languages [OOPSLA 2004, GPCE 2005, OOPSLA 2006, OOPSLA 2008, ATEM 2007] preventing injection attacks [GPCE 2007, SCP 2009] parse table composition [SLE 2008] program transformation program analysis

46 my research 14 extensible and domain-specific languages embedding domain-specific languages [OOPSLA 2004, GPCE 2005, OOPSLA 2006, OOPSLA 2008, ATEM 2007] preventing injection attacks [GPCE 2007, SCP 2009] parse table composition [SLE 2008] program transformation stratego/xt transformation system [PEPM 2006, FI 2006, SCP 2008] program analysis

47 my research 14 extensible and domain-specific languages embedding domain-specific languages [OOPSLA 2004, GPCE 2005, OOPSLA 2006, OOPSLA 2008, ATEM 2007] preventing injection attacks [GPCE 2007, SCP 2009] parse table composition [SLE 2008] program transformation stratego/xt transformation system [PEPM 2006, FI 2006, SCP 2008] grammar engineering [LDTA 2007] program analysis

48 my research 14 extensible and domain-specific languages embedding domain-specific languages [OOPSLA 2004, GPCE 2005, OOPSLA 2006, OOPSLA 2008, ATEM 2007] preventing injection attacks [GPCE 2007, SCP 2009] parse table composition [SLE 2008] program transformation stratego/xt transformation system [PEPM 2006, FI 2006, SCP 2008] grammar engineering [LDTA 2007] program analysis scalable declarative pointer analysis [ISSTA 2009] latest and greatest work, rest of the talk

49 overview 15 what do we do? fully declarative pointer analysis fast, really fast

50 overview 15 what do we do? fully declarative pointer analysis fast, really fast how do we do it? novel, aggressive optimization exposition of indexes

51 overview 15 what do we do? fully declarative pointer analysis fast, really fast how do we do it? novel, aggressive optimization exposition of indexes why do you care? fast sophisticated, simple different

52 overview 15 what do we do? fully declarative pointer analysis fast, really fast how do we do it? novel, aggressive optimization exposition of indexes why do you care? fast sophisticated, simple different why is it relevant? optimization understanding, find bugs

53 program analysis: run faster 16

54 program analysis: software understanding 17

55 program analysis: find bugs 18

56 pointer analysis 19 what objects can a variable point to? program void foo() { a = new A1(); b = id(a); } void bar() { a = new A2(); b = id(a); } A id(a a) { return a; }

57 pointer analysis 19 ˆ ˆ what objects can a variable point to? program void foo() { a = new A1(); b = id(a); } void bar() { a = new A2(); b = id(a); } A id(a a) { return a; } points-to foo:a bar:a new A1() new A2() objects represented by allocation sites

58 pointer analysis 19 ˆ ˆ what objects can a variable point to? program points-to void foo() { foo:a new A1() a = new A1(); bar:a new A2() b = id(a); } id:a new A1(), new A2() void bar() { a = new A2(); b = id(a); } ˆ A id(a a) { return a; }

59 pointer analysis 19 ˆ ˆ ˆ what objects can a variable point to? program points-to void foo() { foo:a new A1() a = new A1(); bar:a new A2() } b = id(a); id:a new A1(), new A2() foo:b new A1(), new A2() void bar() { bar:b new A1(), new A2() a = new A2(); b = id(a); } A id(a a) { return a; }

60 pointer analysis 19 what objects can a variable point to? program void foo() { a = new A1(); b = id(a); } void bar() { a = new A2(); b = id(a); } A id(a a) { return a; } points-to foo:a bar:a id:a foo:b bar:b new A1() new A2() new A1(), new A2() new A1(), new A2() new A1(), new A2() context-sensitive points-to foo:a bar:a id:a (foo) id:a (bar) foo:b bar:b new A1() new A2() new A1() new A2() new A1() new A2()

61 pointer analysis 19 what objects can a variable point to? program void foo() { a = new A1(); b = id(a); } void bar() { a = new A2(); b = id(a); } A id(a a) { return a; } points-to foo:a bar:a id:a foo:b bar:b new A1() new A2() new A1(), new A2() new A1(), new A2() new A1(), new A2() context-sensitive points-to foo:a bar:a id:a (foo) id:a (bar) foo:b bar:b new A1() new A2() new A1() new A2() new A1() new A2() necessarily an approximation full precision too expensive

62 pointer analysis: a complex domain 20

63 pointer analysis: a complex domain 20 flow-sensitive

64 pointer analysis: a complex domain 20 flow-sensitive field-sensitive

65 pointer analysis: a complex domain 20 flow-sensitive context-sensitive field-sensitive

66 pointer analysis: a complex domain 20 flow-sensitive context-sensitive field-sensitive binary decision diagrams

67 pointer analysis: a complex domain 20 inclusion-based unification-based flow-sensitive on-the-fly call graph context-sensitive k-cfa object sensitive field-based field-sensitive heap cloning binary decision diagrams demand-driven

68 algorithms in a 10-page pointer analysis paper 21

69 algorithms in a 10-page pointer analysis paper 21

70 algorithms in a 10-page pointer analysis paper 21

71 algorithms in a 10-page pointer analysis paper 21

72 algorithms in a 10-page pointer analysis paper 21

73 algorithms in a 10-page pointer analysis paper 21

74 algorithms in a 10-page pointer analysis paper 21

75 algorithms in a 10-page pointer analysis paper 21

76 algorithms in a 10-page pointer analysis paper 21 variation points unclear every variant new algorithm correctness unclear incomparable in precision incomparable in performance

77 from algorithms to specification and back 22 algorithm algorithm algorithm algorithm

78 from algorithms to specification and back 22 algorithm policy algorithm policy algorithm policy algorithm policy analysis

79 from algorithms to specification and back 22 algorithm policy algorithm policy algorithm policy specification algorithm policy analysis

80 from algorithms to specification and back 22 algorithm policy algorithm algorithm algorithm policy algorithm algorithm policy specification algorithm algorithm algorithm algorithm policy algorithm analysis synthesis

81 from algorithms to specification

82 what does it mean to be declarative? 23 denoting high-level programming languages which can be used to solve problems without requiring the programmer to specify an exact procedure to be followed. high-level what, not how no control-flow no side-effects specifications, not programs, not algorithms

83 our pointer analysis framework 24 datalog-based pointer analysis framework for java

84 our pointer analysis framework 24 datalog-based pointer analysis framework for java declarative: what, not how

85 our pointer analysis framework 24 datalog-based pointer analysis framework for java declarative: what, not how easier to express sophisticated analyses correctness more clear clear variation points eases exploration of approximations enables aggressive optimization

86 our pointer analysis framework 24 datalog-based pointer analysis framework for java declarative: what, not how easier to express sophisticated analyses correctness more clear clear variation points eases exploration of approximations enables aggressive optimization sophisticated subset-based analysis, fully on-the-fly call graph discovery, field-sensitivity, contextsensitivity, call-site sensitive, object sensitive, thread sensitive, context-sensitive heap abstraction, type filtering, precise exception analysis

87 our pointer analysis framework 24 datalog-based pointer analysis framework for java declarative: what, not how easier to express sophisticated analyses correctness more clear clear variation points eases exploration of approximations enables aggressive optimization sophisticated subset-based analysis, fully on-the-fly call graph discovery, field-sensitivity, contextsensitivity, call-site sensitive, object sensitive, thread sensitive, context-sensitive heap abstraction, type filtering, precise exception analysis support for full semantic complexity of java jvm initialization, reflection analysis, threads, reference queues, native methods, class initialization, finalization, cast checking, assignment compatibility

88 our pointer analysis framework 24 datalog-based pointer analysis framework for java declarative: what, not how easier to express sophisticated analyses correctness more clear clear variation points eases exploration of approximations enables aggressive optimization sophisticated subset-based analysis, fully on-the-fly call graph discovery, field-sensitivity, contextsensitivity, call-site sensitive, object sensitive, thread sensitive, context-sensitive heap abstraction, type filtering, precise exception analysis support for full semantic complexity of java jvm initialization, reflection analysis, threads, reference queues, native methods, class initialization, finalization, cast checking, assignment compatibility enables precision and performance comparison

89 key contributions 25 expressed pointer analysis in full sophistication in datalog - core specification: 250 lines of logic - full specification: 2500 lines of logic

90 key contributions 25 expressed pointer analysis in full sophistication in datalog - core specification: 250 lines of logic - full specification: 2500 lines of logic synthesized efficient algorithms from specification - order of magnitude performance improvement - support all analyses considered interesting

91 key contributions 25 expressed pointer analysis in full sophistication in datalog - core specification: 250 lines of logic - full specification: 2500 lines of logic synthesized efficient algorithms from specification - order of magnitude performance improvement - support all analyses considered interesting approach: heuristics for searching algorithm space - targeted at recursive problem domains

92 key contributions 25 expressed pointer analysis in full sophistication in datalog - core specification: 250 lines of logic - full specification: 2500 lines of logic synthesized efficient algorithms from specification - order of magnitude performance improvement - support all analyses considered interesting approach: heuristics for searching algorithm space - targeted at recursive problem domains demonstrated scalability with explicit representation

93 from algorithms to specification pointer analysis and datalog background

94 program analysis: a domain of mutual recursion 26 var points-to

95 program analysis: a domain of mutual recursion 26 x = y var points-to

96 program analysis: a domain of mutual recursion 26 x = y var points-to

97 program analysis: a domain of mutual recursion 26 x = f() var points-to call graph

98 program analysis: a domain of mutual recursion 26 x = f() var points-to call graph

99 program analysis: a domain of mutual recursion 26 x = y.f() var points-to call graph

100 program analysis: a domain of mutual recursion 26 x = new A() var points-to call graph reachable methods

101 program analysis: a domain of mutual recursion 26 x = new A() var points-to call graph reachable methods

102 program analysis: a domain of mutual recursion 26 x.f = y var points-to call graph field points-to reachable methods

103 program analysis: a domain of mutual recursion 26 x.f = y var points-to call graph field points-to reachable methods

104 program analysis: a domain of mutual recursion 26 x = y.f var points-to call graph field points-to reachable methods

105 program analysis: a domain of mutual recursion 26 throw e var points-to call graph exceptions field points-to reachable methods

106 program analysis: a domain of mutual recursion 26 throw e var points-to call graph exceptions field points-to reachable methods

107 program analysis: a domain of mutual recursion 26 catch(e e) var points-to call graph exceptions field points-to reachable methods

108 program analysis: a domain of mutual recursion 26 g() var points-to call graph exceptions field points-to reachable methods

109 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b;

110 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c

111 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

112 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

113 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

114 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

115 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

116 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

117 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

118 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

119 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo a new A() b new B() c new C() VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

120 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo a new A() b new B() c new C() VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

121 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo a new A() b new B() c new C() a new B() VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

122 datalog: declarative mutual recursion 27 source a = new A(); b = new B(); c = new C(); a = b; b = a; c = b; AssignObjectAllocation a new A() b new B() c new C() Assign b a a b b c VarPointsTo a new A() b new B() c new C() a new B() b new A() c new B() c new A() VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

123 datalog: properties 28 limited logic programming sql with recursion prolog without complex terms (constructors) captures PTIME complexity class strictly declarative as opposed to prolog - conjunction commutative - rules commutative increases algorithm space - enables different execution strategies - enables more aggressive optimization writing datalog is less programming, more specification

124 from algorithms to specification declarative pointer analysis specification

125 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).

126 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj).?base.?field =?from

127 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). ˆ VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj). FieldPointsTo(?baseobj,?field,?obj) <-?base.?field =?from

128 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). ˆ VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj). FieldPointsTo(?baseobj,?field,?obj) <- field?field of object?baseobj may point to object?obj?base.?field =?from

129 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj). field?field of object?baseobj may point to object?obj FieldPointsTo(?baseobj,?field,?obj) <- ˆ StoreField(?from,?base,?field),?base.?field =?from

130 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj). FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?base,?baseobj), ˆ VarPointsTo(?from,?obj).?base.?field =?from?baseobj?obj field?field of object?baseobj may point to object?obj

131 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj). FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?base,?baseobj), VarPointsTo(?from,?obj).?to =?base.?field

132 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj). FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?base,?baseobj), VarPointsTo(?from,?obj). ˆ VarPointsTo(?to,?obj) <-?to =?base.?field

133 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj). ˆ FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?base,?baseobj), VarPointsTo(?from,?obj). VarPointsTo(?to,?obj) <- LoadField(?base,?field,?to),?to =?base.?field

134 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj). ˆ FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?base,?baseobj), VarPointsTo(?from,?obj). VarPointsTo(?to,?obj) <- LoadField(?base,?field,?to), VarPointsTo(?base,?baseobj),?baseobj?to =?base.?field

135 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj). ˆ FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?base,?baseobj), VarPointsTo(?from,?obj).?to =?base.?field VarPointsTo(?to,?obj) <- LoadField(?base,?field,?to), VarPointsTo(?base,?baseobj), FieldPointsTo(?baseobj,?field,?obj).?baseobj

136 example 1: introducing fields 29 VarPointsTo(?var,?obj) <- AssignObjectAllocation(?var,?obj). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj). modularity: introducing new mutually recursive dependencies does not change existing rules FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?base,?baseobj), VarPointsTo(?from,?obj). VarPointsTo(?to,?obj) <- LoadField(?base,?field,?to), VarPointsTo(?base,?baseobj), FieldPointsTo(?baseobj,?field,?obj).

137 example 2: precise exception analysis 30 method invocations: propagated exceptions void f() { --g(); } method invocations: caught exceptions void f() { --try { ----g(); --} catch(e e) {..} }

138 example 2: precise exception analysis 30 method invocations: propagated exceptions ˆ ThrowPointsTo(?caller,?obj) <- void f() { --g(); } ˆ method invocations: caught exceptions VarPointsTo(?param,?obj) <- void f() { --try { ----g(); --} catch(e e) {..} }

139 example 2: precise exception analysis 30 ˆ method invocations: propagated exceptions ThrowPointsTo(?caller,?obj) <- CallGraphEdge(?invocation,?tomethod), void f() { --g(); } method invocations: caught exceptions VarPointsTo(?param,?obj) <- void f() { --try { ----g(); --} catch(e e) {..} }

140 example 2: precise exception analysis 30 ˆ method invocations: propagated exceptions ThrowPointsTo(?caller,?obj) <- CallGraphEdge(?invocation,?tomethod), ThrowPointsTo(?tomethod,?obj), void f() { --g(); } method invocations: caught exceptions VarPointsTo(?param,?obj) <- void f() { --try { ----g(); --} catch(e e) {..} }

141 example 2: precise exception analysis 30 ˆ method invocations: propagated exceptions ThrowPointsTo(?caller,?obj) <- CallGraphEdge(?invocation,?tomethod), ThrowPointsTo(?tomethod,?obj), Type[?obj] =?objtype, void f() { --g(); } method invocations: caught exceptions VarPointsTo(?param,?obj) <- void f() { --try { ----g(); --} catch(e e) {..} }

142 example 2: precise exception analysis 30 ˆ method invocations: propagated exceptions ThrowPointsTo(?caller,?obj) <- CallGraphEdge(?invocation,?tomethod), ThrowPointsTo(?tomethod,?obj), Type[?obj] =?objtype, not exists ExceptionHandler[?objtype,?invocation], void f() { --g(); } method invocations: caught exceptions VarPointsTo(?param,?obj) <- void f() { --try { ----g(); --} catch(e e) {..} }

143 example 2: precise exception analysis 30 ˆ method invocations: propagated exceptions void f() { ThrowPointsTo(?caller,?obj) <- --g(); CallGraphEdge(?invocation,?tomethod), } ThrowPointsTo(?tomethod,?obj), Type[?obj] =?objtype, not exists ExceptionHandler[?objtype,?invocation], Method[?invocation] =?caller. method invocations: caught exceptions VarPointsTo(?param,?obj) <- void f() { --try { ----g(); --} catch(e e) {..} }

144 example 2: precise exception analysis 30 ˆ ˆ method invocations: propagated exceptions ThrowPointsTo(?caller,?obj) <- CallGraphEdge(?invocation,?tomethod), ThrowPointsTo(?tomethod,?obj), Type[?obj] =?objtype, not exists ExceptionHandler[?objtype,?invocation], Method[?invocation] =?caller. method invocations: caught exceptions VarPointsTo(?param,?obj) <- CallGraphEdge(?invocation,?tomethod), ThrowPointsTo(?tomethod,?obj), Type[?obj] =?objtype, void f() { --g(); } void f() { --try { ----g(); --} catch(e e) {..} }

145 example 2: precise exception analysis 30 ˆ method invocations: propagated exceptions ThrowPointsTo(?caller,?obj) <- CallGraphEdge(?invocation,?tomethod), ThrowPointsTo(?tomethod,?obj), Type[?obj] =?objtype, not exists ExceptionHandler[?objtype,?invocation], Method[?invocation] =?caller. method invocations: caught exceptions VarPointsTo(?param,?obj) <- CallGraphEdge(?invocation,?tomethod), ThrowPointsTo(?tomethod,?obj), Type[?obj] =?objtype, ExceptionHandler[?objtype,?invocation] =?handler, void f() { --g(); } void f() { --try { ----g(); --} catch(e e) {..} }

146 example 2: precise exception analysis 30 ˆ method invocations: propagated exceptions ThrowPointsTo(?caller,?obj) <- CallGraphEdge(?invocation,?tomethod), ThrowPointsTo(?tomethod,?obj), Type[?obj] =?objtype, not exists ExceptionHandler[?objtype,?invocation], Method[?invocation] =?caller. void f() { --g(); } method invocations: caught exceptions void f() { --try { VarPointsTo(?param,?obj) <- ----g(); CallGraphEdge(?invocation,?tomethod), --} catch(e e) {..} ThrowPointsTo(?tomethod,?obj), } Type[?obj] =?objtype, ExceptionHandler[?objtype,?invocation] =?handler, ExceptionHandler:FormalParam[?handler] =?param.

147 example 2: precise exception analysis 30 method invocations: propagated exceptions ThrowPointsTo(?caller,?obj) <- CallGraphEdge(?invocation,?tomethod), ThrowPointsTo(?tomethod,?obj), Type[?obj] =?objtype, not exists ExceptionHandler[?objtype,?invocation], Method[?invocation] =?caller. method invocations: caught exceptions VarPointsTo(?param,?obj) <- CallGraphEdge(?invocation,?tomethod), ThrowPointsTo(?tomethod,?obj), Type[?obj] =?objtype, again, ExceptionHandler[?objtype, new mutually?invocation] =?handler, recursive ExceptionHandler:FormalParam[?handler] dependencies do =?param. not change existing rules void f() { --g(); } void f() { --try { ----g(); --} catch(e e) {..} }

148 from algorithms to specification and back 31 algorithm policy algorithm policy algorithm policy specification algorithm policy analysis

149 from algorithms to specification and back 31 algorithm policy algorithm algorithm algorithm policy algorithm algorithm policy specification algorithm algorithm algorithm algorithm policy algorithm analysis synthesis

150 from specification to algorithms

151 from specification to algorithms benchmark

152 benchmark: 1-call-site-sensitive+heap doop paddle analysis time (seconds) antlr bloat chart eclipse hsqldb jython luindex lusearch pmd xalan

153 benchmark: 1-call-site-sensitive+heap full order of magnitude faster! doop paddle 5000 analysis time (seconds) comparable: demonstrated equivalence of results! antlr bloat chart eclipse hsqldb jython luindex lusearch pmd xalan

154 where is the magic? 33 strictly declarative - no coupling of specification and algorithm efficient architecture for datalog engine - but, not a magic tool novel heuristics for searching the algorithm space - targeting recursive logic

155 from specification to algorithms datalog evaluation background

156 datalog: naive evaluation 34 datalog VarPointsTo(?var,?obj) <- AssignObjectAllocation(?obj,?var). VarPointsTo(?to,?obj) <- Assign(?from,?to), VarPointsTo(?from,?obj). naive evaluation (relational algebra) VarPointsTo := AssignObjectAllocation repeat tmp := π to var,heap (VarPointsTo from=var VarPointsTo := VarPointsTo tmp until fixpoint Assign)

157 datalog: naive evaluation 35 VarPointsTo Assign

158 datalog: naive evaluation 35 VarPointsTo Assign

159 datalog: semi-naive evaluation 36 VarPointsTo Assign

160 datalog: semi-naive evaluation 36 VarPointsTo Assign no repeated iteration over computed results

161 datalog: semi-naive evaluation 36 VarPointsTo Assign no repeated iteration over computed results

162 from specification to algorithms datalog engine architecture

163 datalog engine architecture 37 efficient low-level representation "<java.lang.thread: void <clinit>()>" 32-bit int call-graph edge: invocation method 64-bit int relations are multidimensional indexes (b-trees) index-organized tables efficient representation of sparse relations alternative data structures for (partially) dense relations indexes exposed in the language argument order determines index enables us to stay in datalog

164 relations as multidimensional indexes 38 VarPointsTo a new A() b new B() c new C() a new B() b new A() c new B() c new A()

165 relations as multidimensional indexes 38 VarPointsTo a new A() b new B() c new C() a new B() b new A() c new B() c new A() index organized by reverse argument order VarPointsTo(?var,?obj)

166 relations as multidimensional indexes 38 VarPointsTo a new A() b new B() c new C() a new B() b new A() c new B() c new A() index organized by reverse argument order VarPointsTo(?var,?obj) new A(), b new B(), a new A(), a new A(), c new B(), b new B(), c new C(), c

167 relations as multidimensional indexes 38 VarPointsTo new A() a new B() b new C() c new B() a new A() b new B() c new A() c index organized by reverse argument order VarPointsTo(?var,?obj) VarPointsTo(?obj,?var) a, new B() b, new B() a, new A() b, new A() c, new A() c, new B() c, new C()

168 relations as multidimensional indexes 38 VarPointsTo new A() a new B() b new C() c new B() a new A() b new B() c new A() c index organized by reverse argument order VarPointsTo(?var,?obj) VarPointsTo(?obj,?var) a, new B() b, new B() a, new A() b, new A() c, new A() c, new B() c, new C()

169 relations as multidimensional indexes 38 VarPointsTo new A() a new B() b new C() c new B() a new A() b new B() c new A() c index organized by reverse argument order VarPointsTo(?var,?obj) VarPointsTo(?obj,?var) a, new B() b, new B() a, new A() b, new A() c, new A() c, new B() c, new C()

170 relations as multidimensional indexes 38 VarPointsTo new A() a new B() b new C() c new B() a new A() b new B() c new A() c index organized by reverse argument order VarPointsTo(?var,?obj) VarPointsTo(?obj,?var) a, new B() b, new B() a, new A() b, new A() c, new A() c, new B() c, new C()

171 relations as multidimensional indexes 38 VarPointsTo new A() a new B() b new C() c new B() a new A() b new B() c new A() c index organized by reverse argument order VarPointsTo(?var,?obj) VarPointsTo(?obj,?var) a, new B() b, new B() a, new A() b, new A() c, new A() c, new B() c, new C()

172 relations as multidimensional indexes 38 VarPointsTo new A() a new B() b new C() c new B() a new A() b new B() c new A() c index organized by reverse argument order VarPointsTo(?var,?obj) VarPointsTo(?obj,?var) a, new B() b, new B() a, new A() b, new A() c, new A() c, new B() c, new C()

173 relations as multidimensional indexes 38 VarPointsTo new A() a new B() b new C() c new B() a new A() b new B() c new A() c index organized by reverse argument order VarPointsTo(?var,?obj) VarPointsTo(?obj,?var) a, new B() b, new B() a, new A() b, new A() c, new A() c, new B() c, new C()

174 from specification to algorithms heuristics for searching the algorithm space

175 optimization approach: heuristics 39 datalog specification datalog engine inefficient? traditional design of single algorithm major human effort!

176 heuristics for searching algorithm space 40 always use index efficiently

177 heuristics for searching algorithm space 40 always use index efficiently VarPointsTo(?from,?obj) Assign(?from,?to)

178 heuristics for searching algorithm space 40 always use index efficiently VarPointsTo(?from,?obj) Assign(?from,?to) VarPointsTo(?from,?obj) Assign(?to,?from)

179 heuristics for searching algorithm space 40 always use index efficiently never iterate over full views

180 heuristics for searching algorithm space 40 always use index efficiently never iterate over full views Assign(?to,?from) VarPointsTo(?from,?obj)

181 heuristics for searching algorithm space 40 always use index efficiently never iterate over full views Assign(?to,?from) VarPointsTo(?from,?obj) VarPointsTo(?from,?obj) Assign(?to,?from)

182 heuristics for searching algorithm space 40 always use index efficiently never iterate over full views Assign(?to,?from) VarPointsTo(?from,?obj) VarPointsTo(?from,?obj) Assign(?to,?from) Assign(?to,?from) VarPointsTo(?obj,?from)

183 heuristics for searching algorithm space 40 always use index efficiently never iterate over full views never iterate over big input relations

184 heuristics for searching algorithm space 40 always use index efficiently never iterate over full views never iterate over big input relations StoreField(?from,?field,?base) VarPointsTo(?obj,?from) VarPointsTo(?baseobj,?base)

185 from specification to algorithms folding transformation example

186 folding transformation: example 41 specification ˆ FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?baseobj,?base), VarPointsTo(?obj,?from).

187 folding transformation: example 41 specification ˆ FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?baseobj,?base), VarPointsTo(?obj,?from). semi-naive executions ˆ FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?baseobj,?base), VarPointsTo(?obj,?from). ˆ FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?baseobj,?base), VarPointsTo(?obj,?from).

188 folding transformation: example 42 specification FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?baseobj,?base), VarPointsTo(?obj,?from). join orderings VarPointsTo(?baseobj,?base) StoreField(?from,?base,?field) VarPointsTo(?obj,?from) VarPointsTo(?obj,?from) StoreField(?from,?base,?field) VarPointsTo(?baseobj,?base)

189 folding transformation: example 42 specification FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?baseobj,?base), VarPointsTo(?obj,?from). there is no efficient variable ordering! join orderings ˆ VarPointsTo(?baseobj,?base) StoreField(?from,?base,?field) VarPointsTo(?obj,?from) ˆ VarPointsTo(?obj,?from) StoreField(?from,?base,?field) VarPointsTo(?baseobj,?base)

190 folding transformation: example 42 specification FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?baseobj,?base), VarPointsTo(?obj,?from). there is no efficient variable ordering! join orderings VarPointsTo(?baseobj,?base) StoreField(?from,?base,?field) VarPointsTo(?obj,?from) VarPointsTo(?obj,?from) StoreField(?from,?base,?field) VarPointsTo(?baseobj,?base)

191 folding transformation: example 42 specification FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?baseobj,?base), VarPointsTo(?obj,?from). there is no efficient variable ordering! join orderings VarPointsTo(?baseobj,?base) StoreField(?from,?base,?field) VarPointsTo(?obj,?from) VarPointsTo(?obj,?from) StoreField(?from,?base,?field) VarPointsTo(?baseobj,?base)

192 folding transformation: example 43 specification ˆ FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?baseobj,?base), VarPointsTo(?obj,?from).

193 folding transformation: example 43 specification ˆ FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?baseobj,?base), VarPointsTo(?obj,?from). optimized rules FieldPointsTo(?obj,?field,?baseobj) <- VarPointsTo(?obj,?from). ˆ StoreObjectField(?baseobj,?field,?from) <- StoreField(?from,?field,?base), VarPointsTo(?baseobj,?base).

194 folding transformation: example 43 specification FieldPointsTo(?baseobj,?field,?obj) <- StoreField(?from,?base,?field), VarPointsTo(?baseobj,?base), VarPointsTo(?obj,?from). optimized rules ˆ FieldPointsTo(?obj,?field,?baseobj) <- StoreObjectField(?baseobj,?field,?from), VarPointsTo(?obj,?from). ˆ StoreObjectField(?baseobj,?field,?from) <- StoreField(?from,?field,?base), VarPointsTo(?baseobj,?base).

195 algorithm space is gigantic 44 inclusion-based unification-based flow-sensitive on-the-fly call graph context-sensitive k-cfa object sensitive field-based field-sensitive heap cloning binary decision diagrams demand-driven

196 algorithm space is gigantic 44 inclusion-based unification-based flow-sensitive on-the-fly call graph database community: index selection materialized view selection context-sensitive k-cfa object sensitive field-based field-sensitive heap cloning binary decision diagrams demand-driven

197 from algorithms to specification from specification to algorithms

198 fundamental new approach to pointer analysis 45 distilled domain of pointer analysis to its essence declarative specification fully captured the domain: no loss of sophistication - even improved: exception, reflection analysis - declarativiness is a major benefit here developed techniques for deriving efficient algorithms effective heuristics for searching the algorithm space outperforms current hand-written implementations evidence that this is an effective method!

199 future work 46 pointer analysis - new context abstractions - selective context-sensitivity - on-demand analysis - improve evaluation methods

200 future work 46 pointer analysis - new context abstractions - selective context-sensitivity - on-demand analysis - improve evaluation methods program analysis and applications - mixed language analysis - declarative dynamic program analysis - applications: client analyses

201 future work 46 pointer analysis - new context abstractions - selective context-sensitivity - on-demand analysis - improve evaluation methods program analysis and applications - mixed language analysis - declarative dynamic program analysis - applications: client analyses transformation systems, parsing, extensible languages - too much to list here!

202 my research 47 extensible and domain-specific languages embedding domain-specific languages [OOPSLA 2004, GPCE 2005, OOPSLA 2006, OOPSLA 2008, ATEM 2007] preventing injection attacks [GPCE 2007, SCP 2009] parse table composition [SLE 2008] program transformation systems stratego/xt transformation system [PEPM 2006, FI 2006, SCP 2008] grammar engineering [LDTA 2007] program analysis scalable declarative pointer analysis [ISSTA 2009]