Recovering Erlang AST from BEAM bytecode

Size: px

Start display at page:

Download "Recovering Erlang AST from BEAM bytecode"

Claud Hodges
6 years ago
Views:

1 Recovering Erlang AST from BEAM bytecode Dániel Lukács and Melinda Tóth Eötvös Loránd University HU, Faculty of Informatics This research has been supported by the Hungarian Government through the New National Excellence Program of the Ministry of Human Capacities.

2 Motivation Problem statement lib8.beam lib8.beam lib9.beam lib9.beam lib2.erl lib2.erl lib3.erl lib7.beam lib.erl lib6.beam lib3.erl lib7.beam lib.erl lib6.beam mymodule.erl lib5.beam mymodule.erl lib5.beam lib4.erl lib4.erl lib4.beam lib4.beam

3 Goal Problem statement Enable the RefactorErl static analysis system for Erlang to recover information from source dependencies stored as Erlang BEAM bytecode.

RefactorErl Problem statement RefactorErl Static analysis framework for Erlang 26, ELTE Faculty of Informatics, Department of Programming Languages And Compilers

4 RefactorErl Problem statement RefactorErl Static analysis framework for Erlang 26, ELTE Faculty of Informatics, Department of Programming Languages And Compilers Functionality: Semantic queries Refactorings Dependency analysis Code metrics, investigation, duplicated code analysis, and more. Web interface, GUI, CLI

$RefactorErl Problem statement %% Rectangular function rect (T) when abs ( T) < ; abs (T) =:= -> ; rect (_) ->. { function, rect,, 6}. { label,5}. {line,[{ location," example. erl ",}]}.$

5 RefactorErl Problem statement %% Rectangular function rect (T) when abs ( T) < ; abs (T) =:= -> ; rect (_) ->. { function, rect,, 6}. { label,5}. {line,[{ location," example. erl ",}]}. { func_info,{ atom, example },{atom,rect },}. { label,6}. { gc_bif,abs,{f,7},,[{x,}],{x,}}. {test, is_ge,{f,8},[{x,},{ integer,}]}. { label,7}. { gc_bif,abs,{f,9},,[{x,}],{x,}}. {test, is_eq_exact,{f,9},[{x,},{ integer,}]}. { label,8}. {move,{ integer,},{x,}}. return. { label,9}. {move,{ integer,},{x,}}. return.

6 Machine code vs. BEAM Problem statement Machine code Low-level (CPU) Heavily optimized code Complicated loop constructs Dynamic memory allocation Indirect addressing Stack frames Manual process control No metainformation about modules and functions BEAM bytecode High-level (VM) Simplified code Loops always correspond to specific Erlang constructs Managed memory allocation with garbage collection Registers, index displacement Special registers for local variables Process scheduling managed by VM Stores metainformation about modules and functions

7 Goal (revision) Problem statement Enable the RefactorErl static analysis system for Erlang to recover information from source dependencies stored as Erlang BEAM bytecode. Approach: Represent recovered BEAM semantical information in Erlang syntax. Ready for RefactorErl Results can be compared with the original Existing decompilation techniques can be used

8 What is the problem? Problem statement -module(event_handler). -export([handle_event/]). handle_event(event) -> case Event of ok -> done; {message, _} -> done; _ -> unknown_event end. lib8.beam lib3.erl lib2.erl lib.erl lib9.beam lib7.beam lib6.beam {module, event_handler}. %% version = {exports, [{handle_event,},{module_info,},{module_info,}]}. {attributes, []}. {labels, }. {function, handle_event,, 2}. {label,}. {line,[{location,"event_handler.erl",4}]}. {func_info,{atom,event_handler},{atom,handle_event},}. {label,2}. {test,is_tuple,{f,3},[{x,}]}. {test,test_arity,{f,5},[{x,},2]}. {get_tuple_element,{x,},,{x,}}. {test,is_eq_exact,{f,5},[{x,},{atom,message}]}. {jump,{f,4}}. {label,3}. {test,is_eq_exact,{f,5},[{x,},{atom,ok}]}. {label,4}. {move,{atom,done},{x,}}. return. {label,5}. {move,{atom,unknown_event},{x,}}. return. mymodule.erl lib5.beam lib4.erl lib4.beam

9 Is it starightforward? NO! Problem statement -module(event_handler). -export([handle_event/]). handle_event(event) -> case Event of ok -> done; {message, _} -> done; _ -> unknown_event end. {entry,{event_handler,handle_event,}} procedure handle_event [{x,}] at 2: 4 2: 5 if not is_tuple [{x,}] then goto 3; 6 if not test_arity [{x,},2] then goto 5; 7 {x,} := {x,}[]; 8 if not is_eq_exact [{x,},{atom,message}] then goto 5; 3: if not is_eq_exact [{x,},{atom,ok}] then goto 5; lib8.beam lib9.beam lib2.erl lib7.beam 5 5: 6 {x,} := {atom,unknown_event}; 7 return {x,}; 2 4: 3 {x,} := {atom,done}; 4 return {x,}; lib6.beam lib3.erl lib.erl {exit,{event_handler,handle_event,}} mymodule.erl lib5.beam lib4.erl lib4.beam

10 Decompiler workflow Disassembled BEAM Intermediate representation (IR) Control flow graph (CFG) Static single assignment (SSA) Structural analysis Extended A-normal form (ANF) Erlang syntax tree

11 Disassambled BEAM {module, event_handler}. %% version = {exports, [{handle_event,},{module_info,},{module_info,}]}. {attributes, []}. {labels, }. Disassemblers shipped with Erlang/OTP: beam disasm module erlc -S {function, handle_event,, 2}. {label,}. {line,[{location,"event_handler.erl",4}]}. {func_info,{atom,event_handler},{atom,handle_event},}. {label,2}. {test,is_tuple,{f,3},[{x,}]}. {test,test_arity,{f,5},[{x,},2]}. {get_tuple_element,{x,},,{x,}}. {test,is_eq_exact,{f,5},[{x,},{atom,message}]}. {jump,{f,4}}. {label,3}. {test,is_eq_exact,{f,5},[{x,},{atom,ok}]}. {label,4}. {move,{atom,done},{x,}}. return. {label,5}. {move,{atom,unknown_event},{x,}}. return.

12 Intermediate Representation Disassembled BEAM Intermediate representation (IR) Control flow graph (CFG) Static single assignment (SSA) Structural analysis Extended A-normal form (ANF) Explicit syntax Abstraction Internal model Erlang syntax tree {move,{atom,done},{x,}}. {x,} := {atom,done}; {call_ext_only,,{extfunc,erlang,get_module_info,}}. {x,} := {atom,event_handler}; {x,} := call erlang:get_module_info/ [{x,}]; procedure handle_event [{x,}] at 2: : throw {function_clause,{atom,event_handler},{atom,handle_event},}; 2: if not is_tuple [{x,}] then goto 3; if not test_arity [{x,},2] then goto 5; {x,} := {x,}[]; if not is_eq_exact [{x,},{atom,message}] then goto 5; goto 4; 3: if not is_eq_exact [{x,},{atom,ok}] then goto 5; 4: {x,} := {atom,done}; return {x,}; 5: {x,} := {atom,unknown_event}; return {x,};

13 Intermediate Representation An internal model of BEAM semantics. Some BEAM instructions has implicit semantics. Making it explicit reduces possibility for implementation errors. Abstraction layer BEAM semantics is not formally documented BEAM semantics may change

14 Control Flow Graph (CFG) Disassembled BEAM Intermediate representation (IR) Control flow graph (CFG) Static single assignment (SSA) Structural analysis Extended A-normal form (ANF) {entry,{event_handler,handle_event,}} Erlang syntax tree procedure handle_event [{x,}] at 2: Equivalent graph representation of the program flow Consists of blocks (nodes) and flows (edges). Base of further analyses Dominator Static Single Assignment Structuring Simple transformations 4 2: 5 if not is_tuple [{x,}] then goto 3; 6 if not test_arity [{x,},2] then goto 5; 7 {x,} := {x,}[]; 3: 8 if not is_eq_exact [{x,},{atom,message}] then goto 5; if not is_eq_exact [{x,},{atom,ok}] then goto 5; 5 5: 2 4: 6 {x,} := {atom,unknown_event}; 3 {x,} := {atom,done}; 7 return {x,}; 4 return {x,}; {exit,{event_handler,handle_event,}}

15 Static Single Assignment (SSA) Disassembled BEAM Intermediate representation (IR) Control flow graph (CFG) Static single assignment (SSA) Structural analysis Extended A-normal form (ANF) Erlang syntax tree Each symbol is defined only once Calculated based on dominator information SSA Functional representation x := 2 x := f(x)... x# := 2 x#2 := x#3 := φ(x#, x#2) f(x#3)... Cytron, Ron; Ferrante, Jeanne; Rosen, Barry K.; Wegman, Mark N. & Zadeck, F. Kenneth (99), Efficiently computing static single assignment form and the control dependence graph, ACM Transactions on Programming Languages and Systems.

16 Structuring 2 Disassembled BEAM Intermediate representation (IR) Control flow graph (CFG) Static single assignment (SSA) Structural analysis Extended A-normal form (ANF) Erlang syntax tree Identifying high level control structures Region: one entry and exit point Nested regions 2 Cifuentes C. (996), Structuring decompiled graphs. In: Gyimóthy T. (eds) Compiler Construction. CC 996. Lecture Notes in Computer Science, vol 6. Springer, Berlin, Heidelberg

17 Structuring 3 {entry,{event_handler,handle_event,}} procedure handle_event [{x,}] at 2: 4 2: 5 if not is_tuple [{x,}] then goto 3; 6 if not test_arity [{x,},2] then goto 5; 7 {x,} := {x,}[]; 3: 8 if not is_eq_exact [{x,},{atom,message}] then goto 5; if not is_eq_exact [{x,},{atom,ok}] then goto 5; 5 5: 2 4: 6 {x,} := {atom,unknown_event}; 3 {x,} := {atom,done}; 7 return {x,}; 4 return {x,}; Unstructured control flow goto No decomposition Pattern based analysis {exit,{event_handler,handle_event,}} 3 Cifuentes C. (996), Structuring decompiled graphs. In: Gyimóthy T. (eds) Compiler Construction. CC 996. Lecture Notes in Computer Science, vol 6. Springer, Berlin, Heidelberg

18 Structuring 5 Graph Rewriting (TGR) 4 Formal semantics and provable properties Expressive, visualisable tmo_clause: Var var4 recv_tmout (x) x: Block Var var3 Var var4 RecvHead recv_tmo_edge: RecvTmoutClause Var tmo_amount wait_edge: Edge Var var Success true wait_back: Var var4 y: Block Var var9 Var var {test,is_timeout_bit_not_set,[]} tmo_edge: Edge Var var2 Success false 4 Ehrig, Hartmut and Pfender, Michael and Schneider, Hans-Jürgen. Graph-grammars: An algebraic approach, Switching and Automata Theory, 973. SWAT 8. IEEE Conference Record of 4th Annual Symposium on, pp. 67-8, 973, IEEE

19 Structuring 6 Erlang branching structures bool_andalso (left_op) left_op: Block Var lpars Var left_body Var ltest Block Var lpars Var left_body AndExpr Var Var right_body Var rtest left2right: Edge Var var2 Success true clause_group (x) x: Var var68 e: Edge Var var57 If Var e_test e3: Edge [] If AndExpr y: Var e_test Var var69 Var e2_test {entry,{event_handler,handle_event,}} procedure handle_event [{x,}] at 2: 4 2: 5 if not is_tuple [{x,}] then goto 3; 6 if not test_arity [{x,},2] then goto 5; 7 {x,} := {x,}[]; 3: 8 if not is_eq_exact [{x,},{atom,message}] then goto 5; if not is_eq_exact [{x,},{atom,ok}] then goto 5; new_left2other: Edge [] Success true right_op: Block Var var9 Var right_body Var rtest left2join: Edge Var var2 Success false e2: Edge Var var58 If Var e2_test 5 5: 2 4: 6 {x,} := {atom,unknown_event}; 3 {x,} := {atom,done}; 7 return {x,}; 4 return {x,}; right2other: right2join: Edge Edge Var var22 Var var23 Success true Success false z: Var var67 {exit,{event_handler,handle_event,}} other_branch: Var var46 join_branch: Var var45 6 Cifuentes C. (996), Structuring decompiled graphs. In: Gyimóthy T. (eds) Compiler Construction. CC 996. Lecture Notes in Computer Science, vol 6. Springer, Berlin, Heidelberg

20 Structuring 7 Context-analysis tmo_clause: Var var4 recv_tmout (x) x: Block Var var3 Var var4 RecvHead recv_tmo_edge: RecvTmoutClause Var tmo_amount wait_edge: Edge Var var Success true wait_back: Var var4 y: Block Var var9 Var var {test,is_timeout_bit_not_set,[]} tmo_edge: Edge Var var2 Success false {entry,{event_handler,handle_event,}} procedure handle_event [{x,}] at 2: 4 2: 5 if not is_tuple [{x,}] then goto 3; 6 if not test_arity [{x,},2] then goto 5; 7 {x,} := {x,}[]; 3: 8 if not is_eq_exact [{x,},{atom,message}] then goto 5; if not is_eq_exact [{x,},{atom,ok}] then goto 5; 5 5: 2 4: 6 {x,} := {atom,unknown_event}; 3 {x,} := {atom,done}; 7 return {x,}; 4 return {x,}; {exit,{event_handler,handle_event,}} 7 Cifuentes C. (996), Structuring decompiled graphs. In: Gyimóthy T. (eds) Compiler Construction. CC 996. Lecture Notes in Computer Science, vol 6. Springer, Berlin, Heidelberg

X3) X X2 8 Cifuentes C. (996), Structuring decompiled graphs. In: Gyimóthy T.

21 Structuring 8 Identifying regions continuation_edge (head) e2: pdom follow: Var var4 e: e3: dom continuation head: Var var3 if A then (f X) else (f X2) let X3 = if A then else in (f X3) X X2 8 Cifuentes C. (996), Structuring decompiled graphs. In: Gyimóthy T. (eds) Compiler Construction. CC 996. Lecture Notes in Computer Science, vol 6. Springer, Berlin, Heidelberg

22 Structuring 9 {entry,{event_handler,handle_event,}} {entry,{event_handler,handle_event,}} procedure handle_event [{x,}] at 2: 4 2: 5 if not is_tuple [{x,}] then goto 3; 6 if not test_arity [{x,},2] then goto 5; [] {defun_expr,handle_event,[{var,{x,},}],block_end} {cont,step} {upon, {andalso_expr, {test,is_tuple,[{var,{x,},}]}, {andalso_expr, {test,test_arity,[{var,{x,},},2]}, {let_expr, [] {bind_expr,{var,{x,},},{indexed,{var,{x,},},}}, {test,is_eq_exact,[{var,{x,},},{atom,message}]}, block_end []}}}} clause_group {upon,{test,is_tuple,[{var,{x,},}]}} else {upon,{test,is_eq_exact,[{var,{x,},},{atom,ok}]}} 7 {x,} := {x,}[]; 8 if not is_eq_exact [{x,},{atom,message}] then goto 5; 5 5: 6 {x,} := {atom,unknown_event}; 7 return {x,}; 3: if not is_eq_exact [{x,},{atom,ok}] then goto 5; 2 4: 3 {x,} := {atom,done}; 4 return {x,}; [{var,{x,},2}] {let_expr,{bind_expr,{var,{x,},2},{atom,done}},{var,{x,},2},[]} {return,{var,{x,},2}} [{var,{x,},4}] {let_expr,{bind_expr,{var,{x,},4},{atom,unknown_event}},{var,{x,},4},[]} {return,{var,{x,},4}} {exit,{event_handler,handle_event,}} [{{x,},2},{{x,},2}] [{var,{x,},3},{var,{x,},3}] [{{x,},4},{{x,},4}] 9 Cifuentes C. (996), Structuring decompiled graphs. In: Gyimóthy T. (eds) Compiler Construction. CC 996. Lecture Notes in Computer Science, vol 6. Springer, Berlin, Heidelberg

$A-Normal Form Functional-style program representation Translates out of SSA FUN handle_event [x#] = IF WHEN ANDALSO {test,is_tuple,[x#]} ANDALSO {test,test_arity,[x#,2]} LET x# = {indexed,x#,} IN$

23 A-Normal Form Functional-style program representation Translates out of SSA FUN handle_event [x#] = IF WHEN ANDALSO {test,is_tuple,[x#]} ANDALSO {test,test_arity,[x#,2]} LET x# = {indexed,x#,} IN {test,is_eq_exact,[x#,{atom,message}]} -> LET x#2 = {atom,done} IN x#2 WHEN {test,is_eq_exact,[x#,{atom,ok}]} -> LET x#2 = {atom,done} IN x#2 WHEN true -> LET x#4 = {atom,unknown_event} IN x#4 Manuel M.T. Chakravarty, Gabriele Keller, and Patryk Zadarnowski (24), A Functional Perspective on SSA Optimisation Algorithms. Electronic Notes in Theoretical Computer Science, Volume 82, Issue 2, April 24, Pages

24 Code generation Disassembled BEAM Intermediate representation (IR) Control flow graph (CFG) Static single assignment (SSA) Structural analysis Extended A-normal form (ANF) -module(event_handler). -export([handle_event/]). Erlang syntax tree handle_event(x_) -> if (is_tuple(x_) andalso (size(x_) =:= 2 andalso element(, X_) =:= message)) -> X_2 = done, X_2; X_ =:= ok -> X_2 = done, X_2; true -> X_4 = unknown_event, X_4 end. -module(event_handler). -export([handle_event/]). handle_event(event) -> case Event of ok -> done; {message, _} -> done; _ -> unknown_event end.

25 DEMO Conclusions DEMO

26 DEMO Conclusions Module Funs Blocks Total GR Referl Others mnesia event s 22.8 s 27.2 s.734 s mnesia text s s 43.8 s.84 s mnesia index s s 2.8 s.928 s

27 Future Notes Conclusions Pre-structuring List-comprehension Fun-expression Full Erlang coverage Catch patterns Binaries Extended language support (Elixir)

Semantics of an Intermediate Language for Program Transformation

Semantics of an Intermediate Language for Program Transformation Sigurd Schneider Advisors: Sebastian Hack, Gert Smolka Student Session at POPL 2013 Saarland University Graduate School of Computer Science