How to write a GCC plugin with MELT?

Transcription

1 Polytech Tours July 12, 2011, RMLL 2011 These slides are under a Creative Commons Attribution-ShareAlike 3.0 Unported License 1/35

2 About me [] I am finishing my studies at Polytech Tours (France) (both as engineer and researcher student). I have been fortunate to work on (and ) during a school project. I am still working on this as a Google Summer Of Code Project. I am still learning and don t know everything :) Special thanks I specially thanks Alexandre Lissy (my supervisor), Basile Starynkevitch ( creator), and the Google GSOC program. 2/35

3 Plan 1 About me About this talk Foreword about 2 Presentation Internal structure of Plugins 3 Presentation Why a DSL? Using 4 Type of tests Using Modularity 3/35

4 About this talk Questions to the audience: Do you use? Do you use static analysis tools? Or is there never bugs in your code? :) We are going to discuss: Overview of. Good (and bad) reasons to code a plugin. Using to develop more efficiently extensions. Detail about. 4/35

5 Foreword about means mole in Esperanto! The idea of the name is that it digs blindly into (without knowing much where it goes :) ) and calls still found useful informations you need. In fact: a customizable extension (written in ) to run simple analysis in your C/C++ programs. Use case: You want to check that a call to malloc function is followed by a call to free in the same function. You want to check that a call to fopen is immediately followed by a test on his returned pointer. Checking that there is (or not) code after an execx* (execl, execlp, execle, execv, execvp, execvpe) (to check for error for example). 5/35

6 presentation : GNU Compiler Collection: a set of compilers for many architectures and several languages: C, C++, Ada, Fortran, Objective C, Java, Go. The Free Software compiler. Started many years ago (1987: 1.0). Distributed by the FSF (GPL V3). Widely used (Linux, Firefox, Mandriva... ). Current release is (June 2011). 450 regular contributors. Huge and complex. Incomplete documentation. Possibility to write plugins since 4.5 (April 2010). 6/35

7 presentation is still evolving: New processors support + improvements. New languages (Go has been added in 4.6) + conformity to newer language standards New optimizations (LTO with WHOPR 1 ) So, size is increasing (stat for 4.6 with ohcount): Files: Code: lines Comment: 23,6% 1 Link Time Optimization on Whole Program 7/35

8 Internal structure of 8/35

9 Internal structure of 3 parts into Front-end: parse your source code to GENERIC representation. Middle-end: make many optimizations and code analysis ( using GIMPLE representation). Back-End: convert to a processor specific assembly code (using RTL language). The Middle-end is the bigger part of : it factorizes the optimization for the different languages. This is also the main part on which (Middle End Lisp Translator) works. 9/35

10 Middle-end representation Intermediate languages at Middle end level are AST (Abstract Syntax Tree): A node represents an operator, a leaf represents an operand). An operator can be a type declaration, a condition, a function call... Example of AST for the following code: if (a < b){a = 5; } return a; 10/35

11 Middle-end representation There are two main internal representations at Middle-end level: GENERIC/TREE: AST with 180 different possible tree operators. The list of operators can be found in gcc/tree.def. GIMPLE: AST with 36 different possible GIMPLE operators 2. The list of operators can be found in gcc/gimple.def. Example of GENERIC/TREE opcodes 1 / Relational operators. 2 EQ EXPR and NE EXPR are allowed for any types. 3 The others are allowed only for integer (or pointer or enumeral) or real types. 4 In all cases the operands will have the same type, 5 and the value is always the type used by the language for booleans. / 6 DEFTREECODE (LT EXPR, lt expr, tcc comparison, 2) 7 DEFTREECODE (LE EXPR, le expr, tcc comparison, 2) 8 DEFTREECODE (GT EXPR, gt expr, tcc comparison, 2) 9 DEFTREECODE (GE EXPR, ge expr, tcc comparison, 2) 10 DEFTREECODE (EQ EXPR, eq expr, tcc comparison, 2) 11 DEFTREECODE (NE EXPR, ne expr, tcc comparison, 2) 2 Almost half of the Gimple operators are OpenMP specific. 11/35

12 The pass mechanism In the Front-end, Middle-end, Back-end, each operation or transformation of the code is performed by one or several passes. Example of passes: Dead Code Elimination. Mudflap: add Exception, on risky operations, such as dereferencing pointers, dangerous use of standard String functions... There are more than 250 passes in. They are dynamically inserted and the issue is to find where to insert your own pass (-fdump-tree-all, -fdump-rtl-all and -fdump-ipa-all dump every pass into files and are quite useful). 12/35

13 The pass mechanism 13/35

14 The Control Flow Graph (CFG) A directed graph showing the control flow behaviour of a single function: composed of Basic blocks (set of sequential instructions). a single entry and exit block per function. Simple passes are running on each function (and use the CFG). 14/35

15 Middle-end dump C file SSA(Static Single Assignment) representation 1 #include <stdio.h> 2 3 int main(){ 4 FILE file; 5 file= fopen( /root/test fopen cfile.c, 6 r+ ); 7 if(file){ 8 return 0; 9 } 10 else{ 11 return 1; 12 } 13 } 1 main (){ 2 struct FILE file; 3 int D.2531; 4 const char restrict D.2528; 5 const char restrict D.2527; 6 <bb 2>: 7 D = (const char restrict) & r+ [0]; 8 D = (const char restrict) & /root/ test fopen cfile.c [0]; 9 file 4 = fopen (D , D ); 10 if (file 4!= 0B) 11 goto <bb 3>; 12 else 13 goto <bb 4>; 14 <bb 3>: 15 D = 0; 16 goto <bb 5> (<L2>); 17 <bb 4>: 18 D = 1; 19 # D = PHI <D (3), D (4)> 20 <L2>: return D ; 21 } 15/35

16 Plugins How does it work 3? Every plugin should export a plugin init function. Severals callbacks are defined in gcc/gcc-plugin.c Plugins register handlers to theses callbacks with register callback function. Plugins can add their own new passes. To run with a plugin: gcc-4.6 -fplugin=pluginname foo.c However For the moment, there is no stable API! This is strongly discussed within the community /35

17 What can you do with plugins? Example Technically: Add a new pass inside your plugin. In practice: Use the powerful infrastructure to add your own diagnostics or extensions. is a [meta-] plugin. Dehydra / Treehydra: A javascript tool for static C++ analysis (allows to parse the GIMPLE trees) a. gcc-vcg-plugin: Shows internal structure graphically. b... c a ( b c 17/35

18 What can you do with plugins? A plugin is not useful for programmers: They know well the different parts of (doesn t need the plugin abstraction). They want to write core features (always available, nothing specific, heavily integrated in the trunk). They follows the community guidelines (and want their work to be reviewed). Plugins are for people from other projects (You): Projects have their own coding rules and need to follow them (but don t have often automatic tools to check them). Coding rules are specific to each project, so could not be integrated into. It is easier to maintain a plugin than a fork. 18/35

19 Presentation of Goal: Help you to write more easily your own extensions? Started in Quite usable now (with bugs sometimes). Available as a branch 4 (for curious and hacker) or a plugin 5 (for quick and easy use). Motivated for simple static analysis work. A new DSL (Domain Specific Language) full of functionalities. Not really documented (but we work on it!). 4 svn://gcc.gnu.org/svn/gcc/branches/melt-branchgcc-melt 5 Version 0.8 has been released yesterday! Get it at 19/35

20 Why is a lispy langage? From Basile Starynkevitch slides: The internals API are not stable (from one release to the next), and will probably never be. The API is huge and ill-defined. the api contains a lot of more or less simple functions and also non-functions (e.g. macros and even non-expression iterative macros like FOREACH FUNCTION ARGS or FOR EACH BB FN etc.). Interfacing them with scripting-language specific glue functions would be error-prone and inefficient. Glue generators (à la SWIG) don t fit well. The Ggc ( garbage collector) is essential in ; mixing several GC-s(i.e. Ggc and Ocaml s GC) is perilous and brittle. 20/35

21 This explains why is not written in ocaml or python. A new Python binding 6 is appearing. This might be more limited than : Only a reduced set of structures might be available. It will have to deal with Garbage collector issues. has also some advanced features (pattern matching) that you won t have in Python /35

22 How works First you have to write a foo.melt file (which contains your module) 7. Convert your module into C and then into a.so 1 gcc fplugin=melt fplugin arg melt mode=translatetomodule \ 2 fplugin arg melt arg=foo.melt c empty.c Use your module while compiling a source 1 gcc fplugin=melt fplugin arg melt mode=foo \ 2 fplugin arg melt init=foo \ 3 fplugin arg melt module path=$build \ 4 fplugin arg melt source path=$build source.c 7 Every example is for plugin, you can find information about how to use branch on the website 22/35

23 Using is highly inspired by functional programming (Lisp, OCamls... ) A first Hello world example 1 (defun say hello () 2 (code chunk hellochunk #{printf( Saluton, mondo! )}#) 3 ) 4 (say hello) Ok... there is a lot of parenthesis (your text editor will handle them easily). Functions are defined with the defun keyword. A code chunk is a declaration of C code directly into (but we don t use it much usually). 23/35

24 Using Installing a new pass 1 (defun talpo docmd (cmd moduldata) 2 (let ((talpo (instance class gcc gimple pass 3 :named name melt talpo pass 4 :gccpass gate talpo gate 5 :gccpass exec talpo exec 6 :gccpass properties required () 7 ))) 8 (install melt gcc pass talpo after cfg 0) 9 (debug msg talpo talpo mode installed talpo ) 10 (return talpo) 11 )) Two important functions: gate : decides if the pass should be run. exec : do the real work of the pass New pass position relative to another pass ( cfg here). 24/35

25 values and stuff Important! This is specific to (and must be clearly understood!!!). Sometimes we use data coming from (processing GIMPLEs for example): In this case, you need to refer to the C type. We call this a stuff. But we much prefer to use dynamic values (because they are handled transparently by, especially by the garbage collector). We call this a value. 25/35

26 values and stuff Using a stuff 1 (let (( :cstring mystring a dummy C string )) 2 (code chunk test #{printf($mystring)}#) 3 ) let is the keyword to define a local variable. :cstring explicit the type of the variable mystring. Using a value 1 (let (( mystringval mytest )) 2 (errormsg strv Given argument is not a valid test. mystringval) 3 ) Usually, we try to work as much as possible with values! make <types>: box a stuff into a value <type> content: unbox a value as a stuff. 26/35

27 Iterative construct You can iterate over a list... But also over main types (very useful to parse CFG): each cgraph fun body each bb current fun each in gimpleseq eachgimple in basicblock... Iterating over gimpleseq 1 (eachgimple in basicblock 2 (bb) 3 (:gimple g) 4 (is gimple assignation g) 5 ) you can even define your own c-iterators! 27/35

28 Pattern matching on trees Regular expression on a text. pattern-matching on trees (especially representation). Many use cases: Finding every call of a function containing melt in his declaration name. Finding comparison to a NULL pointer. Detecting casts.... How to do? use the match operator. take a object as first parameter (a tree, a GIMPLE, a basic block...). add your different patterns and code to be run if the pattern is verified. You can also add your own pattern if you don t find what you need. 28/35

29 Pattern matching Pattern matching example 1 (defun detect fopen (grdata :gimple g) 2 (match g 3 ( 4?(gimple call 2?lhs 5?(and?callfndcl 6?(tree function decl 7?(cstring same fopen ) 8? )) 9?arg1 10?arg2 11 ) 12 (return (make tree discr tree lhs)) 13 ) 14 ( 15? ((return (make tree discr tree (null tree)))) 16 ) 17 ) 18 ) 29/35

30 8 started with the idea that a static analysis tool can use the powerful functionalities of and must be customized for a project: A test can be easily parameterized by people ignoring (much of) and. With you can check this: for each call to a foo function, result of the call is tested to be (not) NULL/negative/zero. each call to a foo function is immediately followed by a call to a bar function. each call to a foo function is followed by a call to a bar function in the same function body. This is quite simple and limited but can already be useful in many cases! /35

31 Easy to use There are different ways to pass argument to : Using pragma in code file 1 #pragma talpo testnull(fopen) 2 #pragma talpo test followed\by(chroot, chdir) Using direct argument 1 gcc... 2 fplugin arg melt option=talpo arg= (testnull fopen )\ 3 (test followed by chroot chown ) 4... Using a file to list argument? TODO! 31/35

32 Modularity You can implement a new way to read user input. You can implement a new test. 32/35

33 Try it argument 1 #include <stdio.h> 2 int main(void){ 3 FILE test; 4 test=fopen ( test, a ); 5 return 0; 6 } Result 1 gcc Wall fplugin=melt fplugin arg melt mode=talpo \ 2 fplugin arg melt module path= $(talpopath) \ 3 fplugin arg melt source path=. \ 4 fplugin arg melt extra=@$(talpopath)/talpo \ 5 fplugin arg melt option=talpo arg= (testnull fopen ) \ 6 O2 test.c o test.o 1 test simple check cfile.c:5:10: warning: Melt Warning[#221]: Function fopen not followed by a test on his returned pointer. 33/35

34 Try it argument 1 int main(void){ 2 int i=0; 3 FILE test; 4 FILE test2; 5 if( i ==1) 6 test=fopen( toto, a ); 7 else 8 test2=fopen( tata, a ); 9 if (test == NULL) 10 return 1; 11 return 0; 12 } Result 1 test simple check cfile.c:8:10: warning: Melt Warning[#216]: Function fopen not followed by a test on his returned pointer. 34/35

35 Conclusion Free software is about adapting software to your needs: Plugins are great way to personalize. 9 simplifies writing extensions (and is more fun than coding in C). is the result of previous points. I invite you to test. Questions: Was this interesting? Have you idea for a plugin? Have you idea of tests? Are you going to try //? 9 also has a mailing list : gcc-melt@googlegroup.com 35/35

36 Additional slides 36/35

37 Basic Block A basic block is a data representing a part of a function. It has only one input and one output. It is a set (gimple seq) of sequential instructions (without conditions or loops). instruction are GIMPLE This is very useful for analyse and optimisation: We can deduce in which Basic block a variable appears. Basic blocks are hierarchically organised (and we get properties from this). 37/35

38 Using You can declare classes (and instances of them): Class declaration 1 (defclass class talpo test model 2 :super class named 3 :fields ( 4 talpo test model handler ;;the handler is the function used when executing the pass 5 talpo test model help 6 talpo test model nbargs ;;number of arguments 7 )) single-inheritance hierarchy (using super) field name must be unique. no access properties (no protected/private fields) 38/35

39 Debugging in You will need to use debug: To understand structures. To understand results. To find bugs in your code (or in code). For now debug is limited to a file dumping many useful informations. You must pass the option -fplugin-arg-melt-debug when compiling. In your code you can use functions like: debug<type> (such as debugtree or debuggimple) when debugging stuff. debug msg when debugging a value 39/35

40 Debugging in Example of debugging 1 (debugtree test fopen gate cfun decl (cfun decl)) Result debugtree test fopen gate /function decl \ 2 <function decl 0x41a93580 hello 3... //ensemble d informations concernant la fonction \ 4 //hello (argument, retour...). 5 debugtree test fopen gate /function decl \ 7 <function decl 0x41a93600 main 8... //ensemble d informations concernant la fonction main. 40/35

41 primitives Example of primitive 1 (defprimitive get immediate dominator unsafe (:basic block bb) :basic block 2 :doc#{it doesn t check that dominance info are build, use 3 get immediate dominator instead.}# 4 #{($bb)? get immediate dominator (CDI DOMINATORS, $bb) : NULL}# 5 ) 41/35