[function]foo [scope]foo. call. return

Transcription

1 Mapping of Dynamic Language Constructs into Static Abstract Syntax Trees Jakub Míšek, Filip Zavoral Department of Software Engineering Charles University in Prague, Czech Republic Abstract Software solutions performing automatic code analysis are very important, especially for code assistance capabilities or for extracting semantic metadata from the source code. These methods gather syntactic information from the source code and then in general they provide large set of implying semantics. With the increased focus on dynamic languages the problem emerges the whole code semantic is known only at runtime and the analysis has to estimate larger relations. Within the project Phalanger, which is the compiler of the dynamic PHP language into a static environment, we have succeeded with converting most of dynamic constructs into the typical structures used by static code analyzers. That enables processing of most of dynamic language constructs in the classic static way. Moreover described methods simplifies source code prediction and other relevant tasks. Keywords - dynamic language, code analysis, IntelliSense, prediction, PHP, abstract syntax tree I INTRODUCTION It is a common practice to use advanced tools to enhance development processes. For example methods of assistance during writing the source code, predicting the text [4], or analyzing the code for detecting possible issues. These techniques process the syntactic structure (AST, see section II) of the code. Mostly they are designed to work with the code of static languages, because static code containes all the necessary information required for analysis. However the dynamic languages, like PHP or javascript, enable to define and redefine constructs during runtime, change the type of variables, and the source code itself does not describe all the declarations explicitly. Therefore the techniques used in the case of static languages cannot work properly, because there is not complete set of information needed for analysis. Still it is possible to extract specified constructs and expressions from the dynamic language source code, that are typically spreaded through the all source codes, and put them into the static syntactic structure to enable processing the dynamic code in the wellknown classic static way. As a part of the Phalanger project 0[7] we have implemented language integration of PHP dynamic language [8] and.net static language into the Microsoft Visual Studio environment (for the implementation details and dynamic PHP code analysis see [9]). By using the mapping of dynamic language constructs into static language syntax trees, this implementation supports most of the common IntelliSense [3] (or Code Sense) features [10]. Since PHP is a dynamic language, there are no static declarations. In general there are different variables, classes or class members every time the same source code runs. No declaration has a definite meaning until the runtime and until the declaration is used. However IntelliSense works with type information which in dynamic languages is missing until runtime. These are issues for the language integration all successive alternatives on every code fragment have to be populated. The main contribution of this paper is a description of storing information implying from dynamic language constructs into the static abstract syntax trees. Unlike the static languages, the analysis of dynamic languages has to estimate the types and possible declarations from the code fragment semantics [5][11]. These methods also take into account non-static and non-typed implicit and explicit declarations, so that the resulting information covers most of readable dynamic code fragments. II SYNTACTIC ANALYSIS In the first place, syntactic analysis must be performed to determine the structure of the code. As a result of this process, known as a parsing, the abstraction over the source code is created. It has a form of the tree called Abstract Syntax Tree (AST) [2][6]. It contains only the functional parts of the code, like scopes, declarations and expressions, in a form of syntax tree. Irrelevant parts of the code, like comments and white spaces, are mostly ignored, because they have no impact to the syntactic analysis. Still the syntax tree of the dynamic language source code does not contain information like variables declaration, data types and values since these data are not available until runtime. This must be obtained using the subsequent analysis of the AST (see III). [function]foo [scope]foo call A B return Figure 1 Abstract Syntax Tree example. Basicaly there are two types of nodes. The scope node and the expression node. Scope node is like a container for other nodes. It represents a block of code containing other scopes and expressions. Expression node describes an expression (e.g assignment, function call, declaration); it can points to other inner expressions, but no scopes. Every tree node also references corresponding position of the source code text. See Figure 1 for an example of AST. The tree contains all declarations and definitions organized in the same scope structure as in its source code. It is optimized for generating the native code or for searching for = B += C

2 specific declarations visible from the scope in the source code. Note that the parsing results into the tree containing the constructs as they are in the code, no semantic metadata or implying information is typically collected. Therefore in case of dynamic languages, all the implicit declarations or the dynamic type information needed for typical analysis processes is not presented here in this moment. Walking through the tree is typically realized using the visitor pattern. It is a deep-first-search (DFS) graph algorithm, that defines different action for every visited tree node type. It is the main part of the following semantic analysis. The tree is processed and every different node type invokes its own action. During the semantic analysis, dynamic constructs are collected and analyzed, then they are inserted into the tree in a form of static information. III DYNAMIC CONSTRUCTS MAPPING Mapping of constructs from the dynamic language into the classic abstract syntax tree is a kind of semantic analysis. Information implying from the syntax is analyzed and the results are inserted back into the same tree, but in the form of complete static information. Thereafter this preprocessing enables us to work with the syntax tree of the dynamic source code as it is in a static code with some limitations, that are not resolvable until runtime in dynamic languages [9][10]. III.A Tree construction Creation of the Abstract Syntax Tree for the specific source file depends on the origin language (language, file format, grammar). The source has to be analyzed and then particular language elements are converted into the tree nodes subsequently. The typical process consists of the source code parsing and creation of the AST. The visitor pattern is used on the AST and source code elements are then processed in a custom way, like generating native code or just searching for specified tree node containing specified declaration. Typical process is depicted on Figure 2. Source Code text Source Code Parser Abstract Syntax Tree Static Code processing Semantic Analysis (Mapping Dynamic Constructs) Figure 2 Source code analysis process. The tree can be created from various source files, like.net assemblies or PHP source files. Whatever the sources are, the resulting Abstract Syntax Tree implementation unifies them. Once built, additional information is added to the tree by subsequent processing (Semantic Analysis). Note that the initial tree contains all the static declarations; the following semantic analysis gathers dynamically used symbols and then they are inserted into the tree subsequently. This extended tree can be used for further processing, like the normal AST, without considering any dynamicity of the language nature. There must be a separate implementation of tree nodes for every source code construct, e.g. function definition, assignment expression or variable usage. Within the parsing process, the source file is scanned and every relevant construct is converted into the corresponding AST node implementation. In dynamic languages the resulting tree node information is gathered from various source elements in the code. The whole source file, in a parsed form - AST, must be processed and every expression has to be analyzed. As a result it extends some existing node information or add new declaration nodes (implicit declarations). In static languages it is typically sufficient to scan declaration constructs only (class declaration, variable declaration etc.); single expressions are not needed for collecting available symbols and their description. Following source files are used in the implementation of project Phalanger to enable the support for PHP dynamic language and its integration with.net static language. Corresponding source constructs are mapped onto the syntax tree nodes. Using the same architecture of AST, it is possible to use different source files written in different languages, both static and dynamic, within one language integration. Various source files (.NET assembly, PHP source, PHP native extension) might be referenced from another source code file, and language integration mechanism needs not to solve any differences. Therefore defined symbols from many various source files may be used then together. The list of resolved file types follows, they define the complete knowledge of the source code files: PHP source file - The file contains PHP source code and references to another PHP files. The whole PHP project might reference.net assemblies - their symbols must be present in the same way as the PHP constructs are..net assembly - Type declarations are defined in.net assemblies. Every type is placed in a namespace and contains members, such as methods and properties. The declarations are static only..net assembly can reference other.net assemblies. PHP.NET extension - Special case of.net assemblies are PHP extensions generated by Phalanger. These assemblies are just wrappers for PHP native extensions. They contain type declarations within one specified namespace. Every type is described by an attribute which specifies the PHP class name. Methods can be also denoted by.net attributes to specify the regular PHP name or define global functions (the method can be used as a global function). Type properties can be denoted as PHP constants. One of

3 these special.net assemblies also contains most of default PHP functions (PHP class library) and it is referenced by all source codes by default. Native PHP symbols - Native PHP symbols, e.g. "echo", "include", "isset", "_GLOBAL" etc., must be defined within the Abstract Syntax Tree too. The special tree is created and its root node contains these symbol declarations. This tree is referenced by all other source codes. III.B PHP source file The complete creation of the syntax tree from the PHP source code consists of two steps. First of all the code must be parsed and the basic syntax tree created (III.A). In the AST statically declared constructs are defined directly during the parsing. In the second step by analyzing single expressions, implicit declarations and additional information are collected. III.B.1 Static information Many constructs in dynamic languages are static. E.g. class declaration or function definition in PHP can be defined statically. These constructs together with single expressions make the basic structure of the initial Abstract Syntax Tree. This resulting tree reflects the same hierarchical structure of code scopes as it was in the origin source code. As a result statically defined PHP constructs (classes, functions) are mapped into the representation of the tree. The following language constructs are processed directly (static code): Global code - Global code represents the root of AST. It is inherited from the scope node, hence it can contain other scopes and expressions. It has no parent node and it just remembers the full file name of the source code file, for further processing. The inner code is processed and the result nodes are placed within the GlobalScope node. Code scope - Code scopes, such as function bodies, if/else bodies or general code scopes, are inserted into the parent scope node as a child scope (inherited from the scope node). Its child elements are placed within this scope. It is used for encapsulating local declarations of variables, in order to use them only in current and child scopes. The source code hierarchical structure is then preserved. It has not to remember the file name of the source code, because the parent s node file name information can be used. Within the semantic analysis, recognized dynamically declared variables are inserted here, in the same way as the statically declared variables. Function declaration - In PHP, function is declared and defined within the single code block statement. It is stored as two AST nodes - the declaration of the function and code scope representing the body of the function. Function declaration node contains the function name and the function signature (parameters list). The declaration is linked with the corresponding code scope node which is created from the function body scope. Function parameter Function parameters, contained in the function signature, are added into the corresponding function declaration parameters list. Also every parameter is converted into the static variable declaration node. This is added into the function code scope (it represents the static local variable declaration). The variable type can be estimated using the documentary comments above the function, but mostly it has to be done by following semantic analysis. Class declaration - AST contains statically declared classes placed within the current scope node. The class name is typically a simple string literal, but in dynamic languages it can be represented by a general expression. In this case, possible class name has to be estimated within the semantic analysis. Statically declared class members are collected and inserted into the class declaration directly. For dynamically added class members see III.B.2 (Variable use). Import statement - Phalanger adds non-standard keywords into the PHP syntax. Because of.net interoperability there is the "import namespace" statement. The namespace name follows and it is a string literal always, because this must be known during the compilation to generate MSIL code properly and to add correct assembly references into the generated code. Imported namespace is used and added into the current scope node. Therefore the AST is able to point to used namespaces, and following processing can use this information efficiently. III.B.2 Dynamic information In addition to statically defined symbols another tree nodes are added using the semantic analysis of the abstract syntax tree. Also some existing information can be modified. Particular expressions are scanned to find e.g. implicitly declared variables and dynamically added object members. Also possible variable values and estimated types are discovered. The so far built Abstract Syntax Tree is used for that. Process of mapping the Assignment Expression, containing Variable Usage, into the static declaration of used variable is depicted on Figure 3. The mapping collects known information from the expression and stores it into the static representation of the source code (AST). [declaration] A [member] B [type] type of C [value] value of C Figure 3 Mapping of an expression into the static AST. B Is Member Of A Assignment C

4 Following AST nodes (processed expressions) that describe dynamic features of the language are caught using the visitor pattern. Every node type defines corresponding action, that typically extends the known set of knowledge by adding static nodes into the AST. Assignment expression This expression is in the form of L-value is assigned from R-value (L=R). There are the basics of the type resolving in dynamic languages; the set of known information of the R-value is just added into the L-value knowledge. The L-value must be a variable use expression, which must be processed too and its declaration node in the tree must be found or created. The specific tree node describing the variable declaration corresponding to the L-value is described by the set of knowledge from every assignment operation (without duplicities). Include statement - The inclusion statement, and especially its attribute (included file name), has to be estimated. Mostly it is a simple string literal. The file name is updated by the estimated value. Variable use Variable Use expression construct consists of the whole member chain (a->b->c). There is the variable name and the IsMemberOf property which points to the parent object expression and which may be null. If it is null, the Variable Use expression describes some local variable (e.g. $xxx), otherwise the Variable Use expression is its member (e.g. some_parent_expression->xxx). o If it is a member (IsMemberOf is not null) - the IsMemberOf expression must be processed recursively first and the resulting tree nodes representing the parent object are used. Then the object members of the parent object with the name of the current Variable Use can be collected. If there is no such declaration, the new object member with the name of current Variable Use is added into the parent object declaration (dynamic member). Since now, this dynamically added object member can be presented like the static one. o If it is a variable - the corresponding declaration node is found in the tree, starting from the current code scope. If there is no such declaration node, the declaration node for new variable must be created and inserted into the current code scope. Return statement - The return statement is used to describe the corresponding function return value and its possible type. The expression within this statement is analyzed and the result set of known information is added into the function declaration node. It works like the assignment expression with the L-value of function declaration and R-value of the returning expression. Function call - Function calls are processed to gather possible information about input parameters. Input parameters are analyzed and the results are added into the local variables of called function matching the corresponding parameter name. III.B.3 Documentary comments The language parser can catch specially formatted comments above the function and class declaration. These comments are called documentary comments (see Figure 4 for an example). They can describe parameter types in dynamic languages. In PHP the notation of these comments come from the java doc comments. /** * Multiline description * used for symbol defined below * type description type description */ Figure 4 Documentary comment example. When the comments are found, the parser parses them and the result is placed into the corresponding AST node (as annotations). The symbol text description and optionally function return type and parameters description are extracted from the comment. The parameters type and return value type is used as a type hint. The dynamic declaration, which is typeless, seems then like static. III.C.NET assembly Assemblies (MSIL DLL) [12] are referenced by the application - it is possible to use the symbols defined in that assemblies in the dynamic language source code. Symbols from assemblies must be collected in the same way as the other symbols from source files are (types and their methods and properties). The mapping from.net assembly is simpler, because it contains static declarations only and following semantic analysis is not needed. Every referenced assembly is loaded, empty Abstract Syntax Tree is created and particular type declarations are mapped. The assembly is specified by its full assembly name; all assemblies referenced by the application must be processed, including the assemblies referenced by other referenced assemblies. [class] A.X [class] A.Y [class] B.X [namespace] A [namespace] B [class] X [class] Y [class] X Figure 5 Mapping of assembly classes into the AST. As a result there is one AST for every assembly, which contains type declarations managed in the hierarchical structure of namespaces. The result is depicted on Figure 5. It is because of faster search process and compatibility with the source code structure; all types (class declarations) within a specified namespace must be found quickly and in the same way. There may be thousands of class declarations in the assembly stored in an unstructured way, they are not divided into namespaces; every symbol is placed within its namespace declaration node in the AST.

5 III.C.1 Assembly processing In the resulting Abstract Syntax Tree there are only declaration nodes describing particular namespaces, types and their members. In the assembly there are no corresponding elements for the code scopes, also the position in the source code is ignored here. Since there are only static declarations, the process of AST creation consists of simple assembly load and listing its elements using the reflections. Following list describes the mapping from specified assembly elements: System.Type - These objects obtained from the assembly are converted into the type declaration nodes. Only public and visible types are used, the other ones are not accessible and they are not important for analysis. First of all the type namespace is used and the corresponding namespace declaration node is found or created. For the namespace e.g. A::B::C, there is the namespace declaration node "A" in the root of the tree, then its namespace member "B" and then "C" - the last one is used as a container for the types placed within A::B::C. Using the reflections the newly inserted type declaration node is analyzed and its members are populated. System.Reflection.MethodInfo - For this object the function declaration node is created and added into the parent type declaration node. Non public methods are ignored. In.NET (as a static language) every method has a return value and parameters list with the type specified. But there may be more than one method with the same name and different signatures - in this case the previously added function declaration node is found and its list of signatures is extended. System.Reflection.FieldInfo - FieldInfo is the named symbol containing a constant value placed within the type declaration. The declaration node is created and inserted into the parent type declaration node. The new declaration node can contain the field s value for consequent analyses. System.Reflection.PropertyInfo - Property is the named member of the type declaration which has the type specified and is able to get and/or set the value. The declaration node is created and inserted into the parent declaration as a member. The type of the property can be used to obtain possible property members. The created node has to remember if the set and/or get is allowed on the property. The property is mixed into the parent object child nodes list (like the fields or the methods are). System.Reflection.EventInfo - The event is a special object which holds the list of delegates (references to methods). It allows calling all of them within one event call. In C# there are overloaded operators (+= and -=) to adding and removing references into/from the event, however the PHP (and other dynamic languages) is not designed to work with events at all. Phalanger adds special methods to manipulate with the event object. Therefore the language parser and semantic analyzer have to do the same - the declaration node for the event is created and the member functions Add and Remove are added into its child nodes list in order to be able to see them in the static way. III.C.2 XML documentation Assemblies and descriptions of assembly elements are placed separately. Documentary comments of elements from a single assembly are stored in the.net XML documentation file. In order to offer the text description of every declaration node, XML files corresponding to.net assemblies must be loaded and processed. Then particular text descriptions of elements must be obtained from this XML file. III.D Native language symbols Built-in declarations of the PHP language must be present in a form of Abstract Syntax Tree too. They can be processed in the same way and integrated with user-defined symbols. Therefore there must be an extra tree which contains these declarations, mostly native functions such as "echo", "include", "isset", "empty" etc. This extra tree is included by all the PHP source files by default, thus the symbols defined here are visible everywhere. The extra AST must be created in order to present all built-in symbols. In addition there are also standard keywords which can be offered to the user in case of IntelliSense, when he's typing a word (auto-completion) together with the declarations. The keywords can be defined in the syntax tree too. III.D.1 Native symbols definition To create the extra Abstract Syntax Tree, the source database of built-in native symbols is needed. They are used to fill the tree; the process of tree filling (or initialization) depends on the form of the source representation. The best option is to use already defined mapping, e.g. from the PHP source code or.net assembly, then the extra tree creation would be easier. The list below describes possible options: Hardcoded - The built-in PHP declarations can be easily initiated in the parser program itself by creating corresponding declaration nodes directly; they are inserted into the empty extra AST root. This way is fastest, but in the context of programming it is not good-looking way. The program and data are mixed together. PHP source code - Built-in PHP declarations can be defined in an extra PHP source file. The existing mapping from the PHP code can be used to create AST containing these symbols. Existing symbols and their additional information can be easily changed and new symbols can be added without a need of the parser program code modifications..net assembly - The PHP compiler (Phalanger) defines most of the PHP native functions using the special.net assembly (Class Library). This assembly can be processed in the same way as the other assemblies are (III.C) through the reflections. All the built-in functions are placed within the predefined namespace. Therefore this namespace must be imported by all the source files by default. Documentation or another text representation - The amount of natively defined symbols can be obtained from some text representation, desired for human reader, such as online language documentation. Methods of web semantization or another data extraction processes may be used. The source text is parsed, e.g. using regular expressions, and specified

6 elements are collected. These elements can represent a PHP symbol declaration, corresponding symbol description or link to another source text. In this way big amounts of built-in declarations can be extracted from already existing documentation and they can be then kept updated. In the corresponding Phalanger project, declarations that are a part of the language syntax ("echo", "include", etc) are defined in the hardcoded way. The rest of declarations (standard PHP functions and types) are defined in the special.net assembly, because Phalanger libraries provide it. Therefore there are two extra IntelliSense trees containing standard PHP symbols. III.E Optimizing The process of single Abstract Syntax Tree mapping is performed every time the corresponding source code changes, hence it is suitable to implement some optimizations. Also there are trees containing symbols from large static language libraries, e.g. from.net assemblies, which manage typically thousands of symbols. Initializing all of them may be too slow. Some of additional information initialization can be postponed until it is requested, because not all of the source elements must be analyzed and mapped immediately. Also multi-core systems may be utilized and creation of tree nodes can be parallelized in some cases. Even then the database of symbols may not be ready immediately. Especially in case of large system libraries, collecting of all the symbols information takes some time. Still e.g. the language integration must handle user requests immediately; so the initialization process should be executed on a background thread and then serve only the information available in the current moment. III.E.1 Lazy mapping The process of mapping does not need to be done immediately. In fact a single tree node can remember its state and the origin source data element; then the mapping of child elements can be done on-demand and not before the content of the tree node is used. Only the top level tree nodes are initiated immediately and the rest of the tree can be created later. This works for static languages, in dynamic languages all the tree must be created in order to perform semantic analysis. Because the results of semantic analysis can affect other parts of the tree. In case of.net assemblies, only types are added into the tree, their members need not to be scanned immediately. The declaration node describing the type references the source System.Type object instance. When the type members are requested, the type declaration node initialization is finished first. Dynamic languages are typically able to modify local or global declarations everywhere. Dynamic members of global variables can be added within e.g. class member function call. Also global variables can be assigned here. Therefore the class content, its members and member function bodies have to be scanned and processed without postponing. But optionally theses cases might be ignored in order to speed up the process; the symbols information then will not be complete until the content of all the source elements is used and analyzed. III.E.2 Multi-threading Some tasks within the tree creation process may be performed independently. They can run on separate threads, so the process is parallelized. Especially different tree nodes can be created separately, as products of mapping source static elements. For the following semantic analysis of the dynamic language code, the complete tree is needed. Various tree elements are modified and additional information is added, that is then used for the rest of analysis. Its eventual parallelization will require many locks, and then it does not have the desired effect. IV CONCLUSION AND FUTURE WORK The presented paper describes the methods used for automatic analysis of the dynamic language source code. These methods are designed to work independently and to allow using of classic techniques of the static language syntax tree analysis. Proposed methods was successfully implemented within our project Phalanger. They are used for automatic syntactic and semantic prediction of the PHP language during the development process. Dynamic code is processed by mapping of dynamic constructs, and then usual techniques for text prediction in the static way are used. The semantic analysis works even for static languages too. It is able to gather useful information about the source code, such as possible values of variables or possible relations between objects. In case of dynamic languages we are able to estimate possible types of objects. Future enhancements of the designed methods would be able to work with more dynamic language constructs. Also it would be useful to formally specify what kind of semantic information can be automatically extracted from the dynamic language source code before the runtime. ACKNOWLEDGMENTS This work was supported in part by grants GAUK 2010/28910 and SVV REFERENCES [1] Phalanger compiler, compiler.net/ [2] Microsoft DLR, [3] IntelliSense, [4] Predictive text, [5] Tobias Lindahl, Konstantinos Sagonas: Practical Type Inference Based on Success Typings, Proceedings of PPDP, ACM, Venice 2006 [6] Joel Jones, "Abstract Syntax Tree Implementation Idioms", Pattern Languages of Programs 2003, Illinois [7] Abonyi A., Balas D., Beno M., Misek J. and Zavoral F.: Phalanger Improvements, Technical Report, Department of Software Engineering, Charles University in Prague, 2009 [8] M. Achour, F. Betz, A. Dovgal, N. Lopes, H. Magnusson, G. Richter, D. Seguy, J. Vrana: PHP: Hypertext Preprocessor, [9] Misek J., Zavoral F.: Syntactic and Semantic prediction in Dynamic Languages. SERA 2009, Springer SCI, 2009 [10] Misek J.: IntelliSense implementation of a dynamic language, Charles University in Prague, 2009 [11] Erik Meijer and Peter Drayton, Microsoft Corporation: Static Typing Where Possible, Dynamic Typing When Needed: The End of the Cold War Between Programming Languages [12] Duntemann J.: Assembly Language Step by Step. Wiley, 2000