Graph-based analysis of JavaScript source code repositories

Size: px

Start display at page:

Download "Graph-based analysis of JavaScript source code repositories"

Willis Gray
6 years ago
Views:

1 Graph-based analysis of JavaScript source code repositories Gábor Szárnyas Graph Processing FOSDEM 2018

2 JAVASCRIPT Latest standard: ECMAScript 2017

3 STATIC ANALYSIS Static source code analysis is a software testing approach performed without compiling and executing the program itself. Static analysis Codacy, CodeClimate, etc. Development Version Control System Unit and integration tests Compilation

4 STATIC ANALYSIS TOOLS C o lint -> linters Java o FindBugs o PMD JavaScript o ESLint o Facebook Flow o Tern.js o TAJS

5 PERFORMANCE CONSIDERATIONS Checking global rules is computationally expensive Slow for large projects, difficult to integrate even to CI Unit tests Code analysis Workaround #1: no global rules (ESLint) Workaround #2: batching (e.g. 1/day) Unit tests Code analysis Workaround #3: custom algorithms (e.g. Flow)

6 PROJECT GOALS Goal Static analysis for JavaScript applications Design considerations Custom analysis rules o Both global and local o Extensible High-performance o real-time responses

7 ARCHITECTURE AND WORKFLOW

8 PROPOSED APPROACH Design considerations Custom analysis rules High-performance 1. Δ Approach Use a declarative query language Use incremental processing o in lieu of batch execution o file-granularity o maintain results analyzer

9 ARCHITECTURE Main.js Dependency.js Fiterator.js Parser.js ++ VCS Workspace discoverer +--- ChangeProcessor.js +--- CommandParser.js FileIterator.js +--- iterators DepCollector.js FileDiscoverer.js InitIterator.js +--- Main.js +--- whitepages +--- ConnectionMgr.js DependencyMgr.js. Abstract Syntax Tree Abstract Semantic Graph Analysis rules Analysis server Graph database <!> <?> Validation report <.> Client

10 CODE PROCESSING STEPS CODE forráskód code tokenizer tokenek tokens a sequence of statements: parser AST var foo = 1 / 0 scope analyzer ASG

11 CODE PROCESSING STEPS TOKENS code tokens: the shortest meaningful character sequence tokenizer tokens parser AST scope analyzer ASG var foo = 1 / 0 Token Token type var VAR (Keyword) foo IDENTIFIER (Ident) = ASSIGN (Punctuator) 1 NUMBER (NumericLiteral) / DIV (Punctuator) 0 NUMBER (NumericLiteral)

12 CODE PROCESSING STEPS AST Abstract Syntax Tree o Tree representation of o the grammar structure of tokenizer parser code o sequence of tokens. tokens AST items declaration declarators binding BindingIdentifier name = "foo" Module VariableDeclarationStatement VariableDeclaration VariableDeclarator init BinaryExpression operator = "Div" scope analyzer ASG left right LiteralNumericExpression value = 1.0 LiteralNumericExpression value = 0.0

13 CODE PROCESSING STEPS ASG Abstract Semantic Graph o Not necessarily a tree o Has scopes code & semantic info tokenizer o Cross edges parser tokens astnode GlobalScope Module children items items variables declaration declarators declaration declarators references binding declarations init binding Module init AST scope analyzer leftnode right left right ASG

14 AST VS. ASG var foo = 1 / 0 1 LOC -> 20+ nodes

15 PATTERN MATCHING Declarative graph patterns with Cypher VariableDeclarator binding BindingIdentifier name = "foo" be BinaryExpression operator = "Div" Match result MATCH (binding:bindingidentifier) LNExpression value = 1.0 right LNExpression value = 0.0 <-[:binding]-()--> (be:binaryexpression) -[:right]->(right:lnexpression) WHERE be.operator = 'Div' AND right.value = 0.0 RETURN binding

16 WORKFLOW Version control system source code tokenizer Developer s IDE tokens parser AST transformation graph database scope analyzer ASG traceability Git, Visual Studio Code ShapeSecurity Shift Java, Cypher Neo4j

17 USE CASES TYPE INFERENCING function foo(x, y) { return (x + y); } function bar(a, b) { return foo(b, a); } var quux = bar("goodbye", "hello"); Source:

18 USE CASES GLOBAL ANALYSIS Reachability: dead code detection async/await (ECMAScript 2017) potential division by zero

19 TECH DETAILS

20 IMPORTS AND EXPORTS

21 FIXPOINT ALGORITHMS Lots of propagation algorithms Run to completion scheduling o Mix of Java code and Cypher

22 EFFICIENT INITIALIZATION Initial build of the graph with Cypher was slow Generate CSV and bulk load Two files: nodes, relationships $NEO4J_HOME/bin/neo4j-admin import --database=db --nodes=nodes.csv --relationships=relationships.csv 10 speedup

23 REGULAR PATH QUERIES Transitive closure on certain combinations Workaround: o Start transaction o Add proxy relationships o Calculate transitive closure o Rollback transaction A B * opencypher proposal for path patterns (:A)-/[:R1 :R2 :R3]+/->(:B)

24 INCREMENTAL QUERIES

25 OPENCYPHER SYSTEMS The opencypher project aims to deliver a full and open specification of the industry s most widely adopted graph database query language: Cypher. (late 2015) Research prototypes o Graphflow (Univesity of Waterloo) o ingraph (incremental graph engine) incremental processing (Source: Keynote GraphConnect NYC 2017)

26 FOSDEM 2017: INGRAPH

comprehensions J. Marton, G. Szárnyas, D.

28 STATE OF INGRAPH IN 2018 Cover a substantial fragment of opencypher o MATCH, OPTIONAL MATCH, WHERE o WITH, functions, aggregations o CREATE, DELETE Features on the roadmap o MERGE, REMOVE, SET o List comprehensions J. Marton, G. Szárnyas, D. Varró: Formalising opencypher Graph Queries in Relational Algebra, ADBIS, Springer, 2017 G. Szárnyas: Incremental View Maintenance for Property Graph Queries, SIGMOD SRC, 2018

29 RELATED PROJECTS

30 Dirk Mahler, Pushing the evolution of software analytics with graph technology, Neo4j blog, 2017 JQASSISTANT Code comprehension: software to graph

to provide an easy-to-use in-depth insight of your

31 SLIZAA slizaa uses Neo4j/jQAssistant and provides a front end with a bunch of specific tools and viewers to provide an easy-to-use in-depth insight of your software's architecture. Gerd Wütherich, Core concepts, slizaa

32 SLIZAA: ECLIPSE IDE

33 SLIZAA: XTEXT OPENCYPHER Xtext-based grammar Used in the ingraph compiler Now has a scope analyzer Works in the Eclipse IDE and web UI

34 WRAPPING UP

35 PUBLICATIONS Dániel Stein: Graph-based source code analysis of JavaScript repositories, Master s thesis, 2016 Soma Lucz: Static analysis algorithms for JavaScript, Bachelor s thesis, 2017

36 CONCLUSION Some interesting analysis rules require a global view of the code Good use case for graph databases o Property graph o Cypher language Very good use case for incremental queries o Incrementality on multiple levels

37 RELATED RESOURCES Codemodel-Rifle ingraph engine Shape Security s Shift parser Slizaa opencypher Xtext github.com/ftsrg/codemodel-rifle github.com/ftsrg/ingraph github.com/shapesecurity/shift-java github.com/slizaa/slizaa-opencypher-xtext Thanks to Ádám Lippai, Soma Lucz, Dániel Stein, Dávid Honfi and the ingraph team.

38 Ω

39 VISUAL STUDIO CODE INTEGRATION Language Server Protocol (LSP) allows portable implementation

40 USE CASES Control Flow Graph o graph representation of o every possible statement sequence Basis for type inferencing and test generation statement statement if statement statement statement statement CFG error condition done

András Zsámboki, Gábor Szárnyas, József Marton

opencypher TCK András Zsámboki, Gábor Szárnyas, József Marton TCK Overview opencypher TCK The project aims to deliver four types of artifacts: 1. Cypher Reference Documentation 2. Grammar specification