Static Code Analysis in ATLAS. Andrew Washbrook University of Edinburgh Common discussion about software quality analysis 23rd November 2016

Transcription

1 Static Code Analysis in ATLAS Andrew Washbrook University of Edinburgh Common discussion about software quality analysis 23rd November 2016

2 Coverity Commercial static code analysis (and security testing) tool CERN-wide license available Any member of the ATLAS collaboration can view detected code defects though a hosted Coverity Connect platform Split software into several domains Universal case ID (CID) for references across software updates Triaging within platform Coverity defined impact levels Source code navigation

3 Coverity Defect Handling in ATLAS Our coverity static analysis runs twice a week and takes longer than 24 hours to fully complete We do not run in delta mode: each report reflects the entirety of the defect base Coverity weekly processing schedule Coverity defects timeline SOAP interface to extract XML results Also possible to get the results with a one line curl command Combine results with the developer database to allocate the defects Command line tool issues queries web page and lists issues for a specified file

4 Cppcheck Open source tool that performs static analysis of C++ code Focus on bugs instead of stylistic issues Does not detect syntax errors More lightweight than Coverity but may be prone to more false positives. List of checks Error and warning statistics per domain High, Medium, Low Runs twice per week on the ATLAS nightly development release More verbose but picks up on defects not seen by Coverity Can be run by developers across individual packages Selected warnings can be suppressed on request

5 Lizard Recently explored the use of the Lizard code complexity analyzer tool Provides a generic quality indicator to avoid focusing on just fixing Coverity defects Calculates how complex the code 'looks' rather than how complex the code really 'is' Requires syntactically correct code Word cloud generation Provides threshold reporting on the following key code quality metrics: The nloc (lines of code without comments) CCN (the Cyclomatic Complexity Number) Token count of functions Parameter count of functions Cyclomatic Complexity counts the number of paths program flow can take through a given section of code Complexity Number >40 Meaning Structured and well written code High Testability Complex Code Medium Testability Very complex Code Low Testability Not at all testable Very high Cost and Effort Cyclomatic Complexity

6 Lizard Results for ATLAS software Defect summary per domain based on thresholds for CC and NLOC values Number of lines of code without comments per function for each project Cyclomatic Complexity for each Project

7 Code Beautifying: Uncrustify /afs/cern.ch/atlas/offline/external/uncrustify/atlas.cfg Very customisable Aim for consistency, not uniformity Commit fixes/features and code cosmetics separately

8 Other Evaluated Tools IWYU Analysis of #include statements in C and C++ source files Aim to remove superfluous #includes by determining only what is needed and replacing #includes with forward-declares where possible Used to generate IWYU reports for each project in web frontend TCtoolkit Provides: Code Duplication Detector Token Tag Cloud Class Cooccurance matrix (CCOM) Treemap Visualization for Source Monitor Metrics data Noisy: 1 hr to scan ATLAS software release and produces >50Mb of html IWYU web frontend Sanitizer Tools LLVM based tools: AddressSanitizer, LeakSanitizer, ThreadSanitizer, MemorySantizer Need to modify our build process to take full advantage of the tools

9 Future Effort Software Quality Tool Evaluation Look at usefulness of other tools such as clang-tidy, clang-format, gcc checker plugins, OCLint (and many others) Evaluate security-oriented static analysis tools available through CERN (see CERN Computer Security page) CppLint, FlawFinder, RATS, VCG Code Coverage All our important code should have excellent test coverage Can use --coverage compiler option and the gcov tool to quantitatively identify coverage Prioritise coverage by tagging domains, packages and code sections as critical Software Infrastructure Evolution Identify any possible software quality plugins and services in the Git/Jenkins ecosystem e.g. build errors and warnings, tracking package activity, code ownership and provenance Analytics Generate an overwhelming set of data to assess software quality Look at analytics tools to provide visualisation and data mining capabilities Static code analysis results drill down Git statistics monitoring Test coverage indicators

10 Econometric and Ecology Analysis Methods How are econometric values calculated? Are there universal bounds of acceptability for metric output (e.g CBO value)? How easily could this be applied to ATLAS software? Would the application of these methods to ATLAS software be a direct extension of your CHEP studies? Are there any calculation scaling issues with increasing lines of code (and class dependencies)? Are there other laws/tests more suitable than others for our case? How repeatable are the tests? How are metric values interpreted and translated into meaningful actions for software package maintainers?