CERN Web Application Detection

Transcription

1 CERN Web Application Detection Refactoring and release as open source software by Piotr Lizończyk Supervised by Sebastian Łopieński and Dr. Stefan Lüders Summer Students Programme 2015 Geneva, 28. August

2 Table of contents 1. Abstract Project specification What is Web Application Detection (WAD)? Original project goals Additional achievements Initial code assessment and refactoring Determining project usability for public audience Creating environment for code development Code refactoring Improving code maintenance Ensuring compatibility with Python Public release of Web Application Detection Splitting WAD into public and CERN-specific parts Setting up continuous integration Providing license and creating readme Wrapping code as a Python package Publishing the package on Python Package Index (PyPI) Integration with third party solutions Resignation from integration with OpenVAS Integrating WAD with w3af Integration with Kali Linux distribution Adding new features Multiple output formats Login to SSO-protected websites Detecting proxies Bugfixes after release Conclusion and outlook

3 1. Abstract This paper covers my work during my assignment as participant of CERN Summer Students 2015 programme. The project was aimed at refactoring and publication of the Web Application Detection tool, which was developed at CERN and priorly used internally by the Computer Security team. The range of tasks performed include initial refactoring of code, which was developed like a script rather than a Python package, through extracting components that were not specific to CERN usage, the subsequent final release of the source code on GitHub and the integration with third-party software i.e. the w3af tool. Ultimately, Web Application Detection software received positive responses, being downloaded ca times at the time of writing this report. 3

4 2. Project specification 2.1. What is Web Application Detection (WAD)? Web Application Detection is a website fingerprinting tool developed by the Computer Security team at CERN, that allows to scan websites and web servers in search for used technologies and software. The tool is based on an open-source browser extension called Wappalyzer, originally developed by Elbert Alias 1. It was used internally for years until the decision to make some of its parts public has been made. The tool is parsing HTTP responses received from a scanned target, in search for traces, that indicate usage of certain software. Detection results may contain details about the website, including, but not limited to operating system, web server, databases, content delivery networks, programming language, content management systems, frameworks, analytic tools and JavaScript libraries. Over 700 different technologies can be recognized and this number will only grow, as WAD uses the Wappalyzer s database, which is constantly extended. CERN employees have contributed a lot into creation of that database during the period of internal use Original project goals Evaluation of WAD s usage results led to the conclusion, that the tool is powerful and useful enough to be shared with the world-wide community. This was meant to let other people use it and at the same time contribute to it. This project was focused on making necessary modifications in order to make parts of WAD publicly available as open-source software and render it easily integrable with third party tools, such as OpenVAS and w3af vulnerability scanners Additional achievements During my work on the project, multiple goals that were not part of original scope were accomplished. I have successfully managed to refactor most of the codebase, making it more clean and extensible. I have also worked on automating tasks related to maintaining the project, i.e. updating the detection database and checking code correctness. Tests coverage has been improved and a continuous integration system has been set up. The documentation of the project was revamped and enhanced. Numerous features have been added, improving the overall usability of the tool. The public part of WAD now can be run using Python 2.6, 2.7, 3.2, 3.3 and

5 3. Initial code assessment and refactoring 3.1. Determining project usability for public audience The first task was focused on evaluating, whether this kind of project is desired to be publically available. The WAD s predecessor Wappalyzer had a Python wrapper, but it was only a simple script running the original JavaScript implementation over a retrieved website. There was no plain Python counterpart and it seemed to prevent Wappalyzer s usage, not as browser extension, but as a standalone tool or plug-in 2. This leads to another problem, which was the lack of actively developed free opensource website scanners. The only project that has been found was WhatWeb, whose development almost stopped in , while Wappalyzer keeps on receiving continuous support from the original author and many other developers. The reality of web technologies requires stable support, since those technologies evolve very quickly and software is becoming promptly outdated Creating environment for code development This project was moved from the AFS-contained Git repository to the online GitLab repository. Later on, a public respository was created on GitHub 4, which is the most popular storage for open-source projects nowadays. The continuous integration for WAD was set up using Jenkins CI 5. It was configured to automatically build and test the project s code on every commit push to the GitLab repository. Additionally, code quality checks using the pep8 and pylint tools are run and the results of those checks are available for the user to review. Finally, Jenkins builds generate JUnit test results, which allows the user to easily track and analyse failing tests Code refactoring WAD seemed to support Python versions as old as 2.4. I have decided to abandon support of Python older than 2.6, because less than 2% of Python users claim to regularly use those versions 6 and it would be virtually impossible to make the code compatible with Python 3 without doing this. This also allowed to reduce the set of package s dependencies, since some previously external libraries were included later into Python s standard library

6 Due to former WAD s script-like nature, the code wasn t ready to be reused and released as a Python package. It contained global variables for having data shared between files, executable code was contained in one file with tests and callable methods and it didn t follow any popular Python code convention. Those problems were resolved by: Moving all tests to a separate directory, with unique files for each tested Python source file; Using the singleton class for containing shared data in lieu of global variables; Applying pep8 rules to the project s source. Additionally, code that wasn t directly related to the scripts execution was extracted into separate files. Further refactoring was concentrated on improving the quality of code. I have created reusable classes with respect to object oriented programming rules. Many statements were simplified or rewritten to comply with a modern style of Python programming. I have also improved the code s test coverage and repaired failing tests. Some bugfixes had to be applied to resolve problems that were crashing WAD while it was periodically used at CERN Improving code maintenance Wappalyzer s database of detection rules is updated very often, but the procedure of this update in WAD used to be manual. A script was created, which automates that process by: 1. Running tests before the update to ensure proper functioning of code; 2. Checking, if local copies of database checking scripts and schemas are up to date; 3. Saving results of WAD run over given URLs (URL retrieval is automated with another script that was created); 4. Downloading the new rules database and validating it s correctness with respective scripts; 5. Running WAD s test suite using this new database; 6. Showing a difference comparison between databases and letting the user decide if he wants to continue with an update; 7. Re-running a scan over previously checked URLs and comparing the results. If the user approves the results of an update, he is presented with several commands that he can paste into his terminal in order to automatically commit and push changes. 6

7 3.5. Ensuring compatibility with Python 3 With the use of the six 7 package, it was possible to provide compatibility of WAD with Python ranging from version 2.6 to version 3.4. In most cases, the only change that had to be done, was changing Python 2 calls to modules to six wrappers. Unfortunately, I had to rewrite some code and introduce version checking conditionals to make certain functionalities work

8 4. Public release of Web Application Detection 4.1. Splitting WAD into public and CERN-specific parts Not every script that is contained in WAD was suitable to be made public many solutions usage was limited to the CERN web landscape and it would be just polluting the public repository. This led to a project repository division, one of which contains wrappers for WAD s usage at CERN and remains unpublished and the second one, consisting of the actual code of the detector and that is available on Github. The biggest challenge was to ensure compatibility of the newly created public repository with RPM packaging system, that was used to make WAD installable on CERN machines. This was achieved by using git submodules and modifying the makefile responsible for RPM creation. It was also important not to introduce any constraints on the published code that would be a result of existence of CERN-internal WAD wrappers. I have managed to do this by abstracting the interfaces and proper modularization of public code, so if changes have to be done for wrappers to comply with WAD s source, they can use subclasses changing single behaviours etc Setting up continuous integration While Jenkins builds consists of both CERN s WAD wrappers and public WAD code, it was necessary to configure continuous integration for the public part only. I have decided to use Travis 8, which is free for open-source projects and is widely used among them. Information about Travis build status is integrated as an image in WAD s readme, which in turn is shown on the project s main page Providing license and creating readme WAD uses parts of Wappalyzer s source, which is licensed under General Public License version 3. This obliged us to use a GPL compatible license and we decided to release WAD under the same license GPLv3. An extensive readme file was created, covering the project s description, installation procedure and usage details. It also elaborates on the differences between WAD and Wappalyzer. A changelog is integrated with readme, providing information on updates in the code. There was also an Authors file created, in order to gather and acknowledge people that have and will work on this project

9 4.4. Wrapping code as a Python package When moving code to the public repository, a proper directory structure has been created to ensure appropriate format of the Python package. A setup.py file has been created, that incorporates complete information about the package along with installation settings. This enables the package to be installed by setuptools or distutils. Setup.py contained console entry-points for the package, which means that after installation, it can be invoked by executing wad from a terminal, without the need to be located in the package s directory Publishing the package on Python Package Index (PyPI) Finally, the package was uploaded to the Python Package Index, which is a central repository for Python packages. Finishing this step made the project truly public and accessible, because from now on, it could be installed using the pip package manager, which is the default way of obtaining Python libraries and tools. 9

10 5. Integration with third party solutions The project s main goal was to integrate WAD as a plugin with commonly used network security tools Resignation from integration with OpenVAS Integrating WAD with OpenVAS seemed to be the most important task after the publication of code. Although there was no documentation on how to actually develop a new plugin for it, I tried to use a plugin template and already existing plugins to create such a WAD plugin. However, while trying to run OpenVAS with plugins with similar scanning functionality Wapiti and Nikta I ran into multiple problems. Searching for a solution, I have found this statement from Benoit Allard, OpenVAS developer In the meantime, I believe your best bet is to start wapiti manually, parse the logs yourself, and forget that you saw some kind of wapiti integration in OpenVAS... (Same apply for Arachni, and Nikta unfortunately).. This was an answer to a question about problems with running Wapiti and OpenVAS. It seems, that the OpenVAS team decided to move plugins with scanning capabilities to OpenVAS web interface, but this new approach wasn t completely ready to use. Considering the above mentioned reasons, there was no point in trying to create a plugin for OpenVAS. Without any documentation and with an incomplete interface for scanning plugins, a decision to develop this plugin might have lead this project into a dead end Integrating WAD with w3af Initially, I have contacted Andres Riancho, the original author of w3af, on the tool s discussion list. My suggestion of adding WAD as a plugin to w3af was warmly received and I have worked closely with the author during my development of plugin. In fact, addition of the Wappalyzer database to w3af was very desired 9 and WAD was a perfect solution to that need. I have created a merge request to the development branch of w3af and after resolving issues that arose during the creation of the plugin, it was approved to be integrated with next w3af release Integration with Kali Linux distribution Kali Linux, formerly known as Backtrack, is a Linux distribution focused on gathering pentesting tools and unique features that make it very actively used by people involved in network security. Because WAD will become a dependency of w3af, which is integrated into Kali Linux, the WAD will be packaged with Kali Linux and available to use out of the box

11 6. Adding new features After finishing the main goals of the project, there was still time for introduction of new features Multiple output formats I have added different output format for results: comma separated values, JSON and human readable output, which presents results in easy to read way. Output is done through implementing interface of abstract base class, so adding a new format in future will be simple Login to SSO-protected websites This is an improvement for the CERN wrapper, that enables it to use a user s Kerberos token or certificate in order to log into SSO-protected websites and scan them. A cookie required to log in is retrieved using the cern-get-sso-cookie tool, available in CERN s Linux repositories Detecting proxies The detection of proxies seemed to be useful addition to tthe CERN wrapper and it is accomplished by checking for Via HTTP header in the server s response Bugfixes after release After receiving feedback from users and following the development of the w3af plugin, I have fixed several bugs. The first one was rare case of crash, if WAD was ran on Python or newer, and the scanned website has invalid certificate. Another bug was an unexpected thread-unsafeness of code, which resulted in improper behaviour of the WAD plugin in w3af. The last bug was a false positive detection of Perl programming language, if the URL of a website ended with.pl, which is also the top-level domain for Poland. This was fixed by normalizing URLs that are parsed. 11

12 7. Conclusion and outlook Concluding my work on the project, I believe that the Web Application Detection tool is mature and complete software. During this summer, I have managed to incorporate every reasonable idea about extending this project that me and my supervisors had. After gathering feedback from people unrelated to this project, namely the w3af author and some project users, I have introduced minor improvements into the package. The future development of this project might focus on resigning from regex-based detection in favour of tools like BeautifulSoup or lxml. While the current approach works properly and was tested at CERN for a long time, further extensions to the code, along with refactoring, might be achieved by moving to aforementioned libraries. Thanks to being open-source, the development of the public part of WAD might in future rely on contributions from non-cern developers, including the author of this report. The project gathered positive outcome, being starred 12 times on GitHub and downloaded more than 1500 times from PyPI, which is hopefully a good sign for a future success of WAD. Further increase of interest in this project might be achieved with WAD s integration into Kali Linux. 12