ShiftLeft Real-World Runtime Protection Benchmarking
Table of Contents Executive Summary... 02 Testing Approach... 02 ShiftLeft Technology... 04 Test Application... 06 Results... 07 SQL injection exploits (OWASP A1)... 08 Insecure Deserialization exploit (OWASP A8)... 09 Using Components with Known Vulnerabilities: Jackson-Databind Deserialization Exploit (OWASP A9)... 10 HTTP data as file path (OWASP A5)...11 Sensitive Data breach to file (OWASP A3)...12 Conclusion...13 1
EXECUTIVE SUMMARY In the industry s first such test, ShiftLeft subjected its application protection capabilities to the expertise of real-world penetration testing from Cobalt Labs (a leader in penetration testing). During the 14-day penetration test, ShiftLeft was able to detect and block all attempted exploits against a vulnerable test application. This included protection for vulnerabilities such as SQL injection, Java deserialization, and sensitive data leakage, among many others. The results demonstrated that ShiftLeft s unique approach protected the comprehensively application (zero false negatives) while minimizing operational overhead (minimal false positives). Key highlights 1. 2. 3. ShiftLeft blocked all exploits to all the six (6) OWASP Top 10 vulnerabilities present in the test application. The vulnerabilities include A1-SQL Injection, A2-Broken Authentication, A4-XML External Entities, A5-Broken Access Control, A8-Insecure Deserialization, and A9-Known Vulnerabilities. The test application also had an A3-Sensitive Data Exposure vulnerability 1, which was inherently out of scope for external penetration testing. ShiftLeft did detect and protect against exposure of such data. TESTING APPROACH ShiftLeft is the industry s first solution that uses modern code analysis to automatically build a runtime-protection security profile. ShiftLeft s Code Property Graph (CPG) extracts the application s Security DNA to automatically create a custom security profile. The security profile is then enforced, with surgical precision, by ShiftLeft s Microagent in production to protect the application. ShiftLeft s security profile is based on the CPG s ability to analyze the application s entire composition, including: custom business code 3rd party libraries and SDKs OSS libraries OSS frameworks. In order to benchmark ShiftLeft s runtime protection in the most real-world possible scenario, 1 As traditional penetration testing is done from a black-box perspective, most data leakages (barring leaks on web sockets) are impossible to verify from external perspective. Data leaks may happen to logs, DB, files, etc., all of which are not accessible to an external penetration tester. 2
we constructed a testing methodology to mimic the adversarial nature of a hacker. Usually, the testing of a security solution is done in a controlled lab setting where the device-under-test (DUT) is subjected to traffic from well-known penetration tools. Such testing is not comprehensive, as it 1) focuses on measuring efficacy against known attacks without understanding the actual application attack surface, and 2) allows vendors to pre-tune their signature database to provide favorable results during the test. The approach used for this test, hiring white-hat hackers, most closely mimics the real-world scenario where hackers bring all their tools and their know-how to compromise an application. The approach used for this test, therefore, provides the most realistic assessment of the efficacy of a security solution to protect an application. The testing methodology started by developing an application in isolation with numerous OWASP Top 10 vulnerabilities embedded into it. Next, two instances of the application were created. One instance was hosted without any security protection. Another instance was instrumented and protected with ShiftLeft by a second team that had no knowledge of how the application was developed or how it was vulnerable. Finally, Cobalt Labs performed a 14-day penetration test against both applications. Cobalt Labs had 3 white-hat hacking experts attack both applications with any and all tools and methods. 3
SHIFTLEFT TECHNOLOGY ShiftLeft s technology is a unique combination of modern code analysis and runtime security that is customized for every application and all subsequent releases. ShiftLeft s modern code analysis extracts an application s Security DNA from its CPG, which is a detailed graphical representation of security posture based on the application s source code.the Security DNA of an application is used to identify software vulnerabilities and sensitive data flows by combining data flow, control flow, and dependence flow analyses. It also analyzes third-party and OSS dependencies separately from custom code. Furthermore, ShiftLeft s Policy Engine understands which variables are sensitive in order to map critical data flows and identify mishandling and external leakage. Thus, Security DNA defines how an application is vulnerable, whether via known vulnerabilities, unknown vulnerabilities, or data mishandling/leakage situations. THE CPG PYRAMID Service Dependency Graph Component graph - Application, Dependencies Security critical information flows Methods, Types, Call Graph, Type Hierarchy Instruction level -Syntax, control flow, data-flow semantics 4
ShiftLeft s runtime security is enforced by the Microagent that runs inside the application. The Microagent enforces policies, defined in a security profile, that are automatically created from the application s Security DNA to block exploit attempts. Derived from source code and only looking for vulnerable routes, the Microagent is surgical in its precision. Apart from the performance benefits of surgical precision, the Microagent also helps developers prioritize real positive vulnerabilities for the fix cycle. 5
TEST APPLICATION The test application is a simple REST-based multi tenant application emulating the functions of a retail-banking interface, including routes. The application was built with examples of seven (7) of the relevant OWASP Top 10 vulnerabilities embedded into it. Hence, if the penetration testing team were able to breach runtime protection, the application would be exploitable. Test Application Routes GET GET POST POST POST POST POST /account /account/:id /account /checkaccount /checkaccountsimple /account/:id/deposit /account/:id/withdraw POST /account/:id/addinterest GET GET PUT DELETE GET GET GET POST /rawcustomersbyname/:firstname /customers/:id /customers/:id /customers/:id /customers /createcustomer /customersxml /customers In order to make the application more vulnerable, it was built without support for any authorization or authentication scheme. All endpoints specified above can be exercised by any user. Hence, Cross-site Request Forgery (CSRF) and resulting Cross-site Scripting (XSS) were inherently out of scope. Following vulnerabilities are present in the application: Vulnerability category A1-Injection A2-Broken Authentication (*) A3-Sensitive data exposure A4-XML External Entities A5-Broken Access Control A8-Insecure Deserialization A9-Known Vulnerabilities Type SQLi Appropriate cookie protection Leaking data to a file XXE Path traversal Java deserialization A known deserialization issue 6
RESULTS The Cobalt Labs team was able to identify and exploit all of the vulnerabilities present in the unprotected version of test application except for A3-Sensitive Data Exposure, which is inherently out of scope for the external penetration testing they performed. However, during the 14-day testing period, Cobalt Labs was unable to exploit any of the same vulnerabilities in the application protected by ShiftLeft. Cobalt Labs was able to identify the presence of vulnerabilities, but ShiftLeft was able to block their exploit attempts. OWASP Category Vulnerability Type Endpoint Unprotected Application Protected Application Identified Exploited Identified Exploited A1-Injection SQLi /rawcustomersbyname A2-Broken Authentication A4-XML External Entities A5-Broken Access Control A8-Insecure Deserialization A9-Known Vulnerabilities Appropriate cookie protection XXE Path traversal Java deserialization Known OSS vulnerability /admin /customersxml /savesettings /check /checkfast YES YES YES NO YES YES YES NO YES YES YES NO YES YES YES NO YES YES YES NO YES YES YES NO 7
SQL INJECTION EXPLOITS (OWASP A1) ShiftLeft detected and blocked all attempts to exploit the SQL injection vulnerability. There were 6.5K (thousand) SQL injection exploit attempts as a result of fuzzing and all of them were blocked. ShiftLeft also showed the exact line of vulnerable code as well as all the associated flows that were being targeted. This information is extremely valuable for the developers, allowing them to quickly understand why the vulnerability is present and what they can do to fix it. 8
INSECURE DESERIALIZATION EXPLOIT (OWASP A8) The ShiftLeft agent blocked all attempts to exploit the java deserialization vulnerability in the test application. A total of 25 such incidents were detected and blocked in real time. As in the previous case, ShiftLeft identifies the actual line of code and all the associated flows in the test application causing the vulnerability. 9
USING COMPONENTS WITH KNOWN VULNERABILITIES : JACKSON DATABIND DESERIALIZATION EXPLOIT (OWASP A9) ShiftLeft, unlike most SAST or IAST solutions, is able to look at third-party code, identify vulnerabilities, and block attempts to exploit such solutions. In this case, ShiftLeft was able to block attempts to exploit a deserialization vulnerability in the Jackson data-bind library (OSS dependency) included as part of the core application. As in the case of other vulnerabilities, ShiftLeft also identified the line number of the code with the actual vulnerability. 10
HTTP DATA AS FILE PATH (OWASP A5) The ShiftLeft agent, as part of this penetration test, was able to identify and block attempts to exploit a broken-access-control vulnerability present in the test application. ShiftLeft also identified the line number of the code where this vulnerability is present in the application. 11
SENSITIVE DATA BREACH TO FILE (OWASP A3) The ShiftLeft agent is also able to discover numerous sensitive data breaches that are triggered by inadvertent or mal-intentioned code execution. In this case, penetration testers inadvertently executed some code path where personal data was leaked to file amongst many others. Following is an example for sensitive data leakage 12
CONCLUSION Traditional application security is broken into two silos: Application Security Tools (AST) Static Application Security Testing Dynamic Application Security Testing Interactive Application Security Testing Runtime protection tools Web Application Firewall Runtime Application Self Protection Next Generation Firewal Source Code Composition Analysis Each silo creates numerous false positives, which slow down the organization and increase the operational overhead of protecting applications. ShiftLeft is the first company to bring together knowledge of development and production environments via automated workflows that integrated with DevOps accelerate the pace of execution and innovation. As this test demonstrates, understanding source code is the best way to automatically protect an application is runtime, while ensuring comprehensive coverage and operational efficiency of both talent and compute resources. 13