Runtime Application Self-Protection (RASP) Performance Metrics

Transcription

1 Product Analysis June 2016 Runtime Application Self-Protection (RASP) Performance Metrics Virtualization Provides Improved Security Without Increased Overhead Highly accurate. Easy to install. Simple to operate. Nollaig Heffernan Director of Product Delivery

2 Table of Contents Executive Summary 3 Introduction 3 Defining and Measuring Performance 3 How Waratek Works 4 Performance Test Environment 5 Application Under Test 5 Workload Characteristics 5 Test Methodology 5 Test Environment Setup and Tuning 7 Results 8 Conclusion 10 2

3 Executive Summary It is widely accepted that all application runtime security controls introduce a performance overhead when providing protection for an application. In this test case, Waratek s Runtime Application Self Protection (RASP) technology was tested to measure the performance overhead it introduced while providing protection for an application. This paper demonstrates that the overhead of this solution is less than 3% when considering response time and throughput. Furthermore, when an application is under load with 12% attack traffic, memory consumption is improved by 4% compared to running the same Application on a standard unprotected JVM. Introduction In providing protection for applications it is important to understand the cost in terms of performance. Waratek s application security solution is a completely in-process technology that provides highly accurate detection and blocking of malicious application attacks at runtime. Waratek s solution protects applications from sophisticated attacks without having to install network devices or make code changes. Waratek embeds security into the Java run-time layer, not in the app itself. Pre-defined security rules immediately identify and optionally block compromised application behavior with minimal performance impact. A member of the RASP category of solutions, Waratek is uniquely able to secure enterprise applications because it possesses complete information about an application s behavior at a level of granularity network devices cannot. And, because it is tightly wrapped around the application, it seamlessly supports public or private cloud deployments. Defining and Measuring Performance Application architecture and dependences determines the way in which requests are processed. There are a number of factors like application load, file and network IO, and other application and system factors that impact application responsiveness. It is the characteristics of response that define performance. When considering the performance of an application we consider two key metrics here. Response time: This is the most widely used metric of performance and it is simply a direct measure of how long it takes to process a request. Throughput: A straightforward count of the number of requests that the application can process within a defined time interval. 3

4 For Web applications, a count of page impressions or requests per second is often used as a measure of throughput. Resource consumption is also an important consideration, even more important for applications deployed in elastic environments, such as the cloud. With these larger scale and geographically dispersed applications, scalability and performance become inextricably linked that can most easily be described from a resource-centric point of view. The purposes of this test we measure memory consumption for an application protected and unprotected when under load. How Waratek Works Ø Waratek is a simple.jar plug-in that, once installed, creates a secure, virtual container around your entire application stack. Ø A Java Container is a protected in-jvm container with built in application security and quarantine controls. Ø The Java container separates apart the vulnerable JRE code (where the insecure Java APIs reside) from the low-level JVM (the JIT compiler and GC). Ø Application security controls inserted between the Java Container and the JVM protect and quarantine the Java application. Business Logic 3 rd Party Components Platform & Java APIs Ø Every Java container is a completely isolated abstraction of a real JVM. Ø Each Java container has its own classpath, classloaders, namespace, environment variables, heap allocation and resource quotas. Ø All code running inside a Java container is essentially trapped and emulated in the same way as a hardware Virtual Machine Monitor. Ø Every container (guest) has its own JRE version that can be different from the host JVM version. 4

5 Performance Test Environment Application Under Test For the purposes of this test we used a purpose-built vulnerable web application called Spiracle. It can be accessed at This application is used to test and demonstrate Web Application security technologies and provides a good and comprehensive example of everyday requests that can be used to exploit any application. This application will run on any compatible Java EE Application Sever including Apache Tomcat, JBoss, Weblogic and WebSphere, etc. It also supports Oracle, MySql and MS SQL databases. Workload Characteristics Apache JMeter was used as the workload driver to send requests to the system under test and to measure the key metrics captured during the test phase: Response Time and Throughput. The workload was made up of 50 http requests. The Thread Group was composed with 20 threads to simulate 20 concurrent users where each thread performs 30 iterations. This makes a total of (20 x 50) x30 = 30,000 samples. We define this as 1 run. Within each run there are 6 malicious http requests that cover some of the most severe critical vectors as described in the OWASP and SANS Top Five. This represents 12% malicious attack traffic, which would be considered very high for normal application traffic. Test Methodology When determining the performance impact or overhead of the Waratek RASP solution, we re measuring the performance difference between running an application unprotected, vulnerable to exploits versus running protected by Waratek s RASP solution. To achieve a direct comparison, we run the test application on an unprotected standard Java Virtual Machine and also run the same application on Waratek s RASP solution under identical test conditions. As a base lined control, we execute the workload on a standard Hotspot Java Virtual Machine with no protection in place. We then compare the response time and throughput between the application running on a standard JVM and that same application under identical conditions when running protected on the Waratek Security solution. The first loop of the test iteration is a warm-up loop, which engages and warms up the compiler on both JVM s and is not considered part of the test results. Memory consumption was monitored and logged using pidstat. The pidstat command is used for monitoring individual tasks currently being managed by the Linux kernel. 5

6 Memory usage was logged at 3 second intervals from the point where the tomcat application server had started to where a run had completed. Note: On start-up a run was immediately kicked off. Fig.1 Application Stack Comparison, Waratek - Hotspot For the purposes of this test, we use a standard Hotspot Java Virtual Machine version 1.7. The Application will run in an unprotected state on this JVM. The Waratek solution is the bundled deployment option; see Option 2 above under Deployment Options. Spiracle will run on a version 1.7 Hotspot in the Guest Container. The Host Virtual machine of the bundled RASP solution is Hotspot 1.8. Note: this is an apples-to-apples comparison with the app running on identical Hotspot Virtual machine versions. Test Environment Setup and Tuning Apache JMeter is running on a client desktop machine with a default configuration. The JMeter memory allowance was set to Xms 512m Xmx512m. The System under test where the test application and Waratek s RASP solution are running is hosted on a Server instance deployed on a separate internal network. The Oracle database is also running on the same server instance. The Apache Tomcat application server had its memory allowance set to 1G i.e. -Xms1G Xmx1G. In addition, the G1 Garbage Collector was set as the primary Garbage Collector. The Garbage First Garbage Collector (G1 GC) is the low-pause, server-style generational garbage collector for Java Hotspot VM. -XX:+UseG1GC. Using the G1 C will allow us to see more predictable behaviour in the applications behaviour. No other tuning options were selected. 6

7 Fig 2. Test System - Environment Setup 7

8 Results Response Time (milliseconds) Overhead Runs Waratek (m/s) Hotspot (m/s) Average % 8

9 Throughput (Requests per second) - Overhead Runs Waratek (requests/sec) Hotspot (requests/sec) Average % 9

10 Memory (RSS) Consumption - Overhead Runs Waratek (Kb) Hotspot (Kb) Average % Conclusion Waratek s RASP solution is a completely in-process technology that provides highly accurate detection and blocking of malicious application attacks at runtime. As a result of implementing runtime security protections through virtualization, this analysis has demonstrated that the performance overhead associated with providing accurate protection from malicious attack is negligible. Another key finding here; we can observe the benefits of running a legacy application on the Waratek solution whereby a legacy app benefits from the improvements and optimizations within the most current up-to-date version of Java (i.e. a Java 7 guest Application running on a Java 8 host JVM). This is specifically apparent in the memory consumption of the legacy application when under load compared to the baseline standard. 10