Benchmarking Vulnerability Detection Tools for Web Services

Size: px

Start display at page:

Download "Benchmarking Vulnerability Detection Tools for Web Services"

Mark Parks
6 years ago
Views:

1 Benchmarking Vulnerability Detection Tools for Web Services, Marco Vieira {nmsa, ICWS 2010 CISUC Department of Informatics Engineering University of Coimbra, Portugal

2 Outline The problem Benchmarking Approach Benchmark for SQL Injection vulnerability detection tools Benchmarking Example Conclusions and Future Work 2

3 Web Services Web Services are becoming a strategic component in a wide range of organizations Web Services are extremely exposed to attacks Any existing vulnerability will most probably be uncovered/exploited Hackers are moving their focus to applications code Both providers and consumers need to assess services security 3

4 Common vulnerabilities in Web Services 300 Public Web Services analyzed 4

5 Vulnerability detection tools Vulnerability Scanners Easy and widely-used way to test applications searching vulnerabilities Use fuzzing techniques to attack applications Avoid the repetitive and tedious task of doing hundreds or even thousands of tests by hand Static Code Analyzers Analyze the code without actually executing it The analysis varies depending on the tool sophistication Provide a way for highlighting possible coding errors 5

6 Using vulnerability detection tools Tools are often expensive Many tools can generate conflicting results Due to time constraints or resource limitations Developers have to select a tool from the set of tools available Rely on that tool to detect vulnerabilities However Previous work shows that the effectiveness of many of these tools is low How to select the tools to use? 6

7 How to select the tools to use? Existing evaluations have limited value By the limited number of tools used By the representativeness of the experiments Developers urge a practical way to compare alternative tools concerning their ability to detect vulnerabilities The solution: Benchmarking! 7

8 Benchmarking vulnerability detection tools Benchmarks are standard approaches to evaluate and compare different systems according to specific characteristics Evaluate and compare the existing tools Select the most effective tools Guide the improvement of methodologies As performance benchmarks have contributed to improve performance of systems 8

9 Benchmarking Approach Workload: Work that a tool must perform during the benchmark execution Measures: Characterize the effectiveness of the tools Must be easy to understand Must allow the comparison among different tools Procedure: The procedures and rules that must be followed during the benchmark execution 9

10 Workload Services to exercise the Vuln. Detection Tools Domain defined by: Class of web services (e.g., SOAP, REST) Types of vulnerabilities (e.g., SQL Injection, XPath Injection, file execution) Vulnerability detection approaches (e.g., penetrationtesting, static analysis, anomaly detection) Different types of workload can be considered: Real workloads Realistic workloads Synthetic workloads 10

11 Measures Computed from the information collected during the benchmark run Relative measures Can be used for comparison or for improvement and tuning Different tools report vulnerabilities in different ways Precision Recall F-Measure 11

12 Procedure Step 1: Preparation Select the tools to be benchmarked Step 2: Execution Use the tools under benchmarking to detect vulnerabilities in the workload Step 3: Measures calculation Analyze the vulnerabilities reported by the tools and calculate the measures. Step 4: Ranking and selection Rank the tools using the measures Select the most effective tool 12

13 A Benchmark for SQL Injection V. D. tools This benchmark targets the domain: Class of web services: SOAP web services Type of vulnerabilities: SQL Injection Vulnerability detection approaches: penetration-testing, static code analysis, and runtime anomaly detection Workload composed by code from standard benchmarks: TPC-App TPC-W* TPC-C* 13

14 Workload Benchmark Service Name Vuln. Inputs Vuln. Queries LOC Avg. C. Comp. TPC-App TPC-C TPC-W ProductDetail NewProducts NewCustomer ChangePaymentMethod Delivery NewOrder OrderStatus Payment StockLevel AdminUpdate CreateNewCustomer CreateShoppingCart DoAuthorSearch DoSubjectSearch DoTitleSearch GetBestSellers GetCustomer GetMostRecentOrder GetNewProducts GetPassword GetUsername Total

15 Enhancing the workload To create a more realistic workload we created new versions of the services This way, for each web service we have: one version without known vulnerabilities one version with N vulnerabilities N versions with one vulnerable SQL query each This accounts for: Services + Versions Vuln. Inputs Vuln. lines

16 Step 1: Preparation The tools under benchmarking Provider Tool Technique HP WebInspect IBM Rational AppScan Acunetix Web Vulnerability Scanner Penetration testing Univ. Coimbra VS.WS Univ. Maryland FindBugs SourceForge Yasca Static code analysis JetBrains IntelliJ IDEA Univ. Coimbra CIVS-WS Anomaly detection Vulnerability Scanners: VS1, VS2, VS3, VS4 Static code analyzers: SA1, SA2, SA3 16

17 Step 2: Execution Results for Penetration Testing Tool % TP % FP VS % 54.46% VS % 61.22% VS3 1.9% 0% VS % 43.28% 17

18 Step 2: Execution Results for Static Code Analysis and Anomaly Detection Tool % TP % FP CIVS 79.31% 0% SA % 7.69% SA2 100% 36.03% SA % 67.50% 18

19 Step 3: Measures calculation Benchmarking results Tool F-Measure Precision Recall CIVS-WS SA SA SA VS VS VS VS

20 Step 4: Ranking and selection Rank the tools using the measures Select the most effective tool Criteria 1 st 2 nd 3 rd 4 th F-Measure VS1 VS4 VS2 VS3 Inputs Precision VS3 VS4 VS1 VS2 Recall VS1 VS2/VS4 VS3 F-Measure CIVS SA2 SA1 SA3 Queries Precision CIVS SA1 SA2 SA3 Recall SA2 CIVS SA1 SA3 20

21 Benchmark properties Portability Non-intrusiveness Simple to use Repeatability Representativeness 21

22 Conclusions and future work We proposed an approach to benchmark the effectiveness of V. D. tools in web services A concrete benchmark was implemented Targeting tools able to detect SQL Injection A benchmarking example was conducted Results show that the benchmark can be used to assess and compare different tools Future work includes: Extend the benchmark to other types of vulnerabilities Apply the benchmarking approach to define benchmarks for other types of web services 22

23 Questions? Center for Informatics and Systems University of Coimbra 23

24 Benchmark Representativeness Influenced by the representativeness of the workload May not be representative of all the SQL Inj. patterns However, what is important is to compare tools in a relative manner To verify this we replaced the workload by a real workload Constituted by a small set of third-party WS 24

25 Benchmark Representativeness Inputs Queries Criteria 1 st 2 nd 3 rd 4 th F-Measure VS1 VS4 VS2 VS3 Precision VS3/VS4 VS2 VS1 Recall VS1 VS4 VS2 VS3 F-Measure CIVS SA2 SA1 SA3 Precision CIVS SA1 SA2 SA3 Recall SA2/CIVS SA1 SA3 25

Effective Detection of SQL/XPath Injection Vulnerabilities in Web Services

SCC 2009 Effective Detection of SQL/XPath Injection Vulnerabilities in Web Services Nuno Antunes, Nuno Laranjeiro,, Henrique Madeira {nmsa, cnl, mvieira, henrique}@dei.uc.pt CISUC Department of Informatics