OUTLINE PERFORMANCE BENCHMARKING 7/23/18 SUB BENCHMARKING THE SECURITY OF SOFTWARE SYSTEMS OR TO BENCHMARK OR NOT TO BENCHMARK

Transcription

1 BENCHMARKING THE SECURITY OF SOFTWARE SYSTEMS OR TO BENCHMARK OR NOT TO BENCHMARK Department of Informatics Engineering University of Coimbra - Portugal QRS 2018 Lisbon, Portugal July 19 th, 2018 BENCHMARKING according to specific quality attributes Performance benchmarking Well established both in terms of research and application Supported by organizations like TPC and SPEC Mostly for marketing Dependability benchmarking Well established from a research perspective No endorsement from the industry QRS 2018, Lisbon, Portugal, July 19 th, BENCHMARKING according to specific quality attributes Security benchmarking Several works can be found No common approach available yet Performance benchmarks Whetstone Wisconsin Bench TP1 DebitCredit Orange Book TPC & SPEC Release of commercial performance benchmarks SIGDeB CIS Common Criteria Dependability benchmarks Security benchmarks QRS 2018, Lisbon, Portugal, July 19 th, EMBC Research projects on dependability & security benchmarks OUTLINE The past: Performance & Dependability Benchmarking The present: Security Benchmarking Benchmarking the Security of Systems Approach: + Trustworthiness Assessment Example: Benchmarking Web Service Frameworks Benchmarking Security Tools Approach: Vulnerability and Attack Injection Example: Benchmarking Intrusion Detection Systems Challenges and Conclusions QRS 2018, Lisbon, Portugal, July 19 th, PERFORMANCE BENCHMARKING PERFORMANCE BENCHMARKING in terms of performance : Set of representative operations Throughput Response time Latency QRS 2018, Lisbon, Portugal, July 19 th, QRS 2018, Lisbon, Portugal, July 19 th,

2 TPC-C (1992) DEPENDABILITY BENCHMARKING DBMS considering dependability attributes : Database transactions Although some integrity tests are performed, it assumes that nothing fails Transaction rate (tpmc) Price per transaction ($/tpmc) QRS 2018, Lisbon, Portugal, July 19 th, QRS 2018, Lisbon, Portugal, July 19 th, Faultload: Faultload DEPENDABILITY BENCHMARKING Set of representative faults, injected into the system Parameters (fault rates, MTBF, etc.) Performance and/or dependability Both baseline and in the presence of faults Unconditional and/or direct Models Unconditional QRS 2018, Lisbon, Portugal, July 19 th, : Faultload TPC-C transactions Faultload: Operator faults + Software faults + HW component failures Performance: tpmc, $/tpmc, Tf, $/Tf Dependability: Ne, AvtS, AvtC DBENCH-OLTP (2005) QRS 2018, Lisbon, Portugal, July 19 th, DBENCH-OLTP (2005) DBENCH-OLTP (2005) tpmc 4000 Baseline Performance tpmc $/tpmc $ 30 Tf 4000 Performance With Faults Tf $/Tf $ A B C D E F G H I J K A B C D E F G H I J K Does not take into account malicious behaviors % Availability AvtS (Server) 100 AvtC (Clients) (faults = vulnerability + attack) Faultload: Operator faults A B C D E F G H I J K QRS 2018, Lisbon, Portugal, July 19 th, QRS 2018, Lisbon, Portugal, July 19 th,

3 SECURITY BENCHMARKING considering security aspects Benchmarking the Security of Systems / Components Systems that should implement security requirements OS, middleware, server software, etc. Benchmarking Security Tools Tools used to improve the security of systems Penetration testers, static analyzers, IDS, etc. QRS 2018, Lisbon, Portugal, July 19 th, BENCHMARKING SECURITY OF SYSTEMS Does not work if one exposure, wants mean to time benchmark between attacks, how etc.) Attackload: secure different systems are! e.g. Representative does the number attacks of vulnerabilities of a system represent anything? Performance + dependability Security (e.g., number vulnerabilities, attack detection) Attackload Attacking what? Do we know the vulnerabilities? Models What are representative attacks? Unconditional Parameters (vulnerability QRS 2018, Lisbon, Portugal, July 19 th, A DIFFERENT APPROACH A DIFFERENT APPROACH s Security s Security Acceptable Trustworthiness Assessment Security : Apply state-of-the-art techniques and tools to detect vulnerabilities s with vulnerabilities are: Disqualified! Or vulnerabilities are fixed QRS 2018, Lisbon, Portugal, July 19 th, Trustworthiness Assessment: Gather evidences on how much one can trust e.g., best coding practices, development process, bad smells QRS 2018, Lisbon, Portugal, July 19 th, A DIFFERENT APPROACH EXAMPLE: WEB SERVICE FRAMEWORKS s Security Acceptable Trustworthiness Assessment WSFs (testing) Acceptable Assessment (CPU + mem.) Trust. Score Portray trust from a user perspective Dynamic: may change over time Depend on the type of evidences gathered Different for different attack vectors QRS 2018, Lisbon, Portugal, July 19 th, DoS Attacks Coercive Parsing, Malformed XML, Malicious Attachment, etc. Trustworthiness Assessment: Quality model to compute a score QRS 2018, Lisbon, Portugal, July 19 th,

4 QUALITY MODEL SYSTEMS UNDER BENCHMARKING QRS 2018, Lisbon, Portugal, July 19 th, QRS 2018, Lisbon, Portugal, July 19 th, TRUSTWORTHINESS RESULTS BENCHMARKING SECURITY TOOLS Faultload (vulnerabilities + attacks) Data Sec. Tool Faultload: Vulnerabilities are injected Attacks target the injected vulnerabilities Data can be collected for benchmarking security tools Penetration testers, static analyzers, IDS, etc. QRS 2018, Lisbon, Portugal, July 19 th, QRS 2018, Lisbon, Portugal, July 19 th, VULNERABILITY AND ATTACK INJECTION EXAMPLE: BENCHMARKING IDS Security requires a defense in depth approach Coding best practices Testing Static analysis Vulnerability-free code is hard (or even impossible) to achieve... Intrusion detection tools support a post-deployment approach For protecting against known and unknown attacks QRS 2018, Lisbon, Portugal, July 19 th, QRS 2018, Lisbon, Portugal, July 19 th,

5 EVALUATION APPROACH EXAMPLES OF VULNERABILITIES INJECTED Original PHP code Code with injected vulnerability Operation performed $id=intval($_get['id']); $id=$_get['id']; Removed the intval function allowing also non numeric values (i.e. SQL commands) in the $id variable $page = urlencode($page); $page = $page; Removed the urlencode function allowing also alphanumeric values (i.e. SQL commands) in the $page variable QRS 2018, Lisbon, Portugal, July 19 th, QRS 2018, Lisbon, Portugal, July 19 th, EXAMPLES OF ATTACKS SYSTEMS UNDER BENCHMARKING Attack payloads Expected result ' Modifies the structure of the query; usually results in an error or 1=1 Modifies the structure of the query. Overrides the query restrictions by adding a statement that is always true. ' or 'a'='a Modifies the structure of the query. Overrides the query restrictions by adding a statement that is always true. +connection_id()- connection_id() Modifies the query result to Modifies the query result to ASCII('A') Modifies the query result to ASCII(1) Modifies the query result to 0 Tool Architectural Level monitored Detection Approach Data Source Known Technology Limitations ACD Application Anomaly Based Apache Log Only GET method Apache Scalp Application Signature Based Apache Log Only GET method ModSecurity Application Signature Based HTTP traffic - Snort (v2.8 and Network Network Signature Based v2.9) Trafic - GreenSQL Database Signature Based SQL Proxy Trafic MySQL data DB IDS Database Anomaly Based SQL Sniffer Trafic MySQL and Oracle data QRS 2018, Lisbon, Portugal, July 19 th, QRS 2018, Lisbon, Portugal, July 19 th, EXPERIMENTAL SETUP MAIN RESULTS lvl DB Net App Tool P N Pop TP TN FN FP ACD Scalp ModSecurity Review Snort GreenSQL All Reported Prec. Recall Mark. Infor. DB IDS Net Snort QRS 2018, Lisbon, Portugal, July 19 th, QRS 2018, Lisbon, Portugal, July 19 th,

6 WHAT IS WRONG? FIXED! Established benchmarks are mostly for marketing! Strict benchmarking conditions Activation Fixed workload & faultload + Small set of & faultload: May not be representative of the user scenario Fixed! May not satisfy the user needs Decision based on several is difficult! No security benchmark endorsed by any organization or industry QRS 2018, Lisbon, Portugal, July 19 th, Example: Fixed! Benchmarking vulnerability detection tools Typical metric: F-Measure Is this good in all scenarios? Business critical: recall Best effort: F-Measure Minimum effort: Markedness QRS 2018, Lisbon, Portugal, July 19 th, A POTENTIAL APPROACH SCENARIOS AND QUALITY MODELS Benchmarking conditions adaptable to the user needs Include multiple usage scenarios: depend on the scenario Adaptable workload and faultload Use quality models instead of independent Quality models should also adapt to the scenario How to define scenarios? How to define quality models? How to adapt workloads and faultloads to the scenarios? QRS 2018, Lisbon, Portugal, July 19 th, QRS 2018, Lisbon, Portugal, July 19 th, Satisfy industry requirements Representativeness, portability, scalability, nonintrusiveness, low cost, Prevent gaming CHALLENGES Satisfy user requirements Representativeness, usefulness, simplicity of use Adaptable allow gaming Endorsement by TPC, SPEC, How to? Resilience Benchmarking IS THERE A FUTURE? Assess and compare the behavior of components and computer systems when subjected to changes Which resilience? Comparable, consistent, understandable, meaningful, Changeloads: Representative, practical, portable, Trustworthiness Benchmarking What evidences to collect? What? Dynamicity of perception social trust... QRS 2018, Lisbon, Portugal, July 19 th, QRS 2018, Lisbon, Portugal, July 19 th,

7 7/23/18 CONCLUSIONS QUESTIONS? The benchmarking concept is well established! Department of Informatics Engineering University of Coimbra Acceptance by big industry depends on perceived utility for marketing Acceptance by users requires adaptability From a research perspective, performance and dependability benchmarking are well known Security benchmarking approaches are weak New types of benchmarks will bring additional challenges! QRS 2018, Lisbon, Portugal, July 19th, QRS 2018, Lisbon, Portugal, July 19th,