T-110.470 Salausjärjestelmät (Cryptosystems) Testing Final stages of life cycle Security testing 15.11.2004 1 3 Outline Security testing Security testing Some testing techniques Final stages of life cycle Security testing A subset of normal testing, but the approach is different A few important aspects make it more difficult Difficult to measure success of a test Expected result security is intact => how to measure this? Silent failures Security failures are often silent Example: cryptographic key at both ends broken in the same way Asymmetry between attacker and defender Requires us to verify functionality of very obscure scenarios It's difficult to simulate attacks, even in white box testing An attacker may be able to feed corrupted data to an internal interface i.e. reach states that are otherwise almost unreachable This may be very difficult to do using valid inputs 2 4
Testing in general Determining test cases Glen Myers on testing 1. Testing = execution of a program, with the intent of finding errors 2. Good test case = high probability of finding an as-yet-undiscovered error 3. Successful test = uncovers an as-yet-undiscovered error Black box and white box (glass box) testing White box testing more relevant to security critical software Module testing determining test cases Common goal is to execute all paths of control (various definitions) For security critical software, also the unlikely paths are interesting Normal data select to exercise all control paths Boundary data natural boundaries, + {-1, 0, 1} Random data brute force Exceptions unexpected signals and traps, etc. 5 Boundary data Typical programming errors occur on the boundary Often result in off-by-one (or one-off ) errors Inputs often modelled as equivalence classes Examples Loop is iterated once too often Array index check is off-by-one, and allows overwriting one char Boundary value test cases try to cause such errors to occur Determining test cases is usually straightforward Example: function input integer n must be in range 0..99 => test boundary data -1, 0, 1, 98, 99, 100 Invalid input -1 0 99 100 Valid input Invalid input 7 Determining test cases Determining test cases Normal data try to exercise all parts of the code Function coverage All functions executed at least once? Statement coverage (aka line coverage ) All statements in function executed at least once? Decision coverage (aka branch coverage ) All branches evaluated to both true and false? Condition coverage and multiple condition coverage All components of logical expressions evaluated to both true and false? Multiple condition coverage => all combinations of true/false values Path coverage All paths of control evaluated? Issues with correlated conditions and loops Relational operator coverage Example: (x < 10) => tested with (x < 10), (x = 10), (x > 10) 6 Random data Generate random inputs using some useful distribution Both valid and invalid inputs should be generated Must be able to determine success or failure of test Example: against a reference implementation Example: mathematical property (y=sqrt(x) => test y*y=x) Exceptions Generate unexpected signals using various timings Generate signals when previous signals are being handled, etc. Time-dependent and thus difficult to arrange in practice Timing inputs In multi-threaded applications timing is often a major headache Testing timing is important, but difficult 8
Drivers and stubs A module is typically tested using drivers and stubs Provide a prosthetic environment for the module Drivers and stubs can be used to feed unlikely errors into the module E.g. test invalid inputs that cannot be tested in integration or system testing Attacks often involve causing invalid inputs (from above or below) Can be simulated by stubs / drivers doing the same Driver 1... Driver N Module under test Driver X Module under test Using invalid inputs Typical sources of inputs Specific test cases e.g. boundary conditions, known attacks (Pseudo)random inputs brute force for conditions we can't think of Actual inputs but distorted usually possible in system testing only Internal distortion specific testing code added to module Generating (pseudo)random inputs It's a good idea to use a PRNG with a known seed, so a test can be repeated Not always possible if time dependent module Must ensure that statistical distribution of inputs is reasonably sane Usually must use heuristics Example: 50% of tests on short inputs and 50% on long inputs A natural distribution when an attacker is involved? Stub 1... Stub N Stub Y 9 11 Using invalid inputs Some testing techniques 10 Distorted real inputs Examples For 10% of network packets, packet truncation and/or corruption For 5% of memory allocations, claim out of memory Goals We can test whether the implementation (e.g. protocol) can survive and do it's duty even with partially corrupted inputs Exercise code related to input sanity checking (e.g. can detect TLV decoding loop when length=0) Advantages Can generate errors in system states that cannot be achieved with black box testing (e.g. random inputs from outside) Disadvantages Usually feasible to apply in integration or system testing only 12
Using invalid inputs Internal distortion Add distortion code to module under test Code distorts internal data in well chosen locations We assume attacker might be able to do this in real life Example: buffer overflow in internal code Distortion code enabled conditionally for a test build Dangerous ensure that the build identifies itself as a test build Advantages Can exercise internal error handling which is otherwise difficult Simulates the impact of an attack, i.e. measures containment Disadvantages We cannot distort completely arbitrarily if we violate basic invariants, the code is not supposed to deal with it Of course, secure code should detect inconsistency and bail out Penetration testing ( Tiger team ) A third party who attempts to penetrate a system Try to avoid motivation conflict May be black box or white box White box (= source given) more relevant Have to assume that attacker has source, even with closed source (consider Windows source code theft) Advantages Avoids motivation conflict, semi-objective Third party may have better competence Disadvantages Cost and time Results limited by third party's competence (difficult to assess) 13 15 Using invalid inputs Test vectors Determining success (= no security issue) is difficult A test case requires an expected result part for ensuring success But typical effects from testing on invalid inputs include random memory corruption, crashes, etc. => Typical criterion: code is stable (= no crash) Best to combine with a dynamic analysis tool (e.g. Valgrind) Ensure no complaints from the tool during testing May catch e.g. buffer overflows or NULL pointers A valuable technique in practice Simulates conditions that are often very difficult to reach otherwise For some testing, a stub approach can be used instead E.g. out-of-memory conditions 14 Simply test cases for cryptographic primitives and constructs Try to maximize coverage of code and functionality with a reasonable set of tests Randomized tests against a reference implementation Useful in many stages Crypto specification for clarifying interpretation Software implementation for testing correctness of implementation Regression testing for ensuring primitives are still correct May catch e.g. compiler bugs, or differences between compilers Deployed, running system for ensuring integrity of code and data Very important A faulty cryptographic primitive may be very difficult to find Especially if the bug is not total 16
Interoperability testing Testing for Denial-of-Service Test against other implementations Typically used for cryptographic protocols But can also be used for other things (e.g. PGP interoperability) Advantages Tests functionality of cryptographic constructs Catches most differences in cryptographic constructs, e.g. if key derivation algorithms differ Real-world system test Also has marketing value to vendor => motivation Disadvantages Two vendors may misinterpret specification the same way Does not ensure security, just functionality Often costly e.g. interoperability meeting ( bake-off ) Specific to particular standards 17 Denial-of-Service may result from many things Actual exploits of implementation flaws But may also be inherent to a protocol For instance, in IKE Aggressive Mode, responding host must compute Diffie-Hellman after receiving first message If DoS can be caused legitimately, it needs to be measured Use attack software to cause legitimate DoS If DoS unavoidable because of e.g. protocol requirements Desirable result is to have clean and predictable behavior Bounded memory or processor use Recovery to normal after attack ceases If DoS can be avoided, e.g. protocol provides ways to do that Desirable result is to experience only a minimal level of DoS 19 Leak testing Product installer Memory leaks (and other resources leaks) are nasty Software may leak resources in error conditions only, for instance Leaking software is susceptible to Denial-of-Service If out-of-resources condition exposes other bugs, may also be a security issue Leaks can be found in several ways Dynamic analysis tools Monitoring process statistics in UNIX systems Periodic printouts (in event loop, for instance) Most methods require that a leak is triggered Thus often invalid inputs and internal corruption used 18 Product installer is an important system component Goal is to get system into an initial secure state (or fail completely) Important to test effect of media / bus corruption If files are copied incorrectly, can have fatal consequences Broken crypto library can cause various problems Errors in code can have arbitrary impact Installer should verify copy operations and detect copy errors If product does not have integrity protection Consider encapsulating product files for distribution Example: compression improves likelihood of corruption detection Corruption on hard drive less likely than corruption of e.g. floppy or CD Still, better to fix at the source 20
Deployment issues Final stages of lifecycle Physical environment If cryptosystem relies on physical protection, must ensure its adequacy Example: often servers required to be physically protected but organizational policies not good enough (social engineering) Hardware and software environment Hardware and software must match system requirements Requirements in user documentation If operating system not delivered as part of product, a specific, tested version should be used Example: kernel level exploits in some Linux systems => choice of operating system version not trivial Certification is often relative to a certain OS In general, we must check that security assumptions are met 21 23 Deployment issues Overall goals Deployed system configuration is secure Security services and their limitations are understood (e.g. by customer) Required organizational policies and processes are in place Deployed components haven't been tampered with Documentation Like a specification, documentation must be clear, complete, and consistent Must also be easily understandable Difficult, how to ensure administrator understands e.g. rationale for choosing cryptographic parameters? Documentation must provide enough information to... Understand intended usage and limitations of the system Configure system properly to achieve security goals Deployment issues Default configuration Security critical should always be safe out of the box Surprising how often this is not the case If feature X cannot be secure by default, it must be disabled Software may indicate this to administrator to improve usability Should also be noted in configuration ( initial configuration ) A configuration wizard may be used to enter into one of welldefined starting configurations for different usage scenarios Must ensure components haven't been tampered with More difficult than it sounds e.g. how to ensure smartcards are OK? Several techniques, e.g. sealing by manufacturer, difficult to forge For software, various alternatives Network download + signed MD5 sums, or physical trusted media 22 24
Maintenance issues Maintenance needs caused by many factors Hardware breakdown and upgrades => may require software changes New features requested by customer to improve usage Bug fixes (functional or security) Changes to deployment = mini-deployment Must apply testing procedures and re-think deployment scenario Have some essential requirements or assumptions changed compared to time before new version? In addition, must think of backwards compatibility In client-server protocols, is new version compatible with old clients? Do clients need to be upgraded? Does configuration need changing? Incremental or complete update? Incremental must consider security of intermediate configuration Retirement Typical goals Controlled process of dismantling system Cryptography => what to do with system data? a) Ensure it's properly erased (or destroyed) b) Ensure it's properly archived Erasing data properly Hard drives are a particular problem, since they retain a lot of data Example: http://www.ibas.com/ Some methods Disk eraser software (e.g. IBAS ExpertEraser, Linux shred) Strong magnetic field ( Degausser ) Erasing individual files is often very difficult Underlying file system may e.g. relocate files Thus, typically entire partitions / drives are erased 25 27 Maintenance issues Summary Software updates are becoming more and more important Necessitated by penetrate-and-patch method of software development Pressure caused by viruses, worms, and other exploits increasing Market requires quicker fixes (competitive pressure) Typical goal is to fix and deploy before problem become public Common rules of conduct for security incidents is to give vendor time to fix and deploy problem before wider publicity Several models can be used for updating software securely Media deliveries with trusted media (seals, etc) Network updates Digital signatures a key method in providing assurance Unfortunately many systems don't use a secure update mechanism Example: many Linux distributions are insecure in this respect! 26 Security testing Some testing techniques Final stages of life cycle This concludes the main lectures of this course Next time: Visiting lecturer After that: Summary lecture 28
Thank you! Questions? 29