Hybrid Verification in SPARK 2014: Combining Formal Methods with Testing

procedure Array_Indexing_Bug (Buffer : in out String) is -- intended to change "ABCD" into "AABC", -- but high loop bound is bad begin for I in Buffer'First.. Buffer'Last loop Buffer (I + 1) := Buffer (I); end loop; end Array_Indexing_Bug;

"Program testing can be used to show the presence of bugs, but never to show their absence!" -- Edsger Dijkstra, 1970

Contract: agreement between the client and the supplier of a service Program contract: agreement between the caller and the callee subprograms

Each assertion serves three functions: a static proof goal during proving a dynamic consistency check during testing documentation for the human reader Because the same assertion serves all three functions, consistency is guaranteed. See Chalin's "Engineering a Sound Assertion Semantics for the Verifying Compiler".

Buffer overflows in Ada? Easily avoided by programmers (array types carry their bounds) Automatically caught at run-time Integer overflows in Ada? Easily avoided by programmers (using bounded integer types) Automatically caught at run-time Buffer and integer overflows in SPARK? If present, automatically caught by analysis If absent, automatic proof that no such error can occur Buffer overflows and integer overflows are still major sources of pain in C

Two advantages of formal verification: bugs which testing would have caught may be caught earlier bugs which testing would have missed may be caught

Assertions also serve as documentation for human readers

Quantified expressions: (for all I in Vec'Range => Vec (I) <= Max_Element) (for some I in Vec'Range => Vec (I) = Max_Element) Conditional expressions: (if Y /= 0 then X / Y else Integer'Last) (case Adjustment is when Double => 2 * X, when Increment => X + 1, when None => X) The Old attribute: procedure Increment (X : in out Integer) with Pre => X /= Integer'Last, Post => X = X'Old + 1; Expression functions: function Is_Even (X : Natural) return Boolean is (X mod 2 = 0);

Ghost code: useful for assertions in verification useful for consistency checks in testing can be safely and easily disabled in the final build

SPARK 2014 supports a large Ada subset concurrency (tasks and protected objects) object oriented programming (tagged types) recursion dynamic constraints

Combining tests and proofs P is tested P Q P calls Q How do we justify assumptions made during proof? Q calls P Q is proved verification combining tests and proofs should be AT LEAST AS GOOD AS verification based on tests only Copyright 2014 AdaCore Slide: 12

Implicit precondition violation examples: type Rec is record F1, F2 : Integer; end record; procedure P1 (X : Rec) is... ; procedure Q1 is Y : Rec; begin Y.F1 := 123; P1 (Y); -- parameter incompletely initialized end Q1; -------- Buffer : String (1.. 100); procedure P2 (X : in out String) with Global => (In_Out => Buffer) is ; procedure Q2 is begin P2 (X => Buffer); -- unsafe aliasing end Q2; Copyright 2014 AdaCore Slide: 14

Cost of testing Cost of testing greater than cost of development 10% increase each year for avionics software (Boeing META Project) Uneven partitioning: 80% of effort! Uneven quality: 80% of errors traced to 20% of code (NASA Software Safety Guidebook) Need to reduce and focus the cost of testing Copyright 2014 AdaCore Slide: 15

Proving and testing can be combined at different levels At the level of individual run-time checks: Focus test coverage on unproven checks At the level of individual subprograms: Proven subprograms call tested and vice versa. Use proofs to completely replace unit testing and executable contracts during integration testing Copyright 2014 AdaCore Slide: 16

Resources SPARK Pro webpage http://www.adacore.com/sparkpro SPARK community page http://www.spark-2014.org SPARK User s Guide http://docs.adacore.com/spark2014-docs/html/ug AdaCore University http://u.adacore.com Copyright 2014 AdaCore Slide: 17