The COGENT Case for Property-Based Tes=ng

Transcription

1 The OGENT ase for Property-Based Tes=ng Zilin hen, Liam O'onnor, Gabriele Keller, Gerwin Klein and Gernot Heiser PLOS 17

2 What Is OGENT? Reduces the cost of formally verifying systems code Restricted, purely func=onal language Uniqueness type system ase-studies: BilbyFs, ext2, F2FS, VFAT equa=onal OGENT Abstract Spec (Isabelle/HOL) manual manual one-off generated automa=c O onnor et al., IFP 16! Amani et al., ASPLOS 16! ADTs 2

3 Manual Effort BilbyFs functions sync(), iget() and library Effort Isabelle LOP OGENT SLO ost $/SLO LOP/LO 9.25 pm 13,000 1, sync()-specific 3.75 pm 5, iget()-specific 1 pm 1, sel4 12 py 180,000 8, * BilbyFs: totally 4,200 lines of OGENT Amani et al., ASPLOS 16! 3

4 Original sel4 150 bugs Abstract Spec (Isabelle/HOL) 150 bugs Executabe Spec (Haskell derived) 1 week-end 150 bugs 2 weeks 4

5 Present sel4: New Features Abstract Spec (Isabelle/HOL) Executabe Spec (Haskell derived) Unit tests 5

6 Assurance Strength 1. Safe language 2. Unit tes=ng 3. Func=onal Model Property-based checking proof tes=ng 4. Func=onal Model checking proof 5. Func=onal proof Abstract Spec (Isabelle/HOL) an do in sequence orrect code easier to verify an stop anywhere Assurance cost tradeoff generated OGENT automa=c 6

7 What Is Property-Based Tes=ng (PBT)? sort :: [Int] -> [Int]! sort =...!! Input: [4,7,3,5,0] Expected: [0,3,4,5,7]!! Unit tes=ng: Feed in specified inputs ompare outputs to expected prop_length: xs. length xs == length (sort xs)!! prop_elem: xs, x. x xs ==> x (sort xs)!! prop_sorted: {xs length xs >= 2}, i [0..length xs - 2].! (sort xs)!!i <= (sort xs)!!(i+1)!! prop_lib: xs. sort xs == List.sort xs! Property-based tes=ng: Generate random (but biased) inputs Run un=l viola=on found 7

8 OGENT: PBT-ing vs Verifying PBT overhead minimal if verifying anyway Both need (formal) specifica=on Both need (formal) proper=es OGENT total, determinis=c, purely func=onal language Good match to Isabelle & func=onal verifica=on PBT: find bugs early (spec, proper=es, implementa=on) PBT: lightweight and agile alterna=ve to formal proofs PBT and formal verifica=on support each other! 8

9 What Do We Prove? Func=onal correctness: a refinement statement from an abstract specifica=on Defini=on (refinement): Program refines A, if!!!a!. Refinement rela=on R that relates abstract and concrete states Data refinement (informally): de Roever and Engelhardt, 1998! 9

10 K J Proving Refinement To prove data refinement (by simula=on): R; JconcK JabsK; R abs OGENT: purely func=onal, determinis=c, total Refinement statement: abs : X a Y a and conc : X c Y c R X i a i c o a abs i a. R Y o a (conc i c ) (1) corres R a c! o a. R o c abs R X i a i c corres R Y (abs i a ) (conc i c ) (2) R conc R orem like this for each o 10

11 How To Test Refinement R X i a i c corres R Y (abs i a ) (conc i c ) Encode these proper=es as machine-testable specifica=ons Quickheck library in Haskell laessen and Hughes, IFP 00! test data genera=on control distribu=on, sa=sfy invariants combinators in the specifica=on language: forall, (==>), (.&.), (.&&.), (..), etc. counter-example shrinking 11

12 Quickheck Architecture Abstract Spec Embedding (Executable Spec) Generated Manual Semi-Automatic Executable Spec Isabelle/HOL Haskell OGENT Tests Tests Embeddings (ogent) Embedding (Functional ADTs ) ogent Functional ADTs compiled FFI Tests Embeddings () Embedding ( ADTs) ADTs 12

13 Modular Tes=ng Abstract Spec Generated Manual Semi-Automatic Isabelle/HOL Haskell OGENT Embedding (Executable Spec) Executable Spec Embeddings (ogent) Embedding (Functional ADTs ) Tests ogent Tests Functional ADTs compiled FFI Test s Embeddings () Embedding ( ADTs) ADTs 13

14 Modular Tes=ng Abstract Spec Generated Manual Semi-Automatic Isabelle/HOL Haskell OGENT Embedding (Executable Spec) Executable Spec Embeddings (ogent) Embedding (Functional ADTs ) Tests ogent Tests Functional ADTs compiled FFI Test s Embeddings () Embedding ( ADTs) ADTs 14

15 Example data MountState! data FsmState! hs_fsm_init :: (MountState, FsmState)! -> ogent_monad (Either Errode FsmState)! fsm_init : (SysState, MountState!, FsmState take (..))! -> RR SysState FsmState (Errode, FsmState take(..))! cogent_fsm_init :: t21 -> IO t24!! foreign import ccall unsafe "ffi_fsm_init"! c_fsm_init :: Ptr FFI.t21 -> IO (Ptr FFI.t24)! 15 t24* ffi_fsm_init (t21* a1);! t24 fsm_init (t21 a1);! data t21! data t24! 15

16 Example hs_fsm_init :: (MountState, FsmState)! -> ogent_monad (Either Errode FsmState)! a2c_i ::(MountState, FsmState) -> IO t21! rel_o :: Either Errode FsmState -> t24 -> IO Bool! Refines gen_mountstate :: Gen MountState! gen_fsmstate :: Gen FsmState! $ghci > quickheck prop_fsm_init_corres! R X i a i c corres R Y (abs i a ) (conc i c ) +++ OK, passed 100 tests.! prop_fsm_init_corres = monadicio $! forallm gen_mountstate $ \mount_st ->! forallm gen_fsmstate $ \fsm_st -> run $ do! let ia = (mount_st, fsm_st)! ic <- a2c_i ia! oa <- return $ hs_fsm_init (mount_st, fsm_st)! oc <- cogent_fsm_init ic! corresm rel_o oa oc! cogent_fsm_init :: t21 -> IO t24! 16

17 More In The Paper The OGENT ase for Property-Based Testing ract Zilin hen Liam O onnor Gabriele Keller Gerwin Klein Gernot Heiser Data61 and UNSW, Australia first.last@data61.csiro.au 17

18 Future Work More automa=on Glue code between Haskell and Refinement statements R X i a i c corres R Y (abs i a ) (conc i c ) Refinement rela=ons R X, R Y with domain-specific knowledge Isabelle theorems Test results vs. theorem proving in Isabelle Test data generators and shrinking algorithms David R. MacIver, 2016! Jacob Stanley, 2017! Tes=ng kernel modules Full case-studies 18

19 Thank you Trustworthy Systems Gernot Heiser e gernot.heiser@data61.csiro.au w hpp://trustworthy.systems OGENT Project ts.data61.csiro.au/projects/ts/cogent.pml