Testing the Hard Stuff and Staying Sane. John Hughes

Transcription

1 Testing the Hard Stuff and Staying Sane John Hughes

2 Why is testing hard? n features O(n) 23 ) test cases 3 4 tests per triples pairs race of feature conditions of features

3 Don t write tests! Generate them

4 Problem How do we know whether an arbitrary test has passed or failed? expected results can t be used general properties can be checked Property-based testing

5 QuickCheck 1999 invented by Koen Claessen and myself (ICFP 2000), in Haskell, for purely functional code 2006 Quviq founded marketing Erlang version State machines, race condition testing, etc Lots of fun solving challenging testing problems in the field

6 What does it do? API under test A minimal failing example

7 Let s see it in action

8 State Machine Models Erlang API Calls API Calls API Calls API Calls postconditions Erlang Model state Model state Model state Model state Erlang

9 Example put 1 put 2 get put 3 get [] [1] [1,2] [2] [2,3] 1 2

10 Code Fragments: specifying get get_pre(s) -> S#state.ptr /= undefined andalso S#state.contents /= []. get_next(s,_value,_args) -> S#state{contents=tl(S#state.contents)}. get_post(s,_args,res) -> eq(res,hd(s#state.contents)). Precondition State transition Postcondition

11 Time for some tests!

12 Lessons The same property can find many different bugs Minimal failing tests make diagnosis easy

13

14 Doing it for real 500 pages of specifications

15 Theory Car manufacturers should be able to buy code from different providers and have them work seamlessly together

16 Practice VOLVO's experience has been that this is often not the case

17 ~20,000 lines of QuickCheck code Vendor code 3,000 pages of pdfs

18 A Bug in a vendor s CAN stack 1 sent send priority 1 send priority 2 send priority 3 tx_confirm sending 1 queued sending 3 19

19 The Problem CAN bus identifiers determine bus priority

20 A Bug in a vendor s CAN stack send priority 1 sending 1 send priority 2 queued 1 sent send priority 3 tx_confirm sending 3 Failed to mask off the top bit before comparing priorities 21

21 AutoSAR results > 200 issues discovered > 100 bugs in the standard Test code 9x smaller than conventional TTCN- 3 tests and tests more! Test code 3-6x smaller than implementations

22 Riak: a no-sql distributed database Network partition Eventual consistency Data is replicated on 3 different nodes Thanks to: Dave Smith, Jon Meredith, Scott Fritchie

23 Data loss! A Riak bug

24 What does it mean? QuickCheck found An extreme failure scenario Leading to loss of data A failure of eventual consistency So is Riak buggy?

25 "We know there is a lurking bug somewhere in the dets code. We have got 'bad object' and 'premature eof' every other month the last year. We have not been able to track the bug down since the dets files is repaired automatically next time it is opened. Tobbe Törnqvist, Klarna, 2007

26 What is it? Application Mnesia Dets File system Invoicing services for web shops Distributed database: transactions, distribution, replication Tuple storage Race conditions?

27 Imagine Testing This dispenser:take_ticket() dispenser:reset()

28 A Unit Test in Erlang test_dispenser() -> ok = reset(), 1 = take_ticket(), 2 = take_ticket(), 3 = take_ticket(), ok = reset(), 1 = take_ticket(). Expected results

29 Modelling the dispenser reset take take take ok 1 2 3

30 A Parallel Unit Test reset ok take_ticket take_ticket take_ticket Three possible correct outcomes!

31 Another Parallel Test reset take_ticket take_ticket take_ticket take_ticket reset 30 possible correct outcomes!

32 Deciding a Parallel Test reset ok take 1 take 3 take

33 Let s run some tests

34 take_ticket() -> N = read(), Prefix: write(n+1), N+1. Parallel: 1. dispenser:take_ticket() --> 1 2. dispenser:take_ticket() --> 1 Result: no_possible_interleaving

35 dets Tuple store: {Key, Value1, Value2 } Operations: insert(table,listoftuples) delete(table,key) insert_new(table,listoftuples) Model: List of tuples (almost)

36 QuickCheck Specification > 6,000 LOC <100 LOC

37 Bug #1 Prefix: Types: open_file(dets_table,[{type,bag}]) --> Name = name() dets_table Parallel: 1. insert(dets_table,[]) Bool = bool() --> ok insert_new(name, Objects) -> Bool Objects = object() [object()] 2. insert_new(dets_table,[]) --> ok Result: no_possible_interleaving

38 Bug #2 Prefix: open_file(dets_table,[{type,set}]) --> dets_table Parallel: 1. insert(dets_table,{0,0}) --> ok 2. insert_new(dets_table,{0,0}) --> time out =ERROR REPORT==== 4-Oct-2010::17:08:21 === ** dets: Bug was found when accessing table dets_table

39 Bug #3 Prefix: open_file(dets_table,[{type,set}]) --> dets_table Parallel: 1. open_file(dets_table,[{type,set}]) --> dets_table 2. insert(dets_table,{0,0}) --> ok get_contents(dets_table) --> [] Result: no_possible_interleaving!

40 Is the file corrupt?

41 Bug #4 Prefix: open_file(dets_table,[{type,bag}]) --> dets_table close(dets_table) --> ok open_file(dets_table,[{type,bag}]) --> dets_table Parallel: 1. lookup(dets_table,0) --> [] 2. insert(dets_table,{0,0}) --> ok 3. insert(dets_table,{0,0}) --> ok Result: ok premature eof

42 Bug #5 Prefix: open_file(dets_table,[{type,set}]) --> dets_table insert(dets_table,[{1,0}]) --> ok Parallel: 1. lookup(dets_table,0) --> [] delete(dets_table,1) --> ok 2. open_file(dets_table,[{type,set}]) --> dets_table Result: ok false bad object

43 "We know there is a lurking bug somewhere in the dets code. We have got 'bad object' and 'premature eof' every other month the last year. Tobbe Törnqvist, Klarna, 2007 Each bug fixed the day after reporting the failing case

44 Before After Files over 1GB? Rehashing? > 6 weeks of effort! Database with one record! 5 calls to reproduce < 1 day to fix

45 Property-based testing Lets us test more with less effort Finds bugs in everything we test

46 Don t write tests Generate them!

47 Where to find out more Haskell QuickCheck The original ICFP 2000 Quviq QuickCheck Commercial tool & services Many, many tools inspired by QuickCheck ScalaCheck, FsCheck, test.generate

48 What does it feel like? Docs QC Spec Code

49

50 Properties vs test cases Code sizes for the Flexray interface: Lines of code TTCN-3 test cases QuickCheck and Erlang 9x smaller code! and it tests more!

51 Properties vs implementations Code QuickCheck 0 CAN DEM FlexRay COM The test code is 3 6x smaller than the implementation