Software System Design and Implementation

Transcription

1 Software System Design and Implementation Property-based Testing Gabriele Keller The University of New South Wales School of Computer Science and Engineering Sydney, Australia COMP s1

2 Testing in Haskell The language already provides many assurances: Memory safety - no buffer overflow - no access to already freed memory - no access to uninitialised memory - no freeing of already free memory

3 Testing in Haskell The language already provides many assurances: Strong type safety - type correct programs can t go wrong/lead to undefined behaviour

4 Testing in Haskell The language already provides many assurances Purity except where explicitly stated Testing can largely focus on logic bugs It is largely sufficient to test functions in isolation (thanks to purity!) Focus on properties of functions

5 Property-based testing Idea: specify properties formally and test the code against those properties

6 An example property reverse :: [a] -> [a] reverse [] = [] reverse (x:xs) = reverse xs ++ [x] Property: reverse commutes with append -- In predicate logic (more about that next week) xs ys. reverse (xs ++ ys) reverse ys ++ reverse xs -- In Haskell Let's try it! prop_revapp xs ys = reverse (xs ++ ys) == reverse ys ++ reverse xs

7 How can QuickCheck generate test data? We know that it s not possible to generate data of any type

8 What is the type of quickcheck?? an IO action (more about this later)

9 Conceptual benefits of property-based testing Properties get specified formally Encourages to think about the tested code in new ways Increases understanding of tested system Simple, compact test representation Checking is cheap Checks provide feedback to debug the specification Checks find bugs in the code This doesn t mean you should do property based testing instead of unit testing, but in addition to!

10 QuickCheck Specification of tests by properties Randomised test data generation Originally invented in the context of Haskell [Claessen & Hughes] Ported to Erlang, Scheme, Common Lisp, Perl, Python, Ruby, Java, Scala, F#, Standard ML, JavaScript, and C++

11 Core concepts of QuickCheck Specification of program properties Using a simple specification language Properties have a formal, logical meaning Properties also have a well-defined operational meaning Property testing Using random tests produced by test-data generators It is cheap so, it is done often!

12 Random testing in QuickCheck Experience shows that it works well at fine granularity In purely functional code, all dependencies are explicit Has been extended to cover state-based code, too Test-data generators Type driven that is, type-dependent generator selection Built-in default generators for common types Explicit user-control and custom generators are supported

13 Property-based versus unit testing: challenges Does the generated data test the cases that need to be tested? Should use coverage checker in any case Are failures informative? Try to work at a fine granularity and use shrinking How difficult is it to generate test data for user-defined structures? QuickCheck comes with elaborate combinators for test generation

14 Property-based versus unit testing: advantages Repeated testing (as part of nightly builds and regression tests) can improve code coverage Properties are more compact than a set of related unit tests Properties are a form of documentation checked for consistency Properties cover the general case, instead of one or more examples Less testing code needs to be written and maintained Ericsson s AXD301 ATM-switch: 1.5 million lines of Erlang code plus 700,000 lines of conventional testing code

15 A slightly larger example words :: String -> [String] -- break into words unwords :: [String] -> String -- glue words together We would expect (unwords. words) to be the identity Make this into a property prop_words Lessons: Define prop_words! The code s properties may not be what you think at first! Shrinking helps to get to the bottom of bugs in properties and code

16 Testing mergesort A more comprehensive example

17 Mergesort algorithm Recursive divide-and-conquer algorithm 1. If list length smaller than 2, the list is already sorted 2. Split input list into two equal halves 3. Recursively apply mergesort to the two halves 4. Merge the two sorted halves Merging sorted lists is cheap (linear in the length of the lists)

18 What does the call tree for mergesort look like?

19 Merging sorted lists Given two lists in ascending order Produce a list that combines the elements of these two lists and is also in ascending order merge :: Ord a => [a] -> [a] -> [a]

20 Splitting a list into even halves Partition a list into two lists, such that both have (almost) the same length split :: [a] -> ([a], [a])

21 Putting it all together mergesort :: Ord a => [a] -> [a]

22 merge :: Ord a => [a] -> [a] -> [a] mergesort :: Ord a => [a] -> [a] merge must preserve the length of the sorted list: prop_preserveslength Sorting is idempotent: prop_idempotent The first element in a sorted list must be its minimum: prop_minimum Let's define these properties!

23 The sorted list must be ordered: prop_ordered The output must be a permutation of the input: prop_permutation The last element in a sorted list must be its maximum: prop_maximum Interaction of sorting and append: prop_append Let's define these properties, too!

24 Testing against a model For many tricky (especially for optimised algorithms), there is a simpler, less efficient one The tricky algorithm is correct if it produces the same output as the simple one We call the simple (maybe even naive) implementation a model Test the real implementation against the model

25 Constraining generators The function merge must produce an ordered list from two ordered lists How do we capture this in a property?

26 Constraining generators The function merge must produce an ordered list from two ordered lists

27 orderedlist :: (Ord a, Arbitrary a) => Gen [a] forall :: (Show a, Testable prop) => Gen a -> (a -> prop) -> Property

28 Alternative: type-level modifier: newtype OrderedList a Constructor: Ordered :: [a] -> OrderedList a

29 Quality of tests How good are your tests? Have you checked that every special case works correctly? Is all code exercised in the tests? Even if all code is exercised, is it exercised in all contexts?

30 Code coverage A whole family of measures is used to judge the degree of code coverage The various measures have varying precision They vary in the rigour required of tests to achieve full coverage They differ in their suitability for tool support May be combined Let's look at some of the more important ones.

31 Function coverage Has every function been called? Easy to measure Easy to provide tool support Rather coarse grain i.e., misses many possible execution paths

32 Entry/exit coverage Has every possible call and return of the function been executed? Easy to measure Easy to provide tools support Somewhat more comprehensive than plain function coverage

33 Statement/expression coverage Has each statement been executed? Somewhat harder to measure Easy to provide tool support Still fairly coarse grain, but substantially more comprehensive than function coverage or entry/exit coverage

34 Branch/decision coverage Has every control flow alternative been executed? Measuring and tool support as for statement coverage In an imperative language, more comprehensive than statement coverage There may be more than one control flow edge that leads to a given statement In branch/decision coverage, they must all have been taken Related condition coverage: have all conditions been True and False

35 Path coverage Has every possible route through a program been executed? This is much harder to measure Requires that each combination of branches which leads to a unique path to be executed Full path coverage is infeasible Loops may lead to infinite numbers of paths Even when disregarding repetitions, path coverage is still very expensive

36 Haskell Program Coverage (HPC) Coverage checker integrated with GHC Instruments a compiled program to track and log code execution Produces coverage statistics and annotations of source listings Implements function coverage, condition coverage & expression coverage

37 Yellow: functions and expression that were not executed Red: conditions that never evaluated to True Green: conditions that never evaluated to False