QuickCheck, SmallCheck & Reach: Automated Testing in Haskell By Tom Shackell
A Brief Introduction to Haskell Haskell is a purely functional language. Based on the idea of evaluation of mathematical functions rather than on manipulating state. Heavy emphasis on 'functions' and data-types. Higher-order functions. Passing functions to other functions. Very powerful static type system including typeinference.
Some Simple Examples... A program for calculating factorials: fac :: Int -> Int fac 0 = 1 fac n = n * fac (n - 1) And for calculating the length of a list: length :: [a] -> Int length [] = 0 length (x:xs) = 1 + length xs
Data Types A data type for colours data Colour = Red Green Blue For binary trees data Tree a = Empty Node a (Tree a) (Tree a)
Higher Order Functions A function to apply some function to every element of a list: map :: (a -> b) -> [a] -> [b] map f [] = [] map f (x:xs) = f x : map f xs for example map (*2) [3,5,9] = [6,10,18]
Class Polymorphism A function to tested whether a list contains a particular element: elem :: Eq a => a -> [a] -> Bool elem x [] = False elem x (y:ys) = x == y elem x ys
Classes The previous example makes use of the Eq class. class Eq a where (==) :: a -> a -> Bool (/=) :: a -> a -> Bool x /= y = not (x == y) We can define instances of this class to describe how to compare any data type for equality. instance Eq Colour where Red == Red = True Green == Green = True Blue == Blue = True _ == _ = False
QuickCheck
QuickCheck Developed by Koen Claessen and John Hughes. Based on the idea of specifying properties that the programmer expects to be true. These properties are then tested with a large number of random inputs. Properties are expressed in Haskell using combinators defined in the QuickCheck library.
Simple QuickCheck Properties We can express a property such as: prop_revrev :: [Int] -> Bool prop_revrev xs = reverse (reverse xs) == xs We can then ask QuickCheck to verify this property. Main> quickcheck prop_revrev OK: Passed 100 tests Or perhaps it fails. Falsifiable, after 12 tests: [3,5,1]
Conditionals We can specify conditional properties. prop_maxle :: Int -> Int -> Property prop_maxle x y = x <= y ==> max x y == y prop_ordinsert :: Int -> [Int] -> Property prop_ordinsert x xs = ordered xs ==> ordered (insert x xs) This second property has a problem...
Trivial Cases The condition in prop_ordinsert is quite restrictive. Most lists generated randomly are not ordered. Of those that are, shorter lists are a lot more likely that long ones. Our test function will likely only test very small cases.
Generators In fact it's better to define this property using a 'generator'. prop_ordinsert :: Int -> Property prop_ordinsert x = forall orderedlist $ \ xs -> ordered (insert x xs) The generator orderedlist generates a random ordered list of numbers.
Generator Instances We can tell QuickCheck how to generate random data of any type by providing an instance of the arbitrary class. class Arbitrary a where arbitrary :: Gen a for example instance Arbitrary Colour where arbitrary = elements [ Red, Green, Blue ]
Generator Functions We can also define generator functions such as orderedlist orderedlist :: (Arbitrary a, Ord a) => Gen [a] orderedlist = do xs <- arbitrary return (sort xs)
How QuickCheck is implemented QuickCheck is implemented as a Haskell library imported into the program rather than an external tool. quickcheck is a Haskell function that takes a function of any number of arguments. It generates random values for the arguments, passes them to the function and observes the result. Internally the implementation relies on some quite clever use of Haskell's class system.
QuickCheck Used for random testing. Easy to use as it's a standard Haskell library. Generally works best for simple data structures. Works best for properties with simple preconditions. Writing generators can be time consuming and tedious.
SmallCheck
SmallCheck Developed by Colin Runciman at York. Design is similar to QuickCheck. However the focus is on finding small counter examples through exhaustive search of the space. Uses a depth-bound on the size of data generated to control search space explosion.
Usage SmallCheck is used in a very similar way to QuickCheck prop_revrev :: [Bool] -> Bool prop_revrev xs = reverse (reverse xs) == xs Main> smallcheck 5 prop_revrev Completed 63 tests without failure Here we've checked all lists of Bools with length less than or equal to 5.
Existentials SmallCheck introduces the idea of existential quantification. prop_isprefix :: [Bool] -> [Bool] -> Property prop_isprefix xs ys = isprefix xs ys ==> exists $ \ zs -> ys == xs ++ zs SmallCheck will search exhaustively for a zs that matches the criteria specified.
SmallCheck Implemented in much the same way as QuickCheck but with exhaustive searching. Has issues with large search spaces. Still requires user to write generators.
Reach
Reach Developed by Matt Naylor and Colin Runciman Is based on the idea of trying to find an input that causes a target expression to be evaluated. Very much based around the idea of avoiding having to write 'generators'. Is implemented using constraint solving and Functional Logic Programming constructs. Like SmallCheck, makes use of a depth-bound.
Target Expressions In Reach the user specifies a target expression that they would like to be executed. This is done using the target function. f xs ys = if ordered xs && ordered ys then target (xs ++ ys) else... Reach will then search for an input to the function f that causes target to be evaluated.
Property Testing This can be used to test properties just like in QuickCheck or SmallCheck. prop_revrev :: [Int] -> Bool prop_revrev xs = reverse (reverse xs) == xs main xs = refute (prop_revrev xs) refute True = True refute False = target False Reach will search for an input that makes the property false.
Constraint Solving Reach does not use exhaustive search as in SmallCheck. Instead it uses constraint solving. At the top level the main function is evaluated with unbound logical variables as the arguments. Evaluation proceeds as normal Haskell but instead of inspecting data structures computation introduces constraints. Evaluation finishes when the target is reached or the depth-bound exceeded.
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) main a {a=?} Stack
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte a (S Z) {a=?} Stack
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte Z (S Z) lte a (S Z) {a=?} Stack
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte Z (S Z) {a=z} lte a (S Z) {a=?} Stack
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) Target not reached! True {a=z} lte a (S Z) {a=?} Stack
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) (S Z) doesn't match Z lte a (S Z) {a=?} Stack
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte (S b) (S Z) {a=s b, b =?} lte a (S Z) {a=?} Stack
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte (S lte x) b Z (S{a=S Z) {x=s b, b y, = y?} =?} lte a (S Z) {a=?} Stack
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte Z Z {a=s b, b = Z} lte (S lte x) b Z (S{a=S Z) {x=s b, b y, = y?} =?} lte a (S Z) {a=?} Stack
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) Target not reached! True {a = S b, b = Z} lte (S lte x) b Z (S{a=S Z) {x=s b, b y, = y?} =?} lte a (S Z) {a=?} Stack
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte (S c) Z {a=s b, b = S c, c =?} lte (S lte x) b Z (S{a=S Z) {x=s b, b y, = y?} =?} lte a (S Z) {a=?} Stack
data Nat = Z S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) Target reached! Answer found: a = S b, b = S c, c =? a = S (S?) target False {a=s b, b = S c, c =?} lte (S lte x) b Z (S{a=S Z) {x=s b, b y, = y?} =?} lte a (S Z) {a=?} Stack
Reach Can be used for many of the same problems as SmallCheck. Is often much more efficient than SmallCheck, especially in the presence of complex antecedents. No need to write 'generators' which can be a big win in certain applications. Implemented as an external tool rather than a Haskell library.
Conclusion Variety of tools for automated testing in Haskell. The ideas from QuickCheck have been used in several other languages (Erlang, Scheme, Lisp, Python, Ruby, SML). Matt Naylor working on SparseCheck a version of Reach used as a Haskell library.