Testing Functional Code

• Mihai Maruseac
• 20th of June 2012

Why testing?

Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning. - Rick Cook, The Wizardry Compiled

• bugs
• nasty bugs
• heisenbugs
• safety
• tent analogy

Testing in imperative world

• JUnit -> CUnit, PyUnit --> HUnit
• test functions, classes, units
• test driven development

Testing in Haskell :: Overview

• QuickCheck
• SmallCheck
• Reach
• SparseCheck
• Haskell Program Coverage
• Haskell Equational Reasoning Assistant

Testing in Haskell :: Main Idea

• test properties of code
• try to generate counter-examples

• or, use equational reasoning

fmap id = id
fmap (g . h) = (fmap g) . (fmap h)

QuickCheck (1)

• Koen Claessen, John Hughes
• specify properties you want to be true
• test with large number of random inputs

import Test.QuickCheck

propID x = id x == x

• test it

QuickCheck (2)

import Test.QuickCheck

propID x = id x == x

*Main> quickCheck propID
+++ OK, passed 100 tests.

QuickCheck (3)

import Test.QuickCheck

propTake xs n = length (take n xs) == n

• looks good?

QuickCheck (4)

import Test.QuickCheck

propTake xs n = length (take n xs) == n

*Main> quickCheck propTake 
*** Failed! Falsifiable (after 1 test and 1 shrink):     
[]
1

QuickCheck (5)

import Test.QuickCheck

propTake xs n = length (take n xs) == n

*Main> quickCheck propTake 
*** Failed! Falsifiable (after 1 test and 1 shrink):     
[]
1

• need to reduce the test case :)
• or write better properties

QuickCheck (6)

import Test.QuickCheck

propTake xs n = n > 0 && length xs >= n ==> length (take n xs) == n

*Main> quickCheck propTake 
*** Gave up! Passed only 7 tests.
*Main> quickCheck propTake
*** Gave up! Passed only 8 tests.

• harder to satisfy the precondition of the property (before the ==> part)
• generators

QuickCheck (7)

import Test.QuickCheck
import Control.Monad

newtype ListWithLength = L ([Int], Int) deriving (Show, Eq)

instance Arbitrary ListWithLength where
  arbitrary = do
    n <- arbitrarySizedIntegral
    r <- arbitrarySizedIntegral
    let n1 = if n < 0 then -n else n
    let r1 = if r < 0 then -r else r
    l <- replicateM (n1 + r1 + 1) arbitrary
    return $ L (l, (n1 + r1 + 1))

propTake :: ListWithLength -> Bool
propTake (L (xs, n)) = length (take n xs) == n

*Main> quickCheck propID
+++ OK, passed 100 tests.

QuickCheck (8)

• 100 tests are too few in some cases

import Test.QuickCheck

propTake1 xs = length (take 105 xs) < 105

*Main> quickCheck propTake1
+++ OK, passed 100 tests.

QuickCheck (9)

import Test.QuickCheck

deepCheck p = quickCheckWith (stdArgs { maxSize = 1000 }) p
propTake1 xs = length (take 105 xs) < 105

*Main> deepCheck  propTake1
*** Failed! Falsifiable (after 13 tests and 1 shrink):       
[(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(),()]
--** Delete me

QuickCheck (10)

• want to print each testcase

*Main> verboseCheck propTake
Passed:  
[]
0
Failed: 
[()]
-1
*** Failed! Failed:                       
[]
-1
Failed:                                    
[]
1
Passed:                                     
[]
0
Falsifiable (after 2 tests and 2 shrinks):    
[]
1
--** delete me

QuickCheck (11)

• Haskell library imported into the program
• quickCheck - function which receives function with any number of arguments
• generate random values for the arguments and observe function's behaviour
• internally - clever use of Haskell's type system
• random testing
• easy to use
• works best for simple data structures
• works best for simple properties
• generators
  • time consuming and tedious

SmallCheck (1)

• Colin Runciman
• similar design to QuickCheck
• focus on finding small counterexamples
  • exhaustive search of state space
• depth bound on size of generated data
  • prevents combinatorial explosion of space

SmallCheck (2)

import Test.SmallCheck

propID x = reverse (reverse x) == x

*Main> smallCheck 10 propID 
Depth 0:
  Completed 1 test(s) without failure.
Depth 1:
  Completed 2 test(s) without failure.
Depth 2:
  Completed 3 test(s) without failure.
Depth 3:
  Completed 4 test(s) without failure.
Depth 4:
  Completed 5 test(s) without failure.
Depth 5:
  Completed 6 test(s) without failure.
Depth 6:
  Completed 7 test(s) without failure.
Depth 7:
  Completed 8 test(s) without failure.
Depth 8:
  Completed 9 test(s) without failure.
Depth 9:
  Completed 10 test(s) without failure.
Depth 10:
  Completed 11 test(s) without failure.
--* Delete me

SmallCheck (3)

import Test.SmallCheck

prop2 xs ys = isPrefix xs ys ==> exists $ \a -> ys == xs ++ a

isPrefix [] _ = True
isPrefix _ [] = False
isPrefix (x:xs) (y:ys)
  | x == y = isPrefix xs ys
  | otherwise = False

*Main> smallCheck 2 prop2
Depth 0:
  Completed 1 test(s) without failure.
Depth 1:
  Completed 4 test(s) without failure.
  But 1 did not meet ==> condition.
Depth 2:
  Completed 9 test(s) without failure.
  But 3 did not meet ==> condition.
--* Delete me

SmallCheck (4)

• similar to QuickCheck but with exhaustive search
• issues with large search spaces
• generators are still required
• LazySmallCheck

Reach (1)

• Matt Naylor, Colin Runciman
• find input which causes target expression to be evaluated
• avoid writing generators
• constraint solving
• Functional Logic Programming
• depth-bound

Reach (2)

• eval main with unbound arguments
• evaluation just like Haskell
  • instead of doing computation introduce constraints
• finish when depth is exceeded or target is reached

Reach (3)

Reach (3)

Reach (3)

Reach (3)

Reach (3)

Reach (3)

Reach (3)

Reach (3)

Reach (3)