Applying Random Testing to a Base Type Environment Experience Report. Vincent St-Amour Neil Toronto PLT

Transcription

1 Applying Random Testing to a Base Type Environment Experience Report Vincent St-Amour Neil Toronto PLT ICFP September 27th, 2013

2 Γbase e : τ

3 Γbase e : τ

4 Γbase e : τ

5 Γbase e : τ

6 Γbase e : τ

7 first : (Listof A) A Γbase e : τ string-append : String * String

8 + : Γbase e : τ

9 + : /* This macro is used to implement most all binary math and comparison e functions : (!): τ*/ #define GEN_BIN_THING(rettype, name, scheme_name, \ iop, fop, fsop, bn_op, rop, Γbase wrap, combineinf, \...

10 + : Γbase e : τ

11 + : 18% of Typed Racket bugs Γbase e : τ { 10.9% numeric Γbase 6.8% other Γbase

12 + : 18% of Typed Racket bugs Γbase e : τ { 10.9% numeric Γbase 6.8% other Γbase

13 + : Γbase e : τ Use random testing

14 What do these bugs look like? How do we find them? How well did that work?

15 What do these bugs look like?

16 A Type Environment Bug (: sinh (case [Float-Zero Float-Zero] [Positive-Float Positive-Float] [Negative-Float Negative-Float]...))

17 A Type Environment Bug τ₁ τ₂ (: sinh (case [Float-Zero Float-Zero] [Positive-Float Positive-Float] [Negative-Float Negative-Float]...))

18 A Type Environment Bug τ₁ τ₂ 10 5 (: sinh (case [Float-Zero Float-Zero] [Positive-Float Positive-Float] [Negative-Float Negative-Float]...))

19 A Type Environment Bug τ₁ τ₂ 10 5 (: sinh (case [Float-Zero Float-Zero] [Positive-Float Positive-Float] [Negative-Float Negative-Float]...)) cases (integers, complexes, exact rationals)

20 A Type Environment Bug (: sinh (case [Float-Zero Float-Zero] [Positive-Float Positive-Float] [Negative-Float Negative-Float]...))

21 A Type Environment Bug (: sinh (case [Float-Zero Float-Zero] [Positive-Float Positive-Float] [Negative-Float Negative-Float]...)) (sinh e-17) 0.0 : Float-Zero

22 A Type Environment Bug (: sinh (case [Float-Zero Float-Zero] [Nonnegative-Float Nonnegative-Float] [Nonpositive-Float Nonpositive-Float]...)) (sinh e-17) 0.0 : Float-Zero

23 A Type Environment Bug (: * (case... [Positive-Real Positive-Real Positive-Real]...))

24 A Type Environment Bug (: * (case... [Positive-Real Positive-Real Positive-Real]...)) (* 5/ e-324) 0.0 : Float-Zero

25 A Type Environment Bug (: * (case... [Nonnegative-Real Nonnegative-Real Nonnegative-Real]...)) (* 5/ e-324) 0.0 : Float-Zero

26 A Type Environment Bug (: * (case... [Nonnegative-Real Nonnegative-Real Nonnegative-Real]...)) (* 5/ e-324) 0.0 : Float-Zero (* +inf.0 0.0) +nan.0 : Float-Nan

27 A Type Environment Bug (: * (case... [Nonnegative-Real Nonnegative-Real (U Nonnegative-Real Float-Nan)]...)) (* 5/ e-324) 0.0 : Float-Zero (* +inf.0 0.0) +nan.0 : Float-Nan

28 How do we find them?

29 Use random testing PLT Redex

30 (define-language λv [e (e e...) (if0 e e e) x v] [v (λ (x...) e) number] [x (variable-except λ if0)]) PLT Redex

31 PLT Redex (define-language λv [e (e e...) (if0 e e e) x v] [v (λ (x...) e) number] [x (variable-except λ if0)]) (define red (reduction-relation λv...))

32 PLT Redex (define-language λv [e (e e...) (if0 e e e) x v] [v (λ (x...) e) number] [x (variable-except λ if0)]) (redex-check λv v (number? (term v))) counterexample found after 4 attempts: (λ () 1)

33 PLT Redex (define-language λv [e (e e...) (if0 e e e) x v] [v (λ (x...) e) number] [x (variable-except λ if0)]) (redex-check λv v (> (n-google-results (term v)) 20)) counterexample found after 15 attempts: (λ (x y) (+ (λ () 3) 2))

34 Testing Type Preservation e ::= n (+ e e)... Generate arithmetic expressions

35 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ Typecheck using Typed Racket

36 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ e e * v Evaluate using Typed Racket

37 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ e e * v v Γ v : τ' Typecheck the result

38 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ e e * v τ v Γ v : τ' τ' : τ' <: τ <: Check consistency

39 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ e e * v τ v Γ v : τ' τ' : τ' <: τ (sinh e-17) <:

40 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ e e * v : τ' <: τ Positive-Float τ v Γ v : τ' τ' (sinh e-17) <:

41 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ e e * v : τ' <: τ Positive-Float 0.0 τ v Γ v : τ' τ' (sinh e-17) <:

42 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ e e * v : τ' <: τ Positive-Float 0.0 τ v Γ v : τ' τ' (sinh e-17) Float-Zero <:

43 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ e e * v : τ' <: τ Positive-Float 0.0 τ v Γ v : τ' τ' (sinh e-17) Float-Zero <: Float-Zero : Positive-Float

44 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ e e * v : τ' <: τ Positive-Float 0.0 τ v Γ v : τ' τ' (sinh e-17) Float-Zero <: Float-Zero : Positive-Float

45 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ e e * v : τ' <: τ Positive-Float 0.0 τ v Γ v : τ' τ' (sinh e-17) Float-Zero <: Float-Zero : Positive-Float

46 Testing Type Preservation e ::= n (+ e e)... e 57.6% initial Γ e:τ (sinh e-17) e * v 0.0 rejection rate Positive-Float 1.6% after grammar engineering e τ v Γ v : τ' τ' : τ' <: τ Float-Zero <: Float-Zero : Positive-Float

47 Testing Type Preservation e ::= n (+ e e)... e Γ e:τ e e * v : τ' <: τ Positive-Float 0.0 τ v Γ v : τ' τ' (sinh e-17) Float-Zero <: Float-Zero : Positive-Float

48 Testing Type Preservation Random floating-point number generation 25% Laplace distribution 8.75% Close to % Close to 0 5% - 25% Uniform random bits 5% 8.75% Close to 5% NaN

49 Testing Type Preservation Random floating-point number generation 25% Laplace distribution 8.75% Close to % Close to 0 5% - 25% Uniform random bits 5% 8.75% Close to 5% NaN

50 How well did that work?

51 Finding Bugs Existing 10+ kloc test suite Found bugs anyway Small random test cases Smaller than user bug reports Even without test case reduction

52 Confidence When Refactoring Fact: programs evolve over time Follow changes with random testing Success stories NaN refactoring Optimizer rewrite

53 The Take-Away Type environments have bugs too! Random testing can help. Redex makes random testing easy.

54 The Take-Away Type environments have bugs too! Random testing can help. Redex makes random testing easy. Thank You