Counterexample-Driven Genetic Programming

Transcription

1 Counterexample-Driven Genetic Programming Iwo Błądek, Krzysztof Krawiec Institute of Computing Science, Poznań University of Technology Poznań, I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 1 / 58

2 Plan of the Presentation 1 Introduction 2 Counterexample-Driven GP (part 1) 3 Satisfiability Modulo Theories (SMT) 4 SMT-based verification 5 Counterexample-Driven GP (part 2) 6 Experiments I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 2 / 58

3 Program Synthesis Aim of program synthesis is to find a program given its desired behavior. bool isgreater(int x, int y) { if (x > y) return true;? else return false; } a b c? x 2 +? x 2 x = 1, y = 5 x = 5, y = 1 a = 1, b = 0, c = 0 a = 0, b = 0, c = 0 x = 1 x = 0 false true I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 3 / 58

4 Program Specification The desired behavior of a program is described by its specification. The most commonly used types of specifications are: Input-output examples (test cases) x y max(x,y) Formal specification (logical relations between input and output) x,y max(x, y) x max(x, y) y (max(x, y) = x max(x, y) = y) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 4 / 58

5 Program Specification Input-output examples are a special case of formal specification: x,y ([(x = 0 y = 0) = max(x, y) = 0] [(x = 1 y = 0) = max(x, y) = 1] [(x = 4 y = 3) = max(x, y) = 4] ) Or simpler: max(0, 0) = 0 max(1, 0) = 1 max(3, 4) = 4 I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 4 / 58

6 Provably-correct programs To prove that program is correct for all inputs, a formal specification needs to be used (or all possible input-output examples enumerated) In certain applications hoping for algorithm s generalization is not enough: Safety-critical systems (!) Software development Hardware design Finding mathematical structures with certain properties I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 5 / 58

7 Plan of the Presentation 1 Introduction 2 Counterexample-Driven GP (part 1) 3 Satisfiability Modulo Theories (SMT) 4 SMT-based verification 5 Counterexample-Driven GP (part 2) 6 Experiments I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 6 / 58

8 Counterexample-Driven GP First described in: K.Krawiec, I.Błądek, J.Swan, Counterexample-Driven Genetic Programming, Proceedings of the Genetic and Evolutionary Computation Conference. GECCO 17. Berlin, Germany. [1] (Best Paper Award in the Genetic Programming track) Follow-up: submitted to Evolutionary Computation Journal I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 7 / 58

9 Counterexample-Driven GP Goal: Automatic synthesis of programs which are proved to correctly realize a formally defined task Contributions: GP algorithm capable of synthesizing such programs from formal specifications (spec for short) Systematization of spec properties and their consequences for program evaluation I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 8 / 58

10 Counterexample-Driven GP Main ideas: Verify programs generated by GP to guarantee their correctness Incrementally supply GP with test cases created from counterexamples returned by failed verifications GP search Program Fitness Testing T c All passed? Verification Counterexample I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 9 / 58

11 Plan of the Presentation 1 Introduction 2 Counterexample-Driven GP (part 1) 3 Satisfiability Modulo Theories (SMT) 4 SMT-based verification 5 Counterexample-Driven GP (part 2) 6 Experiments I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 10 / 58

12 Satisfiability Problem (SAT) Question: Is a formula in the propositional calculus satisfiable? Examples: a b SAT: a = false, b = true a a b UNSAT I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 11 / 58

13 Satisfiability Modulo Theories (SMT) Question: Is a formula in the first-order logic satisfiable under the background theory T, which defines semantics of a certain set of symbols? Examples: Logic: QF_LIA (Quantifier-Free Linear Integer Arithmetic) x, y, z Z a {false, true} (10 x = 20) a SAT: x = 2, a = true (x < y) (y < z) (z < x) UNSAT (x y) (y z) (z x) SAT: x = 0, y = 0, z = 0 I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 12 / 58

14 Satisfiability Modulo Theories (SMT) Question: Is a formula in the first-order logic satisfiable under the background theory T, which defines semantics of a certain set of symbols? Examples: Logic: NIA (Non-Linear Integer Arithmetic) x, y Z x x SAT: x = 1 x,y (x + y) 2 > x 2 + y 2 UNSAT I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 12 / 58

15 Satisfiability Modulo Theories (SMT) Question: Is a formula in the first-order logic satisfiable under the background theory T, which defines semantics of a certain set of symbols? Examples: Logic: QF S (Quantifier-Free Strings) x, y String str.len(x ++ y) str.len(x) + str.len(y) UNSAT str.substr(x, 0, 2) = str.substr(x, 2, 2) str.at(x, 0) str.at(x, 1) SAT: x = "abab" I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 12 / 58

16 SMT Solvers SMT Solver any software that can check satisfiability of formulas modulo some given theory. There are several freely accessible SMT solvers: CVC4 (open source) MathSAT (free for non-commercial use) Z3, by Microsoft Research (open source) Most SMT solvers accept queries in SMT-LIB language ( which was created to standardize interaction with different solvers. I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 13 / 58

17 Short SMT-LIB course (set-logic LIA) sets theory to Ints and limits considerations to only linear integer arithmetic I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 14 / 58

18 Short SMT-LIB course (set-logic LIA) sets theory to Ints and limits considerations to only linear integer arithmetic (declare-fun x () Int) declares a free variable x of type Int I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 14 / 58

19 Short SMT-LIB course (set-logic LIA) sets theory to Ints and limits considerations to only linear integer arithmetic (declare-fun x () Int) declares a free variable x of type Int (define-fun fun ((a Int)(b Int)) Int (+ a b)) defines a function for later use I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 14 / 58

20 Short SMT-LIB course (set-logic LIA) sets theory to Ints and limits considerations to only linear integer arithmetic (declare-fun x () Int) declares a free variable x of type Int (define-fun fun ((a Int)(b Int)) Int (+ a b)) defines a function for later use (assert (= (fun x 2) 0)) adds an assertion (constraint) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 14 / 58

21 Short SMT-LIB course (set-logic LIA) sets theory to Ints and limits considerations to only linear integer arithmetic (declare-fun x () Int) declares a free variable x of type Int (define-fun fun ((a Int)(b Int)) Int (+ a b)) defines a function for later use (assert (= (fun x 2) 0)) adds an assertion (constraint) (check-sat) a command instructing solver to check satisfiability I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 14 / 58

22 Plan of the Presentation 1 Introduction 2 Counterexample-Driven GP (part 1) 3 Satisfiability Modulo Theories (SMT) 4 SMT-based verification 5 Counterexample-Driven GP (part 2) 6 Experiments I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 15 / 58

23 SMT-based program verification To check program correctness we need to prove: which can be posed as disproving: in Pre(in) = Post(in, p(in)) in Pre(in) = Post(in, p(in)). p a program in inputs to the program Pre(in) precondition. Behavior of the program is defined only for inputs that satisfy Pre(in). Post(in, out) postcondition. Describes the expected behavior of the program. I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 16 / 58

24 An example of SMT verification Task: Synthesize a program max(x, y) returning a maximum of two numbers. max(x, y) = { x, if x y. y, otherwise. Formal specification: max(x, y) x max(x, y) y (max(x, y) = x max(x, y) = y) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 17 / 58

25 An example of SMT verification Incorrect program: if x < y: res = x else: res = y Program encoded as SMT formulas: ; PROGRAM (=> (< x y) (and (= res x) (= res res))) (=> (not (< x y)) (and (= res y) (= res res ))) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 18 / 58

26 An example of SMT verification Verification constraints in SMT-LIB for the incorrect program: ; PRECONDITION (assert true) ; PROGRAM (assert (=> (< x y) (and (= res x) (= res res)))) (assert (=> (not (< x y)) (and (= res y) (= res res )))) ; POSTCONDITION (assert (not (and (>= res x) (>= res y) (or (= res x) (= res y))))) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 19 / 58

27 An example of SMT verification Incorrect program: if x < y: res = x else: res = y Solver result: SAT x = -1 y = 0 This answer means that the program is not correct and solver provides us a counterexample. I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 20 / 58

28 An example of SMT verification Correct program: if x > y: res = x else: res = y Solver result: UNSAT This answer means that the program is correct with respect to the specification. No counterexample was found. I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 21 / 58

29 Plan of the Presentation 1 Introduction 2 Counterexample-Driven GP (part 1) 3 Satisfiability Modulo Theories (SMT) 4 SMT-based verification 5 Counterexample-Driven GP (part 2) 6 Experiments I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 22 / 58

30 Counterexample-Driven GP Main ideas: Verify programs generated by GP to guarantee their correctness Incrementally supply GP with test cases created from counterexamples returned by failed verifications GP search Program Fitness Testing T c All passed? Verification Counterexample I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 23 / 58

31 Program representation Programs in SMT-LIB Language Purely functional Represented in GP as traditional program trees Examples: x (mod x 2) (* 2 (ite (>= x y) 2 (+ x y))) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 24 / 58

32 Search operators Mutation (50%) / Crossover (50%) Only well-typed programs are generated Examples: Mutation: (ite (>= x y) 2 (+ x y)) (ite (< y 0) 2 (+ x y)) Crossover: (mod x 2) (ite (>= x y) 2 (+ x y)) (mod (+ x y) 2) (ite (>= x y) 2 x) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 25 / 58

33 Verification Generated solutions are verified if they pass a certain ratio of test cases (TestsRatio) unsat program correct sat program incorrect Verification query to SMT solver: (set-logic LIA) (define-fun max ((x Int)(y Int)) Int (ite (>= y x) x y)) (declare-fun x () Int) (declare-fun y () Int) (assert (not (and (>= (max x y) x) (>= (max x y) y) (or (= (max x y) x) (= (max x y) y))))) (check-sat) (get-value (x y)) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 26 / 58

34 Example run Log for Median3 problem: Tests: found: 0 total: 0 known outputs: 0 Added test: (Map(a -> -70, b -> 0, c -> -70),Some(-70)) Added test: (Map(a -> -1, b -> -8, c -> 0),Some(-1)) Added test: (Map(a -> 0, b -> 5, c -> 0),Some(0)) Added test: (Map(a -> 0, b -> 0, c -> 1),Some(0)) Added test: (Map(a -> -2, b -> -1, c -> -2),Some(-2)) Added test: (Map(a -> -1, b -> 0, c -> 0),Some(0)) Added test: (Map(a -> 0, b -> 0, c -> -3),Some(0)) Added test: (Map(a -> 0, b -> 1, c -> -1),Some(0)) Tests: found: 8 total: 8 known outputs: 8 Added test: (Map(a -> -1, b -> 0, c -> -1),Some(-1)) Tests: found: 1 total: 9 known outputs: 9 Added test: (Map(a -> -7, b -> -6, c -> -7),Some(-7)) Tests: found: 1 total: 10 known outputs: 10 Tests: found: 0 total: 10 known outputs: 10 Tests: found: 0 total: 10 known outputs: 10 I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 27 / 58

35 Different types of formal specifications Spec 1: f (x, y) x f (x, y) y (f (x, y) = x f (x, y) = y) Spec 2: (in 2 = 0 = f (in 2 ) = 0) (in 2 > 0 = f (in 2 ) = in 2 + 1) Spec 3: f (x, y + 1) = f (x, y) + 1 f (x + 1, y) = f (x, y) + 1 (x = 0 y = 0 = f (x, y) = 0) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 28 / 58

36 Single-invocation property In the constraints every call to synthesis-function has the same arguments. Correctness of a function on a single input does not depend on function s values on other inputs. Can be checked by syntactic analysis. Spec 1: single-invocation f (x, y) x f (x, y) y (f (x, y) = x f (x, y) = y) Invocations: (x, y) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 29 / 58

37 Single-invocation property In the constraints every call to synthesis-function has the same arguments. Correctness of a function on a single input does not depend on function s values on other inputs. Can be checked by syntactic analysis. Spec 3: multiple-invocation f (x, y + 1) = f (x, y) + 1 f (x + 1, y) = f (x, y) + 1 (x = 0 y = 0 = f (x, y) = 0) Invocations: (x, y + 1), (x + 1, y), (x, y) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 29 / 58

38 Single-output property For a single input there is only one correct output. Can be considered locally (single input) or globally (all inputs). Can be checked by a query to SMT solver. Spec 1: single-output f (x, y) x f (x, y) y (f (x, y) = x f (x, y) = y) The maximum of two numbers is strictly defined as a concrete value. Compare this with f (x) 0 property, for which there is infinitely many correct outputs for a single input. I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 30 / 58

39 Single-output property For a single input there is only one correct output. Can be considered locally (single input) or globally (all inputs). Can be checked by a query to SMT solver. Spec 2: multiple-output (in 2 = 0 = f (in 2 ) = 0) (in 2 > 0 = f (in 2 ) = in 2 + 1) For in 2 < 0 outputs of f are undefined, which means that any integer is a valid output. I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 30 / 58

40 Single-output property Query to SMT solver (checking globally): (set-logic LIA) (declare-fun out1 () Int) (declare-fun out2 () Int) (define-fun max 1 ((x Int)(y Int)) Int out1) (define-fun max 2 ((x Int)(y Int)) Int out2) (declare-fun x () Int) (declare-fun y () Int) (assert (>= (max 1 x y) x)) (assert (>= (max 1 x y) y)) (assert (or (= x (max 1 x y)) (= y (max 1 x y)))) (assert (>= (max 2 x y) x)) (assert (>= (max 2 x y) y)) (assert (or (= x (max 2 x y)) (= y (max 2 x y)))) (assert (distinct out1 out2)) (check-sat) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 31 / 58

41 Single-output property Queries to SMT solver (checking locally): (set-logic LIA) (declare-fun out () Int) (define-fun max ((x Int)(y Int)) Int out) (define-fun x () Int (- 1)) (define-fun y () Int 0) (assert (>= (max x y) x)) (assert (>= (max x y) y)) (assert (or (= (max x y) x) (= (max x y) y))) ; ; Added in the 2nd query, after 0 is found to be a correct output. ; If other correct output is found, then single-output does not hold. (assert (distinct out 0)) ; (check-sat) (get-value (out)) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 32 / 58

42 Different types of formal specifications Spec 1: single-invocation, single-output standard evaluation f (x, y) x f (x, y) y (f (x, y) = x f (x, y) = y) Spec 2: single-invocation, multiple-output evaluation by solver (in 2 = 0 = f (in 2 ) = 0) (in 2 > 0 = f (in 2 ) = in 2 + 1) Spec 3: multiple-invocation, single-output evaluation by solver f (x, y + 1) = f (x, y) + 1 f (x + 1, y) = f (x, y) + 1 (x = 0 y = 0 = f (x, y) = 0) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 33 / 58

43 Standard evaluation Program tree is recursively reduced to a constant by function applications on leaves Obtained constant is then compared with the expected output Requires single-invocation for problem, and single-output for the considered input Example: (ite (>= x y) y (+ x y)) x 3, y 4 1. (ite (>= 3 4) 4 (+ 3 4)) 2. (ite (>= 3 4) 4 7) (expected answer) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 34 / 58

44 Evaluation by solver Query to SMT solver: (set-logic LIA) (define-fun max ((x Int)(y Int)) Int (ite (>= x y) y x)) (define-fun x () Int (- 1)) (define-fun y () Int 0) (assert (>= (max x y) x)) (assert (>= (max x y) y)) (assert (or (= (max x y) x) (= (max x y) y)))) (check-sat) sat correct behavior of the program on the test unsat incorrect behavior I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 35 / 58

45 Plan of the Presentation 1 Introduction 2 Counterexample-Driven GP (part 1) 3 Satisfiability Modulo Theories (SMT) 4 SMT-based verification 5 Counterexample-Driven GP (part 2) 6 Experiments I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 36 / 58

46 Benchmarks LIA Benchmarks marked with * were created by us. Other were taken from SyGuS competition ( Name Arity Semantics *CountPos 2, 3, 4 The number of positive arguments *IsSeries 3, 4 Do arguments form an arithmetic series? *IsSorted 4, 5 Are arguments in ascending order? *Median 3 The median of arguments *Range 3 The range of arguments Max 4 The maximum of arguments Search 2, 3, 4 The index of an argument among the other arguments Sum 2, 3, 4 The sum of arguments I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 37 / 58

47 LIA Example: CountPos2 (set-logic LIA) (synth-fun countpositive2 ((a Int) (b Int)) Int) (declare-var a Int) (declare-var b Int) (constraint (=> (and (<= a 0) (<= b 0)) (= (countpositive2 a b) 0))) (constraint (=> (and (> a 0) (<= b 0)) (= (countpositive2 a b) 1))) (constraint (=> (and (<= a 0) (> b 0)) (= (countpositive2 a b) 1))) (constraint (=> (and (> a 0) (> b 0)) (= (countpositive2 a b) 2))) (check-synth) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 38 / 58

48 Benchmarks SLIA Benchmarks are based on those from SyGuS competition. The test-based specification was converted to a formal one covering all inputs. Name Arity Semantics dr-name 1 Extract first name from full name and prepend it with Dr. firstname 1 Extract first name from full name initials 1 Extract initials name from full name lastname 1 Extract last name from full name combine 2 Combine first and last name into full name combine-2 2 Combine first and last name into first name followed by initial combine-3 2 Combine first and last name into initial followed by last name combine-4 2 Combine first and last name into last name followed by initial phone 1 Extract the first triplet of digits from a phone number phone-1 1 Extract the second triplet of digits from a phone number phone-2 1 Extract the third triplet of digits from a phone number phone-3 1 Put first three digits of a phone number in parentheses phone-4 1 Change all - in a phone number to. I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 39 / 58

49 SLIA Example: combine-4 (set-logic SLIA) (synth-fun f ((name String)) String SLIA_GRAMMAR) (declare-var firstname String) (declare-var lastname String) ; (constraint (= (f "Launa" "Withers") "Withers, L.")) ; (constraint (= (f "Lakenya" "Edison") "Edison, L.")) ; (constraint (= (f "Brendan" "Hage") "Hage, B.")) (constraint (= (f firstname lastname) (str.++ lastname ", " (str.at firstname 0) "." ) )) (check-synth) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 40 / 58

50 SLIA Example: dr-name (set-logic SLIA) (synth-fun f ((name String)) String SLIA_GRAMMAR) (define-fun ithsplit((s String)(delim String)(i Int))...) (define-fun precond ((s String)) Bool (and (distinct (str.indexof s " " 0) (- 1))... (distinct (str.at s (- (str.len s) 1)) " "))) (declare-var s String) ; (constraint (= (f "Nancy FreeHafer") "Dr. Nancy")) ; (constraint (= (f "Mariya Sergienko") "Dr. Mariya")) ; (constraint (= (f "Jan Kotas") "Dr. Jan")) (constraint (=> (precond s) (= (str.len (f s)) (+ (str.indexof s " " 0) 4)))) (constraint (=> (precond s) (= (ithsplit (f s) " " 0) "Dr."))) (constraint (=> (precond s) (= (ithsplit (f s) " " 1) (ithsplit s " " 0) ))) (check-synth) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 41 / 58

51 Experiments approaches CDGP GPR our baseline, which adds random tests instead of those found during verification EUSolver enumerates solutions and tries to unify already collected short programs into longer ones CVC4 refutation-based SMT approach to synthesis I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 42 / 58

52 Experiments configurations EvolutionMode x Selection x TestsRatio EvolutionMode: generational, steady state Selection: Tournament7, Lexicase TestsRatio (CDGP): 0.0, 0.25, 0.5, 0.75, 1.0 TestsRatio (GPR): 0.75, 1.0 I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 43 / 58

53 Experiments parameters Parameter Value Number of runs 25 Population size 500 Maximum height of initial programs 5 Maximum height of trees inserted by mutation 5 Maximum height of programs in population 12 Maximum number of generations Maximum runtime in seconds 3600 Probability of mutation 0.5 Probability of crossover 0.5 Tournament size 7 I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 44 / 58

54 Experiments LIA Success rate CDGP GPR EUSolver CVC4 Gener. SteadySt. Gener. SteadySt. Tour Lex Tour Lex Tour Lex Tour Lex CountPos CountPos CountPos IsSeries IsSeries IsSorted IsSorted Max Median Range Search Search Search Sum Sum Sum All I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 45 / 58

55 Experiments LIA Success rate (2) CDGP GPR EUSolver CVC4 Gener. SteadySt. Gener. SteadySt CountPos CountPos CountPos IsSeries IsSeries IsSorted IsSorted Max Median Range Search Search Search Sum Sum Sum All I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 46 / 58

56 Experiments LIA Average runtime CDGP GPR EUSolver CVC4 Gener. SteadySt. Gener. SteadySt. Tour Lex Tour Lex Tour Lex Tour Lex CountPos CountPos CountPos IsSeries IsSeries IsSorted IsSorted Max Median Range Search Search Search Sum Sum Sum All I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 47 / 58

57 Experiments LIA Average runtime (2) CDGP GPR EUSolver CVC4 Gener. SteadySt. Gener. SteadySt CountPos CountPos CountPos IsSeries IsSeries IsSorted IsSorted Max Median Range Search Search Search Sum Sum Sum All I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 48 / 58

58 Experiments LIA Average generated tests (2) CDGP GPR Gener. SteadySt. Gener. SteadySt CountPos CountPos CountPos IsSeries IsSeries IsSorted IsSorted Max Median Range Search Search Search Sum Sum Sum All I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 49 / 58

59 Experiments SLIA Success rate CDGP CVC4 1.5 CVC4 head Gener. SteadySt. Tour Lex Tour Lex dr-name firstname initials lastname name-combine name-combine name-combine name-combine phone phone phone phone phone All I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 50 / 58

60 Experiments SLIA Success rate (2) CDGP CVC4 1.5 CVC4 head Gener. SteadySt dr-name firstname initials lastname name-combine name-combine name-combine name-combine phone phone phone phone phone All I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 51 / 58

61 Experiments SLIA Average runtime CDGP CVC4 1.5 CVC4 head Gener. SteadySt. Tour Lex Tour Lex dr-name firstname initials lastname name-combine name-combine name-combine name-combine phone phone phone phone phone All I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 52 / 58

62 Experiments SLIA Average runtime (2) CDGP CVC4 1.5 CVC4 head Gener. SteadySt dr-name firstname initials lastname name-combine name-combine name-combine name-combine phone phone phone phone phone All I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 53 / 58

63 Experiments SLIA Average generated tests (2) CDGP Gener. SteadySt dr-name firstname initials lastname name-combine name-combine name-combine name-combine phone phone phone phone phone All I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 54 / 58

64 Experiments Average ranks for Friedman test (LIA): GL75 SL75 GT75 ST75 GL5 GL1 GT1 GL25 SL5 GL0 SL1 GT5 SL0 ST SL25 GPRGL1 GPRGT1 GT25 GT0 GPRGL75 ST25 ST1 GPRGT75 ST0 GPRST1 GPRSL75 GPRSL1 GPRST Post-hoc analysis using symmetry test (described by Hollander, et al.) All CDGP configurations with q = 0.75 are better than all GPR configurations (p < 0.05) (except for ST75) I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 55 / 58

65 Experiments Average ranks for Friedman test (SLIA): ST1 5.4 GL SL SL ST GT SL1 7.0 ST ST5 7.5 GT GL GL GT1 9.0 SL5 9.3 GL1 9.4 GT ST GT GL SL Post-hoc analysis using symmetry test (described by Hollander, et al.) Most of pairwise differences are statistically insignificant I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 56 / 58

66 Comparison with formal synthesizers I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 57 / 58

67 Bibliography I [1] Krzysztof Krawiec, Iwo Błądek, and Jerry Swan. Counterexample-driven Genetic Programming. In: Proceedings of the Genetic and Evolutionary Computation Conference. GECCO 17. Berlin, Germany: ACM, 2017, pp I. Błądek, K. Krawiec Counterexample-Driven Genetic Programming 58 / 58