picoq: Parallel Regression Proving for Large-Scale Verification Projects

Transcription

1 picoq: Parallel Regression Proving for Large-Scale Verification Projects Karl Palmskog, Ahmet Celik, and Milos Gligoric The University of Texas at Austin, USA 1 / 29

2 Introduction Verification Using Proof Assistants 1 encode definitions in (higher-order) formalism 2 prove propositions interactively using powerful tactics 3 check soundness of every low-level step proof assistant tactics proof user logic engine proof checker subgoals examples: Coq, HOL4, HOL Light, Isabelle/HOL, Lean, Nuprl,... 2 / 29

3 Introduction Some Large-Scale Proof Assistant Projects Project Year Assistant Check Time LOC 4-Color Theorem 2005 Coq hours 60k Odd Order Theorem 2012 Coq hours 150k Kepler Conjecture 2015 HOL Light days 500k CompCert C compiler 2009 Coq tens of mins 40k Cogent (BilbyFS) 2016 Isabelle/HOL hours 14k Verdi (Raft consensus) 2016 Coq tens of mins 50k problem: long proof checking times 3 / 29

4 Introduction Proof Engineering Techniques For Effective Proving Proof selection: check only proofs affected by changes file/module selection proof selection Examples: Make, Isabelle [ITP 14], icoq [ASE 17] 4 / 29

5 Introduction Proof Engineering Techniques For Effective Proving Proof selection: check only proofs affected by changes file/module selection proof selection Examples: Make, Isabelle [ITP 14], icoq [ASE 17] Proof parallelization: leverage multi-core hardware parallel checking of proofs parallel checking of files Examples: Make, Isabelle [ITP 13], Coq [ITP 15] 4 / 29

6 Introduction Our Contributions taxonomy of regression proving techniques that leverage both selection and parallelism implementation of techniques in tool, picoq, that supports Coq projects (useful for CI, e.g., Travis on GitHub) evaluation using picoq on six open source projects (23 kloc over 22 revisions per project, on average) 5 / 29

7 Background The Coq Proof Assistant (1985-present) based on constructive dependent type theory Gallina programming/specification language Ltac proof tactic language small trusted core checker for programs & proofs 6 / 29

8 Background Coq Source File Example Require Import List. Require Import ListUtil. Import ListNotations. Fixpoint dedup A A_eq_dec (xs : list A) : list A := match xs with [] [] x :: xs if in_dec A_eq_dec x xs then dedup A A_eq_dec xs else x :: dedup A A_eq_dec xs end. Lemma remove_dedup : forall A A_eq_dec (x : A) xs, remove A_eq_dec x (dedup A A_eq_dec xs) = dedup A A_eq_dec (remove A_eq_dec x xs). Proof. induction xs; intros; auto; simpl. repeat (try case in_dec; try case A_eq_dec; simpl; intuition); auto using f_equal. - exfalso. apply n0. apply remove_preserve; auto. - exfalso. apply n. apply in_remove in i; intuition. Qed. Dedup.v 7 / 29

9 Background Coq Source File Example Require Import List. Require Import ListUtil. Import ListNotations. Require statements expressing file dependencies. Fixpoint dedup A A_eq_dec (xs : list A) : list A := match xs with [] [] x :: xs if in_dec A_eq_dec x xs then dedup A A_eq_dec xs else x :: dedup A A_eq_dec xs end. Lemma remove_dedup : forall A A_eq_dec (x : A) xs, remove A_eq_dec x (dedup A A_eq_dec xs) = dedup A A_eq_dec (remove A_eq_dec x xs). Proof. induction xs; intros; auto; simpl. repeat (try case in_dec; try case A_eq_dec; simpl; intuition); auto using f_equal. - exfalso. apply n0. apply remove_preserve; auto. - exfalso. apply n. apply in_remove in i; intuition. Qed. Dedup.v 7 / 29

10 Background Coq Source File Example Require Import List. Require Import ListUtil. Import ListNotations. Fixpoint dedup A A_eq_dec (xs : list A) : list A := match xs with [] [] x :: xs if in_dec A_eq_dec x xs then dedup A A_eq_dec xs else x :: dedup A A_eq_dec xs end. Definition of a recursive function to remove duplicate list elements in Gallina. Lemma remove_dedup : forall A A_eq_dec (x : A) xs, remove A_eq_dec x (dedup A A_eq_dec xs) = dedup A A_eq_dec (remove A_eq_dec x xs). Proof. induction xs; intros; auto; simpl. repeat (try case in_dec; try case A_eq_dec; simpl; intuition); auto using f_equal. - exfalso. apply n0. apply remove_preserve; auto. - exfalso. apply n. apply in_remove in i; intuition. Qed. Dedup.v 7 / 29

11 Background Coq Source File Example Require Import List. Require Import ListUtil. Import ListNotations. Fixpoint dedup A A_eq_dec (xs : list A) : list A := match xs with [] [] x :: xs if in_dec A_eq_dec x xs then dedup A A_eq_dec xs else x :: dedup A A_eq_dec xs end. Lemma remove_dedup : forall A A_eq_dec (x : A) xs, remove A_eq_dec x (dedup A A_eq_dec xs) = dedup A A_eq_dec (remove A_eq_dec x xs). Proof. induction xs; intros; auto; simpl. repeat (try case in_dec; try case A_eq_dec; simpl; intuition); auto using f_equal. - exfalso. apply n0. apply remove_preserve; auto. - exfalso. apply n. apply in_remove in i; intuition. Qed. Dedup.v Statement (type) of a lemma in Gallina. 7 / 29

12 Background Coq Source File Example Require Import List. Require Import ListUtil. Import ListNotations. Fixpoint dedup A A_eq_dec (xs : list A) : list A := match xs with [] [] x :: xs if in_dec A_eq_dec x xs then dedup A A_eq_dec xs else x :: dedup A A_eq_dec xs end. Lemma remove_dedup : forall A A_eq_dec (x : A) xs, remove A_eq_dec x (dedup A A_eq_dec xs) = dedup A A_eq_dec (remove A_eq_dec x xs). Proof. induction xs; intros; auto; simpl. repeat (try case in_dec; try case A_eq_dec; simpl; intuition); auto using f_equal. - exfalso. apply n0. apply remove_preserve; auto. - exfalso. apply n. apply in_remove in i; intuition. Qed. Dedup.v Proof script in Ltac potentially time-consuming to process. 7 / 29

13 Background Coq Proof-Checking Toolchain Legacy Top-Down Proof Checking (1990s) coqc: compilation of source.v files to binary.vo files.vo files contain functions and all proofs file-level parallelism via Make 8 / 29

14 Background Coq Proof-Checking Toolchain Legacy Top-Down Proof Checking (1990s) coqc: compilation of source.v files to binary.vo files.vo files contain functions and all proofs file-level parallelism via Make Quick Compilation and Asynchronous Checking (2015) coqc -quick: compilation of.v files to binary.vio files.vio files contain functions and proof tasks proof tasks checkable asynchronously in parallel 8 / 29

15 Regression Proving Modes for Coq (Taxonomy) Parallelization Selection Granularity None Files Proofs File level f none f file N/A Proof level p none p file p icoq 9 / 29

16 f none Mode: File-Level Parallelization, No Selection Parallelization Selection Granularity None Files Proofs File level f none f file N/A Proof level p none p file p icoq legacy mode used in most GitHub Coq projects no overhead from proof task management or dep. tracking parallelism restricted by file dependency graph overhead from writing proofs to disk 10 / 29

17 f none Mode in Practice ListUtil.v Dedup.v RemoveAll.v 11 / 29

18 f none Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v 11 / 29

19 f none Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vo remove preserve, in remove 11 / 29

20 f none Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vo remove preserve, in remove 2 Dedup.vo dedup, remove dedup 2 RemoveAll.vo remove all, remove all in, remove all preserve 11 / 29

21 p none Mode: Proof-Level Parallelization, No Selection Parallelization Selection Granularity None Files Proofs File level f none f file N/A Proof level p none p file p icoq legacy mode used in some GitHub Coq projects overhead from proof task management parallelism (largely) unrestricted by file dependency graph no overhead from writing proofs to disk and dep. tracking 12 / 29

22 p none Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v 13 / 29

23 p none Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vio remove preserve, in remove 13 / 29

24 p none Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vio remove preserve, in remove 2 Dedup.vio dedup, remove dedup 2 RemoveAll.vio remove all, remove all in, remove all preserve 13 / 29

25 p none Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vio remove preserve, in remove 2 Dedup.vio dedup, remove dedup 2 RemoveAll.vio remove all, remove all in, remove all preserve 3 checking remove preserve 3 checking in remove 3 checking remove dedup 3 checking remove all in 3 checking remove all preserve 13 / 29

26 f file Mode: File-Level Parallelization, File Selection Parallelization Selection Granularity None Files Proofs File level f none f file N/A Proof level p none p file p icoq novel mode that persists file checksums overhead from file dependency tracking parallelism restricted by file dependency graph overhead from writing proofs to disk 14 / 29

27 f file Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v 15 / 29

28 f file Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v 15 / 29

29 f file Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vo remove preserve, in remove 15 / 29

30 f file Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vo remove preserve, in remove 2 Dedup.vo dedup, remove dedup 15 / 29

31 p file Mode: Proof-Level Parallelism, File Selection Parallelization Selection Granularity None Files Proofs File level f none f file N/A Proof level p none p file p icoq novel mode that persists file checksums overhead from file dependency tracking parallelism (mostly) unrestricted by file dependency graph no overhead from writing proofs to disk 16 / 29

32 p file Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v 17 / 29

33 p file Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v 17 / 29

34 p file Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vio remove preserve, in remove 17 / 29

35 p file Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vio remove preserve, in remove 2 RemoveAll.vio remove all, remove all in, remove all preserve 17 / 29

36 p file Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vio remove preserve, in remove 2 RemoveAll.vio remove all, remove all in, remove all preserve 3 checking remove all in 3 checking remove all preserve 17 / 29

37 p icoq Mode: Proof-Level Parallelism, Proof Selection Parallelization Selection Granularity None Files Proofs File level f none f file N/A Proof level p none p file p icoq novel mode that persists file & proof checksums overhead from file & proof dependency tracking parallelism (mostly) unrestricted by file dependency graph no overhead from writing proofs to disk 18 / 29

38 p icoq Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v 19 / 29

39 p icoq Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v 19 / 29

40 p icoq Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v 19 / 29

41 p icoq Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vio remove preserve, in remove 19 / 29

42 p icoq Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vio remove preserve, in remove 2 Dedup.vio dedup, remove dedup 2 RemoveAll.vio remove all, remove all in, remove all preserve 19 / 29

43 p icoq Mode in Practice ListUtil.v remove preserve in remove dedup remove dedup Dedup.v remove all preserve remove all remove all in RemoveAll.v Phase Task Definitions and Lemmas 1 ListUtil.vio remove preserve, in remove 2 Dedup.vio dedup, remove dedup 2 RemoveAll.vio remove all, remove all in, remove all preserve 3 checking in remove 3 checking remove dedup 3 checking remove all in 19 / 29

44 p icoq Workflow with 4-way Parallelization file dep. graph.v files Analysis Checking Collection proof dep. graph 20 / 29

45 p icoq Workflow with 4-way Parallelization Analysis Checking Collection file dep. graph.v files proof dep. graph compilation commands.vio files affected proofs 20 / 29

46 p icoq Workflow with 4-way Parallelization Analysis Checking Collection file dep. graph.v files proof dep. graph compilation commands.vio files affected proofs proof dependencies proofchecking commands 20 / 29

47 p icoq Workflow with 4-way Parallelization Analysis Checking Collection file dep. graph compilation commands proof dependencies new dep. graphs.v files.vio files proof dep. graph affected proofs proofchecking commands storage 20 / 29

48 picoq Tool Implementation extension of icoq toolchain (Java, OCaml, bash) Java task executor for parallel compilation of.vo/.vio files extension of coqc for parallel checking of (and dependency extraction from) specific proofs across files $ coqc -schedule -vio -task - depends - checking 4 \ file1. vio 1,15,16 \ file2. vio 3,10,11,13,20 21 / 29

49 picoq Tool Implementation extension of icoq toolchain (Java, OCaml, bash) Java task executor for parallel compilation of.vo/.vio files extension of coqc for parallel checking of (and dependency extraction from) specific proofs across files $ coqc -schedule -vio -task - depends - checking 4 \ file1. vio 1,15,16 \ file2. vio 3,10,11,13,20 21 / 29

50 picoq Tool Implementation extension of icoq toolchain (Java, OCaml, bash) Java task executor for parallel compilation of.vo/.vio files extension of coqc for parallel checking of (and dependency extraction from) specific proofs across files $ coqc -schedule -vio -task - depends - checking 4 \ file1. vio 1,15,16 \ file2. vio 3,10,11,13,20 21 / 29

51 Evaluation Evaluation: Open Source Git-Based Projects Project LOC Domain Coquelicot real number analysis Finmap 5661 finite sets and maps Flocq floating-point arithmetic Fomegac 2637 formal system metatheory Surface Effects 9621 functional programming languages Verdi distributed systems Avg / 29

52 Evaluation Evaluation: Open Source Git-Based Projects Project LOC #Revs. #Files #Proof Tasks Coquelicot Finmap Flocq Fomegac Surface Effects Verdi Avg / 29

53 Evaluation Evaluation Details Experiment machine: Intel Core i GHz 4 CPU cores 16 GB memory Ubuntu Linux Coq 8.5 Evaluation Setup every build starts from scratch (version control) up to 4 parallel jobs/processes dependency metadata persisted between revisions 24 / 29

54 Evaluation Results with 4-way Parallelization: Coquelicot f none p none f file p file p icoq 40 Time [s] 20 0 befc2aa c83 af196a6f 96153bf b49 7c74911f cc869c0d a94e2add ac f7b7c3f 51ccec65 a21157ac 67c0544f 99d324c8 02aa4835 ef18b8b a43e920b f25236ff e49f02aa fc2b7663 a22616bb a035d3bd 680ca587 Revision 25 / 29

55 Evaluation Results with 4-way Parallelization: Fomegac f none p none f file p file p icoq 30 Time [s] af6b37e ac790c93 a9bcb72a 78a261b b65 ca71b96c 4c1f4b3b 99f338f0 Revision 62d1a9d5 5f156d72 4b9edb51 b d7d688 7a654d7c 26 / 29

56 Evaluation Speedups over f none for 4-way Parallel Checking Speedup factor over f none f none p none f file p file p icoq Coquelicot Finmap Flocq Fomegac Surface Effects Verdi How much faster modes are than the default mode, for each project 27 / 29

57 Evaluation Speedups from Sequential to 4-way Parallel Checking Speedup factor over sequential f none p none f file p file p icoq 0 Coquelicot Finmap Flocq Fomegac Surface Effects Verdi Effect of parallelism on each mode and project 28 / 29

58 Conclusion Conclusion taxonomy of regression proving modes implementation of modes for Coq in tool picoq eval shows speedups for parallelism/selection (up to 28.6 ) Contact us: Karl Palmskog, palmskog@utexas.edu Ahmet Celik, ahmetcelik@utexas.edu Milos Gligoric, gligoric@utexas.edu Resources: Website: GitHub: This work was partially supported by the US National Science Foundation under Grants Nos. CCF and CCF / 29

59 f none vs. p none for 4-way Parallel Checking Coquelicot Finmap Flocq Fomegac Surface Effects Verdi 30 / 29

60 f none vs. f file for 4-way Parallel Checking Coquelicot Finmap Flocq Fomegac Surface Effects Verdi 31 / 29

61 p none vs. p file for 4-way Parallel Checking Coquelicot Finmap Flocq Fomegac Surface Effects Verdi 32 / 29

62 p file vs. p icoq for 4-way Parallel Checking Coquelicot Finmap Flocq Fomegac Surface Effects Verdi 33 / 29

63 Regression Proving in Evolving Projects Typical proving scenario: 1 change definition or lemma statement 2 begin process of re-checking all proofs 3 checking fails much later (for seemingly unrelated proof) Typical testing scenario: 1 change method statements or method signature 2 begin process of re-running all tests 3 testing fails much later (for seemingly unrelated test) 34 / 29

64 Software Engineering Techniques For Effective Testing Test selection: run only tests affected by changes file/class selection method selection hybrid Examples: Ekstazi [ISSTA 15], STARTS [ASE 17], HyRTS [ICSE 18] Test parallelization: leverage multi-core hardware parallel threads parallel processes (VM forking) hybrid Examples: Gradle, Maven, JUnit 35 / 29

65 Regression Proving vs. Regression Testing proof checking is deterministic proof checking has no side effects (e.g., I/O) only file-level deps. relevant for (asynch) proof checking 36 / 29