Improving Coverage Analysis

Size: px

Start display at page:

Download "Improving Coverage Analysis"

Elfrieda Chandler
6 years ago
Views:

1 Improving Coverage Analysis 1

2 Ryan Davis Coding professionally 25 years, 16 years Ruby Founder, Seattle.rb; First & Oldest Ruby Brigade Author: minitest, flog, flay, debride, ruby_parser, etc. ~114 million downloaded gems (114/10648 = 1%!) Developer s Developer: I building tools Run Seattle.rb Consulting Most text of any slide 2

3 Setting Expectations Conceptual / Idea Talk Beginner+ 144 slides = 3-5 spm (easy pace) Shooting for ~30 minutes 3

4 Introduction 4

5 As a Consultant 5

6 I get clients 6

7 Huge/Messy Implementations 7

8 Tiny/No Tests 8

9 Making working on it dangerous/impossible 9

10 Pair 100%? 10

11 (Not Realistic) 11

12 PRs in Limbo 12

13 for Months 13

14 It s Frustrating 14

15 To Be a Fixer 15

16 But Not Fix Things 16

17 Being a Tool Builder 17

18 Lots of tools 18

19 To Help Me 19

20 Flog 20

21 Flay 21

22 Debride 22

23 Minitest 23

24 Minitest-bisect 24

25 But 25

26 Without Tests? 26

27 I m not getting very far 27

28 How to Improve Tests? 28

29 Code Coverage 29

30 What is Code Coverage Analysis? 30

31 Miller & Maloney Communications of the ACM 31

32 in 1963!!! 32

33 Everything good in computing 33

34 is old. 34

35 Like me. 35

36 The Measure of the Amount of the Code that the Tests execute 36

37 Huh? 100% class Adder Covered! def initialize n = n class TestAdder < Minitest::Test def test_add assert_equal 3, Adder.new(1).add(2) def add + m end end end end 37

38 Lots of Types 38

39 C0-C2 C0 Statement/Line coverage C1 Branch coverage C2 Condition coverage 39

40 MyClass#method1 Function MyClass#method2 MyClass#method3 Coverage 40

41 Statement executed? (C0) 41

42 Branches if condition then # truthy else # falsey end exhausted? (C1) 42

43 Conditions if (a or b) and c then # truthy exhausted? else end # falsey (C2) 43

44 (a or b) and c > > > > > > > > > > > > > > 1 and Decision Coverage truthy decision falsey Modified Decision/ Condition Coverage or and Full Condition Coverage a or b c 44

45 Parameter Strings: Null Edge Case Coverage Empty Whitespace Valid Format Invalid Format Single-Byte Multi-Byte 45

46 1 2 Path Coverage 6 46

47 Entry/Exit Coverage def return unless? if then x = y return z end x end 47

48 def initialize str = str end Coverage 48

49 Warning! 49

50 Bad Metrics Ahead! 50

51 This is NOT a Dilbert Cartoon 51

52 Because Fuck Scott Adams 52

53 Soooo 53

54 People Get Caught Up 54

55 In Getting High Scores 55

56 Leading to Gaming the System 56

57 But High Code Coverage 57

58 IS NOT 58

59 Proof of Good Tests 59

60 But we have 110% code coverage, we can't have bugs Actual Quote 60

61 Proof class Adder def initialize = n end def add + m end end 61

62 Proof class TestAdder < Minitest::Test def test_add assert_equal 3, Adder.new(1).add(2) end end 62

63 Proof class TestAdder < Minitest::Test def test_add assert_equal 3, Adder.new(1).add(2) end end 63

64 Proof Still 100% class TestAdder < Minitest::Test Covered!?! def test_add Adder.new(1).add(2) end end 64

65 To the Rescue! TDD 65

66 Red Green Refactor 66

67 Natural Fix 67

68 Many other Benefits 68

69 Current State of the Art 69

70 Coverage 70

71 Nobody Knows About It 71

72 Because 72

73 Ships w/ Ruby 73

74 Coverage require "coverage" Coverage.start # must be first-ish require "the_code" require "the_tests" Tests.run Coverage.result # => { } 74

75 Coverage pp Coverage.result { "/full/path_to_file.rb" => [ nil, # comment/blank ], 0, # not covered 1+ # covered N times } 75

76 Not For Human Consumption 76

77 How does it Work? 77

78 Hooks in VM 78

79 Coverage.start -> sets hash 79

80 Load or Eval 80

81 Extra VM Instructions sasm: +trace ( 12) putspecialobject 1 ( 12) putobject :foo putiseq foo send <callinfo!mid:core#define_method, argc:2, ARGS_SIMPLE>, <callcache>, nil pop +trace ( 16) putself ( 16) send <callinfo!mid:foo, argc:0, FCALL VCALL ARGS_SIMPLE>, <callcache>, nil putobject 10 send <callinfo!mid:+, argc:1, ARGS_SIMPLE>, <callcache>, nil leave sasm: #<ISeq:foo@<compiled>>======================================= +trace ( 13) putself ( 13) send <callinfo!mid:bar, argc:0, FCALL VCALL ARGS_SIMPLE>, <callcache>, nil leave 81

82 tests run code 82

83 increments counters 83

84 Coverage.result -> returns & clears hash 84

85 Coverage.peek_result -> returns hash 85

86 Simplecov 86

87 Code Example require "simplecov" SimpleCov.start # must be first-ish require "the_code" require "the_tests" Tests.run 87

88 Simplecov Uses Coverage Internally. Improves Reporting. 88

89 89

90 90

91 Not Much Else 91

92 The Problem 92

93 But First 93

94 Types of Errors 94

95 Statistics 95

96 Type 1: False Positive 96

97 class X class Y if else end if else end class TestX assert = = 75% 97

98 Type 2: False Negative 98

99 class X class Y if else end if else end??? class TestX assert class TestQ assert n 75% 99

100 Numerator Errors 100

101 Type 3 : Error by Omission 101

102 class X class Y class Z if else end if else end????????????????????????????????????????????????????? class TestX assert n m =??% 102

103 Denominator Error 103

104 OK 104

105 How Coverage Sucks 105

106 Type 1 Errors: Macro Type 1 Errors: Micro Type 2 Errors: None Type 3 Errors: Maybe 106

107 Type 1 Error: Macro class X class Y class Z if else end if else end Never Directly Tested if else end class TestX class TestY class TestZ assert assert assert 107

108 Type 1Error: Micro buggy_response if false # line covered raise "haha" unless true # line covered a? b : c a? b : c a = b rescue c a = b c # line covered # line covered # line covered # line covered has_many :problems # line covered a; b; c; d; e; # line covered i_raise; d; e; # line covered (and so on ) 108

109 No Type 2 Errors in simplecov 109

110 Type 3 Errors class X class Y Never class Z Rails Loaded Lazy if else end if else end if else end Loading class TestX class TestY Don't class TestZ Don't Write Run assert assert assert 110

111 How can it be improved? 111

112 minitest-coverage 112

113 Uses Coverage 113

114 Extends C API (optionally?) Coverage.result= 114

115 New Strategy 115

116 Record Baseline = Fix Error by Omission 116

117 Baseline? a minimum or starting point used for comparisons. 117

118 Loading Everything + Running Nothing = Baseline 118

119 Creating a Baseline ENV["RAILS_ENV"] = "production" require "coverage/start" require "./config/environment.rb" Dir["app/**/*.rb", "lib/**/*.rb"].each do path end require path require "coverage/pruner" # (or just use `minitest_coverage_baseline`) 119

120 Only record coverage for CUT 120

121 Record for CUT class X class Y if else end if else end class TestX class TestY assert assert 121

122 class X if else end Hard to do class TestX assert Map Test Class to Implementation 122

123 Coverage is by Path { "/full/path_to_file.rb" => [ nil, # comment/blank ], 0, # not covered 1+ # covered N times } 123

124 Classes Path % ag class.something app/models/something_else.rb: 1: class Something 124

125 Test Names Impl Names Valuations::UpdateWeightedExit ValuationWeightedExitTest 125

126 Test Names Impl Names app/services/valuations/updated_weighted_exit.rb: module Valuations class UpdateWeightedExit test/unit/services/valuations/weighted_exit_test.rb: class ValuationWeightedExitTest 126

127 Biased Towards False Negatives 127

128 Increased Type 2 Errors 128

129 Fixed by Naming Conventions 129

130 simplecov: 83% 130

131 + Baseline: 51% 131

132 Fixed Type 2s: 62% 132

133 Report Output CompanyTest::Create:... app/models/company.rb: from 0.0% to 28.6% of 228 lines CompanyTest::Modify:... app/models/company.rb: from 28.6% to 45.6% of 228 lines 133

134 Difference in Numbers uncv covr% totl : path % 422 : app/models/report.rb % 319 : app/controllers/transactions_controller.rb % 198 : app/concerns/formula.rb % 180 : app/controllers/workflows_controller.rb % 170 : app/controllers/permissions_controller.rb % 207 : app/helpers/transactions_helper.rb % 159 : app/helpers/reports_helper.rb

135 What s left? 135

136 Hook into simplecov s report generator 136

137 emacs & vim visualization / coloring 137

138 Better error handling: type 2 errors on name mismatches 138

139 class X if else end class TestX Enhancment: Record per-test & map lines covered back assert to each test 139

140 class X if else end class TestX assert Enhancment: then use that to show heavy overlaps 140

141 class X class Y Neat Idea: if else end if else end Do this without filtering to identify class TestX places to isolate/ assert disconnect 141

142 I'd Help! 142

143 Need Tools? Need Help? I m Available! 143

144 Thank You (hire me) 144

Lecture 3. Miscellaneous Ruby and Testing 1 / 40

Lecture 3. Miscellaneous Ruby and Testing 1 / 40 Lecture 3 Miscellaneous Ruby and Testing 1 / 40 Homework 1 Grades were released! TAs provided feedback on best practices, but did not take off points Keep the comments in mind for future assignments! Any