The Impact of Equivalent, Redundant and Quasi Mutants on Database Schema Mutation Analysis. Chris J. Wright Gregory M. Kapfhammer Phil McMinn

Transcription

1 The Impact of Equivalent, Redundant and Quasi Mutants on Database Schema Mutation Analysis Chris J. Wright Gregory M. Kapfhammer Phil McMinn

2 The Impact of Equivalent, Redundant and Quasi Mutants on Database Schema Mutation Analysis Chris J. Wright Gregory M. Kapfhammer Phil McMinn

3 The Impact of Equivalent, Redundant and Quasi Mutants on Database Schema Mutation Analysis Chris J. Wright Gregory M. Kapfhammer Phil McMinn

4 The Impact of Equivalent, Redundant and Quasi Mutants on Database Schema Mutation Analysis Chris J. Wright Gregory M. Kapfhammer Phil McMinn

5 The Impact of Equivalent, Redundant and Quasi Mutants on Database Schema Mutation Analysis Chris J. Wright Gregory M. Kapfhammer Phil McMinn

6 The Impact of Equivalent, Redundant and Quasi Mutants on Database Schema Mutation Analysis Chris J. Wright Gregory M. Kapfhammer Phil McMinn

7

8 Database Schema

9 Database Schema 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 );

10 Database Schema Tables 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 );

11 Database Schema Columns 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 );

12 Database Schema Constraints 1 CREATE TABLE T ( 2 A CHAR, B CHAR, C CHAR, 3 CONSTRAINT UniqueOnColsAandB UNIQUE (A, B) 4 ); 5 6 CREATE TABLE S ( 7 X CHAR, Y CHAR, Z CHAR, 8 CONSTRAINT RefToColsAandB FOREIGN KEY (X, Y) 9 REFERENCES T (A, B) 10 );

13 Why Test a Database Schema?

14 Why Test a Database Schema? Database Schema

15 Why Test a Database Schema? Database Schema DBMS

16 Why Test a Database Schema? Database Schema DBMS

17 Why Test a Database Schema? Third Party Application Web Server Database Schema DBMS

18 Why Test a Database Schema? Third Party Application Web Server Database Schema DBMS

19 How to Test a Database Schema

20 How to Test a Database Schema Generate test data SQL INSERT statements

21 How to Test a Database Schema Generate test data SQL INSERT statements INSERT INTO T(a, b) VALUES('a', 'b');

22 How to Test a Database Schema Generate test data SQL INSERT statements INSERT INTO T(a, b) VALUES('a', 'b');

23 How to Test a Database Schema Generate test data SQL Execute the data against the database INSERT INTO T(a, b) VALUES('a', 'b'); DBMS

24 How to Test a Database Schema Generate test data SQL Execute the data against the database Examine the acceptance of statements / INSERT INTO T(a, b) VALUES('a', 'b'); DBMS

25 Mutation Analysis

26 Application Mutation Analysis

27 Mutation Analysis Application Mutation Operators

28 Mutation Analysis Application Mutation Operators Mutants

29 Mutation Analysis Application Mutation Operators Mutants Test suite

30 Mutation Analysis Application Mutation Operators Mutants Test suite Test results

31 Mutation Analysis Application Mutation Operators Mutants Test suite Test suite Test results

32 Mutation Analysis Application Mutation Operators Mutants Test suite Test suite Test results Test results

33 Mutation Analysis Application Mutation Operators Mutants Test suite Test suite Test results Comparison Test results

34 Mutation Analysis Application Mutation Operators Mutants Test suite Test suite Test results Comparison Test results Mutation Score

35 Mutation Analysis Mutation Operators Application Mutants Test suite Test suite Test results Comparison Test results Mutation Score

36 Database Schema Mutation Operators

37 Database Schema Mutation Operators Primary Key Foreign Key Unique Not Null Check

38 Database Schema Mutation Operators Primary Key Foreign Key Unique Not Null Check

39 Database Schema Mutation Operators Primary Key Column Addition Foreign Key Unique Not Null Check

40 Database Schema Mutation Operators Primary Key Foreign Key Unique Column Addition Column Removal Not Null Check

41 Database Schema Mutation Operators Primary Key Foreign Key Unique Not Null Column Addition Column Removal Column Exchange Check

42 Database Schema Mutation Operators

43 Database Schema Mutation Operators Primary Key Column Addition

44 Database Schema Mutation Operators Primary Key Column Addition 1 CREATE TABLE T ( 2 A CHAR, B CHAR, 3 PRIMARY KEY (A) 4 );

45 Database Schema Mutation Operators Primary Key Column Addition 1 CREATE TABLE T ( 2 A CHAR, B CHAR, 3 PRIMARY KEY (A) 4 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, 3 PRIMARY KEY (A, B) 4 );

46 Database Schema Mutation Operators Primary Key Column Exchange 1 CREATE TABLE T ( 2 A CHAR, B CHAR, 3 PRIMARY KEY (A) 4 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, 3 PRIMARY KEY (B) 4 );

47 Database Schema Mutation Operators Primary Key Column Removal 1 CREATE TABLE T ( 2 A CHAR, B CHAR, 3 PRIMARY KEY (A) 4 ); 1 CREATE TABLE T ( 2 A CHAR, B CHAR, 3 4 );

48 Mutation Analysis Challenges

49 Mutation Analysis Challenges Special classes of mutants

50 Mutation Analysis Challenges Special classes of mutants Equivalent

51 Mutation Analysis Challenges Special classes of mutants Equivalent Redundant

52 Mutation Analysis Challenges Special classes of mutants Equivalent Redundant Quasi-mutants

53 Mutation Analysis Challenges Special classes of mutants Equivalent Redundant Quasi-mutants

54 Equivalent Mutants

55 Equivalent Mutants Functionally identical to non-mutant

56 Equivalent Mutants Functionally identical to non-mutant but syntactically different

57 Equivalent Mutants Functionally identical to non-mutant but syntactically different Cannot be killed

58 Equivalent Mutants Functionally identical to non-mutant but syntactically different Cannot be killed Artificially decrease mutation score

59 Equivalent Mutants Functionally identical to non-mutant but syntactically different Cannot be killed Artificially decrease mutation score

60 Equivalent Mutants

61 Equivalent Mutants Original: 1 CREATE TABLE T ( 2 A CHAR, 3 PRIMARY KEY (A) 4 );

62 Equivalent Mutants Original: 1 CREATE TABLE T ( 2 A CHAR, 3 PRIMARY KEY (A) 4 ); Mutant: 1 CREATE TABLE T ( 2 A CHAR NOT NULL, 3 PRIMARY KEY (A) 4 );

63 Redundant Mutants

64 Redundant Mutants Functionally identical to another mutant

65 Redundant Mutants Functionally identical to but syntactically different

66 Redundant Mutants Functionally identical to but syntactically different May be killed

67 Redundant Mutants Functionally identical to but syntactically different May be killed Artificially alters mutation score

68 Redundant Mutants Functionally identical to but syntactically different May be killed Artificially alters mutation score Reduces efficiency

69 Redundant Mutants Functionally identical to but syntactically different May be killed Artificially alters mutation score Reduces efficiency

70 Types of Equivalence

71 Structural Types of Equivalence

72 Types of Equivalence Structural Functionally irrelevant syntactic differences

73 Types of Equivalence Structural Functionally irrelevant syntactic differences

74 Types of Equivalence Structural Functionally irrelevant syntactic differences Behavioural

75 Types of Equivalence Structural Functionally irrelevant syntactic differences Behavioural Overlap within SQL features

76 Types of Equivalence Structural Functionally irrelevant syntactic differences Behavioural Overlap within SQL features

77 Behavioural Equivalence Patterns

78 Behavioural Equivalence Patterns NOT NULL in CHECK constraints NOT NULL CHECK( IS NOT NULL)

79 Behavioural Equivalence Patterns NOT NULL in CHECK constraints NOT NULL CHECK( IS NOT NULL) 1 CREATE TABLE T ( 2 A CHAR NOT NULL, 3 ); 1 CREATE TABLE T ( 2 A CHAR, 3 CHECK(A IS NOT NULL) 4 );

80 Behavioural Equivalence Patterns

81 Behavioural Equivalence Patterns NOT NULL on PRIMARY KEY columns

82 Behavioural Equivalence Patterns NOT NULL on PRIMARY KEY columns Implicit NOT NULL on PRIMARY KEY

83 Behavioural Equivalence Patterns NOT NULL on PRIMARY KEY columns Implicit NOT NULL on PRIMARY KEY (Only PostgreSQL and HyperSQL)

84 Behavioural Equivalence Patterns NOT NULL on PRIMARY KEY columns

85 Behavioural Equivalence Patterns NOT NULL on PRIMARY KEY columns 1 CREATE TABLE T ( 2 A CHAR, 3 PRIMARY KEY (A) 4 ); 1 CREATE TABLE T ( 2 A CHAR NOT NULL, 3 PRIMARY KEY (A) 4 );

86 Behavioural Equivalence Patterns

87 Behavioural Equivalence Patterns UNIQUE and PRIMARY KEY with shared columns

88 Behavioural Equivalence Patterns UNIQUE and PRIMARY KEY with shared columns 1 CREATE TABLE T ( 2 A CHAR, 3 PRIMARY KEY (A) 4 ); 1 CREATE TABLE T ( 2 A CHAR, 3 PRIMARY KEY (A), 4 UNIQUE (A) 5 );

89 Quasi-mutants

90 Quasi-mutants Operators produce DBMS-agnostic mutants

91 Quasi-mutants Operators produce DBMS-agnostic mutants Some DBMSs have implicit constraints

92 Quasi-mutants Operators produce DBMS-agnostic mutants Some DBMSs have implicit constraints Valid for some DBMSs, invalid for others PostgreSQL HyperSQL SQLite

93 Quasi-mutants Operators produce DBMS-agnostic mutants Some DBMSs have implicit constraints Valid for some DBMSs, invalid for others PostgreSQL HyperSQL SQLite

94 Quasi-mutants PostgreSQL HyperSQL SQLite

95 Quasi-mutants Cannot adversely affect mutation score PostgreSQL HyperSQL SQLite

96 Quasi-mutants Cannot adversely affect mutation score but may preclude some optimisations PostgreSQL HyperSQL SQLite

97 Quasi-mutants Cannot adversely affect mutation score but may preclude some optimisations Remove when DBMS will reject them SQLite PostgreSQL HyperSQL

98 Types of Quasi-mutants

99 Types of Quasi-mutants Representative example

100 Types of Quasi-mutants Representative example DBMS: PostgreSQL, HyperSQL

101 Types of Quasi-mutants Representative example DBMS: PostgreSQL, HyperSQL FK(reference columns) (PK(reference columns) Unique(reference columns))

102 Types of Quasi-mutants Representative example DBMS: PostgreSQL, HyperSQL FK(reference columns) (PK(reference columns) Unique(reference columns))

103 Types of Quasi-mutants Representative example DBMS: PostgreSQL, HyperSQL FK(reference columns) (PK(reference columns) Unique(reference columns))

104 Types of Quasi-mutants Representative example DBMS: PostgreSQL, HyperSQL FK(reference columns) (PK(reference columns) Unique(reference columns))

105 Types of Quasi-mutants Representative example DBMS: PostgreSQL, HyperSQL FK(reference columns) (PK(reference columns) Unique(reference columns))

106 Types of Quasi-mutants Representative example DBMS: PostgreSQL, HyperSQL (PK(reference columns) Unique(reference

107 Detecting Quasi-mutants

108 Detecting Quasi-mutants Submit to DBMS

109 Detecting Quasi-mutants Submit to DBMS 100% accurate

110 Detecting Quasi-mutants Submit to DBMS 100% accurate Convert representation to SQL, submit to database, inspect response

111 Detecting Quasi-mutants Submit to DBMS 100% accurate Convert representation to SQL, submit to database, inspect response Analyse statically

112 Detecting Quasi-mutants Submit to DBMS 100% accurate Convert representation to SQL, submit to database, inspect response Analyse statically Operates directly on representation

113 Detecting Quasi-mutants Submit to DBMS 100% accurate Convert representation to SQL, submit to database, inspect response Analyse statically Operates directly on representation DBMS-specific implementation

114 Empirical Study

115 Empirical Study 1. Quasi-mutant detection DBMS v Static Analysis

116 Empirical Study 1. Quasi-mutant detection DBMS v Static Analysis 2. Equivalent, Redundant and Quasi-mutant removal Efficiency?

117 Empirical Study 1. Quasi-mutant detection DBMS v Static Analysis 2. Equivalent, Redundant and Quasi-mutant removal Efficiency? 3. Equivalent, Redundant and Quasi-mutant removal Effectiveness?

118 Empirical Study

119 16 schemas Empirical Study

120 Empirical Study 16 schemas 2 DBMSs PostgreSQL, HyperSQL

121 Empirical Study 16 schemas 2 DBMSs PostgreSQL, HyperSQL 15 repeat trials

122 Empirical Study 16 schemas 2 DBMSs PostgreSQL, HyperSQL 15 repeat trials

123 Empirical Study Quasi-mutants

124 Empirical Study Quasi-mutants 5 conditions:

125 Empirical Study Quasi-mutants 5 conditions: Postgres (with/without transactions)

126 Empirical Study Quasi-mutants 5 conditions: Postgres (with/without transactions) HyperSQL (with/without transactions)

127 Empirical Study Quasi-mutants 5 conditions: Postgres (with/without transactions) HyperSQL (with/without transactions) Static analysis

128 Empirical Study Quasi-mutants 5 conditions: Postgres (with/without transactions) HyperSQL (with/without transactions) Static analysis

129 Empirical Study Quasi-mutants Scaled Time Taken HyperSQL HyperSQL Trans. Postgres Approach Postgres Trans. Static

130 Empirical Study Mutant Removal

131 Empirical Study Mutant Removal 2 conditions with and without removal

132 Empirical Study Mutant Removal 2 conditions with and without removal 2 metrics

133 Empirical Study Mutant Removal 2 conditions with and without removal 2 metrics Time taken for mutation analysis

134 Empirical Study Mutant Removal 2 conditions with and without removal 2 metrics Time taken for mutation analysis Mutation score

135 Empirical Study Mutant Removal

136 Empirical Study Mutant Removal HyperSQL Time saved Best case: 718ms (23.05%) Worst case: -824ms (-9.71%)

137 Empirical Study Mutant Removal

138 Empirical Study Mutant Removal HyperSQL Time saved

139 Empirical Study Mutant Removal HyperSQL Time saved 9/16 mean time decrease (p < 0.05)

140 Empirical Study Mutant Removal HyperSQL Time saved 9/16 mean time decrease ( 7/16 mean time increase (p < 0.05)

141 Empirical Study Mutant Removal HyperSQL Time saved 9/16 mean time decrease ( 7/16 mean time increase ( Overall, decrease (1.6% mean, 1.4% median)

142 Empirical Study Mutant Removal

143 Empirical Study Mutant Removal PostgreSQL Time saved Best case: 317,208ms (33.71%) Worst case: -3,086ms, (-0.33%)

144 Empirical Study Mutant Removal

145 Empirical Study Mutant Removal PostgreSQL Time saved

146 Empirical Study Mutant Removal PostgreSQL Time saved 14/16 mean time decrease (p < 0.05)

147 Empirical Study Mutant Removal PostgreSQL Time saved 14/16 mean time decrease ( 2/16 mean time increase (p < 0.05)

148 Empirical Study Mutant Removal PostgreSQL Time saved 14/16 mean time decrease ( 2/16 mean time increase ( Overall, decrease (12.7% mean, 11.8% median)

149 Empirical Study Mutant Removal DBMS Median Time saved (ms) Mean

150 Empirical Study Mutant Removal DBMS Median Time saved (ms) Mean HyperSQL

151 Empirical Study Mutant Removal DBMS Median Time saved (ms) Mean HyperSQL Postgres 8,071 50,880

152 Empirical Study Mutant Removal DBMS Median Time saved (ms) Mean HyperSQL Postgres 8,071 50,880 Both ,450

153 Empirical Study Mutant Removal DBMS Median Time saved (%) Mean HyperSQL Postgres Both

154 Empirical Study Mutant Removal

155 Empirical Study Mutant Removal HyperSQL Mutation score 75% Increased 44% Adequate 25% No change

156 Empirical Study Mutant Removal PostgreSQL Mutation score 75% Increased 44% Adequate 25% No change

157 Empirical Study Mutant Removal DBMS Increased (adequate) Scores changed (%) No change HyperSQL 75 (44) 25 Postgres 75 (44) 25 Both 75 (44) 25

158 Conclusion 1. Quasi-mutant detection DBMS v Static Analysis 2. Equivalent, Redundant and Quasi-mutant removal Efficiency? 3. Equivalent, Redundant and Quasi-mutant removal Effectiveness?

159 Conclusion 1. Quasi-mutant detection DBMS v Static Analysis 2. Equivalent, Redundant and Quasi-mutant removal Efficiency? 3. Equivalent, Redundant and Quasi-mutant removal Effectiveness?

160 Conclusion 1. Quasi-mutant detection DBMS v Static Analysis 2. Equivalent, Redundant and Quasi-mutant removal Efficiency? 3. Equivalent, Redundant and Quasi-mutant removal Effectiveness?

161 Conclusion 1. Quasi-mutant detection Improved efficiency 2. Equivalent, Redundant and Quasi-mutant removal Improved efficiency 3. Equivalent, Redundant and Quasi-mutant removal Improved effectiveness

162 Conclusion 1. Quasi-mutant detection Improved efficiency 2. Equivalent, Redundant and Quasi-mutant removal Improved efficiency 3. Equivalent, Redundant and Quasi-mutant removal Improved effectiveness Chris J. Wright