An Exploration of Generalization and Over tting in Genetic Programming: Standard and Geometric Semantic Approaches. Ivo Carlos Pereira Gonçalves

Size: px

Start display at page:

Download "An Exploration of Generalization and Over tting in Genetic Programming: Standard and Geometric Semantic Approaches. Ivo Carlos Pereira Gonçalves"

Erin Pearson
5 years ago
Views:

Associate Professor Carlos Manuel Mira da Fonseca and Principal Investigator Sara Guilherme Oliveira da Silva, and presented

1 Ivo Carlos Pereira Gonçalves An Exploration of Generalization and Over tting in Genetic Programming: Standard and Geometric Semantic Approaches Doctoral thesis submitted to the Doctoral Program in Information Science and Technology, supervised by Associate Professor Carlos Manuel Mira da Fonseca and Principal Investigator Sara Guilherme Oliveira da Silva, and presented to the Department of Informatics Engineering of the Faculty of Sciences and Technology of the University of Coimbra. November 216.

17 p

19 X3/X1 (X2 +.42) X3 2 /X2

21 k = 1.1 k = 1.1

23 1

27 2

30 Training error Generalization error: scenario 1 Generalization error: scenario 2 Generalization optimum Error Iterations

38 * / + X3 X1 X2.42 X3/X1 (X2 +.42)

39 / * X2 X3 X3 X3 2 /X2

40 N

42 Parent 1 Parent 2 Crossover points * / / + * X2 X3 X1 X2.42 X3 X3 Offspring 1 Offspring 2 * / / * + X2 X3 X1 X3 X3 X2.42

43 Parent Random tree Mutation point * * * + * X3 X1 X1 X2.42 X3 X3 Offspring * * * X1 X1 * X3 X3 X3

49 E N

53 ER(n) ER(min) min ER(n) ER(min) min

55 y ymin ymax

63 3

64 α =.5

67 4

69 Training error Generalization error Standard RST RST 5% RST 25% RST 5% RST 75% RST 95% Training error Generalization error Standard RST RST 5% RST 25% RST 5% RST 75% RST 95% Training error Generalization error Standard RST RST 5% RST 25% RST 5% RST 75% RST 95%

70 Overfitting Standard RST RST 5% RST 25% RST 5% RST 75% RST 95% Overfitting Standard RST RST 5% RST 25% RST 5% RST 75% RST 95% Overfitting Standard RST RST 5% RST 25% RST 5% RST 75% RST 95%

71 p p p

72 Generalization error Generalization error Standard RST RST 5% RST 25% RST 5% RST 75% RST 95% Standard RST RST 5% RST 25% RST 5% RST 75% RST 95% x Generalization error Standard RST RST 5% RST 25% RST 5% RST 75% RST 95%

73 Tree size Standard RST RST 5% RST 25% RST 5% RST 75% RST 95% Tree size 2 15 Standard RST RST 5% RST 25% RST 5% RST 75% RST 95% Tree size 2 15 Standard RST RST 5% RST 25% RST 5% RST 75% RST 95%

74 Tree depth Standard RST RST 5% RST 25% RST 5% RST 75% RST 95% Tree depth Standard RST RST 5% RST 25% RST 5% RST 75% RST 95% Tree depth Standard RST RST 5% RST 25% RST 5% RST 75% RST 95%

76 Training error Generalization error Standard RST IA 5% IA 1% IA 15% IA 2% IA 25% Training error Generalization error Standard RST IA 5% IA 1% IA 15% IA 2% IA 25% Training error Generalization error Standard RST IA 5% IA 1% IA 15% IA 2% IA 25%

77 Tree size Standard RST IA 5% IA 1% IA 15% IA 2% IA 25% Tree size 2 15 Standard RST IA 5% IA 1% IA 15% IA 2% IA 25% Tree size Standard RST IA 5% IA 1% IA 15% IA 2% IA 25%

78 Tree depth Standard RST IA 5% IA 1% IA 15% IA 2% IA 25% Tree depth Standard RST IA 5% IA 1% IA 15% IA 2% IA 25% Tree depth Standard RST IA 5% IA 1% IA 15% IA 2% IA 25%

80 Training error Generalization error Standard RST IS 5% IS 1% IS 15% IS 2% IS 25% Training error Generalization error Standard RST IS 5% IS 1% IS 15% IS 2% IS 25% Training error Generalization error Standard RST IS 5% IS 1% IS 15% IS 2% IS 25%

81 Tree size Standard RST IS 5% IS 1% IS 15% IS 2% IS 25% Tree size 2 15 Standard RST IS 5% IS 1% IS 15% IS 2% IS 25% Tree size 2 15 Standard RST IS 5% IS 1% IS 15% IS 2% IS 25%

82 Tree depth Standard RST IS 5% IS 1% IS 15% IS 2% IS 25% Tree depth Standard RST IS 5% IS 1% IS 15% IS 2% IS 25% Tree depth Standard RST IS 5% IS 1% IS 15% IS 2% IS 25%

83 p p p p p p p p

85 Training error Generalization error Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5% Training error Generalization error Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5% Training error Generalization error Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5%

86 Overfitting Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5% Overfitting Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5% Overfitting Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5%

87 Generalization error Generalization error Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5% Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5% x Generalization error Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5%

88 Tree size Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5% Tree size Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5% Tree size Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5%

89 Tree depth Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5% Tree depth Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5% Tree depth Standard RST RI 99% RI 95% RI 75% RI 5% RI 25% RI 5%

90 Generalization error 6 Generalization error Standard RST RI 75% NRI 75% RI 5% NRI 5% Standard RST RI 75% NRI 75% RI 5% NRI 5% x Generalization error Standard RST RI 75% NRI 75% RI 5% NRI 5%

91 p p p

93 5

94 T 1, T 2 : R n R T XO = (T 1 T R ) + ((1 T R ) T 2 ) T R [, 1] T R [, 1] T : R n R ms T M = T + ms (T R1 T R2 ) T R1 T R2

95 T R1 T R2 f(x) = e x [, 1] [ 1, 1] ms [ ms, ms]

96 [ ms, ms]

98 Training error Generalization error GSGP UM MS.1 GSGP UM MS.1 GSGP UM MS 1 GSGP UM MS 1 GSGP UM MS Training error Generalization error GSGP BM MS.1 GSGP BM MS.1 GSGP BM MS 1 GSGP BM MS 1 GSGP BM MS T R1 T R2

99 Training error Generalization error GSGP UM MS.1 GSGP UM MS.1 GSGP UM MS 1 GSGP UM MS 1 GSGP UM MS Training error Generalization error GSGP BM MS.1 GSGP BM MS.1 GSGP BM MS 1 GSGP BM MS 1 GSGP BM MS

100 Training error Generalization error GSGP UM MS.1 GSGP UM MS.1 GSGP UM MS 1 GSGP UM MS 1 GSGP UM MS Training error Generalization error GSGP BM MS.1 GSGP BM MS.1 GSGP BM MS 1 GSGP BM MS 1 GSGP BM MS

101 , x < f(x) = 1, x > 1 x, f(x) = sin(x) [, 1]

102 Training error Generalization error GSGP BM Logistic GSGP BM Linear GSGP BM Sine Training error Generalization error GSGP BM Logistic GSGP BM Linear GSGP BM Sine Training error Generalization error GSGP BM Logistic GSGP BM Linear GSGP BM Sine

103 p p p p p p

104 p p p p

105

106

107 Training error Generalization error Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM Training error Generalization error Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM Training error Generalization error Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM

108 Overfitting Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM Overfitting Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM Overfitting Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM

109 Generalization error Generalization error Standard Standard DL 17 GSGP BM GSGP BM NC SSHC BM GSGP UM GSGP UM NC SSHC UM Generalization error 15 1 Generalization error Standard Standard DL 17 GSGP BM GSGP BM NC SSHC BM GSGP UM GSGP UM NC SSHC UM 11 x Generalization error Generalization error Standard Standard DL 17 GSGP BM GSGP BM NC SSHC BM GSGP UM GSGP UM NC SSHC UM

110 % of crossover improvements over the best individual % of crossover improvements over the best parent Standard Standard DL 17 GSGP UM GSGP BM % of crossover improvements over the best individual % of crossover improvements over the best parent Standard Standard DL 17 GSGP UM GSGP BM % of crossover improvements over the best individual % of crossover improvements over the best parent Standard Standard DL 17 GSGP UM GSGP BM

111 % of mutation improvements over the best individual % of mutation improvements over the parent Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM % of mutation improvements over the best individual % of mutation improvements over the parent Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM % of mutation improvements over the best individual % of mutation improvements over the parent Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM

112 Tree size Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM Tree size Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM Tree size Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM

113 Tree depth Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM Tree depth Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM Tree depth Standard Standard DL 17 GSGP UM GSGP BM GSGP UM NC GSGP BM NC SSHC UM SSHC BM

114 ms [ ms, ms]

115

116 I 2M I 2M = P + R1 ms + R2 ms, ms

117 P RI RI ms P + RI ms = t, P RI t RI ms = (t P), A x = y, RI P t pw

118 P pw + RI ms = t

119 Training error Generalization error SSHC UM SSHC BM SSHC AUM SSHC ABM SSHC DAUM SSHC DABM Training error Generalization error SSHC UM SSHC BM SSHC AUM SSHC ABM SSHC DAUM SSHC DABM Training error Generalization error SSHC UM SSHC BM SSHC AUM SSHC ABM SSHC DAUM SSHC DABM

120 Training error Generalization error SSHC UM SSHC BM SSHC AUM SSHC ABM SSHC DAUM SSHC DABM Training error Generalization error SSHC UM SSHC BM SSHC AUM SSHC ABM SSHC DAUM SSHC DABM Training error Generalization error SSHC UM SSHC BM SSHC AUM SSHC ABM SSHC DAUM SSHC DABM

121 11 11 % of mutation improvements over the best individual SSHC AUM SSHC ABM SSHC DAUM SSHC DABM % of mutation improvements over the best individual SSHC AUM SSHC ABM SSHC DAUM SSHC DABM % of mutation improvements over the best individual SSHC AUM SSHC ABM SSHC DAUM SSHC DABM

122 X = { x 1, x 2,..., x n } t = [t 1, t 2,..., t n ] t i x i x i I( x i ) x i si = [I( x 1 ), I( x 2 ),..., I( x n )] n n X ei = s I t. n

123 A B k e A = k e B k A B ea e B = cos θ = e A e B. C A B k A B sa t = k ( s B t ). t = 1 1 k s A k 1 k s B,

124 A B C I opt = 1 1 k A k 1 k B k k e A 1 / e B 1, e A 2 / e B 2,..., e A n / e B n k A k B

125 A eb = ea k, sb = e B + t. A k A k B k C s B k A k A s B T s B

126 s B T N N T s B T N A T B A A A s B A T A T B s B

127 k A T k k k k k = 1.1 k k k k k k

128

129

130

131 Absolute cosine similarity Absolute cosine similarity Absolute cosine similarity k = 1.1

132

133

134

135 k p k = 1.1

136 Training error Generalization error ACA SSHC SSHC Training error Generalization error ACA SSHC SSHC Training error Generalization error ACA SSHC SSHC k = 1.1

137 k =.9 k =.1

138

139 6 x

140 w w i x i y y = f( x w + bias), f Inputs Weights x 1 w 1 Bias b Activation function x 2 w 2 Σ f Output y x 3 w 3

141 f(x) = x T R1 T R2

142 Input layer Hidden layer Output layer Input 1 Input 2 Input 3 Output Input 4 Input 5

143

144 [ ls, ls] ls [ 1, 1] [ ms, ms] ms Parent Neural Network Random Neural Network Resulting Neural Network

145

146 N N B B N N B B N B

147

148

149 Training error Generalization error LS 1 LS 1 LS Iterations / generations Iterations / generations Training error Generalization error LS 1 LS 1 LS Iterations / generations Iterations / generations Training error Generalization error LS 1 LS 1 LS Iterations / generations Iterations / generations

150 Training error Generalization error SS 1 SS 5 SS 1 SS Iterations / generations Iterations / generations Training error Generalization error SS 1 SS 5 SS 1 SS Iterations / generations Iterations / generations 3 3 Training error 25 2 Generalization error 25 2 SS 1 SS 5 SS 1 SS Iterations / generations Iterations / generations

151

152 p p p p p p p [ ls, ls] ls p

153 Training error Generalization error SP % SP 5% SP 25% SP 5% SP 75% SP 95% Iterations / generations Iterations / generations Training error Generalization error SP % SP 5% SP 25% SP 5% SP 75% SP 95% Iterations / generations Iterations / generations 3 3 Training error 25 2 Generalization error 25 2 SP % SP 5% SP 25% SP 5% SP 75% SP 95% Iterations / generations Iterations / generations

154 Generalization error Generalization error SP % SP 5% SP 25% SP 5% SP 75% SP 95% SP % SP 5% SP 25% SP 5% SP 75% SP 95% Generalization error SP % SP 5% SP 25% SP 5% SP 75% SP 95%

155

156 Training error Generalization error SLM FLS SLM FLS SP 75% SLM FLS SP 95% SSHC BM Iterations / generations Iterations / generations Training error Generalization error SLM FLS SLM FLS SP 75% SLM FLS SP 95% SSHC BM Iterations / generations Iterations / generations 3 3 Training error 25 2 Generalization error 25 2 SLM FLS SLM FLS SP 75% SLM FLS SP 95% SSHC BM Iterations / generations Iterations / generations

157 Generalization error 32 3 Generalization error SLM FLS SLM FLS SP 75% SLM FLS SP 95% SSHC BM SLM FLS SLM FLS SP 75% SLM FLS SP 95% SSHC BM Generalization error SLM FLS SLM FLS SP 75% SLM FLS SP 95% SSHC BM

158 p p p p p p p p p p p

159 Training error Generalization error SLM OLS SLM OLS SP 5% SLM OLS SP 25% SLM OLS SP 5% SLM OLS SP 75% SLM OLS SP 95% SSHC ABM Iterations / generations Iterations / generations Training error Generalization error SLM OLS SLM OLS SP 5% SLM OLS SP 25% SLM OLS SP 5% SLM OLS SP 75% SLM OLS SP 95% SSHC ABM Iterations / generations Iterations / generations Training error Generalization error SLM OLS SLM OLS SP 5% SLM OLS SP 25% SLM OLS SP 5% SLM OLS SP 75% SLM OLS SP 95% SSHC ABM Iterations / generations Iterations / generations

160 Generalization error Generalization error SLM OLS SLM OLS SP 5% SLM OLS SP 25%SLM OLS SP 5%SLM OLS SP 75%SLM OLS SP 95% SLM OLS SLM OLS SP 5% SLM OLS SP 25%SLM OLS SP 5%SLM OLS SP 75%SLM OLS SP 95% Generalization error SLM OLS SLM OLS SP 5% SLM OLS SP 25%SLM OLS SP 5%SLM OLS SP 75%SLM OLS SP 95%

161 p p p p p p p p

162 p p p p p p p p

163 Training error Generalization error SLM FLS NWD SP % SLM FLS NWD SP 5% SLM FLS NWD SP 25% SLM FLS NWD SP 5% SLM FLS NWD SP 75% SLM FLS NWD SP 95% Iterations / generations Iterations / generations Training error Generalization error SLM FLS NWD SP % SLM FLS NWD SP 5% SLM FLS NWD SP 25% SLM FLS NWD SP 5% SLM FLS NWD SP 75% SLM FLS NWD SP 95% Iterations / generations Iterations / generations 3 3 Training error 25 2 Generalization error 25 2 SLM FLS NWD SP % SLM FLS NWD SP 5% SLM FLS NWD SP 25% SLM FLS NWD SP 5% SLM FLS NWD SP 75% SLM FLS NWD SP 95% Iterations / generations Iterations / generations

164 Training error Generalization error SLM OLS NWD SP % SLM OLS NWD SP 5% SLM OLS NWD SP 25% SLM OLS NWD SP 5% SLM OLS NWD SP 75% SLM OLS NWD SP 95% Iterations / generations Iterations / generations Training error Generalization error SLM OLS NWD SP % SLM OLS NWD SP 5% SLM OLS NWD SP 25% SLM OLS NWD SP 5% SLM OLS NWD SP 75% SLM OLS NWD SP 95% Iterations / generations Iterations / generations Training error Generalization error SLM OLS NWD SP % SLM OLS NWD SP 5% SLM OLS NWD SP 25% SLM OLS NWD SP 5% SLM OLS NWD SP 75% SLM OLS NWD SP 95% Iterations / generations Iterations / generations

165

166 1 1 9 % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations

167

168

169 Generalization error Number of iterations / generations % 15% 25% 35% 45% 5% 5% 15% 25% 35% 45% 5% Generalization error Number of iterations / generations % 15% 25% 35% 45% 5% 5% 15% 25% 35% 45% 5% 1 Generalization error Number of iterations / generations % 15% 25% 35% 45% 5% 5% 15% 25% 35% 45% 5%

170 p p

171 Generalization error Number of iterations / generations % 15% 25% 35% 45% 5% 5% 15% 25% 35% 45% 5% Generalization error Number of iterations / generations % 15% 25% 35% 45% 5% 5% 15% 25% 35% 45% 5% 3 1 Generalization error Number of iterations / generations % 15% 25% 35% 45% 5% 5% 15% 25% 35% 45% 5%

172 33 76 Generalization error Number of iterations / generations EDV TIE EDV TIE Generalization error Number of iterations / generations EDV TIE EDV TIE 23 9 Generalization error Number of iterations / generations EDV TIE EDV TIE

173 1 1 9 % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations

174 p

175 Generalization error Number of iterations / generations % 15% 25% 35% 45% 5% 5% 15% 25% 35% 45% 5% Generalization error 4 35 Number of iterations / generations % 15% 25% 35% 45% 5% 5% 15% 25% 35% 45% 5% Generalization error Number of iterations / generations % 15% 25% 35% 45% 5% 5% 15% 25% 35% 45% 5%

176 p p

177 Generalization error Number of iterations / generations SLM FLS EDV SLM FLS TIE SLM OLS EDV SLM FLS EDV SLM FLS TIE SLM OLS EDV Generalization error 4 35 Number of iterations / generations SLM FLS EDV SLM FLS TIE SLM OLS EDV SLM FLS EDV SLM FLS TIE SLM OLS EDV Generalization error Number of iterations / generations SLM FLS EDV SLM FLS TIE SLM OLS EDV SLM FLS EDV SLM FLS TIE SLM OLS EDV

178 1 1 9 % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations

179 1 1 9 % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations % of models per Error Deviation (ED) variation ED decrease ED increase % of models per error variation against current best Error decrease Error increase Iterations / generations Iterations / generations

180 p p p p p p p p p p p p p p

181 p p C

182 Generalization error 35 Number of iterations / generations SSHC BM EDV SSHC BM TIE SSHC ABM EDV SSHC BM EDV SSHC BM TIE SSHC ABM EDV Generalization error Number of iterations / generations SSHC BM EDV SSHC BM TIE SSHC ABM EDV SSHC BM EDV SSHC BM TIE SSHC ABM EDV Generalization error Number of iterations / generations SSHC BM EDV SSHC BM TIE SSHC ABM EDV SSHC BM EDV SSHC BM TIE SSHC ABM EDV

183 p p

184 Generalization error 5 4 Generalization error FLS EDV FLS TIE OLS EDV SVR MLP RT LR FLS EDV FLS TIE OLS EDV SVR MLP RT LR Generalization error FLS EDV FLS TIE OLS EDV SVR MLP RT LR

185

186 Computational time (in seconds) Computational time (in seconds) FLS EDV FLS TIE OLS EDV FLS EDV FLS TIE OLS EDV 25 2 Computational time (in seconds) FLS EDV FLS TIE OLS EDV

187 7

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

arxiv: v1 [cs.ne] 19 Jun 2017

arxiv: v1 [cs.ne] 19 Jun 2017 Unsure When to Stop? Ask Your Semantic Neighbors arxiv:16.6195v1 [cs.ne] 19 Jun 17 ABSTRACT Ivo Gonçalves NOVA IMS, Universidade Nova de Lisboa -312 Lisbon, Portugal igoncalves@novaims.unl.pt Carlos M.