Hardware Neuronale Netzwerke - Lernen durch künstliche Evolution (?)

SKIP - May 2004 Hardware Neuronale Netzwerke - Lernen durch künstliche Evolution (?) S. G. Hohmann, Electronic Vision(s), Kirchhoff Institut für Physik, Universität Heidelberg Hardware Neuronale Netzwerke Evolutionäre Algorithmen und ihre Grenzen Inkrementelle Trainingsstrategien Experimente und Resultate

Inspiration : Biological Neural Networks e.g. the human brain parallel distributed fault tolerant adaptive

Artificial Neural Networks (ANNs) a 1 neuron N net i = ω a j= 1 ji j a 2 ω 2i ω 1i a i a = F i ( neti ) ω 3i a 3 synaptic connection

Artificial Neural Networks (ANNs) input neurons hidden neurons output neurons necessary : find synaptic weights for desired task training : optimization of weights common : supervised learning goal : generalization

Artificial Neural Networks (ANNs) shortcut connections 2 hidden layers feedback connections no hidden layer

Hardware Realization: HAGEN Chip Photo

Hardware Realization: HAGEN Chip Photo

Hardware Realization: Network Model HAGEN chip : contains 4 network blocks network block : implements a fully connected one layer network 128 64 binary inputs binary outputs 8192 analog synapses fully connected threshold activation function : O = Θ( I ), O, I {0,1} i ω ij j (McCulloch-Pitts) j network block weight values configured with 10 bit (+ sign bit) Digital to Analog Converters on chip

Hardware Realization: Network Operation clocked network operation (50 MHz) feedback connections recurrent network or multi-layer architecture inter block connections networks distributed among blocks binary interface interconnect multiple chips for even larger networks

Hardware Realization: Training Setup neural network chip HAGEN custom-made PCI board RAM programmable logic (FPGA) digital controller low-level parts of training algorithm communication via PCI bus host computer user interaction training algorithm (C++)

Training the HAGEN chip consider binary input/output neurons device variations common gradient-based training chip-in-the-loop algorithms well proven evolutionary algorithms: require no knowledge of neuron characteristics can exploit the speed of the hardware

Evolutionary Algorithms (EAs) concept process multiple networks : population of individuals simulate natural selection (based on fitness) produce offspring by recombination (crossover) + introduce variations in form of mutation C. Darwin common : use genetic coding for individuals

Evolutionary Algorithms (EAs) critical aspects fitness function performance measure problem-dependent optimizing fitness optimizing networks? selection scheme simulate natural selection balance selection and diversity genetic coding & operators suited for problem structure promote recombination of advantageous features

Genetic Coding for HAGEN Networks keeping the architecture fixed fix network cycles enable/disable feedback connections set unused synapses to 0 optimize weights of used synapses of needed input and output neurons of involved network blocks

Genetic Coding for HAGEN Networks number [-1023,1023] codes one weight gene linear string of genes codes used synapses of one output neuron chromosome set of chromosomes codes used synapses of each used neuron genome

Genetic Operators used operators two-point crossover two random cut points intermediary genes swapped between individuals new cut points for each chromosome pair + = random uniform mutation affects single genes fixed probability replaces old gene with new random value

Evolutionary Coprocessor hardware acceleration of evolution implemented in configurable logic (FPGA) hosts population (dedicated RAM) performs all genetic manipulations fast processing of genetic material extensive instruction set * programmed from outside (software) flexibility of genetic operators fitness & selection in software * under development

Selection and Reproduction Scheme example: tournament selection processing a population of Φ individuals evaluate each network investigate its fitness fill new population : tournament selection with tournament size τ old population random best new population τ = 3 form mating pairs (random) & apply crossover with probability χ mutate each gene with probability µ

Example : Classification Problems common classification problem : allocate each of the N p instances to one of N c classes by evaluating N i input parameters per instance examples * : predict cellular localization sites of E.coli and yeast proteins classify iris plants input values normalized [0,1]: code as binary n-bit integer values N i E.coli Yeast 7 8 N p 336 1484 10 Iris 4 150 N c 8 3 * data sets obtained from the UCI KDD online archive

Simple Evolutionary Training Approach one output neuron for each class when applying input activate correct neuron & deactivate all others class 1 class 3 class 2 intuitive fitness function : score 1 point for each correct active neuron correct inactive neuron after each applied input good fitness function?

Simple Evolutionary Training Approach experiment (e.g. E.coli ): evolved networks always stay totally inactive! correct output contains more 0s than 1s! correct activation score >> correct inactivation score

Simple Evolutionary Training Approach experiment (e.g. E.coli ): evolved networks always stay totally inactive! correct output contains more 0s than 1s! correct activation score >> correct inactivation score new experiment (again E.coli ): networks always predict most common class! classes not equally distributed (most common : 41%)! need adequate weighting of classes e.g. all classes equally?

Simple Evolutionary Training Approach experiment (e.g. E.coli ): evolved networks always stay totally inactive! correct output contains more 0s than 1s! correct activation score >> correct inactivation score new experiment (again E.coli ): networks always predict most common class! classes not equally distributed (most common : 41%)! need adequate weighting of classes e.g. all classes equally? etc

Discussing The Evolutionary Approach contradiction? real-life applications complex networks = complex search space simple genetic operators, simple fitness measure, simple selection scheme: suited for hardware implementation (EvoCop) fast processing exploit speed of hardware successful training?

Discussing The Evolutionary Approach contradiction? real-life applications complex networks = complex search space simple genetic operators, simple fitness measure, simple selection scheme: suited for hardware implementation (EvoCop) fast processing exploit speed of hardware successful training?

Discussing The Evolutionary Approach contradiction? real-life applications complex networks = complex search space simple genetic operators, simple fitness measure, simple selection scheme: suited for hardware implementation (EvoCop) fast processing exploit speed of hardware successful training? one approach : divide-and-conquer

Incremental Training : Stage I one problem with N classes = N problems with one class + rest train N nets : one for each problem task : activate neurons if input belongs to respective class, remain silent otherwise train nets independently (parallel or sequentially)

Incremental Training : Stage I one problem with N classes = N problems with one class + rest train N nets : one for each problem task : activate neurons if input belongs to respective class, remain silent otherwise train nets independently (parallel or sequentially)

Incremental Training : Stage I one problem with N classes = N problems with one class + rest train N nets : one for each problem task : activate neurons if input belongs to respective class, remain silent otherwise train nets independently (parallel or sequentially)

Incremental Training : Stage I one problem with N classes = N problems with one class + rest train N nets : one for each problem task : activate neurons if input belongs to respective class, remain silent otherwise train nets independently (parallel or sequentially)

Incremental Training : Stage I Cont. all nets can use the same inputs parts of one big net response of the network : class with the highest number of activated output neurons

Incremental Training : Stage II add and train connections between the subnetworks again train each subnetwork one by one the task remains the same

Incremental Training : Stage II add and train connections between the subnetworks again train each subnetwork one by one the task remains the same

Incremental Training : Stage II add and train connections between the subnetworks again train each subnetwork one by one the task remains the same

Incremental Training : Stage II add and train connections between the subnetworks again train each subnetwork one by one the task remains the same

Incremental Training : Stage II add and train connections between the subnetworks again train each subnetwork one by one the task remains the same

Advantages of Incremental Training in each training phase smaller networks = smaller search space simple two-class problem can use simple fitness function can use simple genetic operators suited for hardware implementation (EvoCop) better performance of evolutionary algorithm

Experiments : Cross-Validation setup 10-fold cross-validation test : randomly divide data set into 10 subsets nine for training, one for test of generalization repeat 10 x with different validation subset for each subset : take best net of N (e.g. 10) average over the 10 configurations : classification accuracy A in % generalization accuracy G in % repeat experiment multiple (5) times (for estimation of variance)

Results : E.coli, Yeast and Iris train nets on HAGEN using the EvoCop 6 hidden neurons and 4 outputs per subnetwork population size 10, 2000 generations per phase tournament size 2, mutation 3%, crossover 90% E.coli Yeast Iris classification accuracy A [%] 90.5 +/- 0.4 56.8 +/- 0.3 99.5 +/- 0.1 generalization accuracy G [%] 81.9 +/- 0.9 51.8 +/- 1.3 95.6 +/- 1.3 training all at once [%] 64.7** 34.2** 93.8 +/- 2.5 typical G in literature [%] 86* 60* 96 results much better than if trained all at once results comparable to other systems * Horton, Nakai, Better prediction of protein cellular localization sites with the k nearest neighbors classifier, 1997 ** Results obtained by pure software algorithm without EvoCop. Appears in : Hohmann, Schemmel, Schuermann, Meier Predicting Protein Cellular Localization Sites with a Hardware Analog Neural Network, 2003

Iris Problem : A Trained Network upper left corner of upper left block inputs intermediate outputs 2 network cycles layer 2 layer 1

Iris Problem : A Trained Network upper left corner of upper left block inputs intermediate outputs 2 network cycles phase 1 : subnetwork 1

Iris Problem : A Trained Network upper left corner of upper left block inputs intermediate outputs 2 network cycles phase 2 : subnetwork 2

Iris Problem : A Trained Network upper left corner of upper left block inputs intermediate outputs 2 network cycles phase 3 : subnetwork 3

Iris Problem : A Trained Network upper left corner of upper left block inputs intermediate outputs 2 network cycles phase 4 : subnetwork 1

Iris Problem : A Trained Network upper left corner of upper left block inputs intermediate outputs 2 network cycles phase 5 : subnetwork 2

Iris Problem : A Trained Network upper left corner of upper left block inputs intermediate outputs 2 network cycles phase 6 : subnetwork 3

Iris Problem : A Trained Network upper left corner of upper left block inputs intermediate outputs 2 network cycles network complete

Finish : The Iris Iris Virginica The End Thanks for your attention