NEURAL NETWORKS Typeset by FoilTEX 1
Basic Concepts The McCulloch-Pitts model Hebb s rule Neural network: double dynamics. Pattern Formation and Pattern Recognition Neural network as an input-output device What is the Perceptron and how does it learn? What a Perceptron can and cannot do? Typeset by FoilTEX 2
The McCulloch-Pitts model a a set of weighted input an adder an activation function (here a threshold-dependent step function) can you imagine anything else? Limitations of the MCP neurons nonlinear summation spike train asynchronous update excitatory and inhibitory synapses feedback loops Typeset by FoilTEX 3
Hebb s rule d dt w ij(t) = F (a i, a j ) (1) where F is a functional, and a j and a i are presynaptic and postsynaptic activity functions (i.e., they may include activity levels over some period of time and not just the current activity values. To define specific learning rules, i.e., the form of F, a few points should be clarified. The simplest Hebbian learning rule can be formalized as: d dt w ij(t) = k a i (t) a j (t), k > 0 (2) This rule expresses the conjunction among pre- and postsynaptic elements (using neurobiological terminology) or associative conditioning (in psychological terms), by a simple product of the actual states of pre- and post- synaptic elements, a j (t) and a i (t). Typeset by FoilTEX 4
Neural network: double dynamics. Pattern Formation and Pattern Recognition Figure 1: Activation function Typeset by FoilTEX 5
W / Neural network: double dynamics. Pattern Formation and Pattern Recognition Networks and dynamics parameters ]^`_\acbed'^f_\acb XZY\[ QSRUT V output input DFEGIHJ K LMEAGNOE GIHJPJ e.g. "node dynamics" 0"13245 6 78290:132$45+;=<1>-2$4*5+;?@24A;CB 5 5 "edge dynamics"! "#$&%' ()$*+%,-$. Figure 2: Double dynamics Typeset by FoilTEX 6
Neural network: double dynamics. Pattern Formation and Pattern Recognition SELF ORGANIZATION: Development and Plastcity of Ordered Structures s ij j i retina How to generate topographic order? How topreserve replace it after partial lesions? topographic order optic tectum 0 0 ONTOGENY AND(!) PLASTICITY LGN m L i s R i LL RR LL RR LL RR LL RR cortex (layer IV) ocular dominance columns initial configuration: homogeneous, very small Figure 3: Spatial pattern formation Typeset by FoilTEX 7
Neural network as an input-output device Input vector: x Weights: W Output: y ; y(x, W) Targets: t Activation function: g( ) Error: E deviation between y and t Typeset by FoilTEX 8
STRUCTURE What is the Perceptron and how does it learn? Possible mistakes a neuron can do Error t k y k Perceptron learning rule (numbers of inputs and of neurons are not necessary the same) w ik w ik + η(t k y k )x i if x i = 0 only threshold is adjustable bias node ; x 0 = 1 w 0j : adjustable Typeset by FoilTEX 9
What is the Perceptron and how does it learn? The Perceptron learning algorithm Initialize the weights and threshold to small (positive and negative) random numbers Training For each input vector calculate the output Update the weights according to the Perceptron learning rule: w ik w ik + η(t k y k )x i Recall Compute the activation of each neuron similarly Typeset by FoilTEX 10
What a Perceptron can and cannot do? logical OR operation NN implementation IMPLEMENTATION: choose from the scripts of Chapter 3: http://seat.massey.ac.nz/personal/s.r.marsland/mlbook.html Typeset by FoilTEX 11
Linear Separability What a Perceptron can and cannot do? Perceptron generally tries to find a decision boundary or discriminant function specifically straight line in 2D, plane in 3D and hyperplane in higher dimensions more specifically: separating two groups of neurons - (i) should fire, (ii) should not fire x 1 w T = 0: boundary vector Typeset by FoilTEX 12
Linear Separability What a Perceptron can and cannot do? (Data, target) Perceptron tries to find a straight line if a straight line exists: linearly separable cases more than one output neuron several straight lines Typeset by FoilTEX 13
Exclusive Or (XOR) What a Perceptron can and cannot do? XOR is not linearly separable Perceptron algorithm does not converge Minsky and Papert -> no funding for NN research for 15 years Typeset by FoilTEX 14
Changing dimension! What a Perceptron can and cannot do? does not change the data when is looked at (x, y) just moves 0, 0) along a third dimension : General Message: linear separability is always possible after projecting the data into the correct set of dimensions kernel classifiers support vector machines Typeset by FoilTEX 15