Website: HOPEFIELD NETWORK. Inderjeet Singh Behl, Ankush Saini, Jaideep Verma. ID-

Transcription

1 International Journal Of Scientific Research And Education Volume 1 Issue 7 Pages ISSN (e): Website: HOPEFIELD NETWORK Inderjeet Singh Behl, Ankush Saini, Jaideep Verma ID- isbahl@yahoo.com ID -ankushsaini90@gmail.com ID- vickyjd12@gmail.com Abstract:- Hopfield network is a kind of neural network investigated by John Hopfield in the early 1980s. The Hopfield network has no special input or output neurons (see McCulloch-Pitts), but all are both input and output, and all are connected to all others in both directions (with equal weights in the two directions). Input is applied simultaneously to all neurons which then output to each other and the process continues until a stable state is reached, which represents the network output. ( ) 1. Introduction:- A Hopfield network is a form of recurrent artificial neural network invented by John Hopfield. Hopfield nets serve as content-addressable memory systems with binary threshold nodes. They are guaranteed to converge to a local minimum, but convergence to a false pattern (wrong local minimum) rather than the stored pattern (expected local minimum) can occur. Hopfield networks also provide a model for understanding human memory. A neural network (or more formally artificial neural network) is a mathematical model or computational model inspired by the structure and functional aspects of biological neural networks. It consists of an interconnected group of artificial neurons. The original inspiration for the term Artificial Neural Network came from examination of central nervous systems and their neurons, axons, dendrites and synapses which constitute the processing elements of biological neural networks. The first model of a neuron was presented in 1943 by W. McCulloch and W. Pitts, and in 1958 Rossenblatt conceived the Perception. His work had big repercussion but in 1969 a violent critic by Minsky and Papert was published. The work on neural network was slow down but John Hopfield convinced of the power of neural network came out with his model in 1982 and boost research in this field. Hopfield Network is a particular case of Neural Network. It is based on physics, inspired by spin system. 2. History:- In the beginning of the 1980s Hopfield published two scientific papers, which attracted much interest. This was the starting point of the new era of neural networks, which continues today. Hopfield showed that models of physical systems could be used to solve computational problems. Such systems could be implemented in hardware by combining standard components such as capacitors and resistors. The importance of the different Hopfield networks in practical application is limited due to theoretical limitations of the network structure but, in certain situations, they may form interesting models. Hopfield networks are typically used for classification problems with binary pattern vectors. Inderjeet Singh Behl et al. IJSRE vol 1 issue 6 Nov Page 1

2 The Hopfield network is created by supplying input data vectors, or pattern vectors, corresponding to the different classes. These patterns are called class patterns. In an n-dimensional data space the class patterns should have n binary components {1,-1}; that is, each class pattern corresponds to a corner of a cube in an n- dimensional space. The network is then used to classify distorted patterns into these classes. When a distorted pattern is presented to the network, then it is associated with another pattern. If the network works properly, this associated pattern is one of the class patterns. In some cases (when the different class patterns are correlated), spurious minima can also appear. This means that some patterns are associated with patterns that are not among the pattern vectors. Hopfield networks are sometimes called associative networks since they associate a class pattern to each input pattern. The Neural Networks package supports two types of Hopfield networks, a continuous-time version and a discrete-time version. Both network types have a matrix of weights W defined as where D is the number of class patterns {,..., }, vectors consisting of +/-1 elements, to be stored in the network, and n is the number of components, the dimension, of the class pattern vectors. Discrete-time Hopfield networks have the following dynamics: Eq. (2.26) is applied to one state, x(t), at a time. At each iteration the state to be updated is chosen randomly. This asynchronous update process is necessary for the network to converge, which means that x(t)=sign[w x(t)]. A distorted pattern, x(0), is used as initial state for the Eq. (2.0), and the associated pattern is the state toward which the difference equation converges. That is, starting with x(0) and then iterating Eq. (2.0) gives the associated pattern when the equation converged. For a discrete-time Hopfield network, the energy of a certain vector x is given by It can be shown that, given an initial state vector x(0), x(t) in Eq. (2.26) will converge to a value having minimum energy. Therefore, the minima of Eq. (2.27) constitute possible convergence points of the Hopfield network and, ideally, these minima are identical to the class patterns {,..., }. Hence, one can guarantee that the Hopfield network will converge to some pattern, but one cannot guarantee that it will converge to the right pattern. Note that the energy function can take negative values; this is, however, just a matter of scaling. Adding a sufficiently large constant to the energy expression it can be made positive. The continuous Hopfield network is described by the following differential equation Inderjeet Singh Behl et al. IJSRE vol 1 issue 6 Nov Page 2

3 where x(t) is the state vector of the network, W represents the parametric weights, and is a nonlinearity acting on the states x(t). The weights W are defined in Eq. (2.25). The differential equation, Eq. (2.28), is solved using an Euler simulation. To define a continuous-time Hopfield network, you have to choose the nonlinear function. There are two choices supported by the package, SaturatedLinear and the default nonlinearity of Tanh. For a continuous-time Hopfield network, defined by the parameters given in Eq. (2.25), one can define the energy of a particular state vector x as As for the discrete-time network, it can be shown that given an initial state vector x(0) the state vector x(t) in Eq. (2.28) converges to a local energy minimum. Hence, the minima of Eq. (2.29) constitute the possible convergence points of the Hopfield network and ideally these minima are identical to the class patterns {,..., }. However, there is no guarantee that the minima will coincide with this set of class patterns. 3. Structure:- The units in Hopfield nets are binary threshold units, i.e. the units only take on two different values for their states and the value is determined by whether or not the units' input exceeds their threshold. Hopfield nets normally have units that take on values of 1 or -1, and this convention will be used throughout the article. However, other literature might use units that take values of 0 and 1.Every two units i and j of a Hopfield network have a connection that is described by the connectivity weight. In this sense, the Hopfield network can be formally described as a complete undirected graph, where is a set of McCulloch-Pitts neurons and connectivity weight. is a function that links pairs of nodes to a real value, the The connections in a Hopfield net typically have the following restrictions: (no unit has a connection with itself) (connections are symmetric) Inderjeet Singh Behl et al. IJSRE vol 1 issue 6 Nov Page 3

4 A Hopfield net with four nodes. The requirement that weights be symmetric is typically used, as it will guarantee that the energy function decreases monotonically while following the activation rules, and the network may exhibit some periodic or chaotic behavior if non-symmetric weights are used. However, Hopfield found that this chaotic behavior is confined to relatively small parts of the phase space, and does not impair the network's ability to act as a content-addressable associative memory system. 4. Examples:- In this subsection, Hopfield networks are used to solve some simple classification problems. The first two examples illustrate the use of discrete-time Hopfield models, and the last two examples illustrate the continuous-time version on the same data sets Discrete-Time Two-Dimensional Example:- Load the Neural Networks package. In[1]:=<<NeuralNetworks` In this small example there are two pattern vectors {1,-1} and {-1,1}. Since the vectors are two-dimensional you can display the results to illustrate the outcome. Generate and look at class vectors. In[2]:=x={{1,-1},{-1,1}}; NetClassificationPlot[x] Inderjeet Singh Behl et al. IJSRE vol 1 issue 6 Nov Page 4

5 The two pattern vectors are placed in the corners of the plot. The idea is now that disturbed versions of the two pattern vectors should be classified to the correct undisturbed pattern vector. Define a discrete-time Hopfield network. In[4]:=hop = HopfieldFit[x] Out[4]= Since the discrete Hopfield network is the default type you do not have to specify that you want this type. Some descriptive information is obtained by using NetInformation. In[5]:= Out[5]= A new data pattern may be classified by processing it with the obtained model. Evaluate the network for some disturbed data vectors. In[6]:=hop[{0.4,-0.6}] Out[6]= More information about the evaluation of the Hopfield network on data vectors can be obtained by using NetPlot. The default is to plot the state trajectories as a function of time. Plot the state vectors versus time. In[7]:=NetPlot[hop, {{0.4, -0.6}}] Inderjeet Singh Behl et al. IJSRE vol 1 issue 6 Nov Page 5

6 It might be interesting to obtain the trajectories for further manipulation. They can be obtained using the evaluation rule with the option Trajectories True. Then the trajectories are returned instead of only the final value, which is the default. Obtain the state trajectory. In[8]:= Out[8]= The trajectory is the numerical solution to Eq. (2.26) describing the network; see Section 2.7, Hopfield Network. NetPlot can also be used for several patterns simultaneously. Evaluate two data vectors simultaneously. In[9]:=res = NetPlot[hop, {{0.4, -0.6}, {0.6, 0.7}}] By giving the option DataFormat Energy you obtain the energy decrease from the initial point, the data vector, to the convergence point as a function of the time. Look at the energy decrease. In[10]:=res = NetPlot[hop, {{0.4, -0.6}, {0.6, 0.7}},DataFormat Energy] Try 30 data pattern vectors at the same time. To avoid 30 trajectory and energy plots you can instead choose the option DataFormat ParametricPlot. You can use the command RandomArray from the standard add-on package Statistics`ContinuousDistributions` to generate random vectors. Plot a contour plot with state vector trajectories. Inderjeet Singh Behl et al. IJSRE vol 1 issue 6 Nov Page 6

7 In[11]:=<< Statistics`ContinuousDistributions` x = Random Array[Uniform Distribution[-1, 1], {10, 2}]; Net Plot[hop,x,DataFormat ParametricPlot,PlotRange {-1.2,1.2}] All trajectories converge to {-1,1} or {1,-1}, which are the two pattern vectors used to define the Hopfield network with Hopfield Fit 4.2.Continuous-Time Two-Dimensional Example:- Load the Neural Networks package. In[1]:=<<NeuralNetworks` Consider the same two-dimensional example as for the discrete Hopfield network. There are two pattern vectors {1,-1} and {-1,1}, and the goal is to classify noisy versions of these vectors to the correct vector. Generate and look at class pattern vectors. In[2]:=x={{1,-1},{-1,1}}; NetClassificationPlot[x] The two pattern vectors are placed in the corners of the plot. Define a continuous-time Hopfield network with a saturated linear neuron. Create a continuous-time Hopfield network. In[4]:=hop=HopfieldFit[x,NetType Continuous,Neuron SaturatedLinear] Out[4]= Inderjeet Singh Behl et al. IJSRE vol 1 issue 6 Nov Page 7

8 Provide some information about the Hopfield network. In[5]:= Out[5]= The obtained network can be used right away on any data vector by using the evaluation rule for Hopfield objects. Evaluate the Hopfield network on a data vector. In[6]:=hop[{0.4,-0.6}] Out[6]= Using NetPlot you can plot various information, for example, the state trajectories. Plot the state trajectories. In[7]:=NetPlot[hop,{{0.4, -0.6}}] It might be interesting to obtain the state trajectories of the evaluation of the Hopfield network on the data vectors. This can be done by setting the option Trajectories True in the evaluation of the Hopfield network. Obtain the state trajectory. In[8]:= Out[8]= The trajectory is the numerical solution to Eq. (2.28) describing the network (see Section 2.7, Hopfield Network), computed with a time step given by the variable Dt in the Hopfield object. It was automatically chosen when HopfieldFit was applied, to ensure a correct solution of the differential equation. Inderjeet Singh Behl et al. IJSRE vol 1 issue 6 Nov Page 8

9 The energy surface and the trajectories of the data vectors can provide a vivid illustration of the classification process. This is only possible for two-dimensional continuous-time Hopfield nets. Plot the energy surface together with the trajectories of several data vectors. In[9]:=<<Statistics`ContinuousDistributions` x = RandomArray[UniformDistribution[-1, 1], {30, 2}]; NetPlot[hop,x,DataFormat Surface]. 5. Use of the Hopfield network:- The way in which the Hopfield network is used is as follows. A pattern is entered in the network by setting all nodes to a specific value, or by setting only part of the nodes. The network is then subject to a number of iterations using asynchronous or synchronous updating. This is stopped after a while. The network neurons are then read out to see which pattern is in the network. The idea behind the Hopfield network is that patterns are stored in the weight matrix. The input must contain part of these patterns. The dynamics of the network then retrieve the patterns stored in the weight matrix. This is called Content Addressable Memory (CAM). The network can also be used for auto-association. The patterns that are stored in the network are divided in two parts: cue and association By entering the cue into the network, the entire pattern, which is stored in the weight matrix, is retrieved. In this way the network restores the association that belongs to a given cue. The stage is now almost set for the Hopfield network, we must only decide how we determine the weight matrix. We will do that in the next section, but in general we always impose two conditions on the weight matrix: ' symmetry: w ij = w ji no self connections: w ii = 0 6. Major Application of Hopfield Network:- Inderjeet Singh Behl et al. IJSRE vol 1 issue 6 Nov Page 9

10 Recalling or Reconstructing corrupted patterns Large-scale computational intelligence systems Handwriting Recognition Software Practical applications of HNs are limited because number of training patterns can be at most about 14% the number of nodes in the network. If the network is overloaded -- trained with more than the maximum acceptable number of attractors -- then it won't converge to clearly defined attractors. 7. Shortcomings of Hopfield network:- Training patterns can be at most about 14% the number of nodes in the network. If more patterns are used then the stored patterns become unstable; spurious stable states appear (i.e., stable states which do not correspond with stored patterns). Can sometimes misinterpret the corrupted pattern. 8. Conclusion:- Hopfield network was a breakthrough in neural network and gave a important dynamism to neural network's research. A lot of result are available and nowadays neural system are used in computer science. Their ability to learn by example makes them very flexible. They are also very well suited for real time systems because of their parallel architecture. Still, there are also critics. A. K. Dewdney wrote in 1997 Although neural nets do solve a few toy problems, their powers of computation are so limited that I am surprised anyone takes them seriously as a general problem-solving tool. Of course, a big problem is that neural network needs a lot of training to be efficient. References [1] Jehoshua Bruck. On the convergence properties of the Hopfield model. Proceedings of the IEEE, 78(10), October [2] Raul Rojas. Neural Networks. Springer, [3] Patrick K.Simpson. Artificial Neural Systems. Pergamon Press, [4] David Kriesel. A Brief Introduction to Neural Networks [5]. [6]. Inderjeet Singh Behl et al. IJSRE vol 1 issue 6 Nov Page 10