Sparse Distributed Memory Pattern Data Analysis

Transcription

1 Sparse Distributed Memory Pattern Data Analysis František Grebeníček * grebenic@dcse.fee.vutbr.cz Abstract: This paper discusses, how some statistical properties of pattern data can affect efficiency of Kanerva's Sparse Distributed Memory (SDM). Then, it suggests an method which should improve SDM efficiency. The method looks for optimal SDM parameters according to properties of pattern data. Results of simple experiment are included. Key Words: Sparse Distributed Memory, neural network, pattern recognition 1 Introduction The SDM was developed by Kanerva [1] and it may be regarded either as an extension of a classical random-access memory (RAM) or as a special type of three layer feedforward neural network. The main SDM alterations to the RAM are: The SDM calculates Hamming distances between the reference address and each location address. For each distance which is less or equal to the given radius the corresponding location is selected. The own memory is represented by n m counters (where n is number of locations and m is the input data length) instead of single-bit storage elements. Writing to the memory, instead of overwriting, is as follows: if the i-bit of the input data is 1, the corresponding counters (counters in the selected locations (rows) and in the i-th columns) are incremented, if the i-bit of the input data is 0, the corresponding counters are decremented. Reading (or recall) from the memory is similar: The contents of the selected locations are summed columnwise. Each sum is thresholded. If the sum is greater than or equal to the threshold value the corresponding output bit is set to 1, in the opposite case it is cleared. Note that the thresholds may be zero, if the training input vectors are closed to orthogonal ones. SDM implements transformation from logical space to physical space using distributed data storing. A value corresponding to a logical address is stored into many physical addresses. This way of storing is robust and not deterministic. A memory cell is not addressed directly. If input data (logical addresses) are partially damaged at all, we can get correct output data. Details about principles and implementation can be found in [2], [3], [4], [5], [6]. 2 SDM Analysis This section introduces some mathematical foundations which are significant for SDM. The concepts and properties of the space {0,1} n are adopted from Kanerva [1]. * Department of Computer Science and Engineering, Božetěchova 2, CZ Brno

2 2.1 The Space {0,1} n The SDM works with n-dimensional vectors with binary components. Depending on the context, the vectors are called points, patterns, addresses, words, memory items, data, or events. This section is mostly about the properties of the vector space [1]. Let n be number of dimensions of the space. The number of points, or possible memory items, is then 2 n. We will denote this number by N and will use N and 2 n to stand also for the space itself. 2.2 Concepts Related to the Space {0,1} n Origin, 0 The point with all coordinates 0 is called the origin, 0 = Complement, 'x Norm, x Difference, x y Distance, d(x, y) The complement, or opposite, of point x is the n-tuple that has ones where x has zeros and vice versa. The norm of point x is the number of ones in its binary representation. The difference of two points x and y is the n-tuple that has ones where x and y differ and zeros elsewhere. It is the bitwise 'exclusive or': x y = x y. The difference commutes: x y = y x. The distance between two points x and y is the number of dimensions at which x and y differ. It is called the Hamming distance (its square root is the Euclidean distance) and is expressed in bits. Distance is the norm of the difference: d(x, y) = x y Betweenness, x:y:z Point y is between points x and z if and only if the distance from x to z is the sum of the distances from x to y and from y to z; that is, x:y:z d(x, z) = d(x, y) + d(y, z). It is easily seen that every bit of a point in between is a copy of the corresponding bit of an endpoint. Orthogonality, x y Point x is orthogonal to point y, or the two are perpendicular or indifferent, if and only if the distance between the two is half the number of dimensions: x y d(x, y) = n/2. The distance n/2 is called indifference distance of {0,1} n. If x is orthogonal to y, it is also orthogonal to its complement 'y (x is halfway between y and 'y). Circle, O(r,x) A circle with radius r and center x is the set of points that are at most r bits from x: O(r,x) = {y d(x, y) r}. 2.3 Some Properties of the Space {0,1} n The space N can be represented by the vertices of the unit cube in n-dimensional Euclidean space. The vertices lie on the surface of an n-dimensional sphere with (Euclidean-metric) radius n 2. This gives rise to the sphere analogy. We will call a space spherical if 1. any point x has a unique opposite 'x, 2. the entire space is between any point x and its opposite 'x, and 3. all points are "equal" (meaning that for any two points x and y there is a distancepreserving automorphism of the space that maps x to y, so that from any of its points the space "looks" the same). The surface of a sphere (in Euclidean 3-space) clearly is spherical. According to definition, N is also spherical, since y x ( ) is an automorphism that maps x to y.

3 Because N is spherical, it is helpful to think of it as the surface of a sphere with circumference 2n. All points of N are equally qualified as points of origin, and a point and its complement are like two poles at distance n from each other, with the entire space in between. The points halfway between the poles and perpendicular to them are like the equator. In addition to N's having a finite number of points and a sphere's being continuous, the spaces differ in other important ways. For example, the points of N between two points x and y are a d(x, y)-dimensional subspace of N, whereas the corresponding set on a sphere is a "straight" line. The segments (i.e., the minimal paths between two points) of N are not even unique. Also, the circles on N, as a rule, are not convex. Distribution of the Space N: The number of points that are exactly d bits form an arbitrary point x (say, from the point 0) is the number of ways to choose d coordinates from a total of n coordinates, and is therefore given by the binomial coefficient n n! = (2.1) d d!( n d)! The distribution of N thus is the binomial distribution with parameters n and p, where p = 1/2. The mean of the binomial distribution is n/2, and the variance is n/4. This distribution function will be denoted by N(d). The normal distribution F with mean n/2 and standard deviation n 2 is a good approximation to it: N(d) = Pr{d(x, y) d} F{( d n / 2) / n / 4} (2.2) Tendency to Orthogonality: An outstanding property of N is that most of it lies at approximately the mean (indifference) distance n/2 from a point (and its complement). In other words, most of the space is nearly orthogonal to any given point, and the larger n is, the more pronounced is this effect. The mathematics of it works out as follows: If we divide the mean distance n/2 by the standard deviation of distance n 2, we get that the distance from a point to the bulk of the space (from a pole to the equator) is n standard deviations. For n = 1000 it is 31.6 standard deviations. According to the normal distribution, over of the space lies within 5 standard deviations of the mean or within ( n ± 5) standard deviations from a point of N. With n = 1000, the mean distance is 500 bits, but only about a millionth of the space is closer to the point than 422 bits or farther from it than 578 bits. 3 Data Analysis In this section we concern to distances between pattern addresses. If pattern data are random, then (respecting tendency of orthogonality of N) most of them lies approximately the mean distance n/2 from a point. In this case we can get optimal radius value r solving equation: N(r) = 0.01, see Cibulka [3]. (3.1) Found value is rounded to integer and SDM working with this radius is maximally efficient. But, pattern data from real applications are not random. It causes less SDM efficiency. So, we look for a way how to find optimal radius for non-random pattern data. Let D is matrix of distances between pairs of pattern addresses:

4 D ij = d(t i, t j ), i,j = 1..T (3.2) It is clear that D ij = 0 for i = j. Now we define the mean pattern distance as a mean value of triangular part of matrix D: d mean 2 = TT ( ) T 1 j= 2 i= 1 j We can define the standard pattern deviation in a similar way: D ij (3.3) d dev = 2 T ( T 1) T j j = 2 i= 1 ( D ij d mean ) 2 (3.4) Hypothesis 1: the optimal radius must be less than the mean pattern distance: r opt < d mean (3.5) In the ideal case pattern data are random, mean pattern distance approximately equals to n/2. Probability N(n/2) = 1/2. Clearly, solving equation (3.1) we get radius less then n/2. In the ideal case the hypothesis 1 is performed. We can constitute stricter hypothesis: Hypothesis 2: the optimal radius must be less than the mean pattern distance minus the standard pattern deviation: r opt < d mean d dev (3.6) If pattern data are random, the standard pattern deviation approximately equals to n 2. Probability N ( n / 2 n 2) = F{ 1} =! We can solve equation F{x} = 0.01 for x and find x =! It means that optimal radius for random pattern data is about d mean 2.33d dev. In the ideal case the hypothesis 2 is performed too. 4 Experiment 4.1 Data Images of digits are used for the experiment. Images are pixels large, black and white, digits are rasterized from standard Microsoft Arial Bold font. Figure 4.1 shows the whole pattern set. Table 4.1 shows a triangular part of distance matrix definition (3.2). Mean value (3.3) of distances equals 64 bits. Standard deviation (3.3) equals approximately 22 bits. There is also an additional property: 99 "unused" bits. The unused bit is a bit which is always white or always black in the pattern set. Figure 4.1 Pattern set

5 Table 4.1 Triangular part of distance matrix (values in bits) 4.2 Used SDM Used SDM has both input and output 256 bits width. For the first measuring, the SDM radius is set to 110 bits. It corresponds to line selection probability around Location count is 10000, therefore mean number of selected locations should be about 122. For the second measuring, the SDM radius is set to the mean value (see Hypothesis 1, section 3), i.e. 64 bits. It corresponds to line selection probability around It means that location count should be minimally This condition is unacceptable for realization of original Kanerva's SDM design. Therefore we use a modified form of SDM (Zbořil [8]) where only part of address space is generated and location count can be reduced to For the third measuring, the SDM radius is set to 42 bits, corresponding to Hypothesis 2. Like in the previous case, the modified form of SDM, location count is Results Pattern set (Figure 4.1) was autoassociatively written to corresponding SDM and then the same patterns were read. Results are shown at following figures: Figure 4.2 Measuring 1: Kanerva's SDM desing, radius 110 Figure 4.3 Measuring 2: Modified SDM design, radius 64 Figure 4.4 Measuring 3: Modified SDM design, radius 42 Differences between written and read data were also computed. Following table shows mean error in bits, mean error relative to data width (256 bits), and mean error relative to "used" bits (data width minus "unused" bits, see section 4.1, i.e = 157 bits):

6 SDM, radius Mean error (bits) Mean error (% of 256) Mean error (% of used bits) Kanerva, Modified, Modified, Table 4.2 Mean errors of reading 5 Conclusions The simple experiment shows that two hypotheses introduced in section 3 can be useful for improving SDM efficiency. The first hypothesis allows SDM to generalize partially. The second hypothesis is more strict and eliminates generalization. The modified SDM was used in the experiment. The modification is similar to RCE (Restricted Coulomb Energy) net, as it was discussed in [8]. The new method of radius estimation can be very useful for the modification and makes it more similar to RCE. The relation between these two still requires further research. This project was supported by the GACR grant No. 102/98/0552 (Research and Applications of Heterogenous Models). References 1. Kanerva, P.: Sparse Distributed Memory, The MIT Press, Cambridge, Massachusetts Rogers, D.: Kanerva s sparse distributed memory: An associative memory algorithm well-suited to the Connection Machine. In Proc. Conference on Scientific Application of the connection machine, Moffet Field, CA, 1988, Vol.1, Cibulka, J.: Sparse Distributed Memory: An Analysis, In: ASIS 1997, MARQ, Ostrava, Krnov, 1997, p , ISBN Grebeníček, F.: Data coding for SDM, In: Mosis'98 Proceedings, Vol. 1, MARQ, Ostrava, Hostýn, Czech Republic, 1998, p , ISBN Grebeníček, F.: Neural Net Applications in Image Processing, In: Asis'98 Proceedings, Vol. 2, MARQ, Ostrava, Krnov, Czech Republic, 1998, p , ISBN Zbořil, F.: An Application of the Sparse Distributed Memory, In: Proceedings of the ASIS 1997, MARQ, Ostrava, Krnov, 1997, p , ISBN Zbořil, F.: Analysis of Neural Networks Applications in Heterogeneous Models, In: Proceedings of the ASIS'98, MARQ Ostrava, Krnov, 1998, p , ISBN Zbořil, F.: Sparse Distributed Memory and Restricted Coulomb Energy Classifier, In: Proceedings of the MOSIS'98, MARQ, Ostrava, Sv. Hostýn - Bystřice pod Hostýnem, 1998, p , ISBN Zbořil, F.: Neural Associative Memories, In: Proceedings of the MOSIS'99, MARQ Ostrava, Rožnov pod Radhoštěm, 1999, p , ISBN X