THE discrete multi-valued neuron was presented by N.

Transcription

1 Proceedings of International Joint Conference on Neural Networks, Dallas, Texas, USA, August 4-9, 2013 Multi-Valued Neuron with New Learning Schemes Shin-Fu Wu and Shie-Jue Lee Department of Electrical Engineering National Sun Yat-Sen University Kaohsiung 80424, Taiwan Abstract Multi-valued neuron (MVN) is an efficient technique for classification and regression. It is a neuron with complex-valued weights and inputs/output, and the output of the activation function is moving along the unit circle on the complex plane. Therefore, MVN may have more functionalities than sigmoidal or radial basis function neurons. In some cases, a pair of weighted sums would oscillate between two sectors and the learning process can hardly converge. Besides, many weighted sums may be located around the borders of each sector, which may cause bad performance in classification accuracy. In this paper, we propose two modifications of multivalued neuron. One is involved with moving boundaries and the other one with targets at the center of sectors. Experimental results show that the proposed modifications can improve the performance of MVN and help it to converge more efficiently. Index Terms Multi-Valued Neuron (MVN), complex-valued Neural Network (CVNN), classification, activation function, learning process. I. INTRODUCTION THE discrete multi-valued neuron was presented by N. Aizenberg and I. Aizenberg in [1]. This neuron operates with complex-valued weights. Its inputs and output are mapped onto the complex plane. They are located on the unit circle, and are exactly the k th roots of unity. The activation function of MVN k-valued logic maps a set of the k th roots of unity on itself. Two discrete-valued MVN learning algorithms are presented in [2]. They are based on errorcorrecting learning rule and are derivative-free. This makes MVN have higher functionality than sigmoidal or radial basis function neurons. Although the MVN has higher functionality than other neurons, it has difficulty learning highly non-linear functions. There are various methods to solve this problem, for example, multi-layer MVN (MLMVN) and MVN with periodic activation function (MVN-P). The MVN-based feedforward neural networks was proposed in [3] solving benchmark and real world problems. Rather than building the multilayer structure, MVN-P was introduced in [4] enhancing the ability of a single neuron with periodic activation function. Therefore, it is appealing to modify a single neuron in order to increase its functionality. In this paper, we first consider the discrete and continuous multi-valued neuron which activation function is modified with moving boundary. When learning the basic multi-valued neuron for classification, the weighted sum is moving toward the targeted sector on the complex plane. The locations of the k th roots are fixed on the unit circle that makes k sectors on the complex plane divided equally. The idea of multi-valued neuron with moving boundary is to dynamically change the size of each sector. The desired output and the weighted sum are cooperatively moving toward each other. MVN learning algorithm with this modification can improve the convergence of the learning process. Secondly, we consider improving the classification accuracy of multi-valued neuron by setting the learning target to the center of the targeted sector, not the k th roots of unity. The learning rule of basic MVN leads a weighted sum to a border of the targeted sector and eventually there may be many weighted sums located around the sector boundaries. Therefore, the classification accuracy suffers from this movement of weighted sums. Since the revised algorithm separates the learning instances from the sector boundaries, it can significantly improve the classification accuracy. A. Discrete MVN II. MULTI-VALUED NEURON A discrete-valued MVN is a function mapping from a n-feature input onto a single output. This mapping is described by a multiple-valued (k-valued) function of n-feature instances, f(x 1,..., x n ), which uses n 1 complex-valued weights: f(x 1,..., x n ) = P (ω 0 ω 1 x 1 ω n x n ) (1) where x 1,..., x n are the features of an instance, on which the performed function depends, and ω 0, ω 1,..., ω n are the weights. The values of the function and of the features are complex. They are the k th roots of unity: ɛ j = exp(i2πj/k), j 0, 1,..., k 1, and i is an imaginary unity. P is the activation function of the neuron: P (z) = exp(i2πj/k), if 2πj/k arg(z) < 2π(j 1)/k (2) where j = 0, 1,..., k 1 are values of the k-valued logic, z = ω 0 ω 1 x 1 ω n x n is the weighted sum, and arg(z) is the argument of the complex number z. Eq.(2) is illustrated in Fig. 1. Eq.(2) divides the complex plane into k equal sectors and maps the whole complex plane onto a subset of points belonging to the unit circle. This subset corresponds exactly to a set of the k th roots of unity. The MVN learning is reduced to the movement along the unit circle and is derivative-free. The movement is determined by the error which is the difference between the desired and actual output. The error-correcting learning rule /13/$ IEEE 58

2 Fig. 1: Geometrical interpretation of the discrete-valued MVN activation function. and the corresponding learning algorithm for the discretevalued MVN were described in [2] and modified by I. Aizenberg and C. Moraga [3]: i = ωi r C r (n 1) z r (ɛq ɛ s ) x i, (3) for i = 0, 1,..., n, where x i is the input of i th feature with the components complex-conjugated, n is the number of the input features, ɛ q is the desired output of the neuron, ɛ s = P (z) is the actual output of the neuron (see Fig. 2a), r is the number of the learning epoch, ωi r is the current weighting of the i th feature, i is the following weighting of the i th feature after correction, C r is the constant part of the learning rate (it may always equal to 1), and z r is the absolute value of the weighted sum obtained on the r th epoch. The 1 factor is useful when learning non-linear functions with z r a number of high irregular jumps. Eq.(3) ensures that the corrected weighted sum moves from sector s to sector q (see Fig. 2a). The direction of this movement is determined by the error δ = ɛ q ɛ s. The convergence of the learning algorithm was proven in [6]. (a) Fig. 2: Geometrical interpretation of the MVN learning rule: (a) Discrete-valued MVN, and (b) Continuous-valued MVN. B. Continuous MVN (b) The activation function Eq.(2) is piece-wise discontinuous. This function can be modified and generalized for the continuous case in the following way. When k in Eq.(2), the angle value of the sector (see in Fig. 1) will approach to zero. The activation function is transformed as follows: P (z) = exp(iarg(z)) = z (4) z where z is the weighted sum, arg(z) is the argument of complex number z, and z is the modulus of the complex number z. The activation function Eq.(4) maps the weighted sum into the whole unit circle (see Fig. 2b). Eq.(2) maps only to the discrete subsets of the points belonging to the unit circle. Eq.(2) and Eq.(4) are both not differentiable, but their differentiability is not required for MVN learning. The Learning rule of the continuous-valued MVN is shown as follows: C r i = ωi r (n 1) z r (ɛq ɛ s ) x i = ωi r C r (n 1) z r (ɛq z z ) x i, for i = 0, 1,..., n. III. PROPOSED MODIFICATIONS There are some problems associated with MVN: Weighted Sums Oscillating Between Two Sectors. The speed of convergence depends on the initial weights and the error. In some cases, a pair of weighted sum will oscillate between two sectors. Take Fig. 3 as an example. In this figure, p1 and p2 are two weighted (a) Fig. 3: Oscillation of a pair of weighted sums. sums to be corrected. Note that p1 belongs to the sector 2 and p2 belongs to sector 1. Due to their geometrical presentation and the correcting error, they will repeatedly oscillate between sector 1 and sector 2 (see in Fig. 3a and Fig. 3b). In such case, the learning algorithm can hardly converge. Weighted Sums being Crowded around the Sector Boundaries. Classification accuracy can be improved if the learning instances are separated from the sector boundaries. But the learning target of MVN is on the k th roots of unity which is exactly the boundaries of the k sectors. There may be many weighted sums located around the sector boundaries after numerous learning iterations. Hence, the classification accuracy suffers from this learning algorithm. (b) 59

3 We propose a new approach of MVN learning using moving boundary to solve the first problem. While learning the basic MVN, the k th roots of unity are located fixed on the unit circle and divides the unit circle into k equal sectors. Therefore the correcting error from one to a certain sector is static. The idea of learning MVN with moving boundary is to dynamically change the size of each sector. This new learning rule will iteratively train not only the weights but also the location of the k-valued logic thresholds. We expect this modification can improve the convergence of MVN efficiently. For the second problem, we were inspired by the learning method proposed in [5] and modify the learning rule such that the target of each learning iteration is at the center of target sector, instead of each sector. We expect the distribution of weighted sums will be around the center of each sector by this modified learning rule and the testing accuracy will be improved. A. MVN Learning with Moving Boundaries The activation function of as MVN sets the roots of unity separating the unit circle equally. Let E k be the set of the k th roots of unity: E k = {ɛ 0 k, ɛ 1 k, ɛ 2 k,..., ɛ k 1 k } boundary E k where ɛ k = exp(i2π/k) is the primitive k th root of unity and boundary is the set of boundaries of the k sectors, to be described later. Let K be the set of k-valued logic: K = {0, 1,..., k 1} Let O be a set of continuous values that are located on the unit circle and f(x 1,..., x n ) be a function with the mapping f : O n K. In order to obtain the function f(x 1,..., x n ) : O n K, we can normalize the domain of each feature into a value within the bounded sub-domain D n R n : f(y 1,..., y n ), y j [a j, b j [, a j, b j R, j = 1,..., n y j [a j, b j [ φ j = y j a j b j a j α, α [0, 2π) (5) The feature is transformed using the simple linear transformation in Eq.(5). Note that x j = exp(iφ j ) O, j = 1, 2,..., n is a complex number located on the unit circle. Hence, we obtain the function f(x 1,..., x n ) : O n K. Let the activation function be dynamically developed with boundaries of sectors. We can rewrite the activation function as follows: P d (z, boundary) = boundary(j), arg(boundary(j)) arg(z) < arg(boundary(j 1)) for j = 0, 1,..., k 1, where boundary(j) is the j th root of unity (boundary) on the unit circle. Therefore the boundary can be dynamic during the learning process. It is important to note that boundaries are moving along the unit circle while training, and boundary should be sorted by argument and normalized in each learning step. The idea of MVN learning is the movement of weighted sums, while the idea of MVN learning with moving boundary is the movement of both weighted sums and targeted boundaries (see Fig. 4). In order to obtain the dynamic boundaries, (a) Discrete-valued MVN. (b) Continuous-valued MVN. Fig. 4: Geometrical interpretation of the MVN learning with moving boundary. we introduce a parameter p which represents the proportion of correcting error ([0, 1]) for boundary movement. The learning rule of boundary movement can be expressed as follows: ɛ q r boundary(i) r ɛ q r1 = ɛq r p(ɛ q r ɛ s r) ɛ q r p(ɛ q r ɛ s r) boundary(i) r1 ɛ q r1 (6) where boundary(i) r is the current targeted boundary, and boundary(i) r1 is the updated boundary. The denominator of the corrected term is to normalize the trained boundary to the unit circle. Because error-correcting learning is very similar to competitive learning, they may have some common properties during the learning process. Competitive learning has a serious stability problem when clusters are close together. In certain cases, the weighted sum may invade other sectors, and therefore upset the current classification scheme. It is reasonable that targeted boundary and the weighted sum both contribute to the stability.

4 The learning process of the weights may be based on the same learning rule, Eq.(3), by incorporating the parameter p and applying P d as the dynamic activation function. The Learning rules of discrete and continuous MVN with moving boundary can be written as follows: C r i ωi r (n 1) z r (1 p)(ɛq r P d (z, boundary)) x i i ωi r C r (n 1) z r (1 p)(ɛq r z z ) x i (8) Since the learning algorithm still adopts the k-valued function and the error-correcting learning rule, the convergence of the learning algorithm can be proven based on the convergence of the learning rule Eq.(3) [6]. The implementation of the proposed learning algorithm in one iteration consists of the following steps: procedure MVN-MB(boundary, p) This is one learning epoch with N learning instances j = 1 Let z be the current value of weighted sum. P (z) = ɛ s δ ɛ q ɛ s while j N do Check equation (6) with activation function. for i = 0 to n do / n-feature instance / i ω r i C r (n 1) z r (1 p)δ x i end for ɛ q r1 ɛq r p δ ɛ q r p δ / ɛ q r is the targeted boundary in each learning step / boundary is updated with ɛ q r1 z = ω0 r1 ω1 r1 x 1 ωn r1 x n δ ɛ q r1 P d( z, boundary) / Complex sign function using dynamic boundaries / j = j 1 end while end procedure B. MVN Learning with Targets at the Centre of Sectors This MVN Learning rule is based on the basic discrete/continuous-valued MVN which we mentioned in section II. Let E k be the set of the k th roots of unity: E k = {ɛ 0 k, ɛ 1 k, ɛ 2 k,..., ɛ k 1 k } where ɛ k = exp(i2π/k) is the primitive k th root of unity and boundary is the set of boundaries of k sectors, to be described later. Let K be the set of k-valued logic: K = {0, 1,..., k 1} Let O be a set of continuous values that are located on the unit circle and f(x 1,..., x n ) be a function with the mapping f : O n K. In order to obtain the function f(x 1,..., x n ) : O n K, we can normalize the domain of each feature into a value within the bounded sub-domain D n R n. The (7) feature is transformed using the simple linear transformation in Eq.(5). Note that x j = exp(iφ j ) O, j = 1, 2,..., n is a complex number located on the unit circle. Hence, we obtain the function f(x 1,..., x n ) : O n K. The activation function of this learning algorithm is expressed in Eq.(2). The idea of MVN learning is to move the weighted sums to the target which is exactly the k th roots of unity. In order to increase the classification accuracy, it s reasonable to set the learning targets at the center of each sector (see in Fig. 5). Therefore, we introduce a new set of targets (a) Discrete-valued MVN. (b) Continuous-valued MVN. Fig. 5: Geometrical interpretation of the MVN learning with targets at the center of sectors. τ = {τ 0, τ 1,..., τ k 1 } for weighted sums moving to: τ q = µq µ q, µ q = 1 2 (ɛq ɛ q1 ) (9) where the τ q is the target of the q th sector and ɛ q is the q th roots of unity. There is an exception for binary classification problem, because µ 0 = µ 1 = 0. In this special case, we manually set µ 0 = exp(i π 2 ) and µ1 = exp( i π ). The 2 learning rule is based on the same learning rule, Eq.(3), by incorporating with modified new targets τ. The learning rules of discrete and continuous MVN with targets at the center 61

5 of sectors can be written as follows: i ωi r C r (n 1) z r (τ q P (z))) x i () i ωi r C r (n 1) z r (τ q z z ) x i (11) still adopts the k-valued function and the error-correcting learning rule, the convergence of the learning algorithm can be proven based on the convergence of the learning rule Eq.(3) [6]. The implementation of the proposed learning algorithm in one iteration consists of the following steps: procedure MVN-CS This is one learning epoch with N learning instances j = 1 Check equation (2) with activation function. Let z be the current value of weighted sum. Calculate τ using equation (9). P (z) = ɛ s δ τ q ɛ s while j N do for i = 0 to n do / n-feature instance / i ω r i C r (n 1) z r δ x i end for z = ω0 r1 ω1 r1 x 1 ωn r1 δ τ q P ( z) j = j 1 end while end procedure IV. SIMULATION RESULTS The proposed strategies and learning algorithms are implemented and checked over two benchmark datasets: Glass and Wine [9]. The software simulator is written in Matlab R2011b (64-bit) running on a computer with Intel R Core TM i5-3m 2.5 GHz CPU and 8 GB RAM. A. Glass Dataset The glass identification dataset is taken from the website of UC Irvine Machine Learning Repository [9]. It contains 214 instances and, for each instance, there are realvalued features. The instances belongs to any of the 6 classes indicating one of the following categories: float processed building windows, non-float processed building windows, float processed vehicle windows, non-float processed vehicle windows, containers, and tableware and headlamps. We merge these 6 classes into 3 which are building windows, vehicle windows, and the others. To solve this classification problem, we use 5-fold crossvalidation. The first fold includes 168 instances in the training set and 46 instances in the testing set. The other 4 folds include 172 instances in the training set and 42 instances in the testing set, respectively. After learning with the training set completely with no training error, i.e., training accuracy being 0%, we use the testing set to compute their x n average classification performance. In table (I), the number of learning epochs, the training time, and the testing accuracy involved with each fold are shown. Note that the parameter p is selected between 2 to 4. From this table, we can see that MVN runs faster than the other two versions. However, MVN achieves the lowest testing accuracy among the three methods. MVN with moving boundary runs very slow, but provides slightly better testing accuracy than MVN. MVN with new targets runs slightly slower than MVN, but provides much better testing accuracy than MVN. The distributions of the weighted sums after training are shown in Fig. 6. In Fig. 6(a), the weighted sums after discrete-valued MVN learning are close to certain border of a sector, which may reduce the classification accuracy. Fig. 6(b) shows that the boundaries actually moved during the learning process of MVN with moving boundary, and the moving range depends on the value of parameter p. In Fig. 6(c), the weighted sums moved to center of each sector, which meets our expectation. Sector 1([ π 3, 2π )) is 3 not obvious because the weighted sums in sector 1 are too close to the origin. B. Wine Dataset This dataset is also taken from the website of UC Irvine Machine Learning Repository [9]. It contains 178 instances and, for each instance, there are 13 real-valued features. The instances belongs to any of the 3 classes that indicates 3 different types of wines. To solve this classification problem, we use 5-fold crossvalidation. The first fold has 136 instances in the training set and 42 instances in the testing set. The other 4 folds have 144 instances in the training set and 34 instances in the testing set, respectively.after learning with the training set completely with no training error, i.e., training accuracy being 0%, we use the testing set to compute their average classification performance. In table (II), the number of learning epochs, the training time, and the testing accuracy involved with each fold are shown. Note that the parameter p is selected between 2 to 3. Again, we can see that MVN runs faster than the other two versions. However, MVN achieves the lowest testing accuracy among the three methods. MVN with moving boundary runs very slow, but provides slightly better testing accuracy than MVN. MVN with new targets runs slightly slower than MVN, but provides much better testing accuracy than MVN. The distributions of the weighted sums after training are shown in Fig. 7. In Fig. 7(a), the weighted sums after discrete-valued MVN learning are close to certain border of a sector, which may reduce the classification accuracy. Fig. 7(b) shows that the boundaries actually moved during the learning process of MVN with moving boundary, and the moving range depends on the value of parameter p. In Fig. 7(c), the weighted sums moved to center of each sector, which meets our expectation 62

6 TABLE I: Comparison of the three methods on the Glass dataset MVN MVN with moving boundary MVN with new targets* Epoch Time (sec) Accuracy (%) Epoch Time (sec) Accuracy (%) Epoch Time (sec) Accuracy (%) Average Average Average *:continuous learning rule (Eq.11) (a) MVN. (b) MVN-MB. Fig. 6: Distribution of weighted sums for the Glass dataset. (c) MVN-CS. TABLE II: Comparison of the three methods on the Wine dataset MVN MVN with moving boundary MVN with new targets* Epoch Time (sec) Accuracy (%) Epoch Time (sec) Accuracy (%) Epoch Time (sec) Accuracy (%) Average Average Average *:continuous learning rule (11) V. CONCLUSION We have proposed two new learning schemes for MVN. One is with a dynamic activation function which we call the moving boundary. For it we we introduced a parameter p in the boundary movement process. The other sets the learning targets at the center of the sectors. The simulation results obtained with benchmark datasets show that MVN with our modifications, especially the one with new targets, can improve convergence and testing accuracy. More work will be done to see their effects on MVN-P or the multilayer structure of MVN. REFERENCES [1] N. N. Aizenberg and I. N. Aizenberg, CNN based on multivalued neuron as a model of associative memory for grey scale images, in Proc. Workshop Second Int Cellular Neural Networks and their Applications CNNA-92, 1992, pp [Online]. Available: [2] I. Aizenberg, N. N. Aizenberg, and J. P. Vandewalle, Multi-Valued and Universal Binary Neurons: Theory, Learning and Applications. Springer,

7 (a) MVN. (b) MVN-MB. Fig. 7: Distribution of weighted sums for the Wine dataset. (c) MVN-CS. [3] I. Aizenberg and C. Moraga, Multilayer feedforward neural network based on multi-valued neurons (mlmvn) and a backpropagation learning algorithm, Soft Computing-A Fusion of Foundations, Methodologies and Applications, vol. 11, no. 2, pp , [Online]. Available: [4] I. Aizenberg, Periodic activation function and a modified learning algorithm for the multivalued neuron, vol. 21, no. 12, pp , 20. [Online]. Available: [5] I. Aizenberg, J. Jackson, and S. Alexander, Classification of blurred textures using multilayer neural network based on multi-valued neurons, Neural Networks (IJCNN), The 2011 International Joint Conference on pp.1328,1335, July Aug [Online]. Available: [6] I., Aizenberg, Complex-valued neural networks with multi-valued neurons. Springer, 2011, vol [7] I. Aizenberg, D. V. Paliy, J. M. Zurada, and J. T. Astola, Blur identification by multilayer neural network based on multivalued neurons, vol. 19, no. 5, pp , [Online]. Available: [8] M. T. Hagan, H. B. Demuth, M. H. Beale et al., Neural network design. Thomson Learning Stamford, CT, [9] UCI Machine Learning Repository. 64