Influence of Neighbor Size for Initial Node Exchange of SOM Learning

Transcription

1 FR-E3-3 Tokyo, Japan (September 20-24, 2006) Influence of Neighbor Size for Initial Node Exchange of SOM Learning MIYOSHI Tsutomu Department of Information and Knowledge Engineering, Tottori University, Tottori-shi Koyama-cho Minami 4-101, , Tottori, Japan. mijosxi@ike.tottori-u.ac.jp Abstract Self Organizing Map (SOM) is a kind of neural networks, that learns the feature of input data thorough unsupervised and competitive neighborhood learning. In SOM learning algorithm, every connection weights in SOM feature map are initialized at random values to covers whole space of input data, however, this is also set nodes at random point of feature map independently with data space. The move distance of output nodes increases and learning convergence becomes slow for this. As precedence research, I proposed the method that, initial node exchange by using a part of learning data. In this paper, I investigate how the average move distance of a node would change with the differences in initial size of neighbor area in node exchange process. As a result of experiments, I clarified the relation between average of move distance of nodes and initial neighbor size, drew the expression of relations, and showed the optimal domain of relations. K I. INTRODUCTION ohonen's self Organizing Map (SOM) [1] is a kind of neural networks [2], that algorithm learns the feature of input data thorough unsupervised and competitive neighbourhood learning. SOM is applied to many fields and there are a lot of studies [9]. SOM learning efficiency or learning speed must be essential to put to practical use of SOM, so some techniques are proposed that, batch-learning [10], incremental ordering of data units [11], rough comparison [12], etc. However, the studies about improvement methods based on conventional SOM learning algorithm are not a major stream. According to conventional SOM learning algorithm, SOM learning is influenced by the order of learning data and initial feature map. As precedence research, firstly we reported about the influence of the order of learning data [3-5]. We found that, a difference occurred at learning speed by the order of learning data even if all learning data and initial feature map are the same [3]. And I explained a part of this phenomenon by mathematical analysis [4-5]. Secondly I reported about initial feature map that also influence to SOM learning efficiency. In initialization process, every connection weights in feature map are initialized at random values to covers whole space of input data, however, this is also set nodes to random point of feature map independently with input data space. Unfortunately, learning speed or learning convergence becomes slow is expected for this relation missing, because this relation is self organized when convergence was completed. So, I proposed the method that, initial node exchange by using a part of learning data [6-8]. By using this method, both the average of move distance of all nodes and the time to the completion of learning were shortened to almost half [6]. And there is effect sufficient by performing exchange process by few number of learning data [7-8]. Since node exchange process is performed by adding to the conventional SOM learning algorithm, it is desirable for the load of processing to be smaller. It clarified about two of three factors which affect the load of processing, so, I thought that it was necessary to clarify influence of 2nd factor, initial size of neighbor area. In this paper, I investigate how the average move distance of a node would change with the differences in initial size of neighbor area in node exchange process. Hereafter, Chapter 2 explains SOM, Chapter 3 describes about initial node exchange, Chapter 4 describes experiments, and Chapter 5 describes a conclusion. II. SELF ORGANIZING MAP Kohonen's Self Organizing Maps (SOM) [1] is a kind of neural networks [2], that learns the feature of input data thorough unsupervised and competitive neighbourhood learning. SOM is mapping from a high dimensional space to a low dimensional space (usually a two dimensional map). It provides a feature map that arranged similar classes in near position, so it can visualize the high-dimensional information to a two dimensional feature map. Map representation makes us easy to understand the relation between data. Generally, SOM has two layers, input layer and output layer. The output layer nodes usually forms a two-dimensional grid and input layer nodes are fully connected with those at the output layer nodes. Each connection has connection weight, so every output layer nodes have pattern or vector to be learned

2 The image of SOM is shown in fig.1. Output Layer Connection (Connection Weights) Input Layer A. SOM Learning Algorithm Fig 1 : structure of SOM Output Layer Node Input Layer Node In learning process, when an input pattern or input vector is presented to the input layer as learning data, the output layer nodes compete with each other for the right to be declared the winner. The winner node will be the output layer node whose incoming connection weights are the closest to the input pattern in terms of Euclidean distance. The connection weights of the winner node and its neighbor nodes are then adjusted, i.e., moved closer in the direction of the input pattern. As learning process progresses, learning rate and the size of the neighbor area around the winner node will be made to decrease. So, large number of output layer nodes will be adjusted strongly in early stage of learning process, and only winner node will be adjusted weakly in final stage. SOM learning consists of double loop. By the inside loop, learning data is inputted in order and connection weights are adjusted. By the outside loop, learning rate and the size of the neighbor area around the winner node are made to decrease and an inside loop is repeated until learning is completed. The inside loop scheme of learning algorithm is shown as 1: select input data at random from learning data, 2: input data is presented to input nodes, 3: calculate distance between input data and all output nodes, 4: ordering output nodes by distance, 5: set the first order node to winner node, 6: adjust connection weights of the node, 7: select a neighbor node of winner node in feature map, 8: adjust connection weights of the node, 9: repeat 7 to 8 until all neighbor node are processed, 10: repeat 1 to 9 until all learning data are performed. In SOM learning algorithm, step 6 and step 8 of inside loop, connection weights of winner and its neighbors are moved closer in the direction of learning data or values of input nodes. Each connection weight is calculated by a formula (1). x n =x n 1 g x n 1 y n (1) x n : a connection weight learned by n-th learning data, y n : n-th learning data, g : learning rate. 0 g 1 The outside loop scheme of learning algorithm is shown as 1: initialize all connection weights at random, 2: initialize learning rate and the size of the neighbor area, 3: execute inside loop, 4: decrease learning rate and the size of the neighbor area, 5: repeat 3 to 4 until learning is completed. After learning, each node represents a group of individuals with similar features, the individual data correspond to the same node or to neighboring node. That is, SOM configures the output nodes into a topological representation of the original data, through a process called self organization. B. The Measure of Learning In conventional SOM algorithm, convergence of learning are mainly determined by following two measures: 1: the number of repetition becomes larger than the threshold, 2: the largest distance in all distances between learning data and its winner node becomes smaller than the threshold. At 1st measure, learning convergence itself is thought as important and learning speed is not taken into consideration. Usually long time or large number of repetition is set up as threshold, because it is only just sufficient length for convergence of learning. Learning takes the same time or same repetition limited by threshold, regardless of the performance of learning algorithms. At 2nd measure, distance between learning data and farthest winner node are used as a measure of learning. Since its attention is paid to a node with the slowest convergence, convergence of the whole feature map or whole nodes has not been measured. The average of the move distance of all nodes is proposed as a new measure [6], in order to consider both learning speed and convergence of the whole feature map. Since this will become a constant if learning converges, it can be said that, this is able to measure speed of convergence. And, since this is the average of the move distance instead of the move distance of a specific node, it can be said that, this has measured the whole feature map. The average of the move distance of all nodes at s-th learning loop M s is calculated by a formula (2). s M s = r k =1 j=1 q i=1 p x h,i, j, k x h,i, j 1, k 2 (2) h=1 x h,i, j, k : h-th dimension value of connection weight of the output node of i-th position of feature map learned by j-th learning data in k-th learning loop p : total number of connection weight dimension q : total number of output nodes r : total number of learning data s : number of learning loop

3 A. The Concept III. INITIAL NODE EXCHANGE In initialization process of SOM learning, step1 in the outside loop shown before section, every connection weights in feature map is initialized at random within the domain of each dimension. So, there is no relation between the positions of input space and the one of feature map. Unfortunately, it is expected that, learning speed becomes slow for this relation missing. The concept of initial node exchange is that, the neighbor nodes in input space are gathered to neighbor area in feature map by nodes exchange, without connection weights adjustment. The nodes, that are in the neighbor area of winner node in feature map, are exchanged to the nodes, that are in the neighbor area of winner node in input space. It is necessary to perform node exchange process by using selected learning data, because there are many mapping possibility from input space to feature map and desirable mapping changes depending on learning data. Through enough input data are chosen at random from leaning data, relation between the position of input space and the one of feature map must be made without connection weights adjustment. The image of node exchange process is shown in fig.2. input space feature map (a) initial random map (b) after node exchange Fig.2 : image of node exchange process B. Node Exchange Algorithm exchange input space feature map The scheme of initial node exchange process is shown as 1: select input data, 2: input data is presented to input nodes, 3: calculate distance between input data and all output nodes, 4: ordering output nodes by distance, 5: set the first order node to winner node, 6: select a neighbor node of winner node in feature map, 7: select next order node, 8: exchange their position in feature map or exchange connection weights, 9: repeat 6 to 8 until all neighbor node are processed, 10: repeat 1 to 9 until enough data are performed. initial node exchange process is easy to implement because the inside loop scheme of learning algorithm and initial node exchange process are resemblance. The differences are only step 6 and step 8 of inside loop. Step 6 in inside loop removed because it is connection weights adjustment of winner node. Step 8 in inside loop is rewrote from connection weights adjustment of neighbor nodes to select neighbor node in input space and exchange it to neighbor node in feature map. The leaning algorithm with initial node exchange is easy to realize by inserting node exchange process between feature map initialization and connection weights adjustment in outside loop scheme of learning process. The outside loop scheme of former method is shown as 1: initialize all connection weights at random, 2: initialize learning rate and the size of the neighbor area 3: execute initial node exchange process, 4: decrease learning rate and the size of the neighbor area, 5: repeat 3 to 4 until relation is made, 6: initialize learning rate and the size of the neighbor area, 7: execute inside loop, 8: decrease learning rate and the size of the neighbor area, 9: repeat 7 to 8 until learning is completed. C. Speed Factors of Node Exchange Since node exchange process is performed by adding to the conventional SOM learning algorithm, it is desirable for the load of processing to be smaller. In order to depend for the load of processing on the number of exchange nodes, the number of input data (1st factor), initial size of neighbor area (2nd factor), and the reduction speed of neighbor area (3rd factor) are related to processing load. In the previous paper [12], how the average move distance of nodes would change with the differences in the number of input data (1st factor) is investigated in the conditions that initial size of neighbor area and the reduction speed of neighbor were fixed to the same value of learning process. I reported in the paper, that the effect was the highest from 5% to 10% of the number of input data of the output node of the feature map, and the average move distance of nodes was shortened to about 70%. It is not necessary to repeat in node exchange, since node exchange process is completed by one exchange, although it is necessary to repeat learning process in order to adjust a weight vector gradually. Therefore, regulation of the reduction speed of neighbor area (3rd factor), required of learning process, is unnecessary in node exchange process. Node exchange process is always applied the fastest reduction speed. Since it clarified about two of three factors which affect the load of processing, I thought that it was necessary to clarify influence of the last one factor, initial size of neighbor area (2nd factor). Below, I investigate how the average move distance of a node would change with the differences in initial size of neighbor area. IV. EXPERIMENTS For the experiments, I used common parameters, 10x10 of 100 nodes two dimensional feature map, its connection weights are initialized at random values, learning rate is gradually converged on from 0.01 as learning progresses, neighbor size of learning is also converged from 7x7 area to single node. Learning data are created randomly

4 into 8 classes, the center of classes is one of the corner of 3 dimensional cube, and 20 data per class total 160 data are used. A. self Organization of Feature Map The experiment for confirming that learning is performed satisfactory by the method was conducted. An example of initial feature map is shown in fig.3. All connection weights are initialized at random values. The number, 0 to 7, shows the node belonging to the class in it, and different number shows different class. Fig.3 shows that nodes are distributed at random in initial feature map Fig.3 : initial feature map Fig.4 are the results of performing the method to initial feature map shown in fig.3. Fig.4(a) show that, in spite of not performing connection weights adjustment process, the node of same class has gathered each other by initial node exchange process. Fig.4(b) show that, the self organization of feature map is completely carried out after performing the method (a) after exchange (b) after learning Fig.4 : typical result of the method B. Initial Neighbor Size and Move Distance of Nodes The experiment which investigates how the average move distance of a node would change with the differences in initial size of neighbor area was conducted. Average value of 100 times of experiments is shown in fig.5, and from 0% to 18% part of fig.5 is shown in fig.6. Horizontal axis shows average of move distance of nodes by the percentage which set conventional SOM to 100. Vertical axis shows number of input data for exchange by the percentage which set number of all output nodes belonging to feature map to 100. Each line shows how average of move distance of nodes will change with initial size of neighbor area, and the number of input data. For example, 10% line indicated by lozenge expresses change in case that initial size of neighbor area is 10% of the feature map. From this line it understands that, before learning process, node exchange process is performed with 30% of input data and with 10% of initial neighbor size of feature map, average of move distance of nodes is being shortened to 75% compared with the case where it learns without performing node exchange. average of move distance (%) Fig.5: the average of the move distance and the number of data for exchange According to fig.5, if the number of input data exceeds 30% of feature map, it can observe that average of move distance of nodes becomes rather longer. average of move distance (%) number of input data (%) number of input data (%) Fig.6: from 0% to 18% part of fig.5 10% 25% 50% 80% 120% 170% 225% 290% 360% 440% 10% 25% 50% 80% 120% 170% 225% 290% 360% 440% According to fig.6, when an initial neighbor size is 50% or more, the average of move distance of nodes is the shortest (72% or less) from 4% to 10% of input data, and enough shorter (75% or less) at only 2%. When an initial neighbor size is 25%, the average of move distance of nodes is the shortest (about 72%) from 5% of input data. While there is a

5 small number of input data, average of move distance of node is little longer, but when it sees on the whole, there is almost no difference. When an initial neighbor size is 10%, the average of move distance of nodes is the shortest (about 76%) from 10% of input data. At 30% or less of input data, average of move distance of node is long clearly, and if it exceeds 30%, a difference will almost be lost. From these, it can observe that sufficient effect is acquired, if the product of the initial neighbor size and the number of exchange data are over 100, and the number of input data is under 30. This condition is shown in formula (3). Na 0 Nd 100 and Nd 30 (3) Na 0 : percentage of initial neighbor size, Nd : percentage of number of input data for exchange V. CONCLUSIONS In this paper, I investigate how the average move distance of a node would change with the differences in initial size of neighbor area in node exchange process. As a result of experiments, I clarified the relation between average of move distance of nodes and initial neighbor size, drew the expression of relations, and showed the optimal domain of relations. Sufficient effect is acquired, if the product of the initial neighbor size and the number of exchange data are over 100, and the number of input data is under 30. It is clear that, regulation of the reduction speed of neighbor area (3rd factor) is unnecessary in node exchange process. It becomes a future subject about sturdy of the processing method of the neighbor area optimized to initial node exchange process. [7] MIYOSHI Tsutomu : "Initial Node Exchange and Convergence of SOM Learning," Proceedings of The 6th International Symposium on Advanced Intelligent Systems (ISIS2005), Yeosu, Korea, pp (2005). [8] MIYOSHI Tsutomu : "Initial Node Exchange Using Learning Data and Convergence of SOM Learning," GESTS International Transactions on Computer Science and Engineering, Vol.21, No.1, pp (2005). [9] "Bibliography of SOM papers," Draft version , [10] Makoto KINOUCHI, KUDO Yoshihiro : "Much faster learning algorithm for batch-learning som and its application to bioinformatics." Proceedings of The Workshop on Self-Organizing Maps (WSOM03) Kitakyushu, Japan (2003). [11] Young Pyo Jun, Hyunsoo Yoon, Jung Wan Cho : "L* learning: a fast self-organizing feature map learning algorithm based on incremental ordering." IEICE TRANSACTIONS on Information and Systems, Vol.E76- D, No.6, pp , (1993). [12] Hakaru Tamukou, Keiichi Horio, Takeshi Yamakawa : "Fast learning algorithms for self-organizing map employing rough comparison WTA and its digital hardware implementation." IEICE TRANSACTIONS on Electronics, Vol.E87-C, No.11, pp (2004). REFERENCES [1] Teuvo Kohonen : "Self-Organizing Maps," Springer Verlag, ISBN (1995). [2] Robert Heclt-Nielsen : "Neurocomputing," Addison- Wesley Pub. Co., ISBN (1990). [3] Tsutomu Miyoshi, Hidenori Kawai, Hiroshi Masuyama : "Efficient SOM Learning by Data Order Adjustment," Proceedings of 2002 IEEE World Congress on Computational Intelligence (WCCI2002), pp (2002). [4] MIYOSHI Tsutomu : "Order of Learning Data and Convergence of SOM Learning," Proceedings of The 6th International Symposium on Advanced Intelligent Systems (ISIS2005), Yeosu, Korea, pp (2005). [5] MIYOSHI Tsutomu : "Learning Data Order and Convergence of SOM Learning," GESTS International Transactions on Computer Science and Engineering, Vol.22, No.1, pp (2005). [6] MIYOSHI Tsutomu : "Node Exchange for Improvement of SOM Learning," Proceedings of 9th International Conference on Knowledge-Based Intelligent Information and Engineering Systems (KES2005), Melbourne, Australia, pp (2005)