Brief Review of Self-Organizing Maps

Transcription

1 Bref Revew of Self-Organzng Maps Dubravko Mljkovć Hrvatska elektroprvreda, Zagreb, Croata Abstract - As a partcular type of artfcal neural networks, self-organzng maps (SOMs) are traned usng an unsupervsed, compettve learnng to produce a lowdmensonal, dscretzed representaton of the nput space of the tranng samples, called a feature map. Such a map retans prncple features of the nput data. Self-organzng maps are known for ts clusterng, vsualzaton and classfcaton capabltes. In ths bref revew paper basc tenets, ncludng motvaton, archtecture, math descrpton and applcatons are revewed. I. INTRODUCTION Among numerous neural network archtectures, partcularly nterestng archtecture was ntroduced by Fnsh Professor Teuvo Kohonen n the 1980s, [1,2]. Selforganzng map (SOM), sometmes also called a Kohonen map use unsupervsed, compettve learnng to produce low dmensonal, dscretzed representaton of presented hgh dmensonal data, whle smultaneously preservng smlarty relatons between the presented data tems. Such low dmensonal representaton s called a feature map, hence map n the name. Ths bref revew paper attempts to ntroduce a reader to SOMs, coverng n short basc tenets, underlyng bologcal motvaton, ts archtecture, math descrpton and varous applcatons, [3-10]. II. NEURAL NETWORKS bologcal neurons are replaced wth neuron nput weghts. Adjustment of connecton weghts s performed by some of numerous learnng algorthms. ANNs have very smple prncples, but ther behavor can be very complex. They have a capablty to learn, generalze, assocate data and are fault tolerant. The hstory of the ANNs begns n the 1940s, but the frst sgnfcant step came n 1957 wth the ntroducton of Rosenblatt s perceptron. The evoluton of the most popular ANN paradgms s shown n Fg. 1, [10]. B. Basc Archtectures An artfcal neural network s an nterconnected assembly of smple processng elements, called artfcal neurons (also called unts or nodes), whose functonalty mmcs that of a bologcal neuron, [4]. Indvdual neurons can be combned nto layers, and there are sngle and mult-layer networks, wth or wthout feedback. The most common types of ANNs are shown n Fg. 2, [11]. Among tranng algorthms the most popular s backpropagaton and ts varants. ANNs can be used for solvng a wde varety of problems. Before the use they have to be traned. Durng the tranng, network adjusts ts weghts. In supervsed tranng, nput/output pars are presented to the network by an external teacher and network tres to learn desred nput output mappng. Some neural archtectures (lke SOM) can learn wthout supervson (unsupervsed) from the tranng data wthout specfed nput/output pars. Human and anmal brans are hghly complex, nonlnear and parallel systems, consstng of bllons of neurons ntegrated nto numerous neural networks, [3]. A neural networks wthn a bran are massvely parallel dstrbuted processng system sutable for storng knowledge n forms of past experences and makng t avalable for future use. They are partcularly sutable for the class of problems where t s dffcult to propose an analytcal soluton convenent for algorthmc mplementaton. A. Bologcal Motvaton After mllons of years of evoluton, bran n anmals and humans has evolved nto the massve parallel stack of computng power capable of dealng wth the tremendous varetes of stuatons t can encounter. The bologcal neural networks are natural ntellgent nformaton processors. Artfcal neural networks (ANN) consttute computng paradgm motvated by the neural structure of bologcal systems, [6]. ANNs employ a computatonal approach based on a large collecton of artfcal neurons that are much smplfed representaton of bologcal neurons. Synapses that ensure communcaton among Fgure 1. Evoluton of artfcal neural network paradgms, based on [10] Fgure 2. Most common artfcal neural networks, accordng to [11] 1252 MIPRO 2017/CTS

2 III. SELF-ORGANIZING MAPS Self-organzed map (SOM), as a partcular neural network paradgm has found ts nspraton n selforganzng and bologcal systems. A. Self-Organzed Systems Self-organzng systems are types of systems that can change ther nternal structure and functon n response to external crcumstances and stmul, [12-15]. Elements of such a system can nfluence or organze other elements wthn the same system, resultng n a greater stablty of structure or functon of the whole aganst external fluctuatons, [12]. The man aspects of self-organzng systems are ncrease of complexty, emergence of new phenomena (the whole s more than the sum of ts parts) and nternal regulaton by postve and negatve feedback loops. In 1952 Turng publshed a paper regardng the mathematcal theory of pattern formaton n bology, and found that global order n a system can arse from local nteractons, [13]. Ths often produces a system wth new, emergent propertes that dffer qualtatvely from those of components wthout nteractons, [16]. Self-organzng systems exst n nature, ncludng non-lvng as well as lvng world, they exst n man-made systems, but also n the world of abstract deas, [12]. B. Self-Organzng Map Neural networks of neurons wth lateral communcaton of neurons topologcally organzed as self-organzng maps are common n neurobology. Varous neural functons are mapped onto dentfable regons of the bran, Fg. 3, [17]. In such topographc maps neghborhood relaton s preserved. Bran mostly does not have desred nput-output pars avalable and has to learn n unsupervsed mode. 1. Compettve Process For each nput pattern vector presented to the map, all neurons calculate values of a dscrmnant functon. The neuron that s most smlar to the nput pattern vector s the wnner (best matchng unt, BMU). 2. Cooperatve Process The wnner neuron (BMU) fnds the spatal locaton of a topologcal neghborhood of excted neurons. Neurons from ths neghborhood may then cooperate. 3. Synaptc Adaptaton Provdes that excted neurons can modfy ther values of the dscrmnant functon related to the presented nput pattern vector by the process of weght adjustments. D. Common Topologes SOM topologes can be n one, two (most common) or even three dmensons, [2-10]. Two most used two dmensonal grds n SOMs are rectangular and hexagonal grd. Three dmensonal topologes can be n form of a cylnder or torod shapes. 1-D (lnear) and 2-D grds are llustrated n Fg. 4, wth correspondng SOMs n Fg. 5 and Fg. 6, accordng to [19]. Fgure 4. Most common grds and neuron neghborhoods Fgure 5. 1-D SOM network, accordng to [19]. Fgure 6. 2-D SOM network, accordng to [19]. Fgure 3. Maps n bran, [17] A SOM s a sngle layer neural network wth unts set along an n-dmensonal grd. Most applcatons use twodmensonal and rectangular grd, although many applcatons also use hexagonal grds, and some one, three, or more dmensonal spaces. SOMs produce lowdmensonal projecton mages of hgh-dmensonal data dstrbutons, n whch the smlarty relatons between the data tems are preserved, [18], C. Prncples of Self-Organzaton n SOMs Followng three processes are common to selforganzaton n SOMs, [7,19,20]: IV. LEARNING ALGORITHM In 1982 Professor Kohonen presented hs SOM algorthm, [1]. Further advancement n a feld came wth the Second edton of hs book Self-Organzaton and Assocatve Memory n 1988, [2]. A. Measures of Dstance and Smlarty To determned smlarty between the nput vector and neurons measures of dstance are used. Some popular dstances among nput pattern and SOM unts are, [21]: Eucldan Correlaton Drecton cosne Block dstance In a real applcaton most often squared Eucldean dstance s used, (1): 2 dj x w j (1) MIPRO 2017/CTS 1253

3 B. Neghborhood Functons Neurons wthn a grd nteract among themselves usng a neghbor functon. Neghborhood functons most often assume the form of the Mexcan hat, (2), Fg. 7, that has bologcal motvaton behnd (rejects some neurons n the vcnty to the wnnng neuron) although other functons (Gaussan, cone and cylnder) are also possble, [22]. Orderng algorthm s robust to the choce of functon type f the neghbor radus and learnng rate decrease to zero. The popular choce s the exponental decay. 5. Contnuaton Increment n. Repeat steps 2-4 untl a stoppng crteron s met (e.g. the fxed number of teratons or the map has reached a stable state). For the convergence and stablty to be guaranteed, the learnng rate αn and neghborhood radus rn are decreasng wth each teraton towards the zero, [22]. SOM Sample Hts, Fg. 8, show the number of nput vectors that each unt n SOM classfes, [24] n jm 2 r h w, w, r e (2) g j mn Fgure 7. Mexcan hat functon C. Intalzaton of Self-Organzng Maps Before tranng SOM, unts (.e. ts weghts) should be ntalzed. Common approaches are, [2,23]: 1. Use of random values, completely ndependent of the tranng data set 2. Use of random samples from the nput tranng data 3. Intalzaton that tres to reflect the dstrbuton of the data (Prncpal Components) D. Tranng Self-organzng maps use the most popular algorthm of the unsupervsed learnng category, [2]. The crteron D, that s mnmzed, s the sum of dstances between all nput vectors xn and ther respectve wnnng neuron weghts w calculated at the end of each epoch, (3), [21]: k 2 (3) D x w n 1 nc Tranng of self-organzng maps, [2,18], can be accomplshed n two ways: as sequental or batch tranng. 1. Sequental tranng sngle vector at a tme s presented to the map adjustment of neuron weghts s made after a presentaton of each vector sutable for on-lne learnng 2. Batch tranng whole dataset s before any adjustment to the neuron weghts s made sutable for off-lne learnng Here are steps for the sequental tranng, [3,7,19,22]: 1. Intalzaton Intalze the neuron weghts (teraton step n=0) 2. Samplng Randomly sample a vector xn from the dataset 3. Smlarty Matchng Fnd the best matchng unt (BMU), c, wth weghts wbmu=wc, (4): c n n (4) arg mn x w 4. Updatng Update each unt wth the followng rule: w n 1 w n nhw n, w n, rnx nw n (5) bmu Fgure 8. SOM Sample Hts, [24] Durng the tranng process two phases may be dstngushed, [7,18]: 1. Self-organzng (orderng) phase: Topologcal orderng n the map takes place (roughly frst 1000 teratons). The learnng rate αn and neghborhood radus rn are decreasng. 2. Convergence (fne tunng) phase: Ths s fne tunng that provdes an accurate statstcal representaton of the nput space. It typcally lasts at least (500 x number of neuron) teratons. The smaller learnng rate αn and neghborhood radus rn may be kept fxed (e.g. last values from the prevous phase). After the tranng of the SOM s completed, neurons may be labeled f labeled pattern vectors are avalable. E. Classfcaton Fnd the best matchng unt (BMU), c, (5): (5) c arg mn x w Test pattern x belongs to the class represented by the best matchng unt c. V. PROPERTIES OF SOM After the convergence of SOM algorthm, resultng feature map dsplays mportant statstcal characterstcs of the nput space. They are also able to dscover relevant patterns or features present n the nput data. A. Important Propertes of SOMs SOMs have four mportant propertes, [3,7]: 1. Approxmaton of the Input Space The resultng mappng provdes a good approxmaton to the nput space. SOM also performs dmensonalty reducton by mappng multdmensonal data on SOM grd. 2. Topologcal Orderng Spatal locatons of the neurons n the SOM lattce are topologcally related to the features of the nput space. 3. Densty Matchng The densty of the output neurons n the map approxmates the statstcal dstrbuton of the nput space. Regons of the nput space that contan more tranng vectors are represented wth more output neurons MIPRO 2017/CTS

4 4. Feature Selecton Map extracts the prncpal features of the nput space. It s capable of selectng the best features for approxmaton of the underlyng statstcal dstrbuton of the nput space. B. Representng the Input Space wth SOMs of Varous Topologes 1. 1-D 2D nput data ponts are unformly dstrbuted n a trangle, 1D SOM orderng process shown n Fg. 9, [2]. Fgure 9. 2D to 1D mappng by a SOM (orderng process), [2] 2. 2-D 2D nput data ponts are unformly dstrbuted n a square, 2D SOM orderng process shown n Fg. 10, [3]. VI. APPLICATIONS Despte ts smplcty, SOMs can be used for a varous classes of applcatons, [2,26,27]. Ths n a broad sense ncludes vsualzatons, generaton of feature maps, pattern recognton and classfcaton. Kohonen n [2] came wth the followng categores of applcatons: machne vson and mage analyss, optcal character recognton and scrpt readng, speech analyss and recognton, acoustc and muscal studes, sgnal processng and radar measurements, telecommuncatons, ndustral and other real world measurements, process control, robotcs, chemstry, physcs, desgn of electronc crcuts, medcal applcatons wthout mage processng, data processng lngustc and AI problems, mathematcal problems and neurophysologcal research. Wth such an exhaustve lst provded here, as space permts, t s possble only to menton some of them that are nterestng and popular. A. Speech Recognton The neural phonetc typewrter for Fnnsh and Japanese speech was developed by Kohonen n 1988, [28]. The sgnal from the mcrophone proceeds to acoustc preprocessng, shown n more detal n Fg. 13, formng 15-component pattern vector (values n 15 frequency beans taken every 10 ms), contanng a short tme spectral descrpton of speech. These vectors are presented to a SOM wth the hexagonal lattce of the sze 8 x 12. Fgure 10. 2D to 2D mappng by a SOM (orderng process), [3] 3. Torus SOMs In conventonal SOM, the sze of neghborhood set s not always constant because the map has ts edges. Ths problem can be mtgated by use of torus SOM that has no edges, [25]. However torus SOM, Fg. 11, s not easy to vsualze as there are now mssng edges. Fgure 13. Acoustc preprocessng After tranng resultng phonotopc map s shown n Fg. 14, [7]. Durng speech recognton new pattern vectors are assgned category belongng to a closest prototype n the map. Fgure 11. Torus SOM 4. Herarchcal SOMs After prevous topologes, herarchcal SOMs should also be mentoned. Herarchcal neural networks are composed of multple loosely connected neural networks that form an acyclc graph. The outputs of the lower level SOMs can be used as the nput for the hgher level SOM, Fg. 12, [10]. Such nput can be formed of several vectors from Best Matchng Unts (BMUs) of many SOMs. Fgure 14. Phonotopc map, [7] B. Text Clusterng Text clusterng s the technology of processng a large number of texts that gves ther partton. Preparaton of text for SOM analyss s shown n Fg. 15, [29], and Fgure 12. Herarchcal SOM, [10] Fgure 15. Preparaton of text for SOM analyss, accordng to [29] MIPRO 2017/CTS 1255

5 F. Robotcs Some applcatons of SOMs are control of robot arm, learnng the moton map and solvng travelng salesman problem (mult-goal path plannng problem), Fg. 20, [32]. Fgure 16. Framework for text clusterng, [29] complete framework n Fg. 16, [29]. Massve document collectons can be organzed usng a SOM. It can be optmzed to map large document collectons whle preservng much of the classfcaton accuracy Clusterng of scentfc artcles s llustrated n Fg. 17, [30]. Fgure 20. Travelng Salesman Problem, [32] G. Classfcaton of Satellte Images SOMs can be used for nterpretng satellte magery lke land cover classfcaton. Dust sources can also be spotted n mages usng the SOM as shown n Fg. 21, [33]. Fgure 17. Clusterng of scentfc artcles, [30] C. Applcaton n Chemstry SOMs have found applcatons n chemstry. Illustraton of the output layer of the SOM model usng a hexagonal grd for the combnatoral desgn of cannabnod compounds s shown n Fg. 18, [11]. Fgure 21. Detectng dust sources usng SOMs, [33] H. Psycholngustc Studes One example s the categorzaton of words by ther local context n three word sentences of the type subjectpredcate-object or subject-predcate-predcatve that were constructed artfcally. The words become clustered by SOM accordng to ther lngustc roles n an orderly fashon, Fg. 22, [18]. Fgure 18. Applcaton of SOM n chemstry, [11] D. Medcal Imagng and Analyss Recognton of dseases from medcal mages (ECG, CAT scans, ultrasonc scans, etc.) can be performed by SOMs, [21].Ths ncludes mage segmentaton, Fg. 19, [31], to dscover regon of nterest and help dagnostcs. Fgure 22. SOM n psycholngustc studes, [18] I. Explorng Musc Collectons Smlarty of musc recordngs may be determned by analyzng the lyrcs, nstrumentaton, melody, rhythm, artsts, or emotons they nvoke, Fg. 23, [34]. Fgure 19. Segmentaton of hp mage usng SOM, [31] E. Martme Applcatons SOMs have been wdely for martme applcatons, [22]. One example s analyss of passve sonar recordngs. Also SOMs have been used for plannng shp trajectores Fgure 23. Explorng musc collectons, [34] MIPRO 2017/CTS

6 J. Busness Applcatons Customer segmentaton of the nternatonal tourst market s llustrated n Fg. 24, [35]. Another example s classfyng world poverty (welfare map), [36]. Orderng of tems wth the respect to 39 features descrbng varous qualty-of-lfe factors, such as state of health, nutrton, and educatonal servces s shown n Fg. 25. Countres wth smlar qualty of lfe factors clustered together on a map. Fgure 24. Customer segmentaton of the nternatonal tourst market, [35] Fgure 25. Poverty map based on 39 ndcators from World Bank statstcs (1992), [36] VII. CONCLUSION Self-organzng maps (SOMs) are neural network archtecture nspred by the bologcal structure of human and anmal brans. They become one of the most popular neural network archtecture. SOMs learn wthout external teacher,.e. employ unsupervsed learnng. Topologcally SOMs most often use a two-dmensonal grd, although one-dmensonal, hgher-dmensonal and rregular grds are also possble. SOM maps hgher dmensonal nput onto the lower dmensonal grd whle preservng topologcal orderng present n the nput space. Durng compettve learnng SOM uses lateral nteractons among the neurons to form a semantc map where smlar patterns are mapped closer together than dssmlar ones. They can be used for broad type of applcatons lke vsualzatons, generaton of feature maps, pattern recognton and classfcaton. Humans can t vsualze hgh-dmensonal data, hence SOMs by mappng such data to a two-dmensonal grd are wdely used for data vsualzaton. SOMs are also sutable for generaton of feature maps. Because they can detect clusters of smlar patterns wthout supervson SOMs are a powerful tool for dentfcaton and classfcaton of spato-temporal patterns. SOMs can be used as an analytcal tool, but also n a myrad of real world applcatons ncludng scence, medcne, satellte magng and ndustry. REFERENCES [1] T. Kohonen, Self-organzed formaton of topologcally correct feature maps, Bol. Cybern. 43, pp , 1982 [2] T. Kohonen, Self-Organzng Maps, 2nd ed., Sprnger 1997 [3] S. Haykn, Neural Networks: A Comprehensve Foundaton, 2nd ed., Prentce Hall PTR Upper Saddle Rver, NJ, USA, 1998 [4] K. Gurney, An ntroducton to neural network, UCL Press Lmted, London, UK, 1997 [5] D. Krese, A Bref Introducton to Neural Networks, [6] R. Rojas: Neural Networks, A Systematc Introducton, Sprnger- Verlag, Berln, 1996 [7] J. A. Bullnara, Introducton to Neural Networks - Course Materal and Useful Lnks, [8] C. M. Bshop, Neural Networks for Pattern Recognton, Clarendon Press, Oxford, 1997 [9] R. Eckmller, C. Malsburg, Neural Computers, NATO ASI Seres, Computer and Systems Scences, 1988 [10] P. Hodju and J. Halme, Neural Networks Informaton Homepage, [11] Kátha Mara Honóro and A. B. F. da Slva, Applcatons of artfcal neural networks n chemcal problems, n Artfcal Neural Networks - Archtectures and Applcatons, InTech, 2013 [12] W. Banzhafl, Self-organzng systems, n Encyclopeda of Complexty and Systems Scence, 2009, Sprnger, Hedelberg, [13] A. M. Turng, The chemcal bass of morphogeness, Phlosophcal Transactons of the Royal Socety of London. Seres B, Bologcal Scences, Vol. 237, No.641. pp , Aug. 14, 1952 [14] W. R. Ashby, Prncples of the self-organzng system, E:CO Specal Double Issue Vol. 6, No. 1-2, pp , 2004 [15] C. Fuchs, Self-organzng system, n Encyclopeda of Governance, Vol. 2, SAGE Publcatons, 2006, pp [16] J. Howard, Self-organsaton n bology, n Research Perspectves of the Max Planck Socety, 2010, pp [17] The Wzard of Ads Bran Map - Werncke and Broca, [18] T. Kohonen, MATLAB Implementatons and Applcatons of the Self-Organzng Map, Ungrafa, Helsnk, Fnland, 2014 [19] Bll Wlson, Self-organsaton Notes, 2010, [20] J. Boedecker, Self-Organzng Map (SOM),.ppt, Machne Learnng, Summer 2015, Machne Learnng Lab, Unv. of Freburg [21] L. Grajcarova, J. Mares, P. Dvorak and A. Prochazka, Bomedcal mage analyss usng self-organzng maps, Matlab Conference 2012 [22] V. J. A. S. Lobo, Applcaton of Self-Organzng Maps to the Martme Envronment, Proc. IF&GIS 2009, 20 May 2009, St. Petersburg, Russa, pp [23] A. A. Aknduko and E. M. Mrkes, Intalzaton of self-organzng maps: prncpal components versus random ntalzaton. A case study, Informaton Scences,Vol. 364, Is. C, pp , Oct [24] MathWorks, Self-Organzng Maps, [25] M. Ito, T. Myosh, and H. Masuyama, The characterstcs of the torus self organzng map, Proc. 6th Int. Conf. on Soft Computng (IIZUKA 2000), Izuka, Fukuoka, Japan, Oct. 1-4, 2000, pp [26] M. Johnsson ed., Applcatons of Self-Organzng Maps, InTech, November 21, 2012 [27] J. I. Mwasag (ed.), Self Organzng Maps - Applcatons and Novel Algorthm Desgn, InTech, 2011 [28] T. Kohonen, The neural phonetc typewrter, IEEE Computer 21(3), pp , 1988 [29] Yuan-Chao Lu, Mng Lu and Xao-Long Wang, Applcaton of self-organzng maps n text clusterng: a revew, n Self Organzng Maps - Applcatons and Novel Algorthm Desgn, InTech, 2012 [30] K. W. Boyacka et. all., Supplementary nformaton on data and methods for Clusterng more than two mllon bomedcal publcatons: comparng the accuraces of nne text-based smlarty approaches, PLoS ONE 6(3): e18029, 2011 [31] A. Aslantas, D. Emre and M. Çakroğlu, Comparson of segmentaton algorthms for detecton of hotspots n bone scntgraphy mages and effects on CAD systems, Bomedcal Research, 28 (2), pp , 2017 [32] J. Fagl, Mult-goal path plannng for cooperatve sensng, PhD Thess, Czech Techncal Unversty of Prague, February 2010 [33] D. Lary, Machne Learnng for Scentfc Applcatons, sldes, [34] E. Pampalk, S. Dxon and G. Wdmer, Explorng musc collectons by browsng dfferent vews, Computer Muscal Journal, Vol. 28, No. 2, pp , Summer 2004 [35] J. Z. Bloom, Market segmentaton - a neural network applcaton, Annals of Toursm Research, Vol. 32, No. 1, pp , 2005 [36] World Poverty Map, SOM research page, Unv. of Helsnk, MIPRO 2017/CTS 1257