Internatonal Conference on Logstcs Engneerng, Management and Computer Scence (LEMCS 2014) Image Emotonal Semantc Retreval Based on ELM Pele Zhang, Mn Yao, Shenzhang La College of computer scence & Technology Zhejang Unversty Hangzhou 310027, P.R. Chna zhangpele/myao@zju.edu.cn Pele Zhang, College of computer scence & Technology Zhejang Unversty Hangzhou 310027, P.R. Chna Shenzhang La College of computer scence & Technology Zhejang Unversty Hangzhou 310027, P.R. Chna Junfe Zhuo The Computer Centre Zhejang Unversty Hangzhou 310027, P.R. Chna zhuojf@huaha.net Abstract Image emotonal semantc retreval s one of the mportant subjects n nformaton scence. Ths paper dscusses the bass of emotonal cogntve, the dmon of emotonal expresson and the establshment of scentfc emoton space, and proposes a knd of mproved extreme learnng machne for mage emoton semantc retreval. Fnally, a knd of prototype system for mage emoton semantc retreval has been developed. The expermental results show that prototype system for mage emoton semantc retreval s effectve. Keywords-mage; emoton; mage retreval; semantc retreval; ELM I. ITRODUCTIO Wth the advent of the multmeda age, the pcture as the man medum of nformaton transmsson has become a major tool for people to express emoton, conduct socal actvtes. However, wth the explosve growth of the mages, how to retreval mages effcently s an urgent problem to be solved. The tradtonal mage retreval methods are manly based on the underlyng features of the mages, whch search mages by text-based or content-based markers wthout consderng the semantc nformaton of the mages and havng bgger dfference n semantcs wth human understandng. In other words, there s a huge semantc gap (Semantc Gap) between the underlyng vsual characterstcs of the mages and human search for mages by means of mage semantcs [1-3], unable to meet the demand for people o retreve mages wth semantcs. In order to elmnate semantc gap n semantc mage retreval, semantc-based mage retreval (SBIR for short) has been proposed. Semantcs s the text abstract to mage by human, dfferent semantcs levels stands for dfferent complexty about mage content descrpton n mage retreval [4]. Emoton s the most dffcult level n semantc descrpton. Unlke the underlyng vsual features, mage emotonal semantcs s the closer semantc level wth human emoton recognton for mages. Ths paper forms emotonal space by adjectve pars, and bulds mappng between mage emotonal semantc and the underlyng characterstcs of the mages. At the same tme, a knd of mproved Extreme Learnng Machne s used to execute mage emotonal semantc classfcaton and retreval. Experment results shown ts better retreval accuracy and robustness. II. IMAGE EMOTIOAL SEMATIC RETRIEVAL Image emotonal semantc retreval manly covers followng three key aspects: 1) extracton and dmonalty reducton of mage vsual features; 2) emotonal semantc descrpton and ts relaton wth mage features; 3)classfcaton and retreval of machne learnng based Image emotonal semantc. Many prevous research has focused on the frst aspect, thus, ths thess wll manly analyze the other two. A. Emotonal Semantc Descrpton 1) Choce on Adjectve Space Studes have shown that human emotonal varaton ntty s contnuous [5]. Every sngle poston of emotonal space represents a specfc type of emoton, therefore, f drectly applyng the whole emotonal space to descrbe the emoton of mage, whch wll result n numerous types of emoton. Moreover, each of those emotons s smlar wth the others and accompaned by certan degree of redundancy. Thus, t wll fal to descrbe certan type of human emoton vsually. For ths reason, ths thess puts forward the dea of buldng an adjectvebased emoton space, by selectng orthogonal and representatve adjectves, to construct a coarse-graned space to express emotons. Based on the combnaton of Plutchc emotonal space and PAD dmonal model of emoton, ths thess selects 12 pars of emotonal adjectves wth representatve and orthogonalty. Under certan emotonal dmon, each par of adjectves becomes the bpolar par, and thus forms the correspondng emotonal space wth 12 dmons. Ths method can not only effectvely conde the emotonal complexty of contnuous emotonal space, but 2014. The authors - Publshed by Atlants Press 498
also descrbe human emotons n a more complete way. 12 pars of adjectves are shown n Table I: TABLE I. passonate-ndfferent warm-cold excted-quet orderly-dsorderly mld-pungent brght-dark ADJECTIVE EMOTIOAL SPACES comfortable-uncomfortable harmony-conflcted exquste-coarse serene-nosy clear-vague terrfed-calm 2) Establshment of Tranng Image Lbrary The core part of mage emotonal semantc retreval s the tranng of classfer, whch gves the classfer an ablty to recognze human emoton. The tranng result of classfer depends on the completeness of the tranng mage lbrary, the more accurate mappng between mage and emoton; the tranng result wll be more favorable. Ths paper selects 1288 mages of landscape, characters and buldngs, based on the adjectve emotonal space n table I, those mages are labeled wth dfferent emotonal tag though user emotonal labelng experment. In order to classfy emotons, we need to use some adverbs to extend emoton, makng t more rch and specfc. Quantfcaton of ths method can be descrbe as a decmal between 0 and 1, for example, for warm - cold, very warm, relatvely warm, not cold not warm, relatvely cold, very cold, these fve degrees can be denoted as 0, 0.25, 0.5, 0.75, 1respectvely. Ths method s used to label the emoton of Images. In the process of labelng mage by users, t s very hard to avod subjectvty because of users from dfferent areas may have dfferent understandngs of a same mage. So deferent area users are selected, each user marks the emoton of 100 mages n the mage lbrary; each mage labeled wth a decmal between 0 and 1, stores the label data, and fnally summarzes the experment data to obtan an average emotonal result to represent the emoton of mage. Then accordng to the average emotonal result to classfy mage emoton, for example, when there s an mage n the warm - cold emotonal space, ts average emotonal result s 0.87, we can consder ths mage belongs to lukewarm emoton; whle the average emotonal result s 0.12, t belongs to passonate emoton. By usng ths method we can get the fnal emotonal mage lbrary. B. ELM based Image Emotonal Semantc Classfcaton 1) Introducton of Extreme Learnng Machne Gven detects of sngle hdden layer feed-forward neural network algorthm, t s unable to meet the massve mage data applcaton envronment. Ths paper uses a relatvely new sngle hdden layer feed-forward neural network algorthm called extreme learnng machne[6] algorthm for mage features classfcaton and mage retreval. Steps of Extreme Learnng Machne algorthm: For a gven set of tranng sample for ELM nput n m layer, S {( x, t ) x R, t R }, 1,2,...,, where s the total number of tranng samples, ~ s the number of nodes of hdden layer, g(x) s the actvaton functon, x s the -th sample of nput layer, t s the correspondng output classfy vector of the -th nput sample. a) Randomly generated connecton weghts between the nput layer and the hdden layer w and the threshold value of hdden layer nodes b, where 1,2,... ; b) Calculate the hdden layer output matrx H accordng to the formula H T, where H: g( w1 x1 b1 ) g( w1 x1 b1 ) H, g( w ) ( ) 1 x1 b1 g w1 x1 b1, 1 2,..., T ~ c) Accordng to the formula H T, can be calculated as H 1 T. Unlke normal SLFs learnng algorthm, ELM algorthm does not need to terate W and b, whch respectvely s the connecton weghts between the nput layer and the hdden layer, and the threshold value of hdden layer nodes, to get the optmal soluton. When the algorthm starts, W and b are randomly generated [7], can be calculated from H T, then gets a set of parameters, whch has been proven to be affectve to classfcaton problem. ELM does not need to teratve, so that acheves a sgnfcantly mproved effcency. In practce, the number of tranng samples s far greater than the number of nodes n the hdden layer, H s rectangular matrx, there s not necessarly exsts a set of parameters (W,, b) such that (2.1) and (2.2) are equvalent: H( Wˆ, bˆ) ˆ T mn H( W, b) T (1) j1 ~ 1 w, b, E g( w x b ) t (2) For the formula (2.2), namely the cost functon to obtan the optmal soluton, the tradtonal methods must execute teratve approxmaton, whle ELM algorthm does not requre teratve, all ts need s just smply specfy the approprate W and b, least squares soluton ˆ can be obtaned as: H ˆ T mn H T (3) amely j ˆ 1 T 1 T H T ( H H) H H (4) Where A s a generalzed nverse matrx of H. Huang Guang Bn [8] proves as long as the actvaton functon g(x) nfntely dfferentable, then W and b needn t to update, you can fnd the least-squares soluton of ˆ. Due to the randomness of ELM parameters, dfferent performances of the dfferent parameters are manly reflected n the choce of exctaton functon and threshold. Many scholars have conducted n-depth research n ths area, Huang Guan bn proposed ELM and kernel ELM, whch are used to solve nput problem n complex space; The tranng samples of orgnal ELM are provded onetme, all the samples should nput nto the network at the j 2 499
same tme, so after the tranng, the ELM cannot be changed. Based on ths defcency, Lang [9] proposed Onlne Sequental ELM, n whch tranng samples can be nputted batch by batch. Ths algorthm effectvely compates the shortcomngs of the orgnal ELM that cannot add new tranng samples, makng the system more tme-stve. The above analyss shows that ELM can randomly set the parameters of the network and calculates the other parameter when used to solve classfcaton problem, n ths way can hghly mprove the tranng effcency. However because of the randomness of ELM parameters, the parameters have not been optmzed, whch may reduce the tranng ablty of ELM, n ths paper, we propose a modfed ELM to mprove the accuracy of the orgnal ELM. 2) The Modfcaton of ELM and Performance Analyss a) Modfcaton of Extreme based on Genetc Algorthm As mentoned above,the randomness of parameters of orgnal ELM may reduce the ablty of classfcaton, we propose a new algorthm called GA-ELM, whch combnes ELM wth genetc algorthm. Ths algorthm s used to classfy mage n ths paper. The man process of ths algorthm s as follow: Get 50 dffcult ELM parameters by usng orgnal ELM, use genetc algorthm to modfy these 50 parameters, choose 10 best ELM parameters and form a combned ELM. Genetc Algorthm (GA) [10], a heurstc search method smulatng bologcal evoluton, s wdely used to solve optmzaton and search problems. Genetc algorthm s a knd of evolutonary algorthm, whch manly smulates crossover, mutaton and selecton of chromosome n evoluton of bologcal communtes, and selects the better offsprng. The ntal value of GA s chromosomal sequence of communty, whch s bnary sequence codng from the soluton of practcal problem, ndvduals of ntal group are randomly selected; In the process of GA, ftness coeffcent s used to ndcate the degree of adapton to the envronment of ndvdual, f the ftness coeffcent s larger than standard value set before the algorthm started, we can stop the algorthm, and select best ndvdual from the populaton, whch can be decoded to get the soluton of practcal problem; otherwse, we contnue the algorthm untl the ftness coeffcent larger the standard value. Accordng to the ntal crossover and mutaton probablty, ndvdual s selected to operate chromosome changng. Crossover s a process that parental chromosomes compose to form a new chromosome, whch may be a better chromosome and makng the populaton evaluate toward the drecton of optmal soluton. Mutaton, a process that part of ndvdual chromosome has changed, mantans the bologcal dversty of the populaton. By crossover and mutaton, a new generator s produced, and then repeats the calculaton of ftness coeffcent as mentoned above, untl reach the tmes of evoluton, the algorthm wll stop. The genetc algorthm s appled to optmze the parameters of ELM, f ELM hdden layer nodes are ~, then the connecton weght of nput layer and hdden layer W and the hdden layer node threshold value b can be unted nto the ntal populaton of ndvduals as e,, ~,,,, ~, b, b, b ~, 11 1 n1 n 1 2, Wheren andb j s a random number between [-1,1].50 ELM parameters form an ntal populaton, and the calculaton of accommodaton coeffcent s based on the standard devaton of ndvdual chromosomes and tranng samples, as shown n (2.5) as follows: F j1 ~ 1 g( w x b j ) t Accordng to the accommodaton coeffcent to choose ndvdual, do crossover and mutaton probablty, the last generaton produces sutable alternatve ndvdual to form combned ELM. ELM parameter ndvdual, whch optmzed by usng genetc algorthm, more lager of ts accommodaton coeffcent shows lower tranng standard devaton of correspondng ELM, accordng to whch we wll be able to get better performance of ndvduals. However, for a partcular ELM, ts performance not only determned by ts accommodaton coeffcent, but also the weght of norm of correspondng ELM,.e. related to, smaller the weghts of norm, more superor performance of ELM. Through the above analyss, ths paper choose ndvdual accommodaton coeffcent and mnmum paradgms standard n the selecton of ndvduals, these two parameters are calculated normalzed respectvely, and calculate the sum, as (2.6) shows, sort the result n descendng order, and select top ten ELM to form the combned ELM. f C um max (6) um M M f j1 j1 Where um represents the number of selected ELM, n ths paper um equals 10, C represents the selected ELM, um j 2 2 (5) f represents the accommodaton coeffcent of the -th ELM, represents the weght of norm of -th ELM, where equals 1, 2,..., M, M s the number of ntal ELM, n ths paper M equals 50.By savng the parameters of the selected ten ELMs, a combned ELM can be generated to retreval mage. Orgnal ELM mage usng a sngle network to predct the emotonal category, snce a sngle ELM s performance has randomness and uncertanty, we propose usng a combned ELM to forecast, usng the max probablty output of the combned ELM as the fnal predcton. The predcton process for the combned ELM: For a test sample mage (x, t), usng the -th ELM of the combned ELM to predct, ts correspondng nput values are w and b, the predcton result s o ; Use ten traned ELM of the combned ELM to predct the emotonal classfcaton of testng mage, get ten predct result, summarze the ten 500
O result obtans, calculate the proportonal share of each result, such as (2.7) as follows: um 1 O o (7) um Select the max proportonal share emotonal classfcaton fromo, as seen n (2.8), we can get the classfcaton result of the testng mage. O fnal o max (8) Where o s the -th classfcaton of 1,2,...um. O, b) Performance analyss of modfed ELM To verfy the performance of the combned ELM, we compare t to the performance of orgnal ELM and SVM. The tranng set s some mages of satelltes and letters, ts related parameters as shown n table II, choose 1 g( x) Dataset e x as actvaton functon. 1 TABLE II. Tranng Capacty PARAMETERS OF TRAIIG DATA SET Testng Capacty umber of Hdden layer nodes Satellte 1500 410 100 Letter 12000 4000 200 TABLE III. Dataset Satellte Letter TRAIIG PERFORMACE COMPARISO OF MODIFIED ELM WITH ORIGIAL ELM AD SVM Algorthm Tranng tme(s) Tranng accuracy(%) Modfed ELM 786.42 87.31 Orgnal ELM 0.76 84.69 SVM 6268.36 79.68 Modfed ELM 1889.38 80.07 Orgnal ELM 2.16 78.65 SVM 48865.33 73.85 Through 20 tmes experment, we obtan such tranng result shown n Table III, whch s the average result of 20 experments. From the result, we can see that although the tranng tme of modfed ELM s much longer than orgnal ELM and SVM, due to modfed ELM needs multple tmes of tranng and to modfy the parameter by usng genetc algorthm, generally t needs much shorter tranng tme than SVM. In terms of tranng accuracy, modfed ELM s much hgher than orgnal ELM and SVM on both dataset, whch proved that the mproved ELM can mprove tranng accuracy rate. To verfy the testng performance of the combned ELM, we test the combned ELM based on the dataset shown n Table II, and compare t to the performance of orgnal ELM and SVM. Experments conducted 20 tmes, the average result s shown n Table IV. TABLE IV. Dataset Satellte Letter TESTIG PERFORMACE COMPARISO OF MODIFIED ELM WITH ORIGIAL ELM AD SVM Algorthm Testng accuracy (%) Testng mean square error (%) Modfed ELM 89.24 0.163 Orgnal ELM 86.69 0.186 SVM 83.28 0.218 Modfed ELM 82.96 0.189 Orgnal ELM 79.63 0.233 SVM 77.47 0.692 The above table shows that modfed ELM algorthm has advantages n testng accuracy when comparng wth orgnal ELM and SVM algorthm. Meanwhle, t has much lower mean square error of testng accuracy, whch shows that modfed ELM has a stable performance and sgnfcantly mprove the robustness. III. PROTOTYPE SYSTEM Based on the core algorthms descrbed n the prevous secton, we desgned and mplemented prototype system for mage emotonal semantc retreval based on the mproved ELM. On the prototype system, the user can through two ways, namely text nput retreval and mage nput retreval to obtan the requred emotonal mages. In text nput retreval mode, the user nput hope emoton semantc text, the system matches the mage lbrary created by tranng, fnds out the smlar mage wth user nput on emoton semantcs n the mage lbrary and, and returns the retreved results to the user. Its advantage s fast and accurate. Users can retreve the exact emotonal semantc mage from the system. In mage Input retreval, users upload to retreve mages through the user nterface module. The system extracts the feature vector from the nput mages, executes emoton recognton by traned ELM, maps t to a smlar emotonal semantc classfcaton, and returns the smlar emotonal semantc mages to users. Image nput approach has the advantages of more ntutve and convenent. Users do not enter clear semantc keywords, only by uploadng an mage, the system automatcally complete the emoton recognton and returns the retreval results, realzng ntellgent nteractons between the system and human. Therefore, mage nput retreval can overcome the semantc gap between mage features and the hgh-level emotonal bottom effectvely. Image emotonal semantc retreval system based on ELM s developed by modular development. The prototype system conssts of mage base constructng module, feature extractng module, mage emoton matchng module and user nterface module, the system block dagram s shown as fgure 1. 501
Fgure 1. System Block Dagram A. Image base constructng module The functon of the module ncludes orgnal mage preprocessng, the sze and format of the mage adjustng, mage emoton tag labelng by user annotatons experment, and the correspondng mage and emoton classfcaton matchng ndex creatng by statstcal. B. Feature extractng module The functon of the feature extractng module ncludes extractng mage feature, decreasng the dmon of the feature vectors by PCA, storng the fnal feature vectors n system mage base, and buldng ndex between mage feature vectors and the emoton classfcaton. Snce then, the tranng of the ELM and mage retreval s based on mage feature vectors, no longer on the mage tself so as to reduce the processng tme effectvely and mprove the effcency of retreval. C. Image matchng module Accordng to dfferent choce of emotonal nput mode, the mage emoton matchng process wll also be dfferent. When a user selects a text as emotonal nput, the system reads nput from the user's emotonal keywords, and emoton classfcaton matchng n mage lbrary. When the user selects a sample mage as emotonal nput, the system wll matches the emoton semantcs n sample mage wth mages n system lbrary. D. User nterface module The prototype system provdes users wth a frendly nteractve nterface,.e. two dfferent nput modes, namely the text nput approach and mage nput approach. When users nput mage emotonal text to be retreved, such as nput "happy", the system executes emoton semantc matchng and estmates whether there s the correspondng emoton classfcaton n the system. If successful, t returns the correspondng mages to the user. If no correspondng emoton classfcaton s found, users can upload a sample mage to search the smlar emotonal mages. Fgure 2 and fgure 3 are the results by text search and mage search respectvely. Fgure 2. Result by text search Fgure 3. Result by mage search IV. COCLUSIO In ths paper, mage emoton semantcs has been explored based on ELM, ncludng the bass of emotonal cogntve, the dmon of emotonal expresson, the establshment of scentfc emoton space, the selecton of representatve mages, feature extracton and dmon reducton, and the mprovement of extreme learnng machne. The expermental results show that prototype system for mage emoton semantc retreval acheved ntal success. The future task s to carry out further research on emoton expresson and semantc matchng so as to acheve the practcal level for mage emoton semantc retreval. ACKOWLEDGMET Ths paper s the partal achevement of Project 2013CB329504 supported by atonal Key Basc Research and Development Program (973 program), and project 2012C21002 supported by Scence Technology Department of Zhejang Provnce. REFERECES [1] Dora C, Venkatesh S. Brdgng the Semantc Gap wth Computatonal Meda Aesthetcs. IEEE Multmeda 2003; 10(2):15-17. [2] Aamr S M, Humara. Edge Refnement Method for Content- Based Image Retreval. IEEE, 1999: 921-924. 502
[3] Peter E, Chrstne S. Towards a comprehve survey of the semantc gap n Vsual mage retreval, Lecture notes n computer scence, 2003:163-1 68. [4] Wang W, Yu Y L, Jang S M. Image Retreval by Emotonal Semantcs: A Study of Emotonal Space and Feature Extracton. IEEE, 2006: 3534-3539. [5] Schlosberg, H. Three dmons of emoton. Psychologcal Revew, 1954, 61: 81-88. [6] HUAG G B, ZHU Q Y, SIEW C K. Extreme learnng machne: a new learnng scheme of feedforward neural networks [J]. eurocomputng, 2004, 2(2): 985-990. [7] G-B Huang, L. Chen, and C.-K.Sew, Unversal approxmaton usng ncremental feedforward networks wth arbtrary nput weghs, n Techncal Report ICIS/46/2003, (School of Electrcal and Electronc Engneerng, anyang Technologcal Unversty, Sngapore), Oct. 2003. [8] Huang Gang-bn. Learnng capablty and storage capacty of twohdden-layer feedforward networks[j]. IEEE Trans on eural etworks, 2003, 14(2):274-281. [9] Bartlett PL (1998), the sample complexty of pattern classfcaton wth neural networks: the sze of the weghts s more mportant than the sze of the network. IEEE Trans Info Theory44(2): 525 536. [10] Goldberg D E. Genetc Algorthms n Search, Optmzaton and Machne Learnng [M]. MA: Addson Wesley, 1989: 1-83. 503