Multilayer Neural Networks and Nearest Neighbor Classifier Performances for Image Annotation

Transcription

1 (IJACSA) Internatonal Journal of Advanced Comuter Scence and Alcatons, Vol. 3, No. 11, 01 Multlayer Neural Networs and Nearest Neghbor Classfer erformances for Image Annotaton Mustaha OUJAOURA Laboratory of Informaton rocessng and Telecommuncatons, Comuter Scence Deartment, Faculty of Scence and Technology, Sultan Moulay Slmane Unversty, O Box. 53, Bén Mellal, Morocco. Brahm MINAOUI Laboratory of Informaton rocessng and Telecommuncatons, Comuter Scence Deartment, Faculty of Scence and Technology, Sultan Moulay Slmane Unversty, O Box. 53, Bén Mellal, Morocco. Mohammed FAKIR Laboratory of Informaton rocessng and Telecommuncatons, Comuter Scence Deartment, Faculty of Scence and Technology, Sultan Moulay Slmane Unversty, O Box. 53, Bén Mellal, Morocco. Abstract The exlosve growth of mage data leads to the research and develoment of mage content searchng and ndexng systems. Image annotaton systems am at annotatng automatcally anmage wth some controlled eywords that can be used for ndexng and retreval of mages. Ths aer resents a comaratve evaluaton of the mage content annotaton system by usng the multlayer neural networs and the nearest neghbour classfer. The regon growng segmentaton s used to searate obects, the Hu moments, Legendre moments and Zerne moments whch are used n as feature descrtors for the mage content characterzaton and annotaton.the ETH-80 database mage s used n the exerments here. The best annotaton rate s acheved by usng Legendre moments as feature extracton method and the multlayer neural networ as a classfer. Keywords-Image annotaton; regon growng segmentaton; multlayer neural networ classfer; nearest neghbour classfer; Zerne moments; Legendre moments; Hu moments; ETH-80 database. I. INTRODUCTION As the onlne resources are stll a vtal resource n everyday lfe, develong automatc methods for managng large volumes of dgtal nformaton s ncreasngly mortant. Among these methods, automatc ndexng multmeda data reman an mortant challenge, esecally the mage annotaton [1]. Annotated mages lay a very mortant role n nformaton rocessng. They are useful for an mage retreval based on eywords and mage content []. Manual annotaton s not only borng but also not ractcal n many cases due to the abundance of nformaton. Most mages are, therefore, avalable wthout adeuate annotaton. Automatc mage content annotaton becomes a recent research nterest [3]. It attemts to exlore the vsual characterstcs of mages and assocate them wth mage contents and semantcs. Many classfers have been used for the mage classfcaton and annotaton wthout any erformance s evaluaton. In ths aer, we use the neural networ classfer for mage annotaton, and we comare ts erformance wth the nearest neghbour classfer. We use the same tye of moments as a method of features extracton for each classfer n order to have an obectve comarson between erformances of the consdered classfers. In such case, we used a system that tends to extract obects from each mage and fnd the annotaton terms that descrbe ts ndvdual content. The rest of the aer s organsed as follows. The Secton resents the adoted annotaton system whle the Secton 3 dscusses the rmordal tass of any annotaton and recognton system. They are the mage segmentaton and features extracton roblems n addton to a bref formulaton for Hu, Legendre and Zerne moments as features extracton method. The Secton 4 s reserved for the mage classfcaton and annotaton usng neural networ or the nearest neghbour classfer. Fnally, the Secton 5 resents the exermental annotaton results and the comarng erformance of each used classfer. II. ANNOTATION SYSTEM The automatc annotaton s technues of mage s content attemt to exlore vsual features of mages that descrbe an mage content and assocate them wth ts semantcs. It s an effectve technology to mrove the mage ndexng and searchng n the large volume of nformaton that s avalable n the meda. The algorthms and systems, used for mage annotaton, are commonly dvded nto those tass [4]: Segmentaton and Features extracton; Classfcaton and Annotaton. Inut Image Segmentaton Features Extracton Classfer AnnotatonResults Keywords& Features Database Fgure 1. The bloc dagram of the mage annotaton system a g e

2 (IJACSA) Internatonal Journal of Advanced Comuter Scence and Alcatons, Vol. 3, No. 11, 01 The annotaton system adoted n ths wor s shown n Fg.1. The system has a reference database that contans eywords and features descrtor of mages that are already annotated by exerts (manual offlne annotaton).ths database s used for modellng and tranng the classfer n order to choose the arorate eywords. To acheve ths goal, by usng regon growng segmentaton, the nut mage s frstly segmented nto regons that reresent obects n the mage. The feature vectors of each regon are secondly comuted and extracted from the mage. Those features are fnally feed nto the nut of the classfer, that can be the multlayer neural networs or the nearest neghbor classfer, n order to decde and choose the arorated eywords for annotaton tass. Therefore, the mage content s annotated and the erformance of each classfer can be evaluated. III. IMAGE SEGMENTATION AND FEATURES EXTRACTION The features vector, whch s extracted from the entre mage, loses local nformaton. So, t s necessary to segment an mage nto regons or obects of nterest and use of local characterstcs. The mage segmentaton s the rocess of arttonng a dgtal mage nto multle segments. It s very mortant n many alcatons for any mage rocessng, and t stll remans a challenge for scentsts and researchers. So far, the efforts and attemts are stll beng made to mrove the segmentaton technues. Wth the mrovement of comuter rocessng caabltes, there are several ossble segmentaton technues of an mage: threshold, regon growng, actve contours, level sets, etc... [5]. Among these methods, the regon growng s well suted because of ts smlcty and has been successfully used several tmes as a segmentaton technue of dgtal mages. The regons are teratvely grown by comarng all unallocated neghbourng xels to the regons. The dfference between a xel's ntensty and the xel's mean n one regon s used as a measure of smlarty, t s a redcate that control the evoluton of segmentaton rocess. The xel, wth the smallest dfference measured ths way, s allocated to the resectve regon. Ths rocess contnues untl all xels are allocated to a regon. Whle all the xels n mage are not vsted; 1. Choose an unlabeled xel ;. Set the regon s mean to ntensty of xel ; 3. Consder unlabeled neghborng xels ; If (xel's ntensty - regon's mean) < threshold; a. Affect the xel to the regon labeled by. b. Udate the regon s mean and go bac to ; Else = + 1 and go bac to. End If End Whle Fgure. Regon growng segmentaton algorthm. By assumng that the obects are localsed at the center of mages, the regon growng segmentaton s started from the corner of the mage to solate the obects n the center of the mage. The regon growng segmentaton algorthm used n ths aer s resented n Fg. and an examle of mage segmentaton s n Fg.3. Fgure 3. Examle of mage segmentaton result. After dvdng the orgnal mage nto several dstnct regons that corresond to obects n a scene, the feature vector can be extracted from each regon and can be consdered as a reresentaton of an obect n the entre mage. The feature extracton tas transforms carefully the rch content and large nut data of mages nto a reduced reresentaton set of features n order to decrease the rocessng tme. Not only t enhances the retreval and annotaton accuracy, but the annotaton seed as well, snce a large mage database can be organzed accordng to the classfcaton rule and, therefore, search can be erformed [6]. In the feature extracton method, the reresentaton of the mage content must be consdered n some stuatons such as: translaton, rotaton and change of scale. Ths s the reason that ustfes the use of moments for feature extracton method from the segmented mage. The use of moments for mage analyss and attern recognton was nsred by Hu [7] and Alt [10]. In ths aer, the moments used are: Hu moments. Legendre moments. Zerne moments. A. Hu moments For a dscrete mage of M x N xels wth ntensty functon f(x,, Hu [7] defned the followng seven moments that are nvarant to the change of scale, translaton and rotaton: a g e

3 (IJACSA) Internatonal Journal of Advanced Comuter Scence and Alcatons, Vol. 3, No. 11, Where, And, N1 M x x y y x0 y0 x0 y0 1 N1 M 1 x y N 1 M 1 x0 y0 N 1 M 1 x0 y0 N 1 M 1 x0 y0 N 1 M 1 x0 y0 x y B. Legendre moments Legendre moments were frst ntroduced by Teague [8]. They were used n several attern recognton alcatons [9]. The orthogonal roerty of Legendre olynomals mles no redundancy or overla of nformaton between the moments wth dfferent orders. Ths roerty enables the contrbuton of each moment to be unue and ndeendent of the nformaton n an mage [10]. The Legendre moments for a dscrete mage of M x N xels wth ntensty functon f(x, s the followng [13]: M 1 N1 L ( x ) ( y ) x0 y0 1 1 Where, x and y denote the M N normalzed xel coordnates n the range of [-1, +1], whch are gven by: x M 1 x M 1 y N 1 y N 1 xs the th -order Legendre olynomal defned by: x 0!!!! 1 x even And, the recurrent formula of the Legendre olynomals s: 1 x x x 1 x, 0 x 1 1 x 1 x In ths wor the recurrent formula s used for calculatng Legendre olynomals n order to ncrease the comutaton seed. C. Zerne moments Zerne moments are the mang of an mage onto a set of comlex Zerne olynomals. As these Zerne olynomals are orthogonal to each other, Zerne moments can reresent the roertes of an mage wth no redundancy or overla of nformaton between the moments [11]. Due to these characterstcs, Zerne moments have been utlzed as feature sets n many alcatons [1]. The dscrete form of the Zerne moments of an mage sze M N reresented by f(x, s exressed, n the unt ds ( x y ) 1, as follows [13]: M N Z R, rxy e xy f x, y Where x0 y a g e

4 (IJACSA) Internatonal Journal of Advanced Comuter Scence and Alcatons, Vol. 3, No. 11, 01 s 1 s s! r R, r s0 s! s! s! even,, 0 s the number of xels located n the unt crcle, and the transformed hase xy and the dstance rxy at the xel of coordnates (x, are [13]: r xy xy tan 1 y N 1 N 1 x M 1 M 1 M 1 y N 1 x M 1 N 1 Most of the tme taen for the comutaton of Zerne moments s due to the comutaton of radal olynomals. Therefore, researchers have roosed faster methods that reduce the factoral terms by utlzng the recurrence s relatons on the radal olynomals. In ths aer, we obtaned Zerne moments usng the drect method. IV. IMAGE S CLASSIFICATION AND ANNOTATION As the features are extracted, a sutable classfer must be chosen. Many classfers are used and each classfer s found sutable to classfy a artcular nd of feature vectors deendng on ther characterstcs. Some of these classfers are based on suervsed learnng whch reure an ntensve learnng and a tranng hase of the classfer arameters (arameters of Suort Vector Machnes [14], Boostng [15], arametrc generatve models [16], decson trees[17], and Neural Networ[18]). They are also nown as arametrc classfers. Other classfers base ther classfcaton decson drectly on the data, and reure no learnng and tranng of arameters. These methods are also nown as nonarametrc classfers. The most common nonarametrc classfer, based on dstance estmaton, s the Nearest Neghbour classfer. There are several tyes of the classfer. Based on ther ablty to detect comlex nonlnear relatonshs between deendent and ndeendent varables, the multlayer neural networs are used n ths aer to classfy and annotate mage content. Ther erformances are comared to those of the Nearest Neghbour classfer. A. Multlayer Neural Networ classfer A multlayer neural networ conssts of an nut layer ncludng a set of nut nodes, one or more hdden layers of nodes, and an outut layer of nodes [19]. Fg. 4 shows an examle of a three layer networ, used n ths aer, havng nut layer formed by m nodes, one hdden layer formed by 0 nodes, and outut layer formed by n nodes. Ths neural networ s traned to classfy nuts accordng to target classes. The tranng nut data are loaded from the reference database whle the target data should consst of a vector of all zero values excet for the element that reresents the arorate class. Fgure 4. The three layer neural networ. After random ntalsaton of bas and connecton weghts, the tranng rncle of the neural networ s based on a loo of stes. It starts by roagatng nuts from the nut layer to the outut layer. The error s calculated and bac roagated for each layer of the neural networ n order to udate the bas and connecton weghts. When the bas and connecton weghts are changed, the roagaton of the nuts s reeated untl havng the mnmum error between oututs and targets. Ths s the crteron sto of the neural networ tranng. The nuts Y are resented to the nut layer and roagated to the hdden layer usng the followng formula: Inut E w Inut Layer * b + m Y f m 1 Y w b Then from the hdden layer to the outut layer: Fnally, the oututs are: 0 Y f Yw b 1 O f Hdden Layer n 1 Y w b Where f s the actvaton functon (hyerbolc tangent sgmod) used n the tree layer neural networ, defned by: f ( x) 1 x 1 e At the outut layer, the error between the desred outut T and the actual outut O s calculated by: E 1 O T O w * b The calculated error s roagated to the hdden layer usng the followng formula: + 0 w Outut Layer * b + n Outut O a g e

5 (IJACSA) Internatonal Journal of Advanced Comuter Scence and Alcatons, Vol. 3, No. 11, 01 E n 1Y w. E Then the calculated bac roagaton of error from the hdden layer to the nut layer s: 0 1 E 1Y w. E The bas and the connectons weghts of the nut layer, the hdden layer and the outut layer are adusted by: wl Yl. El bl El 1 for l,, B. Nearest Neghbor classfer The nearest neghbour classfer s used to comare the feature vector of the nut mage and the feature vectors stored n the database. It s obtaned by fndng the dstance between the rototye mage and the database. The class s found by measurng the dstance between a feature vector of nut mage and feature vectors of mages n reference database. The Eucldean dstance measurement s used n ths aer, but other dstance measurement can be also used [14]. Let X 1, X,, X be the class features vectors n the database and X the feature vector of the uery mage. The feature vector wth the mnmum dstance s found to be the closest matchng vector. It s gven by: adoton of Hu moments, nne nut nodes when adotng Zerne moments and ten nut nodes when used Legendre moment as feature extracton method. However, the number of nut nodes for the neural networs can be changed and ncreased when usng Zerne and Legendre moments, as a feature extracton method, to ncrease the accuracy of the annotaton system. The features can be also combned together and feed to the Multlayer neural networ or the nearest neghbour classfer. V. EXERIMENTS AND RESULTS A. Exerments In our exerments, for each regon that reresent an obect from the nut mage, the number of nut features extracted usng Hu nvarants features extracton method s 7 (hu1, hu, hu3, hu4, hu5, hu6, hu7) whle the number of nut features extracted usng the order 4 of Zerne moments s 9 (Z00, Z11, Z0, Z, Z31, Z33, Z40, Z4, Z44) and the number of nut features extracted usng the order 3 of Legendre moments s 10 (L00, L01, L0, L, L10, L11, L1, L0, L1, L). These nuts are feed to the nut of the multlayer neural networ or the nearest neghbour classfer n order to do matchng wth the feature values n the reference database. Then, the arorate eywords are selected and used for annotaton of the nut mage. The fg. 5 shows some examles of mage obects from the ETH-80 mage database used n our exerments. The exerments are based on eght classes of obects (Ale, Car, Cow, Cu, Dog, Horse, ears, and Tomato). The number of rototyes er class s 5. d ( X, X ) mn x x 1,,..., C. Image Annotaton For the mage annotaton, low-level feature vectors are calculated teratvely for each regon n the mage, ether by usng Hu moments or Legendre moments or Zerne moments. These features vector are resented ether to the nearest neghbour classfer or to the Multlayer neural networ that was already traned. When features vector are resented to the nearest neghbour classfer to test matchng wth the feature values n reference database, the label or eyword of mage class wth the mnmum dstance s selected. When features vectors are feed to the nut layer of the Multlayer neural networ, where each of the nut neurons or nodes corresond to each element of the features vector, The outut neurons of the neural networ reresent the class labels of mages to be classfed and annotated. Then, each regon s annotated by the corresondng label that s found by the classfer. The nut layer of the neural networ has a varable number of nut nodes. It has seven nut nodes n the case of Fgure 5. Some examles of obects from the ETH-80 database. The accuracy of mage annotaton system s evaluated by the recson rate whch s the number of the correct results dvded by the number of all returned results. All the exerments are conducted usng the ETH-80 database contanng a set of 8 dfferent obect mages [0]. The roosed system has been mlemented and tested on a Core Duo ersonal comuter usng Matlab Software. B. Results For both of the used classfers (the neural networ classfer and Nearest Neghbour classfer), the annotaton a g e

6 (IJACSA) Internatonal Journal of Advanced Comuter Scence and Alcatons, Vol. 3, No. 11, 01 rates of Hu, Zerne and Legendre Moments, as the feature extracton method, are gven for each obect n Table 1. Ths s the man reason for the modest results obtaned by Zerne moments. TABLE I. OJECTS ANNOTATION RATE OF HU, ZERNIKE AND LEGENDRE MOMENTS USING NEURAL NETWORK AND NEAREST NEIGHBOUR CLASSIFIER Neural networclassfer Nearest neghbourclassfer Obect Hu Zerne Legendre Hu Zerne Legendre Ale 71.0% 78.40% 9.13% 69.1% 76.84% 90.93% Car 68.53% 74.37% 83.11% 67.% 7.87% 81.91% Cow 41.3% 50.00% 61.37% 39.8% 49.13% 60.10% Cu 63.7% 71.0% 81.0% 6.07% 70.37% 80.0% Dog 6.57% 7.% 80.41% 60.87% 70.93% 79.10% Horse 48.60% 56.40% 65.8% 46.80% 55.10% 63.80% ears 69.9% 78.40% 89.38% 66.80% 76.94% 87.85% Fgure 6. Comarson of general annotaton rates. The fg 6 gves the comarson of general annotaton rate for each extracton method and for each classfer. Tomato 67.49% 76.5% 86.40% 65.90% 75.10% 84.70% Average 61.53% 69.70% 79.91% 59.80% 68.41% 78.55% The general annotaton rates and error rates of the neural networ classfer or the Nearest Neghbour classfer based on Hu moments, Zerne moments and Legendre Moments as feature extracton method are gven n Table. The exermental results showed that the annotaton rate of the neural networ classfer based on Legendre moments, Zerne moments and Hu moments s hgher than the annotaton rate of the Nearest Neghbour classfer based on Legendre moments, Zerne moments and Hu moments. TABLE II. Annotaton and Error Rate of Hu, Zerne and Legendre Moments usng Neural Networ and Nearest Neghbour classfer Classfer Neural networ Nearest neghbour Features Extracton method Annotaton Rate Error Rate Hu moments 61.53% 38.47% Zerne moments 69.70%.% Legendre moments 79.91% 0.09% Hu moments 59.80% 40.0% Zerne moments 68.41% 31.59% Legendre moments 78.55% 1.45% (a) The best annotaton rate s acheved when we use the Legendre moments as the features extracton method and the multlayer neural networs as a classfer. However, for the comutaton tme, the Nearest Neghbour classfer based on Hu moments s the best. The annotaton rates of Zerne moments s lower than the annotaton rate of Legendre moments because Zerne moments are comuted only from xels of an mage nsde a unt crcle obtaned by the mang transformaton over a unt crcle. In one hand, those xels outsde a unt crcle are not consdered. On the other hand, only the absolute values of extracted comlex Zerne moments are feed nto the classfer. (b) Fgure 7. Examle of the annotaton results usng Zerne moment features and (a) the multlayer neural networ classfer (b) the Nearest Neghbour classfer. The Fg. 7 resents an examle of annotaton results obtaned usng the resented system and Zerne moment as a a g e

7 (IJACSA) Internatonal Journal of Advanced Comuter Scence and Alcatons, Vol. 3, No. 11, 01 feature extracton method. The Grahcal User Interface s llustrated n Fg. 8. Fgure 8. Grahcal User Interface (GUI). The results are also affected by the accuracy of the mage segmentaton method. In most cases, t s very dffcult to have an automatc deal segmentaton. Ths roblem decreases the annotaton rates. Therefore, any annotaton attemt must consder the mage segmentaton as an mortant ste, not only for automatc mage annotaton system, but also for the other systems whch reure ts use. The Multlayer neural networ classfer based on Legendre moments and Zerne moments s very exensve regardng to the comutaton tme of the features n addton to the tranng tme of the classfer. So, any use of them n a real tme for an onlne mage annotaton system wll be dffcult and mractcable. VI. CONCLUSION In ths aer, we evaluated the mage annotaton erformance of the neural networ classfer and the nearest neghbour classfer based on Hu moments, Legendre moments and Zerne moments as feature extracton methods. The erformance of each classfer and each feature extracton method has been exermentally analyzed. The successful exermental results roved that the roosed mage annotaton system based on the multlayer neural networ classfer gves the best results for some mages that are well and roerly segmented. However, the rocessng tme and the Image segmentaton reman a challenge that needs more attenton n order to ncrease recson and accuracy of the mage annotaton system. Also, the ga between the low-level features and the semantc content of an mage must be reduced and consdered for more accuracy of any mage annotaton system. Other Image segmentaton and other classfers may be consdered n the future wors for other mage databases. REFERENCES [1] N. Vasconcelos and M. Kunt, Content-Based Retreval from Image Databases: Current Solutons and Future Drectons, roc. Int l Conf. Image rocessng, 001. [] A. Smeulders, M. Worrng, S. Santn, A. Guta, and R. Jan, Content- Based Image Retreval: The End of the Early Years, IEEE Trans. attern. Analyss and Machne Intellgence, vol., no. 1, , Dec [3] Le Ye, hl Ogunbona and Janang Wang, Image Content Annotaton Based on Vsual Features, roceedngs of the Eghth IEEE Internatonal Symosum on Multmeda (ISM'06), IEEE comuter socety, San Dego, USA, December 006. [4] Ryszard S. Chora s, Image Feature Extracton Technues and Ther Alcatons for CBIR and Bometrcs Systems, Internatonal Journal Of Bology And Bomedcal Engneerng, Issue 1, Vol. 1,. 6-16, 007. [5] Fran Y. Shh, Shouxan Cheng, Automatc seeded regon growng for color mage segmentaton, Image and Vson Comutng 3, , 005. [6] Zun Yang and C.-C. Jay Kuo, Survey on Image Content Analyss, Indexng, and Retreval Technues and Status Reort of MEG-7, Tamang Journal of Scence and Engneerng, Vol., No. 3, , [7] M. K. Hu, Vsual roblem Recognton by Moment Invarant, IRE. Trans. Inform. Theory, Vol. IT-8, , Feb [8] M.R. Teague, Image analyss va the general theory of moments, J. Ot. Soc. Amer. 70,. 90 9, [9] Chee-Way Chonga,. Raveendranb and R. Muundan, Translaton and scale nvarants of Legendre moments, attern Recognton 37, , 004. [10] F. L. Alt, Dgtal attern Recognton by Moments, J. Assoc. Comutng Machnery, Vol. 9, , 196. [11] Sun-Kyoo Hwang, Who-Yul Km, Anovel aroach to the fast comutaton of Zerne moments, attern Recognton 39, , 006. [1] R. Snan Tumen, M. Emre Acer and T. Metn Sezgn, Feature Extracton and Classfer Combnaton for Image-based Setch Recognton, Eurograhcs Symosum on Setch-Based Interfaces and Modelng,. 1 8, 010. [13] Mustaha Ouaoura, Brahm Mnaou and Mohammed Far. Artcle: Image Annotaton usng Moments and Multlayer Neural Networs. IJCA Secal Issue on Software Engneerng, Databases and Exert Systems SEDEX(1):46-55, Setember 01. ublshed by Foundaton of Comuter Scence, New Yor, USA. [14] Oren Boman, El Shechtman and Mchal Iran, In Defense of Nearest- Neghbor Based Image Classfcaton, IEEE Conference on Comuter Vson and attern Recognton (CVR), June 008. [15] A. Oelt, M. Fussenegger, A. nz, and. Auer. Wea hyotheses and boostng for generc obect detecton and recognton, In ECCV, 004. [16] G. Wang, Y. Zhang, and L. Fe-Fe. Usng deendent regons for obect categorzaton n a generatve framewor. In CVR, 006. [17] A. Bosch, A. Zsserman, and X. Munoz. Image classfcaton usng random forests and ferns. In ICCV, 007. [18].Smard, D. Stenraus, J. C. latt, Best ractces for Convolutonal Neural Networs Aled to Vsual Document Analyss, ICDAR, 0, [19] S. Hayn, Neural networs: a comrehensve foundaton, nd Edton. rentce Hall, New Jersey [0] ETH-80 database mage. [Onlne]. Avalable: htt:// a g e