AXON 2 A visual object recognition system for non-rigid objects

Transcription

1 AXON 2 A visual object ecognition system fo non-igid objects PABLO ALVARADO, PEER DÖRFLER, JOCHEN WICKEL Depatment of echnical Compute Science RWH Aachen, Gemany alvaado ABSRAC his aticle descibes a viewe-centeed, featue-based object ecognition system capable to distinguish up to 200 diffeent objects. It has to cope with images of non-igid objects showing patial occlusion. heefoe, we suggest the exclusive use of colo and textue infomation. o accomplish this, we intoduce some novel featue types, which we have successfully tested on a vaiety of classification poblems. KEY WORDS object ecognition, compute vision, neual netwoks, colo constancy, featue extaction, steeable filtes 1. Intoduction he main task of patten ecognition is to find unambiguous identification labels fo some given data. his might be fo example the ecognition of phonemes in an audio steam o the identification of specific pattens in images. Visual ecognition deals with the poblem of finding labels fo eal-wold objects contained in one o moe images of a scene. An application tageted by ecognition systems is an intelligent electonic catalog [1]. hese can be useful fo example in logistics, when a manufactue needs to egiste objects emitted fom etailes. If these objects lack a poduct label, it is a tedious and time-consuming task to look up the object s poduct identification in a catalog manually. Ou system aims to alleviate this task by allowing the use to pesent the object to a camea system and let the object ecognize etieve a list of the most simila objects. Both systems pefom well on an set of about hunded diffeent objects, but they also show some dawbacks. SEEMORE uses intelligently chosen featues, but equies impovements in its classification methods. DyPERS uses local featues, with impede the discimination between vey simila objects, like the ones depicted in Fig. 5. Ou system aims to avoid these limitations by combining diffeent featues with obust and efficient classification algoithms. 1.2 System desciption he AXON 2 system follows a featue-based, viewe-centeed appoach. Usually, an impotant pat of the ecognition is the analysis of object shape. Most of the known shape featues, howeve, suffe fom a stong sensitivity to patial occlusion and defomation. Futhemoe, in lage object sets, thee ae fom pimitives which ae shaed by many objects, making it had to distinguish among them. Additionally ou main object set is constituted by non-igid objects. Due to this facts, AXON 2 elies exclusively on colo and textue. he poposed object ecognition system woks as depicted in Figue 1. It fist acquies an image fom a camea and feeds it to the colo constancy system. Afte that, diffeent kinds of featue vectos ae extacted (Section 2.). he next step consists of thei classification, using RBF neual netwoks (Section 3.), whose esults ae combined to yield the final ecognition esult. Image Acquisition Colo Constancy 1.1 Related wok Result Classification Featue Extaction Many object ecognition systems have been intoduced peviously. Fo a compehensive suvey of olde systems, see [2]. wo of the most inteesting ecent systems ae SEEMORE [3] and DyPERS [4]. SEEMORE is an anthopomophic featue-based, viewe-centeed system that uses a combination of colo and shape desciptions. Classification is pefomed using a neaest-neighbo appoach. DyPERS uses local featues which ae classified using socalled multidimensional eceptive field histogams. Use Inteface Object Recognition Kenel Figue 1. he achitectue of the AXON 2 system. he colo constancy and featue extaction algoithms ae descibed in Section 2., the classification algoithms in Section 3. he system has been tested in a vaiety of situations

2 > A and with diffeent kinds of test data. Expeimental esults with seveal data sets ae pesented in Section Featue Extaction he example application discussed hee equies featue vectos to be invaiant against tanslation and otation (similaity tansfomations on the image plane) and against changes in the illuminant. Futhemoe, the objects to be ecognized have vey simila, non-igid foms. Due to the fact that the use pesents the objects to the system by hand, patial occlusion also has to be expected. Fo these easons only colo and textue featues ae utilized. he following sections conside the extaction of featue vectos fo the whole image (global featues), but the methods could be applied to smalle egions as well (local featues). Figue 2 shows the acquisition and featue extaction achitectue. Chomaticity Histogam Colo Histogams Image Acquisition Colo Constancy Segmentation MR Enegy Featue MR OGD Featue Figue 2. Achitectue fo the featue extaction 2.1 Colo constancy MR OCOGD Featue Finlayson et. al. [5, 6] have shown that a diagonal tansfomation suffices fo colo constancy if thee conditions ae met: the camea sensos exhibit naow-band colo sensitivity, the lighting geomety is constant, and the acquied scene contains lambetian sufaces only. he use of chomaticity channels to attain invaiance against lighting geomety imposes the loss of all infomation contained in the intensity channel, which can be indispensable fo the ecognition. Humans can indeed ecognize objects ignoing all colo infomation, exclusively using the intensity images. In the emainde of this pape will denote a channel containing specific infomation of an image, e. g. colo. Let be a nom which fulfills It can be easily shown that the nomaliza-. tion of each colo channel!, "# and $%! of the RGB colo space with thei espective noms yields a canonical image which is invaiant against changes in the illuminant colo. In [6], the nom '& ()+* (1) is used. &, - epesents the mean value of all pixels in the channel,. We pefe the nom./& (!02143! - (2) whee 1 is a constant and 3! ( the standad deviation of all elements of. he final ange of canonical values using nom (1) will highly depend on the distibution of the oiginal channel values. In Eq. (2) the standad deviation ensues that most of the nomalized values will be kept in a fixed inteval. In ou expeiments 156* is used. his way, if we assume a nomal distibution, the channel values will be mapped into an element of the inteval [0,1] with a pobability of 7879;: <. 2.2 Object segmentation o simplify image segmentation, a black backgound and a black glove fo holding the object ae used. he eal backgound is mapped into an ideal black backgound applying a simple egion gowing algoithm. A smalle image containing the object alone is passed to the next stages of the featue extaction. 2.3 Chomaticity Histogams Since the intoduction of colo histogamming by Swain and Ballad [7], this pinciple has been applied and enhanced in many object and image ecognition systems (e.g. [3, 8]). he chomaticity histogam exclusively contains infomation about colo, ignoing the intensity. his fact is used in [6] to poduce canonical images invaiant against changes of the lighting geomety, which ae eflected exclusively in the intensity channel. he chomaticity channels! ae defined as = >@?AB?C and D >@?AE?C. he 2D chomaticity histogam used hee contains 16 bins fo each coodinate. o incease the obustness against noise, the 2D histogam is convolved with a Gaussian *=FG* low-pass filte kenel (HJIK)+L MIN)O,IK)+L8P QHJIK)+L MIN)O,IK)+L8P ). Consideing that chomaticity values cove only one half of the histogam bins, the esulting vecto equies no moe than 136 dimensions. Scale invaiance is achieved by nomalizing the volume unde the histogam. All ideally black pixels ae ignoed; this way only the object pixels ae consideed. 2.4 Colo Histogams he chomaticity featue discads valuable infomation fo the ecognition. Consideing the pevious geneation of a canonical image invaiant against changes in the illuminant colo, the intensity infomation can also be obustly evaluated using simple colo histogams, geneated in the following way. Fou 1D histogams with 32 bins each ae calculated fo the ed, geen, blue and luminance channels, whee luminance is defined as RS

3 Ÿ V.UJVW >YXAXC!Z[?.\^]_W >YXAXC!Z. Again, to educe the effects of noise, a a IbFc* low-pass filte kenel (H IN)L,IK)+O4,IN)L P ) is applied to each histogam. he fou histogams ae concatenated into a 128-dimensional vecto. 2.5 Multiesolutional Enegy Featue Using the concepts fo textue channels pesented in [9], a textue-enegy image is geneated. Smith s suggestions fo scale and otation invaiance ae also consideed. A channel is fist tansfomed into its wavelet epesentation with fou esolutions, using a 9-tap symmetic kenel de#fh g9 g O h IN _i.g9 g j g87 i.g49 g :*8L +g49 L IL :N ^g49 :^7@:L4 g9 g :* L4 4i.g49 g8j g 74 g49 g8o h@i^p. he low-pass channel is ignoed, which means that only the thee following bands ae consideed (nine channels in total). he enegy channels ae calculated applying k Kk to each pixel of each spectal channel. hey ae then combined into a single image using the multiscaled addition defined in [10]. hese enegy images ae calculated fo the thee colo channels, " and $. Histogams with 32 bins fo each channel ae concatenated into a 96-dimensional featue vecto. he same noise-eduction measues as fo the colo featues ae applied. Anothe featue appoximates the multiesolutional enegy calculating the enegy of the pixels of a high-pass filteed image. he filte subtacts a low-pass filteed channel fom the oiginal. he used low-pass kenel is a O lmfno l Gaussian with vaiance 31. A 96-dimensional featue vecto is geneated following the same pinciples as fo the multiesolutional enegy featue. 2.6 Multiesolutional OGD Featue he infomation contained in each specific spectal band of a multiesolutional decomposition of an image has been successfully applied (fo example [11, 12]). hese featues, howeve, have a limited applicability in 3D object ecognition due to thei lack of invaiance against scaling and otation. Rotation invaiant and scaling equivaiant featues can be geneated exploiting the steeability popety [13] of oiented Gaussian deivatives (OGD). Let (o_ be a spectal component of a channel p extacted using a steeable filte o, which can be expessed as: o q sutwv s x8 s o (3) whee s x_ denotes the intepolation functions and o s the basis filtes. Let yzo! be the powe of the channel (o! defined as y o q o U Xsutwv sutv U Xs x_ y o^{; } s x8 os with U Xs x8b~ U x_ s x8, yzo {[ } b~(ou (os! and (os = p n o s whee denotes the convolution opeato. Eq. (4) indicates that the powe is also steeable using intepolation functions U X s x8 and basis powe channels y ou Xs. Let mo > be the enegy fo a egion ƒ Fˆ : > o W ] XŠZ( N> y o Œ U Xsutwv (4) U Xs x8 U Xs (5) with U Xs W ] XŠZ( N> yzo^{; }. he total enegy mo > is theefoe also steeable. With U Xs Ž s XU, it follows: > o U twv U XU x8 U XU 0O v U s U Xs x8 U Xs (6) With Eq. (6) and the steeability of the Gaussian deivatives, it can also be poven that the total enegy of a egion in os fo a -th ode Gaussian deivative filte is given by: > o 6 Y m0 U twv UN ;ON ^xši5 U (7) whee the U do not depend on the steeing angle x, i. e. the desciptos U ae otation invaiant. able 1 shows the values of these tems fo the fist and second oiented gaussian deivatives. he oiginal channel can be analyzed applying this method at diffeent esolutions using fo example dyadic pyamid achitectues. his way, diffeent fequency bands can be consideed in the final featue vecto. We employ a dyadic pyamid of OGDs with standad deviation 3 œ3,o fo each level used. Let ž;mo denote the total enegy of a channel, computed by Eq. (7) using a - and th ode OGD steeed towads x with a vaiance 3 a scale Ÿ O_ v. It can be easily poven that W V v Z o ž[mo?, i. e. the space-scaling of an image by Ÿ shifts the enegy in the esolution axis by = ŸK and W V v Z inceases the enegy level by Ÿ. Fo the constucted featue vecto a 2nd ode OGD dyadic pyamid with fou levels ( GL ) is geneated. Fo this task, a I* FªI* OGD filte kenel with vaiance 3 O is employed. In ode to enhance the esolution axis, a second pyamid with 3z«O3 is also ceated. he coefficients U in able 1 ae calculated fo each level of both pyamids. he final vecto is composed of all desciptos

4 ± ½ Ì Ê º Å Ä Ì Ä V able 1. Coefficients fo the enegy with 1st and 2nd OGD ± ³ ± à ½m Æ-Ç È+Æ-É ±³ 1st OGD # q ²±³! µ±b,,¹º(¼n ¾½m - ćä!å Å, 2nd OGD ²±m³ ±# ¹,º-¼n 5½m 4 ± E 4ćÄ + ±# Å 5º, E ½m Æ-Ç È+Æ É Å Æ Ç È+Æ-É Äzº Å Å + Å,¹º Ä,¼n 5½ ¾ Å U soted fist by 3 and then by the index. heefoe, the geneated vecto has 24 dimensions. his pocedue can conside colo infomation if it is applied to each of the, " and $ channels. he final featue vecto has 72 dimensions. 2.7 Opponent Colo Multiesolutional Featue Until now, the colo infomation has been consideed using the, " and $ colo channels sepaately. Anothe appoach to colo analysis is inspied by the opponent colo theoy of human vision. Hee, eceptive fields oganized in a cente-suound fashion compae the stimulus of the cente (e. g. ed) with the stimulus in the suounding (e. g. geen). his can be modeled using the scheme shown in Fig. 3. Colo Image Colo Space cente suound OGD 2 OGD 2 + Figue 3. Geneation of an opponent colo channel opponent colo channel If the enegy o \K Í fo the opponent colo channel is calculated as descibed by Eq. (6), it follows: \K 4Í o 6 \ o 02 Ío ig \Î Í o (8) Compaing this tem with the pevious featues, it is clea that the new infomation gained compaing both cente and suound channels is contained in the opponent colo en- \Î Í. Fo the OGD, this tem can be also expessed egy o as \Î%Í o Ž 0 U twv U + 8 [ON ^xši¾ U (9) Fo the 1st ode OGD, the new desciptos ae: Ž Ï 02 e \+Í ^Ï v Ï ig e Ï \+Í ^Ï v 'Ð8Ñ Ò Xe ^Ï Ð Ó Ï ig e Ï with ^Ï e ^Ï \+Í Ï Xe ^Ï W ] XŠZ N> W ] XŠZ N> W ] XŠZ N> ^Ï \ ^Ï Í e \ ^Ï! e Í Ï ^Ï \ e Í ^Ï!!0 0 Ï Xe Ï ^Ï Í e \ ^Ï Combining the desciptos and v of both input channels (cente and suound " ) and the new opponent colo channel, a featue vecto with 48 dimensions is geneated. 3. Classification he classification in AXON 2 follows a hieachical appoach as shown in Figue 4. In the fist stage, Radial Basis Function (RBF) neual netwoks can be found, one fo each of the featue types descibed in Section 2. he position of the basis functions is detemined by the OLVQ3 taining algoithm [14]. he esult of the output laye is computed using a sigmoid activation function. In the second stage, a combination module geneates a final classification esult fom the answes of the individual netwoks by linea combination [15]. 4. Applications and Results he applicability of object ecognition systems as inteface tools fo accessing electonic catalogues obviously depends on thei ecognition coectness. Howeve, fo inteactive applications it is not geneally necessay to pefom an unique identification. As outlined in the intoduction, we ae pimaily concened with finding simila objects. heefoe, it is sufficient if the system pesents the most simila objects at the top of the esult list. Fo that pupose,

5 Featue1 Featue2 Featue3 RBF netwoks Combine Figue 4. Achitectue fo the classification we use the concept of -best ecognition, which descibes the pobability of an object to be at position o bette in the esult list. AXON 2 has been tested on thee diffeent object sets. he fist one is the COIL database [16], consisting of 100 objects, each one epesented by 72 views. 36 of these views wee used fo taining, the emaining ones wee used fo testing. he second set is composed of views of 42 plush animals. aining was pefomed using 35 andom views fo each object, the test set consisted of 840 andom views, 20 fo each object; both sets wee disjunct. he thid set contains views of 202 plush animals. 40 views, taken fom camea points unifomly distibuted on an uppe hemisphee above the object, seved as taining set. he test set consisted of 36 andom views fo each object. his set contains objects which appea almost identical to each othe (Figue 5). AXON 2 is able to disciminate them, even tough they diffe in shape only. his demonstates that the textue featues intoduced hee indiectly incopoate shape infomation. he whole pocess fom image acquisition to classification esult takes no moe than two seconds on a standad Pentium II PC (350 MHz). Fo the object set of 202 objects, taining times ange fom 20 minutes to two hous pe RBF netwok, depending on the dimensionality of the featue vectos. he esults pesented hee wee obtained using the following configuation: Ô Featues: Chomaticity Histogam, Colo Histogam, Multiesolutional Enegy, Multiesolutional OGD, Multiesolutional OC OGD, and high pass filte vecto, as descibed in Section 2. Ô Classifie: six RBF netwoks, each with six units pe class, pototypes estimated by OLVQ3. Ô Combination by aveaging the classifie esults. able 2 shows the -best ecognition ates fo these thee sets. he COIL database is an easily solvable poblem. he 42 plush objects ae hade to ecognize due to Figue 5. wo objects fom the test set. he two images on uppe side ae scanned fom the manufactue s catalog. he two images on the bottom side show one actual test image fo each object as taken fom AXON 2 s camea. Both test images ae classified coectly. able 2. Recognition ates fo thee diffeent test object sets. Object Set Views 1-Best 2-Best 3-Best COIL % 42 plush obj % 99.9% 100% 202 plush obj % 93.8% 95.8% stong similaities among some of them. he esult with 202 objects suggests that viewe-centeed, featue-based object ecognition is a pomising appoach fo object sets containing seveal hunded objects, even though thee is oom fo impovements. able 3 shows the ecognition esults fo 202 objects, using the best combinations of featues. It can be seen, that always the best esults ae obtained by using both colo and textue featues. he most poweful single featue type is the Colo Histogam, despite its simplicity. he high pass filte featue impoves the esults only maginally. 5. Conclusions and futue wok We have pesented an object ecognition system which is able to yield good ecognition esults fo object sets up to 200 objects despite the pesence of patial occlusion. We have demonstated that the combination of appopiately designed featues with neual classifies can cope with vey difficult ecognition poblems. Even though the pesented appoach explicitly excludes shape featues, the system is capable of identifying objects which diffe almost only in

6 able 3. Best ecognition ates fo diffeent featue combinations. # Featues Ch. Hist. Col. Hist. MR OC OGD MR OGD MR Enegy high pass 1-Best 2-Best 3-Best 1 Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ Õ shape. he successo of the pesented system will ovecome some of its limitations. hese ae on the one hand, its estiction to scenes showing only one object, on the othe hand, its inability to handle object sets containing moe than seveal hunded objects. he expeience fom woking with the system pesented hee makes us confident that the new impoved system will be able to disciminate object sets of seveal thousand objects. 6. Acknowledgements his poject has been suppoted by the Heinz-Nixdof Foundation. We would like to thank Dik Kumbiegel, Ingo Elsen, and Pete Walte, who wee ou co-wokes on this poject and who deseve a geat pat of the cedit fo the poject s success. We also would like to thank ou students who helped implementing the system. Refeences [1] I. Elsen, K.-F. Kaiss, D. Kumbiegel, P. Walte, and J. Wickel. Visual Infomation Retieval fo 3D Poduct Identification. Künstliche Intelligenz, Jan. 1999, 1/99: [2] A. R. Pope. Model-based object ecognition a suvey of ecent eseach. echnical epot 94 04, Univesity of Bitish Columbia, [3] B. W. Mel. SEEMORE: Combining Colo, Shape, and extue Histogamming in a Neually Inspied Appoach to Visual Object Recognition. Neual Computation, May 1997, 9(4): the 4th Intenational Confeence on Compute Vision. IEEE Compute Society, 1993, [6] G. D. Finlayson, B. Schiele, and J. L. Cowley. Compehensive colou image nomalization. In ECCV, 1998, [7] M. Swain and D. H. Ballad. Colo indexing. Intenational Jounal of Compute Vision, 1991, 7(1): [8] M. S. Dew, J. W. Wei, and Z. Li. Illuminationinvaiant colo object ecognition via compessed chomaticity histogams of nomalized images. echnical Repot CMP-R 97-09, Simon Fase Univesity School of Compute Science, [9] J. R. Smith. Integated Spatial and Featue Image Systems: Retieval, Analysis and Compession. PhD thesis, Columbia Univesity, [10] L. Itti, C. Koch, and E. Niebu. A model of saliencybased visual attention fo apid scene analysis. IEEE ansactions on Patten Analysis and Machine Intelligence, Novembe 1998, 20(11): [11] A. Jain and G. Healey. A multiscale epesentation including opponent colo featues fo textue ecognition. IEEE ansaction on Image Pocessing, Janua 1998, 7(1): [12] B. S. Manjunath and W. Y. Ma. extue featues fo bowsing and etieval of image data. IEEE ansactions on Patten Analysis and Machine Intelligence, August 1996, 18(8): [13] W.. Feeman and E. H. Adelson. he design and use of steeable filtes. IEEE ansactions on Patten Analysis and Machine Intelligence, Septembe 1991, 13(9): [14]. Kohonen. Self-Oganizing Maps. Spinge-Velag Belin Heidelbeg New Yok, [15] K. ume and J. Ghosh. heoetical Foundations of Linea and Ode Statistics Combines fo Neual Patten Classifies. echnical Repot R , Cente fo Imaging Science, John Hopkins Univesity, Baltimoe, USA, [16] S. A. Nene, S. K. Naya, and H. Muase. Columbia Object Image Libay (COIL-100). echnical Repot CUCS , Columbia Univesity, [4] B. Schiele and A. Pentland. Pobabilistic object ecognition and localization. In Poc. Int. Conf. on Compute Vision ICCV 99, 1999, [5] G. D. Finlayson, M. Dew, and B. Funt. Diagonal tansfoms suffice fo colo constancy. In Poc. of