Single vs. Population Cell Coding: Gaze Movement Control in Target Search

Transcription

1 Sngle vs. Populaton Cell Codng: Gaze ovement Control n Target Search Jun ao, Layun Qng, Ljuan Duan, and Baxan Zou Abstract Gaze movement plays an mportant role n human vsual search system. In lterature, the wnner-take-all method s wldly used to smulate the controllng of the gaze movement. The wnner-take-all s a type of sngle-cell codng method, whch uses one cell (grandmother cell) or one response to represent an object. However, eye movement s affected by the vsual context whch ncludes more than one object n mages, especally n target search. Therefore, we propose to use the populaton codng wth more than one response rather than the sngle-cell codng on gaze movement control. The proposed method s supported by the theoretcal analyss and experments on a real mage database whch show the populaton-cell-codng mproves the target locatng accuracy by 44.4% only at the cost of codng 22.4% more nformaton than that of sngle-cell-codng. S I. INTRODUCTION ngle and populaton cell codng mechansms have been an argumentatve ssue n understandng human bran and vson functons, whch has been dscussed and debated n the specal ssue of Neuron Vol.24 () n detal. Sngle-cell-codng means usng one cell (grandmother cell) or one response to represent one object, n terms of wnner-take-all mechansm. Populaton-cell-codng uses an ensemble of cells or responses to represent an object, whch mples relevant cells that encodes smlar object categores, are fred wth a certan responses. Generally t s concluded that both mechansms exst n human neural system but play dfferent roles []. Eye gaze movement or varaton plays an mportant role n human vson nformaton acqurng and object searchng system. Some research work smulates the gaze varaton n Ths research s partally sponsored by Natural Scence Foundaton of Chna (No and No ), H-Tech Research and Development Program of Chna (No.2006AA0Z22), Natural Scence Foundaton of Bejng (No ) and Bejng uncpal Educaton Commttee (No.K ). J. ao s wth the Key Laboratory of Intellgent Informaton Processng, Insttute of Computng Technology, Chnese Academy of Scences, Bejng 0090, Chna (e-mal: jmao@ct.ac.cn). L. Qng s wth the School of Informaton Scence and Engneerng, Graduate Unversty of the Chnese Academy of Scences, Bejng 00049, Chna (e-mal: lyqng@gucas.ac.cn). L. Duan s wth the College of Computer Scence and Technology, Bejng Unversty of Technology, Bejng 0024, Chna (e-mal: ljduan@bjut.edu.cn). B. Zou s wth the Department of Informaton Scence and Technology, College of Art and Scence of Bejng Unon Technology, Bejng 00083, Chna (e-mal: zoubx@yg.edu.cn). bottom-up and top-down modes [2-, 23-26]. For example, when one searches objects n mages wthout the knowledge of object scales, orentatons and postons, he may moves hs gaze to the places accordng to the posteror probablty wth the context between objects and envronmental features[7,8,25-26], whch s known as the top-down attenton or the task-drven object searchng. When one recognzes a located object, he may change hs vew pont from one salent regon to another [9,0] to extract key features accordng to the attracton strength or a salency map [] from the most salent to the least salent, whch s known as the bottom-up attenton or the feature-drven vsual searchng. Ether the top-down or the bottom-up method, f they use the decson prncples by the largest salency or probablty, s smlar to wnner-take-all or sngle-cell-codng mechansm for only one largest response s used to make decson. Sngle-cell-codng means usng one cell (grandmother cell) or one response to represent one object. Generally eye movement s affected by the vsual context that ncludes more than one object n mages, especally n the case of target search. So we hypothesze that populaton codng wth more than one response performs better than sngle-cell-codng n controllng the gaze movement. From the prncple of nformatcs, sngle-cell-codng could save large codng quantty at the rsk of losng accuracy for recognton or behavor control, whle the populaton codng keeps nformaton representng more stable and accurate at the cost of more codng quantty. Experments that appled sngle and populaton codng methods for target search on a real mage database verfy the hypothess and the expermental results show that the populaton codng performs almost twce better than the sngle-cell-codng. II. FEATURES DESIGNED FOR CODING VISUAL CONTEXT In a top-down vsual object search system, the context [7, 2-4] between envronmental features and targets s usually to be learned or coded for the future predcton of the postons of the targets. Vsual context s relatve to the spatal relatonshp n terms of dstances ( x, y) between two objects n an mage respectvely n the horzontal and the vertcal drectons, as shown n Fg.. Generally, learnng such context s nevtable to cost a large amount of memory. Choosng or desgnng concse and effcent features seem qute mportant. The psychologcal expermental results show that when human perceve an mage (e.g. a human face mage) only a few of neurons

2 response n hs vsual cortex [5]. Ths s the strategy of sparse codng for human s vsual neural system. athematcally, sparse codng means that an mage can be approxmately represented by a group of sparse coeffcents correspondng to a number of bases or features, whch obey the super-gaussan dstrbuton. any approaches [6-8] have been proposed for fndng such spares bases. In addton, ndependent components of natural scenes are smlar to the sparse bases [9], whch look lke edge flters [20]. 0 f ( x8 xk ) > 0 bk =, ( k=0~7) (2) otherwse In our codng system, for each receptve feld nput x, there are 256 feature neurons extractng the above extended LBP features and only the frst m neurons havng the largest responses wn through the competton and produce outputs wth ther responses r j = f j (x ), where j=~m. To decrease the codng quantty as much as possble, m may be set to for enough sparsty. III. SINGLE AND POPULATION CELL CODING FOR GAZE OVEENT CONTROL Fg.. Vsual context n terms of dstances ( x, y) n horzontal and vertcal drectons respectvely between two objects. A set of features called local bnary patterns (LBP) [2] s wdely used recently. Although t has not been proved to be sparse codng feature, t s smple and effcent on mage feature extracton and classfcaton. We extended the orgnal LBP features to the new features llustrated n Fg.2. (a) Sngle-cell-codng through wnner-take-all n the thrd layer. Fg extend LBP features (receptve feld=3 3 pxels, each of whch s computed by a sum of eght pars of dfferences between pxels(labels=0~7) and the central pxel (label=8). The gray box represents weght whle the black box represents weght. LBP (local bnary patterns) [24] s a knd of bnary code for representng one of 256 patterns for mage blocks of 3 3 pxels. The orgnal LBP code features only output a dscrete number from 0~255 to encode an mage block pattern nstead of producng a contnuous comparable value for a local mage pattern. We extend LBP features by assgnng them contnuous output wth the followng functon: 7 bk r = f (x ) ( ) ( x x ) () j j 8 k k = 0 where the vector x =(x 0 x... x 8 ) represents the -th mage block or receptve feld nput of 3 3 pxels, the term r j represents the j-th feature extracted from the -th mage block, and j s a dscrete number among 0~255, whch corresponds to a 8-bt bnary code: (b 0 b...b k...b 7 ) 2, where (b) Populaton-cell-codng n the thrd layer. Fg.3. Codng structure of vsual context representaton and gaze movement controllng For the convenence of comparson and dscusson, a unfed neural codng structure s desgned for sngle and populaton cell codng, and s llustrated n Fg.3. The codng structure conssts of two parts. The frst part s mage content codng, whch ncludes the frst three layers: the frst layer - nput neurons, the second layer - feature neurons, and the thrd layer sngle or populaton cell codng neurons. The system nputs a local mage from a group of vsual felds at dfferent resolutons, extracts features and encodes the current vsual feld mage n terms of lnkng weghts between the thrd layer and the second layer. The

3 second part s the spatal relatonshp codng part, whch ncludes the last two layers: the thrd layer - codng neurons and the fourth layer movement control neurons. It encodes the spatal relaton between two objects or between an object and ts envronmental ponts n terms of the lnkng weghts between the thrd layer and the fourth layer, whch correspond to the horzontal and vertcal shft dstances (Δx, Δy) from the center poston (x, y) n the current vsual feld to the center of targets. These two parts naturally ncorporate nto an entre one. They cooperate to code the mage content and the spatal relatonshp on current vsual feld mage at a larger resoluton, and then move from the center of the current vsual feld to the center of the target. Ths codng procedure s run n a repeated mode from the largest vsual feld to the smallest vsual feld. A. Codng of vsual feld mage content The codng neuron(s), whether t s sngle cell or populaton cells that wn(s) through mutual competton, represent(s) dfferent vsual feld mage patterns. Wth reference to Fg. 3, the k-th codng neuron receves nputs weghted wth w k,j from the j-th feature neuron wth the j-th feature response r j for ts -th receptve feld mage nput x. Therefore, a codng neuron s response R k =F( x ), for the vsual feld mage x =( x x 2 x N ) whch s composed of the receptve feld nputs x ( N), s: R k =F( x )=F(( x, x 2,, x N )) N m = = j= w k,j f N m j ( x ) = = j= w k,j r j where the weghts w k,j s obtaned at the codng or tranng stage accordng to Hebban rule w k,j =αr k r j, n whch R k s set to to represent the response of the k-th codng neuron generated for representng a new vsual feld mage pattern, α s set as for smplfcaton, r j s the response of the j-th feature neuron for receptve feld nput x, r j =f j ( x ) belongs to the frst m features that have the largest responses among the total feature responses {r j }(j=0~255) at the testng stage. All the weghts w k,j wll be normalzed to length for unfed smlarty computaton and comparson. B. Codng for gaze movement control Gaze movement control s the key aspect for vsual object research, whch s mplemented n the structure that conssts of two layers of neurons: codng neurons and movement control neurons (see Fg.3). The movement control neurons, dvded nto Δ x and Δ y neurons, represent the target poston (Δx, Δy) based on the current gaze pont (x, y) (the center of the current vsual feld mage). For the current vsual feld mage nput, the frst codng neurons whch have the largest responses play a man role n actvatng the movement control neurons. If =, t s the sngle-cell-codng and controllng mechansm, otherwse t s the (4) populaton-cell-codng mechansm. The responses of gaze movement control neurons can be formulated as: RΔx = wx, R, RΔy = wy, R, = = R = R j= where R s the response of the -th codng neuron among the codng neurons. The terms w x, and w y, are the connectng weghts from the -th codng neuron to the movement control neurons n x and y drectons respectvely. Both of them are calculated based on the Hebban rule: w x, =αδx R, R j (5) w y, =αδy R (6) where Δx and Δy are the responses of movement control neurons, whch equal to the dstances from the current gaze pont to the center of the target when R s the response of the -th codng neuron that s generated for representng a new spatal relatonshp at learnng or codng stage from the current vsual feld mage. At learnng or codng stage, the learnng rate parameter α and the response R can be set to for smplfyng calculaton. Thus, formulae (5) can be rewrtten as: RΔx = x R, RΔy = yr (7) = = Formula (7) means that the gaze movement dstances produced at the percepton stage s the sum of the spatal relatonshp coded at the learnng stage, whch s weghted by the response(s) of the sngle codng neuron (=) or the populaton codng neurons( ). C. Algorthm descrpton The system s coded vsual context s preserved n the weghts of the neural codng structure. The Hebban rule s the fundamental learnng or codng rule,.e., w j =αr R j, where w j s connectng weghts; α s the learnng rate; R and R j are the responses of two neurons whch are connected mutually. Codng algorthm s descrbed as follows:. Input a local mage from a vsual feld centered at the gaze pont, and perceve the target s shft dstances (Δ x, Δy); 2. If the perceved result (Δx, Δy) s not correct, generate a new codng neuron (let response R=); else go to 4; 3. Code or compute connectng weghts between the new codng neuron and feature neurons and that between the new codng neuron and two movement control neurons (responses RΔx=Δx and RΔy =Δy) usng the Hebban rule w j =αr R j ; 4. ove current gaze pont to the center of the target n the current vsual feld and change the vsual feld to a smaller one; 5. Go to, untl all scales of vsual felds and all gven ntal startng gaze ponts are coded.

4 Fg.4 descrbes the target search procedure n terms of gaze movements startng from any gven gaze pont that s ntalzed to be the center of the vsual feld at the largest resoluton. Accordng to the vsual context codng memory, the system perceves mage nput from the largest vsual feld to the smallest vsual feld and moves the gazes n a repeated mode untl the system ensures that the center of the vsual feld at the lowest resoluton s the center of the target (.e. the shft dstances are zero (Δx=0, Δy=0)). The codng or target search procedure s llustrated n Fg.5. mage Vsual feld(vf) mage centered at gaze pont(x, y) Gaze moves wth (delta x, delta y) and change vsual feld to a smaller one No Fearue neuron responsng (delta x, delta y)=(0,0) and small VF? Yes target located Sngle or populaton codng neuron esponsng Gaze movement control neuron responsng wth (delta x, delta y) Fg.4. Target search n terms of gaze movement drven by sngle or populaton cell codng. Fg.5. Illustraton of gradual spatal relatonshp codng or target search. D. Learnng propertes of sngle and populaton codng From the above codng or learnng algorthm, t can be learned that the neural structure codes the support weght vectors that dstrbute on the borders between dfferent vsual context categores. Wth references to Fg.6, there are four vsual context classes coded wth weght vectors W, W 2, W 3, and W 4. A vsual feld mage nput X frstly s projected to sngle or populaton classes accordng to ts frst largest projectons to the four class vectors, and then t s mapped to target postons accordng to the connectons from sngle or populaton codng neurons to the movement control neurons. The number of such support weght vectors s dependent on the vsual context categores n mages. For the sngle-cell-codng, the reconstructon of an nput X s equvalent to ts largest projecton to one of the four class vectors. Obvously, the reconstructon error s larger than the error of the populaton codng whch reconstructs the nput X wth a group of largest projectons and makes the reconstructed X close enough to the orgnal nput X, especally when usng the dual-bass method. Ths learnng property theoretcally verfes our hypothess. (a) Fve vsual felds centered at the target center (the eye center). (a) Sngle-cell-codng (b) Populaton-cell-codng Fg.6. Comparson of reconstructon effects between two codng mechansms. (b) Fve vsual feld mages (6 6 pxels, scales=5, 4, 3, 2 and ) sub-sampled from the orgnal mage ( pxels). (c) The spatal relatonshp between the target center and the any gven startng gaze pont. IV. EXPERIENTS ON CODING FOR GAZE OVEENT CONTROL IN TARGET SEARCH Two experments were carred out to expermentally verfy our hypothess. The results of these two experments reflect the effcency and performance of the sngle and the populaton codng and are obtaned by searchng the left eyeball center based on a set of stll face mages n a database provded by the Unversty of Bern [22]. In ths database, totally there are 300 mages ( pxels) wth 30 people (ten mages each person) n ten dfferent poses. The mean radus of the eyeballs of these 30 persons s 4.02 pxels. Fg.7 llustrates the frst ten mages. Fg.7. Face database of the Unversty of Bern (320x24 pxels) (d) encodng the content of vsual feld mages and the spatal relatonshp between target centers and gaze ponts, or searchng the target accordng to the vsual context coded, gradually from largest vsual feld to smaller ones (here two scales of vsual felds are shown). A. Codng Structures We desgned two codng systems respectvely usng the sngle-cell-codng and the populaton-cell-codng

5 mechansms. A group of vsual felds at 5 scales ( , 28 28, 64 64, and 6 6 pxels) are used to nput local mages from the tranng and test mages ( pxels). For each scale or resoluton, there s the same number of 6 6 nput neuron array wth dfferent ntervals (6, 8, 4, 2 and pxels). Thus, there are totally 5 6 6=280 neurons n the frst layer of the neural codng structure. Wth reference to Fg.2, there are 256 knds of LBP features for 5 vsual felds wth dfferent resolutons, and the sze of receptve feld of each feature neuron s 3 3 pxels, whch has /2 overlap between neghborng receptve felds, thus there are totally [6-(3-)] 2 = feature neurons respectvely n two systems, n whch only (/256)=980 neurons (the frst m largest respondng feature neurons, m= for sparsty, see Secton 2) contrbute to actvate the sngle or populaton codng neurons n the thrd layer. The number of codng neurons n the thrd layer s dependent on the natural categores of vsual context patterns that the system learned. The number of gaze movement control neurons n the fourth layer s 2. These two control neurons should output the value n a range from 8 to 7 to represent 6 postons n x and y drectons respectvely, correspondng to 6 6 nput neuron array for all the fve vsual felds n the frst layer. B. Experments on gaze movement for target search Fg.8. Tranng for codng vsual context between the eye center and a group of ntal gaze ponts n an even dstrbuton. Fg.9. Testng for gaze movement for search the eye center from a group of ntal gaze ponts n a random dstrbuton. Two experments were carred out for each system. Therefore, totally four experments were done on the face database from the Unversty of Bern for searchng and locatng left eye centers. As llustrated n Fg.8 and 9, vsual context codng was wth a group of ntal gaze ponts n an even dstrbuton whle testng was wth a group of ntal gaze ponts n a random dstrbuton. The systems were traned or tested from these ntal gaze ponts to code the context or search the eye centers. In the frst experment (Exp. ) for two systems, 30 mages of 30 people (one frontal mage each person) were coded wth 368 ntal gaze ponts on each mage, and the rest of 270 mages were tested at 48 random ntal gaze ponts on each mage. In the second experment (Exp. 2) for two systems, 90 mages of 9 people (0 mages each one) were coded and the rest of 20 mages were tested wth ntal gaze ponts as Exp.. The number of total feature neurons n layer 2, the number of total codng neurons n layer 3, the number of connectons between feature neurons and codng neurons and the average locatng error are lsted n Table I. Codng system TABLE I PERFORANCES OF THE TWO CODING SYSTES (: ILLION) numbe r of feature neuron s n layer 2 number of codng neurons n layer 3 number of connectons between feature neurons and codng neurons () average locatng error/standard devaton of locatng error (pxels) Exp. Exp.2 Exp. Exp.2 Exp. Exp.2 Sngle cell Populaton cell / / / / 2.2 The number of populaton codng neurons whch can actvate movement control neurons s a dynamcal value that s decded by the rato of the sum of the responses of the frst largest respondng codng neurons to the sum of the responses of the total codng neurons. Accordng to our expermental results, the best search accuracy could be obtaned when ths rato s set to %. The mean radus of the eyeballs n the database s approxmately 4.02 pxels. Table I shows the average locatng error decreases of Exp. and Exp.2 are.68 pxels (from 4.65 to 2.97 pxels) and 2.52 pxels (from 4.82 to 2.3 pxels) respectvely by usng two codng mechansms, whch means the average locatng postons by usng sngle codng s outsde the average borders of the eyeball objects and that by usng populaton codng s nsde the borders of the objects. It can be calculated that the populaton codng system reached an average locatng error that s 44.4% lower than the locatng error provded by the sngle-cell-codng system at the cost of 22.4% more codng neurons and connectons. Smultaneously, the decreases of standard devatons of locatng error for Exp. and Exp.2 are 2.43 pxels (from7.09 to 4.66 pxels) and 3.46 pxels (from 5.67 to 2.2 pxels) respectvely by usng two codng mechansms, whch ndcate sgnfcant standard devaton decreases of 34.3% and 6% respectvely. The large mprovements can be found from the above two groups of experment comparson n terms of means and standard devatons.

6 V. DISCUSSION Wnner-take-all or maxmum a posteror prncples used n much work s smlar to the sngle-cell-codng mechansm. In ths paper, we proposed a hypothess that populaton-cell-codng s more sutable for representng the context that contans more than one objects and ther spatal relatonshp and controllng the gaze moton for target search than sngle-cell-codng mechansm. The hypothess s verfed theoretcally and expermentally. Our expermental results ndcated that the populaton-cell-codng made the reconstructed error smaller than the error obtaned by usng of the sngle-cell-codng, and t reached a target locatng accuracy 44.4% hgher than the locatng accuracy of sngle-cell-codng only at the cost of codng 22.4% more nformaton. The sgnfcant decreases of standard devatons of locatng error, 34.3% and 6%, can be also found from the two experment comparsons. Although the experment was carred out on searchng one object, the system can be easly extended to detect multple objects by tranng the system to learn the context between any postons and the objects. Smlar to the codng features used n references [, 26], the ntensty and orentaton features wth a receptve feld of 2 2 pxels had been adopted to mplement the sngle-cell codng system [25]. Compared wth those two knds of features, the extended LBP feature wth a receptve feld of 3 3 pxels n ths paper, s a knd of contrast features, and has a larger codng range. As a result, a less codng quantty can be acheved. For example, usng the extended LBP feature only need 980 codng neurons to acheve the locatng error of 5.02 pxels (see Table I), compared wth the 2250 codng neurons generated to acheve the locatng error of 5.74 pxels [25]. In the future study, both two aspects of the codng quantty and the search accuracy should be enhanced further to make the context codng and vsual searchng technques more practcal. REFERENCES [] Delang Wang, The tme dmenson for scene analyss, IEEE-NN, 6(6), pp , [2] G. Zelnsky, W. Zhang, B. Yu, X. Chen, and D. Samaras, The role of top-down and bottom-up processes n gudng eye movements durng vsual search, Advances n NIPS, [3]. ozer and D. Baldwn, Experence-Guded Search: A Theory of Attentonal Control, NIPS, [4]. Cerf, J. Harel, W. Enhaeuser, and C. Koch, Predctng human gaze usng low-level salency combned wth face detecton, Advances n NIPS, [5] J. Wolfe, Guded Search 4.0: Current progress wth a model of vsual search, n. W. Gray (Ed.), Integrated odels of Cogntve Systems,NY: Oxford, 2007 [6] A.Torralba, A.Olva,. Castelhano and J. Henderson, Contextual gudance of eye movements and attenton n real-world scenes: The role of global features on objects search. Psych. Rev., 3, [7] A. Torralba, Contextual Prmng for Object Detecton, IJCV 53(2): 69-9, [8] K. urphy, A. Torralba, and W.T. Freeman, Usng the forest to see the trees: a graphcal model relatng features, objects, and scenes, Advances n NIPS, [9] I. Rybak, V. Gusakova, A. Golovan, L. Podladchkova, and N. Shevtsova, A model of attenton-guded vsual percepton and recognton, Vson Research, vol.8, pp , 998 [0] T. Shbata, S. Vjayakumar, J. Conradt, S. Schaal, Humanod oculomotor control based on concepts of computatonal neuroscence, Proc. the 2nd Internatonal Conference on Humanod Robots, 200. [] L. Itt and C. Koch, A salency-based search mechansm for overt and covert shfts of vsual attenton, Vson Research, vol. 40, pp , [2] H. Kruppa,. Santana and B. Schele, Fast and Robust Face Fndng va Local Context, Proc. Jont IEEE Internatonal Workshop on Vsual Survellance and Performance Evaluaton of Trackng and Survellance (VS-PETS03), [3] L. Paletta and C. Grendl. Context based object detecton from vdeo, LNCS 2626-Proc. Internatonal Conference on Computer Vson Systems, pp , [4] T. Strat and. Fschler, Context-based Vson: recognzng objects usng nformaton from both 2D and 3D magery, IEEE-PAI vol.3(0), pp , 99 [5]. Yong, S. Yamane, Sparse Populaton codng of faces n the nferotemporal cortex, Scence, vol.256(), pp , 992. [6] B. Olshausen and D. Feld, Sparse codng wth an overcomplete bass set: a strategy employed by V?, Vson Research, vol.37, pp , 997. [7] A. Hyvarne, P.O. Hoyer, A two-layer sparse codng model learn smple and complex cell receptve felds and topography from natural mages, Vson Research, vol.4(8), , [8] D. Lee and H. Seung, Learnng the parts of objects wth nonnegatve matrx factorzaton, Nature, vol.40, pp , 999. [9] S. Hornllo-ellado, R. artín-clemente, C. Puntonet, and J. Górrz, Connectons between ICA and sparse codng revsted, Lecture Notes n Computer Scence, Vol. 352, , [20] A. Bell and T. Sejnowsk, The ndependent components of natural scenes are edge flters, Vson Research, vol.37(23), pp , 997. [2] T. Ahonen, A. Hadd, and. Petkanen, Face recognton wth local bnary patterns, Lecture Notes n Computer Scence-Proc.ECCV, vol. 302, pp , [22] The Face Database of the Unversty of Bern, ftp://amftp.unbe.ch/pub/images/faceimages/, [23] C. Guo and Lmng Zhang, An attenton selecton model wth vsual memory and onlne learnng, Proc. IJCNN, [24]. de Brecht and J. Sak, A neural network mplementaton of a salency map model, Neural Networks, vol.9 (0), , [25] J. ao, L. Qng, L. Duan and W. Gao, Searchng eye centers usng a context-based neural network, Proc. of the 4 th Internatonal Symposum on Neural Networks, [26] L. Lanyon and L. Denham, A model of actve vsual search wth object-based attenton gudng scan paths. Neural Networks, Specal Issue: Vson & Bran, 7(5-6): , 2004.