Space-variant image sampling for foveation

Transcription

1 Space-vaiant image sampling fo foveation Paul Fitzpatick Machine Vision (6.866) Tem Pape MIT ID Abstact In active vision applications, thee can be a conflicting need fo both high acuity and a wide field of view. One possible tade-off in such cases is to povide high acuity in a small cental aea, and to use lowe esolution infomation in a wide field of view to contol whee that cental aea is placed. This aangement is akin to the foveated etina of the human eye. Fo binocula vision unde these cicumstances, it is impotant that both cameas cente the same object. In humans, this is achieved by vegence eye movements. One simple, fast method fo implementing vegence is to use coelation of a log-pola sampling of the images fom the cameas. I will examine a vegence algoithm based on log-pola sampling, and pesent an analysis and implementation of a modified vesion of this algoithm. Intoduction Visual tasks in obotics, paticulaly humanoid obotics, can have a conflicting equiement fo high acuity and a wide field of view. High acuity is needed fo ecognition tasks and fo contolling pecise visually-guided moto movements. A wide field of view is needed fo seach tasks, fo tacking fast o multiple objects, compensating fo involuntay ego-motion, etc. A common tade-off is to sample pat of the visual field at a high enough esolution to suppot the fist set of tasks, and to sample the est of the field at an adequate level to suppot the second set. This is the same tade-off used in animals with foveate vision, such as humans, whee the density of (colo) photoeceptos is highest at the cente and falls off damatically towads the peiphey. It can be implemented by using multiple cameas with diffeent fields of view [6], o specially designed hadwae [5]. Fo a binocula vision system, it is useful to have some way to ensue that both cameas foveate, o cente, the same object. This is called vegence in human vision, and is the only eye movement whee the eyes ae moved towads and away fom each othe athe than moving in lock-step. Humans use many cues fo pefoming vegence, including motion and shading, but the main cue is dispaity. This is the also the main cue that machine Left camea Right camea Vegence angle Saccade Followed by change in vegence Followed by smooth pusuit Figue : This figue shows a simplified decomposition of eye movements in humans. Vegence movements allow the eyes to cente objects at vaying depths in the highest esolution aea of the etina. Smooth pusuit tacks objects acoss the visual field, and inteacts with vegence. Saccades move the eyes apidly to a new pat of the visual field. vision implementations use fo applications in unconstained envionments. Computing dispaity involves establishing a coespondence between the images fom the two cameas, which is a poblem that has been studied extensively in steeo vision. The main baie to binging all this wok to bea on vegence is the sevee estictions on the amount of computation that can be pefomed, since the vegence algoithm must be fast enough to contol the cameas in eal-time. Fo contempoay hadwae, this geneally esticts vegence algoithms to simple, limited, techniques such as coelation of patches of the images, o coelation in the fequency domain. An example of the latte is the cepstal filte [8], a type of filte oiginally developed to analyze signals containing echoes. Anothe way to impove coelation pefomance without moving to the fequency domain is to wok with the log-pola tansfomation of the image. The log-pola tansfom gives a space-vaiant sampling of the image

2 that is most dense aound the cente and least dense towads the peiphey, so that the cente of the image is epesented with a highe esolution than the est of the image. Coelation of such images with each othe amounts to little moe than a weighted coelation of the oiginal untansfomed images, but it nevetheless gives supisingly good esults, and is faste than cuent methods that opeate in the fequency domain. The natue of the log-pola tansfom is descibed in Section 3, and details of an implementation of vegence using the tansfom by Santos-Victo and Benadino [4] ae given in Section 4. The algoithm is analyzed in Section 5. I implemented a vaiant of thei algoithm on an active vision head, the details of which ae given in Section 7, with esults in Section 8. The active vision head I woked with was fom the Cog humanoid obot poject at the MIT AI Lab [6]. The head is binocula, with each eye having both a wide angle camea and a foveal camea with a naow field of view. An object can theefoe be imaged with geatest detail when infomation fom the wide field of view cameas is used to bing the object into the field of view of the foveal cameas. Relevant details of the head ae given in Section 7., when I descibe the details of my implementation. Figue 2: The active vision head 2 Appoaches to vegence Vegence seems a simple poblem than geneal steeo, since it is only necessay to identify a single point of coespondence between the camea images. Unfotunately, this is not an easy task, and typical methods fo establishing coespondences ely on the optimization of a global eo measue fo obustness. In othe wods, to establish a single coespondence eliably it is necessay to solve the total steeo vision poblem, which is computationally expensive. Since vegence has to be done in eal-time, thee ae sevee pactical limits on the computation that can be devoted to identifying the coespondence. In constained envionments, vegence can be quite easy to implement fo example, when thee is a high contast between the object and the backgound, o when the object is moving and is the only pat of the envionment doing so, o when the object is a light bulb [9]. But fo abitay stationay objects in a clutteed envionment, the coespondence poblem is much hade. It is possible to make the coespondence poblem consideably easie by simply changing the hadwae. Fo example, the coespondence poblem becomes inceasingly staightfowad as the distance between the cameas is educed, since the images will diffe less though occlusions, foeshotening and shading effects. Howeve, the most accuate depth measuements can be deduced fom coespondences when the cameas ae fa apat. So the cameas could simply be moved vey close togethe, making coespondences simple to establish, and then moved apat at a ate that allows the coespondences to be updated iteatively, with the seach at each iteation being highly constained by the esults of the pevious iteation. This is the appoach used by FOVEA [2], a foveated vegent active steeo vision system using an expanding baseline. While the goal of that wok was to ceate dense depth maps, the hadwae base would make vegence an easie task. This is not an option fo the Cog poject because of a commitment to humanlike vision, fo easons that will be discussed in Section 9. Othe engineeing appoaches include using multiple (>2) cameas with multiple baselines (fo example, []), o illuminating the scene with caefully engineeed light pattens (fo example, [3]). If the hadwae is taken as a given, then the poblem becomes finding a method fo establishing coespondences that is a good balance between accuacy and speed. The simplest way to seach fo coespondences is to coelate a small patch of the image fom one of the cameas with small patches of the image fom the othe camea, within some aea o along an epipole. This is vey staightfowad to implement, although it has well known failings, such as coping pooly with diffeences in foeshotening effects []. Vegence modules along these lines have been implemented on the Cog active vision head by a numbe of people including Yamato [2]. As a baseline fo compaison, I implemented such a system too, and found it to be vey uneliable. The next step geneally taken afte this appoach is abandoned is to switch to doing coelation in the fequency domain [8]. This equies using the FFT, which takes consideably moe computation then coelation on the aw image. A less computationally intensive altenative that has aisen fom consideations of pimate vision makes use of a log- 2

3 Figue 3: Photoecepto density acoss the etina. Visual acuity is geatest within a egion of about.3,andthen falls off vey apidly. (Taken fom [9]) pola sampling of the image, the natue of which will now be descibed. Figue 4: Log-pola image sampling. The gid size inceases exponentially with distance fom the cente. Thee is a singulaity at the cente whee the gid size goes to zeo, so the gid must be tuncated at some minimum adius. 3 The log-pola tansfom In the etina, the density of photoeceptos gows apidly towads the cente of the visual field (the fovea). This effect can be appoximated using a log-pola tansfomation. This maps the image on to a pola coodinate system whee the distance coodinate is logaithmic: (x, y) = (ρ, θ) ρ = log b = log b x2 + y 2 θ = tan y x The paamete b contols how quickly the density falls off with distance fom the cente. Clealy, thee is a singulaity at the cente of the coodinate system. This is often handled by intoducing anothe paamete ρ min to contol the minimum ρ value that will be used. Anothe choice would be to use ρ = log b ( + a) wheea is some constant to avoid the singulaity when =. Thisis the choice used in models of the actual etinal density [5], and fo this application it has the advantage of not cutting out infomation fom the one pat of the image you cae most about. Figue 4 shows a egula gid in (ρ, θ) space mapped onto an image in Catesian space. Notice how the gid size deceases towads the cente, showing that the cental pat of the image is sampled at the highest esolution. Figue 5 shows an example of a tansfomed image. The log-pola tansfom has some popeties that make it potentially useful fo vision. One popety that stimulated ealy inteest is that scaling and otation Figue 5: An example of a tansfomed image. The image has been shifted in θ to facilitate compaison with the oiginal. The figue on the left shows the oiginal image, in the (x, y) coodinate system. The figue on the ight shows the tansfomed vesion of this image in the (ρ, θ) coodinate system. The gid size is, of couse, much fine than shown in Figue 4. Notice, fo example, that the eas become smalle than the eyes, since they ae futhe fom the cente of the oiginal image. of the image coespond to simple tanslation in (ρ, θ) coodinates. Of couse, the down side of this is that tanslation of the image in Catesian coodinates coesponds to a complicated waping in (ρ, θ) coodinates. But foveation centes an object in a steeotyped way, so in theoy at least this could povide a basis fo scale and otation invaiant object ecognition. Anothe eason fo inteest in log-pola sampling is that it can give a damatic eduction in image size, and hence incease the speed of pocessing the image. This applies only to tasks fo which the cente of the image is all-impotant and the peiphey is less so. 3

4 4 Using the tansfom Hee I give details of how the log-pola tansfomation has been used to implement vegence. The paticula algoithm I descibe is due to Benadino and Santos- Victo [2, 4]. Fist, the log-pola tansfom is applied to the images fom the two cameas. This educes the size of the image significantly (a facto of eight fo Benadino and Santos- Victo), since the peiphey is sampled at a much educed esolution. Then a coelation measue is applied to the two tansfomed images, ove thei full aea. As the cameas move, the sign of the change in the coelation measue is used to contol whethe the cameas should stay moving in the same diection, o switch diections. The speed at which the cameas should move is detemined fom the magnitude of the coelation measue. The computation equied fo this scheme is vey minimal, so vegence can be extemely apid. Unfotunately the cameas can get captued at locations that coespond to local maxima of the coelation function, and the speed of convegence is stongly dependent on faily abitay popeties of the scene. Benadino and Santos-Victo also descibe a moe obust vegence stategy called pepogammed vegence contol, modeled afte a theoy of vegence in humans. Hee they diectly estimate the dispaity, at the pice of moe computation. The dispaity estimate is made by pefoming the coelation fo vaious assumed dispaities between the images, and finding the coelation peak. This makes the system less susceptible to local minima of the coelation measue. Pefoming the coelations equies a complicated waping of the images to eflect tanslation in Catesian space, maps fo which ae calculated off-line. The dispaities tested fo ae sampled with highest density aound zeo and deceasing density fo lage dispaities. The coelation measue used by Santos-Victo and Benadino in [2] is: ρ,θ (I l(ρ, θ) µ (Il ))(I (ρ, θ) µ (I)) ρ,θ (I l(ρ, θ) µ (Il )) 2 ρ,θ (I (ρ, θ) µ (I)) 2 This fist subtacts the mean value fom the images, and then teats them as vectos and calculates the cosine of the angle between them, using this as a measue of similaity. Since the measue is invaiant to bias and scaling applied to the images, thee is less need to caefully calibate the cameas o monito changes in illumination conditions. If I l and I ae teated as andom vaiables, of which the images epesent N N pais of samples, then this measue is exactly the coelation coefficient of the two vaiables. c = σ (I l I ) σ (Il )σ (I) whee σ (Il I ) = E[(I l µ (Il ))(I µ (I))] σ(i 2 l ) = E[(I l µ (Il )) 2 ] σ(i 2 ) = E[(I µ (I)) 2 ] with E(x) being the expected value of the andom vaiable x, empiically estimated by the mean. This coelation is caied out on the image afte it has been tansfomed into its log-pola equivalent, so it is weighted towads measuing the statistics of the cente. It is of couse possible to fold the tansfomation into the coelation measue, and simply apply the combined measue to the untansfomed images. I give the elevant analysis in the next section, since it claifies the natue of the coelation. 5 Analysis Santos-Victo and Benadino in [2] compae the popeties of a paticula similaity measue in Catesian coodinates and log-pola coodinates when the dispaity is low. Fo some eason, pesumably simplicity, the similaity measue is not the one they actually use in thei algoithm. I give hee an analysis fo the coelation metic used in the pape, and show what the metic is expessed in tems of the oiginal Catesian image. Conside the contininuous analog of the coelation measue given ealie: (Il µ (Il ))(I µ (I))dρdθ c = (Il µ (Il)) 2 dρdθ (I µ (I)) 2 dρdθ Witing (x, y) in tems of (ρ, θ) gives: x = b ρ cos θ y = b ρ sin θ The tansfomation between the coodinate systems is then: x x ρ θ da = dxdy = dρdθ y ρ y θ whee the deteminant of the Jacobian is: x x ρ θ y y = ρ θ bρ cos θ log b b ρ sin θ b ρ sin θ log b b ρ cos θ = b 2ρ log b = 2 log b Whee = x 2 + y 2 as usual. So the coelation measue can be ewitten as: c = Il µ (Il ) ( I l µ (Il ) I µ (I ) dxdy ) 2 dxdy ( I µ (I ) ) 2 dxdy 4

5 Ignoing the subtaction of aveages, this is just the coelation of the images afte they have been individually weighted by a facto invesely popotional to the distance fom the cente. Benadino makes an equivalent point fo anothe similaity measue (a Euclidean distance metic). I think it is impotant to stess that this does not imply that the oveall coelation is weighted by. That would be a athe weak weighting, since the total weight along lines of equal distance fom the cente would emain constant. This is undesiable because thee is moe aea in the peiphey of the image than in the cente, so if the weighting doesn t fall fast enough the total weight of the peiphey can be highe than that of the cente. The actual natue of the weighting is claified by expanding the above out fully as: [ RR I 2 l RR 2 dxdy RR Il I RR 2 dxdy 2 dxdy RR Il ( 2 dxdy RR Il RR 2 dxdy 2 dxdy ][ RR RR 2 dxdy I 2 )2 2 dxdy RR I 2 dxdy RR 2 dxdy 2 dxdy RR RR ( I 2 dxdy RR 2 dxdy 2 dxdy)2 This is exactly the fomula fo coelation in Catesian space, but with an exta weighting of appeaing 2 eveywhee. Eveything in the above equation is staightfowad to calculate without making the log-pola tansfomation. Notice that the b paamete of the tansfomation does not appea. The minimum adius paamete ρ min tanslates to leaving a hole in the middle of the aea ove which the integal is pefomed. Clealy, this integal could have been deived by a diect attempt to weight the coelation measue. It is inteesting to now wok backwads, and see what tansfomations diffeent weightings would coespond to. Conside a adially symmetic weight w(), which fo the above case was. We want to find a coodinate system (φ, θ) fo which nomal coelation will give the same 2 esult as a weighted coelation in the Catesian coodinate system. We can constain the coodinate system to be pola: x = (φ)cosθ y = (φ)sinθ The vaiable has its usual meaning, with = x2 + y 2. The deteminant of the Jacobian of this tansfomation is d dφ. As shown ealie, all the integals in the coelation end up divided by this quantity, so it must be is the ecipocal of the desied weighting function w(). So: d dφ = w() w()d = dφ φ = w()d + C ] Fo a weighting of, fo instance, φ = 3 is a solution (ignoing scale factos, which don t affect the coelation). This means that the image is tuned inside out, with points appoaching the cente of the image in Catesian coodinates appeaing out towads infinity in the tansfomed image. Highe powes of ae simila. In fact the weighting is a special case since it leads to 2 the integal of, which is logaithmic instead of a powe of. This suggests that the log-pola tansfomation is not easily tweaked to give stonge weight to the cente it is tied stongly to the weighting of. 2 6 Comments Reseaches who use the log-pola tansfom fo vegence stess its speed advantages. This is indeed a cucial facto fo vegence, since it sits inside a contol loop tying to keep the cameas foveated on an object, and the less latency thee is in the loop the moe tightly the object can be kept foveated and the moe stable the oveall system is. Anothe advantage cited fo the log-pola tansfom is its close match with the natue of foveated tasks, whee the cental aea of the image is all-impotant and the peiphey is less so. It seems to be taken fo ganted that this match also applies to the vegence task, since it too is concened with the cente of the image. But I don t think this is as obvious as it might seem, and contains an implicit assumption about the type of vegence being implemented. By definition, when vegence has any wok to do, it must be the case that the cameas ae not centeed on the same object. To move the cameas so that they ae centeed on the same object, the impotant infomation will in geneal lie away fom the cente of the images fom the cameas. Theefoe it does not seem sensible to thow away infomation fom the peiphey of the images while sampling the cente of the image densely. Vegence can be divided into two phases: tacking and acquiing. When an object is moving slowly, the vegence angle needed to fixate it will change smoothly. If the fame ate is high enough, the coect fixation point will be close to the cente of the images, and so the log-pola sampling scheme etains the impotant infomation. But this fom of vegence is quite simple to implement using any one of a numbe of schemes, including simple coelation ove small patches. The poblem of initially acquiing coect vegence on a new object of inteest o an object that has just appeaed is a moe difficult pat of the vegence poblem. In this case, the cente of the images fom the cuent view of the camea ae no moe likely to be impotant than any othe hoizontally offset position. In these cases then, vegence does not immediately appea to be a good candidate fo a foveated appoach. One agument to dispute this conclusion would be to make 5

6 an appeal to biology, since humans can clealy vege obustly woking fom an appoximately log-pola sampled image. Moe convincingly, it could be agued that the lage the dispaity, the less accuately that dispaity needs to be known, since the vegence contol is pat of a closed loop system and this is exactly what the log-pola tansfom will give, since it has high esolution at the cente and lowe esolution towads the peiphey. But it seems likely that the lowe esolution at the peiphey can only educe the set of cicumstances unde which the qualitatively coect vegence angle will be detected. In my opinion, thee ae two sepaable ideas in Benadino and Santos-Victo s wok that have been somewhat conflated. One is the use of the log-pola tansfom to bias coelation towads measuing the statistics of the cente of the image. Anothe is the use of the log-pola tansfom to educe the size of the image and speed pocessing. In paticula, it is possible to use the tansfom as a weighting scheme only, and to elinquish its use in compessing the image (and also its biological plausibility). Fo example, instead of waping aleady-tansfomed images to poduce the vesions needed fo vaious assumed dispaity values, this could be done fom the oiginal Catesian images. This means thee is no longe a bias towads small dispaities. Doing this sacifices some speed, since thee ae moe opeations ove the lage Catesian images. I consideed it a wothwhile tade-off, so this is the algoithm I implemented. 7 Implementation I implemented a vaiant of Santos-Victo and Benadino s algoithm fo pefoming vegence. The natue of this algoithm, and the vaiation I intoduced wee descibed in the pevious section. This section biefly descibes the popeties of the active vision head on which this wok was done, and outlines some impotant implementation details that have not yet been touched on. 7. Chaacteistics of the head The active vision head I used fo this wok has two pais of colo cameas. One pai has 3 mm lenses, giving wide fields of view (about 2 along the hoizontal). The othe pai has mm lenses, giving naow fields of view (about 25 along the hoizontal). I only made use of the cameas with wide fields of view, and conveted the colo images to geyscale so that the vegence system could be used on othe active vision heads in the lab which cuently have monochome cameas. The head has thee degees of feedom: independent pan fo the left cameas and the ight cameas, and a shaed tilt of the entie head. Fo this wok, only the Object in font of the Cyclopean eye Left image plane Left eye Cyclopean eye Object to the side of the Cyclopean eye s view Right eye Right image plane Figue 6: This shows how the taget fo vegence is chosen. Objects in font of the Cyclopean eye poject onto mio locations in the images fom the left and ight cameas, while othe objects do not. pans wee used. The cameas can pan vey apidly fast enough fo the limiting facto to be the computation to decide when to stop them. The baseline distance between the left and ight cameas is about 3cm (human aveage inte-ocula distance is about 6.5cm). The cameas ae contolled by a set of TMS32C4 digital signal pocessos. Code fo these is witten in Paallel C fom 3L. 7.2 Choosing the taget One issue that has not been mentioned yet in this pape is how to chose the taget on which the cameas ae to vege. In the absence of any highe-level goals, one common answe is to make one of the cameas dominant and vege on whateve appeas in the cente of the image fom that camea. Since moving the dominant camea will change the taget and could cause looping behavio, this means that, in pactice, this camea needs to be held stationay while the othe conveges. This is vey unnatual in appeaance. I chose to make the cameas vege on anything that appeaed diectly in font of the active vision head. Specifically, I imagined a vitual Cyclopean eye between the cameas, facing in a diection intemediate to the two cameas, and defined whateve appeaed at the cente of its view to be the taget fo vegence. When the Cyclopean eye is facing fowads, anything in font of it will be pojected to an equal distance fom the inne side of each image of the left and ight cameas, as shown in Figue 6. When the Cyclopean eye faces off-axis, thee is a simila elationship. This constains the numbe of 6

7 A A C A C B B B Vegence angle (degees) Left eye Right eye Left eye Right eye Left eye Right eye Time (seconds) Figue 7: Sequence of movements in changing fixation fom A to B. Eyes move to cente the new taget, and then to vege on it (Afte [2]). These actions ae somewhat supeimposed. compaisons that need to be made. Thee can be moe than one match if thee is moe than one object diectly in font of the Cyclopean eye. The match whose vegence angle shows that it is closest to the head should be chosen. In my implementation, I don t do this. I simply choose the one with the lagest coelation. If the closest object is not too small, it will occlude, at least patially, the camea s views of objects behind it, and so it likely to have the highest coelation value. I kept the Cyclopean eye facing diectly fowads so that only objects diectly in font of the head would be used as tagets fo vegence. In othe wods, I dove the position of the cameas so that they always otated by equal amounts in opposite diections. Human vision behaves in a way eminiscent of this. Conside the situation shown in Figue 7. When a taget disappeas fom position A and appeas at position B, both eyes move to compensate. It might appea that only one eye need move while the othe could stay stationay. This is tue, but is not what occus. In human vision both eyes tun to cente the object on a line dawn fom the midpoint of the baseline (whee the Cyclopean eye would be), and then vegence bings the object back onto the fovea. The ight eye ends up back whee it stated, but moved though a complicated path to get thee. 7.3 Contol stategy Since vegence is pefomed in a feedback loop with latencies, cae must be taken to ensue that the contolle will be stable. I chose a vey simple contol stategy, whee the velocity of the camea motos was made popotional to the dispaity between the camea images. Figue 8: A ecod of the camea movements coesponding to the sequence of events in Figue. The vegence angle is estimated fom shaft encode ticks, so the scale facto of the gaph is appoximate but its shape is accuate. Between and 5 seconds, the cameas ae veged on the backgound. A human figue entes the foveal egion at 5 seconds. The human extends his hand at seconds, and emoves it at 7 seconds. This tigges a eflex etun to vegence on infinity. Vegence is egained on the human figue at about 2 seconds. At 4 seconds the human figue withdew, and the cameas etuned to veging on the backgound. Note that the shap eflexive loss of vegence does not occu hee. Thee ae many ways this could be impoved, but the fame ate was high enough (5Hz) and the inetia of the cameas was low enough fo this stategy to be stable and still allow fast camea movement. The vegence algoithm made no attempt to monito the positions of the cameas. Position measuements wee used only to monito fo dastic failues of vegence fo example, the cameas diveging beyond the point whee they ae paallel, o conveging so fa as to appoach the had limits of motion of the camea. One detail was that when the cameas wee veged on a vey close object, and that object was emoved vey apidly, too little of the fontal view emained fo thee to be a high enough coelation to attact the cameas back to facing fowad. So when the best coelation measue fell below a theshold, the cameas wee made to eflexively etun to veging on infinity. Usually that theshold was only exceeded when thee was no object in view that could be veged on. 8 Results Figue 8 shows how the vegence angle of the cameas changes ove time as the object being veged on changes. 7

8 The sequence of events is as follows (shown in Figue ). The cameas vege fist on the backgound. Then a human figue entes the scene. The peson extends thei hand vey close to the camea. The hand is then emoved, and the peson leaves. The gaph shows that the vegence system tends to oveshoot slightly, due to the simplicity of the contolle used. It also shows that when the eyes ae highly veged, loss of a taget tigges a eflex that bings the cameas back to veging on infinity befoe contol is etuned to the vegence system. This was necessay because the images in the camea ae often too diffeent in this cicumstance fo the system to egain coect vegence. Figue also shows coelation measuements unde the diffeent cicumstances. Since these apply to veged conditions, the peak is always at zeo dispaity. The peak is shapest when veged on the backgound, and smallest when veged on a vey close object. Figue shows coelation values in veged and nonveged cicumstances. In the non-veged case, the peak is off cente, and indicates how much the ight image should be shifted left, and the left image shifted ight, fo the best match at the cente. While the highest peak is the coect dispaity, thee is a smalle peak coesponding to the dispaity of the backgound. Figue 9 shows a time sequence of the cameas veging on a smoothly moving object. Vegence angle (degees) Time (seconds) Figue 9: This time seies shows the cameas veging on an object moving towads, then away fom the active vision head. Notice the jumps in the gaph. This is due to the low esolution of the images used. The smallest change in dispaity that can be detected is one pixel elative to the vitual Cyclopean eye, o two pixels total, which coesponds to a lage distance the futhe the object is fom the camea. This is why the gaph gets oughe towads the lowe values of the vegence angle, when the object is futhest away. 9 Discussion Vegence using log-pola sampling of the images poved consideably moe obust than any pevious implementation of vegence on the Cog active vision platfom. It is by no means pefect, and suffes fom the pedictable defects of a weighted coelation ove the entie image: it won t wok on small objects, it won t wok with objects that contast pooly with the backgound, and sometimes it just won t wok. My implementation is also not paticulaly pecise, as Figue 9 showed, but this is not intinsic to the algoithm. My pioity was to make vegence obust to sudden changes in dispaity, since tacking small changes in dispaity is a staightfowad task. With a elatively coase but obust vesion of vegence woking, the views fom the foveal cameas will be of appoximately the same pat of the scene, so it should be possible to now take the views fom the foveal cameas and use them fo pecise vegence. Log-pola sampling is such a simple technique that I am almost embaassed to submit a tem pape on the topic, but it is cuently an attactive method fo pefoming vegence. And vegence is an attactive visual behavio to implement, because if it can be done obustly without object ecognition, it helps to solve the figue/gound sepaation poblem. Once the cameas have veged, the object they have veged on can be extacted fom the backgound using, fo example, a zeo dispaity filte [8]. This can be used to implement a obust object tacking behavio [4]. Vegence also facilitates steeo fusion aound the foveation point, since steeo algoithms that will only opeate with small dispaity values can be applied once the cameas have veged [7]. Both these popeties facilitate object ecognition. Othe advantages ae that the vegence angle gives a measue of the distance to the object, and that vegence is a step towads having an object-centeed coodinate system. All these advantages ae elevant fo the Cog poject. Thee is also anothe moe specific and less technical eason fo vegence being impotant fo the type of eseach being pusued fo the Cog poject. Pat of the poject focuses on social inteaction as scaffolding fo leaning [6]. Mechanisms of joint attention, such as pointing and gaze diection, ae a key element of this wok. It is impotant that Cog can infe what a human is inteested in fom thei gaze diection. But the dynamic of social inteaction makes it equally impotant that a human can infe the location to which Cog is paying attention. It is helpful to make Cog s gaze as human-like as possible. Cuently Cog keeps its cameas paallel when it tuns towads an object of inteest. Adding vegence will make its locus of inteest easie fo a human to deduce, by adding infomation about distance. This is paticulaly so because Cog s cameas ae about twice the distance apat as human eyes, exaggeating the diffeence between 8

9 .8 Coelation Assumed dispaity (pixels).8 Coelation Assumed dispaity (pixels).8 Coelation Assumed dispaity (pixels).8 Coelation Assumed dispaity (pixels).8 Coelation Assumed dispaity (pixels) Figue : The above images ae views fom the two cameas taken at five second intevals. Views fom the left camea ae on the left, views fom the ight camea ae on the ight. The cameas ae designated left and ight fom the point of view of a homunculus behind the cameas. The fist pai of images show the cameas in nea-paallel oientation, veging on the distant backgound. The coelation measue, given beside the pai of images, is shaply peaked. The next pai of images show the cameas veged on a face. Objects in the backgound now appea shifted. The coelation measue is less shaply peaked. The next pai of images show the cameas veged on a hand. The human figue now appeas shifted outwads. Notice that the appeaance of the hand diffes significantly between the images. The coelation measue is quite eatic. The following two pais of images show the cameas eveting to veging on the face when the hand is emoved, and then to the backgound when the face is emoved. The images shown hee ae scaled vesions of the pixel images actually used by the vegence system. 9

10 .8 Coelation Assumed dispaity (pixels).8 Coelation Assumed dispaity (pixels) Figue : The uppe pai of images show a toy in font of the cameas with the vegence system disabled. The coelation gaph on the top ight shows that thee is a dispaity of 2 pixels in the convegent diection. This dispaity is with espect to a vitual cyclopean eye, so the total dispaity between the two images is 4 pixels (image width is 64 pixels). Thee is a seconday peak in coelation fo a dispaity close to zeo, coesponding to the backgound. The lowe pai of images show that when the vegence system is enabled, the cameas move to cente the dominant dispaity. paallel and convegent camea angles. One difficulty with the log-pola tansfom fo vegence is that is no obvious way to impove its pefomance. Details such as accounting fo the significant adial distotion of the wide angle cameas used might have some small benefit. Using colo images might help somewhat as well. But thee is no eal theoetical meat to build on. Theefoe, fo this application, the log-pola tansfomation may be just a stop-gap appoach until the pice of computation falls enough to allow the use of moe igoously developed techniques. Refeences [] D. Ballad. Animate vision. Atificial Intelligence, 48(): 27, Febuay 99. [2] A. Benadino and J. Santos-Victo. Coelation based vegence contol using log-pola images. Intenational Symposium on Intelligent Robotic Systems, 996. [3] A. Benadino and J. Santos-Victo. Senso geomety fo dynamic vegence: chaacteization and pefomance analysis. Euopean Confeence on Compute Vision, 996. [4] A. Benadino and J. Santos-Victo. Visual behavious fo binocula tacking. Robotics and Autonomous Systems, to appea, 998. [5] G. Bonmassa and E. Schwatz. Lie goups, space-vaiant fouie analysis and the exponential chip tansfom. Compute Vision and Patten Recognition, 3: , 996. [6]R.A.Books,C.Beazeal,R.Iie,C.C.Kemp,M.Majanovic, B. Scassellati, and M. Williamson. Altenate essences of intelligence. AAAI, 998. [7] C. Capuo, F. Paneai, and G. Sandini. Dynamic vegence using log-pola images. Intenational Jounal of Compute Vision, 24():79 94, August 997. [8] D. Coombs. Real-time gaze holding in binocula obot vision. PhD thesis, Univesity of Rocheste, 992. [9] C. Gaham. Vision and visual peception. John Wiley and Sons, Inc. [] B. K. P. Hon. Robot Vision. MIT Pess, Cambidge, Massachusetts, 986. [] T. Kanade, S.B. Kang, J.A. Webb, and C.L. Zitnick. An active multibaseline steeo vision system with eal-time image captue. In CMU-CS-TR, 994. [2] W. N. Klaquist and A. C. Bovik. Fovea: A foveated vegent active steeo vision system fo dynamic thee-dimensional scene ecovey. IEEE Tansactions on Robotics and Automation, 5:755 77, 998. [3] A. Koschan, V. Rodehost, and K. Spille. Colo steeo vision using hieachical block matching and active colo illumination. Intenational Confeence on Patten Recognition, A: [4] T.J. Olson and D. Coombs. Real-time vegence contol fo binocula obots. Intenational Jounal of Compute Vision, 7():67 89, Novembe 99. [5] S. Rougeaux and Y. Kuniyoshi. Robust eal-time tacking on an active vision head. Poc. Int. Conf. on Intelligent Robots and Systems, 997. [6] B. Scassellati. A binocula, foveated active vision system. Technical Repot 628, MIT Atificial Intelligence Lab Memo. [7] E. Schwatz, D. Geve, and G. Bonmassa. Space-vaiant active vision: Definition, oveview and examples. Neual Netwoks, 8(7/8):297 38, 995. [8] C. Silva and J. Santos-Victo. Egomotion estimation using log-pola images. Intenational Confeence of Compute Vision, 998. [9] A. Takanishi, T. Matsuno, and I. Kato. Development of an anthopomophic head-eye obot with two eyes-coodinated

11 head-eye motion and pusuing motion in the depth diection. In Intenational Confeence on Intelligent Robot and Systems, volume 3, pages V27 28, 997. [2] G. Westheime. Oculomoto contol: the vegence system. In R. A. Monty and J. W. Sendes, editos, Eye movements and psychological pocesses. John Wiley and Sons, Inc. [2] J. Yamato. Tacking moving object by steeo vision head with vegence fo humanoid obot. Maste s thesis, MIT Depatment of Electical Engineeing and Compute Science, 998.