Hitachi Ltd., Production Engineering Research Lab. ABSTRACT

Transcription

1 Real-time omputation of depth from defous Masahiro Watanabe, Shree K.Nayar y and Minori Noguhi Hitahi Ltd., Prodution Engineering Researh Lab. 292 Yoshida-ho, Totsuka, Yokohama 244, Japan y Columbia University, Department of Computer Siene New York, NY 127 ABSTRACT A new range sensing method based on depth from defous is desribed. It uses illumination pattern projetion to give texture to the objet surfae. Then the image of the sene is split into two images with dierent fous settings and sensed simultaneously. The ontrast map of the two images are omputed and ompared pixel by pixel to produe a dense depth map. The illumination pattern and the fous operator to extrat the ontrast map are designed to ahieve nest spatial resolution of the omputed depth map and to maximize response of the fous operator. As the algorithm uses only loal operations suh as onvolution and lookup table, the depth map an be omputed rapidly on a data-ow image proessing hardware. As this projets an illumination pattern and detets the two images with dierent fous setting from exatly the same diretion, it does not share the problem of shadowing and olusion with triangulation based method and stereo. It's speed and auray are demonstrated using a prototype system. The prototype generates range maps at 3 frame/se with a depth resolution of.3% relative to the objet distane. The proposed sensor is omposed of o-the-shelf omponents and outperforms ommerial range sensors through its ability to produe omplete three-dimensional shape information at video rate. Keywords: range sensor, fous, depth from defous, teleentri optis, pipeline proessor, real-time sensor 1 INTRODUCTION For appliations suh as objet reognition, automati CAD model generation and remote visualization, a range sensor whih produes fast and dense depth maps is neessary. In the past, many tehniques for range sensing have been proposed, 11 whih an be ategorized into passive tehniques whih donotuseanative illumination or ative tehniques whih use an ative illumination. Passive tehniques suh as stereo and shape from motion are based on orrespondene mathing between two or more images. From the disparity or the motion vetor extrated from this orrespondene mathing, one an get the range data of the sene. The orrespondene mathing is omputationally expensive. These tehniques also suer from the olusion problem, i.e. one annot get depth data for areas in the sene whih are visible to only one of the amera. Another passive tehnique is depth from fous/defous. Depth from fous uses a sequene of images taken by inrementing the fous setting in small steps. For eah pixel, the fous setting that maximizes image ontrast is determined. This in turn an be used to ompute the depth of the orresponding sene point. 8,11,14,15,4,25 In ontrast, depth from defous uses only two images with dierent optial settings. 18,5,2,22,26 Though depth from fous/defous does not have the problem of olusion, it is also omputationally expensive to get a reliable depth map. 26 This is beause the texture of the objet has a variety of spetrum distribution, and one must analyse arefully to get a reliable fous estimate. Another draw bak that is ommon to the passive tehniques is that one annot ompute depth for sene areas without texture.

2 Popular ative tehniques are based on the priniples of strutured light and time of ight. Popular strutured light methods inlude light striping method, 1 moire interferometry 6 and Fourier-transform prolometry. 23 They are based on triangulation and determines the depth from the deformation of the image of the projeted pattern. They provide reasonable auray. For light striping method, one must projet many sets of light stripe pattern to enode the stripes in order to disriminate the stripes. This makes sensing time long, whih implies that the sene must be stati during the sensing. New hope for light stripe range nding has been instilled by advanes in VLSI. Based on the notion of ell parallelism, 12 a omputational sensor is developed where eah sensor element reords a stripe detetion time-stamp as a single laser stripe sweeps the sene at high speed. Depth maps are produed in as little as 1 mse, though urrent VLSI density limits the total number of ells, and hene spatial depth resolution, to Future advanes in VLSI are expeted to yield high-resolution depth maps at high speeds. For moire interferometry and Fourier-transform prolometry, one needs only one image but the sene should not have a steep depth gap, as it must keep trak of the fringe order. In addition, the depth it gives is just a relative depth, not absolute depth. Another type of ative method, time of ight, uses a modulated laser beam and measures the time for the light toomebak from the objet surfae. 11 Although this method is suitable at getting a rough depth map for relatively far senes, it takes a long time to get a dense depth map as it sans the sene pointbypoint. A sensor whih uses fous error information and ative illumination has been proposed by Rioux et al. 2 and Pentland et al. 19 They projet a matrix of dots 2 or light-stripe pattern. 19 Using the phenomenon that the diameter or the width of the defoused dot or strip gets larger when it is defoused, this dimension is measured from the image and is onverted into a depth value. These sensors are able to detet the depth map of a dynami sene in real-time. However, as they use a oarse matrix of dots or oarse light-stripe pattern, resulting spatial resolution and depth auray are insuient for most real-world appliations. Our approah uses o-axial projetion of a ne illumination pattern onto the sene and detets two images with two CCD sensors that have dierent fous settings. A fous operator, i.e. narrow-band-pass onvolution lter is newly designed to provide estimates of the defous of the projeted illumination pattern. The operator is derived by areful modeling of the illumination, blurring and image sensing and is tuned to respond to the fundamental frequeny of the projeted illumination pattern. The fous operator is applied to the two images to obtain two dierent fous measures at eah image point. The relative defous of eah image point maps to a unique depth estimate. The omputation of depth is a loal operation, whih enables us to realize a frame-rate range sensor. Sine the illumination pattern and the tuned ontrast operator were designed to maximize depth auray and resolution, the sensor produes depth maps of high quality. The o-axial illumination and imaging also results in a shadowless image all surfae regions that are visible to the sensor are also illuminated. A prototype real-time fous range sensor has been developed. Figure 1 shows two brightness images and the omputed depth map of a up with milk owing out of it. Strutures of suh dynami senes an only be reovered by a high-speed sensor. In the previous paper, 17 the authors have disussed this sensor mainly on the basi onept inluding illumination pattern design and the usage of onstant magniation optis. In this paper, we fous on the ontrast operator and real-time depth omputation. The performane of the sensor is demonstrated through several experiments onduted on omplex senes. Quantitative results on the auray, repeatability, and linearity of the sensor are inluded. 2.1 Basi onept 2 DEPTH FROM DEFOCUS Fundamental to depth from defous is the relationship between foused and defoused images. 1 Figure 2 shows the basi image formation geometry. Alllightrays that are radiated by objet point P and pass the aperture A are refrated by the lens to onverge at point Q on the image plane. For a thin lens, the relationship between the objet distane d, foal length of the lens f, and the image distane d i is given by the lens law: 1 d + 1 d i = 1 f : (1) Eah point on the objet plane is projeted onto a single point on the image plane, ausing a lear or foused image I f to be formed. If, however, the sensor plane does not oinide with the image plane and is displaed

3 from it, the energy reeived from P by the lens is distributed over a path on the sensor plane. The result is a blurred image of P. It is lear that a single image does not inlude suient information for depth estimation as two senes defoused to dierent degrees an produe idential images. A solution to depth is ahieved by using two images I 1 and I 2 separated by a known physial distane. 18,21 The problem is redued to analyzing the relative blurring of eah sene point inthetwo images and omputing the distane of its foused image. Then, using d i = ;, the lens law (1) yields depth d of the sene point. Simple as this proedure may appear, several tehnial problems emerge when implementing an algorithm. These inlude (a) aurate estimation of relative defous in the two image, (b) reovery of textured and textureless surfaes, and () ahieving onstant magniation that is invariant to the degree of defous. 2.2 Teleentri optis We begin with the last of the problems mentioned above. In the imaging system shown in Figure 2, the eetive image loation of point P moves along the prinipal ray R as the sensor plane is displaed. This auses a shift in image oordinates of the image of P. This variation in image magniation with defous manifests as orrespondene like problem in depth from defous as the right set of points in images I 1 and I 2 are needed to estimate blurring. We approah the problem from an optial perspetive rather than a omputational one. Consider the image formation model shown in Figure 3. The only modiation made with respet to the model in Figure 2 is the use of the external aperture A. The aperture is plaed at the front-foal plane, i.e. a foal length in front of the prinipal point O of the lens. This simple addition solves the prevalent problem of magniation variation with distane of the sensor plane from the lens. Simple geometrial analysis reveals that a ray of light R from any senepoint that passes through the enter O of aperture A emerges parallel to the optial axis on the image side of the lens (see book. 13 ) As a result, despite blurring, the eetive image oordinates of point P in both images I 1 and I 2 are the same as the oordinate of its foused image Q on I f. The detailed disussion of this is found in the tehnial report Defous funtion and depth estimation The defous funtion is desribed in detail in previous work. 1,9 As in Figure 3, let be the distane between the foused image of a surfae point and its defoused image formed on the sensor plane. The light energy radiated by the surfae point and olleted by the imaging optis is uniformly distributed over a irular path with a radius of a =f on the sensor plane. y This path, also alled the pillbox, is the defous funtion h(x y) =h(x y a f)= f 2 a 2 ( f p x2 + y 2 2a 2 ) (2) where a is the radius of the teleentri lens aperture, and (r) is a retangular funtion whih takes a value 1 for jrj < 1, otherwise. In Fourier domain, the above defous funtion is given by: 2 H(u v) =H(u v a f)= f a p u 2 + v J 1( 2a 2 f p u 2 + v 2 ) (3) where J 1 is the rst-order Bessel funtion. As is evident from the above expression, defous serves as a low-pass lter. The bandwidth of the lter inreases as dereases, i.e. as the sensor plane gets loser to the plane of fous. Figure 4 visualizes the above disussion (a) is the image i(x y) at the foused plane, I f, and its Fourier spetrum I(u v). When the sensor plane is displaed to I 1, the defoused image is the onvolution of the foused image i(x y) with the pill-box h 1 (x y) as in (b). In the Fourier domain, it is the produt of Fourier spetrum of the foused image I(u v) andthefourier transform of the pill-box H 1 (u v). () is the equivalent set to (b) when the sensor is plaed at I 2, i.e. at distane ; from the foused plane I f. As the image is defoused more, the low-pass response of the transfer funtion H 2 (u v) is more notable. Sine we have used the teleentri lens (Figure 3) in our implementation, it's parameters are used in the model. However, the following expressions an be made valid for the onventional lens model (Figure 2) by simply replaing the fator f a by d i a. f addition, the nominal F-number of the lens equals. 2a y This geometri model is valid as far as the lens is not exatly foused and the aberration is small ompared to the radius a =f. 1 In

4 2.4 Ative illumination If one an get the amplitudes g 1 and g 2 of the spetrum of the two defoused images at a predened frequeny as in Figure 4, one an get the depth estimate from g 1 and g 2. Thisisdoneby applying a onvolution operator to the images. But this is not trivial sine the image texture inludes all kinds of frequeny. Unertainty relation 3 tells us that, when we try to ondut a frequeny analysis for a small area, the frequeny resolution redues proportionally to the inverse of the area size. To get a dense depth map, one must get the g 1 and g 2 for a very small area around eah pixel. But this means the operator output is atually an averaged spetrum over a wide band of frequeny. As the response of the defous funtion H depends not only on defous but also on the texture frequeny, this band width of the operator auses error in depth value. If the texture has only one frequeny, the problem is solved. This is the reason why we have introdued ative illumination. The projetion lter pattern has been designed to ahieve nest spatial resolution of the omputed depth map and to maximize response of the fous operator (see papers. 17,16 ) The resulting pattern is a hekerboard pattern with a horizontal period of t x and a vertial period of t y suh that t x =4p x t y =4p y (4) where p x and p y are the CCD pixel pith in horizontal and vertial diretion, respetively. The horizontal and vertial spaing between neighboring elements of the disrete Laplaian kernel (q x q y ) that orresponds to the optimal pattern obeys q x =2p x q y =2p y (5) This means the 33 Laplaian operator kernel has zeros between eah element, and it is atually a 55 kernel. z Figure 5 shows the eet of pattern projetion. (a) is a image of a sene under normal lighting and its spetrum. (b) is the image of the same sene under the oaxial pattern projetion and its spetrum. The spetrum in (b) shows that projeted pattern reates strong peaks in the spetrum at positions (1=t x 1=t y ). 2.5 Depth from two images Now let us introdue following normalized ratio q(x y) = g 1(x y) ; g 2 (x y) g 1 (x y) +g 2 (x y) = H( 1 1 ) ; H( 1 1 ; ) tx ty tx ty H( 1 1 ) +H( 1 1 ; ) tx ty tx ty (6) Here, g 1, g 2 and q are funtions of image oordinate (x y). As shown in Figure 6, q is a monotoni funtion of suh that ;p q p and p 1. This monotoni response is obtained as far as and a are hosen so that the analyzed frequeny (1=t x 1=t y ) is within the main lobe of the defous funtion H s 1 t x t y 2 < :61 f a (7) In pratie, the above relation an be preomputed and stored as a look-up table that maps q omputed at eah image point to a unique. Sine represents the position of the foused image, the lens law (1) yields the depth d of the orresponding sene point. 3.1 Design of the kernel 3 TUNED FOCUS OPERATOR For the purpose of illumination optimization, we used the Laplaian operator as is desribed in the previous papers. 17,16 The resulting illumination pattern has only a single dominant absolute frequeny, (1=t x 1=t y ). Given z In the papers, 17,16 wehaveshown another hekerboard pattern that is not used for the implementation, where (t x t y)=2(p x p y) and (q x q y)=(p x p y). However, this pattern requires perfet registration between illumination pattern and sensor pixel. It is beause the fous measure also depends on the phase between pattern and pixel. In this ase, as the peak frequeny is at the Nyquist frequeny, the phase error annot be ompensated using quadrature operation whih will be desribed in setion 3.2.

5 this, we are in a position to further rene our fous operator so as to minimize the eets of all other frequenies aused either by thephysial texture of the sene or image noise. To this end, let us onsider the properties of the 33 disrete Laplaian (see Figure 7(a) and (b)). We see that though the Laplaian does have peaksat (1=t x 1=t y ), it has a fairly broad bandwidth allowing other spurious frequenies to ontribute to the fous measure. Here, we seek a narrow band operator with sharp peaks at the above four oordinates in frequeny spae. Given that the operator must eventually be disrete and of nite support, there is a limit to the extent to whih it an be tuned. To onstrain the problem, we impose the following onditions. (i) To maximize spatial resolution in omputed depth we fore the operator kernel to be 33 or44. x This is also a requirement from the onvolution hardware of the pipeline proessor we use, whih an exeute up to 88 onvolution. (ii) Sine the fundamental frequeny of the illumination pattern has a symmetri quadrapole arrangement, the fous operator must be either reetion-symmetri or anti-reetion-symmetri about vertial and horizontal axis. (iii) The operator must not respond to any DC omponent in image brightness. This last ondition is satised if the sum of all elements of the operator equals zero. If we use33 operator, ondition (ii) fores the operator to have the struture shown in Figure 7(), and ondition (iii) beomes a +4b +4 = (8) It is also imperative that the response of the operator L(u v) = a +2b ( os 2q x u + os 2q y v )+4 os 2q x u os 2q y v: (9) is not zero at the fundamental frequeny, i.e. L( 1 tx 1 ty ) 6=. This redues to: a ; 4b +4 6= (1) Expressions (8) and (1) imply that b 6=. Without loss of generality, we set b = ;1. Hene, (8) gives a = 4(1;). Therefore, the tuned operator is determined by a single unknown parameter,, as shown in Figure 7(d). The problem then is to nd suh that the operator's Fourier transform has a sharp peak at (1=t x 1=t y ). A rough measure of sharpness is given by the seond-order moment ofthepower jj L(u v) jj 2 with respet to (1=t x 1=t y ): M = = jj L( 1 tx 1 1 ty ) jj 2 Z 2 tx u= Z 2 ty v= [(u 1 ; ) 2 +(v 1 ; ) 2 ] jj L(u ; v; ) jj dv du (11) t x t y t x t y t 2 x + t 2 y ( ; ; 93) t 3 x t 3 y The above measure is =, i.e. when =:658 as shown in Figure 7(e). The resulting tuned fous operator has the response shown in Figure 7(f). It has substantially sharper peaks than the disrete Laplaian. We have also solved an optimization problem for 44 kernel ase. This time it beomes a twoparameter minimization problem after onsidering the symmetri property. The resultant kernel and its spetrum response is shown in Figure 7 (f) and (g), respetively. The above derivation brings to light the fundamental dierene between designing tuned operators in ontinuous and disrete domains. In general, an operator that is deemed optimal in ontinuous domain is most likely sub-optimal for disrete images. 3.2 Quadrature operation As was disussed above, the fous operator passes the spetrum at the frequeny (1=t x 1=t y ) and stops DC spetrum omponent. Let's denote the spetrum of the hekerboard illuminated image (Figure 5 (d)) as G (u v) =gf(u 1 1 ; v; )+(u v; )+(u 1 ; v+ 1 )+(u + 1 v+ 1 )g: (12) t x t y t x t y t x t y t x t y If the operator gain at the illumination frequenies (1=t x 1=t y )is, operator output is the inverse Fourier transform of G (u v) g (x y) =4g os 2 1 x os 2 1 y: (13) t x t y x Here, 33 or44 ounts the pixels with non-zero kernel values. Atual kernel has zero kernel element in-between, resulting in a kernel with a size of 55 or77.

6 Atual values that the disrete fous operator gives are the g (x y) values at disrete sampling positions ( x + mp x y + np y ) g d (m n) =g ( x + mp x y + np y )=4g os ( m x ) os ( n t x y ) (14) t y where m and n are pixel indexing integer values and ( x y ) is the relative shift between illumination and CCD pixel. Here the relationship of equation (4) was used. Equation (14) shows that the operator output g d (m n) is sensitive to the registration, ( x y ). To ope with this problem, we use the fat that g d (m n) at the next pixel has a =2 phase dierene, then It is easily shown that g = 1 p gd (m n) g d (m +1 n) 2 + g d (m n +1) 2 + g d (m +1 n+1) 2 (15) Here we get a fous measure g whih is insensitive to the sub-pixel order mis-registration. This quadrature operation loosens the requirement for the auray to register the CCD's and the illumination pattern, making pixel-order registration enough. 4 REAL TIME RANGE SENSOR Based on the above results, we have implemented the real-time fous range sensor shown in Figure 8. The sene is imaged using a standard 12.5 mm Fujinon lens onverted to teleentri with an additional aperture inside. Aperture diameter is set so that F-number is 6.5. Light rays passing through the lens are split in two diretions using a beam-splitting prism. This produes two images that are simultaneously deteted using two Sony XC- 77RR CCD ameras. The positions of the two ameras are preisely xed suh that one obtains a near-fous image while the other a far-fous image. In this setup a physial displaement of.25mm between the eetive foal lengths of the two CCD ameras orresponds to a sensor depth of eld of 257 mm (a detetable range of mm.) This detetable range of the sensor an be varied by hanging the sensor displaement and the fous distane of the lens. F-number of the optis should be hosen to fulll equation (7). The hekerboard illumination pattern was ethed on a glass plate using mirolithography. The lter was then plaed in the path of a 3 W Xenon ar lamp. The illumination pattern is projeted using a teleentri lens idential to the one used for image formation. A half-mirror is used to ensure that the illumination pattern projets onto the sene via the same optial path used to aquire images. As a result, the pattern is registered with respet to the pixels of the two CCD ameras. Furthermore, the above arrangement ensures that every sene point that is visible to the sensor is also illuminated by it,avoiding shadows and thus undetetable regions. If objets in the sene have a strong speular reetion omponent, ross-polarized lters an be attahed to the illumination and imaging lens to lter out speularities and produe images that mainly inlude the diuse reetion omponent. Images from the two CCD ameras are digitized and proessed using MV2 Dataube image proessing hardware. The present onguration inludes two A/D onverters, one 12-bit onvolver (maximum kernel size:88,) one arithmeti logi unit (ALU) and one 16-bit look up table, whih an be aligned on a pipeline. Data in the pipeline ow at 2MHz. The pipeline also requires 1 2mse as an overhead. For example, for a pixel image, the pipeline is ompleted in about 14.3 mse. This hardware enables simultaneous digitization of the two images, onvolution of both images with the tuned fous operator, and the omputation of a depth map, all within a single frame time of 33 mse with a lag of 33 mse. Figure 9 shows the data ow. The rst and the seond pipeline input the near foused (i 1 ) and far foused (i 2 ) images, respetively, and exeute onvolution with the tuned fous operator and quadrature operation to produe the fous measure image of resolution. Eah pipeline takes 14.2 mse. The fous measure image is sub-sampled in the third pipeline to a resolution of and input to the 16-bit look up table. The look-up table is ongured to take two 8-bit inputs and map eah pair of fous measures (g 1 and g 2 ) to a unique depth estimate d. Here, the normalized ratio of fous measures q in equation (6) is not output. Instead, the depth value d or d i is diretly output. Then the depth map goes through linear and non-linear smoothing and pixel-by-pixel linear alibration in a single pipeline as the look up table, whih takes 4.8 mse. The above three pipelines produe a depth map in 33 mse in total. Image grabbing of the near and far images for the next depth omputation is aomplished parallelly with the above three pipelines.

7 A pixel-by-pixel linear alibration is exeuted to ompensate for the image urvature and vignetting. Image urvature auses an oset of the depth value. Vignetting hanges the depth detetion gain. A planer target is plaed perpendiularly to the optial axis of the sensor at a far position z 1 and a near position z 2 in the ranging depth. Then the depth maps are deteted and smoothed using a spline funtion. Let us denote the smoothed depth map when the target is at z 1 by z d1 (x y). Similarly denote the smoothed depth map when the target is at z 2 by z d2 (x y). Then the alibration gain map in Figure 9 is omputed by The alibration oset map is omputed by z 2 ; z 1 z d2 (x y) ; z d1 (x y) : (16) z 1 z d2 (x y) ; z 2 z d1 (x y) z d2 (x y) ; z d1 (x y) : (17) The sub-sampling for the third pipeline is merely beause of the time restrition. Instead, by giving up simultaneous grabbing of the near and far images, a depth map an be omputed at the same rate if the two images are taken in suession. Still, simultaneous image aquisition is learly advantageous sine it makes the sensor less sensitive tovariations in both illumination and sene struture between frames. With an addition of one more MV2 to the present proessing hardware, it is easy to obtain depth maps at 3 Hz using simultaneous image grabbing. Depth maps produed by the sensor are shipped via video able and visualized as wire-frame plots with 86 meshes at a speed of 18 frame/se on a DEC Alpha workstation. 5 EXPERIMENTS Numerous experiments have been onduted to test the performane of the sensor. Here we briey summarize these results. Figure 1(a) shows near and far foused images of a planar surfae, half of the surfae is textureless while the other half has strong random texture. A omputed depth map of the surfae is shown in Figure 1(b). As expeted the textureless area is estimated almost free of errors while the textured area has small errors due to texture frequenies that lie lose to the illumination frequeny. Several depth maps of the plane in Figure 1(a) were omputed by varying its position in the 25 mm workspae of the sensor. Relative auray and repeatability of the sensor were estimated for both simultaneous and suessive image grabbing ongurations (see Table 1()). The onventional denition of relative auray and repeatability is used, where it is a value relative to the objet distane. Auray is measured by the tting error to a plane. Repeatablity is measured by the standard deviation of the depth values measured at the same position at dierent times. Here we disuss absolute auray (linearity.) Figure 11(a) is the plot of deteted depth for the textured planer target vs. the target depth. Deteted depth is determined by a plane tting to a 55 pixel area in the depth map. The tting errors are also shown as 6 error bars. The deviation from the linearity is 5.2mm (.) The slight urvature is due to the error in the optial parameter. After a alibration of the look up table using a quadrati funtion, the linearity isimproved as is shown in Figure 11(b). The linearity is 2.5 mm. Figure 12 is the same pair of plots as Figure 11, exept that the deteted depth is determined by an average of 15 suessive depth value at a pixel. The linearity is 1. mm with alibration. These results learly demonstrate the superior performane of the sensor over previous implementations of depth from defous. Figure 13 shows a sene with polyhedral objets. The omputed depth map in Figure 13(b) is fairly aurate despite the omplex textural properties of the objets. The only ltering that is applied to the depth map is a 55 smoothing funtion to redue high frequeny noise in omputed depth that results from the low signal-tonoise ratio of the CCD ameras and spurious frequenies aused by surfae texture. All surfae disontinuities and orientation disontinuities are well preserved. The reovered shapes are preise enough for a variety of visual tasks inluding reognition and inspetion. Similar results are shown in Figure 14 where shapes of urved objets are reovered. In the ase of dynami senes, struture an be estimated only by using a real-time sensor. Figure 15 shows an objet's depth map omputed as it rotates on a motorized turntable. Suh depth map sequenes are valuable for automati CAD model generation from sample objets. Computed CAD models are useful not only for visual reognition tasks but also for graphis rendering. In both ases, objet models are more often than not manually designed and input to the system, a proess that is not only tedious but also impratial for

8 I Texture Spetrum i :Texture FT (a) Image texture and its Fourier transform fr (a) x h1 α a'/f f /πα a' y FT H1 Defous Funtion I1 = H1 I.61 f /α a' (b) Pill-box defous model and the Fourier transform of the blured image I1 g1 fr h f /π(β α) a' FT H2 Defous Funtion I2 = H2 I x (β α)a' /f y.61 f /(β α) a' () Pill-box defous model and the Fourier transform of the blured image I2 Figure 4: Image blaring and fous measure. g2 fr (b) Figure 1: (a) Two images of a sene taken using dierent fous settings. (b) A depth map of the sene omputed in 33 mse by the fous range sensor. I 1 I f I 2 α β γ A P R f O a Q d i Figure 2: Image formation and depth from defous. d (a) image under normal lighting and its power spetrum I 1 I f I 2 α β γ A' P Q f R' O f a' O' d i Figure 3: A onstant-magniation imaging system for depth from defous is ahieved by simply plaing an aperture at the front-foal plane of the optis. d (b) image under pattern projetion and its power spetrum Figure 5: Eet of ative pattern projetion.

9 g q 1 _ g = 2 1. g 1 + g 2 p.5. β α SOURCE -.5 CCD2 CCD1 FILTER -p. Figure 6: Relation between fous measures g 1 and g 2 and the defous parameter. 4 L v PRISM IMAGING OPTICS (a) (b) Figure 8: (a) The real-time fous range sensor and its key omponents. (b) The sensor an produe depth maps up to in resolution at 3 Hz. 1 u (a) b (b) b M a b () b (e) (d) (g) (1-) Figure 7: (a) The 33 Laplaian and its (b) Fourier transform. () The kernel struture for a 33 symmetri operator. (d) The kernel of a 33 operator that dose not pass DC omponent (see text). (e) The seond moment M of eah of the operator peaks is minimized when =:658. (f) Response of the tuned fous operator ( =:658) has muh sharper peaks than the Laplaian. (g) 44 optimal operator. (h) Response of (g) L L v (f) v (h) u u 2 Op x y overflow flag Digitizer i1 Conv. Mult. Add. Add. 16-bit LUT g1 fous operator Op. 1 line delay 2 Mult. Op x y Delay 1 pixel delay Opx y + Opx y + Opx y + Opx y sqrt Offset : frame memory Gain : proessing element : proessing pipe-line 2 Op x y overflow flag Digitizer i2 Conv. Mult. Add. Add. 16-bit LUT g2 fous operator sqrt Op. 16-bit Depth=Lut(g, g1) LUT ALU 1x3 median filter or interpolation Mult. Add. Conv. Depth point by point linear alibration Smoothing Mult. 1 line delay 2 Op x y Delay 1 pixel delay Opx y + Opx y + Opx y + Opx y Figure 9: Dataow for the real-time depth omputation

10 (a) (b) Simulatneous Image Grab Suessive Image Grab (a) Depth Auray (rms) Repeatability (rms) Spatial Resolution.24 %.34 %.23 %.29 % 256 x x 48 Speed 3 Hz 3 Hz Delay 33 mse 33 mse () Figure 1: (a) Near foused image of a planar surfae that inludes highly textured and textureless areas. (b) Depth of the surfae omputed using the fous range sensor. () Performane harateristis of the sensor. (b) Figure 13: (a) Near and far foused images of a set of polyhedral objets. (b) Computed depth map. deteted depth (mm) average of 5x5 depth values at the image enter origin:56mm from the sensor error bar : +/-3 σ deteted depth (mm) average of 5x5 depth values at the image enter origin:56mm from the sensor error bar : +/-3 σ -5-5 fitting σ=5.2mm target depth (mm) (a) without alibration Figure 11: Auray and linearity fitting σ=2.5mm target depth (mm) (b) after alibration (a) 3 25 average of 15 depth values at the enter pixel origin:56mm from the sensor error bar : +/-3 σ 3 25 average of 15 depth values at the enter pixel origin:56mm from the sensor error bar : +/-3 σ deteted depth (mm) deteted depth (mm) fitting σ=3.8mm target depth (mm) (a) without alibration -5-5 fitting σ=.97mm target depth (mm) (b) after alibration Figure 12: Repeatability and linearity (b) Figure 14: (a) Near and far foused images of a set of urved objets. (b) Computed depth map.

11 (a) (b) () (d) (e) (f) (a) left-eye image (b) right-eye image Figure 16: Stereo-pair of texture-mapped images are synthesized from the deteted image and depth map. (g) (h) (i) Figure 15: Depth maps generated by the sensor at 3 Hz while an objet rotates on a motorized turntable. large numbers of omplex objets. Figure 16 is an example of graphis rendering. Sine this sensor gives both depth map and image data, We an symthesize images viewed from diretions that are dierent from the sensing diretion. Furthermore, real-time depth omputation learly enhanes the apability of any vision system as it enables reovery of a deforming shape, preise traking of moving objets, and robust navigation in dynami senes. 6 SUMMARY In order to get a dense and aurate depth map at frame rate for both textured and textureless surfae, we have inorporated o-axial ative pattern projetion to depth from defous method. The projetion pattern and fous operator to extrat the ontrast of the projeted pattern has been designed through areful modeling of the optis, sensing and proessing. To solve the pixel order registration problem between the image sensors, we have introdued a teleentri optis for onstant magniation. To solve the sub-pixel-order registration problem between the two image sensors and the illumination pattern, we have introdued quadrature operation whih is applied after the fous operator. All of these results were used to implement a real-time fous range sensor that produes high resolution depth maps at frame rate. This sensor is unique in its ability to produe fast, dense, and preise depth information at a very low ost. With time we expet the sensor to nd appliations ranging from visual reognition and robot ontrol to automati CAD model generation for visualization and virtual reality. 7 ACKNOWLEDGEMENTS This researh was onduted at the Center for Researh in Intelligent Systems, Department of Computer Siene, Columbia University. We would like to thank Dr. Yasuo Nakagawa at Hitahi Ltd. for his enouragement to this researh. 8 REFERENCES [1] M. Born and E. Wolf. Priniples of Optis. London:Permagon, 1965.

12 [2] V. M. Bove, Jr. \Entropy-based depth from fous". Journal of Optial Soiety of Ameria A, 1:561{566, April [3] R. N. Braewell. The Fourier Transform and Its Appliations. MGraw Hill, [4] T. Darrell and K. Wohn. \Pyramid based depth from fous". Pro. of IEEE Conf. on Computer Vision and Pattern Reognition, pages 54{59, June [5] J. Ens and P. Lawrene. \A matrix based method for determining depth from fous". Pro. of IEEE Conf. on Computer Vision and Pattern Reognition, pages 6{69, June [6] J. J. Gibson. The senses onsidered as pereptual systems. Houghton Miin, Boston, [7] A. Gruss, S. Tada, and T. Kanade. \A VLSI smart sensor for fast range imaging". Pro. of ARPA Image Understanding Workshop, pages 977{986, April [8] B. K. P. Horn. \Fousing". Memo 16, AI Lab., Massahusetts Institute of Tehnology, Cambridge, MA, USA, [9] B. K. P. Horn. Robot Vision. The MIT Press, [1] S. Inokuhi, K. Sato, and F. Matsuda. \Range imaging system for 3-d objet reognition". Pro. of 7th Intl. Conf. on Pattern Reognition, pages 86{88, July [11] R. A. Jarvis. \A perspetive on range nding tehniques for omputer vision". IEEE Trans. on Pattern Analysis and Mahine Intelligene, 5(2):122{139, Marh [12] T. Kanade, A. Gruss, and L. R. Carley. \A very fast VLSI rangender". Pro. of Intl. Conf. on Robotis and Automation, pages 1322{1329, April [13] R. Kingslake. Optial System Design. Aademi Press, [14] E. Krotkov. \Fousing". Intl. Journal of Computer Vision, 1:223{237, [15] S. K. Nayar and Y. Nakagawa. \Shape from fous: An eetive approah for rough surfaes". IEEE Trans. on Pattern Analysis and Mahine Intelligene, 16(8):824{831, August [16] S. K. Nayar, M. Watanabe, and M. Noguhi. \Real-time fous range sensor". Tehnial Report CUCS-28-94, Dept. of Computer Siene, Columbia University, New York, NY, USA, November [17] S. K. Nayar, M. Watanabe, and M. Noguhi. \Real-time fous range sensor". Pro. of Intl. Conf. on Computer Vision, pages 995{11, June [18] A. Pentland. \A new sense for depth of eld". IEEE Trans. on Pattern Analysis and Mahine Intelligene, 9(4):523{531, July [19] A. Pentland, S. Sherok, T. Darrell, and B. Girod. \Simple range ameras based on foal error". Journal of Optial Soiety of Ameria A, 11(11):2925{2935, November [2] M. Rioux and F. Blais. \Compat three-dimentional amera for roboti appliation". Journal of Optial Soiety of Ameria A, 3(9):1518{1521, September [21] M. Subbarao and G. Surya. \Appliation of spatial-domian onvolution/deonvolution transform for determining distane from image defous". Pro. of SPIE: Optis, Illumumination, and Image Sensing for Mahine Vision VII, 1822, November [22] G. Surya and M. Subbarao. \Depth from defous by hanging amera aperture: A spatial domain approah". Pro. of IEEE Conf. on Computer Vision and Pattern Reognition, pages 61{67, June [23] M. Takeda, H. Ina, and S. Kobayashi. \Fourier-transform method of fringe-pattern analysis for omputerbased topography and interferometry". Journal of Optial Soiety of Ameria A, pages 156{16, January [24] M. Watanabe and S. K. Nayar. \Teleentri optis for onstant-magniation imaging". Tehnial Report CUCS-26-95, Dept. of Computer Siene, Columbia University, New York, NY, USA, September [25] R. G. Willson and S. A. Shafer. \Modeling and alibration of automated zoom lenses". Tehnial Report CMU-RI-TR-94-3, The Robotis Institute, Carnegie Mellon University, Pittsburgh, PA, USA, January [26] Y. Xiong and S. A. Shafer. \Moment lters for high preision omputation of fous and stereo". Pro. of Intl. Conf. on Robotis and Automation, pages 18{113, August Also, Tehnial Report CMU-RI-TR-94-28, Pittsburgh, PA, USA, September, 1994.