17 Internatonal Conference on Mathematcs, Modellng and Smulaton Technologes and Applcatons (MMSTA 17) ISBN: 978-1-6595-53-8 Dstance Calculaton from Sngle Optcal Image Xao-yng DUAN 1,, Yang-je WEI 1,,* and Ke ZHANG 1 1 College of Computer Scence and Engneerng, Northeastern Unversty, Wenhua Str.3, 11819, Shenyang, Chna State Key Laboratory of Synthetcal Automaton for Process Industres *Correspondng author Keywords: Dstance calculaton, Sngle mage, Oblque straght lne, Slope. Abstract. Accurate and fast dstance calculaton wth respect to a sngle optcal mage s useful for real-tme 3D constructon and acquston, however currently rare dstance calculaton methods theoretcally base on an optcal sngle mage, and the tradtonal dstance calculaton method wth a sngle mage has lmtatons due to assumpton that the step edge n the mage must be strctly horzontal or vertcal, whch s dffcult to fulfll n real applcatons because the slope of a practcal edge could be any other value except for zero and nfnty. In ths paper, a dstance calculaton method wth a sngle defocused mage contanng an oblque step edge s proposed, and no specal camera equpment or unque external condton s requred. Frst, the basc theory of coordnate system rotaton has been ntroduced to smplfy the samplng process of neghbor ponts and elmnate the nfluence resulted from slope of the step edge; one-dmenson pont-spread-functon of the orgnal dstance calculaton method s expended to two-dmenson, takng nto account coordnate transform, and a comprehensve dstance calculaton based on an oblque step edge wth any slope s deduced; The relatonshp between the precson of dstance calculaton and the slope of the step edge s analyzed and proved n theory. Fnally, a seral of smulaton are conducted to valdate our proposed dstance calculaton method, and the expermental results prove the applcablty, valdty and hgh precson of our method. Introducton Real-tme dstance acqurement wth respect to optcal mages s an mportant research feld n many applcatons, such as unmanned surface vehcles (USVs), envronmental montorng, and medcal operatons, to realze autonomy of the used systems. There are many methods to estmate dstance or depth nformaton from D optcal mages, ncludng depth from stereo (DFS), depth from focus (DFF), depth from defocus (DFD) and so on. Compared to DFS and DFF, DFD requres less mages, no match and occluson between mages, and the operaton s smple. The tradtonal DFD methods requre at least two defocused mages wth dfferent camera parameters to calculate depth nformaton usng the relatonshp between burrng degree and depth [1-4]. Therefore, two cameras wth dfferent camera parameters or a camera wth two parameter adjustment sets are requred. However, t s dffcult to modfy the camera parameters durng mage capturng when a real system s operatng. Even f two or more mages are obtaned by two cameras, a tme algnment s requred to assure that these defocused mages are captured at the same tme. Therefore, tradtonal DFD methods are dffcult to be used n real-tme dstance calculaton. In order to calculate dstance for real-tme applcatons, to smplfy tradtonal DFD methods n means of mprovement on hardware or software s wdely researched n recent years. In 1996, Nayar proposed a depth reconstructon method based a sngle optcal mage wth scene lghtng assstance [5], and n 14, Ln proposed the coded aperture method to modfy the camera lens [6]. However, these methods requre to mprove the hardware or add external settngs, therefore ther applcaton s greatly lmt. In recent years, a new method was proposed by Zhou and Cao [7-9], n whch only a 75
defocused mage s captured and the second defocused mage s synthetzed wth a known defocus degree as expected, then the dstance nformaton n the orgnal mage s calculated based on tradtonal DFD fundamentals. However, ths method requres an addtonal step to synthetze the second defocused mage after the frst defocused mage s captured, therefore theoretcally t s not a dstance reconstructon method wth a sngle mage, and of course t stll can not be used n a real-tme dynamc applcaton. Dstance calculaton from a sngle optcal mage wth a step edge s frst proposed by Pentland n 1987[5]. Ths method estmates the blurrng degree by comparng the ntensty dstrbuton on both sdes of a step edge, and obtans dstance between the edge and the camera wth the relatonshp between blurrng degree and dstance. The algorthm s based on an assumpton that the step edge n the mage s horzontal or vertcal. However, n dstance calculaton of real systems whose applcaton envronment s complcated, t s dffcult to assure that the step edge n a dynamc mage s strctly vertcal or horzontal. Therefore, the applcaton felds of ths orgnal method are lmt. In order to calculate dstance nformaton wth a sngle optcal mage captured by a real-tme system, the paper expands the theores of the orgnal dstance calculaton method and solves the problem when the step edge s oblque and the pont-spread-functon s two-dmensonal spreadng, and then proposes an mproved dstance calculaton method based on a sngle mage where the step edge could be postoned at dfferent postons and wth dfferent slopes. Compared to the orgnal method, our method needs only an mage wth a step edge to calculate the dstance nformaton between the edge and the camera, and no assumpton about the step edge s requred. Dstance Calculaton wth a Sngle Image Imagng Model In optcal magng, the object dstance u of a source pont P, the mage dstance v and the focal length f meet the Gauss magng formula shown as Eq. (1), the mage of P s a pont, whch s also called focused mage. If any parameter n Eq. (1) s changed, the mage of P becomes a blurred round spot, called as defocused mage. 1 1 1 u v f (1) Accordng to the prncple of lght propagaton, the ntensty dstrbuton functon n the spot s called as pont-spread-functon (PSD), whch s approxmately equal to a two-dmensonal Gauss functon, 1 x y hx, y e () where x and y respectvely represent the horzontal and vertcal coordnates of pont(x, y) on the magng plane; σ s the varance of Gauss functon, whch s used to represent the blurrng degree. Theoretcally, a defocused mage can be consdered as the summaton of all round spots, therefore t can be expressed as, Gx, yhx, yix, y (3) where G(x, y) represents the defocused mage; I(x, y) represents the focused mage; * represents convoluton. Accordng to the geometry knowledge, the dstance between P and the lens can be calculated wth, fv d v f m F (4) 751
where m represents the coeffcent between r, radus of the defocused spot, and σ; F s the aperture number of the camera. From Eq. (4), t can be seen that wth fxed camera parameters, d s easly calculated f the blurrng degree σ s estmated precsely. In order to obtan σ of each pxel n a defocused mage, ether ts focused mage s already known, or another defocused mage of the dentcal scene wth dfferent camera parameters s used to be compared to. However, n a dynamc applcaton, t s dffcult to fulfll these requrements because there s no tme to vary camera parameters and t s mpossble to capture a defocused mage n an outdoor envronment. If there s a step edge n the defocused mage, the orgnal dstance calculaton method proposed by Pentland can be used to calculate dstance nformaton of pxels on ths edge, because the blurrng degree of each pont can be estmated through estmatng the ntensty varaton of both sdes of the edge. The basc theory s denoted as followng. Orgnal Dstance Calculaton wth a Sngle Image Suppose that the defocused mage has a vertcal step edge at pont x shown as Fg. 1, the ntensty of the step edge can be represented as, k x-x Ix, y k x-x (5) where k s the ntensty value of pont(x, y); δ represents the ntensty dfference. Fgure 1. The dagram of a vertcal edge n an mage. Fgure. The dagram of an oblque edge n a mage. Takng the Laplace transformaton n Eq. (3), we could obtan, xu yv 1 Lx, y h x, yi x, y e I u, vdudv (6) where s Laplace transformaton; L(x, y) represents the rate of ntensty-varaton at pont(x, y). When we estmate the blurrng degree of the step edge, we only need to compare the ntensty dfference between ponts on two sdes of the edge, whle the ntensty dstrbuton of the pxels along the step edge does not need to be consdered. Therefore, the PSD n Eq. () can be smplfed nto an undrectonal functon, and Eq. (6) s transformed nto, xu xx x u 3 3 x Lx, y 1 e du xx xe Takng the absolute and the natural logarthm on both sdes of Eq. (7), we obtan, ln 3 x x Lx, y ln whch can be smplfed as, A x x B C x x x x (7) (8) (9) 75
1 Lxy, where A B ln C ln. 3 x x As a lnear regresson problem about x, the coeffcents A and B can be obtaned usng maxmum lkelhood estmaton, x x xx x x x x C A B C xx (1) where x s the horzontal coordnate of neghbor ponts of x whch s on the step edge; x-x s the dstance between the sample pont x and x. x x represents the average dstance between sample ponts and x; C represents the average of C. Then we can calculate the blurrng degree wth, 1 A (11) Fnally, replace σ n Eq. (4), the dstance nformaton of each pont on the step edge can be obtaned, fv d v f mf A (1) If the step edge s horzontal, the deducton process s smlar and smple because only x s replaced by y. However, t s dffcult to use ths dstance calculaton method n real applcatons due to the assumpton that the step edge s vertcal or horzontal, whch s dffcult to assured. Therefore, n ths paper an mprovement to ths orgnal method has been proposed to expand ts applcaton areas where the step edge s oblque. Dstance Calculaton of Oblque Step Edges There s an oblque step edge n the defocused mage shown as Fg., the ntensty value of the edge can be represented as, I x, y k x-x -ty k x-x -ty (13) where 1/t s the slope of the step edge and tanθ=1/t.. In order to calculate dstance, frst the orgnal coordnate system s rotated α degrees untl the new vertcal axs y s parallel to the step edge. Accordng to the theory of coordnate system rotaton, the new coordnates (x, y ) can be denoted by the old coordnates (x, y) and a rotaton matrx, whch s show as, cos sn x sn cos y (14) x y arctan t (15) Accordng to Eq. (14), the new coordnates of pont (x, y ) can be denoted as, x xcos ysn y xsn ycos (16) Therefore, after rotaton, the ntensty value of the step edge n Eq. (13) can be represented as, 753
k xx Ix, y (17) k xx From Eq. (17), t can be seen that after coordnate system rotaton the nfluence of the slope t has been elmnated and all the calculaton equatons n orgnal method could be used here. Then, replace I(x, y) n Eq. (6) wth I(x, y ), xu yv 1 L x, y e I u, v dudv Accordng to the orgnal method n Secton, Eq. (18) can be smplfed as, (18) xx L x y x x 3 xe where x s equal to,, (19) x x cos () Takng the absolute and the natural logarthm on both sdes of Eq. (19), we obtan, 1 L x, y Axx BCxx A Bln Cln 3 x x (1) L x, y L( xcos ysn, xsn ycos ) () x x xx x x x x C A BC xx where x s the horzontal coordnate of neghbor ponts of x n the new coordnate system; x -x s the dstance between the sample pont x and x ; x x represents the average dstance between the sample pont x and x ; C represents the average of C. Replace x, y, x n A and L of Eq. (), then, A cos sn cos cos sn cos xcosysn xcos xcos ysn xcos x y x x y x C Lx ( cos ysn, xsn ycos ) C ln xcos ysn x cos Replace α wth t n Eq. (4), then A can be calculated wth t. Subsequently, the blurrng degree σ can be calculated wth coordnates n the orgnal coordnate system wth Eq. (11) and Eq. (4). That means even the step edge s oblque, ts dstance to the camera can be calculated wth our method and the calculaton equaton s denoted as, fv d v f mf A If t=,, then, x x xx C Lxy (, ) A A C ln C x x x x xx From Eq. (6), t can be seen that the orgnal dstance calculaton method s only a specal case of our proposed method; when the step edge s not oblque, the calculaton result of our method and the orgnal method s the same. In the followng, the nfluence of the slope t on the dstance calculaton result s analyzed. 754 (3) (4) (5) (6)
Frst, t s reasonable to assume that when we estmate varaton of blurrng degree, the number of sample ponts (x, y) on two sdes of the step edge s the same, thus the average dstance between the sample ponts and the step edge s zero. That means x y x cos sn cos.furthermore, L s the Laplace transformaton of the mage, and t does not nfluenced by coordnate system rotaton, thus t can be taken as a constant T. Therefore, Eq. (4) can be smplfed as, A z T ln z z where z=x cosα- y snα- xcosα. Then, ln z T ln z A z When the rotaton angle α ncreases from toπ/, z s nversely proportonal to α. In detal, the denomnator of Eq. (8) decreases wth the speed of the z 4, whle the numerator decreases wth the speed of the z lnǀzǀ. Because the molecules decrease faster than the denomnator n Eq. (8), A s proportonal to α. Accordng to Eq. (11), σ s nversely proportonal to α and 1/t.. Therefore, when the angel ϴ between the step edge and the horzontal axs s equal to 9, the calculaton result of our method and the orgnal method s the same. However, when ϴ decreases, σ also decreases and A s larger than that of the orgnal method whch has not consdered the nfluence of t. That means when the angle of the step edge between the vertcal axs and the edge becomes smaller, the calculated dstance of the orgnal method s larger than that of our method. Therefore, wth decreasng of ϴ, the precson of the orgnal method becomes much lower, whle the precson of our method wll not nfluenced by the slope of the step edge. Smulaton In order to valdate our proposed algorthm of dstance calculaton, frst, a serals of smulaton are conducted. The camera parameters are as follows: the deal object dstance s 85mm; the focal length s 1.mm; the radus of the lens s.6mm; the aperture number s.. The defocused mages conssts of a rectangle box on a flat substrate s synthetzed; the bottom and top surface of the box s flat, and the heght of the box s 8mm; the dstance between the substrate and the camera s 85mm. Snce the dstance between the top surface and the camera s less than the deal object dstance, the syntheszed mage of the top surface s defocused, whle the mage of the substrate s focused. In our smulaton, the dstance between the camera and the top edge of the box s calculated wth the orgnal method and our method. From the smulaton condton, t can be seen that the ground truth s 5mm. The dagram of our smulaton s shown n Fg. 3. In order to test the robustness of our method, we add the dfferent brghtness levels on the rectangle box and the substrate of the syntheszed mage. (7) (8) Fgure 3. The dagram of our smulaton. Fgure 4. Defocused mage of oblque edge wth slope of.8. 755
Frst, the syntheszed defocused mage composed of a box and a substrate s shown n Fg.4, where the slope of the long top edge of the box s.8. Then, the dstance between the top edge and the camera s calculated wth the orgnal method and our method n ths paper. The calculaton result s shown n Fg.5, where the horzonal axs denotes the pxel number on the edge; the vertcal axs s dstance wth unt of m; the lne wth * s the true dstance value; the lne wth o s the dstance calculated wth the orgnal method; the lne wth + s the dstance calculated wth our method n ths paper. Fgure 5. Dstance calculaton result wth slope of.8. Fgure 6. Dstance calculaton result wth slope of 3.. From Fg.5, t can be seen that when the slope of the step edge s.8, whch s far from the vertcal edge, for most of ponts the calculated dstance of our method s closer to the ground truth than that of the orgnal method. In order to prove the precson of our method, the mean-relatve-error (MRE) and the mean-square-devaton (MSD) of calculated dstance are calculated wth the followng equatons, n d d MRE 1 (9) nd where d s the true dstance; d s the dstance of the pxel ; n s the number of pxels along the step edge. MSD n d d 1 n Wth Eq. (9) and Eq. (3), MRE and MSD are calculated based on the result n Fg. 5. MRE of the orgnal method s.54, and MRE of our method s.9; MSD of the orgnal method s.354, MSD of our method s.363. Therefore, MRE and MSD of our method are both lower than those of the orgnal method. That means the precson of our method s hgher than the orgnal method when the defocused mage has an oblque edge. (3) 756
Fgure 7. Dstance calculaton result wth slope of.. Fgure 8. Dstance calculaton result wth slope of 1.. Second, a seral of smlar smulatons are conducted wth respect to dfferent oblque step edges to research the relatonshp between slope and precson of dstance calculaton. In our smulaton, the slope of the oblque edges s 3.,., and 1., respectvely. The dstance of the top edge s calculated wth the orgnal method and our method, and the calculaton result s shown n Fgs.6-8. Based on these fgures, we also calculate the error value of each pont for both the orgnal method and our proposed method. From Fgs.6-8, t can be seen that the precson of the orgnal method s much easy to be nfluenced by the slope of the step edge, whle the precson of our method sn t oblvously nfluenced by t. In order to compare the nfluence of slope on the orgnal method and our method methods n detal, MRE and MSD of the orgnal method and our method are calculated wth respect to dfferent step edges, and the result s shown n Table 1 and Table. Table 1. MRE. slope Method.8 1.. 3. Orgnal method.54.388.97. Our method.9.87.94.99 dfference.45.31.3.13 Table. MSD slope Method.8 1.. 3. Orgnal method.354.63.1819.1435 Our method.363.354.3.344 dfference.1991.179.1519.191 From Tables 1-, the followng concluson can be obtaned, 1) When the slope of the step edge decreases from 3. to.8, MRE of the orgnal method ncreases from. to.54, whle MRE of our method s.99 when the slope s 3. and t does not vary much wth the varaton of slope. ) When the slope of the step edge decreases from 3. to.8, MSD of the orgnal method ncreases from.1435 to.354, whle MSD of our method s.344 when the slope s 3. and t does not vary much wth the varaton of slope. 3) When the step edge s close to be vertcal, the precson of the orgnal method s smlar to that of our method; however when the step edge s further to be vertcal, the precson of the orgnal method s much lower, whle the precson of our method has not changed. 4) The average MRE and MSD of our method are.95 and.34, respectvely. Whle the average MRE and MSD of the orgnal method are.357 and.191775, respectvely 757
Summary In ths paper, dstance calculaton wth a sngle defocused mage contanng an oblque step edge wthout specal camera equpment or unque external condtons s proposed. The frst contrbuton s to ntroduce coordnate system rotaton to fulfll the requrement of orgnal dstance method and elmnate the nfluence of slope; The second contrbuton s enlargng pont-spread-functon from one-dmenson to two-dmenson, takng nto account coordnate transform, and a comprehensve dstance calculaton based on an oblque step edge wth any slope s deduced; The relatonshp between the precson of dstance calculaton and slope of step edges s analyzed wth both MRE and MSD of dstance calculaton. Fnally, a seral of smulaton s conducted to valdate the dstance calculaton method proposed n ths paper, and the expermental results prove the applcablty, valdty and hgh precson of our method. Acknowledgement Ths research was fnancally supported by the Natonal Key Research and Development Plan (16YFC115) and the Fundamental Research Funds for the Central Unverstes (N1616) and State Key Laboratory of Synthetcal Automaton for Process Industres. References [1] Pentland A.P. A New Sense for Depth of Feld [J]. IEEE Transactons on Pattern Analyss and Machne Intellgence, 1987, pp. 53-531. [] Subbarao M. Parallel Depth Recovery By Changng Camera Parameters [C]. Computer Vson, 1988, pp.149-155. [3] Namboodr V.P, Chaudhur S. On defocus, dffuson and depth estmaton [J]. Pattern Recognton Letters, 7, pp. 311-319. [4] We Y.J, Dong Z.L, Wu C.D. Global depth from defocus wth fxed camera parameters [C]. Internatonal Conference on Mechatroncs and Automaton. 9, pp. 1887-189. [5] Nayar S.K, Watanabe M, Noguch M. Real-tme focus range sensor [J]. IEEE Transactons on Pattern Analyss and Machne Intellgence, 1996, Vol. 18(1), pp. 1186-1198. [6] Ln J.Y, Ln X, J X.Y, Da Q.H. Separable Coded Aperture for Depth from a Sngle Image [J]. IEEE Sgnal Processng Letters, 14, Vol. 1(1), pp. 1471-1475. [7] Zhou S., Sm T. n the recovery of depth from a Sngle defocused mage [C]. Proceedngs of the 13th Internatonal conference Computer Analyss of Images and Patterns, 9, Vol. 57(), pp. 889-897. [8] Mng Y., Jang J.J. Depth recovery from a sngle defocused mage usng a Cauchy-dstrbuton-based pont-spread-functon model [J]. Journal of Image and Graphcs, 15, Vol. (5), pp. 78-714. [9] Cao F.Y., Fang S., Hu Y.J., Wang H., Yang J. Recoverng depth from a sngle natural defocused mage [J]. Journal of Image and Graphcs, 14, Vol. 19(5), pp. 746-754. 758